{"metadata":{"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

đź“šRecurrent Neural Networks for Digit Recognitionđź“š

\n

In this notebook, I have explained about Recurrent Neural Networks (RNN), including implementing them on Digit Recognizer dataset followed by evaluating their performance.

\n
\n
\n đź“Ś   In case this notebook is helpful to you in anyway, please do consider leaving feedback for further improvement :)
\n
\n
","metadata":{"_cell_guid":"16db9e81-fe38-44f0-8a4f-3dba9eba0409","_uuid":"d0469858bd9b00ece464f6267cfad7dc450d5d54"}},{"cell_type":"markdown","source":"

\n
\n WHAT IS A RNN?\n
\n

\n
\n \n
\n\nA **Recurrent Neural Network (RNN)** is a class of artificial neural network where connections between nodes form a directed graph along a temporal sequence. This allows it to exhibit temporal dynamic behavior. Unlike feedforward neural networks, RNNs can use their internal state (memory) to process sequences of inputs. \n\n
\n
\n đź“Ś  They are called “recurrent” because they perform the same task for every element of a sequence, with the output being depended on the previous computations.
\n

\nRecurrent Neural Networks (RNNs) are widely used in various fields due to their ability to process sequential data, such as-\n\n1. **Natural Language Processing (NLP)**: RNNs are used in tasks like language modeling, machine translation, and sentiment analysis where the sequence of the data (words in a sentence) is important.\n\n2. **Speech Recognition**: RNNs can be used to transcribe spoken language into written text, as they can handle varying lengths of input sequences and can model temporal dependencies in the speech signal.\n\n3. **Time Series Prediction**: RNNs can model the temporal dynamics of complex time series data, making them useful in predicting stock prices, weather forecasting, and more.\n\n
","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"}},{"cell_type":"markdown","source":"

\n
\n IMPORTING LIBRARIES\n
\n

\n","metadata":{}},{"cell_type":"code","source":"\nimport numpy as np \nimport pandas as pd \nimport os\nimport matplotlib.pyplot as plt\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\n\nfrom sklearn.model_selection import train_test_split\nfrom IPython.display import display, HTML\n\nfrom torch.utils.data import DataLoader, TensorDataset\n\n# To ignore warinings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2024-05-06T20:09:57.642656Z","iopub.execute_input":"2024-05-06T20:09:57.643086Z","iopub.status.idle":"2024-05-06T20:09:57.651602Z","shell.execute_reply.started":"2024-05-06T20:09:57.643045Z","shell.execute_reply":"2024-05-06T20:09:57.650495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n
\nLOADING THE DATASET
\n

\n","metadata":{}},{"cell_type":"code","source":"# Load data\ndata = pd.read_csv(r\"/kaggle/input/digit-recognizer/train.csv\", dtype=np.float32)\n\n# Split data into features(pixels) and labels(numbers from 0 to 9)\ny = data.label.values\nX = data.loc[:, data.columns != \"label\"].values / 255 # normalization\n\n# Train test split. Size of train data is 80% and size of test data is 20%. \nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Convert to PyTorch tensors\nX_train_tensor = torch.from_numpy(X_train)\ny_train_tensor = torch.from_numpy(y_train).type(torch.LongTensor) # data type is long\n\nX_test_tensor = torch.from_numpy(X_test)\ny_test_tensor = torch.from_numpy(y_test).type(torch.LongTensor) # data type is long\n\n# Define batch size\nbatch_size = 100\n\n# Create PyTorch datasets\ntrain_data = TensorDataset(X_train_tensor, y_train_tensor)\ntest_data = TensorDataset(X_test_tensor, y_test_tensor)\n\n# Create data loaders\ntrain_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True)\ntest_loader = DataLoader(test_data, batch_size=1, shuffle=False)\n","metadata":{"_cell_guid":"55dd8ffd-6011-49a3-a1fe-c6933c4187b7","_uuid":"840f7b1c60d1a2d5b2222a7c53b2b9d08aac9169","execution":{"iopub.status.busy":"2024-05-06T20:10:00.264315Z","iopub.execute_input":"2024-05-06T20:10:00.264646Z","iopub.status.idle":"2024-05-06T20:10:03.974853Z","shell.execute_reply.started":"2024-05-06T20:10:00.264618Z","shell.execute_reply":"2024-05-06T20:10:03.973841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n
\nVISUALIZING DATASET
\n

\n","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(3, 8, figsize=(16, 6)) \ntorch.manual_seed(81)\nvisualize_loader = DataLoader(train_data, batch_size=1, shuffle=True)\n\nfor i, (data, label) in enumerate(visualize_loader):\n data = data.reshape((-1, 28, 28))[:,:,:,None]\n ax[i//8][i%8].imshow(data.cpu().squeeze())\n ax[i//8][i%8].axis('off')\n ax[i//8][i%8].set_title(f\"Label: {label.item()}\", y=-0.3)\n if(i==23): \n break\n\n\n\nplt.subplots_adjust(hspace=0.5)\nplt.savefig('plot2.png')\nplt.close()\n\n# Display the saved plot file in HTML\ndisplay(HTML(\"
\"))\n","metadata":{"execution":{"iopub.status.busy":"2024-05-06T20:10:03.976552Z","iopub.execute_input":"2024-05-06T20:10:03.976868Z","iopub.status.idle":"2024-05-06T20:10:04.609547Z","shell.execute_reply.started":"2024-05-06T20:10:03.976838Z","shell.execute_reply":"2024-05-06T20:10:04.608625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n
\nCREATING RECURRENT NEURAL NETWORK
\n

\n","metadata":{}},{"cell_type":"markdown","source":"We set up a basic Recurrent Neural Network (RNN) using PyTorch, initially defining some hyperparameters, and then finally creating our RNN model. \n\nThe model has an RNN layer and a fully connected (FC) layer. The RNN layer takes in our input and passes it through the hidden layers (with ReLU activation), and the FC layer maps the output of the RNN layer to our desired output dimension. We then initialize our model, define a loss function (CrossEntropyLoss in this case), and set up an optimizer (Stochastic Gradient Descent) for training.","metadata":{}},{"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n\n# Hyperparameters\ninput_dim = 28\nhidden_dim = 100\nlayer_dim = 1\noutput_dim = 10\nlearning_rate = 0.05\n\n# RNN Model\nclass RNNModel(nn.Module):\n def __init__(self, device):\n super(RNNModel, self).__init__()\n self.device = device\n self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity='relu')\n self.fc = nn.Linear(hidden_dim, output_dim)\n \n def forward(self, x):\n h0 = torch.zeros(layer_dim, x.size(0), hidden_dim).to(self.device) # Move h0 to the device\n out, _ = self.rnn(x, h0)\n out = self.fc(out[:, -1, :]) \n return out\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n# Initialize model, loss function, and optimizer\nmodel = RNNModel(device).to(device) # Pass the device to the model\ncriterion = nn.CrossEntropyLoss().to(device) # Move the loss function to the device\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)","metadata":{"execution":{"iopub.status.busy":"2024-05-06T19:49:59.568541Z","iopub.execute_input":"2024-05-06T19:49:59.568877Z","iopub.status.idle":"2024-05-06T19:50:02.011964Z","shell.execute_reply.started":"2024-05-06T19:49:59.568850Z","shell.execute_reply":"2024-05-06T19:50:02.011147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n
\nTRAINING THE RECURRENT NEURAL NETWORK
\n

\nWe define a training loop, training the model for num_epochs number of of epochs, with each epoch representing a complete pass over the training dataset. For each batch of images and labels in the training data, we reshape the images and feed them into our model to get the output predictions.\n

\nWe then calculate the loss, which is the difference between the predictions and the actual labels, and update our model’s parameters to minimize this loss. Every 250 iterations, we evaluate our model’s performance on a test dataset, and record the loss, iteration count, and accuracy. If the iteration count is a multiple of 500, we print out the current iteration, loss, and accuracy","metadata":{}},{"cell_type":"code","source":"seq_dim = 28 \nnum_epochs=30\nmetrics = {'loss': [], 'iteration': [], 'accuracy': []}\ncount = 0\n\n\nfor epoch in range(num_epochs):\n \n for i, (images, labels) in enumerate(train_loader):\n train = Variable(images.view(-1, seq_dim, input_dim)).to(device)\n labels = Variable(labels).to(device)\n images = images.to(device)\n\n optimizer.zero_grad()\n outputs = model(train)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n\n count += 1\n if count % 250 == 0:\n correct, total = 0, 0\n for images, labels in test_loader:\n images = Variable(images.view(-1, seq_dim, input_dim)).to(device)\n labels = labels.to(device)\n outputs = model(images).to(device)\n \n predicted = torch.max(outputs.data, 1)[1]\n total += labels.size(0)\n correct += (predicted == labels).sum()\n\n accuracy = 100 * correct / float(total)\n metrics['loss'].append(loss.item())\n metrics['iteration'].append(count)\n metrics['accuracy'].append(accuracy.item())\n\n if count % 500 == 0:\n print(f'Iteration: {count} Loss: {loss} Accuracy: {accuracy} %')","metadata":{"execution":{"iopub.status.busy":"2024-05-06T19:53:24.041033Z","iopub.execute_input":"2024-05-06T19:53:24.041695Z","iopub.status.idle":"2024-05-06T19:57:10.529261Z","shell.execute_reply.started":"2024-05-06T19:53:24.041665Z","shell.execute_reply":"2024-05-06T19:57:10.528164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n
\nVISUALIZING RNN TRAINING
\n

\n","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Use seaborn style defaults and set the default figure size\nsns.set(rc={'figure.figsize':(16, 6)})\n\nfig, axs = plt.subplots(ncols=2)\n\n# Loss plot\nsns.lineplot(x=metrics['iteration'], y=metrics['loss'], ax=axs[0], linewidth=2.5)\naxs[0].set_title('RNN: Loss vs Number of iteration', fontsize=14, fontweight='bold')\naxs[0].set_xlabel('Number of iteration', fontsize=12)\naxs[0].set_ylabel('Loss', fontsize=12)\naxs[0].grid(True)\n\n# Accuracy plot\nsns.lineplot(x=metrics['iteration'], y=metrics['accuracy'], ax=axs[1], color=\"red\", linewidth=2.5)\naxs[1].set_title('RNN: Accuracy vs Number of iteration', fontsize=14, fontweight='bold')\naxs[1].set_xlabel('Number of iteration', fontsize=12)\naxs[1].set_ylabel('Accuracy', fontsize=12)\naxs[1].grid(True)\n\nplt.tight_layout()\nplt.savefig('graph.png')\nplt.close()\n#plt.show()\ndisplay(HTML(\"
\"))\n","metadata":{"execution":{"iopub.status.busy":"2024-05-05T19:26:56.602098Z","iopub.execute_input":"2024-05-05T19:26:56.602541Z","iopub.status.idle":"2024-05-05T19:26:57.391491Z","shell.execute_reply.started":"2024-05-05T19:26:56.602508Z","shell.execute_reply":"2024-05-05T19:26:57.390428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we will visualize the results on the testing dataset","metadata":{}},{"cell_type":"code","source":"sns.reset_orig()\n\nfig, axes = plt.subplots(3, 8, figsize=(20, 8))\nfig.subplots_adjust(hspace=0.5, wspace=0.7)\n\n# Create an iterator for the test_loader\ndata_iter = iter(test_loader)\n\nfor i in range(24):\n # Get the image and labels\n images, labels = next(data_iter)\n \n # Move images and labels to the device\n images = images.to(device)\n labels = labels.to(device)\n\n # Reshape the image for visualization\n image_visualization = images.view(-1, 28, 28).cpu().numpy()\n\n # Display the image\n axes[i//8][i%8].imshow(image_visualization[0])\n \n outputs = model(images.view(-1, seq_dim, input_dim))\n pred_label = torch.max(outputs.data, 1)[1].item()\n\n # Get the true label\n true_label = labels[0].item()\n\n # Set the title with the predicted and actual labels\n title = f\"Predicted: {pred_label}\\nActual: {true_label}\"\n color = 'green' if pred_label == true_label else 'red'\n axes[i//8][i%8].set_title(title, color=color, y=-0.5)\n\n # Hide the axes\n axes[i//8][i%8].axis('off')\n\nplt.savefig('plot.png')\nplt.close()\n\n# Display the saved plot file in HTML\ndisplay(HTML(\"
\"))\n","metadata":{"execution":{"iopub.status.busy":"2024-05-05T19:52:58.491867Z","iopub.execute_input":"2024-05-05T19:52:58.492286Z","iopub.status.idle":"2024-05-05T19:52:59.395182Z","shell.execute_reply.started":"2024-05-05T19:52:58.492254Z","shell.execute_reply":"2024-05-05T19:52:59.393960Z"},"trusted":true},"execution_count":null,"outputs":[]}]}