{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## INTRODUCTION\n- It’s a Python based scientific computing package targeted at two sets of audiences:\n    - A replacement for NumPy to use the power of GPUs\n    - Deep learning research platform that provides maximum flexibility and speed\n- pros: \n    - Iinteractively debugging PyTorch. Many users who have used both frameworks would argue that makes pytorch significantly easier to debug and visualize.\n    - Clean support for dynamic graphs\n    - Organizational backing from Facebook\n    - Blend of high level and low level APIs\n- cons:\n    - Much less mature than alternatives\n    - Limited references / resources outside of the official documentation\n- I accept you know neural network basics. If you do not know check my tutorial. Because I will not explain neural network concepts detailed, I only explain how to use pytorch for neural network\n- Neural Network tutorial: https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners \n- The most important parts of this tutorial from matrices to ANN. If you learn these parts very well, implementing remaining parts like CNN or RNN will be very easy. \n<br>\n<br>**Content:**\n1. Basics of Pytorch, Linear Regression, Logistic Regression, Artificial Neural Network (ANN), Concolutional Neural Network (CNN)\n    - https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers/code\n1. [Recurrent Neural Network (RNN)](#1)\n1. Long-Short Term Memory (LSTM)\n    - https://www.kaggle.com/kanncaa1/long-short-term-memory-with-pytorch","metadata":{"_cell_guid":"16db9e81-fe38-44f0-8a4f-3dba9eba0409","_uuid":"d0469858bd9b00ece464f6267cfad7dc450d5d54"}},{"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n### Recurrent Neural Network (RNN)\n- RNN is essentially repeating ANN but information get pass through from previous non-linear activation function output.\n- **Steps of RNN:**\n    1. Import Libraries\n    1. Prepare Dataset\n    1. Create RNN Model\n        - hidden layer dimension is 100\n        - number of hidden layer is 1 \n    1. Instantiate Model\n    1. Instantiate Loss\n        - Cross entropy loss\n        - It also has softmax(logistic function) in it.\n    1. Instantiate Optimizer\n        - SGD Optimizer\n    1. Traning the Model\n    1. Prediction","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","jupyter":{"outputs_hidden":true}}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"execution":{"iopub.status.busy":"2026-09-16T17:31:36.059901Z","iopub.execute_input":"2026-09-16T17:31:36.060185Z","iopub.status.idle":"2026-09-16T17:31:36.073050Z","shell.execute_reply.started":"2026-09-16T17:31:36.060123Z","shell.execute_reply":"2026-09-16T17:31:36.072255Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Import Libraries\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nfrom sklearn.model_selection import train_test_split\nfrom torch.utils.data import DataLoader, TensorDataset","metadata":{"_cell_guid":"482946c2-72d8-4489-a094-d6cb8993a912","_uuid":"ceffbb7fe5381f0d2f5f234ea37d1f834843edee","trusted":true,"execution":{"iopub.status.busy":"2026-09-16T17:31:36.074000Z","iopub.execute_input":"2026-09-16T17:31:36.074284Z","iopub.status.idle":"2026-09-16T17:31:36.901381Z","shell.execute_reply.started":"2026-09-16T17:31:36.074230Z","shell.execute_reply":"2026-09-16T17:31:36.900869Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Prepare Dataset\n# load data\ntrain = pd.read_csv(r\"../input/train.csv\",dtype = np.float32)\n\n# split data into features(pixels) and labels(numbers from 0 to 9)\ntargets_numpy = train.label.values\nfeatures_numpy = train.loc[:,train.columns != \"label\"].values/255 # normalization\n\n# train test split. Size of train data is 80% and size of test data is 20%. \nfeatures_train, features_test, targets_train, targets_test = train_test_split(features_numpy,\n                                                                             targets_numpy,\n                                                                             test_size = 0.2,\n                                                                             random_state = 42) \n\n# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable\nfeaturesTrain = torch.from_numpy(features_train)\ntargetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long\n\n# create feature and targets tensor for test set.\nfeaturesTest = torch.from_numpy(features_test)\ntargetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 10000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = TensorDataset(featuresTrain,targetsTrain)\ntest = TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = DataLoader(test, batch_size = batch_size, shuffle = False)\n\n# visualize one of the images in data set\nplt.imshow(features_numpy[10].reshape(28,28))\nplt.axis(\"off\")\nplt.title(str(targets_numpy[10]))\nplt.savefig('graph.png')\nplt.show()","metadata":{"_cell_guid":"55dd8ffd-6011-49a3-a1fe-c6933c4187b7","_uuid":"840f7b1c60d1a2d5b2222a7c53b2b9d08aac9169","trusted":true,"execution":{"iopub.status.busy":"2026-09-16T17:31:36.902279Z","iopub.execute_input":"2026-09-16T17:31:36.902541Z","iopub.status.idle":"2026-09-16T17:31:41.290694Z","shell.execute_reply.started":"2026-09-16T17:31:36.902431Z","shell.execute_reply":"2026-09-16T17:31:41.289932Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create RNN Model\nclass RNNModel(nn.Module):\n    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):\n        super(RNNModel, self).__init__()\n        \n        # Number of hidden dimensions\n        self.hidden_dim = hidden_dim\n        \n        # Number of hidden layers\n        self.layer_dim = layer_dim\n        \n        # RNN\n        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity='relu')\n        \n        # Readout layer\n        self.fc = nn.Linear(hidden_dim, output_dim)\n    \n    def forward(self, x):\n        \n        # Initialize hidden state with zeros\n        h0 = Variable(torch.zeros(self.layer_dim, x.size(0), self.hidden_dim))\n            \n        # One time step\n        out, hn = self.rnn(x, h0)\n        out = self.fc(out[:, -1, :]) \n        return out\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 8000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = TensorDataset(featuresTrain,targetsTrain)\ntest = TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = DataLoader(test, batch_size = batch_size, shuffle = False)\n    \n# Create RNN\ninput_dim = 28    # input dimension\nhidden_dim = 100  # hidden layer dimension\nlayer_dim = 1     # number of hidden layers\noutput_dim = 10   # output dimension\n\nmodel = RNNModel(input_dim, hidden_dim, layer_dim, output_dim)\n\n# Cross Entropy Loss \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer\nlearning_rate = 0.05\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)","metadata":{"_cell_guid":"7fbe419e-7ce2-4d72-bb31-8b27e8161f1b","_uuid":"bb1b6d4fb5504400ed7678d8e95d0a4478b5f409","trusted":true,"execution":{"iopub.status.busy":"2026-09-16T17:31:41.291879Z","iopub.execute_input":"2026-09-16T17:31:41.292170Z","iopub.status.idle":"2026-09-16T17:31:41.367344Z","shell.execute_reply.started":"2026-09-16T17:31:41.292111Z","shell.execute_reply":"2026-09-16T17:31:41.366561Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"seq_dim = 28  \nloss_list = []\niteration_list = []\naccuracy_list = []\ncount = 0\nfor epoch in range(num_epochs):\n    for i, (images, labels) in enumerate(train_loader):\n\n        train  = Variable(images.view(-1, seq_dim, input_dim))\n        labels = Variable(labels )\n            \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and ross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculating gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        \n        count += 1\n        \n        if count % 250 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            # Iterate through test dataset\n            for images, labels in test_loader:\n                images = Variable(images.view(-1, seq_dim, input_dim))\n                \n                # Forward propagation\n                outputs = model(images)\n                \n                # Get predictions from the maximum value\n                predicted = torch.max(outputs.data, 1)[1]\n                \n                # Total number of labels\n                total += labels.size(0)\n                \n                correct += (predicted == labels).sum()\n            \n            accuracy = 100 * correct / float(total)\n            \n            # store loss and iteration\n            loss_list.append(loss.data)\n            iteration_list.append(count)\n            accuracy_list.append(accuracy)\n            if count % 500 == 0:\n                # Print Loss\n                print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data[0], accuracy))","metadata":{"_cell_guid":"32786a5c-0388-412d-b6da-ee5ace604eda","_uuid":"9c935ac4a1d1964b85513da422ebf60085dca0e3","trusted":true,"execution":{"iopub.status.busy":"2026-09-16T17:31:41.368486Z","iopub.execute_input":"2026-09-16T17:31:41.368798Z","iopub.status.idle":"2026-09-16T17:33:43.677461Z","shell.execute_reply.started":"2026-09-16T17:31:41.368741Z","shell.execute_reply":"2026-09-16T17:33:43.676633Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# visualization loss \nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"RNN: Loss vs Number of iteration\")\nplt.show()\n\n# visualization accuracy \nplt.plot(iteration_list,accuracy_list,color = \"red\")\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"RNN: Accuracy vs Number of iteration\")\nplt.savefig('graph.png')\nplt.show()","metadata":{"_cell_guid":"0e527a85-b600-4e40-a0ef-850537db2ab1","_uuid":"0cb7130ea6e22093d6d5cb1284822b0b76b8d66c","trusted":true,"execution":{"iopub.status.busy":"2026-09-16T17:33:43.678569Z","iopub.execute_input":"2026-09-16T17:33:43.678901Z","iopub.status.idle":"2026-09-16T17:33:43.957302Z","shell.execute_reply.started":"2026-09-16T17:33:43.678836Z","shell.execute_reply":"2026-09-16T17:33:43.956547Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Conclusion\nIn this tutorial, we learn: \n1. Basics of pytorch\n1. Linear regression with pytorch\n1. Logistic regression with pytorch\n1. Artificial neural network with with pytorch\n1. Convolutional neural network with pytorch\n    - https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers/code\n1. Recurrent neural network with pytorch\n1. Long-Short Term Memory (LSTM)\n    - https://www.kaggle.com/kanncaa1/long-short-term-memory-with-pytorch\n\n<br> If you have any question or suggest, I will be happy to hear it ","metadata":{"_cell_guid":"eeaf09ff-e125-42ee-99fa-e231e97c4308","_uuid":"0771ccf728b05cf6a5e3804e7d9bc5fa376e7ef8"}}]}