{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":18199,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## INTRODUCTION\n- It’s a Python based scientific computing package targeted at two sets of audiences:\n    - A replacement for NumPy to use the power of GPUs\n    - Deep learning research platform that provides maximum flexibility and speed\n- pros: \n    - Interactively debugging PyTorch. Many users who have used both frameworks would argue that makes pytorch significantly easier to debug and visualize.\n    - Clean support for dynamic graphs\n    - Organizational backing from Facebook\n    - Blend of high level and low level APIs\n- cons:\n    - Much less mature than alternatives\n    - Limited references / resources outside of the official documentation\n- I accept you know neural network basics. If you do not know check my tutorial. Because I will not explain neural network concepts detailed, I only explain how to use pytorch for neural network\n- Neural Network tutorial: https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners \n- The most important parts of this tutorial from matrices to ANN. If you learn these parts very well, implementing remaining parts like CNN or RNN will be very easy. \n<br>\n<br>**Content:**\n1. [Basics of Pytorch](#1)\n    - Matrices\n    - Math\n    - Variable\n1. [Linear Regression](#2)\n1. [Logistic Regression](#3)\n1. [Artificial Neural Network (ANN)](#4)\n1. [Concolutional Neural Network (CNN)](#5)\n1. Recurrent Neural Network (RNN)\n    - https://www.kaggle.com/kanncaa1/recurrent-neural-network-with-pytorch\n1. Long-Short Term Memory (LSTM)\n    - https://www.kaggle.com/kanncaa1/long-short-term-memory-with-pytorch","metadata":{"_uuid":"84c59c4d97b2f0a0eb52a22d74e7939dd51873bf","_cell_guid":"48bb1394-d871-4670-812e-9e80ac87a27a"}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-11-05T23:11:06.256267Z","iopub.execute_input":"2025-11-05T23:11:06.256482Z","iopub.status.idle":"2025-11-05T23:11:06.277033Z","shell.execute_reply.started":"2025-11-05T23:11:06.256434Z","shell.execute_reply":"2025-11-05T23:11:06.276439Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## Basics of Pytorch\n### Matrices\n- In pytorch, matrix(array) is called tensors.\n- 3*3 matrix koy. This is 3x3 tensor.\n- Lets look at array example with numpy that we already know.\n    - We create numpy array with np.numpy() method\n    - Type(): type of the array. In this example it is numpy\n    - np.shape(): shape of the array. Row x Column","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","jupyter":{"outputs_hidden":true}}},{"cell_type":"code","source":"# import numpy library\nimport numpy as np\n\n# numpy array\narray = [[1,2,3],[4,5,6]]\nfirst_array = np.array(array) # 2x3 array\nprint(\"Array Type: {}\".format(type(first_array))) # type\nprint(\"Array Shape: {}\".format(np.shape(first_array))) # shape\nprint(first_array)","metadata":{"_uuid":"d60e9b9f706124117b1d7ffcdeefb45b9aca45e6","_cell_guid":"0a70fde1-b9c4-47c5-aed7-1c863b2fd1e1","trusted":true,"execution":{"iopub.status.busy":"2025-11-05T23:13:35.049516Z","iopub.execute_input":"2025-11-05T23:13:35.049821Z","iopub.status.idle":"2025-11-05T23:13:35.054702Z","shell.execute_reply.started":"2025-11-05T23:13:35.049761Z","shell.execute_reply":"2025-11-05T23:13:35.054009Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- We looked at numpy array.\n- Now examine how we implement tensor(pytorch array)\n- import pytorch library with import torch\n- We create tensor with torch.Tensor() method\n- type: type of the array. In this example it is tensor\n- shape: shape of the array. Row x Column","metadata":{"_uuid":"28a4eb1e95add272552310eacf20740b895e43cd","_cell_guid":"042b8e6f-0be6-43a1-96b4-49b32999a208"}},{"cell_type":"code","source":"# import pytorch library\nimport torch\n\n# pytorch array\ntensor = torch.tensor(array)\nprint(\"Array Type: {}\".format(tensor.type)) # type\nprint(\"Array Shape: {}\".format(tensor.shape)) # shape\nprint(tensor)","metadata":{"_uuid":"126ea635dff4e2a6bc9a828dc560863e3be6aa74","_cell_guid":"b383b085-a18f-4c18-a093-428336b6acf6","trusted":true,"execution":{"iopub.status.busy":"2025-11-05T23:14:20.230354Z","iopub.execute_input":"2025-11-05T23:14:20.230605Z","iopub.status.idle":"2025-11-05T23:14:20.235987Z","shell.execute_reply.started":"2025-11-05T23:14:20.230565Z","shell.execute_reply":"2025-11-05T23:14:20.235234Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Allocation is one of the most used technique in coding. Therefore lets learn how to make it with pytorch.\n- In order to learn, compare numpy and tensor\n    - np.ones() = torch.ones()\n    - np.random.rand() = torch.rand()","metadata":{"_uuid":"fa1375fffab5b879eb828de7261e3e436ec15102","_cell_guid":"e1f7ac29-8aa8-46ff-929f-94f2cedf7541"}},{"cell_type":"code","source":"# numpy ones\nprint(\"Numpy {}\\n\".format(np.ones((2,3))))\n\n# pytorch ones\nprint(torch.ones((2,3)))","metadata":{"_uuid":"2d36d68f3b57eef9d8f0ed94885f3376de92b414","_cell_guid":"741468a5-5d91-48d7-95b0-6d02180d0c09","trusted":true,"execution":{"iopub.status.busy":"2025-11-05T23:14:58.380755Z","iopub.execute_input":"2025-11-05T23:14:58.381021Z","iopub.status.idle":"2025-11-05T23:14:58.386678Z","shell.execute_reply.started":"2025-11-05T23:14:58.380981Z","shell.execute_reply":"2025-11-05T23:14:58.385876Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# numpy random\nprint(\"Numpy {}\\n\".format(np.random.rand(2,3)))\n\n# pytorch random\nprint(torch.rand(2,3))","metadata":{"_uuid":"1e6b8ce52af8a26ffc39fcd751a834ea7c870a2d","_cell_guid":"a578ff9f-df45-4acd-b5ec-2e26b2690adb","trusted":true,"execution":{"iopub.status.busy":"2025-11-05T23:15:11.760046Z","iopub.execute_input":"2025-11-05T23:15:11.760290Z","iopub.status.idle":"2025-11-05T23:15:11.767526Z","shell.execute_reply.started":"2025-11-05T23:15:11.760254Z","shell.execute_reply":"2025-11-05T23:15:11.766717Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Even if when I use pytorch for neural networks, I feel better if I use numpy. Therefore, usually convert result of neural network that is tensor to numpy array to visualize or examine.\n- Lets look at conversion between tensor and numpy arrays.\n    - torch.from_numpy(): from numpy to tensor\n    - numpy(): from tensor to numpy","metadata":{"_uuid":"22b5e44de713f58261bf1ff0b3a52b22a81ef1ef","_cell_guid":"b5177215-45b5-40c1-b838-2d0e3acb48ba"}},{"cell_type":"code","source":"# random numpy array\narray = np.random.rand(2,2)\nprint(\"{} {}\\n\".format(type(array),array))\n\n# from numpy to tensor\nfrom_numpy_to_tensor = torch.from_numpy(array)\nprint(\"{}\\n\".format(from_numpy_to_tensor))\n\n# from tensor to numpy\ntensor = from_numpy_to_tensor\nfrom_tensor_to_numpy = tensor.numpy()\nprint(\"{} {}\\n\".format(type(from_tensor_to_numpy),from_tensor_to_numpy))","metadata":{"_uuid":"c6d3a7b8e0e42fcadecb16264b0563f74d01439a","_cell_guid":"f2cedc86-bd28-4709-906f-e236f4a4dbbe","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Basic Math with Pytorch\n- Resize: view()\n- a and b are tensor.\n- Addition: torch.add(a,b) = a + b\n- Subtraction: a.sub(b) = a - b\n- Element wise multiplication: torch.mul(a,b) = a * b \n- Element wise division: torch.div(a,b) = a / b \n- Mean: a.mean()\n- Standart Deviation (std): a.std()","metadata":{"_uuid":"42cbe3900b733ab12867612d484d6fffaccd5e31","_cell_guid":"6d7038e6-6aaf-4a1e-9204-406ab21082a2"}},{"cell_type":"code","source":"# create tensor \ntensor = torch.ones(3,3)\nprint(\"\\n\",tensor)\n\n# Resize\nprint(\"{}{}\\n\".format(tensor.view(9).shape,tensor.view(9)))\n\n# Addition\nprint(\"Addition: {}\\n\".format(torch.add(tensor,tensor)))\n\n# Subtraction\nprint(\"Subtraction: {}\\n\".format(tensor.sub(tensor)))\n\n# Element wise multiplication\nprint(\"Element wise multiplication: {}\\n\".format(torch.mul(tensor,tensor)))\n\n# Element wise division\nprint(\"Element wise division: {}\\n\".format(torch.div(tensor,tensor)))\n\n# Mean\ntensor = torch.Tensor([1,2,3,4,5])\nprint(\"Mean: {}\".format(tensor.mean()))\n\n# Standart deviation (std)\nprint(\"std: {}\".format(tensor.std()))","metadata":{"_uuid":"66193cb3c790d13b8328c1c1262e1e3c17230bb8","_cell_guid":"e43af8e7-53ab-40bc-a4f8-4cea941c6df0","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Variables\n- It accumulates gradients. \n- We will use pytorch in neural network. And as you know, in neural network we have backpropagation where gradients are calculated. Therefore we need to handle gradients. If you do not know neural network, check my deep learning tutorial first because I will not explain detailed the concepts like optimization, loss function or backpropagation. \n- Deep learning tutorial: https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners\n- Difference between variables and tensor is variable accumulates gradients.\n- We can make math operations with variables, too.\n- In order to make backward propagation we need variables","metadata":{"_uuid":"ff85694eebe8d02701e20d7c15b0ad2974175dd3","_cell_guid":"9fb8b7d4-848a-4d2c-8436-162fd47a0e11"}},{"cell_type":"code","source":"# import variable from pytorch library\nfrom torch.autograd import Variable\n\n# define variable\nvar = Variable(torch.ones(3), requires_grad = True)\nvar","metadata":{"_uuid":"83e3222b53be71e5fc7207da552ce9e9b90486dd","_cell_guid":"fd8ceaa3-f1e2-4761-924e-00a6daca4a82","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Assume we have equation y = x^2\n- Define x = [2,4] variable\n- After calculation we find that y = [4,16] (y = x^2)\n- Recap o equation is that o = (1/2)*sum(y) = (1/2)*sum(x^2)\n- deriavative of o = x\n- Result is equal to x so gradients are [2,4]\n- Lets implement","metadata":{"_uuid":"1cc3de04f98fc14624a18cceb5a84034b1dc29c7","_cell_guid":"f5d54144-0753-4e2a-bac1-ccfff3084f6e"}},{"cell_type":"code","source":"# lets make basic backward propagation\n# we have an equation that is y = x^2\narray = [2,4]\ntensor = torch.Tensor(array)\nx = Variable(tensor, requires_grad = True)\ny = x**2\nprint(\" y =  \",y)\n\n# recap o equation o = 1/2*sum(y)\no = (1/2)*sum(y)\nprint(\" o =  \",o)\n\n# backward\no.backward() # calculates gradients\n\n# As I defined, variables accumulates gradients. In this part there is only one variable x.\n# Therefore variable x should be have gradients\n# Lets look at gradients with x.grad\nprint(\"gradients: \",x.grad)","metadata":{"_uuid":"ff4010e2958a72ce45e43118790f3e20dc2abad6","_cell_guid":"cd73c1cf-d250-48e4-bfb7-ffbe8c03c267","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n### Linear Regression\n- Detailed linear regression tutorial is in my machine learning tutorial in part \"Regression\". I will not explain it in here detailed.\n- Linear Regression tutorial: https://www.kaggle.com/kanncaa1/machine-learning-tutorial-for-beginners\n- y = Ax + B.\n    - A = slope of curve\n    - B = bias (point that intersect y-axis)\n- For example, we have car company. If the car price is low, we sell more car. If the car price is high, we sell less car. This is the fact that we know and we have data set about this fact.\n- The question is that what will be number of car sell if the car price is 100.","metadata":{"_uuid":"6d8fa48e6e641da312175509aae00fea2760cb2c","_cell_guid":"c916b8e5-e078-48de-8bc6-a757022ba65d"}},{"cell_type":"code","source":"# As a car company we collect this data from previous selling\n# lets define car prices\ncar_prices_array = [3,4,5,6,7,8,9]\ncar_price_np = np.array(car_prices_array,dtype=np.float32)\ncar_price_np = car_price_np.reshape(-1,1)\ncar_price_tensor = Variable(torch.from_numpy(car_price_np))\n\n# lets define number of car sell\nnumber_of_car_sell_array = [ 7.5, 7, 6.5, 6.0, 5.5, 5.0, 4.5]\nnumber_of_car_sell_np = np.array(number_of_car_sell_array,dtype=np.float32)\nnumber_of_car_sell_np = number_of_car_sell_np.reshape(-1,1)\nnumber_of_car_sell_tensor = Variable(torch.from_numpy(number_of_car_sell_np))\n\n# lets visualize our data\nimport matplotlib.pyplot as plt\nplt.scatter(car_prices_array,number_of_car_sell_array)\nplt.xlabel(\"Car Price $\")\nplt.ylabel(\"Number of Car Sell\")\nplt.title(\"Car Price$ VS Number of Car Sell\")\nplt.show()","metadata":{"_uuid":"0bed0a61494fab620e639745f0c48b341f665bf8","_cell_guid":"b9a3beb3-9e3c-4502-94c2-fe87ac623ca2","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Now this plot is our collected data\n- We have a question that is what will be number of car sell if the car price is 100$\n- In order to solve this question we need to use linear regression.\n- We need to line fit into this data. Aim is fitting line with minimum error.\n- **Steps of Linear Regression**\n    1. create LinearRegression class\n    1. define model from this LinearRegression class\n    1. MSE: Mean squared error\n    1. Optimization (SGD:stochastic gradient descent)\n    1. Backpropagation\n    1. Prediction\n- Lets implement it with Pytorch","metadata":{"_uuid":"9e7128ffc3fa1b0e545529d5f56d969d45cab78b","_cell_guid":"a6044795-9f61-4d67-a16b-8008903fd482"}},{"cell_type":"code","source":"# Linear Regression with Pytorch\n\n# libraries\nimport torch      \nfrom torch.autograd import Variable     \nimport torch.nn as nn \nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# create class\nclass LinearRegression(nn.Module):\n    def __init__(self,input_size,output_size):\n        # super function. It inherits from nn.Module and we can access everythink in nn.Module\n        super(LinearRegression,self).__init__()\n        # Linear function.\n        self.linear = nn.Linear(input_dim,output_dim)\n\n    def forward(self,x):\n        return self.linear(x)\n    \n# define model\ninput_dim = 1\noutput_dim = 1\nmodel = LinearRegression(input_dim,output_dim) # input and output size are 1\n\n# MSE\nmse = nn.MSELoss()\n\n# Optimization (find parameters that minimize error)\nlearning_rate = 0.02   # how fast we reach best parameters\noptimizer = torch.optim.SGD(model.parameters(),lr = learning_rate)\n\n# train model\nloss_list = []\niteration_number = 1001\nfor iteration in range(iteration_number):\n        \n    # optimization\n    optimizer.zero_grad() \n    \n    # Forward to get output\n    results = model(car_price_tensor)\n    \n    # Calculate Loss\n    loss = mse(results, number_of_car_sell_tensor)\n    \n    # backward propagation\n    loss.backward()\n    \n    # Updating parameters\n    optimizer.step()\n    \n    # store loss\n    loss_list.append(loss.data)\n    \n    # print loss\n    if(iteration % 50 == 0):\n        print('epoch {}, loss {}'.format(iteration, loss.data))\n\nplt.plot(range(iteration_number),loss_list)\nplt.xlabel(\"Number of Iterations\")\nplt.ylabel(\"Loss\")\nplt.show()","metadata":{"_uuid":"8040e01e2bdc25d6fdbff800262f3afe8b9dac3a","_cell_guid":"2b74a84a-29da-44ed-9b5f-649a5c54b8a9","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Number of iteration is 1001.\n- Loss is almost zero that you can see from plot or loss in epoch number 1000.\n- Now we have a trained model.\n- While usign trained model, lets predict car prices.","metadata":{"_uuid":"6f3c764684ae00949ef3f91f5eb75b7bd11c089d","_cell_guid":"b86efed7-1e47-44ad-9223-59c370a42560"}},{"cell_type":"code","source":"# predict our car price \npredicted = model(car_price_tensor).data.numpy()\nplt.scatter(car_prices_array,number_of_car_sell_array,label = \"original data\",color =\"red\")\nplt.scatter(car_prices_array,predicted,label = \"predicted data\",color =\"blue\")\n\n# predict if car price is 10$, what will be the number of car sell\n#predicted_10 = model(torch.from_numpy(np.array([10]))).data.numpy()\n#plt.scatter(10,predicted_10.data,label = \"car price 10$\",color =\"green\")\nplt.legend()\nplt.xlabel(\"Car Price $\")\nplt.ylabel(\"Number of Car Sell\")\nplt.title(\"Original vs Predicted values\")\nplt.show()","metadata":{"_uuid":"ec56085a9c6e91f18042f8a79dede663abf09cf0","_cell_guid":"abecf557-4d1f-4e2a-a466-001abb4d27d2","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n### Logistic Regression\n- Linear regression is not good at classification.\n- We use logistic regression for classification.\n- linear regression + logistic function(softmax) = logistic regression\n- Check my deep learning tutorial. There is detailed explanation of logistic regression. \n    - https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners\n- **Steps of Logistic Regression**\n    1. Import Libraries\n    1. Prepare Dataset\n        - We use MNIST dataset.\n        - There are 28*28 images and 10 labels from 0 to 9\n        - Data is not normalized so we divide each image to 255 that is basic normalization for images.\n        - In order to split data, we use train_test_split method from sklearn library\n        - Size of train data is 80% and size of test data is 20%.\n        - Create feature and target tensors. At the next parts we create variable from these tensors. As you remember we need to define variable for accumulation of gradients.\n        - batch_size = batch size means is that for example we have data and it includes 1000 sample. We can train 1000 sample in a same time or we can divide it 10 groups which include 100 sample and train 10 groups in order. Batch size is the group size. For example, I choose batch_size = 100, that means in order to train all data only once we have 336 groups. We train each groups(336) that have batch_size(quota) 100. Finally we train 33600 sample one time.\n        - epoch: 1 epoch means training all samples one time.\n        - In our example: we have 33600 sample to train and we decide our batch_size is 100. Also we decide epoch is 29(accuracy achieves almost highest value when epoch is 29). Data is trained 29 times. Question is that how many iteration do I need? Lets calculate: \n            - training data 1 times = training 33600 sample (because data includes 33600 sample) \n            - But we split our data 336 groups(group_size = batch_size = 100) our data \n            - Therefore, 1 epoch(training data only once) takes 336 iteration\n            - We have 29 epoch, so total iterarion is 9744(that is almost 10000 which I used)\n        - TensorDataset(): Data set wrapping tensors. Each sample is retrieved by indexing tensors along the first dimension.\n        - DataLoader(): It combines dataset and sample. It also provides multi process iterators over the dataset.\n        - Visualize one of the images in dataset\n    1. Create Logistic Regression Model\n        - Same with linear regression.\n        - However as you expect, there should be logistic function in model right?\n        - In pytorch, logistic function is in the loss function where we will use at next parts.\n    1. Instantiate Model\n        - input_dim = 28*28 # size of image px*px\n        - output_dim = 10  # labels 0,1,2,3,4,5,6,7,8,9\n        - create model\n    1. Instantiate Loss \n        - Cross entropy loss\n        - It calculates loss that is not surprise :)\n        - It also has softmax(logistic function) in it.\n    1. Instantiate Optimizer \n        - SGD Optimizer\n    1. Traning the Model\n    1. Prediction\n- As a result, as you can see from plot, while loss decreasing, accuracy(almost 85%) is increasing and our model is learning(training).    ","metadata":{"_uuid":"20b4762eb8607ed428703c2156c5aefe8b49ff3f","_cell_guid":"49344c72-d0ea-4092-96fe-ed5508ae6e0b"}},{"cell_type":"code","source":"# Import Libraries\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nfrom torch.utils.data import DataLoader\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split","metadata":{"_uuid":"1382c63fe24710d3b2840e7dcf172cddbf533743","_cell_guid":"a0bf0fa7-c527-4fd3-b504-02a88fc94798","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Prepare Dataset\n# load data\ntrain = pd.read_csv(r\"../input/train.csv\",dtype = np.float32)\n\n# split data into features(pixels) and labels(numbers from 0 to 9)\ntargets_numpy = train.label.values\nfeatures_numpy = train.loc[:,train.columns != \"label\"].values/255 # normalization\n\n# train test split. Size of train data is 80% and size of test data is 20%. \nfeatures_train, features_test, targets_train, targets_test = train_test_split(features_numpy,\n                                                                             targets_numpy,\n                                                                             test_size = 0.2,\n                                                                             random_state = 42) \n\n# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable\nfeaturesTrain = torch.from_numpy(features_train)\ntargetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long\n\n# create feature and targets tensor for test set.\nfeaturesTest = torch.from_numpy(features_test)\ntargetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 10000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\ntest = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = DataLoader(test, batch_size = batch_size, shuffle = False)\n\n# visualize one of the images in data set\nplt.imshow(features_numpy[10].reshape(28,28))\nplt.axis(\"off\")\nplt.title(str(targets_numpy[10]))\nplt.savefig('graph.png')\nplt.show()","metadata":{"_uuid":"c6e0d7d3843719091564a580dbe08f67ee0d93ec","_cell_guid":"59cdc9d5-da8f-4d7a-abc5-c62b0008afb0","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create Logistic Regression Model\nclass LogisticRegressionModel(nn.Module):\n    def __init__(self, input_dim, output_dim):\n        super(LogisticRegressionModel, self).__init__()\n        # Linear part\n        self.linear = nn.Linear(input_dim, output_dim)\n        # There should be logistic function right?\n        # However logistic function in pytorch is in loss function\n        # So actually we do not forget to put it, it is only at next parts\n    \n    def forward(self, x):\n        out = self.linear(x)\n        return out\n\n# Instantiate Model Class\ninput_dim = 28*28 # size of image px*px\noutput_dim = 10  # labels 0,1,2,3,4,5,6,7,8,9\n\n# create logistic regression model\nmodel = LogisticRegressionModel(input_dim, output_dim)\n\n# Cross Entropy Loss  \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer \nlearning_rate = 0.001\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)","metadata":{"_uuid":"7c7a7265a23a8101d5ed0c8826dfec3726d6161d","_cell_guid":"03a25584-c567-4b5e-bae1-7bc9e02184fe","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Traning the Model\ncount = 0\nloss_list = []\niteration_list = []\nfor epoch in range(num_epochs):\n    for i, (images, labels) in enumerate(train_loader):\n        \n        # Define variables\n        train = Variable(images.view(-1, 28*28))\n        labels = Variable(labels)\n        \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and cross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculate gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        \n        count += 1\n        \n        # Prediction\n        if count % 50 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            # Predict test dataset\n            for images, labels in test_loader: \n                test = Variable(images.view(-1, 28*28))\n                \n                # Forward propagation\n                outputs = model(test)\n                \n                # Get predictions from the maximum value\n                predicted = torch.max(outputs.data, 1)[1]\n                \n                # Total number of labels\n                total += len(labels)\n                \n                # Total correct predictions\n                correct += (predicted == labels).sum()\n            \n            accuracy = 100 * correct / float(total)\n            \n            # store loss and iteration\n            loss_list.append(loss.data)\n            iteration_list.append(count)\n        if count % 500 == 0:\n            # Print Loss\n            print('Iteration: {}  Loss: {}  Accuracy: {}%'.format(count, loss.data, accuracy))","metadata":{"_uuid":"0cab9c3ec72f73db1b06578fa7a51611141e16da","_cell_guid":"82de08d9-7f3c-4eb9-8a99-9d7a8677799c","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# visualization\nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"Logistic Regression: Loss vs Number of iteration\")\nplt.show()","metadata":{"_uuid":"db87c03e9d263f07eb75f82a914d3e966895a6c1","_cell_guid":"924e9606-e155-4e39-89d3-39f941fd52f8","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n### Artificial Neural Network (ANN)\n- Logistic regression is good at classification but when complexity(non linearity) increases, the accuracy of model decreases.\n- Therefore, we need to increase complexity of model.\n- In order to increase complexity of model, we need to add more non linear functions as hidden layer. \n- I am saying again that if you do not know what is artificial neural network check my deep learning tutorial because I will not explain neural network detailed here, only explain pytorch.\n- Artificial Neural Network tutorial: https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners\n- What we expect from artificial neural network is that when complexity increases, we use more hidden layers and our model can adapt better. As a result accuracy increase.\n- **Steps of ANN:**\n    1. Import Libraries\n        - In order to show you, I import again but we actually imported them at previous parts.\n    1. Prepare Dataset\n        - Totally same with previous part(logistic regression).\n        - We use same dataset so we only need train_loader and test_loader. \n        - We use same batch size, epoch and iteration numbers.\n    1. Create ANN Model\n        - We add 3 hidden layers.\n        - We use ReLU, Tanh and ELU activation functions for diversity.\n    1. Instantiate Model Class\n        - input_dim = 28*28 # size of image px*px\n        - output_dim = 10  # labels 0,1,2,3,4,5,6,7,8,9\n        - Hidden layer dimension is 150. I only choose it as 150 there is no reason. Actually hidden layer dimension is hyperparameter and it should be chosen and tuned. You can try different values for hidden layer dimension and observe the results.\n        - create model\n    1. Instantiate Loss\n        - Cross entropy loss\n        - It also has softmax(logistic function) in it.\n    1. Instantiate Optimizer\n        - SGD Optimizer\n    1. Traning the Model\n    1. Prediction\n- As a result, as you can see from plot, while loss decreasing, accuracy is increasing and our model is learning(training). \n- Thanks to hidden layers model learnt better and accuracy(almost 95%) is better than accuracy of logistic regression model.","metadata":{"_uuid":"ea9eba414f2f0f1e63ef564dc0ee708c753ff51f","_cell_guid":"4d38db05-fad0-468c-9000-20caf5465eca"}},{"cell_type":"code","source":"# Import Libraries\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable","metadata":{"_uuid":"cf25ee4b28129a47bac4c9dc7d932295155b79f7","_cell_guid":"6925f8ed-54b7-4d9a-9801-acd65f213bc9","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create ANN Model\nclass ANNModel(nn.Module):\n    \n    def __init__(self, input_dim, hidden_dim, output_dim):\n        super(ANNModel, self).__init__()\n        \n        # Linear function 1: 784 --> 150\n        self.fc1 = nn.Linear(input_dim, hidden_dim) \n        # Non-linearity 1\n        self.relu1 = nn.ReLU()\n        \n        # Linear function 2: 150 --> 150\n        self.fc2 = nn.Linear(hidden_dim, hidden_dim)\n        # Non-linearity 2\n        self.tanh2 = nn.Tanh()\n        \n        # Linear function 3: 150 --> 150\n        self.fc3 = nn.Linear(hidden_dim, hidden_dim)\n        # Non-linearity 3\n        self.elu3 = nn.ELU()\n        \n        # Linear function 4 (readout): 150 --> 10\n        self.fc4 = nn.Linear(hidden_dim, output_dim)  \n    \n    def forward(self, x):\n        # Linear function 1\n        out = self.fc1(x)\n        # Non-linearity 1\n        out = self.relu1(out)\n        \n        # Linear function 2\n        out = self.fc2(out)\n        # Non-linearity 2\n        out = self.tanh2(out)\n        \n        # Linear function 2\n        out = self.fc3(out)\n        # Non-linearity 2\n        out = self.elu3(out)\n        \n        # Linear function 4 (readout)\n        out = self.fc4(out)\n        return out\n\n# instantiate ANN\ninput_dim = 28*28\nhidden_dim = 150 #hidden layer dim is one of the hyper parameter and it should be chosen and tuned. For now I only say 150 there is no reason.\noutput_dim = 10\n\n# Create ANN\nmodel = ANNModel(input_dim, hidden_dim, output_dim)\n\n# Cross Entropy Loss \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer\nlearning_rate = 0.02\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)","metadata":{"_uuid":"cefd0bb2f23b80f30ca65cbb08859ad81ab12e08","_cell_guid":"3472f1c1-5888-4abe-822c-3a493a5f8be5","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ANN model training\ncount = 0\nloss_list = []\niteration_list = []\naccuracy_list = []\nfor epoch in range(num_epochs):\n    for i, (images, labels) in enumerate(train_loader):\n\n        train = Variable(images.view(-1, 28*28))\n        labels = Variable(labels)\n        \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and ross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculating gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        \n        count += 1\n        \n        if count % 50 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            # Predict test dataset\n            for images, labels in test_loader:\n\n                test = Variable(images.view(-1, 28*28))\n                \n                # Forward propagation\n                outputs = model(test)\n                \n                # Get predictions from the maximum value\n                predicted = torch.max(outputs.data, 1)[1]\n                \n                # Total number of labels\n                total += len(labels)\n\n                # Total correct predictions\n                correct += (predicted == labels).sum()\n            \n            accuracy = 100 * correct / float(total)\n            \n            # store loss and iteration\n            loss_list.append(loss.data)\n            iteration_list.append(count)\n            accuracy_list.append(accuracy)\n        if count % 500 == 0:\n            # Print Loss\n            print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data, accuracy))","metadata":{"_uuid":"c91694f3af94e4e1b76ab01489e186718c70ccd3","_cell_guid":"7550e98b-5011-4d09-88ee-97b0ecbc6f19","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# visualization loss \nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"ANN: Loss vs Number of iteration\")\nplt.show()\n\n# visualization accuracy \nplt.plot(iteration_list,accuracy_list,color = \"red\")\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"ANN: Accuracy vs Number of iteration\")\nplt.show()","metadata":{"_uuid":"c5e2e6da7f1ee801e38358dc28d4c99e32d2b761","_cell_guid":"5579a7d6-7766-4d0f-b9d0-584cb4f28321","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n### Convolutional Neural Network (CNN)\n- CNN is well adapted to classify images.\n- You can learn CNN basics: https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\n- **Steps of CNN:**\n    1. Import Libraries\n    1. Prepare Dataset\n        - Totally same with previous parts.\n        - We use same dataset so we only need train_loader and test_loader. \n    1. Convolutional layer: \n        - Create feature maps with filters(kernels).\n        - Padding: After applying filter, dimensions of original image decreases. However, we want to preserve as much as information about the original image. We can apply padding to increase dimension of feature map after convolutional layer.\n        - We use 2 convolutional layer.\n        - Number of feature map is out_channels = 16\n        - Filter(kernel) size is 5*5\n    1. Pooling layer: \n        - Prepares a condensed feature map from output of convolutional layer(feature map) \n        - 2 pooling layer that we will use max pooling.\n        - Pooling size is 2*2\n    1. Flattening: Flats the features map\n    1. Fully Connected Layer: \n        - Artificial Neural Network that we learnt at previous part.\n        - Or it can be only linear like logistic regression but at the end there is always softmax function.\n        - We will not use activation function in fully connected layer.\n        - You can think that our fully connected layer is logistic regression.\n        - We combine convolutional part and logistic regression to create our CNN model.\n    1. Instantiate Model Class\n        - create model\n    1. Instantiate Loss\n        - Cross entropy loss\n        - It also has softmax(logistic function) in it.\n    1. Instantiate Optimizer\n        - SGD Optimizer\n    1. Traning the Model\n    1. Prediction\n- As a result, as you can see from plot, while loss decreasing, accuracy is increasing and our model is learning(training). \n- Thanks to convolutional layer, model learnt better and accuracy(almost 98%) is better than accuracy of ANN. Actually while tuning hyperparameters, increase in iteration and expanding convolutional neural network can increase accuracy but it takes too much running time that we do not want at kaggle.   \n        ","metadata":{"_uuid":"cd8f261d231acaccd0f0bc8466fc28c1b0c2f567","_cell_guid":"50bbb2e7-15d8-47f5-8f31-25e0d0cb9e29"}},{"cell_type":"code","source":"# Import Libraries\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable","metadata":{"_uuid":"c78fdf4401cb34db0d963df0249bf92144bc5fbd","_cell_guid":"abbc86d1-f677-4d4b-9b7b-09c7f9962288","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create CNN Model\nclass CNNModel(nn.Module):\n    def __init__(self):\n        super(CNNModel, self).__init__()\n        \n        # Convolution 1\n        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0)\n        self.relu1 = nn.ReLU()\n        \n        # Max pool 1\n        self.maxpool1 = nn.MaxPool2d(kernel_size=2)\n     \n        # Convolution 2\n        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)\n        self.relu2 = nn.ReLU()\n        \n        # Max pool 2\n        self.maxpool2 = nn.MaxPool2d(kernel_size=2)\n        \n        # Fully connected 1\n        self.fc1 = nn.Linear(32 * 4 * 4, 10) \n    \n    def forward(self, x):\n        # Convolution 1\n        out = self.cnn1(x)\n        out = self.relu1(out)\n        \n        # Max pool 1\n        out = self.maxpool1(out)\n        \n        # Convolution 2 \n        out = self.cnn2(out)\n        out = self.relu2(out)\n        \n        # Max pool 2 \n        out = self.maxpool2(out)\n        \n        # flatten\n        out = out.view(out.size(0), -1)\n\n        # Linear function (readout)\n        out = self.fc1(out)\n        \n        return out\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 2500\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\ntest = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)\n    \n# Create CNN\nmodel = CNNModel()\n\n# Cross Entropy Loss \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer\nlearning_rate = 0.1\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n","metadata":{"_uuid":"4915535771ffdd33ef480200393216f215b4fc48","_cell_guid":"9ca5af9e-6821-4d60-8084-edb523a39c6b","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# CNN model training\ncount = 0\nloss_list = []\niteration_list = []\naccuracy_list = []\nfor epoch in range(num_epochs):\n    for i, (images, labels) in enumerate(train_loader):\n        \n        train = Variable(images.view(100,1,28,28))\n        labels = Variable(labels)\n        \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and ross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculating gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        \n        count += 1\n        \n        if count % 50 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            # Iterate through test dataset\n            for images, labels in test_loader:\n                \n                test = Variable(images.view(100,1,28,28))\n                \n                # Forward propagation\n                outputs = model(test)\n                \n                # Get predictions from the maximum value\n                predicted = torch.max(outputs.data, 1)[1]\n                \n                # Total number of labels\n                total += len(labels)\n                \n                correct += (predicted == labels).sum()\n            \n            accuracy = 100 * correct / float(total)\n            \n            # store loss and iteration\n            loss_list.append(loss.data)\n            iteration_list.append(count)\n            accuracy_list.append(accuracy)\n        if count % 500 == 0:\n            # Print Loss\n            print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data, accuracy))","metadata":{"_uuid":"f44e02d25698ac1a014795d972a384a3f3003d35","_cell_guid":"99a8903c-da15-496c-96b7-f5402c8fc5f0","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# visualization loss \nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"CNN: Loss vs Number of iteration\")\nplt.show()\n\n# visualization accuracy \nplt.plot(iteration_list,accuracy_list,color = \"red\")\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"CNN: Accuracy vs Number of iteration\")\nplt.show()","metadata":{"_uuid":"44c1ed412d778f3e6b08f11bddc5321f63e408dd","_cell_guid":"ac9e4aee-b8af-4641-8794-bad03b650179","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Conclusion\nIn this tutorial, we learn: \n1. Basics of pytorch\n1. Linear regression with pytorch\n1. Logistic regression with pytorch\n1. Artificial neural network with with pytorch\n1. Convolutional neural network with pytorch\n1. Recurrent neural network with pytorch\n    - https://www.kaggle.com/kanncaa1/recurrent-neural-network-with-pytorch\n1. Long-Short Term Memory (LSTM)\n    - https://www.kaggle.com/kanncaa1/long-short-term-memory-with-pytorch\n\n<br> **If you have any question or suggest, I will be happy to hear it **","metadata":{"_uuid":"6ce8edd6d5fbf0019d6ab189f8ed1fd9cde0ac85","_cell_guid":"a508a4c8-0719-4a8f-847c-bfce1ceef30f"}}]}