{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n### References\n- CNN is well adapted to classify images.\n- You can learn CNN basics: \n     1. https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners\n     1. https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\n     1. https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers\n     1. https://www.kaggle.com/franciscoramirez/pytorch-tutorial-for-deep-learning-lovers  (based on[3])\n     1. https://www.kaggle.com/franciscoramirez/cnn-for-digit-recognition (based on [2, 3])\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n### Convolutional Neural Network (CNN)\n- CNN is well adapted to classify images.\n- You can learn CNN basics: https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\n- **Steps of CNN:**\n    1. Import Libraries\n    1. Prepare Dataset\n        - Totally same with previous parts.\n        - We use same dataset so we only need train_loader and test_loader. \n    1. Convolutional layer: \n        - Create feature maps with filters(kernels).\n        - Padding: After applying filter, dimensions of original image decreases. However, we want to preserve as much as information about the original image. We can apply padding to increase dimension of feature map after convolutional layer.\n        - We use 2 convolutional layer.\n        - Number of feature map is out_channels = 16\n        - Filter(kernel) size is 5*5\n    1. Pooling layer: \n        - Prepares a condensed feature map from output of convolutional layer(feature map) \n        - 2 pooling layer that we will use max pooling.\n        - Pooling size is 2*2\n    1. Flattening: Flats the features map\n    1. Fully Connected Layer: \n        - Artificial Neural Network that we learnt at previous part.\n        - Or it can be only linear like logistic regression but at the end there is always softmax function.\n        - We will not use activation function in fully connected layer.\n        - You can think that our fully connected layer is logistic regression.\n        - We combine convolutional part and logistic regression to create our CNN model.\n    1. Instantiate Model Class\n        - create model\n    1. Instantiate Loss\n        - Cross entropy loss\n        - It also has softmax(logistic function) in it.\n    1. Instantiate Optimizer\n        - SGD Optimizer\n    1. Traning the Model\n    1. Prediction\n- As a result, as you can see from plot, while loss decreasing, accuracy is increasing and our model is learning(training). \n- Thanks to convolutional layer, model learnt better and accuracy(almost 98%) is better than accuracy of ANN. Actually while tuning hyperparameters, increase in iteration and expanding convolutional neural network can increase accuracy but it takes too much running time that we do not want at kaggle.   \n ","metadata":{}},{"cell_type":"code","source":"# Import Libraries\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\n\nimport os\nprint(os.listdir(\"../input/digit-recognizer\"))\n","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:01:23.447541Z","iopub.execute_input":"2023-06-25T12:01:23.448767Z","iopub.status.idle":"2023-06-25T12:01:29.153960Z","shell.execute_reply.started":"2023-06-25T12:01:23.448727Z","shell.execute_reply":"2023-06-25T12:01:29.149430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare Dataset\n# load data\ntrain = pd.read_csv(r\"../input/digit-recognizer/train.csv\",dtype = np.float32)\n\n# split data into features(pixels) and labels(numbers from 0 to 9)\ntargets_numpy = train.label.values\nfeatures_numpy = train.loc[:,train.columns != \"label\"].values/255 # normalization\n\n# train test split. Size of train data is 80% and size of test data is 20%. \nfeatures_train, features_test, targets_train, targets_test = train_test_split(features_numpy,\n                                                                             targets_numpy,\n                                                                             test_size = 0.2,\n                                                                             random_state = 42) \n\n# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable\nfeaturesTrain = torch.from_numpy(features_train)\ntargetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long\n\n# create feature and targets tensor for test set.\nfeaturesTest = torch.from_numpy(features_test)\ntargetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long\n\nprint(len(features_train))\nprint(len(features_test))","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:01:29.156728Z","iopub.execute_input":"2023-06-25T12:01:29.163523Z","iopub.status.idle":"2023-06-25T12:01:34.790711Z","shell.execute_reply.started":"2023-06-25T12:01:29.163470Z","shell.execute_reply":"2023-06-25T12:01:34.789552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create CNN Model\nclass CNNModel(nn.Module):\n    def __init__(self):\n        super(CNNModel, self).__init__()\n        \n        # Convolution 1\n        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0)\n        self.relu1 = nn.ReLU()\n        \n        # Max pool 1\n        self.maxpool1 = nn.MaxPool2d(kernel_size=2)\n     \n        # Convolution 2\n        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)\n        self.relu2 = nn.ReLU()\n        \n        # Max pool 2\n        self.maxpool2 = nn.MaxPool2d(kernel_size=2)\n        \n        # Fully connected 1\n        self.fc1 = nn.Linear(32 * 4 * 4, 10) \n    \n    def forward(self, x):\n        # Convolution 1\n        out = self.cnn1(x)\n        out = self.relu1(out)\n        \n        # Max pool 1\n        out = self.maxpool1(out)\n        \n        # Convolution 2 \n        out = self.cnn2(out)\n        out = self.relu2(out)\n        \n        # Max pool 2 \n        out = self.maxpool2(out)\n        \n        # flatten\n        out = out.view(out.size(0), -1)\n\n        # Linear function (readout)\n        out = self.fc1(out)\n        \n        return out\n\n","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:01:34.792397Z","iopub.execute_input":"2023-06-25T12:01:34.792747Z","iopub.status.idle":"2023-06-25T12:01:34.805113Z","shell.execute_reply.started":"2023-06-25T12:01:34.792717Z","shell.execute_reply":"2023-06-25T12:01:34.803273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 2500\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\ntest = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)\n    \n# Create CNN\nmodel = CNNModel()\nif torch.cuda.is_available():\n    print(\"cuda\")\n    model = model.cuda()\nelse:\n    print(\"cpu\")\nprint(model)\n\n# Cross Entropy Loss \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer\nlearning_rate = 0.1\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:01:34.808169Z","iopub.execute_input":"2023-06-25T12:01:34.808542Z","iopub.status.idle":"2023-06-25T12:01:34.837666Z","shell.execute_reply.started":"2023-06-25T12:01:34.808511Z","shell.execute_reply":"2023-06-25T12:01:34.836187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CNN model training\ncount = 0\nloss_list = []\niteration_list = []\naccuracy_list = []\nfor epoch in range(num_epochs):\n    for i, (images, labels) in enumerate(train_loader):\n        \n        train = Variable(images.view(100,1,28,28))\n        labels = Variable(labels)\n        \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and ross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculating gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        \n        count += 1\n        \n        if count % 50 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            # Iterate through test dataset\n            for images, labels in test_loader:\n                \n                test = Variable(images.view(100,1,28,28))\n                \n                # Forward propagation\n                outputs = model(test)\n                \n                # Get predictions from the maximum value\n                predicted = torch.max(outputs.data, 1)[1]\n                \n                # Total number of labels\n                total += len(labels)\n                \n                correct += (predicted == labels).sum()\n            \n            accuracy = 100 * correct / float(total)\n            \n            # store loss and iteration\n            loss_list.append(loss.data)\n            iteration_list.append(count)\n            accuracy_list.append(accuracy)\n        if count % 500 == 0:\n            # Print Loss\n            print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data, accuracy))","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:01:34.841137Z","iopub.execute_input":"2023-06-25T12:01:34.841702Z","iopub.status.idle":"2023-06-25T12:02:59.912268Z","shell.execute_reply.started":"2023-06-25T12:01:34.841657Z","shell.execute_reply":"2023-06-25T12:02:59.910929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n\nprint(len(test_loader)) # 84 sets of 100 outputs each set\n# print(test_loader)\niteration = 0\nfor images, labels in test_loader:\n    test = Variable(images.view(100,1,28,28))\n    print(\"test\")\n    print(len(images[0].reshape(28,28)))\n    # Forward propagation\n    outputs = model(test)\n    # Get predictions from the maximum value\n    predicted = torch.max(outputs.data, 1)[1]\n    index = 0\n    if predicted[index] == labels[index]:\n        # visualize one of the images in data set\n        # plt.imshow(features_numpy[10].reshape(28,28))\n        plt.imshow(images[index].reshape(28,28))\n        plt.axis(\"off\")\n        plt.title(str(labels[index]) + \" == \" + str(predicted[index]))\n        plt.savefig('graph-test.png')\n        plt.show()\n    print(predicted[0])\n    iteration += 1\n    if iteration == 1:\n        break\nprint(iteration)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:02:59.914464Z","iopub.execute_input":"2023-06-25T12:02:59.914983Z","iopub.status.idle":"2023-06-25T12:03:00.194838Z","shell.execute_reply.started":"2023-06-25T12:02:59.914938Z","shell.execute_reply":"2023-06-25T12:03:00.193355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualization loss \nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"CNN: Loss vs Number of iteration\")\nplt.show()\n\n# visualization accuracy \nplt.plot(iteration_list,accuracy_list,color = \"red\")\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"CNN: Accuracy vs Number of iteration\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T12:03:00.196333Z","iopub.execute_input":"2023-06-25T12:03:00.196791Z","iopub.status.idle":"2023-06-25T12:03:00.764089Z","shell.execute_reply.started":"2023-06-25T12:03:00.196761Z","shell.execute_reply":"2023-06-25T12:03:00.762548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"4\"></a>\n## Convolutional Neural Network \n* CNN is used for image classification, object detection \n* <a href=\"https://ibb.co/kV1j9p\"><img src=\"https://preview.ibb.co/nRkBpp/gec2.jpg\" alt=\"gec2\" border=\"0\"></a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"5\"></a>\n### What is Convolution Operation?\n* We have some image and feature detector(3*3)\n* Feature detector does not need to be 3 by 3 matrix. It can be 5 by 5 or 7 by 7.\n* Feature detector = kernel = filter\n* Feauture detector detects features like edges or convex shapes. Example, if out input is dog, feature detector can detect features like ear or tail of the dog.\n* feature map = conv(input image, feature detector). Element wise multiplication of matrices.\n* feature map = convolved feature\n* Stride = navigating in input image.\n* We reduce the size of image. This is important bc code runs faster. However, we lost information. \n* We create multiple feature maps bc we use multiple feature detectors(filters).\n* Lets look at gimp. Edge detect: [0,10,0],[10,-4,10],[0,10,0]\n* <a href=\"https://imgbb.com/\"><img src=\"https://image.ibb.co/m4FQC9/gec.jpg\" alt=\"gec\" border=\"0\"></a>\n* After having convolution layer we use ReLU to break up linearity. Increase nonlinearity. Because images are non linear.\n* <a href=\"https://ibb.co/mVZih9\"><img src=\"https://preview.ibb.co/gbcQvU/RELU.jpg\" alt=\"RELU\" border=\"0\"></a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"6\"></a>\n### Same Padding\n* As we keep applying conv layers, the size of the volume will decrease faster than we would like. In the early layers of our network, we want to preserve as much information about the original input volume so that we can extract those low level features.\n* input size and output size are same.\n* <a href=\"https://ibb.co/jUPkUp\"><img src=\"https://preview.ibb.co/noH5Up/padding.jpg\" alt=\"padding\" border=\"0\"></a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"7\"></a>\n### Max Pooling\n* It makes down-sampling or sub-sampling (Reduces the number of parameters)\n* It makes the detection of features invariant to scale or orientation changes.\n* It reduce the amount of parameters and computation in the network, and hence to also control overfitting. \n* <a href=\"https://ibb.co/ckTjN9\"><img src=\"https://preview.ibb.co/gsNYFU/maxpool.jpg\" alt=\"maxpool\" border=\"0\"></a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"8\"></a>\n### Flattening\n* <a href=\"https://imgbb.com/\"><img src=\"https://image.ibb.co/c7eVvU/flattenigng.jpg\" alt=\"flattenigng\" border=\"0\"></a>","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"<a id=\"9\"></a>\n### Full Connection\n* Neurons in a fully connected layer have connections to all activations in the previous layer\n* Artificial Neural Network\n* <a href=\"https://ibb.co/hsS14p\"><img src=\"https://preview.ibb.co/evzsAU/fullyc.jpg\" alt=\"fullyc\" border=\"0\"></a>","metadata":{}}]}