{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST-Digit Recognizer CNN with GPU"},{"metadata":{},"cell_type":"markdown","source":"MNIST dataset is consists of handwritten digits between 0 and 9. It is very popular and basic dataset for image classification. Easiness of this dataset makes it very approachable for beginners like me."},{"metadata":{},"cell_type":"markdown","source":"<br>Content:\n* [1-PREPROCESSING](#1)\n    * [Loading Libraries](#2)\n    * [Loading Data](#3)\n    * [Labelling](#4)\n    * [Normalization](#5)\n    * [Reshaping](#6)\n    * [Label Encoding (One-Hot Encoding)](#7)\n    * [Visual Sample](#8)\n    * [Training and Validation Split](#9)\n* [2- MODELLING](#10)\n    * [CNN](#11)\n    * [Model](#12)\n    * [Data Augmentation](#13)\n* [3- Evaluation](#14)\n* [4-Submission](#15)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n# 1- PREPROCESSING"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## Loading Libraries\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport tensorflow as tf\nfrom tensorflow.python.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D\nfrom tensorflow.keras.callbacks import TensorBoard\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.preprocessing.image import ImageDataGenerator\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nimport time\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## Loading Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## Labelling"},{"metadata":{},"cell_type":"markdown","source":"We will be focusing on our train set for our operations. \n* For our independent variables, we drop our **label** on train and to save with **X_train**. \n* For our dependent variable, we select **label** and save it with **Y_train**."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Drop 'label' column\nX_train = train.drop(labels = [\"label\"],axis = 1) \n# put labels into y_train variable\nY_train = train[\"label\"]\n# free some space\ndel train \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_train.nunique() # We have 10 classes which are in the range of 0-9 numbers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## Normalization"},{"metadata":{},"cell_type":"markdown","source":"For making our job easier, we normalize our dataset with **255** which is the max value for an image."},{"metadata":{"trusted":true},"cell_type":"code","source":"# normalization with 255 because max value in image is 255\n\nX_train = X_train/255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a> <br>\n## Reshaping"},{"metadata":{},"cell_type":"markdown","source":"For making our job easier, we normalize our dataset with 255 which are the max value for an image."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Reshape\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a> <br>\n## Label Encoding (One-Hot Encoding)"},{"metadata":{},"cell_type":"markdown","source":"Labels are 10 digits numbers from 0 to 9"},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"8\"></a> <br>\n## Visual Sample"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X_train[0][:,:,0], cmap = \"gray\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"9\"></a> <br>\n## Training and Validation Split\n"},{"metadata":{},"cell_type":"markdown","source":"We will use validation set for looking our accuracy on training."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split the train and the validation set for the fitting\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.2, random_state= 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"10\"></a> <br>\n# 2-MODELLING"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"11\"></a> <br>\n## CNN"},{"metadata":{},"cell_type":"markdown","source":"For our model, we will use CNN with keras which is a tensorflow backend. Also, I used tensorboard for demonstration of my model. Hyper-parameter tuning is vital on CNN because deep learning is a blackbox. We do not know how CNN works deeply. Thus, one of the most important thing on AI is trying over and over again. I used very limited iteration for this operation. For that operation you can go https://github.com/asilkanvar96/MNIST-Digit_Recognizer_CNN_TF_GPU/blob/master/digit_rec_gpu.ipynb  For this notebook, I only put tensorboard images."},{"metadata":{},"cell_type":"markdown","source":"### Validation Accuracy"},{"metadata":{},"cell_type":"markdown","source":"<a href=\"https://ibb.co/PxdKQXQ\"><img src=\"https://i.ibb.co/wyfVsGs/val-acc.png\" alt=\"val-acc\" border=\"0\"></a>"},{"metadata":{},"cell_type":"markdown","source":"### Validation Loss"},{"metadata":{},"cell_type":"markdown","source":"<a href=\"https://ibb.co/fHcnV1N\"><img src=\"https://i.ibb.co/cQmCKg8/val-loss.png\" alt=\"val-loss\" border=\"0\"></a>"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"12\"></a> <br>\n## Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"dense_layers = [1]\nlayer_sizes = [64]\nconv_layers = [3]\n\nfor dense_layer in dense_layers: \n    for layer_size in layer_sizes:\n         for conv_layer in conv_layers:\n            model =  Sequential()\n\n            model.add(Conv2D(layer_size, (3,3), input_shape = (28,28,1), padding = 'Same'))\n            model.add(Activation(\"relu\"))\n            model.add(MaxPooling2D(pool_size = (2,2)))\n\n            for l in range(conv_layer-1):\n                model.add(Conv2D(layer_size, (3,3), padding = 'Same'))\n                model.add(Activation(\"relu\"))\n                model.add(MaxPooling2D(pool_size = (2,2)))\n                model.add(Dropout(0.25))\n                \n            model.add(Flatten()) # converts ourr 3D feature maps to 1D feature vectors\n\n            for l in range(dense_layer):\n                model.add(Dense(layer_size))\n                model.add(Activation(\"relu\"))\n\n            model.add(Dense(10)) # output layer\n            model.add(Activation(\"softmax\"))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a href=\"https://ibb.co/FY6cmcq\"><img src=\"https://i.ibb.co/R62tYt4/3-conv-64-nodes1-dense-1571955873.png\" alt=\"3-conv-64-nodes1-dense-1571955873\" border=\"0\"></a><br /><a target='_blank' href='https://imgbb.com/'></a><br />"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer = \"adam\" , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"13\"></a> <br>\n## Data Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        rotation_range=8,  # randomly rotate images\n        zoom_range = 0.5, # Randomly zoom image \n        width_shift_range=0.08,  # randomly shift images horizontally \n        height_shift_range=0.08,  # randomly shift images vertically \n        shear_range = 0.5,\n        ) \n\ndatagen.fit(X_train)\n\n# fitting model\nhistory = model.fit_generator(datagen.flow(X_train, Y_train, batch_size= 32),\n                              epochs = 10, validation_data = (X_val, Y_val), steps_per_epoch= X_train.shape[0] // 32, validation_steps = X_val.shape[0] // 32)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"14\"></a> <br>\n# 3-Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = model.predict(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(5):\n    plt.grid(False)\n    plt.imshow(X_val[i][:,:,0], cmap = plt.cm.binary)\n    plt.xlabel(\"Actual: \" + str(np.argmax(Y_val[i])))\n    plt.title(\"Prediction\" + str(np.argmax(y_pred[i])))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"15\"></a> <br>\n# 4-Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict results\nresults = model.predict(test)\n\n# select the index with the maximum probability\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_mnist_datagen.csv\",index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}