{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Digit Recognition Using Tensorflow\n\nMNIST Digit Recognition can be regarded as a 'Hello World' into Deep Learning based Computer Vision problems. \nMNIST dataset consists of images of handwritten digits (from 0 to 9). The objective is to recognize any handwritten digit outside the training data. Accuracy is chosen as the performance metric. \n\nI used Keras API for modelling the 5 layered CNN architecture. This is the overview on the steps I followed:\n After data preparation and checking the missing values, I performed  data augmentation - a way to prevent overfittng and to increase the capability to generalise. Then I modelled a Convolutional Neural Network architecture for solving\nOverview: \n 1. Data Preparation\n 2. Image Augmentation\n 3. Modelling CNN\n 4. Training and Validation to get appropriate model\n 5. Testing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Data Preparation\n\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the provided dataset, images are represented as csv data where each entry indicates the intensity value of corresponding pixel. Hence, for a $ 28 \\times 28  $ image, there will be $ 28 \\times 28 = 784 $ pixel values. So the first step is to prepare the data into appropriate form.  ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Reading the train and test data\ntrain = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')\n\n\nprint(train.shape)\nprint(test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 42000 training samples and 28000 testing samples. To get an overview, lets see some samples of training data.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first column of the training sample indicates the label and rest indicates the pixel values. Hence, we need to split the train data into dependent and independent variables and reshape the whole dataset into 2D images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = train.drop(['label'], axis = 1) #dependent variables \ny_train = train['label'] #independent variable\nX_test = test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets see if there are any missing values in train and test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above data is in vectorial form (pixel values arranged in a single array). Hence it should be reshaped into the form of 2D images. This is necessary since we are training the data using CNN. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the image data, higher intensity values are represented as $ 255 $, while lower values are represened by $0$ values. However, it is better to normalize the data before training for faster convergence. A simple way to normalize is to divide the whole data by the maximum value (255).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train  = X_train/255\nX_test  = X_test/255\n#The above step can also be done in image augmentation part","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets see a sample image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nplt.imshow(X_test[7][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To check the distribution of the classes:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.barplot(x = y_train.value_counts().index, y =y_train.value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hence, the dataset is almost balanced. Before proceeding to image augmentation part, there is an additional step : One Hot encoding the class labels. In one hot encoding, a binary vector is created corresponding to each classes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":" y_train  = tf.keras.utils.to_categorical(y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we can proceed to image augmentation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2. Image Augmentation\n\nIn simple terms,image augmentation is done to prevent overfitting. For example, if I feed the trained model with a new image for testing, the model should be able to recognize the image even if it is rotated or enlarged or flipped. Hence from the given data, we could generate additonal training data by performing above operations (but vertical flip is not recommended since it could confuse the model from distinguishing 6 and 9). Also, training data is again split into validation set for evaluating the performance. \n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Creating Training and Validation Datasets\n\nfrom keras.preprocessing.image import ImageDataGenerator\nBS = 32    # Batch Size\n\ntrain_datagen = ImageDataGenerator(\n    shear_range=0.1,\n    zoom_range=0.2,\n    validation_split = 0.2)\n\ntraining_set = train_datagen.flow(\n    x = X_train,\n    y = y_train,\n    batch_size = BS,\n    shuffle = True,\n    subset = 'training')\n\nvalidation_set = train_datagen.flow(\n    x = X_train,\n    y = y_train,\n    batch_size = BS,\n    shuffle = True,\n    subset = 'validation')\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generating test data\n# Do not perform augmentation operation on test data\ntest_datagen = ImageDataGenerator()\ntest_set = test_datagen.flow(\n    x = X_test,\n    y = None,\n    batch_size = BS,\n    shuffle = False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Modelling CNN\n\nI used a 5 layered CNN architecture as the model. At the initial stage, it is prefearrable to use higher filter size ($ 5 \\times 5 $ is used in this case) to capture more information. In the second stage the filter size is decreased to $ 3 \\times 3 $. The pooling size is chosen as  $ 2 $. \n\nRelu is chosen as the activation function in hidden stages for faster convergence and adding non linearity to the network. The output of convolution and maxpooling layers is flattened and is fed as input to a 3 layered fully connected (dense) ANN. Relu is also chosen as the activation in these hidden layers also. At the output the softmax is chosen as the activation function which gives the probability distribution of the labels for a given input.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cnn = tf.keras.models.Sequential()\ncnn.add(tf.keras.layers.Conv2D(filters = 32, kernel_size = 5, activation = 'relu', input_shape = (28,28,1)))\ncnn.add(tf.keras.layers.MaxPool2D(pool_size = 2, strides = 1))\n\ncnn.add(tf.keras.layers.Conv2D(filters = 64, kernel_size = 3, activation = 'relu'))\ncnn.add(tf.keras.layers.MaxPool2D(pool_size = 2, strides = 1))\n\ncnn.add(tf.keras.layers.Flatten())\ncnn.add(tf.keras.layers.Dense(units = 128, activation = 'relu'))\ncnn.add(tf.keras.layers.Dense(units = 128, activation = 'relu'))\ncnn.add(tf.keras.layers.Dense(units = 10, activation = 'softmax'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Training and Validation\n\nInitially, the number of epochs is chosen as 25 (arbitrary value). ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"EPOCHS = 25 # No. of Epochs\ncnn.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\ntest_model = cnn.fit(training_set, validation_data = validation_set, epochs = EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(test_model.history.keys())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting training accuracy and validaiton accuracy \nplt.plot(test_model.history['accuracy'])\nplt.plot(test_model.history['val_accuracy'])\nplt.title('Model Accuracy')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\nplt.legend(['Train Accuracy', 'Validation Accuracy'], loc = 'upper right')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting training loss and validaiton loss\nplt.plot(test_model.history['loss'])\nplt.plot(test_model.history['val_loss'])\nplt.title('Model Loss Performance')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\nplt.legend(['Train Loss', 'Validation Loss'], loc = 'upper right')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initially, for low values of epochs (less than 4), the training accuracy is low which indicates underfitting. \nFor number of epochs around 4, the training and validation accuracies are almost the same. \nAs the value is increased beyond 8, the both patterns begins to diverge whcih clearly indicates overfitting. Hence for optimal model fitting, the number of epochs is chosen  as 4.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Final model\ncnn.fit(training_set, validation_data = validation_set, epochs = 4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Testing\n\nThe final model is used to predict the test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing a sample image \n# test_image = X_test[7][:,:,0]\n# print(type(test_image))\n# test_image = np.expand_dims(test_image, axis = 2)\n# test_image.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict = cnn.predict_classes(test_set)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Writing the test results to a separate dataframe\nsubmission = pd.DataFrame()\nsubmission['ImageId'] = pd.Series(range(1,28001))\nsubmission['Label'] = predict\nsubmission.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.imshow(test_set[0][15][:,:,0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the final model, I was able to achieve **98.79% accuracy**. I think it is possible to improve the accuracy with a better hyperparameter tuning (no. of hidden layers, no of neurons, optimizer, etc.). I admit earlystopping is the more ideal method to find the optimal number of epochs. But I used  my approach to get a more intuitive feeling of what's really happening. ","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}