{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MNIST Handwritten Digit Recognizer using Deep CNN\n![MNIST Dataset](https://miro.medium.com/max/1400/1*26W2Yk3cu2uz_R8BuSb_SA.png)\n   If you got interested in Machine Learning and Data Science and you are looking for a way to learn the concepts of machine learing, you have came to the perfect place to learn and be nutured. Kaggle is perfect for the begginer to get started with machine learning. Handwritten Digit Recognizer using MNIST Dataset is getting started competition for all beginners. I hope my notebook can introduced you to the world of machine learning and future.\n\n   In this kernel, I have created a Deep Convolutional Neural Network (CNN) model to recognize different handwritten digits and classify them. The dataset used here is actually from [Digit Recognition Competition](https://www.kaggle.com/c/digit-recognizer). Let's get started.\n\n**Frameworks**\n* [Tensorflow v2](https://tensorflow.org) - A open sourced machine learning framework from Google.\n* [Keras](https://keras.io) - A open sourced neural network library running on top of tensorflow.\n\n\n### Process flow\n1. [Importing Libraries](#1)\n2. [Preparing the Dataset](#2)\n3. [Model Building](#3)\n4. [Model Fitting](#4)\n5. [Analyzing the model](#5)\n6. [Predicting the test data](#6)\n\n\n","metadata":{"id":"K_YntzlsRNWx"}},{"cell_type":"markdown","source":"## 1. Importing the Libraries <a></a>\nHere we are importing the required libraries for the kernel. ","metadata":{"id":"zQHYkAoVUMKi"}},{"cell_type":"code","source":"# Importing Tensorflow and keras\n#Keras is built into TF 2.0\n\nimport tensorflow as tf\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.image as img\n%matplotlib inline\n\nnp.random.seed(2)\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\n#Setting the Theme of the data visualizer Seaborn\nsns.set(style=\"dark\",context=\"notebook\",palette=\"muted\")\n\n#Tensorflow Version\nprint(\"TensorFlow Version:   \"+tf.version.VERSION)\nprint(\"Keras Version:   \"+tf.keras.__version__)","metadata":{"id":"LYlQnHuARMFQ","outputId":"53fccd35-7077-46f4-d7f5-3e8d77d18508","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Preparing the Dataset\nWe have added the data provided from the MNIST Handwritten Digit Recognition competition. Use the *+ Add data* button on the top right corner to do that. Select the competition. The data will be added to the kernel.We can find the dataset in the right panel under Data. You can copy the path of the train and test data from the right panel. \n\nread_csv is used to return a pandas DataFrame object from conversion of csv file.\n\nThen we select the label column and store it in Y_train which is used for the training. X_train contains the pixel values of the respective labelled image.\n\nWe visualize the total number of data of each class using countplot.Then, we check for missing values in the dataset.","metadata":{"id":"WEPDqrJLUWEi"}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\n","metadata":{"id":"UutK5S_rUGmH","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_train = train['label']\n\n#Dropping Label Column\nX_train = train.drop(labels=['label'],axis=1)\n\n#free up some space\ndel train\n\ngraph = sns.countplot(Y_train)\n \nY_train.value_counts()","metadata":{"id":"05yoL078UJIp","outputId":"87e71ffe-2da5-4e74-d1e9-c66cc4061b84","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Checking for any null or missing values\nX_train.isnull().any().describe()\n\ntest.isnull().any().describe()\n","metadata":{"id":"jGSazKXLXRtA","outputId":"5c6958f8-b12c-44d5-f0e3-dc7eb1546c1e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Normalisation\nNormalisation is done to reduce the scale of the input values. The pixel value ranges from 0 to 255 which specify gradient of gray. The CNN will converge more faster on values 0 to 1 than 0 to 255. So we divide every value by 255 to scale the data from [0..255] to [0..1]. It helps the model to better learning of features by decreasing computational complexities if we have data that scales bigger.","metadata":{"id":"xJQsCTooXv1X"}},{"cell_type":"code","source":"X_train = X_train/255\ntest = test/255","metadata":{"id":"pTFl1nLqXpSy","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reshape\nThe array of pixel values are reshaped into a (28,28,1) matrix. We are feeding the CNN model with input_shape of 28x28x1 matrix.","metadata":{"id":"0Lx2A8K8YAKn"}},{"cell_type":"code","source":"X_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","metadata":{"id":"RgJra9a4X9U1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Label Encoding\nSince the CNN model will give results in a vector of predictions for each classes. The label (numbers) are encoded into hot vector for prediction by the model. So that we can train the CNN with the encoded outputs and the parameters are tuned accordingly","metadata":{"id":"GSbzJTHlYkhS"}},{"cell_type":"code","source":"Y_train = tf.keras.utils.to_categorical(Y_train, num_classes=10)\n#To enable label into hot vector. For Eg.7 -> [0,0,0,0,0,0,0,1,0,0]","metadata":{"id":"U7DvS95KYrVa","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train and Validation Data Split\nWe are segmenting the input data for training into two exclusive data namely, Train and Validation data sets. Train data is used to train the model whereas the validation data is used for cross verification of the model's accuracy and how well the model is generalized for the data other than training data. Validation accuracy and loss will tell us the performance of the model for new data and it will show if there is overfitting or underfitting situation while model training. \n","metadata":{}},{"cell_type":"code","source":"#Spliting Train and test set\nrandom_seed =2\n\nX_train,X_val,Y_train,Y_val = train_test_split(X_train,Y_train,test_size=0.1,\n                                                random_state = random_seed)","metadata":{"id":"UUBksBY12L8V","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Show some example \n\ng = plt.imshow(X_train[0][:,:,0])","metadata":{"id":"KVNgcZ5F3Ap5","outputId":"4e5029cb-8519-42c8-f57f-12d7f35e8eb3","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Model Building","metadata":{"id":"tU1XRCKr3e0n"}},{"cell_type":"markdown","source":"Deep Convolutional Neural Network is a network of artificial neural networks. A model archiecture is the design of the neural networks with which we train the parameters in training process. Here we used LeNet-5 Architecture, it was proposed by Yann LeCun in 1998. Its pretty popular for its minimal structure and easy to train nature. LeNet-5 architecture is suitable for recognition and classification of different classes of objects in small resolution images. \n\nTo learn more about CNN architectures and current state-of-the-art architecture, take a visit [here](https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5)\n\n![Deep CNN Architecture](https://miro.medium.com/max/3744/1*SGPGG7oeSvVlV5sOSQ2iZw.png)","metadata":{}},{"cell_type":"code","source":"#CNN Architecture is In -> [[Conv2D->relu]*2 -> MaxPool2D -> Dropout]*2 -> \n                           #Flatten -> Dense -> Dropout -> Out\nmodel = tf.keras.Sequential()\n\nmodel.add(layers.Conv2D(filters=32, kernel_size=(5,5), padding='Same', \n                       activation=tf.nn.relu, input_shape = (28,28,1)))\nmodel.add(layers.Conv2D(filters=32, kernel_size=(5,5), padding='Same', \n                       activation=tf.nn.relu))\nmodel.add(layers.MaxPool2D(pool_size=(2,2)))\nmodel.add(layers.Dropout(0.25))\n\n\nmodel.add(layers.Conv2D(filters=64, kernel_size=(3,3), padding='Same', \n                       activation=tf.nn.relu, input_shape = (28,28,1)))\nmodel.add(layers.Conv2D(filters=64, kernel_size=(3,3), padding='Same', \n                       activation=tf.nn.relu))\nmodel.add(layers.MaxPool2D(pool_size=(2,2),strides=(2,2)))\nmodel.add(layers.Dropout(0.25))\n\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(256,activation=tf.nn.relu))\nmodel.add(layers.Dropout(0.25))\nmodel.add(layers.Dense(10,activation=tf.nn.softmax))","metadata":{"id":"kpkANohy3TaJ","outputId":"956ee2f4-8139-4189-8f92-b2382119e70d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Optimizers and Annealers\n* Optimizer is the crucial part in a neural network. Optimizer ensure the model reaches the optimium faster. RMSProp optimizer makes the model converge more effectively and faster. It also stricts the model to coverge at global minimum therefore the accuracy of the model will be higher.  \n* Setting the learning rate is very important in a Deep Learning algorithm. Though choosing a good learning rate may yield its goodness, scheduling a reduction in learning rate while training has a great advantage in convergence at global minimum. ReduceLRonPlateau makes the Learning Rate to reduce at a rate by monitoring the learning rate and epoch. It will greatly help the model to achieve maximum accuracy.","metadata":{}},{"cell_type":"code","source":"#Defining Optimizer\n\noptimizer = tf.keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)\n","metadata":{"id":"VuN1-e0uPCWy","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Compiling Model\n\nmodel.compile(optimizer = optimizer, loss='categorical_crossentropy', \n             metrics=[\"accuracy\"])","metadata":{"id":"wiwSpyiYPcIb","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Setting Learning rate annealer\n\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_acc',\n                                           patience=3,\n                                           verbose=1,\n                                           factor=0.5,\n                                           min_lr=0.00001)","metadata":{"id":"FqExgW5DQoab","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> I have set the epoch to 10 for the purpose of kernel. You can bump it upto 30 to see a larger accuracy. ","metadata":{}},{"cell_type":"code","source":"epochs=30\nbatch_size = 112","metadata":{"id":"VpoFy1wQRJf8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Augmentation\nData Augmentation is the process of creating more dataset for training by manipulating the given images. In Deep learning, the availability of large dataset is very vital for the training. Since we have limited real world training samples, we can use data augementation to create more images for training. Data Augmentation involves zoom, rotating, flip, crop and other image manipulations over the available datasets to create further more data for training. Data Augmentation makes the model to classify more generally.","metadata":{"id":"E573js59RQVn"}},{"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X_train)","metadata":{"id":"lTi2iTMKROwt","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Model Fitting\n   Model Fitting or Model Training is where we train our model and evaluate the error parameters. Training process typically take a lot of time when it runs in a CPU. But the training can be speeeded up with the graphics card that have CUDA support. Kaggle have inbuilt limited GPU Support be sure to turn it on when running this cell.","metadata":{"id":"qkX93W-P4iYl"}},{"cell_type":"code","source":"if(tf.test.is_built_with_cuda() == True):\n    print(\"CUDA Available.. Just wait a few moments...\")\nelse: \n    print(\"CUDA not Available.. May the force be with you.\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit the model\nhistory = model.fit_generator(datagen.flow(X_train,Y_train, batch_size=batch_size),\n                              epochs = epochs, validation_data = (X_val,Y_val),\n                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n                              , callbacks=[learning_rate_reduction])","metadata":{"id":"BYvD4vpOz3_L","outputId":"02ebfa24-d959-4b82-8aa5-f83f18c98a0a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. Analyzing the model\nWe can analyse the model using various methods. One of them, is learning graph. Here we plot the losses of both training and validation data in a plot and evaluate it by looking at the trend. For a ideal model, training and validation loss should be low and similar. ","metadata":{}},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","metadata":{"id":"PMKZhU75z-b6","outputId":"4801351f-7ead-469c-a528-f418abc68eb6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Confusion Matrix Plotting\nConfusion Matrix is another way of model evaluation. It is used for grphical representation of performance of the model. It shows the performance of Model in predicting every class. Here you can find the model pretty accurately predict the relevant classes.","metadata":{"id":"oTsAl3Ur3n9d"}},{"cell_type":"code","source":"# Look at confusion matrix \n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","metadata":{"_execution_state":"idle","_uuid":"16e161179bf1b51ba66c39b2cead883f1db3a9c7","_cell_guid":"11361e73-8250-4bf5-a353-b0f8ea83e659","id":"IK6-TyagG4G4","outputId":"29d7f070-8773-4f39-d2fe-6f02d0d2bfa1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Important Error\nThough we may have very high accuracy, any deep learing model cannot correctly predict all the images. So we are viewing the important errors done by our model for perspection.","metadata":{}},{"cell_type":"code","source":"# Display some error results \n\n# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\ndef display_errors(errors_index,img_errors,pred_errors, obs_errors):\n    \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n    n = 0\n    nrows = 2\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n            ax[row,col].set_title(\" Predicted :{} True :{}\".format(pred_errors[error],obs_errors[error]))\n            n += 1\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[-6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","metadata":{"_execution_state":"idle","_uuid":"e7a3d6449b499a29db224e42e950f21ca1ec4e36","_cell_guid":"7b0f31b8-c18b-4529-b0d8-eb4c31e30bbf","id":"tV4wnLTQG4G8","outputId":"718bdde1-4a6b-45f9-c5e9-18e93cb845cb","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6. Predicting the test data\nFinally we are predicting the test dataset for the competition afer done training and performance evaluation. We predict the test data and store it in a csv file for competition submission.","metadata":{}},{"cell_type":"code","source":"# predict results\nresults = model.predict(test)\n\n# select the indix with the maximum probability\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","metadata":{"_execution_state":"idle","_uuid":"7f17e7bf0a54a01a52fef2d554780f6bc6580dc6","_cell_guid":"05ff3b9f-c3bb-4cec-a8c2-2c128e8f15b3","id":"N9KswR97G4G-","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_mnist_datagen.csv\",index=False)","metadata":{"_execution_state":"idle","_uuid":"369dfaab09240f3f12bcff91953ffd315ab84985","_cell_guid":"b5f1f39f-13b8-439a-8913-0f120e3d47a9","id":"qvS8FrQEG4HC","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\nI hope you all enjoyed by notebook. It's my first try of writing one. I have learnt many awesome new things about Deep Learning and the math concepts behind it. I got incredibly amused by all those concepts and sink in the world of deep learning. \n\n## If you find this notebook useful, please do upvote which will encourage me to learn and do more in the future.","metadata":{}}]}