{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Problem Statment \n\nIdentify digits from a dataset of tens of thousands of handwritten images. The MNIST handwritten digit classification problem is a standard dataset used in computer vision and deep learning.\n\nIn this Notebook, we will discover how to develop a convolutional neural network for handwritten digit classification from scratch.\n\nWhat we will implement in this notebook:\n\n* How to develop a test harness to develop a robust evaluation of a model and establish a baseline of performance for a classification task.\n* How to explore extensions to a baseline model to improve learning and model capacity.\n* How to develop a finalized model, evaluate the performance of the final model, and use it to make predictions on new images.\n\n\n## **This task can be divided into the following subtasks.**\n\n1. **Data Preparation**\n2. **Building a CNN Model**\n3. **Evalution of the model**\n4. **Prediction of validation data**\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Loading\n\nIt is a dataset of 60,000 small square 28×28 pixel grayscale images of handwritten single digits between 0 and 9.\n\nThe task is to classify a given image of a handwritten digit into one of 10 classes representing integer values from 0 to 9, inclusively.\n\nWe will load the data and visualize the data. ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.optimizers import Adam, RMSprop\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPool2D,BatchNormalization","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Spliting the data in to X_train and Y_train","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_train=train['label']\n\n# Drop 'label' column\nX_train = train.drop(labels = [\"label\"],axis = 1)\n\nY_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data visualization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.countplot(Y_train)\nplt.title('The distribution of the digits in the dataset', weight='bold', fontsize='18')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we say that all classes have almost equal distribution, We don't have any class imbalance here so we can go forward with our preprocessing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the data\nX_train.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the data\ntest.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Normalization\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Images can be used to rescale pixel values from the range of 0-255 to the range 0-1 preferred for neural network models.\n\nScaling data to the range of 0-1 is traditionally referred to as normalization.\n\nhere we are normalizing the pixel values of grayscale images, e.g. rescale them to the range [0,1]. This involves first converting the data type from unsigned integers to floats, then dividing the pixel values by the maximum value.\n\nIn this case, the ratio is 1/255 or about 0.0039.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255\ntest = test / 255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reshape\n\n![Reshape](https://backtobazics.com/wp-content/uploads/2018/08/numpy-reshape-vector-to-matrix.jpg)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshape image in 3 dimensions (height = 28px, width = 28px , channel = 1)\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is a glimpse of what we will be dealing with:\n\n* Images of handwritten Digits from 0 to 9\n* We will feed those images to the CNN in order to learn and predict the test images.\n* We have below an example of few digit images from this dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train[0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,8))\nfor i in range(50):\n    plt.subplot(5,10,i+1)\n    plt.imshow(X_train[i].reshape((28,28)),cmap='binary')\n    plt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Encoding \n\n![Encoding](https://i.imgur.com/wKtY1Og.png)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The shape of the labels before One Hot Encoding\",Y_train.shape)\nY_train = to_categorical(Y_train, num_classes = 10)\nprint(\"The shape of the labels after One Hot Encoding\",Y_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_train[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split the train and the validation set for the fitting\nrandom_seed = 2\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.3, random_state=random_seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n# Some examples\ng = plt.imshow(X_train[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data augmentation\n\n![Data Augmentation](https://nanonets.com/blog/content/images/2018/11/1_dJNlEc7yf93K4pjRJL55PA--1-.png)\n\n* ImageDataGenerator accepts the original data, randomly transforms it, and returns only the new, transformed data.\n* The Keras deep learning neural network library provides the capability to fit models using image data augmentation via the ImageDataGenerator class.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(zoom_range = 0.1, width_shift_range = 0.1, height_shift_range = 0.1, rotation_range = 10) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Building a CNN Model\n![Cnn Model Architecture](https://miro.medium.com/max/700/1*uAeANQIOQPqWZnnuH-VEyw.jpeg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Constructing a sequential CNN model\n\n* The model type that we will be using is Sequential. Sequential is the easiest way to build a model in Keras. It allows you to build a model layer by layer.\n\n![Convolution neural network](https://miro.medium.com/max/1000/1*vkQ0hXDaQv57sALXAJquxA.jpeg)\n\n* We use the ‘add()’ function to add layers to our model.\n* Our first 3 layers are Conv2D layers. These are convolution layers that will deal with our input images, which are seen as 2-dimensional matrices.\n\n![kernel](https://i.imgur.com/NcyYyaJ.gif)\n\n* Activation is the activation function for the layer. The activation function we will be using for our first 2 layers is the ReLU, or Rectified Linear Activation. This activation function has been proven to work well in neural networks.\n\n![Activation Function](https://miro.medium.com/max/1000/1*4ZEDRpFuCIpUjNgjDdT2Lg.png)\n\n* Batchnormalization layer - Batch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. [More info](https://medium.com/analytics-vidhya/everything-you-need-to-know-about-regularizer-eb477b0c82ba)\n\n![Normalization](https://miro.medium.com/max/1200/1*DmnOhSTIzn04sC0w1d3FPg.png)\n\n* Pooling layers provide an approach to down sampling feature maps by summarizing the presence of features in patches of the feature map. Two common pooling methods are average pooling and max pooling that summarize the average presence of a feature and the most activated presence of a feature respectively.\n\n![Pooling](https://qph.fs.quoracdn.net/main-qimg-cf2833a40f946faf04163bc28517959c)\n\n* Dropoutlayer - A single model can be used to simulate having a large number of different network architectures by randomly dropping out nodes during training.\n![Drop out](https://miro.medium.com/max/700/0*bTMVb8uekPpHxDcm)\n\n* In between the Conv2D layers and the dense layer, there is a ‘Flatten’ layer. Flatten serves as a connection between the convolution and dense layers.\n* ‘Dense’ is the layer type we will use in for our output layer. Dense is a standard layer type that is used in many cases for neural networks.\n* We will have 10 nodes in our output layer, one for each possible outcome (0–9).\n* The activation is ‘softmax’. Softmax makes the output sum up to 1 so the output can be interpreted as probabilities. The model will then make its prediction based on which option has the highest probability.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', input_shape = (28, 28, 1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 32, kernel_size = (5, 5), activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(strides = (2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3, 3), activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 64, kernel_size = (3, 3), activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 64, kernel_size = (5, 5), activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(strides = (2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation = 'relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1024, activation = 'relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation = 'softmax'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compiling the model\n\nWhile compiling the model, three parameters requires loss, the optimizer and metrics.\n\n* categorical_crossentropy is a loss function for categorical variables\n* Use the Adam Optimizer to control the learning rate\n* The metric 'accuracy' is used to measure the performance of the model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam',metrics=['accuracy'],loss='categorical_crossentropy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduction_lr = ReduceLROnPlateau(monitor='val_accuracy',patience=2, verbose=1, factor=0.2, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reduce learning rate when a metric has stopped improving.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"hist = model.fit_generator(datagen.flow(X_train,Y_train,batch_size=32),epochs=20,validation_data = (X_val,Y_val),callbacks=[reduction_lr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss = pd.DataFrame(model.history.history)\nloss[['loss', 'val_loss']].plot()\nloss[['accuracy', 'val_accuracy']].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_loss, final_acc = model.evaluate(X_val, Y_val, verbose=0)\nprint(\"Final loss: {0:.4f}, final accuracy: {1:.4f}\".format(final_loss, final_acc))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = model.predict(X_val, batch_size = 64)\n\ny_pred = np.argmax(y_pred,axis = 1)\ny_pred = pd.Series(y_pred,name=\"Label\")\ny_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn')\nsns.set_style('whitegrid')\nfig = plt.figure(figsize=(10,10))\nax1 = plt.subplot2grid((1,2),(0,0))\ntrain_loss = hist.history['loss']\ntest_loss = hist.history['val_loss']\nx = list(range(1, len(test_loss) + 1))\nplt.plot(x, test_loss, color = 'cyan', label = 'Test loss')\nplt.plot(x, train_loss, label = 'Training losss')\nplt.legend()\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title(' Loss vs. Epoch',weight='bold', fontsize=18)\n\nax1 = plt.subplot2grid((1,2),(0,1))\ntrain_loss = hist.history['loss']\ntest_loss = hist.history['val_loss']\nx = list(range(1, len(test_loss) + 1))\nplt.plot(x, test_loss, color = 'cyan', label = 'Test loss')\nplt.plot(x, train_loss, label = 'Training losss')\nplt.legend()\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title(' Accuracy vs. Epoch',weight='bold', fontsize=18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_val.shape, y_pred.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_val = np.argmax(Y_val,axis = 1)\nY_val = pd.Series(Y_val,name=\"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\ncmatrix = confusion_matrix(Y_val, y_pred)\n\nplt.figure(figsize=(15,8))\nplt.title('Confusion matrix of the test/predicted digits ', weight='bold', fontsize=18)\nsns.heatmap(cmatrix,annot=True,cmap=\"Reds\",fmt=\"d\",cbar=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# #We use np.argmax with y_test and predicted values: transform them from 10D vector to 1D\n# # class_y = np.argmax(Y_val,axis = 1) \n# # class_num=np.argmax(y_pred, axis=1)\n# #Detect the errors\n# errors = (y_pred - Y_val != 0)\n# #Localize the error images\n# predicted_er = y_pred[errors]\n# y_test_er = Y_val[errors]\n# x_test_er = X_val[errors]\n#Plot the misclassified numbers\n# plt.figure(figsize=(15,9))\n\n# for i in range(30):\n#     plt.subplot(5,6,i+1)\n#     plt.imshow(x_test_er[i].reshape((-1,28,28,1)),cmap='binary')\n#     plt.title( np.argmax(predicted_er[i]), size=13, weight='bold', color='red')\n#     plt.axis(\"off\")\n\n\n# plt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test = test.values.reshape(-1, 28, 28, 1) / 255\ny_pred1 = model.predict(test, batch_size = 64)\n\ny_pred1 = np.argmax(y_pred1,axis = 1)\ny_pred1 = pd.Series(y_pred1,name=\"Label\")\ny_pred1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),y_pred1],axis = 1)\nsubmission.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"More reference\n\nhttps://www.pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/\n\nhttps://machinelearningmastery.com/how-to-normalize-center-and-standardize-images-with-the-imagedatagenerator-in-keras/\n\nhttps://towardsdatascience.com/complete-guide-of-activation-functions-34076e95d044\n\nhttps://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-keras-329fbbadc5f5\n\nhttps://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53\n\nhttps://medium.com/analytics-vidhya/everything-you-need-to-know-about-regularizer-eb477b0c82ba\n\nhttps://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ReduceLROnPlateau\n\nhttps://www.pluralsight.com/guides/getting-started-tensorflow","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}