{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Handwritten Digits Recognition & Classification\n\n> This notebook gives a first hand Convolutional Neural Network approach using tensorflow and keras for the task of Handwritten digits recognition and classification<br><br>\n> This dataset is pretty much the 'Hello World!' of Computer Vision<br>\n> Hope this helps beginners interested in Image recognition and classification to have a jump start!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Notes:\n* Kaggle Competition Link for dataset: https://www.kaggle.com/c/digit-recognizer\n* About the dataset: The dataset consists of 10 classes of handwritten numbers between 0-9 as Images.\n* Jump in! 😃","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## Importing all necessary libraries \nimport pandas as pd\nimport numpy as np\n\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport tensorflow as tf\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras import metrics\nfrom tensorflow.keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading the folder architecture of Kaggle to get the dataset path.\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading the Datasets.\nmnist_train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\nmnist_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Printing the dimensions/shape of the given data\nprint(mnist_train.shape, mnist_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that there are 42000 train and 28000 test examples","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Preliminary analysis\nmnist_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist_train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Normalization and Cleaning","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist_train.isna().any().any()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### NOTE:\n* Data is totally clean in this case (since the final result says `False` which means it has no missing values)\n* There is no empty field. Data is clean already.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Data Normalization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# dividing the data into the input and output features to train make the model learn based on what to take in and what to throw out.\nmnist_train_data = mnist_train.loc[:, \"pixel0\":]\nmnist_train_label = mnist_train.loc[:, \"label\"]\n\n# Normalizing the images array to be in the range of 0-1 by dividing them by the max possible value. \n# Here is it 255 as we have 255 value range for pixels of an image. \nmnist_train_data = mnist_train_data/255.0\nmnist_test = mnist_test/255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visulaize a single digit with an array","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's make some beautiful plots.\ndigit_array = mnist_train.loc[3, \"pixel0\":]\narr = np.array(digit_array) \n\n#.reshape(a, (28,28))\nimage_array = np.reshape(arr, (28,28))\n\ndigit_img = plt.imshow(image_array, cmap=plt.cm.binary)\nplt.colorbar(digit_img)\nprint(\"IMAGE LABEL: {}\".format(mnist_train.loc[3, \"label\"]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's make some beautiful plots.\ndigit_array = mnist_train.loc[4, \"pixel0\":]\narr = np.array(digit_array) \n\n#.reshape(a, (28,28))\nimage_array = np.reshape(arr, (28,28))\n\ndigit_img = plt.imshow(image_array, cmap=plt.cm.binary)\nplt.colorbar(digit_img)\nprint(\"IMAGE LABEL: {}\".format(mnist_train.loc[4, \"label\"]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Countplot for each of the 10 digits.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's build a count plot to see the count of all the labels.\nsns.countplot(mnist_train.label)\nprint(list(mnist_train.label.value_counts().sort_index()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of data across the classes of digits is pretty much the same","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting dataframe into arrays\nmnist_train_data = np.array(mnist_train_data)\nmnist_train_label = np.array(mnist_train_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshaping the input shapes to get it in the shape which the model expects to recieve later.\nmnist_train_data = mnist_train_data.reshape(mnist_train_data.shape[0], 28, 28, 1)\nprint(mnist_train_data.shape, mnist_train_label.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Building Process\n> Training a neural network with one input layer, one hidden layer and one output layer for learning the digits in images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# But first import some cool libraries before getting our hands dirty!! \n# TensorFlow is Google's open source AI framework and we are using is here to build model.\n# Keras is built on top of Tensorflow and gives us\n# NO MORE GEEKY STUFF, Know more about them here:  https://www.tensorflow.org     https://keras.io\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Lambda, Flatten, BatchNormalization\nfrom tensorflow.keras.layers import Conv2D, MaxPool2D, AvgPool2D\nfrom tensorflow.keras.optimizers import Adadelta\nfrom keras.utils.np_utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n### When the accuracy or loss starts to plateau during training we can implement the following callbacks \n#### to lower the learning rate and hence make smaller steps as it gets closer to the global optimum\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.callbacks import LearningRateScheduler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encoding train labels","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encoding the labels and making them as the class value and finally converting them as categorical values.\nnclasses = mnist_train_label.max() - mnist_train_label.min() + 1\nmnist_train_label = to_categorical(mnist_train_label, num_classes = nclasses)\nprint(\"Shape of y_train after encoding: \", mnist_train_label.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building a Sequential Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function builds the CNN Necessary for recognition, detailed explanation in the comments\n\ndef build_model(input_shape=(28, 28, 1)):\n    model = Sequential()     ## We need a sequential model obviously (don't require bidirectional, etc)\n    model.add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = input_shape)) # First 2D Convolutional layer\n    model.add(BatchNormalization()) # Activation is Rectified Linear Unit of ReLU for all layers\n    model.add(Conv2D(32, kernel_size = 3, activation='relu')) # Batch Normalization is used along with Dropout\n    model.add(BatchNormalization())\n    model.add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4))\n    ## Dropout Regularization of 0.4 in order to avoid overfitting\n    model.add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4))\n\n    model.add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Flatten())\n    model.add(Dropout(0.4))\n    model.add(Dense(10, activation='softmax')) ## Softmax activation used as this is a multiclass classification task\n    return model    ## The number of units is 10 as there are 10 different classes of digits","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Utility Functions ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def compile_model(model, optimizer='adam', loss='categorical_crossentropy'):\n    model.compile(optimizer=optimizer, loss=loss, metrics=[\"accuracy\"]) # using adam optimization, RMSProp works fine too\n    ## Categorical crossentropy is used as the multiclass loss\n    \ndef train_model(model, train, test, epochs, split):\n    history = model.fit(train, test, shuffle=True, epochs=epochs, validation_split=split)\n    return history ## Data is shuffled during training to avoid inherent bias to the sequence of occurence of an image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training for 50 epochs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training the model using the above function built to build, compile and train the model\ncnn_model = build_model((28, 28, 1)) ## The input is an image odf size 28 X 28\ncompile_model(cnn_model, 'adam', 'categorical_crossentropy')\n\n# train the model for as many epochs as you want but I found training it above 100 will not help us and eventually \n## increase overfitting.\nmodel_history = train_model(cnn_model, mnist_train_data, mnist_train_label, 50, 0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Performance Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_model_performance(metric, validations_metric):\n    plt.plot(model_history.history[metric],label = str('Training ' + metric))\n    plt.plot(model_history.history[validations_metric],label = str('Validation ' + metric))\n    plt.legend()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Plotting the accuracy's\nplot_model_performance('accuracy', 'val_accuracy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training accuracy steadily increased and plateaued while validation accuracy is also consistent. This clearly shows the model is robust!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## Plotting the loss\nplot_model_performance('loss', 'val_loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Transforming testing data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# reshaping the test arrays as we did to train images above somewhere.\nmnist_test_arr = np.array(mnist_test)\nmnist_test_arr = mnist_test_arr.reshape(mnist_test_arr.shape[0], 28, 28, 1)\nprint(mnist_test_arr.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prediction & Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now, since the model is trained, it's time to find the results for the unseen test images.\npredictions = cnn_model.predict(mnist_test_arr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Finally, making the final submissions\npredictions_test = []\n\nfor i in predictions:\n    predictions_test.append(np.argmax(i))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Submitting in the required format\nsubmission =  pd.DataFrame({\n        \"ImageId\": mnist_test.index+1,\n        \"Label\": predictions_test\n    })\n\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope you liked it.<br>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Edit 1: Making the cells more meaningful and clearer with comments\n\nEdit 2: Adding submission file and few more visualizations.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Kindly upvote if it was helpful! 😃","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}