{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Handwritten Digits Classification\n\n> If you are new to Image processing and Deep Learning, then this notebook is for you!<br>\n> It will give you a basic insight about how to work with images and leave you with a good model to get upto 99.7% accurate results on MNIST handwritten images for unseen images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Important Points:\n> Have a look at these points before we continue to read forther!\n* Kaggle Competition Link: https://www.kaggle.com/c/digit-recognizer\n* In this notebook I am using the MNIST Digits dataset which can be accessed from the link above.\n* About the dataset: The dataset consists of 10 classes of handwritten Images pictures each with a number between 0-9.\n* I hope you enjoy reading this notebook! 😃","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport tensorflow as tf\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras import metrics\nfrom tensorflow.keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading the folder architecture of Kaggle to get the dataset path.\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading the Train and Test Datasets.\nmnist_train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\nmnist_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's see the shape of the train and test data\nprint(mnist_train.shape, mnist_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Looking at a few rows from the data isn't a bad idea.\nmnist_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# and yeah, here you will see the basic statistical insights of the numerical features of train data.\nmnist_train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Cleaning and Normalization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist_train.isna().any().any()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### NOTE:\n* Data is totally clean in this case (since the final result says `False` which means it has no missing values)\n* There is no empty field. Data is clean already.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Data Normalization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# dividing the data into the input and output features to train make the model learn based on what to take in and what to throw out.\nmnist_train_data = mnist_train.loc[:, \"pixel0\":]\nmnist_train_label = mnist_train.loc[:, \"label\"]\n\n# Notmailzing the images array to be in the range of 0-1 by dividing them by the max possible value. \n# Here is it 255 as we have 255 value range for pixels of an image. \nmnist_train_data = mnist_train_data/255.0\nmnist_test = mnist_test/255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visulaize a single digit with an array","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's make some beautiful plots.\ndigit_array = mnist_train.loc[3, \"pixel0\":]\narr = np.array(digit_array) \n\n#.reshape(a, (28,28))\nimage_array = np.reshape(arr, (28,28))\n\ndigit_img = plt.imshow(image_array, cmap=plt.cm.binary)\nplt.colorbar(digit_img)\nprint(\"IMAGE LABEL: {}\".format(mnist_train.loc[3, \"label\"]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PCA Implementation on MNIST Digits\n\n### 1) Using manual approach:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### a) Compute standardization of data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n\nstandardized_scalar = StandardScaler()\nstandardized_data = standardized_scalar.fit_transform(mnist_train_data)\nstandardized_data.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### b) Calculate covariance matrix S(dxd)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cov_matrix = np.matmul(standardized_data.T, standardized_data)\ncov_matrix.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### c) Calculate Eigen values and eigen vectors","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.linalg import eigh\n\nlambdas, vectors = eigh(cov_matrix, eigvals=(782, 783))\nvectors.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vectors = vectors.T\nvectors.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### d) Calculate unit vectors U1=V1 and new coordinates","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_coordinates = np.matmul(vectors, standardized_data.T)\nprint(new_coordinates.shape)\nnew_coordinates = np.vstack((new_coordinates, mnist_train_label)).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_new = pd.DataFrame(new_coordinates, columns=[\"f1\", \"f2\", \"labels\"])\ndf_new.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### e) Plot FacetGrid using seaborn","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.FacetGrid(df_new, hue=\"labels\", size=6).map(plt.scatter, \"f1\", \"f2\").add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2) Using Sci-kit Learn library:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import decomposition\n\npca = decomposition.PCA()\npca.n_components = 2\npca_data = pca.fit_transform(standardized_data)\npca_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_data = np.vstack((pca_data.T, mnist_train_label)).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_PCA = pd.DataFrame(new_coordinates, columns=[\"f1\", \"f2\", \"labels\"])\ndf_PCA.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.FacetGrid(df_new, hue=\"labels\", size=12).map(plt.scatter, \"f1\", \"f2\").add_legend()\nplt.savefig(\"PCA_FacetGrid.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PCA Dimension Reduction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pca.n_components = 784\npca_data = pca.fit_transform(standardized_data)\npercent_variance_retained = pca.explained_variance_ / np.sum(pca.explained_variance_)\n\ncum_variance_retained = np.cumsum(percent_variance_retained)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(1, figsize=(10, 6))\nplt.clf()\nplt.plot(cum_variance_retained, linewidth=2)\nplt.axis(\"tight\")\nplt.grid()\nplt.xlabel(\"number of compoments\")\nplt.ylabel(\"cumulative variance retained\")\nplt.savefig(\"pca_cumulative_variance.png\")\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Manipulation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's build a count plot to see the count of all the labels.\nsns.countplot(mnist_train.label)\nprint(list(mnist_train.label.value_counts().sort_index()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting dataframe into arrays\nmnist_train_data = np.array(mnist_train_data)\nmnist_train_label = np.array(mnist_train_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshaping the input shapes to get it in the shape which the model expects to recieve later.\nmnist_train_data = mnist_train_data.reshape(mnist_train_data.shape[0], 28, 28, 1)\nprint(mnist_train_data.shape, mnist_train_label.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Building Process\n> Training a neural network with one input layer, one hidden layer and one output layer for learning the digits in images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# But first import some cool libraries before getting our hands dirty!! \n# TensorFlow is Google's open source AI framework and we are using is here to build model.\n# Keras is built on top of Tensorflow and gives us\n# NO MORE GEEKY STUFF, Know more about them here:  https://www.tensorflow.org     https://keras.io\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Lambda, Flatten, BatchNormalization\nfrom tensorflow.keras.layers import Conv2D, MaxPool2D, AvgPool2D\nfrom tensorflow.keras.optimizers import Adadelta\nfrom keras.utils.np_utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.callbacks import LearningRateScheduler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encoding train labels","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encoding the labels and making them as the class value and finally converting them as categorical values.\nnclasses = mnist_train_label.max() - mnist_train_label.min() + 1\nmnist_train_label = to_categorical(mnist_train_label, num_classes = nclasses)\nprint(\"Shape of ytrain after encoding: \", mnist_train_label.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building a Sequential Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Warning!!! Here comes the beast!!!\n\ndef build_model(input_shape=(28, 28, 1)):\n    model = Sequential()\n    model.add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = input_shape))\n    model.add(BatchNormalization())\n    model.add(Conv2D(32, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4))\n\n    model.add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4))\n\n    model.add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Flatten())\n    model.add(Dropout(0.4))\n    model.add(Dense(10, activation='softmax'))\n    return model\n\n    \ndef compile_model(model, optimizer='adam', loss='categorical_crossentropy'):\n    model.compile(optimizer=optimizer, loss=loss, metrics=[\"accuracy\"])\n    \n    \ndef train_model(model, train, test, epochs, split):\n    history = model.fit(train, test, shuffle=True, epochs=epochs, validation_split=split)\n    return history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training the model using the above function built to build, compile and train the model\ncnn_model = build_model((28, 28, 1))\ncompile_model(cnn_model, 'adam', 'categorical_crossentropy')\n\n# train the model for as many epochs as you want but I found training it above 80 will not help us and eventually increase overfitting.\nmodel_history = train_model(cnn_model, mnist_train_data, mnist_train_label, 80, 0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Performance Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_model_performance(metric, validations_metric):\n    plt.plot(model_history.history[metric],label = str('Training ' + metric))\n    plt.plot(model_history.history[validations_metric],label = str('Validation ' + metric))\n    plt.legend()\n    plt.savefig(str(metric + '_plot.png'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_model_performance('accuracy', 'val_accuracy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_model_performance('loss', 'val_loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Transforming testing data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# reshaping the test arrays as we did to train images above somewhere.\nmnist_test_arr = np.array(mnist_test)\nmnist_test_arr = mnist_test_arr.reshape(mnist_test_arr.shape[0], 28, 28, 1)\nprint(mnist_test_arr.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prediction & Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now, since the model is trained, it's time to find the results for the unseen test images.\npredictions = cnn_model.predict(mnist_test_arr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Finally, making the final submissions assuming that we have to submit it in any comptition. P)\npredictions_test = []\n\nfor i in predictions:\n    predictions_test.append(np.argmax(i))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission =  pd.DataFrame({\n        \"ImageId\": mnist_test.index+1,\n        \"Label\": predictions_test\n    })\n\nsubmission.to_csv('my_submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Read my other notebooks at:\nhttps://www.kaggle.com/blurredmachine/notebooks\n\nI hope you like it.<br>\nI am continuously working on this notebook to keep an easy approache for beginners to understand the concepts easily.<br>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Consider upvoting if it was helpful! 😃","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}