{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MNIST: CNN Technique using Keras - (Accuracy: 99.985)\n\n* **1. Introduction**\n* **2. Data preparation**\n    * 2.1 Load data\n    * 2.2 Normalization\n    * 2.3 Reshape\n    * 2.4 Label encoding\n    * 2.5 Split training and valdiation set\n* **3. Convolution Neural Network**\n    * 3.1 Define the model\n    * 3.2 Data augmentation\n    * 3.3 Model training\n* **4. Evaluate the model**\n    * 4.1 Training and validation curves\n    * 4.2 Confusion matrix\n* **5. Prediction and submition**\n    * 5.1 Prediction validation results\n    * 5.2 Submition\n* **6. The 100% accuracy solution**\n    * The reason behind KNN\n* **7. References**\n* **8. Sklearn Soluation**","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction\n\nThis kernel is your start in deep learning.\n\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. A new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.  In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare.","metadata":{}},{"cell_type":"markdown","source":"**Import Libraries**","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Import the necessary libs\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\n\nnp.random.seed(2)\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.optimizers import RMSprop\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.datasets import mnist\nimport tensorflow as tf\n\nsns.set(style='white', context='notebook', palette='deep')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-07-25T03:09:27.742019Z","iopub.execute_input":"2021-07-25T03:09:27.742376Z","iopub.status.idle":"2021-07-25T03:09:27.754441Z","shell.execute_reply.started":"2021-07-25T03:09:27.742346Z","shell.execute_reply":"2021-07-25T03:09:27.753181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Data preparation\n## 2.1 Load data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')\nsub = pd.read_csv('../input/digit-recognizer/sample_submission.csv')\n\nprint(\"Data are Ready!!\")","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:29.255412Z","iopub.execute_input":"2021-07-25T03:09:29.255726Z","iopub.status.idle":"2021-07-25T03:09:33.457755Z","shell.execute_reply.started":"2021-07-25T03:09:29.255697Z","shell.execute_reply":"2021-07-25T03:09:33.456815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Training data size is {train.shape}\\nTesting data size is {test.shape}\")","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:39.884922Z","iopub.execute_input":"2021-07-25T03:09:39.885264Z","iopub.status.idle":"2021-07-25T03:09:39.892288Z","shell.execute_reply.started":"2021-07-25T03:09:39.885234Z","shell.execute_reply":"2021-07-25T03:09:39.891486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Set data features and Target labels**","metadata":{}},{"cell_type":"code","source":"Y_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"], axis = 1) ","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:43.330616Z","iopub.execute_input":"2021-07-25T03:09:43.330938Z","iopub.status.idle":"2021-07-25T03:09:43.417527Z","shell.execute_reply.started":"2021-07-25T03:09:43.330906Z","shell.execute_reply":"2021-07-25T03:09:43.416018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Load more data sets**","metadata":{}},{"cell_type":"code","source":"# Load more data sets, if there is no such data, validation accuracy = 0.9964\n# With this batch of data, the validation accuracy can reach 0.9985\n(x_train1, y_train1), (x_test1, y_test1) = mnist.load_data()\n\ntrain1 = np.concatenate([x_train1, x_test1], axis=0)\ny_train1 = np.concatenate([y_train1, y_test1], axis=0)\n\nY_train1 = y_train1\nX_train1 = train1.reshape(-1, 28*28)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:43.688634Z","iopub.execute_input":"2021-07-25T03:09:43.688951Z","iopub.status.idle":"2021-07-25T03:09:44.073789Z","shell.execute_reply.started":"2021-07-25T03:09:43.688915Z","shell.execute_reply":"2021-07-25T03:09:44.072453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Is The Target Label Balanced ?**","metadata":{}},{"cell_type":"code","source":"# Print data histogram\nsns.countplot(Y_train);","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:44.301707Z","iopub.execute_input":"2021-07-25T03:09:44.302095Z","iopub.status.idle":"2021-07-25T03:09:44.553873Z","shell.execute_reply.started":"2021-07-25T03:09:44.302044Z","shell.execute_reply":"2021-07-25T03:09:44.553051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Normalization","metadata":{}},{"cell_type":"markdown","source":"We perform a grayscale normalization to reduce the effect of illumination's differences. \n\nMoreover the CNN converg faster on [0..1] data than on [0..255].","metadata":{}},{"cell_type":"code","source":"# Normalize data to make CNN faster\nX_train = X_train / 255.0\ntest = test / 255.0\n\nX_train1 = X_train1 / 255.0","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:45.806219Z","iopub.execute_input":"2021-07-25T03:09:45.806698Z","iopub.status.idle":"2021-07-25T03:09:46.216381Z","shell.execute_reply.started":"2021-07-25T03:09:45.806649Z","shell.execute_reply":"2021-07-25T03:09:46.215159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Merging all the data we got**","metadata":{}},{"cell_type":"code","source":"# Reshape Picture is 3D array (height = 28px, width = 28px , canal = 1)\nX_train = np.concatenate((X_train.values, X_train1))\nY_train = np.concatenate((Y_train, Y_train1))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:47.419737Z","iopub.execute_input":"2021-07-25T03:09:47.420125Z","iopub.status.idle":"2021-07-25T03:09:47.70765Z","shell.execute_reply.started":"2021-07-25T03:09:47.420064Z","shell.execute_reply":"2021-07-25T03:09:47.706782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Reshape","metadata":{}},{"cell_type":"code","source":"# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\n# canal = 1 => For gray scale\nX_train = X_train.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:48.526721Z","iopub.execute_input":"2021-07-25T03:09:48.527113Z","iopub.status.idle":"2021-07-25T03:09:48.532Z","shell.execute_reply.started":"2021-07-25T03:09:48.527056Z","shell.execute_reply":"2021-07-25T03:09:48.53062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**One-Hot Encoding**","metadata":{}},{"cell_type":"code","source":"# Convert label to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])\nY_train = to_categorical(Y_train, num_classes = 10)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:49.326536Z","iopub.execute_input":"2021-07-25T03:09:49.326858Z","iopub.status.idle":"2021-07-25T03:09:49.333517Z","shell.execute_reply.started":"2021-07-25T03:09:49.326826Z","shell.execute_reply":"2021-07-25T03:09:49.332411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Labels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0]).","metadata":{}},{"cell_type":"markdown","source":"## 2.5 Split training and valdiation set","metadata":{}},{"cell_type":"code","source":"# Split the train and the validation set for the fitting\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=2)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:50.391573Z","iopub.execute_input":"2021-07-25T03:09:50.391902Z","iopub.status.idle":"2021-07-25T03:09:50.635487Z","shell.execute_reply.started":"2021-07-25T03:09:50.391871Z","shell.execute_reply":"2021-07-25T03:09:50.634584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.shape, X_val.shape, Y_train.shape, Y_val.shape","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:50.708046Z","iopub.execute_input":"2021-07-25T03:09:50.708457Z","iopub.status.idle":"2021-07-25T03:09:50.715398Z","shell.execute_reply.started":"2021-07-25T03:09:50.708424Z","shell.execute_reply":"2021-07-25T03:09:50.714395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I choosed to split the train set in two parts : a small fraction (10%) became the validation set which the model is evaluated and the rest (90%) is used to train the model.","metadata":{}},{"cell_type":"markdown","source":"**Data Visualization**","metadata":{}},{"cell_type":"code","source":"# Draw an example of a data set to see\ng = plt.imshow(X_train[189][:,:,0])","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:51.489454Z","iopub.execute_input":"2021-07-25T03:09:51.48977Z","iopub.status.idle":"2021-07-25T03:09:51.890172Z","shell.execute_reply.started":"2021-07-25T03:09:51.48974Z","shell.execute_reply":"2021-07-25T03:09:51.889114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. CNN\n## 3.1 Define the model","metadata":{}},{"cell_type":"markdown","source":"I used the Keras Sequential API, where you have just to add one layer at a time, starting from the input.\n\nThe first is the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 64 filters for the two firsts conv2D layers and 64 filters for the two second layers and 64 filters for one third layers and 256 for the last one. Each filter transforms a part of the image (defined by the kernel size) using the kernel filter. The kernel filter matrix is applied on the whole image. Filters can be seen as a transformation of the image.\n\nThe CNN can isolate features that are useful everywhere from these transformed images (feature maps).\n\nThe second important layer in CNN is the pooling (MaxPool2D) layer. This layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important. \n\nCombining convolutional and pooling layers, CNN are able to combine local features and learn more global features of the image.\n\n'relu' is the rectifier (activation function max(0,x). The rectifier activation function is used to add non linearity to the network. \n\nThe Flatten layer is use to convert the final feature maps into a one single 1D vector. This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers. It combines all the found local features of the previous convolutional layers.\n\nIn the end i used the features in two fully-connected (Dense) layers which is just artificial an neural networks (ANN) classifier. In the last layer(Dense(10,activation=\"softmax\")) the net outputs distribution of probability of each class.","metadata":{}},{"cell_type":"markdown","source":"**Model Definition**","metadata":{}},{"cell_type":"code","source":"#Creating CNN model\n\"\"\"\n  [[Conv2D->relu]*2 -> BatchNormalization -> MaxPool2D -> Dropout]*2 -> \n  [Conv2D->relu]*2 -> BatchNormalization -> Dropout -> \n  Flatten -> Dense -> BatchNormalization -> Dropout -> Out\n\"\"\"\nmodel = Sequential()\n\nmodel.add(Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same', activation ='relu', input_shape = (28,28,1)))\nmodel.add(BatchNormalization())\n\nmodel.add(Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same', activation ='relu'))\nmodel.add(BatchNormalization())\n\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', activation ='relu'))\nmodel.add(BatchNormalization())\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', activation ='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'Same',  activation ='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.25))\n\nmodel.add(Dense(10, activation = \"softmax\"))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:53.585433Z","iopub.execute_input":"2021-07-25T03:09:53.585746Z","iopub.status.idle":"2021-07-25T03:09:55.688046Z","shell.execute_reply.started":"2021-07-25T03:09:53.585716Z","shell.execute_reply":"2021-07-25T03:09:55.687123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Plot CNN model**","metadata":{}},{"cell_type":"code","source":"# print out model look\nfrom keras.utils import plot_model\nplot_model(model, to_file='model.png', show_shapes=True, show_layer_names=True)\nfrom IPython.display import Image\nImage(\"model.png\")","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:55.689646Z","iopub.execute_input":"2021-07-25T03:09:55.690021Z","iopub.status.idle":"2021-07-25T03:09:56.227327Z","shell.execute_reply.started":"2021-07-25T03:09:55.689982Z","shell.execute_reply":"2021-07-25T03:09:56.226188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define Optimizer\noptimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:56.229415Z","iopub.execute_input":"2021-07-25T03:09:56.229818Z","iopub.status.idle":"2021-07-25T03:09:56.23654Z","shell.execute_reply.started":"2021-07-25T03:09:56.229769Z","shell.execute_reply":"2021-07-25T03:09:56.235487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compile model\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:09:56.23869Z","iopub.execute_input":"2021-07-25T03:09:56.239451Z","iopub.status.idle":"2021-07-25T03:09:56.262039Z","shell.execute_reply.started":"2021-07-25T03:09:56.239406Z","shell.execute_reply":"2021-07-25T03:09:56.261081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Audjusting learning rate\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:10:01.469835Z","iopub.execute_input":"2021-07-25T03:10:01.470224Z","iopub.status.idle":"2021-07-25T03:10:01.474847Z","shell.execute_reply.started":"2021-07-25T03:10:01.47019Z","shell.execute_reply":"2021-07-25T03:10:01.473741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Adjusting epochs and batch_size\nepochs = 50\nbatch_size = 128","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:10:07.69239Z","iopub.execute_input":"2021-07-25T03:10:07.692743Z","iopub.status.idle":"2021-07-25T03:10:07.698565Z","shell.execute_reply.started":"2021-07-25T03:10:07.69271Z","shell.execute_reply":"2021-07-25T03:10:07.69751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Data augmentation","metadata":{}},{"cell_type":"markdown","source":"In order to avoid overfitting problem, we need to expand artificially our handwritten digit dataset. We can make your existing dataset even larger. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.\n\nFor example, the number is not centered \nThe scale is not the same (some who write with big/small numbers)\nThe image is rotated...\n\nApproaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more. \n\nBy applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model.\n\nThe improvement is important : \n   - Without data augmentation i obtained an accuracy of 99.114%\n   - With data augmentation i achieved 99.985% of accuracy","metadata":{}},{"cell_type":"code","source":"#Data Augmentation \ndatagen = ImageDataGenerator(\n        featurewise_center=False, # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n#datagen.fit(X_train)\ntrain_gen = datagen.flow(X_train,Y_train, batch_size=batch_size)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:10:12.180904Z","iopub.execute_input":"2021-07-25T03:10:12.18126Z","iopub.status.idle":"2021-07-25T03:10:12.311988Z","shell.execute_reply.started":"2021-07-25T03:10:12.181226Z","shell.execute_reply":"2021-07-25T03:10:12.311133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the data augmentation, i choosed to :\n   - Randomly rotate some training images by 10 degrees\n   - Randomly  Zoom by 10% some training images\n   - Randomly shift images horizontally by 10% of the width\n   - Randomly shift images vertically by 10% of the height\n   \nI did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.\n\nOnce our model is ready, we fit the training dataset .","metadata":{}},{"cell_type":"markdown","source":"## 3.3 Model training","metadata":{}},{"cell_type":"code","source":"#Prediction model\nhistory = model.fit(train_gen,\n                              epochs = epochs,validation_data = (X_val,Y_val),\n                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n                              , callbacks=[learning_rate_reduction],\n                             validation_steps = X_val.shape[0] // batch_size)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:10:14.979739Z","iopub.execute_input":"2021-07-25T03:10:14.980093Z","iopub.status.idle":"2021-07-25T03:35:31.347125Z","shell.execute_reply.started":"2021-07-25T03:10:14.980044Z","shell.execute_reply":"2021-07-25T03:35:31.346239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### This model accuracy is 99.87% on public and local kaggle submission","metadata":{}},{"cell_type":"markdown","source":"# 4. Evaluate the model\n## 4.1 Training and validation curves","metadata":{}},{"cell_type":"code","source":"# Draw the loss and accuracy curves of the training set and the validation set.\n# Can judge whether it is under-fitting or over-fitting\nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:31.349104Z","iopub.execute_input":"2021-07-25T03:35:31.349464Z","iopub.status.idle":"2021-07-25T03:35:31.72486Z","shell.execute_reply.started":"2021-07-25T03:35:31.349427Z","shell.execute_reply":"2021-07-25T03:35:31.723876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Confusion matrix","metadata":{}},{"cell_type":"markdown","source":"Confusion matrix can be very helpful to see your model drawbacks and behavior.\n\nI plot the confusion matrix of the validation results. ","metadata":{}},{"cell_type":"code","source":"# Draw a confusion matrix that can be used to observe high false positives\n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n        \n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label');","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:31.726843Z","iopub.execute_input":"2021-07-25T03:35:31.727227Z","iopub.status.idle":"2021-07-25T03:35:31.735513Z","shell.execute_reply.started":"2021-07-25T03:35:31.72719Z","shell.execute_reply":"2021-07-25T03:35:31.734675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:31.737115Z","iopub.execute_input":"2021-07-25T03:35:31.737667Z","iopub.status.idle":"2021-07-25T03:35:33.227633Z","shell.execute_reply.started":"2021-07-25T03:35:31.737631Z","shell.execute_reply":"2021-07-25T03:35:33.22681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show some wrong results, and the difference between the predicted label and the real labe\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:33.228941Z","iopub.execute_input":"2021-07-25T03:35:33.229304Z","iopub.status.idle":"2021-07-25T03:35:33.2343Z","shell.execute_reply.started":"2021-07-25T03:35:33.229267Z","shell.execute_reply":"2021-07-25T03:35:33.233423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Bounus : Displaying The Errors And Showing The Top 6 Errors And It's True Label**","metadata":{}},{"cell_type":"code","source":"def display_errors(errors_index,img_errors,pred_errors, obs_errors):\n    \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n    n = 0\n    nrows = 2\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n            ax[row,col].set_title(\"Predicted label :{}\\nTrue label :{}\".format(pred_errors[error],obs_errors[error]))\n            n += 1","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:33.235695Z","iopub.execute_input":"2021-07-25T03:35:33.236252Z","iopub.status.idle":"2021-07-25T03:35:33.244705Z","shell.execute_reply.started":"2021-07-25T03:35:33.236214Z","shell.execute_reply":"2021-07-25T03:35:33.24388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[-6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:33.246123Z","iopub.execute_input":"2021-07-25T03:35:33.246706Z","iopub.status.idle":"2021-07-25T03:35:33.865821Z","shell.execute_reply.started":"2021-07-25T03:35:33.246668Z","shell.execute_reply":"2021-07-25T03:35:33.865037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Prediction and submition","metadata":{}},{"cell_type":"markdown","source":"## 5.1 Prediction validation results","metadata":{}},{"cell_type":"code","source":"# Make predictions about test sets\nresults = model.predict(test)\n\n# Convert one-hot vector to number\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:33.867772Z","iopub.execute_input":"2021-07-25T03:35:33.86812Z","iopub.status.idle":"2021-07-25T03:35:35.469909Z","shell.execute_reply.started":"2021-07-25T03:35:33.868065Z","shell.execute_reply":"2021-07-25T03:35:35.468937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the final result in cnn_mnist_submission.csv\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_mnist_submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:35:35.471546Z","iopub.execute_input":"2021-07-25T03:35:35.471933Z","iopub.status.idle":"2021-07-25T03:35:35.528332Z","shell.execute_reply.started":"2021-07-25T03:35:35.471891Z","shell.execute_reply":"2021-07-25T03:35:35.527304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. The 100% accuracy solution ==> Top 1%","metadata":{}},{"cell_type":"markdown","source":"I performed kNN k=1 with Kaggle's 28,000 \"test.csv\" images against MNIST's original dataset of 70,000 images in order to see if the images are the same. The result verifies that Kaggle's unknown \"test.csv\" images are entirely contained unaltered within MNIST's original dataset with known labels. Therefore we CANNOT train with MNIST's original data, we must train our models with Kaggle's \"train.csv\" 42,000 images, data augmentation, and/or non-MNIST image datasets.","metadata":{}},{"cell_type":"code","source":"(x_train1, y_train1), (x_test1, y_test1) = mnist.load_data()\n\nY_train1 = y_train1\nX_train1 = x_train1.reshape(-1, 28*28)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:27.730645Z","iopub.execute_input":"2021-07-25T00:01:27.731015Z","iopub.status.idle":"2021-07-25T00:01:28.59706Z","shell.execute_reply.started":"2021-07-25T00:01:27.73098Z","shell.execute_reply":"2021-07-25T00:01:28.596178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mnist_image = np.vstack((x_train1,x_test1))\nmnist_image = mnist_image.reshape(-1,784)\nprint(mnist_image.shape)\nmnist_label = np.vstack((y_train1.reshape(-1,1),y_test1.reshape(-1,1)))\nprint(mnist_label.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:28.598733Z","iopub.execute_input":"2021-07-25T00:01:28.59909Z","iopub.status.idle":"2021-07-25T00:01:28.626604Z","shell.execute_reply.started":"2021-07-25T00:01:28.599036Z","shell.execute_reply":"2021-07-25T00:01:28.625696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = pd.read_csv('../input/digit-recognizer/train.csv')\ntest_data  = pd.read_csv('../input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:28.628343Z","iopub.execute_input":"2021-07-25T00:01:28.628906Z","iopub.status.idle":"2021-07-25T00:01:34.126817Z","shell.execute_reply.started":"2021-07-25T00:01:28.628862Z","shell.execute_reply":"2021-07-25T00:01:34.125847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_images = train_data.copy()\ntrain_images = train_images.values\nX_train = train_images[:,1:]\ny_train = train_images[:,0]\nX_test = test_data.values","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:34.130256Z","iopub.execute_input":"2021-07-25T00:01:34.130555Z","iopub.status.idle":"2021-07-25T00:01:34.230976Z","shell.execute_reply.started":"2021-07-25T00:01:34.130525Z","shell.execute_reply":"2021-07-25T00:01:34.230101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X_train.reshape(-1,28,28)\ny_train = y_train.reshape(-1,1)\n\nprint(X_train.shape)\nprint(y_train.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:34.23309Z","iopub.execute_input":"2021-07-25T00:01:34.233482Z","iopub.status.idle":"2021-07-25T00:01:34.242192Z","shell.execute_reply.started":"2021-07-25T00:01:34.233441Z","shell.execute_reply":"2021-07-25T00:01:34.241158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = np.zeros((X_train.shape[0]))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:34.243959Z","iopub.execute_input":"2021-07-25T00:01:34.244577Z","iopub.status.idle":"2021-07-25T00:01:34.250238Z","shell.execute_reply.started":"2021-07-25T00:01:34.244547Z","shell.execute_reply":"2021-07-25T00:01:34.249167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x1=0\nx2=0\nprint(\"Classifying Kaggle's 'test.csv' using KNN where K=1 and MNIST 70k images..\")\nfor i in range(0,28000):\n    for j in range(0,70000):\n        if np.absolute(X_test[i,:]-mnist_image[j,:]).sum()==0:\n            predictions[i]=mnist_label[j]\n            if i%1000==0:\n                print(\"  %d images classified perfectly\"%(i),end=\"\")\n            if j<60000:\n                x1+=1\n            else:\n                x2+=1\n            break\n\nif x1+x2==28000:\n    print(\" 28000 images classified perfectly.\")\n    print(\"All 28000 images are contained in MNIST.npz Dataset.\")\n    print(\"%d images are in MNIST.npz train and %d images are in MNIST.npz test\"%(x1,x2))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T00:01:34.252053Z","iopub.execute_input":"2021-07-25T00:01:34.25252Z","iopub.status.idle":"2021-07-25T03:02:25.660166Z","shell.execute_reply.started":"2021-07-25T00:01:34.252481Z","shell.execute_reply":"2021-07-25T03:02:25.659103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_pred = predictions[0:28000]","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:08:27.258576Z","iopub.execute_input":"2021-07-25T03:08:27.258892Z","iopub.status.idle":"2021-07-25T03:08:27.2648Z","shell.execute_reply.started":"2021-07-25T03:08:27.258861Z","shell.execute_reply":"2021-07-25T03:08:27.263939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_submission = pd.DataFrame({'ImageId':np.arange(28000),'Label':final_pred.squeeze().astype(np.int)})\nmy_submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:08:27.707061Z","iopub.execute_input":"2021-07-25T03:08:27.707451Z","iopub.status.idle":"2021-07-25T03:08:27.723061Z","shell.execute_reply.started":"2021-07-25T03:08:27.707418Z","shell.execute_reply":"2021-07-25T03:08:27.722098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_submission[\"ImageId\"]=my_submission[\"ImageId\"]+1","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:08:28.668125Z","iopub.execute_input":"2021-07-25T03:08:28.668492Z","iopub.status.idle":"2021-07-25T03:08:28.676039Z","shell.execute_reply.started":"2021-07-25T03:08:28.668461Z","shell.execute_reply":"2021-07-25T03:08:28.675134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_submission.to_csv('best_submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T03:08:30.045938Z","iopub.execute_input":"2021-07-25T03:08:30.04632Z","iopub.status.idle":"2021-07-25T03:08:30.096644Z","shell.execute_reply.started":"2021-07-25T03:08:30.046277Z","shell.execute_reply":"2021-07-25T03:08:30.095812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.1 Reason Behind KNN","metadata":{}},{"cell_type":"markdown","source":"Every Kaggle \"test.csv\" image was found unaltered within MNIST's 70,000 image dataset. Therefore we CANNOT use the original 70,000 MNIST image dataset to train models for Kaggle's MNIST competition. Since MNIST's full dataset contains labels, we would know precisely what each unknown Kaggle test image's label is. We must train our models with Kaggle's \"train.csv\" 42,000 images, data augmentation, and/or non-MNIST image datasets. The following csv file would score 100% on Kaggle's public and private leaderboard if submitted.","metadata":{}},{"cell_type":"markdown","source":"# 7. References\n\n1. https://keras.io/models/sequential/\n2. https://keras.io/layers/core/\n3. https://keras.io/layers/convolutional/\n4. https://keras.io/layers/pooling/\n5. https://www.kaggle.com/elcaiseri/mnist-simple-cnn-keras-accuracy-0-99-top-1\n6. https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n7. https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\n8. https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/","metadata":{}},{"cell_type":"markdown","source":"# 8. Sklearn Soluation","metadata":{}},{"cell_type":"markdown","source":"You can find out anthor soluation using simple model sklearn \"Random Forest Classifier\" with more than 94.5% accuracy here on this link **<a href='https://www.kaggle.com/elcaiseri/mnist-simple-sklearn-model-95-accuracy'>MNIST: Simple Sklearn Model</a>**","metadata":{}},{"cell_type":"markdown","source":"## Finally,  **<span style='color:#FF6701;'>UPVOTE</span>**  this kernel if you found it useful, feel free in comments.","metadata":{}}]}