{"cells":[{"metadata":{"_uuid":"f2156d1dd26a1243e18512002e10872c5bd7271e","_cell_guid":"d4c08f48-fe23-4ddb-ac46-d97f05397514"},"cell_type":"markdown","source":"# MNSIT digit recognizer\n\n\n* **1. Introduction**\n* **2. Data preparation**\n    * 2.1 Load data\n    * 2.2 Check for null and missing values\n    * 2.3 Normalization\n    * 2.4 Reshape\n    * 2.5 Label encoding\n    * 2.6 Split training and valdiation set\n* **3. CNN**\n    * 3.1 Define the model\n    * 3.2 Set the optimizer and annealer\n    * 3.3 Data augmentation\n* **4. Evaluate the model**\n    * 4.1 Training and validation curves\n    * 4.2 Confusion matrix\n* **5. Prediction and submition**\n    * 5.1 Predict and Submit results"},{"metadata":{"_uuid":"e9aff3cf1bb8daa73bec67b970d12195677679f3","_cell_guid":"eb88b372-a6e5-40c8-a1c6-c03799165490"},"cell_type":"markdown","source":"# 1. Introduction\n\nThis is a 5 layers Sequential Convolutional Neural Network for digits recognition trained on MNIST dataset. I choosed to build it with keras API (Tensorflow backend) which is very intuitive. Firstly, I will prepare the data (handwritten digits images) then i will focus on the CNN modeling and evaluation.\n\nI achieved 99.671% of accuracy with this CNN trained in 2h30 on a single CPU (i5 2500k). For those who have a >= 3.0 GPU capabilites (from GTX 650 - to recent GPUs), you can use tensorflow-gpu with keras. Computation will be much much faster !!!\n\n**For computational reasons, i set the number of steps (epochs) to 2, if you want to achieve 99+% of accuracy set it to 30.**\n\nThis Notebook follows three main parts:\n\n* The data preparation\n* The CNN modeling and evaluation\n* The results prediction and submission\n\n\n\n\n<img src=\"http://img1.imagilive.com/0717/mnist-sample.png\" ></img>"},{"metadata":{"_execution_state":"idle","_uuid":"72334cb006d02a4bcfc2a2fe622524eba824c6f8","_cell_guid":"f67b9393-8ea1-4e23-b856-2ce149cfe421","collapsed":true,"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\n\nnp.random.seed(2)\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.optimizers import RMSprop\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\n\nsns.set(style='white', context='notebook', palette='deep')","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"86061d98eccaa02efe0dab0fa3884e71fcf4c310","_cell_guid":"6d2fb3e6-ab71-4974-b5a2-4af1ebdb99f4"},"cell_type":"markdown","source":"# 2. Data preparation\n## 2.1 Load data"},{"metadata":{"_execution_state":"idle","_uuid":"84bbd5ab8d7895bd430d5ecfe2f7ddf77baa7b74","_cell_guid":"5e51d00e-62fd-4141-bf73-50ac4f2da7d0","collapsed":true,"trusted":false},"cell_type":"code","source":"# Load the data\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"1213b979d5ed3e0d13824d17d694c79d2ece92fa","_cell_guid":"86570a36-5c20-460a-9dfd-2070548532a7","collapsed":true,"trusted":false},"cell_type":"code","source":"Y_train = train[\"label\"]\n\n# Drop 'label' column\nX_train = train.drop(labels = [\"label\"],axis = 1) \n\n# free some space\ndel train \n\ng = sns.countplot(Y_train)\n\nY_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7b1b1d36243c885e57374c8b60c5a7e10abe922","_cell_guid":"5aea4062-1790-4987-b739-c4bebd79030f"},"cell_type":"markdown","source":"We have similar counts for the 10 digits."},{"metadata":{"_uuid":"5d77934302869925c19128c77e247b3c8ca84d71","_cell_guid":"5b7d4b66-a140-4fcc-a889-bcef007c880a"},"cell_type":"markdown","source":"## 2.2 Check for null and missing values"},{"metadata":{"_execution_state":"idle","_uuid":"cdf27c27e2a5b15e6d7bfc70de7a18c08f3feb7a","_cell_guid":"ececaa00-2ae3-4d13-b631-438df085b030","collapsed":true,"trusted":false},"cell_type":"code","source":"# Check the data\nX_train.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"a0089bb7ec9aec76373db475399aea24699ae989","_cell_guid":"92572e80-8543-4e72-8767-5c9be8381b04","collapsed":true,"trusted":false},"cell_type":"code","source":"test.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0bee59691c2df0b275c78e38e7f9907d02ac038","_cell_guid":"09d04cae-4245-4659-85dd-ef48531da295"},"cell_type":"markdown","source":"I check for corrupted images (missing values inside).\n\nThere is no missing values in the train and test dataset. So we can safely go ahead."},{"metadata":{"_uuid":"2954681f25f0dcbe986e6914396cdbce61db591f","_cell_guid":"6812040d-80ad-43d2-a571-275f4f20067b"},"cell_type":"markdown","source":"## 2.3 Normalization"},{"metadata":{"_uuid":"0ecf4b52510ab7957d0d4eb646c0aa1ba5986273","_cell_guid":"159d5854-437a-4d0f-bc1e-fc3f7e43d178"},"cell_type":"markdown","source":"We perform a grayscale normalization to reduce the effect of illumination's differences. \n\nMoreover the CNN converg faster on [0..1] data than on [0..255]."},{"metadata":{"_execution_state":"idle","_uuid":"b5d4f8fcf2a967e2c7d57daedf95aa8c5ab7f8cb","_cell_guid":"cdc4340b-6e24-4e12-be99-ac806098ff17","collapsed":true,"trusted":false},"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a66741bf1ac597094f3a3166877008feef27c519","_cell_guid":"7413df94-bcb9-4f75-b174-c127d4445766"},"cell_type":"markdown","source":"## 2.3 Reshape"},{"metadata":{"_execution_state":"idle","_uuid":"f0a6ad80dab8e0f2c2e46165ccd9cd82dd162bc3","_cell_guid":"34b6a5f7-8fd2-4387-8ef4-c9dc19584fed","collapsed":true,"trusted":false},"cell_type":"code","source":"# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4fb5553e188d9956f5d8b3a5d275ab00ea667ce","_cell_guid":"8decd1ce-7b7e-431d-8458-eaca18e0e1f7"},"cell_type":"markdown","source":"Train and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values. We reshape all data to 28x28x1 3D matrices. \n\nKeras requires an extra dimension in the end which correspond to channels. MNIST images are gray scaled so it use only one channel. For RGB images, there is 3 channels, we would have reshaped 784px vectors to 28x28x3 3D matrices. "},{"metadata":{"_uuid":"39b7a31e843bac6b705461bcce89da216b91799e","_cell_guid":"bdb422e2-bdec-444f-97a5-283a1e54bf2c"},"cell_type":"markdown","source":"## 2.5 Label encoding"},{"metadata":{"_execution_state":"idle","_uuid":"cabefd1478d5c1bdfe57fd6a34395340916a854c","_cell_guid":"4b7f3e78-44dc-4561-b1f0-9429ee024cf4","collapsed":true,"trusted":false},"cell_type":"code","source":"# Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])\nY_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcfb688587dfc6feafd27442a3505e35dc01b82d","_cell_guid":"ae068bd8-b12e-4768-8a7e-0fc865dd7562"},"cell_type":"markdown","source":"Labels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])."},{"metadata":{"_uuid":"d8abbbf31483b94e1b29d07c4c8253d1311648a7","_cell_guid":"32152fc3-a570-4d64-8a7d-6c689a4acd33"},"cell_type":"markdown","source":"## 2.6 Split training and valdiation set "},{"metadata":{"_execution_state":"idle","_uuid":"6e51c925c6e0f1b936679c9649fef345c853555f","_cell_guid":"3a698301-9759-4279-ae48-fd980f89ea53","collapsed":true,"trusted":false},"cell_type":"code","source":"# Set the random seed\nrandom_seed = 2","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"b779ac76d8317647db92d5a88b4098d212d72884","_cell_guid":"dcd25ebb-d845-4d32-9867-082e352b1396","collapsed":true,"trusted":false},"cell_type":"code","source":"# Split the train and the validation set for the fitting\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=random_seed)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72ed54a305eebf80e8b8b0f7eabde1332a5a85a3","_cell_guid":"3c99964f-4500-4f1c-947d-c67e644e34db"},"cell_type":"markdown","source":"I choosed to split the train set in two parts : a small fraction (10%) became the validation set which the model is evaluated and the rest (90%) is used to train the model.\n\nSince we have 42 000 training images of balanced labels (see 2.1 Load data), a random split of the train set doesn't cause some labels to be over represented in the validation set. Be carefull with some unbalanced dataset a simple random split could cause inaccurate evaluation during the validation. \n\nTo avoid that, you could use stratify = True option in train_test_split function (**Only for >=0.17 sklearn versions**)."},{"metadata":{"_uuid":"60eed15ec5bc0d354385301789ecb8538fc02267","_cell_guid":"adbeacf0-0dc0-4675-b2df-9c9663750f32"},"cell_type":"markdown","source":"We can get a better sense for one of these examples by visualising the image and looking at the label."},{"metadata":{"_execution_state":"idle","_uuid":"e0dae8943d3d35f075dba3d7ba31bde1d4bf2ff4","_cell_guid":"5f76131b-4ba0-45f1-a98c-bd4e7d561793","collapsed":true,"trusted":false},"cell_type":"code","source":"# Some examples\ng = plt.imshow(X_train[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5fa18b37a9acd9e098bac1d12264b0dd4310fdd3","_cell_guid":"d5265777-aeb3-449d-b171-d88cad74c0a4"},"cell_type":"markdown","source":"# 3. CNN\n## 3.1 Define the model"},{"metadata":{"_uuid":"7697570491420f957f6e4d3569d51410b5277250","_cell_guid":"504fa00e-148c-4364-9b68-218b3aaedfdb"},"cell_type":"markdown","source":"I used the Keras Sequential API, where you have just to add one layer at a time, starting from the input.\n\nThe first is the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 32 filters for the two firsts conv2D layers and 64 filters for the two last ones. Each filter transforms a part of the image (defined by the kernel size) using the kernel filter. The kernel filter matrix is applied on the whole image. Filters can be seen as a transformation of the image.\n\nThe CNN can isolate features that are useful everywhere from these transformed images (feature maps).\n\nThe second important layer in CNN is the pooling (MaxPool2D) layer. This layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important. \n\nCombining convolutional and pooling layers, CNN are able to combine local features and learn more global features of the image.\n\nDropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting. \n\n'relu' is the rectifier (activation function max(0,x). The rectifier activation function is used to add non linearity to the network. \n\nThe Flatten layer is use to convert the final feature maps into a one single 1D vector. This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers. It combines all the found local features of the previous convolutional layers.\n\nIn the end i used the features in two fully-connected (Dense) layers which is just artificial an neural networks (ANN) classifier. In the last layer(Dense(10,activation=\"softmax\")) the net outputs distribution of probability of each class."},{"metadata":{"_execution_state":"idle","_uuid":"f7991ef6871a26f9fa57acdcd460a69bab53e804","_cell_guid":"1e0f3f88-2ad7-459e-8e02-aecc5f3511ae","collapsed":true,"trusted":false},"cell_type":"code","source":"# Set the CNN model \n# my CNN architechture is In -> [[Conv2D->relu]*2 -> MaxPool2D -> Dropout]*2 -> Flatten -> Dense -> Dropout -> Out\n\nmodel = Sequential()\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation = \"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7065512cf892ba49f0c06e239e3a20ff13667ef2","_cell_guid":"0e26b6a3-ef9d-4ae5-9d79-d5e4b073251a"},"cell_type":"markdown","source":"## 3.2 Set the optimizer and annealer\n\nOnce our layers are added to the model, we need to set up a score function, a loss function and an optimisation algorithm.\n\nWe define the loss function to measure how poorly our model performs on images with known labels. It is the error rate between the oberved labels and the predicted ones. We use a specific form for categorical classifications (>2 classes) called the \"categorical_crossentropy\".\n\nThe most important function is the optimizer. This function will iteratively improve parameters (filters kernel values, weights and bias of neurons ...) in order to minimise the loss. \n\nI choosed RMSprop (with default values), it is a very effective optimizer. The RMSProp update adjusts the Adagrad method in a very simple way in an attempt to reduce its aggressive, monotonically decreasing learning rate.\nWe could also have used Stochastic Gradient Descent ('sgd') optimizer, but it is slower than RMSprop.\n\nThe metric function \"accuracy\" is used is to evaluate the performance our model.\nThis metric function is similar to the loss function, except that the results from the metric evaluation are not used when training the model (only for evaluation)."},{"metadata":{"_execution_state":"idle","_uuid":"420c704367b397b8255fefe9d882b35ac8929b95","_cell_guid":"a4c55409-6a65-400a-b5e8-a1dc535429c0","collapsed":true,"trusted":false},"cell_type":"code","source":"# Define the optimizer\noptimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"b4c0288622227a3cd05479aa765e324dbb852f34","_cell_guid":"0bd89b79-1a74-40da-af9a-7a76a8b96ff2","collapsed":true,"trusted":false},"cell_type":"code","source":"# Compile the model\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4d89b29d55b328e8c446880505c54e674eeec38","_cell_guid":"6bc49be3-11e5-48fb-b70f-601aec0f0c86"},"cell_type":"markdown","source":"<img src=\"http://img1.imagilive.com/0717/learningrates.jpg\"> </img>"},{"metadata":{"_uuid":"bb90d48f4426d1baeb0404f2431e04e3cade59df","_cell_guid":"35b826b6-c516-47d0-8469-f8d04a102118"},"cell_type":"markdown","source":"In order to make the optimizer converge faster and closest to the global minimum of the loss function, i used an annealing method of the learning rate (LR).\n\nThe LR is the step by which the optimizer walks through the 'loss landscape'. The higher LR, the bigger are the steps and the quicker is the convergence. However the sampling is very poor with an high LR and the optimizer could probably fall into a local minima.\n\nIts better to have a decreasing learning rate during the training to reach efficiently the global minimum of the loss function. \n\nTo keep the advantage of the fast computation time with a high LR, i decreased the LR dynamically every X steps (epochs) depending if it is necessary (when accuracy is not improved).\n\nWith the ReduceLROnPlateau function from Keras.callbacks, i choose to reduce the LR by half if the accuracy is not improved after 3 epochs."},{"metadata":{"_execution_state":"idle","_uuid":"c4a5b4e462ec5362c47eef4fcc7956fd4e203307","_cell_guid":"b5987a18-6bbe-42a2-9d31-333ebc4f7af1","collapsed":true,"trusted":false},"cell_type":"code","source":"# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"26b0647c46efdb6b1096cf7335a7bf2a3417543a","_cell_guid":"970db455-b393-4b25-806d-92c6766c12c0","collapsed":true,"trusted":false},"cell_type":"code","source":"epochs = 1 # Turn epochs to 30 to get 0.9967 accuracy\nbatch_size = 86","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f24df64b223e0177c94025b6767ab19b722c5386","_cell_guid":"241a0f4f-13f5-4b13-be1e-4e3e4a714c06"},"cell_type":"markdown","source":"## 3.3 Data augmentation "},{"metadata":{"_uuid":"9e498b91419439f0fa791e595f202d9a0d56ad6b","_cell_guid":"e2d41e30-0724-40fb-a901-750e514ba9f9"},"cell_type":"markdown","source":"In order to avoid overfitting problem, we need to expand artificially our handwritten digit dataset. We can make your existing dataset even larger. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.\n\nFor example, the number is not centered \nThe scale is not the same (some who write with big/small numbers)\nThe image is rotated...\n\nApproaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more. \n\nBy applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model.\n\nThe improvement is important : \n   - Without data augmentation i obtained an accuracy of 98.114%\n   - With data augmentation i achieved 99.67% of accuracy"},{"metadata":{"_execution_state":"idle","_uuid":"d4e9e1ade3c04f9ca4d8cd44e799f9e09524d5a1","_cell_guid":"ade93d2d-90c6-4401-af95-f7d65f8c0a20","collapsed":true,"trusted":false},"cell_type":"code","source":"# Without data augmentation i obtained an accuracy of 0.98114\n#history = model.fit(X_train, Y_train, batch_size = batch_size, epochs = epochs, \n#          validation_data = (X_val, Y_val), verbose = 2)","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"21d6192c87d92d497c797656474bccd9cefc5647","_cell_guid":"b342befe-1a6f-44bf-8dab-28033a729122","collapsed":true,"trusted":false},"cell_type":"code","source":"# With data augmentation to prevent overfitting (accuracy 0.99286)\n\ndatagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51f16d0a5b9d9373438474e7defa7348359d7c18","_cell_guid":"22f80b8a-d4f6-4a34-b33d-ff7334f45d94"},"cell_type":"markdown","source":"For the data augmentation, i choosed to :\n   - Randomly rotate some training images by 10 degrees\n   - Randomly  Zoom by 10% some training images\n   - Randomly shift images horizontally by 10% of the width\n   - Randomly shift images vertically by 10% of the height\n   \nI did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.\n\nOnce our model is ready, we fit the training dataset ."},{"metadata":{"_execution_state":"idle","_uuid":"cf36b3d029f95b553be02d612e097a9769ee8252","_cell_guid":"b453af8d-9736-43e3-b486-7a1cd7dd8909","collapsed":true,"trusted":false},"cell_type":"code","source":"# Fit the model\nhistory = model.fit_generator(datagen.flow(X_train,Y_train, batch_size=batch_size),\n                              epochs = epochs, validation_data = (X_val,Y_val),\n                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n                              , callbacks=[learning_rate_reduction])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a1834f2a9f2db15dcaba4a84004b9627d714469","_cell_guid":"e758621d-b27b-40ff-a93f-bebd2e0e5243"},"cell_type":"markdown","source":"# 4. Evaluate the model\n## 4.1 Training and validation curves"},{"metadata":{"_execution_state":"idle","_uuid":"3a831860dd5bb65c8ead1ddcf4ae18ae20dd7f3e","_cell_guid":"eb4b1b73-cbd4-40e8-9790-066fcef4c4c0","collapsed":true,"trusted":false},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba77212f82b781b0e5a4e494897b18c31c5e30a3","_cell_guid":"69df90d7-f2fd-497d-a0b2-57c497a3e645"},"cell_type":"markdown","source":"The code below is for plotting loss and accuracy curves for training and validation. Since, i set epochs = 2 on this notebook .\nI'll show you the training and validation curves i obtained from the model i build with 30 epochs (2h30)"},{"metadata":{"_uuid":"9f2920cbbfb6f1f566fac4afdd4030c75ee5bf66","_cell_guid":"bacff684-79bb-4ede-83c1-ccb7cf92df77"},"cell_type":"markdown","source":"<img src=\"http://img1.imagilive.com/0717/mnist_099671_train_val_loss_acc.png\"></img>\n\nThe model reaches almost 99% (98.7+%) accuracy on the validation dataset after 2 epochs. The validation accuracy is greater than the training accuracy almost evry time during the training. That means that our model dosen't not overfit the training set.\n\nOur model is very well trained  !!! \n\n<img src=\"http://img1.imagilive.com/0717/accuracies1de.jpg\"/>"},{"metadata":{"_uuid":"63698d7d51381b33892ce164b0f21930abb3e937","_cell_guid":"de1c65bd-4a88-4351-9f4b-562e72e7e0fd"},"cell_type":"markdown","source":"## 4.2 Confusion matrix"},{"metadata":{"_uuid":"3306d29b732341663e50866140dc569360701a81","_cell_guid":"5688faa0-b33b-4e92-b125-7fa0b37e7df3"},"cell_type":"markdown","source":"Confusion matrix can be very helpfull to see your model drawbacks.\n\nI plot the confusion matrix of the validation results."},{"metadata":{"_execution_state":"idle","_uuid":"16e161179bf1b51ba66c39b2cead883f1db3a9c7","_cell_guid":"11361e73-8250-4bf5-a353-b0f8ea83e659","collapsed":true,"trusted":false},"cell_type":"code","source":"# Look at confusion matrix \n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ecb928433299b163ecc1f6c4e66d4ddcf38fe898","_cell_guid":"1b8a5cdc-9122-4e31-b9fa-0f6b57d33fc8"},"cell_type":"markdown","source":"Here we can see that our CNN performs very well on all digits with few errors considering the size of the validation set (4 200 images).\n\nHowever, it seems that our CNN has some little troubles with the 4 digits, hey are misclassified as 9. Sometime it is very difficult to catch the difference between 4 and 9 when curves are smooth."},{"metadata":{"_execution_state":"idle","_uuid":"afd59cae1115188b77abd3471e5e89790cef80a0","_cell_guid":"ef54d686-6f79-4d96-a5a0-a64657bd742e"},"cell_type":"markdown","source":"Let's investigate for errors. \n\nI want to see the most important errors . For that purpose i need to get the difference between the probabilities of real value and the predicted ones in the results."},{"metadata":{"_execution_state":"idle","_uuid":"e7a3d6449b499a29db224e42e950f21ca1ec4e36","_cell_guid":"7b0f31b8-c18b-4529-b0d8-eb4c31e30bbf","collapsed":true,"trusted":false},"cell_type":"code","source":"# Display some error results \n\n# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\ndef display_errors(errors_index,img_errors,pred_errors, obs_errors):\n    \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n    n = 0\n    nrows = 2\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n            ax[row,col].set_title(\"Predicted label :{}\\nTrue label :{}\".format(pred_errors[error],obs_errors[error]))\n            n += 1\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[-6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"afc408bd5545a6a2b2d4e04989890546263cb642","_cell_guid":"d21c4caf-e699-4647-8ef4-e60e868607ae"},"cell_type":"markdown","source":"The most important errors are also the most intrigous. \n\nFor those six case, the model is not ridiculous. Some of these errors can also be made by humans, especially for one the 9 that is very close to a 4. The last 9 is also very misleading, it seems for me that is a 0."},{"metadata":{"_execution_state":"idle","_uuid":"7f17e7bf0a54a01a52fef2d554780f6bc6580dc6","_cell_guid":"05ff3b9f-c3bb-4cec-a8c2-2c128e8f15b3","collapsed":true,"trusted":false},"cell_type":"code","source":"# predict results\nresults = model.predict(test)\n\n# select the indix with the maximum probability\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","execution_count":null,"outputs":[]},{"metadata":{"_execution_state":"idle","_uuid":"369dfaab09240f3f12bcff91953ffd315ab84985","_cell_guid":"b5f1f39f-13b8-439a-8913-0f120e3d47a9","collapsed":true,"trusted":false},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_mnist_datagen.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ec3bcf47cd49afea61aa9ed60ced64de8fc2d3d","_cell_guid":"b1c2790e-4313-4b47-937a-b4eaaae6d845"},"cell_type":"markdown","source":"\n\n** you found this notebook helpful or you just liked it , some upvotes would be very much appreciated - That will keep me motivated :)**\n"}],"metadata":{"kernelspec":{"language":"python","name":"python3","display_name":"Python 3"},"language_info":{"pygments_lexer":"ipython3","mimetype":"text/x-python","nbconvert_exporter":"python","version":"3.6.1","codemirror_mode":{"version":3,"name":"ipython"},"file_extension":".py","name":"python"}},"nbformat":4,"nbformat_minor":1}