{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"},"cell_type":"markdown","source":"## Computer Science II\n### Assignment C: Classifying handwritten digits using a Convolutional Neural Network\n\n- Introduction video to Neural Networks [here](https://www.youtube.com/watch?v=OT1jslLoCyA&index=1&list=PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU) and another one [here ](https://www.youtube.com/watch?v=bxe2T-V8XRs&t=6s)\n\n\n  \n  - We will use the MNIST problem dataset for recognizing handwritten digits.\n  - Every image is a 28 by 28 pixel square (784 pixels total).\n  - Thus, this is a digit recognition task, there are 10 classes to predict (0-9).\n  - the data files contain gray-scale images.\n  - the training data set has 785 columns, the first column called 'label' is the digit that is drawn by the user, the rest of the columns contain the pixel values of the associated image.\n  - The dataset can be found [here](http://yann.lecun.com/exdb/mnist/)\n  \n  \n![](https://i2.wp.com/dataaspirant.com/wp-content/uploads/2017/05/Mnist-database-hand-written-digits.png?w=530)  \n  \n  \n  "},{"metadata":{"collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0"},"cell_type":"markdown","source":"## Step 1: Setting up the environment\n\nWe first have to import the dataset as well as all the other librarier we will be using.\nFor this can use Keras as a very convenient way to import the dataset since it is build in.\n\nThe MINST dataset has 70.000 images in total:\n\n- 60.000 training examples\n- 10.000 test examples"},{"metadata":{"_uuid":"5840f6acd4b3c741f521b7f62c27b6ad93fc53eb","_cell_guid":"ca22f43f-d177-4ef4-a53f-531b44d50b7c"},"cell_type":"markdown","source":"### What is Keras?\n\nKeras is an open source neural network library API, written in python and capable of running on top of TensorFlow, CNTK, or Theano, which are libraries or frameworks designed to be used for machine learning.\nKeras was developed with a focus on making it possible to build fast ML models without having to stress about implementation. It enables fast experimentation with deep neural networks, and focuses on being user-friendly, modular, and extensible.\n"},{"metadata":{"collapsed":true,"_uuid":"fadec5cbed8232d95e5d5c3218cee1f1f8ba8800","_cell_guid":"136273e6-819c-4827-bcab-86309fbeb29d","trusted":false},"cell_type":"code","source":"# import all libraries \nimport pandas as pd # for displaying and importing the data\nimport numpy as np # for the linear algebra (matrix operations)\nimport matplotlib.pyplot as plt # plotting\nimport matplotlib.image as mpimg # image plotting\nimport seaborn as sns # to make our plots pretty\n%matplotlib inline\n\nnp.random.seed(2) # setting the starting parameters\n\nfrom sklearn.model_selection import train_test_split # used to split training set in validation and training set\nfrom sklearn.metrics import confusion_matrix # to make the confusion matrix\nimport itertools # for efficient iteration\n\n# using keras \nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential # to build a NN\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D # methods used from the keras framework\nfrom keras.optimizers import SGD # Stochastic Graidient Descent\nfrom keras.callbacks import ReduceLROnPlateau # annealing LR\n\n# Seaborn specifications\nsns.set(color_codes=True) \nsns.set(style='white', context='notebook')\nsns.set_palette(\"Blues\",10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d55c0d4bb7414ba9d778d900c18ef08cd1755b4f","_cell_guid":"9c682b85-0719-4e06-97b5-4c19a31fd647"},"cell_type":"markdown","source":"### A note on random.seed\n\nIn Python, the random.seed function generates “random numbers.” However, note that random numbers are not truly random. The numbers generated are **pseudorandom**, meaning the numbers are generated by a complicated formula that makes it look random. In order to generate numbers, the formula takes the previous value generated as its input. If there is no previous value generated, it often takes the time as a first value.\nThat is why we seed the generator — to make sure that we always get the same random numbers. We provide a fixed value that the number generator can start with, which is 2 in this case."},{"metadata":{"_uuid":"cf1e5592ec94b6d1a6a7f50a0e3287fa2d90c79d","_cell_guid":"d31d4b5d-0cf0-456a-9baf-02d6c02d3cc2"},"cell_type":"markdown","source":"### GPU's vs CPU's\n\nWhen you are training models on big datasets, you want to use a GPU. A GPU, which stands for Graphics Processing Unit, is different than a CPU, a Computer Processing Unit. For normal uses, a CPU is perfect. But GPU's are great for doing a lot of the same operations. And because during the training process we are doing tons of matrix operations, using a GPU is way faster than when you would use a CPU. \n\n*Analogy*\n\nA CPU is like an Opel Astra, its  quite fast and a nice car to drive in. However, when you want to transfer a lot of freight from A to B, your car would look like the image below, not very practical. Instead, you would better use a mini-truck (GPU) that can transport a lot at the same time. In that way, you'll be much faster. \n\n\n![](https://www.motogoloco.com/blog/wp-content/uploads/2014/07/car-full-of-stuff.jpg)\n\n\n[here](https://i2.wp.com/www10.mcadcafe.com/blogs/jeffrowe/files/2017/03/CPU-and-GPU.png?w=925&ssl=1) is a neat overview on the exact differences"},{"metadata":{"_uuid":"77a068af3b2169925d0aecd087d326a955e1c033","_cell_guid":"a9f9eb85-33e3-44e7-a78a-1dd074c77e4b"},"cell_type":"markdown","source":"## Step 2: Loading and Preparing the Data; Feature Engineering\n\nIn datascience the quality of your data very important. \n\nBefore you want to start building and training the model, you want to know how your data looks like and if there are things that you should change."},{"metadata":{"collapsed":true,"_uuid":"93259934e48074f0f931329369dd83a998d8afd2","_cell_guid":"24bc0e28-3201-4c32-92f2-6318d90aaf38","trusted":false},"cell_type":"code","source":"# First we load the data using pandas\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b074e4f1a9267bb86961598f94a821e4a27c5273","_cell_guid":"b8f1c7c1-2573-4d9c-98db-4b34d1e77404"},"cell_type":"markdown","source":"### 2 A) Investigate the distribution of the data\n\nWe imported two datasets, train and test. The first column of both of the datasets, called \"label\" is the digit that was drawn (i.e. the true label). The rest of the columns are the pixel values \n\n\nSince we will be classifying handwritten digits, its important to know how many examples of each digit we have in our database. \n\nIf there is an unequal distribution of images, it might impact the results of our model."},{"metadata":{"collapsed":true,"_uuid":"9e632a8472f2e8623e46323ee9f46db10ddc093a","_cell_guid":"23ef5504-4ab3-42af-a17a-5037325a36c0","trusted":false},"cell_type":"code","source":"Y_train = train[\"label\"]\n\n# Drop 'label' column\nX_train = train.drop(labels = [\"label\"],axis = 1) \n\n# free some space\ndel train \n\ng = sns.countplot(Y_train)\n\nY_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"ad02d31cc59cc67e398337b52378696c8820e4aa","_cell_guid":"f54c6aaf-0a13-4d70-bdf2-d29c2d6c4d6e"},"cell_type":"markdown","source":"Now that we know that all data is about equally distributed, we can proceed with confidence."},{"metadata":{"_uuid":"8b7dd55e8688dcdea1ac414b6d148dcf6fdaceee","_cell_guid":"aad12082-5e25-4777-881c-1eab4ad4013e"},"cell_type":"markdown","source":"### 2 B) Normalising the data\n\nYou want to make sure that your data is in the same scale / form to increase the integrity of the data.\nWe call this normalising the data. We want the data to be on a scale from 0 to 1.\n\nIf we would not Normalise our data, larger data points in the data can cause instability in our Neural Network.\n"},{"metadata":{"collapsed":true,"_uuid":"f052de25dfe6e58d76c04fcb44683f6662ab7fb9","_cell_guid":"5d08805d-8693-40be-a1f9-f048db324558","trusted":false},"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"829c0550b4ec12bc9d6a47cd158ab20c57062f97","_cell_guid":"14a87bc8-3581-4fb1-a34e-2881c05ba8fa"},"cell_type":"markdown","source":"### 2 C) Reshape \n\nIn the MNIST dataset, each entry in the set is a one dimensional vector with 784 elements. This is created by taking the 2D matrix of pixels, and flattening it, so that it's just one long row. \n\nThe input to a CNN has to be a 4D tensor, the dimensions of this tensor should be: (number of images) x (height in pixels) x (width in pixels) x (number of color channels).\nThe input has to be 4D because Convolutional filters are 2D sliding windows which have to go through each channel of each image, and slide over the images in a 2D plane to make their calculations.\nIn the MNIST dataset, the images should be 28 pixels by 28 pixels by 1 color channel (instead of 3 for colour images). So we **reshape** the 1D vector of 784 pixel values to match the dimensions that can go into the CNN."},{"metadata":{"collapsed":true,"_uuid":"fa1c69a931da50a4ef7aa655a66e6ead75edadd8","_cell_guid":"9abcd37d-4e42-49c3-ba3e-f5c8289dae8a","trusted":false},"cell_type":"code","source":"# Reshape image to 3 dimensions (height = 28px, width = 28px , canal = 1)\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bebbeb157fb2269be5814a97df3479b2bfa458dc","_cell_guid":"8c8aa417-5dc3-400d-8dee-2330785b73e7"},"cell_type":"markdown","source":"### 2 D) Encode the Labels\n\nMachine learning algorithms can not work with categorical data directly, the data must be converted to numbers first. We call the process of representing categorical data as binary (0/1) vectors **one hot encoding**.\n\nThe process is as follows:\n- We make a table with a column for each category\n- the value in the column is either 1 (if it is) or 0 (if not)\n\nExplanatory video [here](https://www.youtube.com/watch?v=v_4KWmkwmsU)\n\n![Example table](https://i.imgur.com/mtimFxh.png)\n"},{"metadata":{"collapsed":true,"_uuid":"4c812581fd47e4ed196f8c833ef969bac7efe0ed","_cell_guid":"79423a79-07f5-4cd2-832e-6e9daf49c85f","trusted":false},"cell_type":"code","source":"# Now we encode the categorical labels to one hot vectors (e.g. : 3 -> [0,0,0,1,0,0,0,0,0,0])\nY_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f4403f01cbc185504d2e514549209f4c76ed240","_cell_guid":"aadd6933-5f5c-42e3-941c-51f501daef2c"},"cell_type":"markdown","source":"### 2 E) Split Training Set to Train and Validation Set\n\nIn Machine Learning, we always split our dataset into 3 sets, a training, validation and test set. \n\n- **Training set**: this is the set you use to train the model over and over again.\n- **Validation set**: a set of data you use to validate the model during training. The validation may give us information that can help us with adapting hyperparameters. \n- **Test set**: the set used to test the model after it has been trained.\n\n![How we divide our dataset](https://dziganto.github.io/assets/images/train-validate-test.png?raw=true)\n\nSince our dataset is now divided into two sets: a training and test set, we will separate the training set into a train and validation set. 90% of the data will be used to train and 10% to validate the model.\n\n*Why do we do this?*\n\nThe main reason is to **avoid overfitting** which is when our model focuses to much on the *specific images* in our training set. If it gets too adapted (too fitted) it is not able to generalise on other datasets.\nHence the name \"overfitting\", the model simply is too fitted to the data it learned from.\n\n*Analogy*\n\nA student is studying for an exam, it is training (*training set*) on several homework exercises in the book. After making his weekly exercises, the student tests his understanding of the materials by making the weekly quizzes (*validation set*) after many weeks (epochs) of learning, the student is examined with an exam (*test set*) which reveals its true understanding of the subject at hand. The student has an idea on how well it is prepared even before it did the exam through the weekly quizzes. If they went well each week, the student is likely to have a good score on the final as well. \nHowever, if the student only focuses too much on the same exercises in the book, it will not do well on the final, as the student won't be able to **generalise** its knowledge to different problems.\n"},{"metadata":{"collapsed":true,"_uuid":"5f284e6a28f65b0c7246bbf672bd466b1d005442","_cell_guid":"6d7f7c14-db88-43eb-bed4-ecbf7ac92d23","trusted":false},"cell_type":"code","source":"# Set random seed\nrandom_seed = 2","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"f3dec4c9685f45e8792f9c43451e4a6980c57b05","_cell_guid":"4b2f6d97-45ea-4ffe-a4dc-540509c03629","trusted":false},"cell_type":"code","source":"# Split the training set to train and validation\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=random_seed)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90b17745e8a1fa7ff8b730c53f706a4025c9976c","_cell_guid":"2327ba93-eaf5-4153-8657-8c95154e4bd5"},"cell_type":"markdown","source":"### 2 F) Plot examples from the dataset\n\nIt is always good to display some images and get familiar with the dataset.\n\nChange the number after *X_train[ ]* to display a different image from the dataset, the number represents the nth number in the dataset. "},{"metadata":{"collapsed":true,"_uuid":"b63263316c54f5dac34197c85069509cbed09981","_cell_guid":"7a1396a3-151d-42c3-a806-95cd8d7d2083","trusted":false},"cell_type":"code","source":"# example images\nimage = plt.imshow(X_train[6][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c85cea348e98a74ab0b2d49152ecba2aee805a0f","_cell_guid":"5873d55d-d88f-4b70-87cb-50ed4d4a9bc1"},"cell_type":"markdown","source":"## Step 3: Building the Model\n\n### 3 A) Define the Network\n\nNow it is time to build our Neural Network, we are using the[ Keras framework](https://keras.io/getting-started/sequential-model-guide/) to do this.\n\n\n- for building a Neural Network in Keras we use the Sequential API, which means that we just build up our model layer by layer. \n\nWhen using Keras to build a model, you go through these steps:\n\n![](https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2016/07/5-Step-Life-Cycle-for-Neural-Network-Models-in-Keras.png)"},{"metadata":{"_uuid":"61163c49cfd8fb90ab51d4afb809617c49639dff","_cell_guid":"3d338881-d9e7-4d51-9295-40bbd5ac18f3"},"cell_type":"markdown","source":"### Convolutional Neural Networks\n\nBecause we are dealing with image data, we will use a Convolutional Neural Network. Daphne wrote [this](https://medium.freecodecamp.org/an-intuitive-guide-to-convolutional-neural-networks-260c2de0a050) article about the basics.\n\nOn a very high level, this is what happens in a CNN. If you want to know exactly how it works, we would recommend to read the article.\nBut in general, there have two phases, a feature extraction part (these are the convolutional layers) and a classification part (these are fully connected layers).\n\n![](https://cdn-images-1.medium.com/max/800/1*tJ1Rkl5xw_5izEZXmNfh5Q.png)\n\n"},{"metadata":{"_uuid":"65f6cd4990e73774373c3e946f7201b14c4d04ec","_cell_guid":"acfd7683-d450-4dbb-b1ca-df67002041fc"},"cell_type":"markdown","source":"This is the architecture of our model:\n\n![](https://scontent-amt2-1.xx.fbcdn.net/v/t1.0-9/32858355_2004117536327614_4250562419400638464_n.jpg?_nc_cat=0&oh=760543b80489acd2848bed2c5b077c59&oe=5B82B65A)"},{"metadata":{"collapsed":true,"_uuid":"cdb501018ed707d8d3154d0260d457d00ec71bc7","_cell_guid":"ae61ccb7-7c09-4c37-8498-3761d08b91ef","trusted":false},"cell_type":"code","source":"model = Sequential() # to build the NN\n\n# 2 Conv Layers, followed by the relu activation function and maxpooling\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(MaxPool2D(pool_size =(2, 2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(MaxPool2D(pool_size =(2, 2))) \nmodel.add(Dropout(0.25))\n\n# flatten the data to 1 dimensional data\nmodel.add(Flatten())\n\n# Fully connected layer 1\nmodel.add(Dense(1000, activation='relu'))\n\n# Fully connected layer 2, this is the output layer\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.summary() # give use a neat summary of the model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f62dbecc1dcbe6c188c756d79df4b91147b758b","_cell_guid":"30d7d36c-b7b3-4297-a3dc-fefa9ce3e159"},"cell_type":"markdown","source":"## Step 4: Training the Model"},{"metadata":{"collapsed":true,"_uuid":"90e51da2f66be84f656255e5ecc8e82ed8e88cbe","_cell_guid":"271b0219-5133-46ea-a0fb-d81386554e4a"},"cell_type":"markdown","source":"### 4 A) Scores, Optimisation and Loss \n\nTo measure how our network is doing, we need to understand two important concepts:\n\n- The **Loss function** is a function that looks at the scores (S) and tells us how bad quantitatively the weights are. In a way it quantifies our unhappiness with the score across the training data. The loss can also be described by the error rate between the prediction our model made and the true label.\n\n- **Optimisation** is basically a function that will try to minimize the loss as much as possible. A nice analogy is a person (i.e. the algorithm) walking down a mountain, it doesn't know where exactly the valley is, but could still get to the valley by taking a step, feel with his feet, and take a step again. If the person would iterate this many times it will eventually reach the bottom.\n\n![Illustration made by me](https://cdn-images-1.medium.com/max/800/1*djBjys3plxcVGo8AFVasMA.png)\n\n\nNow we set up the model itself, the next step is to construct a score, a loss function and an optimization algorithm."},{"metadata":{"_uuid":"7d5e8e0e39ee6c1d210852522004e9e42ff00582","_cell_guid":"a5fafe55-6a5e-4ad0-b482-2cd011bbea5b"},"cell_type":"markdown","source":"\n- For the loss function we choose the **Categorical Cross Entropy** calculation, which is the sum of the true labels multiplied by the natural log of the predicted probabilities.\n\n![](https://cdn-images-1.medium.com/max/800/0*YRAt7P06fL7TObX-.png)"},{"metadata":{"_uuid":"9cff338986e15365df92287a49e508674a86d944","_cell_guid":"79a64ca5-71e5-466e-b8f9-1d0c50763c3b"},"cell_type":"markdown","source":"- As optimisation function we choose the 'sgd' from Keras. This is because we are currently most familiar with this optimizer. \n**Stochastic Gradient Descent**, sometimes reffered to as incremental gradient descent is an interative, stochastic (i.e. containing randomness) approximation of the gradient descent optimisation.\n\n- A **metric ** is the function that will be used to judge how our model is doing, but only for evaluation of the model, not for training. We use the **accuracy** function. "},{"metadata":{"collapsed":true,"_uuid":"a9e8148c0deeccb7c633e733e5382a06881acb6d","_cell_guid":"f47108e4-3bf7-4545-a66b-539bb46b8603","trusted":false},"cell_type":"code","source":"# First we define the optimizer we will use \noptimizer = SGD(lr=0.01, momentum=0.0, decay=0.0, nesterov=False)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"43bb010e50680e67a105504fc79b9846fb0f61ff","_cell_guid":"e805122e-8bc1-478e-878b-fd293454b3de","trusted":false},"cell_type":"code","source":"# Then compile the model (note: step 2 in the Keras workflow process)\n# in this part we configure the learning process\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc92f71be2e5073d29417895c4b355775448f69b","_cell_guid":"5fca1806-5416-4a5a-93e1-5e53e1557ce3"},"cell_type":"markdown","source":"We want the model to get to the global minimum as fast as possible. We can use learning rate annealing which slows down the learning rate as it gets closer to the minimum.\n\nThe ReduceLROnPlateau (from Keras.callbacks) method reduces the learning rate when a metric has stopped improving. \nThis can increase the accuracy of our model since models often benefit from reducing the learning rate once learning stagnates.\nWe choose to reduce the LR by 0.5 if the accuracy is not improved after 3 epochs."},{"metadata":{"collapsed":true,"_uuid":"482b43d693f9d6d262830077db19f6818c041a5b","_cell_guid":"36bafa27-6585-48cd-9126-4f770b0b3d61","trusted":false},"cell_type":"code","source":"# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0907d422fcbbe612343e9d3be188f56665a56aef","_cell_guid":"9866f8df-81d8-443b-90d2-4ddcd77eb134"},"cell_type":"markdown","source":"Bactches and Epochs\n\n- The** batch size** is the number of samples that are passed through the network at one time. Or in other words, the number of training examples in one epoch. We train our network in batches rather than by everything at once for better performance and because it is easier for the computer to process.\n- An **epoch** is complete pass (so forward and backward) through the dataset at hand. It should not be confused with an iteration, which is just one update of the neural net model's parameters"},{"metadata":{"collapsed":true,"_uuid":"e3eeca0f2bd085c0d7cc3fe8332a66b681c1b1ed","_cell_guid":"10f73d2d-305c-4983-a6e3-ea67a87b1828","trusted":false},"cell_type":"code","source":"epochs = 10\nbatch_size = 16","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"71f1bdb50dfd6e0f98c12622c1a1b3aec4e117a0","_cell_guid":"3fea90b6-1eab-42b6-a2e7-371f5d38f45a"},"cell_type":"markdown","source":"### 4 B) Start Training"},{"metadata":{"collapsed":true,"_uuid":"adce7ca2af97ad1971458e7f65e815a219713f14","_cell_guid":"b1175ccb-0fcf-4b97-b76f-e7069d241bae","trusted":false},"cell_type":"code","source":"# before training we want to specify the batch sizes the network will be trained on and the epochs\n# model.fit(data, labels, epochs, batch_size)\nhistory = model.fit(X_train, Y_train, batch_size = batch_size, epochs = epochs, \n                    validation_data = (X_val, Y_val), verbose = 2)\n# Lets train!","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d1f778c87daa55072f422ada17445668049cb6e","_cell_guid":"24049676-cef0-4f73-a159-f0be006dda71"},"cell_type":"markdown","source":"As you can see, after 10 epochs we have an accuract of 99.0% !"},{"metadata":{"_uuid":"d82d67c0a61fe853f65adc28dd21e0597a4de001","_cell_guid":"012240ee-5e7b-4dff-8563-d7119dd07768"},"cell_type":"markdown","source":" ## Step 5: Evaluate the Model\n\n\n\nSo we have just trained the model, lets review what it actually means for a model to \"learn\".\n\nWhen the model is first initialised it is set with random weights, the model will spit out the output for a given input at the end of the network and then compute the **loss or error** of that output by looking at what the model predicted for that input, versus what the inputs true label is.\n\nNext, the model will compute the **gradient** (slope) of the loss function with respect to every single weight that has been set. Once it has computed this value, the value will by multiplied (ie the dot product will be taken) by the learning rate. The** learning rate** can be seen as the *step size* by which we take go down the slope of the loss function and is usually a value like 0.001. \n\nThe output of   *d(loss) / d(weight * LR*  is now the input for the next layer in the NN. This process happens for each of the weights in the model for every time data passes through. \n\nSince all of these weights are continuously updated with each epoch, the weights are getting closer and closer to their optimised values while Stochastic Gradient Descent works to minimise the loss function.\n\n*This process of updating the weights is what is meant when a model is learning*. The model is learning what values to assign to each **weight **based on how these cumulative changes are affecting** loss function**.\n\n\n![](https://www.embedded-vision.com/sites/default/files/technical-articles/CadenceCNN/Figure2.jpg)\n\n"},{"metadata":{"_uuid":"e57cf2bc0364d1b4784555615551ba66292f5e63","_cell_guid":"ca1de7d8-aae7-4d37-a251-092728cc458a"},"cell_type":"markdown","source":"We want the training loss always above our validation loss or our validation accuracy above out training accuracy, otherwise it means our model is overfitting, we can use this image from the fast.ai course as a guide, this is what we aim for. \nBecause remember, **the point of machine learning is making a model that can generalise.**\n\n![Guideline](http://img1.imagilive.com/0717/accuracies1de.jpg)"},{"metadata":{"_uuid":"ce22d14aa334c6bdc13146cabe0dbc2ee6824f05","_cell_guid":"ae1e0442-f473-4b91-b935-e6cd31e90ca2"},"cell_type":"markdown","source":"### 5 A) Plot the training/ validation loss and accuracy"},{"metadata":{"collapsed":true,"_uuid":"21b01503a17397446969062766af53b586a12a01","_cell_guid":"4645df9c-3e84-4af0-a876-06e9c003e7e1","trusted":false},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='r', label=\"Training loss\") # a blue line for the Training loss\nax[0].plot(history.history['val_loss'], color='b', label=\"Validation loss\",axes =ax[0]) # red for the validation loss\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='r', label=\"Training accuracy\") \nax[1].plot(history.history['val_acc'], color='b',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b1b9da6191c1ad61604656b322306612bc13830","_cell_guid":"bd177c71-86b9-4bed-bb3f-83c455ac5b10"},"cell_type":"markdown","source":"You can see in the graphs that there is no overfitting, the validation accuracy is about the same as the training accuracy."},{"metadata":{"collapsed":true,"_uuid":"bb9dd6bbb8c6b5a1148e9dc6e1b26f4ac7bc5d55","_cell_guid":"22f432a4-e1e5-4b26-bda8-8b6c0cb78c4f"},"cell_type":"markdown","source":"### 5 B) Plot the Confusion Matrix\n\nTo get a better understanding of the performance of our model, we plot a confusion matrix. A confusion matrix is basically a technique to summarize how well our model predicts the images it has never seen before, it visualises the performance. It will give us a better understanding of the exact classification mistakes it made.\n\nReffering back to the student analogy, this is were the student reviews the exam and carefully looks at his mistakes. It is an important step to get insight in why we (the model) made certain mistakes.\n"},{"metadata":{"collapsed":true,"_uuid":"22cade466f2be63f2123b4e8b2f6d4efed8dc216","_cell_guid":"b2d7d986-8d2e-4be0-8f5c-460625e94c33","trusted":false},"cell_type":"code","source":"# Plot the confusion matrix using the scikit learn library\n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Oranges):   \n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n\"\"\" Compute the Confusion Matrix \"\"\"\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\n# range is the number of classes we have\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7656d0938885f94c8dd4180f12decfa9635d947e","_cell_guid":"776a3b91-9b2f-4b87-a6d4-70bf68a7c029"},"cell_type":"markdown","source":"### 5 C) Show some correct images\n\nLets look at some examples of correctly classified images."},{"metadata":{"collapsed":true,"_uuid":"8ed0df3c2cc978eee9250e2ff5de70db79e5f931","_cell_guid":"3cfb9cfb-c211-4f80-97ab-ae10e8987f45","trusted":false},"cell_type":"code","source":"# Lets display some correct results\n\n# if there is an error, or in other words if the difference between the prediction and the true label > 0\ncorrect = (Y_pred_classes - Y_true == 0) # define what is correct\n\n# specify the correct ones from each set\nY_pred_classes_correct = Y_pred_classes[correct]\nY_pred_correct = Y_pred[correct]\nY_true_correct = Y_true[correct]\nX_val_correct = X_val[correct]\n\ndef display_correct(correct_index,image_correct,pred_correct, obs_correct):\n    \"\"\" showing 6 images that were correctly classified\"\"\"\n    n = 0\n    nrows = 2\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n    for row in range(nrows):\n        for col in range(ncols):\n            correct = correct_index[n]\n            ax[row,col].imshow((image_correct[correct]).reshape((28,28)))\n            ax[row,col].set_title(\"Predicted label - {}\\n True label - {}\".format(pred_correct[correct],obs_correct[correct]))\n            n += 1\n            \n# Probabilities of the wrong predicted numbers\nY_pred_correct_prob = np.max(Y_pred_correct,axis = 1) # np.max flattens the nD array and returns the maximum\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_correct = np.diagonal(np.take(Y_pred_correct, Y_true_correct, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_correct = Y_pred_correct_prob - true_prob_correct\n\n# Sorted list of the delta prob errors\nsorted_delta_correct = np.argsort(delta_pred_true_correct)\n\n# Take the first 6 elements from the top errors\nmost_important_correct = sorted_delta_correct[-6:]\n\n# Display the top 6 errors\ndisplay_correct(most_important_correct, X_val_correct, Y_pred_classes_correct, Y_true_correct)\n\nplt.tight_layout() # make sure the padding around each images is sufficient","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"605abe63f6c3fb98035e633861750379aeab482a","_cell_guid":"9eade8ca-0899-4e6c-9a8d-0c71c50f62b6"},"cell_type":"markdown","source":"### 5 D) Show some wrongly misclassified images \n\nNow we know which images the model found relatively difficult to classify, we can display the images from the test data."},{"metadata":{"collapsed":true,"_uuid":"1ba8e0086b29ad0ca29b4b3456aa6b534c35e9c0","_cell_guid":"ca376aa5-85a5-4e82-8a3a-6cf66b528298","trusted":false},"cell_type":"code","source":"# Lets display some error results\n\n# if there is an error, or in other words if the difference between the prediction and the true label > 0\nerrors = (Y_pred_classes - Y_true != 0)\n\n# pick the errors from each set\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\ndef display_errors(errors_index,image_errors,pred_errors, obs_errors):\n    \"\"\" showing 6 images that were incorrectly classified\"\"\"\n    n = 0\n    nrows = 2\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row,col].imshow((image_errors[error]).reshape((28,28)))\n            ax[row,col].set_title(\"Predicted label - {}\\n True label - {}\".format(pred_errors[error],obs_errors[error]))\n            n += 1\n            \n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1) # np.max flattens the nD array and returns the maximum\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_delta_errors = np.argsort(delta_pred_true_errors)\n\n# Take the first 6 elements from the top errors\nmost_important_errors = sorted_delta_errors[-6:]\n\n# Display the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)\n\nplt.tight_layout() # make sure the padding around each images is sufficient","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"91edafbb2050e6f387450ca0637c7640fc3b87a1","_cell_guid":"6a1e257b-7835-4d23-aa70-9f1e4e08b02c"},"cell_type":"markdown","source":"### Result\n\nWe have a pretty accurate model now. But there is always room for improvement. By adding different kind of regularisation methods or using a method called transfer learning for example the model could be made even more accurate.\n\n"},{"metadata":{"collapsed":true,"_uuid":"dfddafbe28c648e1bc3bd7fd4924758a3748782b","_cell_guid":"b0cee769-552d-429b-a070-d029b0223c4b"},"cell_type":"markdown","source":"### Main References\n\n- Keras Library https://keras.io \n- Keras Library https://keras.io/datasets/#mnist-database-of-handwritten-digits\n- Scikit Learn http://scikit-learn.org/stable/\n- Intro to Computer Vision ​https://www.kaggle.com/jannesklaas/13-intro-to-computer-vision\n- How to build an image classifier https://www.kaggle.com/jannesklaas/14-building-an-image-classifier\n- Transfer learning https://www.kaggle.com/jannesklaas/14-2-building-using-pre-trained-models\n- Kaggle competition MINST top n dataset https://www.kaggle.com/daphnecor/introduction-to-cnn-keras-0-997-top-6/edit\n- Kaggle competition MNIST top n https://www.kaggle.com/flagma/mnist-and-simple-keras-cnn\n- Siamese Networks https://www.coursera.org/learn/convolutional-neural-networks\n- One-shot learning https://towardsdatascience.com/one-shot-learning-face-recognition-using-siamese-neural-network-a13dcf739e"},{"metadata":{"collapsed":true,"_uuid":"03e3970a135309cdc3a69400ca49241f5c6c1f22","_cell_guid":"13a48495-9237-4835-bc96-60079b2bfb64","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"a933ca53d447437c3ac56b40b3195d721cf754a0","_cell_guid":"32b9b70b-85a2-4269-94a6-c6b05206c7a2","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"3d858feda6cadaaae3601beeab3a37cb8c21143d","_cell_guid":"1ade59c6-de6f-47a3-8858-77031b4afe1b","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}