{"cells":[{"metadata":{"_uuid":"a4229d26429f6fdf9dd42b26b8b02a4c4175b707"},"cell_type":"markdown","source":"# CNN implementation on MNIST dataset in Keras- Detailed explanation \n
\nFirst version - 25/07/2018 \nLast version - 24/11/2018\n
\n \n1. **Introduction** \n \n2. **Data pre-processing** \n 2.1. Load data \n 2.2. Check shape, data type \n 2.3. Extract xtrain, ytrain \n 2.4. Mean and std of classes \n 2.5. Check nuls and missing values \n 2.6. Check nuls and missing values \n 2.6. Visualization \n 2.7. Normalization \n 2.8. Reshape \n 2.9. One hot encoding of label \n 2.10. Split training and validation sets \n \n3. **CNN** \n 3.1. Define model architecture \n 3.2. Compile the moedl \n 3.3. Set other parameters \n 3.4. Fit model \n 3.5. Plot loss and accuracy \n 3.6. Plot confusion matrix \n 3.7. Plot errors \n \n4. **Predict and save to csv** \n \n "},{"metadata":{"_uuid":"d5780ab91c4dbd65ba303299ef03e596f55fff5c"},"cell_type":"markdown","source":"# 1. **Introduction**\n\nThis is my first CNN kernel and as such, I believe the [Digit Recognizer dataset/competition](https://www.kaggle.com/c/digit-recognizer) is a very suitable set of images for a beginner CNN project, considering the image size is homogeneous across all images (not common in real-world problems), that the size is small (28x28) so no resizing required, they are in grayscale and they are already in a csv, which can be easily read into a dataframe. \n\n\n\nGiven the comfort that this dataset provides and taking inspration from very popular kernels such as [yassineghouzam's kernel](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6) and [poonaml's kernel](https://www.kaggle.com/poonaml/deep-neural-network-keras-way) among others, I've created my own kernel joining what I have found most useful from each kernel, as well as adding what I have learnt in the process and other notes that may be helpful for others or for future me.\n\nThe kernel consists in 3 main parts:\n * *Data preparation*: Firstly, even if the input data is already quite clean as mentioned before, it still needs some preparation and pre-processing in order to be in an appropriate format to then later be fed to the NN. This includes data separation, reshaping and visualization which might give insight to the data scientist as to the nature of the images.\n * *CNN*: Afterwards, the NN is defined (this is where Keras comes in), the convolutional steps added, NN parameters initialized, and the model trained. This part takes the most time in a ML project.\n * *Evaluation*: Once the model is trained, it's interesting to evaluate the model performance by seeing the progress of the loss and extract some conclusions, that the model is overfitting, or if there is high variance for instance.\n\nIf you find some errors in theoretical concepts, comments of any kind or suggestions, please do let me know :)"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"collapsed":true},"cell_type":"code","source":"# import libraries\nimport numpy as np # linear algebra, matrix multiplications\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"# 2. **Data pre-processing**\n## 2.1 Load data\n\n * *train*: \nthis is the data used to train the CNN. \nthe image data and their corresponding class is provided. \nthe CNN learns the weights to create the mapping from the image data to their corresponding class. \n\n\n * *test*: \nthis is the data used to test the CNN. \nonly the image data is provided. \nthe prediction is submitted to the competition and depending on the accuracy, a score is obtained."},{"metadata":{"_uuid":"5c4c2fa7eeffa51a4c2294cac109bed54fda7dca","trusted":true,"collapsed":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6dca7aa2c39302f7f2e568c01e7d44ecbf12dc1a"},"cell_type":"markdown","source":"## 2.3. Check shape, data type\n\n * *train*: \nthe train dataframe contains data from 42k images. \nthe data from each image is streched out in 1D with 28*28 = 784 pixels. \nthe first column is the label/class it belongs to, the digit it represents. \n\n\n * *test*: \nthe test dataframe contains data from 28k images. \nthis data shall be fed to the CNN so that it's new data, that the CNN has never seen before. \nsame as in the train dataset, image data is streched out in 1D with 784 pixels. \nthere is no label information, that is the goal of the competition, predicting labels as well as possible."},{"metadata":{"_uuid":"8a7cb2975ffe4514b061c379a9bf8cc37c48f9ed","trusted":true,"collapsed":true},"cell_type":"code","source":"print(train.shape)\nntrain = train.shape[0]\n\nprint(test.shape)\nntest = test.shape[0]\n\ntrain.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ffd9d5ea1aa332b428fa668215c7c84c28f3dae","trusted":true,"collapsed":true},"cell_type":"code","source":"# check data type\nprint(train.dtypes[:5]) # all int64, otherwise do train = train.astype('int64')\nprint(train.dtypes[:5]) # all int64, otherwise do test = test.astype('int64')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c71f725e3a5fb0bcca79a6797f48556888797e7"},"cell_type":"markdown","source":"## 2.4 Extract xtrain, ytrain \nThe CNN will be fed xtrain and it will learn the weights to map xtrain to ytrain"},{"metadata":{"_uuid":"f9d2f465a2abfab92687cd5c08404f7e8aef8ae6","trusted":true,"collapsed":true},"cell_type":"code","source":"# array containing labels of each image\nytrain = train[\"label\"]\nprint(\"Shape of ytrain: \", ytrain.shape)\n\n# dataframe containing all pixels (the label column is dropped)\nxtrain = train.drop(\"label\", axis=1)\n\n# the images are in square form, so dim*dim = 784\nfrom math import sqrt\ndim = int(sqrt(xtrain.shape[1]))\nprint(\"The images are {}x{} squares.\".format(dim, dim))\n\nprint(\"Shape of xtrain: \", xtrain.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e5f24bd732ba81028c32812eccec43ce50db629","trusted":true,"collapsed":true},"cell_type":"code","source":"ytrain.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4b74cdf8150c9a6a4e6910bfa0f1441162d4f19"},"cell_type":"markdown","source":"## 2.5. Mean and std of the classes"},{"metadata":{"_uuid":"6eb1fc174dc86ce277e2b8fc858ea1067c8174ee","trusted":true,"collapsed":true},"cell_type":"code","source":"import seaborn as sns\nsns.set(style='white', context='notebook', palette='deep')\n\n# plot how many images there are in each class\nsns.countplot(ytrain)\n\nprint(ytrain.shape)\nprint(type(ytrain))\n\n# array with each class and its number of images\nvals_class = ytrain.value_counts()\nprint(vals_class)\n\n# mean and std\ncls_mean = np.mean(vals_class)\ncls_std = np.std(vals_class,ddof=1)\n\nprint(\"The mean amount of elements per class is\", cls_mean)\nprint(\"The standard deviation in the element per class distribution is\", cls_std)\n\n# 68% - 95% - 99% rule, the 68% of the data should be cls_std away from the mean and so on\n# https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule\nif cls_std > cls_mean * (0.6827 / 2):\n print(\"The standard deviation is high\")\n \n# if the data is skewed then we won't be able to use accurace as its results will be misleading and we may use F-beta score instead.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33027153afbdb5887955c8f39e29c000e374b573"},"cell_type":"markdown","source":"> Summary\n\nShape of *xtrain* is: (42000, 784) \nShape of *ytrain* is: (42000, ) \nShape of *test* is: (28000, 784) \n\nnumber of classes = 10, the distribution of the pictures per class has a mean of 4200 images and a std of 237 images. \nThe digit 1 has the most representation (4684 images) and the digit 5 the least (3795 images). This data can be seen by printing *vals_class* \nThis corresponds to a small standard deviation (5.64%) so there is no class imbalance. In case there was, other techniques would have to be considered but this is outside the scope of this notebook."},{"metadata":{"_uuid":"7b07fc7612d5ce54f82d9debdce7273fd9a3f735"},"cell_type":"markdown","source":"## 2.6. Check nuls and missing values\n```python\ndf.isnull()\n```\nreturns a boolean df with true if value is NaN and false otherwise. \n```python\ndf.isnull().any()\n```\nreturns a df with 1 col and ncol rows where each row says if there is a NaN value present in that col. \n```python\ndf.isnull().any().any()\n```\nreturns a bool with True if any of the df.isnull().any() rows is True"},{"metadata":{"_uuid":"03c7f67d1575a25008ebd244872a262f4a4495ce","trusted":true,"collapsed":true},"cell_type":"code","source":"def check_nan(df):\n print(df.isnull().any().describe())\n print(\"There are missing values\" if df.isnull().any().any() else \"There are no missing values\")\n\n if df.isnull().any().any():\n print(df.isnull().sum(axis=0))\n \n print()\n \ncheck_nan(xtrain)\ncheck_nan(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c960d91af825d5cf5d291ef019b779876e5ccb06"},"cell_type":"markdown","source":"## 2.7. Visualization\n\nThe first nine images in the dataset (which are not ordered by digit) are plotted, just for visualization. There is only one color channel (grayscale) and moreover the pixels are binarized, meaning that hey are either black (with value 0) or white (255). This makes the classification problem easier. Imagine that the CNN received colored digits, either solid, gradient, or digits with many colors. Probably some part of the neural network would focus on learning to tell the digits apart by looking at the colors, when the actual difference between the digits is in their shape."},{"metadata":{"_uuid":"9a22f34f81cb94af5787f06448e326058e61122d","trusted":true,"collapsed":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n%matplotlib inline\n\n# convert train dataset to (num_images, img_rows, img_cols) format in order to plot it\nxtrain_vis = xtrain.values.reshape(ntrain, dim, dim)\n\n# https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplot.html\n# subplot(2,3,3) = subplot(233)\n# a grid of 3x3 is created, then plots are inserted in some of these slots\nfor i in range(0,9): # how many imgs will show from the 3x3 grid\n plt.subplot(330 + (i+1)) # open next subplot\n plt.imshow(xtrain_vis[i], cmap=plt.get_cmap('gray'))\n plt.title(ytrain[i]);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2051502a7a5443f14679c836f5f3bfabd8d0bdac"},"cell_type":"markdown","source":"## 2.8. Normalization\nPixels are represented in the range [0-255], but the NN converges faster with smaller values, in the range [0-1] so they are normalized to this range.\n"},{"metadata":{"_uuid":"76e6f0a067bb7cabf2cfa079a3917f2391693872","collapsed":true,"trusted":true},"cell_type":"code","source":"# Normalize the data\nxtrain = xtrain / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e07e4cdd4acf0ae14bb0dcb633fd2e020ff3e3d6"},"cell_type":"markdown","source":"## 2.9. Reshape"},{"metadata":{"_uuid":"9685eb7e78736876af2f032c53dd2f2d2ea5382b","trusted":true,"collapsed":true},"cell_type":"code","source":"# reshape of image data to (nimg, img_rows, img_cols, 1)\ndef df_reshape(df):\n print(\"Previous shape, pixels are in 1D vector:\", df.shape)\n df = df.values.reshape(-1, dim, dim, 1) \n # -1 means the dimension doesn't change, so 42000 in the case of xtrain and 28000 in the case of test\n print(\"After reshape, pixels are a 28x28x1 3D matrix:\", df.shape)\n return df\n\nxtrain = df_reshape(xtrain) # numpy.ndarray type\ntest = df_reshape(test) # numpy.ndarray type","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d94d64556e53d62e3822542f4575d82f1132bd28"},"cell_type":"markdown","source":"> Note\n\nIn real world problems, the dimensions of images could diverge from this particular 28x28x3 set in two ways:\n * Images are usually much bigger\n\nIn this case all images are 28x28x1, but in another problem I'm working on, I have images of 3120x4160x3, so much bigger and in RGB. Usually images are resized to much smaller dimensions, in my case I'm resizing them to 64x64x3 but they can be made much smaller depending on the problem. In this MNIST dataset there is no such problem since the dimensions are already small.\n * Images don't usually have the same dimensions\n \nDifferent dimension images are a problem since dense layers at the end of the CNN have a fixed number of neurons, which cannot be dynamically changed. This means that the layer expects fixed image dimensions, which means all images must be resized to the same dimensions before training. There is another option, namely, using a FCN (fully convoluted network) which consits solely of convolutional layers and a very big pooling in the end, so each image can be of any size, but this architecture isn't as popular as the CNN + FC (fully connected) layers which is the one I'm familiarized with. \nThere are various methods to make images have the same dimensions:\n * resize to a fixed dimension\n * add padding to some images and resize\n * ... \n\nIn my other problem I have scanned pictures, so I trim the whitespace and resize afterwards. Being this a beginner-friendly dataset, all digits are the same size, binarized and well centered so no need to worry about resizing.\n\n"},{"metadata":{"_uuid":"ac10231c13c9f52440bbb76ed6bc285c402d9b5e"},"cell_type":"markdown","source":"## 2.10. One hot encoding of label\n\nAt this point in the notebook the labels vary in the range [0-9] which is intuitive, but in order to define the type of loss for the NN later, which in this case is categorical_crossentropy (reason is explained in section 2), the targets should be in categorical format (=one hot-vectors): ex : 2 -> [0,0,1,0,0,0,0,0,0,0]\n\nytrain before \n0 1 \n1 0 \n2 1 \n3 4 \n4 0 \n\nwhere the first column is the index,\n\nytrain after \n[[0. 1. 0. ... 0. 0. 0.] \n [1. 0. 0. ... 0. 0. 0.] \n [0. 1. 0. ... 0. 0. 0.] \n ... \n [0. 0. 0. ... 1. 0. 0.] \n [0. 0. 0. ... 0. 0. 0.] \n [0. 0. 0. ... 0. 0. 1.]]\n"},{"metadata":{"_uuid":"508a23dfec7dde6ea68e2904277687283a956327","trusted":true,"collapsed":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical\n\nprint(type(ytrain))\n# number of classes, in this case 10\nnclasses = ytrain.max() - ytrain.min() + 1\n\nprint(\"Shape of ytrain before: \", ytrain.shape) # (42000,)\n\nytrain = to_categorical(ytrain, num_classes = nclasses)\n\nprint(\"Shape of ytrain after: \", ytrain.shape) # (42000, 10), also numpy.ndarray type\nprint(type(ytrain))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e078f5545f1387ae0b49f14fee7ac1d47e5bcae"},"cell_type":"markdown","source":"## 2.11. Split training and validation sets\n\nThe available data is 42k images. If the NN is trained with these 42k images, it might overfit and respond poorly to new data. Overfitting means that the NN doesn't generalize for the digits, it just learns the differences in those 42k images. When faced with new digits slightly different, the performance decreases considerably. This is not a good outcome, since the goal of the NN is to learn from the training set digits so that it does well on the **new digits**.\n\nIn order to avoid submitting the predictions and risking a bad performance, and to determine whether the NN overfits, a small percentage of the train data is separated and named validation data. The ratio of the split can vary from 10% in small datasets to 1% in cases with 1M images.\n\nThe NN is then trained with the remaining of the training data, and in each step/epoch, the NN is tested against the validation data and we can see its performance. That way we can watch how the loss and accuracy metrics vary during training, and in the end determine where there is overfitting and take action (more on this later). For example, the results I had after the 20th epoch with a certain CNN architecture which turned out to overfit:\n\n> loss: 0.0066 - acc: 0.9980 - val_loss: 0.0291 - val_acc: 0.9940\n\nIn this example and without getting much into detail, the __training loss__ is very low while the __val_loss__ is 4 times higher, and the __training accuracy__ is a little higher than the __val_acc__. The accuracy difference is not that much, partly because we are talking about 0.998 vs 0.994, which is exceptionally high, but the difference in loss suggests an overfitting problem.\n\nComing back to the general idea, the **val_acc** is the important metric. The NN might do very well with trained data but the goal is that the NN learns to generalize other than learning the training data \"by heart\". If the NN does well with val data, it's probable that it generalizes well to a certain extent and it will do well with the test data. (more on this in section 2 regarding CNNs)."},{"metadata":{"_uuid":"631056d188bbeec02e02e7f564030b0f8d92178e"},"cell_type":"markdown","source":"__random_state__ in train_test_split ensures that the data is pseudo-randomly divided. \nIf the images were ordered by class, activating this feature guarantees their pseudo-random split. \nThe seed means that every time this pseudo-randomization is applied, the distribution is the same.\n\n__stratify__ in train_test_split ensures that there is no overrepresentation of classes in the val set. \nIt is used to avoid some labels being overrepresented in the val set. \n> Note: only works with sklearn version > 0.17"},{"metadata":{"_uuid":"117f9749a1511c0a238a0dece0f684bd27b41cef","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n# fix random seed for reproducibility\nseed = 2\nnp.random.seed(seed)\n\n# percentage of xtrain which will be xval\nsplit_pct = 0.1\n\n# Split the train and the validation set\nxtrain, xval, ytrain, yval = train_test_split(xtrain,\n ytrain, \n test_size=split_pct,\n random_state=seed,\n shuffle=True,\n stratify=ytrain\n )\n\nprint(xtrain.shape, ytrain.shape, xval.shape, yval.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"237545703337e0e18dc1a7f40daa9edff5d9dec3"},"cell_type":"markdown","source":"> Summary\n\nThe available data is now divided as follows:\n* **Train data**: images (xtrain) and labels (ytrain), 90% of the available data\n* **Validation data**: images (xval) and labels (yval), 10% of the available data"},{"metadata":{"_uuid":"a6f751f76f859c7b51b846f81dc79baa7609e934"},"cell_type":"markdown","source":"# 3. **CNN**\n\nIn this section the CNN is defined, including architecture, optimizers, metrics, learning rate reductions, data augmentation... Then it is compiled and fit to the training set."},{"metadata":{"_uuid":"162a9388b958cf25272870d290a616be47fd85ee","trusted":true,"collapsed":true},"cell_type":"code","source":"from keras import backend as K\n\n# for the architecture\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Lambda, Flatten, BatchNormalization\nfrom keras.layers import Conv2D, MaxPool2D, AvgPool2D\n\n# optimizer, data generator and learning rate reductor\nfrom keras.optimizers import Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ee343a11f8881814ba57a48bdf1a69752c7cc34"},"cell_type":"markdown","source":"## 3.1. Define model architecture\n\nBelow is an example CNN architecture:\n\n![CNN architecture](https://cdn-images-1.medium.com/max/1000/1*xlOZHo8svfDWDyxFFnMurQ.png)\n\nMy final CNN architechture is:\n> In → [ [Conv2D → relu]\\*2 → MaxPool2D → Dropout ]\\*2 → Flatten → Dense → Dropout → Out\n\nI'd like to encourage everyone who wants to learn about CNNs to begin with a simpler one, such as\n\n> In → [Conv2D → relu] → MaxPool2D → Flatten → Dense → Out\n\n, check the performance and keep adding layers or tweaking the parameters until you reach an architecture (that may or may not be like mine) with a __val_acc__ of 0.996 more or less, trying to improve that takes much more time and it's really about the details, but of course feel free to try it out. I just encourage that you build your own model and do your own tests, instead of looking at an already well-performing model and using that.\n\nIn my case I started with the simple architecture, kept a log where I wrote down the loss and accuracy results, changed one thing at a time, checked performance and how it changed regarding the previous version, wrote down the changes I had made and how the result changed, and made further changes based on that.\n\nMore info on CNN architectures here: [How to choose CNN Architecture MNIST](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist)"},{"metadata":{"_uuid":"f53f5e983c9e8769c12180ac9f1bc1c5a5d21fc3"},"cell_type":"markdown","source":"### Architecture layers\n\nYou can read about the theory of CNNs on the Internet from people more knowledgeable than me and who surely explain it much better. So I will skip the theory explanation for the Conv2D, MaxPool2D, Flatten and Dense layers and I will focus on smaller details.\n\n * Conv2D\n \n * __filters__: usually on the first convolutional layers there are less filters, and more deeper down the CNN. Usually a power of 2 is set, and in this case 16 offered poorer performance and I didn't want to make a big CNN with 64 or 128 filters for digit classification.\n \n * __kernel_size__: this is the filter size, usually (3,3) or (5,5) is set. I advise setting one, building the architecture and changing it to see if it affects the performance though it usually doesn't.\n \n * __padding__: two options\n \n * valid padding: no padding, the image shrinks after convolution: n - f + 1\n * same padding: padding of 2, the image doesn't shrink after convolution: p = (f-1)/2 → (n+2) - f(=3) + 1 = n\n \n * __activation__: ReLU is represented mathematically by max(0,X) and offers good performance in CNNs (source: the Internet)\n\n * MaxPool2D: the goal is to reduce variance/overfitting and reduce computational complexity since it makes the image smaller. two pooling options\n \n * MaxPool2D: extracts the most important features like edges\n * AvgPool2D: extracts smooth features\n \n My personal conclusion then is that for binarized images, with noticeable edge differences, MaxPool performs better.\n \n \n * Dropout: you can read the theory on the Internet, it's a useful tool to reduce overfitting. The net becomes less sensitive to the specific weights of neurons and is more capable of better generalization and less likely to overfit to the train data. The optimal dropout value in Conv layers is 0.2, and if you want to implement it in the dense layers, its optimal value is 0.5: [Dropout in ML](https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5)"},{"metadata":{"_uuid":"bb1b773c6bf7baf33756dd265f3a2c1bc662b220","trusted":true,"collapsed":true},"cell_type":"code","source":"model = Sequential()\n\ndim = 28\nnclasses = 10\n\nmodel.add(Conv2D(filters=32, kernel_size=(5,5), padding='same', activation='relu', input_shape=(dim,dim,1)))\nmodel.add(Conv2D(filters=32, kernel_size=(5,5), padding='same', activation='relu',))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(filters=64, kernel_size=(5,5), padding='same', activation='relu'))\nmodel.add(Conv2D(filters=64, kernel_size=(5,5), padding='same', activation='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Flatten())\nmodel.add(Dense(120, activation='relu'))\nmodel.add(Dense(84, activation='relu'))\nmodel.add(Dense(nclasses, activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7983fec4500a27ff54adc40535405ba71e9fd80a","trusted":true,"collapsed":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"191698b8e851c0d7e5cde2eea6016fd865af3f55"},"cell_type":"markdown","source":"This summary shows the summary of the model, displaying each layer with the shape of the output as well as the number of parameters it needs. The first dense layer is the one with the most parameters, since it maps the 3136 outputs of the Flatten layer to the 120 neurons of the Dense layer. Since the layer is a fully connected layer, the number of parameters is: 120 * 3136 + 120.\n\nThe amount of trainable parameters is roughly half a million, which is not that much considering the architecture has medium size and the input dimensions (28,28,3) are small. "},{"metadata":{"_uuid":"5544fac92a5c0a32cf2431a69c092903a212f959"},"cell_type":"markdown","source":"## 3.2. Compile the model\n\n * **Optimizer**: it represents the gradient descent algorithm, whose goal is to minimize the cost function to approach the minimum point. **Adam** optimizer is one of the best-performing algorithms: [Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980v8). The default learning rate for the Adam optimizer is 0.001. Another optimizer choice may be RMSprop or SGD.\n\n * **Loss function**: It is a measure of the overall loss in the network after assigning values to the parameters during the forward phase so it indicates how well the parameters were chosen during the forward propagation phase. This loss function requires the labels to be encoded as one-hot vectors which is why this step was taken back in 1.8. \n\n * **Metrics**: this refers to which metric the network should achieve, the most common one being 'accuracy' but there are other metrics to measure the performance other than accuracy, such as precision or recall or F1 score. The choice depends on the problem itself. Where high recall means low number of false negatives , High precision means low number of false positives and F1 score is a trade off between them: [Precision-Recall](http://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html). Depending on the problem, accuracy may not be the best metric. Suppose a binary classification problem where there are much more 0 values than 1, and therefore it's crucial that the predicted 1's are mostly correct. A network that just outputs 0 every time would get very high accuracy but the model still wouldn't perform well. Take the popular example:\n \n > A ML company has built a tool to identify terrorists among the population and they claim to have 99.99% accuracy. When inspecting their product, turns out they just output 0 in every case. Since there is only one terrorist for every 10000 people (this is made up, I actually have no idea what the probability is but all I know is that it's very low), the company has a very high precision, but there's no need of a ML tool for that. With the class imbalance being so high, accuracy is not a good metric anymore and other options should be considered."},{"metadata":{"_uuid":"5785f24f35271268d9616e45cdde37ddf4547921","trusted":true,"collapsed":true},"cell_type":"code","source":"model.compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6bf54846016d6953c5eff1ea126b318a32206a51"},"cell_type":"markdown","source":"## 3.3. Set other parameters \n\n### Learning rate annealer\n\nThis is a useful tool which reduces the learning rate when there is a plateau on a certain value, which you can specify. In this case the monitoring value is __val_acc__. When there is no change in __val_acc__ in 3 epochs (patience), the learning rate is multiplied by 0.5 (factor). If the learning rate has the value of min_lr, it stops decreasing."},{"metadata":{"_uuid":"e8a38a240404358164f63020aceb591c3907ffdc","trusted":true,"collapsed":true},"cell_type":"code","source":"lr_reduction = ReduceLROnPlateau(monitor='val_acc', \n patience=3, \n verbose=1, \n factor=0.5, \n min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ed29b562d85bb487fe56451436bfd65684bd6c6"},"cell_type":"markdown","source":"### Data augmentation\n\nData augmentation is a technique used to artificially make the training set bigger. There are a number of options for this, the most common ones include rotating images, zooming in a small range and shifting images horizontally and vertically. \n\nBeware that activating some features may be confusing for the network, imagine that when taking img1 and flipping it, it may be very similar to img2 which has a different label. With the digits 6 and 9 for example, if you take either and flip it vertically and horizontally, it becomes the other. So if you do that with the digit 9, flip it in both edges and tell the network that the digit is still a 9 when it actually is very similar to the images of the digit 6, the performance will drop considerably. So take into account the images and how activating the features may affect the labeling."},{"metadata":{"_uuid":"e1379cb5bd05278c3726b39df441a5eb1dcd9dc5","collapsed":true,"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n featurewise_center=False, # set input mean to 0 over the dataset\n samplewise_center=False, # set each sample mean to 0\n featurewise_std_normalization=False, # divide inputs by std of the dataset\n samplewise_std_normalization=False, # divide each input by its std\n zca_whitening=False, # apply ZCA whitening\n rotation_range=30, # randomly rotate images in the range (degrees, 0 to 180)\n zoom_range = 0.1, # Randomly zoom image \n width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)\n height_shift_range=0.1, # randomly shift images vertically (fraction of total height)\n horizontal_flip=False, # randomly flip images\n vertical_flip=False) # randomly flip images\n\ndatagen.fit(xtrain)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ebea61c92df8ebe85c0fe42360775e1de002b76"},"cell_type":"markdown","source":"### Epochs and batch_size\n\n * **Epochs**: based on my experiments, the loss and accuracy get into a plateau at around the 10th epoch, so I usually set it to 15. \n * **Batch_size**: I skip the theory which you can read it on the Internet. I recommend that you try changing it and seeing the change in the loss and accuracy, in my case a batch_size of 16 turned out to be disastrous and the best case occurred when I set it to 64. "},{"metadata":{"_uuid":"2fd7356e907ad7001a84890191271f59975ff49e","collapsed":true,"trusted":true},"cell_type":"code","source":"epochs = 15\nbatch_size = 64","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"095c87f6e1063bbed5aa2ecfc51215768319491e"},"cell_type":"markdown","source":"## 3.4 Fit the model\n\nSince there is data augmentation, the fitting function changes from fit (when there is no data augmentation) to fit_generator. The first input argument is slightly different. Otherwise you can specify the verbosity, number of epochs, validation data if any, any callbacks you want to include... This is one of the most time consuming cells in the notebook, and its running time depends on the number of epochs specified, number of trainable parameters in the network and input dimensions. Changing the batch_size also conrtibutes to changes in time, the bigger the batch_size, the faster the epoch.\n\nIn a GPU such as the one Kaggle ofers, the training is done in 16s per epoch adding up to a total of 400s.\n\n> Note: remember to create and compile the model all over again whenever you change something, such as batch_size or epochs or anything related to the CNN. if you don't and just run the fit cell, it will continue training on the old network."},{"metadata":{"_uuid":"3a87fb83b3d6654ee0b2a54e65a8b9c00ec48cf4","scrolled":false,"trusted":true,"collapsed":true},"cell_type":"code","source":"history = model.fit_generator(datagen.flow(xtrain,ytrain, batch_size=batch_size),\n epochs=epochs, \n validation_data=(xval,yval),\n verbose=1, \n steps_per_epoch=xtrain.shape[0] // batch_size, \n callbacks=[lr_reduction])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"556d85363f20ddeda657143c437327de757b8b23"},"cell_type":"markdown","source":"## 3.5. Plot loss and accuracy\n\nAfter training the model, it's useful to plot the loss and accuracy in training and validation to see its progress and detect problems. In this particular case with this particular network, the training loss decreases, which means the network is learning, and there is no substantial difference between the training loss and validation loss wich indicates no overfitting. At this levels where the loss is so low and accuracy is so high there really is no bias or variance problem, but if you want to improve results further you could approach a bias problem, in other words, that the training loss is too high. To reduce this the recommended solutions are making a bigger network and training for a longer time. Feel free to tweak the network or epochs."},{"metadata":{"_uuid":"5876a7e97bfb916020e9c04a6a6056bc96f98669","trusted":true,"collapsed":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"Validation loss\",axes =ax[0])\nax[0].grid(color='black', linestyle='-', linewidth=0.25)\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nax[1].grid(color='black', linestyle='-', linewidth=0.25)\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0577813580eb3ead164e8742e53768db2b176cbd"},"cell_type":"markdown","source":"## 3.6. Plot confusion matrix\n\nI imported this from [yassineghouzam's kernel](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6). The confusion matrix is a nclasses x nclasses matrix (nclasses is the number of classes/labels in your classification problem). The vertical axis shows the actual or true labels, while the horizontal axis shows the predicted labels, that is, the labels that the network has predicted. \n\nIn an ideal case, the matrix would be an identity matrix. All the points in the matrix would be 0 except the ones in the diagonal. This would happen if the network predicted the correct label every time and each label were predicted correctly every time, but it rarely happens. A more common situation is that there are many values in the diagonal, many occurrences of correctly labeled images, while there are some scattered wrong-labeled-images. \n\nIn this case the wrong values seem to be randomly disrtibuted, which gives no information about how to proceed. If there were a bigger number of visible errors, such as the digit 4 being mistaken by the digit 9 several times, it would be intuitive to understand what is happening, since depending on how someone writes the digit 4, it might be very similar to 9.\n\nThe case of the most popular digit, the digit 1, is also noticeable (popular I mean there are 463 images corresponding to the digit 1, more than any other digit). In this case, the matrix shows that if the network predicts the digit 2, then it is always correct (the second column is all 0 except at digit 2), the network has perfect precision. However, the digit 2 is not always correctly labeled (second row), so the network doesn't have perfect recall. All in all, the more images available for a class, the less errors the network usually does. In an unrelated project I'm working with a small and imbalanced dataset and there's one class very under-represented and I'm having difficulties to predict it correctly."},{"metadata":{"_uuid":"63a0bda9010ac311a6a9c845343c64ebcba8ddc3","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport itertools\n\ndef plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues):\n \"\"\"\n This function prints and plots the confusion matrix.\n Normalization can be applied by setting `normalize=True`.\n \"\"\"\n plt.imshow(cm, interpolation='nearest', cmap=cmap)\n plt.title(title)\n plt.colorbar()\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n if normalize:\n cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n thresh = cm.max() / 2.\n for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n plt.text(j, i, cm[i, j],\n horizontalalignment=\"center\",\n color=\"white\" if cm[i, j] > thresh else \"black\")\n\n plt.tight_layout()\n plt.ylabel('True label')\n plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nypred_onehot = model.predict(xval)\n# Convert predictions classes from one hot vectors to labels: [0 0 1 0 0 ...] --> 2\nypred = np.argmax(ypred_onehot,axis=1)\n# Convert validation observations from one hot vectors to labels\nytrue = np.argmax(yval,axis=1)\n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(ytrue, ypred)\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes=range(nclasses))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bde5a08ce48c6d0602e8931a2d59dd76f99fd1d7"},"cell_type":"markdown","source":"## 3.7. Plot errors\n\nAnoher useful approach is to plot the errors or wrongly labeled images, hoping it provides some intuition about what the network might be doing wrong. The cell outputs just 6 error images, you can change the range in the code to see other images. After running the cell, almost all images are very confusing and it would be difficult for a human to label them correctly. The third digit could be 1 or 7, the fourth a 0 or 6, the last could be either a 3 or a 8.. Even humans could make mistakes when labelling these images.\n\nGenerally speaking, a network might have high bias (high training loss) and you could spend hours trying to decrease it, but the network may have reached the best human result. If the best digit-recognizer can only achieve 90% accuracy (assume that this is true for the sake of the example), the network won't be able to do much better. So when the network reaches this point, it's usually the limit of what it can do. "},{"metadata":{"_uuid":"f5f8df7abc5d6278caa4098f5d39ea7566a55ccb","trusted":true,"collapsed":true},"cell_type":"code","source":"errors = (ypred - ytrue != 0) # array of bools with true when there is an error or false when the image is cor\n\nypred_er = ypred_onehot[errors]\nypred_classes_er = ypred[errors]\nytrue_er = ytrue[errors]\nxval_er = xval[errors]\n\ndef display_errors(errors_index, img_errors, pred_errors, obs_errors):\n \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n n = 0\n nrows = 2\n ncols = 3\n fig, ax = plt.subplots(nrows, ncols, sharex=True, sharey=True)\n for row in range(nrows):\n for col in range(ncols):\n error = errors_index[n]\n ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n ax[row,col].set_title(\"Predicted label :{}\\nTrue label :{}\".format(pred_errors[error],obs_errors[error]))\n n += 1\n \n# Probabilities of the wrong predicted numbers\nypred_er_prob = np.max(ypred_er,axis=1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_er = np.diagonal(np.take(ypred_er, ytrue_er, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_er = ypred_er_prob - true_prob_er\n\n# Sorted list of the delta prob errors\nsorted_delta_er = np.argsort(delta_pred_true_er)\n\n# Top 6 errors. You can change the range to see other images\nmost_important_er = sorted_delta_er[-6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_er, xval_er, ypred_classes_er, ytrue_er)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04ece1be5e18b0bbda311c49d6e3566d49460452"},"cell_type":"markdown","source":"# 4. **Predict and save to csv**\n\nOnce you are happy with your network, use the test data to create the prediction, this cell creates the csv format necessary to submit to the [Digit Recognizer](https://www.kaggle.com/c/digit-recognizer). \n\n> Remember that the csv is only created when you Commit the kernel, not when you run the cell in the editor. Make sure you commit the kernel, that it is successfull, and then in the kernel page (clicking the left arrows on the top left), there will be a tab called __Output__ where you can find your csv and submit it directly to the competition."},{"metadata":{"_uuid":"f85a38f96f2fb9c26119f6a2a32ce84ee0751c91","trusted":true,"collapsed":true},"cell_type":"code","source":"predictions = model.predict_classes(test, verbose=1)\n\nsubmissions = pd.DataFrame({\"ImageId\": list(range(1,len(predictions)+1)),\n \"Label\": predictions})\n\nsubmissions.to_csv(\"mnist2908.csv\", index=False, header=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"580778ade72465b34fd1665ca0871326fe71b855"},"cell_type":"markdown","source":"And that's it! I hope the explanations were informative and the code was clean and well-formatted. If there are any comments or suggestions whatsoever, if something is not clear or if you have any (positive) criticism towards the kernel, don't hesitate to tell me! "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}