{"cells":[{"metadata":{"_uuid":"25b233bff0670cfe4c40a6ab2977917be19c0e1a"},"cell_type":"markdown","source":"1. [Introduction](#introduction)<br>\n2. [Visualazing images](#image)<br>\n3. [How does a Convolution Neural Network works ?](#cnn) <br>\n4. [Training our model](#training)<br>\n5. [Result and conclusion](#result)<br>"},{"metadata":{"_uuid":"b374e5eb61f00d9baf334c4df0c5834213b0cadd"},"cell_type":"markdown","source":"<a id='introduction'>1. Introduction</a><br>\nFrom the phone industry to military and government, cameras with artificial intelligence have become in widespread use. Phone makers have integrated this technology so that users could unlock their phone or pay just by taking a glimpse at their device, governments are using it to build giants surveillance network. I have always wondered how a computer could recognize something on an image. In this article, we will build a simple algorithm capable of recognizing handwritten numbers. Our goal is to correctly identify digits from a dataset of tens of thousands of handwritten image. In order to do so, we are going to create a neural network called Convolution Neural Network (CNN) and train it on our Dataset. They have proved to be very efficient on image recognition problem. "},{"metadata":{"_uuid":"b8a73abc50f421e33ae5287e5a749fdc626b5401"},"cell_type":"markdown","source":"We will use the Keras library in order to build our model."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nfrom keras.layers import Dense, Conv2D,MaxPooling2D,Flatten,Dropout\nfrom keras.layers.core import Activation\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nfrom keras.models import Sequential\nfrom keras.utils import to_categorical\nfrom keras.callbacks import EarlyStopping ,ReduceLROnPlateau\nfrom keras.models import load_model\nfrom keras.optimizers import SGD, RMSprop\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f23a120e7b954eca35c7fb57b98ff1095c97f79"},"cell_type":"markdown","source":"<a id='image'>2. Visualazing images</a><br>\nThe data files train.csv and test.csv contain gray-scale images of hand-drawn digits, from zero through nine. There are 42000 entries in our dataset. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image. Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"collapsed":true},"cell_type":"code","source":"df_train=pd.read_csv(\"../input/train.csv\")\ndf_test=pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27621ed41b6631be7c558dc1f1801db13df44d9b"},"cell_type":"code","source":"df_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f05c23eafa20fd600418a708759908d3f6752045"},"cell_type":"code","source":"df_train.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0e92904cd4a2f26809f1e343ce24b6674e8ed75"},"cell_type":"markdown","source":"We save the images in a numpy array 'predictors' and the corresponding digit in the array 'target'."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"9559b42e37f95a2ec19aa73be396ec5cb3961f43"},"cell_type":"code","source":"predictors=np.array(df_train.drop(columns=['label']))\ntarget=np.array(df_train[\"label\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"53786b2ccd63ac92230a48f4a7e678dff390926c"},"cell_type":"code","source":"test=np.array(df_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f2a95e441000bc7801b4960bb5cdce9350eff5e"},"cell_type":"markdown","source":"We have to reshape our predictors array in order to visualize the 28x28 px images. "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"f5dcf939cb7dbea4ed48615ca811db1dcd21313d"},"cell_type":"code","source":"images=predictors.reshape(-1,28,28)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ea37bd6ef26775639cc092b84d93c8bcbb95641"},"cell_type":"code","source":"images[0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c7591ba63624a636e8f9ea39c495ba58251d8a6"},"cell_type":"code","source":"f, axs = plt.subplots(1,10,figsize=(15,15))  \naxs.ravel()\nfor k in range(0,10):\n    axs[k].imshow(images[k],cmap=\"gray_r\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b525ea09133c092d7f5fa3620d6725d7880aae3"},"cell_type":"markdown","source":"In order to visualize a set of handwritten digit 3, we will filter our training dataset and save the result in the 'image_three' array."},{"metadata":{"trusted":true,"_uuid":"e4b1a3e6eab68307371cd47698510d60252ca9bd","collapsed":true},"cell_type":"code","source":"image_three=df_train[df_train[\"label\"]==3]\nimage_three=np.array(image_three.drop(columns=['label']))\nimage_three=image_three.reshape(-1,28,28)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75ef6fe9cf4e7a862c8025178f5dd80eda754403"},"cell_type":"code","source":"f, axs = plt.subplots(5,5,figsize=(15,15))  \naxs.ravel()\ni=0\nfor k in range(0,5):\n    for j in range(0,5):\n        axs[j,k].imshow(image_three[i],cmap=\"gray_r\")\n        i+=1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9bc186170924bb68a5e2753c0703b16efe332ffa"},"cell_type":"markdown","source":"<a id=\"cnn\" >3. How does a Convolution Neural Network works ?</a>\n\nWhat's a convolution Neural Network ? \nConvolution Neural Networks are a category of neural networks that are very effective in image recognition and classification. \nA Convolutional Neural Network is typically composed by four operations : \n1. Convolution\n1. Non linearity \n1. Pooling \n1. Classification "},{"metadata":{"_uuid":"b878e3e6a609260ee87f9fdac23d1624cb38c19b"},"cell_type":"markdown","source":"**Convolution** <br>\nThe purpose of this operation is to extract features from the input images. \nLet's consider a 5x5 image whose pixel values are 0 or 1. Now, let's consider a 3x3 matrix. The convolution consists in sliding the matrix (called **kernel** or **filter**) over the input image and multiply the matrixes which are overlapping. The number of pixels by which we slide our filter on the input matrix is called the **stride**.\nBy using different types of filter, we can identify different features in the image ( curves, edges ...). \nDuring this step, the input image will be filtered by different kernels. The output will be the result of the input image's filtering. The number of filters applied during the convolution is called the **Depth** and the output matrixes are called **Convolved features** or **Feature map** .\nAs convolution is a linear operation ( multiplication of matrixes ) and our model will be trained on non-linear data, we should apply a non linear operation to this layer's output. \n\n**Non linearity** \nThe function that is applied to Convolved Features  is called 'RELU' ( stands for Rectified Linear Unit ). It consists in replacing all the negatives pixels by 0 :\n$$ Relu=max(0,input)$$. \nRelu is a non linear operation\n\nLet's see what does a feature map look like ! We will apply this operation on an image and display the output. \n"},{"metadata":{"trusted":true,"_uuid":"65b2a9795164c5c55f65aa1cc7579dc3da8be13e","collapsed":true},"cell_type":"code","source":"model_test = Sequential()\nmodel_test.add(Conv2D(50, kernel_size=5, padding=\"same\",input_shape=(28, 28, 1),activation='relu'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2bb11d5ca837936ca3b6cff9bd16b1c17d145b32"},"cell_type":"code","source":"image=image_three[0]\nplt.imshow(image,cmap=\"gray_r\")\nimage=image.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c5224975b55c57f2329c60c7a124b43378a8c1d","collapsed":true},"cell_type":"code","source":"image.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99688c39cc21c28ca2483be075ff40e012595987"},"cell_type":"code","source":"conv_image=model_test.predict(image)\nconv_image=np.squeeze(conv_image,axis=0)\nconv_image.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b64ed83f984e79f5c699ec40d3190e67335ce0a"},"cell_type":"code","source":"f, axs = plt.subplots(5,10,figsize=(15,15))  \naxs.ravel()\ni=0\nfor k in range(0,10):\n    for j in range(0,5):\n        axs[j,k].imshow(conv_image[:,:,i],cmap=\"gray_r\")\n        i+=1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d195469e02c54a8ef8aa37c619e1d006d9cca72"},"cell_type":"markdown","source":"**Pooling** <br>\nWe don't want our model to overfit our training data, indeed it must be capable of correctly identifying numbers from another dataset. \nIn order to avoid overfitting, we will try to decrease the level of details of our Feature map. This operation is called **Pooling** or **subsampling**. This step is quite similar to the convolution step. It consists in sliding a filter over the input image and apply a function on the window which is flown over by the filter. The most common pooling function are max or average. \n\nLet's visualize the output of this operation !"},{"metadata":{"trusted":true,"_uuid":"1f07b086a5e6e95ffad99c4e75531f85fd1b1250"},"cell_type":"code","source":"model_test.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\nconv_image=model_test.predict(image)\nconv_image=np.squeeze(conv_image,axis=0)\nconv_image.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65e5db5efd4db1fdd304d3ec67c5527b8adb268c"},"cell_type":"code","source":"f, axs = plt.subplots(5,10,figsize=(15,15))  \naxs.ravel()\ni=0\nfor k in range(0,10):\n    for j in range(0,5):\n        axs[j,k].imshow(conv_image[:,:,i],cmap=\"gray_r\")\n        i+=1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3f108fff8fc95c255035754d6fd17b214fdef34"},"cell_type":"markdown","source":"**Classification** <br>\nEventually the output of the pooling feature is given to a fully connected layer. The purpose of this operation is to use those features to classify the input image.\nHere, this task can have ten possible outputs. In the output layer of the neural network, we use an activation function called **softmax**.  This function transforms a vector of arbitrary real values into a vector of real values where each entry is in the range (0, 1], and all the entries add up to 1. "},{"metadata":{"_uuid":"e8c268657037d678c2c0c0aa0846bdf1dc795df2"},"cell_type":"markdown","source":"<a id='training'>4. Training our model</a><br>"},{"metadata":{"_uuid":"ed41a3368090e1f49d23639f5cb7e919b86b741e"},"cell_type":"markdown","source":"As Images with high pixel values would have a bigger impact on our model's training then low pixels value images, we have to normalize the data set. Hence, each image will contribute equally to our model loss. \nWe also tranform our target value ( digit ) into a vector to match the softmax function output. \n$$ 6=[0,0,0,0,0,0,1,0,0,0]$$. "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"e85c1df90168fb588cca95067996406a33296dbd"},"cell_type":"code","source":"predictors=predictors/255\ntest=test/255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1f55c64ad472eb3e847405d27f0ac9df5ba4daf9"},"cell_type":"code","source":"target=to_categorical(target)\nprint(target)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"312a1169d156052003959b5c8c8119e3966dd70b"},"cell_type":"markdown","source":"If we train our model on the training set and evaluate its accuracy on the same data, our model would likely perform better than it actually does on unseen data. In order to avoid overfitting, we will split our train set in two subsets : 80% of the training set will be used for training and 20% will be used for testing our model. We use the train_test_split function to generate those subsets."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"80f7407a324cc6fb04a6d769172347dd08c4bd6f"},"cell_type":"code","source":"predictors=predictors.reshape(-1,28,28,1)\ntest=test.reshape(-1,28,28,1)\nX_train, X_test, y_train, y_test = train_test_split(predictors,target,test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"221c5fececff05c0e1acb39ace4852fc0017a1e7"},"cell_type":"markdown","source":"The dropout layer consists in randomly dropping nodes, the aim is to avoid overfitting.<br> \nThe flatten layer consists in transforming the input array to an output array of size (*,500)"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"268ae99b35048971de310fbf9bc0d1781a08ef03"},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(50, kernel_size=5, padding=\"same\",input_shape=(28, 28, 1), activation = 'relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(80, kernel_size=5, padding=\"same\", activation = 'relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Flatten())\nmodel.add(Dense(500))\nmodel.add(Activation(\"relu\"))\n\nmodel.add(Dense(10))\nmodel.add(Activation(\"softmax\"))\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec08d80687068de02bf0117969550a3e11d03009"},"cell_type":"code","source":"model.fit(X_train, y_train, epochs= 15 , batch_size=200, validation_split = 0.2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a64b505944781082fb11c48aa5e6c6cae111d009"},"cell_type":"markdown","source":"<a id='result'>5. Result and conclusion</a><br>"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"75530b6dddb469384dd3bb9a3b19e53d69a9f185"},"cell_type":"markdown","source":"Our model can predict handwritten's digit value with 99% accuracy. Let's test our model !"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"b338b4ed16acfafdff68ca3135242f69ccd9761a"},"cell_type":"code","source":"result_test=model.predict(predictors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cf247bc016efa7bc13dcbc83979b776579b9449"},"cell_type":"code","source":"# select the index with the maximum probability\nresult_test = np.argmax(result_test,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"007d0310ab88680854e6275a2a9ba3af84f911f4"},"cell_type":"markdown","source":"Our model has predicted that the handwritten digit of the 456th image corresponds to a 5 !"},{"metadata":{"trusted":true,"_uuid":"45f3017556342a36a7709414b68fa5eb428698f2"},"cell_type":"code","source":"result_test[456]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8182e991401b093de48ce512c93c873a88b05413"},"cell_type":"markdown","source":"As we can see below, our model has correctly predicted the outcome :)"},{"metadata":{"trusted":true,"_uuid":"2ca7d480aae2eb3d48ce3a8157aecb90e15e0516"},"cell_type":"code","source":"plt.imshow(predictors[456].reshape(28,28),cmap=\"gray_r\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2dc6fbdde2dc2bb220066688001132f1f4495ee2"},"cell_type":"code","source":"results=model.predict(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"80b3492386705716e767bd79016cf84510c24815"},"cell_type":"code","source":"results=np.argmax(results,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"481b2f5ec934877b513bb90c9774a0900541eaac"},"cell_type":"code","source":"results = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"result.csv\",index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}