{"cells":[{"metadata":{"_uuid":"d3b4064ea3dc4a7465b572f4d7c1cb0e30149ac7"},"cell_type":"markdown","source":"# **Digit Recognizer with Convolutional Neural Networks (CNN)**\n\n### **Content**\n* [Introduction](#1)\n* [Data Preparation](#2)\n * [Normalization](#3)\n * [Reshape](#4) \n * [Label Encoding](#5) \n * [Train - Test Split](#6) \n* [CNN](#7) \n* [Conclusion](#8)"},{"metadata":{"_uuid":"46cd638e2689d5b30ca276b12c80fbd249c36150"},"cell_type":"markdown","source":" \n## **Introduction**\n\n![CNN](http://i64.tinypic.com/15mldu9.jpg)\n\n* A convolutional neural network (CNN) is a type of artificial neural network used in image recognition and object detection. That is specifically designed to process pixel data.\n* It has feature detector for scanning images. This is also called filter. It detects edges or convex shapes.\n* After detection operation, filtered image is obtained. (Convolved feature)\n* While detection operation a feature detector matrix scans picture.\n* After filtering layer it uses activation functions like ReLU to break linearity.\n* Repetitive filtering operations cause image size to decrease. \"Same Padding\" method is used for prevent decreasing of image size.\n* **Max Pooling** :Continuously reduce the spatial size of the representation to reduce the amount of parameters and computation in the network. Pooling layer operates on each feature map independently. It also controls overfitting.\n* **Flattening** : Convert matrices to one dimensional vectors.\n* **Full Connection** : All neurons are connected with each other. They are all connected from previous to next layer.\n\n\n![](http://)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nwarnings.filterwarnings('ignore')\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f12f8152c8fa6e54cde3023e9c734c03f86a78e"},"cell_type":"markdown","source":" \n### **Data Preparation**\n* We'll read train data from csv for model trainin. And test for submission"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3352deb412151d895eca007cfe2b57f2ef121786"},"cell_type":"code","source":"# Read test data\ntest = pd.read_csv(\"../input/test.csv\")\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"436152c4270b81fa1b6be7382187556171105751"},"cell_type":"code","source":"# Get digin labels to yTrain\nyTrain = train[\"label\"]\n\n# Drop 'label' column\nxTrain = train.drop(labels = [\"label\"], axis = 1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d376e34105797fcce85bceb22e840574318eee3b"},"cell_type":"code","source":"plt.figure(figsize = (15,7))\ng = sns.countplot(yTrain, palette = \"icefire\")\nplt.title(\"Number of digits\")\nyTrain.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"533ee5533f2c2e282fb6770cb5c8c478577ba2ea"},"cell_type":"code","source":"def drawImage(imgArray): \n imgArray = imgArray.reshape((28, 28))\n plt.imshow(imgArray, cmap = 'gray')\n plt.title(train.iloc[0,0])\n plt.axis(\"off\")\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa4283ea002c17db2e2f6677e2bc44ccb605a0f2"},"cell_type":"code","source":"# plot some samples\ndrawImage(xTrain.iloc[9].as_matrix())\ndrawImage(xTrain.iloc[5].as_matrix())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5188ed1d6a169bcc8650dbf4b8b36a019e6e0a2a"},"cell_type":"markdown","source":" \n## **Normalization**\n\n* We'll compress the data to the range of [0, 1]\n* CNN works faster on [0, 1] rather than [0, 255]"},{"metadata":{"trusted":true,"_uuid":"a36993eeb6250f3dc7148adaffd9d59145ba8703"},"cell_type":"code","source":"xTrain = xTrain / 255.0\ntest = test / 255.0\nprint(\"xTrain shape: \",xTrain.shape)\nprint(\"test shape: \",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"770057338c6d4b8caa16d8f685911d6ce71aa615"},"cell_type":"markdown","source":" \n## **Reshape**\n\n* Keras needs a dimension for channels. We have gray scaled data so it needs 1 channel. We'll transform 28x28x1 3D matrix."},{"metadata":{"trusted":true,"_uuid":"986c05f185935fbae17e431ee7f092cc2a0a1139"},"cell_type":"code","source":"xTrain = xTrain.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)\nprint(\"xTrain shape: \",xTrain.shape)\nprint(\"test shape: \",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65c2b08ac27daf2874a8afd0ab59b4778a3a809f"},"cell_type":"markdown","source":" \n## **Label Encoding**\n\n* We'll encode the labels to vectors.\n* Labels are 1,2,3,4.. after convertion they like 1-> [0,1,0,0,0,0,0,0,0,0]"},{"metadata":{"trusted":true,"_uuid":"c21f905a1f8668747afb73730c08f4baed012e40"},"cell_type":"code","source":"# one-hot-encoding\nfrom keras.utils.np_utils import to_categorical\nyTrain = to_categorical(yTrain, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf55e7fa20eac6eee921e61c322c36ec9237cd38"},"cell_type":"markdown","source":" \n## **Train - Test Split**\n\n* Train - Test Split for training and validation"},{"metadata":{"trusted":true,"_uuid":"18581dab2d7a1cb851b97d93fadc12a03ca56405"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nxTrain, xVal, yTrain, yVal = train_test_split(xTrain, yTrain, test_size = 0.1, random_state = 2)\nprint(\"xTrain shape\",xTrain.shape)\nprint(\"xVal shape\",xVal.shape)\nprint(\"yTrain shape\",yTrain.shape)\nprint(\"yVal shape\",yVal.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1667fc9e9ec792403d2a63a1ba6b485125d91faa"},"cell_type":"code","source":"# Draw an example\ndrawImage(xTrain[2][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"712dc7417742df6bd3b990b6723fc3e7ccabd81f"},"cell_type":"markdown","source":" \n# **CNN**\n\n* We'll use Keras library for building convolutional neural networks."},{"metadata":{"trusted":true,"_uuid":"40fff63db79e2979138ccd9e97fa09bb3269e61f"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.optimizers import RMSprop,Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.callbacks import LearningRateScheduler\n\nmodel = Sequential()\n\nmodel.add(Conv2D(filters = 64, kernel_size = (4,4),padding = 'Same', activation = 'relu', input_shape = (28,28,1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 32, kernel_size = (4,4),padding = 'Same', activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size = (2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size=(2,2), strides = (2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size = (2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(1024, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28e49fe4a1fcb4942bbbc30a84035795f0653e3a"},"cell_type":"markdown","source":"### **Optimizer**"},{"metadata":{"trusted":true,"_uuid":"f586b27122bf798fd97d26d0d0329436df09058b"},"cell_type":"code","source":"optimizer = Adam(lr = 0.0001, beta_1 = 0.9, beta_2 = 0.999)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cac97798104eb1a04e3a92bd7a9fb495e8797901"},"cell_type":"markdown","source":"### **Model Compilation**\n* Categorical Cross-Entropy method will be used for loss function. \n* We will use categorical cross-entropy for multi class labels."},{"metadata":{"trusted":true,"_uuid":"f2af61064149b08567f4865e107c4a12e2291a17"},"cell_type":"code","source":"model.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics = [\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b55092e926fe13c0ef284861a10f9fcb46a2e52d"},"cell_type":"markdown","source":"### **Epochs and Batch Size Definition**"},{"metadata":{"trusted":true,"_uuid":"d74ad5b1b76c745acb543a7cc1a8cc1becb5aa0d"},"cell_type":"code","source":"epochs = 30 # 1 epoch means 1 forward and 1 backward pass.\nbatch_size = 385 # Number of training samples for one forward/backward pass.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af4d3a7ef379b42d0b7f3eacbd1a6d630ef4a737"},"cell_type":"markdown","source":"### **Data Augmentation**\n\n![](http://i68.tinypic.com/346kytk.jpg)\n\n* To avoid overfitting, we will expand artificially our digit dataset. \n* This way we can diversify the data\n* After this operation we will have various shapes of our digit images."},{"metadata":{"trusted":true,"_uuid":"d698f92e93d72c228911288adccdc638caf00409"},"cell_type":"code","source":"datagen = ImageDataGenerator(\n featurewise_center = False, # set input mean to 0 over the dataset\n samplewise_center = False, # set each sample mean to 0\n featurewise_std_normalization = False, # divide inputs by std of the dataset\n samplewise_std_normalization = False, # divide each input by its std\n zca_whitening = False, # dimesion reduction\n rotation_range = 10, # randomly rotate images in the range 10 degrees\n zoom_range = 0.1, # Randomly zoom image 1%\n width_shift_range = 0.1, # randomly shift images horizontally 1%\n height_shift_range = 0.1, # randomly shift images vertically 1%\n horizontal_flip = False, # randomly flip images\n vertical_flip = False) # randomly flip images\n\ndatagen.fit(xTrain)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b03b8f831bc044de6e51e7d59f297e5f97cb104"},"cell_type":"markdown","source":"### **Setting Annealer**\n\n* It is a learning rate optimizer method provided by Keras. It optimizes learning rate during epoch steps."},{"metadata":{"trusted":true,"_uuid":"7b1ee37b69d092aa16d721fa1242f5fe906dac3c"},"cell_type":"code","source":"annealer = LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7f9e900d145346a47885ac7aa04c92b9e1539c8"},"cell_type":"code","source":"# fit the model\nhistory = model.fit_generator(datagen.flow(xTrain,\n yTrain, \n batch_size = batch_size), \n epochs = epochs, \n validation_data = (xVal, yVal), \n steps_per_epoch = xTrain.shape[0] // batch_size,\n callbacks = [annealer])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"967dcf167588e1f583c8eab397bfff0dbfada2ca"},"cell_type":"markdown","source":"Below plot shows loss values against epochs."},{"metadata":{"trusted":true,"_uuid":"44eac7981c44c9f84b844ea68d5ea553bbbb1d6e"},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nplt.plot(history.history['val_loss'], color = 'b', label = \"validation loss\")\nplt.title(\"Test Loss\")\nplt.xlabel(\"Number of Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"550e5a4207cb375d476b08bd4ff72ecffe61fa9d"},"cell_type":"code","source":"# Final Accuracy\nfinal_loss, final_acc = model.evaluate(xVal, yVal, verbose = 0)\nprint(\"Final loss: {0:.4f}, final accuracy: {1:.4f}\".format(final_loss, final_acc))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"859482185746c419f02cde4ebbf09c3f49d859c5"},"cell_type":"markdown","source":"### **Evaluation of Model**"},{"metadata":{"trusted":true,"_uuid":"fa85894de171e0f0f84adb09abb8e30e82bce2b0"},"cell_type":"code","source":"# Predict the values from the validation dataset\nY_pred = model.predict(xVal)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(yVal,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nf,ax = plt.subplots(figsize=(10, 10))\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.01,cmap=\"Greens\",linecolor=\"gray\", fmt= '.1f',ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3ba3d22e8327980011d4f0e90a2f403dd0a3754"},"cell_type":"markdown","source":"Results can be improved by adding filters or increasing epochs. "},{"metadata":{"trusted":true,"_uuid":"b517456658d72ede0ca005675b5228f1590777a2"},"cell_type":"code","source":"# predict results\nresults = model.predict(test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results, name = \"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d78368d8a13a681c754efd8c724e66d29f386dd"},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"cnn_mnist_datagen.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e40c31522d2439eadc916643fefcd046e2e569b8"},"cell_type":"markdown","source":" \n## **Conclusion**\n* We used Keras for building Convolutional Neural Network model. We got approximately %99 accuracy.\n* *If you have a suggestion, I'd be happy to read it.*"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}