{"cells":[{"metadata":{"_uuid":"047a3aeb22d4a2f692a220f24d2845882a3514ed"},"cell_type":"markdown","source":"# Cihan Yatbaz\n###  10 / 12 / 2018\n---\n\n\n1.  [Introduction:](#0)\n2. [Preparing Dataset :](#1)\n3. [Normalization, Reshape and Label Encoding:](#2)\n4. [Train Test Split  :](#3)\n5. [Convolutional Neural Network (CNN)  :](#4)\n    1. [Define Optimizer  :](#5)\n    1. [Compile Model  :](#6)\n    1. [Epochs and Batch Size  :](#7)\n    1. [Data Augmentation  :](#8)\n    1. [Fit the model  :](#9)\n    1. [Evaluate the Model  :](#10)\n6. [CONCLUSION :](#11)"},{"metadata":{"_uuid":"f7d28b490518e77d49a088cc656f146dd417bf33"},"cell_type":"markdown","source":"<a id=\"0\"></a> <br>\n## 1) Introduction\n* We will be working on this kernel Degit Recognizer data. We'll introduce 80% of the Degit Recognizer we have, and we will try to predict the remaining 20%.\n* In this Kernel we will do the Convolutional Neural Network (CNN) step by step.\n* Let's start by creating our libraries\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns \nimport matplotlib.pyplot as plt\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ddc68b18946c0736b86cf40c7fb3836d96ef6c4"},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## 2) Preparing Dataset"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# read train\ntrain = pd.read_csv(\"../input/train.csv\")\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"593a8bb329509665cfedaf04d8c97cba569f87cb"},"cell_type":"code","source":"# read test\ntest = pd.read_csv(\"../input/test.csv\")\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f3e90e0f4642778f845693d8d4b42a7fdec83594"},"cell_type":"code","source":"# put labels into y_train variable \nY_train = train[\"label\"]\n\n# Drop label column\nX_train = train.drop(labels=[\"label\"], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98352c1144257ea310bec1d6a4f3e4cdbea35af3"},"cell_type":"code","source":"# visualize number of digits classes\nplt.figure(figsize =(15,10))\ng = sns.countplot(Y_train, palette=\"icefire\")\nplt.title(\"Number of digit classes\")\nY_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ce84b109b5cfd68ce9ea29a77819ab242c29855"},"cell_type":"code","source":"# plot for the three number\nimg = X_train.iloc[9].as_matrix() # as_matrix: Converting to Matrix\nimg = img.reshape((28,28))\nplt.imshow(img, cmap='gray')\nplt.title(train.iloc[0,0])\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e60fb4ced08665011e1fa1785024da5b3de14ee"},"cell_type":"code","source":"# plot for the seven number\nimg = X_train.iloc[6].as_matrix()\nimg = img.reshape((28,28))\nplt.imshow(img, cmap='gray')\nplt.title(train.iloc[3,0])\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ace75fcd5a9d036edc177e90b33efa87ff42e88"},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## 3) Normalization, Reshape and Label Encoding\n* Normalization\n    * Increases CNN's operating speed.\n* Reshape\n    * Keras works with 3D matrix does not work with 2D so it must be 3D.\n* Label Encoding\n    * Encode labels to one hot vectors\n        * 2 => [0,0,1,0,0,0,0,0,0,0]\n        * 4 => [0,0,0,0,1,0,0,0,0,0]\n---\n* The maximum value that a picture can take is 355, so we divide by 255. And we're normalizing x_train and y_train."},{"metadata":{"trusted":true,"_uuid":"3861df5598e7d3526afada67f4e08410f69bd6e1"},"cell_type":"code","source":"X_train = X_train / 255.0\ntest = test / 255.0\nprint(\"X_train shape: \", X_train.shape)\nprint(\"test shape: \", test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a38d437bbeb3c68dd1e148261836c0b95cda6b8c"},"cell_type":"code","source":"# reshape\nX_train = X_train.values.reshape(-1,28,28,1)\nterst = test.values.reshape(-1,28,28,1)\nprint(\"x_train shape: \", X_train.shape)\nprint(\"test shape: \", test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"436d90dc9a34dd9d47b7a84cf8be657c1b7c406a"},"cell_type":"code","source":"# Label encoding \nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding**\nY_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"17887413abd0c895bd973711319d4b0df89fb043"},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 4) Train Test Split\n* We'll introduce 80% of the Degit Recognizer we have, and we will try to predict the remaining 20%."},{"metadata":{"trusted":true,"_uuid":"dfbbd44f99573448c09b9c23e7b1dcf3538b6583"},"cell_type":"code","source":"# split the train and the validation set for the fittig\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, Y_train, Y_test = train_test_split(X_train, Y_train, test_size=0.1, random_state=2)\nprint(\"X_train shape: \", X_train.shape)\nprint(\"X_test shape: \", X_test.shape)\nprint(\"Y_train shape: \", Y_train.shape)\nprint(\"Y_test shape: \", Y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ec4759dcb007b000320685289ad0adf27f518bc"},"cell_type":"code","source":"# examples for the eight\nplt.imshow(X_train[0][:,:,0], cmap='gray')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b493b81ced036ba814d47b72b46f0a9a890f79c0"},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 5) Convolutional Neural Network (CNN)\n* Cnn is generally used for image classification.\n* We will do step by step.\n"},{"metadata":{"trusted":true,"_uuid":"5f6b55e8a475182086b6ffde26d357ba096fbf15"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical  # convert to one-hot-encoding\nfrom keras.models import Sequential # Sequential: A structure with layers in it.\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\nmodel = Sequential()\n#\nmodel.add(Conv2D(filters=8, kernel_size=(5,5), padding='Same', activation='relu', input_shape=(28,28,1)))\nmodel.add(MaxPool2D(pool_size=(2,2))) # Max pooling: Transfer the max values ​​in our image to the pooling layer.\nmodel.add(Dropout(0.25)) \n#\nmodel.add(Conv2D(filters=16, kernel_size=(3,3), padding='Same', activation='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2))) \nmodel.add(Dropout(0.25))\n# fully connected\nmodel.add(Flatten())# flatting : Doing a straight vector by extending our matrix\nmodel.add(Dense(256, activation=\"relu\"))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))  # softmax: It is a more generalized version of Sigmoid.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c1806a3a2bab0992738ab61ba7d78915d3c6526a"},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n### A) Define Optimizer\n---\n* Adam optimizer : Adaptive momentum. If we use Adam, learning_rate is not fixed. It updates Learning_rate and enables us to learn more quickly."},{"metadata":{"trusted":true,"_uuid":"d66715f61166422ee9436032de4eac8840cda4e2"},"cell_type":"code","source":"# define the optimizer \noptimizer = Adam(lr=0.003, beta_1=0.9, beta_2=0.999)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e55e001cda023eb2936c1210ff2bf2a3c32ea0f5"},"cell_type":"markdown","source":"<a id=\"6\"></a>\n### B) Compile Model\n"},{"metadata":{"trusted":true,"_uuid":"4c45d1a172bae956bd2ad0653e91af5d6df0fa9b"},"cell_type":"code","source":"# loss: If the error is too many we update the weight, until the error is minimized\n# categorical_crossentropy : If the classification is more than 2, we use categorical_crossentropy.\nmodel.compile(optimizer = optimizer, loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"372c4adf276603030d72bf935b10fc56a565d6c1"},"cell_type":"markdown","source":"<a id=\"7\"></a>\n### C) Epochs and Batch Size"},{"metadata":{"trusted":true,"_uuid":"6801effd3e1b2cdf580a8880b422885265630aa5"},"cell_type":"code","source":"epochs = 100 # for better result increase the epochs\nbatch_size = 500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd7bac14ca0d7a65fbce893e502549984217c898"},"cell_type":"markdown","source":"<a id=\"8\"></a>\n### D) Data Augmentation\n* In order to prevent Over Fitting, we are making minor changes to our training data and through these changes we are reproducing imprint like img.\n"},{"metadata":{"trusted":true,"_uuid":"c5ccadbeedfea5c157512c804495401b422d4865"},"cell_type":"code","source":"# Data augmentation\ndatagen = ImageDataGenerator(\n    featurewise_center = False, # set input mean to 0 over the dataset\n    samplewise_center = False,  # set each sample mean to 0 \n    featurewise_std_normalization = False,  # divide inputs by std of the dataset\n    samplewise_std_normalization = False,   # divide each input by its std\n    zca_whitening = False,  # dimension reduction\n    rotation_range = 0.5,  # Randomly rotate images in the range 5 degrees\n    zoom_range = 0.5,   # Randomly zoom image 5%\n    width_shift_range = 0.5,  # Randomly shift images horizontally 5%\n    height_shift_range = 0.5,  # Randomly shift images vertically 5%\n    horizontal_flip = False, # Randomly flip images\n    vertical_flip = False) # Randomly flip images\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ede7f155aada6c8a32eb120112d6dabce5b303fc"},"cell_type":"markdown","source":"<a id=\"9\"></a>\n### E) Fit the Model\n"},{"metadata":{"trusted":true,"_uuid":"c138e0cdd58c1c6f9734cbfa883b2c65f5998b08"},"cell_type":"code","source":"history = model.fit_generator(datagen.flow(X_train, Y_train, batch_size),\n                             epochs = epochs, validation_data = (X_test, Y_test), steps_per_epoch = X_train.shape[0] // batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6440cdf800abc5e9c3dce5673420faadd64e99a7"},"cell_type":"markdown","source":"<a id=\"10\"></a>\n### F) Evaluate the Model\n* Test Loss visualization\n* Confusion matrix"},{"metadata":{"trusted":true,"_uuid":"4d939cac1657b271b0f55afe1aba1059053ffa87"},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation\nplt.plot(history.history['val_loss'], color='r', label= \"validation loss\")\nplt.title(\"Test Loss\")\nplt.xlabel(\"Number of epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2634ecadaf9ba3e524b957374bea0c73bf52f43"},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation\nplt.plot(history.history['val_acc'], color='b', label= \"validation accuracy\")\nplt.title(\"Test Accuracy\")\nplt.xlabel(\"Number of epochs\")\nplt.ylabel(\"Accuracy\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ab5938f03bceee1dc9166c8588672176097dfc6f"},"cell_type":"code","source":"#Confusion Matrix\n# Predict the values from the validation dataset\ny_pred = model.predict(X_test)\n# Convert predictions classes to one hot vectors\ny_pred_classes = np.argmax(y_pred, axis=1)\n# Convert validation observation to one hot vectors\ny_true = np.argmax(Y_test, axis=1)\n# Compute the confussion matrix\nconfusion_mtx = confusion_matrix(y_true, y_pred_classes)\n# plot the confusion matrix\nf,ax = plt.subplots(figsize=(16,8))\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.01, cmap=\"Blues\", linecolor=\"Green\", fmt='.1f', ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16f2f54c1ff4ed66f051d0c809a2defa8ea0977d"},"cell_type":"markdown","source":"<a id=\"11\"></a> <br>\n> # CONCLUSION \n* If you want a more detailed kernel. Check out DATAI TEAM's Convolutional Neural Network (CNN) Tutorial Kernel. https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\n---\n<br> **Thank you for your votes and comments.**                                                                                                                                             \n<br>**If you have any suggest, May you write for me, I will be happy to hear it.**"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}