{"cells":[{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"# CONTENT\n\n* [1. Intoduction](#1)\n* [2. Importing Packages and Collecting Data](#2)\n* [3. Data Description and Preparation](#3)\n* [4. Data Preprocessing](#4)\n * [4.1 Normalize Images](#4.1)\n * [4.2 Label encoding of Target varible](#4.2)\n * [4.3 Reshape](#4.3)\n * [4.4 Split training and valdiation set](#4.4)\n* [5. Convolutional Neural Network](#5)\n * [5.1 Define the model](#5.1)\n * [5.2 Outline the model](#5.2)\n * [5.3 Data augmentation](#5.3)\n * [5.4 Compile and Train the model](#5.4)\n * [5.5 Evaluate the model](#5.5)\n* [Prediction and Submission ]()"},{"metadata":{},"cell_type":"markdown","source":"# 1. Intoduction \nThis is a 5 layers Sequential Convolutional Neural Network for digits recognition trained on Kannada MNIST dataset. I choosed to build it with keras API (Tensorflow backend) which is very intuitive. Firstly, I will prepare the data (handwritten digits images) then i will focus on the CNN modeling and evaluation.\n\n"},{"metadata":{},"cell_type":"markdown","source":"# 2. Importing Packages and Collecting Data "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"'''Importing Data Manipulattion Moduls'''\nimport numpy as np\nimport pandas as pd\n\n'''Seaborn and Matplotlib Visualization'''\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\n'''Importing preprocessing libraries'''\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\n'''Display markdown formatted output like bold, italic bold etc.'''\nfrom IPython.display import Markdown\ndef bold(string):\n display(Markdown(string))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Importing tensorflow libraries'''\nimport tensorflow as tf \nprint(tf.__version__)\n\nfrom tensorflow.keras import layers, models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Read in train and test data from csv files'''\ntrain = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('../input/Kannada-MNIST/test.csv')\nsample_sub = pd.read_csv(\"../input/Kannada-MNIST/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Data Description and Preparation "},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Train and test data at a glance.'''\nbold('**Preview of Train Data:**')\ndisplay(train.head(3))\nbold('**Preview of Test Data:**')\ndisplay(test.head(3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Ckecking for null and missing values'''\nbold('**Train Data**')\ndisplay(train.isnull().any(). describe())\nbold('**Test Data**')\ndisplay(test.isnull().any(). describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Seting X and Y'''\ny_train = train['label']\n\n# Drop 'label' column\nX_train = train.drop('label', axis = 1)\n\nX_test = test.drop('id', axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"Let's have a final look at our data\"\"\"\nbold('**Data Dimension for Model Building:**')\nprint('Input matrix dimension:', X_train.shape)\nprint('Output vector dimension:',y_train.shape)\nprint('Test data dimension:', X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Visualizating the taget distribution'''\nplt.figure(figsize = (8,8))\nsns.countplot(y_train, palette='Paired')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Cool!!**"},{"metadata":{"trusted":true},"cell_type":"code","source":"images = train.iloc[:,1:].values\nimages = images.astype(np.float)\n\n# convert from [0:255] => [0.0:1.0]\nimages = np.multiply(images, 1.0 / 255.0)\n\nimage_size = images.shape[1]\nprint('image_size => {0}'.format(image_size))\n\n# in this case all images are square\nimage_width = image_height = np.ceil(np.sqrt(image_size)).astype(np.uint8)\n\nprint('image_width => {0}\\nimage_height => {1}'.format(image_width, image_height))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Displaying image'''\n# display image\ndef display(img):\n \n # (784) => (28,28)\n one_image = img.reshape(image_width,image_height)\n \n plt.axis('off')\n plt.imshow(one_image, cmap='binary')\n\n# output image \ndisplay(images[8])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**It's Eight in Kannada**"},{"metadata":{},"cell_type":"markdown","source":"# 4. Data Preprocessing "},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Normalize Images \nRescale pixel values from the range of 0-255 to the range 0-1 preferred for neural network models.\n\nScaling data to the range of 0-1 is traditionally referred to as normalization.\n\nThis can be achieved by setting the rescale argument to a ratio by which each pixel can be multiplied to achieve the desired range.\n\nIn this case, the ratio is 1/255 or about 0.0039."},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Normalizing the data'''\nX_train = X_train / 255.0\nX_test = X_test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Label encoding of Target varible \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''convert class labels from scalars to one-hot vectors'''\n# 0 => [1 0 0 0 0 0 0 0 0 0]\n# 1 => [0 1 0 0 0 0 0 0 0 0]\n# ...\n# 9 => [0 0 0 0 0 0 0 0 0 1]\ny_train = tf.keras.utils.to_categorical(y_train, num_classes = 10, dtype='uint8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Reshape \n\nTrain and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values. We reshape all data to 28x28x1 3D matrices.\n\nKeras requires an extra dimension in the end which correspond to channels. MNIST images are gray scaled so it use only one channel. For RGB images, there is 3 channels, we would have reshaped 784px vectors to 28x28x3 3D matrices."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4 Split training and valdiation set \nI choosed to split the train set in two parts : a small fraction (10%) became the validation set which the model is evaluated and the rest (90%) is used to train the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Set the random seed'''\nseed = 44\n'''Split the train and the validation set for the fitting'''\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = 0.1, random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Convolutional Neural Network \n"},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Define the model ****\n\n**What is Convolution Operation?**\n* The first is the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 32 filters for the two firsts conv2D layers and 64 filters for the two last ones. \n* We have some image and feature detector(5*5)\n* Feature detector does not need to be 5 by 5 matrix. It can be 3 by 3 or 7 by 7.\n* Feature detector = kernel = filter\n * Feauture detector detects features like edges or convex shapes. Example, if out input is dog, feature detector can detect features like ear or tail of the dog.\n* feature map = conv(input image, feature detector). Element wise multiplication of matrices.\n* feature map = convolved feature\n* Stride = navigating in input image.\n* We reduce the size of image. This is important bc code runs faster. However, we lost information.\n* We create multiple feature maps bc we use multiple feature detectors(filters).\n* Lets look at gimp. Edge detect: [0,10,0],[10,-4,10],[0,10,0]\n\n* After having convolution layer we use ReLU to break up linearity. Increase nonlinearity. Because images are non linear.\n\n\n**MaxPool2D**\n* The second important layer in CNN is the pooling (MaxPool2D) layer. \n* This layer simply acts as a downsampling filter. \n* It looks at the 2 neighboring pixels and picks the maximal value. \n* These are used to reduce computational cost, and to some extent also reduce overfitting.\n\n\n**Dropout**\n* Dropout is a technique where randomly selected neurons are ignored during training.\n* Dropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample.\n* This technique also improves generalization and reduces the overfitting.\n\n\n**Flattening**\n* The Flatten layer is use to convert the final feature maps into a one single 1D vector. \n* This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers.\n\n\n**Full Connection**\n* Neurons in a fully connected layer have connections to all activations in the previous layer\n* Artificial Neural Network\n"},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Outline the model "},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Set the CNN model'''\n# CNN architechture is In -> [[Conv2D->relu]*2 -> MaxPool2D -> Dropout]*2 -> Flatten -> Dense -> Dropout -> Out\nmodel = models.Sequential()\nmodel.add(layers.Conv2D(32, (5, 5), activation='relu', input_shape=(28,28,1)))\nmodel.add(layers.Conv2D(32, (5, 5), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Dropout(0.25))\n \nmodel.add(layers.Conv2D(64, (5, 5), activation='relu'))\nmodel.add(layers.Conv2D(64, (5, 5), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Dropout(0.25))\n \nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(256, activation='relu'))\nmodel.add(layers.Dropout(0.25))\nmodel.add(layers.Dense(10, activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.3 Data augmentation \nIn order to avoid overfitting problem, we need to expand artificially our handwritten digit dataset. We can make your existing dataset even larger. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.\n\nFor example, the number is not centered The scale is not the same (some who write with big/small numbers) The image is rotated...\n\nApproaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more.\n\nBy applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = tf.keras.preprocessing.image.ImageDataGenerator(\n featurewise_center=False, # set input mean to 0 over the dataset\n samplewise_center=False, # set each sample mean to 0\n featurewise_std_normalization=False, # divide inputs by std of the dataset\n samplewise_std_normalization=False, # divide each input by its std\n zca_whitening=False, # apply ZCA whitening\n rotation_range=10, # randomly rotate images in the range (degrees, 0 to 180)\n zoom_range = 0.1, # Randomly zoom image \n width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)\n height_shift_range=0.1, # randomly shift images vertically (fraction of total height)\n horizontal_flip=False, # randomly flip images\n vertical_flip=False) # randomly flip images\n\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5.4 Compile and Train the model "},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam',\n loss='categorical_crossentropy',\n metrics=['accuracy'])\n\nhistory = model.fit(X_train, y_train, batch_size = 1000, epochs = 10, validation_data = (X_val, y_val), verbose = 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.5 Evaluate the model "},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Training and validation curves'''\nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''confusion matrix'''\nimport seaborn as sns\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nf,ax = plt.subplots(figsize=(8, 8))\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.01,cmap=\"Greens\",linecolor=\"gray\", fmt= '.1f',ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction and Submission "},{"metadata":{"trusted":true},"cell_type":"code","source":"'''predict results'''\nresults = model.predict(X_test)\n\n'''select the indix with the maximum probability'''\nresults = np.argmax(results,axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub['label'] = results\nsample_sub.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}