{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Digit Recognition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Digit racognition using CNN.\n\nI am using data from analytics vidya digit recognition competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Importing required libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as  np\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* **pandas** - we use pandas to handle our csv files\n* **matplotlib & seaborn** - used for charting and plotting","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* **sklearn** - Popular ML library.We will use it for splitting our data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout,BatchNormalization\nfrom keras.optimizers import RMSprop,Adam\nfrom keras.callbacks import ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* **Keras** : Popular Deep learning library,we will use it to build our CNN Network.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Loading data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_X = pd.read_csv('../input/digit-recognition-dataset/Digit Recognition/trian_data1.csv')\ntest_X = pd.read_csv('../input/digit-recognition-dataset/Digit Recognition/test_data1.csv')\ntrain_y = pd.read_csv('../input/digit-recognition-dataset/Digit Recognition/train.csv')\ntest= pd.read_csv('../input/digit-recognition-dataset/Digit Recognition/Test.csv')\nsample_sub = pd.read_csv('../input/digit-recognition-dataset/Digit Recognition/Sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Understanding the train and test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train dataset has {} rows and {} columns'.format(train_X.shape[0],train_X.shape[1]))\nprint('test dataset has {} rows and {} columns'.format(test_X.shape[0],test_X.shape[1]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_X.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_X.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Pixel 0 to Pixel 783**: These are the pixel values of the image metrics.That is each row contains 28 * 28 = 784 (0-783 here) values here.Each one of these values indicates the pixel value at i x 28 + j th pixel position in the image metric.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**train_y** file contains a target value i.e **label** for train data\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y.head()\ntrain_y = train_y.iloc[:,1]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_y.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Checking Target class distribution.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train_y.value_counts()\nsns.barplot(y.index,y)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Normalize pixel values\n\nFor most image data, the pixel values are integers with values between 0 and 255.\n\nNeural networks process inputs using small weight values, and inputs with large integer values can disrupt or slow down the learning process. As such it is good practice to normalize the pixel values so that each pixel value has a value between 0 and 1.\n\nIt is valid for images to have pixel values in the range 0-1 and images can be viewed normally.\n\nThis can be achieved by dividing all pixels values by the largest pixel value; that is 255. This is performed across all channels, regardless of the actual range of pixel values that are present in the image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_X = train_X /255\ntest_X =test_X /255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reshape","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_X= train_X.values.reshape(-1,28,28,1)\ntest_X = test_X.values.reshape(-1,28,28,1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The shape of train set now is',train_X.shape)\nprint('The shape of test set now is',test_X.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Encoding Target Values\n\n\n\nNow we will encode our target value.Keras inbuild library to_categorical() is used to do the on-hot encoding.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y = to_categorical(train_y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Splitting train and test data\n\nNow we will split out training data into train and validation data. 20 percent of the training data will be used for validation purpose.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train,X_test,y_train,y_test = train_test_split(train_X,train_y,random_state = 42 , test_size=0.20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X_train[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generating more data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In order to avoid overfitting problem , we need to expand our dataset artificially.\n\nWe can do it by some **data augmentation techniques**.\n\nBy applying these techniques we can double or triple the number of training examples and create a very robust model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n            featurewise_center = False, # set input mean to 0 over the dataset\n            samplewise_center = False,  # set each sample mean to 0\n            featurewise_std_normalization = False, # divide inputs by std of the dataset\n            samplewise_std_normalization = False,  # divide each input by its std\n            zca_whitening = False,   # apply ZCA whitening\n            rotation_range = 10,     # randomly rotate images in the range (degrees, 0 to 180)\n            zoom_range = 0.1,       # Randomly zoom image \n            width_shift_range = 0.1,  # randomly shift images horizontally (fraction of total width)\n            height_shift_range = 0.1, # randomly shift images vertically (fraction of total height)\n            horizontal_flip = False,  # randomly flip images\n            vertical_flip = False     # randomly flip images\n)\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Modelling\n\n### CNN","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same',activation ='relu'))\n\nmodel.add(BatchNormalization(momentum = .05))\n\nmodel.add(MaxPool2D(pool_size=(2,2)))\n\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', activation ='relu'))\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', activation ='relu'))\nmodel.add(BatchNormalization(momentum=0.05))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', activation ='relu'))\nmodel.add(BatchNormalization(momentum=.05))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = 'relu'))\nmodel.add(Dropout(0.4))\nmodel.add(Dense(10, activation = \"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Optimizer\n\n\nIn simpler terms, optimizers shape and mold your model into its most accurate possible form by futzing with the weights. The loss function is the guide to the terrain, telling the optimizer when it’s moving in the right or wrong direction.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = Adam(learning_rate=0.001 , beta_1=0.9 ,beta_2 = 0.999)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer=optimizer , loss=['categorical_crossentropy'],metrics = ['accuracy'])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Leraning rate reduction\n\n\n\nIn order to make the optimizer converge faster and closest to the global minimum of the loss function, i used an annealing method of the learning rate (LR).\n\nWith the ReduceLROnPlateau function from Keras.callbacks, i choose to reduce the LR by half if the accuracy is not improved after 3 epochs.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_acc',\n                                            patience = 5 ,\n                                            verbose = 1,\n                                            factor = 0.5 , \n                                            min_lr = 0.00001)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fitting Our Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 20\nbatch_size = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(datagen.flow(X_train,y_train,batch_size = batch_size),\n                              epochs = epochs ,\n                              validation_data = (X_test,y_test),\n                              verbose = 2,\n                              steps_per_epoch = X_train.shape[0]//batch_size,\n                              callbacks =[learning_rate_reduction])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluatin our approach using graph","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax=plt.subplots(2,1)\nfig.set\nx=range(1,1+epochs)\nax[0].plot(x,history.history['loss'],color='red')\nax[0].plot(x,history.history['val_loss'],color='blue')\n\nax[1].plot(x,history.history['accuracy'],color='red')\nax[1].plot(x,history.history['val_accuracy'],color='blue')\nax[0].legend(['trainng loss','validation loss'])\nax[1].legend(['trainng acc','validation acc'])\nplt.xlabel('Number of epochs')\nplt.ylabel('accuracy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pre_test=model.predict(X_test)\ny_pre_test=np.argmax(y_pre_test,axis=1)\ny_test=np.argmax(y_test,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"conf=confusion_matrix(y_test,y_pre_test)\nconf=pd.DataFrame(conf,index=range(0,10),columns=range(0,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nconf\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.set(font_scale=1.4)#for label size\nsns.heatmap(conf, annot=True,annot_kws={\"size\": 16},cmap=plt.cm.Blues)# font size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some Misclassified Images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x=(y_pre_test-y_test!=0).tolist()\nx=[i for i,l in enumerate(x) if l!=False]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax=plt.subplots(1,4,sharey=False,figsize=(15,15))\n\nfor i in range(4):\n    ax[i].imshow(X_test[x[i]][:,:,0])\n    ax[i].set_xlabel('Real {}, Predicted {}'.format(y_test[x[i]],y_pre_test[x[i]]))\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predicting for test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pre_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_y = model.predict(test_X)\ntest_y =np.argmax(test_y,axis=1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test1 = test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test1 = test1.iloc[:,0:1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output = pd.DataFrame({'filename': test1.iloc[1:,0],\n                     'label': test_y})\noutput.to_csv('submission1.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Reference and credit:\n\n1.[Analytics Vidya](http://www.analyticsvidhya.com/blog/2018/12/guide-convolutional-neural-network-cnn/)\n\n2.[Kaggle notebook](http://www.kaggle.com/shahules/indian-way-to-learn-cnn)\n\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}