{"cells":[{"metadata":{},"cell_type":"markdown","source":"> Currently i am writing this notebook my rank is `300` in MNIST with score of `.99567`9\n16/10/20\n\n# MNIST - Digit Recognition using Tensorflow\n\n### Contents\n\n* 1. Importing Libraries\n* 2. Importing Dataset\n    * 2.1 Histogram\n    * 2.1 Plotting Figures\n* 3. Data Preprocessing\n    * 3.1 Converting Labels to Categorical\n    * 3.2 Reshaping\n    * 3.3 Data Normalization\n    * 3.4 Spliting Data into Training Set & Validation Set\n    * 3.5 Data Argumentation\n* 4. CNN Model\n    * 4.1 Creating Model\n    * 4.2 Model Training\n    * 4.3 Model Accuracy and Loss Plot\n    * 4.4 Confusion Matrix and Classification Report\n* 5. Final Prediction"},{"metadata":{},"cell_type":"markdown","source":"# 1. Importing Libraries"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport tensorflow as tf\nfrom keras.preprocessing.image import ImageDataGenerator, load_img\nfrom keras.layers import Conv2D, BatchNormalization, Activation, MaxPooling2D, GlobalAveragePooling2D, Dense, Flatten, Dropout\nfrom keras.optimizers import RMSprop, Adam, SGD\nfrom keras import regularizers\nfrom keras.callbacks import CSVLogger, ModelCheckpoint, ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Importing Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')\ntrain.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train set: ',train.shape, \"\\t Test Set\", test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So there are 42k images in training set and 28k images in testing set"},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train['label']\n# droping label cloumn in training set\ntrain.drop('label', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1 Histogram"},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.countplot(y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The plot shows the frequency of Labels, the most fequent label is 1"},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Plotting Figures"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,5))\nfor i in range(40):\n    plt.subplot(4,10,i+1)\n    img = train.iloc[i,:].values.reshape(28,28)\n    plt.imshow(img)\n    plt.axis('off')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3 Data Preprocessing"},{"metadata":{},"cell_type":"markdown","source":"### 3.1 Converting Labels to Categorical"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical \ny = to_categorical(y, num_classes = 10)\ny[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Labels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors (ex : 9 -> [0,0,0,0,0,0,0,0,0,1])."},{"metadata":{},"cell_type":"markdown","source":"### 3.2 Reshaping "},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.values.reshape(train.shape[0], 28, 28, 1)\ntest = test.values.reshape(test.shape[0], 28, 28, 1)\nprint('Reshaped Train set: ',train.shape, \" & Reshaped Test Set\", test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"MNIST images are gray scaled so it use only 1 channel, we would reshape the image to 28x28x1"},{"metadata":{},"cell_type":"markdown","source":"### 3.3 Data Normalization\n\nWe have to perform normalization on  images [0..1] data than on [0..255]. to reduce the effect of illumination's differences.\n\nThe Normailzed data learns faster as there is less gradient to cover"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.astype(\"float32\")/255.0\ntest = test.astype(\"float32\")/255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Spliting Data into Training Set & Validation Set\nWe have 42k images in training set we can split it into training and validation sets, so that we can be sure that the model is not ovefitted on training data. I have splites 20 % of data to validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_val, y_train, y_val = train_test_split(train, y, test_size=0.25, random_state=0)\n\nprint(\"Number of samples in Training set :\", X_train.shape[0])\nprint(\"Number of samples in Validation set :\", X_val.shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.5 Data Argumentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rotation_range=10,\n                                   zoom_range=0.1,\n                                   width_shift_range=0.1,\n                                   height_shift_range=0.1\n                                  )\n\ntraining_set = train_datagen.flow(X_train, y_train,\n                                  batch_size=64\n                                 )\n\nval_datagen = ImageDataGenerator()\nval_set = val_datagen.flow(X_val, y_val,\n                           batch_size=64\n                          )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Randomly rotate some training images by 10 degrees\n* Randomly Zoom by 10% some training images\n* Randomly shift images horizontally by 10% of the width\n* Randomly shift images vertically by 10% of the height\n\nSince vertical_flip nor horizontal_flip could have lead to misclassify symetrical numbers such as 6 and 9, so i did not use it"},{"metadata":{},"cell_type":"markdown","source":"# 4. CNN Model"},{"metadata":{},"cell_type":"markdown","source":"### 4.1 Creating Model\nI tried several model and this has given me the best accuracy so far"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential()\n\nmodel.add(Conv2D(64, kernel_size=(5,5), padding='same', activation='relu', input_shape=(28,28,1)))\nmodel.add(Conv2D(64, kernel_size=(5,5), padding='same', activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n\nmodel.add(Conv2D(128, kernel_size=(3,3), activation='relu', padding='same'))\nmodel.add(Conv2D(128, kernel_size=(3,3), activation='relu', padding='same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.5))\n\nmodel.add(Dense(256, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.3))\n\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy']\n             )\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# If the model is not improving on validation, we need to reduce the learning rate, If val loss is not improved in 4 epoch then lr will be reduced \nreduce_lr = ReduceLROnPlateau(monitor='val_loss', \n                              factor=0.2, \n                              patience=4, \n                              verbose=1, \n                              min_delta=0.0001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Model Training "},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch = training_set.n // training_set.batch_size\nvalidation_steps = val_set.n // val_set.batch_size\n\nhist = model.fit(x=training_set,\n                 validation_data=val_set,\n                 epochs=35,\n                 callbacks=[reduce_lr],\n                 steps_per_epoch=steps_per_epoch,\n                 validation_steps=validation_steps\n                )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"i might have run it more a bit longer i think 25 or 20 epochs will do fine"},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Model Accuracy and Loss Plot"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,5))\nplt.subplot(1,2,2)\nplt.plot(hist.history['accuracy'])\nplt.plot(hist.history['val_accuracy'])\nplt.title('Model Accuracy')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\nplt.legend(['train', 'test'], loc='upper left')\n\nplt.subplot(1,2,1)\nplt.plot(hist.history['loss'])\nplt.plot(hist.history['val_loss'])\nplt.title('model Loss')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, acc_val = model.evaluate(val_set)\n_, acc_tr = model.evaluate(val_set)\nprint(\"\\nFinal Accuracy on training set : {:.2f}% & accuracy on validation is set: {:.2f}%\".format(acc_tr*100, acc_val*100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"now the final accuracy of model on test and training set satisfactiory"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import plot_model\nplot_model(model, show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.4 Confusion Matrix and Classification Report"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_pred = model.predict(val_set)\nval_pred = np.argmax(val_pred, axis=1)\ny_val = np.argmax(y_val, axis=1)\n\nfrom sklearn.metrics import confusion_matrix, classification_report\nprint(\"Confusion Matrix\")\ncm = confusion_matrix(y_val, val_pred)\nprint(cm)\nprint(\"Classification Report\")\nprint(classification_report(y_val, val_pred))\n\n#g = sns.heatmap(cm, cmap='Blues')\nplt.figure(figsize=(8,8))\nplt.imshow(cm, interpolation='nearest')\nplt.colorbar()\ntarget_names = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\ntick_mark = np.arange(len(target_names))\n_ = plt.xticks(tick_mark, target_names)\n_ = plt.yticks(tick_mark, target_names)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Final Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = model.predict(test)\nres = np.argmax(pred, axis=1)\nsubmission = pd.DataFrame({\"ImageId\":[i+1 for i in range(len(test))],\n                           \"Label\": res})\nsubmission.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Your feedback in comments is much appreciated, Comment if you have any doubts or for inprovement\n* Please UPVOTE if you LIKE this notebook, it will keep me motivated"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}