{"cells":[{"metadata":{},"cell_type":"markdown","source":"Started on 24 Sep 2019"},{"metadata":{},"cell_type":"markdown","source":"## Introduction"},{"metadata":{},"cell_type":"markdown","source":"#### This notebook uses a Convolutional Neural Network (CNN) to classify Kannada digits, from 1 through 9."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Examine the data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# load data from csv files\ntrain_df = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest_df = pd.read_csv('../input/Kannada-MNIST/test.csv')\ndig_df = pd.read_csv('../input/Kannada-MNIST/Dig-MNIST.csv')\nprint(train_df.shape, test_df.shape, dig_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* In this 'Kannada MNIST' competition, there are 60,000 labelled images in the train set ('train.csv') and 5,000 unlabelled images in the test set ('test.csv'). There is also an additional set of 10,240 labelled images ('Dig-MNIST.csv') which can be used as another dev set."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['label'].value_counts().sort_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dig_df['label'].value_counts().sort_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create arrays from dataframes\nX = train_df.drop(['label'], axis=1).values\nY = train_df['label']\nX_test = test_df.drop(['id'], axis=1).values\nX_dig = dig_df.drop(['label'], axis=1).values\nY_dig = dig_df['label']\nprint(X.shape, Y.shape, X_test.shape)\nprint(X_dig.shape, Y_dig.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Here are some examples of the Kannada digits from the train dataset with the labels:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# look at some of the digits from train dataset\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"label=%d\" % Y[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Kannada digit 6 & 9 are very similar. So are digit 3 & 7. I expect these digits would be the sources of misclassification."},{"metadata":{},"cell_type":"markdown","source":"#### And some examples of the digits from the dig dataset:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# look at some of the digits from dig dataset\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_dig[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"label=%d\" % Y_dig[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### And some from the test dataset:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# look at some of the digits from test dataset\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_test[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My eyeball classification for the above digits would be:\n* 3 0 2 6 7 7 1 9 3 4\n* 8 8 0 9 8 1 5 1 5 9\n* 3 7 6 0 2 0 8 7 0 0\n* 8 9 2 3 2 4 6 0 3 8"},{"metadata":{},"cell_type":"markdown","source":"#### Let's set up a dev set from the train data for checking the performance of the CNN classifier."},{"metadata":{"trusted":true},"cell_type":"code","source":"# set up a dev set to check the performance of the CNN classifier\nfrom sklearn.model_selection import train_test_split\nX_train, X_dev, Y_train, Y_dev = train_test_split(X, Y, test_size=0.1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(X_train.shape, Y_train.shape)\nprint(X_dev.shape, Y_dev.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the data for use in CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"# prepare the data for CNN\n\n# reshape flattened data into 3D tensor & standardize the values in the datasets by dividing by 255\nn_x = 28\ntrain_img = X_train.reshape((-1, n_x, n_x, 1)).astype('float32')/255.\ndev_img = X_dev.reshape((-1, n_x, n_x, 1)).astype('float32')/255.     # similarly for dev set\ntest_img = X_test.reshape((-1, n_x, n_x, 1)).astype('float32')/255.   # similarly for test set\ndig_img = X_dig.reshape((-1, n_x, n_x, 1)).astype('float32')/255.     # similarly for dig set\nprint(train_img.shape, dev_img.shape, test_img.shape, dig_img.shape)\n\n# one-hot encode the labels in Y_train, Y_dev, Y_dig\nfrom keras.utils.np_utils import to_categorical\ntrain_labels = to_categorical(Y_train)\ndev_labels = to_categorical(Y_dev)\ndig_labels = to_categorical(Y_dig)\nprint(train_labels.shape, dev_labels.shape, dig_labels.shape)\nprint(Y_dig[8], dig_labels[8])\nplt.figure(figsize=(1,1))\nplt.imshow(X_dig[8].reshape((28,28)),cmap=plt.cm.binary)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create CNN Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# use Keras data generator to augment the training set\n\nfrom keras_preprocessing.image import ImageDataGenerator\ndata_augment = ImageDataGenerator(rotation_range=10, zoom_range=0.1, \n                                 width_shift_range=0.1, height_shift_range=0.1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# build the CNN from keras\nfrom keras import models\nfrom keras import layers\n\nmodel = models.Sequential()\nmodel.add(layers.Conv2D(32, kernel_size=3, padding='same', activation='relu', input_shape=(28, 28, 1)))\nmodel.add(layers.Conv2D(32, kernel_size=3, padding='same', activation='relu'))\nmodel.add(layers.BatchNormalization(momentum=0.15))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2)))\nmodel.add(layers.Conv2D(32, kernel_size=5, padding='same', activation='relu'))\nmodel.add(layers.Dropout(0.4))\nmodel.add(layers.Conv2D(64, kernel_size=3, padding='same', activation='relu'))\nmodel.add(layers.Conv2D(64, kernel_size=3, padding='same', activation='relu'))\nmodel.add(layers.BatchNormalization(momentum=0.15))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2)))\nmodel.add(layers.Conv2D(64, kernel_size=5, padding='same', activation='relu'))\nmodel.add(layers.Dropout(0.4))\nmodel.add(layers.Conv2D(128, kernel_size=3, padding='same', activation='relu'))\nmodel.add(layers.Conv2D(128, kernel_size=3, padding='same', activation='relu'))\nmodel.add(layers.BatchNormalization(momentum=0.15))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2)))\nmodel.add(layers.Conv2D(128, kernel_size=5, padding='same', activation='relu'))\nmodel.add(layers.Dropout(0.4))\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(128, activation='relu'))\nmodel.add(layers.Dropout(0.4))\nmodel.add(layers.Dense(64, activation='relu'))\nmodel.add(layers.Dropout(0.4))\nmodel.add(layers.Dense(10, activation='softmax'))\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compile the model\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', \n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set a learning rate annealer\nfrom keras.callbacks import ReduceLROnPlateau\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy',patience=3,factor=0.5,min_lr=0.00001,\n                                           verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Run the model using the train and validation datasets, and capture histories to visualise the performance."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Train and validate the model\nepochs = 75\nbatch_size = 64\nhistory = model.fit_generator(data_augment.flow(train_img, train_labels, batch_size=batch_size), \n                              epochs=epochs, steps_per_epoch=train_img.shape[0]//batch_size, \n                              validation_data=(dev_img, dev_labels), callbacks=[learning_rate_reduction])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# plot and visualise the training and validation losses\nloss = history.history['loss']\ndev_loss = history.history['val_loss']\nepochs = range(1, len(loss) + 1)\n\nfrom matplotlib import pyplot as plt\nplt.plot(epochs, loss, 'bo', label='training loss')\nplt.plot(epochs, dev_loss, 'b', label='validation loss')\nplt.title('Training and Validation Loss')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Error Analysis"},{"metadata":{},"cell_type":"markdown","source":"#### Do error analysis on the predictions for X_dig"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(dig_img, dig_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Accuracy for the dig dataset is above 85%."},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_dig = model.predict(dig_img)\ndig_df['pred'] = np.argmax(pred_dig, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# look at those that were classified wrongly in X_dig\ndig_df['correct'] = dig_df['label'] - dig_df['pred']\nerrors = dig_df[dig_df['correct'] != 0]\nerror_list = errors.index\nprint('Number of errors is ', len(errors))\nprint('The indices are ', error_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Some of the wrong predictions for dig dataset:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plot images of some of the wrong predictions for X_dig\nplt.figure(figsize=(15,10))\nfor i in range(60):\n    plt.subplot(6, 10, i+1)\n    plt.imshow(X_dig[error_list[i]].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"true={}\\npredict={}\".format(dig_df['label'][error_list[i]], \n                                           dig_df['pred'][error_list[i]]), y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at those that were predicted wrongly, there are quite several difficult and ambiguous ones."},{"metadata":{},"cell_type":"markdown","source":"## Make Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict on test set\npredictions = model.predict(test_img)\nprint(predictions.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set the predicted labels to be the one with the highest probability\npredicted_labels = np.argmax(predictions, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Here are some examples of the predictions made:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# look at some of the predictions for test_X\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(test_img[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"predict=%d\" % predicted_labels[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Looks reasonable...."},{"metadata":{"trusted":true},"cell_type":"code","source":"# create submission file\nsubmission = pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nsubmission['label'] = predicted_labels\n# generate submission file in csv format\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}