{"cells":[{"metadata":{"_uuid":"c70dc539-2e6d-40f9-9a28-6083a10e5e49","_cell_guid":"e4ac114f-9456-44e2-87e3-ba6a9b6aed3c","trusted":true,"id":"yQ9u1XznGtLr"},"cell_type":"markdown","source":"# Introduction\n\n\nFor this project we need to classify images of hand written digits which are in Kannada. Kannada is a language spoken predominantly by people of Karnataka in southwestern India. The language has roughly 45 million native speakers and is written using the Kannada script (see Wikipedia).\n\nThe format is similar to MNIST in terms of how the data is structured."},{"metadata":{"_uuid":"e159af88-aa68-477e-a6d1-ddc49ff7f949","_cell_guid":"bec208fe-8377-4ef7-85d5-9a0ee24b4374","trusted":true,"id":"eScRlYdMJYXG"},"cell_type":"markdown","source":"Packages and Libraries:\n\nFor this project, we will make use of the tensorflow, keras, and scikit-learn libraries for machine learning. Numpy and Pandas for scientific computing and data manipulation. Seaborn and Matplotlib will be used for visualizations and Exploratory Data Analysis (EDA).\n\n\n\n*   tensorflow==2.2.0-rc2\n* keras==2.3.0\n*   scikit-learn==0.22.1\n*   numpy==1.18.2\n*   pandas==0.25.3\n* seaborn==0.10.0\n*   matplotlib==3.2.1"},{"metadata":{"_uuid":"04afb775-cd1b-448d-96d0-2e459df4557f","_cell_guid":"70eae4a3-d3f0-48a6-93b6-490bf1aece94","trusted":true,"id":"mXIPqjY16R5J"},"cell_type":"markdown","source":"# Exploratory Data Analysis"},{"metadata":{"_uuid":"0b1c3fc4-761c-470d-9043-7fd354f9627d","_cell_guid":"55b6e239-5a10-4288-aa90-691a74eafee9","trusted":true,"id":"c8KLQ4AUuqyS","outputId":"e1eebf37-545f-4cf4-93a3-0367285de222"},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport sklearn\nimport tensorflow as tf\n\n\nfrom sklearn import cluster\nfrom sklearn import metrics\nfrom sklearn.metrics import mean_squared_error, r2_score\nfrom sklearn.preprocessing import scale\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.cluster import AgglomerativeClustering\nfrom sklearn.linear_model import LogisticRegression\nfrom pandas.plotting import scatter_matrix\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\n\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import cross_val_score\n\n\n\n# load training & test datasets\ntrain = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('../input/Kannada-MNIST/test.csv')\ntrain.columns","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"a73eca12-67b9-4c20-9b78-3f653ab49d25","_cell_guid":"53119f43-2440-42b6-b616-21dfd32a2192","trusted":true,"id":"6YDVjjnm20Yl","outputId":"f0e6eea7-1175-4029-d46d-879018f4ee6f"},"cell_type":"code","source":"train.head()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"8ba4903b-98b2-480f-b111-cb3ca00d46c5","_cell_guid":"a913bfc3-d7c0-4366-9d1b-b12e0af45f9c","trusted":true,"id":"Sfl_bI8Xil7K","outputId":"d04c873e-b914-4d9b-d656-9a71757ce59c"},"cell_type":"code","source":"test.head()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"de105df3-72b9-4df3-af71-3185798d0aa2","_cell_guid":"749e8000-fa1a-41ee-ba03-2d10ef61f12e","trusted":true,"id":"KU4HfCl48SE5","outputId":"f1ca6126-0112-4842-949c-73a813a3f283"},"cell_type":"code","source":"#Getting the shape of the pixel data\nprint( 'Shape of the data is :' , train.loc[2,'pixel0':].shape)\n\nprint('Max pixel value:' ,train.loc[2,'pixel0':].max())\n\nprint('Min pixel value:' ,train.loc[2,'pixel0':].min())","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"2b845bf2-9ca2-4926-9d98-292379cadd20","_cell_guid":"d0b59d13-7c0e-423d-bdfa-c0ed0a83ae24","trusted":true,"id":"sZs24YSz7x4B"},"cell_type":"markdown","source":"Exploring the training data set we have a label column which shows the label of the pixel data in the following columns.\nThere are 784 columns for pixel data (28*28) with values ranging from 0-255"},{"metadata":{"_uuid":"ade4a18e-afda-48ef-a3a8-42a53120f93e","_cell_guid":"1695ed0f-99bc-47a7-8b55-f20908c6c573","trusted":true,"id":"SBzuotPv9p1h"},"cell_type":"markdown","source":"### Visualizing Kannada digits"},{"metadata":{"_uuid":"1111d95e-e388-456c-b447-a17b77438836","_cell_guid":"544cd696-9683-4e22-b527-b6765ab83c76","trusted":true,"id":"x0ir_6-RusfT","outputId":"85843832-2638-4b3e-eed4-8f94b7597769"},"cell_type":"code","source":"plt.figure()\nplt.imshow(train.loc[2,'pixel0':].values.reshape((28,28)))\nplt.colorbar()\nplt.grid(False)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"b3f7905b-0a34-4bcf-98a5-a57d174f0d79","_cell_guid":"6541697c-d213-4ed4-b3fb-540f927f263c","trusted":true,"id":"R_rTor5sPQGb","outputId":"e4b9b457-25e6-4b69-c314-35c1634a976a"},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nfor i in range(25):\n\tplt.subplot(5,5,i+1)\n\tplt.xticks([])\n\tplt.yticks([])\n\tplt.grid(False)\n\tplt.imshow(train.loc[i,'pixel0':].values.reshape((28,28)) ,  cmap=plt.cm.binary)\n\tplt.xlabel(train.loc[i,'label'])","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"bdf09444-a3c0-43fa-a455-7c0530ea8229","_cell_guid":"7d9aeefd-ace4-4c03-9433-ea6db79bb4c3","trusted":true,"id":"cnPYOxq79_ct"},"cell_type":"markdown","source":"### Visualizing Embeddings with T-SNE \n\nt-SNE is a dimensionality reduction algorithm which is often used for visualization. It learns a mapping from a set of high-dimensional vectors, to a space with a smaller number of dimensions (usually 2), which is hopefully a good representation of the high-dimensional space."},{"metadata":{"_uuid":"0c7b171a-fa07-4d15-b02b-5b61acef6c58","_cell_guid":"2071d827-5ed9-411d-b3ac-023918985eb2","trusted":true,"id":"jAs4a8DbP4Bz","outputId":"8b86d73f-baae-46fb-9c73-d599826b0614"},"cell_type":"code","source":"from sklearn.manifold import TSNE\n\n# Sample from the training set\nsample_size = 8000\n\nnp.random.seed(2018)\nidx = np.random.choice(60000, size=sample_size, replace=False)\ntrain_sample = train.loc[idx,'pixel0':]\nlabel_sample = train.loc[idx,'label']\n\ntrain_sample\n# Generate 2D embedding with TSNE\nembeddings = TSNE(verbose=2).fit_transform(train_sample)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"526fa770-ea04-4077-8084-0e43a82d707d","_cell_guid":"90855f15-813f-427d-b62e-c3297d9984cf","trusted":true,"id":"HNAi-lwecmQF","outputId":"bcad6b49-efe3-4499-d8e6-f9e19f506fb9"},"cell_type":"code","source":"\n# Visualize TSNE embedding\nvis_x = embeddings[:, 0]\nvis_y = embeddings[:, 1]\n\nplt.figure(figsize=(10,7))\nplt.scatter(vis_x, vis_y, c=label_sample, cmap=plt.cm.get_cmap(\"jet\", 10), marker='.')\nplt.colorbar(ticks=range(10))\nplt.clim(-0.5, 9.5)\nplt.show()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"d505358d-abcf-4ebb-98e2-af8c7d440fec","_cell_guid":"e2c20907-4896-4d8e-ba9d-ccc3ad26a4cb","trusted":true,"id":"PNUWR3n8WCg6","outputId":"0cbcc52d-e374-4eed-af5e-7b971d845e68"},"cell_type":"code","source":"y_train = train['label']\n\ny_train","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"d6c4bd68-143d-4ebc-808b-24b2cd3e5ac4","_cell_guid":"08ac40da-44ee-4a4e-b198-68399fb931ac","trusted":true,"id":"FCcZ0pZkVNd3","outputId":"3efa4516-1ad2-4dc1-b69a-d47dde8478ce"},"cell_type":"code","source":"train['label'].value_counts()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"6333bfdb-3a9d-4540-938e-e96cf4a14c1d","_cell_guid":"917d4612-25eb-4233-a3e8-65fa3a139c62","trusted":true,"id":"2oB-rqwIaikG","outputId":"4513adb7-37b0-4863-b257-c3c7c797ebe0"},"cell_type":"code","source":"g = sns.countplot(y_train)\n\ny_train.value_counts()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"fbe3beca-64c8-4851-97a6-b06e9a74d733","_cell_guid":"50cea818-17b4-47e5-9c08-dcd94e1826d9","trusted":true,"id":"yA6I-9zSf6VE"},"cell_type":"markdown","source":"**Classification Problem**"},{"metadata":{"_uuid":"348f4bcc-fefb-4db4-a88b-472c5d84c125","_cell_guid":"bba92731-7ebf-4eab-ab40-05da5751d5ca","trusted":true,"id":"iWNTs3H_KbD1"},"cell_type":"markdown","source":"* For this problem, we decided to use Convolutional Neural Network (CNN) which are a class of deep neural networks that have several proven applications in analyzing visual imagery particularly in image classification. The name \"Convolutional Neural Network\" indicates that the network uses convolution which is a specialized kind of linear operation instead of general matrix multiplication in at least one of their layers.\n* A neural network is clasified as Convolutional if it has more than one convolutional layer. A typical architecure includes convolutional layers, pooling layers, fully connected layers and normalization layers.\n* The filters are used to extract the features of the image and the fully connected layer is responsible for classification according to extracted features."},{"metadata":{"_uuid":"2815b072-767f-4044-9efd-767eb25eeb5b","_cell_guid":"418e7b10-2be2-4712-982b-0b5eddd58544","trusted":true,"id":"G-EmXVSomeAQ"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\n%matplotlib inline\n\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.utils import to_categorical\n\nnp.random.seed(2)\n\nfrom sklearn.model_selection import train_test_split,GridSearchCV\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nsns.set(style='white', context='notebook', palette='deep')","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"ecd821a5-1a76-43f1-bb1e-37d104380720","_cell_guid":"862457eb-fe5c-4dba-a227-e0c96ed37fb8","trusted":true,"id":"c0pTO_V8hqkm","outputId":"d5d6177d-71d7-4f16-c256-1312806c8e7a"},"cell_type":"code","source":"train","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"57e987ba-b719-45f1-8232-3f79614cfbfd","_cell_guid":"e39e0c0d-f9d1-4e39-a516-c25a9d78a204","trusted":true,"id":"JHVmUGtZiQP7","outputId":"049ef995-db43-421d-ee58-eb17c62df541"},"cell_type":"code","source":"test","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"edc7d7df-6930-4858-8026-bc6964f9b725","_cell_guid":"a582d5dd-bb73-4d49-b3ab-6a2cfee198f2","trusted":true,"id":"f-BPWh7AX_Ur","outputId":"b94f62f8-100f-4543-c5d4-3afa8e13855c"},"cell_type":"code","source":"#Seperating all the features and target for training data\ntrain_data = train.loc[:,'pixel0':]\ntrain_label = train.loc[:,'label']\nprint(f\"train_data shape :{train_data.shape}\")\nprint(f\"train_label shape :{train_label.shape}\")","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"aacbc435-022a-4aab-bcaf-c83e94c1cbda","_cell_guid":"4890f1f4-9c98-4330-aba2-2e8c66812584","trusted":true,"id":"-oVoILCiYE3R"},"cell_type":"code","source":"#Normalize\nX = train_data.values / 255.0\nX_test = test.loc[:,'pixel0':].values / 255.0\ny = train_label","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"f4ec6074-ed67-4373-899e-43b7e6e71e77","_cell_guid":"cadd5848-195c-4045-8e53-39b237c7a558","trusted":true,"id":"bMTRkg8ncLUw"},"cell_type":"code","source":"X_train,X_val,y_train,y_val = train_test_split(X,y,test_size = 0.1)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"35cfad17-50ef-41cb-8f61-4416709915c0","_cell_guid":"a3da89b3-f657-4954-95ec-80038d24a840","trusted":true,"id":"Qu_gZvuwfvxT","outputId":"efa57d65-7f8b-4131-dee9-18d011adcb39"},"cell_type":"code","source":"X_train.shape","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"2b42661f-61ec-4095-baeb-ffc57ed14d38","_cell_guid":"6286d751-f876-49d8-a5dd-1aa70471f01c","trusted":true,"id":"AJQ-gPjseCo7"},"cell_type":"code","source":"#input reshape\ninput_shape = (-1,28,28,1)\nX_train = X_train.reshape(input_shape)\nX_val = X_val.reshape(input_shape)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"450b1440-acb1-46f0-98f1-c4766e3314a3","_cell_guid":"42b179d0-0c00-4574-a3d8-5d09fdf63d2a","trusted":true,"id":"d7UC_29neVrK"},"cell_type":"code","source":"X_test = X_test.reshape(-1,28,28,1)\n# result = model.predict_classes(X_test)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"bcca3b42-10a9-4f09-8f66-4a5febf485d0","_cell_guid":"90da394a-f4db-4f67-b6f0-e26df3e1bfe4","trusted":true,"id":"JR-OuK1YeDkF"},"cell_type":"code","source":"#Now let us encode our labels\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"880c6046-aaac-4289-b8f8-054fa1a1f1c6","_cell_guid":"c90f3454-d4b5-402d-9623-3f52e27c3450","trusted":true,"id":"YJG9Oss2eGNL","outputId":"94b17059-d018-4b5e-c310-42dc19551294"},"cell_type":"code","source":"#Now we have categoricaly encoded our labels\nprint(y_train.shape)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"e5babb0a-0bca-432b-9f49-b87a4fd8a166","_cell_guid":"6c5111e8-5d75-4634-aa48-fe0ee7c4d4dc","trusted":true,"id":"CbaXP5yG6oPx"},"cell_type":"markdown","source":"**Building the Model**"},{"metadata":{"_uuid":"758c64cf-6ae7-4b2f-b7cd-fe90059f1593","_cell_guid":"b0b5efa4-7e2e-45c4-a5f4-38588dd0da3c","trusted":true,"id":"AgPozul8kSHy"},"cell_type":"markdown","source":"We decided to follow an experimental approach to tackle this problem. we tried different filter sizes introduced in different layers of the network, several combinations of activations, optimizers and paddings."},{"metadata":{"_uuid":"4bd1f13b-5530-4014-b119-ff6c37424b0f","_cell_guid":"8bbc4175-b0f5-4b0d-b3b6-3f06f1ede8ec","trusted":true,"id":"k1GeuI2yRbVF"},"cell_type":"code","source":"def DL_Model(filter1_size=64, filter2_size=32 , activation='relu', optimizer='Adam', padding='Same'):\n  model = tf.keras.models.Sequential()\n  model.add(tf.keras.layers.Conv2D(filter1_size,(3,3),padding = padding,activation = activation,\n                                  input_shape = (28,28,1)))\n  model.add(tf.keras.layers.Conv2D(filter1_size,(3,3),padding = padding,activation=activation))\n  model.add(tf.keras.layers.Dropout(0.2))\n  model.add(tf.keras.layers.MaxPool2D(pool_size=(2,2)))\n  model.add(tf.keras.layers.Conv2D(filter2_size,(3,3),padding = padding,activation=activation))\n  model.add(tf.keras.layers.Conv2D(filter2_size,(3,3),padding = padding,activation=activation))\n  model.add(tf.keras.layers.Dropout(0.25))\n  model.add(tf.keras.layers.MaxPool2D(pool_size=(2,2)))\n  model.add(tf.keras.layers.Flatten())\n  model.add(tf.keras.layers.Dense(128,activation=activation))\n  model.add(tf.keras.layers.Dropout(0.25))\n  model.add(tf.keras.layers.Dense(256,activation=activation))\n  model.add(tf.keras.layers.Dropout(0.25))\n  model.add(tf.keras.layers.Dense(10,activation='softmax'))\n  model.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])\n  return model","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"a51a3eb8-885c-42ea-9574-c32cbf973090","_cell_guid":"ca94701e-92f7-4660-b72c-8c38107039ee","trusted":true,"id":"xXuZdUnXeNhH"},"cell_type":"code","source":"# activation = ['relu','softmax', 'tanh', 'sigmoid', 'linear']\n# padding = ['Valid','Same']\n# optimizer = ['Adam', 'SGD',  'Adamax','RMSprop']\n\nfilter1_size = [64]\nfilter2_size = [64]\nactivation = ['relu','tanh','sigmoid']\npadding = ['Same']\noptimizer = ['Adam','RMSprop']","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"2d896a98-5639-4566-802a-25550f3ba935","_cell_guid":"bd25e8c5-5a32-4361-8e0f-ec461ef580a2","trusted":true,"id":"t1gtbpqrnehK"},"cell_type":"markdown","source":"GridSearch was used to find the model combination that gave the best accuracy in the training data. Each model was trained for 10 epochs using a batch size of 128. \nThe outcome of the evaluation will present the best combination of paramters that gives the best score for that model. This is done by calculating the Mean Absolute Error (MAE) for each model after the epochs."},{"metadata":{"_uuid":"7a408375-bf86-460c-8c66-b0393973700b","_cell_guid":"14ddc625-a28d-4f3b-b4b8-36017255ce6d","trusted":true,"id":"DmH8w-e7wH_6"},"cell_type":"code","source":"# param_grid = dict(filter1_size = filter1_size , filter2_size = filter2_size , activation = activation, padding = padding, optimizer = optimizer)\n\n# clf = tf.keras.wrappers.scikit_learn.KerasClassifier(build_fn= DL_Model, epochs= 10, batch_size=128, verbose= 3)\n# model = GridSearchCV(estimator= clf, param_grid=param_grid, n_jobs=-1, verbose=3)\n\n# epochs = 10\n# batch_size = 128\n# model.fit(X_train,y_train,\n#           validation_data=(X_val,y_val),\n#           epochs=epochs,\n#           batch_size=batch_size)\n\n# print(\"Max Accuracy Registred: {} using {}\".format(round(model.best_score_,3), \n#                                                    model.best_params_))","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"b7017722-d59e-40b6-8382-4524c89fa7d1","_cell_guid":"052a0b51-3402-4987-8eaf-d73e1b30a679","trusted":true,"id":"iURaDmlNoMud"},"cell_type":"markdown","source":"Using **relu** activation, filter size of **64** across all convolutional layers, an **Adam** optimizer and the **Same** padding gave the highest accuracy in the specified number of epochs which shows that this combination has potential to produce accurate predictions and should be advanced for further training."},{"metadata":{"_uuid":"e60f9275-87a1-481f-9b9d-846e5b7840aa","_cell_guid":"8a65773a-d59d-4030-be0c-dc3aff9f9f16","trusted":true,"id":"kNUD9LbiStWf"},"cell_type":"code","source":"model=DL_Model(64,64,'relu','Adam','Same')","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"fdac66d1-0bab-4e2c-815a-1bb90dfa52a4","_cell_guid":"4c36316e-a2ff-4411-921b-0c06d839f3d1","trusted":true,"id":"dGuxdcnEUdWt"},"cell_type":"code","source":"epochs = 40\nbatch_size = 128\nmodel.fit(X_train,y_train,\n          validation_data=(X_val,y_val),\n          epochs=epochs,\n          batch_size=batch_size)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"2fe019d4-547b-473e-a284-7eca9f2c2313","_cell_guid":"1de04018-a4bc-479f-a336-cfe7deab6e92","trusted":true,"id":"NkoXPe5ipn8a"},"cell_type":"markdown","source":"Running the training for more iterations did not improve the training accuracy and only caused it to oscillate between 0.998 and 0.9992. similarly the validation accuracy did not exceed 0.9978 at any point."},{"metadata":{"_uuid":"13080c69-fa53-4b19-af52-cd47c5c26764","_cell_guid":"2ee48145-afb1-4b72-8bb0-6f17536b4dec","trusted":true,"id":"HmXrIMylRsAX"},"cell_type":"code","source":"#lets just evaluate the model\nmodel.evaluate(X_val,y_val)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"fedc0bfd-2841-4e29-ad20-a038ac9163d7","_cell_guid":"2fba1f60-69f3-453e-9670-308c976b5b6c","trusted":true,"id":"26M68QwqI_Z5"},"cell_type":"markdown","source":"# Data Augmentation and Reducing Learning rate\n\n\n**Data Augmentation**\n* On classification tasks on image datasets data augmentation is a common way to increase the generalization of the model.\n* With the ImageDataGenerator on Keras, we can handle this objective easily.\n\n\n **Learning Rate reduction**\n\n* Included a feature that reduces the learning rate as the training advances whenever validation accuracy decreases beyond a certain threshold by a factor. This callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced."},{"metadata":{"_uuid":"4adecb29-1dca-44df-9130-937533c6e9ba","_cell_guid":"16613608-234b-4990-a892-b487a486a804","trusted":true,"id":"-y-dYLkr4vLj"},"cell_type":"code","source":"model=DL_Model(64,64,'relu','Adam','Same')","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"6b33e76c-e34b-468f-bd40-c9bc313bedcf","_cell_guid":"4990123a-5d38-4619-94b3-5d8778bbb4e2","trusted":true,"id":"iEk5G5u8W8wT"},"cell_type":"code","source":"early_stopping_monitor = tf.keras.callbacks.EarlyStopping(\n    monitor='val_loss',\n    min_delta=0,\n    patience=6,\n    verbose=0,\n    mode='auto',\n    baseline=None,\n    restore_best_weights=True\n)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"34416a0b-c8dd-4351-9a99-e008fa472d3c","_cell_guid":"61eae5d7-b304-49b8-92f6-e6e7b05aa552","trusted":true,"id":"8pw22T0DRmdv"},"cell_type":"code","source":"#This function reduces the learning rate as the training advances whenever validation accuracy decrease.\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_accuracy', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"e6f42cf9-7c8f-40e8-a41b-19a6f4a406fe","_cell_guid":"47546cea-df47-46da-9bbf-e1838018e40a","trusted":true,"id":"5_P-_YJARoo8"},"cell_type":"code","source":"epochs = 30 \nbatch_size = 86","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"f63dbc0b-ac50-4627-a5b9-5ef20c16f47a","_cell_guid":"592916a7-cf19-4b8d-988c-6048e109fe20","trusted":true,"id":"NffRZa-2THZb"},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X_train)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"e1b71d0e-3cb7-4b94-a9b4-1f483f6a432e","_cell_guid":"7d5111e6-a6ea-49cb-a721-26dc456202f6","trusted":true,"id":"XV8VKujYwM-i","outputId":"6dc8cd23-f4c0-4afe-c1cd-55ea3dc32880"},"cell_type":"code","source":"X_train.shape","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"5a05dbcc-6a42-484f-ba62-55c189549c5b","_cell_guid":"0a548224-8a80-43cd-9732-b83715ed098f","trusted":true,"id":"vWJrp19aTRfx","outputId":"580628ce-980e-4c12-8d02-da39d83131aa"},"cell_type":"code","source":"history = model.fit(datagen.flow(X_train,y_train, batch_size=batch_size),\n                              epochs = epochs, validation_data = (X_val,y_val),\n                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n                              , callbacks=[learning_rate_reduction])","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"4187c8ee-a731-4364-a349-081356b0a997","_cell_guid":"0b867919-95ae-48ca-8403-16315deb05e0","trusted":true,"id":"YH8SLD9p_HKc","outputId":"1da5fcbc-3f60-42ce-fc79-63af6b92130a"},"cell_type":"code","source":"model.evaluate(X_val,y_val)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"081d9009-f6e4-4af2-bb0e-927a12ebd660","_cell_guid":"0c699fbd-5f03-428e-a7d9-b1f5e0e8e5aa","trusted":true,"id":"0D5_ptwM-Clq"},"cell_type":"code","source":"%matplotlib inline\ndef PlotLoss(his, epoch):\n    plt.style.use(\"ggplot\")\n    plt.figure()\n    plt.plot(np.arange(0, epoch), his.history[\"loss\"], label=\"train_loss\")\n    plt.plot(np.arange(0, epoch), his.history[\"val_loss\"], label=\"val_loss\")\n    plt.title(\"Training Loss\")\n    plt.xlabel(\"Epoch #\")\n    plt.ylabel(\"Loss\")\n    plt.legend(loc=\"upper right\")\n    plt.show()\n\ndef PlotAcc(his, epoch):\n    plt.style.use(\"ggplot\")\n    plt.figure()\n    plt.plot(np.arange(0, epoch), his.history[\"accuracy\"], label=\"train_acc\")\n    plt.plot(np.arange(0, epoch), his.history[\"val_accuracy\"], label=\"val_accuracy\")\n    plt.title(\"Training Accuracy\")\n    plt.xlabel(\"Epoch #\")\n    plt.ylabel(\"Accuracy\")\n    plt.legend(loc=\"upper right\")\n    plt.show()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"e321f713-f858-46f1-ad2d-ce922adde9b7","_cell_guid":"647259f0-94f2-4bb6-8e44-0ac7e4aa3dd9","trusted":true,"id":"ORX8uXm0OQP0"},"cell_type":"markdown","source":"### Plotting Loss and Accuracy\n\nA smooth improvement of the loss and accuracy functions over the number of epochs can be observed. The model to converges gradually and \nplateau near the end"},{"metadata":{"_uuid":"271be339-ba65-4690-8487-d32f8c6ea13c","_cell_guid":"40b7ac82-df45-4280-b5b8-a812f1d652a0","trusted":true,"id":"nQF2NWQl-Fzs","outputId":"65c09972-d26c-4991-bcd2-71af60225adf"},"cell_type":"code","source":"PlotLoss(history, 30)\nPlotAcc(history, 30)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"248f1da2-33d0-48d9-a1c1-58befa561bbf","_cell_guid":"feaae860-9a18-421d-b888-4117ac5ae468","trusted":true,"id":"1_NPzOSA-mXt","outputId":"1e72e2ce-3ed7-4210-a7ec-8c2730787543"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\ny_val_predicted = model.predict_classes(X_val)\ny_val_actual=np.argmax(y_val, axis=-1)\ncm = confusion_matrix(y_val_actual, y_val_predicted)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"c8c61da7-45d6-4021-adac-24556cec1512","_cell_guid":"2da8ef54-ee3d-4f93-930d-62a4f4f8ad72","trusted":true,"id":"O8LRgTZyBPYq","outputId":"38c69d1d-3673-4d29-f57b-d05a8d85fcbe"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Actual\")\nplt.show()","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"d0ef4228-74d0-48b4-bca5-789d01c3fa87","_cell_guid":"3d346a04-8647-4928-8803-ea7a825d282f","trusted":true,"id":"Nzyt8OTmxRfe"},"cell_type":"code","source":"result = model.predict_classes(X_test)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"1da6027b-8e42-4ed2-9837-719141000800","_cell_guid":"786b2255-b8f7-458c-be18-32df710eb36f","trusted":true,"id":"7bvWBKQHyKkm"},"cell_type":"code","source":"result","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"476698fb-419e-419e-9b22-6b712ca3c99b","_cell_guid":"1823b2d6-d16b-43d9-a32d-c8521d992595","trusted":true,"id":"rxwdv60tyOSX"},"cell_type":"code","source":"sub_df = test[['id']]\nsub_df['label'] = result","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"3d33489e-2cb5-4b6a-9185-26421225a87a","_cell_guid":"d97c8bbc-e9fd-4ac8-bca9-7a9d9350ebc1","trusted":true,"id":"Wr8yDgQfyTJt"},"cell_type":"code","source":"sub_df.to_csv('submission.csv',index=False)","execution_count":0,"outputs":[]},{"metadata":{"_uuid":"b5b1bfaf-258d-4c7c-b27d-d31dc85f9730","_cell_guid":"30a725e8-a44d-4754-afbc-0ec6b5d4daea","trusted":true,"id":"KeJvS12hsrmr"},"cell_type":"markdown","source":"# **Results of the final model**\n* The accuracy of the model on the validation dataset was 99.80%\n* It's performance on the unseen kaggle dataset gave an accuracy of 98.16%"},{"metadata":{"_uuid":"7f6e7226-8a8e-42fc-9f83-b20e6dafe560","_cell_guid":"fef4dc74-7017-4709-9759-f4f143d1f1f3","trusted":true,"id":"pfD3BD0cY-VX"},"cell_type":"markdown","source":"The generalization of this model proved to be adequate and it did not overfit the traning dataset."},{"metadata":{"_uuid":"78eb8032-8c56-4c87-b11d-8124bb54ccb2","_cell_guid":"a084dc9c-0b2b-40f2-9184-7026a8b92a05","trusted":true,"id":"HvAU_PKK6wtS"},"cell_type":"markdown","source":"# Conclusion"},{"metadata":{"_uuid":"d54a7714-05fb-471c-a603-b0d7331c0693","_cell_guid":"dee6cbac-0174-47fc-a709-5ce351620f7f","trusted":true,"id":"YjY3-VHFC87X"},"cell_type":"markdown","source":"Convolutional Neural Networks proved to be a suitable choice for this type of image classification task. Based on the experimentation using GridSearch and adjusting the hyperparameter like activation, filter size and optimizer combination I was able to get a high accuracy on the training data. Using data augmentation and learning rate reduction led to an increase in the validation accuracy from 99.78% to 99.80%."},{"metadata":{"_uuid":"661c288f-32a8-4439-8b7c-b17f316c9baf","_cell_guid":"4286e712-6b52-4ad4-aafe-045346b34d5c","trusted":true,"id":"n4Iee1t2DAat"},"cell_type":"markdown","source":"However, I believe that the model's performance could be enhanced if it's possible to restore weights obtained during the epoch that gave the highest validation accuracy instead of calling the model after running all the epochs due to avoidance of overfitting the training data."}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":4}