{"cells":[{"metadata":{},"cell_type":"markdown","source":"## MNIST Digit Recognizer. Tensorflow / Keras CNN\nImporting modules"},{"metadata":{"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function, unicode_literals\n\n%matplotlib inline \n\n# Importing modules\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nfrom matplotlib import image as mp_image\n\nimport imageio\nimport imgaug as ia\nfrom imgaug import augmenters\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nfrom skimage import morphology\nfrom skimage import exposure\n\nfrom mlxtend.plotting import plot_confusion_matrix\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Model, Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Importing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing data\ntrain_data = pd.read_csv(r\"../input/digit-recognizer/train.csv\")\ntest_data = pd.read_csv(r\"../input/digit-recognizer/test.csv\")\n\n# Selecting train data and labels\nX_train_raw = train_data.loc[:, \"pixel0\":].to_numpy() / 255 # Transformed pixel values\ny_train_raw = train_data.loc[:, \"label\"].to_numpy() # Non-encoded labels\n# Test data\nX_test_raw = test_data.loc[:, \"pixel0\":].to_numpy() / 255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Recreating images\n\nX_train = [] # List of train images\nfor pixels in X_train_raw:\n    image = pixels.reshape(28,28)\n    X_train.append(image)\nX_train = np.array(X_train)\n\nX_test = [] # List of test images\nfor pixels in X_test_raw:\n    image = pixels.reshape(28,28)\n    X_test.append(image)\nX_test = np.array(X_test)\n\n# Review Images\nwidth = 6\nheight = 6\nfig = plt.figure(figsize = (width, height))\ncolumns = 4\nrows = 5\nfor i in range(1, columns * rows + 1):\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(X_train[np.random.randint(low = 0, high = 784)], cmap='Greys_r')\nplt.show()\n\nprint(\"Shape of the training data:\", X_train.shape)\nprint(\"Shape of the test data:\", X_test.shape)\nprint(\"Shape of the labels:\", y_train_raw.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation"},{"metadata":{},"cell_type":"markdown","source":"To improve the classifier, it is viable to create additional, modified images, which we can add (not replace) for subsequent training. Modifications should not deform the image, such it may change the original label, like rotating 6 -> 9."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adding image rotation, shift, zoom\naugs_one = []\nfor i in X_train:\n    rotate = augmenters.Affine(translate_percent={\"x\": (-0.12, 0.12), \"y\": (-0.12, 0.12)}, rotate=(-12, 12),\n                               scale={\"x\": (0.8, 1.1), \"y\": (0.8, 1.1)})\n    aug_image = rotate.augment_image(i)\n    augs_one.append(aug_image)\n\n# Review Augmented Images\nwidth = 6\nheight = 6\nfig = plt.figure(figsize = (width, height))\ncolumns = 4\nrows = 5\nfor i in range(1, columns * rows + 1):\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(augs_one[np.random.randint(low = 0, high = 784)], cmap='Greys_r')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Additional image augmentation technique - Erosion (optional)\nOptional, not added to overall training set. Would like additional feedback on it's implications in such datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adding erosion\naugs_two = []\nfor i in X_train:\n    aug_image = morphology.binary_erosion(i)\n    augs_two.append(aug_image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Summary of Augmented Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_augs = np.concatenate((X_train, augs_one), axis=0) # Combine original and augmented images\ny_augs = np.concatenate((y_train_raw, y_train_raw), axis=0)\nprint(\"Shape of the augmented and original data:\", X_augs.shape)\nprint(\"Shape of the augmented and original labels:\", y_augs.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Normalizing Images (optional)\nIdeally, the image should have relatively even distribution of values, indicating good contrast and making it easier to extract analytical information, as such normalization is preferred."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_eq = []\nfor i in X_augs:\n    # Histogram Equalization\n    image_eq = exposure.equalize_hist(i)\n    X_train_eq.append(image_eq)\n\n# Review Equalized Images\nwidth = 6\nheight = 6\nfig = plt.figure(figsize = (width, height))\ncolumns = 4\nrows = 5\nfor i in range(1, columns * rows + 1):\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(X_train_eq[np.random.randint(low = 0, high = 784)], cmap='Greys_r')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tensorflow / Keras model\nFour stage CNN model, with addition of drouputs to decrease overfitting, using functional API."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transform y-labels into categories for categorical crossentropy\ny_train = tf.keras.utils.to_categorical(y_augs, num_classes=10, dtype='float32')\n\n# Add channels (reshaping data for CNN)\nX_train = np.expand_dims(X_augs, -1)\nX_test = np.expand_dims(X_test, -1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.backend.clear_session()\n\n# Tensorflow CNN\n# 1st Stage\nX_input = keras.Input(shape=(28, 28, 1), name=\"input\")\ndata = layers.Conv2D(filters = 32, kernel_size = (5, 5), strides = (1, 1),\n                     padding = \"Same\", activation = \"relu\")(X_input)\ndata = layers.Conv2D(filters = 64, kernel_size = (5, 5), strides = (1, 1),\n                     padding = \"Same\", activation = \"relu\")(data)\ndata = layers.MaxPool2D(pool_size = (2, 2), strides = (2, 2),\n                        padding = \"Same\")(data)\ndata = layers.Dropout(rate = 0.25)(data)\n# 2nd Stage\ndata = layers.Conv2D(filters = 128, kernel_size = (3, 3), strides = (1, 1),\n                     padding = \"Same\", activation = \"relu\")(data)\ndata = layers.Conv2D(filters = 128, kernel_size = (3, 3), strides = (1, 1),\n                     padding = \"Same\", activation = \"relu\")(data)\ndata = layers.Dropout(rate = 0.25)(data)\ndata = layers.MaxPool2D(pool_size = (2, 2), strides = (2, 2),\n                        padding = \"Same\")(data)\n# 3rd Stage\ndata = layers.Conv2D(filters = 256, kernel_size = (5, 5), strides = (1, 1),\n                     padding = \"Same\", activation = \"relu\")(data)\ndata = layers.MaxPool2D(pool_size = (2, 2), strides = (2, 2),\n                        padding = \"Same\")(data)\ndata = layers.Dropout(rate = 0.25)(data)\n# 4th Stage\ndata = layers.Flatten()(data)\ndata = layers.Dense(units = 512, activation = \"relu\")(data)\ndata = layers.Dropout(rate = 0.5)(data)\ndata = layers.Dense(units = 10, activation = \"softmax\")(data)\n\n# \nmodel = Model(inputs = X_input, outputs = data)\n\n# Compile model\nmodel.compile(optimizer = tf.compat.v2.optimizers.Adam(learning_rate=0.0005), loss = \"categorical_crossentropy\",\n              metrics = [\"accuracy\"])\n# Summarize\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Callbacks\nThree callbacks were implemented : \n1. To stop learning if validation loss would not improve; \n2. Saving model with highest validation accuracy; \n3. Reducing learning rate gradually to get close to local minima;"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining callbacks\n# Getting model with lowest validation accuracy\naccurate_model = ModelCheckpoint(\"model_weights.h5\", monitor=\"val_accuracy\", verbose=1,\n                                 save_best_only=True, mode=\"max\")\n\n# Stopping learning with no improvment\nloss_stop = EarlyStopping(monitor=\"val_loss\", min_delta=0.00001,\n                          patience = 10, mode=\"min\", verbose=1)\n\n# Reducing learning rate with increase of validation accuracy\nreduce_lr = ReduceLROnPlateau(monitor=\"val_accuracy\", factor=0.4,\n                              patience=3, min_lr=0.0000001, mode = \"auto\", verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Training model\nValidation accuracy at 10th epoch should be around 99%, at 30th epoch 99.6%"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(x = X_train, y = y_train, epochs = 10, verbose = 1,\n                    validation_split = 0.2, \n                    callbacks = [reduce_lr, loss_stop, accurate_model])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model can be run for additional epochs, to improve validation accuracy"},{"metadata":{},"cell_type":"markdown","source":"#### DataFrame Results\nOverview of learning results"},{"metadata":{"trusted":true},"cell_type":"code","source":"# DataFrame results of learning\ninfo = {'Loss': history.history[\"loss\"], 'Validation Loss': history.history[\"val_loss\"],\\\n        \"Accuracy\" : history.history[\"acc\"], \"Validation Accuracy\" : history.history[\"val_acc\"]}\n\ndeep_statistics = pd.DataFrame(data=info)\n\n# Top 5 best results from learning based on validation loss\nbest_results = deep_statistics[(deep_statistics[\"Validation Accuracy\"] >= 0.90) & (deep_statistics[\"Accuracy\"] >= 0.90)\\\n                               & (deep_statistics[\"Validation Loss\"] <= 0.1) & (deep_statistics[\"Loss\"] <= 0.1)]\nbest_results.sort_values(\"Validation Loss\").head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Visualizing results\nComparing Training VS Validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame(history.history)\ndf.rename(columns={\"acc\":\"Training\", \"val_acc\":\"Validation\",\n                   \"loss\":\"Train Loss\", \"val_loss\":\"Validation Loss\"}, inplace = True)\n\nsns.set(font_scale=1.2)\nsns.set_style(\"whitegrid\")\n\npalette = sns.diverging_palette(10, 220, sep=80, n=2)\n\n# Training VS Validation plot\nfigure, axes = plt.subplots(nrows = 1, ncols = 2, figsize = (20,6))\naxes[0].set(xlabel=\"Epoch\", ylabel=\"Accuracy\")\nsns.lineplot(data=df[[\"Training\", \"Validation\"]], marker = \"o\", ax=axes[0], palette=palette, legend=\"full\")\naxes[1].set(xlabel=\"Epoch\", ylabel=\"Loss\")\nsns.lineplot(data=df[[\"Train Loss\", \"Validation Loss\"]], marker = \"d\", ax=axes[1], palette=palette, legend=\"full\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Data Prediction\nConfusion matrix will tell us with which predictions model had the biggest trouble and made most mistakes "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Re-evaluating train set\ntrain_predictions = model.predict(X_train)\nnumbers = [np.where(number == np.amax(number)) for number in train_predictions] # Get predictions labels\nnumbers_ravel = np.concatenate(numbers).ravel() # List of predicted numbers\n\n# Comparing labels\nresult = confusion_matrix(y_augs, numbers_ravel)\n\nclasses = [\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\"]\n\nfig, ax = plot_confusion_matrix(conf_mat = result,\n                                colorbar = True,\n                                show_absolute = True,\n                                show_normed = True,\n                                class_names = classes, figsize = (10, 10))\nplt.ylabel('True label')\nplt.xlabel('Predicted label')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on confusion matrix, the biggest issue is to differentate between label 4 and 9, as well as 3 and 5"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Gettting predictions of test set\npredictions = model.predict(X_test)\nnumbers = [np.where(number == np.amax(number)) for number in predictions] # Get predictions labels\nnumbers = np.concatenate(numbers).ravel() # List of predicted numbers\nImageId = np.arange(1, X_test.shape[0]+1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Conclusion\nModel stops around 99.65% validation accuracy, as it hits a hard plato. I am open to suggestions and critique on how to improve the overall design of the model (or what's incorrect) and what additional data augmentation techniques to use. Thanks!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"}},"nbformat":4,"nbformat_minor":1}