{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Digit Recognizer for Pros\n\n[Digit Recognizer](https://www.kaggle.com/c/digit-recognizer) is a `Kaggle` competition where using the dataset you have to create a `classifier` that can classify handwritten images into digits.\n\nHere `no pre-trained CNN or predefined architecture` is used, this is a `custom` CNN architecture.\n\n**While doing this we'll go through**\n- Data augmentation using `ImageDataGenerator`\n- Building `custom` CNN architecture\n- Visualizing CNN (`filters` and `feature maps`)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import itertools\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport tensorflow as tf\nfrom tensorflow import keras\n\nfrom tensorflow.keras import Sequential, Model\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.layers import Flatten, Dense, Conv2D, Lambda, MaxPooling2D, Dropout, BatchNormalization\n\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading the dataset\ntrain_df = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_df  = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://media.giphy.com/media/3o7TKUM3IgJBX2as9O/giphy.gif)"},{"metadata":{},"cell_type":"markdown","source":"## 🏋️‍♀️ Data preprocessing"},{"metadata":{},"cell_type":"markdown","source":"Looking at `labels` count"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nplt.title('Number of digit classes')\ng = sns.countplot(train_df.label,palette='icefire')\ng.set(xlabel='Numbers', ylabel='Count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Shuffling the training dataframe\ntrain_df = train_df.sample(frac=1).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = list(train_df.columns)\ncols.remove('label')\n\nx = train_df[cols]\ny = train_df['label']\n\n# Splitting the dataset into training and validation(dev) sets\nx_train, x_dev, y_train, y_dev = train_test_split(x, y, test_size=0.1, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing dataset (this is the set on which we'll do predictions and then submit it)\nx_test = test_df[cols]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Training set size: {len(x_train)}')\nprint(f'Validation set size: {len(x_dev)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(x_train.shape)\nprint(x_train.values.reshape(-1, 28, 28).shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshaping the datasets to feed images of 28X28 pixels to our neural network\n# And also scaling the images\nx_train = x_train.values.reshape(-1, 28, 28) / 255\nx_dev   = x_dev.values.reshape(-1, 28, 28) / 255\nx_test  = x_test.values.reshape(-1, 28, 28) / 255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(x_train.shape)\nprint(np.expand_dims(x_train, axis=-1).shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adding an additional dimension of channel (1 as images are grayscale)\nx_train = np.expand_dims(x_train, axis=-1)\nx_dev   = np.expand_dims(x_dev, axis=-1)\nx_test  = np.expand_dims(x_test, axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(x_train[0].reshape((28, 28)), cmap=plt.cm.binary)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Looking at first 25 training examples\n\nplt.figure(figsize=(10, 10))\nfor i in range(25):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    \n    # reshaping the images as (28, 28, 1) is an invalid shape to plot imgs\n    plt.imshow(x_train[i].reshape((28, 28)), cmap=plt.cm.binary)\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### 👨‍👨‍👦‍👦 Data Augmentation\n\n![](https://media.giphy.com/media/fzZzoftMBR8is/giphy.gif)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augmentation(x_data, y_data, batch_size):\n    datagen = ImageDataGenerator(\n        featurewise_center=False,            # set input mean to 0 over the dataset\n        samplewise_center=False,             # set each sample mean to 0\n        featurewise_std_normalization=False, # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,                 # apply ZCA whitening\n        rotation_range=10,                   # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1,                    # Randomly zoom image \n        width_shift_range=0.1,               # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,              # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,               # randomly flip images\n        vertical_flip=False,                 # randomly flip images\n    )\n    \n    \n    datagen.fit(x_data)\n    train_data = datagen.flow(x_data, y_data, batch_size=batch_size, shuffle=True)\n    \n    return train_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 64\naug_train_data = data_augmentation(x_train, y_train, BATCH_SIZE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 🧠 Modelling"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model():\n    # Neural Network Architecture\n    layers = [\n        Conv2D(filters=96, kernel_size=(11, 11), strides=2, activation='relu', input_shape=(28, 28, 1)),\n        MaxPooling2D(pool_size=(3, 3), strides=2),\n        Conv2D(filters=256, kernel_size=(5, 5), padding='same', activation='relu'),\n        Flatten(),\n        Dense(9216, activation='relu'),\n        Dense(4096, activation='relu'),\n        Dense(4096, activation='relu'),\n        Dense(10, activation='softmax')\n    ]\n\n    model = Sequential(layers)\n\n    optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n    model.compile(\n        optimizer=optimizer,\n        loss='sparse_categorical_crossentropy',\n        metrics=['accuracy']\n    )\n\n    return model\n\n\nmodel = build_model()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\n    ReduceLROnPlateau(monitor=\"loss\",factor=0.1, patience=2, min_lr=0.000001, verbose=1),\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    aug_train_data, \n    steps_per_epoch=x_train.shape[0] // BATCH_SIZE,\n    batch_size=BATCH_SIZE,\n    validation_data=(x_dev, y_dev), \n    epochs=50,\n    callbacks=callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 🪂 Plotting model's performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'][1:], label='train acc')\nplt.plot(history.history['val_accuracy'][1:], label='validation acc')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend(loc='lower right')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['loss'][1:], label='train loss')\nplt.plot(history.history['val_loss'][1:], label='validation loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend(loc='lower right')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualizing CNN\n\nTo know more read the following posts: [post_1](https://www.analyticsvidhya.com/blog/2018/03/essentials-of-deep-learning-visualizing-convolutional-neural-networks/), [post_2](https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/) and [post_3](https://www.kaggle.com/arpitjain007/guide-to-visualize-filters-and-feature-maps-in-cnn).\n\nTo `visualize how CNN and Max pooling works` go through the following [kernel](https://www.kaggle.com/akashsdas/how-does-convolutions-work).\n\n"},{"metadata":{},"cell_type":"markdown","source":"**Visualize filters**"},{"metadata":{},"cell_type":"markdown","source":"Plotting the `96th` filter of the `1st conv layer`."},{"metadata":{"trusted":true},"cell_type":"code","source":"top_layer = model.layers[0]\nplt.imshow(top_layer.get_weights()[0][:, :, :, 95].squeeze(), cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Shape of 1st conv layer weights: {model.layers[0].get_weights()[0].shape}')\nprint(f'Shape of 2nd conv layer weights: {model.layers[2].get_weights()[0].shape}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_filters_for_conv_layer(model, layer_index, num_columns=5, cmap='binary', how_many='all'):\n    layer = model.layers[layer_index]\n    filter_weights = layer.get_weights()[0]\n    \n    num_filters = layer.filters if how_many == 'all' else how_many\n    num_rows = (num_filters // num_columns) + (num_filters % num_columns)\n    # example:\n    # num_rows = (96 // 5) + (96 % 5) == 20 (to plot all the filters)\n    \n    f, axs = plt.subplots(num_rows, num_columns, figsize=(20, 5 * num_rows))\n    row_count = 0  # to plot num_columns figs in an individual row\n    \n    if not isinstance(axs, np.ndarray):\n        # When num_cloumns == how_many\n        axs = np.array(axs)  # to make axs iterable\n        # list can also be inplace np.array but since plt.subplots axs output is of type np.ndarray I kept \n        \n    for idx, row_ax in enumerate(axs):\n        # plotting filters in a row\n        for i, ax in enumerate(row_ax):\n            if row_count + i >= num_filters:\n                break\n                \n            if len(filter_weights.shape) == 4:\n                if filter_weights.shape[2] == 1:\n                    # For plotting filters whose weight shape is == (kernel_size_x, kernel_size_y, 1, #filters)\n                    # example: (11, 11, 1, 96)\n                    ax.imshow(filter_weights[:, :, :, row_count + i].squeeze(), cmap=cmap)\n                else:\n                    # For plotting filters whose weight shape is == (kernel_size_x, kernel_size_y, num > 1, #filters)\n                    # example: (5, 5, 96, 256)\n                    # because if ax.imshow(filter_weights[:, :, :, row_count + i].squeeze(), cmap=cmap)\n                    # is used then we'll have array of (5, 5, 96) which is invalid image data for plotting 2D image\n                    # (in above case where `filter_weights.shape[2] == 1` there we'll end up with (11, 11, 1) which\n                    # after applying the `squeeze` function will be (11, 11) which is valid image data) so in \n                    # that case we'll just plot (5, 5) plot in the first 3D array i.e. (5, 5, 0, row_count + i) \n                    # => this is what we'll plot. To plot (5, 5, row_count + i, 0) just change indexing from\n                    # [:, :, 0, row_count + i] to [:, :, row_count + i, 0]\n                    ax.imshow(filter_weights[:, :, 0, row_count + i].squeeze(), cmap=cmap)\n                    \n                # For generalization this can be used, but to understand why 0 need to be used,\n                # using the above way\n                # ax.imshow(filter_weights[:, :, 0, row_count + i].squeeze(), cmap=cmap)\n            else:\n                break\n                            \n        # increasing row_count by num_columns\n        row_count += num_columns ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using `cmap` as `sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)` instead of `binary`, just to make the plot look `beautiful as you`. Also there are `96` filters in the `1st conv` layer so only plotting the first `20` filters."},{"metadata":{},"cell_type":"markdown","source":"Visualizing `only first 20` filters in the `1nd conv layer`. Here `sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)` is used as `cmap`."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_filters_for_conv_layer(\n    model, \n    0, \n    num_columns=4,\n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True), \n    how_many=20\n)  \n\n# 11X11 filters","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualizing `only first 10` filters in the `2nd conv layer`. Here `binary` is used as `cmap`."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_filters_for_conv_layer(\n    model, \n    2, \n    num_columns=5,\n    how_many=10\n)  \n\n# 5X5 filters","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizing feature maps**"},{"metadata":{"trusted":true},"cell_type":"code","source":"[idx for idx in range(len(model.layers)) if 'conv' in model.layers[idx].name]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `feature maps` of a `CNN` capture the result of `applying the filters` to an input image i.e at each layer, the feature map is the output of that layer. The reason for visualising a feature map for a specific input image is to try to gain some understanding of what features our CNN detects."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_feature_maps_for_single_conv_layer(model, layer_id, input_img, num_columns=10, cmap='binary'):\n    ref_model = Model(inputs=model.inputs, outputs=model.layers[layer_id].output)\n    feature_map = ref_model.predict(input_img)\n    \n    num_filters = feature_map[0].shape[2]\n    num_rows = (num_filters // num_columns) + (num_filters % num_columns)\n\n    fig = plt.figure(figsize=(16, 2 * num_rows))\n    ix = 1\n    for _ in range(num_rows):\n        for _ in range(num_columns):\n            if ix == num_filters:\n                break\n        \n            # specify subplot and turn of axis\n            ax = plt.subplot(num_rows, num_columns, ix)\n            ax.set_xticks([])\n            ax.set_yticks([])\n        \n            # plot filter channel in grayscale\n            plt.imshow(feature_map[0, :, :, ix-1], cmap=cmap)\n            ix += 1\n            \n    # show the figure\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize `feature maps` for `7th` image in `x_train`."},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_feature_maps_for = 7","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(x_train[visualize_feature_maps_for].reshape((28, 28)), cmap=plt.cm.binary)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    0, \n    x_train[visualize_feature_maps_for][np.newaxis, ...], \n    num_columns=8,\n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_feature_maps_for_single_conv_layer(\n    model, \n    2, \n    x_train[visualize_feature_maps_for][np.newaxis, ...],\n    num_columns=8,\n    cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## 🔮 Evaluation"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Here model is evaluated on the `validation` dataset."},{"metadata":{},"cell_type":"markdown","source":"`plot_confusion_matrix` function prints and plots the confusion matrix. Normalization can be applied by setting `normalize=True`."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix'):\n    plt.figure(figsize=(8, 8))\n    \n    plt.imshow(cm, interpolation='nearest', cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True))\n    plt.title(title)\n    plt.colorbar(fraction=0.046, pad=0.04)\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predict labels for validation dataset\ny_pred = model.predict(x_dev)\n\n# Convert predictions classes to one hot vectors \ny_pred_classes = np.argmax(y_pred, axis=1) \n\n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(y_dev, y_pred_classes) \n\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes=range(10)) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Verify the results\n\npredictions = y_pred\n\nprint(predictions[0]) # Confidence matrix\n\nprint('Predicted digit is: ' + str(np.argmax(predictions[0])))\nprint('Accuracy is: ' + str(np.max(predictions[0] * 100)) + '%')\n\n# Actual Digit\nplt.imshow(x_dev[0].reshape((28, 28)), cmap=plt.cm.binary)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seeing first 25 validation images predictions\nplt.figure(figsize=(12, 14))\nfor i in range(25):\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.imshow(x_dev[i].reshape((28, 28)), cmap=plt.cm.binary)\n    \n    plt.xlabel(\n        'Predicted digit is: ' + str(np.argmax(predictions[i])) + \n        '\\n' + \n        'Accuracy is: ' + str(np.max(predictions[i] * 100)) + '%'\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## 🔮 Predictions on test set and 📧 submitting the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(x_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in submission.index:\n    submission['Label'][i] = np.argmax(predictions[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"sample_submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **🎁 Saving the model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('model')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\nI'll wrap things up there. If you want to find some other answers then go ahead `edit` this kernel. If you have any `questions` then do let me know.\n\nIf this kernel helped you then don't forget to 🔼 `upvote` and share your 🎙 `feedback` on improvements of the kernel.\n\n![](https://media.giphy.com/media/xjZtu4qi1biIo/giphy.gif)\n\n---"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}