{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST - Neural Network Experimentation\n\nGreetings! In this notebook, we experiment with various neural network configurations applied to the task of classifying MNIST digits. \n\nWe start with a dense (fully connected) network and play with using different amounts of dropout to promote model generalization. Then we implement a LeNet-5-inspired convolutional neural network and experiment with additional conv blocks and a new activation function called Mish.\n\nThanks for reading and please leave a comment below if this helped you in your digit classification :)"},{"metadata":{},"cell_type":"markdown","source":"### Load the Good Stuff"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os, datetime\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.pylab as pylab\nfrom sklearn.model_selection import cross_validate, train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import confusion_matrix\nimport keras\nimport tensorflow\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Conv2D, MaxPooling2D, Dropout, Flatten\nfrom keras.optimizers import SGD\nfrom keras.layers.normalization import BatchNormalization\n\n# Load TensorBoard\nfrom keras.callbacks import TensorBoard\n%load_ext tensorboard\n\n# Set plotting defaults\nparams = {'legend.fontsize': 'large',\n          'figure.figsize': (10, 8),\n         'axes.labelsize': 'large',\n         'axes.titlesize':'large',\n         'xtick.labelsize':'large',\n         'ytick.labelsize':'large'}\npylab.rcParams.update(params)\nsns.set(style=\"white\")\n\n# Prevent Pandas from truncating displayed dataframes\npd.set_option('display.max_rows', None)\npd.set_option('display.max_columns', None)\n\n# Seed randomness for reproducibility\nfrom numpy.random import seed\nSEED = 42\nseed(SEED)\ntensorflow.random.set_seed(SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 10 digits\nnum_classes = 10\n\n# Load master copies of data - these remain pristine\ntrain_ = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_ = pd.read_csv(\"../input/digit-recognizer/test.csv\")\nsample_submission = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\n\n# Take copies of the master dataframes\ntrain = train_.copy()\ntest = test_.copy()\n\n# Separate the target variable from the digits\ny = train.pop(\"label\")\n\n# Scale the data to [0-1]\ntrain = train / np.max(np.max(train))\ntest = test / np.max(np.max(test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(train, y, test_size=0.2, random_state=SEED)\ny_train = keras.utils.to_categorical(y_train, num_classes=num_classes)\ny_valid = keras.utils.to_categorical(y_valid, num_classes=num_classes)\n\ny = keras.utils.to_categorical(y, num_classes=num_classes)\n\n# For CNNs - reshape vector into 2D matrix\nX_train_2D = X_train.values.reshape(-1,28,28,1) # Keras requires an extra dimension representing the image channel\nX_valid_2D = X_valid.values.reshape(-1,28,28,1)\ntrain_2D = train.values.reshape(-1,28,28,1)\ntest_2D = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Helper Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"class NeuralNetwork():\n    \"\"\"Wrap neural network functions into a handy class.\"\"\"\n    \n    def __init__(self, name, batch_size, epochs, optimizer, verbose):\n        self.name = name\n        self.batch_size = batch_size\n        self.epochs = epochs\n        self.verbose = verbose\n        self.model = Sequential()\n        self.optimizer = optimizer\n        \n    def add_(self, layer):\n        self.model.add(layer)\n\n    def compile_and_fit(self):\n        self.model.compile(loss=\"categorical_crossentropy\", optimizer=self.optimizer, metrics=[\"accuracy\"])\n        self.history = self.model.fit(x=X_train,\n                                      y=y_train,\n                                      batch_size=self.batch_size,\n                                      epochs=self.epochs,\n                                      verbose=self.verbose,\n                                      validation_data=(X_valid, y_valid))\n        self.val_loss = self.history.history[\"val_loss\"]\n        self.val_accuracy = self.history.history[\"val_accuracy\"]\n    \n    def plot_learning_curves(self):\n        fig, ax = plt.subplots(1, 2, figsize=(15, 5))\n        \n        sns.lineplot(x=range(0,len(self.val_accuracy)), y=self.val_accuracy, ax=ax[0], label=\"Validation Accuracy\")\n        sns.lineplot(x=range(0,len(self.val_loss)), y=self.val_loss, ax=ax[1], label=\"Validation Loss\")\n\n        ax[0].set_xlabel(\"# of Epochs\")\n        ax[1].set_xlabel(\"# of Epochs\")\n\n        plt.suptitle(\"Learning Curves: {}\".format(self.name))\n        plt.show()\n\n    def evaluate_(self):\n        return self.model.evaluate(x=X_valid, y=y_valid, batch_size=self.batch_size)\n    \n    def save(self, filename):\n        self.model.save(\"working/\"+filename+\".hd5\")\n        \n    def predict_(self, test):\n        return self.model.predict(test, batch_size=self.batch_size).argmax(axis=1)\n        \n    def summary_(self):\n        return self.model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_output(model, test_data, filename, save=False):\n    \"\"\"Generate output dataframe (and .csv file) of predictions for Kaggle submission.\"\"\"\n    \n    try:\n        predictions = model.predict_(test_data)\n    except:\n        predictions = model.predict(test_data).argmax(axis=1)\n    \n    output = sample_submission.copy()\n    output[\"Label\"] = predictions\n\n    if save:\n        output.to_csv(filename + \".csv\", index=False)\n    return output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intermediate Net\n\nFirst, let's create a fully connected network comprised of only dense layers. Using such a configuration achieves a score of `0.97` on the Kaggle leaderboard."},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_size = 128\nn_epochs = 20\nlearning_rate = 0.1\nverbose = 0 # set this = 1 to see training progress\noptimizer = \"adam\"\n\nmodel = NeuralNetwork(\"3 Relu Hidden Layers + Batch Norm + Dropout (Adam Optimizer)\", batch_size, n_epochs, optimizer, verbose)\nmodel.add_(Dense(64, activation=\"relu\", input_shape=(784,)))\nmodel.add_(BatchNormalization())\n\nmodel.add_(Dense(64, activation=\"relu\"))\nmodel.add_(BatchNormalization())\n\nmodel.add_(Dense(64, activation=\"relu\"))\nmodel.add_(BatchNormalization())\nmodel.add_(Dropout(0.2))\n\nmodel.add_(Dense(10, activation=\"softmax\"))\n\nmodel.summary_()\nmodel.compile_and_fit()\nmodel.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"results = model.evaluate_()\nresults","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"predictions = model.predict_(test)\npredictions[0:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can preview individual digits in the test set and observe whether any of the model predictions are correct."},{"metadata":{"trusted":false},"cell_type":"code","source":"preview_index = 4 # The index of the digit in the training set to preview\n\nfig, ax = plt.subplots(figsize=(5,5))\nimage = test.iloc[preview_index].values.reshape((28,28))\nplt.imshow(image, cmap=\"Greys\")\nplt.axis(\"off\")\n\nplt.suptitle(\"Previewing Digit #{}\".format(preview_index + 1), y=0.9)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Compare Varying Amounts of Dropout\n\nDropout is a powerful technique for preventing overfitting. Often trial and error is the best way to determine how many rounds of dropout to incorporate into a network. \n\nFor example, below are three networks: one with one round of dropout, one with two rounds of dropout and one with three rounds of dropout. Based on validation metrics, we can see that having two rounds of dropout yields optimal accuracy. Similarly, you could analyze configurations with more or less dropout per layer (i.e. adjust the 20% parameter)."},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_size = 128\nn_epochs = 50\nverbose = 0\noptimizer = \"adam\"\n\n# 1 round of dropout\nmodel_1d = NeuralNetwork(\"3 Relu Hidden Layers + Batch Norm + 1 x Dropout (Adam Optimizer)\", batch_size, n_epochs, optimizer, verbose)\nmodel_1d.add_(Dense(64, activation=\"relu\", input_shape=(784,)))\nmodel_1d.add_(BatchNormalization())\n\nmodel_1d.add_(Dense(64, activation=\"relu\"))\nmodel_1d.add_(BatchNormalization())\n\nmodel_1d.add_(Dense(64, activation=\"relu\"))\nmodel_1d.add_(BatchNormalization())\nmodel_1d.add_(Dropout(0.2))\n\nmodel_1d.add_(Dense(10, activation=\"softmax\"))\n\nmodel_1d.compile_and_fit()\nmodel_1d.plot_learning_curves()\n\n# 2 rounds of dropout\nmodel_2d = NeuralNetwork(\"3 Relu Hidden Layers + Batch Norm + 2 x Dropout (Adam Optimizer)\", batch_size, n_epochs, optimizer, verbose)\nmodel_2d.add_(Dense(64, activation=\"relu\", input_shape=(784,)))\nmodel_2d.add_(BatchNormalization())\n\nmodel_2d.add_(Dense(64, activation=\"relu\"))\nmodel_2d.add_(BatchNormalization())\nmodel_2d.add_(Dropout(0.2))\n\nmodel_2d.add_(Dense(64, activation=\"relu\"))\nmodel_2d.add_(BatchNormalization())\nmodel_2d.add_(Dropout(0.2))\n\nmodel_2d.add_(Dense(10, activation=\"softmax\"))\n\nmodel_2d.compile_and_fit()\nmodel_2d.plot_learning_curves()\n\n# 3 rounds of dropout\nmodel_3d = NeuralNetwork(\"3 Relu Hidden Layers + Batch Norm + 3 x Dropout (Adam Optimizer)\", batch_size, n_epochs, optimizer, verbose)\nmodel_3d.add_(Dense(64, activation=\"relu\", input_shape=(784,)))\nmodel_3d.add_(BatchNormalization())\nmodel_3d.add_(Dropout(0.2))\n\nmodel_3d.add_(Dense(64, activation=\"relu\"))\nmodel_3d.add_(BatchNormalization())\nmodel_3d.add_(Dropout(0.2))\n\nmodel_3d.add_(Dense(64, activation=\"relu\"))\nmodel_3d.add_(BatchNormalization())\nmodel_3d.add_(Dropout(0.2))\n\nmodel_3d.add_(Dense(10, activation=\"softmax\"))\n\nmodel_3d.compile_and_fit()\nmodel_3d.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"model_1d.evaluate_()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"model_2d.evaluate_()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"model_3d.evaluate_()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(model_1d.history.history[\"val_accuracy\"], label=\"1 x Dropout\")\nplt.plot(model_2d.history.history[\"val_accuracy\"], label=\"2 x Dropout\")\nplt.plot(model_3d.history.history[\"val_accuracy\"], label=\"3 x Dropout\")\nplt.legend()\nplt.title(\"Impact of Dropout on Neural Network Learning\")\nplt.ylabel(\"Validation Accuracy\")\nplt.xlabel(\"Epoch\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CNN v1\n\nConvolutional neural networks are uniquely adept at extracting spatial information. Below is a simple CNN with two convolutional layers (inspired by LeNet-5 architecture)."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 128\nn_epochs = 10\nverbose = 1\noptimizer = \"adam\"\n\n# cnn = NeuralNetwork(\"CNN\", batch_size, n_epochs, optimizer, verbose)\ncnn = Sequential()\n\n# 1st convolutional layer\ncnn.add(Conv2D(32, kernel_size=(3,3), activation=\"relu\", input_shape=(28,28,1)))\n\n# 2nd convolutional layer\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\"))\n\n# reduce computational complexity\ncnn.add(MaxPooling2D(pool_size=(2,2)))\n\n# dropout\ncnn.add(Dropout(0.25))\n\n# dimensionality reduction\ncnn.add(Flatten())\n\n# dense layer\ncnn.add(Dense(128, activation=\"relu\"))\n        \n# dropout\ncnn.add(Dropout(0.5))\n\n# output\ncnn.add(Dense(10, activation=\"softmax\"))\n\ncnn.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cnn.compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\ncnn_history = cnn.fit(x=X_train_2D,\n                      y=y_train,\n                      batch_size=batch_size,\n                      epochs=n_epochs,\n                      verbose=verbose,\n                      validation_data=(X_valid_2D, y_valid))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(cnn_history.history[\"val_accuracy\"], label=\"CNN\")\nplt.legend()\nplt.title(\"CNN Learning\")\nplt.ylabel(\"Validation Accuracy\")\nplt.xlabel(\"Epoch\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CNN v2: Adding More Conv Blocks\n\nTo extract even more complex spatial information from an image, we can increase the number of convolutional blocks (composed of convolutional layers, max pooling layers and dropout layers). This configuration earns around a `0.992` on the leaderboard."},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n\nbatch_size = 128\nn_epochs = 10\nverbose = 1\noptimizer = \"adam\"\n\ncnn = Sequential()\n\ncnn.add(Conv2D(32, kernel_size=(3,3), activation=\"relu\", input_shape=(28,28,1)))\ncnn.add(Conv2D(32, kernel_size=(3,3), activation=\"relu\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\"))\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Flatten())\ncnn.add(Dense(256, activation=\"relu\"))\ncnn.add(Dropout(0.5))\n\ncnn.add(Dense(10, activation=\"softmax\"))\n\ncnn.compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\ncnn_history = cnn.fit(x=X_train_2D,\n                      y=y_train,\n                      batch_size=batch_size,\n                      epochs=n_epochs,\n                      verbose=1,\n                      validation_data=(X_valid_2D, y_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CNN v3: Increase L1/L2 Filter Size & Use Same Padding\n\nFurther adjustments can be made to the network above, including: increasing the filter size in the first conv block and setting `padding=\"Same\"`."},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_size = 128\nn_epochs = 10\nverbose = 1\noptimizer = \"adam\"\n\ncnn = Sequential()\n\ncnn.add(Conv2D(32, kernel_size=(5,5), activation=\"relu\", padding=\"Same\", input_shape=(28,28,1)))\ncnn.add(Conv2D(32, kernel_size=(5,5), activation=\"relu\", padding=\"Same\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\", padding=\"Same\"))\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\", padding=\"Same\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Flatten())\ncnn.add(Dense(256, activation=\"relu\"))\ncnn.add(Dropout(0.5))\n\ncnn.add(Dense(10, activation=\"softmax\"))\n\ncnn.compile(optimizer=optimizer, loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\ncnn_history = cnn.fit(x=X_train_2D,\n                      y=y_train,\n                      batch_size=batch_size,\n                      epochs=n_epochs,\n                      verbose=1,\n                      validation_data=(X_valid_2D, y_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CNN v4: Experiment with a New Activation Function (Mish)\n\nSomeone on Twitter tipped me off to two new activation functions that are gaining popularity: Swish and Mish. I decided to try Mish, however didn't see any performance gains relative to our `relu`-based network."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Keras Implementation of Mish Activation Function.\n# Author: https://github.com/digantamisra98/Mish\n\n# Import Necessary Modules.\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nfrom keras.engine.base_layer import Layer\nfrom keras import backend as K\n\nclass Mish(Layer):\n    '''\n    Mish Activation Function.\n    .. math::\n        mish(x) = x * tanh(softplus(x)) = x * tanh(ln(1 + e^{x}))\n    Shape:\n        - Input: Arbitrary. Use the keyword argument `input_shape`\n        (tuple of integers, does not include the samples axis)\n        when using this layer as the first layer in a model.\n        - Output: Same shape as the input.\n    Examples:\n        >>> X_input = Input(input_shape)\n        >>> X = Mish()(X_input)\n    '''\n\n    def __init__(self, **kwargs):\n        super(Mish, self).__init__(**kwargs)\n        self.supports_masking = True\n\n    def call(self, inputs):\n        return inputs * K.tanh(K.softplus(inputs))\n\n    def get_config(self):\n        base_config = super(Mish, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n    def compute_output_shape(self, input_shape):\n        return input_shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_size = 128\nn_epochs = 10\nverbose = 1\noptimizer = \"adam\"\n\ncnn = Sequential()\n\ncnn.add(Conv2D(32, kernel_size=(5,5), activation=Mish(), padding=\"Same\", input_shape=(28,28,1)))\ncnn.add(Conv2D(32, kernel_size=(5,5), activation=Mish(), padding=\"Same\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=Mish(), padding=\"Same\"))\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=Mish(), padding=\"Same\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Flatten())\ncnn.add(Dense(256, activation=Mish()))\ncnn.add(Dropout(0.5))\n\ncnn.add(Dense(10, activation=\"softmax\"))\n\ncnn.compile(optimizer=optimizer, loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\ncnn_history = cnn.fit(x=X_train_2D,\n                      y=y_train,\n                      batch_size=batch_size,\n                      epochs=n_epochs,\n                      verbose=1,\n                      validation_data=(X_valid_2D, y_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training on All Training Data\n\nBased on validation performance, I decided to submit CNN v3 as my final model, training on the entire training dataset to eke out as much performance as possible.\n\nThe model below earned a public score of around `0.993`. Not bad for minimal architectural trial and error!"},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_size = 128\nn_epochs = 10\nverbose = 0\noptimizer = \"adam\"\n\ncnn = Sequential()\n\ncnn.add(Conv2D(32, kernel_size=(5,5), activation=\"relu\", padding=\"Same\", input_shape=(28,28,1)))\ncnn.add(Conv2D(32, kernel_size=(5,5), activation=\"relu\", padding=\"Same\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\", padding=\"Same\"))\ncnn.add(Conv2D(64, kernel_size=(3,3), activation=\"relu\", padding=\"Same\"))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.25))\n\ncnn.add(Flatten())\ncnn.add(Dense(256, activation=\"relu\"))\ncnn.add(Dropout(0.5))\n\ncnn.add(Dense(10, activation=\"softmax\"))\n\ncnn.compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\ncnn_history = cnn.fit(x=train_2D,\n                      y=y,\n                      batch_size=batch_size,\n                      epochs=n_epochs,\n                      verbose=verbose)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"output = generate_output(cnn, test_2D, \"cnn\", save=True)\noutput.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks for reading and, until next time, happy coding :)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":4}