{"cells":[{"metadata":{"colab":{},"colab_type":"code","id":"5ZCfC6utfVxg","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt # data visualisation\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, AvgPool2D, MaxPool2D\n\n# Import train & test data\ntrain = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv', )\n\n# Global variables\nnum_classes = 10 # classification classes 0-9\nimg_rows, img_cols = 28, 28 # image_size\nstyles = [':','-.','--','-',':','-.','--','-',':','-.','--','-']","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"QUUsAdKhMwKi"},"cell_type":"markdown","source":"\n# Data Preprocessing"},{"metadata":{"colab":{},"colab_type":"code","id":"tvUaQ7n2byn_","trusted":true},"cell_type":"code","source":"# Data Preprocessing\ndef data_preprocessing(train, test):\n  sns.countplot(train.iloc[:, 0].values)\n  # 1. One hot encoding\n  Y_train = to_categorical(train.label, num_classes) # convert vector to binary class matrix -> categorical_crossentropy \n  X_train_as_array = train.values[:,1:] # indexing everything after the 'label' col -> numpy array\n  \n  # 2. Normalizing the train and test images \n  X_train_shaped_array = X_train_as_array.reshape(train.shape[0], img_rows, img_cols, 1) # Reshaping into a 4D array indexed by \n  X_test = test.values.reshape(test.shape[0], img_rows, img_cols, 1) # image number, img_row, img_col & channel (1 = gray for RGB = 3)\n  # 3. Rescaling\n  X_train = X_train_shaped_array / 255\n  X_test = X_test / 255\n  return X_train, Y_train, X_test","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"4AlHiP62wm65"},"cell_type":"markdown","source":"I wrote a data preprocessing function that will handle all data preprocessing. We will also use it for extracting labels and reshape the pixel intensity data back to a 28x28 grid before applying our models. Keras has a to_categorical function, by passing the target variable ('label') & the number of different values, the function will return a one hot encoded version of the target that represents the 10 possible values of the target with 10 separate binary columns. The pixel values are gray scale between 0 and 255. Dividing by 255 will scale/normalize the input values to the range of 0 and 1. We do this because CNNs converge faster on [0, 1] data than on [0, 255]. This also improves optimization with the default parameters for the atom optimizer."},{"metadata":{"colab":{},"colab_type":"code","id":"nRHSz98Hv2vk","trusted":true},"cell_type":"code","source":"X_train, Y_train, X_test = data_preprocessing(train, test)","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"DUFrrdCk2ODh","trusted":true},"cell_type":"code","source":"# Split training set into 10% validation & 90% training data\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=2)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"JmJ74jEB8v8X"},"cell_type":"markdown","source":"\n# CNN Architecture\nConvolutional Neural Networks are a category of Neural Networks that have proven very effective in areas such as image recognition and classification. In this part of the kernel we will start with a baseline CNN model and improve it by running experiments till we find the most accurate and efficient CNN architecture for the classifying MNIST handwritten digits.\n"},{"metadata":{"colab_type":"text","id":"hJmh_5VnrsYN"},"cell_type":"markdown","source":"### CNN Baseline model\n\n1.   The baseline model is a simple CNN model with the first hidden layer being a convolution layer called Conv2D. This layer has  filters with a size of 3x3 and a rectifier activation function (ReLu). The input layer has 784 neurons/units (28x28) and it expects the input_shape which we have defined in the data_preprocessing function.\n2.    The second layer is a max pooling layer called MaxPool2D. We use this layer to reduce the amount of parameters which will help reduce the computational complexity.\n3.   The third layer converts the 2D matrix data into a vector called Flatten. This allows the output to be processed by standard fully conntected layers.\n4.   The fourth layer is a Dense layer, it is recommended to use this later before the output layer to help with classification.\n3.   The output layer consists of 10 neurons for the 10 classes and a softmax activation function that allows the output layer to turn the outputs into probabalities and allow one class of the 10 to be selected as the model's output prediction. Logarithmic loss is used as the loss function (called categorical_crossentropy in Keras) and the adam algorithm (best performing on the MNIST dataset) is used to learn the weights."},{"metadata":{"colab":{},"colab_type":"code","id":"nejeEGxA4ayW","trusted":true},"cell_type":"code","source":"# Define a baseline model\ndef baseline_model():\n  # create model\n  model = Sequential()\n  # Input layer with 784 neurons & the hidden layer with 24 filters of size 3x3\n  model.add(Conv2D(24, kernel_size = (3, 3), \n                   input_shape=(img_rows, img_cols, 1), \n                   activation = 'relu'))\n  # Max Pooling 2 by 2 filter, helps reducing parameters\n  model.add(MaxPool2D())\n  # Convert output of previous layers into 1D representation for each image\n  model.add(Flatten())\n  # Helps with classification\n  model[i].add(Dense(128, activation='relu'))\n  # Output layer with 10 classes (0-9) with softmax activation\n  model.add(Dense(num_classes, activation='softmax'))\n  # Compile model\n  model.compile(optimizer = 'adam', \n                loss = 'categorical_crossentropy', \n                metrics = ['accuracy'])\n  return model","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"uKDS_fSTQBWK"},"cell_type":"markdown","source":"### Convolution-Pooling Pairs\n\nThe first step is to define the number of convolution-pooling pairs. We will experiment with 1, 2 or 3 pairs. The input image size reduces after each pair, after the first pair the input image goes from 28x28 to 14x14, after the second pair 7x7 and after the third pair 4x4. so for the 4th pair the image would be too small. It is recommended to increase the number of filter after each pair.\n\n1.   CP-1: 784 - 24CP - 128 - 10\n2.   CP-2: 784 - 24CP - 32CP - 128 - 10\n3.   CP-3: 784 - 24CP - 32CP - 40CP - 128 - 10\n\n\n\n"},{"metadata":{"colab":{},"colab_type":"code","id":"mGt_kRyb1ioe","trusted":false},"cell_type":"code","source":"# Defining CNN models - convolution-pooling pairs\nnum_cnns = 3\nmodel = [0] * num_cnns \n\nfor i in range(1):\n  # CNN with 1 convolution-pooling pair\n  model[i] = Sequential()\n  model[i].add(Conv2D(24, kernel_size = (3, 3), input_shape=(img_rows, img_cols, 1), padding = 'same', activation = 'relu'))\n  model[i].add(MaxPool2D())\n  # CNN with 2 convolution-pooling pairs\n  if (i > 0):\n    model[i].add(Conv2D(32, kernel_size = (3, 3), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n  # CNN with 3 convolution-pooling pairs\n  if (i > 1):\n    model[i].add(Conv2D(40, kernel_size = (3, 3), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n  # Convert output of previous layers into 1D representation for each image\n  model[i].add(Flatten())\n  # Helps with classification\n  model[i].add(Dense(128, activation='relu'))\n  # Output layer with 10 classes (0-9) with softmax activation\n  model[i].add(Dense(num_classes, activation='softmax'))\n  # Compile model\n  model[i].compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"colab_type":"code","id":"Jn5dOxqOCyBm","outputId":"ddda9f0d-50bc-4817-9e6d-6b2bb4f6c164","trusted":false},"cell_type":"code","source":"# Training CNN models - convolution-pooling pairs\nhistory = [0] * num_cnns\nmodels = ['CP-1', 'CP-2', 'CP-3']\n\nfor i in range(num_cnns):\n  history[i] = model[i].fit(X_train, Y_train, validation_data = (X_val, Y_val), batch_size = 64, epochs = 100)\n  print('{0}: Train accuracy = {1:.5f}, Validation accuracy = {2:.5f}'.format(models[i], max(history[i].history['acc']), max(history[i].history['val_acc'])))","execution_count":null,"outputs":[]},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":350},"colab_type":"code","id":"yJpFhqAkKcgh","outputId":"392cbc03-9423-4845-e971-38263c929c6a","trusted":false},"cell_type":"code","source":"# Graph - convolution-pooling pairs\nplt.figure(figsize=(10,5))\nfor i in range(num_cnns):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('Model Accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(models, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"uJfWrPGDUTeV"},"cell_type":"markdown","source":"From the above experiment, we can see that 2 pairs performs the best. 3 pairs seems to get closer to 2 pairs after some epochs. But we will stick with 2 pairs to reduce computational complexity."},{"metadata":{"colab_type":"text","id":"hecB1WPrWIAs"},"cell_type":"markdown","source":"### Number of Convolution Filters\nFinding the best fitting number of convolutions in a CNN is through trial and error. However will not go over 32 convolutions as that would translate in computational cost. We will not only look at the best performing but also the most efficient.\n\n1.   8 Filters: 784 - 8CP - 16CP - 128 - 10\n2.   16 Filters: 784 - 16CP - 32CP - 128 - 10\n3.   24 Filters: 784 - 24CP - 48CP - 128 - 10\n4.   32 Filters: 784 - 32CP - 64CP - 128 - 10\n"},{"metadata":{"colab":{},"colab_type":"code","id":"onGYrftkXSfk","trusted":false},"cell_type":"code","source":"# Defining CNN models - number of convolution filters\nnum_cnns = 4\nmodel = [0] * num_cnns \n\nfor i in range(num_cnns):\n    model[i] = Sequential()\n    model[i].add(Conv2D(i*8+16, kernel_size = (3, 3), input_shape=(img_rows, img_cols, 1), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n    model[i].add(Conv2D(i*16+32, kernel_size = (3, 3), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n    model[i].add(Flatten())\n    model[i].add(Dense(128, activation='relu'))\n    model[i].add(Dense(num_classes, activation='softmax'))\n    model[i].compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"W4wUiwfNZ4j2","trusted":false},"cell_type":"code","source":"# Training CNN models - number of convolution filters\nhistory = [0] * num_cnns\nmodels = ['8 Filters', '16 Filters', '24 Filters', '32 Filters']\n\nfor i in range(num_cnns):\n  history[i] = model[i].fit(X_train, Y_train, validation_data = (X_val, Y_val), batch_size = 64, epochs = 100)\n  print('{0}: Train accuracy = {1:.5f}, Validation accuracy = {2:.5f}'.format(models[i], max(history[i].history['acc']), max(history[i].history['val_acc'])))","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"R6ecOHuPeFX1","trusted":false},"cell_type":"code","source":"# Graph - number of convolution filters\nplt.figure(figsize=(15,5))\nfor i in range(num_cnns):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('Model Accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(models, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"UAtK4X0PN3Tk"},"cell_type":"markdown","source":"From the above experiment, we can observe that the CNN model with 16 filters is performing just slightly better than 32 filters over 16 epochs but to ensure that we would get a better accuracy over more epochs, we will go with 32 filters. We could try and take a larger number but that would only increase the computational cost. 32 is the hotspot."},{"metadata":{"colab_type":"text","id":"P1-NMaC9qWvP"},"cell_type":"markdown","source":"### Number of Dense Units\nI have already mentioned that is recommended to add a Dense layer just before the output layer, this helps with better classification. The number of neurons/units in this layer is also chosen with trial and error. We will experiment with some numbers and see which one offers the best results.\n\n1.   0N: 784 - 32CP - 64CP - 0 - 10\n2.   32N: 784 - 32CP - 64CP - 32 - 10\n3.   64N: 784 - 32CP - 64CP - 64 - 10\n4.   128N: 784 - 32CP - 64CP - 128 - 10\n5.   256N: 784 - 32CP - 64CP - 256 - 10\n6.   512N: 784 - 32CP - 64CP - 512 - 10\n7.   1024NN: 784 - 32CP - 64CP - 2048 - 10\n"},{"metadata":{"colab":{},"colab_type":"code","id":"IenDcukwqxIj","trusted":false},"cell_type":"code","source":"# Defining CNN models - number of dense units\nnum_cnns = 7\nmodel = [0] * num_cnns \n\nfor i in range(num_cnns):\n    model[i] = Sequential()\n    model[i].add(Conv2D(32, kernel_size = (3, 3), input_shape=(img_rows, img_cols, 1), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n    model[i].add(Conv2D(64, kernel_size = (3, 3), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n    model[i].add(Flatten())\n    if (i > 0):\n        model[i].add(Dense(2**(i+4), activation='relu'))\n    model[i].add(Dense(num_classes, activation='softmax'))\n    model[i].compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"I4ykEy2KrQfG","trusted":false},"cell_type":"code","source":"# Training CNN models - number of dense units\nhistory = [0] * num_cnns\nmodels = ['0N', '32N', '64N', '128N', '256N', '512N', '1024N']\n\nfor i in range(num_cnns):\n  history[i] = model[i].fit(X_train, Y_train, validation_data = (X_val, Y_val), batch_size = 64, epochs = 100)\n  print('{0}: Train accuracy = {1:.5f}, Validation accuracy = {2:.5f}'.format(models[i], max(history[i].history['acc']), max(history[i].history['val_acc'])))","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"Fv_wTqdQFyZh","trusted":false},"cell_type":"code","source":"# Graph - number of convolution filters\nplt.figure(figsize=(10,5))\nfor i in range(num_cnns):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('Model Accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(models, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"coHawVhOHJqe"},"cell_type":"markdown","source":"From this experiment, we can conclude that a Dense layer with 128 units has the best performance. Dense layers with more units would generally perform better over time but they're not worth the computational cost nor is adding more dense layers."},{"metadata":{"colab_type":"text","id":"Yh6fwbjxIhEi"},"cell_type":"markdown","source":"### Dropout\nDropout is a technique used to prevent a model from overfitting. Dropout works by randomly setting the outgoing edges of hidden units (neurons that make up hidden layers) to 0 at each update of the training phase. This will prevent our network from overfitting thus helping our network generalize better. It is recommended to do this after each layer.\n\n\n\n1.   D0 = 0%\n2.   D10 = 10%\n3.   D20 = 20%\n4.   D30 = 30%\n5.   D40 = 40%\n6.   D50 = 50%\n\n"},{"metadata":{"colab":{},"colab_type":"code","id":"hrkXLRWeKnfM","trusted":false},"cell_type":"code","source":"# Defining CNN models - dropout amount\nnum_cnns = 6\nmodel = [0] * num_cnns \n\nfor i in range(num_cnns):\n    model[i] = Sequential()\n    model[i].add(Conv2D(32, kernel_size = (3, 3), input_shape=(img_rows, img_cols, 1), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n    model[i].add(Dropout(i*0.1))\n    model[i].add(Conv2D(64, kernel_size = (3, 3), padding = 'same', activation = 'relu'))\n    model[i].add(MaxPool2D())\n    model[i].add(Dropout(i*0.1))\n    model[i].add(Flatten())\n    model[i].add(Dense(128, activation='relu'))\n    model[i].add(Dropout(i*0.1))\n    model[i].add(Dense(num_classes, activation='softmax'))\n    model[i].compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"h75Zu3MCL-zL","trusted":false},"cell_type":"code","source":"# Training CNN models - dropout amount\nhistory = [0] * num_cnns\nmodels = ['D0', 'D10', 'D20', 'D30', 'D40', 'D50']\n\nfor i in range(num_cnns):\n  history[i] = model[i].fit(X_train, Y_train, validation_data = (X_val, Y_val), batch_size = 64, epochs = 100)\n  print('{0}: Train accuracy = {1:.5f}, Validation accuracy = {2:.5f}'.format(models[i], max(history[i].history['acc']), max(history[i].history['val_acc'])))","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"CEJRS7DvSMCb","trusted":false},"cell_type":"code","source":"# Graph - dropout amount\nplt.figure(figsize=(10,5))\nfor i in range(num_cnns):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('Model Accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(models, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"vkfA9G8a3ET_"},"cell_type":"markdown","source":"From the experiment above, we can observe that our accuracy increases best at a 40% dropout."},{"metadata":{"colab_type":"text","id":"GgmnJU2D_CRt"},"cell_type":"markdown","source":"# Evaluate & Submit"},{"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":281},"colab_type":"code","id":"-sedtbEMAEvy","outputId":"21c6581f-a1f5-4a26-efdf-60a68c837a0b","trusted":false},"cell_type":"code","source":"# Evaluate improved model for 10 images\ntest_pred = model[4].predict(X_test)\nresults = test_pred.argmax(axis = 1)\n\nfor i in range(1, 10):\n  index = np.random.randint(1, 28001)\n  plt.subplot(3, 3, i)\n  plt.imshow(X_test[index].reshape(28, 28))\n  plt.title(\"Predicted Label: {}\".format(results[index]))\nplt.subplots_adjust(hspace=1.2, wspace=1.2)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"KUZmH6JX8coR","trusted":false},"cell_type":"code","source":"# Submit\nresults = pd.Series(results, name='Label')\nsubmission = pd.concat([pd.Series(range(1, 28001), name = 'ImageId'), results], axis = 1)\nsubmission.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"2xkQMFot3nHI"},"cell_type":"markdown","source":"# Conclusion\nDefining the architecture of Convolutional Neural Networks is a process of trial and error. It's also a random process because each time you run the same code, you get different results. To counter this, the experiments should be run multiple times and an average should be taken of these results. In the case of this course project, it has not been done because that was not the main objective of the course. The main objective was to improve on a baseline CNN architecture. The best performing CNN architecture for classifying MNIST handwritten digits obtained in this project is 784 - 32CP - 64CP - 128 - 10 met 40% dropout."},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nsample_submission = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")","execution_count":0,"outputs":[]}],"metadata":{"colab":{"collapsed_sections":["Yh6fwbjxIhEi"],"name":"Kaggle Digit Recognizer.ipynb","provenance":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"}},"nbformat":4,"nbformat_minor":1}