{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n### Main goal\nThe main goal of this project is to explore multiple CNN architectures and evaluate their accuracy and efficiency for the famous MNIST dataset.\nThe secondary goal is to predict labels for `test.csv` using the best model and submit the results to the competition.\n\n## Data\nFrom [Digit Recognizer Dataset](https://www.kaggle.com/c/digit-recognizer/data) Description:\n>The data files train.csv and test.csv contain gray-scale images of hand-drawn digits, from zero through nine.\n\n>Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\n>The training data set, (train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n\n>Each pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n\n## Plan\n1. Load and explore data\n2. Clean and prepare data for Neural Networks\n    * Scale\n    * Reshape\n    * Encode labels\n    * Split into training and validation set\n    * Add Data Augmentation to avoid overfitting\n3. Neural Networks\n    * Fully Connected NN\n    * Convolutional NN (experiment with different architectures)\n4. Evaluate models and create the submission file\n5. Key Findings and suggestions for next steps\n\n### References\n1. [Convolutional Neural Network (CNN) Tutorial](https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial) by DATAI\n2. [Deep Neural Network Keras way](https://www.kaggle.com/poonaml/deep-neural-network-keras-way) by POONAM LIGADE\n3. [Introduction to CNN Keras - 0.997 (top 6%)](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6) by YASSINE GHOUZAM\n4. [How to choose CNN Architecture MNIST](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist) by CHRIS DEOTTE\n5. [MNIST with Keras for Beginners(.99457)](https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457) by ADITYA SONI\n","metadata":{}},{"cell_type":"markdown","source":"# Loading and Processing Data","metadata":{}},{"cell_type":"code","source":"# loading libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nfrom keras.utils.np_utils import to_categorical\nfrom keras.utils.layer_utils import count_params\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Activation, Flatten\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.optimizers import Adam\n\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau, Callback","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# to compare the computation time between models\nimport time\n\nclass TimeHistory(Callback):\n    def on_train_begin(self, logs={}):\n        self.times = []\n\n    def on_epoch_begin(self, epoch, logs={}):\n        self.epoch_time_start = time.time()\n\n    def on_epoch_end(self, epoch, logs={}):\n        self.times.append(time.time() - self.epoch_time_start)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading data sets\ntrain_data = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_data = pd.read_csv(\"../input/digit-recognizer/test.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# looking at the shape of the data\nprint(\"Train data size: \", train_data.shape)\nprint(\"Test data size: \", test_data.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check for NAs\nprint('NAs in train set: ', train_data.isnull().sum().sum())\nprint('NAs in test set: ', test_data.isnull().sum().sum())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Structure of the data:**\n* Each row represents one image (28x28 pixels) of a hand-drawn digit, from zero through nine\n* First column *'label'* represents the digit drawn in the image \n* Each column (except first) represents one pixel in the image\n* Values in the cells represent the lightness of the pixel (grayscale)\n    >This pixel-value is an integer between 0 and 255, inclusive.    \n    [Digit Recognizer Dataset](https://www.kaggle.com/c/digit-recognizer/data)","metadata":{}},{"cell_type":"markdown","source":"**Data cleaning and preprocessing:**\n1. Extract labels from train set into separate target dataframe `y_train`\n2. Scale values in `X_train` and `X_test`: new_value = old_value / 255\n3. Reshape each data entry (image) into (28x28x1) matrices\n4. Encode *'label'* (one-hot encoding)\n5. Split train set into train and validation sets\n6. Add Data Augmentation: artificially expand handwritten digit dataset by zooming, rotating and shifting original images to avoid overfitting","metadata":{}},{"cell_type":"code","source":"# define target and features\ny_train = train_data['label'].copy()\n\nX_train = train_data.drop(columns = 'label').copy()\nX_test = test_data.copy() # no labels here\n\nprint('X_train shape: ', X_train.shape)\nprint('X_test shape: ', X_test.shape)\nprint('y_train shape: ', y_train.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# look at the distribution of labels in the train dataset\ny_train.value_counts(normalize=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.barplot(x=y_train.value_counts().index, y=y_train.value_counts())\nplt.title('Distribution of classes in the train dataset')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All 10 classes are represented in the train dataset equally: about 10% or 4000 enteries of each.","metadata":{}},{"cell_type":"code","source":"# Scale values\nX_train = X_train / 255.0\nX_test = X_test / 255.0\n\n# Check scaling\nprint('X_train maximum value: ', X_train.values.max())\nprint('X_train minimum value: ', X_train.values.min())\nprint('-' * 40)\nprint('X_test maximum value: ', X_test.values.max())\nprint('X_test minimum value: ', X_test.values.min())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshape\nX_train = X_train.values.reshape((X_train.shape[0],) + (28, 28, 1))\nX_test = X_test.values.reshape((X_test.shape[0],) + (28, 28, 1))\n\n# Check new shapes\nprint('X_train shape: ', X_train.shape)\nprint('X_test shape: ', X_test.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# One-hot encoding\ny_train = to_categorical(y_train, num_classes = 10)\nprint('y_train shape: ', y_train.shape)\ny_train[0:5]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize images with labels\nfig, axList = plt.subplots(5, 5, sharex=False, sharey=False)\naxList = axList.flatten()\nfig.set_size_inches(10, 10)\n\nfor i,ax in enumerate(axList):\n    ax.imshow(X_train[i], cmap=plt.get_cmap('gray'))\n    ax.set_title(np.argmax(y_train[i]))\n            \nfig.tight_layout()\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split train data into the train and the validation set for the fitting\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = 0.1, random_state = 42)\n\nprint('X_train shape: ', X_train.shape)\nprint('X_val shape: ', X_val.shape)\nprint('y_train shape: ', y_train.shape)\nprint('y_val shape: ', y_val.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data augmentation\ndatagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # dimesion reduction\n        rotation_range=5,  # randomly rotate images in the range 5 degrees\n        zoom_range = 0.1, # Randomly zoom image 10%\n        width_shift_range=0.1,  # randomly shift images horizontally 10%\n        height_shift_range=0.1,  # randomly shift images vertically 10%\n        horizontal_flip=False,  # randomly flip images horizontally\n        vertical_flip=False)  # randomly flip images vertically\n\ndatagen.fit(X_train)\n\n# source: https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Neural Network Models\n## Fully Connected Neural Network\nThis model doesn't consider the spatial arrangement of pixels in the image. It treats all inputs interchangeably.    \n    \nArchitecture:\n* Flatten\n* Dense 1 (512 units, 'relu')\n* Dropout (0.4)\n* Dense 2 (256 units, 'relu')\n* Dropout (0.5)\n* Dense - output (10 units, 'softmax')","metadata":{}},{"cell_type":"code","source":"# build and compile model\nmodel_fcnn = Sequential([\n    Flatten(input_shape = (28,28,1)),\n    Dense(512, activation='relu'),\n    Dropout(0.4),\n    Dense(256, activation='relu'),\n    Dropout(0.5),\n    Dense(10, activation='softmax')\n])\n\nmodel_fcnn.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at the model summary\nmodel_fcnn.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# set up fitting parameters\n# optimizer\noptimizer = Adam(learning_rate = 0.001, beta_1 = 0.9, beta_2 = 0.999)\n\n# epochs and batch size\nepochs = 20 \nbatch_size = 250\n\n# define learning rate reduction\nlearning_rate_reduction = ReduceLROnPlateau(monitor = 'val_acc', \n                                            patience = 3, \n                                            verbose = 1, \n                                            factor = 0.5, \n                                            min_lr = 0.0001)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit the model\ntime_callback = TimeHistory()\nfcnn_history1 = model_fcnn.fit(datagen.flow(X_train, y_train, batch_size = batch_size),\n                               epochs = epochs, validation_data = (X_val, y_val),\n                               verbose = 1, steps_per_epoch = X_train.shape[0] // batch_size, \n                               callbacks = [learning_rate_reduction, time_callback])\ntimes = time_callback.times","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot Loss and Accuracy\nn = len(fcnn_history1.history[\"loss\"])\n\nfig = plt.figure(figsize=(12, 6))\nax = fig.add_subplot(1, 2, 1)\nax.plot(range(n), (fcnn_history1.history[\"loss\"]),'r--', label=\"Train Loss\")\nax.plot(range(n), (fcnn_history1.history[\"val_loss\"]),'b-', label=\"Validation Loss\")\nax.legend()\nax.set_title('Loss over iterations')\n\nax = fig.add_subplot(1, 2, 2)\nax.plot(range(n), (fcnn_history1.history[\"accuracy\"]),'r--', label=\"Train Acc\")\nax.plot(range(n), (fcnn_history1.history[\"val_accuracy\"]),'b-', label=\"Validation Acc\")\nax.legend(loc='lower right')\nax.set_title('Accuracy over iterations')\n\nfig.suptitle('Fully Connected Neural Network')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = model_fcnn.evaluate(X_val, y_val, verbose=0)\nprint('Test score:', round(score[0], 4))\nprint('Test accuracy:', round(score[1], 4))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit the model again to see if score will improve\nfcnn_history2 = model_fcnn.fit(datagen.flow(X_train, y_train, batch_size = batch_size),\n                               epochs = 10, validation_data = (X_val, y_val),\n                               verbose = 1, steps_per_epoch = X_train.shape[0] // batch_size, \n                               callbacks = [learning_rate_reduction])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n = len(fcnn_history1.history[\"loss\"])\nm = len(fcnn_history2.history['loss'])\nfig, ax = plt.subplots(figsize=(12, 6))\n\nax.plot(range(n), fcnn_history1.history[\"loss\"],'r', marker='s', label=\"Train Loss - Run 1\")\nax.plot(range(n, n+m), fcnn_history2.history[\"loss\"], 'hotpink', marker='s', label=\"Train Loss - Run 2\")\n\nax.plot(range(n), fcnn_history1.history[\"val_loss\"],'b', marker='o', label=\"Validation Loss - Run 1\")\nax.plot(range(n, n+m), fcnn_history2.history[\"val_loss\"], 'LightSkyBlue', marker='o',  label=\"Validation Loss - Run 2\")\n\nax.legend()\nax.set_title(\"Loss Values for Fully Connected Neural Network\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = model_fcnn.evaluate(X_val, y_val, verbose=0)\nscores = pd.DataFrame(data = {'model': ['fully connected nn'],  \n                              'test score': [round(score[0], 4)], \n                              'test accuracy': [round(score[1], 4)],\n                              'sec per epoch': [round(sum(times)/len(times), 2)],\n                              'trainable weights': [count_params(model_fcnn.trainable_weights)]})\nscores","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convolutional Neural Networks\nTo design my CNNs I will use the following principles:\n1. I will use only 3x3 convolutions, but will apply 2 Convolutional layers in a row for each \"cell\" to follow the idea of VGG Networks: multiple 3x3 kernels will simulate bigger kernels with fewer weights to adjust.\n2. Each \"convolutional cell\" will consist of:\n    * 2 convolutional layers with {n} feature maps using 3x3 filter and stride 1 ({n}C3);\n    * 1 pooling layer using 2x2 filter and stride 2 (P2);\n    * 1 dropout layer (Dp).\n3. I will experiment with the number of cells and the number of feature maps in my CNN:\n    * 1-cell CNN:   \n        * Small:   \n            (28x28x1) -> **[8C3 - 8C2 - P2 - Dp]** -> (14x14x32) -> **Flatten** -> **Dense(512, relu)** -> **Dp** -> **Dense(10, softmax)**  \n        * Big:   \n            (28x28x1) -> **[32C3 - 32C2 - P2 - Dp]** -> (14x14x32) -> **Flatten** -> **Dense(512, relu)** -> **Dp** -> **Dense(10, softmax)**  \n    * 2-cell CNN:   \n        * Small:   \n            (28x28x1) -> **[8C3 - 8C2 - P2 - Dp]** -> (14x14x32) -> **[16C3 - 16C2 - P2 - Dp]** -> (7x7x64) -> **Flatten** -> **Dense(512, relu)** -> **Dp** -> **Dense(10, softmax)**\n        * Big:   \n            (28x28x1) -> **[32C3 - 32C2 - P2 - Dp]** -> (14x14x32) -> **[64C3 - 64C2 - P2 - Dp]** -> (7x7x64) -> **Flatten** -> **Dense(512, relu)** -> **Dp** -> **Dense(10, softmax)**\n        \n### 1-cell Small CNN","metadata":{}},{"cell_type":"code","source":"# build and compile model\nmodel_1s_cnn = Sequential([\n    Conv2D(filters = 8, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)),\n    Conv2D(filters = 8, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    MaxPooling2D(pool_size = (2,2), strides = (2,2)),\n    Dropout(0.4),\n    Flatten(),\n    Dense(512, activation = 'relu'),\n    Dropout(0.5),\n    Dense(10, activation = 'softmax')\n])\n\nmodel_1s_cnn.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at the model summary\nmodel_1s_cnn.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit the model\ntime_callback = TimeHistory()\ncnn1s_history1 = model_1s_cnn.fit(datagen.flow(X_train, y_train, batch_size = batch_size),\n                               epochs = epochs, validation_data = (X_val, y_val),\n                               verbose = 1, steps_per_epoch = X_train.shape[0] // batch_size, \n                               callbacks = [learning_rate_reduction, time_callback])\ntimes = time_callback.times","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot Loss and Accuracy\nn = len(cnn1s_history1.history[\"loss\"])\n\nfig = plt.figure(figsize=(12, 6))\nax = fig.add_subplot(1, 2, 1)\nax.plot(range(n), (cnn1s_history1.history[\"loss\"]),'r--', label=\"Train Loss\")\nax.plot(range(n), (cnn1s_history1.history[\"val_loss\"]),'b-', label=\"Validation Loss\")\nax.legend()\nax.set_title('Loss over iterations')\n\nax = fig.add_subplot(1, 2, 2)\nax.plot(range(n), (cnn1s_history1.history[\"accuracy\"]),'r--', label=\"Train Acc\")\nax.plot(range(n), (cnn1s_history1.history[\"val_accuracy\"]),'b-', label=\"Validation Acc\")\nax.legend(loc='lower right')\nax.set_title('Accuracy over iterations')\n\nfig.suptitle('CNN small model with 1 cell')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = model_1s_cnn.evaluate(X_val, y_val, verbose=0)\nscores = scores.append({'model': '1-cell small cnn', \n                        'test score': round(score[0], 4), \n                        'test accuracy': round(score[1], 4),\n                        'sec per epoch': round(sum(times)/len(times), 2),\n                        'trainable weights': count_params(model_1s_cnn.trainable_weights)}, \n                        ignore_index=True)\nscores","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1-cell Big CNN","metadata":{}},{"cell_type":"code","source":"# build and compile model\nmodel_1b_cnn = Sequential([\n    Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)),\n    Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    MaxPooling2D(pool_size = (2,2), strides = (2,2)),\n    Dropout(0.4),\n    Flatten(),\n    Dense(512, activation = 'relu'),\n    Dropout(0.5),\n    Dense(10, activation = 'softmax')\n])\n\nmodel_1b_cnn.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at the model summary\nmodel_1b_cnn.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit the model\ntime_callback = TimeHistory()\ncnn1b_history1 = model_1b_cnn.fit(datagen.flow(X_train, y_train, batch_size = batch_size),\n                               epochs = epochs, validation_data = (X_val, y_val),\n                               verbose = 1, steps_per_epoch = X_train.shape[0] // batch_size, \n                               callbacks = [learning_rate_reduction, time_callback])\ntimes = time_callback.times","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot Loss and Accuracy\nn = len(cnn1b_history1.history[\"loss\"])\n\nfig = plt.figure(figsize=(12, 6))\nax = fig.add_subplot(1, 2, 1)\nax.plot(range(n), (cnn1b_history1.history[\"loss\"]),'r--', label=\"Train Loss\")\nax.plot(range(n), (cnn1b_history1.history[\"val_loss\"]),'b-', label=\"Validation Loss\")\nax.legend()\nax.set_title('Loss over iterations')\n\nax = fig.add_subplot(1, 2, 2)\nax.plot(range(n), (cnn1b_history1.history[\"accuracy\"]),'r--', label=\"Train Acc\")\nax.plot(range(n), (cnn1b_history1.history[\"val_accuracy\"]),'b-', label=\"Validation Acc\")\nax.legend(loc='lower right')\nax.set_title('Accuracy over iterations')\n\nfig.suptitle('CNN big model with 1 cell')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = model_1b_cnn.evaluate(X_val, y_val, verbose=0)\nscores = scores.append({'model': '1-cell big cnn', \n                        'test score': round(score[0], 4), \n                        'test accuracy': round(score[1], 4),\n                        'sec per epoch': round(sum(times)/len(times), 2),\n                        'trainable weights': count_params(model_1b_cnn.trainable_weights)}, \n                        ignore_index=True)\nscores","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2-cell Small CNN","metadata":{}},{"cell_type":"code","source":"# build and compile model\nmodel_2s_cnn = Sequential([\n    Conv2D(filters = 8, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)),\n    Conv2D(filters = 8, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    MaxPooling2D(pool_size = (2,2), strides = (2,2)),\n    Dropout(0.4),\n    \n    Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    MaxPooling2D(pool_size = (2,2), strides = (2,2)),\n    Dropout(0.4),\n    \n    Flatten(),\n    Dense(512, activation = 'relu'),\n    Dropout(0.5),\n    Dense(10, activation = 'softmax')\n])\n\nmodel_2s_cnn.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at the model summary\nmodel_2s_cnn.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit the model\ntime_callback = TimeHistory()\ncnn2s_history1 = model_2s_cnn.fit(datagen.flow(X_train, y_train, batch_size = batch_size),\n                               epochs = epochs, validation_data = (X_val, y_val),\n                               verbose = 1, steps_per_epoch = X_train.shape[0] // batch_size, \n                               callbacks = [learning_rate_reduction, time_callback])\ntimes = time_callback.times","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot Loss and Accuracy\nn = len(cnn2s_history1.history[\"loss\"])\n\nfig = plt.figure(figsize=(12, 6))\nax = fig.add_subplot(1, 2, 1)\nax.plot(range(n), (cnn2s_history1.history[\"loss\"]),'r--', label=\"Train Loss\")\nax.plot(range(n), (cnn2s_history1.history[\"val_loss\"]),'b-', label=\"Validation Loss\")\nax.legend()\nax.set_title('Loss over iterations')\n\nax = fig.add_subplot(1, 2, 2)\nax.plot(range(n), (cnn2s_history1.history[\"accuracy\"]),'r--', label=\"Train Acc\")\nax.plot(range(n), (cnn2s_history1.history[\"val_accuracy\"]),'b-', label=\"Validation Acc\")\nax.legend(loc='lower right')\nax.set_title('Accuracy over iterations')\n\nfig.suptitle('CNN small model with 2 cells')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = model_2s_cnn.evaluate(X_val, y_val, verbose=0)\nscores = scores.append({'model': '2-cell small cnn', \n                        'test score': round(score[0], 4), \n                        'test accuracy': round(score[1], 4),\n                        'sec per epoch': round(sum(times)/len(times), 2),\n                        'trainable weights': count_params(model_2s_cnn.trainable_weights)}, \n                        ignore_index=True)\nscores","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2-cell Big CNN","metadata":{}},{"cell_type":"code","source":"# build and compile model\nmodel_2b_cnn = Sequential([\n    Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)),\n    Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    MaxPooling2D(pool_size = (2,2), strides = (2,2)),\n    Dropout(0.4),\n    \n    Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'),\n    MaxPooling2D(pool_size = (2,2), strides = (2,2)),\n    Dropout(0.4),\n    \n    Flatten(),\n    Dense(512, activation = 'relu'),\n    Dropout(0.5),\n    Dense(10, activation = 'softmax')\n])\n\nmodel_2b_cnn.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at the model summary\nmodel_2b_cnn.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit the model\ntime_callback = TimeHistory()\ncnn2b_history1 = model_2b_cnn.fit(datagen.flow(X_train, y_train, batch_size = batch_size),\n                               epochs = epochs, validation_data = (X_val, y_val),\n                               verbose = 1, steps_per_epoch = X_train.shape[0] // batch_size, \n                               callbacks = [learning_rate_reduction, time_callback])\ntimes = time_callback.times","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot Loss and Accuracy\nn = len(cnn2b_history1.history[\"loss\"])\n\nfig = plt.figure(figsize=(12, 6))\nax = fig.add_subplot(1, 2, 1)\nax.plot(range(n), (cnn2b_history1.history[\"loss\"]),'r--', label=\"Train Loss\")\nax.plot(range(n), (cnn2b_history1.history[\"val_loss\"]),'b-', label=\"Validation Loss\")\nax.legend()\nax.set_title('Loss over iterations')\n\nax = fig.add_subplot(1, 2, 2)\nax.plot(range(n), (cnn2b_history1.history[\"accuracy\"]),'r--', label=\"Train Acc\")\nax.plot(range(n), (cnn2b_history1.history[\"val_accuracy\"]),'b-', label=\"Validation Acc\")\nax.legend(loc='lower right')\nax.set_title('Accuracy over iterations')\n\nfig.suptitle('CNN big model with 2 cells')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = model_2b_cnn.evaluate(X_val, y_val, verbose=0)\nscores = scores.append({'model': '2-cell big cnn', \n                        'test score': round(score[0], 4), \n                        'test accuracy': round(score[1], 4),\n                        'sec per epoch': round(sum(times)/len(times), 2),\n                        'trainable weights': count_params(model_2b_cnn.trainable_weights)}, \n                        ignore_index=True)\nscores","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Best model evaluation","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nax.plot(cnn1s_history1.history[\"val_accuracy\"],'r-', label=\"1-cell small\")\nax.plot(cnn1b_history1.history[\"val_accuracy\"],'g--', label=\"1-cell big\")\nax.plot(cnn2s_history1.history[\"val_accuracy\"],'b-.', label=\"2-cell small\")\nax.plot(cnn2b_history1.history[\"val_accuracy\"],'m:', label=\"2-cell big\")\nax.set(title='Validation Accuracy for CNNs')\nax.legend()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# confusion matrix\n# predict values (probabilities)\ny_pred = model_2b_cnn.predict(X_val)\n# predict classes\ny_pred_classes = np.argmax(y_pred, axis = 1)\n# convert true classes to one vector with 10 categories\ny_true = np.argmax(y_val, axis = 1) \n\n# calculate confusion matrix\ncm = confusion_matrix(y_true, y_pred_classes) \n\n# plot confusion matrix\nfig,ax = plt.subplots(figsize = (8, 8))\nsns.heatmap(cm, annot = True, linewidths=0.01, cmap=\"Blues\", linecolor=\"gray\", fmt= '.1f', ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix for CNN model with 3 cells\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Take a closer look at the errors\n# source: https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457\n\n# find indeces of errors\nerrors = (y_pred_classes - y_true != 0)\n\ny_pred_classes_errors = y_pred_classes[errors]\ny_pred_errors = y_pred[errors]\ny_true_errors = y_true[errors]\nX_val_errors = X_val[errors]\n\ndef display_errors(errors_index, img_errors, pred_errors, obs_errors):\n    \"\"\" This function shows 16 images with their predicted and real labels\"\"\"\n    n = 0\n    nrows = 4\n    ncols = 4\n    fig, ax = plt.subplots(nrows, ncols, sharex=True, sharey=True)\n    fig.set_size_inches(8, 8)\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row, col].imshow((img_errors[error]).reshape((28,28)), cmap=plt.get_cmap('gray'))\n            ax[row, col].set_title(\"Predicted: {}\\nTrue: {}\".format(pred_errors[error], obs_errors[error]))\n            n += 1\n    fig.tight_layout()\n    fig.show()  \n\n# Probabilities of the wrong predicted numbers\ny_pred_errors_prob = np.max(y_pred_errors, axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(y_pred_errors, y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = np.argsort(y_pred_errors_prob - true_prob_errors)\n\n# Top 25 errors \nmost_important_errors = delta_pred_true_errors[-16:]\n\n# Show the top 25 errors\ndisplay_errors(most_important_errors, X_val_errors, y_pred_classes_errors, y_true_errors)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submission","metadata":{}},{"cell_type":"code","source":"# perform prediction on the test set\nresults_1b = model_1b_cnn.predict(X_test)\nresults_1b = np.argmax(results_1b, axis = 1)\n\nresults_2b = model_2b_cnn.predict(X_test)\nresults_2b = np.argmax(results_2b, axis = 1)\n\n# create submission files\nsubmission_1b = pd.DataFrame({\"ImageId\": list(range(1,len(results_1b)+1)),\n                           \"Label\": results_1b})\nsubmission_1b.to_csv(\"MNIST-CNN_1b.csv\", index=False)\n\n\nsubmission_2b = pd.DataFrame({\"ImageId\": list(range(1,len(results_2b)+1)),\n                           \"Label\": results_2b})\nsubmission_2b.to_csv(\"MNIST-CNN_2b.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Findings \n1. As expected, the Fully Connected model performed worse than any CNN model.\n2. Out of all CNN architectures I tried, CNNs with more feature maps (\"Big\") showed more accurate predictions on the validation set.\n3. The number of \"convolutional cells\" didn't seem to affect the performance of CNN models much for this particular dataset. Though the images are pretty small (28x28 pixels), therefore the model might not benefit from extra convolutional layers.\n4. Out of the 2 most successful models (1-cell Big CNN and 2-cell Big CNN), the shallower one (1-cell Big CNN) requires less computational time per epoch (38 sec vs 60 sec for 2-cell Big CNN), even though it has more trainable weights.   \n\n**Conclusion:**\nOut of all models I tried in this notebook, the 1-cell Big CNN model showed the best results for the MNIST dataset.","metadata":{}}]}