{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## CNN Experiment\n\nIn this notebook I will try to experiment with the architecture of a CNN in order to build a good enough model to get me at around 99% accuracy (if possible).\nBeing the third notebook I publish, this is again inspired from the works of others that I will mention here: \\\nhttps://www.kaggle.com/code/yassineghouzam/introduction-to-cnn-keras-0-997-top-6 \\\nhttps://www.kaggle.com/code/poonaml/deep-neural-network-keras-way \\\nhttps://www.kaggle.com/code/cdeotte/how-to-choose-cnn-architecture-mnist","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import Sequential\nfrom keras import Input\nfrom keras.layers import Conv2D, Dense, BatchNormalization, Flatten, MaxPool2D, Dropout, Activation, InputLayer\nfrom keras.utils.np_utils import to_categorical","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load test and train and check what we deal with\ntrain = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')\n\ntrain.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check if we have any null values\nprint('Training null values = ', train.isnull().sum().sum())\nprint('Test null values = ', test.isnull().sum().sum())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# No null values, it means that we can go ahead and preprocess our data\n\nX = train.iloc[:, 1:]\ny = train.iloc[:, 0]\n\ny","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# See what kind of labels we have\ny.value_counts()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# One hot encode labels and standardize the features\ny = to_categorical(y.values, num_classes=10)\n\nX = X / 255.0\nX_test = test / 255.0\n\ny[0]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshape training/test images so they can be fed as 28x28x1 images to the model\nX = np.reshape(X.values, (-1, 28, 28, 1))\nX_test = np.reshape(X_test.values, (-1, 28, 28, 1))\n\nprint(X.shape, X_test.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.1, random_state=7)\nprint(X_train.shape, y_train.shape, X_val.shape, y_val.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## What is the general approach\n\nFrom what I have read here and articles on the internet, I saw that CNNs are the best approach to image classification problems. This is because convolution layers make it easy to spot shapes and more complex features, while at the sam time, trying to keep parameters as small as can be.\nSo in the next few cells I will try to answer the following questions:\n\n1. What is an optimal number of convolutional layers?\n2. Batch size\n3. Does batch normalization help?\n4. Does Dropout help? If so, how many Dropout layers are optimal and what values do they have?\n5. How many maxpooling layers?\n6. How many and how big should the end dense layers be?\n7. Does data augmentation help (time vs accuracy improvement)\n8. After reaching an 'optimal' model, does inspecting it tell us anything more? Can I improve the architecture?\n\n\nI will start with a simple model that will be comprised of Convolution - Dense - Output. So initially my models will have the convolution parts (which I will test to find the optimal) and 2 dense layers, one used for learning (initially with 256 neurons) and one for the output using softmax activation function.\nLet's see what we will find.","metadata":{}},{"cell_type":"markdown","source":"### 1. How many convolutional layers are optimal?\n\nFor this question I will train 4 models, each for 10 epochs and see how much it improves based on time spent training.\nI will use the following formula: ImprRate = (ModelValAcc - BaseValAcc) * 100 / (ModelTimeTrainingInMinutes - BaseTimeTrainingInMinutes),\nWhere BaseValAcc is the accuracy that I get on the validation set for the basic model that I train (without modifications) and BaseTimeTrainingInMinutes is the time it takes to train such a basic model. The other values correspond to models we are training. ","metadata":{}},{"cell_type":"code","source":"import time\n\nEPOCH_SIZE = 10\n\nmodels_1 = [0] * 4\ntimes_1 = [0] * 4\nhistorys_1 = [0] * 4\n\nfor i in range(len(models_1)):\n    models_1[i] = Sequential()\n    models_1[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=X_train.shape[1:]))\n    for j in range(i):\n        models_1[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\n    models_1[i].add(Flatten())\n    models_1[i].add(Dense(units=256, activation='relu'))\n    models_1[i].add(Dense(units=10, activation='softmax'))\n    models_1[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow\nprint(tensorflow.__version__)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_1)):\n    start_time = time.time()\n    historys_1[i] = models_1[i].fit(X_train, y_train, batch_size=512, epochs=EPOCH_SIZE,\n                                   validation_data=(X_val, y_val))\n    times_1[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historys_1[0].history['accuracy'][-1]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_1)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_1[i].history['accuracy'][-1], historys_1[i].history['val_accuracy'][-1], times_1[i]))\n    \n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range (1, 4):\n    print((historys_1[i].history['val_accuracy'][-1] - historys_1[0].history['val_accuracy'][-1]) * 600 / (times_1[i] - times_1[0]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like we had the best accuracy improvement with the model number 3 out of 4. That is when we had 3 convolutional layers.\nBecause all this is so computationally heavy, I will check the maxpooling question first to see if we can get a better time for our training (since training our model costed us 10 minutes)","metadata":{}},{"cell_type":"markdown","source":"### 5. How many maxpooling layers are optimal?\n\nWe found that our network works well with 3 convolution layers. Let's see if we can reduce time without severly reducing accuracy by adding some maxpooling layers.\nLet's try 5 models here:\n\n1. Maxpooling after every convolutional layer\n2. Maxpooling after the first 2 conv layers\n3. Maxpooling after 1st and 3rd\n4. Same as 1 but with increased kernel size\n5. Same as 1 but with different kernel sizes","metadata":{}},{"cell_type":"code","source":"models_5 = [0] * 5\ntimes_5 = [0] * 5\nhistorys_5 = [0] * 5\n\nfor i in range(len(models_5) - 2):\n    models_5[i] = Sequential()\n    models_5[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=X_train.shape[1:]))\n    models_5[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_5[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\n    if i !=3: \n        models_5[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_5[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\n    if i !=2: \n        models_5[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_5[i].add(Flatten())\n    models_5[i].add(Dense(units=256, activation='relu'))\n    models_5[i].add(Dense(units=10, activation='softmax'))\n    models_5[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\n# Model 4    \nmodels_5[3] = Sequential()\nmodels_5[3].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=X_train.shape[1:]))\nmodels_5[3].add(MaxPool2D(pool_size=3, strides=2))\nmodels_5[3].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\nmodels_5[3].add(MaxPool2D(pool_size=3, strides=2))\nmodels_5[3].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))   \nmodels_5[3].add(MaxPool2D(pool_size=2, strides=2))\nmodels_5[3].add(Flatten())\nmodels_5[3].add(Dense(units=256, activation='relu'))\nmodels_5[3].add(Dense(units=10, activation='softmax'))\nmodels_5[3].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\n# Model 5    \nmodels_5[4] = Sequential()\nmodels_5[4].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=X_train.shape[1:]))\nmodels_5[4].add(MaxPool2D(pool_size=3, strides=1))\nmodels_5[4].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\nmodels_5[4].add(MaxPool2D(pool_size=3, strides=1))\nmodels_5[4].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))   \nmodels_5[4].add(MaxPool2D(pool_size=2, strides=1))\nmodels_5[4].add(Flatten())\nmodels_5[4].add(Dense(units=256, activation='relu'))\nmodels_5[4].add(Dense(units=10, activation='softmax'))\nmodels_5[4].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_5)):\n    start_time = time.time()\n    historys_5[i] = models_5[i].fit(X_train, y_train, batch_size=512, epochs=EPOCH_SIZE,\n                                   validation_data=(X_val, y_val))\n    times_5[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_5)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_5[i].history['accuracy'][-1], historys_5[i].history['val_accuracy'][-1], times_5[i]))\n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aaand it looks like number 2 is the winner.\nLet's go ahead now and explore our next options.","metadata":{}},{"cell_type":"markdown","source":"### 2. Batch size\n\nThis will definitely affect the timing, but let's see how it also affects accuracy.\nWe have a total of 37800 training samples. Let's test what we get using the following number of batches:\n\n1. 3780\n2. 1024\n3. 512\n4. 256\n5. 128\n6. 64","metadata":{}},{"cell_type":"code","source":"models_2 = [0] * 6\ntimes_2 = [0] * 6\nhistorys_2 = [0] * 6\n\nbatch_size = [3780, 1024, 512, 256, 128, 64]\n\nfor i in range(len(models_2)):\n    models_2[i] = Sequential()\n    models_2[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=X_train.shape[1:]))\n    models_2[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_2[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\n    models_2[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_2[i].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\n    models_2[i].add(Flatten())\n    models_2[i].add(Dense(units=256, activation='relu'))\n    models_2[i].add(Dense(units=10, activation='softmax'))\n    models_2[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_2)):\n    start_time = time.time()\n    historys_2[i] = models_2[i].fit(X_train, y_train, batch_size=batch_size[i], epochs=EPOCH_SIZE,\n                                   validation_data=(X_val, y_val))\n    times_2[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_2)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_2[i].history['accuracy'][-1], historys_2[i].history['val_accuracy'][-1], times_2[i]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Perfect. Now we can go with the 4th model. So batch_size=256 works fine","metadata":{}},{"cell_type":"markdown","source":"### 3. Batch normalization\n\nLet's see what happens in the next scenarios:\n\n0. No BatchNorm\n1. BatchNorm before all layers\n2. BatchNorm only before conv layers\n3. BatchNorm only before dense layers","metadata":{}},{"cell_type":"code","source":"models_3 = [0] * 4\ntimes_3 = [0] * 4\nhistorys_3 = [0] * 4\n\nfor i in range(len(models_3)):\n    models_3[i] = Sequential()\n    models_3[i].add(Conv2D(filters=32, kernel_size=(3, 3), input_shape=X_train.shape[1:]))\n    if i in [1, 2]:\n        models_3[i].add(BatchNormalization())\n    models_3[i].add(Activation('relu'))\n    models_3[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_3[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    if i in [1, 2]:\n        models_3[i].add(BatchNormalization())\n    models_3[i].add(Activation('relu'))\n    models_3[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_3[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    if i in [1, 2]:\n        models_3[i].add(BatchNormalization())\n    models_3[i].add(Activation('relu'))\n    models_3[i].add(Flatten())\n    models_3[i].add(Dense(units=256))\n    if i in [1, 3]:\n        models_3[i].add(BatchNormalization())\n    models_3[i].add(Activation('relu'))\n    models_3[i].add(Dense(units=10))\n    if i in [1, 3]:\n        models_3[i].add(BatchNormalization())\n    models_3[i].add(Activation('softmax'))\n    models_3[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_3)):\n    start_time = time.time()\n    historys_3[i] = models_3[i].fit(X_train, y_train, batch_size=256, epochs=EPOCH_SIZE,\n                                   validation_data=(X_val, y_val))\n    times_3[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_3)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_3[i].history['accuracy'][-1], historys_3[i].history['val_accuracy'][-1], times_3[i]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The best one accuracy vs training ratio is model 3. Which means it is best to use batchnorm just for the dense layers.","metadata":{}},{"cell_type":"markdown","source":"### 4. Does Dropout help? If so, how many Dropout layers are optimal and what values do they have?\n\nI will try to see what happens in these different scenarios\n\n0. No Dropout\n1. Dropout of 0.5 before each conv and dense layer\n2. Dropout of 0.4 before every conv layer\n3. Dropout of 0.4 before each dense layer\n4. Dropout before last 2 conv layers and before first dense layer\n\nAfter Finding the best one out of these six, if the result is different than 0, I will also experiment with the dropout rate.","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 256\n\nmodels_4 = [0] * 5\ntimes_4 = [0] * 5\nhistorys_4 = [0] * 5\n\nfor i in range(len(models_4)):\n    models_4[i] = Sequential()\n    models_4[i].add(Input(shape=X_train.shape[1:]))\n    if i in [1, 2]:\n        models_4[i].add(Dropout(.5))\n    models_4[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_4[i].add(Activation('relu'))\n    models_4[i].add(MaxPool2D(pool_size=2, strides=2))\n    if i in [1, 2, 4]:\n        models_4[i].add(Dropout(.5))\n    models_4[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_4[i].add(Activation('relu'))\n    models_4[i].add(MaxPool2D(pool_size=2, strides=2))\n    if i in [1, 2, 4]:\n        models_4[i].add(Dropout(.5))\n    models_4[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_4[i].add(Activation('relu'))\n    models_4[i].add(Flatten())\n    if i in [1, 3, 4]:\n        models_4[i].add(Dropout(.5))\n    models_4[i].add(Dense(units=256))\n    models_4[i].add(BatchNormalization())\n    models_4[i].add(Activation('relu'))\n    if i in [1, 3]:\n        models_4[i].add(Dropout(.5))\n    models_4[i].add(Dense(units=10))\n    models_4[i].add(BatchNormalization())\n    models_4[i].add(Activation('softmax'))\n    models_4[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_4)):\n    start_time = time.time()\n    historys_4[i] = models_4[i].fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=EPOCH_SIZE,\n                                   validation_data=(X_val, y_val))\n    times_4[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_4)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_4[i].history['accuracy'][-1], historys_4[i].history['val_accuracy'][-1], times_4[i]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Due to the nature of the layer, it is normal that in 10 epochs it cannot achiever a lot of accuracy. Let's also plot the validation graphs to see what happens.\nWhatever the result, I would like to keep at least a little bit of dropout in our model, to avoid overfitting.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,6))\ncolors = ['r', 'g', 'b', 'y', 'm']\nnames = [('Model_' + str(x)) for x in range(0, 5)]\nfor i in range(len(models_4)):\n    plt.plot(historys_4[i].history['val_accuracy'], colors[i])\nplt.title('Model validation accuracy')\nplt.ylabel('Val accuracy')\nplt.legend(names, loc='upper left')\n\naxes = plt.gca()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I want to try again for models 2 and 3, but with 30 epochs. For each of them I will try 2 different dropout rates: 0.3 and 0.6\n# Afterwards I will experiment with the one I find the best with dropout rate\nmodels_41 = [0] * 4\ntimes_41 = [0] * 4\nhistorys_41 = [0] * 4\n\nfor i in range(len(models_41)):\n    models_41[i] = Sequential()\n    models_41[i].add(Input(shape=X_train.shape[1:]))\n    if i in [0, 1]:\n        models_41[i].add(Dropout(0.3 + (i%2)*0.3))\n    models_41[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_41[i].add(Activation('relu'))\n    models_41[i].add(MaxPool2D(pool_size=2, strides=2))\n    if i in [0, 1]:\n        models_41[i].add(Dropout(0.3 + (i%2)*0.3))\n    models_41[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_41[i].add(Activation('relu'))\n    models_41[i].add(MaxPool2D(pool_size=2, strides=2))\n    if i in [0, 1]:\n        models_41[i].add(Dropout(0.3 + (i%2)*0.3))\n    models_41[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_41[i].add(Activation('relu'))\n    models_41[i].add(Flatten())\n    if i in [2, 3]:\n        models_41[i].add(Dropout(0.3 + (i%2)*0.3))\n    models_41[i].add(Dense(units=256))\n    models_41[i].add(BatchNormalization())\n    models_41[i].add(Activation('relu'))\n    if i in [2, 3]:\n        models_41[i].add(Dropout(0.3 + (i%2)*0.3))\n    models_41[i].add(Dense(units=10))\n    models_41[i].add(BatchNormalization())\n    models_41[i].add(Activation('softmax'))\n    models_41[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_41)):\n    start_time = time.time()\n    historys_41[i] = models_41[i].fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=30,\n                                   validation_data=(X_val, y_val))\n    times_41[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,6))\ncolors_41 = ['r', 'g', 'b', 'y']\nnames_41 = [('Model_' + str(x)) for x in range(0, 4)]\nfor i in range(len(models_41)):\n    plt.plot(historys_41[i].history['val_accuracy'], colors_41[i])\nplt.title('Model validation accuracy')\nplt.ylabel('Val accuracy')\nplt.legend(names_41, loc='upper left')\nplt.ylim(0.96, 1)\naxes = plt.gca()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# It looks like model 3 performs the best. Let's try it with different dropout rates\nmodels_42 = [0] * 5\ntimes_42 = [0] * 5\nhistorys_42 = [0] * 5\n\nfor i in range(len(models_42)):\n    models_42[i] = Sequential()\n    models_42[i].add(Input(shape=X_train.shape[1:]))\n    models_42[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_42[i].add(Activation('relu'))\n    models_42[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_42[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_42[i].add(Activation('relu'))\n    models_42[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_42[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_42[i].add(Activation('relu'))\n    models_42[i].add(Flatten())\n    models_42[i].add(Dropout(0.4 + i*0.1))\n    models_42[i].add(Dense(units=256))\n    models_42[i].add(BatchNormalization())\n    models_42[i].add(Activation('relu'))\n    models_42[i].add(Dropout(0.4 + i*0.1))\n    models_42[i].add(Dense(units=10))\n    models_42[i].add(BatchNormalization())\n    models_42[i].add(Activation('softmax'))\n    models_42[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_42)):\n    start_time = time.time()\n    historys_42[i] = models_42[i].fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=30,\n                                   validation_data=(X_val, y_val))\n    times_42[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,6))\ncolors_42 = ['r', 'g', 'b', 'y']\nnames_42 = [('Model_' + str(x)) for x in range(0, 4)]\nfor i in range(len(models_42)-1):\n    plt.plot(historys_42[i].history['val_accuracy'], colors_42[i])\nplt.title('Model validation accuracy')\nplt.ylabel('Val accuracy')\nplt.legend(names_42, loc='upper left')\nplt.ylim(0.98, 1)\naxes = plt.gca()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historys_42","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like the model with dropout rate 0.4 was the best","metadata":{}},{"cell_type":"markdown","source":"### 6. How many and how big should the end dense layers be?\n\nWe will look at the next scenarios\n\n0. 1 Dense layer before softmax\n1. 2 Dense layers before softmax\n2. 3 Dense layers before softmax\n3. 4 Dense layers before softmax","metadata":{}},{"cell_type":"code","source":"# It looks like model 3 performs the best. Let's try it with different dropout rates\nmodels_50 = [0] * 4\ntimes_50 = [0] * 4\nhistorys_50 = [0] * 4\n\ndef add_dense_layer(model):\n    model.add(Dropout(0.4))\n    model.add(Dense(units=256))\n    model.add(BatchNormalization())\n    model.add(Activation('relu'))\n\nfor i in range(len(models_50)):\n    models_50[i] = Sequential()\n    models_50[i].add(Input(shape=X_train.shape[1:]))\n    models_50[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_50[i].add(Activation('relu'))\n    models_50[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_50[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_50[i].add(Activation('relu'))\n    models_50[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_50[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_50[i].add(Activation('relu'))\n    models_50[i].add(Flatten())\n    models_50[i].add(Dropout(0.4))\n    models_50[i].add(Dense(units=256))\n    models_50[i].add(BatchNormalization())\n    models_50[i].add(Activation('relu'))\n    for j in range(i):\n        add_dense_layer(models_50[i])\n    models_50[i].add(Dropout(0.4))\n    models_50[i].add(Dense(units=10))\n    models_50[i].add(BatchNormalization())\n    models_50[i].add(Activation('softmax'))\n    models_50[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_50)):\n    start_time = time.time()\n    historys_50[i] = models_50[i].fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=30,\n                                   validation_data=(X_val, y_val))\n    times_50[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_50)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_50[i].history['accuracy'][-1], historys_50[i].history['val_accuracy'][-1], times_50[i]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like 2 dense layers before softmax works best. Now let's try with different number of neurons combinations\n\n0. 256 for both\n1. 512 for both\n2. 256 and 512\n3. 512 and 1024\n4. 1024 and 512\n5. 512 and 256\n6. 256 and 128","metadata":{}},{"cell_type":"code","source":"models_51 = [0] * 7\ntimes_51 = [0] * 7\nhistorys_51 = [0] * 7\n\nunit_numbers = [[256, 256], [512, 512], [256, 512], [512, 1024], [1024, 512], [512, 256], [256, 128]]\n\ndef add_two_dense_layers(model, unit_number_list):\n    model.add(Dropout(0.4))\n    model.add(Dense(units=unit_number_list[0]))\n    model.add(BatchNormalization())\n    model.add(Activation('relu'))\n    model.add(Dropout(0.4))\n    model.add(Dense(units=unit_number_list[1]))\n    model.add(BatchNormalization())\n    model.add(Activation('relu'))\n\nfor i in range(len(models_51)):\n    models_51[i] = Sequential()\n    models_51[i].add(Input(shape=X_train.shape[1:]))\n    models_51[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_51[i].add(Activation('relu'))\n    models_51[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_51[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_51[i].add(Activation('relu'))\n    models_51[i].add(MaxPool2D(pool_size=2, strides=2))\n    models_51[i].add(Conv2D(filters=32, kernel_size=(3, 3)))\n    models_51[i].add(Activation('relu'))\n    models_51[i].add(Flatten())\n    add_two_dense_layers(models_51[i], unit_numbers[i])\n    models_51[i].add(Dropout(0.4))\n    models_51[i].add(Dense(units=10))\n    models_51[i].add(BatchNormalization())\n    models_51[i].add(Activation('softmax'))\n    models_51[i].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(models_51)):\n    start_time = time.time()\n    historys_51[i] = models_51[i].fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=30,\n                                   validation_data=(X_val, y_val))\n    times_51[i] = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(historys_51)):\n    print('CNN {0}: train accuracy = {1:.4f}, validation accuracy = {2:.4f}, time = {3:.4f} seconds'.format(\n        i, historys_51[i].history['accuracy'][-1], historys_51[i].history['val_accuracy'][-1], times_51[i]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,6))\ncolors_51 = ['r', 'g', 'b', 'y', 'c', 'k', 'm']\nnames_51 = [('Model_' + str(x)) for x in range(0, 7)]\nfor i in range(len(models_51)):\n    plt.plot(historys_51[i].history['val_accuracy'], colors_51[i])\nplt.title('Model validation accuracy')\nplt.ylabel('Val accuracy')\nplt.legend(names_51, loc='upper left')\nplt.ylim(0.987, 0.996)\naxes = plt.gca()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The one with 512 and 256 looks like it has the best validation accuracy and it also looks like it has an increasing validation accuracy, graphically speaking, so I will choose this architecture.\n\nSo we have reached the end of our model architecture choice. What I will do now is move on to number 7.\nI will train my model for a maximum of 100 epochs. 1 of the models will have the initial dataset that we were given, while for the other I will use data augmentation (simple stuff, shifts, zoom in, zoom out etc.).\nI will then evaluate the results for each of them on the competition.\nAfter that I will check again the model and see if it would make sense to add anything.","metadata":{}},{"cell_type":"markdown","source":"###  7. Does data augmentation help (time vs accuracy improvement)\nI will try to see what happens in these different scenarios\n\nAs stated above, I will train 2 models for a maximum 100 epochs. I will use a callback to stop the training if validation accuracy starts decreasing.\nOne of the model will be trained on the initial data set.\nThe other model will be trained on the augmented dataset.","metadata":{}},{"cell_type":"code","source":"model_7 = Sequential()\nmodel_7.add(Input(shape=X_train.shape[1:]))\nmodel_7.add(Conv2D(filters=32, kernel_size=(3, 3)))\nmodel_7.add(Activation('relu'))\nmodel_7.add(MaxPool2D(pool_size=2, strides=2))\nmodel_7.add(Conv2D(filters=32, kernel_size=(3, 3)))\nmodel_7.add(Activation('relu'))\nmodel_7.add(MaxPool2D(pool_size=2, strides=2))\nmodel_7.add(Conv2D(filters=32, kernel_size=(3, 3)))\nmodel_7.add(Activation('relu'))\nmodel_7.add(Flatten())\nadd_two_dense_layers(model_7, [512, 256])\nmodel_7.add(Dropout(0.4))\nmodel_7.add(Dense(units=10))\nmodel_7.add(BatchNormalization())\nmodel_7.add(Activation('softmax'))\nmodel_7.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.callbacks import EarlyStopping\nfrom keras.preprocessing.image import ImageDataGenerator\n\nearly_callback = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True, min_delta=0.0001)\n\ndatagen = ImageDataGenerator(\n    rotation_range=20,\n    zoom_range=0.1,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2)\n\ndatagen.fit(X_train)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train on non augmented data\n\nstart_time = time.time()\nhistorys_71 = model_7.fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=100, \n                          validation_data=(X_val, y_val), callbacks=[early_callback])\ntimes_71 = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_7.save('model_7_normal.h5')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = model_7.predict(X_test, workers=-1)\npredictions = np.argmax(predictions, axis=1)\n\npredictions_df = pd.DataFrame(predictions, index=(test.index.values+1), columns=['Label'])\npredictions_df.index.name = 'ImageId'\npredictions_df.to_csv('model_7_normal.csv')\npredictions_df.to_csv('submission.csv')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_7 = Sequential()\nmodel_7.add(Input(shape=X_train.shape[1:]))\nmodel_7.add(Conv2D(filters=32, kernel_size=(3, 3)))\nmodel_7.add(Activation('relu'))\nmodel_7.add(MaxPool2D(pool_size=2, strides=2))\nmodel_7.add(Conv2D(filters=32, kernel_size=(3, 3)))\nmodel_7.add(Activation('relu'))\nmodel_7.add(MaxPool2D(pool_size=2, strides=2))\nmodel_7.add(Conv2D(filters=32, kernel_size=(3, 3)))\nmodel_7.add(Activation('relu'))\nmodel_7.add(Flatten())\nadd_two_dense_layers(model_7, [512, 256])\nmodel_7.add(Dropout(0.4))\nmodel_7.add(Dense(units=10))\nmodel_7.add(BatchNormalization())\nmodel_7.add(Activation('softmax'))\nmodel_7.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\nstart_time = time.time()\nhistorys_72 = model_7.fit_generator(datagen.flow(X_train, y_train, batch_size=BATCH_SIZE), \n                                    epochs=100, \n                                    validation_data=(X_val, y_val), \n                                    callbacks=[early_callback], \n                                    steps_per_epoch=X_train.shape[0]//BATCH_SIZE)\ntimes_72 = time.time() - start_time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_7.save('model_7_augmented.h5')\n\npredictions = model_7.predict(X_test, workers=-1)\npredictions = np.argmax(predictions, axis=1)\n\npredictions_df = pd.DataFrame(predictions, index=(test.index.values+1), columns=['Label'])\npredictions_df.index.name = 'ImageId'\npredictions_df.to_csv('model_7_augmented.csv')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}