{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"markdown","source":"# What is the best CNN architecture for MNIST?\nThere are so many choices for CNN architecture. How do we choose the best one? First we must define what **best** means. The best may be the simplest, or it may be the most efficient at producing accuracy while minimizing computational complexity. In this kernel, we will run experiments to find the most accurate and efficient CNN architecture for classifying MNIST handwritten digits.\n\nThe best known MNIST classifier found on the internet achieves 99.8% accuracy!! That's amazing. The best Kaggle kernel MNIST classifier achieves 99.75% [posted here][1]. This kernel demostrates the experiments used to determine that kernel's CNN architecture.\n[1]:https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Basic CNN structure\nA typical CNN design begins with feature extraction and finishes with classification. Feature extraction is performed by alternating convolution layers with subsambling layers. Classification is performed with dense layers followed by a final softmax layer. For image classification, this architecture performs better than an entirely fully connected feed forward neural network.\n![extract](http://playagricola.com/Kaggle/extract.png)\n"},{"metadata":{"_uuid":"f4d45939b302ddb1224c14779ff43fde5856343e"},"cell_type":"markdown","source":"# Notation in this Kaggle kernel\nThroughout this kernel, we'll use the following notation:\n * **24C5** means a convolution layer with 24 feature maps using a 5x5 filter and stride 1\n * **24C5S2** means a convolution layer with 24 feature maps using a 5x5 filter and stride 2 \n * **P2** means max pooling using 2x2 filter and stride 2\n * **256** means fully connected dense layer with 256 units \n \n# Keras API\n In Keras, to add a convolutional layer, you write  `model.add(Conv2D(filters=48,kernel_size=5,strides=1,padding='same',activation='relu')) `  \nWhat do all these terms mean?\n * **filters** is the number of desired feature maps.  \n * **kernel_size** is the size of the convolution kernel. A single number 5 means a 5x5 convolution.  \n * **strides** the new layer maps will have a size equal to the previous layer maps divided by strides. Leaving this blank results in strides=1.\n * **padding** is either 'same' or 'valid'. Leaving this blank results in padding='valid'. If padding is 'valid' then the size of the new layer maps is reduced by kernel_size-1. For example, if you perform a 5x5 convolution on a 28x28 image (map) with padding='valid', then the next layer has maps of size 24x24. If padding is 'same', then the size isn't reduced.\n * **activation** is applied during forward propagation. Leaving this blank results in no activation.\n   \n You can always view all of the layer sizes in your convolutional network by issuing the command `model.summary()`"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"437f141e31c4a4e14a8cd93942353eb1671365b9","collapsed":true},"cell_type":"code","source":"# LOAD LIBRARIES\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, AvgPool2D, BatchNormalization, Reshape\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nimport matplotlib.pyplot as plt\n\n# LOAD THE DATA\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\n\n# PREPARE DATA FOR NEURAL NETWORK\nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"],axis = 1)\nX_train = X_train / 255.0\nX_test = test / 255.0\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\nY_train = to_categorical(Y_train, num_classes = 10)\n\n# GLOBAL VARIABLES\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x, verbose=0)\nstyles=[':','-.','--','-',':','-.','--','-',':','-.','--','-']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe4337041127f8e40356909198b1c91ad6116253"},"cell_type":"markdown","source":"# 1. How many convolution-subsambling pairs?\nFirst question, how many pairs of convolution-subsampling should we use? For example, our network could have 1, 2, or 3:\n * 784 - **[24C5-P2]** - 256 - 10\n * 784 - **[24C5-P2] - [48C5-P2]** - 256 - 10\n * 784 - **[24C5-P2] - [48C5-P2] - [64C5-P2]** - 256 - 10  \n   \nIt's typical to increase the number of feature maps for each subsequent pair as shown here."},{"metadata":{"_uuid":"ecaf17096ebd89622602ade46a49b68f5e0ecbd5"},"cell_type":"markdown","source":"# Experiment 1\nLet's see whether one, two, or three pairs is best. We are not doing four pairs since the image will be reduced too small before then. The input image is 28x28. After one pair, it's 14x14. After two, it's 7x7. After three it's 4x4 (or 3x3 if we don't use padding='same'). It doesn't make sense to do a fourth convolution."},{"metadata":{"trusted":true,"_uuid":"c11a7bc48ad8cb7deaeed7edc517f4cb88ee941c","collapsed":true},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 3\nmodel = [0] *nets\n\nfor j in range(3):\n    model[j] = Sequential()\n    model[j].add(Conv2D(24,kernel_size=5,padding='same',activation='relu',\n            input_shape=(28,28,1)))\n    model[j].add(MaxPool2D())\n    if j>0:\n        model[j].add(Conv2D(48,kernel_size=5,padding='same',activation='relu'))\n        model[j].add(MaxPool2D())\n    if j>1:\n        model[j].add(Conv2D(64,kernel_size=5,padding='same',activation='relu'))\n        model[j].add(MaxPool2D(padding='same'))\n    model[j].add(Flatten())\n    model[j].add(Dense(256, activation='relu'))\n    model[j].add(Dense(10, activation='softmax'))\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e58d53aa20b07ef77878b9dbc5d61b5e613fea6","_kg_hide-input":true,"collapsed":true},"cell_type":"code","source":"# CREATE VALIDATION SET\nX_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.333)\n# TRAIN NETWORKS\nhistory = [0] * nets\nnames = [\"(C-P)x1\",\"(C-P)x2\",\"(C-P)x3\"]\nepochs = 20\nfor j in range(nets):\n    history[j] = model[j].fit(X_train2,Y_train2, batch_size=80, epochs = epochs, \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        names[j],epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"812c755eabd49f01d022d81e6c18e24c86bf541c","collapsed":true},"cell_type":"code","source":"# PLOT ACCURACIES\nplt.figure(figsize=(15,5))\nfor i in range(nets):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(names, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8165d70653b2a210231ca5148e2a39ad636909e6"},"cell_type":"markdown","source":"## Summary\nFrom the above experiment, it seems that 3 pairs of convolution-subsambling is slightly better than 2 pairs. However for efficiency, the improvement doesn't warrant the additional computional cost, so let's use 2."},{"metadata":{"_uuid":"811a6fc71dbfbc2b5c784c0028a4de10e95c5ddb"},"cell_type":"markdown","source":"# 2. How many feature maps?\nIn the previous experiement, we decided that two pairs is sufficient. How many feature maps should we include? For example, we could do\n * 784 - [**8**C5-P2] - [**16**C5-P2] - 256 - 10\n * 784 - [**16**C5-P2] - [**32**C5-P2] - 256 - 10\n * 784 - [**24**C5-P2] - [**48**C5-P2] - 256 - 10\n * 784 - [**32**C5-P2] - [**64**C5-P2] - 256 - 10\n * 784 - [**48**C5-P2] - [**96**C5-P2] - 256 - 10  \n * 784 - [**64**C5-P2] - [**128**C5-P2] - 256 - 10  "},{"metadata":{"_uuid":"19cee794a265cc6236172017b8742a9c332f9706"},"cell_type":"markdown","source":"# Experiment 2"},{"metadata":{"trusted":true,"_uuid":"7f137de935d61fd9a7a1e611152cda0958f251af","collapsed":true},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 6\nmodel = [0] *nets\nfor j in range(6):\n    model[j] = Sequential()\n    model[j].add(Conv2D(j*8+8,kernel_size=5,activation='relu',input_shape=(28,28,1)))\n    model[j].add(MaxPool2D())\n    model[j].add(Conv2D(j*16+16,kernel_size=5,activation='relu'))\n    model[j].add(MaxPool2D())\n    model[j].add(Flatten())\n    model[j].add(Dense(256, activation='relu'))\n    model[j].add(Dense(10, activation='softmax'))\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"47266925cecaada755e575fa1a072708b81094a4","collapsed":true},"cell_type":"code","source":"# CREATE VALIDATION SET\nX_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.333)\n# TRAIN NETWORKS\nhistory = [0] * nets\nnames = [\"8 maps\",\"16 maps\",\"24 maps\",\"32 maps\",\"48 maps\",\"64 maps\"]\nepochs = 20\nfor j in range(nets):\n    history[j] = model[j].fit(X_train2,Y_train2, batch_size=80, epochs = epochs, \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        names[j],epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"15da8cf30e3329bd8bcd273b184f7da5e2495d3e","collapsed":true},"cell_type":"code","source":"# PLOT ACCURACIES\nplt.figure(figsize=(15,5))\nfor i in range(nets):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(names, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b9dbd0b831671ed8f9ecf054e14c3a83027c391"},"cell_type":"markdown","source":"## Summary\nFrom the above experiement, it appears that 32 maps in the first convolutional layer and 64 maps in the second convolutional layer is the best. Architectures with more maps only perform slightly better and are not worth the additonal computation cost."},{"metadata":{"_uuid":"9512821135490c52ebf738fdc37394c768b3911b"},"cell_type":"markdown","source":"# 3. How large a dense layer?\nIn our previous experiment, we decided on 32 and 64 maps in our convolutional layers. How many dense units should we use? For example we could use\n * 784 - [32C5-P2] - [64C5-P2] - **0** - 10\n * 784 - [32C5-P2] - [64C5-P2] - **32** - 10\n * 784 - [32C5-P2] - [64C5-P2] - **64** - 10\n * 784 - [32C5-P2] - [64C5-P2] - **128** -10\n * 784 - [32C5-P2] - [64C5-P2] - **256** - 10\n * 784 - [32C5-P2] - [64C5-P2] - **512** -10\n * 784 - [32C5-P2] - [64C5-P2] - **1024** - 10\n * 784 - [32C5-P2] - [64C5-P2] - **2048** - 10"},{"metadata":{"_uuid":"01c032ab2297ace6d896bd5644b1be193bc8acca"},"cell_type":"markdown","source":"# Experiment 3"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d7a14f4981962ebefa5bc75c1f8edea5613648d4"},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 8\nmodel = [0] *nets\n\nfor j in range(8):\n    model[j] = Sequential()\n    model[j].add(Conv2D(32,kernel_size=5,activation='relu',input_shape=(28,28,1)))\n    model[j].add(MaxPool2D())\n    model[j].add(Conv2D(64,kernel_size=5,activation='relu'))\n    model[j].add(MaxPool2D())\n    model[j].add(Flatten())\n    if j>0:\n        model[j].add(Dense(2**(j+4), activation='relu'))\n    model[j].add(Dense(10, activation='softmax'))\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"50edbd024130a96d09aa24de6eb9bb2cf75162cd","collapsed":true},"cell_type":"code","source":"# CREATE VALIDATION SET\nX_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.333)\n# TRAIN NETWORKS\nhistory = [0] * nets\nnames = [\"0N\",\"32N\",\"64N\",\"128N\",\"256N\",\"512N\",\"1024N\",\"2048N\"]\nepochs = 20\nfor j in range(nets):\n    history[j] = model[j].fit(X_train2,Y_train2, batch_size=80, epochs = epochs, \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        names[j],epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"collapsed":true,"_uuid":"23d6fcd92c5c767bf322946c08ee5cf5436e39cf"},"cell_type":"code","source":"# PLOT ACCURACIES\nplt.figure(figsize=(15,5))\nfor i in range(nets):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(names, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba80e130bbb060945fd6180b89002e67f942ae9c"},"cell_type":"markdown","source":"## Summary\nFrom this experiment, it appears that 128 units is the best. Dense layers with more units only perform slightly better and are not worth the additional computational cost. (I also tested using two consecutive dense layers instead of one, but that showed no benefit over a single dense layer.) "},{"metadata":{"_uuid":"3bf4785e93a84d5e3d7cceb52adfcf8bb24f86ea"},"cell_type":"markdown","source":"# 4. How much dropout?\nDropout will prevent our network from overfitting thus helping our network generalize better. How much dropout should we add after each layer?\n * 0%, 10%, 20%, 30%, 40%, 50%, 60%, or 70%"},{"metadata":{"_uuid":"0d2738a7fdadd227cf64d03d7c5570088778c5f8"},"cell_type":"markdown","source":"# Experiment 4"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"7d03cf0f878e1c75338bd8620aa995c7d8048446"},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 8\nmodel = [0] *nets\n\nfor j in range(8):\n    model[j] = Sequential()\n    model[j].add(Conv2D(32,kernel_size=5,activation='relu',input_shape=(28,28,1)))\n    model[j].add(MaxPool2D())\n    model[j].add(Dropout(j*0.1))\n    model[j].add(Conv2D(64,kernel_size=5,activation='relu'))\n    model[j].add(MaxPool2D())\n    model[j].add(Dropout(j*0.1))\n    model[j].add(Flatten())\n    model[j].add(Dense(128, activation='relu'))\n    model[j].add(Dropout(j*0.1))\n    model[j].add(Dense(10, activation='softmax'))\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"collapsed":true,"_uuid":"46d1ddefaf6b2bc6480136ba0840d8923bab4526"},"cell_type":"code","source":"# CREATE VALIDATION SET\nX_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.333)\n# TRAIN NETWORKS\nhistory = [0] * nets\nnames = [\"D=0\",\"D=0.1\",\"D=0.2\",\"D=0.3\",\"D=0.4\",\"D=0.5\",\"D=0.6\",\"D=0.7\"]\nepochs = 30\nfor j in range(nets):\n    history[j] = model[j].fit(X_train2,Y_train2, batch_size=80, epochs = epochs, \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        names[j],epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"collapsed":true,"_uuid":"57790fc692a802e3b5fa1c2fc197d06590299c01"},"cell_type":"code","source":"# PLOT ACCURACIES\nplt.figure(figsize=(15,5))\nfor i in range(nets):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(names, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ccacc2aadcab70d6671b2d268cc0c91d2e48255"},"cell_type":"markdown","source":"## Summary\nFrom this experiment, it appears that 40% dropout is the best."},{"metadata":{"_uuid":"85d4b9a0b15675052a20a09edea12376286eda59"},"cell_type":"markdown","source":"# 5. Advanced features\nInstead of using one convolution layer of size 5x5, you can mimic 5x5 by using two consecutive 3x3 layers and it will be more nonlinear. Instead of using a max pooling layer, you can subsample by using a convolution layer with strides=2 and it will be learnable. Lastly, does batch normalization help? And does data augmentation help? Let's test all four of these\n * replace '32C5' with '32C3-32C3'  \n * replace 'P2' with '32C5S2'\n * add batch normalization\n * add data augmentation"},{"metadata":{"_kg_hide-input":true,"trusted":true,"collapsed":true,"_uuid":"fdfe0baaa31e65282ec778e5cbb0ab3c320af072"},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 5\nmodel = [0] *nets\n\nj=0\nmodel[j] = Sequential()\nmodel[j].add(Conv2D(32,kernel_size=5,activation='relu',input_shape=(28,28,1)))\nmodel[j].add(MaxPool2D())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Conv2D(64,kernel_size=5,activation='relu'))\nmodel[j].add(MaxPool2D())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Flatten())\nmodel[j].add(Dense(128, activation='relu'))\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Dense(10, activation='softmax'))\nmodel[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nj=1\nmodel[j] = Sequential()\nmodel[j].add(Conv2D(32,kernel_size=3,activation='relu',input_shape=(28,28,1)))\nmodel[j].add(Conv2D(32,kernel_size=3,activation='relu'))\nmodel[j].add(MaxPool2D())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Conv2D(64,kernel_size=3,activation='relu'))\nmodel[j].add(Conv2D(64,kernel_size=3,activation='relu'))\nmodel[j].add(MaxPool2D())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Flatten())\nmodel[j].add(Dense(128, activation='relu'))\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Dense(10, activation='softmax'))\nmodel[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nj=2\nmodel[j] = Sequential()\nmodel[j].add(Conv2D(32,kernel_size=5,activation='relu',input_shape=(28,28,1)))\nmodel[j].add(Conv2D(32,kernel_size=5,strides=2,padding='same',activation='relu'))\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Conv2D(64,kernel_size=5,activation='relu'))\nmodel[j].add(Conv2D(64,kernel_size=5,strides=2,padding='same',activation='relu'))\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Flatten())\nmodel[j].add(Dense(128, activation='relu'))\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Dense(10, activation='softmax'))\nmodel[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nj=3\nmodel[j] = Sequential()\nmodel[j].add(Conv2D(32,kernel_size=3,activation='relu',input_shape=(28,28,1)))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(32,kernel_size=3,activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(32,kernel_size=5,strides=2,padding='same',activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Conv2D(64,kernel_size=3,activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(64,kernel_size=3,activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(64,kernel_size=5,strides=2,padding='same',activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Flatten())\nmodel[j].add(Dense(128, activation='relu'))\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Dense(10, activation='softmax'))\nmodel[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"5cb6505352d68087e0c9c76e817f1d46505cf6c4"},"cell_type":"code","source":"j=4\nmodel[j] = Sequential()\n\nmodel[j].add(Conv2D(32,kernel_size=3,activation='relu',input_shape=(28,28,1)))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(32,kernel_size=3,activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(32,kernel_size=5,strides=2,padding='same',activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Dropout(0.4))\n\nmodel[j].add(Conv2D(64,kernel_size=3,activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(64,kernel_size=3,activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Conv2D(64,kernel_size=5,strides=2,padding='same',activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Dropout(0.4))\n\nmodel[j].add(Flatten())\nmodel[j].add(Dense(128, activation='relu'))\nmodel[j].add(BatchNormalization())\nmodel[j].add(Dropout(0.4))\nmodel[j].add(Dense(10, activation='softmax'))\n\nmodel[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9221a2e1e88540eef7acbe96b7b1fc4399cef4df","_kg_hide-input":true,"collapsed":true},"cell_type":"code","source":"# CREATE VALIDATION SET\nX_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.2)\n# TRAIN NETWORKS 1,2,3,4\nhistory = [0] * nets\nnames = [\"basic\",\"32C3-32C3\",\"32C5S2\",\"both+BN\",\"both+BN+DA\"]\nepochs = 35\nfor j in range(nets-1):\n    history[j] = model[j].fit(X_train2,Y_train2, batch_size=64, epochs = epochs,  \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        names[j],epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))\n    \n# CREATE MORE TRAINING IMAGES VIA DATA AUGMENTATION\ndatagen = ImageDataGenerator(\n        rotation_range=10,  \n        zoom_range = 0.1,  \n        width_shift_range=0.1, \n        height_shift_range=0.1)\n# TRAIN NETWORK 5\nj = nets-1\nhistory[j] = model[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64), \n    epochs = epochs, steps_per_epoch = X_train2.shape[0]//64,\n    validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\nprint(\"CNN {0}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n    names[j],epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"baa858ecc8022a5c08f0dd1106892b573a6ea919","_kg_hide-input":true,"collapsed":true},"cell_type":"code","source":"# PLOT ACCURACIES\nplt.figure(figsize=(15,5))\nfor i in range(nets):\n    plt.plot(history[i].history['val_acc'],linestyle=styles[i])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(names, loc='upper left')\naxes = plt.gca()\naxes.set_ylim([0.98,1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6858ea7ba2e65c62a081c04ea6e257775d0617f"},"cell_type":"markdown","source":"## Summary\nFrom this experiment, we see that each of the four advanced features improve accuracy. The first model uses no advanced features. The second uses only the double convolution layer trick. The third uses only the learnable subsambling layer trick. The third model uses both of those techniques plus batch normalization. The last model employs all three of those techniques plus data augmentation and achieves the best accuracy of 99.5%! (Or more if we train longer.) (Experiments determing the best data augmentation hyper-parameters are posted at the end of the kernel [here][1].)\n\n[1]:https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist"},{"metadata":{"_uuid":"1f73dc58ecf486b4899239dad072ead1e5ab7835"},"cell_type":"markdown","source":"# Conclusion\nTraining convolutional neural networks is a random process. This makes experiments difficult because each time you run the same experiment, you get different results. Therefore, you must run your experiments dozens of times and take an average. This kernel was run dozens of times and it seems that the best CNN architecture for classifying MNIST handwritten digits is 784 - [32C5-P2] - [64C5-P2] - 128 - 10 with 40% dropout. Afterward, more experiments show that replacing '32C5' with '32C3-32C3' improves accuracy. And replacing 'P2' with '32C5S2' improves accuracy. And adding batch normalizaiton and data augmentation improve the CNN. The best CNN found from the experiments here becomes\n *  **784 - [32C3-32C3-32C5S2] - [64C3-64C3-64C5S2] - 128 - 10** \n * with 40% dropout, batch normalization, and data augmentation added\n\nYou can see an ensemble of these CNNs achieve 99.75% accuracy by [clicking here][1]. Thanks for reading my kernel. If anyone has ideas to improve CNN architecture for classifying MNIST handwritten digits, please comment. Also I encourage people to fork this kernel, change the code, and test new CNN architectures of your design.\n\n[1]:https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist"},{"metadata":{"_uuid":"c190184093076f2c6381030e508c69fc57d2e070"},"cell_type":"markdown","source":"# Predict \"test.csv\" and submit to Kaggle"},{"metadata":{"trusted":true,"_uuid":"67f481323e08a7a945b9b8f3060493594cdd0be6","collapsed":true},"cell_type":"code","source":"# TRAIN OUR BEST NET MORE\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** (x+epochs))\nmodel[4].fit_generator(datagen.flow(X_train,Y_train, batch_size=64), epochs = 25, \n    steps_per_epoch = X_train.shape[0]//64, callbacks=[annealer], verbose=0)\n\n# SUBMIT TO KAGGLE\nresults = model[4].predict(X_test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"MNIST-CNN.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"778c475a8ec8fc9381cf19b8f2e2f52f7cf6a547"},"cell_type":"markdown","source":"![result](http://playagricola.com/Kaggle/HowToResult.png)"},{"metadata":{"_uuid":"8d4f29eaadc84fae347f86210e0b1cd22cc896a3"},"cell_type":"markdown","source":"# CNN Statistics\nThe best CNN in this notebook (model 4 of experiment 5) was trained and evaluated 1,500 times!! (on the original MNIST dataset with 60k-train/10k-test split using the code template [here][1] on GitHub.) Below is a histogram of its accuracy. The maximum accuracy was 99.81% with average accuracy 99.641% and standard deviation 0.047. (An ensemble of 15 CNNs was also evaluated and results are posted [here][2].)\n![hist](http://playagricola.com/Kaggle/histSingle2.png)\n\n[1]:https://github.com/cdeotte/MNIST-CNN-99.75\n[2]:https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}