{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Accuracy=99.75% using 25 Million Training Images!!\nIt's amazing that convolutional neural networks can classify handwritten digits so accurately. In this kernel, we witness an ensemble of 15 CNNs classify Kaggle's MNIST digits after training on Kaggle's 42,000 images in \"train.csv\" plus 25 million more images created by rotating, scaling, and shifting Kaggle's images. Learning from 25,042,000 images, this ensemble of CNNs achieves 99.75% classification accuracy. This kernel uses ideas from the best published models found on the internet. Advanced techniques include data augmentation, nonlinear convolution layers, learnable pooling layers, ReLU activation, ensembling, bagging, decaying learning rates, dropout, batch normalization, and adam optimization. ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true}},{"cell_type":"code","source":"# LOAD LIBRARIES\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-29T19:07:05.424855Z","iopub.execute_input":"2021-08-29T19:07:05.42508Z","iopub.status.idle":"2021-08-29T19:07:06.405244Z","shell.execute_reply.started":"2021-08-29T19:07:05.425025Z","shell.execute_reply":"2021-08-29T19:07:06.404495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Kaggle's 42,000 training images","metadata":{"_uuid":"cd31c62c12088bfa6f2b26dcecc714182627c767"}},{"cell_type":"code","source":"# LOAD THE DATA\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","metadata":{"_uuid":"d71b3fa2b10620dc8870352fc18d4548f824a88a","execution":{"iopub.status.busy":"2021-08-29T19:07:11.612433Z","iopub.execute_input":"2021-08-29T19:07:11.612724Z","iopub.status.idle":"2021-08-29T19:07:18.409471Z","shell.execute_reply.started":"2021-08-29T19:07:11.612664Z","shell.execute_reply":"2021-08-29T19:07:18.408738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PREPARE DATA FOR NEURAL NETWORK\nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"],axis = 1)\nX_train = X_train / 255.0\nX_test = test / 255.0\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\nY_train = to_categorical(Y_train, num_classes = 10)","metadata":{"_kg_hide-input":true,"_uuid":"b3c56055d1ba56d28d982f9647c33439c46753ff","execution":{"iopub.status.busy":"2021-08-29T19:07:25.457009Z","iopub.execute_input":"2021-08-29T19:07:25.457303Z","iopub.status.idle":"2021-08-29T19:07:26.187839Z","shell.execute_reply.started":"2021-08-29T19:07:25.457248Z","shell.execute_reply":"2021-08-29T19:07:26.186775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n# PREVIEW IMAGES\nplt.figure(figsize=(15,4.5))\nfor i in range(30):  \n    plt.subplot(3, 10, i+1)\n    plt.imshow(X_train[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","metadata":{"_kg_hide-input":true,"_uuid":"b95ca2c1e71cb5457684eff3c35bb8d68b4a0f97","execution":{"iopub.status.busy":"2021-08-29T19:07:28.931375Z","iopub.execute_input":"2021-08-29T19:07:28.931681Z","iopub.status.idle":"2021-08-29T19:07:30.013936Z","shell.execute_reply.started":"2021-08-29T19:07:28.931631Z","shell.execute_reply":"2021-08-29T19:07:30.012975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate 25 million more images!!\nby randomly rotating, scaling, and shifting Kaggle's 42,000 images.","metadata":{"_uuid":"cfcb89d7d2dab632986e80d9f68d194c3c1c9e9f"}},{"cell_type":"code","source":"# CREATE MORE IMAGES VIA DATA AUGMENTATION\ndatagen = ImageDataGenerator(\n        rotation_range=10,  \n        zoom_range = 0.10,  \n        width_shift_range=0.1, \n        height_shift_range=0.1)","metadata":{"_uuid":"1e61e07d14b9b012748fdaac9eaf02e5263a475e","execution":{"iopub.status.busy":"2021-08-29T19:07:33.129278Z","iopub.execute_input":"2021-08-29T19:07:33.129772Z","iopub.status.idle":"2021-08-29T19:07:33.139004Z","shell.execute_reply.started":"2021-08-29T19:07:33.129571Z","shell.execute_reply":"2021-08-29T19:07:33.13814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PREVIEW AUGMENTED IMAGES\nX_train3 = X_train[9,].reshape((1,28,28,1))\nY_train3 = Y_train[9,].reshape((1,10))\nplt.figure(figsize=(15,4.5))\nfor i in range(30):  \n    plt.subplot(3, 10, i+1)\n    X_train2, Y_train2 = datagen.flow(X_train3,Y_train3).next()\n    plt.imshow(X_train2[0].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\n    if i==9: X_train3 = X_train[11,].reshape((1,28,28,1))\n    if i==19: X_train3 = X_train[18,].reshape((1,28,28,1))\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","metadata":{"_kg_hide-input":true,"_uuid":"fcf6daaae4424b95978856d7e75271c97b971c71","execution":{"iopub.status.busy":"2021-08-29T19:07:35.766646Z","iopub.execute_input":"2021-08-29T19:07:35.766934Z","iopub.status.idle":"2021-08-29T19:07:36.936847Z","shell.execute_reply.started":"2021-08-29T19:07:35.766878Z","shell.execute_reply":"2021-08-29T19:07:36.936027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build 15 Convolutional Neural Networks!","metadata":{"_uuid":"9ea116cd3688cb26ac79b9fecc7309a1aebf3b63"}},{"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 15\nmodel = [0] *nets\nfor j in range(nets):\n    model[j] = Sequential()\n\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = (28, 28, 1)))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.4))\n\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.4))\n\n    model[j].add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Flatten())\n    model[j].add(Dropout(0.4))\n    model[j].add(Dense(10, activation='softmax'))\n\n    # COMPILE WITH ADAM OPTIMIZER AND CROSS ENTROPY COST\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","metadata":{"_uuid":"f6703f3f53c659e95579122755454899d842722a","execution":{"iopub.status.busy":"2021-08-29T19:07:42.8896Z","iopub.execute_input":"2021-08-29T19:07:42.889884Z","iopub.status.idle":"2021-08-29T19:07:56.725833Z","shell.execute_reply.started":"2021-08-29T19:07:42.889835Z","shell.execute_reply":"2021-08-29T19:07:56.72498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Architectural highlights\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/2020/LeNet5.png)\n\nThe CNNs in this kernel follow [LeNet5's][1] design (pictured above) with the following improvements:  \n* Two stacked 3x3 filters replace the single 5x5 filters. These become nonlinear 5x5 convolutions\n* A convolution with stride 2 replaces pooling layers. These become learnable pooling layers.\n* ReLU activation replaces sigmoid.\n* Batch normalization is added\n* Dropout is added\n* More feature maps (channels) are added\n* An ensemble of 15 CNNs with bagging is used  \n  \nExperiments [(here)][2] show that each of these changes improve classification accuracy.\n\n[1]:http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf\n[2]:https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist","metadata":{"_uuid":"843d2cb58465b81404c47559ceaf96c139ff82da"}},{"cell_type":"markdown","source":"# Train 15 CNNs","metadata":{"_uuid":"e433661c7762b947c0fbfc4ad3f5e1d2e056312c"}},{"cell_type":"code","source":"# DECREASE LEARNING RATE EACH EPOCH\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)\n# TRAIN NETWORKS\nhistory = [0] * nets\nepochs = 45\nfor j in range(nets):\n    X_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.1)\n    history[j] = model[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64),\n        epochs = epochs, steps_per_epoch = X_train2.shape[0]//64,  \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        j+1,epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","metadata":{"_uuid":"9f1dd8a54aa0fab8530c0095f7d4c4b35984ea6d","execution":{"iopub.status.busy":"2021-08-29T19:09:14.65437Z","iopub.execute_input":"2021-08-29T19:09:14.654661Z","iopub.status.idle":"2021-08-29T19:19:33.234441Z","shell.execute_reply.started":"2021-08-29T19:09:14.654605Z","shell.execute_reply":"2021-08-29T19:19:33.23336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensemble 15 CNN predictions and submit","metadata":{"_uuid":"28b78b6502d2d1c993555725383f3e30728fa5be"}},{"cell_type":"code","source":"# ENSEMBLE PREDICTIONS AND SUBMIT\nresults = np.zeros( (X_test.shape[0],10) ) \nfor j in range(nets):\n    results = results + model[j].predict(X_test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"MNIST-CNN-ENSEMBLE.csv\",index=False)","metadata":{"_uuid":"6e4e01ffe692c34c555bdbf5d606611f9a128b9c","execution":{"iopub.status.busy":"2021-08-29T19:19:38.958421Z","iopub.execute_input":"2021-08-29T19:19:38.958707Z","iopub.status.idle":"2021-08-29T19:19:53.278606Z","shell.execute_reply.started":"2021-08-29T19:19:38.95866Z","shell.execute_reply":"2021-08-29T19:19:53.27736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PREVIEW PREDICTIONS\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_test[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"predict=%d\" % results[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","metadata":{"_kg_hide-input":true,"_uuid":"4a9d7710a3b69c2b48bc1687e5b6a27a7076f40a","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle Result\n![result](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/2020/MNIST-result-DA4a.png)\n\nWow, its amazing that convolution neural networks can classify handwritten digits so accurately; 99.75% is as good as a human can classify!! This ensemble of 15 CNNs was trained with Kaggle's \"train.csv\" 42,000 images plus 25 million more images created by rotating, scaling, and shifting Kaggle's \"train.csv\" images.","metadata":{"_uuid":"b4d3e3246313e8bcb15c6b485540368d9297537b"}},{"cell_type":"markdown","source":"# How much more accuracy is possible?\nNot much. Here are the best published MNIST classifiers found on the internet:\n* 99.79% [Regularization of Neural Networks using DropConnect, 2013][1]\n* 99.77% [Multi-column Deep Neural Networks for Image Classification, 2012][2]\n* 99.77% [APAC: Augmented PAttern Classification with Neural Networks, 2015][3]\n* 99.76% [Batch-normalized Maxout Network in Network, 2015][4]\n* **99.75% [This Kaggle published kernel, 2018][12]**\n* 99.73% [Convolutional Neural Network Committees, 2011][13]\n* 99.71% [Generalized Pooling Functions in Convolutional Neural Networks, 2016][5]\n* More examples: [here][7], [here][8], and [here][9]  \n  \nOn Kaggle's website, there are no published kernels more accurate than 99.70% besides the one you're reading. The few you will find posted were trained on the full original MNIST dataset (of 70,000 images) which contains known labels for Kaggle's unknown \"test.csv\" images so those models aren't actually that accurate. For example, [one kernel achieves 100% accuracy][10] training on the original MNIST dataset.  \n  \nBelow is a annotated histogram of Kaggle submission scores. Each bar has range 0.1%. There are spikes at 99.1% and 99.6% accuracy corresponding with using convolutional neural networks. Frequency count decreases as scores exceed 99.69%, hitting a low at 99.8% which is just past the highest possible accuracy. Then frequency count spikes again at accuracies of 99.9% and 100.0% corresponding to mistakenly training with the full original MNIST dataset.  \n  \n![hist](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/2020/KaggleMNISThist3.png)\n\n[1]:https://cs.nyu.edu/~wanli/dropc/dropc.pdf\n[2]:http://people.idsia.ch/~ciresan/data/cvpr2012.pdf\n[3]:https://arxiv.org/abs/1505.03229\n[4]:https://arxiv.org/abs/1511.02583\n[5]:https://arxiv.org/abs/1509.08985\n[7]:http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html\n[8]:http://yann.lecun.com/exdb/mnist/\n[9]:https://en.wikipedia.org/wiki/MNIST_database\n[10]:https://www.kaggle.com/cdeotte/mnist-perfect-100-using-knn/\n[12]:https://www.kaggle.com/cdeotte/35-million-images-0-99757-mnist\n[13]:http://people.idsia.ch/~ciresan/data/icdar2011a.pdf\n[14]:http://www.mva-org.jp/Proceedings/2015USB/papers/14-21.pdf","metadata":{"_uuid":"372eaad5737aeef94f084c9c3317f6537db2cf2f"}},{"cell_type":"markdown","source":"# How well can a human classify?\nTake the following quiz. Here are 50 of the most difficult images from Kaggle's \"test.csv\". For each image, write down a guess as to what digit it is. Then click the link below to see the correct answers. Hint: Nothing on the bottom row is what it seems and the top 4 rows contain 9 different digits!!  Good luck!  \n  \n    \n![quiz](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/2020/unknown.png)  \n  \n  \nClick [here][1] for the answers. The ambiguity and/or mislabeling of certain images is why classifiers cannot achieve accuracy greater than 99.8%. Roughly speaking your overall accuracy on the entire MNIST test dataset would be equal to 100% minus 0.01% times the quantity you got wrong in this quiz.\n\n[1]:http://playagricola.com/Kaggle/answers.png","metadata":{"_uuid":"1381453d438dbfa2ef72b50f2ba23ea0622078ac"}},{"cell_type":"markdown","source":"# Credits\nThe code here was inspired by the following outstanding Kaggle kernels (in addition to the publications above).\n\n* [Yassine Ghouzam][1] - [Introduction to CNN Keras - 0.997 (top 6%)][2]\n* [Peter Grenholm][5] - [Welcome to deep learning (CNN 99%)][6]\n* [Ding Li][3] - [Digits Recognition With CNN Keras][4]\n* [Aditya Soni][7] - [MNIST with Keras for Beginners(.99457)][8]\n\n[1]:https://www.kaggle.com/yassineghouzam\n[2]:https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n[3]:https://www.kaggle.com/dingli\n[4]:https://www.kaggle.com/dingli/digits-recognition-with-cnn-keras\n[5]:https://www.kaggle.com/toregil\n[6]:https://www.kaggle.com/toregil/welcome-to-deep-learning-cnn-99/\n[8]:https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457/\n[7]:https://www.kaggle.com/adityaecdrid","metadata":{"_uuid":"1d55f8c054432beabeed62024a998234c7cdb7b6"}},{"cell_type":"markdown","source":"# CNN Performance\nHow can we evaluate the performance of a neural network? A trained neural network performs differently each time you train it since the weights are randomly initialized. Therefore, to assess a neural network's performance, we must train it many times and take an average of accuracy. The ensemble in this notebook was trained and evaluated 100 times!! (on the original MNIST dataset with 60k/10k split using the code template [here][1] on GitHub.) Below is a histogram of its accuracy.  \n  \nThe maximum accuracy of an individual CNN was 99.81% with average accuracy 99.641% and standard deviation 0.047. The maximum accuracy of an ensemble of fifteen CNNs was 99.79% with average accuracy 99.745% and standard deviation 0.020.  \n  \n![hist](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/2020/histBoth5.png)\n\n## Data augmentation hyper-parameters\nTo determine the best hyper-parameters for data augmentation, grid search was used. Below is the accuracy of an ensemble (of 15 CNNs) with various data augmentation settings. The columns are `rotation` and `zoom`. The rows are `w_shift` and `h_shift`. For example: row 2, column 4 is `r = 15, z = 0.15, w = 0.1, h = 0.1`. Each cell is the average of 6 trials:  \n  \n            0      5     10      15     20     25     30  \n    0     99.70  99.70  99.70  99.70  99.69  99.65  99.62\n    0.1   99.73  99.73  99.75  99.75  99.72  99.67  99.64 \n    0.2                 99.72  99.72\n\nBelow is the accuracy of a single CNN with various data augmentation settings. Each cell is the average of 30 trials.  \n  \n            0      5     10      15     20     25     30  \n    0     99.57  99.58  99.62  99.62  99.62  99.57  99.52\n    0.1   99.62  99.63  99.65  99.65  99.63  99.58  99.52  \n    0.2                 99.62  99.62\n  \nLastly, I calculated the variance of the MNIST training images. The average center in pixels = (14.9, 15.2). The standard deviation of centers in pixels = (0.99, 1.34). That means that a setting of `w_shift = 0.07` together with `h_shift = 0.09` contains 95% of the centers. Similar analysis shows that a setting of `rotation_range = 13` together with `zoom_range = 0.13` contains 95% of the images.   \n  \nBased on this analysis, the settings of `rotation_range = 10, zoom_range = 0.10, w_shift = 0.1, and h_shift = 0.1` were chosen.\n\n[1]:https://github.com/cdeotte/MNIST-CNN-99.75","metadata":{"_uuid":"e716712230dda4fc08bc4705cfbbb3c9a9942228"}}]}