{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"markdown","source":"## Accuracy=99.75% using 25 Million Training Images!!\n\nIt's amazing that convolutional neural networks can classify handwritten digits so accurately. In this kernel, we witness an ensemble of 15 CNNs classify Kaggle's MNIST digits after training on Kaggle's 42,000 images in \"train.csv\" plus 25 million more images created by rotating, scaling, and shifting Kaggle's images. Learning from 25,042,000 images, this ensemble of CNNs achieves 99.75% classification accuracy. This kernel uses ideas from the best published models found on the internet. Advanced techniques include data augmentation, nonlinear convolution layers, learnable pooling layers, ReLU activation, ensembling, bagging, decaying learning rates, dropout, batch normalization, and adam optimization. \n\n# 2천5백만개의 이미지로 학습시켜서 정확도 99.75% !! \n\n#### CNN 신경망이 손글씨로 된 숫자를 이렇게 정확하게 분류할 수 있다는 것은 놀라운 일입니다. 이 커널에서는 15개 CNN의 앙상블이 \"train.csv\"에서 Kaggle의 4만2000개 이미지와 회전, 스케일링, 스케일링으로 생성된 2500만 개의 이미지를 학습한 후 Kaggle의 MNIST 숫자를 분류하는 것을 볼 수 있습니다. 이 CNN 앙상블은 25,042,000개의 이미지로부터 학습하여 99.75%의 분류 정확도를 달성합니다. 이 커널은 인터넷에서 발견된 가장 잘 알려진 모델의 아이디어를 사용합니다. 고급 기법으로는 data augmentation, nonlinear convolution layers, learnable pooling layers, ReLU activation, ensembling, bagging, decaying learning rates, dropout, batch normalization, and adam optimization 등이 있습니다."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# LOAD LIBRARIES\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b794c1a43edfc86d217be7727e9e021b427476d3"},"cell_type":"markdown","source":"#### 위 코드는 해당 커널에서 필요한 라이브러리와 함수들을 선언하는 부분입니다. "},{"metadata":{"_uuid":"cd31c62c12088bfa6f2b26dcecc714182627c767"},"cell_type":"markdown","source":"# Load Kaggle's 42,000 training images\n# Kaggle의 4만2천개의 학습 이미지들을 로드하기"},{"metadata":{"_uuid":"71b0cb9a7c8a38f3e752ca7e2d6455a91714b49b"},"cell_type":"markdown","source":"#### 커널의 input 폴더 아래에 있는 train data와 test data를 로드합니다. "},{"metadata":{"trusted":true,"_uuid":"d71b3fa2b10620dc8870352fc18d4548f824a88a"},"cell_type":"code","source":"# LOAD THE DATA\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e823e7c835ae67823e45b67b1effbf8564384f5a"},"cell_type":"markdown","source":"#### 뉴럴 네트워크를 만들기 위한 데이터를 준비합니다."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b3c56055d1ba56d28d982f9647c33439c46753ff"},"cell_type":"code","source":"# PREPARE DATA FOR NEURAL NETWORK\nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"],axis = 1)\nX_train = X_train / 255.0\nX_test = test / 255.0\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\nY_train = to_categorical(Y_train, num_classes = 10) # convert class vectors to binary class matrices","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3a1feaaffd8f418578fa22a80e5392cee31c797"},"cell_type":"markdown","source":"#### 이미지를 출력해서 확인합니다. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b95ca2c1e71cb5457684eff3c35bb8d68b4a0f97"},"cell_type":"code","source":"import matplotlib.pyplot as plt\n# PREVIEW IMAGES\nplt.figure(figsize=(15,4.5))\nfor i in range(30):  \n    plt.subplot(3, 10, i+1)\n    plt.imshow(X_train[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cfcb89d7d2dab632986e80d9f68d194c3c1c9e9f"},"cell_type":"markdown","source":"## Generate 25 million more images!!\nby randomly rotating, scaling, and shifting Kaggle's 42,000 images.\n\n# 2천5백만개의 이미지 생성 !!\n\n#### Kaggle의 4만2천개의 사진을 랜덤하게 rotating하고, scaling하고 shifting해서 2천5백만개의 이미지를 생성합니다. \n\n## Data Augmentation\n\nData Augmentation 이란, 이름 해석에서도 알 수 있듯이 데이터를 확장시키는 것을 의미합니다. 일반적으로 Machine Learning 이나 Deep Learning 에서는 데이터 분류를 위해 모델을 학습시킬 때, 매우 많은 양의 데이터가 필요합니다. 하지만 대게 직접 데이터를 수집하는 것은 상당히 많은 비용이 드는 작업이며, 경우에 따라 데이터가 매우 부족한 경우도 있습니다. 따라서 이러한 문제를 해결하기 위해 Data Augmentation 기술이 필요합니다. Data Augmentation 은 적은 양의 데이터를 바탕으로 여러 조작을 통해 데이터의 양을 늘리는 작업으로, 대표적인 Augmentation 방법은 다음과 같다.\n\n* flip\n* crop\n* rotate\n* shift\n\nData Augmentation 을 적용하기 위해, 위에서 선언한 ImageDataGenerator 라는 함수를 활용해봅시다. "},{"metadata":{"trusted":true,"_uuid":"1e61e07d14b9b012748fdaac9eaf02e5263a475e"},"cell_type":"code","source":"# CREATE MORE IMAGES VIA DATA AUGMENTATION\ndatagen = ImageDataGenerator(\n        rotation_range=10,  \n        zoom_range = 0.10,  \n        width_shift_range=0.1, \n        height_shift_range=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60036fdc64932be35b148a448601e2e42e3f520e"},"cell_type":"markdown","source":"#### Augmentation을 적용하여 생성된 이미지를 확인합니다. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"fcf6daaae4424b95978856d7e75271c97b971c71"},"cell_type":"code","source":"# PREVIEW AUGMENTED IMAGES\nX_train3 = X_train[9,].reshape((1,28,28,1))\nY_train3 = Y_train[9,].reshape((1,10))\nplt.figure(figsize=(15,4.5))\nfor i in range(30):  \n    plt.subplot(3, 10, i+1)\n    X_train2, Y_train2 = datagen.flow(X_train3,Y_train3).next()\n    plt.imshow(X_train2[0].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\n    if i==9: X_train3 = X_train[11,].reshape((1,28,28,1))\n    if i==19: X_train3 = X_train[18,].reshape((1,28,28,1))\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ea116cd3688cb26ac79b9fecc7309a1aebf3b63"},"cell_type":"markdown","source":"## Build 15 Convolutional Neural Networks!\n# 15개의 CNN 구축 ! \n\n\n>#### Conv2D() 함수의 인자로\n    ...\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu'))\n    ...\n* 첫 번째 숫자인 32는 32개의 필터를 적용하겠다는 의미입니다.\n* kernel_size는 필터의 크기를 의미합니다.\n* input_shape는 (행, 열, 색상)을 의미합니다. 흑백의 경우 1의 값을 가집니다. RGB의 경우 3의 값을 가집니다.\n\n>#### Dropout()는 특정 노드에 학습이 지나치게 몰리는 것을 방지하기 위해 랜덤하게 일부 노드를 꺼주는 역할을 합니다. Dropout을 통해 과적합을 조금 더 효과적으로 회피할 수 있습니다.\n    ...\n    model[j].add(Dropout(0.4))\n    ...\n\n>#### 지금까지 작업했던 이미지는 2차원 배열인데, Flattern() 함수를 통해 1차원 배열로 바꿔줄 수 있습니다.\n    ...\n    model[j].add(Flatten())\n    ..."},{"metadata":{"trusted":true,"_uuid":"f6703f3f53c659e95579122755454899d842722a"},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 15\nmodel = [0] *nets\nfor j in range(nets):\n    model[j] = Sequential()\n\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = (28, 28, 1)))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.4))\n\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.4))\n\n    model[j].add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Flatten())\n    model[j].add(Dropout(0.4))\n    model[j].add(Dense(10, activation='softmax'))\n\n    # COMPILE WITH ADAM OPTIMIZER AND CROSS ENTROPY COST\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"843d2cb58465b81404c47559ceaf96c139ff82da"},"cell_type":"markdown","source":"## Architectural highlights\n\n![LeNet5](http://playagricola.com/Kaggle/LeNet5.png)\nThe CNNs in this kernel follow [LeNet5's][1] design (pictured above) with the following improvements:  \n* Two stacked 3x3 filters replace the single 5x5 filters. These become nonlinear 5x5 convolutions\n* A convolution with stride 2 replaces pooling layers. These become learnable pooling layers.\n* ReLU activation replaces sigmoid.\n* Batch normalization is added\n* Dropout is added\n* More feature maps (channels) are added\n* An ensemble of 15 CNNs with bagging is used  \n\nExperiments [(here)][2] show that each of these changes improve classification accuracy.\n\n# 구조적인 강조\n![LeNet5](http://playagricola.com/Kaggle/LeNet5.png)\n>### 이 커널에서 사용하는 CNN은 [LeNet5's][1] (위 그림과 같은)의 디자인을 따르되 아래 개선 사항을 포함합니다 : \n* 두 개의 적층 3x3 필터가 단일 5x5 필터를 대체합니다. 이것들은 비선형 5x5의 조합이 됩니다.\n* 2계단과의 공진이 풀링 레이어를 대체합니다. 학습 가능한 풀링 레이어가 됩니다.\n* ReLU 활성화는 Sigmoid를 대체합니다.\n* 드롭아웃 추가\n* 드롭다운이 추가되었습니다.\n* 피쳐 맵(채널)이 추가됨\n* 15개의 CNN ensemble과 bagging 사용\n  실험들 [여기][2]는 위의 각 개선 사항들이 분류 정확도를 향상시킨다는 것을 보여줍니다.\n\n\n[1]:http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf\n[2]:https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist"},{"metadata":{"_uuid":"e433661c7762b947c0fbfc4ad3f5e1d2e056312c"},"cell_type":"markdown","source":"## Train 15 CNNs\n\n# 15개 CNN 학습"},{"metadata":{"trusted":true,"_uuid":"9f1dd8a54aa0fab8530c0095f7d4c4b35984ea6d"},"cell_type":"code","source":"# DECREASE LEARNING RATE EACH EPOCH\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)\n# TRAIN NETWORKS\nhistory = [0] * nets\nepochs = 45\nfor j in range(nets):\n    X_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.1)\n    history[j] = model[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64),\n        epochs = epochs, steps_per_epoch = X_train2.shape[0]//64,  \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        j+1,epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28b78b6502d2d1c993555725383f3e30728fa5be"},"cell_type":"markdown","source":"# Ensemble 15 CNN predictions and submit"},{"metadata":{"trusted":true,"_uuid":"6e4e01ffe692c34c555bdbf5d606611f9a128b9c"},"cell_type":"code","source":"# ENSEMBLE PREDICTIONS AND SUBMIT\nresults = np.zeros( (X_test.shape[0],10) ) \nfor j in range(nets):\n    results = results + model[j].predict(X_test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"MNIST-CNN-ENSEMBLE.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4a9d7710a3b69c2b48bc1687e5b6a27a7076f40a","collapsed":true},"cell_type":"code","source":"# PREVIEW PREDICTIONS\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_test[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"predict=%d\" % results[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4d3e3246313e8bcb15c6b485540368d9297537b"},"cell_type":"markdown","source":"# Kaggle Result\n![result](http://playagricola.com/Kaggle/MNIST-result-DA4a.png)\nWow, its amazing that convolution neural networks can classify handwritten digits so accurately; 99.75% is as good as a human can classify!! This ensemble of 15 CNNs was trained with Kaggle's \"train.csv\" 42,000 images plus 25 million more images created by rotating, scaling, and shifting Kaggle's \"train.csv\" images."},{"metadata":{"_uuid":"372eaad5737aeef94f084c9c3317f6537db2cf2f"},"cell_type":"markdown","source":"# How much more accuracy is possible?\nNot much. Here are the best published MNIST classifiers found on the internet:\n* 99.79% [Regularization of Neural Networks using DropConnect, 2013][1]\n* 99.77% [Multi-column Deep Neural Networks for Image Classification, 2012][2]\n* 99.77% [APAC: Augmented PAttern Classification with Neural Networks, 2015][3]\n* 99.76% [Batch-normalized Maxout Network in Network, 2015][4]\n* **99.75% [This Kaggle published kernel, 2018][12]**\n* 99.73% [Convolutional Neural Network Committees, 2011][13]\n* 99.71% [Generalized Pooling Functions in Convolutional Neural Networks, 2016][5]\n* More examples: [here][7], [here][8], and [here][9]  \n  \nOn Kaggle's website, there are no published kernels more accurate than 99.70% besides the one you're reading. The few you will find posted were trained on the full original MNIST dataset (of 70,000 images) which contains known labels for Kaggle's unknown \"test.csv\" images so those models aren't actually that accurate. For example, [one kernel achieves 100% accuracy][10] training on the original MNIST dataset.  \n  \nBelow is a annotated histogram of Kaggle submission scores. Each bar has range 0.1%. There are spikes at 99.1% and 99.6% accuracy corresponding with using convolutional neural networks. Frequency count decreases as scores exceed 99.69%, hitting a low at 99.8% which is just past the highest possible accuracy. Then frequency count spikes again at accuracies of 99.9% and 100.0% corresponding to mistakenly training with the full original MNIST dataset.  \n  \n![hist](http://playagricola.com/Kaggle/KaggleMNISThist3.png)\n\n[1]:https://cs.nyu.edu/~wanli/dropc/dropc.pdf\n[2]:http://people.idsia.ch/~ciresan/data/cvpr2012.pdf\n[3]:https://arxiv.org/abs/1505.03229\n[4]:https://arxiv.org/abs/1511.02583\n[5]:https://arxiv.org/abs/1509.08985\n[7]:http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html\n[8]:http://yann.lecun.com/exdb/mnist/\n[9]:https://en.wikipedia.org/wiki/MNIST_database\n[10]:https://www.kaggle.com/cdeotte/mnist-perfect-100-using-knn/\n[12]:https://www.kaggle.com/cdeotte/35-million-images-0-99757-mnist\n[13]:http://people.idsia.ch/~ciresan/data/icdar2011a.pdf\n[14]:http://www.mva-org.jp/Proceedings/2015USB/papers/14-21.pdf"},{"metadata":{"_uuid":"1381453d438dbfa2ef72b50f2ba23ea0622078ac"},"cell_type":"markdown","source":"# How well can a human classify?\nTake the following quiz. Here are 50 of the most difficult images from Kaggle's \"test.csv\". For each image, write down a guess as to what digit it is. Then click the link below to see the correct answers. Hint: Nothing on the bottom row is what it seems and the top 4 rows contain 9 different digits!!  Good luck!  \n  \n    \n![quiz](http://playagricola.com/Kaggle/unknown.png)  \n  \n  \nClick [here][1] for the answers. The ambiguity and/or mislabeling of certain images is why classifiers cannot achieve accuracy greater than 99.8%.\n\n[1]:http://playagricola.com/Kaggle/answers.png"},{"metadata":{"_uuid":"1d55f8c054432beabeed62024a998234c7cdb7b6"},"cell_type":"markdown","source":"# Credits\nThe code here was inspired by the following outstanding Kaggle kernels (in addition to the publications above).\n\n* [Yassine Ghouzam][1] - [Introduction to CNN Keras - 0.997 (top 6%)][2]\n* [Peter Grenholm][5] - [Welcome to deep learning (CNN 99%)][6]\n* [Ding Li][3] - [Digits Recognition With CNN Keras][4]\n* [Aditya Soni][7] - [MNIST with Keras for Beginners(.99457)][8]\n\n[1]:https://www.kaggle.com/yassineghouzam\n[2]:https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n[3]:https://www.kaggle.com/dingli\n[4]:https://www.kaggle.com/dingli/digits-recognition-with-cnn-keras\n[5]:https://www.kaggle.com/toregil\n[6]:https://www.kaggle.com/toregil/welcome-to-deep-learning-cnn-99/\n[8]:https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457/\n[7]:https://www.kaggle.com/adityaecdrid"},{"metadata":{"_uuid":"e716712230dda4fc08bc4705cfbbb3c9a9942228"},"cell_type":"markdown","source":"# CNN Performance\nHow can we evaluate the performance of a neural network? A trained neural network performs differently each time you train it since the weights are randomly initialized. Therefore, to assess a neural network's performance, we must train it many times and take an average of accuracy. The ensemble in this notebook was trained and evaluated 100 times!! (on the original MNIST dataset with 60k/10k split using the code template [here][1] on GitHub.) Below is a histogram of its accuracy.  \n  \nThe maximum accuracy of an individual CNN was 99.81% with average accuracy 99.641% and standard deviation 0.047. The maximum accuracy of an ensemble of fifteen CNNs was 99.79% with average accuracy 99.745% and standard deviation 0.020.  \n  \n![hist](http://playagricola.com/Kaggle/histBoth5.png)\n\n## Data augmentation hyper-parameters\nTo determine the best hyper-parameters for data augmentation, grid search was used. Below is the accuracy of an ensemble (of 15 CNNs) with various data augmentation settings. The columns are `rotation` and `zoom`. The rows are `w_shift` and `h_shift`. For example: row 2, column 4 is `r = 15, z = 0.15, w = 0.1, h = 0.1`. Each cell is the average of 6 trials:  \n  \n            0      5     10      15     20     25     30  \n    0     99.70  99.70  99.70  99.70  99.69  99.65  99.62\n    0.1   99.73  99.73  99.75  99.75  99.72  99.67  99.64 \n    0.2                 99.72  99.72\n\nBelow is the accuracy of a single CNN with various data augmentation settings. Each cell is the average of 30 trials.  \n  \n            0      5     10      15     20     25     30  \n    0     99.57  99.58  99.62  99.62  99.62  99.57  99.52\n    0.1   99.62  99.63  99.65  99.65  99.63  99.58  99.52  \n    0.2                 99.62  99.62\n  \nLastly, I calculated the variance of the MNIST training images. The average center in pixels = (14.9, 15.2). The standard deviation of centers in pixels = (0.99, 1.34). That means that a setting of `w_shift = 0.07` together with `h_shift = 0.09` contains 95% of the centers. Similar analysis shows that a setting of `rotation_range = 13` together with `zoom_range = 0.13` contains 95% of the images.   \n  \nBased on this analysis, the settings of `rotation_range = 10, zoom_range = 0.10, w_shift = 0.1, and h_shift = 0.1` were chosen.\n\n[1]:https://github.com/cdeotte/MNIST-CNN-99.75"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}