{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Adapted from MNIST competition\nThis is an implementation of Kaggle Grandmaster Chris Deotte's [notebook](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist) in the original MNIST competition. It has been slightly modified to fit the time requirements of this competition, and to take advantage of the holdout set here for further validation."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# LOAD LIBRARIES\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Kaggle's training images"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# LOAD THE DATA\ntrain = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('../input/Kannada-MNIST/test.csv')\nholdout = pd.read_csv('../input/Kannada-MNIST/Dig-MNIST.csv')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# PREPARE DATA FOR NEURAL NETWORK\nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"],axis = 1)\nX_train = X_train / 255.0\nX_test = test / 255.0\nX_train = X_train.values.reshape(-1,28,28,1)\n# prepare the test data set by removing the id\nX_test.drop(labels=['id'], axis=1, inplace=True)\nX_test = X_test.values.reshape(-1,28,28,1)\nY_train = to_categorical(Y_train, num_classes = 10)\n\nlen_test = len(X_test)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"X_holdout = pd.read_csv('../input/Kannada-MNIST/Dig-MNIST.csv')\n# prepare the holdout data set\ny_holdout = X_holdout['label']\nX_holdout.drop(labels=['label'], axis=1, inplace=True)\n\n#prepare holdout data\nX_holdout = X_holdout.astype('float32') / 255.\nX_holdout = X_holdout.values.reshape(X_holdout.shape[0], 28, 28, 1).astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### View some of the images"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n# PREVIEW IMAGES\nplt.figure(figsize=(15,4.5))\nfor i in range(30):  \n    plt.subplot(3, 10, i+1)\n    plt.imshow(X_train[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Generate more images!!\nby randomly rotating, scaling, and shifting Kaggle's training images."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# CREATE MORE IMAGES VIA DATA AUGMENTATION\ndatagen = ImageDataGenerator(\n        rotation_range=10,  \n        zoom_range = 0.10,  \n        width_shift_range=0.1, \n        height_shift_range=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# PREVIEW AUGMENTED IMAGES\nX_train3 = X_train[9,].reshape((1,28,28,1))\nY_train3 = Y_train[9,].reshape((1,10))\nplt.figure(figsize=(15,4.5))\nfor i in range(30):  \n    plt.subplot(3, 10, i+1)\n    X_train2, Y_train2 = datagen.flow(X_train3,Y_train3).next()\n    plt.imshow(X_train2[0].reshape((28,28)),cmap=plt.cm.binary)\n    plt.axis('off')\n    if i==9: X_train3 = X_train[11,].reshape((1,28,28,1))\n    if i==19: X_train3 = X_train[18,].reshape((1,28,28,1))\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build Multiple Convolutional Neural Networks!\nThe time limit on running gpu kernels in this competition means that we will run less than the 15 kernels the original kernel had in the MNIST competition."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 4\nmodel = [0] *nets\nfor j in range(nets):\n    model[j] = Sequential()\n\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = (28, 28, 1)))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.4))\n\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.4))\n\n    model[j].add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Flatten())\n    model[j].add(Dropout(0.4))\n    model[j].add(Dense(10, activation='softmax'))\n\n    # COMPILE WITH ADAM OPTIMIZER AND CROSS ENTROPY COST\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Architectural highlights\n![LeNet5](http://playagricola.com/Kaggle/LeNet5.png)\nThe CNNs in this kernel follow [LeNet5's][1] design (pictured above) with the following improvements:  \n* Two stacked 3x3 filters replace the single 5x5 filters. These become nonlinear 5x5 convolutions\n* A convolution with stride 2 replaces pooling layers. These become learnable pooling layers.\n* ReLU activation replaces sigmoid.\n* Batch normalization is added\n* Dropout is added\n* More feature maps (channels) are added\n* An ensemble of 15 CNNs with bagging is used  \n  \nExperiments [(here)][2] show that each of these changes improve classification accuracy.\n\n[1]:http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf\n[2]:https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist"},{"metadata":{},"cell_type":"markdown","source":"# Train Multiple CNNs"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# DECREASE LEARNING RATE EACH EPOCH\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)\n# TRAIN NETWORKS\nhistory = [0] * nets\nepochs = 45\nfor j in range(nets):\n    X_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.1)\n    history[j] = model[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64),\n        epochs = epochs, steps_per_epoch = X_train2.shape[0]//64,  \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        j+1,epochs,max(history[j].history['accuracy']),max(history[j].history['val_accuracy']) ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble Multiple CNN predictions and submit"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from sklearn.metrics import accuracy_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ENSEMBLE PREDICTIONS AND SUBMIT\nresults = np.zeros( (X_test.shape[0],10) )\nresults_holdout = np.zeros( (X_holdout.shape[0],10) )\nfor j in range(nets):\n    results = results + model[j].predict(X_test)\n    results_holdout = results_holdout + model[j].predict(X_holdout)\n\nresults_holdout = np.argmax(results_holdout,axis = 1)\nholdout_accuracy = accuracy_score(y_holdout, results_holdout)\nprint( \" Holdout Accuracy = %3.4f\"% (holdout_accuracy))\n\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"label\")\nsubmission = pd.concat([pd.Series(range(0,len_test),name = \"id\"),results],axis = 1)\nsubmission.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREVIEW PREDICTIONS\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_test[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"predict=%d\" % results[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Final Thoughts\nAs the original kernel shows, there is an upper limit beyond which it isn't possible to classify accurately. For the other MNIST competition this is around 99.8%. Validation scores for this kernel have been around that level but the private leaderboard scores have been lower. Further, the public leaderboard scores for this kernel are below some other publicly available kernels in this competition. It is possible that this is random fluctuation, however, the consistency with which other models have scored slightly higher makes this seem unlikely. \n\nThe code here is clearly a good solution to this competition and is based on the excellent results this same approach achieved in the other MNIST competition. I have created discussions [here](https://www.kaggle.com/c/Kannada-MNIST/discussion/111460) and [here](https://www.kaggle.com/c/Kannada-MNIST/discussion/111462) that explore why the accuracy on the public leaderboard may be lower than the validation accuracy for this competition."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}