{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"#Import packages\n\nimport seaborn as sns \nsns.set(font_scale=1.5)\nimport pandas as pd\nimport os \nimport keras\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import model_from_json\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nfrom keras.utils import to_categorical\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout,BatchNormalization\nfrom keras.callbacks import LearningRateScheduler\nfrom scipy.ndimage.filters import gaussian_filter\nfrom scipy.ndimage.interpolation import map_coordinates\nimport matplotlib.pyplot as plt\nimport os\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **CNN Ensemble!**\n\n**Purpose**\n\nThe purpose of this notebook is to share an ensemble method consisting of three convolutional neural networks for the Kannada Mnist challenge. Running this workbook as is achieves 98.60% accuracy on the leaderboard, and increasing the number of unique CNNs in the ensemble can achieve a score of 98.90%. \n\nThis is my first noteback and I'm looking to learn like everyone else so any and all feedback is welcomed! Also a quick shoutout to this great kernal https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist that inspired both the ensembling and architecture! \n\nAlong the way we will see\n1. A brief history of the original MNIST dataset\n2. Check out the Kannada dataset\n3. Data augmentations including elastic distortions\n4. A set of three convolutional neural networks \n5. Submit some predictions and hear how to improve accuracy"},{"metadata":{},"cell_type":"markdown","source":"**A brief history of digit classifcation**\nThankfully for us this competition is very similar to the original MNIST competition, meaning we have all the key learnings of that competition at our finger tips. A great survey of MNIST progress is provided here https://www.researchgate.net/publication/334957576_A_Survey_of_Handwritten_Character_Recognition_with_MNIST_and_EMNIST. Below we plot a key table from the report which shows the error rates for a range of very competitive methods. Not surprising we see the much loved CNN with data augmentation taking out all the top spots for lowest error rates, and that's what we'll focus on below!"},{"metadata":{"trusted":true},"cell_type":"code","source":"summaryResults = pd.read_csv('/kaggle/input/summarypaper/SummaryPaper.csv').sort_values(by = ['ErrorRate'])\nplt.figure(figsize=(18, 17))\nax = sns.barplot(x=\"ErrorRate\", y='Method',hue = 'Category', data=summaryResults, dodge = False)\nplt.legend(loc='upper right')\nplt.setp(ax.get_legend().get_texts(), fontsize='22') # for legend text\nplt.setp(ax.get_legend().get_title(), fontsize='32') # for legend title\nplt.xlabel(\"Error rate\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The Kannada Data Set**\nLet's get started and take a look at these images! Like the Mnist dataset we have 28x28 pixel images, all stored as a vector of length 784. Below we read them all in, reshape them and change the scale from 0 to 255 to be -0.5 to 0.5 and then look at a few samples!"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train_path = \"../input/Kannada-MNIST/train.csv\"\ntest_path = \"../input/Kannada-MNIST/test.csv\"\ntrain = pd.read_csv(train_path)\ntest = pd.read_csv(test_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"#Split the data up into labels and images, and also scale the images to be between -0.5 and 0.5.\ntrain_images = train.iloc[:,1:]/255 - 0.5\ntrain_labels = train.iloc[:,0]\ntest_images = test.iloc[:,1:]/255 - 0.5\ntest_id = test.iloc[:,0]\n\n#Since we're using CNNs we need everything shaped as an image - i.e a 28*28*1 matrix. \n#This represents pixel height, width and \"channel\". For greyscale images channel =1, for RGB channel =3. \nX_train=train_images.values.reshape(-1,28,28,1)\nX_test=test_images.values.reshape(-1,28,28,1)\nY_train=to_categorical(train_labels)\n\n##Let's plot a few sample images\nfig=plt.figure(figsize=(16, 16))\nfor i in range(1, 5):\n    img = X_train[i].squeeze()\n    fig.add_subplot(1, 5, i)\n    plt.imshow(img,cmap=plt.cm.binary)\n    plt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Image Augmentation** The image augmentation used is typical to many other kernals with affine transforms, that is shifts and rotations. An elastic deformation as described by Simard, Steinkraus, and Plattin (2003) in \"Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis\" is also included using code and suggested parameters from https://www.kaggle.com/babbler/mnist-data-augmentation-with-elastic-distortion as it sounded like an interesting addition! After running with/without the elastic deformation it seemed a higher score is achieved without, so in the actual image gen I comment out the call to it. "},{"metadata":{},"cell_type":"markdown","source":"Elastic deformation code:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def elastic_transform(image, alpha_range, sigma, random_state=None):\n    \"\"\"Elastic deformation of images as described in [Simard2003]_.\n    .. [Simard2003] Simard, Steinkraus and Platt, \"Best Practices for\n       Convolutional Neural Networks applied to Visual Document Analysis\", in\n       Proc. of the International Conference on Document Analysis and\n       Recognition, 2003.\n       \n   # Arguments\n       image: Numpy array with shape (height, width, channels). \n       alpha_range: Float for fixed value or [lower, upper] for random value from uniform distribution.\n           Controls intensity of deformation.\n       sigma: Float, sigma of gaussian filter that smooths the displacement fields.\n       random_state: `numpy.random.RandomState` object for generating displacement fields.\n    \"\"\"\n    import numpy as np\n    from scipy.ndimage.filters import gaussian_filter\n    from scipy.ndimage.interpolation import map_coordinates\n\n    if random_state is None:\n        random_state = np.random.RandomState(None)\n        \n    if np.isscalar(alpha_range):\n        alpha = alpha_range\n    else:\n        alpha = np.random.uniform(low=alpha_range[0], high=alpha_range[1])\n\n    shape = image.shape\n    dx = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma) * alpha\n    dy = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma) * alpha\n\n    x, y, z = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]), np.arange(shape[2]), indexing='ij')\n    indices = np.reshape(x+dx, (-1, 1)), np.reshape(y+dy, (-1, 1)), np.reshape(z, (-1, 1))\n\n    return map_coordinates(image, indices, order=1, mode='reflect').reshape(shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Image augmentation set up:"},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        rotation_range=10,  #This randomly rotates images\n        zoom_range = 0.10,  #This randomly zooms images\n        width_shift_range=0.1, #This randomly shifts images vertically\n        height_shift_range=0.1 #This randomly shifts images horizontally \n        #preprocessing_function=lambda x: elastic_transform(x, alpha_range=[8,10], sigma=3) #Defined above\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Build the three models!** \nThe architecture of the models and the ensembling process is taken from https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist exactly as is. Not very creative I know, but it looks like a lot of research was done in choosing this structure here so why mess with a good thing. "},{"metadata":{"trusted":true},"cell_type":"code","source":"nets=3\nmodel = [0] * nets\nfor j in range(nets):\n    model[j] = Sequential()\n\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = (28, 28, 1)))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.1))\n\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Dropout(0.15))\n\n    model[j].add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model[j].add(BatchNormalization())\n    model[j].add(Flatten())\n    model[j].add(Dropout(0.2))\n    model[j].add(Dense(10, activation='softmax'))\n\n    #Compile all the pieces of our model together\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nmodel[0].summary()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The models are trained using a decaying learning rate. The number of epochs was set to 30 in the interest of having time to build 3 nets in the competitions runtime. Better results can be achieved by increasing this number. There is also room to experiment with early stopping. "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"decay = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)\nepochs = 30\nbatch_size = 64\n\nhistory = [0] * nets\nfor j in range(nets):\n    \n    ##Train the models using different subsets of the training data\n    X_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.1)\n    history[j] = model[j].fit_generator(datagen.flow(X_train2, Y_train2, batch_size=batch_size),\\\n                                        epochs = epochs,steps_per_epoch = X_train2.shape[0]//64,\\\n                                        validation_data = (X_val2,Y_val2),callbacks = [decay], verbose = True)\n    ##Save the model output incase we want to reuse it later\n    model_json = model[j].to_json()\n    with open(\"model\"+str(j)+\".json\", \"w\") as json_file:\n        json_file.write(model_json)\n        # serialize weights to HDF5\n    model[j].save_weights(\"modelw\"+str(j)+\".h5\")\n    print(\"Saved model to disk\")\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's make some predictions!** \nThis is the easy bit - now we just let each of the models make it's predction and add them all together. We choose the highest score from the result as our final prediction. "},{"metadata":{"trusted":true},"cell_type":"code","source":"##Initialise a matrix with rows = number of test images and columns = number of classifications (10)\npredictions = np.zeros( (X_test.shape[0],10) ) \n\n##Loop through the different Neural networks to add together the predictions\nfor j in range(nets):\n    predictions = predictions + model[j].predict(X_test)\n    \ntest_labels = np.argmax(predictions, axis=1)\n\nsubmission = pd.DataFrame(test_labels)\nsubmission.index = test_id\nsubmission.columns = [\"label\"]\nsubmission.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Results** \n\nRunning this piece as it is resulted in a score of 98.6%. Like mentioned previously I've since used essentially the same code with an increase in both epochs and models and that has provided an accuracy of 98.9%. \n\nNote that since the competition only allows a certain amount of compute time for more complicated models I've found running multiple commits and saving the models as outputs works best. Then the models can be read back in for the final run. This process is outlined here https://jovianlin.io/saving-loading-keras-models/. \n\nIdeas for the future: \n* Changing the model architecture and transformation parameters. This architecture was chosen as it performed well on the original MNIST set, but might not be the best choice for this set. \n\nLet me know your thoughts and suggestions! "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}