{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Kannada MNIST CNN\n\nKannada is a language spoken predominantly by people of Karnataka in southwestern India. The language has roughly 45 million native speakers and is written using the Kannada script.\n\nThe goal of this competition is to use Machine Learning to correctly label hand-written digits written in the Kannada script. The Kannada dataset format is based on the dataset used in the original MNIST dataset where the objective was the same but the hand-written digits were in Arabic numerals.\n\n### If you find this code helpful please <span style=\"color:red\">upvote</span>"},{"metadata":{},"cell_type":"markdown","source":"## Initial Setup\n\nLoad the most widely used libraries numpy and pandas and check what input files are available to the notebook."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Libraries\n\nLoad other useful libraries particularly the ones that we need to build the Convolutional Neural Network (CNN). This is an effective type of model for dealing with image recognition. In this competition access to the internet is not permitted for the kernel. We will exhibit building a model from scratch rather than using a pre-trained model."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import datetime\nfrom keras.models import Sequential\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.decomposition import PCA\nfrom keras import regularizers\nfrom keras.layers import Input, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D\nfrom keras.layers import AveragePooling2D, Dropout, GlobalMaxPooling2D, GlobalAveragePooling2D\nfrom keras.models import Model\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.optimizers import Adam\nfrom keras.callbacks import ReduceLROnPlateau\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data\n\nFirstly let's read the data that the models will be working on."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(('Start: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\nprint(('Data Load and Preprocessing: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\nX_train = pd.read_csv('../input/Kannada-MNIST/train.csv')\nX_test = pd.read_csv('../input/Kannada-MNIST/test.csv')\nX_holdout = pd.read_csv('../input/Kannada-MNIST/Dig-MNIST.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Prep\nA few preprocessing steps to get the data ready for our Machine Learning model. (Note X_test has a column called 'id' which X_train and X_holdout do not have)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"id_train = X_train.index.values\ny_train = X_train['label']\ny_valid_pred = 0*y_train\n\nX_train.drop(labels=['label'], axis=1, inplace=True)\n\n# prepare the test data set by removing the id\nX_test.drop(labels=['id'], axis=1, inplace=True) #remove the column that is not in the X_train and X_holdout\n\n#prepare test data\nX_test = X_test.astype('float32') / 255.\nX_test = X_test.values.reshape(X_test.shape[0], 28, 28, 1).astype('float32')\n\n# prepare the holdout data set\ny_holdout = X_holdout['label']\nX_holdout.drop(labels=['label'], axis=1, inplace=True)\n\n#prepare holdout data\nX_holdout = X_holdout.astype('float32') / 255.\nX_holdout = X_holdout.values.reshape(X_holdout.shape[0], 28, 28, 1).astype('float32')\n\n\nmy_init = 'glorot_uniform'\nmy_activ = 'relu'\nmy_optimiser = 'adam'\nmy_epsilon = 1e-8\nnb_classes = 10","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Visualisation\n\nBelow is a representation of what the Kannada digits look like to us. We can see that they have been captured with low resolution images."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig,ax=plt.subplots(5,10)\nfor i in range(5):\n    for j in range(10):\n        ax[i][j].imshow(X_test[np.random.randint(0,X_test.shape[0]),:,:,0],cmap=plt.cm.binary)\n        ax[i][j].axis('off')\nplt.subplots_adjust(wspace=0, hspace=0)        \nfig.set_figwidth(16)\nfig.set_figheight(8)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Principal Component Analysis\n\nAbove we saw what the digits look like to humans. Here we see how ML models can go about learning to distinguish features. The PCA method seeks to obtain the optimal directions (or eigenvectors) that captures the most variance (spreads out the data points the most) whilst reducing the number of features. It may be informative (and cool) to visualise these directions and their associated eigenvalues. For the purposes of this notebook and for speed, I will invoke PCA to extract the top 30 eigenvalues from the digit dataset. So effectively we are reducing the original 28x28 image which is 784 features to just 30 features. We then visually compare the top eigenvalues i.e. those that explain the most variance, to some of the other smaller ones to see if we can glean any insights as follows:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Invoke SKlearn's PCA method\nn_components = 30\npca = PCA(n_components=n_components).fit(X_train.values)\n\neigenvalues = pca.components_.reshape(n_components, 28, 28)\n\n# Extracting the PCA components ( eignevalues )\n#eigenvalues = pca.components_.reshape(n_components, 28, 28)\neigenvalues = pca.components_","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"n_row = 4\nn_col = 7\n\n# Plot the first eignenvalues\nplt.figure(figsize=(13,12))\nfor i in list(range(n_row * n_col)):\n    offset =0\n    plt.subplot(n_row, n_col, i + 1)\n    plt.imshow(eigenvalues[i].reshape(28,28), cmap='jet')\n    title_text = 'Eigenvalue ' + str(i + 1)\n    plt.title(title_text, size=6.5)\n    plt.xticks(())\n    plt.yticks(())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### What the plots show us\n\nThe subplots above portray the top 30 optimal directions or principal component axes that the PCA method has decided to generate for our digit dataset. Of interest is when one compares the first component \"Eigenvalue 1\" to the 28th component \"Eigenvalue 28\", it is obvious that more complicated directions or components are being generated in the search to maximise variance in the new feature subspace. For the CNN below we will use the full 784 features but the illustration above gives a visualisation about how ML models can learn features on an image."},{"metadata":{},"cell_type":"markdown","source":"### Model Function\n\nDefine a function that returns the model. This will set the model architecture and will prevent the need to type this all out repeatedly for the validation and prediction stages.\n\nWe will build a CNN of several layers. This is not a simple NN but it certainly isn't complex either relative to pretrained models that can contain over 100 layers.\n\nThe model uses a softmax output to output the probabilities for each of the 10 digits"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def build_network(input_shape):    \n    model = Sequential()\n    # For an explanation on conv layers see http://cs231n.github.io/convolutional-networks/#conv\n    # For an explanation on pooling layers see http://cs231n.github.io/convolutional-networks/#pool\n    # By default the stride/subsample is 1 and there is no zero-padding.\n    # use padding=\"same\" if you want to preserve dimensions\n    \n    #model.add(ZeroPadding2D(padding=(2, 2), data_format=\"channels_last\", input_shape=input_shape))\n    model.add(Conv2D(filters=64, kernel_size=(5, 5), strides=(1, 1), input_shape=input_shape, padding=\"same\", activation='relu', kernel_initializer=my_init))\n    model.add(MaxPooling2D(pool_size=(2, 2), padding='valid'))    \n    model.add(Dropout(0.25))\n    model.add(Conv2D(filters=128, kernel_size=(5, 5), strides=(1, 1), padding=\"same\", activation='relu', kernel_initializer=my_init))\n    model.add(MaxPooling2D(pool_size=(2, 2), padding='valid'))    \n    model.add(Dropout(0.25))\n    model.add(Conv2D(filters=256, kernel_size=(3, 3), strides=(1, 1), padding=\"same\", activation='relu', kernel_initializer=my_init))\n    model.add(MaxPooling2D(pool_size=(2, 2), padding='valid'))    \n    model.add(Dropout(0.25))\n    \n    # Flatten the 3D output to 1D tensor for a fully connected layer to accept the input\n    model.add(Flatten())\n    \n    \n    #Fully Connected Layer\n    model.add(Dense(256, kernel_initializer=my_init))\n    model.add(Activation(my_activ))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.3)) #dropout is a type of regularisation. Regularisation helps to control overfitting\n    #Fully Connected Layer\n    model.add(Dense(128, kernel_initializer=my_init))\n    model.add(Activation(my_activ))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n    \n    #Output layer\n    model.add(Dense(nb_classes, activation='softmax')) \n\n    model.compile(optimizer=my_optimiser,\n        loss='categorical_crossentropy',\n        metrics=['accuracy'])\n    \n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Augmentation Settings\n\nCNN's can often produce better results if they are fed not just the original input images. Here we take the input image data and make some specific types of adjustments. This process is called augmentation. For example if an image is rotated 1 degree to the right in many cases it will still be identifiable to a human as the same object. Similarly a model can be trained to still recognise the image. Again if we as humans zoom in slightly to the image often it is still recognisable. Once again a model can be trained to still be able to recognise the image. All these slight variations often allow the model to be trained in a more robust way. There are many other types of ways to adjust the image. Not all of them will be used here but there are some notes in the code below to briefly outline the types of augmentations that can be done."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=12,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.25, # Randomly zoom image \n        shear_range=15, #move top of image along without moving the bottom or vice versa\n        width_shift_range=0.25,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.25,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False,  # randomly flip images\n        data_format=\"channels_last\" \n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Charting Function\n\nLater we will draw some charts. Here we set up the function we can call when we need to do that."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def PlotAcc(his, epoch):\n    plt.style.use(\"ggplot\")\n    plt.figure()\n    plt.plot(np.arange(1, epoch + 1), his.history[\"accuracy\"], label=\"train_accuracy\")\n    plt.plot(np.arange(1, epoch + 1), his.history[\"val_accuracy\"], label=\"val_accuracy\")\n    plt.title(\"Training and Validation Accuracy\")\n    plt.xlabel(\"Epoch #\")\n    plt.ylabel(\"Accuracy\")\n    plt.legend(loc=\"lower right\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Utility Function\n\nAnother function that we will use later."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def epoch_cv(df_cv_per_epoch_val_acc, fold_num):\n    #Find the best epoch by the best single and the best moving average\n    #Update the index to start at 1 with epoch 1 for the validation accuracy data\n    if fold_num==0:\n        df_cv_per_epoch_val_acc.index += 1 #needed so length of values matches length of index\n    df_cv_per_epoch_val_acc['mean_val_acc'] = df_cv_per_epoch_val_acc.mean(axis=1)\n    #Calculate an epoch moving average\n    num_epochs = df_cv_per_epoch_val_acc.shape[0]\n    \n    for i in range(1, num_epochs+1):\n        #print(i)\n        if i<moving_average_period+1:\n            df_cv_per_epoch_val_acc.at[i, 'moving_average'] = df_cv_per_epoch_val_acc.iloc[:i]['mean_val_acc'].mean()\n        else:\n            df_cv_per_epoch_val_acc.at[i, 'moving_average'] = df_cv_per_epoch_val_acc.iloc[i-moving_average_period:i]['mean_val_acc'].mean()\n    \n    #Locate the Best Epoch Number (not the value but the epoch number) by the Mean per epoch\n    best_epoch_cv = df_cv_per_epoch_val_acc['mean_val_acc'].idxmax()\n    #Locate the Best Epoch Number (not the value but the epoch number) by the Moving Average\n    best_epoch_cv_by_moving_average = df_cv_per_epoch_val_acc['moving_average'].idxmax()\n    \n    return df_cv_per_epoch_val_acc, best_epoch_cv, best_epoch_cv_by_moving_average","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Parameters\n\nHere are some parameters that will be used to control what the notebook does later and what inputs are fed to the model and validation process."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"Run_CV = \"Y\"\nRun_Holdout_Validation = \"Y\"\nRun_Kaggle_Pred = \"Y\"\nn_epochs = 75 #333 #number of epochs\nmy_batch_size = 128\nmy_verbose = 0 #how much information keras shows per epoch 0 shows least, 1 shows moving arrows as each epoch progresses, 2 displays accuracy at the end of each epoch\nK = 4 #number of folds\nlen_test = len(X_test)\nmoving_average_period = 10\nmean_chart_lower_epoch_bound = 30","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Something to speed things up a little\n\nThe allowable run-time for models using a GPU in this competition is 2 hours. It is good practice to think about the compute time of models. One thing that can speed up the run-time of a model is to start with a higher learning rate and as the model increases its accuracy decrease the learning rate. If the learning rate is low all the time the model will be accurate but it will take a long time to train. If the learning rate is high all the time it will train quickly but will lose some accuracy. Decreasing the learning rate as the model progresses is one type of attempt to get good accuracy without long compute times."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy', patience=4, verbose=my_verbose, factor=0.55, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Execute\nOK let's get going with our Machine Learning model!"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\" Epochs: \" + str(n_epochs))\nprint(\" Cross Validation Requested: \" + Run_CV)\nprint(\" Kaggle Prediction Requested: \" + Run_Kaggle_Pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Cross-Validation\n\nValidation is very important in Machine Learning. Cross-Validation tends to be more robust than a single train test split validation. However, cross-validation is also much more time consuming. In practice for neural networks, especially with large datasets, cross-validation may not be practical and it may be better to do validation on a single train test split of the data. This notebook is an example of cross-validation and also uses the Dig-MNIST data for holdout validation at the end."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"if Run_CV==\"Y\":\n    print(\" Batch Size: \" + str(my_batch_size))\n    print(\" Number of K-Folds: \" + str(K))\n\n\n    print(('Fold Preparation: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n    kfold = StratifiedKFold(n_splits = K, \n                            random_state = 2007, \n                            shuffle = True) \n\n    oof_pred = None\n    df_cv_per_epoch_train_acc = pd.DataFrame()\n    df_cv_per_epoch_val_acc = pd.DataFrame()\n    \n    print(('KFold Model Starting: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n    \n    for i, (f_ind, outf_ind) in enumerate(kfold.split(X_train, y_train)):\n        my_optimiser = Adam(lr=0.004, beta_1=0.9, beta_2=0.999, epsilon=my_epsilon, decay=0.0, amsgrad=False)\n        \n        # Create data for this fold\n        X_train_f, X_val_f = X_train.loc[f_ind].copy(), X_train.loc[outf_ind].copy()\n    \n        # Normalize and reshape\n        X_train_f = X_train_f.astype('float32') / 255.\n        X_train_f = X_train_f.values.reshape(X_train_f.shape[0], 28, 28, 1).astype('float32') #Fabien Tence suggests this shape suits Tensorflow but Theano requires 1, 28, 28\n        X_val_f = X_val_f.astype('float32') / 255.\n        X_val_f = X_val_f.values.reshape(X_val_f.shape[0], 28, 28, 1).astype('float32') #Fabien Tence suggests this shape suits Tensorflow but Theano requires 1, 28, 28\n\n        #Identify the input_shape - the cnn needs this\n        input_shape = X_train_f.shape[1:]\n        nnet_model = build_network(input_shape)\n     \n        y_train_f, y_val_f = y_train[f_ind], y_train[outf_ind]\n        y_train_f = y_train_f.values\n        y_val_f = y_val_f.values\n        y_val_f_series = y_val_f\n\n        y_train_f = to_categorical(y_train_f, num_classes = nb_classes)\n        y_val_f = to_categorical(y_val_f, num_classes = nb_classes)\n    \n        print(('Augmenting Data: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))    \n        datagen.fit(X_train_f) #This step must be after reshaping\n        # Run model for this fold\n        print(('Model Fitting: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n        print('Fold: ' + str(i))\n        history = nnet_model.fit_generator(datagen.flow(X_train_f,y_train_f, batch_size=my_batch_size), epochs=n_epochs, verbose=my_verbose, \n                                           steps_per_epoch=X_train.shape[0] // my_batch_size, validation_data=(X_val_f,y_val_f), callbacks=[learning_rate_reduction])\n                \n        df_cv_per_epoch_train_acc['fold_'+str(i)] = history.history['accuracy']\n        df_cv_per_epoch_val_acc['fold_'+str(i)] = history.history['val_accuracy']\n        \n        # Generate validation predictions for this fold\n        y_preds = nnet_model.predict(X_val_f)\n        if Run_Holdout_Validation==\"Y\":\n            if i==0:\n                holdout_preds = nnet_model.predict(X_holdout)\n            else:\n                holdout_preds = holdout_preds + nnet_model.predict(X_holdout)\n        \n        if Run_Kaggle_Pred==\"Y\":\n            if i==0:\n                test_preds = nnet_model.predict(X_test)\n            else:\n                test_preds = test_preds + nnet_model.predict(X_test)\n        \n        y_preds_series = np.argmax(y_preds,axis = 1)\n        y_preds_series = pd.Series(y_preds_series,name=\"label\")\n\n        fold_accuracy = accuracy_score(y_val_f_series, y_preds_series)\n\n        print( \" Fold Accuracy = %3.6f\"% (fold_accuracy)) # Report the accuracy of the prediction\n\n        if oof_pred is None:\n            oof_pred = y_preds_series\n            oof_pred_ids = outf_ind\n        else:\n            oof_pred = np.hstack((oof_pred, y_preds_series))\n            oof_pred_ids = np.hstack((oof_pred_ids, outf_ind))\n            \n        df_cv_per_epoch_train_acc, best_train_epoch_cv, best_train_epoch_cv_by_moving_average = epoch_cv(df_cv_per_epoch_train_acc, i)\n        df_cv_per_epoch_val_acc, best_epoch_cv, best_epoch_cv_by_moving_average = epoch_cv(df_cv_per_epoch_val_acc, i)\n        print(\"Best Single Epoch: \" + str(best_epoch_cv))\n\n        df_cv_per_epoch_val_acc['mean_train_acc'] = df_cv_per_epoch_train_acc['mean_val_acc']\n        \n        PlotAcc(history, n_epochs) # plot the accuracy for this fold\n            \n    #Output CV Epoch Data\n    df_cv_per_epoch_val_acc.to_csv('df_cv_epoch_{:%Y%m%d%H%M%S}.csv'.format(datetime.datetime.now()), index=True)\n    \n    #Deal with oof preds\n    oof_pred = np.column_stack((oof_pred_ids, oof_pred))\n    df_oof_pred = pd.DataFrame(oof_pred,index=oof_pred[:,0])\n    df_oof_pred.columns = ['id', 'label']\n    df_oof_pred = df_oof_pred.sort_values(by=('id'), ascending=True)\n    \n    y_valid_pred = df_oof_pred['label'].values\n    \n    oof_accuracy = accuracy_score(y_train, y_valid_pred)\n    print( \" Overall Out-of-Fold Accuracy = %3.4f\"% (oof_accuracy))   \nelse:\n    print(\"Cross-Validation skipped\")\n\nif Run_Kaggle_Pred==\"Y\":\n    test_preds = test_preds / K\n    #format prediction\n    results = np.argmax(test_preds,axis = 1)\n    results = pd.Series(results,name=\"label\")\n\n    print(('Writing Prediction: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n    submission = pd.concat([pd.Series(range(0,len_test),name = \"id\"),results],axis = 1)\n    submission.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plotting the Mean Accuracy\n\nWe know that the training accuracy is very low on the first few epochs so we'll exclude some initial epochs from the chart to show more clearly what is happening on later epochs."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"lbound = min(n_epochs-1, mean_chart_lower_epoch_bound)\nubound = max(n_epochs, lbound) + 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(14,8))\nplt.plot(np.arange(lbound, ubound), df_cv_per_epoch_val_acc[\"mean_train_acc\"][lbound-1:], label=\"mean_train_acc\")\nplt.plot(np.arange(lbound, ubound), df_cv_per_epoch_val_acc[\"mean_val_acc\"][lbound-1:], label=\"mean_val_acc\")\nplt.title(\"Mean Accuracy across folds\")\nplt.xlabel(\"Epoch #\")\nplt.ylabel(\"Accuracy\")\nplt.legend(loc=\"lower right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Your <span style=\"color:green\">upvote</span> encourages people to share their code."},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREVIEW PREDICTIONS\nplt.figure(figsize=(15,6))\nfor i in range(40):  \n    plt.subplot(4, 10, i+1)\n    plt.imshow(X_test[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"predict=%d\" % results[i],y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Holdout Validation\n\nIn the original MNIST competition validation scores achieved whilst training the model were in line with the accuracy scores on the test set on the public leaderboard. In this competition to date they have not been. In this kernel validation accuracy has been around 0.997, but the leaderboard scores for has been around 0.987. At the time this notebook is being prepared near the end of the competition the highest score on the PLB is 0.9918.\n\nHere, we use a holdout set to get a different perspective on validation accuracy. Fortunately the competition organisers have provided the Dig-MNIST dataset for validation. The competition notes state that accuracy on this holdout set has been reported to be much lower than on the test set. This is because the images were scanned in a different way. So we will keep that in mind when we interpret the resulting accuracy."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"if Run_Holdout_Validation==\"Y\":\n    holdout_preds = holdout_preds / K\n    #format prediction\n    results_holdout = np.argmax(holdout_preds,axis = 1)\n    results_holdout = pd.Series(results_holdout,name=\"label\") \n\n    holdout_accuracy = accuracy_score(y_holdout, results_holdout)\n    print( \" Holdout Accuracy = %3.5f\"% (holdout_accuracy))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Where were the errors on the holdout set?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# build confusion matrix\nconf = confusion_matrix(y_holdout, results_holdout)\nconf = pd.DataFrame(conf, index=range(0,10), columns=range(0,10))\n\n# plot the confusion matrix\nplt.figure(figsize=(12,10))\nsns.heatmap(conf, cmap=\"coolwarm\", annot=True , fmt=\"d\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Summary\n\nIf the model ran then a summary of the model architecture will be shown. Otherwise a warning will show that the model didn't run."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"if Run_Kaggle_Pred==\"Y\" or Run_CV==\"Y\":\n    print(nnet_model.summary())\nelse:\n    print(\"Select Cross-Validation Run, Kaggle Prediction, or both.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Blended Validation\n\nIn the previous run of this kernel we got a good approximation to the accuracy on the test set (public leaderboard) by taking a weighted average of the cross-validation and holdout validation values. The weights to this are overfit to values I have seen in previous runs of this kernel. I am not sure how well this will generalise either to later attempts on the public leaderboard let alone to the private leaderboard. Still it may be an improvement on the cross-validation accuracy which seems to have approached a frontier. What validation methods have you tried?"},{"metadata":{"trusted":true},"cell_type":"code","source":"cv_weight = 7\nholdout_weight = 1\n\nblended_validation_accuracy = ((cv_weight * oof_accuracy) + (holdout_weight * holdout_accuracy)) / (cv_weight + holdout_weight)\nprint( \"Blended Validation Accuracy = %3.5f\"% (blended_validation_accuracy))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This value should not be taken as an accurate guide to the public leaderboard score or to the final private leaderboard accuracy. It is just one attempt to find a reliable validation approach. Finding a good validation approach is an important part of Machine Learning."},{"metadata":{},"cell_type":"markdown","source":"### Comments on Accuracy\n\nAccuracy on the holdout set is lower. This could be due to overfitting to the training & validation data. However we know these images were captured in a different way. This could increase the complexity of getting high accuracy predictions on this holdout set. This situation might benefit from more complex CNN's than were required for the original MNIST."},{"metadata":{},"cell_type":"markdown","source":"## Further comments after the competition ended\n\nThe blended accuracy approach above did a reasonable job on this dataset of estimating the final result. That isn't surprising since it was mostly weighted to the cv score! On version 21 of this kernel, the blended validation score was 0.9872, the public score was 0.9876, and the private leaderboard score was 0.9886. "},{"metadata":{},"cell_type":"markdown","source":"### Credits\n\nA lot of the ideas in the kernel came from working on the original MNIST competition. There are many useful kernels there that could be of interest. However, this competition is a unique challenge in itself and directly applying a solution from the other competition does not appear to be the optimal solution. \n* In this competition the [notebook](https://www.kaggle.com/jinbao/kannada-mnist-baseline/notebook?scriptVersionId=21092956) by [Jinbao](https://www.kaggle.com/jinbao) was helpful in a few different ways. \n* Also this [notebook](https://www.kaggle.com/ankur1401/kannada-digit-recognizer) by [Ankur](https://www.kaggle.com/ankur1401).\n* This [notebook by Ronald](https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits) was the single most helpful as it provided a slight improvement on other similar CNN architectures.\n* This [notebook](https://www.kaggle.com/ilyamich/kannada-mnist-choosing-the-right-optimizer) was also useful in reminding me of the value of a confusion matrix.\n* This [notebook](https://www.kaggle.com/arthurtok/interactive-intro-to-dimensionality-reduction) from the original MNIST competition has some fascinating visualisations."},{"metadata":{},"cell_type":"markdown","source":"#### Please <span style=\"color:blue\">Upvote</span> if you learned something new or found this interesting."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(('Run Completed: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}