{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Optical Digit Recognition - Transfer Learning\nThis is the original MNIST dataset. Currently on Kaggle there is another MNIST challenge. The kernel here is adapted from a [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script) used in that other challenge.\n\nLet's see how well a model architecture designed on that data will perform on this MNIST dataset.\n\n### If you find this kernel helpful please <span style=\"color:green\">upvote</span>"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Libraries\nLoad other useful libraries particularly the ones that we need to build the Convolutional Neural Network (CNN) which is an effective type of model for dealing with image recognition. In this competition access to the internet is not permitted for the kernel. We will exhibit building a model from scratch rather than using a pre-trained model."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom pathlib import Path\nimport datetime\nfrom keras.models import Sequential\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import accuracy_score\nfrom keras import regularizers\nfrom keras.layers import Input, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D\nfrom keras.layers import AveragePooling2D, Dropout, GlobalMaxPooling2D, GlobalAveragePooling2D\nfrom keras.models import Model\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.optimizers import Adam\nfrom keras.callbacks import ReduceLROnPlateau, LearningRateScheduler\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT = Path(\"../input/digit-recognizer\")\nos.listdir(INPUT)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the data\nX_train = pd.read_csv(INPUT/\"train.csv\")\nX_test = pd.read_csv(INPUT/\"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\" Training Data Shape: \" + str(X_train.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\" Test Data Shape: \" + str(X_test.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"id_train = X_train.index.values\ny_train = X_train['label']\ny_valid_pred = 0*y_train\n\nX_train.drop(labels=['label'], axis=1, inplace=True)\n\n# prepare the test data set by removing the id\n#X_test.drop(labels=['id'], axis=1, inplace=True)\n\n#prepare test data\nX_test = X_test.astype('float32') / 255.\nX_test = X_test.values.reshape(X_test.shape[0], 28, 28, 1).astype('float32')\n\nmy_init = 'glorot_uniform'\nmy_activ = 'relu'\nmy_optimiser = 'adam'\nmy_epsilon = 1e-8\nnb_classes = 10","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Visualisation\nBelow is a representation of what the Kannada digits look like. We can see that they have been captured with low resolution images."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax=plt.subplots(5,10)\nfor i in range(5):\n    for j in range(10):\n        ax[i][j].imshow(X_test[np.random.randint(0,X_test.shape[0]),:,:,0],cmap=plt.cm.binary)\n        ax[i][j].axis('off')\nplt.subplots_adjust(wspace=0, hspace=0)        \nfig.set_figwidth(15)\nfig.set_figheight(7)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Function\nDefine a function that returns the model. This will set the model architecture and will prevent the need to type this all out repeatedly for the validation and prediction stages.\n\nWe will build a CNN of several layers. This is not a simple NN but it certainly isn't complex either relative to pretrained models that can contain over 100 layers.\n\nThe model uses a softmax output to output the probabilities for each of the 10 digits"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_network(input_shape):    \n    model = Sequential()\n    # For an explanation on conv layers see http://cs231n.github.io/convolutional-networks/#conv\n    # For an explanation on pooling layers see http://cs231n.github.io/convolutional-networks/#pool\n    # By default the stride/subsample is 1 and there is no zero-padding.\n    # use padding=\"same\" if you want to preserve dimensions\n    \n    #model.add(ZeroPadding2D(padding=(2, 2), data_format=\"channels_last\", input_shape=input_shape))\n    model.add(Conv2D(filters=64, kernel_size=(5, 5), strides=(1, 1), input_shape=input_shape, padding=\"same\", activation='relu', kernel_initializer=my_init))\n    model.add(MaxPooling2D(pool_size=(2, 2), padding='valid'))    \n    model.add(Dropout(0.25))\n    model.add(Conv2D(filters=128, kernel_size=(5, 5), strides=(1, 1), padding=\"same\", activation='relu', kernel_initializer=my_init))\n    model.add(MaxPooling2D(pool_size=(2, 2), padding='valid'))    \n    model.add(Dropout(0.25))\n    model.add(Conv2D(filters=256, kernel_size=(3, 3), strides=(1, 1), padding=\"same\", activation='relu', kernel_initializer=my_init))\n    model.add(MaxPooling2D(pool_size=(2, 2), padding='valid'))    \n    model.add(Dropout(0.25))\n    \n    # Flatten the 3D output to 1D tensor for a fully connected layer to accept the input\n    model.add(Flatten())\n    \n    \n    #Fully Connected Layer\n    model.add(Dense(1024, kernel_initializer=my_init))\n    model.add(Activation(my_activ))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4)) #dropout is a type of regularisation. Regularisation helps to control overfitting\n    #Fully Connected Layer\n    model.add(Dense(512, kernel_initializer=my_init))\n    model.add(Activation(my_activ))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25))\n    \n    #Output layer\n    model.add(Dense(nb_classes, activation='softmax')) \n\n    model.compile(optimizer=my_optimiser,\n        loss='categorical_crossentropy',\n        metrics=['accuracy'])\n    \n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentation Settings\nCNN's can often produce better results if they are fed not just the original input images. Here we take the input image data and make some specific types of adjustments. This process is called augmentation. For example if an image is rotated 1 degree to the right in many cases it will still be identifiable to a human as the same object. Similarly a model can be trained to still recognise the image. Again if we as humans zoom in slightly to the image often it is still recognisable. Once again a model can be trained to still be able to recognise the image. All these slight variations often allow the model to train in a more robust way. There are many other types of ways to adjust the image. Not all of them will be used here but there are some notes in the code below to briefly outline the types of augmentations that can be done."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=15,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.2, # Randomly zoom image \n        shear_range=20, #move top of image along without moving the bottom or vice versa\n        width_shift_range=0.2,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.2,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False,  # randomly flip images\n        data_format=\"channels_last\"         )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Charting Function\nLater we will draw some charts. Here we set up the function we can call when we need to do that."},{"metadata":{"trusted":true},"cell_type":"code","source":"def PlotLoss(his, epoch):\n    plt.style.use(\"ggplot\")\n    plt.figure()\n    plt.plot(np.arange(1, epoch + 1), his.history[\"loss\"], label=\"train_loss\")\n    plt.plot(np.arange(1, epoch + 1), his.history[\"val_loss\"], label=\"val_loss\")\n    plt.title(\"Training Loss\")\n    plt.xlabel(\"Epoch #\")\n    plt.ylabel(\"Loss\")\n    plt.legend(loc=\"upper right\")\n    plt.show()\n\ndef PlotAcc(his, epoch):\n    plt.style.use(\"ggplot\")\n    plt.figure()\n    plt.plot(np.arange(1, epoch + 1), his.history[\"accuracy\"], label=\"train_accuracy\")\n    plt.plot(np.arange(1, epoch + 1), his.history[\"val_accuracy\"], label=\"val_accuracy\")\n    plt.title(\"Training and Validation Accuracy\")\n    plt.xlabel(\"Epoch #\")\n    plt.ylabel(\"Accuracy\")\n    plt.legend(loc=\"lower right\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Utility Function\nAnother function that we will use later"},{"metadata":{"trusted":true},"cell_type":"code","source":"def epoch_cv(df_cv_per_epoch_val_acc, fold_num):\n    #Find the best epoch by the best single and the best moving average\n    #Update the index to start at 1 with epoch 1 for the validation accuracy data\n    if fold_num==0:\n        df_cv_per_epoch_val_acc.index += 1 #needed so length of values matches length of index\n    df_cv_per_epoch_val_acc['mean_val_acc'] = df_cv_per_epoch_val_acc.mean(axis=1)\n    #Calculate an epoch moving average\n    num_epochs = df_cv_per_epoch_val_acc.shape[0]\n    \n    for i in range(1, num_epochs+1):\n        #print(i)\n        if i<moving_average_period+1:\n            df_cv_per_epoch_val_acc.at[i, 'moving_average'] = df_cv_per_epoch_val_acc.iloc[:i]['mean_val_acc'].mean()\n        else:\n            df_cv_per_epoch_val_acc.at[i, 'moving_average'] = df_cv_per_epoch_val_acc.iloc[i-moving_average_period:i]['mean_val_acc'].mean()\n    \n    #Locate the Best Epoch Number (not the value but the epoch number) by the Mean per epoch\n    best_epoch_cv = df_cv_per_epoch_val_acc['mean_val_acc'].idxmax()\n    #Locate the Best Epoch Number (not the value but the epoch number) by the Moving Average\n    best_epoch_cv_by_moving_average = df_cv_per_epoch_val_acc['moving_average'].idxmax()\n    \n    return df_cv_per_epoch_val_acc, best_epoch_cv, best_epoch_cv_by_moving_average","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Parameters\nHere are some parameters that will be used to control what the notebook does later and what inputs are fed to the model and validation process"},{"metadata":{"trusted":true},"cell_type":"code","source":"Run_CV = \"Y\"\nRun_Kaggle_Pred = \"Y\"\nn_epochs = 80 #333 #number of epochs\nmy_batch_size = 96\nmy_verbose = 0 #how much information keras shows per epoch 0 shows least, 1 shows moving arrows as each epoch progresses, 2 displays accuracy at the end of each epoch\nK = 6 #number of folds\nlen_test = len(X_test)\nmoving_average_period = 10\nmean_chart_lower_epoch_bound = 20\ninitial_learningrate = 2.2e-3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Something to speed things up a little\nIt is good practice to think about the compute time of models. One thing that can speed up the run-time of a model is to start with a higher learning rate and as the model increases its accuracy decrease the learning rate. If the learning rate is low all the time the model will be accurate but it will take a long time to train. If the learning rate is high all the time it will train quickly but will lose some accuracy. Decreasing the learning rate as the model progresses is one type of attempt to get good accuracy without long compute times."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set a learning rate annealer\ndef lr_decay(epoch):#lrv\n    return initial_learningrate * 0.99 ** epoch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Execute\nOK let's get going with this modelling!"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\" Epochs: \" + str(n_epochs))\nprint(\" Cross Validation Requested: \" + Run_CV)\nprint(\" Kaggle Prediction Requested: \" + Run_Kaggle_Pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Cross-Validation\nValidation is very important in Machine Learning. Cross-Validation tends to be more robust than a single train test split validation. However, cross-validation is also much more time consuming. In practice for neural networks, especially with large datasets, cross-validation may not be practical and it may be better to do validation on a single train test split of the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"if Run_CV==\"Y\":\n    print(\" Batch Size: \" + str(my_batch_size))\n    print(\" Number of K-Folds: \" + str(K))\n\n\n    print(('Fold Preparation: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n    kfold = StratifiedKFold(n_splits = K, \n                            random_state = 2002, \n                            shuffle = True) \n\n    oof_pred = None\n    df_cv_per_epoch_train_acc = pd.DataFrame()\n    df_cv_per_epoch_val_acc = pd.DataFrame()\n    \n    print(('KFold Model Starting: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n    \n    for i, (f_ind, outf_ind) in enumerate(kfold.split(X_train, y_train)):\n        my_optimiser = Adam(lr=0.004, beta_1=0.9, beta_2=0.999, epsilon=my_epsilon, decay=0.0, amsgrad=False)\n        \n        # Create data for this fold\n        X_train_f, X_val_f = X_train.loc[f_ind].copy(), X_train.loc[outf_ind].copy()\n        #X_train_f = X_train_f.values\n        #X_val_f = X_val_f.values\n    \n        # Normalize and reshape\n        X_train_f = X_train_f.astype('float32') / 255.\n        X_train_f = X_train_f.values.reshape(X_train_f.shape[0], 28, 28, 1).astype('float32') #Fabien Tence suggests this shape suits Tensorflow but Theano requires 1, 28, 28\n        X_val_f = X_val_f.astype('float32') / 255.\n        X_val_f = X_val_f.values.reshape(X_val_f.shape[0], 28, 28, 1).astype('float32') #Fabien Tence suggests this shape suits Tensorflow but Theano requires 1, 28, 28\n\n        #Identify the input_shape - the cnn needs this\n        input_shape = X_train_f.shape[1:]\n        #nnet_model = build_model(input_shape=input_shape, classes = nb_classes)\n        #nnet_model.compile(loss='categorical_crossentropy', optimizer=my_optimiser, metrics=['accuracy'])\n        nnet_model = build_network(input_shape)\n     \n        y_train_f, y_val_f = y_train[f_ind], y_train[outf_ind]\n        y_train_f = y_train_f.values\n        y_val_f = y_val_f.values\n        y_val_f_series = y_val_f\n\n        y_train_f = to_categorical(y_train_f, num_classes = nb_classes)\n        y_val_f = to_categorical(y_val_f, num_classes = nb_classes)\n    \n        print(('Augmenting Data: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))    \n        datagen.fit(X_train_f) #This step must be after reshaping\n        # Run model for this fold\n        print(('Model Fitting: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n        print('Fold: ' + str(i))\n        history = nnet_model.fit_generator(datagen.flow(X_train_f,y_train_f, batch_size=my_batch_size), epochs=n_epochs, verbose=my_verbose, \n                                           steps_per_epoch=X_train.shape[0] // my_batch_size, validation_data=(X_val_f,y_val_f), callbacks=[LearningRateScheduler(lr_decay)])\n                \n        df_cv_per_epoch_train_acc['fold_'+str(i)] = history.history['accuracy']\n        df_cv_per_epoch_val_acc['fold_'+str(i)] = history.history['val_accuracy']\n        \n        # Generate validation predictions for this fold\n        y_preds = nnet_model.predict(X_val_f)\n        \n        if Run_Kaggle_Pred==\"Y\":\n            if i==0:\n                test_preds = nnet_model.predict(X_test)\n            else:\n                test_preds = test_preds + nnet_model.predict(X_test)\n        \n        y_preds_series = np.argmax(y_preds,axis = 1)\n        y_preds_series = pd.Series(y_preds_series,name=\"label\")\n\n        fold_accuracy = accuracy_score(y_val_f_series, y_preds_series)\n\n        print( \" Fold Accuracy = %3.6f\"% (fold_accuracy)) # Report the accuracy of the prediction\n\n        if oof_pred is None:\n            oof_pred = y_preds_series\n            oof_pred_ids = outf_ind\n        else:\n            oof_pred = np.hstack((oof_pred, y_preds_series))\n            oof_pred_ids = np.hstack((oof_pred_ids, outf_ind))\n            \n        df_cv_per_epoch_train_acc, best_train_epoch_cv, best_train_epoch_cv_by_moving_average = epoch_cv(df_cv_per_epoch_train_acc, i)\n        df_cv_per_epoch_val_acc, best_epoch_cv, best_epoch_cv_by_moving_average = epoch_cv(df_cv_per_epoch_val_acc, i)\n        print(\"Best Single Epoch: \" + str(best_epoch_cv))\n        #print(\"Best Epoch Moving Average Period: \" + str(moving_average_period) + \" || Best Epoch: \"  + str(best_epoch_cv_by_moving_average))\n\n        df_cv_per_epoch_val_acc['mean_train_acc'] = df_cv_per_epoch_train_acc['mean_val_acc']\n        \n        PlotAcc(history, n_epochs) # plot the accuracy for this fold\n        \n    \n    #Output CV Epoch Data\n    df_cv_per_epoch_val_acc.to_csv('df_cv_epoch_{:%Y%m%d%H%M%S}.csv'.format(datetime.datetime.now()), index=True)\n    \n    #Deal with oof preds\n    oof_pred = np.column_stack((oof_pred_ids, oof_pred))\n    df_oof_pred = pd.DataFrame(oof_pred,index=oof_pred[:,0])\n    df_oof_pred.columns = ['ImageId', 'Label']\n    df_oof_pred = df_oof_pred.sort_values(by=('ImageId'), ascending=True)\n    \n    y_valid_pred = df_oof_pred['Label'].values\n    \n    oof_accuracy = accuracy_score(y_train, y_valid_pred)\n    print( \" Overall Out-of-Fold Accuracy = %3.4f\"% (oof_accuracy))   \nelse:\n    print(\"Cross-Validation skipped\")\n\nif Run_Kaggle_Pred==\"Y\":\n    test_preds = test_preds / K\n    #format prediction\n    results = np.argmax(test_preds,axis = 1)\n    results = pd.Series(results,name=\"Label\")\n\n    print(('Writing Prediction: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))\n    submission = pd.concat([pd.Series(range(1,len_test+1),name = \"ImageId\"),results],axis = 1)\n    submission.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting the Mean Accuracy¶\nWe know that the training accuracy is very low on the first few epochs so we'll display the chart from the 5th epoch only (or lower if few epochs are selected)."},{"metadata":{"trusted":true},"cell_type":"code","source":"lbound = min(n_epochs-1, mean_chart_lower_epoch_bound)\nubound = max(n_epochs, lbound) + 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nplt.plot(np.arange(lbound, ubound), df_cv_per_epoch_val_acc[\"mean_train_acc\"][lbound-1:], label=\"mean_train_acc\")\nplt.plot(np.arange(lbound, ubound), df_cv_per_epoch_val_acc[\"mean_val_acc\"][lbound-1:], label=\"mean_val_acc\")\nplt.title(\"Mean Accuracy across folds\")\nplt.xlabel(\"Epoch #\")\nplt.ylabel(\"Accuracy\")\nplt.legend(loc=\"lower right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Summary\nIf the model ran then a summary of the model architecture will be shown. Otherwise a warning will show that the model didn't run."},{"metadata":{"trusted":true},"cell_type":"code","source":"if Run_Kaggle_Pred==\"Y\" or Run_CV==\"Y\":\n    print(nnet_model.summary())\nelse:\n    print(\"Select Cross-Validation Run, Kaggle Prediction, or both.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### If you got this far please <span style=\"color:blue\">upvote</span>"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(('Finish: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}