{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30042,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognizer using LeNet-5 with improvements\n\nIn this notebook I will use [LeNet-5](https://en.wikipedia.org/wiki/LeNet) digit recognizer model.\n\n1. First I will implement the basic [LeNet-5](http://yann.lecun.com/exdb/publis/pdf/lecun-95a.pdf) which was proposed by Yann LeCun et al. in 1989.\n2. Second, I will add some improvements by replacing tanh with ReLu activation and using dropout regularization (I later removed dropout and put BatchNormalization).\n3. Third, I will use augmentation to add some more random samples based on the provided samples, to add more generalization, and also because the boosting requires a lot of data.\n3. Fourth, I will use a boosting technique with LeNet5. There is a lot of literature about boosted LeNet4 and I've never seen boosted LeNet5 in papers or implementations, so I was curious to implement it by myself.\n\nNote that I only used the provided kaggle dataset, which is a partial MNIST database (with the test list being the other unprovided MNIST part). \nMany submitted results use the entire MNIST database for training, which I consider cheating, and less challenging. I prefer the challenge.\n","metadata":{}},{"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Read and prepare the input data","metadata":{}},{"cell_type":"code","source":"# Read data from files\n\ntrain_pd = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_pd = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nfrom keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\n\n# Extract the labels from the dataframe\nlabel_nums = train_pd[\"label\"]\nlabel = to_categorical(label_nums, num_classes = 10)\n\n# Extract and normalize the data \ntrain = train_pd.drop(labels = [\"label\"],axis = 1) / 255.0\ntest = test_pd / 255.0\n\n# Reshape the data\n(rows, cols, channels) = (28,28,1)\n\ndef ReShape (DF, rows, cols, channels):\n    return DF.values.reshape(DF.shape[0],rows,cols,channels)\n\ntrain = ReShape (train, rows, cols, channels)\ntest = ReShape (test, rows, cols, channels)\n\n# Build training and validation sets\nX_train, X_val, Y_train, Y_val = train_test_split(train,label, test_size = 0.1)\n\nprint (\"X_train shape:\" , X_train.shape, \"  Y_train shape:\", Y_train.shape)\nprint (\"X_val shape:\", X_val.shape, \"  Y_val shape:\", Y_val.shape)\nprint (\"test shape:\", test.shape)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Show some input samples","metadata":{}},{"cell_type":"code","source":"# Plotting examples of training data\n\nimport matplotlib.pyplot as plt\nfrom collections import defaultdict\nfrom random import sample\n\ndef plot_digits(x,y,num_rows=10, start_range=0):\n    figw, figh = 12.0, 12.0\n\n    fig, axes = plt.subplots(num_rows, 10, figsize=(figw, figh))\n    plt.subplots_adjust(wspace = 0.2/figw, hspace = 0.2/figh )\n\n    counter = defaultdict(int)\n\n    for i in sample(range(start_range,len(x)), 1000):\n        label = np.argmax(y[i])\n        if counter[label] >= num_rows: continue\n        img = x[i].reshape((rows,cols))\n        axes[counter[label], label].imshow(img, cmap='Greys')\n        axes[0, label].set_title(label)\n        counter[label] += 1\n\n    [ax.set_axis_off() for ax in axes.ravel()]\n    plt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot random digits from the training dataset\nplot_digits(X_train,Y_train)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Build LeNet-5 CNN models","metadata":{}},{"cell_type":"code","source":"# Read keras libraries\n\nfrom keras.layers import Activation,Dropout,Dense,Conv2D,AveragePooling2D,MaxPooling2D,Flatten,ZeroPadding2D,BatchNormalization\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.losses import categorical_crossentropy\nfrom keras.callbacks import EarlyStopping\n\n#from tensorflow.keras import Model\n\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## LeNet-5","metadata":{}},{"cell_type":"code","source":"# LeNet-5 model\nclass LeNet5(Sequential):\n    def __init__(self, input_shape=(rows, cols, channels),activation='tanh',pooling='avg',dropout = 0, name=\"Base\"):\n        super().__init__(name='LeNet5_'+name)\n        self.add(Conv2D(6, kernel_size=(5, 5), strides=(1, 1), activation=activation, input_shape=input_shape, padding=\"same\"))\n        if pooling == 'avg':\n            self.add(AveragePooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n        else:\n            self.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n        self.add(Conv2D(16, kernel_size=(5, 5), strides=(1, 1), activation=activation, padding='valid'))\n        if pooling == 'avg':\n            self.add(AveragePooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n        else:\n            self.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n        if dropout > 0:\n            self.add((Dropout(rate = dropout)))\n        self.add(Flatten())\n        self.add(Dense(120, activation=activation))\n        self.add(Dense(84, activation=activation))\n        self.add(Dense(10, activation='softmax'))\n\n        self.compile(optimizer='adam',\n                    loss=categorical_crossentropy,\n                    metrics=['accuracy'])\n        \n\n\n# LeNet-5 improved model with BatchNormalization\nclass LeNet5BN(Sequential):\n    def __init__(self, input_shape=(rows, cols, channels), name=\"ImprovedBN\"):\n        super().__init__(name='LeNet5_'+name)\n        self.add(Conv2D(6, kernel_size=(5, 5), strides=(1, 1), input_shape=input_shape, padding=\"same\"))\n        self.add(BatchNormalization())\n        self.add(Activation('relu'))\n        self.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n        self.add(Conv2D(16, kernel_size=(5, 5), strides=(1, 1), padding='valid'))\n        self.add(BatchNormalization())\n        self.add(Activation('relu'))\n        self.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n        self.add(Flatten())\n        self.add(Dense(120))\n        self.add(BatchNormalization())\n        self.add(Activation('relu'))\n        self.add(Dense(84))\n        self.add(BatchNormalization())\n        self.add(Activation('relu'))\n        self.add(Dense(10, activation='softmax'))\n\n        self.compile(optimizer='adam',\n                    loss=categorical_crossentropy,\n                    metrics=['accuracy'])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model_LeNet5_base = LeNet5()\nmodel_LeNet5_base.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train the model\nearly_stop = EarlyStopping(patience=5, monitor='val_accuracy', restore_best_weights=True)\nmodel_LeNet5_base.fit(X_train, y=Y_train, \n                      epochs=25, batch_size=128, \n                      validation_data=(X_val, Y_val),\n                      callbacks=[early_stop])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds1 = model_LeNet5_base.evaluate(x = X_val, y = Y_val)\nprint (\"Loss = \" + str(preds1[0]))\nprint (\"Val Accuracy = \" + str(preds1[1]))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Improved LeNet-5 model\nI improved the basic model by:\n1. Replacing tanh activation function with Rectified Linear Unit - ReLu.\n2. Used max pooling instead of average.\n3. Used dropout for more generalization (especially important since the database is partial).\n4. Used also a model with BatchNormalization.","metadata":{}},{"cell_type":"code","source":"#model_LeNet5 = LeNet5(activation='relu',pooling='max',dropout = 0.2, name=\"Improved\")\nmodel_LeNet5 = LeNet5BN(name=\"Improved_BN\")\nmodel_LeNet5.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"early_stop = EarlyStopping(patience=5, monitor='val_accuracy', restore_best_weights=True)\nmodel_LeNet5.fit(X_train, y=Y_train, \n                    epochs=25, batch_size=128, \n                    validation_data=(X_val, Y_val),\n                    callbacks=[early_stop])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds2 = model_LeNet5.evaluate(x = X_val, y = Y_val)\nprint (\"Loss = \" + str(preds2[0]))\nprint (\"Val Accuracy = \" + str(preds2[1]))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see some improvements in the model accuracy compared to the basic model.","metadata":{}},{"cell_type":"markdown","source":"### Analyze LeNet-5 results","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report, confusion_matrix\nimport matplotlib\nimport matplotlib.pyplot as plt\n\n# Plot Confusion Matrix and classification errors\n\ndef analyze_results(model_name, y_pred, y_true, x):\n\n    conf_mat = confusion_matrix(y_true, y_pred, normalize=None)\n    class_report = classification_report(y_true, y_pred)\n    \n    # Plot the confusion matrix\n    \n    fig, ax = plt.subplots(figsize=(11, 9))\n    im = ax.imshow(conf_mat, cmap=\"OrRd\")\n\n    # We want to show all ticks...\n    ax.set_xticks(np.arange(10))\n    ax.set_yticks(np.arange(10))\n    # ... and label them with the respective list entries\n    ax.set_xticklabels(np.arange(10))\n    ax.set_yticklabels(np.arange(10))\n\n    # Loop over data dimensions and create text annotations.\n    for i in range(10):\n        for j in range(10):\n            num_ij = int(conf_mat[i, j])\n            text_c = \"w\" if num_ij > conf_mat.mean() else \"b\"\n            #text_s = str(int(num_ij)) if (num_ij==0) or (num_ij==1) else \"%.3f\" % num_ij\n            text = ax.text(j, i, num_ij, ha=\"center\", va=\"center\", color=text_c)\n\n    ax.set_title(f\"Confusion Matrix for model {model_name}\")\n    ax.set_ylabel('True Label')\n    ax.set_xlabel('Predicted Label')\n    \n    fig.tight_layout()\n    plt.show()\n    \n    print (class_report)\n    \n    # Gather all the missed predictions for further analysis\n    faults = []\n    for i,label in enumerate(y_true):\n        if label != y_pred[i]:\n            fault = {'label': label, 'pred': y_pred[i], 'idx': i,'img': x[i].reshape((28,28))}\n            faults.append(fault)\n    \n    accuracy = (len(y_true) - len(faults)) / len(y_true)\n    print (f\"Accuracy for model {model_name}: {accuracy}\")\n    \n    return conf_mat, class_report, accuracy, faults\n\n\ndef analyze_model(model, x, y):\n    predicts = model.predict(x,batch_size=None, verbose=0)\n    y_pred = np.argmax(predicts, axis=1)\n    y_true = np.argmax(y, axis=1)\n    conf_mat, class_report, accuracy, faults = analyze_results(model.name, y_pred, y_true, x)\n    return conf_mat, class_report, accuracy, faults\n\n\n# Draw images of faults\ndef show_faults(faults):\n    figw, figh = 16.0, 14.0\n\n    max_faults = 11\n    fig, axes = plt.subplots(10,max_faults, squeeze = False, figsize=(figw, figh))\n\n    counter = defaultdict(int)\n\n    for fault in faults:\n        label = fault['label']\n        pred = fault['pred']\n        img = fault['img']\n        if counter[label] >= max_faults: continue\n        axes[label, counter[label]].imshow(img, cmap='Greys')\n        axes[label, counter[label]].set_title(str(label)+'. pred'+str(pred))\n        counter[label] += 1\n\n    [ax.set_axis_off() for ax in axes.ravel()]\n    plt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"conf_mat, class_report, accuracy, faults = analyze_model(model_LeNet5, X_val, Y_val)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Print the digits with wrong predictions\nshow_faults(faults)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Some of the faults are really hard to recognise even for a human. Other faults might be due to a too small training set (the training set in the challenge is smaller than the full MNIST database, and I don't want to cheat using the full MNIST).\nI'll address this issue using augmented data.","metadata":{}},{"cell_type":"markdown","source":"## Data Augmentation\nI will add more samples to the training dataset by making random changes to the provided dataset.\n\nNote that using augmented data is not as good as fresh new data, especially in CNN, but I need more samples for boosting and it's better than nothing.\nI only used some roation, width shift, shear and zoom. I can't apply too many changes (such as flipping, or 180 rotation) because the shape would not be a digit anymore, or would become another digit (9 can turn into 6 but the label remains '9').","metadata":{}},{"cell_type":"code","source":"\nfrom keras.preprocessing.image import ImageDataGenerator\n\ndatagen = ImageDataGenerator(rotation_range=20,\n                             width_shift_range=0.1,\n                             shear_range=0.2,\n                             zoom_range=0.1,\n                            data_format=\"channels_last\")\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Generate new augmented samples offline\n\nX_train_aug = X_train\nY_train_aug = Y_train\nprint (\"Original train size: \" , X_train_aug.shape)\n\ntarget_size = 32200 # Add new samples to the current 37800 training samples\nbatch_size = 200\n\ndigits_gen = datagen.flow(X_train, Y_train, batch_size=batch_size)\n\nfor i in range(target_size//batch_size):   \n    batch = digits_gen.next()\n    X_train_aug = np.append(X_train_aug, batch[0], axis=0)\n    Y_train_aug = np.append(Y_train_aug, batch[1], axis=0)\n\nprint (\"Augmented train size: \" , X_train_aug.shape)\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot some new samples\n\nplot_digits(X_train_aug, Y_train_aug, start_range=40000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# model_LeNet5_aug = LeNet5(activation='relu',pooling='max',dropout = 0.2, name=\"Augmented\")\nmodel_LeNet5_aug = LeNet5BN(name=\"Augmented_BN\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train the model with the augmented dataset\n\nearly_stop = EarlyStopping(patience=5, monitor='loss', restore_best_weights=True)\nbatch_size = 128\n\nmodel_LeNet5_aug.fit (X_train_aug, Y_train_aug, batch_size=batch_size,\n                       epochs=100,\n                       validation_data=(X_val, Y_val),\n                       callbacks=[early_stop])\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds3 = model_LeNet5_aug.evaluate(x = X_val, y = Y_val)\nprint (\"Loss = \" + str(preds3[0]))\nprint (\"Val Accuracy = \" + str(preds3[1]))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"conf_mat, class_report, accuracy, faults  = analyze_model(model_LeNet5_aug, X_val, Y_val)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"show_faults(faults)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Ussually I get better results with the augmented data (but not always).\n\nNow I'll try to improve results with boosted LeNet-5...","metadata":{}},{"cell_type":"markdown","source":"## Boosted LeNet-5\nBased on the idea of boosted LeNet-4... I haven't seen it implemented with LeNet-5.\n\nFor boosted LeNet we need a lot of data.\nI will use the augmented training data and produce 3 datasets for training 3 LeNet-5 networks:\n1. Net1: Train with augmented data\n2. Net2: Train with data consists of: 50% - all errors from Net1. 50% - data without errors.\n3. Net3: Train with data consists of all data that gave different predictions from Net1 and Net2.\n\nThe result prediction will be the sum of all softmax predictions.","metadata":{}},{"cell_type":"code","source":"# Build a new training set for network 2 from previous model based on success and failed samples\n\ny_pred = model_LeNet5_aug.predict(X_train_aug,batch_size=None, verbose=0)\n\nmask = np.argmax(Y_train_aug, axis=1) == np.argmax(y_pred, axis=1)\nx_hit = X_train_aug[mask]\ny_hit = Y_train_aug[mask]\nx_miss = X_train_aug[~mask]\ny_miss = Y_train_aug[~mask]\n\nnum_misses = y_miss.shape[0]\nsample_idx = sample(range(y_hit.shape[0]), y_miss.shape[0])  # random index samples from hits in the same amount of misses\nx_hit_reduced = x_hit[sample_idx] # choose random samples of hits with the same number of samples as misses\ny_hit_reduced = y_hit[sample_idx]\n\n# Create new training samples for network 2 with shuffled hit and miss samples\nX_train2 = np.append(x_miss,x_hit_reduced, axis=0)\nY_train2 = np.append(y_miss,y_hit_reduced, axis=0)\nidx_shuffle = list(range(X_train2.shape[0]))\nnp.random.shuffle(idx_shuffle)\nX_train2 = X_train2[idx_shuffle]\nY_train2 = Y_train2[idx_shuffle]\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Execute LeNet network 2\n\n# model_LeNet5_2 = LeNet5(activation='relu',pooling='max',dropout = 0.2, name=\"Net2\")\nmodel_LeNet5_2 = LeNet5BN(name=\"Net2\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train network 2\n\nearly_stop = EarlyStopping(patience=5, monitor='loss', restore_best_weights=True)\nbatch_size = 128\n\nmodel_LeNet5_2.fit (X_train2, Y_train2, \n                       epochs=100,\n                       callbacks=[early_stop])\n\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Build a new training set for network 3 based on differences between network 1 and network 2\n\ny_pred1 = model_LeNet5_aug.predict(X_train_aug, batch_size=None, verbose=0)\ny_pred2 = model_LeNet5_2.predict(X_train_aug, batch_size=None, verbose=0)\n\nmask = np.argmax(y_pred1, axis=1) == np.argmax(y_pred2, axis=1)\nX_train3 = X_train_aug[~mask] # Network3 train on samples with different predictions\nY_train3 = Y_train_aug[~mask]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Execute LeNet network 3\n\n# model_LeNet5_3 = LeNet5(activation='relu',pooling='max',dropout = 0.2, name=\"Net3\")\nmodel_LeNet5_3 = LeNet5BN(name=\"Net3\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train network 3\nearly_stop = EarlyStopping(patience=5, monitor='loss', restore_best_weights=True)\nbatch_size = 128\n\nmodel_LeNet5_3.fit (X_train3, Y_train3, \n                       epochs=100,\n                       callbacks=[early_stop])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get prediction result based on the sum of the 3 LeNet5 networks\n\ndef boosted_lenet5 (x):\n    pred1 = model_LeNet5_aug.predict(x,batch_size=None, verbose=0)\n    pred2 = model_LeNet5_2.predict(x,batch_size=None, verbose=0)\n    pred3 = model_LeNet5_3.predict(x,batch_size=None, verbose=0)\n\n    final_pred = np.argmax(pred1+pred2+pred3, axis = 1)\n    return final_pred","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Testing validation set on boosted model\n\nfinal_pred = boosted_lenet5(X_val)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Analyze results\n\nconf_mat, class_report, accuracy, faults = analyze_results(\"boosted LeNet5\", final_pred, np.argmax(Y_val, axis=1), X_val)\n\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Show the errors\n\nshow_faults(faults)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Summery","metadata":{}},{"cell_type":"code","source":"# Build Results table\nacc = [preds1[1], preds2[1], preds3[1], accuracy]\nmodels = ['LeNet5 Base', 'Improved', 'Augmented', 'Boosted LeNet5']\ntbl = {\"Model\": models, \"Accuracies\": acc}\ntbl_df = pd.DataFrame(tbl)\ntbl_df","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Conclusion\nBoosted LeNet-5 improved the accuracy of LeNet-5 in most of my runs (but not always!).","metadata":{}},{"cell_type":"markdown","source":"### Submitting result","metadata":{}},{"cell_type":"code","source":"test_label = boosted_lenet5(test)\nresult = {\"ImageId\":range(1,1+test_label.shape[0]) , \"Label\":test_label}\nsubmission = pd.DataFrame(result)\nsubmission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}