{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Neural Networks by example\nWelcome to my first notebook on Kaggle. I did record my notes so it might help others in their journey to understand Neural Networks by examples (in this case using the famous MNIST handwritten digits example, the \"Hello world\" of Deep Learning.) After seeing many youtube video's and various courses on Neural Networks found the Kaggle Keras course and examples helping me a lot to move from powerpoint understanding to run my own Neural Networks. Many thanks to this community! The least I could do is to contribute back, hence this notebook.\n
\n
Image from Josef Steppan on Wikipedia
\n \nFor starters would like to recommend a couple of video's: \nFound these series of 3Blue1Brown videos that explain Neural Networks very nice using the handwritten digits recognition example \n[https://www.youtube.com/watch?v=aircAruvnKk!](https://www.youtube.com/watch?v=aircAruvnKk!) \nThere is also an online book available: \n[http://neuralnetworksanddeeplearning.com/chap1.html](http://neuralnetworksanddeeplearning.com/chap1.html) \nAlso watch this video \"Tensorflow without a PHD\", it is a long one (2.5 hours) and it gives you an fantastic overview (don't try to remember all details immediately), it inspired me in the steps taken in this blog: \n[https://www.youtube.com/watch?v=vq2nnJ4g6N0!](https://www.youtube.com/watch?v=vq2nnJ4g6N0!) \nAnd finally the deep learning course on Kaggle is fantastic and a must-do: \n[https://www.kaggle.com/learn/deep-learning](https://www.kaggle.com/learn/deep-learning) \nI would like to acknowledge this tutorial for providing ideas and code, learning by example: \n[https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6/notebook](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6/notebook) \nAnd this notebook on CNN hyper parameter tuning: \n[https://www.kaggle.com/charel/how-to-choose-cnn-architecture-mnist](https://www.kaggle.com/charel/how-to-choose-cnn-architecture-mnist)\n\n \n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true}},{"cell_type":"markdown","source":"# Table of Contents \n\nIn the examples below we move through various implementation solutions to improve accuracy. \n1. [Dense Neural Network (95% accuracy)](#sec1) \n1. [Convolutional Neural Network (98% accuracy)](#sec2) \n1. [Convolutional Neural Network with data augmentation, dropout, etc (99.3% accuracy)](#sec3) \n1. [Hyper parameter tuning (99.57% accuracy)](#sec4) \n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport itertools\n\nfrom tensorflow.python import keras\nfrom tensorflow.python.keras.models import Sequential\nfrom tensorflow.keras import layers\nfrom tensorflow.python.keras.layers import Dense, Flatten, Conv2D, Dropout, BatchNormalization, MaxPool2D\nfrom tensorflow.python.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras.utils import to_categorical # convert to one-hot-encoding\nfrom tensorflow.keras.optimizers import RMSprop\n\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nnp.random.seed()\nsns.set(style='white', context='notebook', palette='deep')\n\nprint(\"Done\")","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.status.busy":"2022-01-13T15:52:11.982109Z","iopub.execute_input":"2022-01-13T15:52:11.982696Z","iopub.status.idle":"2022-01-13T15:52:15.038724Z","shell.execute_reply.started":"2022-01-13T15:52:11.982648Z","shell.execute_reply":"2022-01-13T15:52:15.037443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the training data\ndataset = pd.read_csv(\"../input/train.csv\")\n\n#Load the test data for the competition submission\ncompetition_dataset = pd.read_csv(\"../input/test.csv\")\n\ndataset.describe\n\n# Note that Tensorflow Keras nowadays has some readiliy available datasets (https://keras.io/api/datasets/) \n# including the MNIST dataset. So, you could simply import it as:\n\n# (train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()","metadata":{"_uuid":"c77d91789a40b502535c57d635539e13a9ec6e96","execution":{"iopub.status.busy":"2022-01-13T15:52:18.217408Z","iopub.execute_input":"2022-01-13T15:52:18.217878Z","iopub.status.idle":"2022-01-13T15:52:23.944573Z","shell.execute_reply.started":"2022-01-13T15:52:18.217792Z","shell.execute_reply":"2022-01-13T15:52:23.943555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A label is the thing we're predicting\nlabel = dataset[\"label\"]\n\n# A feature is an input variable, in this case a 28 by 28 pixels image\n# Drop 'label' column\nfeature = dataset.drop(labels = [\"label\"],axis = 1)\n\n# let's check we have a good distribution of the handwritten digits\ng = sns.countplot(label)\n\n","metadata":{"_uuid":"dd70309efda8436918f80b4e2665886059a67ad6","execution":{"iopub.status.busy":"2022-01-13T15:52:31.482603Z","iopub.execute_input":"2022-01-13T15:52:31.482993Z","iopub.status.idle":"2022-01-13T15:52:31.873772Z","shell.execute_reply.started":"2022-01-13T15:52:31.482922Z","shell.execute_reply":"2022-01-13T15:52:31.872978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# free some space\ndel dataset \n\nprint(\"Done\")","metadata":{"_uuid":"8c77700bed4ff4c64e5745b4d24e88c09508de08","execution":{"iopub.status.busy":"2022-01-13T15:52:36.178562Z","iopub.execute_input":"2022-01-13T15:52:36.178896Z","iopub.status.idle":"2022-01-13T15:52:36.184199Z","shell.execute_reply.started":"2022-01-13T15:52:36.178831Z","shell.execute_reply":"2022-01-13T15:52:36.183413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\" This shows 6 random images with their labels\"\"\"\nnrows = 2\nncols = 3\nfig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\nfor row in range(nrows):\n for col in range(ncols):\n rand_example = np.random.choice(feature.index)\n ax[row,col].imshow(feature.loc[rand_example].values.reshape((28,28)), cmap='gray_r')\n ax[row,col].set_title(\"Label: {}\".format(label.loc[rand_example]))","metadata":{"_uuid":"4bfd8ab4cd477352c7b0d7fe3bc5b7533df36d81","execution":{"iopub.status.busy":"2022-01-13T15:52:38.952911Z","iopub.execute_input":"2022-01-13T15:52:38.953263Z","iopub.status.idle":"2022-01-13T15:52:39.520053Z","shell.execute_reply.started":"2022-01-13T15:52:38.953203Z","shell.execute_reply":"2022-01-13T15:52:39.519125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])\nlabel = to_categorical(label, num_classes = 10)\n\n# Normalize between 0 and 1 the data (The pixel-value is an integer between 0 and 255)\nfeature = feature / 255.0\ncompetition_dataset = competition_dataset / 255.0\n\nprint(\"Done\")","metadata":{"_uuid":"300fe0d56cdbb305e9b0bcbef2edf8b454cb886c","execution":{"iopub.status.busy":"2022-01-13T15:52:48.441869Z","iopub.execute_input":"2022-01-13T15:52:48.442216Z","iopub.status.idle":"2022-01-13T15:52:49.810743Z","shell.execute_reply.started":"2022-01-13T15:52:48.442162Z","shell.execute_reply":"2022-01-13T15:52:49.809778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Model 1 - Dense Neural Network \n\n","metadata":{"_uuid":"515405e98cfc9a15267d86f0145615fb4a7a2de2"}},{"cell_type":"code","source":"# Split the dataset into train and validation set\n# Keep 10% for the validation and 90% for the training\n# Stratify is argument to keep trainingset evenly balanced ofver the labels (eg validation set not only the digit 5)\n\nfeature_train, feature_val, label_train, label_val = train_test_split(feature, label, test_size = 0.1, stratify=label)","metadata":{"_uuid":"4939a182b48bf191d4c7db97ba4ba2c5834d74b9","execution":{"iopub.status.busy":"2022-01-13T15:52:54.051303Z","iopub.execute_input":"2022-01-13T15:52:54.051682Z","iopub.status.idle":"2022-01-13T15:52:55.017509Z","shell.execute_reply.started":"2022-01-13T15:52:54.051618Z","shell.execute_reply":"2022-01-13T15:52:55.016458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# First model is a dense neural network model with 5 layers\ninputs = keras.Input(shape=(784,))\nx = layers.Dense(200, activation='relu')(inputs)\nx = layers.Dense(100, activation='relu')(x)\nx = layers.Dense(60, activation='relu')(x)\nx = layers.Dense(30, activation='relu')(x)\noutputs = layers.Dense(10, activation='softmax')(x)\nmodel_1 = keras.Model(inputs,outputs, name='Dense neural network')\n\n'''\n# Since this is a simple sequential model it could also have been specified as \nmodel_1 = Sequential()\nmodel_1.add(Dense(200, activation = \"relu\", input_shape = (784,)))\nmodel_1.add(Dense(100, activation = \"relu\"))\nmodel_1.add(Dense(60, activation = \"relu\"))\nmodel_1.add(Dense(30, activation = \"relu\"))\nmodel_1.add(Dense(10, activation = \"softmax\"))\n'''\n\n# Define the optimizer and compile the model\noptimizer = optimizers.SGD(lr=0.03, clipnorm=5.)\nmodel_1.compile(optimizer= optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])\n\n# Print the model\nprint (model_1.summary())\n","metadata":{"_uuid":"f416f53abb559e2b398506fb11b968a37b47661c","execution":{"iopub.status.busy":"2022-01-13T16:08:12.912944Z","iopub.execute_input":"2022-01-13T16:08:12.913315Z","iopub.status.idle":"2022-01-13T16:08:13.090523Z","shell.execute_reply.started":"2022-01-13T16:08:12.913255Z","shell.execute_reply":"2022-01-13T16:08:13.089705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Or show the model in a plotted graph\nkeras.utils.plot_model(model_1, \"Dense Neural Network.png\",show_shapes=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-13T16:08:29.298063Z","iopub.execute_input":"2022-01-13T16:08:29.298621Z","iopub.status.idle":"2022-01-13T16:08:29.456628Z","shell.execute_reply.started":"2022-01-13T16:08:29.298518Z","shell.execute_reply":"2022-01-13T16:08:29.455708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# With this model you should be able to achieve around 95.5% accuracy\n\nhistory = model_1.fit(feature_train, label_train, batch_size = 100, epochs = 8, \n validation_data = (feature_val, label_val), verbose = 1)\n","metadata":{"_uuid":"f67ebab43ca975e5817a091a57b9b486c3a46701","execution":{"iopub.status.busy":"2022-01-13T16:05:15.043825Z","iopub.execute_input":"2022-01-13T16:05:15.044245Z","iopub.status.idle":"2022-01-13T16:05:40.487421Z","shell.execute_reply.started":"2022-01-13T16:05:15.044178Z","shell.execute_reply":"2022-01-13T16:05:40.486674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","metadata":{"_uuid":"70a02842afdca037ad142ee33251257a1e07b971","execution":{"iopub.status.busy":"2022-01-13T16:05:51.698264Z","iopub.execute_input":"2022-01-13T16:05:51.698952Z","iopub.status.idle":"2022-01-13T16:05:52.022667Z","shell.execute_reply.started":"2022-01-13T16:05:51.698880Z","shell.execute_reply":"2022-01-13T16:05:52.021680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check some of this data we got wrong. Below the code to get the confusion matrix ","metadata":{"_uuid":"fa189e26157014458016d81737764c116bac5338"}},{"cell_type":"code","source":"# Look at confusion matrix, the visualization of the performance of an algorithm \n# SW gracefully copied from other notebooks on Kaggle\n\ndef plot_confusion_matrix(cm, classes,\n normalize=False,\n title='Confusion matrix',\n cmap=plt.cm.Blues):\n \"\"\"\n This function prints and plots the confusion matrix.\n Normalization can be applied by setting `normalize=True`.\n \"\"\"\n plt.imshow(cm, interpolation='nearest', cmap=cmap)\n plt.title(title)\n plt.colorbar()\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n if normalize:\n cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n thresh = cm.max() / 2.\n for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n plt.text(j, i, cm[i, j],\n horizontalalignment=\"center\",\n color=\"white\" if cm[i, j] > thresh else \"black\")\n\n plt.tight_layout()\n plt.ylabel('True label')\n plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model_1.predict(feature_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(label_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","metadata":{"execution":{"iopub.status.busy":"2022-01-13T16:05:55.442168Z","iopub.execute_input":"2022-01-13T16:05:55.442523Z","iopub.status.idle":"2022-01-13T16:05:56.318208Z","shell.execute_reply.started":"2022-01-13T16:05:55.442461Z","shell.execute_reply":"2022-01-13T16:05:56.317280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Model 2 - Convolutional network\nStill only 95-96%, how come? \nAnswer: because we have mapped a 2D picture to a single long array, therefor loosing critical information. Convolution networks to the rescue for 2D images\n\n","metadata":{"_uuid":"c3b61d7d0e16f053ef1b2e5f757d27b2d7653bec","trusted":true}},{"cell_type":"code","source":"# First let's reshape the array into a 28*28 picture with 1 color channel (b/w picture)\n#Take a random example to print it before and after the conversion\nrand_example = np.random.choice(1000)\n_, ax = plt.subplots()\nax.imshow(feature.loc[rand_example].values.reshape(28, 28), cmap='gray_r')\nax.set_title(\"Before\")\nax.grid(False)\n\nfeature = feature.values.reshape(-1,28,28,1)\ncompetition_dataset = competition_dataset.values.reshape(-1,28,28,1)\n\n_, ax = plt.subplots()\ng = plt.imshow(feature[rand_example][:,:,0], cmap='gray_r')\nax.set_title(\"After\")\nax.grid(False)\n","metadata":{"_uuid":"27f077a9842ba3f6e6e64424ff083f44a689c7f3","execution":{"iopub.status.busy":"2022-01-13T16:24:43.944760Z","iopub.execute_input":"2022-01-13T16:24:43.945476Z","iopub.status.idle":"2022-01-13T16:24:44.259081Z","shell.execute_reply.started":"2022-01-13T16:24:43.945420Z","shell.execute_reply":"2022-01-13T16:24:44.257922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the dataset into train and validation set\n# Keep 10% for the validation and 90% for the training\n# Stratify is argument to keep trainingset evenly balanced ofver the labels (eg validation set not only the digit 5)\n\nfeature_train, feature_val, label_train, label_val = train_test_split(feature, label, test_size = 0.1, stratify=label)","metadata":{"_uuid":"e5534828c163d5805f37be8e9cbf5b2ef4fa2945","execution":{"iopub.status.busy":"2022-01-13T16:24:48.323208Z","iopub.execute_input":"2022-01-13T16:24:48.323567Z","iopub.status.idle":"2022-01-13T16:24:50.740675Z","shell.execute_reply.started":"2022-01-13T16:24:48.323517Z","shell.execute_reply":"2022-01-13T16:24:50.739009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Second model is a 3 layer convolutional network model with two dense layers at the end\n\ninputs = keras.Input(shape=(28,28,1))\nx = layers.Conv2D(filters = 4, kernel_size = (5,5), strides = 1, padding = 'Same', activation ='relu')(inputs)\nx = layers.Conv2D(filters = 8, kernel_size = (4,4), strides = 2, padding = 'Same', activation ='relu')(x)\nx = layers.Conv2D(filters = 12, kernel_size = (4,4), strides = 2, padding = 'Same', activation ='relu')(x)\nx = layers.Flatten()(x)\nx = layers.Dense(200, activation='relu')(x)\noutputs = layers.Dense(10, activation='softmax')(x)\nmodel_2 = keras.Model(inputs,outputs, name='Convolutional neural network')\n\n# Define the optimizer and compile the model\noptimizer = optimizers.SGD(lr=0.03, clipnorm=5.)\nmodel_2.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nprint (model_2.summary())\n","metadata":{"_uuid":"58c083510604fce028dea04d392ea4a29f068ace","execution":{"iopub.status.busy":"2022-01-13T16:25:14.920001Z","iopub.execute_input":"2022-01-13T16:25:14.920490Z","iopub.status.idle":"2022-01-13T16:25:15.109171Z","shell.execute_reply.started":"2022-01-13T16:25:14.920428Z","shell.execute_reply":"2022-01-13T16:25:15.108300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# With this model you should be able to achieve around 98% accuracy\n\n\nhistory = model_2.fit(feature_train, label_train, batch_size = 100, epochs = 16, \n validation_data = (feature_val, label_val), verbose = 1)\n","metadata":{"_uuid":"1bcb71b907f13517ec782295565cc9d3298d992a","execution":{"iopub.status.busy":"2022-01-13T16:25:19.462187Z","iopub.execute_input":"2022-01-13T16:25:19.462527Z","iopub.status.idle":"2022-01-13T16:28:22.540599Z","shell.execute_reply.started":"2022-01-13T16:25:19.462487Z","shell.execute_reply":"2022-01-13T16:28:22.539708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","metadata":{"_uuid":"e05a1fe99431fbe82614e1f9e4c2864e3d50187a","execution":{"iopub.status.busy":"2022-01-13T16:28:32.652636Z","iopub.execute_input":"2022-01-13T16:28:32.652967Z","iopub.status.idle":"2022-01-13T16:28:32.979219Z","shell.execute_reply.started":"2022-01-13T16:28:32.652927Z","shell.execute_reply":"2022-01-13T16:28:32.978444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at confusion matrix, the visualization of the performance of an algorithm \n\n# Predict the values from the validation dataset\nY_pred = model_2.predict(feature_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(label_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","metadata":{"execution":{"iopub.status.busy":"2022-01-13T16:28:39.565106Z","iopub.execute_input":"2022-01-13T16:28:39.565429Z","iopub.status.idle":"2022-01-13T16:28:40.902653Z","shell.execute_reply.started":"2022-01-13T16:28:39.565383Z","shell.execute_reply":"2022-01-13T16:28:40.901895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Model 3 Overfitting\nNoticed the gap between accuracy in the test and validation data, test data even reaching 99.9%? The model is training so well on the test data that it overfits. It is becoming so well specialized to the input dataset, it is actually getting worse for examples outside the test dataset. Solution directions: \n1) Need more data, Data augmentation with the Keras ImageGenerator to the rescue \n2) Overfitting techniques, Dropout to the rescue, applied significant drop-out in the 200 neurons dense layer \n3) More degrees of freedom, let's get a richer model \nLastly, I also applied a more advanced optimizer: Adam optimizer in this case\n\n\n","metadata":{"_uuid":"d4baffe9f9fae1151de886b17d16268f2559856d"}},{"cell_type":"code","source":"# Generate 22 million more images by randomly rotating, scaling, and shifting 42,000 (-10% validation set) images\ndatagen = ImageDataGenerator(\n rotation_range=10, # randomly rotate images in the range (degrees, 0 to 180)\n zoom_range = 0.1, # Randomly zoom image \n width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)\n height_shift_range=0.1, # randomly shift images vertically (fraction of total height)\n featurewise_center=False, # do not set input mean to 0 over the dataset\n samplewise_center=False, # do not set each sample mean to 0\n featurewise_std_normalization=False, # no divide inputs by std of the dataset\n samplewise_std_normalization=False, # no divide each input by its std\n zca_whitening=False, # No ZCA whitening\n horizontal_flip=False, # no horizontal flip images\n vertical_flip=False) # no vertical flip images, no 6 and 9 mismatches :-)\n\ndatagen.fit(feature_train)\n\nprint(\"Done\")","metadata":{"_uuid":"1680b47313e8a0d7384c1b0ab55eb6c47bd12256","execution":{"iopub.status.busy":"2022-01-13T16:29:33.205778Z","iopub.execute_input":"2022-01-13T16:29:33.206154Z","iopub.status.idle":"2022-01-13T16:29:33.401228Z","shell.execute_reply.started":"2022-01-13T16:29:33.206092Z","shell.execute_reply":"2022-01-13T16:29:33.400278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Third model is a 3 layer convolutional network model with one dense layer at the end, it contains more neurons, has dropout applied in the dense layer, \n# data augmentation and the adam optimizer\n\ninputs = keras.Input(shape=(28,28,1))\nx = layers.Conv2D(filters = 6, kernel_size = (6,6), strides = 1, padding = 'Same', activation ='relu')(inputs)\nx = layers.Conv2D(filters = 12, kernel_size = (5,5), strides = 2, padding = 'Same', activation ='relu')(x)\nx = layers.Conv2D(filters = 24, kernel_size = (4,4), strides = 2, padding = 'Same', activation ='relu')(x)\nx = layers.Flatten()(x)\nx = layers.Dense(200, activation='relu')(x)\nx = layers.Dropout(0.40)(x)\noutputs = layers.Dense(10, activation='softmax')(x)\nmodel_3 = keras.Model(inputs,outputs, name='Convolutional neural network with data augmentation')\n\n# Define the optimizer and compile the model\nmodel_3.compile(optimizer = 'adam' , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nprint (model_3.summary())","metadata":{"_uuid":"9d5e5f1722b605d599e6f8fb5009819c739843eb","execution":{"iopub.status.busy":"2022-01-13T16:31:33.418966Z","iopub.execute_input":"2022-01-13T16:31:33.419475Z","iopub.status.idle":"2022-01-13T16:31:33.632366Z","shell.execute_reply.started":"2022-01-13T16:31:33.419255Z","shell.execute_reply":"2022-01-13T16:31:33.631510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# With this model you should be able to achieve around 99.3% accuracy\n# change epochs to 35 to have a full run\n\nhistory = model_3.fit(datagen.flow(feature_train,label_train, batch_size=100),\n epochs = 8, validation_data = (feature_val, label_val),\n verbose = 2)","metadata":{"_uuid":"098b4ccb84cac9245e19741324503a628d41c054","execution":{"iopub.status.busy":"2022-01-13T16:32:16.100430Z","iopub.execute_input":"2022-01-13T16:32:16.100826Z","iopub.status.idle":"2022-01-13T16:34:56.604202Z","shell.execute_reply.started":"2022-01-13T16:32:16.100762Z","shell.execute_reply":"2022-01-13T16:34:56.603056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","metadata":{"_uuid":"c11e07c9db2345dac1737bdf85cf10ac59293c11","execution":{"iopub.status.busy":"2022-01-13T16:36:52.337335Z","iopub.execute_input":"2022-01-13T16:36:52.337682Z","iopub.status.idle":"2022-01-13T16:36:52.656750Z","shell.execute_reply.started":"2022-01-13T16:36:52.337644Z","shell.execute_reply":"2022-01-13T16:36:52.655877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look at confusion matrix, the visualization of the performance of an algorithm \n\n# Predict the values from the validation dataset\nY_pred = model_3.predict(feature_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(label_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","metadata":{"execution":{"iopub.status.busy":"2022-01-13T16:36:57.018362Z","iopub.execute_input":"2022-01-13T16:36:57.018688Z","iopub.status.idle":"2022-01-13T16:36:58.594529Z","shell.execute_reply.started":"2022-01-13T16:36:57.018641Z","shell.execute_reply":"2022-01-13T16:36:58.593678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Hyper parameter tuning\nNotice that gap between training accuracy and validation accuracy is gone, in fact the validation accuracy outperforms nicely the training loss. Also from 98% to 99.3% is a big step, long live the data augmentation. \nNext step is to tune the hyper parameters, how many layers? How many neurons in a layer? What type of optimizer? What learning rate? Levels of data augmentation? Etc etc\n\nThe hyperparameter tuning is still an \"art\" based on experience, best practices, etc. Some work is being do to automate this as well with eg auto-Keras\n\nSome practices I found on Kaggle and have applied below: \n* Take more filters to have better feature extraction, higher deeper in the network\n* Batch normalization between convolution layers helps\n* Dropout between convolution layers helps, 40% seems to be a good choice\n* Take a double Conv2D layer e.g. Instead of a 5x5 Conv2D layer take 2 3*3 sequential Conv2D layers\n* Either use alternating convolution layers (Conv2D) with MaxPool2D layers, or use Conv2D with strides 2\n\nNote: training (convolutional) neural networks is a random process. Each time you train you get different results. That makes experiments difficult , you must run your experiments dozens of times and take an average. \n\n\n","metadata":{"_uuid":"88e491494a99a7ae55f3b847ce461e5520667b80"}},{"cell_type":"code","source":"# Fourth model with hyper parameter tuning \n\ninputs = keras.Input(shape=(28,28,1))\nx = layers.Conv2D(filters = 32, kernel_size = (5,5), strides = 1, padding = 'Same', activation ='relu')(inputs)\nx = layers.BatchNormalization()(x)\nx = layers.Conv2D(filters = 32, kernel_size = (5,5), strides = 1, padding = 'Same', activation ='relu')(x)\nx = layers.BatchNormalization()(x)\nx = layers.Dropout(0.40)(x)\nx = layers.Conv2D(filters = 64, kernel_size = (3,3), strides = 2, padding = 'Same', activation ='relu')(x)\nx = layers.BatchNormalization()(x)\nx = layers.Conv2D(filters = 64, kernel_size = (3,3), strides = 2, padding = 'Same', activation ='relu')(x)\nx = layers.BatchNormalization()(x)\nx = layers.Dropout(0.40)(x)\nx = layers.Flatten()(x)\nx = layers.Dense(256, activation='relu')(x)\nx = layers.Dropout(0.40)(x)\noutputs = layers.Dense(10, activation='softmax')(x)\n\nmodel_4 = keras.Model(inputs,outputs, name='Convolutional neural network with hyper parameter tuning')\n\n# Define the optimizer and compile the model\noptimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)\nmodel_4.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])\n\n# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n patience=3, \n verbose=1, \n factor=0.5, \n min_lr=0.00001)\nprint (model_4.summary())","metadata":{"_uuid":"efdf7cfa1184799c84851e2a8751661cf537b50a","execution":{"iopub.status.busy":"2022-01-13T16:45:03.044030Z","iopub.execute_input":"2022-01-13T16:45:03.044371Z","iopub.status.idle":"2022-01-13T16:45:03.553270Z","shell.execute_reply.started":"2022-01-13T16:45:03.044331Z","shell.execute_reply":"2022-01-13T16:45:03.552155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# With this model you should be able to achieve around 99.6to 99.7% accuracy\n# change epochs to 35 to have a full run\n\nhistory = model_4.fit(datagen.flow(feature_train,label_train, batch_size=100),\n epochs = 35, validation_data = (feature_val, label_val),\n verbose = 2, callbacks=[learning_rate_reduction])","metadata":{"_uuid":"090a7b9118703b5f5b062ec2877b322c014efa3d","execution":{"iopub.status.busy":"2022-01-13T16:45:28.053625Z","iopub.execute_input":"2022-01-13T16:45:28.054228Z","iopub.status.idle":"2022-01-13T16:53:53.007359Z","shell.execute_reply.started":"2022-01-13T16:45:28.054177Z","shell.execute_reply":"2022-01-13T16:53:53.006297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Look at confusion matrix, the visualization of the performance of an algorithm \n\n\n# Predict the values from the validation dataset\nY_pred = model_4.predict(feature_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(label_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) \n","metadata":{"_uuid":"7a99a16cd18cf6c0953a50aec05cab5b1e0d7ca6","execution":{"iopub.status.busy":"2022-01-13T16:54:44.768856Z","iopub.execute_input":"2022-01-13T16:54:44.769227Z","iopub.status.idle":"2022-01-13T16:54:49.485117Z","shell.execute_reply.started":"2022-01-13T16:54:44.769176Z","shell.execute_reply":"2022-01-13T16:54:49.484017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not that many mistakes (and take a look below in the type of mistakes, I would argue also human operators would have a hard time to read them right and some of the True labels I would dispute). Anyhow reading the confusion matrix: 9 and 4 got confused as well as 9 and 7, makes sense that these got confused more often. Food for thought for further analyses in case you want to go for the world record. I'll stop here for this notebook, showed in a number of steps to get a Neural Network to > 99.6%. Hope you enjoyed it, if so please vote-up the kernel (top-right of the screen).","metadata":{"_uuid":"935e92a3ea93e9860cbe0c394c0c53c15145420f"}},{"cell_type":"code","source":"# Display some error results \n\n# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = feature_val[errors]\n\ndef display_errors(errors_index,img_errors,pred_errors, obs_errors):\n \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n n = 0\n nrows = 2\n ncols = 3\n fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n for row in range(nrows):\n for col in range(ncols):\n error = errors_index[n]\n ax[row,col].imshow((img_errors[error]).reshape((28,28)), cmap='gray_r')\n ax[row,col].set_title(\"Pred: {}; True: {}\".format(pred_errors[error],obs_errors[error]))\n n += 1\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[-6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","metadata":{"_uuid":"6f9a78138872c09eaca76140c04e02d499b7df34","execution":{"iopub.status.busy":"2022-01-13T16:54:55.592200Z","iopub.execute_input":"2022-01-13T16:54:55.592605Z","iopub.status.idle":"2022-01-13T16:54:56.171626Z","shell.execute_reply.started":"2022-01-13T16:54:55.592552Z","shell.execute_reply":"2022-01-13T16:54:56.170619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's submit the results to Kaggle.","metadata":{"_uuid":"df9fa2321e3684855f6642f2564c5774772fef60"}},{"cell_type":"code","source":"# predict results\nresults = model_4.predict(competition_dataset)\n\n# select the indix with the maximum probability\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","metadata":{"_uuid":"0d84997ddb0d5b5c3dffe91ad2e2ceb1d8e8c5e9","execution":{"iopub.status.busy":"2022-01-13T16:55:21.660395Z","iopub.execute_input":"2022-01-13T16:55:21.661109Z","iopub.status.idle":"2022-01-13T16:55:47.165802Z","shell.execute_reply.started":"2022-01-13T16:55:21.661037Z","shell.execute_reply":"2022-01-13T16:55:47.164787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"submission_MNIST.csv\",index=False)\n","metadata":{"_uuid":"93d09d393eb50eaff6bedf7614a7797870f3c390","_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-01-13T16:56:15.427229Z","iopub.execute_input":"2022-01-13T16:56:15.427645Z","iopub.status.idle":"2022-01-13T16:56:15.963494Z","shell.execute_reply.started":"2022-01-13T16:56:15.427589Z","shell.execute_reply":"2022-01-13T16:56:15.962640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hit the \"save version\" button at the top of your notebook screen and select \"Save & Run All (Commit)\". Wait for your kernel to complete it's run. View the newly created kernel (e.g. open it as a new notebook) and go to the tab output, there you will find the submit button.\n","metadata":{}}]}