{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nOften considered the \"Hello World\" of machine learning the MNIST handwritten digits dataset consists of 60,000 grayscale images of size 28x28 pixels. The digits are laregly from American high school students. Each image is labelled according to the digit it represents in the range [0,9]. \n\nThis notebook will use TensorFlow via the Keras backend to employ a basic convolutional neural network (CNN) to solve this multi-class single-label image classification task. The topics areas discussed in this ntebook are:\n\n- Loading of images from pixel grayscale values.\n- Splitting the data into a training and validation set whilst avoiding biases and leakage.\n- Developing a sequential CNN using the Keras API and tuning hyperparameters to optimise performance.\n- Using data augmentation to reduce overfitting and increase the size of the training sample.\n- Understand how to critically analyse the performance of a CNN.\n- How to save and load models to avoid needless re-training.\n\nIf you found this notebook useful please consider giving it an upvote. ","metadata":{}},{"cell_type":"markdown","source":"# Loading Libraries","metadata":{}},{"cell_type":"code","source":"# Data I/O\nimport pandas as pd\nimport os\n\n# Math for linear/tensor algebra\nimport numpy as np\nimport tensorflow as tf\n\n# Machine learning specifics\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom keras.utils import to_categorical\nfrom sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay\n\n# Plotting and aesthetics\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2023-09-17T11:07:23.684793Z","iopub.execute_input":"2023-09-17T11:07:23.685407Z","iopub.status.idle":"2023-09-17T11:07:23.692800Z","shell.execute_reply.started":"2023-09-17T11:07:23.685373Z","shell.execute_reply":"2023-09-17T11:07:23.691303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reading Data\nHere I read in the data from the CSV files. One should take notice of the format of these images, we are provided with a dataframe with pixel grayscale values in the range [0, 255]. Currently this is a 'flattened' representation and we will need to reshape this later for use in the CNN.","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\n\n# Display a few entries of the dataframe to see what we are working with\ntrain.head(3)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:23:54.500249Z","iopub.execute_input":"2023-09-17T10:23:54.500838Z","iopub.status.idle":"2023-09-17T10:24:00.036016Z","shell.execute_reply.started":"2023-09-17T10:23:54.500804Z","shell.execute_reply":"2023-09-17T10:24:00.035075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check to see how many images are in each dataset\nprint(f\"Number of images in the training set = {len(train)} and in the test set = {len(test)}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:24:00.037787Z","iopub.execute_input":"2023-09-17T10:24:00.038425Z","iopub.status.idle":"2023-09-17T10:24:00.043362Z","shell.execute_reply.started":"2023-09-17T10:24:00.038390Z","shell.execute_reply":"2023-09-17T10:24:00.042447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we check for missing values and check the range of the values in the dataset.","metadata":{}},{"cell_type":"code","source":"print(f\"Missing training values = {np.count_nonzero(np.logical_or(train.isna().any(), train.isnull().any()))}\")\nprint(f\"Missing test values = {np.count_nonzero(np.logical_or(test.isna().any(), test.isnull().any()))}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:24:00.045762Z","iopub.execute_input":"2023-09-17T10:24:00.046364Z","iopub.status.idle":"2023-09-17T10:24:00.084806Z","shell.execute_reply.started":"2023-09-17T10:24:00.046333Z","shell.execute_reply":"2023-09-17T10:24:00.083717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Exploration\nLet's have a look at a few of the images and their labels to understand what these data really are. This should also allow us to build some intuition around the data.","metadata":{}},{"cell_type":"code","source":"# We know from the dataset description each row here is a flattened image with size 28 x 28\nX, y = train.drop(\"label\", axis=1).values, train[\"label\"].values # Drop is not in-place","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:24:56.951596Z","iopub.execute_input":"2023-09-17T10:24:56.951975Z","iopub.status.idle":"2023-09-17T10:24:57.042865Z","shell.execute_reply.started":"2023-09-17T10:24:56.951946Z","shell.execute_reply":"2023-09-17T10:24:57.041822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(0) # Let's set the seed to zero arbitrarily for reproducibility\n\n# Generate a selection of random images to view\nN_RANDOM_SAMPLES = 25\nimg_index = np.random.randint(0, X.shape[0], size=N_RANDOM_SAMPLES)\nimgs = X[img_index].reshape(N_RANDOM_SAMPLES, 28, 28, 1)\nimg_labels = y[img_index]\n\nfig, ax = plt.subplots(5, 5, figsize=(18, 18))\naxs = ax.flatten()\nfor i, img in enumerate(imgs):\n    axs[i].imshow(img, cmap=\"gray\")\n    axs[i].set_title(f\"{img_labels[i]}\")\n    axs[i].axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:24:58.622166Z","iopub.execute_input":"2023-09-17T10:24:58.622848Z","iopub.status.idle":"2023-09-17T10:25:00.598040Z","shell.execute_reply.started":"2023-09-17T10:24:58.622816Z","shell.execute_reply":"2023-09-17T10:25:00.597103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now understand the distribution of classes. We will need to weight our samples otherwise so our model doesn't skew towards or away from a particular class that it is under/over represented.","metadata":{}},{"cell_type":"code","source":"# The neatest way to do this is via seaborn directly\nlabels = ['Zero','One','Two', 'Three', 'Four', 'Five', 'Six', 'Seven', 'Eight', 'Nine']\nlabel, count = np.unique(y, return_counts=True)\nuniques = pd.DataFrame(data=count, index=labels, columns=['Count'])\n\nplt.figure(figsize=(14,4))\nsns.barplot(data=uniques, x=labels, y='Count',width=0.4).set_title('Class Distribution in Training Data',fontsize=15)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:06.805159Z","iopub.execute_input":"2023-09-17T10:30:06.805705Z","iopub.status.idle":"2023-09-17T10:30:07.219062Z","shell.execute_reply.started":"2023-09-17T10:30:06.805668Z","shell.execute_reply":"2023-09-17T10:30:07.218111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see from this distribution the 10 classes have roughly equal weight, each representing about 10% of the total data. This is excellent news since we probably don't need to weight our samples!","metadata":{}},{"cell_type":"markdown","source":"# Data Preparation\nTypical of machine learning problems the data must be prepared before we can use it. In this case this involved normalising the values (i.e. mapping the grayscale values [0,255] onto the domain [0,1]. We also elect to one-hot encode the target lables, this is not strictly necessary. If the labels were not one-hot encoded one would need to use the spare categorical crossentropy loss function in the call to ```model.compile```.","metadata":{}},{"cell_type":"code","source":"# Define functions to do the data preparation (ensures consistency given we are not using...\n# ...a pipeline)\n\ndef prepare_images(images):\n    return images.reshape(-1, 28, 28, 1).astype(np.float32) / 255.\n\ndef encode_labels(labels):\n    return tf.keras.utils.to_categorical(y)\n\nX = prepare_images(X)\ny = encode_labels(y)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:10.454854Z","iopub.execute_input":"2023-09-17T10:30:10.455348Z","iopub.status.idle":"2023-09-17T10:30:10.557261Z","shell.execute_reply.started":"2023-09-17T10:30:10.455308Z","shell.execute_reply":"2023-09-17T10:30:10.556070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Do the same for the test data now since we are going to be using it anyway later.","metadata":{}},{"cell_type":"code","source":"X_test = prepare_images(test.values)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:12.889870Z","iopub.execute_input":"2023-09-17T10:30:12.890236Z","iopub.status.idle":"2023-09-17T10:30:12.939854Z","shell.execute_reply.started":"2023-09-17T10:30:12.890202Z","shell.execute_reply":"2023-09-17T10:30:12.938761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split training data into a training & validation sample\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=0)\nprint(f\"Number of samples in training set = {y_train.shape[0]}\")\nprint(f\"Number of samples in validation set = {y_val.shape[0]}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:14.562998Z","iopub.execute_input":"2023-09-17T10:30:14.563858Z","iopub.status.idle":"2023-09-17T10:30:14.874316Z","shell.execute_reply.started":"2023-09-17T10:30:14.563812Z","shell.execute_reply":"2023-09-17T10:30:14.873197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the CNN\nThe model here is a final version after I spent considerable hours tuning and deciding on the best model architecture. If you look through some of the top notebooks on Kaggle you will find all tend towards an architecture similar to this.","metadata":{}},{"cell_type":"code","source":"model = Sequential([\n    Conv2D(32, (5,5), padding='same', activation='relu', input_shape=(28, 28, 1)),\n    Conv2D(32, (5,5), padding='same', activation='relu'),\n    MaxPool2D((2,2)),\n    Conv2D(64, (3,3), padding='same', activation='relu'),\n    Conv2D(64, (3,3), padding='same', activation='relu'),\n    MaxPool2D((2,2)),\n    Flatten(),\n    Dense(512, activation='relu'),\n    Dropout(0.25),\n    Dense(10, activation='softmax')\n])\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:17.864478Z","iopub.execute_input":"2023-09-17T10:30:17.864842Z","iopub.status.idle":"2023-09-17T10:30:20.930518Z","shell.execute_reply.started":"2023-09-17T10:30:17.864811Z","shell.execute_reply":"2023-09-17T10:30:20.929761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Using Callbacks\nA great way to quickly converge on an optimal solution is to use callbacks. These can adjust hyperparameters during training to help prevent overfitting or for your model to converge on a minimum loss. In the case of this notebook we elect to use 2 callbacks:\n\n- **EarlyStopping**: Automatically stops training after a chosen metric fails to improve after a specified number of epochs (referred to as the patience of the callback).\n- **ReduceLROnPlateau**: Reduce the learning-rate of the model by a given factor when a specific metric has failed to improve in a given number of epochs. This effectively makes the gradient descent algorithm make smaller 'steps' in the training hyperplane with the idea being it should be easier to converge on the minimum your model has already approached (rather than overshooting it with a large learning rate).","metadata":{}},{"cell_type":"code","source":"# Custom optimizer\noptimizer = Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)\n\n# Define custom callbacks\nlr_callback = ReduceLROnPlateau(\n    monitor='val_accuracy', \n    patience=3,\n    factor=0.5, \n    min_delta=1e-6,\n    min_lr=1e-6,\n    verbose=1\n)\n\nes_callback = EarlyStopping(\n    monitor='val_loss', \n    min_delta=1e-7,\n    patience=5, \n    restore_best_weights=True\n)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:34.928520Z","iopub.execute_input":"2023-09-17T10:30:34.928901Z","iopub.status.idle":"2023-09-17T10:30:34.935466Z","shell.execute_reply.started":"2023-09-17T10:30:34.928870Z","shell.execute_reply":"2023-09-17T10:30:34.934288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:37.930777Z","iopub.execute_input":"2023-09-17T10:30:37.931317Z","iopub.status.idle":"2023-09-17T10:30:37.954068Z","shell.execute_reply.started":"2023-09-17T10:30:37.931282Z","shell.execute_reply":"2023-09-17T10:30:37.953069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Augmentation\nWe can exploit data augmentation to further improve our model, essentially making it more robust to overfitting. Augmentation involes manipulating your input in some way for example rotating, zooming and cropping in order to produce more training samples. This also helps to make your model translationally, rotationally and aspect invariant.","metadata":{}},{"cell_type":"code","source":"img_gen = ImageDataGenerator(\n    featurewise_center=False,\n    samplewise_center=False,\n    featurewise_std_normalization=False,\n    samplewise_std_normalization=False,\n    zca_whitening=False,\n    rotation_range=20, # Don't rotate more than this or 6's will look like 9's\n    width_shift_range=0.25,\n    height_shift_range=0.25,\n    horizontal_flip=False,\n    vertical_flip=False,\n    fill_mode='nearest'\n)\n\nimg_gen.fit(X_train)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:48.981578Z","iopub.execute_input":"2023-09-17T10:30:48.981969Z","iopub.status.idle":"2023-09-17T10:30:49.025112Z","shell.execute_reply.started":"2023-09-17T10:30:48.981940Z","shell.execute_reply":"2023-09-17T10:30:49.024182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's have a look at exactly what the augmentation is doing on a given sample\naugmented_img_idx = np.random.randint(low=0, high=X_train.shape[0], size=1)\nx = X_train[augmented_img_idx]\n\nfig, ax = plt.subplots(3, 3, figsize = (9, 9))\naxs = ax.flatten()\nfor i, batch in enumerate(img_gen.flow(x, batch_size=1)):\n    if i >= 9:\n        break\n    axs[i].imshow(batch[0], cmap = 'gray')\n    axs[i].axis('off')","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:51.385553Z","iopub.execute_input":"2023-09-17T10:30:51.386432Z","iopub.status.idle":"2023-09-17T10:30:51.804336Z","shell.execute_reply.started":"2023-09-17T10:30:51.386387Z","shell.execute_reply":"2023-09-17T10:30:51.803076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Fitting\nI will fit the model twice once without and once with augmentation to show the power this seemingly simple choice can make.","metadata":{}},{"cell_type":"code","source":"# Note I do not use the EarlyStopping callback here to make the graphic comparison easier\nhistory = model.fit(X_train, y_train, batch_size=128, epochs=30, \n                    validation_data=(X_val, y_val),\n                    steps_per_epoch=X_train.shape[0] // 128, \n                    callbacks = [lr_callback])","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:30:59.268171Z","iopub.execute_input":"2023-09-17T10:30:59.268560Z","iopub.status.idle":"2023-09-17T10:32:22.106388Z","shell.execute_reply.started":"2023-09-17T10:30:59.268531Z","shell.execute_reply":"2023-09-17T10:32:22.105329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'll just define quickly a utility function to plot the accuracy/loss values over the training period since I will want to do this a few times.","metadata":{}},{"cell_type":"code","source":"def plot_history(history):\n    nepochs = len(history.history['accuracy'])\n    epochs = np.arange(1, nepochs + 1, 1)\n    fig, ax = plt.subplots(1, 2, figsize=(10, 4), sharex=True)\n    ax[0].plot(epochs, history.history['accuracy'], color=\"blue\", linestyle=\"--\", label=\"Training Accuracy\")\n    ax[0].plot(epochs, history.history['val_accuracy'], color=\"red\", linestyle=\"-\", label=\"Validation Accuracy\")\n    ax[1].plot(epochs, history.history['loss'], color=\"blue\", linestyle=\"--\", label=\"Training Loss\")\n    ax[1].plot(epochs, history.history['val_loss'], color=\"red\", linestyle=\"-\", label=\"Validation Loss\")\n    ax[1].set_xlabel(\"Epoch\")\n    ax[0].set_ylabel(\"Accuracy\")\n    l0 = ax[0].legend(frameon=True, fancybox=False, edgecolor=\"k\")\n    l0.get_frame().set_alpha(1)\n    l1 = ax[1].legend(frameon=True, fancybox=False, edgecolor=\"k\")\n    l1.get_frame().set_alpha(1)\n    ax[0].grid()\n    ax[1].grid()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:37:49.555543Z","iopub.execute_input":"2023-09-17T10:37:49.556665Z","iopub.status.idle":"2023-09-17T10:37:49.567794Z","shell.execute_reply.started":"2023-09-17T10:37:49.556623Z","shell.execute_reply":"2023-09-17T10:37:49.566112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Results (Before Augmentation)\nWe can see from the plots you could probably call the job \"done\" at this point. There is little sign of overtraining and our model performs exceptionally well with ~99.5% accuracy on both the training and the validation data. But we might be able to do better with augmentation.\n\nRemember: the accuracy is a good metric to use in this case because our classes are not imbalanced.","metadata":{}},{"cell_type":"code","source":"plot_history(history)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:37:53.105665Z","iopub.execute_input":"2023-09-17T10:37:53.106027Z","iopub.status.idle":"2023-09-17T10:37:53.578964Z","shell.execute_reply.started":"2023-09-17T10:37:53.105997Z","shell.execute_reply":"2023-09-17T10:37:53.577717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Peak model validation accuracy = {np.max(history.history['val_accuracy']):.4f}\")\ntrain_acc_at_val_peak = history.history['accuracy'][np.argmax(history.history['val_accuracy'])]\nprint(f\"Model training accuracy at validation peak = {train_acc_at_val_peak:.4f}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:37:56.786984Z","iopub.execute_input":"2023-09-17T10:37:56.787914Z","iopub.status.idle":"2023-09-17T10:37:56.794185Z","shell.execute_reply.started":"2023-09-17T10:37:56.787865Z","shell.execute_reply":"2023-09-17T10:37:56.793232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see the problem here instantly from the plots - our model is overfitting to the training data. This means the model does not generalize well (as can be seen from the validation metrics). Augmentation should help prevent this.","metadata":{}},{"cell_type":"markdown","source":"# Model Fitting (With Augmentation)","metadata":{}},{"cell_type":"code","source":"# Make a new model with the same architecture because tensorflow will re-train \nmodel2 = Sequential([\n    Conv2D(32, (5,5), padding='same', activation='relu', input_shape=(28, 28, 1)),\n    Conv2D(32, (5,5), padding='same', activation='relu'),\n    MaxPool2D((2,2)),\n    Conv2D(64, (3,3), padding='same', activation='relu'),\n    Conv2D(64, (3,3), padding='same', activation='relu'),\n    MaxPool2D((2,2)),\n    Flatten(),\n    Dense(512, activation='relu'),\n    Dropout(0.25),\n    Dense(10, activation='softmax')\n])\n\nmodel2.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:38:46.695888Z","iopub.execute_input":"2023-09-17T10:38:46.696284Z","iopub.status.idle":"2023-09-17T10:38:46.804436Z","shell.execute_reply.started":"2023-09-17T10:38:46.696253Z","shell.execute_reply":"2023-09-17T10:38:46.803642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:39:31.383764Z","iopub.execute_input":"2023-09-17T10:39:31.384127Z","iopub.status.idle":"2023-09-17T10:39:31.399395Z","shell.execute_reply.started":"2023-09-17T10:39:31.384096Z","shell.execute_reply":"2023-09-17T10:39:31.398463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aug_history = model2.fit(img_gen.flow(X_train, y_train, batch_size=128), epochs=30, \n                             validation_data=(X_val, y_val),\n                             steps_per_epoch=X_train.shape[0] // 128, \n                             callbacks = [lr_callback])","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:39:33.272051Z","iopub.execute_input":"2023-09-17T10:39:33.272438Z","iopub.status.idle":"2023-09-17T10:47:04.781361Z","shell.execute_reply.started":"2023-09-17T10:39:33.272408Z","shell.execute_reply":"2023-09-17T10:47:04.780069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history(aug_history)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:47:04.783427Z","iopub.execute_input":"2023-09-17T10:47:04.783898Z","iopub.status.idle":"2023-09-17T10:47:05.230008Z","shell.execute_reply.started":"2023-09-17T10:47:04.783862Z","shell.execute_reply":"2023-09-17T10:47:05.229062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Peak model validation accuracy = {np.max(aug_history.history['val_accuracy']):.4f}\")\ntrain_acc_at_val_peak = aug_history.history['accuracy'][np.argmax(aug_history.history['val_accuracy'])]\nprint(f\"Model training accuracy at validation peak = {train_acc_at_val_peak:.4f}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:47:34.022984Z","iopub.execute_input":"2023-09-17T10:47:34.023363Z","iopub.status.idle":"2023-09-17T10:47:34.029937Z","shell.execute_reply.started":"2023-09-17T10:47:34.023332Z","shell.execute_reply":"2023-09-17T10:47:34.028891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interestingly we actually seemed to do noticeably worse with augmentation! Though given the statistics the validation set is being evaluated on (~8000 images) an 0.0019 percentage difference in accuracy classification amounts to about 1 image.","metadata":{}},{"cell_type":"markdown","source":"# Save Model\nThe TensorFlow reccomended format is the new ```.keras``` format. ","metadata":{}},{"cell_type":"code","source":"model2.save('/kaggle/working/model.keras')","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:47:44.654114Z","iopub.execute_input":"2023-09-17T10:47:44.655208Z","iopub.status.idle":"2023-09-17T10:47:44.731128Z","shell.execute_reply.started":"2023-09-17T10:47:44.655139Z","shell.execute_reply":"2023-09-17T10:47:44.730111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Evaluation\nFinally a very brief evaluation of the model performance so we can understand how our model truly performs. In any real-world application this truly is the must crucial step. As an example in Particle Physics understanding the defecits of your model is crucial in discovering New Physics phenomena.","metadata":{}},{"cell_type":"code","source":"# Load in the previously trained model\nloaded_model = tf.keras.models.load_model('/kaggle/working/model.keras')","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:47:59.351779Z","iopub.execute_input":"2023-09-17T10:47:59.352389Z","iopub.status.idle":"2023-09-17T10:47:59.619570Z","shell.execute_reply.started":"2023-09-17T10:47:59.352345Z","shell.execute_reply":"2023-09-17T10:47:59.618478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# First we need to generate predictions so we can evaluate the model\ny_train_pred = np.argmax(loaded_model.predict(X_train), axis=1)\ny_val_pred = np.argmax(loaded_model.predict(X_val), axis=1)\ny_test_pred = np.argmax(loaded_model.predict(X_test), axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:56:54.219301Z","iopub.execute_input":"2023-09-17T10:56:54.219856Z","iopub.status.idle":"2023-09-17T10:56:59.776687Z","shell.execute_reply.started":"2023-09-17T10:56:54.219824Z","shell.execute_reply":"2023-09-17T10:56:59.775693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we actually look at these predictions they are an array with shape (n,10) where the 10 classes have 0 or 1 classifications. This is why we can lazily apply the argmax to get the singular class predicted value.","metadata":{}},{"cell_type":"code","source":"# Display a classification report (useful to get a quick overview of model performance)\nprint(classification_report(np.argmax(y_val, axis=1), y_val_pred, target_names=labels))","metadata":{"execution":{"iopub.status.busy":"2023-09-17T10:57:17.698641Z","iopub.execute_input":"2023-09-17T10:57:17.699091Z","iopub.status.idle":"2023-09-17T10:57:17.730362Z","shell.execute_reply.started":"2023-09-17T10:57:17.699055Z","shell.execute_reply":"2023-09-17T10:57:17.729319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our model is incredibly accurate the precision and recall and f1-score (the harmonic mean between both precision and recall) is ~1 for all categories. We can be confident this model will perform well when presented with the test data.","metadata":{}},{"cell_type":"code","source":"# Let's get the indices of all the models incorrect predictions in the validation data\nmask = ~(y_val_pred == np.argmax(y_val, axis=1))\nprint(f\"Number of validation images incorrectly classified = {np.count_nonzero(mask)}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-17T11:02:02.280659Z","iopub.execute_input":"2023-09-17T11:02:02.281015Z","iopub.status.idle":"2023-09-17T11:02:02.287445Z","shell.execute_reply.started":"2023-09-17T11:02:02.280985Z","shell.execute_reply":"2023-09-17T11:02:02.286161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have a mask of all the incorrectly classified images we can take a look at some of them to see if what the model is struggling with - or whether the handwriting really is that bad.","metadata":{}},{"cell_type":"code","source":"# Index of all images wrongly classified\nwrong_indices = np.where(mask)[0]\n\n# Pick 9 random samples\nsamples = np.random.choice(wrong_indices, size=9, replace=False)\n\nfig, ax = plt.subplots(3, 3, figsize=(9, 9))\naxs = ax.flatten()\nfor i, j in enumerate(samples):\n    axs[i].imshow(X_val[j], cmap=\"gray\")\n    axs[i].axis('off')\n    axs[i].set_title(f\"Pred={y_val_pred[j]} True={np.argmax(y_val, axis=1)[j]}\", fontsize=14)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T11:03:16.817882Z","iopub.execute_input":"2023-09-17T11:03:16.818271Z","iopub.status.idle":"2023-09-17T11:03:17.536708Z","shell.execute_reply.started":"2023-09-17T11:03:16.818239Z","shell.execute_reply":"2023-09-17T11:03:17.535106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Even to a human like me, what the model has got wrong is fairly understandable. A lot of these pieces of handwriting are difficult to read and the models prediction seems quite reasonable.\n\nNow we can look at the classification (confusion) matrix. This gives us a fast representation of how the model is classifyning each class. E.g. are the '6' digits often wrongly classified as a '9' when they are misclassified? Let's have a look...","metadata":{}},{"cell_type":"code","source":"confusion_mtx = confusion_matrix(np.argmax(y_val, axis=1), y_val_pred) \n\nsns.set_style('ticks')\nf,ax = plt.subplots(figsize = (20,8),dpi=200)\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.1, cmap = \"gist_yarg_r\", linecolor=\"black\", fmt='.0f', ax=ax,cbar=False, xticklabels=labels, yticklabels=labels)\n\nplt.xlabel(\"Predicted Label\",fontsize=10)\nplt.ylabel(\"True Label\",fontsize=10)\nplt.title(\"Confusion Matrix\",fontsize=13)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-17T11:08:22.995319Z","iopub.execute_input":"2023-09-17T11:08:22.995883Z","iopub.status.idle":"2023-09-17T11:08:24.183034Z","shell.execute_reply.started":"2023-09-17T11:08:22.995842Z","shell.execute_reply":"2023-09-17T11:08:24.182025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"# See what the submission should look like\nsubmission_sample = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\nsubmission_sample.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-12T19:52:45.499686Z","iopub.execute_input":"2023-09-12T19:52:45.505429Z","iopub.status.idle":"2023-09-12T19:52:45.548123Z","shell.execute_reply.started":"2023-09-12T19:52:45.505380Z","shell.execute_reply":"2023-09-12T19:52:45.546979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make predictions CSV and save it\nsubmission = pd.DataFrame(columns=[\"ImageId\", \"Label\"])\nsubmission[\"Label\"] = y_test_pred\nsubmission[\"ImageId\"] = np.arange(1, 28001, 1)\nsubmission.to_csv('/kaggle/working/submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-09-17T11:11:35.708088Z","iopub.execute_input":"2023-09-17T11:11:35.709177Z","iopub.status.idle":"2023-09-17T11:11:35.775907Z","shell.execute_reply.started":"2023-09-17T11:11:35.709109Z","shell.execute_reply":"2023-09-17T11:11:35.774940Z"},"trusted":true},"execution_count":null,"outputs":[]}]}