{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit recognizer project","metadata":{}},{"cell_type":"markdown","source":"According to the documentation of this competition, we have:\nThe data files train.csv and test.csv contain gray-scale images of hand-drawn digits, from zero through nine.\n\nEach image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\nWe will build a custom CNN (Convolutional Neural Network) or also called convenet, using Tensorflow and Keras API. Finally, to understand how our CNN work we will also used the shap library.","metadata":{}},{"cell_type":"code","source":"# Imports\nimport os\nimport random\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sb\nimport shap\nimport tensorflow as tf\n\nfrom keras.callbacks import EarlyStopping\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow import keras","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:24:15.623582Z","iopub.execute_input":"2023-06-25T20:24:15.624348Z","iopub.status.idle":"2023-06-25T20:24:28.316733Z","shell.execute_reply.started":"2023-06-25T20:24:15.624308Z","shell.execute_reply":"2023-06-25T20:24:28.315478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Functions\ndef show_image(data: np.array, target: np.array) -> None:\n    \"\"\"\n    This function take the a random image from the \n    the array that contains images, and plot it,\n    with as title the truth value of the image.\n    \"\"\"\n    # Reshape the data, to make 28x28 images\n    images = data.reshape(-1, 28, 28) \n    # -1 tells numpy that the value must be inferred \n    # from the length of the array and remaining dimensions.\n    \n    idx = random.randint(0, len(images))\n    \n    # Plot a random image\n    plt.figure(figsize=(3,3))\n    plt.imshow(images[idx])\n    plt.title(f\"Real number: {target[idx]}\")\n    plt.show()\n    \ndef predict_image(data: np.array, target: np.array) -> None:\n    \"\"\"\n    This function take the a random image from the \n    the array that contains test images, and plot it,\n    with as title the predicted value of the image.\n    \"\"\"\n    # Reshape the data, to make 28x28 images\n    images = data.reshape(-1, 28, 28) \n    # -1 tells numpy that the value must be inferred \n    # from the length of the array and remaining dimensions.\n    \n    idx = random.randint(0, len(images))\n    \n    # Plot a random image\n    plt.figure(figsize=(3,3))\n    plt.imshow(images[idx])\n    plt.title(f\"Predicted number: {target[idx]}\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:24:32.083675Z","iopub.execute_input":"2023-06-25T20:24:32.085162Z","iopub.status.idle":"2023-06-25T20:24:32.095843Z","shell.execute_reply.started":"2023-06-25T20:24:32.085111Z","shell.execute_reply":"2023-06-25T20:24:32.094676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"# Check files avaliable\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:24:34.891530Z","iopub.execute_input":"2023-06-25T20:24:34.892121Z","iopub.status.idle":"2023-06-25T20:24:34.901051Z","shell.execute_reply.started":"2023-06-25T20:24:34.892086Z","shell.execute_reply":"2023-06-25T20:24:34.899767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read train, test and submission csv\ndf_train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndf_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\ndf_submission = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\n\n# Display dimensions of each dataframe\nprint(f\"The dimension of the train df is: {df_train.shape}\")\nprint(f\"The dimension of the test df is: {df_test.shape}\")\nprint(f\"The dimension of the submission df is: {df_submission.shape}\")","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:24:36.680620Z","iopub.execute_input":"2023-06-25T20:24:36.681301Z","iopub.status.idle":"2023-06-25T20:24:41.937742Z","shell.execute_reply.started":"2023-06-25T20:24:36.681264Z","shell.execute_reply":"2023-06-25T20:24:41.936634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we have the 28k images in the test sample that need to be predict. The result will be store in the submission dataframe, and return as csv to kaggle for evaluation.","metadata":{}},{"cell_type":"code","source":"# Repartition of the classes across the dataset\nplt.figure(figsize=(12,5))\nsb.countplot(data=df_train, x='label')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:24:50.981523Z","iopub.execute_input":"2023-06-25T20:24:50.982025Z","iopub.status.idle":"2023-06-25T20:24:51.344018Z","shell.execute_reply.started":"2023-06-25T20:24:50.981966Z","shell.execute_reply":"2023-06-25T20:24:51.342821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have a fairly balanced dataset here, no need to apply some methods to deal with imbalanced dataset.","metadata":{}},{"cell_type":"code","source":"# Extract the target from the data of the train dataframe\n# And convert into numpy array\nX_train = df_train.drop(labels='label', axis=1).values\ny_train = df_train['label'].values\nX_test = df_test.values","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:25:23.250436Z","iopub.execute_input":"2023-06-25T20:25:23.250998Z","iopub.status.idle":"2023-06-25T20:25:23.348147Z","shell.execute_reply.started":"2023-06-25T20:25:23.250933Z","shell.execute_reply":"2023-06-25T20:25:23.346928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display an image as example\nshow_image(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:25:24.934444Z","iopub.execute_input":"2023-06-25T20:25:24.935530Z","iopub.status.idle":"2023-06-25T20:25:25.115510Z","shell.execute_reply.started":"2023-06-25T20:25:24.935453Z","shell.execute_reply":"2023-06-25T20:25:25.114464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"markdown","source":"The first step here is to reshape the data. Since the first convolutional layer from Keras need a 3D input, we will use the reshape function from Numpy and turn the train and test set to 28x28x1 tensor.","metadata":{}},{"cell_type":"code","source":"# Reshape the data\n# We need to turn them to 28x28x1 tensors because the first convolutional layer needs a 3D input.\nX_train = X_train.reshape(-1, 28, 28, 1)\nX_test = X_test.reshape(-1, 28, 28, 1)\n\n# Check the shape of the train set\nX_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:25:27.800511Z","iopub.execute_input":"2023-06-25T20:25:27.801587Z","iopub.status.idle":"2023-06-25T20:25:27.810775Z","shell.execute_reply.started":"2023-06-25T20:25:27.801499Z","shell.execute_reply":"2023-06-25T20:25:27.809413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In a typical gray scale image, white pixels are encoded as the value 255, and black pixels are encoded as zero. We will re-scale the data, so that each image has pixel values in range from zero to one. To go from a range of 0–255 to zero to one, you just have to divide every pixel value by 255. \n\nThis step is called normalization, and it’s common practice in many deep learning techniques. The reason we typically want normalized pixel values is because neural networks rely on gradient calculations. It will helps these gradient calculations stay consistent, and not get so large that they slow down the training.","metadata":{}},{"cell_type":"code","source":"# Basic normalization\nX_train = X_train / 255.0\nX_test = X_test / 255.0","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:25:29.670234Z","iopub.execute_input":"2023-06-25T20:25:29.670973Z","iopub.status.idle":"2023-06-25T20:25:29.845087Z","shell.execute_reply.started":"2023-06-25T20:25:29.670934Z","shell.execute_reply":"2023-06-25T20:25:29.843972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Last step is devide our data to a train and validation set. It is a very important step, since the validation set will help us to analyze any overfitting by the model.","metadata":{}},{"cell_type":"code","source":"# Split the data into a train and validation set\n(X_train,\n X_val,\n y_train,\n y_val) = train_test_split(X_train, y_train, test_size=0.1, random_state=123)\n\n# Check shape of the data\nprint(X_train.shape)\nprint(X_val.shape)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:25:32.308383Z","iopub.execute_input":"2023-06-25T20:25:32.309244Z","iopub.status.idle":"2023-06-25T20:25:32.683680Z","shell.execute_reply.started":"2023-06-25T20:25:32.309204Z","shell.execute_reply":"2023-06-25T20:25:32.682403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CNN modeling","metadata":{}},{"cell_type":"markdown","source":"We will use Keras and Tensorflow to build our Convolutional Neural Network (CNN). Convolution is the mathematical operation that gives the layers of a CNN their unique structure. A CNN used for image classification consists of two parts: a convolutional base and a dense head. \n\nThe base is used to extract the features from an image. It is formed primarily of layers performing the convolution operation, and the pooling operation.\n\nThe head is used to determine the class of the image. It is formed primarily of dense layers, but might include other layers like dropout.\n\nFirst we will define our model using Keras.Sequential:","metadata":{}},{"cell_type":"code","source":"# Define the model (CNN)\nmodel = keras.Sequential([\n    # Use default padding='valid' and strides=(1, 1)\n    keras.layers.Conv2D(32, (3, 3), input_shape=(28, 28, 1), activation='relu'),\n    keras.layers.Conv2D(32, (3, 3), activation='relu'),\n    keras.layers.MaxPooling2D(),\n    keras.layers.BatchNormalization(),\n    keras.layers.SpatialDropout2D(0.2),\n    \n    keras.layers.Conv2D(64, (3, 3), activation='relu'),\n    keras.layers.Conv2D(64, (3, 3), activation='relu'),\n    keras.layers.MaxPooling2D(), \n    keras.layers.BatchNormalization(),\n    keras.layers.SpatialDropout2D(0.2),\n    \n    # Flatten the result and start the dense NN\n    keras.layers.Flatten(),\n    keras.layers.Dense(256, activation='relu'),\n    keras.layers.BatchNormalization(),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(128, activation='relu'),\n    keras.layers.BatchNormalization(),\n    keras.layers.Dropout(0.5),\n    keras.layers.Dense(10, activation='sigmoid')\n])\n\n# Check the model\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:26:57.270300Z","iopub.execute_input":"2023-06-25T20:26:57.270709Z","iopub.status.idle":"2023-06-25T20:26:57.488225Z","shell.execute_reply.started":"2023-06-25T20:26:57.270674Z","shell.execute_reply":"2023-06-25T20:26:57.487386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using the summary method, we can see a summary of each layer of our model. And the number of parameters.\n\nThe next step is compiling the model. We will use the \"adam\" optimizer since it is the most efficient in this case, and the accuracy as metrics because it is the metrics used for this competition.\n\nTo choose the loss, we can see that we are in a multiclass classification problem. So we should use categorical crossentropy as loss function. But it needs the target to be one hot encoded, whereas sparse categorical crossentropy perform the same crossentropy calculation of error, without requiring that the target variable be one hot encoded prior training. So we will use the sparse categorical crossentropy as loss function here.\n\nMoreover, we will defined early stopping, based on the loss of the validation set. Early stopping is used to stop the training when adding more epochs does not improve the general performance of the model. It is important to monitor the training using the validation set, otherwise the model will probably overfit the data.","metadata":{}},{"cell_type":"code","source":"# Compile the model\nmodel.compile(\n    optimizer='adam',\n    loss='sparse_categorical_crossentropy',\n    metrics=['accuracy']\n)\n\n# Define early stopping in case of 2 bad epochs\nstop_train = EarlyStopping(\n    monitor='val_loss', \n    patience=2,\n    restore_best_weights=True\n)\n\n# Train the model with 30 epochs\nmodel.fit(\n    X_train,\n    y_train,\n    epochs=30,\n    validation_data=(X_val, y_val),\n    callbacks=[stop_train]\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:27:01.357915Z","iopub.execute_input":"2023-06-25T20:27:01.358308Z","iopub.status.idle":"2023-06-25T20:28:56.476956Z","shell.execute_reply.started":"2023-06-25T20:27:01.358273Z","shell.execute_reply":"2023-06-25T20:28:56.475912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see that the trainin process of our model stopped by the callbacks of Keras, which refer to the early stopping that we have defined. We can see that the loss of the validation set did not improve two consecutive times, so the trainin stop here.","metadata":{}},{"cell_type":"code","source":"# Analyze the train and validation accuracy\nplt.figure(figsize=(12,5))\nplt.plot(\n    model.history.history['accuracy'],\n    label=\"Train accuracy\"\n)\nplt.plot(\n    model.history.history['val_accuracy'],\n    label=\"Validation accuracy\"\n)\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"Train versus validation accuracy\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:29:04.856958Z","iopub.execute_input":"2023-06-25T20:29:04.857361Z","iopub.status.idle":"2023-06-25T20:29:05.116503Z","shell.execute_reply.started":"2023-06-25T20:29:04.857325Z","shell.execute_reply":"2023-06-25T20:29:05.115430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Analyze the train and validation accuracy\nplt.figure(figsize=(12,5))\nplt.plot(\n    model.history.history['loss'],\n    label=\"Train loss\"\n)\nplt.plot(\n    model.history.history['val_loss'],\n    label=\"Validation loss\"\n)\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Loss\")\nplt.title(\"Train versus validation loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:29:09.289792Z","iopub.execute_input":"2023-06-25T20:29:09.290821Z","iopub.status.idle":"2023-06-25T20:29:09.531577Z","shell.execute_reply.started":"2023-06-25T20:29:09.290770Z","shell.execute_reply":"2023-06-25T20:29:09.530448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First of all, we can see that we are not overfit the data, since the curves are very close to each others. However, we can expect to overfit the model if we add more epochs to our training, so used early stop was a good idea to prevent this.\n\nMoreover we can observe very good result, around 99% of precision on the validation set. Toc check this we can calculated each prediction based on the prediction probabilities, and compute the accuracy score.","metadata":{}},{"cell_type":"code","source":"# Prediction on the validation set\ny_pred = [np.argmax(proba) for proba in model.predict(X_val)]\naccuracy_score(y_val, y_pred)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:29:13.051617Z","iopub.execute_input":"2023-06-25T20:29:13.052742Z","iopub.status.idle":"2023-06-25T20:29:13.743268Z","shell.execute_reply.started":"2023-06-25T20:29:13.052701Z","shell.execute_reply":"2023-06-25T20:29:13.742215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we said before, we have around 99% of precision using this model. We can visualize the result in confusion matrix (and may be observe some number with which our model struggle more).","metadata":{}},{"cell_type":"code","source":"# Confusion matrix\nplt.figure(figsize=(8,8))\nsb.heatmap(confusion_matrix(y_val, y_pred), annot=True, fmt = 'g')\nplt.xlabel(\"Reality\")\nplt.ylabel(\"Model prediction\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:29:17.040068Z","iopub.execute_input":"2023-06-25T20:29:17.041179Z","iopub.status.idle":"2023-06-25T20:29:17.647093Z","shell.execute_reply.started":"2023-06-25T20:29:17.041125Z","shell.execute_reply":"2023-06-25T20:29:17.645851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that only few errors are made by the model, with some very closed number in their design (for example, the 2 and the 7 can be very similar).\n\nTo understand better how our model works, we will use shap values later in this notebook.\n\nFirst we will save our model using keras.","metadata":{}},{"cell_type":"code","source":"# Save model\nmodel.save(filepath=\"mnist_cnn\", save_format=\"tf\")","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:29:23.804001Z","iopub.execute_input":"2023-06-25T20:29:23.805051Z","iopub.status.idle":"2023-06-25T20:29:27.375437Z","shell.execute_reply.started":"2023-06-25T20:29:23.804997Z","shell.execute_reply":"2023-06-25T20:29:27.374373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now our model is saved in tf format. We can load it using the command:  \n`model = keras.models.load_model(\"mnist_cnn\")`","metadata":{}},{"cell_type":"markdown","source":"# Model interpretation","metadata":{}},{"cell_type":"markdown","source":"To interpret the model we can use shap values to understand how our model classify number. Shap values (an acronym from SHapley Additive exPlanations) break down a prediction to show the impact of each feature (i.e., each pixel here). It allows us to interpret the impact of having a certain value for a given feature in comparison to the prediction we'd make if that feature took some baseline value. \n\n**Unfortunately, the shap values are not yet available with the latest version of tensorflow. You have to wait for an update of the library to run the cells below.**","metadata":{}},{"cell_type":"code","source":"# Select a set of background examples to take an expectation over\n#background = X_train[np.random.choice(X_train.shape[0], 100, replace=False)]\n\n# Explain predictions of the model on three images\n#expl = shap.DeepExplainer(model, background)\n# Get the shap values for four images \n#shap_values = expl.shap_values(X_train[1:5])\n\n# Plot the feature attributions\n#shap.image_plot(shap_values, -X_train[1:5])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the visualization:\n- Red pixels represent positive SHAP values that contributed to classifying that image as that particular class.\n- Blue pixels represent negative SHAP values that contributed to not classifying that image as that particular class.\n\nNote also, that the first number represent the number to classify, and after each one represent the sequence of 0 to 9.\n\nSo, for example at the image of 0, we can see that in the first position (i.e., 0) we have red pixel for the number curve and for the absence of high pixel values (since 0 is white, or nothing and 255 is black) in the middle. Whereas in the nine position (i.e., 8), we can see bleu pixel in the middle, indicating this lack of high pixel values contributes to the image not being classified as 8.\n\nWe can do several analysis like this, to better undestand how the model work.\n\nNow, we will predict the image of the test set, and display one of our prediction to see if the model perform well (since the large precision of it, we can expect in a good prediction).","metadata":{}},{"cell_type":"code","source":"# Prediction on the test set\ny_pred = [np.argmax(proba) for proba in model.predict(X_test)]\n\n# Example of prediction\npredict_image(X_test, y_pred)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:30:20.864021Z","iopub.execute_input":"2023-06-25T20:30:20.864402Z","iopub.status.idle":"2023-06-25T20:30:21.042547Z","shell.execute_reply.started":"2023-06-25T20:30:20.864367Z","shell.execute_reply":"2023-06-25T20:30:21.041438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So our work is ended. We just have to save our result in a csv file and submit this file to be analyzed and scored by Kaggle.","metadata":{}},{"cell_type":"code","source":"# Replace the label (all 0) with the prediction\ndf_submission['Label'] = y_pred\n\n# Save as csv the result\ndf_submission.to_csv('submission.csv', index=False)\n!head submission.csv # Use Linux command to show the first row of our file","metadata":{"execution":{"iopub.status.busy":"2023-06-25T20:30:27.361700Z","iopub.execute_input":"2023-06-25T20:30:27.362419Z","iopub.status.idle":"2023-06-25T20:30:28.457602Z","shell.execute_reply.started":"2023-06-25T20:30:27.362381Z","shell.execute_reply":"2023-06-25T20:30:28.456218Z"},"trusted":true},"execution_count":null,"outputs":[]}]}