{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\n# Helper libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction\n\nThis Notebook follows three main parts:\n\n* The data preparation\n* The CNN modeling and evaluation\n* The results prediction and submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport itertools\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.losses import categorical_crossentropy\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Activation,BatchNormalization,Conv2D,Dense,Dropout,Flatten,MaxPooling2D\nfrom tensorflow.keras.callbacks import LearningRateScheduler,ModelCheckpoint,EarlyStopping\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Data preparation\n\n## 2.1 Load data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Train shape: {train.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Test shape: {test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_label = train[\"label\"]\ntrain_data = train.drop(\"label\",axis = 1) \n\nsns.countplot(train_label)\ntrain_label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Check for null and missing values\n\nI check for corrupted images (missing values inside)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the data\ntrain_data.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the data\ntest.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Normalization\n\nWe perform a grayscale normalization to reduce the effect of illumination's differences. \n\nMoreover the CNN converg faster on [0..1] data than on [0..255]."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the data\nX_train = train_data / 255.0\nX_test = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Reshape\n\nTrain and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values. We reshape all data to 28x28x1 3D matrices. \n\nKeras requires an extra dimension in the end which correspond to channels. MNIST images are gray scaled so it use only one channel. For RGB images, there is 3 channels, we would have reshaped 784px vectors to 28x28x3 3D matrices."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\n\nprint(f\"X_train data shape: {X_train.shape}\")\nprint(f\"X_test data shape: {X_test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.5 Label encoding\n\nLabels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])\nY_train = tf.keras.utils.to_categorical(train_label, num_classes = 10)\nprint(f\"Y_train data shape: {Y_train.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.6 Split training and valdiation set\n\nI choosed to split the train set in two parts : a small fraction (20%) became the validation set which the model is evaluated and the rest (80%) is used to train the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can get a better sense for one of these examples by visualising the image and looking at the label.\n\nFunction to plot one random digit along with its label"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_random_digit():\n    random_index = np.random.randint(0,X_train.shape[0])\n    plt.imshow(X_train[random_index][:,:,0], cmap='gray')\n    index = tf.argmax(Y_train[random_index], axis=0)\n    plt.title(index.numpy())\n    plt.axis(\"Off\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=[2,2])\nplot_random_digit()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Execute the cell multiple times to see random examples.\n\nLooking at 50 samples at one go"},{"metadata":{"trusted":true},"cell_type":"code","source":"# preview the images\nplt.figure(figsize=[10,6])\nfor i in range(50):\n    plt.subplot(5, 10, i+1)\n    plt.imshow(X_train[i][:,:,0], cmap='gray')\n    plt.axis('Off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. CNN\n## 3.1 Data augmentation\n\nApproaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more. \n\nBy applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180) \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        shear_range = 0.1, # Shear Intensity (Shear angle in counter-clockwise direction in degrees)\n        zoom_range = 0.1, # randomly zoom image\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\ntrain_datagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Define the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential([\n    Conv2D(32, 5, padding='same', input_shape=(28, 28, 1)),\n    Activation('relu'),\n    \n    Conv2D(32, 5, padding='same'),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),\n    \n    Conv2D(64, 3, padding='same'),\n    Activation('relu'),\n    \n    Conv2D(64, 3, padding='same'),\n    Activation('relu'),\n    MaxPooling2D(2, 2),\n    Dropout(0.25),\n    \n    Flatten(),\n    Dense(256),\n    Activation('relu'),\n    Dropout(0.5),\n    Dense(10, activation=\"softmax\"),\n])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.utils.plot_model(model, to_file=\"model.png\", show_shapes=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Set the optimizer and annealer\n\nSpecify the training configuration (optimizer, loss, metrics)"},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate=0.001\nbatch_size = 64\nepochs = 50\nsteps_per_epoch = X_train.shape[0] // batch_size\nvalidation_steps = X_val.shape[0] // batch_size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the optimizer\noptimizer = Adam(lr=learning_rate)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compile the model\nmodel.compile(optimizer=optimizer,\n              loss=categorical_crossentropy,\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next function reduces the learning rate as the training advances."},{"metadata":{"trusted":true},"cell_type":"code","source":"def scheduler(epoch):\n    return learning_rate * 0.99 ** epoch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr_scheduler = LearningRateScheduler(scheduler)\n\nmodel_checkpoint  = ModelCheckpoint('model_best_checkpoint.h5', save_best_only=True,\n                                    save_weights_only=True, monitor='val_loss', verbose=1)\n\nearly_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1)\n\ncallbacks_list = [lr_scheduler, model_checkpoint, early_stopping]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# training the model\nhistory = model.fit_generator(\n      train_datagen.flow(X_train,Y_train,batch_size=batch_size),\n      steps_per_epoch=steps_per_epoch,\n      epochs=epochs,\n      validation_data=(X_val,Y_val),\n      validation_steps=validation_steps,\n      verbose=1,\n      callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Evaluate the model\n## 4.1 Training and validation curves"},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy = history.history['accuracy']\nval_accuracy = history.history['val_accuracy']\n\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs_range = range(len(accuracy))\n\nplt.figure(figsize=(16, 4))\nplt.subplot(1, 2, 1)\nplt.plot(epochs_range, accuracy, label='Training Accuracy')\nplt.plot(epochs_range, val_accuracy, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.title('Training and Validation Accuracy')\n\nplt.subplot(1, 2, 2)\nplt.plot(epochs_range, loss, label='Training Loss')\nplt.plot(epochs_range, val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.title('Training and Validation Loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Confusion matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('model_best_checkpoint.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = model.evaluate(X_val, Y_val, batch_size=batch_size, verbose=0)\n\nprint(f\"\\nLoss: {results[0]}\")\nprint(f\"Accuracy: {results[1]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at confusion matrix \n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display some error results \n\n# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\ndef display_errors(errors_index,img_errors,pred_errors, obs_errors):\n    \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n    n = 0\n    nrows = 2\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n            ax[row,col].set_title(\"Predicted label :{}\\nTrue label :{}\".format(pred_errors[error],obs_errors[error]))\n            n += 1\n\n# Probabilities of the wrong predicted\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[-6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict results\nresults = model.predict(X_test)\n\n# select the indix with the maximum probability\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating the submission file\nsubmission = pd.DataFrame({\"ImageId\":[i+1 for i in range(len(X_test))],\n                           \"Label\": results})\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"submission_5.csv\", index=False, header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# saving the model\nmodel.save(\"MNIST_MODEL.h5\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}