{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"This is my first Computer Vision model. It's a custom CNN that was built following a simple iterative process : \n- get a baseline model quickly\n- whenever the model suffers from bias, either train it longer or make it more complex (with more parameters to train)\n- whenever the model does not generalize well, either add regularization techniques (I used drop-out) or use a bigger training set.\n- and iterate...\n\nThis simple \"recipe\" worked well on this dataset :\n- v1 = baseline model : accuracy = **98.99%** on test set, some difficulties to generalize well.\n- v2 = v1 + drop-out : **99,22%** on test set, some difficulties to generalize well.\n- v3 = v2 + data augmentation : **99,4%** on test set, bias problem appeared with this bigger dataset\n- v4 = v3 + additional ConvNet blocks and FC layers : **99,45%** on the test set, bias problem fixed but again some difficulties to generalize well.\n- v5 = v4 + data augmentation : **99,61%** on test set.\n\n**At this level of performance, the model performs (almost) as well as human beings : images that are misclassified by the model are not clean and not easy to recognize for human beings too.**","metadata":{}},{"cell_type":"markdown","source":"# Agenda","metadata":{}},{"cell_type":"markdown","source":"1. [EDA](#1)\n\n2. [Dataset preparation with image data augmentation](#2)\n\n3. [Model Creation & Training](#3)\n\n    3.1. [Model creation](#3.1)\n\n    3.2. [Model training with CV](#3.2) \n\n    3.3 [Model performance evaluation](#3.3)\n\n4. [Error Analysis](#4)\n\n    4.1. [Confusion matrix](#4.1)\n    \n    4.2. [Display a sample of images with bad predicitions](#4.2)\n    \n    4.3. [Statistics about the \"confidence score\" for correct and bad predictions](#4.3)\n\n5. [Submit Predictions](#5)\n","metadata":{}},{"cell_type":"code","source":"# Imports\nimport os, warnings\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Load data sets\nds_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nds_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-04T08:13:50.108697Z","iopub.execute_input":"2021-06-04T08:13:50.109173Z","iopub.status.idle":"2021-06-04T08:13:55.679856Z","shell.execute_reply.started":"2021-06-04T08:13:50.109068Z","shell.execute_reply":"2021-06-04T08:13:55.679072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. EDA <a id='1'/>","metadata":{}},{"cell_type":"markdown","source":"A quick analysis of the data set shows :\n\n- The training data set has 42000 images of 28x28 pixels.  \n\n- The test data set has 28000 images of 28x28 pixels.\n\n- The data set is pretty well balanced with a minimum of 3795 images representing a 5 and a maximum of 4684 images representing a 1.\n\n- Some pixels are 0 for all images. E.g. the pixels in the 4 corners of the image are always 0.\n\n- The data set was produced by different writers.\n\n- Some handwritten digits are much cleaner and easier to recognize than others. E.g. the 9th image is a 5 but it could be also a 9. The 20th image is a 5 but could be also a 6...","metadata":{}},{"cell_type":"code","source":"ds_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:13:55.681480Z","iopub.execute_input":"2021-06-04T08:13:55.681825Z","iopub.status.idle":"2021-06-04T08:13:55.711584Z","shell.execute_reply.started":"2021-06-04T08:13:55.681792Z","shell.execute_reply":"2021-06-04T08:13:55.710585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train.shape[0]","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:13:55.712965Z","iopub.execute_input":"2021-06-04T08:13:55.713315Z","iopub.status.idle":"2021-06-04T08:13:55.718786Z","shell.execute_reply.started":"2021-06-04T08:13:55.713280Z","shell.execute_reply":"2021-06-04T08:13:55.717862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train['label'].value_counts().sort_values()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:13:55.720430Z","iopub.execute_input":"2021-06-04T08:13:55.720826Z","iopub.status.idle":"2021-06-04T08:13:55.732579Z","shell.execute_reply.started":"2021-06-04T08:13:55.720793Z","shell.execute_reply":"2021-06-04T08:13:55.731751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_zeros_pixels = (ds_train.drop('label',axis=1) == 0).all().replace(to_replace=[True, False], value=[0,1]).values.reshape(28,28)\nplt.imshow(all_zeros_pixels)\nplt.title('PIXELS WITH 0 FOR ALL IMAGES')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:13:55.736304Z","iopub.execute_input":"2021-06-04T08:13:55.736581Z","iopub.status.idle":"2021-06-04T08:13:56.005003Z","shell.execute_reply.started":"2021-06-04T08:13:55.736551Z","shell.execute_reply":"2021-06-04T08:13:56.004129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig=plt.figure(figsize=(20, 5))\ncolumns = 10\nrows = 2\nfor i in range(0, columns*rows):\n    label = ds_train['label'].iloc[i]\n    img = ds_train.drop('label',axis=1).iloc[i].values.reshape(28,28)\n    fig.add_subplot(rows, columns, i+1)\n    plt.title('LABEL = ' + str(label))\n    plt.axis('off')\n    plt.imshow(img)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:13:56.007176Z","iopub.execute_input":"2021-06-04T08:13:56.007674Z","iopub.status.idle":"2021-06-04T08:13:58.326040Z","shell.execute_reply.started":"2021-06-04T08:13:56.007636Z","shell.execute_reply":"2021-06-04T08:13:58.325332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Dataset preparation & Image data augmentation <a id='2'/>","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.metrics import confusion_matrix\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom kerastuner.tuners import RandomSearch\nfrom tensorflow.keras.layers.experimental import preprocessing\n\n# Reproducability\n# check https://keras.io/getting_started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development for details.\ndef set_seed(seed=42):\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\nset_seed()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:13:58.327446Z","iopub.execute_input":"2021-06-04T08:13:58.327818Z","iopub.status.idle":"2021-06-04T08:14:03.451255Z","shell.execute_reply.started":"2021-06-04T08:13:58.327777Z","shell.execute_reply":"2021-06-04T08:14:03.450464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's :\n\n- split the training set into X_train with the pixel columns and y_train with the corresponding labels\n\n- change X_train format to the one expected by Keras. ","metadata":{}},{"cell_type":"code","source":"X_train = ds_train.copy()\n\n# Creating the X_train dataframe with the pixel columns only and the y_train with the labels\ny_train = X_train['label']\nX_train.drop('label', axis=1, inplace=True)\n\n# Reshaping X_train and X_val to Keras input format\nX_train = X_train.to_numpy().reshape(42000,28,28,1)","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:14:03.453368Z","iopub.execute_input":"2021-06-04T08:14:03.453624Z","iopub.status.idle":"2021-06-04T08:14:03.638711Z","shell.execute_reply.started":"2021-06-04T08:14:03.453599Z","shell.execute_reply":"2021-06-04T08:14:03.637808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's define a function that generates the augmented samples and add them to the original samples:","metadata":{}},{"cell_type":"code","source":"#############################################################################\n# input :\n# X = ndarray as expected by Keras (n_samples,height,width,n_channels)\n# y = 1-d array (n_samples)\n# output :\n# X = ndarray as expected by Keras (10 x n_samples,height,width,n_channels)\n# y = 1-d array (10 x n_samples)\n#############################################################################\ndef data_augmentation(X, y):\n\n    # Defining the data augmentations using Keras preprocessing layers \n    data_augmentation1 = keras.Sequential([\n        preprocessing.RandomTranslation(height_factor=0.1, width_factor=0.1, fill_mode='constant'),\n        preprocessing.RandomRotation(factor=0.1, fill_mode='constant')\n    ])\n\n    data_augmentation2 = keras.Sequential([\n        preprocessing.RandomTranslation(height_factor=0.1, width_factor=0.1, fill_mode='constant'),\n        preprocessing.RandomZoom(height_factor=0.15, width_factor=0.15, fill_mode='constant')\n    ])\n    \n    data_augmentation3 = keras.Sequential([\n        preprocessing.RandomTranslation(height_factor=0.1, width_factor=0.1, fill_mode='constant'),\n        preprocessing.RandomZoom(height_factor=0.15, width_factor=0.15, fill_mode='constant'),\n        preprocessing.RandomRotation(factor=0.1, fill_mode='constant')\n    ])\n\n    # Generating the augmented samples\n    X_new1_1 = data_augmentation1(X)\n    X_new1_2 = data_augmentation1(X)\n    X_new1_3 = data_augmentation1(X)\n    X_new2_1 = data_augmentation2(X)\n    X_new2_2 = data_augmentation2(X)\n    X_new2_3 = data_augmentation2(X)\n    X_new3_1 = data_augmentation3(X)\n    X_new3_2 = data_augmentation3(X)\n    X_new3_3 = data_augmentation3(X)\n    \n    # Concatenating X with the augmented samples\n    X = np.concatenate((X, X_new1_1, X_new1_2, X_new1_3, X_new2_1, X_new2_2, X_new2_3, X_new3_1, X_new3_2, X_new3_3)) \n    y = pd.concat([y, y.copy(), y.copy(), y.copy(), y.copy(), y.copy(), y.copy(), y.copy(), y.copy(), y.copy()], ignore_index=True) \n    \n    return X, y","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:14:03.640870Z","iopub.execute_input":"2021-06-04T08:14:03.641399Z","iopub.status.idle":"2021-06-04T08:14:03.651487Z","shell.execute_reply.started":"2021-06-04T08:14:03.641358Z","shell.execute_reply":"2021-06-04T08:14:03.650648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we are going to check the data_augmentation function on 10 images. To do this, I'm going to :  \n- select the 10 first images of the original training set\n- apply the data_augmentation function to these 10 images\n- As a result, I will get 100 images as a result : 10 original images + 90 augmented samples (each image has 9 augmented samples)\n- display the the 10 first images of the original training set and their 90 corresponding augmented samples","metadata":{}},{"cell_type":"code","source":"X10 = X_train[0:10]\ny10 = y_train[0:10]\nX100, y100 = data_augmentation(X10, y10)\n\nfig=plt.figure(figsize=(20,20))\npos = 1\nfor i in range(0, 10):\n    for j in range(i+0, i+100, 10):\n        fig.add_subplot(10, 10, pos)\n        plt.imshow(tf.squeeze(X100[j]))\n        plt.title('Label = ' + str(y100[j]))\n        plt.axis('off')\n        pos = pos + 1\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:14:03.653819Z","iopub.execute_input":"2021-06-04T08:14:03.654084Z","iopub.status.idle":"2021-06-04T08:14:10.422065Z","shell.execute_reply.started":"2021-06-04T08:14:03.654060Z","shell.execute_reply":"2021-06-04T08:14:10.421127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Model creation and training <a id='3' />","metadata":{}},{"cell_type":"markdown","source":"## 3.1 Model creation  <a id='3.1'/>","metadata":{}},{"cell_type":"code","source":"def build_model():\n    \n    inputs = tf.keras.Input(shape=(28, 28, 1))\n    \n    # Normalizing inputs\n    x = inputs\n    x = tf.keras.layers.BatchNormalization()(x)\n    \n    # ConvNet - this for loop creates 4 conv_blocks\n    for i in range(4):\n        x = tf.keras.layers.Convolution2D(256, kernel_size=(3, 3), padding=\"same\")(x)\n        x = tf.keras.layers.ReLU()(x)\n        x = tf.keras.layers.MaxPool2D()(x)\n        x = tf.keras.layers.Dropout(0.3)(x)\n   \n    # Head - this for loop creates 4 layers\n    x = tf.keras.layers.Flatten()(x)\n    for i in range(4):\n        x = tf.keras.layers.Dense(256, activation=\"relu\")(x)\n        x = tf.keras.layers.Dropout(0.3)(x)\n    \n    # Output layer\n    outputs = tf.keras.layers.Dense(10, activation=\"softmax\")(x)\n\n    # Returning a compiled model\n    model = tf.keras.Model(inputs, outputs)\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n        loss=\"sparse_categorical_crossentropy\",\n        metrics=\"accuracy\"\n    )\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:14:10.423218Z","iopub.execute_input":"2021-06-04T08:14:10.423518Z","iopub.status.idle":"2021-06-04T08:14:10.432929Z","shell.execute_reply.started":"2021-06-04T08:14:10.423488Z","shell.execute_reply":"2021-06-04T08:14:10.432124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Model training with CV <a id='3.2'/>","metadata":{}},{"cell_type":"code","source":"N_EPOCHS = 100\n\n# I use Cross Validation with N_SPLITS. \nN_SPLITS = 5\n\n# N_ITERATION allows me to run less iterations to save time.\n# If N_ITERATION < N_SPLITS then the number of trainings/evaluations will stop earlier.\nN_ITERATION = 1","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:14:10.434108Z","iopub.execute_input":"2021-06-04T08:14:10.434594Z","iopub.status.idle":"2021-06-04T08:14:10.442613Z","shell.execute_reply.started":"2021-06-04T08:14:10.434555Z","shell.execute_reply":"2021-06-04T08:14:10.441825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating hist_df to store history objects for each training / split\nhist_df = pd.DataFrame(columns=['iteration', 'history'])\niteration = 1\nindex = 0\n\n# This boolean variable is used to save one model only. \nsaved_model = False\n\n# Reshaping X_train from Keras input format (42000, 28, 28, 1) to (n_samples, n_features) format (42000, 784) as expected by StratifiedKFold.split() function\nX_train = X_train.reshape(42000, 784)\n\n# Training and evaluating the model 5 times, each time with a different training/validation set\nskf = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=42)\nfor train_index, val_index in skf.split(X_train, y_train): # returns indices to build 10 splits of X_train, y_train, X_val, y_val\n    \n    # Getting the training set and validation set before data augmentation\n    X_train_, X_val_ = X_train[train_index], X_train[val_index]\n    X_train_ = X_train_.reshape(33600,28,28,1) # Reshaping X_train to Keras input format\n    X_val_ = X_val_.reshape(8400,28,28,1) # Reshaping X_val to Keras input format\n    y_train_, y_val_ = y_train[train_index], y_train[val_index]\n    \n    # Generating augmented samples\n    X_train_, y_train_ = data_augmentation(X_train_, y_train_) #X_train_ is now (336000,28,28,1) and y_train_ is now (336000)\n    \n    # Building the model\n    model = build_model()\n    \n    # Training and evaluating each model for this split\n    history = model.fit(x=X_train_, y=y_train_, validation_data=(X_val_, y_val_), epochs=N_EPOCHS, batch_size=64)\n    \n    # Saving the trained model as a saved model file -- only one model is saved\n    if(saved_model == False):\n        model.save('model')\n        saved_model = True\n    \n    # Storing the history objects into a dataframe \n    hist_df.loc[index, 'iteration'] = iteration\n    hist_df.loc[index, 'history'] = history\n    \n    if(iteration == N_ITERATION):\n        break\n        \n    index = index + 1\n    iteration = iteration + 1","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:14:10.444183Z","iopub.execute_input":"2021-06-04T08:14:10.444519Z","iopub.status.idle":"2021-06-04T08:57:22.123497Z","shell.execute_reply.started":"2021-06-04T08:14:10.444484Z","shell.execute_reply":"2021-06-04T08:57:22.122597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 Model performance evaluation <a id='3.3'/>","metadata":{}},{"cell_type":"code","source":"hist = []\nfor i in range(N_ITERATION):\n    hist.append(pd.DataFrame(hist_df[hist_df['iteration']==(i+1)]['history'][i].history))\n    if i==0:\n        hist_full = hist[0]\n    else:\n        hist_full = pd.concat([hist_full, hist[i]])\n\n# Dropping the 1st EPOCHS of each iteration because their losses are high and their accuracies are low \nhist_full.drop([0,1,2], inplace=True)  # 3 EPOCHS dropped / iteration     \n\n# Displaying CV metrics\nfig,axes=plt.subplots(1,2,figsize=(20,6))\nsns.lineplot(data=hist_full[['loss','val_loss']], dashes=False, ax=axes[0])\naxes[0].axhline(0.05, ls='--')\naxes[0].axhline(0, ls='--')\nsns.lineplot(data=hist_full[['accuracy', 'val_accuracy']], dashes=False, ax=axes[1])\naxes[1].axhline(0.99, ls='--')\naxes[1].axhline(0.995, ls='--')\naxes[1].axhline(0.996, ls='--')\naxes[1].axhline(0.997, ls='--')\naxes[1].axhline(0.998, ls='--')\naxes[1].axhline(1, ls='--')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:57:22.125623Z","iopub.execute_input":"2021-06-04T08:57:22.125913Z","iopub.status.idle":"2021-06-04T08:57:22.537560Z","shell.execute_reply.started":"2021-06-04T08:57:22.125884Z","shell.execute_reply":"2021-06-04T08:57:22.536892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Error Analysis <a id='4'/>","metadata":{}},{"cell_type":"markdown","source":"## 4.1. Confusion matrix <a id='4.1'/>","metadata":{}},{"cell_type":"code","source":"# Reloading the saved model -- this is the model trained in the 1st loop of the CV\nmodel = keras.models.load_model('model')\n\n# Retrieving the validation set -- corresponding to the 1st loop of the CV\nskf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nfor _, val_index in skf.split(X_train, y_train): # returns indices to build 10 splits of X_train, y_train, X_val, y_val\n    X_val = X_train[val_index]\n    y_val = y_train[val_index]\n    break\n\nX_val = X_val.reshape(8400,28,28,1)    \n    \n# Making predictions\nscores = model.predict(X_val)\ny_pred = np.argmax(scores, axis=1)\n\n# Displaying the confusion matrix\nplt.figure(figsize = (14,7))\nsns.heatmap(confusion_matrix(y_val, y_pred), annot=True, fmt='d', cmap='YlOrBr')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:57:22.538623Z","iopub.execute_input":"2021-06-04T08:57:22.538862Z","iopub.status.idle":"2021-06-04T08:57:25.804682Z","shell.execute_reply.started":"2021-06-04T08:57:22.538837Z","shell.execute_reply":"2021-06-04T08:57:25.803862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2. Display a sample of images with bad predicitions <a id='4.2'/>","metadata":{}},{"cell_type":"markdown","source":"Let's display misclassified images:","metadata":{}},{"cell_type":"code","source":"errors = pd.DataFrame({'y_val':y_val, 'prediction':y_pred})[y_val!=y_pred]\n\nif(errors.shape[0]>20):\n\n    fig=plt.figure(figsize=(20, 20))\n    columns = 4\n    rows = 5\n    j = 0\n    # Loop on each X_val entry for which the prediction is not correct\n    for i in (errors.index):\n        label = ds_train['label'].iloc[i]\n        predict = errors.loc[i, 'prediction']\n        img = ds_train.drop('label',axis=1).iloc[i].values.reshape(28,28)\n        fig.add_subplot(rows, columns, j+1)\n        j = j + 1\n        plt.title('GROUND TRUTH = ' + str(label) + ' PREDICT = ' + str(predict))\n        plt.imshow(img)\n        if(j==20):\n            break #Loop is broken after 20 iterations\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:57:25.805927Z","iopub.execute_input":"2021-06-04T08:57:25.806272Z","iopub.status.idle":"2021-06-04T08:57:29.194789Z","shell.execute_reply.started":"2021-06-04T08:57:25.806243Z","shell.execute_reply":"2021-06-04T08:57:29.193954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.3. Statistics about the \"confidence score\" for bad predictions <a id='4.3'/>","metadata":{}},{"cell_type":"code","source":"predictions = pd.DataFrame({'y_val':y_val, 'y_pred':y_pred, 'scores':np.max(scores, axis=1)})\n\nfig,axes=plt.subplots(1,2,figsize=(20,4))\nsns.histplot(predictions[predictions['y_val']==predictions['y_pred']]['scores'], kde=False, ax=axes[0]).set_title('Confidence score distribution for correct predictions')\nsns.histplot(predictions[predictions['y_val']!=predictions['y_pred']]['scores'], kde=False, ax=axes[1]).set_title('Confidence score distribution for bad predictions')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:57:29.196360Z","iopub.execute_input":"2021-06-04T08:57:29.196694Z","iopub.status.idle":"2021-06-04T08:57:29.499374Z","shell.execute_reply.started":"2021-06-04T08:57:29.196659Z","shell.execute_reply":"2021-06-04T08:57:29.498648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Submit Predicitions <a id='5'/>","metadata":{}},{"cell_type":"code","source":"#################################################\n# Let's retrain our model with the whole dataset\n#################################################\n\n# Creating the X_train dataframe with the pixel columns only and the y_train with the labels\nX_train = ds_train.copy()\ny_train = X_train['label']\nX_train.drop('label', axis=1, inplace=True)\n\n# Reshaping X_train and X_val to Keras input format\nX_train = X_train.to_numpy().reshape(42000,28,28,1)\n\n# Generating augmented samples\nX_train, y_train = data_augmentation(X_train, y_train) #X_train_ is now (420000,28,28,1) and y_train_ is now (420000)\n\n# Building the model\nmodel = build_model()\n\n# Training the model on the whole training set\nmodel.fit(x=X_train, y=y_train, epochs=N_EPOCHS, batch_size=64)\n\n#################################################\n# Let's generate the predictions on the test set\n#################################################\n\nX_test = ds_test.copy()\nX_test = X_test.to_numpy().reshape(28000,28,28,1)\npredictions = model.predict(X_test)\n\n##################################\n# Let's submit the new predictions\n##################################\noutput = pd.DataFrame({'ImageId': list(range(1, 28001)), 'Label': np.argmax(predictions, axis=1)})\noutput.to_csv('my_submission.csv', index=False)\nprint(\"Your submission was successfully saved!\")","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:57:29.500723Z","iopub.execute_input":"2021-06-04T08:57:29.501163Z","iopub.status.idle":"2021-06-04T09:24:15.871472Z","shell.execute_reply.started":"2021-06-04T08:57:29.501119Z","shell.execute_reply":"2021-06-04T09:24:15.869259Z"},"trusted":true},"execution_count":null,"outputs":[]}]}