{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# General Information\n\nThis notebook aim to test the performance of transfer learning on this hand-writting digit dataset. \n\nTwo well-known CNN model will be tested: VGG16 and ResnNet50.\n\nTo start, we will implement only feature extraction from both model and evaluate using the accuracy and f1 score metric. If the accuracy & loss are not good enough, I would implement partial fine-tuning or full fine-tuning.\n\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"# Importing Libraries","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport os\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nimport cv2 as cv\nimport tensorflow as tf\nimport keras\nfrom keras import backend as K\nfrom tensorflow.keras.utils import img_to_array\nfrom keras.applications.resnet import ResNet50\nfrom keras.applications.resnet import preprocess_input\nfrom keras.layers import Conv2D, MaxPooling2D, Dense, Input, Flatten, Dropout, UpSampling2D, GlobalAveragePooling2D\nfrom keras.layers import RandomFlip, RandomRotation, RandomZoom\nfrom keras.models import Model\nfrom keras.optimizers import Adam, RMSprop\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom sklearn.metrics import classification_report, confusion_matrix\nimport pickle\nfrom tensorflow.keras.models import Sequential\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:20:59.531945Z","iopub.execute_input":"2023-03-21T04:20:59.532352Z","iopub.status.idle":"2023-03-21T04:21:08.109433Z","shell.execute_reply.started":"2023-03-21T04:20:59.532317Z","shell.execute_reply":"2023-03-21T04:21:08.108596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:21:08.111003Z","iopub.execute_input":"2023-03-21T04:21:08.111695Z","iopub.status.idle":"2023-03-21T04:21:08.125448Z","shell.execute_reply.started":"2023-03-21T04:21:08.111667Z","shell.execute_reply":"2023-03-21T04:21:08.123850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_csv = pd.read_csv(\"train.csv\")\ntest_csv = pd.read_csv(\"test.csv\")\ntrain_csv.head(3)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T05:46:00.564837Z","iopub.execute_input":"2023-03-21T05:46:00.565282Z","iopub.status.idle":"2023-03-21T05:46:03.368644Z","shell.execute_reply.started":"2023-03-21T05:46:00.565233Z","shell.execute_reply":"2023-03-21T05:46:03.367296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_csv.head(3)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T04:19:18.825745Z","iopub.execute_input":"2023-03-21T04:19:18.826037Z","iopub.status.idle":"2023-03-21T04:19:18.844020Z","shell.execute_reply.started":"2023-03-21T04:19:18.826011Z","shell.execute_reply":"2023-03-21T04:19:18.843048Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_classes = len(train_csv['label'].unique())\nnum_classes","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:19:18.845571Z","iopub.execute_input":"2023-03-21T04:19:18.845893Z","iopub.status.idle":"2023-03-21T04:19:18.857938Z","shell.execute_reply.started":"2023-03-21T04:19:18.845860Z","shell.execute_reply":"2023-03-21T04:19:18.856905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMG_SIZE = (28,28,1)\n# Sample digit image\nsample_images = train_csv.iloc[0:5,1:].values.reshape((-1,)+IMG_SIZE)\nfig, axs = plt.subplots(1, sample_images.shape[0])\nfor i,img in enumerate(sample_images):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {train_csv.iloc[i,0]}')\n    axs[i].xaxis.label.set_color('red')","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:21:12.113403Z","iopub.execute_input":"2023-03-21T04:21:12.113670Z","iopub.status.idle":"2023-03-21T04:21:12.513005Z","shell.execute_reply.started":"2023-03-21T04:21:12.113646Z","shell.execute_reply":"2023-03-21T04:21:12.511658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparation of Data\n\n- X : from 28 * 28 grey-scale pixels to 75 * 75 RGB-scale pixels \n- y: from a column of numbers to an array of 10 one-hot-encoding classes.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# Preprocessing Data\nX = train_csv.iloc[:,1:].values.reshape((-1,)+(28,28,1)).astype(np.float32)\nlist = []\nfor i in range(len(X)):\n    X_rgb = cv.cvtColor(X[i], cv.COLOR_GRAY2RGB)\n    X_rgb = cv.resize(X_rgb, (75,75))\n    list.append(X_rgb)\nX_rgb = np.array(list)\n# Normalization\nX_rgb /= 255\n\ny = train_csv.iloc[:,0].astype(np.int32)\ny = pd.get_dummies(y)\n\nprint(f'X shape: {format(X_rgb.shape)}')\nprint(f'y shape: {format(y.shape)}')","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:21:15.783207Z","iopub.execute_input":"2023-03-21T04:21:15.783861Z","iopub.status.idle":"2023-03-21T04:21:17.473549Z","shell.execute_reply.started":"2023-03-21T04:21:15.783812Z","shell.execute_reply":"2023-03-21T04:21:17.471856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_submission = test_csv.iloc[:,:].values.reshape((-1,)+(28,28,1)).astype(np.float32)\nlist = []\nfor i in range(len(X_submission)):\n    X_t = cv.cvtColor(X_submission[i], cv.COLOR_GRAY2RGB)\n    X_t = cv.resize(X_t, (75,75))\n    list.append(X_t)\nX_submission = np.array(list)\n# Normalization\nX_submission /= 255\n\nprint(f'X test shape: {format(X_submission.shape)}')","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:46:43.792316Z","iopub.execute_input":"2023-03-21T05:46:43.792943Z","iopub.status.idle":"2023-03-21T05:46:46.886186Z","shell.execute_reply.started":"2023-03-21T05:46:43.792890Z","shell.execute_reply":"2023-03-21T05:46:46.884545Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train/Val/Test data splitting","metadata":{}},{"cell_type":"code","source":"# Splitting Dataset\nX_train, X_val, y_train, y_val = train_test_split(X_rgb,y,test_size=0.2)\nX_val, X_test, y_val, y_test = train_test_split(X_val,y_val,test_size=0.5)\n\nprint(f'X_train shape: {X_train.shape}')\nprint(f'y_train shape: {y_train.shape}')\nprint(f'X_val shape: {X_val.shape}')\nprint(f'y_val shape: {y_val.shape}')\nprint(f'X_test shape: {X_test.shape}')\nprint(f'y_test shape: {y_test.shape}')","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T05:29:06.094037Z","iopub.execute_input":"2023-03-21T05:29:06.094431Z","iopub.status.idle":"2023-03-21T05:29:06.729752Z","shell.execute_reply.started":"2023-03-21T05:29:06.094391Z","shell.execute_reply":"2023-03-21T05:29:06.728143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### X_train and X_val transformed into tensors","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"X_train = tf.convert_to_tensor(X_train)\nX_val = tf.convert_to_tensor(X_val)\nX_test = tf.convert_to_tensor(X_test)\ny_train = tf.convert_to_tensor(y_train)\ny_val = tf.convert_to_tensor(y_val)\ny_test = tf.convert_to_tensor(y_test)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T05:29:17.267957Z","iopub.execute_input":"2023-03-21T05:29:17.268347Z","iopub.status.idle":"2023-03-21T05:29:18.746341Z","shell.execute_reply.started":"2023-03-21T05:29:17.268319Z","shell.execute_reply":"2023-03-21T05:29:18.745063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## F1 score common function","metadata":{}},{"cell_type":"code","source":"def recall_m(y_true, y_pred):\n    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n    recall = true_positives / (possible_positives + K.epsilon())\n    return recall\n\ndef precision_m(y_true, y_pred):\n    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n    precision = true_positives / (predicted_positives + K.epsilon())\n    return precision\n\ndef f1_m(y_true, y_pred):\n    precision = precision_m(y_true, y_pred)\n    recall = recall_m(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:24:50.692797Z","iopub.execute_input":"2023-03-21T04:24:50.693185Z","iopub.status.idle":"2023-03-21T04:24:50.701194Z","shell.execute_reply.started":"2023-03-21T04:24:50.693153Z","shell.execute_reply":"2023-03-21T04:24:50.700070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading pre-trained models (ResNet50/VGG16)","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"# ResNet50","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"### Feature extraction","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"model = Sequential()\n\nmodel.add(tf.keras.applications.resnet50.ResNet50(input_shape = (75, 75, 3), \n                                include_top = False, \n                                weights = 'imagenet'))\n\nmodel.add(Flatten())\nmodel.add(Dense(128, activation='relu'))\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.layers[0].trainable = False\n\nmodel.compile(optimizer=keras.optimizers.Adam(learning_rate=0.0001), loss='categorical_crossentropy', metrics=['accuracy', f1_m])","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:25:51.588954Z","iopub.execute_input":"2023-03-21T04:25:51.589352Z","iopub.status.idle":"2023-03-21T04:25:56.962525Z","shell.execute_reply.started":"2023-03-21T04:25:51.589319Z","shell.execute_reply":"2023-03-21T04:25:56.961431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2023-03-21T04:25:56.964051Z","iopub.execute_input":"2023-03-21T04:25:56.964497Z","iopub.status.idle":"2023-03-21T04:25:56.995801Z","shell.execute_reply.started":"2023-03-21T04:25:56.964469Z","shell.execute_reply":"2023-03-21T04:25:56.995053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train,y_train, validation_data=(X_val,y_val), epochs=5)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T04:26:25.270009Z","iopub.execute_input":"2023-03-21T04:26:25.270800Z","iopub.status.idle":"2023-03-21T04:58:50.174246Z","shell.execute_reply.started":"2023-03-21T04:26:25.270754Z","shell.execute_reply":"2023-03-21T04:58:50.172773Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pickle.dump(model, open('model_resnet50_feature_extraction.sav', 'wb'))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:06:04.675430Z","iopub.execute_input":"2023-03-21T05:06:04.675862Z","iopub.status.idle":"2023-03-21T05:06:06.073166Z","shell.execute_reply.started":"2023-03-21T05:06:04.675821Z","shell.execute_reply":"2023-03-21T05:06:06.071849Z"},"pycharm":{"name":"#%%\n"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.evaluate(X_test,y_test)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:30:11.074190Z","iopub.execute_input":"2023-03-21T05:30:11.075239Z","iopub.status.idle":"2023-03-21T05:30:46.580613Z","shell.execute_reply.started":"2023-03-21T05:30:11.075158Z","shell.execute_reply":"2023-03-21T05:30:46.579479Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(np.array(X_test[4]), interpolation='nearest')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:29:24.677967Z","iopub.execute_input":"2023-03-21T05:29:24.678377Z","iopub.status.idle":"2023-03-21T05:29:24.830262Z","shell.execute_reply.started":"2023-03-21T05:29:24.678343Z","shell.execute_reply":"2023-03-21T05:29:24.829488Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.array(np.argmax(y_test[4]))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:29:35.961280Z","iopub.execute_input":"2023-03-21T05:29:35.961669Z","iopub.status.idle":"2023-03-21T05:29:35.970430Z","shell.execute_reply.started":"2023-03-21T05:29:35.961635Z","shell.execute_reply":"2023-03-21T05:29:35.969652Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"acc_res50_feature = history.history['accuracy']\nval_acc_res50_feature = history.history['val_accuracy']\n\nloss_res50_feature = history.history['loss']\nval_loss_res50_feature = history.history['val_loss']\n\nplt.figure(figsize=(8, 8))\nplt.subplot(2, 1, 1)\nplt.grid(color='b', linestyle='dotted', linewidth=0.5)\nplt.plot(acc_res50_feature, 'o-',label='Training Accuracy', color='b')\nplt.plot(val_acc_res50_feature, 'o-',label='Validation Accuracy', color='g')\nplt.legend(loc='lower right')\nplt.ylabel('Accuracy')\nplt.ylim([min(plt.ylim()),1])\nplt.title('Training and Validation Accuracy')\n\nplt.subplot(2, 1, 2)\nplt.grid(color='b', linestyle='dotted', linewidth=0.5)\nplt.plot(loss_res50_feature, 'o-',label='Training Loss', color='b')\nplt.plot(val_loss_res50_feature, 'o-',label='Validation Loss', color='g')\nplt.legend(loc='upper right')\nplt.ylabel('Cross Entropy')\n# plt.ylim([0,1.0])\nplt.title('Training and Validation Loss')\nplt.xlabel('epoch')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:07:04.673966Z","iopub.execute_input":"2023-03-21T05:07:04.675261Z","iopub.status.idle":"2023-03-21T05:07:05.520342Z","shell.execute_reply.started":"2023-03-21T05:07:04.675204Z","shell.execute_reply":"2023-03-21T05:07:05.519410Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission of ResNet50 result","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"result_test = model.predict(X_submission)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:47:06.378336Z","iopub.execute_input":"2023-03-21T05:47:06.378792Z","iopub.status.idle":"2023-03-21T05:50:57.654390Z","shell.execute_reply.started":"2023-03-21T05:47:06.378753Z","shell.execute_reply":"2023-03-21T05:50:57.653357Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = np.argmax(result_test,1)\nresult","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:19:33.624206Z","iopub.execute_input":"2023-03-21T06:19:33.624820Z","iopub.status.idle":"2023-03-21T06:19:33.633291Z","shell.execute_reply.started":"2023-03-21T06:19:33.624779Z","shell.execute_reply":"2023-03-21T06:19:33.631881Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:51:11.030238Z","iopub.execute_input":"2023-03-21T05:51:11.030643Z","iopub.status.idle":"2023-03-21T05:51:11.054529Z","shell.execute_reply.started":"2023-03-21T05:51:11.030609Z","shell.execute_reply":"2023-03-21T05:51:11.053149Z"},"pycharm":{"name":"#%%\n"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Label'] = np.argmax(result_test,1)\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-03-21T05:52:51.275395Z","iopub.execute_input":"2023-03-21T05:52:51.275805Z","iopub.status.idle":"2023-03-21T05:52:51.292889Z","shell.execute_reply.started":"2023-03-21T05:52:51.275771Z","shell.execute_reply":"2023-03-21T05:52:51.290865Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission_resnet50.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:14:19.287538Z","iopub.execute_input":"2023-03-21T06:14:19.289119Z","iopub.status.idle":"2023-03-21T06:14:19.318496Z","shell.execute_reply.started":"2023-03-21T06:14:19.289041Z","shell.execute_reply":"2023-03-21T06:14:19.317583Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Displaying Test images and predicted label","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"IMG_SIZE = (28,28,1)\n# Sample digit image\nsample_images = test_csv.iloc[0:10,0:].values.reshape((-1,)+IMG_SIZE)\nsample_images2 = test_csv.iloc[10:20,0:].values.reshape((-1,)+IMG_SIZE)\nsample_images3 = test_csv.iloc[20:30,0:].values.reshape((-1,)+IMG_SIZE)\nfig, axs = plt.subplots(1, sample_images.shape[0], figsize=(12,12))\nfor i,img in enumerate(sample_images):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {result[i]}')\n    axs[i].xaxis.label.set_color('red')\nfig, axs = plt.subplots(1, sample_images2.shape[0],figsize=(12,12))\nfor i,img in enumerate(sample_images2):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {result[i + 10]}')\n    axs[i].xaxis.label.set_color('red')\nfig, axs = plt.subplots(1, sample_images3.shape[0],figsize=(12,12))\nfor i,img in enumerate(sample_images3):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {result[i + 20]}')\n    axs[i].xaxis.label.set_color('red')","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:22:03.532459Z","iopub.execute_input":"2023-03-21T06:22:03.533241Z","iopub.status.idle":"2023-03-21T06:22:04.474869Z","shell.execute_reply.started":"2023-03-21T06:22:03.533182Z","shell.execute_reply":"2023-03-21T06:22:04.473777Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Accuracy : 0.93% on 30 random examples","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"## Confusion Matrix ResNet50","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"import seaborn as sns\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\naccuracies = []\ndef confusion(y_true,y_pred):\n    fig = plt.figure(figsize=(8,8)) # Set Figure\n    mat = confusion_matrix(y_true, y_pred) # Confusion matrix\n\n    # Plot Confusion matrix\n    sns.set(font_scale=1)\n    sns.heatmap(mat.T, square=True, annot=True, cbar=True, cmap=\"crest\",fmt=\"d\")\n    plt.title(\"Confusion Matrix\")\n    plt.xlabel('Predicted Values')\n    plt.ylabel('True Values');\n    plt.show();\n    print(accuracy_score(y_true,y_pred))\n    accuracies.append(accuracy_score(y_true,y_pred))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:36:29.495229Z","iopub.execute_input":"2023-03-21T06:36:29.495650Z","iopub.status.idle":"2023-03-21T06:36:29.502830Z","shell.execute_reply.started":"2023-03-21T06:36:29.495612Z","shell.execute_reply":"2023-03-21T06:36:29.502134Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = model.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:29:02.320911Z","iopub.execute_input":"2023-03-21T06:29:02.322248Z","iopub.status.idle":"2023-03-21T06:29:44.200029Z","shell.execute_reply.started":"2023-03-21T06:29:02.322185Z","shell.execute_reply":"2023-03-21T06:29:44.198774Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.argmax(np.array(y_test),1)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:31:19.719407Z","iopub.execute_input":"2023-03-21T06:31:19.720045Z","iopub.status.idle":"2023-03-21T06:31:19.729663Z","shell.execute_reply.started":"2023-03-21T06:31:19.719999Z","shell.execute_reply":"2023-03-21T06:31:19.728058Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"confusion(np.argmax(np.array(y_test),1),np.argmax(y_pred,1))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:36:37.021468Z","iopub.execute_input":"2023-03-21T06:36:37.022555Z","iopub.status.idle":"2023-03-21T06:36:37.526443Z","shell.execute_reply.started":"2023-03-21T06:36:37.022512Z","shell.execute_reply":"2023-03-21T06:36:37.524952Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# VGG16","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"model_vgg16 = Sequential()\n\nmodel_vgg16.add(tf.keras.applications.vgg16.VGG16(input_shape = (75, 75, 3), \n                                include_top = False, \n                                weights = 'imagenet'))\n\nmodel_vgg16.add(Flatten())\nmodel_vgg16.add(Dense(128, activation='relu'))\nmodel_vgg16.add(Dense(10, activation='softmax'))\n\nmodel_vgg16.layers[0].trainable = False\n\nmodel_vgg16.compile(optimizer=keras.optimizers.Adam(learning_rate=0.0001), loss='categorical_crossentropy', metrics=['accuracy', f1_m])","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:43:00.961612Z","iopub.execute_input":"2023-03-21T06:43:00.962145Z","iopub.status.idle":"2023-03-21T06:43:03.648013Z","shell.execute_reply.started":"2023-03-21T06:43:00.962101Z","shell.execute_reply":"2023-03-21T06:43:03.646532Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_vgg16.summary()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:43:04.826058Z","iopub.execute_input":"2023-03-21T06:43:04.826437Z","iopub.status.idle":"2023-03-21T06:43:04.850352Z","shell.execute_reply.started":"2023-03-21T06:43:04.826402Z","shell.execute_reply":"2023-03-21T06:43:04.849376Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_vgg16 = model_vgg16.fit(X_train,y_train, validation_data=(X_val,y_val), epochs=5)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T06:43:14.743146Z","iopub.execute_input":"2023-03-21T06:43:14.743557Z","iopub.status.idle":"2023-03-21T07:58:04.909147Z","shell.execute_reply.started":"2023-03-21T06:43:14.743520Z","shell.execute_reply":"2023-03-21T07:58:04.908341Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pickle.dump(model_vgg16, open('model_vgg16_feature_extraction.sav', 'wb'))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T07:58:04.911212Z","iopub.execute_input":"2023-03-21T07:58:04.911776Z","iopub.status.idle":"2023-03-21T07:58:05.302702Z","shell.execute_reply.started":"2023-03-21T07:58:04.911740Z","shell.execute_reply":"2023-03-21T07:58:05.301109Z"},"pycharm":{"name":"#%%\n"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loaded_model_vgg16 = pickle.load(open('model_vgg16_feature_extraction.sav', 'rb'))","metadata":{"execution":{"iopub.status.busy":"2023-03-20T11:10:54.691749Z","iopub.execute_input":"2023-03-20T11:10:54.692218Z","iopub.status.idle":"2023-03-20T11:10:55.684852Z","shell.execute_reply.started":"2023-03-20T11:10:54.692174Z","shell.execute_reply":"2023-03-20T11:10:55.683336Z"},"pycharm":{"name":"#%%\n"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## VGG16 result visualization","metadata":{}},{"cell_type":"code","source":"acc_vgg16 = history_vgg16.history['accuracy']\nval_acc_vgg16 = history_vgg16.history['val_accuracy']\n\nloss_vgg16 = history_vgg16.history['loss']\nval_loss_vgg16 = history_vgg16.history['val_loss']\n\nplt.figure(figsize=(8, 8))\nplt.subplot(2, 1, 1).set_facecolor('white')\nplt.grid(color='b',linestyle='dotted', linewidth=0.5)\nplt.plot(acc_vgg16, 'o-',label='Training Accuracy', color='b')\nplt.plot(val_acc_vgg16, 'o-',label='Validation Accuracy', color='g')\nplt.legend(loc='lower right')\nplt.ylabel('Accuracy')\nplt.ylim([min(plt.ylim()),1])\nplt.title('Training and Validation Accuracy')\n\nplt.subplot(2, 1, 2).set_facecolor('white')\nplt.grid(color='b', linestyle='dotted', linewidth=0.5)\nplt.plot(loss_vgg16, 'o-',label='Training Loss', color='b')\nplt.plot(val_loss_vgg16, 'o-',label='Validation Loss', color='g')\nplt.legend(loc='upper right')\nplt.ylabel('Cross Entropy')\n# plt.ylim([0,1.0])\nplt.title('Training and Validation Loss')\nplt.xlabel('epoch')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:00:07.245830Z","iopub.execute_input":"2023-03-21T08:00:07.246199Z","iopub.status.idle":"2023-03-21T08:00:07.623783Z","shell.execute_reply.started":"2023-03-21T08:00:07.246171Z","shell.execute_reply":"2023-03-21T08:00:07.622002Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submitting VGG16 result","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"result_test_vgg16 = model_vgg16.predict(X_submission)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:00:36.316415Z","iopub.execute_input":"2023-03-21T08:00:36.316852Z","iopub.status.idle":"2023-03-21T08:09:53.319801Z","shell.execute_reply.started":"2023-03-21T08:00:36.316808Z","shell.execute_reply":"2023-03-21T08:09:53.318203Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_vgg16 = np.argmax(result_test_vgg16,1)\nresult_vgg16","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:10:04.305796Z","iopub.execute_input":"2023-03-21T08:10:04.306372Z","iopub.status.idle":"2023-03-21T08:10:04.313942Z","shell.execute_reply.started":"2023-03-21T08:10:04.306341Z","shell.execute_reply":"2023-03-21T08:10:04.313078Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:10:07.153087Z","iopub.execute_input":"2023-03-21T08:10:07.153524Z","iopub.status.idle":"2023-03-21T08:10:07.178814Z","shell.execute_reply.started":"2023-03-21T08:10:07.153484Z","shell.execute_reply":"2023-03-21T08:10:07.177524Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Label'] = result_vgg16\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:10:24.686234Z","iopub.execute_input":"2023-03-21T08:10:24.686611Z","iopub.status.idle":"2023-03-21T08:10:24.700882Z","shell.execute_reply.started":"2023-03-21T08:10:24.686576Z","shell.execute_reply":"2023-03-21T08:10:24.699444Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission_vgg16.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:10:35.937185Z","iopub.execute_input":"2023-03-21T08:10:35.937595Z","iopub.status.idle":"2023-03-21T08:10:35.972381Z","shell.execute_reply.started":"2023-03-21T08:10:35.937559Z","shell.execute_reply":"2023-03-21T08:10:35.971246Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Displaying test.csv prediction","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"IMG_SIZE = (28,28,1)\n# Sample digit image\nsample_images = test_csv.iloc[0:10,0:].values.reshape((-1,)+IMG_SIZE)\nsample_images2 = test_csv.iloc[10:20,0:].values.reshape((-1,)+IMG_SIZE)\nsample_images3 = test_csv.iloc[20:30,0:].values.reshape((-1,)+IMG_SIZE)\nfig, axs = plt.subplots(1, sample_images.shape[0], figsize=(12,12))\nfor i,img in enumerate(sample_images):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {result_vgg16[i]}')\n    axs[i].xaxis.label.set_color('red')\nfig, axs = plt.subplots(1, sample_images2.shape[0],figsize=(12,12))\nfor i,img in enumerate(sample_images2):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {result_vgg16[i + 10]}')\n    axs[i].xaxis.label.set_color('red')\nfig, axs = plt.subplots(1, sample_images3.shape[0],figsize=(12,12))\nfor i,img in enumerate(sample_images3):\n    axs[i].imshow(img)\n    axs[i].set_xlabel(f'label: {result_vgg16[i + 20]}')\n    axs[i].xaxis.label.set_color('red')","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### accuracy : 100% on 30 examples","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"## Confusion Matrix VGG16","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"y_pred = model_vgg16.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:12:41.769298Z","iopub.execute_input":"2023-03-21T08:12:41.769700Z","iopub.status.idle":"2023-03-21T08:14:05.710343Z","shell.execute_reply.started":"2023-03-21T08:12:41.769667Z","shell.execute_reply":"2023-03-21T08:14:05.709027Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"confusion(np.argmax(np.array(y_test),1),np.argmax(y_pred,1))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:14:09.701375Z","iopub.execute_input":"2023-03-21T08:14:09.702091Z","iopub.status.idle":"2023-03-21T08:14:10.185930Z","shell.execute_reply.started":"2023-03-21T08:14:09.702038Z","shell.execute_reply":"2023-03-21T08:14:10.183661Z"},"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nThe Hand-written Digit Recognizer competition provided an opportunity to consolidate computer vision and supervised classification algorithm skills. The implementation of transfer learning techniques using two different CNN architectures, VGG16 and ResNet50, allowed for a comparison of their performance on the dataset. VGG16 outperformed ResNet50 with an accuracy of 0.978, which is slightly better than traditional algorithms like SVM and K-NN. ResNet50 achieved an accuracy of 0.962, still comparable to non-parametric models. \n\nWhile both CNN models performed well, VGG16 proved to be the more efficient choice for this particular competition. Additionally, we could further fine-tune the VGG16 model by fine-tuning its layers to get better results, but the need for such complexity is not necessary since it would require a large computational cost to gain 0.01-0.015 in accuracy (i.e 0.99).","metadata":{}}]}