{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"***\n***\n\n DIGIT RECOGNIZER - CNN
\n
\n\n***\n***\n\nThis is my second Digit Recognizer Notebook. It has 0.9947 score.\n\nThe first one is with SVM + PCA + Hyperparameter Tuning. (https://www.kaggle.com/code/darkwyvern/digit-recognizer-pca-svm)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-17T13:10:31.904382Z","iopub.execute_input":"2022-05-17T13:10:31.904662Z","iopub.status.idle":"2022-05-17T13:10:31.916769Z","shell.execute_reply.started":"2022-05-17T13:10:31.904632Z","shell.execute_reply":"2022-05-17T13:10:31.916166Z"}}},{"cell_type":"markdown","source":"\n Index
\n
\n\n1. [Import Libraries & Load Data](#1)\n1. [Some Configuration](#2)\n1. [Exploration](#3)\n1. [Process The Data](#4)\n1. [Model](#5)\n1. [Data Augmentation](#6)\n1. [Fit The model](#7)\n1. [Error Visualisation](#8)\n1. [Prediction](#9)\n1. [Submission](#10)","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D,BatchNormalization\nfrom keras.models import Sequential\nfrom keras.optimizers import adam_v2\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.utils.np_utils import to_categorical\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import MinMaxScaler\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint,ReduceLROnPlateau\nfrom sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:07:50.487696Z","iopub.execute_input":"2022-05-27T20:07:50.487933Z","iopub.status.idle":"2022-05-27T20:07:57.208560Z","shell.execute_reply.started":"2022-05-27T20:07:50.487862Z","shell.execute_reply":"2022-05-27T20:07:57.207766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndata_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:08:03.143699Z","iopub.execute_input":"2022-05-27T20:08:03.144592Z","iopub.status.idle":"2022-05-27T20:08:10.474738Z","shell.execute_reply.started":"2022-05-27T20:08:03.144542Z","shell.execute_reply":"2022-05-27T20:08:10.473947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Let pandas show all columns\npd.options.display.width = 0\n\n# Seaborn theme setings\nsns.set_theme(palette=\"deep\")","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:08:13.419256Z","iopub.execute_input":"2022-05-27T20:08:13.419514Z","iopub.status.idle":"2022-05-27T20:08:13.426383Z","shell.execute_reply.started":"2022-05-27T20:08:13.419485Z","shell.execute_reply":"2022-05-27T20:08:13.423437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Since [Digit Recogizer](https://www.kaggle.com/competitions/digit-recognizer/data?select=train.csv) explains clearly, we are not exploring database structure.\n\nEvery row has a 28x28 image which projected to a row with 784 column. Label column shows the digit drawn by the user. Other columns consists of values between 0-255 indicating lightness/darkness of the related pixel.\n\nWe know there is no missing data by the nature of the data (also checked Competition Data Tab). Otherwise it would be a imputation hell.\n\nLet's see some example images.","metadata":{}},{"cell_type":"code","source":"# Prepare x_train and y_train\nx_train = data_train.drop(labels=[\"label\"], axis=1)\ny_train = data_train[\"label\"]\n\ndef get_image_matrix(row, dataset):\n row_values = dataset.iloc[row].values\n image_matrix = row_values.reshape(28, 28)\n return image_matrix\n\nfig, ax = plt.subplots(1, 5, figsize=(12, 9), dpi=120)\nplt.setp(ax, xticks=[], yticks=[])\n\nax_num=0\nfor i in [1,225,648,75,138]:\n ax[ax_num].imshow(get_image_matrix(i, x_train))\n ax_num+=1","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:08:17.516485Z","iopub.execute_input":"2022-05-27T20:08:17.516780Z","iopub.status.idle":"2022-05-27T20:08:17.868132Z","shell.execute_reply.started":"2022-05-27T20:08:17.516747Z","shell.execute_reply":"2022-05-27T20:08:17.867461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Do MinMax Scaling on both train and test data.\nscaler = MinMaxScaler()\nx_train[:] = scaler.fit_transform(x_train)\ndata_test[:] = scaler.transform(data_test)","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:08:21.268581Z","iopub.execute_input":"2022-05-27T20:08:21.268853Z","iopub.status.idle":"2022-05-27T20:09:25.106638Z","shell.execute_reply.started":"2022-05-27T20:08:21.268824Z","shell.execute_reply":"2022-05-27T20:09:25.105883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshape the data for CNN.\nx_train = x_train.values.reshape(-1, 28, 28, 1)\ndata_test = data_test.values.reshape(-1, 28, 28, 1)\n\n# We need 10 classes from 0 to 9 (inclusive)\ny_train = to_categorical(y_train, num_classes=10)\n\n# Train and Test Split\nx_train, x_test, y_train, y_test = train_test_split(x_train, y_train, test_size=0.33, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:09:32.697095Z","iopub.execute_input":"2022-05-27T20:09:32.697431Z","iopub.status.idle":"2022-05-27T20:09:33.400370Z","shell.execute_reply.started":"2022-05-27T20:09:32.697391Z","shell.execute_reply":"2022-05-27T20:09:33.399605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"CNN will first look for edges, lines, curves etc. Hence, I guessed which features needs to be recognized by first convolutional layer. So I start with 16 features. I think more depth is necessary for first convolutional layer, so I choose kernel size as (5,5). Then more feature will be needed in the following layers.\n\nI have experimented with layers and the below one is slightly better than my first layer model.\n\nThough Dropout rate 0.50 is recommended, (through a basic search) it seems lower levels work better. I will use 0.25.\n\nI prefer BatchNormalization in convolutional layers instead of Dropout.\n\nI will use recommended Max Pooling parameters.\n\nMax Pooling will reduce the number of features. So I am doubling the filters parameters to balance it as everbody does.","metadata":{}},{"cell_type":"code","source":"model = Sequential()\n\n# I see two Conv2D increases performance here.\nmodel.add(Conv2D(filters=16, kernel_size=(5, 5), padding=\"Same\", activation=\"relu\", input_shape=(28, 28, 1)))\nmodel.add(Conv2D(filters=16, kernel_size=(5, 5), padding=\"Same\", activation=\"relu\", input_shape=(28, 28, 1)))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(filters=32, kernel_size=(4, 4), padding=\"Same\", activation=\"relu\"))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=(2, 2)))\n\nmodel.add(Conv2D(filters=64, kernel_size=(3, 3), padding=\"Same\", activation=\"relu\"))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=(2, 2)))\n\nmodel.add(Flatten())\nmodel.add(Dense(1024, activation=\"relu\"))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(10, activation=\"softmax\"))","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:09:37.727574Z","iopub.execute_input":"2022-05-27T20:09:37.728054Z","iopub.status.idle":"2022-05-27T20:09:40.612383Z","shell.execute_reply.started":"2022-05-27T20:09:37.727995Z","shell.execute_reply":"2022-05-27T20:09:40.611696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will use Adam as optimizer. It seems to work well on digit recognizing. I will stick to recommended parameter values.\n\nFor better results I will use 1000 epochs along with batch_size 350, since it mostly makes an EarlyStop before 250 epochs.\n\nI have tried Adadelta for adaptive learning rates. Then I changed it to Adam optimizer after seeing no performance gain with my model. I am aware minimum learning rate (min_lr) in ReduceLROnPlateau callback is a bit extreme. Since EarlyStopping is on duty, it will eventually stop if there is no decrease in val_loss.\n\nI will both monitor val_loss and val_accuracy via ModelCheckpoint. I see best val_loss value is not far from val_loss value of best val_accuracy. Hence, I will use best val_accuracy in my model against a little bit confidence loss in predictions.","metadata":{}},{"cell_type":"code","source":"optimizer = adam_v2.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)\n\nmodel.compile(optimizer=optimizer, loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n\nepochs = 1000\nbatch_size = 350\n\ncallbacks = [EarlyStopping(monitor='val_loss', patience=100,verbose=1),\n ModelCheckpoint(filepath='best_model_loss.h5', monitor='val_loss', save_best_only=True,verbose=0),\n ModelCheckpoint(filepath='best_model_accuracy.h5', monitor='val_accuracy', save_best_only=True,verbose=0),\n ReduceLROnPlateau(monitor=\"val_loss\",factor=0.1, patience=25,verbose=1,mode=\"min\",min_lr=0.000000001,cooldown=1)]","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:10:23.251275Z","iopub.execute_input":"2022-05-27T20:10:23.251536Z","iopub.status.idle":"2022-05-27T20:10:23.266451Z","shell.execute_reply.started":"2022-05-27T20:10:23.251508Z","shell.execute_reply":"2022-05-27T20:10:23.265714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"By creating new versions of the images by data augmentation train dataset size will expand for better results.","metadata":{}},{"cell_type":"code","source":"# data augmentation\ndatagen = ImageDataGenerator(\n rotation_range=15, # Rotate images in range 15 degrees at random\n zoom_range=0.1, # Zoom images 10% at random\n width_shift_range=0.1, # Shift images horizontally 10% at random\n height_shift_range=0.1) # Shift images vertically 10% at random\n\ndatagen.fit(x_train)","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:09:55.339796Z","iopub.execute_input":"2022-05-27T20:09:55.340073Z","iopub.status.idle":"2022-05-27T20:09:55.413088Z","shell.execute_reply.started":"2022-05-27T20:09:55.340040Z","shell.execute_reply":"2022-05-27T20:09:55.412359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"history = model.fit(datagen.flow(x_train, y_train, batch_size=batch_size), epochs=epochs,\n validation_data=(x_test, y_test), steps_per_epoch=x_train.shape[0] // batch_size, verbose=0,callbacks=callbacks)\n\n# plot training history for loss\nplt.plot(history.history['loss'], label='loss train')\nplt.plot(history.history['val_loss'], label='loss test')\nplt.legend()\nplt.show()\n\n# plot training history for accuracy\nplt.plot(history.history['accuracy'], label='acc train')\nplt.plot(history.history['val_accuracy'], label='acc test')\nplt.legend()\nplt.show()\n\n\nmodel.load_weights('best_model_loss.h5')\n# evaluate the model for val_loss\ntrain_accuracy = model.evaluate(x_train, y_train, verbose=0)\ntest_accuracy = model.evaluate(x_test, y_test, verbose=0)\n\nprint()\nprint(\"*\"*75)\nprint(\"train (val_loss): \", train_accuracy)\nprint(\"test (val_loss): \", test_accuracy)\nprint(\"*\"*75)\n\n\nmodel.load_weights('best_model_accuracy.h5')\n# evaluate the model val_accuracy\ntrain_accuracy = model.evaluate(x_train, y_train, verbose=0)\ntest_accuracy = model.evaluate(x_test, y_test, verbose=0)\n\nprint()\nprint(\"*\"*75)\nprint(\"train (val_accuracy): \", train_accuracy)\nprint(\"test (val_accuracy): \", test_accuracy)\nprint(\"*\"*75)","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:10:52.183438Z","iopub.execute_input":"2022-05-27T20:10:52.183689Z","iopub.status.idle":"2022-05-27T20:29:11.398551Z","shell.execute_reply.started":"2022-05-27T20:10:52.183661Z","shell.execute_reply":"2022-05-27T20:29:11.396597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"First we will make predictions for x_test to see them in the confusion matrix. The below confusion matrix shows us most false predictions are from the digits that can not be recognized even by a human. (like 3 vs 8, 7 vs 1).","metadata":{}},{"cell_type":"code","source":"results = model.predict(x_test)\n#Get the max prediction value (the best prediction out of the 10 predictions in the np array)\nresults = np.argmax(results,axis = 1)\ny_test=np.argmax(y_test,axis = 1)\n\ncm=confusion_matrix(y_test,results)\ncmd=ConfusionMatrixDisplay(cm,display_labels=(np.arange(10)))\nfig, ax = plt.subplots(figsize=(10,10))\nax.grid(False)\ncmd.plot(ax=ax)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:53:29.822469Z","iopub.execute_input":"2022-05-27T20:53:29.822743Z","iopub.status.idle":"2022-05-27T20:53:31.111494Z","shell.execute_reply.started":"2022-05-27T20:53:29.822716Z","shell.execute_reply":"2022-05-27T20:53:31.110831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will visualize incorrect predictions:","metadata":{}},{"cell_type":"code","source":"correct_pred = np.nonzero(results == y_test)[0]\nincorrect_pred = np.nonzero(results != y_test)[0]\n\nsize=len(incorrect_pred)\n\nfig, axes = plt.subplots(10,10, sharex=True, sharey=True, figsize=(20, 20)) \nplt.setp(axes, xticks=[], yticks=[])\naxes=axes.flatten()\n\nfor i, incorrect in enumerate(incorrect_pred):\n axes[i].imshow(x_test[incorrect], cmap='gray', interpolation='none')\n axes[i].set_title(\"P:{} - T:{}\".format(results[incorrect], y_test[incorrect]))\n\nfor ax in axes[size:]:\n ax.remove()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:46:22.960922Z","iopub.execute_input":"2022-05-27T20:46:22.961606Z","iopub.status.idle":"2022-05-27T20:46:28.826244Z","shell.execute_reply.started":"2022-05-27T20:46:22.961570Z","shell.execute_reply":"2022-05-27T20:46:28.825532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It clearly shows these are hard to predict even for humans.In the light of these facts we can say the model works fine.\n\nIt is Convolutional Neural Network's success, not mine. All the above code shows how powerfull it is. Even with a simple setup and a little bit tuning you can reach high prediction values.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Now we are making the real prediction.","metadata":{}},{"cell_type":"code","source":"results = model.predict(data_test)\n#Get the max prediction value (the best prediction out of the 10 predictions in the np array)\nresults = np.argmax(results,axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:47:06.865140Z","iopub.execute_input":"2022-05-27T20:47:06.865401Z","iopub.status.idle":"2022-05-27T20:47:08.603691Z","shell.execute_reply.started":"2022-05-27T20:47:06.865372Z","shell.execute_reply":"2022-05-27T20:47:08.602957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"results = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-27T20:47:12.193472Z","iopub.execute_input":"2022-05-27T20:47:12.193938Z","iopub.status.idle":"2022-05-27T20:47:12.241469Z","shell.execute_reply.started":"2022-05-27T20:47:12.193902Z","shell.execute_reply":"2022-05-27T20:47:12.240782Z"},"trusted":true},"execution_count":null,"outputs":[]}]}