{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n***\n***\n\n DIGIT RECOGNIZER
\n
\n\n***\n***\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-04-18T21:14:20.618527Z","iopub.execute_input":"2022-04-18T21:14:20.618863Z","iopub.status.idle":"2022-04-18T21:14:20.646056Z","shell.execute_reply.started":"2022-04-18T21:14:20.618759Z","shell.execute_reply":"2022-04-18T21:14:20.645191Z"}}},{"cell_type":"markdown","source":"🐧🐧🐧 Yet another digit recognizer notebook among hundreds. 🐧🐧🐧\n\nSince my main objective is to test/improve my Machine Learning abilities (especially PCA and Hyperparameter Tuning), I will use SVM + PCA + GridSearchCV in this notebook. I prefer SVM because a quick search shows SVM is better on digit recognizing. I also tested with many alghorithms with default parameters. \n\nYou can see here many (possibly) primitive approachs such as exhaustively searching optimal PCA component count from 1 to 190 (That I did on my computer by making predictions with each PCA component count. I do not want to bring that monster to this notebook for the sake of simplicity) or tuning parameters step by step in GridSearchCV. I have tried RandomSearch, then decided to use GridSearchCV.\n\nI see CNN is also used for digit recognizing. Yet my goal is improving myself one step further each time.\n\nI may use small sample size for GridSearchCV. Yet I do not feel comfortable with this, I just want to see the results this way. \n\nPlease advise me if all my approach on GridSearchCV is reasonable.\n\nIn the future, I am thinking coming with better PCA and Hyperparameter Tuning solutions for this dataset. I love this dataset, but my computer does not.\n\n**I will be happy for any comment, especially the ones pointing my mistakes or giving advices.**","metadata":{}},{"cell_type":"markdown","source":"\n Index
\n
\n\n1. [Import Libraries & Load Data](#1)\n1. [Some Configuration](#2)\n1. [Exploration](#3)\n1. [PCA - Principal Component Analysis](#4)\n1. [Model](#5)\n1. [Hyperparameter Tuning](#6)\n1. [Prediction](#7)\n1. [Submit](#8)","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport sys\nimport os\n\nimport sklearn.neighbors._base\nfrom sklearn.decomposition import PCA as sklearnPCA\nfrom sklearn.svm import SVC\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split, GridSearchCV\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import classification_report\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:03:11.058544Z","iopub.execute_input":"2022-05-08T18:03:11.059399Z","iopub.status.idle":"2022-05-08T18:03:12.684197Z","shell.execute_reply.started":"2022-05-08T18:03:11.059218Z","shell.execute_reply":"2022-05-08T18:03:12.683156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndata_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\nlabels = data_train[\"label\"]\nlabels = labels.astype('int')\n\nfeatures = data_train.drop(\"label\", axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:03:12.686308Z","iopub.execute_input":"2022-05-08T18:03:12.686607Z","iopub.status.idle":"2022-05-08T18:03:18.58004Z","shell.execute_reply.started":"2022-05-08T18:03:12.686573Z","shell.execute_reply":"2022-05-08T18:03:18.579088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Let pandas show all columns\npd.options.display.width = 0\n\n# For better viusalisation increase horizontal space for subplots\n#plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.4)\n\n# To avoid some errors\nsys.modules['sklearn.neighbors.base'] = sklearn.neighbors._base\n\n# Seaborn theme setings\nsns.set_theme(style = \"whitegrid\", palette=\"deep\")","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:03:18.581504Z","iopub.execute_input":"2022-05-08T18:03:18.581777Z","iopub.status.idle":"2022-05-08T18:03:18.588321Z","shell.execute_reply.started":"2022-05-08T18:03:18.581737Z","shell.execute_reply":"2022-05-08T18:03:18.587441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Since [Digit Recogizer](https://www.kaggle.com/competitions/digit-recognizer/data?select=train.csv) explains clearly, we are not exploring database structure.\n\nEvery row has a 28x28 image which projected to a row with 784 column. Label column shows the digit drawn by the user. Other columns consists of values between 0-255 indicating lightness/darkness of the related pixel.\n\nWe know there is no missing data by the nature of the data (also checked Competition Data Tab). Otherwise it would be a imputation hell.\n\nLet's see some example images.","metadata":{}},{"cell_type":"code","source":"def get_image_matrix(row, dataset):\n row_values = dataset.iloc[row].values\n image_matrix = row_values.reshape(28, 28)\n return image_matrix\n\nfig, ax = plt.subplots(1, 5, figsize=(12, 9), dpi=120)\nplt.setp(ax, xticks=[], yticks=[])\n\nax_num=0\nfor i in [1,42,783,27000,127]:\n ax[ax_num].imshow(get_image_matrix(i, features))\n ax_num+=1","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:03:18.590239Z","iopub.execute_input":"2022-05-08T18:03:18.590489Z","iopub.status.idle":"2022-05-08T18:03:18.856465Z","shell.execute_reply.started":"2022-05-08T18:03:18.59046Z","shell.execute_reply":"2022-05-08T18:03:18.855509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"I have searched for optimal PCA component count from 1 to 190. As I have an average PC, I have stopped it at 190. Then I use SVM with default parameters. And I see it produces good results between 52-56.\n\nWhile experimenting, I tried making 28x28 matrix and taking sum of each row and column separately then added it to the row data produced by PCA. It does not improve results meaningfully. So I chose plain PCA approach. \n\nExperimenting with full dataset is resource hungry and I cannot move freely in the Kaggle notebook environment. ","metadata":{}},{"cell_type":"code","source":"n_comp = 52\npca = sklearnPCA(n_components=n_comp) #,whiten=True)\npca_train = pca.fit_transform(features)\npca_test = pca.transform(data_test)\n\nresult_train = pd.DataFrame(pca_train, columns=['PCA_%i' % i for i in range(n_comp)])\nresult_test = pd.DataFrame(pca_test, columns=['PCA_%i' % i for i in range(n_comp)]) ","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:03:18.857983Z","iopub.execute_input":"2022-05-08T18:03:18.858296Z","iopub.status.idle":"2022-05-08T18:03:24.15289Z","shell.execute_reply.started":"2022-05-08T18:03:18.858256Z","shell.execute_reply":"2022-05-08T18:03:24.151919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"scaler = StandardScaler()\ndata_train = scaler.fit_transform(result_train)\ndata_test = scaler.transform(result_test)\n\nx_train, x_test, y_train, y_test = train_test_split(result_train, labels, test_size=0.33, random_state=42)\n\nsvm = SVC(random_state=42)\n\nsvm.fit(x_train, y_train)\naccuracy_train = round(svm.score(x_train, y_train) * 100, 2)\naccuracy_test = round(svm.score(x_test, y_test) * 100, 2)\n\nprint(\"Training Accuracy: % {}\".format(accuracy_train))\nprint(\"Testing Accuracy: % {}\".format(accuracy_test))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:03:24.154937Z","iopub.execute_input":"2022-05-08T18:03:24.155673Z","iopub.status.idle":"2022-05-08T18:04:06.370321Z","shell.execute_reply.started":"2022-05-08T18:03:24.15562Z","shell.execute_reply":"2022-05-08T18:04:06.369262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"I am doing hyperparameter tuning in 2 steps. First trying to find the C value and then fine tuning C value. Because:\n* I do not want to experiment more with this resource hungry dataset. \n* I may use a small sample, but I do not like the idea with no rational basis.\n\nWhen I ran it on my computer: \n* I know the key parameters here are 'C', 'gamma' and 'kernel'. I do not want to use more parameters/values. \n* In the full list of kernel parameters the best one was 'rbf'. I simply use 'rbf' here.\n* For gamma, using values instead of ['auto','scale'] did not give good results. ","metadata":{}},{"cell_type":"code","source":"def print_grid_results(grid):\n # print best parameter after tuning\n print('*'*75)\n print(grid.best_params_)\n print('*'*75)\n\n # print classification report\n print(classification_report(y_test, grid_predictions))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:04:06.371931Z","iopub.execute_input":"2022-05-08T18:04:06.372308Z","iopub.status.idle":"2022-05-08T18:04:06.378255Z","shell.execute_reply.started":"2022-05-08T18:04:06.372271Z","shell.execute_reply":"2022-05-08T18:04:06.377444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"param_grid = {'C':[0.1, 1, 10],\n 'gamma':['auto','scale'],\n 'kernel':['rbf'], \n 'random_state':[42]\n }\n\ngrid = GridSearchCV(SVC(), param_grid, refit=True, verbose=0)\n\n# fitting the model for grid search\ngrid.fit(x_train, y_train)\n\ngrid_predictions = grid.predict(x_test)\n\nprint_grid_results(grid)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:04:06.379884Z","iopub.execute_input":"2022-05-08T18:04:06.380205Z","iopub.status.idle":"2022-05-08T18:31:58.381884Z","shell.execute_reply.started":"2022-05-08T18:04:06.380163Z","shell.execute_reply":"2022-05-08T18:31:58.380857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Fine tuning 'C' parameter (checking if it is between 10 and 20).**\n\n---","metadata":{}},{"cell_type":"code","source":"param_grid = {'C':np.arange(10,21,1),\n 'gamma':['scale'],\n 'kernel':['rbf'],\n 'random_state':[42]\n }\n\ngrid = GridSearchCV(SVC(), param_grid, refit=True, verbose=0)\n\n# fitting the model for grid search\ngrid.fit(x_train, y_train)\n\ngrid_predictions = grid.predict(x_test)\n\nprint_grid_results(grid)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:31:58.383311Z","iopub.execute_input":"2022-05-08T18:31:58.383559Z","iopub.status.idle":"2022-05-08T18:41:03.051554Z","shell.execute_reply.started":"2022-05-08T18:31:58.383528Z","shell.execute_reply":"2022-05-08T18:41:03.050554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Fine tuning 'C' parameter (checking if it is between 1 and 10).** \n\n---","metadata":{}},{"cell_type":"code","source":"param_grid = {'C':np.arange(1,10,1),\n 'gamma':['scale'],\n 'kernel':['rbf'],\n 'random_state':[42]\n }\n\ngrid = GridSearchCV(SVC(), param_grid, refit=True, verbose=0)\n\n# fitting the model for grid search\ngrid.fit(x_train, y_train)\n\ngrid_predictions = grid.predict(x_test)\n\nprint_grid_results(grid)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:41:03.053989Z","iopub.execute_input":"2022-05-08T18:41:03.05426Z","iopub.status.idle":"2022-05-08T18:48:37.833978Z","shell.execute_reply.started":"2022-05-08T18:41:03.054228Z","shell.execute_reply":"2022-05-08T18:48:37.833018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"svm = SVC(C=7, gamma='scale', kernel='rbf', random_state=42)\n\nsvm.fit(x_train, y_train)\naccuracy_train = round(svm.score(x_train, y_train) * 100, 2)\naccuracy_test = round(svm.score(x_test, y_test) * 100, 2)\n\nprint(\"Training Accuracy: % {}\".format(accuracy_train))\nprint(\"Testing Accuracy: % {}\".format(accuracy_test))","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:49:56.943603Z","iopub.execute_input":"2022-05-08T18:49:56.943981Z","iopub.status.idle":"2022-05-08T18:50:34.514254Z","shell.execute_reply.started":"2022-05-08T18:49:56.943942Z","shell.execute_reply":"2022-05-08T18:50:34.513616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"predictions = svm.predict(result_test)\n\ndata_submission = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\ndata_submission['Label'] = predictions\ndata_submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T18:50:45.329776Z","iopub.execute_input":"2022-05-08T18:50:45.330147Z","iopub.status.idle":"2022-05-08T18:51:04.733009Z","shell.execute_reply.started":"2022-05-08T18:50:45.330107Z","shell.execute_reply":"2022-05-08T18:51:04.732149Z"},"trusted":true},"execution_count":null,"outputs":[]}]}