{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Objective of Handwritten Digit Recognition\n\nA classic problem in the field of pattern recognition is that of handwritten digit recognition. Suppose that you have images of handwritten digits ranging from 0-9 written by various people in boxes of a specific size - similar to the application forms in banks and universities.\n\nThe goal is to develop a model that can correctly identify the digit (between 0-9) written in an image. \n\nWe are required to develop a model using Support Vector Machine which should correctly classify the handwritten digits from 0-9 based on the pixel values given as features. Thus, this is a 10-class classification problem. "},{"metadata":{},"cell_type":"markdown","source":"### Data Description\n\nFor this problem, we use the MNIST data which is a large database of handwritten digits. The 'pixel values' of each digit (image) comprise the features, and the actual number between 0-9 is the label. "},{"metadata":{},"cell_type":"markdown","source":"### 1. Data Understanding and Exploration\n\nLet's first have a look at the dataset and understand the size, attribute names etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"# libraries\nimport pandas as pd\nimport numpy as np\nimport gc\nfrom sklearn.svm import SVC\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import GridSearchCV\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.preprocessing import scale\nfrom sklearn.metrics import classification_report\nimport os\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\ngc.collect()\n\n# hide warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# dataset\ndigits = pd.read_csv('../input/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"digits.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Inspect the elements of the dataset\nprint(digits.shape)\nprint(digits.columns)\nprint(digits.info())\nprint(digits.describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(digits.columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Data Cleaning\n\nIn this we wil check the presence of NULL values and outliers if any exist"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Checking for the NULL values\nprint(digits.isnull().values.any())\nprint(digits.isnull().values.sum())\nprint(digits.isnull().sum()*100/digits.shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this, we can clearly see the absence of NULL values"},{"metadata":{"trusted":true},"cell_type":"code","source":"digits_mean = digits.groupby('label').mean()\ndigits_mean.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Through a heat map, we can find out the correlation between the variables\nplt.figure(figsize=(18, 10))\nsns.heatmap(digits_mean)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since, the data is too large so we can not compute anything from the heat map"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Considering label as the target value, will drop it from the main data set\nlabel = digits.label\ndigits=digits.drop('label',axis=1)\nprint(\"Data Shape: \",digits.shape)\nprint(\"Label Shape: \",label.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the each pixels in the figure of size 28,28\nfor x in range(0,9):\n    train_0=digits[label==x]\n    digits_new=[]\n    for idx in train_0.index:\n        val=train_0.loc[idx].values.reshape(28,28)\n        digits_new.append(val)\n    plt.figure(figsize=(25,25))   \n    for x in range(1,5):\n        ax1=plt.subplot(1, 20, x)\n        ax1.imshow(digits_new[x],cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(label)\nplt.show","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Analysing Outliers\ndigits.describe(percentiles=[0.1,0.25,0.5,0.75,0.9,0.99])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.boxplot(x=label) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see that there are no outlier present in the data. Hence, we are ready for the modelling."},{"metadata":{"trusted":true},"cell_type":"code","source":"digits.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3. Data Preparation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# scaling the features\n# note that the scale function standardises each column, i.e.\n# x = x-mean(x)/std(x)\n\nfrom sklearn.preprocessing import scale\nX = scale(digits)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train test split\nX_train, X_test, y_train, y_test = train_test_split(X, label, test_size = 0.8, random_state = 40)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the data set is too large so we have splitted the train n test in the ration of 2:8  that is 20% train data and 80% test data"},{"metadata":{},"cell_type":"markdown","source":"### 4. Model Building\n\nLet's fist build two basic models - linear and non-linear with default hyperparameters, and compare the accuracies."},{"metadata":{"trusted":true},"cell_type":"code","source":"# linear model\n\nmodel_linear = SVC(kernel='linear')\nmodel_linear.fit(X_train, y_train)\n\n# predict\ny_pred = model_linear.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# confusion matrix and accuracy\n# As the labels are not binary so we have used another average option to calculate precision and recall i.e 'micro'\n\n# accuracy\nprint(\"accuracy:\", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), \"\\n\")\n\n# precision\nprint(\"precision\", metrics.precision_score(y_true = y_test, y_pred = y_pred, average='micro'), \"\\n\")\n\n# recall/sensitivity\nprint(\"recall\", metrics.recall_score(y_true = y_test, y_pred = y_pred, average='micro'), \"\\n\")\n\n# cm\nprint(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# As this is the 10-class classification problem so Class-wise Accuracy shows the precision, recall and f1-score of all the 10 classes\n# Through which we can estimate the accuracy\nprint(classification_report(y_true=y_test, y_pred=y_pred ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The linear model gives approx. 91% accuracy. Let's look at a sufficiently non-linear model with randomly chosen hyperparameters."},{"metadata":{"trusted":true},"cell_type":"code","source":"# non-linear model\n# using rbf kernel, C=1, default value of gamma\n\n# model\nnon_linear_model = SVC(kernel='rbf')\n\n# fit\nnon_linear_model.fit(X_train, y_train)\n\n# predict\ny_pred = non_linear_model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# confusion matrix and accuracy\n# As the labels are not binary so we have used another average option to calculate precision and recall i.e 'micro'\n\n# accuracy\nprint(\"accuracy:\", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), \"\\n\")\n\n# precision\nprint(\"precision\", metrics.precision_score(y_true = y_test, y_pred = y_pred, average='micro'), \"\\n\")\n\n# recall/sensitivity\nprint(\"recall\", metrics.recall_score(y_true = y_test, y_pred = y_pred, average='micro'), \"\\n\")\n\n# cm\nprint(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# As this is the 10-class classification problem so Class-wise Accuracy shows the precision, recall and f1-score of all the 10 classes\n# Through which we can estimate the accuracy\nprint(classification_report(y_true=y_test, y_pred=y_pred ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see that the value of precision, recall and f1-score is good and maintaining a value approx 90%+. Hence we can say that the class-wise accuracy will also be in range.                                                                              \nThe non-linear model gives approx. 94% accuracy. Thus, going forward, let's choose hyperparameters corresponding to non-linear models."},{"metadata":{},"cell_type":"markdown","source":"### Grid Search: Hyperparameter Tuning\n\nLet's now tune the model to find the optimal values of C and gamma corresponding to an RBF kernel. We'll use 5-fold cross validation"},{"metadata":{},"cell_type":"markdown","source":"Out of parameters below, we will be playing with Gamma and C, where\n\nGamma is the parameter of a Gaussian Kernel (to handle non-linear classification)\nand C the parameter for the soft margin cost function, also known as cost of misclassification. A large C gives you low bias and high variance and vice versa."},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating a KFold object with 5 splits \nfolds = KFold(n_splits = 5, shuffle = True, random_state = 101)\n\n# specify range of hyperparameters\n# Set the parameters by cross-validation\nhyper_params = [ {'gamma': [1e-2, 1e-3, 1e-4],\n                     'C': [1, 10, 100, 1000]}]\n\n\n# specify model\nmodel = SVC(kernel=\"rbf\")\n\n# set up GridSearchCV()\nmodel_cv = GridSearchCV(estimator = model, \n                        param_grid = hyper_params, \n                        scoring= 'accuracy', \n                        cv = folds, \n                        verbose = 1,\n                        n_jobs=-1,\n                        return_train_score=True)      \n\n# fit the model\nmodel_cv.fit(X_train, y_train)    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cv results\ncv_results = pd.DataFrame(model_cv.cv_results_)\ncv_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting C to numeric type for plotting on x-axis\ncv_results['param_C'] = cv_results['param_C'].astype('int')\n\n# # plotting\nplt.figure(figsize=(16,6))\n\n# subplot 1/3\nplt.subplot(131)\ngamma_01 = cv_results[cv_results['param_gamma']==0.01]\n\nplt.plot(gamma_01[\"param_C\"], gamma_01[\"mean_test_score\"])\nplt.plot(gamma_01[\"param_C\"], gamma_01[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.01\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='upper left')\nplt.xscale('log')\n\n# subplot 2/3\nplt.subplot(132)\ngamma_001 = cv_results[cv_results['param_gamma']==0.001]\n\nplt.plot(gamma_001[\"param_C\"], gamma_001[\"mean_test_score\"])\nplt.plot(gamma_001[\"param_C\"], gamma_001[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.001\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='upper left')\nplt.xscale('log')\n\n\n# subplot 3/3\nplt.subplot(133)\ngamma_0001 = cv_results[cv_results['param_gamma']==0.0001]\n\nplt.plot(gamma_0001[\"param_C\"], gamma_0001[\"mean_test_score\"])\nplt.plot(gamma_0001[\"param_C\"], gamma_0001[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.0001\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='upper left')\nplt.xscale('log')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This plot reveals some interesting insights:\n- **High values of gamma** lead to **overfitting** (especially at high values of C); note that the training accuracy at gamma=0.01 and C=1000 reaches almost 99% \n- The **training data increases with higher gamma**, though the **test scores are comparable** (at sufficiently high cost, i.e. C > 10)\n- The least amount of overfitting (i.e. difference between train and test accuracy) occurs at low gamma, i.e. a quite *simple non-linear model*\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# printing the optimal accuracy score and hyperparameters\nbest_score = model_cv.best_score_\nbest_hyperparams = model_cv.best_params_\n\nprint(\"The best test score is {0} corresponding to hyperparameters {1}\".format(best_score, best_hyperparams))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building and Evaluating the Final Model\n\nLet's now build and evaluate the final model, i.e. the model with highest test accuracy."},{"metadata":{"trusted":true},"cell_type":"code","source":"# model with optimal hyperparameter : C=10 and gamma =0.001\n\n# model\nmodel = SVC(C=10, gamma=0.001, kernel=\"rbf\")\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\n# metrics\nprint(metrics.confusion_matrix(y_test, y_pred), \"\\n\")\nprint(\"accuracy\", metrics.accuracy_score(y_test, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model with different hyperparameters: C=10 and gamma =0.01\n\n# model\nmodel = SVC(C=10, gamma=0.01, kernel=\"rbf\")\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\n# metrics\nprint(metrics.confusion_matrix(y_test, y_pred), \"\\n\")\nprint(\"accuracy\", metrics.accuracy_score(y_test, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model with optimal hyperparameter : C=100 and gamma =0.001\n\n# model\nmodel = SVC(C=100, gamma=0.001, kernel=\"rbf\")\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\n# metrics\nprint(metrics.confusion_matrix(y_test, y_pred), \"\\n\")\nprint(\"accuracy\", metrics.accuracy_score(y_test, y_pred))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On checking with diiferent hyperparameters value found that the best value of C and gamma is 10 and 0.001"},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\nThe accuracy achieved using a non-linear kernel (~0.95) is higher than that of a linear one (~0.91). We can conclude that the problem is highly non-linear in nature."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.0"}},"nbformat":4,"nbformat_minor":1}