{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nprint(os.listdir(\"../input\"))\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import validation_curve\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import GridSearchCV\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ntrain_data = pd.read_csv(\"../input/train.csv\") \ntest_data = pd.read_csv(\"../input/test.csv\")\n\n\ntrain_data.shape\ntest_data.shape\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n**MNIST Digits - Classification Using SVM****\n\n\n**Objective**\nWe will develop a model using Support Vector Machine which should correctly classify the handwritten digits from 0-9 based on the pixel values given as features. Thus, this is a 10-class classification problem.\n\n**Data Description**\nFor this problem, we use the MNIST data which is a large database of handwritten digits. The 'pixel values' of each digit (image) comprise the features, and the actual number between 0-9 is the label.\n\nSince each image is of 28 x 28 pixels, and each pixel forms a feature, there are 784 features. MNIST digit recognition is a well-studied problem in the ML community, and people have trained numerous models (Neural Networks, SVMs, boosted trees etc.) achieving error rates as low as 0.23% (i.e. accuracy = 99.77%, with a convolutional neural network).\n\nBefore the popularity of neural networks, though, models such as SVMs and boosted trees were the state-of-the-art in such problems.\n\nWe'll first explore the dataset a bit, prepare it (scale etc.) and then experiment with linear and non-linear SVMs with various hyperparameters.\n\nWe'll divide the analysis into the following parts:\n\n**Data understanding and cleaning\nData preparation for model building\nBuilding an SVM model - hyperparameter tuning, model evaluation etc.**\n\n"},{"metadata":{},"cell_type":"markdown","source":"**Data Understanding**\n\nLet's first load the data and understand the attributes meanings, shape of the dataset etc.\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import validation_curve\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import GridSearchCV\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(\"../input/train.csv\") #reading the csv files using pandas\ntest_data = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.shape # print the dimension or shape of train data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.shape # print the dimension or shape of test data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head() # printing first five columns of train_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"test_data.shape # print the dimension or shape of test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head() # printing first five columns of test_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# there are no missing values in the dataset \n\ntrain_data.isnull().sum().head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_data.isnull().sum().head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# about the dataset\n\n# dimensions\nprint(\"Dimensions: \",test_data.shape, \"\\n\")\n\n# data types\nprint(test_data.info())\n\n# head\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# about the dataset\n\n# dimensions\nprint(\"Dimensions: \",train_data.shape, \"\\n\")\n\n# data types\nprint(train_data.info())\n\n# head\ntrain_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_data.columns)\nprint(test_data.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"order = list(np.sort(train_data['label'].unique()))\nprint(order)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Visualizing the number of class and counts in the datasets\n\n\nsns.countplot(train_data[\"label\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Visualizing the number of class and counts in the datasets\nplt.plot(figure = (16,10))\ng = sns.countplot( train_data[\"label\"], palette = 'icefire')\nplt.title('NUmber of digit classes')\ntrain_data.label.astype('category').value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting some samples as well as converting into matrix\n\nfour = train_data.iloc[3, 1:]\nfour.shape\nfour = four.values.reshape(28,28)\nplt.imshow(four, cmap='gray')\nplt.title(\"Digit 4\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nseven = train_data.iloc[6, 1:]\nseven.shape\nseven = seven.values.reshape(28, 28)\nplt.imshow(seven, cmap='gray')\nplt.title(\"Digit 7\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"********Data Preparation**\n\nLet's conduct some data preparation steps before modeling. Firstly, let's see if it is important to rescale the features, since they may have varying ranges. \n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"# average feature values\nround(train_data.drop('label', axis=1).mean(), 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case, the average values do not vary a lot (e.g. having a diff of an order of magnitude). Nevertheless, it is better to rescale them."},{"metadata":{"trusted":true},"cell_type":"code","source":"## Separating the X and Y variable\n\ny = train_data['label']\n\n## Dropping the variable 'label' from X variable \nX = train_data.drop(columns = 'label')\n\n## Printing the size of data \nprint(train_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Normalization\n\nX = X/255.0\ntest_data = test_data/255.0\n\nprint(\"X:\", X.shape)\nprint(\"test_data:\", test_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# scaling the features\nfrom sklearn.preprocessing import scale\nX_scaled = scale(X)\n\n# train test split\nX_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size = 0.3, train_size = 0.2 ,random_state = 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"******Model Building**\n\nLet's fist build two basic models - linear and non-linear with default hyperparameters, and compare the accuracies.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# linear model\n\nmodel_linear = SVC(kernel='linear')\nmodel_linear.fit(X_train, y_train)\n\n# predict\ny_pred = model_linear.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# confusion matrix and accuracy\n\nfrom sklearn import metrics\nfrom sklearn.metrics import confusion_matrix\n# accuracy\nprint(\"accuracy:\", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), \"\\n\")\n\n# cm\nprint(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The linear model gives approx. 91% accuracy. Let's look at a sufficiently non-linear model with randomly chosen hyperparameters."},{"metadata":{"trusted":true},"cell_type":"code","source":"# non-linear model\n# using rbf kernel, C=1, default value of gamma\n\n# model\nnon_linear_model = SVC(kernel='rbf')\n\n# fit\nnon_linear_model.fit(X_train, y_train)\n\n# predict\ny_pred = non_linear_model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# confusion matrix and accuracy\n\n# accuracy\nprint(\"accuracy:\", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), \"\\n\")\n\n# cm\nprint(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The non-linear model gives approx. 93% accuracy. Thus, going forward, let's choose hyperparameters corresponding to non-linear models"},{"metadata":{},"cell_type":"markdown","source":"**Grid Search: Hyperparameter Tuning**\n\nLet's now tune the model to find the optimal values of C and gamma corresponding to an RBF kernel. We'll use 5-fold cross validation.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating a KFold object with 5 splits \nfolds = KFold(n_splits = 5, shuffle = True, random_state = 10)\n\n# specify range of hyperparameters\n# Set the parameters by cross-validation\nhyper_params = [ {'gamma': [1e-2, 1e-3, 1e-4],\n                     'C': [5,10]}]\n\n\n# specify model\nmodel = SVC(kernel=\"rbf\")\n\n# set up GridSearchCV()\nmodel_cv = GridSearchCV(estimator = model, \n                        param_grid = hyper_params, \n                        scoring= 'accuracy', \n                        cv = folds, \n                        verbose = 1,\n                        return_train_score=True)      \n\n# fit the model\nmodel_cv.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cv results\ncv_results = pd.DataFrame(model_cv.cv_results_)\ncv_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting C to numeric type for plotting on x-axis\ncv_results['param_C'] = cv_results['param_C'].astype('int')\n\n# # plotting\nplt.figure(figsize=(16,8))\n\n# subplot 1/3\nplt.subplot(131)\ngamma_01 = cv_results[cv_results['param_gamma']==0.01]\n\nplt.plot(gamma_01[\"param_C\"], gamma_01[\"mean_test_score\"])\nplt.plot(gamma_01[\"param_C\"], gamma_01[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.01\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='upper left')\nplt.xscale('log')\n\n# subplot 2/3\nplt.subplot(132)\ngamma_001 = cv_results[cv_results['param_gamma']==0.001]\n\nplt.plot(gamma_001[\"param_C\"], gamma_001[\"mean_test_score\"])\nplt.plot(gamma_001[\"param_C\"], gamma_001[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.001\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='upper left')\nplt.xscale('log')\n\n\n# subplot 3/3\nplt.subplot(133)\ngamma_0001 = cv_results[cv_results['param_gamma']==0.0001]\n\nplt.plot(gamma_0001[\"param_C\"], gamma_0001[\"mean_test_score\"])\nplt.plot(gamma_0001[\"param_C\"], gamma_0001[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.0001\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='upper left')\nplt.xscale('log')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the plot above, we can observe that (from higher to lower gamma / left to right):\n\nAt very high gamma (0.01), the model is achieving 100% accuracy on the training data, though the test score is quite low (<80%). Thus, the model is overfitting.\n\nAt gamma=0.001, the training and test scores are comparable at around C=1, though the model starts to overfit at higher values of C\n\nAt gamma=0.0001, the model does not overfit till C=10 but starts showing signs at C=100. Also, the training and test scores are slightly lower than at gamma=0.001.\n\nThus, it seems that the best combination is gamma=0.001 and C=15 (the plot in the middle), which gives the highest test accuracy (~94%) while avoiding overfitting.\n\nLet's now build the final model and see the performance on test data."},{"metadata":{},"cell_type":"markdown","source":"Let's now choose the best hyperparameters. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# printing the optimal accuracy score and hyperparameters\nbest_score = model_cv.best_score_\nbest_hyperparams = model_cv.best_params_\n\nprint(\"The best test score is {0} corresponding to hyperparameters {1}\".format(best_score, best_hyperparams))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Building and Evaluating the Final Model**\n\nLet's now build and evaluate the final model, i.e. the model with highest test accuracy.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# model with optimal hyperparameters\n\n# model\nmodel = SVC(C=10, gamma=0.001, kernel=\"rbf\")\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\n# metrics\nprint(\"accuracy\", metrics.accuracy_score(y_test, y_pred), \"\\n\")\nprint(metrics.confusion_matrix(y_test, y_pred), \"\\n\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Conclusion\n\nThe accuracy achieved using a non-linear kernel (~0.94) is mush higher than that of a linear one (~0.91). We can conclude that the problem is highly non-linear in nature.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}