{"cells":[{"metadata":{"_uuid":"a5f1f1b6bef14c85ef5d08f0e54c20a92eb560fc"},"cell_type":"markdown","source":"# Classification\nThis notebook tries to create a running example in Python that captures the concepts explained in week 4, by **Professor Devavrat Shah** in the **[M.I.T. Professional Education course Machine Learning: From Data to Decissions](https://professionalonline1.mit.edu/machine-learning/index.php)**.\n\nClassification is a supervised learning type of problem, in which we try to predict a class given a number of features. There are many types of classification methods, in this notebook we will go through:\n* Logistics Regression\n* Linear Discriminant Analysis\n* Quadratic Discriminant Analysis\n* Nearest Neighbors\n\nWe are going to work on a typical classification problem, mentioned in the course wich is identifying digits from a set of images. There is a well known dataset for this problem in particular, as Kaggle says, MNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision.\n\n### Attention !!!:\n> This is a **multiclass** problem where we have as many labels as digits (decimal system, so from 0 to 9, 10 of them), **but in order to make it easy to see concepts like confusion matrix etc I will just run it with digits 0 and 1.**"},{"metadata":{"_uuid":"da62b5ce848f0d3e69210524be401dd12e4fdab7"},"cell_type":"markdown","source":"## 0. Import Libraries Data Filtering and Exploration"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt, matplotlib.image as mpimg\nimport seaborn as sns\n\nfrom sklearn.model_selection import train_test_split, cross_val_score\nfrom sklearn.linear_model import LogisticRegression, LogisticRegressionCV\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn import metrics\n\nN_IMAGES_PER_CLASS = 100 # Number of images to use: change this value for longer/shorter times higher/lower accuracy\nSEED = 22\nDIGIT_0 = 3\nDIGIT_1 = 8","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9611064e9550b367d17f7ca725661c0fcb76499f"},"cell_type":"markdown","source":"**Please NOTE:**\n> I am taking as value 0 the digit 3, and as value 1 the digit 8. So when prediction = 1 it means that its detected an 8, and when prediction is 0 it will have detected a 3. I am using these 2 numbers as they are more similar in shape than a 0 and a 1. Tests with 0 and 1 were incredibly accurate and thus it wouldnt show the effect of threshold and the differences between the different models.\n\nLet us import the data and check how many samples we have from each digit."},{"metadata":{"trusted":true,"_uuid":"034349455f87ff636330135ecd6290e18973d8bf"},"cell_type":"code","source":"data = pd.read_csv('../input/train.csv')\nsns.countplot(x = 'label', data = data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9072001dda5205412023972c082673f4c51754fa"},"cell_type":"markdown","source":"I will make another dataset just for 0 and 1, and just take a sample of them (to start with I will take same number of samples from 0 and 1, as defined in global variable N_IMAGES_PER_CLASS = `N_IMAGES_PER_CLASS`>"},{"metadata":{"trusted":true,"_uuid":"738b0f5bc19cdfc12a69f4742429f79a65ee4776"},"cell_type":"code","source":"# By grouping on the label and then sampling, we can keep sample equally among both labels. \ndata01 = data[data.label.isin([DIGIT_0, DIGIT_1])] \ndata01 = data01.groupby('label').apply(lambda x: x.sample(n=N_IMAGES_PER_CLASS, replace=False, random_state = SEED))\nX = data01.iloc[:,1:]\ny = data01.label\nsns.countplot(x = data01.label, data = data01)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10d912f3f439b662431af7d2151aa7fa8c30345e"},"cell_type":"markdown","source":"To view the images, since they are currently one-dimensional, we need to reshape them so that it is two-dimensional (28x28 pixels)\nThen, we plot the image and label with matplotlib. You can change the value of variable i to check out other images and labels."},{"metadata":{"trusted":true,"_uuid":"6d0aa700aed1e80a57812946b390983e9b428e7d"},"cell_type":"code","source":"i=1\nimg = data01.iloc[i,1:].values.reshape((28,28))\nplt.imshow(img,cmap='gray')\nplt.title(data01.iloc[i,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b4f0135e9cc68a2fc78f432b475828bb0665da2"},"cell_type":"markdown","source":"We can see the values of the pixels and see how they distribute over the range of values (typically images have one layer per color of values 0-255, in our case just one layer as its in gray shades). As we can see most of the images are black (value of 0) and the information comes mostly around 255 for the white number but its quite sparse as data."},{"metadata":{"trusted":true,"_uuid":"ad53f4e3f5b97e31dbb5c4854cee3a207af03bac"},"cell_type":"code","source":"sns.distplot(X.values.reshape(1,-1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2182a07e4b72dee22cba04cd5f68e80292550626"},"cell_type":"markdown","source":"Ok, we split the features (X = pixels in the image) and the labels (y) into training and testing datasets, and start working with this information now. I will also prepare another dataset and train/test data with unbalanced classes, where there will be 10% of 1 and 90% of 0.  This we will use to see how the different classification methods fare when the classes are unevenly represented.\n> As it is easily trained and I want to see impact on True Prediction Rate and False Prediction Rate later on. I am going to use a huge sample of Test data of more than half the Training Data."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true},"cell_type":"code","source":"# To make it work as binary for digits other than 1\ny.update(pd.Series(0, index = y[y == DIGIT_0].index))\ny.update(pd.Series(1, index = y[y == DIGIT_1].index))\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.8, random_state=SEED, stratify = y)\n\ndata01_uneven = pd.concat([data01[data01.label == 0], data01[data01.label == 1].iloc[:round(N_IMAGES_PER_CLASS*0.2/0.8)]])\nX_utrain, X_utest, y_utrain, y_utest = train_test_split(data01_uneven.iloc[:,1:], data01_uneven.label, \n                                                        test_size=0.8, stratify = data01_uneven.label ,random_state=SEED)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5c2cc084d8e628dc08b4b29399ccb0ad41a5cc7"},"cell_type":"markdown","source":"## 1. Logistic Regression\nNotice that to get the roc_curve you need to get predict_proba as it will work through the different thresholds with the probabilities predicted by the model."},{"metadata":{"trusted":true,"_uuid":"c0c7b8e8029b01905d610d8575867a8fe7745a32"},"cell_type":"code","source":"lr = LogisticRegression(solver = 'liblinear', random_state = SEED)\nlr.fit(X_train, y_train)\nlr_bal_score = lr.score(X_test, y_test)\nprint(\"Score on Train data:\",lr.score(X_train, y_train), \"Score on Test data =\", lr_bal_score)\ny_p = lr.predict(X_test)\ny_pred = lr.predict_proba(X_test)\nfpr, tpr, thresholds  = metrics.roc_curve(y_test, y_pred[:,1])\nlr_roc_auc = metrics.roc_auc_score(y_test, y_pred[:,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52f54843a79894db1051d73b6174ef489e702f6d"},"cell_type":"code","source":"plt.figure()\nlw = 2\nplt.plot(fpr, tpr, color='darkorange', label='ROC curve (area = %0.2f)' % lr_roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver operating characteristic example')\nplt.legend(loc=\"lower right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02b8a44a84f7592667138abe6e6a30f2c728228b"},"cell_type":"markdown","source":"With the following Confusion Matrix, and True Positive Rate / False Positive Rate"},{"metadata":{"trusted":true,"_uuid":"db23ce46c34fff90aac3667353ba64862f4f7602"},"cell_type":"code","source":"cm_lr = metrics.confusion_matrix(y_test, y_p)\nprint(cm_lr)\nprint(\"TP:\", cm_lr[1,1]/(cm_lr[0,1]+cm_lr[1,1]), \"FP:\", cm_lr[0,1]/(cm_lr[0,0]+cm_lr[0,1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb0772e9d855a8e352138e8c7a340cc812724651"},"cell_type":"markdown","source":"## 1.1 Logistics Regression on unbalanced data:\nRemember we use only 10% of the DIGIT_1 values:"},{"metadata":{"trusted":true,"_uuid":"03843d008e97ebc8033e22a5905c7b223d0d6aa3"},"cell_type":"code","source":"sns.countplot(x = y_utrain)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"99c3e232a2e9fe03e51ad147bdc3bd10d48eec5d"},"cell_type":"code","source":"lr.fit(X_utrain, y_utrain)\nlr_unbal_score = lr.score(X_utest, y_utest)\nprint(\"Score on Train data:\",lr.score(X_utrain, y_utrain), \"Score on Test data =\", lr_unbal_score)\ny_p = lr.predict(X_utest)\ny_pred = lr.predict_proba(X_utest)\nfpr, tpr, thresholds  = metrics.roc_curve(y_utest, y_pred[:,1])\nlr_unbal_roc_auc = metrics.roc_auc_score(y_utest, y_pred[:,1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5ae615f8fc9edd88de94519c8c5042f4d3f79c7"},"cell_type":"markdown","source":"**Test Data Accuracy Score is misleading here** it is higher than the one for a balanced dataset but only because a default of doing ALL 0, would already give a score of 0.8. So we need to look at AUC score and our TP FP to better interpret results. As it can be seen TP has gone down as well as the AUC score."},{"metadata":{"trusted":true,"_uuid":"93fc4b46004dbf163ad84c7a078e54f38b27a631"},"cell_type":"code","source":"cm_lr = metrics.confusion_matrix(y_utest, y_p)\nprint(cm_lr)\nprint(\"TP:\", cm_lr[1,1]/(cm_lr[0,1]+cm_lr[1,1]), \"FP:\", cm_lr[0,1]/(cm_lr[0,0]+cm_lr[0,1]), \"AUC:\", lr_unbal_roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"902aebd5bf311cc2862be575a303c876e33f6578"},"cell_type":"markdown","source":"## 2. Linear Discriminant Analysis\nRemember that Discriminant Analysis makes the hypothesis that attributes (that is the values of each pixel point in this case) are distributed as per a Gaussian Distribution."},{"metadata":{"trusted":true,"_uuid":"38eef20c8c501bd95d2b82763a7862bb75e97a74"},"cell_type":"code","source":"lda = LinearDiscriminantAnalysis()\nlda.fit(X_train, y_train)\nlda_bal_score = lr.score(X_test, y_test)\nprint(\"Score on Train data:\",lda.score(X_train, y_train), \"Score on Test data =\", lda_bal_score)\ny_p = lda.predict(X_test)\ny_pred = lda.predict_proba(X_test)\nfpr, tpr, thresholds  = metrics.roc_curve(y_test, y_pred[:,1])\nlda_roc_auc = metrics.roc_auc_score(y_test, y_pred[:,1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67de1edb52ad1d3d2d33b69a3341c11b5fe7b7d9"},"cell_type":"markdown","source":"As it can be seen in the ROC graph below threshold needs "},{"metadata":{"trusted":true,"_uuid":"e876b75dcfd74808ded3374fa08d4a265674f250"},"cell_type":"code","source":"plt.figure()\nlw = 2\nplt.plot(fpr, tpr, color='darkorange', label='ROC curve (area = %0.2f)' % lr_roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver operating characteristic example')\nplt.legend(loc=\"lower right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c722704981768dde11cf3d85560830cca2982ef2"},"cell_type":"code","source":"cm_lda = metrics.confusion_matrix(y_test, y_p)\nprint(cm_lda)\nprint(\"TP:\", cm_lda[1,1]/(cm_lda[0,1]+cm_lda[1,1]), \"FP:\", cm_lda[0,1]/(cm_lda[0,0]+cm_lda[0,1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ec710f9179239df2938a84b9650a2460d5842bb"},"cell_type":"markdown","source":"We can see that the False Positive Rate is much much higher than in the case of Logistic Regression, as well as the True Positive Rate being much lower, so in this case it looks less appealing. Let us see though what happens when the classes samples are unbalanced."},{"metadata":{"_uuid":"e8434f7323e7dd111c13626317070045692d30b2"},"cell_type":"markdown","source":"## 2.1 Linear Discriminant Analysis on Unbalanced Data\nAs it can be seen below, Linear Discriminant Analysis starts to perform better than logistic Regression on unbalanced Data. Here we see an improvement on the AUC score over the unbalanced result of LR. "},{"metadata":{"trusted":true,"_uuid":"a6e7844c4becf4e7e847446b0a233277668d9ee2"},"cell_type":"code","source":"lda.fit(X_utrain, y_utrain)\nlda_unbal_score = lr.score(X_utest, y_utest)\nprint(\"Score on Train data:\",lda.score(X_train, y_train), \"Score on Test data =\", lda_unbal_score)\ny_p = lda.predict(X_utest)\ny_pred = lda.predict_proba(X_utest)\nfpr, tpr, thresholds  = metrics.roc_curve(y_utest, y_pred[:,1])\nlda_unbal_roc_auc = metrics.roc_auc_score(y_utest, y_pred[:,1])\ncm_lda = metrics.confusion_matrix(y_utest, y_p)\nprint(cm_lda)\nprint(\"TP:\", cm_lda[1,1]/(cm_lda[0,1]+cm_lda[1,1]), \"FP:\", cm_lda[0,1]/(cm_lda[0,0]+cm_lda[0,1]), \"AUC:\", lda_unbal_roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f1c1e192c805414ebbed4dbca2ea08610d7baff"},"cell_type":"markdown","source":"## 3. Quadratic Discriminant Analysis\nQuadratic discriminant analysis, does not assume the same Standard Deviation for all the features as Linear Discriminant Analysis does. This allows it to fit better in some cases."},{"metadata":{"trusted":true,"_uuid":"002d72b7f81d5b11f132726b5f77759f44454d6e"},"cell_type":"code","source":"qda = QuadraticDiscriminantAnalysis()\nqda.fit(X_train, y_train)\nqda_bal_score = qda.score(X_test, y_test)\ny_p = qda.predict(X_test)\ny_pred = qda.predict_proba(X_test)\nfpr, tpr, thresholds  = metrics.roc_curve(y_test, y_pred[:,1])\nqda_roc_auc = metrics.roc_auc_score(y_test, y_pred[:,1])\nprint(\"Score on Train data:\",qda.score(X_train, y_train), \"Score on Test data =\", qda_bal_score, \"AUC:\",qda_roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1261b8a8e421e05521d7a72553a3d3cd9afe4d0f"},"cell_type":"code","source":"plt.figure()\nlw = 2\nplt.plot(fpr, tpr, color='darkorange', label='ROC curve (area = %0.2f)' % lr_roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver operating characteristic example')\nplt.legend(loc=\"lower right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb01e8178223da0705d96c09e046cf20fd37fbbc"},"cell_type":"markdown","source":"As we see above, QDA is performing very badly in this case giving no predictive value."},{"metadata":{"trusted":true,"_uuid":"56aabef3ed7aed5678cdaafc577b0edaf1ff7442"},"cell_type":"code","source":"cm_qda = metrics.confusion_matrix(y_test, y_p)\nprint(cm_qda)\nprint(\"TP:\", cm_qda[1,1]/(cm_qda[0,1]+cm_qda[1,1]), \"FP:\", cm_qda[0,1]/(cm_qda[0,0]+cm_qda[0,1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba8ccb72c93e2a10100422a29e0718277c9ae193"},"cell_type":"markdown","source":"## 3.1 Quadratic Discriminant Analysis on unbalanced Data\nAs we can see below, Quadratic starts performing better on unbalanced data, but it's still giving us worse results than Linear Discriminant Analysis in this case. \nWe are running on very little data though (20% of training data, that is about 4 samples which might be too few to find correct Standard Deviations for the 1 class)"},{"metadata":{"trusted":true,"_uuid":"32e7898540de11e922454d8b5e3bdd6e23bc0974"},"cell_type":"code","source":"qda.fit(X_utrain, y_utrain)\nqda_unbal_score = lr.score(X_utest, y_utest)\nprint(\"Score on Train data:\",qda.score(X_train, y_train), \"Score on Test data =\", qda_unbal_score)\ny_p = qda.predict(X_utest)\ny_pred = qda.predict_proba(X_utest)\nfpr, tpr, thresholds  = metrics.roc_curve(y_utest, y_pred[:,1])\nqda_unbal_roc_auc = metrics.roc_auc_score(y_utest, y_pred[:,1])\ncm_qda = metrics.confusion_matrix(y_utest, y_p)\nprint(cm_qda)\nprint(\"TP:\", cm_qda[1,1]/(cm_qda[0,1]+cm_qda[1,1]), \"FP:\", cm_qda[0,1]/(cm_qda[0,0]+cm_qda[0,1]), \"AUC:\", qda_unbal_roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"abe3fbda0548e95f00ebb4cff06d9cdf868a560d"},"cell_type":"markdown","source":"## 4. K-Nearest Neighbors\nWe will try to find the optimal value of K using Cross Validation\n> Check the [following post \"Complete Guide to Nearest Neighbors\"](https://kevinzakka.github.io/2016/07/13/k-nearest-neighbor/#parameter-tuning-with-cross-validation) for more information. Some of the code below was taken from it with variations.\n\n> **NOTE** In the code below, we use as scoring method roc_auc so as to maximize the AUC area with the k chosen for the nearest neighbors. \n\n> **NOTE 2** Do not confuse the k used for the cross validation (cv = 5) from the k used with the Nearest Neighbors, the parameter we are trying to optimize."},{"metadata":{"trusted":true,"_uuid":"f394d26fd79d317da29c96a995025623796209bf"},"cell_type":"code","source":"# subsetting just the odd ones so that voting is always determining\nneighbors = range(1,20,2)\n\n# empty list that will hold cv scores\ncv_scores = []\ny_pred = []\n\n# perform 10-fold cross validation\nfor k in neighbors:\n    knn = KNeighborsClassifier(n_neighbors=k)\n    scores = cross_val_score(knn, X_train, y_train, cv=5, scoring='roc_auc')\n    cv_scores.append(scores.mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8fa704480a5f249fd7ca8415bcc7397bac433491"},"cell_type":"code","source":"# changing to misclassification error\nMSE = [1 - x for x in cv_scores]\n\n# determining best k\noptimal_k = neighbors[MSE.index(min(MSE))]\nprint(\"The optimal number of neighbors is\",optimal_k, \"with an AUC score of\", scores[optimal_k])\n\n# plot misclassification error vs k\nplt.plot(neighbors, MSE)\nplt.xlabel('Number of Neighbors K')\nplt.ylabel('Misclassification Error')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f73812fd0bb4d2ce7bf5434c8a5270710d9242a8"},"cell_type":"markdown","source":"Let us then run this k-NN model with its optimal k. And check the results:"},{"metadata":{"trusted":true,"_uuid":"84f89e2f4a888c40f2d2745ab4573357c6a5afcf"},"cell_type":"code","source":"knn = KNeighborsClassifier(n_neighbors=3)\nknn.fit(X_train, y_train)\nknn_bal_score = knn.score(X_test, y_test)\ny_p = knn.predict(X_test)\ny_pred = knn.predict_proba(X_test)\nfpr, tpr, thresholds  = metrics.roc_curve(y_test, y_pred[:,1])\nknn_roc_auc = metrics.roc_auc_score(y_test, y_pred[:,1])\nprint(\"Score on Train data:\",knn.score(X_train, y_train), \"Score on Test data =\", knn_bal_score, \"AUC:\",knn_roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df536f43149e85428b5fd4623af9a7593e64ec53"},"cell_type":"markdown","source":"As you can see the scores are different than in the CV above, that is because here we are running that model on the test data, and these are the results for the Test Data."},{"metadata":{"trusted":true,"_uuid":"4a0f0af557f41651950e347a37473a53a9e2fe94"},"cell_type":"code","source":"plt.figure()\nlw = 2\nplt.plot(fpr, tpr, color='darkorange', label='ROC curve (area = %0.2f)' % lr_roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver operating characteristic example')\nplt.legend(loc=\"lower right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12ef7624da88914cf9f2d7a53255691d1b9b3b0d"},"cell_type":"code","source":"cm_knn = metrics.confusion_matrix(y_test, y_p)\nprint(cm_knn)\nprint(\"TP:\", cm_knn[1,1]/(cm_knn[0,1]+cm_knn[1,1]), \"FP:\", cm_knn[0,1]/(cm_knn[0,0]+cm_knn[0,1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55dd5df44bbec6206d66a3fc7f892d4cbc38968c"},"cell_type":"markdown","source":"## 4.1 Let's see how k-NN fares with unbalanced data\nPlease see that I need to change the range of neighbors as we cannot have more neighbors than samples exist in the less frequent class."},{"metadata":{"trusted":true,"_uuid":"25cccd6ce8201eb3d0d096dc5f2fea4c627c8f8b"},"cell_type":"code","source":"# subsetting just the odd ones so that voting is always determining\nneighbors = range(1,6,2)\n\n# empty list that will hold cv scores\ncv_scores = []\ny_pred = []\n\n# perform 10-fold cross validation\nfor k in neighbors:\n    knn = KNeighborsClassifier(n_neighbors=k)\n    scores = cross_val_score(knn, X_utrain, y_utrain, cv=5, scoring='roc_auc')\n    cv_scores.append(scores.mean())\n# changing to misclassification error\nMSE = [1 - x for x in cv_scores]\n\n# determining best k\noptimal_k = neighbors[MSE.index(min(MSE))]\nprint(\"The optimal number of neighbors is\",optimal_k, \"with an AUC score of\", scores[optimal_k])\n\n# plot misclassification error vs k\nplt.plot(neighbors, MSE)\nplt.xlabel('Number of Neighbors K')\nplt.ylabel('Misclassification Error')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9da20a098ce7d3cf2c05550b3f906176b81bbcbc"},"cell_type":"code","source":"knn = KNeighborsClassifier(n_neighbors=3)\nknn.fit(X_utrain, y_utrain)\nknn_unbal_score = knn.score(X_utest, y_utest)\ny_p = knn.predict(X_utest)\ny_pred = knn.predict_proba(X_utest)\nfpr, tpr, thresholds  = metrics.roc_curve(y_utest, y_pred[:,1])\nknn_unbal_roc_auc = metrics.roc_auc_score(y_utest, y_pred[:,1])\nprint(\"Score on Train data:\",knn.score(X_utrain, y_utrain), \"Score on Test data =\", knn_unbal_score, \"AUC:\",knn_unbal_roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2785136ca757dbfe319bf833573747bfcfb5576e"},"cell_type":"markdown","source":"## Recap\nLet's put all our results together, just for easier comparison:\n* When comparing Balanced and Unbalanced accuracy bear in mind that that unbalanced data has DIGIT_1 20% of data and DIGIT_0 80% of data. So a dumb result of DIGIT_0 for all samples would already give an accuracy of 0.8\n* For Balanced data, we would probably use Logistic Regression out of these models for this particular problem. as it gives us both the highest Accuracy and the highest AUC. \n* For unbalanced data, the best out of the models we have considered is LDA.\n* It doesnt cease to amaze me that with such simple methods such as k_NN or logistics regression and just a few samples (I have used just 20 for the training for each class on the balanced examples and 20 and 4 on the unbalanced), you can get such high accuracy and AUC results.\n\nAt any rate, please bear in mind that we have used a minimum amount of data. kNN, keeps all data in memory so if we were to use all the images it would probably start being a bit too heavy on the memory, and maybe unpractical. So, depending on the sample size that we were to use this list would probably change. \n\nIf you want to do some testing, change the global variable N_IMAGES_PER_CLASS,  you will see that with a bit higher values you will start getting 1.00 accuracy and 1.00 AUC in most cases. Also, try changing the digits, if you try with 0 and 1, you will see that both accuracy and AUC also go up with regards to this results (in which I have used 3 and 8).\n\nI hope that you have enjoyed this notebook and that it has been of use to you. Please do not hesitate to send me corrections or improvements over it.\n\nThank you for reading.\nBenigno."},{"metadata":{"trusted":true,"_uuid":"4eaf6563f3327b4490ed7b27873611ea115bab80"},"cell_type":"code","source":"results = pd.DataFrame(data = {'Balanced Accuracy':[], 'Balanced AUC':[], 'Unbalanced Acc':[], 'Unbalanced AUC':[]})\nresults = results.append(pd.Series(name = 'Logistic Regression', data = [lr_bal_score, lr_roc_auc, lr_unbal_score, lr_unbal_roc_auc], index = results.columns))\nresults = results.append(pd.Series(name = 'LDA', data = [lda_bal_score, lda_roc_auc, lda_unbal_score, lda_unbal_roc_auc], index = results.columns))\nresults = results.append(pd.Series(name = 'QDA', data = [qda_bal_score, qda_roc_auc, qda_unbal_score, qda_unbal_roc_auc], index = results.columns))\nresults = results.append(pd.Series(name = 'kNN', data = [knn_bal_score, knn_roc_auc, knn_unbal_score, knn_unbal_roc_auc], index = results.columns))\nresults","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}