{"cells":[{"metadata":{"_uuid":"58fb3529652cde4094326f9d88af6db017f2fade"},"cell_type":"markdown","source":"I spent a good amont of time to apply PCA+SVM in the attempt to reach (possibly) 0.99 accuracy (to date, I failed). However, I decided to write a kernel to raise the question on whether reaching a high accuracy is really meaningful.\nIn fact, during my effort I met what seems to me mistakes in the train sample labels. I'm going to show what I mean. \n\nLet's implement a PCA+SVM model\n"},{"metadata":{"trusted":true,"_uuid":"7ab59754ebfc19d5aaedfe88ced0a4d95a224773"},"cell_type":"code","source":"#import necessary packages\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt, matplotlib.image as mpimg\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import svm\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.decomposition import PCA\n\n%matplotlib inline\n\n#useful function to write results\ndef write_results(predicted, name = 'predicted.csv'):\n    df = pd.DataFrame(predicted)\n    df.index.name='ImageId'\n    df.index+=1\n    df.columns=['Label']\n    df['Label']=df['Label'].astype(int)\n    df.to_csv(name, header=True)\n    \n# useful function to plot digits\ndef plot_digits(list):\n    plt.figure(figsize=(20,20))\n    nn=0\n\n    for idx in list:\n        nn+=1\n        plt.subplot(10,10,nn)\n        plt.title(\"label:{}\".format(y_df.iloc[idx]))\n        #plt.title(\"index: {}\".format(img.index[count]))\n        grid_data = X_df.iloc[idx].values.reshape(28,28)\n        plt.imshow(grid_data, interpolation = \"none\", cmap = \"afmhot\")\n        plt.xticks([])\n        plt.yticks([])\n    plt.show()\n#########","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0099551d0752955636510906572f7eb4994fb64"},"cell_type":"markdown","source":"now upload the data and split them in train and test samples "},{"metadata":{"trusted":true,"_uuid":"ec8be106bef2f273d7c2f2e98da0e4ad7762b84e"},"cell_type":"code","source":"labeled_images = pd.read_csv('../input/train.csv')\nX_df = labeled_images.iloc[0:,1:]\ny_df = labeled_images.iloc[0:,0]\n#split to train and test data\ntrain_images, test_images,train_labels, test_labels = train_test_split(X_df, y_df, train_size=0.67, random_state=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7245c83c461766dcb0b2625a909c101876a4b0f"},"cell_type":"markdown","source":"I use Pipeline to find out the best parameters that optimize the performance of the SVM model. As you can see from the code, I searched through a grid of parameters the best ones and found that the best results (accuracy~0.984) are obtained with PCA n_components=45 and SVM parameters C=5, gamma=0.02. However I will here underfit the data a little in order to get more digits on which the classifier fails on the train sample. Thus, I use C=1."},{"metadata":{"trusted":true,"_uuid":"9e06738790fa76df13c3f4dd582d53fee78acb6e"},"cell_type":"code","source":"pipe0 = Pipeline([(\"pca\",PCA(whiten=True)),(\"clf\", svm.SVC(kernel='rbf'))])\n### grid of parameters I tested. The best results are obtained with PCA n_components=45, C=5, gamma=0.02. \n#params={'pca__n_components': [25, 30, 35, 40, 45, 50],'clf__C': [1, 2, 5, 10, 30, 60], 'clf__gamma': [0.02]}\n###\nparams={'pca__n_components': [45],'clf__C': [1], 'clf__gamma': [0.02]}\ngrid_search = GridSearchCV(pipe0, param_grid = params, cv=3, scoring='accuracy', n_jobs=-1)\ngrid_search.fit(train_images,train_labels.values.ravel())\n\nprediction0_test=grid_search.predict(test_images)\nprediction0_train=grid_search.predict(train_images)\n\nprint('Grid best parameter (max. accuracy): ', grid_search.best_params_)\nprint('Grid best score (accuracy): ', grid_search.best_score_)\nprint('Test score (accuracy): ', grid_search.score(test_images, test_labels.values.ravel()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9c7142f04c02f25234c206197b268c5b2f41100"},"cell_type":"markdown","source":"Now I want to see how the misclassified digits look like"},{"metadata":{"trusted":true,"_uuid":"e62378f78a9d6cf55aeaa93773c35315f81f2d4e"},"cell_type":"code","source":"mask_wrong_train=(prediction0_train!=train_labels.values.ravel())\nprint('the model fails on {} digits out of {}'.format(mask_wrong_train.sum(), len(train_labels)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"205506892a9e0e66b941670eeb62d5cedc99a9dd"},"cell_type":"markdown","source":"I plot some of these digits"},{"metadata":{"trusted":true,"_uuid":"713888f55844c70d333592ba0940ab9d471ea7f0"},"cell_type":"code","source":"list=[]\nfor num in np.arange(0,10):\n    train_labels_wrong=train_labels[mask_wrong_train]\n    flag=(train_labels_wrong==num)\n    for i in train_labels_wrong.index[flag][:10]:\n        list.append(i)\n\nplot_digits(list)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44453995e17ca8ed23f144610b4d21ba2b0171d0"},"cell_type":"markdown","source":"One can see that some of the images in the train sample are messy. Some of them are not digits at all: for instance, the following \"digits\" do not look like any digit I know:"},{"metadata":{"trusted":true,"_uuid":"8ef806bb3c5321a1361e3b6799c71f8110726087"},"cell_type":"code","source":"plot_digits([7505, 7362, 6515, 24499, 18534, 23604, 23237, 33466, 26479, 31649, 7514, 28560, 7810, 19634])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4260daaa0fe0a6a577dfe81eb57b9a6edab4676"},"cell_type":"markdown","source":"Some digits are not well framed and part of the digit is cut off:"},{"metadata":{"trusted":true,"_uuid":"090906ecbf5d55ade54b3df1d91bff4a06b5eb40"},"cell_type":"code","source":"plot_digits([16124, 38159, 29538, 21695, ])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2745a1eab9068d1050ded95d358f2920acc826f"},"cell_type":"markdown","source":"Other digits have wrong labels:"},{"metadata":{"trusted":true,"_uuid":"d29302d06f25ef1efc7f715e67c3dad68e61d7ad"},"cell_type":"code","source":"plot_digits([23299, 41691, 25172, 15219, 5747, 4226, 25946])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"274af4f1793b0745f015231fa9821f316c804d2c"},"cell_type":"markdown","source":"With this, I want to suggest that it may be not meaningful to get very high accuracy when the labels from which the model learns are wrong and the train samples contains things that are not digits. Maybe I'm wrong and the cleaning of the sample it's my duty. \n\nBut what about the test sample? Are the test sample labels correct? If the test set of the public leaderboard contains mislabelled digits (and this is very likely because the train set does it) I have to predict a wrong digit to get the \"right\" label. Anyhow, I guess I can't predict digits if they are not digits...\n\nI would be glad to hear your comments about my analysis.\n\nNow that I finished to complain ;) I propose my model, which, despite my effort to do something original, it's just a plain boring PCA + SVM. I apply the procedure to the full dataset"},{"metadata":{"trusted":true,"_uuid":"f1525497c355eac3c91bf451074aae1e4fa61175"},"cell_type":"code","source":"pipe = Pipeline([(\"pca\",PCA(n_components=45, whiten=True)),(\"clf\", svm.SVC(kernel='rbf', C=5, gamma=0.02))])\n\n# fit the model\npipe.fit(X_df,y_df.values.ravel())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33fcbc4ede23363b2366df6037971fc94f009a15"},"cell_type":"code","source":"# upload the test set\nXtest=pd.read_csv('../input/test.csv')\npredicted_test=pipe.predict(Xtest)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ebd40b4093f9fe7e9a449fecfea96d7d21e7b008"},"cell_type":"code","source":"write_results(predicted_test, name='predicted.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e24af917135fd5baa5d89166e7f9419723d1149"},"cell_type":"markdown","source":"With this result I got accuracy=0.98428 on the public leaderboard."},{"metadata":{"trusted":true,"_uuid":"931f1ecbfc220c71852cc3f2ecb61af39a254f1b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}