{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1 align=center><font size=5>Digit Recognition. PCA + SVM </font></h1>\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Needed libraries and modules\nimport pandas as pd                                  # data processing, reading CSV file, dealing with dataframes etc\nimport numpy as np                                   # linear algebra functionalities \nimport seaborn as sns                                # visualization library\nimport matplotlib.pyplot as plt                      # visualization library\n\n#Modeling\nfrom sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import accuracy_score\n\nimport math \nimport time\nimport multiprocessing\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Table of Contents\n\n* [1. Loading and checking the data ](#Section1)\n* [2. Dimensionality reduction (PCA)](#Section2)\n* [3. SVM](#Section2)\n   "},{"metadata":{},"cell_type":"markdown","source":" # 1. Loading and checking the data <a class=\"anchor\" id=\"Section1\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Loading training and test set to dataframes\npath_train = '../input/digit-recognizer/train.csv'\npath_test = '../input/digit-recognizer/test.csv'\n\ndf_train = pd.read_csv(path_train)\ndf_test = pd.read_csv(path_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Converting train and test sets to numpy arrays\n\n#Train set\nX_train = df_train.iloc[:, 1:].to_numpy()\nY_train = df_train.iloc[:, 0].to_numpy()\n\n#Test set \nX_test = df_test.to_numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Amount of examples in the train set is:\", Y_train.size)\nprint(\"Shape of the train set is:\", X_train.shape)\nprint(\"Shape of the test set is:\", X_test.shape)\nprint(\"Each example is an image, i.e.  a squared matrix of dimemsion:\", math.sqrt(X_train.shape[1]))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot a few pictures"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Function to plot n^2 random images in an array of nxn\nimport random\n\ndef PlotingPictures(Data, size1, size2, n, label = [], rand = False):\n    \"\"\"\n    This functions plots n^2 images of size (size1 x size2) in a squared grid of order n.\n    \n    Parameters:\n     1. Data: a list of images of order (k, a, b), that stores k images of order (a,b)\n     2. size1 and size2: the shape of the images to be dispayed (size1xsize2 = axb)\n     3. n: the size of the squared grid in which we will polt the images\n     4. label: set of labels for the images (default empty list)\n     5. rand: indicates whether we want to display random images or the first n^2 images\n    \"\"\"\n    # width, height in inches\n    fig = plt.figure(figsize=(1.5*n, 1.5*n))  \n    \n    if rand:\n        #Taking n^2 random indices for the images to be displayed\n        sample = random.sample(range(Data.shape[0]), pow(n,2))\n    else:\n        sample = range(pow(n,2))\n    \n    for i in range(pow(n,2)):\n        #making a sub plot, each sumplot at a time, and ploting the corresponding image\n        sub = fig.add_subplot(n, n, i + 1)\n        plt.subplots_adjust(hspace = 3/(n))\n        if len(label) !=0:\n            sub.set_title(\"Label: \" + str(label[sample[i]]))\n        sub.imshow(Data[sample[i]].reshape(size1,size2), interpolation='nearest', cmap='gray')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PlotingPictures(X_train, 28, 28, 4, Y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Dimensionality reduction <a class=\"anchor\" id=\"Section2\"></a>"},{"metadata":{},"cell_type":"markdown","source":"The total amount of black pixels is often predominant\nand the caligraphic signs needed to write the 10 digits are similar.  Therefore, it's reasonable to assume a\nhigh correlation and a low variance on several components. That is why we will perform dimensionality reduction before training our models.\n\nWe want to keep 95% of the variance.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA\n\npca = PCA(n_components=0.95, svd_solver = 'full')\n\n#Fit the model with X_train.\npca.fit(X_train/255)\n\n#Apply dimensionality reduction to train and test sets.\nX_pca_train = pca.transform(X_train/255)\nX_pca_test = pca.transform(X_test/255)\n\ndim_pca= pca.n_components_\n\nprint('Shape of train set after dimnesionality reduction:', X_pca_train.shape)\nprint('Shape of trest set after dimnesionality reduction:', X_pca_test.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see  which part of the total variance is carried by each single component."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting the variance ratio per single component\n\nplt.bar(range(dim_pca), pca.explained_variance_ratio_, align='center', alpha=0.5)\nplt.ylabel('Variance ratio')\nplt.xlabel('Components')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's us recover the original images after dimensionality reduction. This is just to make sure visually most if the  information is still preserved."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_rebuilt_train = pca.inverse_transform(X_pca_train)\nPlotingPictures(X_rebuilt_train, 28, 28, 4, Y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Support Vector Machine <a class=\"anchor\" id=\"Section3\"></a>"},{"metadata":{},"cell_type":"markdown","source":"Grid search"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#Setting parameters\nparam_grid = [{'kernel': ['rbf'],\n             'C': [0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8, 25.6] }]\n\nGrid = GridSearchCV(estimator=SVC(), param_grid=param_grid, scoring='accuracy', cv=4, n_jobs=multiprocessing.cpu_count())\n\nGrid.fit(X_pca_train, Y_train)\n\nBestRM=Grid.best_estimator_\nprint('Best model:\\n', BestRM)\n\n#Accuracy of best model found after CV\nprint('Accuracy of best model', Grid.best_score_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look closer to the CV accuracies for the different values of regularization parameter C."},{"metadata":{"trusted":true},"cell_type":"code","source":"Grid.cv_results_['mean_test_score']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(param_grid[0]['C'], Grid.cv_results_['mean_test_score'])\nplt.title('CV_acurracy vs C') \nplt.xlabel('Regularization parameter C')\nplt.ylabel('CV accuracy')\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that accuracy remains relatively stable above 0.98 for C > 1.6"},{"metadata":{},"cell_type":"markdown","source":"Let's make predictions on the test set and plot some pictures"},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_hat_test = BestRM.predict(X_pca_test)\nPlotingPictures(X_test, 28, 28, 6, Y_hat_test, rand=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\nsample_sub['Label'] = Y_hat_test\nsample_sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub.to_csv('SVM_submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}