{"cells":[{"metadata":{"_cell_guid":"88525aed-c0ad-472c-8d2e-2c949a5bd2b3","_uuid":"846a4bfeb71482bc6146299aa7e8b0b8791974d4"},"cell_type":"markdown","source":"## Intoduction\nlink to competition: [digit recogniger competition](https://www.kaggle.com/c/digit-recognizer)\n\nMy Kaggle Profile: [Amit Vikram | Kaggle](https://www.kaggle.com/amitkvikram)\n\n- It gives me immense pleasure to launch this kernel. For your sake of convenience I got accuracy of **0.98442**.\n\n- Here we will use PCA for dimensionality reduction and then train the data using \"Logistic Regression with solver lbfgs\" and  \"SVM\". "},{"metadata":{"_cell_guid":"a69922cc-96d5-48ed-8682-7e7437f99fad","_uuid":"213689418e78580706c5db6c6728e97da0c0ed7a"},"cell_type":"markdown","source":"## 1. Import data"},{"metadata":{"_cell_guid":"d7d3f5e4-85ac-4709-8fca-b271ee7f8279","_uuid":"c703701cf260091b1eafe3eab2c6a1ec27f13e26","collapsed":true,"trusted":false},"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport scipy.io\nimport warnings\nwarnings.filterwarnings('ignore')\nTrain = pd.read_csv(\"../input/train.csv\").values\nTest = pd.read_csv(\"../input/test.csv\").values","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"bf2db49d-3d6c-4730-b6ff-d68e8702f547","_uuid":"4e4a10748c2657787c376127ec8bf97417c5fbc1","trusted":false},"cell_type":"code","source":"Train.shape","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"12e83e9a-5409-4e58-81f7-352b3ed22c53","_uuid":"439f73047e4e7258d8b91bbf68aa6a9a9d96aebe","trusted":false},"cell_type":"code","source":"Test.shape","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"65bf5b45-a07f-4c84-8b39-6a6d147eb3b1","_uuid":"ac2616dedfa16e143bfca92a12418efb165cef88","trusted":false},"cell_type":"code","source":"X = (Train[:,1:])\nY = (Train[:,0])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7d4c88e4-8903-475e-8d32-4008ee5e54e5","_uuid":"4656f32f4da01679dde2e6c971987591124cd648"},"cell_type":"markdown","source":"## 2. Dimensionality Reduction using PCA"},{"metadata":{"_cell_guid":"56ed120f-46fc-42a1-96e9-5792e28a1f4a","_uuid":"15b7655f2bd79d59864f9aa030d8b102d97e5934"},"cell_type":"markdown","source":"#### a. Plot graph of component vs. cumulative explained variance .\nThis graph will help us in choosing the no of components for training our data."},{"metadata":{"_cell_guid":"4ec458b5-5c7a-4520-948c-38e9b36e7726","_uuid":"d6121880623ed33c299386d04831a96b0a02193b","trusted":false},"cell_type":"code","source":"from sklearn.decomposition import PCA\n# ## Removing the column with variance 0\nvariance = np.var(X, axis = 0)>1000\nprint(variance.shape)\nX = X[:, variance]\nTest = Test[:,variance]\nprint(X.shape)\n# ##Calculate Principal Components\npca = PCA()\npca.fit(X)\nexplained_variance = pca.explained_variance_ratio_\n# ##Calculate cumulative explained ration\ncum_explained_variance = [np.sum(explained_variance[:i+1]) for i in range (0, 201, 25)]\nX_axis = [i for i in range(0, 201,25)]\n\n##Plot Graph\nfig = plt.figure(figsize = (5.841, 7.195), dpi=100)\nplt.plot(X_axis, cum_explained_variance, 'ro')\nplt.grid(True, which = 'both')\nplt.yticks(cum_explained_variance)\nplt.xticks(X_axis)\nplt.ylabel(\"Explained Variance Ratio\")\nplt.xlabel(\"No. of Components\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"256448a6-43bb-4ad7-90af-dc38359a8cb7","_uuid":"370a73a0b9d38e88f08c644c33373812c2a713ce"},"cell_type":"markdown","source":"**So Looking on the above graph, 50 components comprise 80% variance. So first we will go with 50 componets**"},{"metadata":{"_cell_guid":"7aa8e43d-d81d-412c-94e6-c7576fa2d3ac","_uuid":"1f4d3b481531a4af55ed9ea27cbde3cdf868d44f"},"cell_type":"markdown","source":"### Splitting data for training and testing\n- Training data: 80%\n- Test data: 20%"},{"metadata":{"_cell_guid":"7e555966-45d2-4713-8640-6d414bb6e291","_uuid":"f00576135b9504cc80d39b2a3a767b9543e3b9a2","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, Y, test_size  = 0.20, random_state  = 0)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"28097d9a-29e6-4870-8bac-c996eb773ef5","_uuid":"cab35e9ce318f2ac5c66c0ec1570ce52a0dea03d"},"cell_type":"markdown","source":"### define normalize function for normalizing the data, PrincipalComponents function to return top n principal components."},{"metadata":{"_cell_guid":"87dc2b37-1094-4df3-b720-a1c290b8e3c4","_uuid":"d0b11576f6897d361122ed06c74257b7b7dd41c5","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn import linear_model\nfrom sklearn.preprocessing import MinMaxScaler\n\ndef normalize(sigma2, mean2, X):\n    X = (X-mean2)/sigma2\n    return X\n\ndef PolynomialFeatures1(X):\n    X_2 = np.square(X)\n    X = np.column_stack((X, X_2))\n    return X\n\nJ1 = []\nJ2 = []\n\n##Take n principal components\ndef PrincipalComponents(n):\n    pca = PCA(n_components= n)\n    X_train1 = pca.fit_transform(X_train)\n    X_test1 = pca.transform(X_test)\n    return X_train1, X_test1\n\n## Logistic Regression\n\ndef LogisticRegression(X_train2, y_train2, X_test2, y_test2, penalty):\n    print(\"penalty= \", penalty)\n    regr = linear_model.LogisticRegression(solver='lbfgs',max_iter=1000, C=penalty)\n    regr.fit(X_train2, y_train2)\n    score1 = regr.score(X_train2, y_train2)\n    score2 = regr.score(X_test2, y_test2)\n    print(score1, score2)\n    Prediction = regr.predict(X_test2)\n    return score1, score2, Prediction\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ce08b04f-3e9a-4f60-b100-b1cb8f6c096e","_uuid":"a9bb8cb09e95e5c051a29ee829ede3ace7a39b90"},"cell_type":"markdown","source":"**So usually variance of 0.8 is sufficient to explain the variation in data, so we will first train data by taking the top n principal components which can explaine the variance of 0.8**"},{"metadata":{"_cell_guid":"1b1e0d3a-e931-42bc-8b27-64e64b729a0c","_uuid":"206b7691f80c4d2d0d666b8571c3993fc82c4768","collapsed":true,"trusted":false},"cell_type":"code","source":"X_train1, X_test1 = PrincipalComponents(0.8)   # Getting principal components\nJ1 = []\nJ2 = []\nfor i in range(20000, X_train1.shape[0], 1500):\n    score1, score2, Prediction = LogisticRegression(X_train1[:i+1,:], y_train[:i+1, ], \n                                                    X_test1[:i+1,:], y_test[:i+1], 0.1)\n    J1.append(1-score1)\n    J2.append(1-score2)\n    \n    \nplt.plot(J2, 'b-', label = \"CV error\")\nplt.plot(J1, 'r-', label =\"training error\")\nplt.legend()\nplt.ylabel(\"Error\")\nplt.xlabel(\"No of Iterations\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1f6a8ad3-c379-4631-88ec-c326c59a1e85","_uuid":"def7c34d94be0b8472ee21a0fe20b8c71de9a5fc"},"cell_type":"markdown","source":"- We got a very irregular graph and not a satisfied accuracy and the accuracy doesn't increase with the increaing training data. So it seems that we are suffering from a high bias. Let us try to train data with top n components explaining the variance of 0.9"},{"metadata":{"_cell_guid":"8a4c8e47-3371-4907-8a2d-2ff5a8c8a6b0","_uuid":"5b765242b1b78d8d18ccd788ce652a608286a9c9","collapsed":true,"trusted":false},"cell_type":"code","source":"X_train1, X_test1 = PrincipalComponents(0.9)\nJ1 = []\nJ2 = []\nfor i in range(20000, X_train1.shape[0], 1500):\n    score1, score2, Prediction = LogisticRegression(X_train1[:i+1,:],\n                                                    y_train[:i+1, ], X_test1[:i+1,:], y_test[:i+1],0.1)\n    J1.append(1-score1)\n    J2.append(1-score2)\n    \n    \nplt.plot(J2, 'b-', label = \"CV error\")\nplt.plot(J1, 'r-', label =\"training error\")\nplt.legend()\nplt.ylabel(\"Error\")\nplt.xlabel(\"No of Iterations\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"39180d2c-5084-4946-9396-fb907976ed14","_uuid":"65507cf388cc47b3d3f94c5f03e85dacb4bdb08c"},"cell_type":"markdown","source":"** We can see that accuracy didn't increase much so it doesn't seems a gud idea. Let us again take top n components explaining the variance of 0.8 and also include thieir polynomials(degree = 2)**."},{"metadata":{"_cell_guid":"b84cf435-0bab-40bc-b39e-e4bec724b6ec","_uuid":"a8b4e172d0721b998da8ef2d15dbf422868b606f"},"cell_type":"markdown","source":"#### Add polynomial features"},{"metadata":{"_cell_guid":"4ebf86ce-2811-4d1b-b589-d4c5da8ab8ae","_uuid":"97aac9ccf64b88cc20a1bec1e7817b4409dde8b8"},"cell_type":"markdown","source":"##### 1.Logistic Regression"},{"metadata":{"_cell_guid":"c1097439-f1b0-44d8-a684-210b6100be00","_uuid":"736e43cfcb1289d7e4179a57174737004b14b1d5","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import PolynomialFeatures\n\npoly = PolynomialFeatures(include_bias = False)\nX_train1, X_test1 = PrincipalComponents(0.81)\nprint(X_test1.shape, X_train1.shape)\n\nX_train1 = poly.fit_transform(X_train1)\nX_test1 = poly.transform(X_test1)\n\nprint(X_test1.shape, X_train1.shape)\n\nsigma = np.std(X_train1, axis = 0)\nmean = np.mean(X_train1, axis = 0)\n\nX_train1 = normalize(sigma , mean , X_train1)\nX_test1 = normalize(sigma , mean, X_test1)\n\nJ1 = []\nJ2 = []\nfor i in range(25000, X_train1.shape[0], 1500):\n    score1, score2, Prediction = LogisticRegression(X_train1[:i+1,:], \n                                                    y_train[:i+1, ], X_test1[:i+1,:], y_test[:i+1],0.1)\n    J1.append(1-score1)\n    J2.append(1-score2)\n    \n    \nplt.plot(J2, 'b-', label = \"CV error\")\nplt.plot(J1, 'r-', label =\"training error\")\nplt.legend()\nplt.ylabel(\"Error\")\nplt.xlabel(\"No of Iterations\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e6ccb3d2-5f7f-49d9-958e-e468e390278c","_uuid":"70f4c14c12d299725cd638f317e19bf9b2226971"},"cell_type":"markdown","source":"** We got way better accuracy here and is data over  fitting .... may be but since we are getting good result on cross validation data also, It's gud to go with this.**"},{"metadata":{"_cell_guid":"090e4276-d3f7-4a2d-8844-9a3dbda14d79","_uuid":"62c956377ea1617f81f03567b83a8829cec6f260"},"cell_type":"markdown","source":"#### Saving data trained with logistic regression"},{"metadata":{"_cell_guid":"562be36a-4f21-407e-a140-4fd4be41ba2e","_uuid":"53efdbcca699d15c29bcf43e3a521d35a9beb6af","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import PolynomialFeatures\n# poly = PolynomialFeatures(include_bias = False)\n\nX1 = X.copy()\nTest1 = Test.copy()\n\npca = PCA(n_components= 0.8388)\nX1 = pca.fit_transform(X1)\nTest1 = pca.transform(Test)\n\nprint(X1.shape, Test1.shape)\n\nX1 = poly.fit_transform(X1)\nTest1 = poly.fit_transform(Test1)\n\nprint(X1.shape, Test1.shape)\n\nsigma = np.std(X1, axis = 0)\nmean = np.mean(X1, axis = 0)\n\nX1 = normalize(sigma, mean, X1)\nTest1 = normalize(sigma , mean, Test1)\nprint(X1.shape, Test1.shape)\n\n\nregr = linear_model.LogisticRegression(solver='lbfgs',max_iter=1000, C=0.1)\nregr.fit(X1, Y)\nscore1 = regr.score(X1, Y)\nprint(score1)\nPrediction = regr.predict(Test1)\nimage_id = np.arange(1,Prediction.shape[0]+1)\npd.DataFrame({\"ImageId\": image_id, \"Label\": Prediction}).to_csv('out_reg1.csv', \n                                                                      index=False, header=True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"67ea2748-8374-4308-9844-7ca7bf3cedca","_uuid":"85b5fcfa01bf54ae2a52c0d125be67a6629e4204"},"cell_type":"markdown","source":"#### I submitted the data and got an accuracy of 0.9815. Now we will try to train data with SVM and kernel = 'rbf'. Note we will not use polynomial features with SVM since SVM maps the data in higher dimensions so there is no point in including the polynomial features, also SVM doesn't perform well with too many features."},{"metadata":{"_cell_guid":"896fb162-eb83-4e41-9d88-e8c54fd11b41","_uuid":"7b4f05f69a1f1024e93250c06d7183890fcdb783"},"cell_type":"markdown","source":"##### 2.SVM"},{"metadata":{"_cell_guid":"c05b7039-0587-4f30-bb45-c44d94ee2326","_uuid":"e219b8a43580b09cd276d1667842f9a6bcf3a6f1","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.svm import SVC\n\ndef supportVM(X_train2, y_train2, X_test2, y_test2, penalty):\n    regr = SVC(kernel ='rbf', C=penalty)\n    regr.fit(X_train2, y_train2)\n    score1 = regr.score(X_train2, y_train2)\n    score2 = regr.score(X_test2, y_test2)\n    print(score1, score2)\n    Prediction = regr.predict(X_test2)\n    return score1, score2, Prediction\n\nX_train1, X_test1 = PrincipalComponents(0.81)\nprint(X_test1.shape, X_train1.shape)\n\nsigma = np.std(X_train1, axis = 0)\nmean = np.mean(X_train1, axis = 0)\n\nX_train1 = normalize(sigma , mean , X_train1)\nX_test1 = normalize(sigma , mean, X_test1)\n\nJ1 = []\nJ2 = []\nfor i in range(25000, X_train1.shape[0], 1500):\n    score1, score2, Prediction = supportVM(X_train1[:i+1,:], \n                                                    y_train[:i+1, ], X_test1[:i+1,:], y_test[:i+1],10)\n    J1.append(1-score1)\n    J2.append(1-score2)\n    \nplt.plot(J2, 'b-', label = \"CV error\")\nplt.plot(J1, 'r-', label =\"training error\")\nplt.legend()\nplt.ylabel(\"Error\")\nplt.xlabel(\"No of Iterations\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f6dee075-618d-404c-a000-f97386ed3706","_uuid":"213f8d2c96984228251f6ace3d12278bc20883e4"},"cell_type":"markdown","source":"- So it seems that this model is performin a little bit better, So we will go with it, Let's save the prediction and save it."},{"metadata":{"_cell_guid":"2252dd9f-3855-4085-9700-57f51d9aa3e7","_uuid":"5c4f0ba0e675137874402a241dfe21060a8eb4e2"},"cell_type":"markdown","source":"##### Saving prediction trained with SVM"},{"metadata":{"_cell_guid":"d32d8b1d-0874-4289-bcb9-8e6b199a3a0c","_uuid":"7fd4eadd286b560eb468bbf51d50d1b173890fbf","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.svm import SVC\n# poly = PolynomialFeatures(include_bias = False)\n\nX1 = X.copy()\nTest1 = Test.copy()\n\npca = PCA(n_components= 0.8388)\nX1 = pca.fit_transform(X1)\nTest1 = pca.transform(Test)\n\nprint(X1.shape, Test1.shape)\n\nprint(X1.shape, Test1.shape)\n\nsigma = np.std(X1, axis = 0)\nmean = np.mean(X1, axis = 0)\n\nX1 = normalize(sigma, mean, X1)\nTest1 = normalize(sigma , mean, Test1)\nprint(X1.shape, Test1.shape)\n\n\nregr = SVC(kernel ='rbf', C=10)\nregr.fit(X1, Y)\nscore1 = regr.score(X1, Y)\nprint(score1)\nPrediction = regr.predict(Test1)\nPrediction = regr.predict(Test1)\nimage_id = np.arange(1,Prediction.shape[0]+1)\npd.DataFrame({\"ImageId\": image_id, \"Label\": Prediction}).to_csv('out_svm.csv', \n                                                                      index=False, header=True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"389826f3-5e97-4634-a1dd-5f340547de4e","_uuid":"6d86f34a1ee5ad3c6855578c202f1f24c828530e"},"cell_type":"markdown","source":"** I got an accuracy of 0.98442 on leaderboad with above model**.\nI am learning CNN, So in future I will use that to train data and most probably I will launch a kernel also .Please give your valuable feedback."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}