{"cells": [{"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "58604734-edb8-4600-b577-7696b8d15992", "_uuid": "b2d0a950e9202624f3e124428ad2b37c27f91443"}, "outputs": [], "execution_count": null, "source": "This is my first attempt to data science competitions. I will try using Principal Component Analysis (PCA) for dimension reduction, followed by Support Vector Classifier (SVC) with Radial Basis Function (RBF) as my kernel function. Let's see how this scores!"}, {"cell_type": "code", "metadata": {"trusted": false, "_execution_state": "idle", "_cell_guid": "3adcbdf0-fd1e-4a07-b599-6efa1b840a3d", "_uuid": "7ca3c4a71c875052cc934b69e7236332c11eb0b8"}, "outputs": [], "execution_count": null, "source": "# imports\nimport warnings\nwarnings.simplefilter(action=\"ignore\", category=FutureWarning)\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom matplotlib import cm\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.decomposition import PCA\nfrom sklearn.svm import SVC"}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "6cf6f0e0-1797-40cf-9aeb-72ba60fde298", "_uuid": "734dc221a31c6ebc3be6f2547ca9fb8dc791b7a2"}, "outputs": [], "execution_count": null, "source": "# data locations\ntrain_loc = \"../input/train.csv\"\ntest_loc = \"../input/test.csv\""}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "b675d80f-bce8-4076-85dd-54ef9f9da4d5", "_uuid": "54274318f4f5b44b059452d9048309a3537c1f2e"}, "outputs": [], "execution_count": null, "source": "We will load the train and test data separately. Store the data as numpy ndarrays for easier computations."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "c0a984dd-c220-44d2-af54-dbf3dc13e637", "_uuid": "45939e691d8bfe552e27575b495e3a9bf304b601"}, "outputs": [], "execution_count": null, "source": "train = pd.read_csv(train_loc)\nX = train.drop(\"label\", axis=1).as_matrix()\ntrain_size = X.shape[0]\ny = train[\"label\"].values"}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "59fa4043-f4d1-4e58-acfa-26462553836f", "_uuid": "0b120589cf66c1e01d46ecc37b6d805c62300a51"}, "outputs": [], "execution_count": null, "source": "test = pd.read_csv(test_loc).as_matrix()\ntest_size = test.shape[0]"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "99b496f8-0a55-41f1-934a-a0c9b105e4fd", "_uuid": "16a02ff3ac0c5f53368b8c0b34fbb4748756b13f"}, "outputs": [], "execution_count": null, "source": "We will use 6 images to be visualized, so we can confirm their results manually. Storing 6 random images from the test set for verification."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "61600166-db6d-46a5-9a66-6d2cb7f65330", "_uuid": "9fb2fc9c44157c484418d809b771fdc4852cc438"}, "outputs": [], "execution_count": null, "source": "img_idx = np.random.randint(test_size, size=6)\nfig = plt.figure()\nfor i in range(6):\n    instance = test[img_idx[i]]\n    assert(len(instance) == 784) # making sure we have correct image instances\n    img = instance.reshape(28, 28)\n    plt.subplot(230 + i + 1) # creating a 2x3 image-grid for the images\n    plt.title(img_idx[i] + 1) # ImageId\n    plt.axis(\"off\")\n    plt.imshow(img, cmap=cm.gray)"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "14d8dfb6-1fb4-4255-b559-eada42c51ab1", "_uuid": "f742157be95de4755f36c6e08c1841021bff5346"}, "outputs": [], "execution_count": null, "source": "Now that we have the data loaded, we will reserve part of our training set for validation. This is to verify our model, and to obtain scores. "}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "41862d11-bb08-4b2f-b589-06c2e6a912b7", "_uuid": "d852c08b9169a619d99c0483e8ded6e0c15be312"}, "outputs": [], "execution_count": null, "source": "# X_train, y_train are the training sets and labels used in our model\n# X_test, y_test are the validation sets for our model\nX_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, \n                                                   random_state=50)"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "743c00fa-3daf-4112-8a33-0a74ac976f38", "_uuid": "dd38170d9ab25f6145f048392d5dc3bc2cbfe217"}, "outputs": [], "execution_count": null, "source": "This dataset is huge! The size of training data is 42000 images each having 784 pixels. Therefore, it is necessary we reduce the dimensionality of the instances from 784 to some optimum, that will speed up the classification, by not losing the necessary information. We use Principal Component Analysis (PCA) to do so, and transform our train, validation and test data sets."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "b1aac03c-e918-4834-a1cb-217b1f521299", "_uuid": "faf37c44d7c5395df68edb3f6a7a2a49955b69df"}, "outputs": [], "execution_count": null, "source": "# n_components=33 was determined by running a brute-force\n# over various values, and comparing the best scores\n# very time consuming\npca = PCA(n_components=33, whiten=True)\nX_train = pca.fit_transform(X_train)\nX_test = pca.transform(X_test)\ntest = pca.transform(test)"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "70835906-264c-4b06-a509-6e7ea558d80f", "_uuid": "f34d64495fbe5694006abe9252d4ced470a2eed5"}, "outputs": [], "execution_count": null, "source": "Now we use the Support Vector Classifier (SVC) with Radial Basis Function as our kernel function, for modeling our test data."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "41bd61b7-28b0-40a7-8db0-8b08c77afc8f", "_uuid": "d416e542a7d4f39c156b10cddd74d3351521d5f1"}, "outputs": [], "execution_count": null, "source": "svc = SVC(C=10, kernel=\"rbf\", # the parameters C and gamma were obtained \n          gamma=0.05,        # by using cross validation and grid search method\n          class_weight=\"balanced\", \n          random_state=50)"}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "1d6e7e26-825b-4d13-9eb4-d3bacd2c6808", "_uuid": "a32dbe7490ab8a19d1da2c6542959c31b09da1e3"}, "outputs": [], "execution_count": null, "source": "svc.fit(X_train, y_train)\nscore = svc.score(X_test, y_test)\nprint(\"Score {0:.4f}\".format(score))"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "b35acccb-e63f-4cea-9edb-5ae977b386c2", "_uuid": "127eb29c354b9bf27309c3bc938648ba03c5fbdf"}, "outputs": [], "execution_count": null, "source": "Voila! We have our model validated. We should have an accuracy around 0.985 using our model. That's a great start! \nWe now use the model on the test data for predicting the labels."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "b8a669a7-9688-4064-b2c9-cdf4eed97353", "_uuid": "44d950e431ff5b75528b9928afc5de22eaf7a522"}, "outputs": [], "execution_count": null, "source": "labels = svc.predict(test)"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "8c717f51-9200-4b1f-b30c-ee252561cd70", "_uuid": "955fb197410549745a8f570709a70150045953f0"}, "outputs": [], "execution_count": null, "source": "We will now save our results as a DataFrame."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "2dc620ce-e880-4356-88a5-9ef316c03654", "_uuid": "a54cc3567849f4422a6e30fa3bcf8614a873700c"}, "outputs": [], "execution_count": null, "source": "res = pd.DataFrame(columns=[\"ImageId\", \"Label\"])\nres[\"ImageId\"] = pd.Series(1 + np.arange(test_size))\nres[\"Label\"] = pd.Series(labels)"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "381fcc9a-df8d-49c4-8e32-2d4b49a81ade", "_uuid": "486b719ca57300b83838c8da0fa932f1d667ee20"}, "outputs": [], "execution_count": null, "source": "Next we display the results for the 6 images displayed above, for manual verification."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "6874b29c-2295-4aea-884e-d22d3720098d", "_uuid": "166d9c06b6fe032109779f91470fd4b762640e3e"}, "outputs": [], "execution_count": null, "source": "res.iloc[img_idx]"}, {"cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "1bb323e9-fee1-4ac2-9cf0-45350724c7f5", "_uuid": "6177d95423c021f58068437b0507d0b0b75b825d"}, "outputs": [], "execution_count": null, "source": "Saving the results as csv file..."}, {"cell_type": "code", "metadata": {"trusted": false, "collapsed": false, "_execution_state": "idle", "_cell_guid": "15043536-8a9c-424e-80cd-8bdca0f88e07", "_uuid": "1f384d8fea2310ba9e970afc4761deb86fd1a3df"}, "outputs": [], "execution_count": null, "source": "res.to_csv(\"submission.csv\", index=False)"}, {"cell_type": "code", "metadata": {"collapsed": false, "_execution_state": "idle", "_uuid": "ed323103cc820be5431f380ab977e097c7eea83c"}, "outputs": [], "execution_count": null, "source": ""}], "metadata": {"kernelspec": {"display_name": "Python 3", "name": "python3", "language": "python"}, "language_info": {"nbconvert_exporter": "python", "file_extension": ".py", "name": "python", "mimetype": "text/x-python", "codemirror_mode": {"version": 3, "name": "ipython"}, "pygments_lexer": "ipython3", "version": "3.6.1"}}, "nbformat": 4, "nbformat_minor": 0}