{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "aae05b0b-22c5-1df7-c1ff-0a762a75d432" }, "source": [ "Let's get started!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5a4dd6c8-2e8e-0977-a774-c44a199f5942" }, "outputs": [], "source": [ "# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "# print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output.\n", "df = pd.read_csv('../input/train.csv')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "243c16fa-5149-a19d-70a9-8801eee1273e" }, "outputs": [], "source": [ "import numpy as np\n", "# separate X and Y\n", "Y = df.iloc[:, 0].values # target label, np.ndarray\n", "X = df.iloc[:, 1:].values.astype(float) # inputs, n x 284 size, np.ndarray\n", "\n", "from sklearn.preprocessing import StandardScaler\n", "# standarize the input\n", "scaler = StandardScaler().fit(X) # for later scaling of test data\n", "X_std = StandardScaler().fit_transform(X)\n", "# to get the covariant matrix, for the later PCA\n", "cov = np.cov(X_std.T)\n", "# to get the eigen-vectors and eigen-values\n", "eig_vals, eig_vecs = np.linalg.eig(cov)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7eecbd71-33cb-f696-d6dd-912d8097d211" }, "outputs": [], "source": [ "# sort the eigen-value, eigen-vector pairs\n", "eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(eig_vals.shape[0])]\n", "eig_pairs.sort(key = lambda x: x[0], reverse=True) " ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "67e57d14-bf32-aa29-7395-db1b8f465622" }, "source": [ "Now we have obtained the *n* eigen-values (*n* = 784 in this case), and the accompanying *n* eigen-vectors ( each eigen-vector is a *n* x 1 vector), the next step in PCA is to choose the first *d* largest eigen-values, and project the original data into the *d*-dimensional subspace consisting the corresponding eigen-vectors. Note that *d*< *n*. In the process, we reduce the each 1 x *n* original data entry to a smaller (shorter) 1 x *d* entry. \n", "\n", "The question is then, what is a good value of *d*?\n", "\n", "Remembering that, for a 'good' vector-base, all the projections of a high-dimension data onto this 'good' vector-base (i.e. the 'coordinate indices' of the data on this vector base) should spread out (i.e. having large variance). For example, considering a 2-dimensional vector space, x-y, and I have 10 different data points that *all* lie on the x-axis. Making it even simple, let me call them (1, 0), (2, 0), ..., (10, 0). In this silly example, one can quickly tell how to perform PCA: we can just use x-coordinate indices to represent all the data, hence reducing the 2-D space to 1-D space. Why? Because there is no spread (or small variance, well in this case, zero variance) on the y-coordinate indices. \n", "\n", "Therefore, let's quantum leap from 2-dimensional to *n*-dimensional space, we will say, we want the projections to the chosen *d*-dimensional subspace maintain as much variance of the original *n*-dimensional data as possible. Humm.. how about let's chose a value *d* that can maintain 90% of original variance?\n", "\n", "We already know, by the virtue of eigen-values, if the original data is projected onto the eigen-vector with the largest eigen-value, then the resulting indices will have the largest variance. Hence, we should definitely keep this eigen-vector in our final subspace. Therefore, I can just simply start to recruit the eigen-vectors in this fashion (with descending order of the corresponding eigen-values), and keep an eye on the accumulated eigen-values: once it has reached 90% of sum of all eigen-values, I will stop, and I have my *d*.\n", "\n", "Let's visualize the above process." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d991fc05-93fb-1362-cdc4-1ec6f5704a0e" }, "outputs": [], "source": [ "# calculate sum of eigen-values\n", "eig_vals = np.abs(eig_vals)\n", "eig_vals = -np.sort(-eig_vals)\n", "eig_cum = 100*np.cumsum(eig_vals)/sum(eig_vals)\n", "eig_df = pd.DataFrame(eig_vals, columns=['eigen-value'])\n", "eig_df['cum_per'] = eig_cum\n", "\n", "# plotting\n", "import bokeh.charts as bkc\n", "import bokeh.plotting as bkp\n", "from bokeh.models import LinearAxis, Range1d, HoverTool\n", "\n", "bkp.output_notebook()\n", "source = bkc.ColumnDataSource(eig_df)\n", "# Create a HoverTool: hover\n", "p1 = bkp.figure(plot_height = 300, plot_width = 700, \n", " x_axis_label = 'eigen-value number', y_axis_label = 'eigen-value',\n", " tools = [\"pan, box_zoom, reset, resize, hover\"], toolbar_location=\"above\")\n", "p1.vbar(x = 'index', top = 'eigen_value', width = 0.5, source = source, \n", " color = 'blue', alpha = 0.5)\n", "hover = p1.select(dict(type = HoverTool))\n", "hover.tooltips = dict([(\"Index\",\"@index\"), (\"Value\", \"@eigen_value{0.00}\")])\n", "# Setting the second y axis range name and range\n", "#p.extra_y_ranges = {\"cum_per\": Range1d(start=0, end=100)}\n", "# Adding the second axis to the plot. \n", "#p.add_layout(LinearAxis(y_range_name=\"cum_per_axis\"), 'right')\n", "# Using the aditional y range \n", "#p.line(x = 'index', y = 'cum_per', source=source,\n", "# color=\"green\", y_range_name=\"cum_per_axis\")\n", "\n", "bkp.show(p1)\n", "\n", "p2 = bkp.figure(plot_height = 300, plot_width = 700, \n", " x_axis_label = 'eigen-value number', y_axis_label = 'cum sum (%)',\n", " tools = [\"pan, box_zoom, reset, resize, hover\"], toolbar_location=\"above\")\n", "p2.circle(x = 'index', y = 'cum_per', source = source, alpha = 0.5, color = 'red')\n", "hover = p2.select(dict(type = HoverTool))\n", "hover.tooltips = dict([(\"Index\",\"@index\"), (\"Value\", \"@cum_per\")])\n", "bkp.show(p2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "020d92ae-0d73-5542-53fd-885ecd1f39df" }, "outputs": [], "source": [ "# find the 90% index\n", "d = eig_df['cum_per'].searchsorted(90,side='right')[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "77a31c7d-ee2e-3692-7bc6-b3390411e03c" }, "outputs": [], "source": [ "# perform PCA\n", "from sklearn.decomposition import PCA as sklearnPCA\n", "sklearn_pca = sklearnPCA(n_components=d)\n", "pca = sklearn_pca.fit(X_std)\n", "X_sklearn = sklearn_pca.fit_transform(X_std)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "75aec83b-6c52-9c3d-1d1c-9f16f88ffa9a" }, "outputs": [], "source": [ "# split the training set\n", "mask = [True,]*int(0.8*X_sklearn.shape[0])\n", "mask.extend([False,]*(X_sklearn.shape[0] - int(0.8*X_sklearn.shape[0])))\n", "mask = np.random.permutation(mask)\n", "X_train, Y_train = X_sklearn[mask], Y[mask] \n", "X_xv, Y_xv = X_sklearn[~mask], Y[~mask] \n", "\n", "# train a Neural Network\n", "from sklearn.neural_network import MLPClassifier\n", "mlp = MLPClassifier(hidden_layer_sizes=(100,20), max_iter=50, alpha=1e-4,\n", " solver='sgd', verbose=10, tol=1e-4, random_state=1,\n", " learning_rate_init=.1)\n", "\n", "mlp.fit(X_train, Y_train)\n", "print(\"\\nTraining set score: %f\" % mlp.score(X_train, Y_train))\n", "print(\"Test set score: %f\" % mlp.score(X_xv, Y_xv))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "00580d4c-69e6-3733-c9cf-aa8d0dd687d0" }, "outputs": [], "source": [ "# read the test data\n", "df_test = pd.read_csv('../input/test.csv')\n", "X_test = df_test.iloc[:, :].values.astype(float) # inputs, n x 284 size, np.ndarray\n", "\n", "# perform the scaling and the PCA on the test data\n", "X_test_std = scaler.transform(X_test)\n", "X_final = pca.transform(X_test_std)\n", "\n", "# predict with the train NN\n", "Y_final = mlp.predict(X_final)\n", "\n", "# save the output\n", "df_output = pd.DataFrame(np.arange(1, Y_final.shape[0]+1), columns=['ImageId'])\n", "df_output['Label'] = Y_final\n", "df_output.to_csv('output.csv', index=False, header=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "27040890-93d0-2570-2880-49955ba92fd9" }, "outputs": [], "source": "" } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }