{"cells":[{"metadata":{"_uuid":"2df33e5389533544ca69f36129ffba91f1adb6e5","_cell_guid":"ce7d6fa0-5c61-0d94-9941-9eef9ea68077"},"cell_type":"markdown","source":"*CAVEAT: Sorry but just note this notebook can be a bit slow to load probably due to the Plotly embeddings displaying a large number of points*\n\n#Introduction\n\nThere already exists a plethora of notebooks discussing the merits of dimensionality reduction methods, in particular the Big 3 of PCA (Principal Component Analysis), LDA ( Linear Discriminant Analysis) and TSNE ( T-Distributed Stochastic Neighbour Embedding). Quite a handful of these have compared one to the other but few have gathered all 3 in one go. Therefore this notebook will aim to provide an introductory exposition on these 3 methods as well as to portray their visualisations interactively and hopefully more intuitively via the Plotly visualisation library. The chapters are structuredas follows:\n\n 1. **Principal Component Analysis ( PCA )**  - Unsupervised, linear method\n\n\n 2. **Linear Discriminant Analysis (LDA)** - Supervised, linear method\n\n\n 3. **t-distributed Stochastic Neighbour Embedding (t-SNE)** - Nonlinear, probabilistic method\n\nLets go."},{"metadata":{"_uuid":"4bee068051d8d31245eb44efcec101faa7519b36","_execution_state":"idle","_cell_guid":"bbbfbfb7-4cfd-7c9f-523a-1c4d474e7919","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport seaborn as sns\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\nimport matplotlib\n%matplotlib inline\n\n# Import the 3 dimensionality reduction methods\nfrom sklearn.manifold import TSNE\nfrom sklearn.decomposition import PCA\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30a04ee76d6633d67d30446dd84c2fd3160ec1c1","_cell_guid":"f0ef3f94-be7c-8108-279e-41f8c5a28185"},"cell_type":"markdown","source":"**Curse of Dimensionality & Dimensionality Reduction**\n\nThe term \"Curse of Dimensionality\" has been oft been thrown about, especially when PCA, LDA and TSNE is thrown into the mix. This phrase refers to how our perfectly good and reliable Machine Learning methods may suddenly perform badly when we are dealing in a very high-dimensional space. But what exactly do all these 3 acronyms do? They are essentially transformation methods used for dimensionality reduction. Therefore, if we are able to project our data from a higher-dimensional space to a lower one while keeping most of the relevant information, that would make life a lot easier for our learning methods."},{"metadata":{"_uuid":"0c95e9c20988e56008516e7a38c9ed4da033314b","_cell_guid":"8da70ed8-177c-a295-6069-7352a3ff22d6"},"cell_type":"markdown","source":"# MNIST Dataset\n\nFor the purposes of this interactive guide, the MNIST (Mixed National Institute of Standards and Technology) computer vision digit dataset was chosen partly due to its simplicity and also surprisingly deep and informative research that can be done with the dataset. So let's load the training data and see what we have"},{"metadata":{"_uuid":"813379999b963b572b5b9631e36296dcdaee483a","_execution_state":"idle","_cell_guid":"2781f1bf-44b8-bb42-fa9e-c4f5233d4ed5","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest=pd.read_csv('../input/test.csv')\nprint(train.head())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(test.head())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b0c47777a5f3114b4962d5fcdb2032f2f503a87","_execution_state":"idle","_cell_guid":"d3ff48ed-51af-d315-df10-64024922a932","trusted":true},"cell_type":"code","source":"print(train.shape)\nprint(train.shape)\nsns.countplot(x='label', data=train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a0fdbc22dc99b05f10d7839cfc4a1a428a5e5dc","_cell_guid":"6b142c78-a513-0a7c-a8fd-83c55b81a74b"},"cell_type":"markdown","source":"The MNIST set consists of 42,000 rows and 785 columns. There are 28 x 28 pixel images of digits  ( contributing to 784 columns) as well as one extra label column which is essentially a class label to state whether the row-wise contribution to each digit gives a 1 or a 9. Each row component contains a value between one and zero and this describes the intensity of each pixel. "},{"metadata":{"_uuid":"2bb683c300bddb2ae2ca9915a2c7cdcfd0294872","_cell_guid":"be6a870b-3b43-7384-55f4-3ed8c5464c10"},"cell_type":"markdown","source":"**Pearson Correlation Plot**\n\nSince we are still having the problem that our dataset consists of a relatively large number of features (columns),  it is perfect time to introduce Dimensionality Reduction methods. Before we start off, let's conduct some cleaning of the train data by saving the label feature and then removing it from the dataframe"},{"metadata":{},"cell_type":"raw","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"# save the labels to a Pandas series target\ntarget = train['label']\n# Drop the label feature\ntrain = train.drop(\"label\",axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Xs = train.values.reshape(-1,28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(Xs[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Xs.max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Xs = Xs / 255\ntest= test / 255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = test.values.reshape(-1, 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(test[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(Xs, target, test_size=0.3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, MaxPool2D, Flatten","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.utils import to_categorical","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_cat_train = to_categorical(y_train, 10)\ny_cat_test = to_categorical(y_test, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(filters=32, kernel_size=(4,4), input_shape=(28,28,1), activation='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2)))\n\nmodel.add(Flatten())\n\nmodel.add(Dense(units=128, activation='relu'))\nmodel.add(Dense(units=10, activation='softmax'))\n\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"early_stop = EarlyStopping(monitor='val_loss', patience=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_cat_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(X_train, y_cat_train, epochs=5, callbacks=[early_stop])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3617c18ebb55b6d3b5e57bfb7b4d4231a28ef9e","_cell_guid":"e28f547a-7691-4664-ae63-ee1fefde9e76"},"cell_type":"markdown","source":"# 1. Principal Component Analysis (PCA)\n\nIn a nutshell, PCA is a linear transformation algorithm that seeks to project the original features of our data onto a smaller set of features ( or subspace ) while still retaining most of the information. To do this the algorithm tries to find the most appropriate directions/angles ( which are the principal components ) that maximise the variance in the new subspace. Why maximise the variance though? \n\nTo answer the question, more context has to be given about the PCA method. One has to understand that the principal components are orthogonal to each other ( think right angle ). As such when generating the covariance matrix ( measure of how related 2 variables are to each other ) in our new subspace, the off-diagonal values of the covariance matrix will be zero and only the diagonals ( or eigenvalues) will be non-zero. It is these diagonal values that represent the *variances* of the principal components that we are talking about or information about the variability of our features. \n\nTherefore when PCA seeks to maximise this variance, the method is trying to find directions ( principal components ) that contain the largest spread/subset of data points or information ( variance ) relative to all the data points present. For a brilliant and detailed description on this, check out this stackexchange thread: \n\n[PCA and proportion of variance explained][1] by amoeba\n\n  [1]: http://stats.stackexchange.com/a/140579/3277"},{"metadata":{"_uuid":"777890904ab0b2c4fe772191f25781c9719ceb48","_cell_guid":"ec9f7dff-e524-340b-fbba-09a47af726d4"},"cell_type":"markdown","source":"### Calculating the Eigenvectors\n\nNow it may be informative to observe how the variances look like for the digits in the MNIST dataset. Therefore to achieve this, let us calculate the eigenvectors and eigenvalues of the covarience matrix as follows:"},{"metadata":{"_uuid":"701c41fb0ddbc20fbf86f0fec5e778e95c284a77","_execution_state":"idle","_cell_guid":"62e6276a-dac3-8131-aeb3-1e37588c538b","trusted":true},"cell_type":"code","source":"# Standardize the data\nfrom sklearn.preprocessing import StandardScaler\nX = train.values\nX_std = StandardScaler().fit_transform(X)\n\n# Calculating Eigenvectors and eigenvalues of Cov matirx\nmean_vec = np.mean(X_std, axis=0)\ncov_mat = np.cov(X_std.T)\neig_vals, eig_vecs = np.linalg.eig(cov_mat)\n# Create a list of (eigenvalue, eigenvector) tuples\neig_pairs = [ (np.abs(eig_vals[i]),eig_vecs[:,i]) for i in range(len(eig_vals))]\n\n# Sort the eigenvalue, eigenvector pair from high to low\neig_pairs.sort(key = lambda x: x[0], reverse= True)\n\n# Calculation of Explained Variance from the eigenvalues\ntot = sum(eig_vals)\nvar_exp = [(i/tot)*100 for i in sorted(eig_vals, reverse=True)] # Individual explained variance\ncum_var_exp = np.cumsum(var_exp) # Cumulative explained variance","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fb55e246b170981440c4eab01ef98924213cf13","_cell_guid":"e78d6ebc-5a06-57ee-ca95-70976aa54fe6"},"cell_type":"markdown","source":"Now having calculated both our Individual Explained Variance and Cumulative Explained Variance values, let's use the Plotly visualisation package to produce an interactive chart to showcase this."},{"metadata":{"_uuid":"862a68c5415a187d5f36f44e886d90554fa2a0dd","_cell_guid":"4610eb60-0877-8e8d-8dc7-99fdef818dfb","trusted":true},"cell_type":"code","source":"trace1 = go.Scatter(\n    x=list(range(784)),\n    y= cum_var_exp,\n    mode='lines+markers',\n    name=\"'Cumulative Explained Variance'\",\n#     hoverinfo= cum_var_exp,\n    line=dict(\n        shape='spline',\n        color = 'goldenrod'\n    )\n)\ntrace2 = go.Scatter(\n    x=list(range(784)),\n    y= var_exp,\n    mode='lines+markers',\n    name=\"'Individual Explained Variance'\",\n#     hoverinfo= var_exp,\n    line=dict(\n        shape='linear',\n        color = 'black'\n    )\n)\nfig = tls.make_subplots(insets=[{'cell': (1,1), 'l': 0.7, 'b': 0.5}],\n                          print_grid=True)\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2,1,1)\nfig.layout.title = 'Explained Variance plots - Full and Zoomed-in'\nfig.layout.xaxis = dict(range=[0, 80], title = 'Feature columns')\nfig.layout.yaxis = dict(range=[0, 60], title = 'Explained Variance')\n# fig['data'] = []\n# fig['data'].append(go.Scatter(x= list(range(784)) , y=cum_var_exp, xaxis='x2', yaxis='y2', name = 'Cumulative Explained Variance'))\n# fig['data'].append(go.Scatter(x= list(range(784)) , y=cum_var_exp, xaxis='x2', yaxis='y2', name = 'Cumulative Explained Variance'))\n\n# fig['data'] = go.Scatter(x= list(range(784)) , y=cum_var_exp, xaxis='x2', yaxis='y2', name = 'Cumulative Explained Variance')]\n# fig['data'] += [go.Scatter(x=list(range(784)), y=var_exp, xaxis='x2', yaxis='y2',name = 'Individual Explained Variance')]\n\n# # fig['data'] = data\n# # fig['layout'] = layout\n# # fig['data'] += data2\n# # fig['layout'] += layout2\n# py.iplot(fig, filename='inset example')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec573d8f4cdec34b175175bd1b2c7046d37dedf3","_cell_guid":"31ecbb1a-c10e-696b-1367-00cca3af1ce2"},"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*"},{"metadata":{"_uuid":"044e8a973587d22ed711c6fd59d6ba3aa58b5d84","_cell_guid":"a5e0c94e-e947-e69e-6856-70a65b17226b"},"cell_type":"markdown","source":"**Takeaway from the Plot**\n\nThere are two plots above, a smaller one embedded within the larger plot. The smaller plot ( Green and Red) shows the distribution of the Individual and Explained variances across all features while the larger plot ( Golden and black ) portrays a zoomed section of the explained variances only.\n\nAs we can see, out of our 784 features or columns approximately 90% of the Explained Variance can be described by using just over 200 over features. So if one wanted to implement a PCA on this, extracting the top 200 features would be a very logical choice as they already account for the majority of the data.\nIn the section below, I will use the immensely powerful Sklearn toolkit and its built-in PCA method. Unfortunately for brevity I will not be covering how to implement PCA from scratch, partly due to the multitude of resources already available. One excellent article to check out for this would be:\n\n[Principal Component Analysis in 3 Simple Steps][1] by Sebastian Raschka \n\n  [1]: http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html"},{"metadata":{"_uuid":"a22ad858c0ed030f0d7ccb3628d7250107f06017","_cell_guid":"2b275af4-75f0-c8c9-6290-6915aca59f32"},"cell_type":"markdown","source":"**Visualizing the Eigenvalues**\n\nAs alluded to above, since the PCA method seeks to obtain the optimal directions (or eigenvectors) that captures the most variance ( spreads out the data points the most ). Therefore it may be informative ( and cool) to visualise these directions and their associated eigenvalues. For the purposes of this notebook and for speed, I will invoke PCA to only extract the top 30 eigenvalues ( using Sklearn's .components_ call) from the digit dataset and visually compare the top 5 eigenvalues to some of the other smaller ones to see if we can glean any insights as follows:"},{"metadata":{"_uuid":"0a0258f5ceb12a3bb479854b1eee7f49d77929c0","_execution_state":"idle","_cell_guid":"fbadd9b0-7ded-4123-5305-bc4efa1f442d","trusted":true},"cell_type":"code","source":"# Invoke SKlearn's PCA method\nn_components = 30\npca = PCA(n_components=n_components).fit(train.values)\n\neigenvalues = pca.components_.reshape(n_components, 28, 28)\n\n# Extracting the PCA components ( eignevalues )\n#eigenvalues = pca.components_.reshape(n_components, 28, 28)\neigenvalues = pca.components_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2009f34b080f277f960d33659d8e10d5e5e2b699","_execution_state":"idle","_cell_guid":"8a416e93-6c34-77c5-46df-0bacffa85d79","trusted":true},"cell_type":"code","source":"n_row = 4\nn_col = 7\n\n# Plot the first 8 eignenvalues\nplt.figure(figsize=(13,12))\nfor i in list(range(n_row * n_col)):\n    offset =0\n    plt.subplot(n_row, n_col, i + 1)\n    plt.imshow(eigenvalues[i].reshape(28,28), cmap='jet')\n    title_text = 'Eigenvalue ' + str(i + 1)\n    plt.title(title_text, size=6.5)\n    plt.xticks(())\n    plt.yticks(())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8540cc2b0fe3bf8cbb68c291a0dd5932c86083ad","_cell_guid":"b9e9a313-512f-9447-b2be-9d6ba24a1fab"},"cell_type":"markdown","source":"**Takeaway from the Plots**\n\nThe subplots above portray the top 30 optimal directions or principal component axes that the PCA method has decided to generate for our digit dataset. Of interest is when one compares the first component \"Eigenvalue 1\" to the 28th component \"Eigenvalue 28\", it is obvious that more complicated directions or components are being generated in the search to maximise variance in the new feature subspace."},{"metadata":{"_uuid":"2999204986538d328107e3bfc84daa74a46653e4","_cell_guid":"c99fd7fe-ac7f-3587-9d63-94b8d117ef0b"},"cell_type":"markdown","source":"**Visualising the MNIST Digit set on its own**\n\nNow just for the fun and curiosity of it, let's plot the actual MNIST digit set to see what the underlying dataset actually represents, rather than being caught up with just looking at 1 and 0's."},{"metadata":{"_uuid":"ea560c57efd0c57948d2ecbd2ac19eb4008a27b9","_cell_guid":"5782183e-6370-302e-fe43-d5641f6b0463","trusted":true},"cell_type":"code","source":"# plot some of the numbers\nplt.figure(figsize=(14,12))\nfor digit_num in range(0,70):\n    plt.subplot(7,10,digit_num+1)\n    grid_data = train.iloc[digit_num].values.reshape(28,28)  # reshape from 1d to 2d pixel array\n    plt.imshow(grid_data, interpolation = \"none\", cmap = \"afmhot\")\n    plt.xticks([])\n    plt.yticks([])\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"599261d28d78a8996fabfb025c69401ccaf4b270","_cell_guid":"08b6de1e-2527-9728-462d-aa523e6c6444"},"cell_type":"markdown","source":"Phew, they are definitely digits all right. So let's proceed onto the main event."},{"metadata":{"_uuid":"a78027bf4d6eef8067e1a8c27ed1181255fe35cf","_cell_guid":"e397d459-99cf-553c-e546-de4a40189274"},"cell_type":"markdown","source":"###PCA Implementation via Sklearn\n\nNow using the Sklearn toolkit, we implement the Principal Component Analysis algorithm as follows:"},{"metadata":{"_uuid":"710528b4eef44b1fa2475a0f6d688b55c7ab2a24","_cell_guid":"de0e84b4-a031-3d0d-cb13-e782345984bc","trusted":true},"cell_type":"code","source":"# Delete our earlier created X object\ndel X\n# Taking only the first N rows to speed things up\nX= train[:6000].values\ndel train\n# Standardising the values\nX_std = StandardScaler().fit_transform(X)\n\n# Call the PCA method with 5 components. \npca = PCA(n_components=5)\npca.fit(X_std)\nX_5d = pca.transform(X_std)\n\n# For cluster coloring in our Plotly plots, remember to also restrict the target values \nTarget = target[:6000]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e339b0bb6e5a0a959882a7fd42d672cf27731549","_cell_guid":"055bc33a-a723-ee81-0486-bf60b0009c7b"},"cell_type":"markdown","source":"What the chunk of code does above is to first normalise the data (actually no need to do so for this data set as they are all 1's and 0's) using Sklearn's convenient StandardScaler call.\n\nNext we invoke Sklearn's inbuilt PCA function by providing into its argument *n_components*, the number of components/dimensions we would like to project the data onto. In practise, one would educate and motivate the choice of components for example by looking at the proportion of variance captured vs each feature's eigenvalue, such as in our Explained Variance plots. To be honest, there are a multitude of papers in the literature with research on what should be a good indicator on choice of components. Here are some references for the interested: However for the essence of this notebook being a guide of sorts, I have just decided to take a PCA on 5 components ( against perhaps taking 200 over components).\n\nFinally I call both fit and transform methods which fits the PCA model with the standardised digit data set and then does a transformation by applying the dimensionality reduction on the data."},{"metadata":{"_uuid":"7606ae9da733df4fce38d641bb11b318c10dd85e","_cell_guid":"71ae5da5-57a1-51b3-0cfb-0b446d9513ce"},"cell_type":"markdown","source":"###Interactive visualisations of PCA representation\n\nWhen it comes to these dimensionality reduction methods, scatter plots are most commonly implemented because they allow for great and convenient visualisations of clustering ( if any existed ) and this will be exactly what we will be doing as we plot the first 2 principal components as follows:"},{"metadata":{"_uuid":"0d36b2e6443b3e1744eba21e2f4c0e399d7ece20","_cell_guid":"e3beda90-8af6-2491-9c30-ee4a6cbd171c","trusted":true},"cell_type":"code","source":"trace0 = go.Scatter(\n    x = X_5d[:,0],\n    y = X_5d[:,1],\n#     name = Target,\n#     hoveron = Target,\n    mode = 'markers',\n    text = Target,\n    showlegend = False,\n    marker = dict(\n        size = 8,\n        color = Target,\n        colorscale ='Jet',\n        showscale = False,\n        line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        ),\n        opacity = 0.8\n    )\n)\ndata = [trace0]\n\nlayout = go.Layout(\n    title= 'Principal Component Analysis (PCA)',\n    hovermode= 'closest',\n    xaxis= dict(\n         title= 'First Principal Component',\n        ticklen= 5,\n        zeroline= False,\n        gridwidth= 2,\n    ),\n    yaxis=dict(\n        title= 'Second Principal Component',\n        ticklen= 5,\n        gridwidth= 2,\n    ),\n    showlegend= True\n)\n\n\nfig = dict(data=data, layout=layout)\npy.iplot(fig, filename='styled-scatter')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccfcc18ad5a030322423a229e3f86a75e904244b","_cell_guid":"a00e1f9a-b6d9-2d33-f3aa-f0c1498bc3ab"},"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*"},{"metadata":{"_uuid":"21e744544e883f5a1f8d7f87bb349c22fb1bb86b","_cell_guid":"60b92aaf-2a4b-d6cf-c619-5950c1c24aa1"},"cell_type":"markdown","source":"**Takeaway from the Plot**\n\nAs observed from the scatter plot, you can just about make out a few discernible clusters evinced from the collective blotches of colors. These clusters represent the underlying digit that each data point should contribute to and one may therefore be tempted to think that it was quite a piece of cake in implementing and visualising PCA in this section.\n\nHowever, the devil lies in the tiny details of the python implementation because as alluded to earlier, PCA is actually in fact an unsupervised method which does not depend on class labels. I have sneakily snuck in class labelings whilst generating the scatter plots therefore resulting in the clusters of colours as you see them."},{"metadata":{"_uuid":"e4dcc3eac49185a8cb14f76286fe6920f577d249","_cell_guid":"dd4ed15c-3fb1-4dc0-b897-dbfb34227aec"},"cell_type":"markdown","source":"###K-Means Clustering to identify possible classes\n\nImagine just for a moment that we were not provided with the class labels to this digit set because after all PCA is an unsupervised method. Therefore how would we be able to separate out our data points in the new feature space? We can apply a clustering algorithm on our new PCA projection data and hopefully arrive at distinct clusters which would tell us something about the underlying class separation in the data. \n\nTo start off, we set up a KMeans clustering method with Sklearn's *KMeans* call and use the *fit_predict* method to compute cluster centers and predict cluster indices for the first and second PCA projections (to see if we can observe any appreciable clusters)."},{"metadata":{"_uuid":"8034c5a96dfa75f9f2b76f86289e2ac36887b372","_cell_guid":"f936510e-920f-ab32-952d-3ae28b86a2e3","trusted":true},"cell_type":"code","source":"from sklearn.cluster import KMeans # KMeans clustering \n# Set a KMeans clustering with 9 components ( 9 chosen sneakily ;) as hopefully we get back our 9 class labels)\nkmeans = KMeans(n_clusters=9)\n# Compute cluster centers and predict cluster indices\nX_clustered = kmeans.fit_predict(X_5d)\n\ntrace_Kmeans = go.Scatter(x=X_5d[:, 0], y= X_5d[:, 1], mode=\"markers\",\n                    showlegend=False,\n                    marker=dict(\n                            size=8,\n                            color = X_clustered,\n                            colorscale = 'Portland',\n                            showscale=False, \n                            line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        )\n                   ))\n\nlayout = go.Layout(\n    title= 'KMeans Clustering',\n    hovermode= 'closest',\n    xaxis= dict(\n         title= 'First Principal Component',\n        ticklen= 5,\n        zeroline= False,\n        gridwidth= 2,\n    ),\n    yaxis=dict(\n        title= 'Second Principal Component',\n        ticklen= 5,\n        gridwidth= 2,\n    ),\n    showlegend= True\n)\n\ndata = [trace_Kmeans]\nfig1 = dict(data=data, layout= layout)\n# fig1.append_trace(contour_list)\npy.iplot(fig1, filename=\"svm\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d56a803caa184b448251f2049b49a242f309d207","_cell_guid":"ba5dda9f-bb59-9d3b-bdee-05ca234f2dea"},"cell_type":"markdown","source":"**Takeaway from the Plot**\n\nVisually, the clusters generated by the KMeans algorithm appear to provide a clearer demarcation amongst clusters as compared to naively adding in class labels into our PCA projections. This should come as no surprise as PCA is meant to be an unsupervised method and therefore not optimised for separating different class labels. This particular task however is accomplished by the very next method that we will talk about."},{"metadata":{"_uuid":"a4b09ba3598d504c2af5d68d372c0683e1b2114b","_cell_guid":"d95e7384-75bf-4bd7-957a-bb5b976d2cda"},"cell_type":"markdown","source":"# 2. Linear Discriminant Analysis (LDA)\n\nLDA, much like PCA is also a linear transformation method commonly used in dimensionality reduction tasks. However unlike the latter which is an unsupervised learning algorithm, LDA falls into the class of supervised learning methods. As such the goal of LDA is that with available information about class labels, LDA will seek to maximise the separation between the different classes by computing the component axes (linear discriminants ) which does this."},{"metadata":{"_uuid":"edeb0f15442865f9afedd0dcaaa26d63239b2c9d","_cell_guid":"4a6f6d80-45e8-2ba6-ae30-2dc002a370da","trusted":true},"cell_type":"code","source":"from IPython.display import display, Math, Latex","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b6d1ee2d07016d8894c7df6e0b07f13ed2ef9ec","_cell_guid":"05107565-572f-8bc6-fb44-d90561ff4f7e"},"cell_type":"markdown","source":"### LDA Implementation from Scratch\n\nThe objective of LDA is to preserve the class separation information whilst still  reducing the dimensions of the dataset. As such implementing the method from scratch can roughly be split into 4 distinct stages as below. As an aside, since this section will be quite equation heavy therefore we will also be embedding some mathematical equations into the upcoming sections. The good thing about IPython notebook is that you can render your equations (LaTeX) automatically by putting them within the **$$** symbol, courtesy of the use of MathJax - a JavaScript equation display engine.\n\n**A. Projected Means**\n\nSince this method was designed to take into account class labels we therefore first need to establish a suitable metric with which to measure the 'distance' or separation between different classes. Let's assume that we have a set of data points *x* that belong to one particular class *w*. Therefore in LDA the first step is to the project these points onto a new line, Y that contains the class-specific information via the transformation  \n\n$$Y = \\omega^\\intercal x $$\n\nWith this the idea is to find some method that maximises the separation of these new projected variables. To do so, we first calculate the projected mean.\n\n**B.  Scatter Matrices and their solutions**\nHaving introduced our projected means, we now need to find a function that can represent the difference between the means and then maximise it. Like in linear regression, where the most basic case is to find the line of best fit we need to find the equivalent of the variance in this context. And hence this is where we introduce scatter matrices where the scatter is the equivalent of the variance.\n\n$$ \\tilde{S}^{2} = (y - \\tilde{mu})^{2}$$\n\n\n**C.  Selecting Optimal Projection Matrices**\n\n**D.  Transforming features onto new subspace**"},{"metadata":{"_uuid":"f5b2bda1a9e24b739332426c64c225a2ff903f6a","_cell_guid":"c4608299-d551-758a-0759-d7cd09bb7bed"},"cell_type":"markdown","source":"*SECTION STILL UNDER-WAY*"},{"metadata":{"_uuid":"d2cd51cad2d47f771c07350fde71d893040a50fb","_cell_guid":"58b10a24-b7ae-f62d-d879-62283b2e70e9"},"cell_type":"markdown","source":"###LDA Implementation via Sklearn\n\nHaving gone through the nitty-gritty details of the LDA implementation in theory, let us now implement the method in practise. Surprise, surprise we find that the Sklearn toolkit also comes with its own inbuilt LDA function and hence we invoke an LDA model as follows:"},{"metadata":{"_uuid":"1257fba566db293d32a9c108827f538406efd657","_cell_guid":"0007390c-f795-3c20-c100-56bb8d8a98b6","trusted":true},"cell_type":"code","source":"lda = LDA(n_components=5)\n# Taking in as second argument the Target as labels\nX_LDA_2D = lda.fit_transform(X_std, Target.values )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5c641d7989fbf1ad7d193f664134f61b72dc7772","_cell_guid":"8bea4e96-efb5-5229-de62-aaebeaa3a3e4"},"cell_type":"markdown","source":"The syntax for the LDA implementation is very much akin to that of PCA whereby one calls the fit and transform methods which fits the LDA model with the data and then does a transformation by applying the LDA dimensionality reduction to it. However since LDA is a supervised learning algorithm , there is a second argument to the method that the user must provide and this would be the class labels, which in this case is the target labels of the digits."},{"metadata":{"_uuid":"9da97173a301b123a681607caeca5b52edb8dd0b","_cell_guid":"e2937679-f1ac-70d8-0d8c-c1b406a00607"},"cell_type":"markdown","source":"###Interactive visualisations of LDA representation"},{"metadata":{"_uuid":"97be08521701f2981a8d84c8b402b18ff816989b","_cell_guid":"27306c3e-5e63-a2fb-e448-4a7e2bdd46c3","trusted":true},"cell_type":"code","source":"8# Using the Plotly library again\ntraceLDA = go.Scatter(\n    x = X_LDA_2D[:,0],\n    y = X_LDA_2D[:,1],\n#     name = Target,\n#     hoveron = Target,\n    mode = 'markers',\n    text = Target,\n    showlegend = True,\n    marker = dict(\n        size = 8,\n        color = Target,\n        colorscale ='Jet',\n        showscale = False,\n        line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        ),\n        opacity = 0.8\n    )\n)\ndata = [traceLDA]\n\nlayout = go.Layout(\n    title= 'Linear Discriminant Analysis (LDA)',\n    hovermode= 'closest',\n    xaxis= dict(\n         title= 'First Linear Discriminant',\n        ticklen= 5,\n        zeroline= False,\n        gridwidth= 2,\n    ),\n    yaxis=dict(\n        title= 'Second Linear Discriminant',\n        ticklen= 5,\n        gridwidth= 2,\n    ),\n    showlegend= False\n)\n\nfig = dict(data=data, layout=layout)\npy.iplot(fig, filename='styled-scatter')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72bda03b8436a71318007a1d6633c173a3eeab55","_cell_guid":"c584b009-4da0-435d-2c07-ba7c6a68f0c4"},"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*"},{"metadata":{"_uuid":"4118f7aaeea3aa1c38c63b1364224f08f6d025d8","_cell_guid":"586ee60c-6fc0-bd09-7538-76e89a8b6112"},"cell_type":"markdown","source":"From the scatter plot above, we can see that the data points are more clearly clustered when using LDA with as compared to implementing PCA with class labels. This is an inherent advantage in having class labels to supervise the method with. In short picking the right tool for the right job."},{"metadata":{"_uuid":"4ce19a7210f72ca995209cb7cb308a8eb38c5940","_cell_guid":"2b82a157-9ed2-434d-f0d2-bc4ce2d7dd35"},"cell_type":"markdown","source":"#3. T-SNE ( t-Distributed Stochastic Neighbour Embedding )\n\nThe t-SNE method has become widely popular ever since it was introduced by van der Maaten and Hinton in 2008. Unlike the previous two linear methods of PCA and LDA discussed above, t-SNE is a non-linear, probabilistic dimensionality reduction method.\n\nThe internal mechanisms of the algorithm \nTherefore instead of looking at directions/axes which maximise information or class separation, T-SNE aims to convert the Euclidean distances between points into conditional probabilities. A Student-t distribution is then applied on these probabilities which serve as metrics to calculate the similarity between one datapoint to another. \n\nHowever this brief summary does no justice in any manner or shape to the original t-SNE paper by Maaten and Hinton so please do check the original out [here][1].\n\n  [1]: http://www.cs.toronto.edu/~hinton/absps/tsne.pdf"},{"metadata":{"_uuid":"a0ec94827504c34633463548a25e043dc5df0cca","_cell_guid":"b5014801-7844-c15a-ba52-0b56db349bfc","trusted":true},"cell_type":"code","source":"# Invoking the t-SNE method\ntsne = TSNE(n_components=2)\ntsne_results = tsne.fit_transform(X_std) \nprint(tsne_results)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80b3d48e65b0ca81a5348151890ae91a95db4570","_cell_guid":"b5a3b5ce-1f73-38d8-c796-85781af080ba"},"cell_type":"markdown","source":"Having invoked the t-SNE algorithm by simply calling *TSNE()* we fit the digit data to the model and reduce its dimensions with *fit_transform*. Finally let's plot the first two components in the new feature space in a scatter plot"},{"metadata":{"_uuid":"9e20d7ce20113cbee610e974c8c16590ec8e7a1c","_cell_guid":"a7ac8bb6-f011-1a9f-df75-6c701d6a4d72","trusted":true},"cell_type":"code","source":"traceTSNE = go.Scatter(\n    x = tsne_results[:,0],\n    y = tsne_results[:,1],\n    name = Target,\n     hoveron = Target,\n    mode = 'markers',\n    text = Target,\n    showlegend = True,\n    marker = dict(\n        size = 8,\n        color = Target,\n        colorscale ='Jet',\n        showscale = False,\n        line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        ),\n        opacity = 0.8\n    )\n)\ndata = [traceTSNE]\n\nlayout = dict(title = 'TSNE (T-Distributed Stochastic Neighbour Embedding)',\n              hovermode= 'closest',\n              yaxis = dict(zeroline = False),\n              xaxis = dict(zeroline = False),\n              showlegend= False,\n\n             )\n\nfig = dict(data=data, layout=layout)\npy.iplot(fig, filename='styled-scatter')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_classes = model.predict_classes(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_classes.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import accuracy_score, classification_report","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy_score(y_test, predict_classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(classification_report(y_test, predict_classes))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_classes.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission=pd.read_csv('testlabels.csv')\nsubmission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df_predict = model.predict_classes(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df_predict.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_predict = pd.Series(test_df_predict, name='Label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1, 28001), name='ImageId'), test_predict], axis=1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('testlabels.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final=pd.read_csv('testlabels.csv')\n\nprint(final)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_download_link(title=\"Download CSV file\",filename=\"testlabels.csv\"):\n    html='<a href={filename}>{title}</a>'\n    html=html.format(title=title,filename=filename)\n    return HTML(html)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\ncreate_download_link(filename='testlabels.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee5d09d5e1e441bdca9f649d828650b58f7e31ba","_cell_guid":"1fb052f0-b629-c64f-6fc9-599d03b6e427"},"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*"},{"metadata":{"_uuid":"4de036df06c419a2bcb00f08182b87f7fabfda11","_cell_guid":"60ec5748-d1cc-2ace-8ef3-3e52a6e4647b"},"cell_type":"markdown","source":"**Takeaway from the Plots**\n\nFrom the t-SNE scatter plot the first thing that strikes is that clusters ( and even subclusters ) are very well defined and segregated resulting in Jackson-Pollock like Modern Art visuals, even more so than the PCA and LDA methods. This ability to provide very good cluster visualisations can be boiled down to the topology-preserving attributes of the algorithm. \n\nHowever t-SNE is not without its drawbacks. Multiple local minima may occur as the algorithm is identifying clusters/sub-clusters and this can be evinced from the scatter plot, where we can see that clusters of the same colour exist as 2 sub-clusters in different areas of the plot."},{"metadata":{"_uuid":"3b3d1bd37e10e27a7cbdd2a9aedb7cd68b5b6436","_cell_guid":"8a4ff611-c401-9919-8faf-27d5b311a5db"},"cell_type":"markdown","source":"# Concluding Remarks\n\nIn conclusion, this notebook has introduced and briefly covered three different dimensionality reduction methods commonly used by ML practitioners - PCA, LDA and t-SNE. We touched on the concepts of finding principal components and linear discriminants as well as the topology preserving capabilities of t-SNE. We've also discussed the relative merits of using supervised and unsupervised methods as well as the KMeans clustering technique when it comes to an unsupervised scenario.  \n\nApart from these three common reduction methods, there exists a whole host of other dimensionality reduction methods not discussed in this notebook. Just to name a few, they include methods like Sammon's Mapping, Multi-dimensional Scaling or even some graph based visualisation methods.  \n\nI hope this notebook has been useful especially with regards to introducing the concepts of dimensionality reduction if it is new to you. And the key takeaway would be that each method has its own pros and cons and are not to be used as a one-size-fits-all/I-will-only-use-my-favourite method but rather to be implemented as the situation calls for it. \n\nPeace out "}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"_change_revision":0,"_is_fork":false},"nbformat":4,"nbformat_minor":4}