{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n  **Principal Component Analysis ( PCA )\n","metadata":{"_uuid":"2df33e5389533544ca69f36129ffba91f1adb6e5","_cell_guid":"ce7d6fa0-5c61-0d94-9941-9eef9ea68077"}},{"cell_type":"code","source":"# import all the libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport seaborn as sns\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\nimport matplotlib\n%matplotlib inline\n\n# Import the 3 dimensionality reduction methods\nfrom sklearn.manifold import TSNE\nfrom sklearn.decomposition import PCA\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA","metadata":{"_uuid":"4bee068051d8d31245eb44efcec101faa7519b36","_execution_state":"idle","_cell_guid":"bbbfbfb7-4cfd-7c9f-523a-1c4d474e7919","execution":{"iopub.status.busy":"2022-04-18T02:08:46.559847Z","iopub.execute_input":"2022-04-18T02:08:46.561015Z","iopub.status.idle":"2022-04-18T02:08:46.579791Z","shell.execute_reply.started":"2022-04-18T02:08:46.560593Z","shell.execute_reply":"2022-04-18T02:08:46.578197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Curse of Dimensionality & Dimensionality Reduction**\n-how our perfectly good and reliable Machine Learning methods may suddenly perform badly when we are dealing in a very high-dimensional space.\nDimensionality reduction refers to techniques for reducing the number of input variables in training data","metadata":{"_uuid":"30a04ee76d6633d67d30446dd84c2fd3160ec1c1","_cell_guid":"f0ef3f94-be7c-8108-279e-41f8c5a28185"}},{"cell_type":"markdown","source":"# MNIST Dataset\n\nWe hav chosen the MNIST (Mixed National Institute of Standards and Technology) computer vision digit dataset was chosen partly due to its simplicityand informative research that can be done with the dataset.","metadata":{"_uuid":"0c95e9c20988e56008516e7a38c9ed4da033314b","_cell_guid":"8da70ed8-177c-a295-6069-7352a3ff22d6"}},{"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntrain.head()","metadata":{"_uuid":"813379999b963b572b5b9631e36296dcdaee483a","_execution_state":"idle","_cell_guid":"2781f1bf-44b8-bb42-fa9e-c4f5233d4ed5","execution":{"iopub.status.busy":"2022-04-18T02:08:46.653286Z","iopub.execute_input":"2022-04-18T02:08:46.653639Z","iopub.status.idle":"2022-04-18T02:08:51.743607Z","shell.execute_reply.started":"2022-04-18T02:08:46.65358Z","shell.execute_reply":"2022-04-18T02:08:51.742284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train.shape)","metadata":{"_uuid":"5b0c47777a5f3114b4962d5fcdb2032f2f503a87","_execution_state":"idle","_cell_guid":"d3ff48ed-51af-d315-df10-64024922a932","execution":{"iopub.status.busy":"2022-04-18T02:08:51.745558Z","iopub.execute_input":"2022-04-18T02:08:51.745858Z","iopub.status.idle":"2022-04-18T02:08:51.750448Z","shell.execute_reply.started":"2022-04-18T02:08:51.745811Z","shell.execute_reply":"2022-04-18T02:08:51.749617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The MNIST set consists of 42,000 rows and 785 columns. \nT 28 x 28 pixel images of digits - contributing to 784 columns - one extra label column which is essentially a class label to state whether the row-wise contribution to each digit gives a 1 or a 9. Each row component contains a value between one and zero and this describes the intensity of each pixel. ","metadata":{"_uuid":"8a0fdbc22dc99b05f10d7839cfc4a1a428a5e5dc","_cell_guid":"6b142c78-a513-0a7c-a8fd-83c55b81a74b"}},{"cell_type":"markdown","source":"**Pearson Correlation Plot**\n\nSince we are still having the problem that our dataset consists of a relatively large number of features (columns),  it is perfect time to introduce Dimensionality Reduction methods. Before we start off, let's conduct some cleaning of the train data by saving the label feature and then removing it from the dataframe","metadata":{"_uuid":"2bb683c300bddb2ae2ca9915a2c7cdcfd0294872","_cell_guid":"be6a870b-3b43-7384-55f4-3ed8c5464c10"}},{"cell_type":"code","source":"# save the labels to a Pandas series target\ntarget = train['label']\n# Drop the label feature\ntrain = train.drop(\"label\",axis=1)","metadata":{"_uuid":"8e92dba75c5d2886b7343de960907659513ee60e","_execution_state":"idle","_cell_guid":"94b43e7c-fcca-c363-d10a-9407c1dda252","execution":{"iopub.status.busy":"2022-04-18T02:08:51.752539Z","iopub.execute_input":"2022-04-18T02:08:51.752825Z","iopub.status.idle":"2022-04-18T02:08:52.211259Z","shell.execute_reply.started":"2022-04-18T02:08:51.752771Z","shell.execute_reply":"2022-04-18T02:08:52.21043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Principal Component Analysis (PCA)\n\nPCA is a linear transformation algorithm that seeks to project the original features of our data onto a smaller set of features while still retaining most of the information. To do this the algorithm tries to find the most appropriate principal components/angles that maximise the variance in the new subspace. \n\nWe need to maximise the variance as more context has to be given about the PCA method. \n-One important point is that,principal components are orthogonal to each other. \n-when generating the covariance matrix(measure of how related 2 variables are to each other ) in our new subspace, \n    -the off-diagonal values of the covariance matrix will be zero here.\n    -only the diagonals ( or eigenvalues) will be non-zero. \n    -It is these diagonal values that represent the *variances* of the principal components that we are talking about or information about the variability of our features. ","metadata":{"_uuid":"b3617c18ebb55b6d3b5e57bfb7b4d4231a28ef9e","_cell_guid":"e28f547a-7691-4664-ae63-ee1fefde9e76"}},{"cell_type":"markdown","source":"### Calculating the Eigenvectors\n\nNow - to observe how the variances look like for the digits in the MNIST dataset.let us calculate the eigenvectors and eigenvalues of the covarience matrix as follows:","metadata":{"_uuid":"777890904ab0b2c4fe772191f25781c9719ceb48","_cell_guid":"ec9f7dff-e524-340b-fbba-09a47af726d4"}},{"cell_type":"code","source":"# Standardize the data\nfrom sklearn.preprocessing import StandardScaler#sclearn-scikit-classification,regression,clustering,dimensionality reduction.\nX = train.values\nX_std = StandardScaler().fit_transform(X)\n\n# Calculating Eigenvectors and eigenvalues of Covariance matirx\nmean_vec = np.mean(X_std, axis=0)#find the mean\ncov_mat = np.cov(X_std.T)#find the covariance matrix\neig_vals, eig_vecs = np.linalg.eig(cov_mat)#we can find the eigen vector\n\n# Create a list of (eigenvalue, eigenvector) tuples\neig_pairs = [ (np.abs(eig_vals[i]),eig_vecs[:,i]) for i in range(len(eig_vals))]\n\n# Sort the eigenvalue, eigenvector pair from high to low\neig_pairs.sort(key = lambda x: x[0], reverse= True)\n\n# Calculation of Explained Variance from the eigenvalues\ntot = sum(eig_vals)\nvar_exp = [(i/tot)*100 for i in sorted(eig_vals, reverse=True)] # Individual explained variance\ncum_var_exp = np.cumsum(var_exp) # Cumulative explained variance\n#shows the discrepancy between a model and actual data.","metadata":{"_uuid":"701c41fb0ddbc20fbf86f0fec5e778e95c284a77","_execution_state":"idle","_cell_guid":"62e6276a-dac3-8131-aeb3-1e37588c538b","execution":{"iopub.status.busy":"2022-04-18T02:08:52.212909Z","iopub.execute_input":"2022-04-18T02:08:52.213222Z","iopub.status.idle":"2022-04-18T02:08:55.45353Z","shell.execute_reply.started":"2022-04-18T02:08:52.213165Z","shell.execute_reply":"2022-04-18T02:08:55.452352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now - we have - Individual Explained Variance and Cumulative Explained Variance values, using the Plotly visualisation package-  produce an interactive chart to showcase this.","metadata":{"_uuid":"6fb55e246b170981440c4eab01ef98924213cf13","_cell_guid":"e78d6ebc-5a06-57ee-ca95-70976aa54fe6"}},{"cell_type":"raw","source":"#### format wih the the insets is given below\ntrace1 = go.Scatter(\n    x=list(range(784)),\n    y= cum_var_exp,\n    mode='lines+markers',\n    name=\"'Cumulative Explained Variance'\",\n\n    line=dict(\n        shape='spline',\n        color = 'goldenrod'\n    )\n)\ntrace2 = go.Scatter(\n    x=list(range(784)),\n    y= var_exp,\n    mode='lines+markers',\n    name=\"'Individual Explained Variance'\",\n\n    line=dict(\n        shape='linear',\n        color = 'black'\n    )\n)\nfig = tls.make_subplots(insets=[{'cell': (1,1), 'l': 0.7, 'b': 0.5}],\n                          print_grid=True)\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2,1,1)\nfig.layout.title = 'Explained Variance plots - Full and Zoomed-in'\nfig.layout.xaxis = dict(range=[0, 80], title = 'Feature columns')\nfig.layout.yaxis = dict(range=[0, 60], title = 'Explained Variance')\n","metadata":{"_uuid":"862a68c5415a187d5f36f44e886d90554fa2a0dd","_cell_guid":"4610eb60-0877-8e8d-8dc7-99fdef818dfb","execution":{"iopub.status.busy":"2022-04-18T02:08:55.455115Z","iopub.execute_input":"2022-04-18T02:08:55.455694Z","iopub.status.idle":"2022-04-18T02:08:55.599317Z","shell.execute_reply.started":"2022-04-18T02:08:55.455626Z","shell.execute_reply":"2022-04-18T02:08:55.598201Z"}}},{"cell_type":"markdown","source":"","metadata":{"_uuid":"ec573d8f4cdec34b175175bd1b2c7046d37dedf3","_cell_guid":"31ecbb1a-c10e-696b-1367-00cca3af1ce2"}},{"cell_type":"markdown","source":"","metadata":{"_uuid":"044e8a973587d22ed711c6fd59d6ba3aa58b5d84","_cell_guid":"a5e0c94e-e947-e69e-6856-70a65b17226b"}},{"cell_type":"markdown","source":"**Visualizing the Eigenvalues**\n\nPCA method seeks to obtain the optimal directions (or eigenvectors) that captures the most variance ( spreads out the data points the most ). Will invoke PCA to only extract the top 30 eigenvalues ( using Sklearn's .components_ call) from the digit dataset and visually compare the top 5 eigenvalues to some of the other smaller ones to see if we can get any insights as follows:","metadata":{"_uuid":"a22ad858c0ed030f0d7ccb3628d7250107f06017","_cell_guid":"2b275af4-75f0-c8c9-6290-6915aca59f32"}},{"cell_type":"code","source":"# Invoke SKlearn's PCA method\nn_components = 30\npca = PCA(n_components=n_components).fit(train.values)\n\neigenvalues = pca.components_.reshape(n_components, 28, 28)\n\n# Extracting the PCA components ( eignevalues )\n#eigenvalues = pca.components_.reshape(n_components, 28, 28)\neigenvalues = pca.components_","metadata":{"_uuid":"0a0258f5ceb12a3bb479854b1eee7f49d77929c0","_execution_state":"idle","_cell_guid":"fbadd9b0-7ded-4123-5305-bc4efa1f442d","execution":{"iopub.status.busy":"2022-04-18T02:08:55.600872Z","iopub.execute_input":"2022-04-18T02:08:55.60129Z","iopub.status.idle":"2022-04-18T02:08:59.616039Z","shell.execute_reply.started":"2022-04-18T02:08:55.6012Z","shell.execute_reply":"2022-04-18T02:08:59.615174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_row = 4\nn_col = 7\n\n# Plot the first 8 eignenvalues\nplt.figure(figsize=(13,12))\nfor i in list(range(n_row * n_col)):\n    offset =0\n    plt.subplot(n_row, n_col, i + 1)\n    plt.imshow(eigenvalues[i].reshape(28,28), cmap='jet')\n    title_text = 'Eigenvalue ' + str(i + 1)\n    plt.title(title_text, size=6.5)\n    plt.xticks(())\n    plt.yticks(())\nplt.show()","metadata":{"_uuid":"2009f34b080f277f960d33659d8e10d5e5e2b699","_execution_state":"idle","_cell_guid":"8a416e93-6c34-77c5-46df-0bacffa85d79","execution":{"iopub.status.busy":"2022-04-18T02:08:59.617089Z","iopub.execute_input":"2022-04-18T02:08:59.617468Z","iopub.status.idle":"2022-04-18T02:09:00.766157Z","shell.execute_reply.started":"2022-04-18T02:08:59.617426Z","shell.execute_reply":"2022-04-18T02:09:00.76513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Takeaway from the Plots**\n\nThe subplots above portray the top 30 optimal directions that the PCA method has decided to generate for our digit dataset.  Intersetingly is when one compares the first component \"Eigenvalue 1\" to the 28th component \"Eigenvalue 28\", it is obvious that more complicated directions or components are being generated in the search to maximise variance in the new feature subspace.","metadata":{"_uuid":"8540cc2b0fe3bf8cbb68c291a0dd5932c86083ad","_cell_guid":"b9e9a313-512f-9447-b2be-9d6ba24a1fab"}},{"cell_type":"markdown","source":"**Visualising the MNIST Digit set on its own**","metadata":{"_uuid":"2999204986538d328107e3bfc84daa74a46653e4","_cell_guid":"c99fd7fe-ac7f-3587-9d63-94b8d117ef0b"}},{"cell_type":"code","source":"# plot some of the numbers\nplt.figure(figsize=(14,12))\nfor digit_num in range(0,70):\n    plt.subplot(7,10,digit_num+1)\n    grid_data = train.iloc[digit_num].as_matrix().reshape(28,28)  # reshape from 1d to 2d pixel array\n    plt.imshow(grid_data, interpolation = \"none\", cmap = \"afmhot\")\n    plt.xticks([])\n    plt.yticks([])\nplt.tight_layout()","metadata":{"_uuid":"ea560c57efd0c57948d2ecbd2ac19eb4008a27b9","_cell_guid":"5782183e-6370-302e-fe43-d5641f6b0463","execution":{"iopub.status.busy":"2022-04-18T02:09:00.767798Z","iopub.execute_input":"2022-04-18T02:09:00.768408Z","iopub.status.idle":"2022-04-18T02:09:04.128966Z","shell.execute_reply.started":"2022-04-18T02:09:00.768342Z","shell.execute_reply":"2022-04-18T02:09:04.128203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{"_uuid":"599261d28d78a8996fabfb025c69401ccaf4b270","_cell_guid":"08b6de1e-2527-9728-462d-aa523e6c6444"}},{"cell_type":"markdown","source":"###PCA Implementation via Sklearn\n\nNow using the Sklearn toolkit, we implement the Principal Component Analysis algorithm as follows:","metadata":{"_uuid":"a78027bf4d6eef8067e1a8c27ed1181255fe35cf","_cell_guid":"e397d459-99cf-553c-e546-de4a40189274"}},{"cell_type":"code","source":"# Delete our earlier created X object\ndel X\n# Taking only the first N rows to speed things up\nX= train[:6000].values\ndel train\n# Standardising the values\nX_std = StandardScaler().fit_transform(X)\n\n# Call the PCA method with 5 components. \npca = PCA(n_components=5)\npca.fit(X_std)\nX_5d = pca.transform(X_std)\n\n# For cluster coloring in our Plotly plots, remember to also restrict the target values \nTarget = target[:6000]","metadata":{"_uuid":"710528b4eef44b1fa2475a0f6d688b55c7ab2a24","_cell_guid":"de0e84b4-a031-3d0d-cb13-e782345984bc","execution":{"iopub.status.busy":"2022-04-18T02:09:04.129985Z","iopub.execute_input":"2022-04-18T02:09:04.130356Z","iopub.status.idle":"2022-04-18T02:09:04.750224Z","shell.execute_reply.started":"2022-04-18T02:09:04.130314Z","shell.execute_reply":"2022-04-18T02:09:04.748983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What the chunk of code does above is to first normalise the data (actually no need to do so for this data set as they are all 1's and 0's) using Sklearn's convenient StandardScaler call.\n\nNext we invoke Sklearn's inbuilt PCA function by providing into its argument *n_components*, the number of components/dimensions we would like to project the data onto. In practise, one would educate and motivate the choice of components for example by looking at the proportion of variance captured vs each feature's eigenvalue, such as in our Explained Variance plots. To be honest, there are a multitude of papers in the literature with research on what should be a good indicator on choice of components. Here are some references for the interested: However for the essence of this notebook being a guide of sorts, I have just decided to take a PCA on 5 components ( against perhaps taking 200 over components).\n\nFinally I call both fit and transform methods which fits the PCA model with the standardised digit data set and then does a transformation by applying the dimensionality reduction on the data.","metadata":{"_uuid":"e339b0bb6e5a0a959882a7fd42d672cf27731549","_cell_guid":"055bc33a-a723-ee81-0486-bf60b0009c7b"}},{"cell_type":"markdown","source":"###Interactive visualisations of PCA representation\n\nWhen it comes to these dimensionality reduction methods, scatter plots are most commonly implemented because they allow for great and convenient visualisations of clustering ( if any existed ) and this will be exactly what we will be doing as we plot the first 2 principal components as follows:","metadata":{"_uuid":"7606ae9da733df4fce38d641bb11b318c10dd85e","_cell_guid":"71ae5da5-57a1-51b3-0cfb-0b446d9513ce"}},{"cell_type":"code","source":"trace0 = go.Scatter(\n    x = X_5d[:,0],\n    y = X_5d[:,1],\n#     name = Target,\n#     hoveron = Target,\n    mode = 'markers',\n    text = Target,\n    showlegend = False,\n    marker = dict(\n        size = 8,\n        color = Target,\n        colorscale ='Jet',\n        showscale = False,\n        line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        ),\n        opacity = 0.8\n    )\n)\ndata = [trace0]\n\nlayout = go.Layout(\n    title= 'Principal Component Analysis (PCA)',\n    hovermode= 'closest',\n    xaxis= dict(\n         title= 'First Principal Component',\n        ticklen= 5,\n        zeroline= False,\n        gridwidth= 2,\n    ),\n    yaxis=dict(\n        title= 'Second Principal Component',\n        ticklen= 5,\n        gridwidth= 2,\n    ),\n    showlegend= True\n)\n\n\nfig = dict(data=data, layout=layout)\npy.iplot(fig, filename='styled-scatter')","metadata":{"_uuid":"0d36b2e6443b3e1744eba21e2f4c0e399d7ece20","_cell_guid":"e3beda90-8af6-2491-9c30-ee4a6cbd171c","execution":{"iopub.status.busy":"2022-04-18T02:09:04.752113Z","iopub.execute_input":"2022-04-18T02:09:04.75419Z","iopub.status.idle":"2022-04-18T02:09:05.092875Z","shell.execute_reply.started":"2022-04-18T02:09:04.754115Z","shell.execute_reply":"2022-04-18T02:09:05.092169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*","metadata":{"_uuid":"ccfcc18ad5a030322423a229e3f86a75e904244b","_cell_guid":"a00e1f9a-b6d9-2d33-f3aa-f0c1498bc3ab"}},{"cell_type":"markdown","source":"**Takeaway from the Plot**\n\nAs observed from the scatter plot, you can just about make out a few discernible clusters evinced from the collective blotches of colors. These clusters represent the underlying digit that each data point should contribute to and one may therefore be tempted to think that it was quite a piece of cake in implementing and visualising PCA in this section.\n\nHowever, the devil lies in the tiny details of the python implementation because as alluded to earlier, PCA is actually in fact an unsupervised method which does not depend on class labels. I have sneakily snuck in class labelings whilst generating the scatter plots therefore resulting in the clusters of colours as you see them.","metadata":{"_uuid":"21e744544e883f5a1f8d7f87bb349c22fb1bb86b","_cell_guid":"60b92aaf-2a4b-d6cf-c619-5950c1c24aa1"}},{"cell_type":"markdown","source":"###K-Means Clustering to identify possible classes\n\nImagine just for a moment that we were not provided with the class labels to this digit set because after all PCA is an unsupervised method. Therefore how would we be able to separate out our data points in the new feature space? We can apply a clustering algorithm on our new PCA projection data and hopefully arrive at distinct clusters which would tell us something about the underlying class separation in the data. \n\nTo start off, we set up a KMeans clustering method with Sklearn's *KMeans* call and use the *fit_predict* method to compute cluster centers and predict cluster indices for the first and second PCA projections (to see if we can observe any appreciable clusters).","metadata":{"_uuid":"e4dcc3eac49185a8cb14f76286fe6920f577d249","_cell_guid":"dd4ed15c-3fb1-4dc0-b897-dbfb34227aec"}},{"cell_type":"code","source":"from sklearn.cluster import KMeans # KMeans clustering \n# Set a KMeans clustering with 9 components ( 9 chosen sneakily ;) as hopefully we get back our 9 class labels)\nkmeans = KMeans(n_clusters=9)\n# Compute cluster centers and predict cluster indices\nX_clustered = kmeans.fit_predict(X_5d)\n\ntrace_Kmeans = go.Scatter(x=X_5d[:, 0], y= X_5d[:, 1], mode=\"markers\",\n                    showlegend=False,\n                    marker=dict(\n                            size=8,\n                            color = X_clustered,\n                            colorscale = 'Portland',\n                            showscale=False, \n                            line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        )\n                   ))\n\nlayout = go.Layout(\n    title= 'KMeans Clustering',\n    hovermode= 'closest',\n    xaxis= dict(\n         title= 'First Principal Component',\n        ticklen= 5,\n        zeroline= False,\n        gridwidth= 2,\n    ),\n    yaxis=dict(\n        title= 'Second Principal Component',\n        ticklen= 5,\n        gridwidth= 2,\n    ),\n    showlegend= True\n)\n\ndata = [trace_Kmeans]\nfig1 = dict(data=data, layout= layout)\n# fig1.append_trace(contour_list)\npy.iplot(fig1, filename=\"svm\")","metadata":{"_uuid":"8034c5a96dfa75f9f2b76f86289e2ac36887b372","_cell_guid":"f936510e-920f-ab32-952d-3ae28b86a2e3","execution":{"iopub.status.busy":"2022-04-18T02:09:05.093895Z","iopub.execute_input":"2022-04-18T02:09:05.094374Z","iopub.status.idle":"2022-04-18T02:09:06.260425Z","shell.execute_reply.started":"2022-04-18T02:09:05.094324Z","shell.execute_reply":"2022-04-18T02:09:06.259723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Takeaway from the Plot**\n\nVisually, the clusters generated by the KMeans algorithm appear to provide a clearer demarcation amongst clusters as compared to naively adding in class labels into our PCA projections. This should come as no surprise as PCA is meant to be an unsupervised method and therefore not optimised for separating different class labels. This particular task however is accomplished by the very next method that we will talk about.","metadata":{"_uuid":"d56a803caa184b448251f2049b49a242f309d207","_cell_guid":"ba5dda9f-bb59-9d3b-bdee-05ca234f2dea"}},{"cell_type":"markdown","source":"# 2. Linear Discriminant Analysis (LDA)\n\nLDA, much like PCA is also a linear transformation method commonly used in dimensionality reduction tasks. However unlike the latter which is an unsupervised learning algorithm, LDA falls into the class of supervised learning methods. As such the goal of LDA is that with available information about class labels, LDA will seek to maximise the separation between the different classes by computing the component axes (linear discriminants ) which does this.","metadata":{"_uuid":"a4b09ba3598d504c2af5d68d372c0683e1b2114b","_cell_guid":"d95e7384-75bf-4bd7-957a-bb5b976d2cda"}},{"cell_type":"code","source":"from IPython.display import display, Math, Latex","metadata":{"_uuid":"edeb0f15442865f9afedd0dcaaa26d63239b2c9d","_cell_guid":"4a6f6d80-45e8-2ba6-ae30-2dc002a370da","execution":{"iopub.status.busy":"2022-04-18T02:09:06.261451Z","iopub.execute_input":"2022-04-18T02:09:06.261835Z","iopub.status.idle":"2022-04-18T02:09:06.265643Z","shell.execute_reply.started":"2022-04-18T02:09:06.26177Z","shell.execute_reply":"2022-04-18T02:09:06.265033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### LDA Implementation from Scratch\n\nThe objective of LDA is to preserve the class separation information whilst still  reducing the dimensions of the dataset. As such implementing the method from scratch can roughly be split into 4 distinct stages as below. As an aside, since this section will be quite equation heavy therefore we will also be embedding some mathematical equations into the upcoming sections. The good thing about IPython notebook is that you can render your equations (LaTeX) automatically by putting them within the **$$** symbol, courtesy of the use of MathJax - a JavaScript equation display engine.\n\n**A. Projected Means**\n\nSince this method was designed to take into account class labels we therefore first need to establish a suitable metric with which to measure the 'distance' or separation between different classes. Let's assume that we have a set of data points *x* that belong to one particular class *w*. Therefore in LDA the first step is to the project these points onto a new line, Y that contains the class-specific information via the transformation  \n\n$$Y = \\omega^\\intercal x $$\n\nWith this the idea is to find some method that maximises the separation of these new projected variables. To do so, we first calculate the projected mean.\n\n**B.  Scatter Matrices and their solutions**\nHaving introduced our projected means, we now need to find a function that can represent the difference between the means and then maximise it. Like in linear regression, where the most basic case is to find the line of best fit we need to find the equivalent of the variance in this context. And hence this is where we introduce scatter matrices where the scatter is the equivalent of the variance.\n\n$$ \\tilde{S}^{2} = (y - \\tilde{mu})^{2}$$\n\n\n**C.  Selecting Optimal Projection Matrices**\n\n**D.  Transforming features onto new subspace**","metadata":{"_uuid":"5b6d1ee2d07016d8894c7df6e0b07f13ed2ef9ec","_cell_guid":"05107565-572f-8bc6-fb44-d90561ff4f7e"}},{"cell_type":"markdown","source":"*SECTION STILL UNDER-WAY*","metadata":{"_uuid":"f5b2bda1a9e24b739332426c64c225a2ff903f6a","_cell_guid":"c4608299-d551-758a-0759-d7cd09bb7bed"}},{"cell_type":"markdown","source":"###LDA Implementation via Sklearn\n\nHaving gone through the nitty-gritty details of the LDA implementation in theory, let us now implement the method in practise. Surprise, surprise we find that the Sklearn toolkit also comes with its own inbuilt LDA function and hence we invoke an LDA model as follows:","metadata":{"_uuid":"d2cd51cad2d47f771c07350fde71d893040a50fb","_cell_guid":"58b10a24-b7ae-f62d-d879-62283b2e70e9"}},{"cell_type":"code","source":"lda = LDA(n_components=5)\n# Taking in as second argument the Target as labels\nX_LDA_2D = lda.fit_transform(X_std, Target.values )","metadata":{"_uuid":"1257fba566db293d32a9c108827f538406efd657","_cell_guid":"0007390c-f795-3c20-c100-56bb8d8a98b6","execution":{"iopub.status.busy":"2022-04-18T02:09:06.266695Z","iopub.execute_input":"2022-04-18T02:09:06.267161Z","iopub.status.idle":"2022-04-18T02:09:08.014229Z","shell.execute_reply.started":"2022-04-18T02:09:06.267106Z","shell.execute_reply":"2022-04-18T02:09:08.013133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The syntax for the LDA implementation is very much akin to that of PCA whereby one calls the fit and transform methods which fits the LDA model with the data and then does a transformation by applying the LDA dimensionality reduction to it. However since LDA is a supervised learning algorithm , there is a second argument to the method that the user must provide and this would be the class labels, which in this case is the target labels of the digits.","metadata":{"_uuid":"5c641d7989fbf1ad7d193f664134f61b72dc7772","_cell_guid":"8bea4e96-efb5-5229-de62-aaebeaa3a3e4"}},{"cell_type":"markdown","source":"###Interactive visualisations of LDA representation","metadata":{"_uuid":"9da97173a301b123a681607caeca5b52edb8dd0b","_cell_guid":"e2937679-f1ac-70d8-0d8c-c1b406a00607"}},{"cell_type":"code","source":"8# Using the Plotly library again\ntraceLDA = go.Scatter(\n    x = X_LDA_2D[:,0],\n    y = X_LDA_2D[:,1],\n#     name = Target,\n#     hoveron = Target,\n    mode = 'markers',\n    text = Target,\n    showlegend = True,\n    marker = dict(\n        size = 8,\n        color = Target,\n        colorscale ='Jet',\n        showscale = False,\n        line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        ),\n        opacity = 0.8\n    )\n)\ndata = [traceLDA]\n\nlayout = go.Layout(\n    title= 'Linear Discriminant Analysis (LDA)',\n    hovermode= 'closest',\n    xaxis= dict(\n         title= 'First Linear Discriminant',\n        ticklen= 5,\n        zeroline= False,\n        gridwidth= 2,\n    ),\n    yaxis=dict(\n        title= 'Second Linear Discriminant',\n        ticklen= 5,\n        gridwidth= 2,\n    ),\n    showlegend= False\n)\n\nfig = dict(data=data, layout=layout)\npy.iplot(fig, filename='styled-scatter')","metadata":{"_uuid":"97be08521701f2981a8d84c8b402b18ff816989b","_cell_guid":"27306c3e-5e63-a2fb-e448-4a7e2bdd46c3","execution":{"iopub.status.busy":"2022-04-18T02:09:08.01619Z","iopub.execute_input":"2022-04-18T02:09:08.016939Z","iopub.status.idle":"2022-04-18T02:09:08.378299Z","shell.execute_reply.started":"2022-04-18T02:09:08.016869Z","shell.execute_reply":"2022-04-18T02:09:08.377563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*","metadata":{"_uuid":"72bda03b8436a71318007a1d6633c173a3eeab55","_cell_guid":"c584b009-4da0-435d-2c07-ba7c6a68f0c4"}},{"cell_type":"markdown","source":"From the scatter plot above, we can see that the data points are more clearly clustered when using LDA with as compared to implementing PCA with class labels. This is an inherent advantage in having class labels to supervise the method with. In short picking the right tool for the right job.","metadata":{"_uuid":"4118f7aaeea3aa1c38c63b1364224f08f6d025d8","_cell_guid":"586ee60c-6fc0-bd09-7538-76e89a8b6112"}},{"cell_type":"markdown","source":"#3. T-SNE ( t-Distributed Stochastic Neighbour Embedding )\n\nThe t-SNE method has become widely popular ever since it was introduced by van der Maaten and Hinton in 2008. Unlike the previous two linear methods of PCA and LDA discussed above, t-SNE is a non-linear, probabilistic dimensionality reduction method.\n\nThe internal mechanisms of the algorithm \nTherefore instead of looking at directions/axes which maximise information or class separation, T-SNE aims to convert the Euclidean distances between points into conditional probabilities. A Student-t distribution is then applied on these probabilities which serve as metrics to calculate the similarity between one datapoint to another. \n\nHowever this brief summary does no justice in any manner or shape to the original t-SNE paper by Maaten and Hinton so please do check the original out [here][1].\n\n  [1]: http://www.cs.toronto.edu/~hinton/absps/tsne.pdf","metadata":{"_uuid":"4ce19a7210f72ca995209cb7cb308a8eb38c5940","_cell_guid":"2b82a157-9ed2-434d-f0d2-bc4ce2d7dd35"}},{"cell_type":"code","source":"# Invoking the t-SNE method\ntsne = TSNE(n_components=2)\ntsne_results = tsne.fit_transform(X_std) ","metadata":{"_uuid":"a0ec94827504c34633463548a25e043dc5df0cca","_cell_guid":"b5014801-7844-c15a-ba52-0b56db349bfc","execution":{"iopub.status.busy":"2022-04-18T02:09:08.379411Z","iopub.execute_input":"2022-04-18T02:09:08.37986Z","iopub.status.idle":"2022-04-18T02:11:17.482105Z","shell.execute_reply.started":"2022-04-18T02:09:08.379807Z","shell.execute_reply":"2022-04-18T02:11:17.481249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Having invoked the t-SNE algorithm by simply calling *TSNE()* we fit the digit data to the model and reduce its dimensions with *fit_transform*. Finally let's plot the first two components in the new feature space in a scatter plot","metadata":{"_uuid":"80b3d48e65b0ca81a5348151890ae91a95db4570","_cell_guid":"b5a3b5ce-1f73-38d8-c796-85781af080ba"}},{"cell_type":"code","source":"traceTSNE = go.Scatter(\n    x = tsne_results[:,0],\n    y = tsne_results[:,1],\n    name = Target,\n     hoveron = Target,\n    mode = 'markers',\n    text = Target,\n    showlegend = True,\n    marker = dict(\n        size = 8,\n        color = Target,\n        colorscale ='Jet',\n        showscale = False,\n        line = dict(\n            width = 2,\n            color = 'rgb(255, 255, 255)'\n        ),\n        opacity = 0.8\n    )\n)\ndata = [traceTSNE]\n\nlayout = dict(title = 'TSNE (T-Distributed Stochastic Neighbour Embedding)',\n              hovermode= 'closest',\n              yaxis = dict(zeroline = False),\n              xaxis = dict(zeroline = False),\n              showlegend= False,\n\n             )\n\nfig = dict(data=data, layout=layout)\npy.iplot(fig, filename='styled-scatter')","metadata":{"_uuid":"9e20d7ce20113cbee610e974c8c16590ec8e7a1c","_cell_guid":"a7ac8bb6-f011-1a9f-df75-6c701d6a4d72","execution":{"iopub.status.busy":"2022-04-18T02:11:17.483784Z","iopub.execute_input":"2022-04-18T02:11:17.484064Z","iopub.status.idle":"2022-04-18T02:11:17.533343Z","shell.execute_reply.started":"2022-04-18T02:11:17.484011Z","shell.execute_reply":"2022-04-18T02:11:17.532002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*PLEASE CLICK AND MOVE THE SCATTER PLOTS ABOVE. THEY ARE INTERACTIVE. DOUBLE CLICK TO GET BACK TO THE ORIGINAL VIEW*","metadata":{"_uuid":"ee5d09d5e1e441bdca9f649d828650b58f7e31ba","_cell_guid":"1fb052f0-b629-c64f-6fc9-599d03b6e427"}},{"cell_type":"markdown","source":"**Takeaway from the Plots**\n\nFrom the t-SNE scatter plot the first thing that strikes is that clusters ( and even subclusters ) are very well defined and segregated resulting in Jackson-Pollock like Modern Art visuals, even more so than the PCA and LDA methods. This ability to provide very good cluster visualisations can be boiled down to the topology-preserving attributes of the algorithm. \n\nHowever t-SNE is not without its drawbacks. Multiple local minima may occur as the algorithm is identifying clusters/sub-clusters and this can be evinced from the scatter plot, where we can see that clusters of the same colour exist as 2 sub-clusters in different areas of the plot.","metadata":{"_uuid":"4de036df06c419a2bcb00f08182b87f7fabfda11","_cell_guid":"60ec5748-d1cc-2ace-8ef3-3e52a6e4647b"}},{"cell_type":"markdown","source":"# Concluding Remarks\n\nIn conclusion, this notebook has introduced and briefly covered three different dimensionality reduction methods commonly used by ML practitioners - PCA, LDA and t-SNE. We touched on the concepts of finding principal components and linear discriminants as well as the topology preserving capabilities of t-SNE. We've also discussed the relative merits of using supervised and unsupervised methods as well as the KMeans clustering technique when it comes to an unsupervised scenario.  \n\nApart from these three common reduction methods, there exists a whole host of other dimensionality reduction methods not discussed in this notebook. Just to name a few, they include methods like Sammon's Mapping, Multi-dimensional Scaling or even some graph based visualisation methods.  \n\nI hope this notebook has been useful especially with regards to introducing the concepts of dimensionality reduction if it is new to you. And the key takeaway would be that each method has its own pros and cons and are not to be used as a one-size-fits-all/I-will-only-use-my-favourite method but rather to be implemented as the situation calls for it. \n\nPeace out ","metadata":{"_uuid":"3b3d1bd37e10e27a7cbdd2a9aedb7cd68b5b6436","_cell_guid":"8a4ff611-c401-9919-8faf-27d5b311a5db"}}]}