{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n**Dimension Reduction:**\nThere are many ways to achieve dimensionality reduction, but most of these techniques fall into one of two classes \n* Feature Elimination\n* Feature Extraction\n\n**Principal Component Analysis (PCA):** \nPCA is a dimension reduction technique extensionally used for visualization of high dimensional data. It is a feature extraction technique — it combines all input variable in a specific way, drop the least important variables while retaining the most valuable ones. In this method we calculate eigenvectors and eigenvalues of the covariance matrix. Once eigenvectors are found from the covariance matrix, the nextstep is to order them by eigenvalue, highest to lowest. This gives you the components in order of significance.The eigenvector with the highest eigenvalue is the principal component of the data set.\n\n**t-Distributed Stochastic Neighbor Embedding (t-SNE):**\nt-SNE is a non-linear, unsupervised technique primarily used for data exploration and visualization of high-dimensional data. It provides you an intuition of how the data is arranged in a high-dimensional space. The t-SNE algorithm calculates a similarity measure between points in high dimensional space using Gaussian distribution then in the low dimensional space using Cauchy distribution. Finally it measures the probability distribution of the two dimensional spaces by Kullback-Liebler divergence and optimize the KL cost function using gradient descent.\n\n**PCA vs t-SNE:**\nt-SNE differs from PCA by preserving only small pairwise distances or local similarities whereas PCA is concerned with preserving large pairwise distances to maximize variance. PCA is a linear dimension reduction technique that seeks to maximize variance and preserves large pairwise distances. In other words, things that are different end up far apart. This can lead to poor visualization especially when dealing with non-linear manifold structures. \n\n> *In this notebook we will see the comparision between PCA and t-SNE on MNIST digits dataset*\n\n\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn import manifold\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn import decomposition","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Import data as Input\n\n* MNIST digits dataset is used for the purpose which consist of 784 pixel values and 42000 entries in train data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data = pd.read_csv('../input/digit-recognizer/train.csv')\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_labels = data.label\ndf_data = data.drop('label', axis = 1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Count plot for the labels \nsns.countplot(df_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Standardizing the data into a 2D array of shape 10000x784 as 10000 images have been extracted of shape 28x28 pixels. Flattening 28x28 pixels give 784 data points.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#extracting top 10000 data points \ndf_data = df_data.head(10000)\ndf_labels = df_labels.head(10000)\npixel_df = StandardScaler().fit_transform(df_data)\npixel_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_data = pixel_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Implementation of PCA**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = decomposition.PCA(n_components = 2, random_state = 42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_data = pca.fit_transform(sample_data)\nprint(\"shape of pca_reduced.shape = \", pca_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# attaching the label for each 2-d data point \npca_data = np.column_stack((pca_data, df_labels))\n\n# creating a new data frame for plotting of data points\npca_df = pd.DataFrame(data=pca_data, columns=(\"X\", \"Y\", \"labels\"))\nprint(pca_df.head(10))\nsns.FacetGrid(pca_df, hue=\"labels\", size=6).map(plt.scatter, 'X', 'Y').add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Implementation of t-SNE**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne = manifold.TSNE(n_components = 2, random_state = 42, verbose = 2, n_iter = 2000)\ntransformed_data = tsne.fit_transform(sample_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"#Creation of new dataframe for plotting of data points\ntsne_df = pd.DataFrame(\n    np.column_stack((transformed_data, df_labels)),\n    columns = ['x', 'y', 'labels'])\ntsne_df.loc[:, 'labels']= tsne_df.labels.astype(int)\nprint(tsne_df.head(10))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grid = sns.FacetGrid(tsne_df, hue='labels', size = 8)\ngrid.map(plt.scatter, 'x', 'y').add_legend()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}