{"cells":[{"metadata":{},"cell_type":"markdown","source":"
Getting started with Dimensionality Reduction Techniques in Python
\n\n
A 3 part serieson Dimensionality reduction techniques using the Kannada MNIST dataset
\n
\n\n\n

Key Objectives: In this series of notebooks, we shall study about three Dimensionality reduction techniques using the Kannada MNIST dataset. The techniques are PCA, t-SNE and UMAP.

\n\n\n* [Part 1: Visualizing Kannada MNIST with PCA](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-pca)\n* [Part 2: Visualizing Kannada MNIST with t-SNE](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-t-sne)\n* [Part 3: Visualizing Kannada MNIST with UMAP](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-umap-technique)\n
\n\n
What is Dimensionality Reduction
\n
\n\nA lot of Machine Learning problems consists of hundreds to thousands of features. having such a large number of features poses certain problems mainly :\n\n>* Slows down the training process\n>* It becomes hard to find a good solution\n\nThis problem is also sometimes termed as **The Curse of Dimensionality** and **Dimensionality Reduction** or **Dimension reduction** is the process of reducing the number of random variables under consideration by obtaining a set of principal variables[1]. \n\nIn other words, the goal is to take something that is very high dimensional and get it down to something that is easier to work with, without losing much of the information.\n\n### Importance of Dimensionality Reduction :\n\n* Getting down to two or three features can help us visualize our data which is an important part of data analysis\n* Often a lot of dimensionality in the data is redundant and we can get rid of that that can be sueful for the machine learning process.\n* Reducing the dimensionality can also help us in visualising the data easily.\n\nFor instance, the famous MNIST dataset is 784 dimensional when we unfold those digits into long vectors and we shouldn't really need 784 dimensions to describe a datapoint in this dataset. There should be some compact representation of this dataset and we should still be able to get some meaninful result.\n
\n\n
Main Approaches for Dimensionality Reduction
\n
\n\n \nThere are two main approaches to reducing dimensionality: **Projection** and **Manifold Learning**.\n\n* **Projection** : This technique deals with projecting every data point which is in high dimension, onto a subspace suitable lower-dimensional space in a way which approximately preserves the distances between the points[[2](https://en.wikipedia.org/wiki/Random_projection)]. For instance the figure below, the points in 3D are projected onto a 2D plane. This is a lower-dimensional (2D) subspace of the high-dimensional (3D) space and the axes correspond to new features z1 and z2 (the coordinates of the projections on the plane).\n\n![](https://i0.wp.com/www.analyticsvidhya.com/wp-content/uploads/2015/07/Image-4.png?w=556&ssl=1)\n\nSource : [Beginners Guide To Learn Dimension Reduction Techniques(https://www.analyticsvidhya.com/blog/2015/07/dimension-reduction-methods/)\n\nKeep in mind that projection may not always be the best method to achieve dimensional reduction.\n\n- **Manifold Learning** : [Manifold learning](https://scikit-learn.org/stable/modules/manifold.html) is an approach to non-linear dimensionality reduction. Algorithms for this task are based on the idea that the dimensionality of many data sets is only artificially high. \n\nJake VanderPlas explains Manifold Learning in a very intuitive way in his book :Python Data Science handbook and here is an excerpt from the book itself;\n\n*\"manifold learning—a class of unsupervised estimators that seeks to describe datasets as low-dimensional manifolds embedded in high-dimensional spaces. When you think of a manifold, I'd suggest imagining a sheet of paper: this is a two-dimensional object that lives in our familiar three-dimensional world, and can be bent or rolled in that two dimensions. In the parlance of manifold learning, we can think of this sheet as a two-dimensional manifold embedded in three-dimensional space.*\n\n*Rotating, re-orienting, or stretching the piece of paper in three-dimensional space doesn't change the flat geometry of the paper: such operations are akin to linear embeddings. If you bend, curl, or crumple the paper, it is still a two-dimensional manifold, but the embedding into the three-dimensional space is no longer linear. Manifold learning algorithms would seek to learn about the fundamental two-dimensional nature of the paper, even as it is contorted to fill the three-dimensional space.\"*\n
\n\n
Part1: Principal Component Analysis(PCA) in Python
\n
\nPCA is a very common technique for dimensionality reduction. The idea behind it is very simple:\n* Identify a Hyperplane that lies closest to the data\n* Project the data onto the hyperplane.\n\n![](https://i.stack.imgur.com/Q7HIP.gif)\n\n[Projecting 2D-data to a line (PCA](https://i.stack.imgur.com/Q7HIP.gif)\n\nHowever, it is important to choose the right hyperplane so that when the data is projected onto it, it the maximum amount of variation or information about how the original data is distributed. In other words, the axis that minimizes the mean squared distance between the original dataset and its projection onto that axis.\n\n\n### Principal Components\n\nThe axis that explains the maximum amount of variance int he training set is called the principal components. The axis othogonal to this axis is called the second principal component. Thus in 2D, there will be 2 principal components. However, for a higher dimensions, PCA would find a third component orthogonal to the other two components and so on.\n\n![](https://hackernoon.com/hn-images/1*WRKdN-NYF0mMumhfOXVa2Q.png)\n\nSource : [A Layman’s Introduction to Principal Components](https://hackernoon.com/a-laymans-introduction-to-principal-components-2fca55c19fa0)\n\n## Implementing PCA using Scikit Learn\n\n[Scikit-Learn’s](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) PCA class implements PCA using SVD decomposition. Let's apply PCA on Kannada MNIST data set for visualization.\n\n## 1. PCA for Visualisation\n\n\nAn effective way to visualize high-dimensional data is to represent each data object by a two-dimensional point in such a way that similar objects are represented by nearby points, and that dissimilar objects are represented by distant points. The resulting two-dimensional points can be visualized in a scatter plot. This leads to a map of the data that reveals the underlying structure of the objects, such as the presence of clusters.Let's see how we can use PCA to do that."},{"metadata":{},"cell_type":"markdown","source":"### Loading the necessary Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\n# For plotting\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\nsns.set(style='white', context='notebook', rc={'figure.figsize':(14,10)})\n\n#For standardising the dat\nfrom sklearn.preprocessing import StandardScaler\n\n#PCA\nfrom sklearn.decomposition import PCA","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading and Reading the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n\n\ntrain = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('../input/Kannada-MNIST/test.csv')\ntrain.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Setting the label and the feature columns\ny = train.loc[:,'label'].values\nx = train.loc[:,'pixel0':].values\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Standardizing the data\nstandardized_data = StandardScaler().fit_transform(x)\nprint(standardized_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Importing and Apply PCA\n\npca = PCA(n_components=2) # project from 784 to 2 dimensions\nprincipalComponents = pca.fit_transform(x)\nprincipal_df = pd.DataFrame(data = principalComponents\n , columns = ['principal component 1', 'principal component 2'])\nprincipal_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Explaining the Variance ratio\nprint('Explained variation per principal component: {}'.format(pca.explained_variance_ratio_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the first two principal components of each point to learn about the data:\n\nplt.scatter(principalComponents[:, 0], principalComponents[:, 1], s= 5, c=y, cmap='Spectral')\nplt.gca().set_aspect('equal', 'datalim')\nplt.colorbar(boundaries=np.arange(11)-0.5).set_ticks(np.arange(10))\nplt.title('Visualizing Kannada MNIST through PCA', fontsize=24);\nplt.xlabel('Principal Component 1')\nplt.ylabel('Principal Component 2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The full data is a 784-dimensional point cloud, and these colored points are the projection of each data point along the directions with the largest variance. \n\nThis representation makes it possible for us to see the layout of the digits in two dimensions. Another important thing to note here is that we have achieved this in an unsupervised manner—that is, without reference to the labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"## Using 3 Principal Components\n\npca = PCA(n_components=3) # project from 784 to 2 dimensions\nprincipalComponents = pca.fit_transform(x)\nprincipal_df = pd.DataFrame(data = principalComponents\n , columns = ['principal component 1', 'principal component 2', 'principal component 3'])\nprincipal_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Explained variation per principal component: {}'.format(pca.explained_variance_ratio_))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above graph shows that eventhough the components were able to hold some information, however, it isn't able to clearly set apart different digits. Fortunately, there are other dimensionality reduction techniques which we will see in future kernels that do a better job than PCA."},{"metadata":{},"cell_type":"markdown","source":"## Choosing the right number of components\n\nThe thumb rule is to choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%). However, if your goal is to visualise the dataset then make sure not to choose more than 2 or 3 dimensions, since the goal then would be to reduce the dimensions."},{"metadata":{"trusted":true},"cell_type":"code","source":"\npca = PCA(n_components=0.95)\nX_reduced = pca.fit_transform(x)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You could also plot the explained variance as a function of the number of dimensions. There will an elbow in the curve, where the explained variance stops growing fast. "},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = PCA().fit(x)\nplt.plot(np.cumsum(pca.explained_variance_ratio_))\nplt.xlabel('number of components')\nplt.ylabel('cumulative explained variance');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. PCA for for Noise Filtering\n\nPCA can also be used to filter unwanted noise from the data. The idea behind this concept is that components which have variance higher than that of noise should remain reasonably affected by noise. Thus if we reconstruct the data with only the largest group of principal components, we can easily filter out the noise from our data"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Plotting the original train data without noise\n\ndef plot_digits(data):\n fig, axes = plt.subplots(4, 10, figsize=(10, 4),\n subplot_kw={'xticks':[], 'yticks':[]},\n gridspec_kw=dict(hspace=0.1, wspace=0.1))\n for i, ax in enumerate(axes.flat):\n ax.imshow(data[i].reshape(28, 28),\n cmap='binary', interpolation='nearest',\n clim=(0, 16))\nplot_digits(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Introducing some random noise\n\nnp.random.seed(42)\nnoisy = np.random.normal(x, 10)\nplot_digits(noisy)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Training a PCA on noisy data and letting the projection preserve 50% of the variance:\n\npca = PCA(0.50).fit(noisy)\npca.n_components_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here 50% of the variance amounts to 24 principal components. Now we compute these components, and then use the inverse of the transform to reconstruct the filtered digits:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#using the inverse of the transform to reconstruct the filtered digits:\n\ncomponents = pca.transform(noisy)\nfiltered = pca.inverse_transform(components)\nplot_digits(filtered)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Drawbacks of PCA"},{"metadata":{},"cell_type":"markdown","source":"The main drawback of PCA is that it is highly influenced by outliers in the data. There are a couple of variants in Scikit-learn like RandomizedPCA and SParsePCA. Have a look at documentation to know more.\n\n\n## Unearthing Bias using PCA\nSometimes models learn things that are a cause for concern especially when machine learning models are used to make decisions for humans. PCA and other dimensionality reduction techniques can really help us to understand the relationship between points in large datasets. Here is how I used PCA to visualise the Word2Vec model and searched for the word **Engineer** to see its nearest neighbours. I also fixed an axis that goes from **man to woman** so words close to man lie towards left while words similar to woman will be found on the right. Let’s look at the results for our anchor word which is Engineer, given the above axis.\n\n![](https://miro.medium.com/max/856/1*swOOiho9ontVxPurXKncsw.png)\n\nIt appears that the word engineer is already closer to man than woman. The words closer to man are in orange and include astronomer, physicist, mathematician while words like dancer, songwriter, teacher appear closer to woman.\nHow about changing the anchor word to math? Are the results affected? \n\n![](https://miro.medium.com/max/1190/1*rsPf7BqmM2lbHY4lqynIAQ.png)\n\nWe have words like computational, geometry, arithmetic next to man, while the nearest neighbours to woman are music, teaching, philosophy etc.\nImagine if a machine learning algorithm trained on this dataset is used to predict how good someone is at their job related to art or math? Also, what will happen if a company relies on such an algorithm to hire potential engineers? The model might mistakenly believe that gender affected how good a candidate they were and the resulting decisions will be gender biased.\n\n**Note :** : The tool used for the above projections is called [Embedding Projector](http://projector.tensorflow.org/) : a web application tool that interactively visualizes embeddings by reading them from our model and rendering them in two or three dimensions. \n\n## References\n\n* Python Data Scinece Handbook by Jake VanderPlas\n* Hands on Machine Learning with Scikit-Learn, Keras & Tensorflow by Aurelien Geron\n\n### Additional Reading : \n * Colah's blog on the topic is an excellent resource . : https://colah.github.io/posts/2014-10-Visualizing-MNIST/ \n * If you want to get into the mathematics see this great page : https://www.math.hmc.edu/calculus/tutorials/eigenstuff/\n * Visualizing Bias in Data using Embedding Projector : https://towardsdatascience.com/visualizing-bias-in-data-using-embedding-projector-649bc65e7487"},{"metadata":{},"cell_type":"markdown","source":"\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}