{"cells":[{"metadata":{},"cell_type":"markdown","source":"
Getting started with Dimensionality Reduction Techniques in Python
\n\n
A 3 part serieson Dimensionality reduction techniques using the Kannada MNIST dataset
\n
\n\n\n

Key Objectives: In this series of notebooks, we shall study about three Dimensionality reduction techniques using the Kannada MNIST dataset. The techniques are PCA, t-SNE and UMAP.

\n\n\n* [Part 1: Visualizing Kannada MNIST with PCA](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-pca)\n* [Part 2: Visualizing Kannada MNIST with t-SNE](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-t-sne)\n* [Part 3: Visualizing Kannada MNIST with UMAP](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-umap-technique)\n
\n\n\n**This is the last and the final part of the **Dmensionality Reduction Techniques** notebooks that I have been doing. \nIn this motebook, we will explore a new technique called **UMAP** which is a pretty new technique(2018) in the DImensionality Reduction arsenal. We shall also understand how UMAP is a better alternative to t-SNE.\n\n
Part3: Uniform Manifold Approximation and Projection(UMAP) in Python
\n
\n\n## Table of Contents\n\n* What is UMAP\n* UMAP vs t-SNE\n* UMAP under the hood\n* UMAP with Python\n* Further Readings\n\n\n## UMAP\n\n**[UMAP]([https://umap-learn.readthedocs.io/en/latest/basic_usage.html](https://umap-learn.readthedocs.io/en/latest/basic_usage.html))** or **Uniform Manifold Approximation and Projection** is a general purpose manifold learning and dimension reduction algorithm. UMAP is a nonlinear dimensionality reduction method and is very effective for visualizing clusters or groups of data points and their relative proximities.it is a pretty fast and scalable and can be applied directly to sparse matrices thereby eliminating the need to use `Truncated SVD` as a prior pre-processing step. Leland McInnes, one of the creators of UMAP says that UMAP is an attempt to create something like t-SNE but with stronger mathematical foundations. \n\n### Paper\nI won't go into the mathematical foundations behind UMAP but for a more detailed explanation of the algorithm the paper can be found [here](https://arxiv.org/abs/1802.03426).\n\n### UMAP vs t-SNE\nLet's see some of the major points as to why UMAP tends to be the algorithm of choice\n\n* UMAP is faster than t-SNE, a problem we encountered while using t-SNE in the previous notebook where we had to sample the data for this very reason.\n* At capturing the global structure of the data, UMAP does a better job than t-SNE.\n* UMAP can also be used for preprocessing while t-SNE does't have major use outside visualisation.\n* \n## UMAP under the hood\n\nThis method uses the concept of k-nearest neighbor and optimizes the results using stochastic gradient descent. The algorithm first calculates the distance between the points in high dimensional space and then projects them onto the low dimensional space. It then calculates the distance between these points again in this low dimensional space followed by using Stochastic Gradient Descent to minimize the difference between these distances.\n\n## UMAP with Python"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading necessary libraries\n\nimport numpy as np\nimport pandas as pd\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# For plotting\nimport matplotlib.pyplot as plt\nimport matplotlib.patheffects as PathEffects\nimport seaborn as sns\n%matplotlib inline\nsns.set(style='white', context='notebook', rc={'figure.figsize':(14,10)})\n\n#For standardising the dat\nfrom sklearn.preprocessing import StandardScaler\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading in the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('../input/Kannada-MNIST/test.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Setting the label and the feature columns\ny = train.loc[:,'label'].values\nx = train.loc[:,'pixel0':].values\n\nprint(x.shape)\nprint(y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Subsetting a data for faster execution\n\nx_subset = x[0:10000]\ny_subset = y[0:10000]\n\nprint(np.unique(y_subset))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To use UMAP for this task we need to first construct a UMAP object that will do the job for us. That is as simple as instantiating the class. So let’s import the umap library and do that. Next we shall use the fit_transform method which first calls fit and then returns the transformed data as a numpy array."},{"metadata":{"trusted":true},"cell_type":"code","source":"import umap\nreducer = umap.UMAP(random_state=42)\nembedding = reducer.fit_transform(x_subset)\nembedding.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Basic UMAP Parameters:\nUMAP has several hyperparameters that can have a significant impact on the resulting embedding. ILet's have a brief look at some of them:\n\n- `n_neighbors`\n\nThis parameter controls how UMAP balances local versus global structure in the data. This means low values of n_neighbours forces UMAP to focus on very local structures while the higher values will make UMAP focus on the larger neighbourhoods. Look at the comparison below for various values of n_neighbours.\n\n![enter image description here](https://imgur.com/KLTzQpy.png)\n- `min_dist`\n\nThe `min_dist` parameter controls how tightly UMAP is allowed to pack points together.Lower values mean the points will be clustered closely and vice versa.\n\n- `n_components`\n\nThis parameter allows the user to determine the dimensionality of the reduced dimension space.\n\n- `metric`\nThis parameter essentially controls how distance is computed in the ambient space of the input data.\n"},{"metadata":{},"cell_type":"markdown","source":"## Visualizing UMAP as a scatterplot\n\nwe can plot the embedding as a standard scatterplot and color by the target array ("},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(reducer.embedding_[:, 0], reducer.embedding_[:, 1], s= 5, c=y_subset, cmap='Spectral')\nplt.gca().set_aspect('equal', 'datalim')\nplt.colorbar(boundaries=np.arange(11)-0.5).set_ticks(np.arange(10))\nplt.title('Visualizing Kannada MNIST with UMAP', fontsize=24);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"UMAP has successfully captured the digit classes. Some digit classes blend into one another but aren't so obvious to a human eye. To get a better idea of why UMAP chose to do this it is helpful to see the actual digits involve. One can do this using bokeh and mouseover tooltips of the images."},{"metadata":{},"cell_type":"markdown","source":"## Using Bokeh to visualize UMAP\n\nTo get a better idea of why UMAP chose to do this it is helpful to see the actual digits involve. One can do this using bokeh and mouseover tooltips of the images.\n\nSource of code : https://umap-learn.readthedocs.io/en/latest/basic_usage.html"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Encoding all the images for inclusion in a dataframe.\nfrom io import BytesIO\nfrom PIL import Image\nimport base64\n\n\ndef embeddable_image(data):\n img_data = 255 - 15 * data.astype(np.uint8)\n image = Image.fromarray(img_data, mode='L').resize((28,28), Image.BICUBIC)\n buffer = BytesIO()\n image.save(buffer, format='png')\n for_encoding = buffer.getvalue()\n return 'data:image/png;base64,' + base64.b64encode(for_encoding).decode()\n\n# loading up bokeh and other tools to generate a suitable interactive plot.\n\nfrom bokeh.plotting import figure, show, output_notebook\nfrom bokeh.models import HoverTool, ColumnDataSource, CategoricalColorMapper\nfrom bokeh.palettes import Spectral10\n\noutput_notebook()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Generating the plot itself with a custom hover tooltip \n\nx_subset_reshape = x_subset.reshape(10000,28,28)\n\ndigits_df = pd.DataFrame(embedding, columns=('x', 'y'))\ndigits_df['digit'] = [str(x) for x in y_subset]\ndigits_df['image'] = list(map(embeddable_image, x_subset_reshape))\n\n\ndatasource = ColumnDataSource(digits_df)\ncolor_mapping = CategoricalColorMapper(factors=[str(9 - x) for x in y_subset],\n palette=Spectral10)\n\nplot_figure = figure(\n title='UMAP projection of the Kannada MNIST dataset',\n plot_width=600,\n plot_height=600,\n tools=('pan, wheel_zoom, reset')\n)\n\nplot_figure.add_tools(HoverTool(tooltips=\"\"\"\n
\n
\n \n
\n
\n Digit:\n @digit\n
\n
\n\"\"\"))\n\nplot_figure.circle(\n 'x',\n 'y',\n source=datasource,\n color=dict(field='digit', transform=color_mapping),\n line_alpha=0.6,\n fill_alpha=0.6,\n size=4\n)\nshow(plot_figure)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Comparison of Dimension Reduction Techniques\nHere is a comparison of the three different Dimensionality reduction techniques using Kannada MNIST dataset.\n\n![](https://imgur.com/iMjNF0Q.png)\n"},{"metadata":{},"cell_type":"markdown","source":"We can see that the UMAP does a great job in separating digits as compared to t-SNE or PCA. As mentioned in UMAP’s GitHub repository, it often performs better at preserving aspects of the global structure of the data than t-SNE. This means that it can often provide a better “big picture” view of the data as well as preserving local neighbor relations."},{"metadata":{},"cell_type":"markdown","source":"## Using UMAP technique to understand data\n\nWe shall now examine how a Word2Vec model that has been trained to work with text, groups the meanings of words. It will be interesting to see these groupings since they will reveal a lot about the relationships between the words. \n\n[Wikipedia](https://en.wikipedia.org/wiki/Word2vec) describes Word2vec as a group of related models that are used to produce [word embeddings](https://en.wikipedia.org/wiki/Word_embedding). Word2vec takes as its input a large [corpus of text](https://en.wikipedia.org/wiki/Text_corpus) and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space.\n\n![](https://imgur.com/kM4rg2D.png)\n\nNotice how similar meaning words appear together and can be easily visualised.\n\nThe projection has been created using the [Embedding projector](http://projector.tensorflow.org/) tool. You can read more about the tool [here](https://towardsdatascience.com/visualizing-bias-in-data-using-embedding-projector-649bc65e7487)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}