{"cells":[{"metadata":{},"cell_type":"markdown","source":"<div align='center'><font size=\"6\" color=\"#FFA500\">Visualizing Kannada MNIST with t-SNE</font></div>\n\n<div align='center'><font size=\"4\" color=\"#FFA500\">A 3 part series on Dimensionality reduction techniques using the Kannada MNIST dataset</font></div>\n<hr>\n\n\n## Drawbacks of PCA\n\nIn my last kernel titled [Visualizing Kannada MNIST with PCA](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-pca), I demonstrated how we could visualize the Kannada MNIST dataset by reducing the 784 dimensions into 2 thereby, making it easy to be viewed by human eye. We used PCA for reducing the dimensionality of the dataset. PCA essentially tries project the original data onto the directions where variance is maximum. In our case it projcected the data onto 2 Dimensions which could then be easily visualised.\n\nHowever, the visualisations produced by PCA was not able to do such a good job in differentiating all the digits.This is because PCA is a **linear projection**, which means it can’t capture non-linear dependencies. In this notebook we shall explore another Dimensionality reduction technique called **t-SNE** and see if it gives us better results as compared to PCA\n<hr>\n<div align='left'><font size=\"4\" color=\"#FFA500\">Part2: t-SNE(T-distributed stochastic neighbour embedding) in Python</font></div>\n<hr>\n\n## Table of Contents\n\n* What is t-SNE\n* Embeddings]\n* t-SNE under the hood\n* t-SNE with Scikit learn\n* Interactively visualising t-SNE with Bokeh\n* Further Readings\n\n\n\n## <a name=\"tsne\"></a>t-SNE\n\n**t-SNE** or **T-distributed stochastic neighbour embedding** takes a high dimensional data set and reduces it to a low dimensional graph that retains a lot of the original information. It does so by giving each data point a location in a two or three-dimensional map. This technique finds clusters in data thereby making sure that an embedding preserves the meaning in the data. \nt-SNE reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. \n\n## <a name=\"embeddings\"></a>Embeddings\nAn **embedding** is essentially a low-dimensional space into which a high dimensional vector can be translated. During translation, an embedding preserves the semantic relationship of the inputs by placing similar inputs close together in the embedding space. Let’s try and wrap our head around this concept with examples. Here is a grab from the creators of the [Embedding projector](https://projector.tensorflow.org/), a tool which enables us to visualise high dimensional data easily.\n\n<iframe src=\"https://player.vimeo.com/video/340677521\" width=\"640\" height=\"352\" frameborder=\"0\" allow=\"autoplay; fullscreen\" allowfullscreen></iframe>\n<p><a href=\"https://vimeo.com/340677521.mp4\">Embeddings</a> from <a href=\"https://vimeo.com/user97669696\">Parul Pandey</a> on <a href=\"https://vimeo.com\">Vimeo</a>.</p>\n\n## t-SNE under the hood\n**t-SNE**, was proposed by Geoffry Hinton’s and Laurens van der Maaten  back in 2008. Their paper titled [Visualizing Data using t-SNE](http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf) is an essential read for soembody trying to understand t-SNE. Here is how t-SNE basically works:\n\n1.  First, a probability distribution is created in a high dimensional space. This means if we pick a point in the dataset, we define the probability of picking another point as a neighbour.\n2. Next, a low dimensional space is then created that has the same(or as near as possible) probability distribution as the high Dimensional space.\n\n## t-SNE with Scikit learn\n\nScikit-learn has an implementation of t-SNE available, and you can check its documentation [here](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html). \n\n\n\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"### Loading the necessary libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from time import time\n\nimport numpy as np\nimport pandas as pd\n\n\n# For plotting\nfrom matplotlib import offsetbox\nimport matplotlib.pyplot as plt\nimport matplotlib.patheffects as PathEffects\nimport seaborn as sns\nimport plotly.graph_objects as go\n\n%matplotlib inline\nsns.set(style='white', context='notebook', rc={'figure.figsize':(14,10)})\n\n#For standardising the dat\nfrom sklearn.preprocessing import StandardScaler\n\n#PCA\nfrom sklearn.manifold import TSNE\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reading in the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train=  pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest=  pd.read_csv('../input/Kannada-MNIST/test.csv')\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Setting the label and the feature columns","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train.loc[:,'label'].values\nx = train.loc[:,'pixel0':].values\n\nprint(x.shape)\nprint(y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Plotting the original train data\n\ndef plot_digits(data):\n    fig, axes = plt.subplots(4, 10, figsize=(10, 4),\n                             subplot_kw={'xticks':[], 'yticks':[]},\n                             gridspec_kw=dict(hspace=0.1, wspace=0.1))\n    for i, ax in enumerate(axes.flat):\n        ax.imshow(data[i].reshape(28, 28),\n                  cmap='binary', interpolation='nearest',\n                  clim=(0, 16))\nplot_digits(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Standardizing the data\nstandardized_data = StandardScaler().fit_transform(x)\nprint(standardized_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# t-SNE is consumes a lot of memory so we shall use only a subset of our dataset. \n\nx_subset = x[0:10000]\ny_subset = y[0:10000]\n\nprint(np.unique(y_subset))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Applyting t-SNE on the data","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%time\ntsne = TSNE(random_state = 42, n_components=2,verbose=0, perplexity=40, n_iter=300).fit_transform(x_subset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is a brief explanation of the parameter.\n\n* **n_components (default: 2)**: Dimension of the embedded space.\n* **verbose (default: 0)** : Verbosity level.\n* **perplexity (default: 30)**: The perplexity is related to the number of nearest neighbors that are used in other manifold learning algorithms. Consider selecting a value between 5 and 50.\n* **n_iter (default: 1000)**: Maximum number of iterations for the optimization. Should be at least 250.\n\nThere are other parameters which can be tunes. refer to the [documentation](http://)https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html for detailed info.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Visualizing the t-SNE","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(tsne[:, 0], tsne[:, 1], s= 5, c=y_subset, cmap='Spectral')\nplt.gca().set_aspect('equal', 'datalim')\nplt.colorbar(boundaries=np.arange(11)-0.5).set_ticks(np.arange(10))\nplt.title('Visualizing Kannada MNIST through t-SNE', fontsize=24);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nAn important point to note here is that t-SNE is computationally expensive, hence it is mentioned in its documentation that :\n\n*It is highly recommended to use another dimensionality reduction method (e.g. PCA for dense data or TruncatedSVD for sparse data) to reduce the number of dimensions to a reasonable amount (e.g. 50) if the number of features is very high.*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Using PCA to reduce dimensions of data before feeding to t-SNE  algorithm","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA\npca_50 = PCA(n_components=50)\npca_result_50 = pca_50.fit_transform(x_subset)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Using the output of PCA as input for t-SNE\n%time\npca_tsne = TSNE(random_state = 42, n_components=2, verbose=0, perplexity=40, n_iter=300).fit_transform(pca_result_50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow!! the time taken for the algorithm to run has been considerably reduced.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#visualising t-SNE again \n\nplt.scatter(pca_tsne[:, 0], pca_tsne[:, 1], s= 5, c=y_subset, cmap='Spectral')\nplt.gca().set_aspect('equal', 'datalim')\nplt.colorbar(boundaries=np.arange(11)-0.5).set_ticks(np.arange(10))\nplt.title('Visualizing Kannada MNIST through t-SNE', fontsize=24);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualising TSNE in 3D\n\nLet's try and visualise the output as a 3D scatter plot:\n## With Matplotlib","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%time\npca_tsne2 = TSNE(random_state = 42, n_components=3, verbose=0, perplexity=40, n_iter=300).fit_transform(pca_result_50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from mpl_toolkits.mplot3d import Axes3D\n\nfig = plt.figure()\nax = fig.add_subplot(111, projection='3d')\n\nax.scatter(pca_tsne2[:, 0], pca_tsne2[:, 1],pca_tsne2[:,2], s= 5, c=y_subset, cmap='Spectral')\nplt.title('Visualizing Kannada MNIST through t-SNE in 3D', fontsize=24);\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## With Plotly","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x=pca_tsne2[:, 0]\ny=pca_tsne2[:, 1]\nz=pca_tsne2[:, 2]\n\nfig = go.Figure(data=[go.Scatter3d(\n    x=x,\n    y=y,\n    z=z,\n    mode='markers',\n    marker=dict(\n        size=12,\n        color=x,                # set color to an array/list of desired values\n        colorscale='Spectral',   # choose a colorscale\n        opacity=0.8\n    )\n)])\n\n# tight layout\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=0))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x=pca_tsne2[:, 0]\ny=pca_tsne2[:, 1]\nz=pca_tsne2[:, 2]\n\nfig = go.Figure(data=[go.Scatter3d(\n    x=x,\n    y=y,\n    z=z,\n    mode='markers',\n    marker=dict(size=3, symbol=\"circle\"),\n    )\n])\n\n# tight layout\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=0))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Using Bokeh to visualize t-SNE\nWe see that t-SNE has successfully captured the classes. To get a better idea of why t-SNE chose to do this it is helpful to see the actual digits involve. One can do this using bokeh and mouseover tooltips of the images.\n\nSource of code : https://umap-learn.readthedocs.io/en/latest/basic_usage.html\n\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Encoding all the images for inclusion in a dataframe.\n\nfrom io import BytesIO\nfrom PIL import Image\nimport base64\n\n\ndef embeddable_image(data):\n    img_data = 255 - 15 * data.astype(np.uint8)\n    image = Image.fromarray(img_data, mode='L').resize((28,28), Image.BICUBIC)\n    buffer = BytesIO()\n    image.save(buffer, format='png')\n    for_encoding = buffer.getvalue()\n    return 'data:image/png;base64,' + base64.b64encode(for_encoding).decode()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# loading up bokeh and other tools to generate a suitable interactive plot.\n\nfrom bokeh.plotting import figure, show, output_notebook\nfrom bokeh.models import HoverTool, ColumnDataSource, CategoricalColorMapper\nfrom bokeh.palettes import Spectral10\n\noutput_notebook()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Generating the plot itself with a custom hover tooltip \n\nx_subset_reshape = x_subset.reshape(-1,28,28)\n\ndigits_df = pd.DataFrame(pca_tsne, columns=('x', 'y'))\ndigits_df['digit'] = [str(x) for x in y_subset]\ndigits_df['image'] = list(map(embeddable_image, x_subset_reshape))\n\n\ndatasource = ColumnDataSource(digits_df)\ncolor_mapping = CategoricalColorMapper(factors=[str(9 - x) for x in y_subset],\n                                       palette=Spectral10)\n\nplot_figure = figure(\n    title='t-SNE projection of the Kannada MNIST dataset',\n    plot_width=600,\n    plot_height=600,\n    tools=('pan, wheel_zoom, reset')\n)\n\nplot_figure.add_tools(HoverTool(tooltips=\"\"\"\n<div>\n    <div>\n        <img src='@image' style='float: left; margin: 5px 5px 5px 5px'/>\n    </div>\n    <div>\n        <span style='font-size: 16px; color: #224499'>Digit:</span>\n        <span style='font-size: 18px'>@digit</span>\n    </div>\n</div>\n\"\"\"))\n\nplot_figure.circle(\n    'x',\n    'y',\n    source=datasource,\n    color=dict(field='digit', transform=color_mapping),\n    line_alpha=0.6,\n    fill_alpha=0.6,\n    size=4\n)\nshow(plot_figure)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Hover over the plot to actually see the digits in the clusters**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's see a comparison of the PCA and the t-SNE technique on the same dataset\n\n![](https://imgur.com/iO8ahC6.png);![](https://imgur.com/Suki6Px.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Thus, we can see t-SNE does a better job as compared to PCA when it comes to visualising High Dimensional datasets.Similar digits are clustered together. If we were to use a clutering algorithm on this output, we can easily assign labels to unseen test data.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n### Other notebooks in this series:\n\n* [Part 1: Visualizing Kannada MNIST with PCA](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-pca)\n* [Part 2: Visualizing Kannada MNIST with t-SNE](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-t-sne)\n* [Part 3: Visualizing Kannada MNIST with UMAP](https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-umap-technique)\n<hr>\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Further Readings\n[The original paper by Hinton and  Maaten](http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf)\n\n[A blog post by distilled on using t-SNE effectively (and understanding its pitfalls)](https://distill.pub/2016/misread-tsne/)\n\n[The scikit-learn docs](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}