{"cells":[{"metadata":{"id":"Hmxv3CVF7EED"},"cell_type":"markdown","source":"# Visualizing High Dimensional MNIST Dataset with PCA and t_SNE"},{"metadata":{"id":"nZdXvsX-86Sk"},"cell_type":"markdown","source":"#### **Introduction**\n \nHumans can't visualize more then three dimensions but many a times data is available in more then three dimensions so to visualize the data in 2 or 3 dimension which humans can visualize. we reduce the high dimensional data to 2/3 dimensions through Dimensionality reduction techniques like PCA and t-SNE."},{"metadata":{"id":"sYMAwjJJ4c_u"},"cell_type":"markdown","source":"#### Objective : we will take a data set of 784 dimensions and conver it in to 2 and 3 dimensions to visualize it.# "},{"metadata":{"id":"urP4oA003Wer","trusted":true},"cell_type":"code","source":"#importing library \nimport numpy as np\nimport pandas as pd\n\n# for visualization purpose\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# for preprocessing purpose\nfrom sklearn.preprocessing import StandardScaler\n\n# for doing PCA\nfrom sklearn import decomposition\n\n# for t-SNE implemention \nfrom sklearn.manifold import TSNE","execution_count":null,"outputs":[]},{"metadata":{"id":"WF2td94t4_YW"},"cell_type":"markdown","source":"### MNIST dataset \nThe dataset train.csv contain gray-scale images of hand-drawn digits, from zero through nine. Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive. \nThe data set(train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image."},{"metadata":{"id":"momLMnZSnlRv"},"cell_type":"markdown","source":"Data source : https://www.kaggle.com/c/digit-recognizer/data"},{"metadata":{"id":"X93t_zFA8TD0","outputId":"2498ff05-ce17-435d-d843-340f9bb6dc3f","trusted":true},"cell_type":"code","source":"# importing the dataset \ndata_image = pd.read_csv('../input/digit-recognizer/train.csv')\n\n# saving all the label in to variable \"labels\"\nlabels = data_image['label']\n\n# saving the pixel data in to \"image_vec\" variable\nimage_vec = data_image.drop('label',axis=1)\n\ndata_image.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"k1UxPzog8ccV","outputId":"e975468d-b25b-4fc7-d64c-bcba5e99c89b","trusted":true},"cell_type":"code","source":"# let's look at the first image in the dataset \"image_vec\" and its respective label in the dataset \"labels\"\n# we can change the below variable \"no\" to see other images\nno = 0 \nplt.figure(figsize = (3,3))\ngrid_data = image_vec.iloc[no].to_numpy().reshape(28,28)\nplt.imshow(grid_data, interpolation = 'none', cmap='gray')\nplt.show()\nprint(50*\"*\")\nprint(labels[no])\nprint(50*\"*\")","execution_count":null,"outputs":[]},{"metadata":{"id":"cnjPbOJk7Nnw","outputId":"d2dd15d9-1b71-4287-8919-1c968ab29894","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nfor i in range(25):\n plt.subplot(5,5,i+1)\n plt.xticks([])\n plt.yticks([])\n plt.grid(False)\n grid_data = image_vec.iloc[i].to_numpy().reshape(28,28)\n plt.imshow(grid_data, cmap=plt.cm.binary)\n plt.xlabel(labels[i])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"r2SDjk4z6Qxb","outputId":"e2bb2fb8-0d59-4d57-ca22-b211f83b3cb9","trusted":true},"cell_type":"code","source":"## First 10 images in the dataset\nplt.figure(figsize=(14,14))\nfor digit_num in range(0,10):\n plt.subplot(1,10,digit_num+1)\n grid_data = image_vec.iloc[digit_num].to_numpy().reshape(28,28) # reshape from 1d to 2d pixel array\n plt.imshow(grid_data, interpolation = \"none\", cmap = \"gray\")\n plt.xticks([])\n plt.yticks([])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"A_i8fD7xDboQ"},"cell_type":"markdown","source":"There is 42000 such images each with 784 features. To visualize these data points in 2 and 3 dimension lets reduce the dimensions of the data from 784 to 2 and 3 dimension. "},{"metadata":{"id":"_71xl1FdcTr1"},"cell_type":"markdown","source":"### Standardize the Data\nThe results of PCA depend on the scaling of the variables. This can be cured by scaling each feature by its standard deviation, so that one ends up with dimensionless features with unital variance."},{"metadata":{"id":"TKUctqWOR4Z8","outputId":"d960f5fb-2fa8-411e-e6f3-06fe0def1e37","trusted":true},"cell_type":"code","source":"# before going forward let's standardize the data with mean zero and variance one\n\nstandardized_data = StandardScaler().fit_transform(image_vec)\nprint(standardized_data.shape) ","execution_count":null,"outputs":[]},{"metadata":{"id":"Ezmmba0FFX-5"},"cell_type":"markdown","source":"# **PCA without sklearn and with sklearn :** \nPrincipal component analysis (PCA) is the process of computing the principal components(eigen vectors) and projecting each data point onto only the first few principal components to obtain lower-dimensional data while preserving as much of the data's variation as possible. "},{"metadata":{"id":"fRlFVPXYDaQq","outputId":"b1b888ab-905d-4430-a603-b1a5ac77b360","trusted":true},"cell_type":"code","source":"# PCA with out sklearn librery \n\n#find the co-variance matrix which is : A^T * A\nsample_data = standardized_data\n\ncovariance_matrix = np.matmul(sample_data.T , sample_data)\n\nprint ( \"The shape of variance matrix = \", covariance_matrix.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"id":"CT75j1t9CN_o","outputId":"019ccf77-55f0-498d-ce53-ee888074281b","trusted":true},"cell_type":"code","source":"# finding the top two eigen-values and corresponding eigen-vectors \n# for projecting onto a 2-Dim space.\n\nfrom scipy.linalg import eigh \n \n# eigh function will return the eigen values in asending order\n# taking top 2 eigenvalues and eigenvectors.\nvalues, vectors = eigh(covariance_matrix, eigvals=(782,783))\n\nprint('Largest eigen value is {}'.format(values[1]))\nprint('Second largest eigen is {}'.format(values[0]))\n","execution_count":null,"outputs":[]},{"metadata":{"id":"e4UWZ5mC9wSw","outputId":"86d77899-b0b6-4e0c-c34b-7b6028f592bb","trusted":true},"cell_type":"code","source":"# projecting the original data points on the plane \n# formed by two principal eigen vectors.\n\nnew_coordinates = np.matmul(vectors.T, sample_data.T)\n\nprint (\" resultanat new data point's shape is \", vectors.T.shape, \"X\", sample_data.T.shape,\" = \", new_coordinates.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"XKPEqWGeV59L","outputId":"b4fd1f27-0c95-487f-cec2-b97c78f27ebb","trusted":true},"cell_type":"code","source":"# appending label to the 2d projected data\nnew_coordinates = np.vstack((new_coordinates, labels)).T\n\n# creating a new data frame for ploting the labeled points.\ndataframe = pd.DataFrame(data=new_coordinates, columns=(\"2nd_principal\", \"1st_principal\", \"label\"))\nprint(dataframe.head())","execution_count":null,"outputs":[]},{"metadata":{"id":"Bo3-p5-HcBHQ","outputId":"af22fb9a-af39-4fd4-aecf-abf5b7d03438","trusted":true},"cell_type":"code","source":"# ploting the 2d data points with seaborn\nsns.FacetGrid(dataframe, hue=\"label\", height=6).map(plt.scatter, '1st_principal', '2nd_principal').add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"OE1HrZ17cfDl","outputId":"aaa6e9f3-d5d9-47ca-e17f-410fbcb90c77","trusted":true},"cell_type":"code","source":"### PCA with sklearn \npca = decomposition.PCA()\npca.n_components = 2\npca_data = pca.fit_transform(sample_data)\n\n# pca_data will contain the 2-d projects of sample data on the top 2 eigen vector direction\nprint(\"shape of pca_data.shape = \", pca_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"CnOqgZQ7liTc","outputId":"ac2d13d1-e4cf-4e54-ec00-9f69beb79845","trusted":true},"cell_type":"code","source":"# attaching the label for each 2-d data point \npca_data = np.vstack((pca_data.T, labels)).T\n\n# creating a new data fram which help us in ploting the result data\npca_df = pd.DataFrame(data=pca_data, columns=(\"1st_principal\", \"2nd_principal\", \"label\"))\nsns.FacetGrid(pca_df, hue=\"label\", height=6).map(plt.scatter, '1st_principal', '2nd_principal').add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"9SJaJwuwpkC-","trusted":true},"cell_type":"code","source":"### PCA with sklearn for projection of data on top 3 eigen vector. \npca = decomposition.PCA()\npca.n_components = 3\npca_data_3 = pca.fit_transform(sample_data)","execution_count":null,"outputs":[]},{"metadata":{"id":"3wo4Ppkdr1lT","trusted":true},"cell_type":"code","source":"pca_data_3 = np.vstack((pca_data_3.T, labels)).T\npca_df_3 = pd.DataFrame(data = pca_data_3, columns=(\"1st_principal\", \"2nd_principal\",\"3rd_principal\", \"label\"))","execution_count":null,"outputs":[]},{"metadata":{"id":"tiXaoF-8rySC","outputId":"1ca3b527-0856-4cb1-84f5-3cedbc5fe9bf","trusted":true},"cell_type":"code","source":"import plotly.express as px\nfig = px.scatter_3d(pca_df_3, x='1st_principal', y='2nd_principal', z='3rd_principal',\n color='label')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"ov5owT1QW6xA"},"cell_type":"markdown","source":"Below is the graph between Number of components and cumulatuive variance explained. we can see that 400 features explaines more the 95% of variance in the data. "},{"metadata":{"id":"_QYnt8HcnYvg","outputId":"e7078258-c506-4895-9b61-0fd36417bf56","trusted":true},"cell_type":"code","source":"pca = decomposition.PCA()\npca = pca.fit(standardized_data)\nplt.grid()\nplt.plot(np.cumsum(pca.explained_variance_ratio_))\nplt.xlabel('Number of Components')\nplt.ylabel('Cumuative variance explained');","execution_count":null,"outputs":[]},{"metadata":{"id":"9TYTgBgvdwTp"},"cell_type":"markdown","source":"### **Conclusion of PCA Results on MNIST Dataset :**\n The PCA performed well when it comes to seperating some digits like 1's form 0's but perrformed not so good when seperating all the digits in the MNIST dataset this is because PCA is linear projection technique so it does not take non liner dependencies in to consideration. "},{"metadata":{"id":"HdKPLA4Tjw4R"},"cell_type":"markdown","source":"# **t-SNE with sklearn:**\nt-distributed stochastic neighbor embedding (t-SNE) is a machine learning algorithm for visualization based on Stochastic Neighbor Embedding \nIt is a nonlinear reduction technique well-suited for embedding high-dimensional data for visualization in a low-dimensional space of two or three dimensions.\n\nWe will not be doing t-SNE from scrach as in the case of PCA becaues its math is non-trivial but a high level overview of t-SNE is that it tries to preserve the tropology of the data. For every point, it constructs a notion of which other points are its neighbors, trying to make all points have the same number of neighbors. Then it tries to embed them in to low dimensional space so that those points all have the same number of neighbors. "},{"metadata":{"id":"4-y6ewJeqdHU","outputId":"76fece7d-9843-40a5-e1a9-209849aefe16","trusted":true},"cell_type":"code","source":"print(standardized_data.shape)\nprint(labels.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"jJyoervGwcwH"},"cell_type":"markdown","source":"Two parameters perplexity and n_iter(number of iterations) are the two most important parameters for the t-SNE algorithm\n\n* **Perplexity :** perplexity, loosely says how to balance attention between local and global aspects of your data. The parameter is, in a sense, a guess about the number of close neighbors each point has. The perplexity value has a complex effect on the resulting pictures. Typical values are between 5 and 50. Getting the most from t-SNE may mean analyzing multiple plots with different perplexities..\n \n* **n_iter :** Maximum number of iterations for the optimization we iterate until we reaching a stable configuration of plot.\n"},{"metadata":{"id":"4h-DKzeKduvA","outputId":"664e8a53-96de-408a-b2c6-31d8222ea2ec","trusted":true},"cell_type":"code","source":"\nmodel = TSNE(n_components=2, random_state=0)\n# configuring the parameteres\n# the number of components = 2\n# default perplexity = 30\n# default learning rate = 200\n\n# default Maximum number of iterations for the optimization = 1000\n\ntsne_data = model.fit_transform(standardized_data)\n\n# creating a new data frame which help us in ploting the result data\ntsne_data = np.vstack((tsne_data.T, labels.T)).T\ntsne_df = pd.DataFrame(data=tsne_data, columns=(\"Dimension_1\", \"Dimension_2\", \"label\"))\n\n# Ploting the result of tsne\nsns.FacetGrid(tsne_df, hue=\"label\", height=6).map(plt.scatter, 'Dimension_1', 'Dimension_2').add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"SB4LaLqQSvB6"},"cell_type":"markdown","source":"### **Conclusion of t-SNE results on MNIST**\nt-SNE has gives much better result as compared to PCA. Visualization form t-SNE is much more distinguishable and all the digits are well clustered in groups."},{"metadata":{"id":"4SltOX-5Z76U"},"cell_type":"markdown","source":"References:\n\n\n* http://colah.github.io/posts/2014-10-Visualizing-MNIST/\n* https://distill.pub/2016/misread-tsne/\n\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}