{"cells":[{"metadata":{},"cell_type":"markdown","source":"# t-SNE vs PCA vs LDA\nI recently got to know about t-SNE and was very suprised to see the difference in the results among t-SNE and PCA, So I wanted to share what I found in the implementation\nof this amazing algorithm.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Stage-1 Loading the dataset\nwe will use the \"digit recognizer\" dataset to see the visual differences of the outcomes after dimansanality reduction from PCA and t-SNE.\n\n1) Import the normal Libraries\n\n2) Load the dataset\n\n3) Split the dataset in dependent and independent variables","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndata.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label = data['label']\nlabel.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dependent = data.drop('label',axis=1)\ndependent.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualising a row of the data and cross-checking the loaded data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(7,7))\nrow_index = 100\n\ngrid_data = np.array(dependent.iloc[row_index]).reshape(28,28)\nplt.imshow(grid_data, interpolation = 'none', cmap= \"gray\")\nplt.show()\nprint(label.iloc[row_index])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stage-2 Standardization of Data\n\nWell PCA tries to preserve the maximum variance along a dimension by projecting the features on that dimension so it becomes necessary to scale the features to not give a particular feature advantage.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dependent_scaled = StandardScaler().fit_transform(dependent)\ndependent_scaled.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stage-3 Understanding and Applying PCA\n\nKeeping it brief PCA tries to maximize the varience (range of feature) so as to keep the most important features of datasets preserved, but it has some flaws in it and that is it can't preserve the structure of the dataset which is solved by t-SNE.\n\nexample :- For simplisity let us take a 2-dim dataset and reduce it to a 1-dim dataset.\n\n\nIn the left image we could see some 2-dim. dataset without standardisation.\n\nIn the right side image after standardisation the PCA looks for such a dimension/vector that the variance after the projection on the new_plane is maximum. In this case that new_plane is f' and this new_plane will become our new 1-Dim/featured dataset\n\n![Untitled.png](attachment:Untitled.png)","attachments":{"Untitled.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAr8AAAE4CAYAAABSRJsTAAAmnElEQVR4Xu3da5Bc5X3ncUlAlso6ZZcdSwgEeJHESCNGl5EmJUBYBIE2cbmcWKlgk3LtVtkVVcqQlInLe6l9wVZSbLwEJcEYh0CQEbINCAsCiitOgp21MUJyUopuIyEk24BGM9Oa+7Wne/r0f8//6em59PV09+nT5+nzffGpufXTTSWe3/z09HNZJJIWAAAAIAoW5X4DAAAAaFaUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUXwAAAEQG5RcAAACRQfkFAABAZFB+AQAAEBmUX4SYI05yVAZ6u6Wnp1t6B8Yk6TgFHgcAAOAN5Rch5Ug6OSxdb+yRezs6ZEvHNtlx/345PTwu086EjMR6pKf7knS7enp7ZCQ+Lel07nMAAAAsRPlFSCVlYuCoPH7fRll//1458c670jswKonpUYmd/478r4/fIVs3tcumjRukvX2D/PF3TsulEQowAAAojfKLkIrLSM9B+fK6FvnUI4elbzzpfm9aJodPyoH//il5+MApOf/eJXn3nePy3AOrZM269fKVv7skI1MsiwAAAMVRfhE+6ZQ48ZicP/mU7F65Uj7x0Pek8+ddMjA+JanUlEwMxWR0KjPLm54eke7/97/lN1beJL/9V/8q/RNakgs8JwAAgFB+EUaJUel7/WHZcftaWXr11fKhFS2y+fa75IH9p2R4cnrhY6fdxx7+qvzmqpWUXwCWYVMv0AiUX4RPiZlfJz3/D4Mj6XivnN+3W1pa2uTBV7pY9gDAEmzqBRqF8ouQmpTh7gPyYEuL7NpzRPrNmt+Fj0mnJ6Tvwn750i3rpPXTfy4/6RmTaf44ALACm3qBRqH8IqRKl990elpGLp2Sb31po7Rv/B159McXZSTJrC8AW7CpF2gUyi9Cqnj5zRbfb39pk3RscYvvD34mffGU/qTA8wBAyJRY2sWmXqD+KL8IqSLl1/2jMd77tnz7wTZpb2+TB7/9tvSOU3wBWIRNvUBDUX4RUoXKryPT493y5p9/Wtatuk7u2/PP8m9vvy+XZjaF9Pb2y3iSIgwg5ErM/LKpF6g/yi9CKi6jsdfkoW3b5AtPHpNBM9sxJWOX/1H+9Pb1smrFClnZukE2bmqX9nb1a3LHjvtl/+lBmUzlPhcAhE2hf+AvfAybeoH6oPwipBxJ63E/l2MyODsbMvO9eUcAzcfMLwB7lC6/bOoF6ofyCwBA4IqXXzb1AvVF+QUAIHBFyi+beoG6o/wCABC4QuWXTb1AECi/AAAEjk29QKNQfgEACFx1m3pH4wn3Y4/5OuY+znFyzgUGUBblFwAAC2jR7ew8JW1tbbJ8+XLzUb+mAAOVofwCAGABnenVwrtkyRJZtGiR+ahf6/dzHwugOMovrKGzGxryvN0HIIo0+3TGV4tvln6t3899LIDiKL+wAm/3AYg6yi/gD8ovrMDbfQCijvIL+IPyCyuEO/Qzu7Qzcn8GAP4Idw4C9qD8wgrhDH237E7FxYnFxOm6KE5Pt6QnxkVSuhSDIgzAX+HMQcA+lF9YIXyh70h6bFQSrx6S4e33yFD7VhnetkPiTz8jqQsXZgpw7hgAqF74chDeOOIkR2Wgt1t6erqld8D92zEdl/HBmFwemZg54zl3DOqJ8gsrhC70p+Km+A5t3S4DN6yWgRUrZeD61W4JvlXij31d0uNj+WMAoAahy0F44Eg6OSxdb+yRezs6ZEvHNtlx/7Py01+8JU/u3iZ3PHRIYqPxAuNQT5RfWCFcoe+YpQ464ztbfLPcAjy87S5xLscKjAOA6oUrB+FNUiYGjsrj922U9ffvlRPvvGtmfkcH35MX/rhVWr/yknQPU36DRvmFFcIV+o5Z46tLHRYU3xk6++vEeguMA4DqhSsH4U1cRnoOypfXtcinHjksfeNJ8/2p4Uvy0ldaZR3ltyEov7BCuELfMZvbdI2vzvRSfgEEIVw5iLLSKXHiMTl/8inZvXKlfOKh70nnz7tkYHxKJoe65Ltu+d3wP16WnhHKb9Aov7BC2EJfT3XQzW1adBcU4BtWm+UQ6b7LeWMAoBZhy8FmppvQYvGUdE9Om49VbUpLjErf6w/LjtvXytKrr5YPrWiRzbffJQ/sPyWxWJd8///cJb+/919lYCIzG4zgUH5hhdCFfmranOqgm9t0ja+WYKXFN3HwFZGpyfwxAFCD0OVgk9Ki2zmclLbX+mT5S5dl46E+U4BzH1dWiZlfvZ10ajQmQxNT1RVr1ITyCyuEL/QdU4D1VAfd3KbLHJSZ8TXFlzAD4K/w5WDzyRbfdW7xvWJ/ryx6rscUYJ0Bzn2sN5My3H1AHmxpkV17jkj/zJpfNBblF1Yg9AFEHTlYX4WKr37UGeCqZn4Nym8YUX5hhXCEvt/XGPv9fACaWThysDkVK776tX6/+qUJlN8wovzCCo0Nfb+vMfb7+QBEQWNzsHnVr/gqym8YUX5hhcaFvt/XGPv9fACionE52LzqW3wV5TeMKL+wQsNC3+9rjP1+PgCR0bAcbFL1L74qLqOx1+ShbdvkC08ek0GONQsFyi+s0JjQr/QaYzcop91gi09m6OcLljFU+nwAMKcxOdicgim+ypG0MyEjbrYP6hFnvj0vakH5hRUaE/perzF2w8yZlvRAvyTfOiJTLxww9PN0X59bghOZx3h+vtz/DgBoVA42n+CKL8KK8gsrNCb0PV5jrBdevHNOxv7wQRnafJsMtm4ytOSOfX63TB/9qUgyU4A9PV/efwcANCoHmwvFF4ryCys0KvTLXWOsyxTS/X0y9gcPyOCaDe5jVs09xv18cPUtMvrZz5kZX73tp9zzcS0ygGIalYPNguKLLMovrNCw0C9zjbGe3JD80Y8zSxnmF9/5BXh9hyReO2Q2u5V7Pq5FBlBMw3KwCVB8MR/lF1ZoXOg7Ja8xNjO5f/tNGVy7Mb/4ztAZ4cnHn8ic5Vvm+RZukAOAOY3LQbvlFl+17EDM3NxG8Y0myi+sENbQ10I7+dd/k1nyUKD4mvK7rl2mnn8xcwJEgecAAC/CmoNhVqj46ucnh5LmymKKbzRRfmGF0IZ+YkoS//B9GdzQ4RbdAssebrxZhu/cKamzZ8yJEHnjAcCj0OZgSBUrvsz2gvILK4Q29NMpcS5elJFdn5GBm9YuLMBafG/7dZl6/oW5JQ+54wHAo9DmYAhRfFEK5RdWCG/ouyGaiEvyBz+UkU/uMkscBlvWm6POdMbXFN+RYfehhC2A2oQ3B8OF4otyKL+wQrhD3zEnOaTOn5f43n0y+Zdfcz8+K6kznZkZX8IWgA/CnYPhQPGFF5RfWKExoa9BmZX7s1zuY5yUpCcnzEkO+jGzxrfc2EpeA0CUNSYH7UHxhVeUX1gh2NDPzOQ6sZi5nEJvZTMzuCkvZdarIF4DQDMJNgftQvFFJSi/sEJwoe+YiysSrx4yF0/o5RV6HbHeyqaXU2TKae6YSgXxGgCaTXA5aBeKLypF+YUVAgv9qbgppUNbt5srh82pDdevNrew6a1suqQhb0ylgngNAE0nsBy0CMUX1aD8wgrBhL5jliHobOxsKc1yy6leR6y3suWPq0QQrwGgGQWTg/ag+KJalF9YIZjQd8z6W12GkHdZhUtnZvUq4vxxc+PLb16r9TUARFUwOWgHii9qQfmFFYIJfcdsPNP1tzoL672YVrJ5rdrXABB1weRg+FF8USvKL6wQVOhradWNZ1pCF5TTG1abpQrpvss5YyrfvFb5awBAcDkYZhRf+IHyCysEFvpuWdXSqhvPdP2tFlSlpTRx8BWRqcmFj69m81qlrwEAEmAOhhTFF36h/MIKwYW+Y8qpllbdeKZLEJSZjTWldH7AVrt5rZLXAICM4HIwfCi+8BPlF1YIZ+izeQ1AcMKZg/VH8YXfKL+wQjhDn81rAIITzhysL4ov6oHyCyuENfTZvAYgKGHNwXqh+KJeKL+wQmhDn81rAAIS2hysA4ov6onyCyuEN/Sr3bym38/K/RkA5AtvDvqL4ot6o/zCCs0T+k4FF2IAwJzmycHiKL4IAuUXVmiO0K/8QgwAyGqOHCyO4ougUH5hhaYI/WouxACAGU2Rg0VQfBEkyi+sYH/oV3shBgBk2J+DhVF8ETTKL6xgf+hzIQaA2tiVg9429VJ80QiUX1gh+ND3FtyFFRrLhRgAahN8DlbD+6Zeii8ahfILKwQX+t6DO1/psVyIAaAWweVgtbxv6qX4opEov7BCMKHvPbjzeRjLhRgAahBMDtbA46bevOL7XI+0vtwjpy9PSCqZlPITDUBtKL+wQiCh7zG4C/I01qnyQgwACCgHq+ZtU29e8d3XLWu/eU6O/9Nhib/+Q0ke/ambiX0i0wnznPmvA9SO8gsr1D/0vQV3/rhaxwKAN/XPwVqU39Q73dubV3zXPHFcjv6XP5L+zbfJYNsW87ixz++WabcES5ICjPqg/MIK9Q/98sFdfENaLWMBwJv652AtSm/q7d90q5w6d2nBUoe1e9+Wo7/3RelrWe+OWZV5rPtxcPUtMvrZz5lclXSqwGsBtaH8wgr1D/3SwV26wNYyFgC8qX8O1qbYpt4+9/PDHTtN2c1ubms92C3HX/sXtxRvnSu+WVqA13dI4rVDZklZ7usAtaL8wgpBhH6x4PZyGkMtYwHAiyBysCYFNvXqjK8W3zWPviVL9l2aPdXhdPeITDy9VwbXbsybMFCDa9bL5GOPl95rAVSJ8gsrBBL6BYLb82kMtYwFAA8CycGaLNzUq2t8damDzvjOL7667jc1Pi6Tf/03bsndkFd8TfltccvvX32N8ou6oPzCCsGEfi2nMdQyFgDKCyYH/ZF3qkPuOb6JKUn8w/dlcEOHW3Zzlj1o+V3XLlPPvygSZ+IA/qP8wgo2hT4A1IMtOVi2+Orj0ilxLl6UkV2fkYGb1i4swDfeLMN37pTU2TMiTqnz1YHqUH5hBVtCHwDqxYYc9FR8DZ39jUvyBz+UkU/uMjO9utRhsHWTKb5TL7yYuSGTd81QB5RfWMGG0AeAegp7DnovvlmOOc0hdf68xPfuk8m//Jr78VlJnenMFN+CY4DaUX5hhfCEvoZxVu7PiqlmDAAsFJ4cXEiLbSyekpNDOVcWH+iS0wPxIsU3y/2Zk5L05ITZM6EfM0sdSo0BakP5hRUaH/qZGQq9yU0PXtdzfc3MRKpUSFczBgAKa3wO5svO9ra5pXfZgZgpvnqyw9pvnJBjz7wkyfMXZjIvfyzQKJRfWKGxoe9IemxUEq8eMkeX6U1ueqGFnuurx5sVDvZqxgBAceVzsJZ3mSofm7vMYdFzPab46pm+h7fcIwObbzPHP3JcGcKG8gsrlA/9OpqKmxI7tHW7ubTC7Ea+frU5x7dosFczBgBKKJ6DXTW8y1TdO1Sliu+b7XebW9008/Tccz3+MXc80EiUX1iheOjXu/w65o+Czt7OltisosFezRgAKK1gDl5zjVw6f67Kd5mqe4fKU/GdyTyud0cYUX5hhYKhH1T57bpo/ijkHsJePNirGQMApRXMwQ9/WN598qnq3mWq4h2qQsVXN7fpGl9d6jC/+JJ3CCvKL6xQMPSDKr893WY2RP8oeCuy1YwBgNIK5uBVvyRnW9qqeJep8neoChZf96Oe6qCb23SN74LMc59Xn9/cdJnz2pWuLwb8RPmFFQqGfiDlN23Wv+nbgFpavQV7dWMAoJTcHFzsWnHllXLm2o/l/SNblf6HdmXvUBUrvvq1HmempzrobLGWZh2rNOsSB1+ZueI985rVrC8G/Eb5hRVyQz/I8qvBrOvfygd7jWMAoITcHDTl96qr5Oyq1ireZfL+DlWp4jt7gYWbebpMQmeLdZwy/8g3WZeZ5a1mfTFQD5RfWCE39AMtvxrcZYPdjzEAUFzBHPzgh+Tnf/JwVe8yeXmHylPxLfDceapYXwzUC+UXVigY+oGVXwBovII5uHSpdL3xRnXvMpV5hyoVn/Sn+FaxvhioJ8ovrFAw9Cm/ACKkaA5efL/Kd5mKv0Nliu9Qwofim3mdStYXA/VG+YUVioY+5RdARASTg45/Sx3mP6fH9cVAECi/sEL9Qz977E7loR7sOABRVd8cdMy63Om+y3I6NibrXvWr+GZ4WV+cOwaoF8ovrFC/0M8EfuVH7wQ9DkDU1TMH9SSG+KG/lyP37pa13zzna/E1yqwvLrk2GfAZ5RdWqE/oV3v0TtDjAKBeOZg2/yA3xfe3/qus2XPEXFWcvbJ47bPvyOmBqdqKr+EUXV9cfm0y4C/KL6xQl9Cv9uidoMcBgNQpB93SqUsddMY3W3zVR79xRlofeVPe+s37ZPoySxLQXCi/sIL/oV/t0TtBjwOAjMpzsPzeAp3R1TW+utQhW3zXPPqWvLFlp7y9pkP6t9zOZjQ0HcovrFB56JdT7dE7QY8DgAzvOehtb8HsqQ4zm9uyxffN9rulb2ZTGtmEZkT5hRW8h75X1R69E/Q4AMjwloPe9hbkHmd2xXM9suaJ43J4yz2zxZeTGNCsKL+wgrfQr0y1R+8EPQ4AlKcc9LC3IK/4ulpfvCjHvvEtGbjjbvPYuZMYXhaJsyENzYXyCyt4Cv1KVXv0TtDjAEC85GD5vQXT7s9zi685ymwoIan5JzH0ui5eFOf99zIfiyydAGxE+YUVyod+Nao9eifocQDgJQdL7y3o33SrnDp3Kb/45p3h623pBGAryi+sUD70AaC5lc/B4nsLdB3v4Y6dsnbv22WKb9rT0onc/zbAJpRfWKF86ANAc/OSg4X2Fmjx1RMc9CQHPdGhZPH1sHSCYxlhO8ovrOAl9AGgmXnKwZy9BbrUQWd8vRVfVXrpBCfToBlQfmEFT6EPAE3MWw7O7S3QzW26xleXOngrvpnxxZZOUH7RLCi/sIK30AeA5lVJDhY6zqx88c0otHTC4FhGNAnKL6xQSegDQDPymoO1FF+DYxnR5Ci/sILX0AeAZuUlB2suvgbHMqK5UX5hBS+hDwDNrFwO+lN8geZH+YUVyoU+ADS7UjlI8QW8o/zCCqVCP/exAILnONMSi/WY30nUx/Hjx2TZ0qUm/xbP0K+PnzghJwfis8V30bOX5Fef7pTXz74nXZfynwfB0d8J/d3I/X1BY1F+YQUNEcovEE76x72z85S0tbWZ30vUx7JlS+WXr7hSlrlWXHmlXOt+/JWrfkmWtd8hy545O1t8F/3Zj2TJ9Wtl6TX5z4Fg6e+E/m5QgMOF8hswZkeqozMeS2dmPLL0a/1+7mNRO2YrUAn934v+kV+yZMmC31H4R2d5P+D+33fXL39Ajlxzg5y59mNydPnH5HdWb5arv/rjTOmdKb6LrmuRRfz/IhT0d0J/N/R3JPf3Bo1D+Q0QsyPV06Kb+4dVv9bv5z4WtWO2ApXQfzDp/25y//DDP/9h8WJTfP99+Y3SN3Nl8U/a75brH3lTFu+j+IaZ/m7o70ju7w0ah/IbIGZHYAtmK1AJsq3+dKmDzvhmi++bbvHNXllM8Q0vsjScKL8BYnYENmG2Al7xrlb9tV97rZxbcVNe8VUfffSwLG+/Q5Zfe13eODQW76KFE+U3QMyOwBbMVqBS7GeoRdeM3O/P/bz3xHHpu/0uObzlngXFVz9/c+dn5NLp0wXGodH82z/hiJMclYHebunp6ZbegTFJOhxhVy3Kb4CYHakea36DxWwFUG9ucZmKixOLidN1URy30KQnxs3NaoVuUEu5Pzv27Muy9hsnFhTfw1t2ysCd/zlz+1rea6A5OJJODkvXG3vk3o4O2dKxTXbcv19OD09ISn+eTokTHzJFOzYywbnOHlB+A8bsSHU47SFY/s1WAMjnlpmxUUm8ekiGt98jQ+1bZXjbDok//YykLlyYKcBzjzcXWAwlpPVAl1zxXI9Z4/uR//sTeXnjr8vZ1o0yeeC7M9cO574OmkNSJgaOyuP3bZT19++VE++8OzPz61ZfZ0rGe8/Lqf1/JG0bN8nG/3lIYqPxAs+B+Si/sIIWMp2RnF9+9Wv9fu5jASDUpuKm+A5t3S4DN6yWgRUrZeD61W4JvlXij31d0uNjs4/Nvblt9hzf61rko1deJeuvu05633tXCs0Wo1nEZaTnoHx5XYt86pHD0jeezHw/nZCRSz+Sv/jdDbK+ZYV88MMflWsfeEm6hym/5VB+YQXKL4Dm4JilDjrjO1t8s9wCPLztLnEux8xjixXf+ac6kINNzixpiMn5k0/J7pUr5RMPfU86f94lA+NTkpwYkFP7vig7tt0rf3Lg2/LffmO13Pwlyq8XlF9YgfILoDk4Zo2vLnVYUHxn6OyvE+vNK75KryzWm9vmH2dGDja5xKj0vf6w7Lh9rSy9+mr50IoW2Xz7XfLA/lMyNJGQ5Hi/XHb/sRTrOSyPfrpFWh6k/HpB+YUVKL8AmoNjNrfpGl+d6S1Ufqd7e/OKr37++tn3zJXF5GCElJj5ndvYlpTx/iOyZxfl1yvKL6xA+QXQLPRUB93cpkV3QQHWZRAfv1tOne/OK75ahrsukYPRNCnD3QfkwZYW2bXniPRn1/zOovxWivILK1B+ATSN1LQ51UE3t+kaXy3Bpgi7xffYMy9lTnXIKb46y0cORhXl12+UX1iB0AfQPBxTgPVUB93cpmt8damDzvhmjzPLLb46jhyMKsqv3yi/sAKhXy/cGgQ0WqHNbbnFV5GDUUX59RvlF1Yg9OuBW4OARvNafBU5GFVFyq+b0XraQ2/v+3Kh8+/loU+slJW7n5Lj534hvbEhieslGHnPBUX5hRUI/Xrg1iCgkSopvoocjKq4jMZek4e2bZMvPHlMBicy5Tc1OSin939Rdtzxa7Jp4y3S8p+ul+tXrZUNG2+VO3f+mbzeNyqJvOeCovzCCoR+PXBrENAYjqQSCekcTMi6V7PFt0fWvRKTzqHCxVeRg1HlSNqZkJHLMRmcf8TZ7Mxv7hX13cz8lkH5hRUIfZ+VODuSW4OAenKL78S4nDhySloPdpviu2TfJVn7zFk5dfoX4kznrufUopNBDgL+oPzCCoS+z7g1CGgInfE1xfeF982pDlp81zz6lhz++G/J+GNPmBMgMo91C+9U3FyFrDfC6cUY3T87L9ddcw05CNSI8gsrUH59VmLml1uDgPowa3wHE5kZ33nF9832u6XvxpvNmb969Jl5m3tsVBKvHpLh7feYq5D1Rrj3H/6qbP7Ir8oV5CBQE8ovrED5rYciO4hnUX6ByswtUZj/fS29sXhKTg4lZ9f4Lii+M7e86UUXeuavzvhq8R3auj1z65ve/uY+5v3Vt8hXPvgR+Y+Ll5CDQA0ov7AC5bceKL+AP/KXKOgVxnqRRfZEh7bX+mTZgVjR4ju//Orz6IzvbPGd0e/6t2tukKVXXEEOAjWg/MIKlN96oPwCtSu8RCH+9DOSPH9BTg/EZ48yW2Rubusxm9t0ja8udZgtt27R1fHmxje3QOvzzC++WWeWf0yWUX6BmlB+YQXKbz1QfoGaFVmiMLD5Njn2+D5pffHivOLba44z01MddHObrvHV2V6lxTdx8BWR+KSZOdYCbZ6H8gv4jvILK1B+66FI+eXWIMAjp+ASBV3KcHjLPbLmieNmY9ts8dULLPQc3+mkOdXBzPKaZQ69ku677BbpSfdpHbNkQmeOtRTPL8B9rqPXsOwBqBXlF1ag/NYDtwYBtXHyliho8dW1vLqmV9f2Lii+RW5uy5OaltSFCxJ/7OsLZod/sX6L3PeBD8rVixeTg0ANKL+wAuW3Hrg1CKiNs2CJgi/Fd+Z5tQDnzg5fOnlSblh2jSwmB4GaUH5hBcovgPCYO9Isu0RB1/iapQ6ei2/hY9FKIQcBf1B+YQVCH0DjFTjSbHREkm+fM5vbdI1v+eJb4DlmjkUrV4TJQcAflF9YgdAH0FiFjzSbeGqvnDz5M2l9/r38zW0Fim+h59CZY13jmynAua87hxwE/EH5hRUIfQANVeBIs74bW+St7b9tzu3V83tLF9/Cz6FrhXUzm25u0zW+ea87DzkI+IPyCysQ+gAaJ/9Is8o3t+U/xyz3ufRUB93clv/ac8hBwB+UX1iB0AfQOAuPNKu8+OY/R67s1cb54+aQg4A/KL+wAqEPoHHmjjTTpQ6VF9+Fz1Ho5jbKLxAcyi+sQOgDaCQ9kWHi6b1mje+C4utq/W63dA4lShTfuecodHObLoPQ5RDmlrcC47LIQcAflF9YgdAH0Eh6JfGpznfN5rb5M75afE+88e+SSkzljclT5OY2Lb6Jg69krjfOHTMPOQj4g/ILKxD6ABpFZ3Q7h5Ky7pWYOdVBS++yAzFp06UOQ4mZ4lt61jfDKXhzm5nxNcW39HOQg4A/KL+wAqEPoBFM8R1OmjW9Wnqz63tPumU4Fk+VXergJ3IQ8AflF1Yg9AEErWDxffWyp/W99UAOAv6g/MIKhD6AIOUWX13nq+t9T3a+K9Pd3q8k9hM5CPiD8gsrEPoAglKo+OoJD4c7dkr/psquJPYTOQj4g/ILKxD6AIJQrPjq2b56uUX2RjavVxL7iRwE/EH5hRUIfQD+0uUKWZnv5RZfpUsddMZ3tvhWeCWxn8hBwB+UX1iB0AfgD7fsTsXFicXMdcN665qu39VzfM1xZjmb23SNry51yL2RzeutbH4iBwF/UH5hBUIfQO0cSY+NSuLVQ+ZiiaH2zPpdvblNL7DInOM7d5yZnuqgm9tquZLYT+Qg4A/KL6xA6AOo2VTcFN+hrdvNlcJaYPtubDFXFuvyhuwFFqb4DifNMoharyT2EzkI+IPyCysQ+gBq45ilDlpYZ4uvW2Z1I5tuaNONbbnF14yr8UpiP5GDgD8ov7ACoQ+gNo5Z46tLHTwX35lxtVxJ7CdyEPAH5RdWIPQB1MYxm9t0/a4udZhffM0FFs++I6cHphpyc5tX5CDgD8ovrEDoA6iVrt/VzW26xnd+8V2z54gc+d3fl+n+/rwxYUIOAv6g/MIKhD6AWulxZnqqg25uyy510BlfLb7xl//O4/pdR8R9HolPZujnAS19IAcBf1B+YQVCH0AtzAUWeo6vOc5s7lQHXepgZnzLrt91f+ZMS3qgX5JvHZGpFw4Y+nm6r88twYky42tHDgL+oPzCCoQ+gGoVurmt8Oa2EvTUh3fOydgfPihDm2+TwdZNhm6gG/v8bpk++lORZH0LMDkI+IPyCysQ+gCq4Uvx1csx+vtk7A8ekME1G2Tg+lXzrjleJYOrb5HRz37OnCYh6VSB8f4gBwF/UH5hBUIfQKX8Kb5pM6Ob/PEbmWPS5hff+QV4fYckXjtkLtLIG+8TchDwB+UXViD0AVTCt+LrSscnJf6t75hlDnnFd4bOCE8+/oQ5USJ3vF/IQcAflF9YgdAH4JWfxddITEni+/8ogxs63KJbYOZXy++6dpl6/sXMCRC5431CDgL+oPzCCoQ+AC98L74qnRLn4kUZ2fUZGbhpbX4BvvFmGb5zp6TOnjEnQuSN9wk5CPiD8gsrEPoAyqlL8TXcsYm4JH/wQxn55C4zyzvYsj6jdZMpvlMvvDiz5KGW1ymNHAT8QfmFFQh9AKXUr/hmOWYzW+r8eYnv3SeTf/GYEd/7rKTOdGaKry+vUxw5CPiD8gsrEPoAiql/8c3Siy5Skp6ckPT4WIb7eWapg5+vUxg5CPiD8gsrEPoACgmu+DYeOQj4g/ILKxD6AHJFqfgqchDwB+UXViD0AcwXteKryEHAH5RfWIHQB5AVxeKryEHAH5RfWIHQB6CiWnwVOQj4g/ILKxD6AKJcfBU5CPiD8gsrEPpAtEW9+CpyEPAH5RdWIPSB6KL4ZpCDgD8ov7ACoQ9EE8V3DjkI+IPyCysQ+kD0UHwXIgcBf1B+YQVCH4gWim8+chDwB+UXViD0geig+BZGDgL+oPzCCoQ+EA0U3+LIQcAflF9YgdAHmh/FtzRyEPAH5RdWIPSB5qXFNhZPyckhim8p5CDgD8ovrEDoA80pO9vb5hbdZQdiFN8SyEHAH5RfWIHQB5pP7jKHRc/1UHxLIAcBf1B+YQVCH2guFN/KkYOAPyi/sAKhDzQXXeOrSx0ovt6Rg4A/KL+wAqEPNJfuyWlZ/tJlim8FyEHAH5RfWIHQB5qLzvxuPNRnCrDOAFN8yyMHAX9QfmEFQh9oLtnjzXQGWD9SfMsjBwF/UH5hBUIfQNSRg4A/KL+wAqEPIOrIQcAflF9YgdAHEHXkIOAPyi+sQOgDiDpyEPAH5RdWIPQBRB05CPiD8gsrEPoAoo4cBPxB+YUVCH0AUUcOAv6g/MIKhD6AqCMHAX9QfmEFQh9A1MViPdLW1iZLliwxGagf9Wv9fu5jARRH+YUVKL8Aos5xpqWz85QpvJp/+lG/1u/nPhZAcZRfWIHyCwCZAqwzvZp9+pHiC1SO8gsrUH4BAIAfKL+wAuUXAAD4gfILK1B+AQCAHyi/sALlFwAA+IHyCytQfgEAgB8ov7AC51sCAAA/UH5hBc63BAAAfqD8whqcbwkAAGpF+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGZRfAAAARAblFwAAAJFB+QUAAEBkUH4BAAAQGf8fU2gL4KlxiwoAAAAASUVORK5CYII="}},"execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"PCA class has a parameter \"n_components\" which decides the number of dimensions we want after reduction.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = PCA(n_components = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_data = pca.fit_transform(dependent_scaled)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have to add our \"pca_data\" and \"label\" to plot them over labels.\n\nTo do so we will use numpy function of vertical stack and then convert them into a dataframe.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_pca_data = np.vstack((pca_data.T,label)).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_pca_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_pca_df = pd.DataFrame(data = new_pca_data, columns = ('1st','2nd','label'))\nnew_pca_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stage-4 Plotting PCA results ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set_style(style='dark')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nsns.FacetGrid(new_pca_df, hue='label', height=8 ) \\\n    .map(plt.scatter, '1st', '2nd') \\\n    .add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"well as you can see that the results are not that great. Majority of the points are colliding in a cluster or so and that happens because in the higher dimension they have a proper structure which is completey lost when trying to apply PCA. Well don't get disappointed the next thing up (t-SNE) does a amazing job.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## PCA in 3-D ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_3d = PCA(n_components=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_3d_data = pca_3d.fit_transform(dependent_scaled)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_pca_3d_data = np.vstack((pca_3d_data.T,label)).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_pca_3d_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_pca_3d_df = pd.DataFrame(data = new_pca_3d_data, columns = ('1st','2nd','3rd','label'))\nnew_pca_3d_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter_3d(new_pca_3d_df,\n    x='1st',\n    y='2nd',\n    z='3rd',\n    color='label', title='PCA')\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We could see that even the 3-D don't provide much of a seperation. It is really messy.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Let's See how does  kernel PCA fare (to reduce complexity we will use only 10k rows)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import KernelPCA","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kernel_pca_3d_linear = KernelPCA(n_components=3, kernel='linear', n_jobs=-1)\nkernel_pca_3d_rbf = KernelPCA(n_components=3, kernel='rbf', n_jobs=-1)\nkernel_pca_3d_poly = KernelPCA(n_components=3, kernel='poly', n_jobs=-1)\nkernel_pca_3d_cosine = KernelPCA(n_components=3, kernel='cosine', n_jobs=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kernel_pca_3d_linear_df = kernel_pca_3d_linear.fit_transform(dependent_scaled[:10000])\nkernel_pca_3d_rbf_df = kernel_pca_3d_rbf.fit_transform(dependent_scaled[:10000])\nkernel_pca_3d_poly_df = kernel_pca_3d_poly.fit_transform(dependent_scaled[:10000])\nkernel_pca_3d_cosine_df = kernel_pca_3d_cosine.fit_transform(dependent_scaled[:10000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kernel_pca_3d_linear_df = pd.DataFrame(data = kernel_pca_3d_linear_df, columns = ('1st','2nd','3rd'))\nkernel_pca_3d_rbf_df = pd.DataFrame(data = kernel_pca_3d_rbf_df, columns = ('1st','2nd','3rd'))\nkernel_pca_3d_poly_df = pd.DataFrame(data = kernel_pca_3d_poly_df, columns = ('1st','2nd','3rd'))\nkernel_pca_3d_cosine_df = pd.DataFrame(data = kernel_pca_3d_cosine_df, columns = ('1st','2nd','3rd'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kernel_pca_3d_linear_df['label'] = label[:10000]\nkernel_pca_3d_rbf_df['label'] = label[:10000]\nkernel_pca_3d_poly_df['label'] = label[:10000]\nkernel_pca_3d_cosine_df['label'] = label[:10000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot1 = px.scatter_3d(kernel_pca_3d_linear_df,\n    x='1st',\n    y='2nd',\n    z='3rd',\n    color='label', title='linear')\n\nplot2 = px.scatter_3d(kernel_pca_3d_rbf_df,\n    x='1st',\n    y='2nd',\n    z='3rd',\n    color='label', title='rbf')\n\nplot3 = px.scatter_3d(kernel_pca_3d_poly_df,\n    x='1st',\n    y='2nd',\n    z='3rd',\n    color='label', title='poly')\n\nplot4 = px.scatter_3d(kernel_pca_3d_cosine_df,\n    x='1st',\n    y='2nd',\n    z='3rd',\n    color='label',title='cosine')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot1.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot2.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot3.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot4.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### None of the kernel PCA give satisfactory seperation, now let's try LDA ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.discriminant_analysis import LinearDiscriminantAnalysis ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda = LinearDiscriminantAnalysis(n_components=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_data = lda.fit_transform(dependent_scaled, label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_df = pd.DataFrame(lda_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_df['label'] = label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_df.columns = ['1st', '2nd', 'label']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.FacetGrid(lda_df, hue='label', height=8) \\\n    .map(plt.scatter, '1st', '2nd') \\\n    .add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_3d = LinearDiscriminantAnalysis(n_components=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lda_3d_df = lda_3d.fit_transform(dependent_scaled[:10000],label[:10000])\nlda_3d_df = pd.DataFrame(lda_3d_df, columns=['1st','2nd','3rd'])\nlda_3d_df['label'] = label[:10000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot = px.scatter_3d(lda_3d_df, x='1st', y='2nd', z='3rd', color='label', title='LDA-3D')\nplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### LDA gave much better results than PCA and kernel_pca, but can we get something even better, let's try TSNE.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Stage-5 Understanding and Applying t-SNE\n\nt-SNE = Student's t distributed Stoschastic Neighbourhood Embedding\n\nIn brief t-SNE assigns high probabilty over a pair to the points which are closer to each other in high dimension and low probabilty to those which are far apart. Now it tries to replicate that probabilty distribution in the low dimension and that helps it in preserving the structure.\n\nIn t-SNE centering shouldn't matter since the algorithm only operates on distances between points, however rescaling is necessary if you want the different dimensions to be treated with equal importance, since the 2-norm will be more heavily influenced by dimensions with large variance so we will still use standardised data.\n\nWell TSNE is and iterative algorithm.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.manifold import TSNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"TSNE has 2 very important hyperparameters.\n\n1) Perplexity => this refers to how many points does u want to consider for finding the probabilty of them being close by.\n\n2) n_iter => number of iterations u want to perform before aborting.\n\nYou should generally take high value for n_iter (say 2000 to 5000), and  observe the t-SNE plot wheather it is reaching to some stable form after n_iter number of iterations.\nIn short try different perplexity with constant high iterations.\n\nAfter trying different values I am going for the default values of \"perplexity\" and \"n_iter\" in this case.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"n_components means to what dimensions you want to reduce the dataset to.\nTSNE support only 2 and 3 dimensions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne= TSNE(n_components = 2, n_jobs=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dependent_scaled.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne_data = tsne.fit_transform(dependent_scaled[:10000])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again to plot the results we will have to make a new dataset with labels and tsne_data and the make a dataframe.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_tsne_df = pd.DataFrame(data = tsne_data, columns = ('1st','2nd'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_tsne_df['label'] = label[:10000]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stage-6 Plotting the t-SNE \n\nfinally we will see how good does t-SNE work.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.FacetGrid(new_tsne_df, hue='label', height=8) \\\n    .map(plt.scatter, '1st', '2nd') \\\n    .add_legend()\nplt.show()\n# and BINGO!!","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see in compare to PCA t-SNE has done tarrific work in dimensanality reduncation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne3d = TSNE(n_components = 3, n_jobs = -1).fit_transform(dependent_scaled[:10000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne_df = pd.DataFrame(tsne3d, columns=['1st', '2nd', '3rd'])\ntsne_df['label'] = label[:10000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter_3d(tsne_df, x='1st', y='2nd', z='3rd', color='label', title='TSNE 3-D')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Amazing results. We could easily see the different layers of clysters.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Hope you liked it.\nIf found useful please upvote 👍👍\n\nHere is a link for deeper visualisation for t-SNE, I recommend you visit : https://distill.pub/2016/misread-tsne/ with this you will be able to tune TSNE for better results.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}