{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Working with MNIST data and Visualize using t-SNE\n\n## Background\nMNIST data is most famous dataset available for us which has 60,000 samples of hand-written digits (0 to 9). Out of which, 42,000 in the training set and 28,000 is testing test. \nThe digits are already converted to vectors of 784 data points. Each data point will  be considered as feature in this dataset\n\n\n## Problem statement\n\n### Higher Dimension\nIn this kernel I would like to focus on visualization of data points of these samples more than actual classifications. As we all know the task of representation of datapoints with features more than 2 or 3 is not new. It is commom problem to plot the graphs with higher dimentional features. Here we have 784 features. \n\n### Expensive TSNE Computation\nWhoever working with SKLearn might have already figured out, TSNE computation is very expensive. For example, fitting 10,000 MNIST data with perplexity=30 and 1000 iteration would take around 3/4 min to complete. \n\n## Available solutions \nBest solution to represent the higher dimensions data is to perform Dimensionality reduction exercise. Following are solutions available\n\n1. **Feature Eliminations**: Each and every feature is removed and/or added to the solution and check the error results. Which ever feature had least positive impact on results or most negative impact, are eliminated from the set of features. \n\n2. **Factor Analysis**: Among the features, check the correlations between the features themselves. If we find correlated features, we can keep one among them, rest can be removed \n\n3. **Principal Component Analysis PCA**:  Most of common approach is PCA, where, we can project the data points orthogonally on principal component and reduce dimensions\n\n4. **T Distributed Stochastic Neighbourhood Embedding (T-SNE)**: This is most advanced and latest technology where data points are embedded on lower dimesion by keeping local topology and not concerning with gloabl configuration. Also this is similar to graph based technic. \n\n### GPU Solution\nTo tackle the problem of slow TSNE computation, we need to find a library which uses GPU effectively. One of the best available library which uses GPU for tsne computation is  ```tsnecuda``` https://github.com/CannyLab/tsne-cuda. However, Kaggle does not comes with tsnecuda and other pre-requisites installed in GPU enabled kernels. This notebook helps to install required libraries and pre-prequisites for running ```tsnecuda```\n\n\n## Basic checks\nLet us practice on TSNE techniche in this notebook and try to reduce 784 dimensions of MNIST to 2 dimensions and plot it on 2D graphics. Also let us perform those tsne computations using GPU using ```tsnecuda``` library"},{"metadata":{},"cell_type":"markdown","source":"First of all, let us see if GPU is present"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!nvidia-smi","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next check what is the version of cuda in this GPU "},{"metadata":{"trusted":true},"cell_type":"code","source":"!cat /usr/local/cuda/version.txt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Installation of Pre-requisites\n\n1. One of the pre-requisite library for tsnecuda to run is ```faiss```. Let us isntall the same for the version of *cuda*"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Passing Y as input while conda asks for confirmation, we use yes command\n!yes Y | conda install faiss-gpu cudatoolkit=10.0 -c pytorch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Now let us install *tsnecuda* from the sources\nAlso we found *libfaiss.so* was not found while running, but this file comes as part of sources. So, we move that to ```/usr/local/cuda/lib64```"},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget https://anaconda.org/CannyLab/tsnecuda/2.1.0/download/linux-64/tsnecuda-2.1.0-cuda100.tar.bz2\n!tar xvjf tsnecuda-2.1.0-cuda100.tar.bz2 --wildcards 'lib/*'\n!tar xvjf tsnecuda-2.1.0-cuda100.tar.bz2 --wildcards 'site-packages/*'\n!cp -r site-packages/* /opt/conda/lib/python3.6/site-packages/\n# !export LD_LIBRARY_PATH=\"/kaggle/working/lib/\" \n!cp /kaggle/working/lib/libfaiss.so /usr/local/cuda/lib64/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. We found another library missing, *openblas*, we install that now."},{"metadata":{"trusted":true},"cell_type":"code","source":"!apt search openblas\n!yes Y | apt install libopenblas-dev \n# !printf '%s\\n' 0 | update-alternatives --config libblas.so.3 << 0\n# !apt-get install libopenblas-dev \n!rm -rf  ./*","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Start the main objective of notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom keras.datasets import mnist\nimport matplotlib.pyplot as plt\nfrom tsnecuda import TSNE\nfrom sklearn.decomposition import PCA\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\nX_train=X_train.reshape(60000,-1)\nX_test=X_test.reshape(len(X_test),-1)\nX_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\nX_train=X_train.reshape(60000,-1)/255\nX_test=X_test.reshape(len(X_test),-1)/255\nX=np.concatenate((X_train, X_test), axis=0)\ny=np.concatenate((y_train, y_test), axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X[1116].reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 接下來使用t-SNE做降維，sklearn t-SNE中較重要的參數:\n- n_components: 降維之後的維度\n- perpexity: 最佳化過程中考慮鄰近點的多寡，default 30，原始paper建議5-50\n- n_iter: 迭代次數，預設1000"},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne = TSNE(n_components=2, perplexity=42, n_iter=200000)\ntrain_reduced = tsne.fit_transform(X_train)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.scatter(train_reduced[:, 0], train_reduced[:, 1], c=y_train, alpha=0.5,\n            cmap=plt.cm.get_cmap('nipy_spectral', 10))\n\nplt.colorbar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XGBoost(regression) Fit t-SNE model\n> 這一步驟是學出一個模型可以直接將784維的(input)資料預測t-sne過後的2D資料。"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.multioutput import MultiOutputRegressor\nimport xgboost\n\nxgb = xgboost.XGBRegressor(colsample_bytree=0.4,\n                 gamma=0,                 \n                 learning_rate=0.09,\n                 max_depth=6,\n                 min_child_weight=1.5,\n                 n_estimators=5000,                                                                    \n                 reg_alpha=0.75,\n                 reg_lambda=0.45,\n                 subsample=0.6,\n                 seed=42,\n                 objective ='reg:squarederror',\n                 tree_method='gpu_hist',\n                 predictor='cpu_predictor')\nxgbModel=MultiOutputRegressor(xgb)\n\nxgbModel.fit(X_train, train_reduced)\ntrainPred=xgbModel.predict(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 內部測試"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import mean_absolute_error\nfrom math import sqrt\nprint(\"Score: \",xgbModel.score(X_train, train_reduced))\nprint(\"MAE: \",mean_absolute_error(trainPred,train_reduced))\nprint(\"MSE: \",(mean_squared_error(trainPred,train_reduced)))\nprint(\"RMSE: \",sqrt(mean_squared_error(trainPred,train_reduced)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.scatter(trainPred[:, 0], trainPred[:, 1], c=y_train, alpha=0.5,\n            cmap=plt.cm.get_cmap('nipy_spectral', 10))\n\nplt.colorbar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 外部測試"},{"metadata":{"trusted":true},"cell_type":"code","source":"testPred=xgbModel.predict(X_test)\nplt.figure(figsize=(8,6))\nplt.scatter(testPred[:, 0], testPred[:, 1], c=y_test, alpha=0.5,\n            cmap=plt.cm.get_cmap('nipy_spectral', 10))\n\nplt.colorbar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 儲存XGBoost(regression) Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\nimport gzip\nwith gzip.GzipFile('./xgb(regression)-42-12000-scale-all.pgz', 'w') as f:\n    pickle.dump(xgbModel, f)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 輸出KNN(784D->2D)預測"},{"metadata":{"trusted":true},"cell_type":"code","source":"X=np.concatenate((X_train, X_test), axis=0)\ny=np.concatenate((y_train, y_test), axis=0)\npred=xgbModel.predict(X)\n\npdData = pd.DataFrame(pred, columns = [\"x1\", \"x2\"])\npdData[\"y\"]=y\npdData.to_csv('./Result-xgb-42-12000-scale-all.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XGboost(classfication) Model\n這一部分是利用KNN所預測出來的2D資料去學習一個預測0~9數字的分類模型"},{"metadata":{"trusted":true},"cell_type":"code","source":"#讀取資料\ndata = pd.read_csv(\"./Result-xgb-42-12000-scale-all.csv\") #load the dataset\nX=data[['x1','x2']].values\ny=data[['y']].values.reshape(-1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" ### 切割訓練集與測試集"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 模型初始化與學習"},{"metadata":{"trusted":true},"cell_type":"code","source":"from xgboost import XGBClassifier\nfrom sklearn.metrics import accuracy_score\n\n# fit model no training data\nxgbModel = XGBClassifier(tree_method='gpu_hist',predictor='cpu_predictor')\nxgbModel.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# make predictions for test data\ny_pred = xgbModel.predict(X_test)\n# evaluate predictions\naccuracy = accuracy_score(y_test, y_pred)\nprint(\"Accuracy: %.2f%%\" % (accuracy * 100.0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 儲存XGboost(classfication) Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\nimport gzip\nwith gzip.GzipFile('./xgb(classfication)-42-12000-scale-all.pgz', 'w') as f:\n    pickle.dump(xgbModel, f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}