{"cells":[{"metadata":{"_kg_hide-input":true,"_uuid":"1f3ce828d92a81de0587edc498a4baa462dbb272"},"cell_type":"markdown","source":"This is a kernel to explore the outliers for the [Elo Merchant Category Recommendation Competition](https://www.kaggle.com/c/elo-merchant-category-recommendation) . It can also serve as a reall good introduction to **dimensionality reduction** for visualisation and model training purposes."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.decomposition import PCA,KernelPCA\nfrom sklearn.svm import SVC\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import KFold, train_test_split\nfrom sklearn import metrics\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4d27c17fd7f7b612f800c1d2adfe4106f1645d4"},"cell_type":"markdown","source":"Most of the features are obtained from [You're Going to Want More Categories](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737) which I believe is a greate starter kernel for anyone wanting to join in the fun! I have also added a few features of my own from the merchants dataset which seemed largely unexplored but to no avail!~"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"complete_features = pd.read_csv(\"../input/cached-features/complete_features.csv\", parse_dates=[\"first_active_month\"])\ncomplete_features_test = pd.read_csv(\"../input/cached-features/complete_features_test.csv\", parse_dates=[\"first_active_month\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e04c5f31232b5580c3801c08b347270c58249656"},"cell_type":"code","source":"target = complete_features['target']\ndrops = ['card_id', 'first_active_month', 'target']\n# to_remove = [c for c in complete_features if 'new' in c]\nuse_cols = [c for c in complete_features.columns if c not in drops]\nfeatures = list(complete_features[use_cols].columns)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a6d611859acc433057828c2b820690e4f658966"},"cell_type":"markdown","source":"So the process here will be pretty simple I am using PCA ([Principal Components Analysis ](https://en.wikipedia.org/wiki/Principal_component_analysis)) to reduce the dimensionality to the first two Principal Components. A good way to think about this process is that it reduces our features into two uncorrelated features which are ideal for visualisations. These two features (=projections) carry the most variance of our dataset. Going a bit meta, PCA finds linear projections in our feature space to which the variance is maximized."},{"metadata":{"trusted":true,"_uuid":"2e0c24a8ae44e622cb4d52b89434bc6fa8e3c2f8"},"cell_type":"code","source":"complete_features[features] = complete_features[features].fillna(0)\ncomplete_features_test[features] = complete_features_test[features].fillna(0)\npca = PCA(n_components=2)\nreduced_data = pca.fit_transform(complete_features[features])\nprint(reduced_data.shape)\nprint(pca.explained_variance_ratio_)\nplt.plot(reduced_data[:,0],reduced_data[:,1],'.b')\nplt.show()\nreduced_df = pd.DataFrame({'x' : reduced_data[:,0], 'y' : reduced_data[:,1], 'c': complete_features.target.apply(lambda x : x<-20).astype(int)})\nplt.scatter(x=reduced_df['x'], y=reduced_df['y'],c=reduced_df['c'], s=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b3fe6669fd0aa673a0208a3c1cd173fed486bca"},"cell_type":"markdown","source":"So this visualisation here needs a bit of thinking. We first see that data points are spread almost uniformly across this 2D space. Most importantly that our outliers are not easily distinguishable as they are spread across the whole space and not concentrated anywhere. I make the following observations :\n* We are potentially lacking signals that the top competitors have identified and are able to produce a better distinction between the outliers and non-outliers as the outliers are essentially what produce most of the error\n* We have added too much noise with our features and we need to eliminate a number of them\n\n**BUT** a drawback of PCA is that it performs LINEAR projections so let's try using something differen't called Kernel PCA ([Kernel Principal Components Analysis](https://en.wikipedia.org/wiki/Kernel_principal_component_analysis) which takes advantages of the kernel trick to produce non-linear projections. However, as computing the Kernel Matrix is expensive we can only do this for a sample of our data. "},{"metadata":{"trusted":true,"_uuid":"505749af21ea30518e1870430c1d6800c1a5f047"},"cell_type":"code","source":"kpca = KernelPCA(n_components=2, kernel='rbf')\nnp.random.seed(3)\ns = pd.concat([complete_features[complete_features['target']>-20].sample(2207),complete_features[complete_features['target']<-20]]).reset_index(drop='index')\n\nreduced_data_d = kpca.fit_transform(s[features])\nprint(reduced_data_d.shape)\nplt.plot(reduced_data_d[:,0],reduced_data_d[:,1],'.b')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"29729ffe5e70e61491b95c224aec57b4c21df9e8"},"cell_type":"code","source":"reduced_df_d = pd.DataFrame({'x' : reduced_data_d[:,0], 'y' : reduced_data_d[:,1], 'c': s.target.apply(lambda x : x<-20).astype(int)})\nplt.scatter(x=reduced_df_d['x'], y=reduced_df_d['y'],c=reduced_df_d['c'], s=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67ce539e54de04130869dfcb37d5f4acd034c51d"},"cell_type":"markdown","source":"Now this seems a bit better but it is still noisy. Let's try and build some **classifiers** to see how well we can classifiy outliers. I used SVC and RandomForests which to get a sense of what is going on"},{"metadata":{"trusted":true,"_uuid":"ff59281dd8b0a432c64a1fd6741f1ae8078fba53"},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(\n    reduced_df_d[['x','y']].values, s.target.apply(lambda x : -1 if x<-20 else 1).values, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65c102fb871d5fd917cfaa9466ed851b49d0d436"},"cell_type":"code","source":"svm = SVC(C=1,gamma=1.3, tol=0.0001, kernel='rbf', degree=5)\nsvm.fit(X_train, y_train)\nrf = RandomForestClassifier(n_estimators=100,max_depth=5)\nrf.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6228284c0323db6f774e58a363302fb6d39b3ae5"},"cell_type":"code","source":"\ndef make_meshgrid(x, y, h=.002):\n    \"\"\"Create a mesh of points to plot in\n\n    Parameters\n    ----------\n    x: data to base x-axis meshgrid on\n    y: data to base y-axis meshgrid on\n    h: stepsize for meshgrid, optional\n\n    Returns\n    -------\n    xx, yy : ndarray\n    \"\"\"\n    x_min, x_max = x.min() - 0.1, x.max() + 0.1\n    y_min, y_max = y.min() - 0.1, y.max() + 0.1\n    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n                         np.arange(y_min, y_max, h))\n    return xx, yy\n\n\ndef plot_contours(ax, clf, xx, yy, **params):\n    \"\"\"Plot the decision boundaries for a classifier.\n\n    Parameters\n    ----------\n    ax: matplotlib axes object\n    clf: a classifier\n    xx: meshgrid ndarray\n    yy: meshgrid ndarray\n    params: dictionary of params to pass to contourf, optional\n    \"\"\"\n    if not isinstance(clf, RandomForestClassifier):\n        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n        Z = Z.reshape(xx.shape)\n        out = ax.contourf(xx, yy, Z, **params)\n        return out\n    else:\n        for tree in clf.estimators_:\n            Z = tree.predict(np.c_[xx.ravel(), yy.ravel()])\n            Z = Z.reshape(xx.shape)\n            plt.contourf(xx, yy, Z, alpha=1. /len(clf.estimators_), cmap=plt.cm.coolwarm)\n\n    \n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"057395d0bde4b874965f5defe98c69854b2b1b2a"},"cell_type":"markdown","source":"Let's plot the boundaries for SVMs"},{"metadata":{"trusted":true,"_uuid":"b315bf87c4f0366c040fb518de2c119504b5a863","scrolled":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,10))\nX0, X1 = X_train[:,0],X_train[:,1]\nxx, yy = make_meshgrid(X0, X1)\n\nplot_contours(ax, svm, xx, yy,\n                  cmap=plt.cm.coolwarm, alpha=0.8)\nax.scatter(X0, X1, c=y_train, cmap=plt.cm.coolwarm, s=25, edgecolors='k')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"55b29364fef428392eabae764cdddbebde1b00fd"},"cell_type":"code","source":"print(metrics.classification_report(y_train, svm.predict(X_train)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58dba6e48e4d1963af965c192a5b4f95cbd027fe"},"cell_type":"markdown","source":"Hmmm, SVC is not doing a pretty good job 0 precision and 0 recall for the outlier class"},{"metadata":{"trusted":true,"_uuid":"cc32c7efa81f26cb66467a629b103994c466f001"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,10))\n\nplot_contours(ax, rf, xx, yy,\n                  cmap=plt.cm.coolwarm, alpha=1.0 / len(rf.estimators_))\nax.scatter(X0, X1, c=y_train, cmap=plt.cm.coolwarm, s=25, edgecolors='k')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ab2310887c43483afd753d44394afe22cdb08dfe"},"cell_type":"code","source":"print(metrics.classification_report(y_train, rf.predict(X_train)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b0dfb1bc598391d89d684d33068759d8f08564b2"},"cell_type":"markdown","source":"RandomForest are much better, still not as high as we would like but is still good enough. We can now use this classifier by first transforming the rest of our data using KPCA and adding an extra feature called 'outlier_proba'. I will be extending this kernel later on to do this."},{"metadata":{"trusted":true,"_uuid":"e411de22f82d2f064bb25a8f326f5d2608c92bcb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}