{"cells":[{"metadata":{"_uuid":"02a3e45c5f02234fb0f80f904530745f2521ec24"},"cell_type":"markdown","source":"# Feature selecture using target permutation\n\nThis notebook is a straightforward adaptation of [Olivier's kernel](https://www.kaggle.com/ogrellier/feature-selection-with-null-importances), where Olivier proposes a methodology to select the most relevant features of the model. As outlined by [Peter Hurford](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73937), just keeping the right features may help you model to score better. To obtain exhaustive details on the method's implementation, you should refer to the original kernel. **By the way, if you feel like upvoting this kernel, please upvote first the original one !**\n\n### Notebook  Content\n1. [Creating a scoring function](#1)\n1. [Build the benchmark for feature importance](#2)\n1. [Display distribution examples](#3)\n1. [Score features](#4)\n1. [Save data](#5)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"72224d3f98b149d5906423f65de2d2905bb1051e"},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error\nimport gc\n%matplotlib inline\nwarnings.simplefilter(action='ignore', category=FutureWarning)\ngc.enable()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d91bcd66b56e0932b32ac14666ef2093d48a98d9"},"cell_type":"markdown","source":"First, we load the data, which has been pre-procesed in [another kernel](https://www.kaggle.com/fabiendaniel/elo-world):"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/elo-world/train.csv\", index_col=0)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"## 1. Create a scoring function"},{"metadata":{"trusted":true,"_uuid":"c02fba591fb8d4cce6d26fcb1840491d0db420fd"},"cell_type":"code","source":"def get_feature_importances(data, shuffle, seed=None):\n    # Gather real features\n    train_features = [f for f in data if f not in ['target', 'card_id', 'first_active_month']]\n    categorical_feats = [c for c in train_features if 'feature_' in c]\n    # Go over fold and keep track of CV score (train and valid) and feature importances\n    \n    # Shuffle target if required\n    y = data['target'].copy()\n    if shuffle:\n        # Here you could as well use a binomial distribution\n        y = data['target'].copy().sample(frac=1.0)\n    \n    # Fit LightGBM in RF mode, yes it's quicker than sklearn RandomForest\n    dtrain = lgb.Dataset(data[train_features], y, free_raw_data=False, silent=True)\n    lgb_params = {\n        'num_leaves': 129,\n        'min_data_in_leaf': 148, \n        'objective':'regression',\n        'max_depth': 9,\n        'learning_rate': 0.005,\n        \"min_child_samples\": 24,\n        \"boosting\": \"gbdt\",\n        \"feature_fraction\": 0.7202,\n        \"bagging_freq\": 1,\n        \"bagging_fraction\": 0.8125 ,\n        \"bagging_seed\": 11,\n        \"metric\": 'rmse',\n        \"lambda_l1\": 0.3468,\n        \"random_state\": 133,\n        \"verbosity\": -1\n    }\n    \n    # Fit the model\n    clf = lgb.train(params=lgb_params,\n                    train_set=dtrain,\n                    num_boost_round=850,\n                   # categorical_feature=categorical_feats\n                   )\n\n    # Get feature importances\n    imp_df = pd.DataFrame()\n    imp_df[\"feature\"] = list(train_features)\n    imp_df[\"importance_gain\"] = clf.feature_importance(importance_type='gain')\n    imp_df[\"importance_split\"] = clf.feature_importance(importance_type='split')\n    imp_df['trn_score'] = mean_squared_error(clf.predict(data[train_features]), y)**0.5\n    \n    return imp_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc9f900dfd0f7a9d66f667b5fab97e3909d1dd69"},"cell_type":"markdown","source":"## 2. Build the benchmark for feature importance"},{"metadata":{"trusted":true,"_uuid":"dd3013ef82533daee46466e8881c823551e04e1f"},"cell_type":"code","source":"# Seed the unexpected randomness of this world\nnp.random.seed(123)\n# Get the actual importance, i.e. without shuffling\nactual_imp_df = get_feature_importances(data=train, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"144870952be97135b13f9455774452884af3a324"},"cell_type":"code","source":"actual_imp_df.sort_values('importance_gain', ascending=False)[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f06986b86e56a6d1d0ff0f9285feba2d864495f"},"cell_type":"code","source":"null_imp_df = pd.DataFrame()\nnb_runs = 100\nimport time\nstart = time.time()\ndsp = ''\nfor i in range(nb_runs):\n    # Get current run importances\n    imp_df = get_feature_importances(data=train, shuffle=True)\n    imp_df['run'] = i + 1 \n    # Concat the latest importances with the old ones\n    null_imp_df = pd.concat([null_imp_df, imp_df], axis=0)\n    # Erase previous message\n    for l in range(len(dsp)):\n        print('\\b', end='', flush=True)\n    # Display current run and time used\n    spent = (time.time() - start) / 60\n    dsp = 'Done with %4d of %4d (Spent %5.1f min)' % (i + 1, nb_runs, spent)\n    print(dsp, end='', flush=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1375ede6b7c9b6ed20b65788ea135528f589c6ef"},"cell_type":"markdown","source":"## 3. Display distribution examples\n\nA few plots are better than any words"},{"metadata":{"trusted":true,"_uuid":"035963dd118761a93eb4bc62c98c120e251ed4c6"},"cell_type":"code","source":"def display_distributions(actual_imp_df_, null_imp_df_, feature_):\n    plt.figure(figsize=(13, 6))\n    gs = gridspec.GridSpec(1, 2)\n    # Plot Split importances\n    ax = plt.subplot(gs[0, 0])\n    a = ax.hist(null_imp_df_.loc[null_imp_df_['feature'] == feature_, 'importance_split'].values,\n                label='Null importances')\n    ax.vlines(x=actual_imp_df_.loc[actual_imp_df_['feature'] == feature_, 'importance_split'].mean(), \n               ymin=0, ymax=np.max(a[0]), color='r',linewidth=10, label='Real Target')\n    ax.legend()\n    ax.set_title('Split Importance of %s' % feature_.upper(), fontweight='bold')\n    plt.xlabel('Null Importance (split) Distribution for %s ' % feature_.upper())\n    # Plot Gain importances\n    ax = plt.subplot(gs[0, 1])\n    a = ax.hist(null_imp_df_.loc[null_imp_df_['feature'] == feature_, 'importance_gain'].values, label='Null importances')\n    ax.vlines(x=actual_imp_df_.loc[actual_imp_df_['feature'] == feature_, 'importance_gain'].mean(), \n               ymin=0, ymax=np.max(a[0]), color='r',linewidth=10, label='Real Target')\n    ax.legend()\n    ax.set_title('Gain Importance of %s' % feature_.upper(), fontweight='bold')\n    plt.xlabel('Null Importance (gain) Distribution for %s ' % feature_.upper())\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fda2d1925d3fc2467fbfd946ed667926d7557d1"},"cell_type":"code","source":"actual_imp_df.sort_values('importance_gain', ascending=False)[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e6857791d01201566391a5033486c0c5347bd461"},"cell_type":"code","source":"display_distributions(actual_imp_df_=actual_imp_df, null_imp_df_=null_imp_df, feature_='new_installments_min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d623d11e5b44b3c0a9bf3aa16e04192f2d331e4f"},"cell_type":"code","source":"display_distributions(actual_imp_df_=actual_imp_df, null_imp_df_=null_imp_df, feature_='new_month_lag_min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2d6d61bfc4b2bf51cb48890833edbeb05d8ff1c5"},"cell_type":"code","source":"display_distributions(actual_imp_df_=actual_imp_df, null_imp_df_=null_imp_df, feature_='auth_category_1_sum')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"222c4028d990c103bd490e9183991adec500faf3"},"cell_type":"code","source":"display_distributions(actual_imp_df_=actual_imp_df, null_imp_df_=null_imp_df, feature_='hist_month_diff_mean')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f90300cf0ecbf96f34c33e9f791f41095a85c0e"},"cell_type":"markdown","source":"From the above plot, **as stated by Olivier**, the power of the exposed feature selection method is demonstrated. In particular it is well known that :\n - Any feature sufficient variance can be used and made sense of by tree models. You can always find splits that help scoring better\n - Correlated features have decaying importances once one of them is used by the model. The chosen feature will have strong importance and its correlated suite will have decaying importances\n \n The current method allows to :\n  - Drop high variance features if they are not really related to the target\n  - Remove the decaying factor on correlated features, showing their real importance (or unbiased importance)"},{"metadata":{"_uuid":"941fa5fd79eeebf2d7217d177905364d83b38b15"},"cell_type":"markdown","source":"## 4. Score features\n\nThere are several ways to score features : \n - Compute the number of samples in the actual importances that are away from the null importances recorded distribution.\n - Compute ratios like Actual / Null Max, Actual  / Null Mean,  Actual Mean / Null Max\n \nHere, **following Olivier,** we use the log actual feature importance divided by the 75 percentile of null distribution."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"fdc03d0d315c9f92c74b7cfd9361b9201af342ee"},"cell_type":"code","source":"feature_scores = []\nmax_features = 300\nfor _f in actual_imp_df['feature'].unique():\n    f_null_imps_gain = null_imp_df.loc[null_imp_df['feature'] == _f, 'importance_gain'].values\n    f_act_imps_gain = actual_imp_df.loc[actual_imp_df['feature'] == _f, 'importance_gain'].mean()\n    gain_score = np.log(1e-10 + f_act_imps_gain / (1 + np.percentile(f_null_imps_gain, 75)))  \n    f_null_imps_split = null_imp_df.loc[null_imp_df['feature'] == _f, 'importance_split'].values\n    f_act_imps_split = actual_imp_df.loc[actual_imp_df['feature'] == _f, 'importance_split'].mean()\n    split_score = np.log(1e-10 + f_act_imps_split / (1 + np.percentile(f_null_imps_split, 75)))  \n    feature_scores.append((_f, split_score, gain_score))\n\nscores_df = pd.DataFrame(feature_scores, columns=['feature', 'split_score', 'gain_score'])\n\nplt.figure(figsize=(16, 25))\ngs = gridspec.GridSpec(1, 2)\n# Plot Split importances\nax = plt.subplot(gs[0, 0])\nsns.barplot(x='split_score', y='feature', data=scores_df.sort_values('split_score', ascending=False).iloc[0:max_features], ax=ax)\nax.set_title('Feature scores wrt split importances', fontweight='bold', fontsize=14)\n# Plot Gain importances\nax = plt.subplot(gs[0, 1])\nsns.barplot(x='gain_score', y='feature', data=scores_df.sort_values('gain_score', ascending=False).iloc[0:max_features], ax=ax)\nax.set_title('Feature scores wrt gain importances', fontweight='bold', fontsize=14)\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b134e76d266cb563d2d7c32fd591f9332d2f6545"},"cell_type":"markdown","source":"## 4. Save the data"},{"metadata":{"trusted":true,"_uuid":"d3b89a0da637bcbb42c052204d5e841b0dde2d2b"},"cell_type":"code","source":"null_imp_df.to_csv('null_importances_distribution_rf.csv')\nactual_imp_df.to_csv('actual_importances_ditribution_rf.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6c9990a6ff1b6f0a4264d532f727b17d89b9735"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}