{"cells":[{"metadata":{},"cell_type":"markdown","source":"... continued from [this kernel]().\n\n**Outline**\n\n- Feature Engineering (see the [previous kernel](http://www.kaggle.com/brendanhasz/elo-feature-engineering-and-feature-selection#feature-engineering))\n- Feature Aggregations (see the [previous kernel](http://www.kaggle.com/brendanhasz/elo-feature-engineering-and-feature-selection#feature-aggregations))\n- [Feature Selection](#feature-selection)\n  - [Mutual Information](#mutual-information)\n  - [Permutation-based Feature Importance](#permutation-based-feature-importance)\n- [Conclusion](#conclusion)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\n\nfrom scipy.stats import spearmanr\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.impute import SimpleImputer\n\nfrom catboost import CatBoostRegressor\n\n# Plot settings\n%matplotlib inline\n%config InlineBackend.figure_format = 'svg'\nsns.set()\n\n!pip install git+http://github.com/brendanhasz/dsutils.git\nfrom dsutils.encoding import one_hot_encode\nfrom dsutils.encoding import TargetEncoderCV\nfrom dsutils.printing import print_table\nfrom dsutils.evaluation import permutation_importance_cv\nfrom dsutils.evaluation import plot_permutation_importance","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load data containing all the features\nfname = '../input/elo-feature-engineering-and-feature-selection/card_features_all.feather'\ncards = pd.read_feather(fname)\ncards.set_index('card_id', inplace=True)\n\n# Test data\ntest = cards['target'].isnull()\nX_test = cards[test].copy()\ndel X_test['target']\n\n# Training data\ny_train = cards.loc[~test, 'target'].copy()\nX_train = cards[~test].copy()\ndel X_train['target']\n\n# Clean up\ndel cards\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Selection\n\nMachine learning models don't generally perform well when they're given a huge number of features, many of which are not very informative.  The more superfluous features we give our model to train on, the more likely it is to overfit!  To prune out features which could confuse our predictive model, we'll perform some feature selection. \n\nIdeally, we'd fit our model a bunch of different times, using every possible different combination of features, and use the set of features which gives the best cross-validated results. But, there are a few problems with that approach.  First, that would lead to overfitting to the training data.  Second, and perhaps even more importantly, it would take forever.\n\nThere are a bunch of different ways to perform feature selection in a less exhaustive, but more expedient manner.  Forward selection, backward selection, selecting features based on their correlation with the target variable, and \"embedded\" methods such as Lasso regressions (where the model itself performs feature selection during training) are all options.  However, here we'll use two different methods: the mutual information between each feature and the target variable, and the permutation-based feature importance of each feature.\n\n\n### Mutual Information\n\nTo get some idea of how well each feature corresponds to the target (the loyalty score), we can compute the [mutual information](https://en.wikipedia.org/wiki/Mutual_information) between each feature and the target. Let's make a function to compute the mutual information between two vectors."},{"metadata":{"trusted":true},"cell_type":"code","source":"def mutual_information(xi, yi, res=20):\n    \"\"\"Compute the mutual information between two vectors\"\"\"\n    ix = ~(np.isnan(xi) | np.isinf(xi) | np.isnan(yi) | np.isinf(yi))\n    x = xi[ix]\n    y = yi[ix]\n    N, xe, ye = np.histogram2d(x, y, res)\n    Nx, _ = np.histogram(x, xe)\n    Ny, _ = np.histogram(y, ye)\n    N = N / len(x) #normalize\n    Nx = Nx / len(x)\n    Ny = Ny / len(y)\n    Ni = np.outer(Nx, Ny)\n    Ni[Ni == 0] = np.nan\n    N[N == 0] = np.nan\n    return np.nansum(N * np.log(N / Ni))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The mutual information represents the amount of information that can be gained about one variable by knowing the value of some other vairable.  Obviously this is very relevant to the task of feature selection: we want to choose features which knowing the value of will give us as much information as possible about the target variable.\n\nPractically speaking, the nice thing about using mutual information instead of, say, the correlation coefficient, is that it is sensitive to nonlinear relationships.  We'll be using nonlinear predictive models (like gradient boosted decision trees), and so we don't want to limit the features we select to be only ones which have a linear relationship to the target variable.  Notice how the sin-like relationship in the middle plot below has a high mutual information, but not a great correlation coefficient."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show mutual information vs correlation\nx = 5*np.random.randn(1000)\ny = [x + np.random.randn(1000),\n     2*np.sin(x) + np.random.randn(1000),\n     x + 10*np.random.randn(1000)]\nplt.figure(figsize=(10, 4))\nfor i in range(3):    \n    plt.subplot(1, 3, i+1)\n    plt.plot(x, y[i], '.')\n    rho, _ = spearmanr(x, y[i])\n    plt.title('Mutual info: %0.3f\\nCorr coeff: %0.3f'\n              % (mutual_information(x, y[i]), rho))\n    plt.gca().tick_params(labelbottom=False, labelleft=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll use the mutual information of the quantile-transformed aggregation scores (just so outliers don't mess up the mutual information calculation).  So, we'll need a function to perform the [quantile transform](https://en.wikipedia.org/wiki/Quantile_normalization), and one to compute the mutual information after applying the quantile transform:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def quantile_transform(v, res=101):\n    \"\"\"Quantile-transform a vector to lie between 0 and 1\"\"\"\n    x = np.linspace(0, 100, res)\n    prcs = np.nanpercentile(v, x)\n    return np.interp(v, prcs, x/100.0)\n    \n    \ndef q_mut_info(x, y):\n    \"\"\"Mutual information between quantile-transformed vectors\"\"\"\n    return mutual_information(quantile_transform(x),\n                              quantile_transform(y))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can compute the mutual information between each feature and the loyalty score."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Compute the mutual information\ncols = []\nmis = []\nfor col in X_train:\n    mi = q_mut_info(X_train[col], y_train)\n    cols.append(col)\n    mis.append(mi)\n    \n# Print mut info of each feature\nprint_table(['Column', 'Mutual_Information'],\n            [cols, mis])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's only bother keeping the features with the top 200 mutual information scores."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create DataFrame with scores\nmi_df = pd.DataFrame()\nmi_df['Column'] = cols\nmi_df['mut_info'] = mis\n\n# Sort by mutual information\nmi_df = mi_df.sort_values('mut_info', ascending=False)\ntop200 = mi_df.iloc[:200,:]\ntop200 = top200['Column'].tolist()\n\n# Keep only top 200 columns\nX_train = X_train[top200]\nX_test = X_test[top200]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"permutation-based-feature-importance\"></a>\n### Permutation-based Feature Importance\n\nA different way to select features is to try and train a model using *all* the features, and then determine how heavily the model's performance depends on the features.  But, we'll need to use a model which can handle a lot of features without overfitting too badly (i.e., an unregularized linear regression wouldn't be a good idea here).  So, we'll use a gradient boosted decision tree, specifically [CatBoost](http://catboost.ai/).  \n\nLet's create a data processing and prediction pipeline.  First, we'll target encode the categorical columns (basically just set each category to the mean target value for samples having that category - see my [post on target encoding](https://brendanhasz.github.io/2019/03/04/target-encoding.html)).  Then, we can normalize the data and impute missing data (we'll just fill in missing data with the median of the column).  Finally, we can use CatBoost to predict the loyalty scores from the features we've engineered."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Regression pipeline\ncat_cols = [c for c in X_train if 'mode' in c] \nreg_pipeline = Pipeline([\n    ('target_encoder', TargetEncoderCV(cols=cat_cols)),\n    ('scaler', RobustScaler()),\n    ('imputer', SimpleImputer(strategy='median')),\n    ('regressor', CatBoostRegressor(loss_function='RMSE', \n                                    verbose=False))\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can measure how heavily the model depends on various features by using permutation-based feature importance.  Basically, we train the model on all the data, and then measure its error after shuffling each row.  When the model's error increases a lot after shuffling a row, that means that the feature which was shuffled was important for the model's predictions.\n\nThe advantage of permutation-based feature importance is that it gives a super-clear view and a single score as to how important each feature is.  The downside is that this score is intrinsically linked to the model.  Whereas computing the mutual information between the features and the target only depends on the data, permutation-based feature importance scores depend on the data, the model being used, and the interaction between the two.  If your model can't fit the data very well, your permutation scores will be garbage!\n\nLuckily CatBoost nearly always does a pretty good job of prediction, even in the face of lots of features!  So, let's compute the permutation-based feature importance for each feature (the complete code is [on my GitHub](https://github.com/brendanhasz/dsutils/blob/master/src/dsutils/evaluation.py#L126))."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n\n# Compute the cross-validated feature importance\nimp_df = permutation_importance_cv(\n    X_train, y_train, reg_pipeline, 'rmse', n_splits=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, we can plot the importance scores for each feature.  These scores are just the difference between the model's error with no shuffled features and the error with the feature of interest shuffled.  So, larger scores correspond to features which the model needs to have a low error."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the feature importances\nplt.figure(figsize=(8, 100))\nplot_permutation_importance(imp_df)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we'll want to save the features so that we can use them to train a model to predict the loyalty scores.  Let's save the top 100 most important features to a [feather](https://github.com/wesm/feather) file, so that we can quickly load them back in when we do the modeling.  First though, we need to figure out which features *are* the ones with the best importance scores."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get top 100 most important features\ndf = pd.melt(imp_df, var_name='Feature', value_name='Importance')\ndfg = (df.groupby(['Feature'])['Importance']\n       .aggregate(np.mean)\n       .reset_index()\n       .sort_values('Importance', ascending=False))\ntop100 = dfg['Feature'][:100].tolist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, we can save those features (and the corresponding target variable!) to a feather file."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save file w/ 100 most important features\ncards = pd.concat([X_train[top100], X_test[top100]])\ncards['target'] = y_train\ncards.reset_index(inplace=True)\ncards.to_feather('card_features_top100.feather')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"conclusion\"></a>\n## Conclusion\n\nNow that we've engineered features for each card account, the next thing to do is create models to predict the target value from those features.  In [the next kernel](https://www.kaggle.com/brendanhasz/elo-modeling), we'll try different modeling techniques to see which gives the best predictions."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}