{"cells":[{"metadata":{},"cell_type":"markdown","source":"[Elo](https://elo.com.br/) is a Brazillian debit and credit card brand.  They offer credit and prepaid transactions, and have paired up with merchants in order offer promotions to cardholders.  In order to offer more relevant and personalized promotions, in a [recent Kaggle competition](https://www.kaggle.com/c/elo-merchant-category-recommendation), Elo challenged Kagglers to predict customer loyalty based on transaction history.  Presumably they plan to use a loyalty-predicting model in order to determine what promotions to offer to customers based on how certain offers are predicted to effect card owners' card loyalty.\n\nIn a [previous kernel](https://www.kaggle.com/brendanhasz/elo-data-cleaning-and-eda), we loaded and cleaned the data, and performed some exploratory data analysis.  In this kernel, we'll engineer new features about the card accounts, compute aggregate statistics about transactions made with each card, and then select which features to use in a predictive model.\n\n\n**Outline**\n\n- [Feature Engineering](#feature-engineering)\n- [Feature Aggregations](#feature-aggregations)\n- [Feature Selection](#feature-selection)\n  - [Mutual Information](#mutual-information)\n  - [Permutation-based Feature Importance](#permutation-based-feature-importance)\n- [Conclusion](#conclusion)\n\n\nLet's first load the packages we'll use:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\n\nfrom scipy.stats import spearmanr\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.impute import SimpleImputer\n\nfrom catboost import CatBoostRegressor\n\n# Plot settings\n%matplotlib inline\n%config InlineBackend.figure_format = 'svg'\nsns.set()\n\n!pip install git+http://github.com/brendanhasz/dsutils.git\nfrom dsutils.encoding import one_hot_encode\nfrom dsutils.encoding import TargetEncoderCV\nfrom dsutils.printing import print_table\nfrom dsutils.evaluation import permutation_importance_cv\nfrom dsutils.evaluation import plot_permutation_importance","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we can load and clean the data.  (This was done in detail in the [previous kernel](https://www.kaggle.com/brendanhasz/elo-data-cleaning-and-eda) about data cleaning and EDA.)"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Cards data\n\n# Datatypes for each column\n# no nulls in test/train except for ONE ROW in test! (first_active_month)\ndtypes = {\n  'card_id':            'str',     # 201917 unique vals\n  'target':             'float32', # -33.22 thru ~18\n  'first_active_month': 'str',     # 2011-10 thru 2018-02\n  'feature_1':          'uint8',   # 1 thru 5\n  'feature_2':          'uint8',   # 1 thru 3\n  'feature_3':          'uint8',   # 0 and 1\n}\ntrain = pd.read_csv('../input/train.csv',\n                    usecols=dtypes.keys(),\n                    dtype=dtypes)\ndel dtypes['target']\ntest = pd.read_csv('../input/test.csv',\n                   usecols=dtypes.keys(),\n                   dtype=dtypes)\n\n# Add target col to test\ntest['target'] = np.nan\n\n# Merge test and train\ncards = pd.concat([train, test])\n\ndel train, test\ngc.collect()\n\n# Convert first_active_month to datetime\ncards['first_active_month'] = pd.to_datetime(cards['first_active_month'],\n                                             format='%Y-%m')\n\n# Make card_id the index\ncards.set_index('card_id', inplace=True)\ngc.collect()\ncards.sample(10)\n\n\n# --------------------------------------------------\n\n# Merchants data\n\n# Datatypes of each column\n# (don't load cols which are in transactions data, just use those vals)\n# Nulls: NO nulls except for 13 rows in avg_sales_lag{3,6,12}\ndtypes = {\n  'merchant_id':                 'str',     # 334633 unique values\n  'merchant_group_id':           'uint32',  # 1 thru 112586 (w/ some missing, ~109k uniques)\n  'numerical_1':                 'float32', # ~ -0.06 thru ~ 183.8 (only 951 unique vals?)\n  'numerical_2':                 'float32', # roughly the same as above\n  'most_recent_sales_range':     'str',     # A, B, C, D, or E\n  'most_recent_purchases_range': 'str',     # A, B, C, D, or E\n  'avg_sales_lag3':              'float32', # most between 0 and 2, if you transform by 1/x, all but 3 are between 0 and 4\n  'avg_purchases_lag3':          'float32', # most between 0 and 2, if you transform by 1/x, all but 3 are between 0 and 4\n  'active_months_lag3':          'uint8',   # 1 to 3 \n  'avg_sales_lag6':              'float32', # similar to avg_sales_lag3\n  'avg_purchases_lag6':          'float32', # similar to avg_purchases_lag3\n  'active_months_lag6':          'uint8',   # 1 to 6\n  'avg_sales_lag12':             'float32', # similar to avg_sales_lag3\n  'avg_purchases_lag12':         'float32', # similar to avg_purchases_lag3\n  'active_months_lag12':         'uint8',   # 1 to 12\n  'category_4':                  'str',     # Y or N\n}\n\n# Load the data\nmerchants = pd.read_csv('../input/merchants.csv',\n                        usecols=dtypes.keys(),\n                        dtype=dtypes)\n\n# Map merchant_id to integer\nmerch_id_map = dict(zip(\n    merchants['merchant_id'].values,\n    merchants['merchant_id'].astype('category').cat.codes.values\n))\n\ndef preprocess_merch_data(df):\n    \n    # Convert merchant ID to numbers\n    df['merchant_id'] = df['merchant_id'].map(merch_id_map).astype('float32')\n\n    # Inverse transforms\n    inversions = [\n        'avg_sales_lag3',\n        'avg_sales_lag6',\n        'avg_sales_lag12',\n        'avg_purchases_lag3',\n        'avg_purchases_lag6',\n        'avg_purchases_lag12',\n    ]\n    for col in inversions:\n        df[col] = 1.0/df[col]\n\n    # Encode categorical columns\n    bool_map = {'Y': 1, 'N': 0}\n    five_map = {'A': 0, 'B': 1, 'C': 2, 'D': 3, 'E': 4}\n    conversions = [\n        ('category_4', bool_map, 'uint8'),\n        ('most_recent_sales_range', five_map, 'uint8'),\n        ('most_recent_purchases_range', five_map, 'uint8')\n    ]\n    for col, mapper, new_type in conversions:\n        df[col] = df[col].map(mapper).astype(new_type)\n        \n    # Clean up\n    gc.collect()\n\n# Preprocess the merchants data\npreprocess_merch_data(merchants)\n\n\n# ------------------------------------------------------\n\n# Transactions data\n\n# Datatypes of each column\n# only NaNs are in category_3, merchant_id, and category_2\ndtypes = {\n    'authorized_flag':      'str',     # Y or N\n    'card_id':              'str',     # 325540 unique values\n    'city_id':              'int16',   # -1 then 1 to 347 (is -1 supposed to be nan?)\n    'category_1':           'str',     # Y or N\n    'installments':         'int8',    # -25, then -1 thru 12 (-1 supposed to be nan?)\n    'category_3':           'str',     # A, B, C, and nan (ordinal?)\n    'merchant_category_id': 'int16',   # 2 to 891\n    'merchant_id':          'str',     # 334633 unique values and nans (164697 nans!)\n    'month_lag':            'int8',    # -13 thru 0\n    'purchase_amount':      'float32', # min: -0.746, med: -0.699, max: 11269.667\n    'purchase_date':        'str',     # YYYY-MM-DD hh:mm:ss\n    'category_2':           'float32', # 1 thru 5 and nan (ordinal?)\n    'state_id':             'int8',    # -1 then 1 thru 24\n    'subsector_id':         'int8'     # 1 thru 41\n}\n\n# Load the data\nhist_trans = pd.read_csv('../input/historical_transactions.csv', \n                         usecols=dtypes.keys(),\n                         dtype=dtypes)\nnew_trans = pd.read_csv('../input/new_merchant_transactions.csv', \n                        usecols=dtypes.keys(),\n                        dtype=dtypes)\n\ndef preprocess_trans_data(df):\n    \n    # Convert merchant_id to numbers\n    df['merchant_id'] = df['merchant_id'].map(merch_id_map).astype('float32')\n\n    # Convert purchase_date to datetime\n    df['purchase_date'] = df['purchase_date'].str.slice(0, 19)\n    df['purchase_date'] = pd.to_datetime(df['purchase_date'],\n                                         format='%Y-%m-%d %H:%M:%S')\n\n    # Encode categorical columns\n    bool_map = {'Y': 1, 'N': 0}\n    three_map = {'A': 0, 'B': 1, 'C': 2}\n    conversions = [\n        ('authorized_flag', bool_map, 'uint8'),\n        ('category_1', bool_map, 'uint8'),\n        ('category_3', three_map, 'float32'), #has NaNs so have to use float\n    ]\n    for col, mapper, new_type in conversions:\n        df[col] = df[col].map(mapper).astype(new_type)\n        \n    # Clean up\n    gc.collect()\n\n# Preprocess the transactions data\npreprocess_trans_data(hist_trans)\npreprocess_trans_data(new_trans)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"feature-engineering\"></a>\n## Feature Engineering\n\nAt this point we have four tables:\n\n* `hist_trans` - contains details about historical transactions\n* `new_trans` - contains details about newer transactions\n* `merchants` - contains information about the merchants\n* `cards` - contains information about the card accounts\n\nEventually, we'll want one large table where each row corresponds to a card account, and each column corresponds to a feature of that account.  There's not a whole lot of information in the `cards` table as is, so we'll have to engineer features of each card using the information in the other three tables.  The idea is that information about how often an individual is making transactions, when, for how much, with what merchants,  with what *types* of merchants - and so on - will be informative as to how likely that individual is to have a high loyalty score.\n\nEach transaction in `hist_trans` and `new_trans` has a `card_id`, which we can use to find all the transactions from a given card account in the `cards` dataset (there is also a `card_id` column in that table).  Similarly, each transaction also has a corresponding `merchant_id`, which we can use to look up the information about the corresponding merchant in the `merchants` dataset (that table also has a `merchant_id` column).\n\nThe first thing to do is merge the transactions datasets with the merchants dataset on the `merchant_id` column.  That is, for each transaction, we need to look up the information about the merchant which participated in that transaction in the merchants dataset, and append it to the transactions data.\n\nThe pandas merge function makes this pretty easy:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Merge transactions with merchants data\nhist_trans = pd.merge(hist_trans, merchants, on='merchant_id')\nnew_trans = pd.merge(new_trans, merchants, on='merchant_id')\n\n# Clean up\ndel merchants\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we'll want to encode some of the simpler categorical columns.  The `category_2` and `category_3` columns have only 5 and 3 unique values (and `NaN`s), so we'll simply one-hot encode them."},{"metadata":{"trusted":true},"cell_type":"code","source":"# One-hot encode category 2 and 3\ncat_cols = ['category_2', 'category_3']\nhist_trans = one_hot_encode(hist_trans,\n                            cols=cat_cols)\nnew_trans = one_hot_encode(new_trans,\n                           cols=cat_cols)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, we can create some time-based features about the purchases.  For example: the hour of the day the purchase was made, the day of the week, the week of the year, the month, whether the purchase was made on a weekend, the time of the purchase relative to when the card owner was first active, etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Time-based features for purchases\nref_date = np.datetime64('2017-09-01')\none_hour = np.timedelta64(1, 'h')\nfor df in [hist_trans, new_trans]:\n    tpd = df['purchase_date']\n    df['purchase_hour'] = tpd.dt.hour.astype('uint8')\n    df['purchase_day'] = tpd.dt.dayofweek.astype('uint8')\n    df['purchase_week'] = tpd.dt.weekofyear.astype('uint8')\n    df['purchase_dayofyear'] = tpd.dt.dayofyear.astype('uint16')\n    df['purchase_month'] = tpd.dt.month.astype('uint8')\n    df['purchase_weekend'] = (df['purchase_day'] >=5 ).astype('uint8')\n    df['purchase_time'] = ((tpd - ref_date) / one_hour).astype('float32')\n    df['ref_date'] = ((tpd - pd.to_timedelta(df['month_lag'], 'M')\n                          - ref_date ) / one_hour).astype('float32')\n\n    # Time sime first active\n    tsfa = pd.merge(df[['card_id']], \n                    cards[['first_active_month']].copy().reset_index(),\n                    on='card_id', how='left')\n    df['time_since_first_active'] = ((tpd - tsfa['first_active_month'])\n                                     / one_hour).astype('float32')\n    \n    # Clean up\n    del tsfa\n    del df['purchase_date']\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we need to convert the `first_active_month` column (a datetime) to a month.  This way at this point all of our data will be in a numerical format."},{"metadata":{"trusted":true},"cell_type":"code","source":"cards['first_active_month'] = (12*(cards['first_active_month'].dt.year-2011) + \n                               cards['first_active_month'].dt.month).astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"feature-aggregations\"></a>\n## Feature Aggregations\n\nNow we can engineer features for each card account by applying aggregation functions on the transaction data corresponding to each card.  First, we need to group the transactions by the card account which was used to make them:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group transactions by card id\nhist_trans = hist_trans.groupby('card_id', sort=False)\nnew_trans = new_trans.groupby('card_id', sort=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll also need to define some custom aggregation functions which will allow us to better extract information from the transactions.  The first is a function which computes the entropy given some categorical data.  A feature corresponding to entropy could be informative - for example, it could be that card accounts with high entropy over the merchants they use their card with are more likely to be more loyal card users than those who only use their card with a single merchant (and therefore have low entropy)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def entropy(series):\n    \"\"\"Categorical entropy\"\"\"\n    probs = series.value_counts().values.astype('float32')\n    probs = probs / np.sum(probs)\n    probs[probs==0] = np.nan\n    return -np.nansum(probs * np.log2(probs))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another aggregation function which could be useful is one which computes the mean difference between *consecutive* items in a series.  For example, given a column with the purchase date, this function would compute the mean time between purchases.  This could conceivably be a good predictor of how likely an individual is to be a loyal card user: individuals who use their cards regularly and frequently are probably more likely to be loyal."},{"metadata":{"trusted":true},"cell_type":"code","source":"def mean_diff(series):\n    \"\"\"Mean difference between consecutive items in a series\"\"\"\n    ss = series.sort_values()\n    return (ss - ss.shift()).mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The period of a sequence of transactions could also be a useful feature.  That is, the difference between the minimum and maximum value.  For example, customers who have been making purchases over a long period of time (the difference between the date of their first and last purchases is large) may be more likely to be loyal card users."},{"metadata":{"trusted":true},"cell_type":"code","source":"def period(series):\n    \"\"\"Period of a series (max-min)\"\"\"\n    return series.max() - series.min()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we'll create a function to compute the mode (just because pandas' default mode function doesn't handle `NaN`s well, or cases where there are two equally most common elements)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def mode(series):\n    \"\"\"Most common element in a series\"\"\"\n    tmode = series.mode()\n    if len(tmode) == 0:\n        return np.nan\n    else:\n        return tmode[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can actually compute the aggregations.  We'll define a list of aggregation functions to perform for each datatype:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Aggregations to perform for each predictor type\nbinary_aggs = ['sum', 'mean', 'nunique']\ncategorical_aggs = ['nunique', entropy, mode]\ncontinuous_aggs = ['min', 'max', 'sum', 'mean', 'std', 'skew', mean_diff, period]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And then, using those lists, we'll define a dictionary containing which aggregation functions to apply on which columns of the transactions data.  I've occasionally added a mean() aggregation function when it seems like the variable could be ordinal (but we're not sure because of the anonymous feature names!)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Aggregations to perform on each column\naggs = {\n    'authorized_flag':             binary_aggs,\n    'city_id':                     categorical_aggs,\n    'category_1':                  binary_aggs,\n    'installments':                continuous_aggs,\n    'category_3_nan':              ['mean'],\n    'category_3_0.0':              ['mean'],\n    'category_3_1.0':              ['mean'],\n    'category_3_2.0':              ['mean'],\n    'category_2_nan':              ['mean'],\n    'category_2_1.0':              ['mean'],\n    'category_2_2.0':              ['mean'],\n    'category_2_3.0':              ['mean'],\n    'category_2_4.0':              ['mean'],\n    'category_2_5.0':              ['mean'],\n    'merchant_category_id':        categorical_aggs,\n    'merchant_id':                 categorical_aggs,\n    'month_lag':                   continuous_aggs,\n    'purchase_amount':             continuous_aggs,\n    'purchase_time':               continuous_aggs + ['count'],\n    'purchase_hour':               categorical_aggs + ['mean'],\n    'purchase_day':                categorical_aggs + ['mean'],\n    'purchase_week':               categorical_aggs + continuous_aggs,\n    'purchase_month':              categorical_aggs + continuous_aggs,\n    'purchase_weekend':            binary_aggs,\n    'ref_date':                    continuous_aggs,\n    'time_since_first_active':     continuous_aggs,\n    'state_id':                    categorical_aggs,\n    'subsector_id':                categorical_aggs,\n    'merchant_group_id':           categorical_aggs,\n    'numerical_1':                 continuous_aggs,\n    'numerical_2':                 continuous_aggs,\n    'most_recent_sales_range':     categorical_aggs + ['mean'], #ordinal?\n    'most_recent_purchases_range': categorical_aggs + ['mean'], #orindal?\n    'avg_sales_lag3':              continuous_aggs,\n    'avg_purchases_lag3':          continuous_aggs,\n    'active_months_lag3':          continuous_aggs,\n    'avg_sales_lag6':              continuous_aggs,\n    'avg_purchases_lag6':          continuous_aggs,\n    'active_months_lag6':          continuous_aggs,\n    'avg_sales_lag12':             continuous_aggs,\n    'avg_purchases_lag12':         continuous_aggs,\n    'active_months_lag12':         continuous_aggs,\n    'category_4':                  binary_aggs,\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, phew, *now* we can actually compute the aggregations.  This'll take a while!"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Perform each aggregation\nfor col, funcs in aggs.items():\n    for func in funcs:\n        \n        # Get name of aggregation function\n        if isinstance(func, str):\n            func_str = func\n        else:\n            func_str = func.__name__\n            \n        # Name for new column\n        new_col = col + '_' + func_str\n            \n        # Compute this aggregation\n        cards['hist_'+new_col] = hist_trans[col].agg(func).astype('float32')\n        cards['new_'+new_col] = new_trans[col].agg(func).astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After blindly doing a bunch of aggregations on a dataset, it's usually a good idea to check for non-informative columns.  That is, columns which are all `NaN`, only contain one unique value, etc.  Let's check for those and remove them from the dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"def remove_noninformative(df):\n    \"\"\"Remove non-informative columns (all nan, or all same value)\"\"\"\n    for col in df:\n        if df[col].isnull().all():\n            print('Removing '+col+' (all NaN)')\n            del df[col]\n        elif df[col].nunique()<2:\n            print('Removing '+col+' (only 1 unique value)')\n            del df[col]\n\nremove_noninformative(cards)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have one, *giant* table, where each row corresponds to a card account for which we want to predict the loyalty, and each column corresponds to a feature of that account.  Unfortunately, what with all the aggregations we performed, we now have well over 400 features!"},{"metadata":{"trusted":true},"cell_type":"code","source":"cards.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we'll split the data into test and training data, as well as X and y data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test data\ntest = cards['target'].isnull()\nX_test = cards[test].copy()\ndel X_test['target']\n\n# Training data\ny_train = cards.loc[~test, 'target'].copy()\nX_train = cards[~test].copy()\ndel X_train['target']\n\n# Save file w/ all features\ncards.reset_index(inplace=True)\ncards.to_feather('card_features_all.feather')\n\n# Clean up \ndel cards\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"feature-selection\"></a>\n## Feature Selection\n\nSee [this kernel](https://www.kaggle.com/brendanhasz/elo-feature-engineering-and-feature-selection-2) for the executed code for this section (running this entire kernel exceeds the Kaggle runtime limits, so I had to split it into two kernels).\n\nMachine learning models don't generally perform well when they're given a huge number of features, many of which are not very informative.  The more superfluous features we give our model to train on, the more likely it is to overfit!  To prune out features which could confuse our predictive model, we'll perform some feature selection. \n\nIdeally, we'd fit our model a bunch of different times, using every possible different combination of features, and use the set of features which gives the best cross-validated results. But, there are a few problems with that approach.  First, that would lead to overfitting to the training data.  Second, and perhaps even more importantly, it would take forever.\n\nThere are a bunch of different ways to perform feature selection in a less exhaustive, but more expedient manner.  Forward selection, backward selection, selecting features based on their correlation with the target variable, and \"embedded\" methods such as Lasso regressions (where the model itself performs feature selection during training) are all options.  However, here we'll use two different methods: the mutual information between each feature and the target variable, and the permutation-based feature importance of each feature."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"mutual-information\"></a>\n### Mutual Information\n\nTo get some idea of how well each feature corresponds to the target (the loyalty score), we can compute the [mutual information](https://en.wikipedia.org/wiki/Mutual_information) between each feature and the target. Let's make a function to compute the mutual information between two vectors."},{"metadata":{"trusted":true},"cell_type":"code","source":"def mutual_information(xi, yi, res=20):\n    \"\"\"Compute the mutual information between two vectors\"\"\"\n    ix = ~(np.isnan(xi) | np.isinf(xi) | np.isnan(yi) | np.isinf(yi))\n    x = xi[ix]\n    y = yi[ix]\n    N, xe, ye = np.histogram2d(x, y, res)\n    Nx, _ = np.histogram(x, xe)\n    Ny, _ = np.histogram(y, ye)\n    N = N / len(x) #normalize\n    Nx = Nx / len(x)\n    Ny = Ny / len(y)\n    Ni = np.outer(Nx, Ny)\n    Ni[Ni == 0] = np.nan\n    N[N == 0] = np.nan\n    return np.nansum(N * np.log(N / Ni))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The mutual information represents the amount of information that can be gained about one variable by knowing the value of some other vairable.  Obviously this is very relevant to the task of feature selection: we want to choose features which knowing the value of will give us as much information as possible about the target variable.\n\nPractically speaking, the nice thing about using mutual information instead of, say, the correlation coefficient, is that it is sensitive to nonlinear relationships.  We'll be using nonlinear predictive models (like gradient boosted decision trees), and so we don't want to limit the features we select to be only ones which have a linear relationship to the target variable.  Notice how the sin-like relationship in the middle plot below has a high mutual information, but not a great correlation coefficient."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show mutual information vs correlation\nx = 5*np.random.randn(1000)\ny = [x + np.random.randn(1000),\n     2*np.sin(x) + np.random.randn(1000),\n     x + 10*np.random.randn(1000)]\nplt.figure(figsize=(10, 4))\nfor i in range(3):    \n    plt.subplot(1, 3, i+1)\n    plt.plot(x, y[i], '.')\n    rho, _ = spearmanr(x, y[i])\n    plt.title('Mutual info: %0.3f\\nCorr coeff: %0.3f'\n              % (mutual_information(x, y[i]), rho))\n    plt.gca().tick_params(labelbottom=False, labelleft=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll use the mutual information of the quantile-transformed aggregation scores (just so outliers don't mess up the mutual information calculation).  So, we'll need a function to perform the [quantile transform](https://en.wikipedia.org/wiki/Quantile_normalization), and one to compute the mutual information after applying the quantile transform:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def quantile_transform(v, res=101):\n    \"\"\"Quantile-transform a vector to lie between 0 and 1\"\"\"\n    x = np.linspace(0, 100, res)\n    prcs = np.nanpercentile(v, x)\n    return np.interp(v, prcs, x/100.0)\n    \n    \ndef q_mut_info(x, y):\n    \"\"\"Mutual information between quantile-transformed vectors\"\"\"\n    return mutual_information(quantile_transform(x),\n                              quantile_transform(y))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can compute the mutual information between each feature and the loyalty score."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n%%time\n\n# Compute the mutual information\ncols = []\nmis = []\nfor col in X_train:\n    mi = q_mut_info(X_train[col], y_train)\n    cols.append(col)\n    mis.append(mi)\n    \n# Print mut info of each feature\nprint_table(['Column', 'Mutual_Information'],\n            [cols, mis])\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's only bother keeping the features with the top 200 mutual information scores."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n# Create DataFrame with scores\nmi_df = pd.DataFrame()\nmi_df['Column'] = cols\nmi_df['mut_info'] = mis\n\n# Sort by mutual information\nmi_df = mi_df.sort_values('mut_info', ascending=False)\ntop200 = mi_df.iloc[:200,:]\ntop200 = top200['Column'].tolist()\n\n# Keep only top 200 columns\nX_train = X_train[top200]\nX_test = X_test[top200]\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"permutation-based-feature-importance\"></a>\n### Permutation-based Feature Importance\n\nA different way to select features is to try and train a model using *all* the features, and then determine how heavily the model's performance depends on the features.  But, we'll need to use a model which can handle a lot of features without overfitting too badly (i.e., an unregularized linear regression wouldn't be a good idea here).  So, we'll use a gradient boosted decision tree, specifically [CatBoost](http://catboost.ai/).  \n\nLet's create a data processing and prediction pipeline.  First, we'll target encode the categorical columns (basically just set each category to the mean target value for samples having that category - see my [post on target encoding](https://brendanhasz.github.io/2019/03/04/target-encoding.html)).  Then, we can normalize the data and impute missing data (we'll just fill in missing data with the median of the column).  Finally, we can use CatBoost to predict the loyalty scores from the features we've engineered."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n# Regression pipeline\ncat_cols = [c for c in X_train if 'mode' in c] \nreg_pipeline = Pipeline([\n    ('target_encoder', TargetEncoderCV(cols=cat_cols)),\n    ('scaler', RobustScaler()),\n    ('imputer', SimpleImputer(strategy='median')),\n    ('regressor', CatBoostRegressor(verbose=False))\n])\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can measure how heavily the model depends on various features by using permutation-based feature importance.  Basically, we train the model on all the data, and then measure its error after shuffling each row.  When the model's error increases a lot after shuffling a row, that means that the feature which was shuffled was important for the model's predictions.\n\nThe advantage of permutation-based feature importance is that it gives a super-clear view and a single score as to how important each feature is.  The downside is that this score is intrinsically linked to the model.  Whereas computing the mutual information between the features and the target only depends on the data, permutation-based feature importance scores depend on the data, the model being used, and the interaction between the two.  If your model can't fit the data very well, your permutation scores will be garbage!\n\nLuckily CatBoost nearly always does a pretty good job of prediction, even in the face of lots of features!  So, let's compute the permutation-based feature importance for each feature (the complete code is [on my GitHub](https://github.com/brendanhasz/dsutils/blob/master/src/dsutils/evaluation.py#L126))."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n%%time \n\n# Compute the cross-validated feature importance\nimp_df = permutation_importance_cv(\n    X_train, y_train, reg_pipeline, 'rmse', n_splits=2)\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, we can plot the importance scores for each feature.  These scores are just the difference between the model's error with no shuffled features and the error with the feature of interest shuffled.  So, larger scores correspond to features which the model needs to have a low error."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n# Plot the feature importances\nplt.figure(figsize=(8, 100))\nplot_permutation_importance(imp_df)\nplt.show()\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we'll want to save the features so that we can use them to train a model to predict the loyalty scores.  Let's save the top 100 most important features to a [feather](https://github.com/wesm/feather) file, so that we can quickly load them back in when we do the modeling.  First though, we need to figure out which features *are* the ones with the best importance scores."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n# Get top 100 most important features\ndf = pd.melt(imp_df, var_name='Feature', value_name='Importance')\ndfg = (df.groupby(['Feature'])['Importance']\n       .aggregate(np.mean)\n       .reset_index()\n       .sort_values('Importance', ascending=False))\ntop100 = dfg['Feature'][:100].tolist()\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, we can save those features (and the corresponding target variable!) to a feather file."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n# Save file w/ 100 most important features\ncards = pd.concat([X_train[top100], X_test[top100]])\ncards['target'] = y_train\ncards.reset_index(inplace=True)\ncards.to_feather('card_features_top100.feather')\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"conclusion\"></a>\n## Conclusion\n\nNow that we've engineered features for each card account, the next thing to do is create models to predict the target value from those features.  In [the next kernel](https://www.kaggle.com/brendanhasz/elo-modeling), we'll try different modeling techniques to see which gives the best predictions."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}