{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# TL'DR\nThis is the third part in a series of tutorials showcasing how to automate much of feature engineering.\n\n- In [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), we illustrated *how to estimate the best performance you can achieve consistently (i.e. in-sample and out-of-sample) using a set of features, in a single line of code, and without training any model.*\n\n- In [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), we explained why, when it comes to features, sometimes less is more. Then, building on [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), *we illustrated how to filter out redundant and non-informative features in a model-free fashion.*\n\n- In Part III (this tutorial), ***we illustrate how*** the model-free feature selection algorithm of [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii) may be used ***to shrink any model towards avoiding redundant and non-informative features.***\n\n*Please upvote and share it you found it useful.*\n\n# Table of Contents\n\n- **I. [Background](#background)**\n - **I.1 [The Limits of Recursive Feature Selection](#rfe)**\n - **I.2 [The Key To Effective Feature Selection](#key)** \n - **I.3 [How To Add Shrinkage To Any Regressor](#solution)** \n- **II. [Application](#application)**\n - **II.1 [Getting Started With KXY](#setup)**\n - **II.2 [Experimental Setup](#exp)**\n     - **II.2.a [The set of candidate features](#features)**\n     - **II.2.b [How we construct training/testing sets](#split)**\n - **II.3 [How to Add Effective Feature Selection to Any Regressor Class in Python](#one-liner)**\n     - **II.3.a [Training  and predicting with KXY feature selection](#train-pred)**\n     - **II.3.b [Visualizing the learned features ranking](#viz)**\n - **II.4 [Effectiveness Assessment](#effectiveness)**\n    - **II.4.a [High compression rate](#number)**\n    - **II.4.b [Improved performance](#performance)**\n    - **II.4.c [Slower performance decay](#decay)**\n- **III. [Conclusion](#conclusion)**\n    \n    \n# I. Background <a name=\"background\"></a>\n\nIn [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), we argued that using too many features might degrade model performance, and make models costlier to build, harder to explain, and costlier to maintain. This is because a large number of features will often include **non-informative features** and/or **redundant features**. \n\n**Non-informative features** are features that we can safely remove from a set of candidate features without incurring any performance loss, no matter what other features are in the set. \n\n**Redundant features**, on the other hand, are features that we can safely remove from a set of candidate features without incurring any performance loss, ***so long as*** some other features in the candidate set are kept.\n\n\n## I.1 The Limits of RFE (*Recursive Features Elimination*) <a name=\"rfe\"></a>\n\nWhile features selection isn't really a new topic, commonly used approaches such as the celebrated *Recursive Features Elimination (RFE)*, are far from being satisfactory.\n\nA major limitation of RFE is that it cannot detect redundant features, even when seemingly powerful feature importance scores such as SHAP and permutation-based scores are used. \n\nAs an illustration, if we duplicate the most important feature (as per SHAP) in a set of features, and retrain our model, assuming non-degeneracy due to dependent features, the duplicated feature is likely going to be in the new top-2 most important features (again as per SHAP). As such, SHAP-based RFE will have a hard time eliminating the duplicate feature, even though it is completely redundant. \n\nHad we used a permutation-based importance score instead, the situation would have been worse. In effect, the model would have been trained with the two features having identical values. Randomly shuffling values of one of the two features, and not values of the other, would result in a test distribution that is not representative of the true data generating distribution, which would render RFE unsound.\n\n\n## I.2 The Key to Effective Feature Selection <a name=\"key\"></a>\n\nTo understand what it would take to effectively trim off redundant and non-informative features, we recall two important facts.\n\nFirst, as discussed in [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), the highest performance achievable when using features vector $z=(z_1, \\dots, z_d)$ to predict target $y$ is an increasing function of the mutual information $I(y; z)$.\n\nSecond, by the *mutual information chain rule*, for $d>1$ and for any permutation $\\{\\pi_1, \\dots, \\pi_d\\}$ of $\\{1, \\dots, d \\}$: $$\\begin{align}I(y; z_1, \\dots, z_d) &= I\\left(y; z_1\\right) + \\sum_{i=2}^d I\\left(y; z_i | z_1, \\dots, z_{i-1}\\right) \\\\ &= I\\left(y; z_{\\pi_1}\\right) + \\sum_{i=2}^d I\\left(y; z_{\\pi_i} | z_{\\pi_1}, \\dots, z_{\\pi_{i-1}}\\right)\\end{align}.$$ \n\nThe mutual information term $I(y; z_{\\pi_1})$ reflects the highest performance that may be achieved when using feature $z_{\\pi_1}$ by itself to predict target $y$.\n\nThe conditional mutual information term $I\\left(y; z_{\\pi_i} | z_{\\pi_1}, \\dots, z_{\\pi_{i-1}}\\right) := I\\left(y; z_{\\pi_1}, \\dots, z_{\\pi_i}\\right)-I\\left(y; z_{\\pi_1}, \\dots, z_{\\pi_{i-1}}\\right)$ reflects the increase in the highest performance achievable that is due *solely* to adding feature $z_{\\pi_i}$ to features $(z_{\\pi_1}, \\dots, z_{\\pi_{i-1}})$.\n\nThese two facts suggest that trimming off redundant and non-informative features amounts to looking for a special permutation $\\{s_1, \\dots, s_d\\}$ of $\\{1, \\dots, d\\}$ such that $$I(y; z) \\approx I\\left( y; z_{s_1}, \\dots, z_{s_q} \\right) = I\\left(y; z_{s_1}\\right) + \\sum_{i=2}^q I\\left(y; z_{s_i} | z_{s_1}, \\dots, z_{s_{i-1}} \\right)$$ for a small $q < d$. \n\n$dI := I(y; z) - I\\left( y; z_{s_1}, \\dots, z_{s_q} \\right):= I\\left(y; z_{s_{q+1}}, \\dots, z_{s_d} \\vert z_{s_1}, \\dots, z_{s_q} \\right)$ reflects the *loss of juice* resulting from feature selection, while $1-\\frac{q}{d}$ represents the associated compression rate. \n\nFor a given *loss of juice* $dI$, the smaller $q$ (i.e. the higher the compression rate), the more effective the permutation $\\{s_1, \\dots, s_d\\}$.\n\n## I.3 How To Add Shrinkage To Any Regressor <a name=\"solution\"></a>\n\nGiven a number of features $q$ or compression rate, finding the features subset $\\{s_1, \\dots, s_q\\}$ that will yield the smallest *loss of juice* has combinatorial complexity $O(C_d^q)$! \n\nTo appreciate how bad this is, when $d=50$ and $q=10$, we would need to estimate more than ten billion mutual information terms to find the best subset of $10$ features. \n\nWhat's worse is that, in practice, we would not know what compression rate we would afford without giving up too much performance. Searching for the right $q$ to use will add to the computational intractability.\n\nTo circumvent this combinatorial complexity, and inspired by the chain rule $I(y; z_1, \\dots, z_d) = I\\left(y; z_{\\pi_1}\\right) + \\sum_{i=2}^d I\\left(y; z_{\\pi_i} | z_{\\pi_1}, \\dots, z_{\\pi_{i-1}}\\right)$, we squeeze as much mutual information into the terms on the right hand side, sequentially, and one term at a time. \n\nThe resulting greedy algorithm proceeds as follows:\n $$s_1 := \\underset{k \\in [1, d]}{\\text{argmax}} I(y; z_k),$$ $$\\forall i > 1, ~ s_i := \\underset{k \\in [1, d], ~ k \\notin \\{s_1, \\dots, s_{i-1}\\}}{\\text{argmax}} I(y; z_k | z_{s_1}, \\dots, z_{s_{i-1}}).$$\n\nThis is essentially what the model-free variable selection approach described in [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii) does. It learns the permutation $\\{s_1, \\dots, s_d\\}$, and returns the highest performances achievable using $(z_{s_1}, \\dots, z_{s_i})$ to predict $y$, namely $\\bar{\\rho}_{s_i} := \\bar{\\rho}\\left( P_{y; z_{s_1}, \\dots, z_{s_i}} \\right)$, for all $s_i$.\n\nOnce we have the result of the model-free variable selection algorithm of [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), we can add feature selection to any regressor in such a way that strikes the right balance between keeping the number of features $q$ small and model performance. \n\nFirst we initialize $q$ to the smallest number of features required to achieve a certain fraction $\\alpha$ (e.g. $0.9$) of the overall achievable performance: $\\bar{\\rho}\\left( P_{y; z_{s_1}, \\dots, z_{s_q}} \\right) = \\alpha \\bar{\\rho}\\left( P_{y; z_{s_1}, \\dots, z_{s_d}} \\right).$ \n\nThen we train our regressor using $(z_{s_1}, \\dots, z_{s_q})$ and we increase $q$, one feature at a time, until doing so no longer increases model performance on a validation set, or we hit a hard cap on the number of features we could use.\n\n\n\n\n\n**Reference:**\n\n- [1]<a name=\"paper1\"></a> Samo, Y.L.K., 2021. LeanML: A Design Pattern To Slash Avoidable Wastes in Machine Learning Projects. arXiv preprint arXiv:2107.08066.\n- [2]<a name=\"paper2\"></a> Samo, Y.L.K., 2021, March. Inductive Mutual Information Estimation: A Convex Maximum-Entropy Copula Approach. In International Conference on Artificial Intelligence and Statistics (pp. 2242-2250). PMLR.\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# II. Application <a name=\"application\"></a>\n\n## II.1 Getting Started with KXY <a name=\"setup\"></a>\nWe will use the ``kxy`` package. It requires an API key. See [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i) on how to get yours.","metadata":{}},{"cell_type":"code","source":"!pip install kxy -U","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-15T00:04:54.230328Z","iopub.execute_input":"2021-12-15T00:04:54.230705Z","iopub.status.idle":"2021-12-15T00:05:08.536064Z","shell.execute_reply.started":"2021-12-15T00:04:54.230610Z","shell.execute_reply":"2021-12-15T00:05:08.535343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nimport os\nimport numpy as np\nimport pandas as pd\nimport pprint as pp\nimport pylab as plt\nimport kxy","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:05:08.538248Z","iopub.execute_input":"2021-12-15T00:05:08.538794Z","iopub.status.idle":"2021-12-15T00:05:09.584263Z","shell.execute_reply.started":"2021-12-15T00:05:08.538760Z","shell.execute_reply":"2021-12-15T00:05:09.583389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nkxy_api_key = user_secrets.get_secret('KXY_API_KEY')\nos.environ['KXY_API_KEY'] = kxy_api_key","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:05:09.585552Z","iopub.execute_input":"2021-12-15T00:05:09.586000Z","iopub.status.idle":"2021-12-15T00:05:10.006931Z","shell.execute_reply.started":"2021-12-15T00:05:09.585968Z","shell.execute_reply":"2021-12-15T00:05:10.006291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II.2 Experimental Setup <a name=\"exp\"></a>\n### II.2.a  The set of candidate features we will be using <a name=\"features\"></a>\n\nHere we generate a set of 56 candidate features, temporal and cross-sectional.","metadata":{}},{"cell_type":"code","source":"TRAIN_CSV = '/kaggle/input/g-research-crypto-forecasting/train.csv'\n\ndef nanmaxmmin(a, axis=None, out=None):\n    ''' '''\n    return np.nanmax(a, axis=axis, out=out)-np.nanmin(a, axis=axis, out=out)\n\n\ndef get_features(df):\n    ''' \n    An example function generating a candidate list of features.\n    '''\n    features = df[['Count', 'Open', 'High', 'Low', 'Close', \\\n        'Volume', 'VWAP','timestamp', 'Target', 'Asset_ID']].copy()\n    # Upper shadow\n    features['UPS'] = (df['High']-np.maximum(df['Close'], df['Open']))\n    features['UPS'] = features['UPS'].astype(np.float32)\n    \n    # Lower shadow\n    features['LOS'] = (np.minimum(df['Close'], df['Open'])-df['Low'])\n    features['LOS'] = features['LOS'].astype(np.float32)\n    \n    # High-Low range\n    features['RNG'] = ((features['High']-features['Low'])/features['VWAP'])\n    features['RNG'] = features['RNG'].astype(np.float32)\n    \n    # Daily move\n    features['MOV'] = ((features['Close']-features['Open'])/features['VWAP'])\n    features['MOV'] = features['MOV'].astype(np.float32)\n    \n    # Close vs. VWAP\n    features['CLS'] = ((features['Close']-features['VWAP'])/features['VWAP'])\n    features['CLS'] = features['CLS'].astype(np.float32)\n    \n    # Log-volume\n    features['LOGVOL'] = np.log(1.+features['Volume'])\n    features['LOGVOL'] = features['LOGVOL'].astype(np.float32)\n    \n    # Log-count\n    features['LOGCNT'] = np.log(1.+features['Count'])\n    features['LOGCNT'] = features['LOGCNT'].astype(np.float32)\n    \n    # Volume/Count\n    features['VOL2CNT'] = features['Volume']/(1.+features['Count'])\n    features['VOL2CNT'] = features['VOL2CNT'].astype(np.float32)\n    \n    # Drop raw inputs\n    features.drop(columns=['Open', 'High', 'Low', 'Close', 'Volume', 'VWAP', \\\n        'Count'], errors='ignore', inplace=True)\n    \n    # Enrich the features dataframe with some temporal feautures \n    # (specifically, some stats on the last hour worth of bars)\n    features = features.kxy.temporal_features(max_lag=14, \\\n                exclude=['timestamp', 'Target'],\n                groupby='Asset_ID')\n    \n    # Enrich the features dataframe context around the time\n    # (e.g. hour, day of the week, etc.)\n    time_features = features.kxy.process_time_columns(['timestamp'])\n    features.drop(columns=['timestamp'], errors='ignore', inplace=True)\n    features = pd.concat([features, time_features], axis=1)\n    \n    return features","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:05:10.008093Z","iopub.execute_input":"2021-12-15T00:05:10.008359Z","iopub.status.idle":"2021-12-15T00:05:10.026244Z","shell.execute_reply.started":"2021-12-15T00:05:10.008329Z","shell.execute_reply":"2021-12-15T00:05:10.025618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    # Reading candidate features as external data\n    PATH = '/kaggle/input/gresearch-features-v2' # '/kaggle/input/d/leanboosting/gresearch-features'\n    training_features = pd.read_parquet('%s/2018Q1.parquet' % PATH)\nexcept:\n    PATH = '/kaggle/output'\n    try:\n        # Reading candidate features from disk\n        training_features = pd.read_parquet('%s/2018Q1.parquet' % PATH)\n    except:\n        # Regenerating candidate features\n        df_train = pd.read_csv(TRAIN_CSV)\n        training_features = get_features(df_train)\n        del df_train\n        # Saving to disk\n        months = [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]\n        for year in range(2018, 2022):\n            year_df = training_features[training_features['timestamp.YEAR()']==year]\n            for q in range(1, 5):\n                df = year_df[year_df['timestamp.MONTH()'].isin(months[q-1])]\n                df.to_parquet('%s/%dQ%d.parquet' % (PATH, year, q))","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:05:10.028043Z","iopub.execute_input":"2021-12-15T00:05:10.028633Z","iopub.status.idle":"2021-12-15T00:05:15.041318Z","shell.execute_reply.started":"2021-12-15T00:05:10.028594Z","shell.execute_reply":"2021-12-15T00:05:15.040344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Complete list of all candidate features","metadata":{}},{"cell_type":"code","source":"# Printing all feautures\nall_features = sorted([_ for _ in training_features.columns])\npp.pprint(all_features)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:05:15.042551Z","iopub.execute_input":"2021-12-15T00:05:15.042866Z","iopub.status.idle":"2021-12-15T00:05:15.142135Z","shell.execute_reply.started":"2021-12-15T00:05:15.042826Z","shell.execute_reply":"2021-12-15T00:05:15.141327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.2.b How we construct training/testing sets <a name=\"split\"></a>\n\nTo evaluate our approach, we build rolling training/testing sets. \n\nThe first training set we use is the first quarter of 2018, and we use it to predict the second quarter of 2018. \n\nThereafter, we build a model to predict moves in quarter $k+1$ using a training set obtained by sampling 80% of data from quarter $k$ and 20% of the training set we previously used to predict moves in quarter $k$.\n\nThis AR(1)-style of constructing training sets allows us to keep historical data going far back in the past, while avoiding sample size explosion and while giving more importance/weight to recent observations.\n","metadata":{}},{"cell_type":"code","source":"bitcoin_asset_id = 1\nquarters = ['%dQ%d' % (year, quarter) for year in range(2018, 2022) \\\n            for quarter in range(1, 5)]\nquarters.remove('2021Q4')\ntraining_data = pd.read_parquet('%s/2018Q1.parquet' % PATH)\ntraining_data = training_data.astype(np.float32)\n\nfor i in range(1, len(quarters)-4):\n    testing_data = pd.read_parquet('%s/%s.parquet' % (PATH, quarters[i]))\n    testing_data = testing_data.astype(np.float32)\n    # Update the training set with \n    training_data = training_data.sample(frac=0.2, random_state=28)\n    testing_data = testing_data.sample(frac=0.8, random_state=28)\n    training_data = pd.concat([training_data, testing_data], axis=0)\n    gc.collect()\n    \nmonths_df = training_data['timestamp.YEAR()'].astype(int).astype(str) + \\\n    training_data['timestamp.MONTH()'].astype(int).astype(str).apply(lambda x: x.zfill(2))\nmonths_df = pd.DataFrame(months_df, columns=[\"Month\"])\ntraining_data.drop(columns=['timestamp.MONTH()', 'timestamp.YEAR()'], inplace=True)\n\n# Test set\ntesting_data = pd.read_parquet('%s/%s.parquet' % (PATH, quarters[-3]))\ntesting_data = testing_data.astype(np.float32)\ntesting_data.drop(columns=['timestamp.MONTH()', 'timestamp.YEAR()'], inplace=True)    \ntesting_x = testing_data.drop(columns=['Target'])\ntesting_y = testing_data['Target']\n\nbitcoin_training_data = training_data[training_data['Asset_ID']==bitcoin_asset_id]\nbitcoin_testing_data = testing_data[testing_data['Asset_ID']==bitcoin_asset_id]\nbitcoin_testing_x = bitcoin_testing_data.drop(columns=['Target'])\nbitcoin_testing_y = bitcoin_testing_data['Target']\n\n# Decay sets\nq2_2021_data = pd.read_parquet('%s/%s.parquet' % (PATH, quarters[-2]))\nq2_2021_data = q2_2021_data.astype(np.float32)\nq2_2021_data.drop(columns=['timestamp.MONTH()', 'timestamp.YEAR()'], inplace=True)\n    \nq2_2021_x = q2_2021_data.drop(columns=['Target'])\nq2_2021_y = q2_2021_data['Target']\n\nq3_2021_data = pd.read_parquet('%s/%s.parquet' % (PATH, quarters[-1]))\nq3_2021_data = q3_2021_data.astype(np.float32)\nq3_2021_data.drop(columns=['timestamp.MONTH()', 'timestamp.YEAR()'], inplace=True)\n    \nq3_2021_x = q3_2021_data.drop(columns=['Target'])\nq3_2021_y = q3_2021_data['Target']\n\nbitcoin_q2_2021_data = q2_2021_data[q2_2021_data['Asset_ID']==bitcoin_asset_id]\nbitcoin_q2_2021_x = bitcoin_q2_2021_data.drop(columns=['Target'])\nbitcoin_q2_2021_y = bitcoin_q2_2021_data['Target']\n\nbitcoin_q3_2021_data = q3_2021_data[q3_2021_data['Asset_ID']==bitcoin_asset_id]\nbitcoin_q3_2021_x = bitcoin_q3_2021_data.drop(columns=['Target'])\nbitcoin_q3_2021_y = bitcoin_q3_2021_data['Target']","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:05:15.143633Z","iopub.execute_input":"2021-12-15T00:05:15.143872Z","iopub.status.idle":"2021-12-15T00:06:37.480584Z","shell.execute_reply.started":"2021-12-15T00:05:15.143841Z","shell.execute_reply":"2021-12-15T00:06:37.479415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"to_plot = months_df.groupby(\"Month\").size()\nto_plot = to_plot/to_plot.sum()\nto_plot.plot(kind=\"bar\", figsize=(15, 10))\naxis = plt.gca()\n_ = axis.set_ylabel('Proportion of Observations', fontsize=15)\n_ = axis.set_xlabel('Month', fontsize=15)\n_ = axis.set_title('Composition of Training Data', fontsize=18)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:06:37.482050Z","iopub.execute_input":"2021-12-15T00:06:37.482319Z","iopub.status.idle":"2021-12-15T00:06:38.167260Z","shell.execute_reply.started":"2021-12-15T00:06:37.482280Z","shell.execute_reply":"2021-12-15T00:06:38.166321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## II.3 How to Add Effective Feature Selection to Any Regressor Class in Python <a name=\"one-liner\"></a>\n\nThe approach described in the background section is implemented in the ``kxy`` package. \n\n#### Training \nThe syntax is ``training_data_df.kxy.fit(target_column, learner_func, problem_type='regression')`` and it works on any pandas DataFrame object, so long as you import the ``kxy`` package. \n\n``learner_func`` is a function that takes a single optional parameter ``n_vars``, and returns an instance of a regressor class expecting ``n_vars`` features, and following the ``sklearn`` API (i.e. ``m.fit(x, y)`` to fit the instance, and ``m.predict(x)`` to make predictions).\n\n#### Predictions \nThe syntax to make predictions is ``target_predictions_df = training_data_df.kxy.predict(testing_data_df)``, where ``testing_data_df`` is a dataframe with test features (and no target values), and ``target_predictions_df`` is the dataframe with a single column (whose name is the same as the target) containing predictions, and sharing the same index as ``testing_data_df``.\n\n\n#### Working with existing ML libraries\n\nThe ``kxy`` package contains a range of utility functions that allow you to use any supervised learner in ``sklearn``, ``xgboost``, ``lightgbm``, ``tensorflow`` and many more frameworks as base learner. \n\nThe code snippet below shows you how to use ``lightgbm`` regressors as base learners. \n\nFor other libraries see ``kxy.learning.base_learners``.\n\n","metadata":{}},{"cell_type":"code","source":"from kxy.learning import get_lightgbm_learner_learning_api, get_sklearn_learner\n# Constructing a LightGBM base regressor\nparams = {\n    'objective': 'rmse',  \n    'boosting_type': 'gbdt',\n    'n_jobs': -1,\n    'learning_rate': 0.1,\n    'verbose': -1,\n}\nlightgbm_regressor_cls = get_lightgbm_learner_learning_api(params, num_boost_round=2000, \\\n    early_stopping_rounds=5, split_random_seed=28)\nlasso_regressor_cls = get_sklearn_learner('sklearn.linear_model.LassoCV')\nlr_regressor_cls = get_sklearn_learner('sklearn.linear_model.LinearRegression')\n","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:06:38.168323Z","iopub.execute_input":"2021-12-15T00:06:38.168538Z","iopub.status.idle":"2021-12-15T00:06:39.343489Z","shell.execute_reply.started":"2021-12-15T00:06:38.168513Z","shell.execute_reply":"2021-12-15T00:06:39.342662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Experimental setup\n\nWe focus on cross-sectional predictions (i.e. we want to train a single model to predict all crypto-currencies), and bitcoin predictions for brevity.\n\nWe run our training/testing split logic until Q4 2020. We use Q1 2021 as testing set, and we use Q2 and Q3 2021 to compare performance decay with and without feature selection.","metadata":{}},{"cell_type":"markdown","source":"### II.3.a Training  and predicting with KXY feature selection <a name=\"train-pred\"></a>","metadata":{}},{"cell_type":"code","source":"ASSET_CSV = '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'\nasset_details = pd.read_csv(ASSET_CSV)\n\n# Scoring function\ndef gresearch_score(true_df, pred_df):\n    ''' '''\n    res = {}\n    total_weight = 0.0\n    score = 0.0\n    for asset in asset_details.to_dict(orient='records'):\n        try:\n            true_df_ = true_df[true_df['Asset_ID']==asset['Asset_ID']][['Target']]\n            if not true_df_.empty:\n                pred_df_ = pred_df[true_df['Asset_ID']==asset['Asset_ID']]\n                w = asset['Weight']\n                t = true_df_['Target'].values-np.nanmean(true_df_['Target'].values)\n                p = pred_df_['Target'].values-np.nanmean(pred_df_['Target'].values)\n                corr = np.nanmean(t*p)/(np.nanstd(t)*np.nanstd(p))\n                if not np.isnan(corr):\n                    score += corr*w\n                    total_weight += w\n                    res[asset['Asset_Name']] = '%.4f' % corr\n        except:\n            continue\n    score = score/total_weight\n    res['Overall (Weighted)'] = '%.4f' % score\n    return res","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:06:39.344727Z","iopub.execute_input":"2021-12-15T00:06:39.344955Z","iopub.status.idle":"2021-12-15T00:06:39.365745Z","shell.execute_reply.started":"2021-12-15T00:06:39.344929Z","shell.execute_reply":"2021-12-15T00:06:39.365068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training with effective feature selection\nlr_training_data = training_data.copy()\nshrinkage_results = training_data.kxy.fit('Target', lightgbm_regressor_cls, \\\n    problem_type='regression', missing_value_imputation=True, \\\n    train_frac=0.8)\npp.pprint(shrinkage_results)\nlr_shrinkage_results = lr_training_data.kxy.fit('Target', lr_regressor_cls, \\\n    problem_type='regression', missing_value_imputation=True, \\\n    train_frac=0.8)\npp.pprint(lr_shrinkage_results)\n\nlr_bitcoin_training_data = bitcoin_training_data.copy()\nbitcoin_shrinkage_results = bitcoin_training_data.kxy.fit('Target', \n    lightgbm_regressor_cls, problem_type='regression', \\\n    missing_value_imputation=True, train_frac=0.8)\npp.pprint(bitcoin_shrinkage_results)\nlr_bitcoin_shrinkage_results = lr_bitcoin_training_data.kxy.fit('Target', \n    lr_regressor_cls, problem_type='regression', \\\n    missing_value_imputation=True, train_frac=0.8)\npp.pprint(lr_bitcoin_shrinkage_results)\n\n# Predictions\nshrinkage_predicted_y = training_data.kxy.predict(testing_x)\nshrinkage_perf = gresearch_score(testing_data, shrinkage_predicted_y)\n\nshrinkage_predicted_q2_2021 = training_data.kxy.predict(q2_2021_x)\nshrinkage_predicted_q3_2021 = training_data.kxy.predict(q3_2021_x)\n\nlr_shrinkage_predicted_y = lr_training_data.kxy.predict(testing_x)\nlr_shrinkage_perf = gresearch_score(testing_data, lr_shrinkage_predicted_y)\n\nlr_shrinkage_predicted_q2_2021 = lr_training_data.kxy.predict(q2_2021_x)\nlr_shrinkage_predicted_q3_2021 = lr_training_data.kxy.predict(q3_2021_x)\n\nbitcoin_shrinkage_predicted_y = bitcoin_training_data.kxy.predict(bitcoin_testing_x)\nbitcoin_shrinkage_perf = gresearch_score(bitcoin_testing_data, \\\n                                         bitcoin_shrinkage_predicted_y)\n\nbitcoin_shrinkage_predicted_q2_2021 = bitcoin_training_data.kxy.predict(bitcoin_q2_2021_x)\nbitcoin_shrinkage_predicted_q3_2021 = bitcoin_training_data.kxy.predict(bitcoin_q3_2021_x)\n\nlr_bitcoin_shrinkage_predicted_y = lr_bitcoin_training_data.kxy.predict(bitcoin_testing_x)\nlr_bitcoin_shrinkage_perf = gresearch_score(bitcoin_testing_data, \\\n                                         lr_bitcoin_shrinkage_predicted_y)\n\nlr_bitcoin_shrinkage_predicted_q2_2021 = lr_bitcoin_training_data.kxy.predict(bitcoin_q2_2021_x)\nlr_bitcoin_shrinkage_predicted_q3_2021 = lr_bitcoin_training_data.kxy.predict(bitcoin_q3_2021_x)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:06:39.370202Z","iopub.execute_input":"2021-12-15T00:06:39.370641Z","iopub.status.idle":"2021-12-15T00:11:35.423808Z","shell.execute_reply.started":"2021-12-15T00:06:39.370591Z","shell.execute_reply":"2021-12-15T00:11:35.422634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.3.b Visualizing the learned permutations $\\{s_1, \\dots, s_q\\}$ <a name=\"viz\"> </a>\n\n#### Learned $\\{s_1, \\dots, s_q\\}$ for cross-sectional LightGBM models","metadata":{}},{"cell_type":"code","source":"training_data.kxy.predictor.variable_selection_results","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:11:35.425775Z","iopub.execute_input":"2021-12-15T00:11:35.426661Z","iopub.status.idle":"2021-12-15T00:11:35.522945Z","shell.execute_reply.started":"2021-12-15T00:11:35.426552Z","shell.execute_reply":"2021-12-15T00:11:35.522185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"####  Learned $\\{s_1, \\dots, s_q\\}$ for bitcoin LightGBM models","metadata":{}},{"cell_type":"code","source":"bitcoin_training_data.kxy.predictor.variable_selection_results","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:11:35.524532Z","iopub.execute_input":"2021-12-15T00:11:35.524798Z","iopub.status.idle":"2021-12-15T00:11:35.578254Z","shell.execute_reply.started":"2021-12-15T00:11:35.524741Z","shell.execute_reply":"2021-12-15T00:11:35.577110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II.4 Effectiveness Assessment <a name=\"effectiveness\"></a>\n\n#### Training and testing using all 56 features\n","metadata":{}},{"cell_type":"code","source":"from warnings import simplefilter\nfrom sklearn.exceptions import ConvergenceWarning\nsimplefilter(\"ignore\", category=ConvergenceWarning)\n\n# Training using all features\nm_all_features = lightgbm_regressor_cls()\nfeature_columns = [_ for _ in training_data.columns if _ != 'Target']\nx_train = training_data[feature_columns].values\ny_train = training_data['Target'].values\nx_train[np.isnan(x_train)] = 0.0\ny_train[np.isnan(y_train)] = 0.0\nx_test = testing_data[feature_columns].values\ny_test = testing_data['Target'].values\nx_test[np.isnan(x_test)] = 0.0\ny_test[np.isnan(y_test)] = 0.0\nq2_2021_x_test = q2_2021_x[feature_columns].values\nq3_2021_x_test = q3_2021_x[feature_columns].values\nm_all_features.fit(x_train, y_train)\nm_lasso_all_features = lasso_regressor_cls()\nm_lasso_all_features.fit(x_train, y_train)\nm_lr_all_features = lr_regressor_cls()\nm_lr_all_features.fit(x_train, y_train)\n\nbitcoin_all_features = lightgbm_regressor_cls()\nfeature_columns = [_ for _ in bitcoin_training_data.columns if _ != 'Target']\nbitcoin_x_train = bitcoin_training_data[feature_columns].values\nbitcoin_y_train = bitcoin_training_data['Target'].values\nbitcoin_x_test = bitcoin_testing_data[feature_columns].values\nbitcoin_y_test = bitcoin_testing_data['Target'].values\nbitcoin_q2_2021_x_test = bitcoin_q2_2021_x[feature_columns].values\nbitcoin_q3_2021_x_test = bitcoin_q3_2021_x[feature_columns].values\nbitcoin_all_features.fit(bitcoin_x_train, bitcoin_y_train)\nbitcoin_lasso_all_features = lasso_regressor_cls()\nbitcoin_lasso_all_features.fit(x_train, y_train)\nbitcoin_lr_all_features = lr_regressor_cls()\nbitcoin_lr_all_features.fit(x_train, y_train)\n\n# Prediction using all features\nall_predicted_y = m_all_features.predict(x_test)\nall_predicted_y = pd.DataFrame(all_predicted_y, index=testing_data.index, \\\n                               columns=['Target'])\nall_features_perf = gresearch_score(testing_data, all_predicted_y)\nlasso_all_predicted_y = m_lasso_all_features.predict(x_test)\nlasso_all_predicted_y = pd.DataFrame(lasso_all_predicted_y, index=testing_data.index, \\\n                               columns=['Target'])\nlasso_all_features_perf = gresearch_score(testing_data, lasso_all_predicted_y)\nlr_all_predicted_y = m_lr_all_features.predict(x_test)\nlr_all_predicted_y = pd.DataFrame(lr_all_predicted_y, index=testing_data.index, \\\n                               columns=['Target'])\nlr_all_features_perf = gresearch_score(testing_data, lr_all_predicted_y)\n\nall_predicted_q2_2021 = m_all_features.predict(q2_2021_x_test)\nall_predicted_q3_2021 = m_all_features.predict(q3_2021_x_test)\nall_predicted_q2_2021 = pd.DataFrame(all_predicted_q2_2021, \\\n                            index=q2_2021_data.index, columns=['Target'])\nall_predicted_q3_2021 = pd.DataFrame(all_predicted_q3_2021, \\\n                            index=q3_2021_data.index, columns=['Target'])\n\nlasso_all_predicted_q2_2021 = m_lasso_all_features.predict(q2_2021_x_test)\nlasso_all_predicted_q3_2021 = m_lasso_all_features.predict(q3_2021_x_test)\nlasso_all_predicted_q2_2021 = pd.DataFrame(lasso_all_predicted_q2_2021, \\\n                            index=q2_2021_data.index, columns=['Target'])\nlasso_all_predicted_q3_2021 = pd.DataFrame(lasso_all_predicted_q3_2021, \\\n                            index=q3_2021_data.index, columns=['Target'])\n\nlr_all_predicted_q2_2021 = m_lr_all_features.predict(q2_2021_x_test)\nlr_all_predicted_q3_2021 = m_lr_all_features.predict(q3_2021_x_test)\nlr_all_predicted_q2_2021 = pd.DataFrame(lr_all_predicted_q2_2021, \\\n                            index=q2_2021_data.index, columns=['Target'])\nlr_all_predicted_q3_2021 = pd.DataFrame(lr_all_predicted_q3_2021, \\\n                            index=q3_2021_data.index, columns=['Target'])\n\nbitcoin_all_predicted_y = bitcoin_all_features.predict(bitcoin_x_test)\nbitcoin_all_predicted_y = pd.DataFrame(bitcoin_all_predicted_y, \\\n                                       index=bitcoin_testing_data.index, \\\n                                       columns=['Target'])\nbitcoin_all_features_perf = gresearch_score(bitcoin_testing_data, \\\n                                            bitcoin_all_predicted_y)\n\nbitcoin_lasso_all_predicted_y = bitcoin_lasso_all_features.predict(bitcoin_x_test)\nbitcoin_lasso_all_predicted_y = pd.DataFrame(bitcoin_lasso_all_predicted_y, \\\n                                       index=bitcoin_testing_data.index, \\\n                                       columns=['Target'])\nbitcoin_lasso_all_features_perf = gresearch_score(bitcoin_testing_data, \\\n                                            bitcoin_lasso_all_predicted_y)\n\nbitcoin_lr_all_predicted_y = bitcoin_lr_all_features.predict(bitcoin_x_test)\nbitcoin_lr_all_predicted_y = pd.DataFrame(bitcoin_lr_all_predicted_y, \\\n                                       index=bitcoin_testing_data.index, \\\n                                       columns=['Target'])\nbitcoin_lr_all_features_perf = gresearch_score(bitcoin_testing_data, \\\n                                            bitcoin_lr_all_predicted_y)\n\nbitcoin_all_predicted_q2_2021 = bitcoin_all_features.predict(bitcoin_q2_2021_x_test)\nbitcoin_all_predicted_q2_2021 = pd.DataFrame(bitcoin_all_predicted_q2_2021, \\\n                            index=bitcoin_q2_2021_data.index, columns=['Target'])\nbitcoin_all_predicted_q3_2021 = bitcoin_all_features.predict(bitcoin_q3_2021_x_test)\nbitcoin_all_predicted_q3_2021 = pd.DataFrame(bitcoin_all_predicted_q3_2021, \\\n                            index=bitcoin_q3_2021_data.index, columns=['Target'])\n\nbitcoin_lasso_all_predicted_q2_2021 = bitcoin_lasso_all_features.predict(bitcoin_q2_2021_x_test)\nbitcoin_lasso_all_predicted_q2_2021 = pd.DataFrame(bitcoin_lasso_all_predicted_q2_2021, \\\n                            index=bitcoin_q2_2021_data.index, columns=['Target'])\nbitcoin_lasso_all_predicted_q3_2021 = bitcoin_lasso_all_features.predict(bitcoin_q3_2021_x_test)\nbitcoin_lasso_all_predicted_q3_2021 = pd.DataFrame(bitcoin_lasso_all_predicted_q3_2021, \\\n                            index=bitcoin_q3_2021_data.index, columns=['Target'])\n\nbitcoin_lr_all_predicted_q2_2021 = bitcoin_lr_all_features.predict(bitcoin_q2_2021_x_test)\nbitcoin_lr_all_predicted_q2_2021 = pd.DataFrame(bitcoin_lr_all_predicted_q2_2021, \\\n                            index=bitcoin_q2_2021_data.index, columns=['Target'])\nbitcoin_lr_all_predicted_q3_2021 = bitcoin_lr_all_features.predict(bitcoin_q3_2021_x_test)\nbitcoin_lr_all_predicted_q3_2021 = pd.DataFrame(bitcoin_lr_all_predicted_q3_2021, \\\n                            index=bitcoin_q3_2021_data.index, columns=['Target'])","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:11:35.582024Z","iopub.execute_input":"2021-12-15T00:11:35.582395Z","iopub.status.idle":"2021-12-15T00:13:04.802938Z","shell.execute_reply.started":"2021-12-15T00:11:35.582363Z","shell.execute_reply":"2021-12-15T00:13:04.801797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training with RFE using the same number of features as KXY\ndef rfe(x, y, n_vars, x_columns):\n    ''' '''\n    x_ = x.copy()\n    d = x_.shape[1]\n    x_columns_ = list(x_columns)\n    \n    m = lightgbm_regressor_cls()\n    m.fit(x_, y)        \n    importances = m._model.feature_importance(importance_type='split')\n    least_important = np.argmin(importances)\n    importances = [_ for _ in importances]\n    importance_df = pd.DataFrame(sorted(zip(importances, x_columns_)), \\\n                                 columns=['Importance','Feature'])\n    importance_df = importance_df.sort_values('Importance', ascending=False)\n    \n    while d > n_vars:\n        least_important = np.argmin(importances)\n        x_ = np.delete(x_, least_important, axis=1)\n        x_columns_.pop(least_important)\n        d = x_.shape[1]\n        m = lightgbm_regressor_cls()\n        m.fit(x_, y)        \n        importances = m._model.feature_importance(importance_type='split')\n        least_important = np.argmin(importances)\n        importances = [_ for _ in importances]\n        importance_df = pd.DataFrame(sorted(zip(importances, x_columns_)), \\\n                                     columns=['Importance','Feature'])\n        importance_df = importance_df.sort_values('Importance', ascending=False)\n\n        \n    return m, importance_df, x_columns_","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:13:04.808936Z","iopub.execute_input":"2021-12-15T00:13:04.810480Z","iopub.status.idle":"2021-12-15T00:13:04.878088Z","shell.execute_reply.started":"2021-12-15T00:13:04.810426Z","shell.execute_reply":"2021-12-15T00:13:04.877496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LightGBM + RFE (cross-sectional models)\nn_features = len(training_data.kxy.predictor.selected_variables)\nrfe_m, rfe_importances, rfe_selected_columns = rfe(x_train, y_train, n_features, feature_columns)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:13:04.879378Z","iopub.execute_input":"2021-12-15T00:13:04.880046Z","iopub.status.idle":"2021-12-15T00:14:16.149082Z","shell.execute_reply.started":"2021-12-15T00:13:04.880014Z","shell.execute_reply":"2021-12-15T00:14:16.148500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LightGBM + RFE (Bitcoin models)\nbitcoin_n_features = len(bitcoin_training_data.kxy.predictor.selected_variables)\nbitcoin_rfe_m, bitcoin_rfe_importances, bitcoin_rfe_selected_columns = rfe(\n        x_train, y_train, bitcoin_n_features, feature_columns)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:14:16.150226Z","iopub.execute_input":"2021-12-15T00:14:16.151018Z","iopub.status.idle":"2021-12-15T00:17:21.554510Z","shell.execute_reply.started":"2021-12-15T00:14:16.150972Z","shell.execute_reply":"2021-12-15T00:17:21.553426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test_rfe = testing_data[rfe_selected_columns].values\nrfe_predicted_y = rfe_m.predict(x_test_rfe)\nrfe_predicted_y = pd.DataFrame(rfe_predicted_y, \\\n        index=testing_data.index, columns=['Target'])\nrfe_features_perf = gresearch_score(testing_data, rfe_predicted_y)\n\nrfe_q2_2021_x_test = q2_2021_x[rfe_selected_columns].values\nrfe_q3_2021_x_test = q3_2021_x[rfe_selected_columns].values\n\nrfe_predicted_q2_2021 = rfe_m.predict(rfe_q2_2021_x_test)\nrfe_predicted_q3_2021 = rfe_m.predict(rfe_q3_2021_x_test)\nrfe_predicted_q2_2021 = pd.DataFrame(rfe_predicted_q2_2021, \\\n                            index=q2_2021_data.index, columns=['Target'])\nrfe_predicted_q3_2021 = pd.DataFrame(rfe_predicted_q3_2021, \\\n                            index=q3_2021_data.index, columns=['Target'])\n\nx_test_bitcoin_rfe = testing_data[bitcoin_rfe_selected_columns].values\nbitcoin_rfe_predicted_y = bitcoin_rfe_m.predict(x_test_bitcoin_rfe)\nbitcoin_rfe_predicted_y = pd.DataFrame(bitcoin_rfe_predicted_y, \\\n        index=testing_data.index, columns=['Target'])\nbitcoin_rfe_features_perf = gresearch_score(testing_data, bitcoin_rfe_predicted_y)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:21.555980Z","iopub.execute_input":"2021-12-15T00:17:21.556209Z","iopub.status.idle":"2021-12-15T00:17:27.415980Z","shell.execute_reply.started":"2021-12-15T00:17:21.556181Z","shell.execute_reply":"2021-12-15T00:17:27.415404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.4.a High compression rate <a name=\"number\"></a>","metadata":{}},{"cell_type":"code","source":"all_features = {'Bitcoin Model': x_train.shape[1], 'Cross-Sectional Model': x_train.shape[1]}\n\nlightgbm_compression_rate = {\\\n    'Bitcoin Model': 1.-len(bitcoin_training_data.kxy.predictor.selected_variables)/x_train.shape[1],\\\n    'Cross-Sectional Model': 1.-len(training_data.kxy.predictor.selected_variables)/x_train.shape[1]}\n\nlr_compression_rate = {\\\n    'Bitcoin Model': 1.-len(lr_bitcoin_training_data.kxy.predictor.selected_variables)/x_train.shape[1],\\\n    'Cross-Sectional Model': 1.-len(lr_training_data.kxy.predictor.selected_variables)/x_train.shape[1]}\n\ncompression_rate = pd.DataFrame([lightgbm_compression_rate, lr_compression_rate], \\\n                                index=['LightGBM', 'Linear Regression']).astype(float)\ncompression_rate.T.plot.bar(figsize=(15, 10), fontsize=15, color=['b', 'g'])\naxis = plt.gca()\n_ = axis.set_ylabel(r'''$1-\\frac{q}{d}$''', fontsize=15)\n_ = axis.set_title('KXY Feature Selection Compression Rate', fontsize=18)\nfor item in axis.get_legend().get_texts():\n    item.set_fontsize(15)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:27.417185Z","iopub.execute_input":"2021-12-15T00:17:27.417584Z","iopub.status.idle":"2021-12-15T00:17:27.893009Z","shell.execute_reply.started":"2021-12-15T00:17:27.417555Z","shell.execute_reply":"2021-12-15T00:17:27.892144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.4.b Improved performance <a name=\"performance\"></a>","metadata":{}},{"cell_type":"code","source":"lightgbm_all_results = {'Bitcoin Model': bitcoin_all_features_perf['Bitcoin'], \\\n               'Cross-Sectional Model': all_features_perf['Overall (Weighted)']}\n\nlasso_results = {'Bitcoin Model': bitcoin_lasso_all_features_perf['Bitcoin'], \\\n               'Cross-Sectional Model': lasso_all_features_perf['Overall (Weighted)']}\n\nlr_results = {'Bitcoin Model': bitcoin_lr_all_features_perf['Bitcoin'], \\\n               'Cross-Sectional Model': lr_all_features_perf['Overall (Weighted)']}\n\nlightgbm_selected_results = {'Bitcoin Model': bitcoin_shrinkage_perf['Bitcoin'], \\\n    'Cross-Sectional Model': shrinkage_perf['Overall (Weighted)']}\n\nlr_selected_results = {'Bitcoin Model': lr_bitcoin_shrinkage_perf['Bitcoin'], \\\n    'Cross-Sectional Model': lr_shrinkage_perf['Overall (Weighted)']}\n\nlightgbm_rfe_results = {'Bitcoin Model': bitcoin_rfe_features_perf['Bitcoin'], \\\n    'Cross-Sectional Model': rfe_features_perf['Overall (Weighted)']}\n\n","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:27.894131Z","iopub.execute_input":"2021-12-15T00:17:27.894340Z","iopub.status.idle":"2021-12-15T00:17:27.936685Z","shell.execute_reply.started":"2021-12-15T00:17:27.894314Z","shell.execute_reply":"2021-12-15T00:17:27.935821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The figure below illustrates that, when using linear models, ``KXY``'s feature selection algorithm is competitive relative to LASSO.","metadata":{}},{"cell_type":"code","source":"cross_sectional_results = pd.DataFrame([lr_results, lasso_results, lr_selected_results], \\\n                    index=['Linear Regression Using All Features', \\\n                        'Linear Regression With LASSO Feature Selection', \\\n                        'Linear Regression With KXY Feature Selection']).astype(float)\ncross_sectional_results.T.plot.bar(figsize=(15, 10), fontsize=15, \\\n                                   color=[ 'r', 'y', 'g'])\naxis = plt.gca()\n_ = axis.set_ylabel('Pearson Correlation', fontsize=15)\n_ = axis.set_title('Effect of Feature Selection in Linear Models on Testing Performance\\n'\n                   '(Training Data Until Q4 2020, Testing Q1 2021)', fontsize=18)\nfor item in axis.get_legend().get_texts():\n    item.set_fontsize(15)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:27.937965Z","iopub.execute_input":"2021-12-15T00:17:27.938182Z","iopub.status.idle":"2021-12-15T00:17:28.290888Z","shell.execute_reply.started":"2021-12-15T00:17:27.938156Z","shell.execute_reply":"2021-12-15T00:17:28.290078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The figure below illustrates that ``KXY``'s feature selection algorithm is more effective than RFE when using LightGBM, and that it yields better performance than not using any feature selection.","metadata":{}},{"cell_type":"code","source":"cross_sectional_results = pd.DataFrame([lightgbm_all_results, lightgbm_rfe_results, \\\n                                        lightgbm_selected_results], \\\n                    index=['LightGBM Using All Features', \\\n                        'LightGBM With RFE Feature Selection', \\\n                        'LightGBM With KXY Feature Selection']).astype(float)\ncross_sectional_results.T.plot.bar(figsize=(15, 10), fontsize=15, \\\n                                   color=['r', 'y', 'g'])\naxis = plt.gca()\n_ = axis.set_ylabel('Pearson Correlation', fontsize=15)\n_ = axis.set_title('Effect of Feature Selection in LightGBM on Testing Performance\\n'\n                   '(Training Data Until Q4 2020, Testing Q1 2021)', fontsize=18)\nfor item in axis.get_legend().get_texts():\n    item.set_fontsize(15)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:28.292298Z","iopub.execute_input":"2021-12-15T00:17:28.292533Z","iopub.status.idle":"2021-12-15T00:17:28.650932Z","shell.execute_reply.started":"2021-12-15T00:17:28.292504Z","shell.execute_reply":"2021-12-15T00:17:28.650118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### II.4.c Slower performance decay <a name=\"decay\"></a>","metadata":{}},{"cell_type":"code","source":"# Q2 2021 Performance\nshrinkage_perf_q2_2021 = gresearch_score(q2_2021_data, shrinkage_predicted_q2_2021)\nall_perf_q2_2021 = gresearch_score(q2_2021_data, all_predicted_q2_2021)\nlasso_all_perf_q2_2021 = gresearch_score(q2_2021_data, lasso_all_predicted_q2_2021)\nlr_all_perf_q2_2021 = gresearch_score(q2_2021_data, lr_all_predicted_q2_2021)\nlr_shrinkage_perf_q2_2021 = gresearch_score(q2_2021_data, lr_shrinkage_predicted_q2_2021)\nrfe_features_q2_2021_perf = gresearch_score(q2_2021_data, rfe_predicted_q2_2021)\n\n# Q3 2021 Performance\nshrinkage_perf_q3_2021 = gresearch_score(q3_2021_data, shrinkage_predicted_q3_2021)\nall_perf_q3_2021 = gresearch_score(q3_2021_data, all_predicted_q3_2021)\nlasso_all_perf_q3_2021 = gresearch_score(q3_2021_data, lasso_all_predicted_q3_2021)\nlr_all_perf_q3_2021 = gresearch_score(q3_2021_data, lr_all_predicted_q3_2021)\nlr_shrinkage_perf_q3_2021 = gresearch_score(q3_2021_data, lr_shrinkage_predicted_q3_2021)\nrfe_features_q3_2021_perf = gresearch_score(q3_2021_data, rfe_predicted_q3_2021)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:28.652342Z","iopub.execute_input":"2021-12-15T00:17:28.652566Z","iopub.status.idle":"2021-12-15T00:17:37.386264Z","shell.execute_reply.started":"2021-12-15T00:17:28.652538Z","shell.execute_reply":"2021-12-15T00:17:37.385411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Cross-section models\nlightgbm_all_results = {'2021Q1': all_features_perf['Overall (Weighted)'], \\\n               '2021Q2': all_perf_q2_2021['Overall (Weighted)'], \\\n               '2021Q3': all_perf_q3_2021['Overall (Weighted)']}\n\nlightgbm_selected_results = {'2021Q1': shrinkage_perf['Overall (Weighted)'], \\\n               '2021Q2': shrinkage_perf_q2_2021['Overall (Weighted)'], \\\n               '2021Q3': shrinkage_perf_q3_2021['Overall (Weighted)']}\n\nlasso_results = {'2021Q1': lasso_all_features_perf['Overall (Weighted)'], \\\n               '2021Q2': lasso_all_perf_q2_2021['Overall (Weighted)'], \\\n               '2021Q3': lasso_all_perf_q3_2021['Overall (Weighted)']}\n\nlr_results = {'2021Q1': lr_all_features_perf['Overall (Weighted)'], \\\n               '2021Q2': lr_all_perf_q2_2021['Overall (Weighted)'], \\\n               '2021Q3': lr_all_perf_q3_2021['Overall (Weighted)']}\n\nlr_selected_results = {'2021Q1': lr_shrinkage_perf['Overall (Weighted)'], \\\n               '2021Q2': lr_shrinkage_perf_q2_2021['Overall (Weighted)'], \\\n               '2021Q3': lr_shrinkage_perf_q3_2021['Overall (Weighted)']}\n\nlightgbm_rfe_results = {'2021Q1': rfe_features_perf['Overall (Weighted)'], \\\n               '2021Q2': rfe_features_q2_2021_perf['Overall (Weighted)'], \\\n               '2021Q3': rfe_features_q3_2021_perf['Overall (Weighted)']}","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:37.387708Z","iopub.execute_input":"2021-12-15T00:17:37.387945Z","iopub.status.idle":"2021-12-15T00:17:37.437864Z","shell.execute_reply.started":"2021-12-15T00:17:37.387916Z","shell.execute_reply":"2021-12-15T00:17:37.436973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The figure below illustrates that, when using linear models, the *juice* in features selected by ``KXY`` lasts longer than the *juice* in features selected by (cross-validated) LASSO.","metadata":{}},{"cell_type":"code","source":"cross_sectional_results = pd.DataFrame([lr_results, lasso_results, lr_selected_results], \\\n            index=['No Feature Selection', \\\n                    'LASSO Feature Selection', \\\n                    'KXY Feature Selection']).astype(float)\ncross_sectional_results.T.plot.bar(figsize=(15, 10), fontsize=15, color=['r', 'y', 'g'])\naxis = plt.gca()\n_ = axis.set_ylabel('Pearson Correlation', fontsize=15)\n_ = axis.set_title(\\\n    'Effect of Feature Selection in Linear Models on Performance Decay\\n(Training Data Until Q4 2020)',\\\n    fontsize=18)\nfor item in axis.get_legend().get_texts():\n    item.set_fontsize(15)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:37.439601Z","iopub.execute_input":"2021-12-15T00:17:37.439819Z","iopub.status.idle":"2021-12-15T00:17:37.794973Z","shell.execute_reply.started":"2021-12-15T00:17:37.439792Z","shell.execute_reply":"2021-12-15T00:17:37.794159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The figure below illustrates that LightGBM with ``KXY``'s feature selection significantly outperform LightGBM with RFE and LightGBM without feature selection, two quarters in a row after model training.","metadata":{}},{"cell_type":"code","source":"cross_sectional_results = pd.DataFrame([lightgbm_all_results, lightgbm_rfe_results, \\\n                                        lightgbm_selected_results], \\\n            index=['No Feature Selection', \\\n                    'RFE Feature Selection', \\\n                    'KXY Feature Selection']).astype(float)\ncross_sectional_results.T.plot.bar(figsize=(15, 10), fontsize=15, color=['r', 'y', 'g'])\naxis = plt.gca()\n_ = axis.set_ylabel('Pearson Correlation', fontsize=15)\n_ = axis.set_title(\\\n    'Effect of Feature Selection in LightGBM on Performance Decay\\n(Training Data Until Q4 2020)',\\\n    fontsize=18)\nfor item in axis.get_legend().get_texts():\n    item.set_fontsize(15)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:37.796170Z","iopub.execute_input":"2021-12-15T00:17:37.796396Z","iopub.status.idle":"2021-12-15T00:17:38.160801Z","shell.execute_reply.started":"2021-12-15T00:17:37.796360Z","shell.execute_reply":"2021-12-15T00:17:38.160212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# III. Conclusion <a name=\"conclusion\"></a>\n\n*We propose an approach to seemlessly add effective feature selection to any regressor in Python in a single line of code.*\n\nWe show that, in this competition, our approach is competitive with LASSO when the regressor is a linear, and achieves a 20% reduction in the number of features.\n\nWe find that tree-based regressors (LightGBM to be specific), have the potential to outperform linear models in this competition. However, to unlock this potential, it is crucial to mitigate overfitting with effective feature selection. \n\nWe find that the ``KXY`` feature selection approach significantly outperforms RFE, and that RFE performs similarly to not including any feature selection (in this competition).\n\nA major downside of non-linear models in this competition is that they tend to decay faster than linear models. However, thanks to``KXY``'s feature selection, LightGBM consistently outperforms linear models up to 6 months after model training. ","metadata":{}},{"cell_type":"markdown","source":"## III.1 Preparing Submission","metadata":{}},{"cell_type":"code","source":"# Construct training data\nsubmission_training_data = pd.read_parquet('%s/2018Q1.parquet' % PATH)\nsubmission_training_data = submission_training_data.astype(np.float32)\n\nfor i in range(1, len(quarters)-1):\n    testing_data = pd.read_parquet('%s/%s.parquet' % (PATH, quarters[i]))\n    testing_data = testing_data.astype(np.float32)\n    # Update the training set with \n    submission_training_data = submission_training_data.sample(frac=0.2, random_state=28)\n    testing_data = testing_data.sample(frac=0.8, random_state=28)\n    submission_training_data = pd.concat([submission_training_data, testing_data], axis=0)\n    gc.collect()\n    \nmonths_df = submission_training_data['timestamp.YEAR()'].astype(int).astype(str) + \\\n    submission_training_data['timestamp.MONTH()'].astype(int).astype(str).apply(lambda x: x.zfill(2))\nmonths_df = pd.DataFrame(months_df, columns=[\"Month\"])\nsubmission_training_data.drop(columns=['timestamp.MONTH()', 'timestamp.YEAR()'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:17:38.161759Z","iopub.execute_input":"2021-12-15T00:17:38.161958Z","iopub.status.idle":"2021-12-15T00:18:38.360082Z","shell.execute_reply.started":"2021-12-15T00:17:38.161932Z","shell.execute_reply":"2021-12-15T00:18:38.359196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit a LightGBM regressor with KXY feature selection\nsubmission_results = submission_training_data.kxy.fit('Target', lightgbm_regressor_cls, \\\n    problem_type='regression', missing_value_imputation=True, \\\n    train_frac=0.8)","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:18:38.361316Z","iopub.execute_input":"2021-12-15T00:18:38.361538Z","iopub.status.idle":"2021-12-15T00:27:51.044902Z","shell.execute_reply.started":"2021-12-15T00:18:38.361511Z","shell.execute_reply":"2021-12-15T00:27:51.043819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save selected variables and the trained model\nimport lightgbm\nimport pickle as pkl\npkl.dump(submission_training_data.kxy.predictor.selected_variables, open('selected_variables.sav', 'wb'))\nmodel = submission_training_data.kxy.predictor.models[0]._model\nmodel.save_model('lightgbm_regressor.sav', num_iteration=model.best_iteration) ","metadata":{"execution":{"iopub.status.busy":"2021-12-15T00:27:51.046510Z","iopub.execute_input":"2021-12-15T00:27:51.046760Z","iopub.status.idle":"2021-12-15T00:27:51.325047Z","shell.execute_reply.started":"2021-12-15T00:27:51.046729Z","shell.execute_reply":"2021-12-15T00:27:51.324166Z"},"trusted":true},"execution_count":null,"outputs":[]}]}