{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# TL'DR\nThis is the second part in a series of tutorials showcasing how to automate much of feature engineering.\n\nIn [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), we illustrated *how to estimate the best performance you can achieve consistently (i.e. in-sample and out-of-sample) using a set of features, in a single line of code, and without training any model.*\n\nIn Part II (this tutorial), we explain why, when it comes to features, sometimes less is more. Then, building on [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), ***we illustrate how to filter out redundant and non-informative features in a model-free fashion.***\n\nIn [Part III](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-iii), we will build on this tutorial to show you how to seamlessly add shrinkage/feature selection to any regressor in Python.\n\n*Please upvote and share it you found it useful.*\n\n# Table of Contents\n\n- **I. [Background](#background)**\n - **I.1 [When It Comes to Features, Sometimes Less Is More](#less-is-more)**\n - **I.2 [Why Model-Free Feature Selection Matters](#model-free-is-important)** \n - **I.3 [A Simple Model-Free Feature Selection Algorithm](#solution)** \n- **II. [Application](#application)**\n - **II.1 [Getting Started](#setup)**\n - **II.2 [The Set of Candidate Features](#features)**\n - **II.3 [Model-Free Feature Selection in a Single Line of Code](#one-liner)**\n   - **II.3.a [Model-free feature selection for cross-sectional predictions](#cross-sectional)**\n   - **II.3.b [Model-free feature selection for single-asset predictions](#cross-sectional)**\n\n# I. Background <a name=\"background\"></a>\nIn [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i) we argued that a good feature engineering pipeline should achieve two seemingly conflicting objectives. \n\nFirst, it should turn raw inputs $x$ into a feature vector $z$ that is related to the target $y$ in a way that is consistent with the types of patterns that models in our toolbox can actually learn. \n\nSecond, the pipeline should ensure that features $z$ remain as insightful about the target $y$ as raw inputs $x$ are. By the [data processing inequality](https://en.wikipedia.org/wiki/Data_processing_inequality), any feature transformation is bound to lose some *juice* that was in $x$ for predicting $y$, but a good feature transformation should keep this loss of juice to a minimum, while making the *juice* easier to extract using models in our toolbox.\n\nGiven a vector of candidate features $z$, we showed in [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i) how to use the ``kxy`` package to compute the best performance achievable consistently (i.e. in-sample and out-of-sample) using ***all*** candidate features $z$ to predict the target $y$. However, when it comes to features, oftentimes less is more. \n\nIn this tutorial, we will show you how to select the best features in a candidate set, without compromising on predictive power.\n\n## I.1 The Issue With Too Many Features <a name=\"less-is-more\"></a>\n\nLarge feature sets often include features that are redundant or non-informative. \n\n**Non-informative features** are features that cannot contribute to improving the performance of a regressor. Formally, these are features that are statistically *(unconditionally)* independent from the target,  ***and*** statistically independent from the target *conditional on/given **any other feature(s)** in the set*.\n\n**Redundant features** on the other hand are features that, while they might be informative about the target, they cannot contribute to improving the performance of a regressor when used in conjunction with *some other features* in the set of candidates. Formally, these are features for which there exists *a subset of other features conditional on/given which* they are stastically independent from the target.\n\nWorking with redundant or useless features presents several challenges.\n\n* **Less effective:** Some models perform poorly in the presence of redundant features (e.g. OLS with linearly dependent features). Additionally, while non-informative features do not affect the theoretical best performance achievable, in practice, working with non-informative features may degrade model performance. Not only will model training be more prone to overfitting in the presence of non-informative features, but in general it would tend to give non-zero *weights* to features that shouldn't have any. \n* **Less efficient:** memory and compute resources needed to train or run a production model increase with the number of features the model uses.\n* **Harder to explain:** The more features a model uses, the harder it is to explain what it does, especially when some features are redundant or non-informative.\n* **More outages/downtime:** Different features could be served in production by different pipelines (e.g. the raw inputs might come from different databases or tables, processing could be done by different processes etc.). In such as case, the more features a deployed model uses, the more feature delivery system outages the model will be exposed to, which will eventually increase downtime as any feature outage will likely take down prediction capabilities.\n* **Faster drift:** Data distributions drift over time, causing production models to be retrained. The higher the number of features, the quicker the drift will arise, and the more often production models will need to be retrained.\n\nWhen engineering features, care should always be taken to avoid non-informative and redundant features.\n\n## I.2 The Importance of Model-Free Feature Selection <a name=\"model-free-is-important\"></a>\n\nBecause the choice of features (in particular the inclusion or not of redundant or non-informative features) affects model performance, feature selection is best done before and independently from model training. \n\nTying feature selection to model training leaves a few questions unanswered and open to guesses. Say for instance that the trained model didn't perform as well as the theoretical-best (as per [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i)). Why could that be? Is it because of a poor choice of hyper-parameters or model-class? Or did the presence of redundant and/or non-informative features give the optimizer a hard time?\n\nPopular (model-based) feature selection methods can hardly answer these questions. \n\nLet's review a few of these approaches so as to illustrate why.\n\n- **LASSO:** Features to which LASSO gives big weights are features that are linearly related to the target. LASSO can easily give a non-informative feature a bigger weight than a very informative feature, simply because that very informative feature is in a non-linear relationship with the target. Thus, while LASSO works great to learn linear patterns between features and the target (including in the presence of redundant and/or non-informative features), it does not extend to non-linear relationships.\n\n- **PCA:** PCA can be used to linearly transform a feature set into a new set of features that are mutually decorrelated (i.e. not linearly related). New features are constructed so as to maximize the amount of variance of the original feature set they account for, so that we may only retain $q < d$ of the $d$ new features while accounting for almost all the variance of the original set. However, decorrelation of features does not rule out feature redundancy. Two features could be decorrelated but mutually redundant. E.g. If $x$ is a standard Gaussian, $x$ and $x^2$ are decorrelated, but clearly $x^2$ is redundant when $x$ is known. As a more practical example, a longitude (resp. a latitude) intuitively ought to statistically independent from a house number, but knowing both the longitude and the latitude makes the house number redundant for geo-localization. Moreover, preserving the variance of the original feature set does not imply preserving its predective power for forecasting the target. Thus, PCA can neither reliably remove redundant features, nor can it remove non-informative features or preserve the theoretical-best performance achievable by the initial set of features.\n\n- **Recursive Features Elimination (RFE):** RFE selects features by starting with all candidate features and removing features one at a time based on a model-specific feature importance score. This approach can be resource-intensive and, depending on the feature importance score used, it can also present additional limitations. For instance, most methods (e.g. *tree-based/impurity-based feature importance*, *permutation-based feature importance*, and *SHAP*) do not detect non-informative features. Instead, they detect features that the specific trained model to which they are applied does not rely on. However, because a trained model does not rely on a feature does not make it intrinsically non-informative. The model could be poorly trained, or the model class could be inadequate. These methods aren't capable of detecting redundant features either. Let's take the extreme example of identical features. A model trained with two identical features might expect them to remain identical out-of-sample. Applying permutation-based feature importance on these two features will likely underestimate their importance to the trained model overall, but more importantly both will be given the same importance, even though one is redundant relative to the other. SHAP will also give both features the same importance. In fact, theoretically, SHAP cannot not account for redundant features at all. The second most important features as per SHAP could very well be totally useless when the most important feature is known.\n\n\n\n## I.3 A Simple Model-Free Feature Selection Algorithm <a name=\"solution\"></a>\nBuilding on how to estimate the theoretical-best performance achievable using an input vector to predict a target, which we recalled in [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), [[1]](#paper1) proposed the following simple model-free variable selection algorithm.\n\n#### Algorithm (Model-Free Variable Selection)\n - **Inputs**: \n     - $z \\in \\mathbb{R}^d$: features\n     - $y \\in \\mathbb{R}$: target\n     - $u \\to \\bar{\\rho}(P_{y, u})$: function calculating the highest Pearson's correlation between prediction and target achievable using features $u$ to predict target $y$.\n - **Outputs**: \n     - Selection orders $S$: indices of features selected in decreasing order of importance\n     - Running achievable performance $P$: the highest Pearson correlation achievable associated to $S$.\n - **Initialization**: $z^\\prime = []; S = []; P = [];$\n - For $i$ from $1$ to $d$: \n     - $s_i = \\underset{k \\in [1, d], ~ k \\notin S}{\\text{argmax}} ~~~ \\bar{\\rho}\\left(P_{y, z^\\prime + [z[k]]}\\right)$\n     - $S = S + [s_i]$ \n     - $z^\\prime = z^\\prime + [z[s_i]]$\n     \nIn plain English, the first feature selected is the feature that, when used by itself, has the highest achievable performance; in this case the highest achievable Pearson correlation between the prediction and the target. More generally, the $i$-th feature selected is the one, out of all $(d-i+1)$ features not yet selected, such that when it is used in conjuction with all $(i-1)$ previously selected features, it yields the highest achievable performance. The algorithm returns features in the order they were selected, from the most important to the least important, as well as the highest performance achievable using the top-$i$ features for any $i \\in [1, d]$. \n\nNote that, by construction, at each step, this algorithm gives no importance to feautures that are either non-informative, or redundant with respect to previously selected features.\n\nIn Part III, we will see how this algorithm can be used to turn any regressor into a boosted-regressor that avoids redundant and non-informative features.\n\n\n\n**Reference:**\n\n- [1]<a name=\"paper1\"></a> Samo, Y.L.K., 2021. LeanML: A Design Pattern To Slash Avoidable Wastes in Machine Learning Projects. arXiv preprint arXiv:2107.08066.\n- [2]<a name=\"paper2\"></a> Samo, Y.L.K., 2021, March. Inductive Mutual Information Estimation: A Convex Maximum-Entropy Copula Approach. In International Conference on Artificial Intelligence and Statistics (pp. 2242-2250). PMLR.\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# II. Application <a name=\"application\"></a>\n\n## II.1 Getting Started <a name=\"setup\"></a>\nWe will use the ``kxy`` package. It requires an API key. See [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i) on how to get yours.","metadata":{}},{"cell_type":"code","source":"!pip install kxy -U","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:02:57.993898Z","iopub.execute_input":"2021-11-19T23:02:57.99661Z","iopub.status.idle":"2021-11-19T23:03:08.836736Z","shell.execute_reply.started":"2021-11-19T23:02:57.995169Z","shell.execute_reply":"2021-11-19T23:03:08.835401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport pprint as pp\nimport kxy","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:03:08.839168Z","iopub.execute_input":"2021-11-19T23:03:08.839461Z","iopub.status.idle":"2021-11-19T23:03:09.902463Z","shell.execute_reply.started":"2021-11-19T23:03:08.839428Z","shell.execute_reply":"2021-11-19T23:03:09.901489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nkxy_api_key = user_secrets.get_secret('KXY_API_KEY')\nos.environ['KXY_API_KEY'] = kxy_api_key","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:03:09.9054Z","iopub.execute_input":"2021-11-19T23:03:09.905739Z","iopub.status.idle":"2021-11-19T23:03:10.402933Z","shell.execute_reply.started":"2021-11-19T23:03:09.905694Z","shell.execute_reply":"2021-11-19T23:03:10.40201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II.2 The Set of Candidate Features We Will Be Using <a name=\"features\"></a>\n\nHere we generate a set of 37 candidate features, temporal and cross-sectional.","metadata":{}},{"cell_type":"code","source":"TRAIN_CSV = '/kaggle/input/g-research-crypto-forecasting/train.csv'\ndf_train = pd.read_csv(TRAIN_CSV)\n\ndef nanmaxmmin(a, axis=None, out=None):\n    ''' '''\n    return np.nanmax(a, axis=axis, out=out)-np.nanmin(a, axis=axis, out=out)\n\n\ndef get_features(df):\n    ''' \n    An example function generating a candidate list of features.\n    '''\n    features = df[['Count', 'Open', 'High', 'Low', 'Close', \\\n                                    'Volume', 'VWAP','timestamp', 'Target', 'Asset_ID']].copy()\n    # Upper shadow\n    features['UPS'] = (df['High']-np.maximum(df['Close'], df['Open']))\n    features['UPS'] = features['UPS'].astype(np.float16)\n    \n    # Lower shadow\n    features['LOS'] = (np.minimum(df['Close'], df['Open'])-df['Low'])\n    features['LOS'] = features['LOS'].astype(np.float16)\n    \n    # High-Low range\n    features['RNG'] = ((features['High']-features['Low'])/features['VWAP'])\n    features['RNG'] = features['RNG'].astype(np.float16)\n    \n    # Daily move\n    features['MOV'] = ((features['Close']-features['Open'])/features['VWAP'])\n    features['MOV'] = features['MOV'].astype(np.float16)\n    \n    # Close vs. VWAP\n    features['CLS'] = ((features['Close']-features['VWAP'])/features['VWAP'])\n    features['CLS'] = features['CLS'].astype(np.float16)\n    \n    # Log-volume\n    features['LOGVOL'] = np.log(1.+features['Volume'])\n    features['LOGVOL'] = features['LOGVOL'].astype(np.float16)\n    \n    # Log-count\n    features['LOGCNT'] = np.log(1.+features['Count'])\n    features['LOGCNT'] = features['LOGCNT'].astype(np.float16)\n    \n    # Drop raw inputs\n    features.drop(columns=['Open', 'High', 'Low', 'Close', 'Volume', 'VWAP', \\\n                           'Count'], errors='ignore', inplace=True)\n    \n    # Previous target !!WARNING: THIS FEATURE IS NOT TRADEABLE !!\n    features['PREVTARGET'] = features.groupby('Asset_ID')['Target'].shift(1)\n    \n    # Enrich the features dataframe with some temporal feautures \n    # (specifically, some stats on the last hour worth of bars)\n    features = features.kxy.temporal_features(max_lag=3, \\\n                exclude=['LOGVOL', 'LOGCNT', 'timestamp', 'Target'],\n                groupby='Asset_ID')\n    \n    # Enrich the features dataframe context around the time\n    # (e.g. hour, day of the week, etc.)\n    time_features = features.kxy.process_time_columns(['timestamp'])\n    features.drop(columns=['timestamp'], errors='ignore', inplace=True)\n    features = pd.concat([features, time_features], axis=1)\n    \n    return features","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:03:10.404764Z","iopub.execute_input":"2021-11-19T23:03:10.405004Z","iopub.status.idle":"2021-11-19T23:04:06.966568Z","shell.execute_reply.started":"2021-11-19T23:03:10.404977Z","shell.execute_reply":"2021-11-19T23:04:06.965618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    # Reading candidate features from disk\n    training_features = pd.read_parquet('../input/cross-asset-featuresparquet/cross_asset_features.parquet')\n    \nexcept:\n    # Randomly select 10% of days to speed-up features generation\n    df_train['DAYSTR'] = pd.to_datetime(df_train['timestamp'], unit='s').apply(lambda x: x.strftime(\"%Y%m%d\"))\n    all_days = list(set([_ for _ in df_train['DAYSTR'].values]))\n    selected_days = np.random.choice(all_days, size=int(len(all_days)/10), replace=False)\n    df_train = df_train[df_train['DAYSTR'].isin(selected_days)]\n    df_train.drop(columns=['DAYSTR'], errors='ignore', inplace=True)\n    # Generating candidate features\n    training_features = get_features(df_train)\n    # Saving to disk\n    to_save = training_features.astype(np.float32)\n    to_save.to_parquet('cross_asset_features.parquet')\n    del to_save\ntraining_features","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:06.967782Z","iopub.execute_input":"2021-11-19T23:04:06.967999Z","iopub.status.idle":"2021-11-19T23:04:09.749637Z","shell.execute_reply.started":"2021-11-19T23:04:06.967973Z","shell.execute_reply":"2021-11-19T23:04:09.748822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Printing all feautures\nall_features = sorted([_ for _ in training_features.columns])\npp.pprint(all_features)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:09.750755Z","iopub.execute_input":"2021-11-19T23:04:09.750955Z","iopub.status.idle":"2021-11-19T23:04:09.759868Z","shell.execute_reply.started":"2021-11-19T23:04:09.750931Z","shell.execute_reply":"2021-11-19T23:04:09.7567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II.3 Model-Free Feature Selection in a Single Line of Code <a name=\"one-liner\"></a>\nThe syntax is ``features_df.kxy.variable_selection(target_column, problem_type='regression')`` and it works on any pandas DataFrame object, so long as you import the ``kxy`` package.\n\n### II.3.a Case 1: For Cross-Asset Predictions <a name=\"cross-sectional\"></a>\nThis is how to select features using ``kxy`` when you intend to build a single model to trade all cryptocurrencies.","metadata":{}},{"cell_type":"code","source":"cs_variable_selection_df = training_features.kxy.variable_selection('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:09.761108Z","iopub.execute_input":"2021-11-19T23:04:09.761406Z","iopub.status.idle":"2021-11-19T23:04:32.423864Z","shell.execute_reply.started":"2021-11-19T23:04:09.761374Z","shell.execute_reply":"2021-11-19T23:04:32.422799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cs_variable_selection_df['Running Achievable Pearson Correlation'] = \\\n    cs_variable_selection_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\ncs_variable_selection_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:32.425355Z","iopub.execute_input":"2021-11-19T23:04:32.425622Z","iopub.status.idle":"2021-11-19T23:04:32.447169Z","shell.execute_reply.started":"2021-11-19T23:04:32.42559Z","shell.execute_reply":"2021-11-19T23:04:32.446388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.3.b Case 2: For Single-Asset Predictions <a name=\"single-asset\"></a>\n\nThis is how to select features using ``kxy`` when you intend to build one model per cryptocurrency.","metadata":{}},{"cell_type":"code","source":"ASSET_CSV = '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'\nasset_details = pd.read_csv(ASSET_CSV)\nasset_details.set_index(['Asset_ID'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:32.448751Z","iopub.execute_input":"2021-11-19T23:04:32.449348Z","iopub.status.idle":"2021-11-19T23:04:32.480474Z","shell.execute_reply.started":"2021-11-19T23:04:32.449307Z","shell.execute_reply":"2021-11-19T23:04:32.479479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Bitcoin Models","metadata":{}},{"cell_type":"code","source":"asset_id = asset_details[asset_details['Asset_Name']=='Bitcoin'].index[0]\ndf = training_features[training_features['Asset_ID']==asset_id]\nbtc_variable_selection_df = df.kxy.variable_selection('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:32.483858Z","iopub.execute_input":"2021-11-19T23:04:32.48425Z","iopub.status.idle":"2021-11-19T23:04:37.034848Z","shell.execute_reply.started":"2021-11-19T23:04:32.484173Z","shell.execute_reply":"2021-11-19T23:04:37.033535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc_variable_selection_df['Running Achievable Pearson Correlation'] = \\\n    btc_variable_selection_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\nbtc_variable_selection_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:37.037007Z","iopub.execute_input":"2021-11-19T23:04:37.037432Z","iopub.status.idle":"2021-11-19T23:04:37.058953Z","shell.execute_reply.started":"2021-11-19T23:04:37.037376Z","shell.execute_reply":"2021-11-19T23:04:37.057593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Ethereum Models","metadata":{}},{"cell_type":"code","source":"asset_id = asset_details[asset_details['Asset_Name']=='Ethereum'].index[0]\ndf = training_features[training_features['Asset_ID']==asset_id]\neth_variable_selection_df = df.kxy.variable_selection('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:37.061231Z","iopub.execute_input":"2021-11-19T23:04:37.061607Z","iopub.status.idle":"2021-11-19T23:04:42.298002Z","shell.execute_reply.started":"2021-11-19T23:04:37.061553Z","shell.execute_reply":"2021-11-19T23:04:42.29729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth_variable_selection_df['Running Achievable Pearson Correlation'] = \\\n    eth_variable_selection_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\neth_variable_selection_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:42.29981Z","iopub.execute_input":"2021-11-19T23:04:42.300401Z","iopub.status.idle":"2021-11-19T23:04:42.319004Z","shell.execute_reply.started":"2021-11-19T23:04:42.300346Z","shell.execute_reply":"2021-11-19T23:04:42.31823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Litecoin Models","metadata":{}},{"cell_type":"code","source":"asset_id = asset_details[asset_details['Asset_Name']=='Litecoin'].index[0]\ndf = training_features[training_features['Asset_ID']==asset_id]\nltc_variable_selection_df = df.kxy.variable_selection('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:42.320437Z","iopub.execute_input":"2021-11-19T23:04:42.321002Z","iopub.status.idle":"2021-11-19T23:04:46.636513Z","shell.execute_reply.started":"2021-11-19T23:04:42.320948Z","shell.execute_reply":"2021-11-19T23:04:46.635661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ltc_variable_selection_df['Running Achievable Pearson Correlation'] = \\\n    ltc_variable_selection_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\nltc_variable_selection_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:46.637987Z","iopub.execute_input":"2021-11-19T23:04:46.638215Z","iopub.status.idle":"2021-11-19T23:04:46.654059Z","shell.execute_reply.started":"2021-11-19T23:04:46.638169Z","shell.execute_reply":"2021-11-19T23:04:46.652989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Dogecoin Models","metadata":{}},{"cell_type":"code","source":"asset_id = asset_details[asset_details['Asset_Name']=='Dogecoin'].index[0]\ndf = training_features[training_features['Asset_ID']==asset_id]\ndoge_variable_selection_df = df.kxy.variable_selection('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:46.655371Z","iopub.execute_input":"2021-11-19T23:04:46.655781Z","iopub.status.idle":"2021-11-19T23:04:50.786378Z","shell.execute_reply.started":"2021-11-19T23:04:46.655736Z","shell.execute_reply":"2021-11-19T23:04:50.785347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"doge_variable_selection_df['Running Achievable Pearson Correlation'] = \\\n    doge_variable_selection_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\ndoge_variable_selection_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:50.788121Z","iopub.execute_input":"2021-11-19T23:04:50.788418Z","iopub.status.idle":"2021-11-19T23:04:50.806075Z","shell.execute_reply.started":"2021-11-19T23:04:50.788382Z","shell.execute_reply":"2021-11-19T23:04:50.805329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Maker Models","metadata":{}},{"cell_type":"code","source":"asset_id = asset_details[asset_details['Asset_Name']=='Maker'].index[0]\ndf = training_features[training_features['Asset_ID']==asset_id]\nmkr_variable_selection_df = df.kxy.variable_selection('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:50.807816Z","iopub.execute_input":"2021-11-19T23:04:50.808077Z","iopub.status.idle":"2021-11-19T23:04:54.351144Z","shell.execute_reply.started":"2021-11-19T23:04:50.808045Z","shell.execute_reply":"2021-11-19T23:04:54.350428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mkr_variable_selection_df['Running Achievable Pearson Correlation'] = \\\n    mkr_variable_selection_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\nmkr_variable_selection_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:04:54.352083Z","iopub.execute_input":"2021-11-19T23:04:54.352356Z","iopub.status.idle":"2021-11-19T23:04:54.371079Z","shell.execute_reply.started":"2021-11-19T23:04:54.352322Z","shell.execute_reply":"2021-11-19T23:04:54.369871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Remark\n\nAs can be seen above, **PREVTARGET** (and related features) seem to contribute a great deal to the highest achievable performance. This is not surprising because they have look-ahead bias. Indeed, trading bars in this competition are minute bars, but the target is calculated using log-returns over the next $15$ minutes. So, the previous target actually peeks into $14$ minutes in the future.\n\nHere's the cross-asset variable selection analysis without **PREVTARGET** features.","metadata":{}},{"cell_type":"code","source":"clean_features = training_features.drop(\n    columns=[_ for _ in training_features.columns if 'PREVTARGET' in _], errors='ignore')\nclean_feature_names = sorted([_ for _ in clean_features.columns])\npp.pprint(clean_feature_names)\nclean_ca_data_valuation_df = clean_features.kxy.variable_selection(\n    'Target', problem_type='regression')\nclean_ca_data_valuation_df['Running Achievable Pearson Correlation'] = \\\n    clean_ca_data_valuation_df['Running Achievable R-Squared'].apply(lambda x: '%.2f' % np.sqrt(float(x)))\nclean_ca_data_valuation_df","metadata":{"execution":{"iopub.status.busy":"2021-11-19T23:18:14.911234Z","iopub.execute_input":"2021-11-19T23:18:14.911569Z","iopub.status.idle":"2021-11-19T23:18:29.762841Z","shell.execute_reply.started":"2021-11-19T23:18:14.911538Z","shell.execute_reply":"2021-11-19T23:18:29.761674Z"},"trusted":true},"execution_count":null,"outputs":[]}]}