{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# TL'DR\nThis is the first part in a series of tutorials showcasing how to automate much of feature engineering. \n\nIn Part I (this tutorial), we show you ***how to estimate the best performance you can achieve consistently (i.e. in-sample and out-of-sample) using a set of features, in a single line of code, and without training any model.***\n\nIn [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), we will build on this tutorial to show you *how to seamlessly remove non-informative and redundant features from a set of candidate features.*\n\nIn [Part III](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-iii), we will build on Part II and show you how to seamlessly add shrinkage/feature selection to any regressor in Python.\n\n*Please upvote and share it you found it useful.*\n\n# Table of Contents\n\n- **I. [Math Background](#background)**\n - **I.1 [Problem Formulation](#problem)**\n - **I.2 [Solution](#solution)**\n- **II. [Application](#application)**\n - **II.1 [Getting Started](#setup)**\n   - **II.1.a [How to install the kxy package](#install)**\n   - **II.1.b [How to get and use a kxy API key](#api)**\n   - **II.1.c [The set of features we are using in this tutorial](#features)**\n - **II.2 [How to Calculate the Highest Performance Achievable in a Single Line of Code](#one-liner)**\n   - **II.2.a [Highest performance achievable by a cross-sectional model](#cross-sectional)**\n   - **II.2.b [Highest performance achievable by single-asset models](#cross-sectional)**\n - **II.3 [Summary of Findings](#summary)**\n\n# I. Background <a name=\"background\"></a>\nTo set the stage, we formally discuss challenges specific to feature engineering, and we recall new ML findings upon which this tutorial is based.\n\n\n## I.1 Problem Formulation (Memoryless Case) <a name=\"problem\"></a>\nFormally, we are interested in predicting a target variable $y \\in \\mathbb{R}$ given some raw inputs $x \\in \\mathbb{R}^d$.\n\nAlthough all the *juice* we need to predict $y$ is fully contained in the raw inputs $x$, in practice, the tabular models we use to extract this juice can only learn specific types of input(s)-output patterns. \n\nFor instance, linear models (e.g. OLS, LASSO, Ridge, ElasticNet etc.) can only capture linear relationships between inputs and the target. Similarly, decision tree regressors work best in situations where a *good global regressor* can be learned by partitioning the input space into $m$ disjoint subsets using simple decision rules, and learning $m$ *very simple local regressors*. Along the same lines, kernel-Ridge regressors and Gaussian process regressors with smooth kernels assume that inputs that are close to each other (in Euclidean distance) will always have targets that are also close to each other (in Euclidean distance). The same holds for nearest neighbors regressors. Beyond smoothness, kernel-based regressors can also be used to learn global patterns in situations where inputs that are close to each other in a given norm, will have outputs that are close to each other in Euclidean norm. The norm can be chosen to learn periodic or seasonal patterns, with or without a trend, to name but a few.\n\nOftentimes, the natural representation of raw inputs $x$ is inconsistent with the types of patterns our regression models rely on, even when $x$ is very informative about $y$. \n\nFor instance, $y$ might not be linear in $x$, but it could be linear in $x^2$. In such a case, a linear regressor trained on $(y, x)$ would perform poorly, but a linear regressor trained on $(y, x^2)$ could perform very well. \n\nThe same applies to tree-based models. Decision rules are typically as simple as whether a variable is above a certain threshold (i.e. whether an input point falls above or below a hyperplane in the input space). However, the right partition of the raw inputs space can sometimes be hard to generate using hyperplanes. Consider for instance the generative model (a.k.a. the ground truth) $y = \\mu_0 + \\epsilon ~~\\text{  if  }~~ x_1^2 + x_2^2 < 1 ~~\\text{  else  }~~ \\mu_1 + \\epsilon,$ where $\\mu_0 \\neq \\mu_1$ are two constants and $\\epsilon$ a noise term. \n\nIf we train a regression tree directly on inputs $(x_1, x_2)$, then we will need to chain a lot of conditions of the form $x_1 < a$ and $x_2 < b$ to mimic the condition $x_1^2 + x_2^2 < 1$. If it is unclear why, imagine having to draw a circle (the true decision boundary) using intersections of vertical and horizontal lines. You'll end up with a lot of rectangles, each corresponding to a separate local regressor. Training that many regressors could result in overfitting as each rectangle wouldn't have enough data to reliably estimate $\\mu_0$ or $\\mu_1$. \n\nThat said, if we introduce the feature $z:= x_1^2+x_2^2$ and train the tree using $z$ as input, then with only a single rule, namely $z < 1$, we can recover the true decision rule and the true model.\n\nIn general, a successful feature engineering pipeline requires three ingredients:\n\n- **A) Valuable Data -** Raw inputs $x$ need to be informative about the target of interest $y$. No model or feature transformation can compensate for the fact that the raw inputs aren't informative about the target.\n- **B) Valuable Features -** The features $z:=f(x)$ that will be fed as inputs to our models need to be as informative about the target $y$ as possible. By the [data processing inequality](https://en.wikipedia.org/wiki/Data_processing_inequality), any feature transformation is necessarily *juice-reducing*, so we need to keep the reduction of *juice* to a minimum.\n- **C) Features-Model Adequacy -** The relationship between features $z$ and $y$ should be consistent with the types of patterns models in our toolbox can reliably learn.\n\nRequirement C) is typically met by looking for features that are likely to have a simpler relationship to the target, one that models in our toolbox can reliably learn. In this competition for instance, the open (resp. high, low, close) price is unlikely to reveal much about the target, when used by itself. However, differences or ratios of these quantities indicate magnitudes of certain types of moves that may be informative about the target (in particular during trending markets).\n\nOn the other hand, meeting requirement B) by trial-and-error can be very time-consuming. This usually entails trying our features on a range of models and seeing what performance we get. In this approach, when trained models do not yield the desired performance, it is never clear which of our feature transformation and our model training is to blame. Is the poor performance due to a poor choice of model hyperparameters, using the wrong class of models, or the fact that our feature transformation missed out on a lot of juice that was originally in raw inputs $x$? Even when our trained models performed well enough out-of-sample, it would still be useful to know if we could do better. \n\n\n\n## I.2 Solution (Memoryless Case) <a name=\"solution\"></a>\n\nTo meet requirement B) above without resorting to a trial-and-error approach, we need a cost-effective way of estimating the best performance we could reliably achieve when using a set of features $z$ to predict a target $y$ (e.g. the highest Pearson's correlation $\\bar{\\rho}(P_{y, z})$ between prediction and target). \n\n$\\bar{\\rho}(P_{y, z})$ tells us how much juice our features still have, while comparing $\\bar{\\rho}(P_{y, z})$ to $\\bar{\\rho}(P_{y, x})$ (i.e. the highest Pearson's correlation achievable between predictions of $y$ made using $x$, no matter the feature transformation), we may determine how much juice that is in $x$ about $y$ our features $z$ are missing out on. \n\nNow, how exactly do we calculate $\\bar{\\rho}(P_{y, z})$? A *good* regression model $\\mathcal{M}$ would usually take the form $y = g(z) + \\epsilon,$ where the residual $\\epsilon$ is at least mean-zero and decorrelated from the prediction $g(z)$; mean-zero because otherwise the model would be biased, and decorrelated residuals because otherwise there will be residual juice in $z$ about $y$ that $g(z)$ is not accounting for.\n\nUsing such a model, the correlation between the prediction and the target reads $$\\text{Corr}(y, g(z)) = \\frac{\\text{Cov}(y, g(z))}{\\sqrt{\\text{Var}(y)\\text{Var}(g(z))}} = \\sqrt{\\frac{\\text{Var}(g(z))}{\\text{Var}(y)}} := \\sqrt{R^2(\\mathcal{M})},$$ where $R^2(\\mathcal{M})$ is the R-squared of our model $\\mathcal{M}$. \n\nThus, $$\\bar{\\rho}(P_{y, z}) := \\sqrt{\\bar{R}^2(P_{y, z})}$$ where $\\bar{R}^2\\left(P_{y, z}\\right)$ is the highest R-squared achievable using $z$ to predict $y$.\n\nIn [[1]](#paper1), it was shown that $\\bar{R}^2\\left(P_{y, z}\\right) := 1- e^{-2I(y; z)},$ where $I(y; z)$ is the mutual information between $y$ and $z$.  \n\nThe **kxy** Python package we use in this tutorial implements this formula, using the mutual information estimator of [[2]](#paper2).\n\n\n## I.3 Extension to Time Series\n\nSo far we have only considered memoryless problems. These are problems where, although observations might have been collected in a chronological order, the order in which, or times at which, observations were collected is irrelevant to solve the problem. Time series problems on the other hand exhibit memory that could be useful to predict the target.\n\nNonetheless, the two formalisms are fairly similar. In effect, most time series regression models (e.g. ARIMA, RNN, LSTM etc.) are conceptually made of two building blocks: \n\n- A set of temporal features that encode the full extent to which time matters (i.e. what temporal patterns the model will attempt to exploit).\n- A memoryless or tabular regression model that uses the temporal features above as inputs.\n\nGood temporal features are typically found either by trial-and-error (e.g. using EDA), or learned implicitly and jointly with the regression model (e.g. LSTM). Either way, once a set of candidate features (temporal and/or cross-sectional) has been found, we may estimate the highest performance achievable using the result of the previous section. \n\n\n## I.4 Effect of Nonstationarity\n\nStrictly speaking, the extension to time series above requires the mutual information $I\\left(y_t, z_t\\right)$ to be the same for any time $t$.\n\nIn practice, good temporal features $z_t$ should be such that their dependency structure with $y_t$  (a.k.a. the copula $C(y_t, z_t)$) does not change over time, or at least changes slowly relative to the amount of observations required to train the regression model. When this is not the case, model performance will decay rapidly in production. \n\nThe foregoing pratical requirement is stronger than our theoretical requirement that $I\\left(y_t, z_t\\right)$ be time-invariant because $I\\left(y_t, z_t\\right)$ is fully determined by $C(y_t, z_t)$.\n\nCoincidentally, when applied to time series, the mutual information estimator [[2]](#paper2) only requires the joint copula between target and features $C(y_t, z_t)$ to be invariant over time or slow-changing. It does not require $\\{y_t, x_t\\}$ or $\\{y_t, z_t\\}$ to be stationary.\n\n\n\n**Reference:**\n\n- [1]<a name=\"paper1\"></a> Samo, Y.L.K., 2021. LeanML: A Design Pattern To Slash Avoidable Wastes in Machine Learning Projects. arXiv preprint arXiv:2107.08066.\n- [2]<a name=\"paper2\"></a> Samo, Y.L.K., 2021, March. Inductive Mutual Information Estimation: A Convex Maximum-Entropy Copula Approach. In International Conference on Artificial Intelligence and Statistics (pp. 2242-2250). PMLR.\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# II. Application\nBecause mutual information estimation is compute-intensive, we implemented [[1]](#paper1) as a serverless Function-As-A-Service tool, the ``kxy`` Python package. \n\nAll analyses provided in the ``kxy`` package are available as simple method calls from any pandas dataframe object; all you need is to install and import the ``kxy`` Python package. \n\nOur backend does the work for you, and you only get charged for the compute resource used to execute your function. \n\n``kxy`` *is compeletely free to use in Kaggle competitions. Just sign up and use the promotional code KAGGLE21.*\n\n## II.1 Getting Started <a name=\"setup\"></a>\n### II.1.a Installation <a name=\"install\"></a>","metadata":{}},{"cell_type":"code","source":"!pip install kxy -U","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:38:15.138235Z","iopub.execute_input":"2021-11-24T11:38:15.138605Z","iopub.status.idle":"2021-11-24T11:38:26.878458Z","shell.execute_reply.started":"2021-11-24T11:38:15.138511Z","shell.execute_reply":"2021-11-24T11:38:26.877308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport pprint as pp\nimport kxy","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:38:26.880164Z","iopub.execute_input":"2021-11-24T11:38:26.880503Z","iopub.status.idle":"2021-11-24T11:38:28.025155Z","shell.execute_reply.started":"2021-11-24T11:38:26.880454Z","shell.execute_reply":"2021-11-24T11:38:28.023893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.1.b Working with your KXY API key <a name=\"api\"></a>\n\nTo get an API key, simply create an account [here](https://www.kxy.ai/portal). Once done, you'll find your API key [here](https://www.kxy.ai/portal/profile/identity/). \n\nYour API key should either be defined through the **KXY_API_KEY** environment variable, or by excecuting the command ``kxy configure <your_api_key>`` after installing the **kxy** Python package. \n\n**Note:** API keys should not be shared. If you are using the ``kxy`` package in a public notebook, we recommend recording your API key as a Kaggle secret and setting the ``KXY_API_KEY`` environment variable using the corresponding Kaggle secret. See the code-block below.","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nkxy_api_key = user_secrets.get_secret('KXY_API_KEY')\nos.environ['KXY_API_KEY'] = kxy_api_key","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:38:28.026715Z","iopub.execute_input":"2021-11-24T11:38:28.027177Z","iopub.status.idle":"2021-11-24T11:38:28.169108Z","shell.execute_reply.started":"2021-11-24T11:38:28.027125Z","shell.execute_reply":"2021-11-24T11:38:28.167987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.1.c Generating Candidate Features <a name=\"features\"></a>\n\nHere we generate a set of 37 candidate features, temporal and cross-sectional.","metadata":{}},{"cell_type":"code","source":"TRAIN_CSV = '/kaggle/input/g-research-crypto-forecasting/train.csv'\ndf_train = pd.read_csv(TRAIN_CSV)\n\ndef nanmaxmmin(a, axis=None, out=None):\n    ''' '''\n    return np.nanmax(a, axis=axis, out=out)-np.nanmin(a, axis=axis, out=out)\n\n\ndef get_features(df):\n    ''' \n    An example function generating a candidate list of features.\n    '''\n    features = df[['Count', 'Open', 'High', 'Low', 'Close', \\\n                                    'Volume', 'VWAP','timestamp', 'Target', 'Asset_ID']].copy()\n    # Upper shadow\n    features['UPS'] = (df['High']-np.maximum(df['Close'], df['Open']))\n    features['UPS'] = features['UPS'].astype(np.float16)\n    \n    # Lower shadow\n    features['LOS'] = (np.minimum(df['Close'], df['Open'])-df['Low'])\n    features['LOS'] = features['LOS'].astype(np.float16)\n    \n    # High-Low range\n    features['RNG'] = ((features['High']-features['Low'])/features['VWAP'])\n    features['RNG'] = features['RNG'].astype(np.float16)\n    \n    # Daily move\n    features['MOV'] = ((features['Close']-features['Open'])/features['VWAP'])\n    features['MOV'] = features['MOV'].astype(np.float16)\n    \n    # Close vs. VWAP\n    features['CLS'] = ((features['Close']-features['VWAP'])/features['VWAP'])\n    features['CLS'] = features['CLS'].astype(np.float16)\n    \n    # Log-volume\n    features['LOGVOL'] = np.log(1.+features['Volume'])\n    features['LOGVOL'] = features['LOGVOL'].astype(np.float16)\n    \n    # Log-count\n    features['LOGCNT'] = np.log(1.+features['Count'])\n    features['LOGCNT'] = features['LOGCNT'].astype(np.float16)\n    \n    # Drop raw inputs\n    features.drop(columns=['Open', 'High', 'Low', 'Close', 'Volume', 'VWAP', \\\n                           'Count'], errors='ignore', inplace=True)\n    \n    # Previous target !! WARNING: THIS FEATURE IS NOT TRADEABLE !!\n    features['PREVTARGET'] = features.groupby('Asset_ID')['Target'].shift(1)\n    \n    # Enrich the features dataframe with some temporal feautures \n    # (specifically, some stats on the 4 bars worth of bars)\n    features = features.kxy.temporal_features(max_lag=3, \\\n                exclude=['LOGVOL', 'LOGCNT', 'timestamp', 'Target'],\n                groupby='Asset_ID')\n    \n    # Enrich the features dataframe context around the time\n    # (e.g. hour, day of the week, etc.)\n    time_features = features.kxy.process_time_columns(['timestamp'])\n    features.drop(columns=['timestamp'], errors='ignore', inplace=True)\n    features = pd.concat([features, time_features], axis=1)\n    \n    return features","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:38:28.173075Z","iopub.execute_input":"2021-11-24T11:38:28.173676Z","iopub.status.idle":"2021-11-24T11:39:31.881957Z","shell.execute_reply.started":"2021-11-24T11:38:28.173623Z","shell.execute_reply":"2021-11-24T11:39:31.881241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    # Reading candidate features from disk\n    training_features = pd.read_parquet('../input/cross-asset-featuresparquet/cross_asset_features.parquet')\n    \nexcept:\n    # Randomly select 10% of days to speed-up features generation\n    df_train['DAYSTR'] = pd.to_datetime(df_train['timestamp'], unit='s').apply(lambda x: x.strftime(\"%Y%m%d\"))\n    all_days = list(set([_ for _ in df_train['DAYSTR'].values]))\n    selected_days = np.random.choice(all_days, size=int(len(all_days)/10), replace=False)\n    df_train = df_train[df_train['DAYSTR'].isin(selected_days)]\n    df_train.drop(columns=['DAYSTR'], errors='ignore', inplace=True)\n    # Generating candidate features\n    training_features = get_features(df_train)\n    # Saving to disk\n    to_save = training_features.astype(np.float32)\n    to_save.to_parquet('cross_asset_features.parquet')\n    del to_save\ntraining_features","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:39:31.883154Z","iopub.execute_input":"2021-11-24T11:39:31.883875Z","iopub.status.idle":"2021-11-24T11:39:35.155984Z","shell.execute_reply.started":"2021-11-24T11:39:31.883835Z","shell.execute_reply":"2021-11-24T11:39:35.154938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Printing all feautures\nall_features = sorted([_ for _ in training_features.columns])\npp.pprint(all_features)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:39:35.15748Z","iopub.execute_input":"2021-11-24T11:39:35.157728Z","iopub.status.idle":"2021-11-24T11:39:35.169608Z","shell.execute_reply.started":"2021-11-24T11:39:35.157698Z","shell.execute_reply":"2021-11-24T11:39:35.16809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II.2 How To Calculate Highest Performance Achievable <a name=\"one-liner\"></a>\nThe syntax is ``features_df.kxy.data_valuation(target_column, problem_type='regression')`` and it works on any pandas DataFrame object, so long as you import the ``kxy`` package.\n\n### II.2.a Case 1: Cross-Asset Models <a name=\"cross-sectional\"></a>\nThis is the maximum performance we may achieve with the features above using a single model to trade all crypto-currencies.","metadata":{}},{"cell_type":"code","source":"ca_data_valuation_df = training_features.kxy.data_valuation('Target', problem_type='regression')","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:39:35.171132Z","iopub.execute_input":"2021-11-24T11:39:35.171378Z","iopub.status.idle":"2021-11-24T11:40:03.111728Z","shell.execute_reply.started":"2021-11-24T11:39:35.171349Z","shell.execute_reply":"2021-11-24T11:40:03.110714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ca_data_valuation_df['Achievable Pearson Correlation'] = '%.2f' % np.sqrt(\n    float(ca_data_valuation_df['Achievable R-Squared'].iloc[0]))\nca_data_valuation_df","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:40:03.11321Z","iopub.execute_input":"2021-11-24T11:40:03.1136Z","iopub.status.idle":"2021-11-24T11:40:03.128293Z","shell.execute_reply.started":"2021-11-24T11:40:03.113555Z","shell.execute_reply":"2021-11-24T11:40:03.127357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II.2.b Case 2: Single-Asset Models <a name=\"single-asset\"></a>\n\nThis is the maximum performance we may achieve with the features above using one model per crypto-currency.","metadata":{}},{"cell_type":"code","source":"ASSET_CSV = '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'\nasset_details = pd.read_csv(ASSET_CSV)\nasset_details.set_index(['Asset_ID'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:40:03.130044Z","iopub.execute_input":"2021-11-24T11:40:03.130874Z","iopub.status.idle":"2021-11-24T11:40:03.151941Z","shell.execute_reply.started":"2021-11-24T11:40:03.130818Z","shell.execute_reply":"2021-11-24T11:40:03.151015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = {}\nresults['== Cross-Asset =='] = [\\\n        '%.2f' % np.sqrt(float(ca_data_valuation_df['Achievable R-Squared'].iloc[0])), \\\n        ca_data_valuation_df['Achievable R-Squared'].iloc[0]]\nfor asset_id in range(14):\n    asset_name = asset_details.loc[asset_id]['Asset_Name']\n    df = training_features[training_features['Asset_ID']==asset_id]\n    data_valuation_df = df.kxy.data_valuation('Target', problem_type='regression')\n    print(asset_name)\n    print(data_valuation_df)\n    print('\\n\\n')\n    results[asset_name] = [\\\n        '%.2f' % np.sqrt(float(data_valuation_df['Achievable R-Squared'].iloc[0])), \\\n        data_valuation_df['Achievable R-Squared'].iloc[0]]\n    \nbreakdown_df = pd.DataFrame.from_dict(results, orient='index', columns=[\n    'Achievable Pearson Correlation', 'Achievable R-Squared'])","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:40:03.154251Z","iopub.execute_input":"2021-11-24T11:40:03.154544Z","iopub.status.idle":"2021-11-24T11:41:03.809972Z","shell.execute_reply.started":"2021-11-24T11:40:03.154513Z","shell.execute_reply":"2021-11-24T11:41:03.808954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II.3 Summary of Findings <a name=\"summary\"></a>","metadata":{}},{"cell_type":"code","source":"breakdown_df","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:41:03.81151Z","iopub.execute_input":"2021-11-24T11:41:03.811846Z","iopub.status.idle":"2021-11-24T11:41:03.825371Z","shell.execute_reply.started":"2021-11-24T11:41:03.8118Z","shell.execute_reply":"2021-11-24T11:41:03.82423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Update\n\n**PREVTARGET** features have look-ahead bias, and as such they can't be traded. Check out [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii) to find out more. \n\nIn the meantime, here's the cross-asset achievable performance without **PREVTARGET** features.","metadata":{}},{"cell_type":"code","source":"clean_features = training_features.drop(\n    columns=[_ for _ in training_features.columns if 'PREVTARGET' in _], errors='ignore')\nclean_feature_names = sorted([_ for _ in clean_features.columns])\npp.pprint(clean_feature_names)\nclean_ca_data_valuation_df = clean_features.kxy.data_valuation(\n    'Target', problem_type='regression')\nclean_ca_data_valuation_df['Achievable Pearson Correlation'] = '%.2f' % np.sqrt(\n    float(clean_ca_data_valuation_df['Achievable R-Squared'].iloc[0]))\nclean_ca_data_valuation_df","metadata":{"execution":{"iopub.status.busy":"2021-11-24T11:41:03.826958Z","iopub.execute_input":"2021-11-24T11:41:03.827628Z","iopub.status.idle":"2021-11-24T11:41:24.876682Z","shell.execute_reply.started":"2021-11-24T11:41:03.827562Z","shell.execute_reply":"2021-11-24T11:41:24.875559Z"},"trusted":true},"execution_count":null,"outputs":[]}]}