{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Target reconstruction: what's up with 2018-01-03 14:29:00?\n# UPDATE: \n### There's nothing special about that date except that it is 3749 steps after the train set starts! Adding in a timestamp at 2018-01-01 00:00 while calculating the Target completely changes the value of the four rows mentioned below. See [updated notebook](https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota) and [renewed discussion](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778#1626395).\n\n# Old notebook below:\n\nThis is a quick note to draw attention to one timestamp that the heroic target-reconstruction work collected in alexfir's [notebook](https://www.kaggle.com/alexfir/recreating-target/) isn't reproducing.\nThe plus side is that - excluding just 4 rows from 2018-01-03 14:29:00 - the reconstruction is pretty much perfect.\n\n## Summary statistics:\n\n**Calculated on all rows:**\n\n* Mean absolute error: 6.0e-10\n* Max absolute error: 3.8e-03\n* Std absolute error: 1.4e-06\n\n\n**Calculated on all rows except [3748, 3899904, 9762453, 11718653]:**\n\n* Mean absolute error: 8.2e-16\n* Max absolute error: 2.8e-15\n* Std absolute error: 8.5e-14\n\n**The question is: why does it have a large error for those 4 rows, and is perfect (up to precision) everywhere else?**\n","metadata":{}},{"cell_type":"markdown","source":"# Preliminaries","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\n\ndata_path = '../input/g-research-crypto-forecasting/train.csv'\ndtypes = {\n    'timestamp': np.int64,\n    'Asset_ID': np.int8,\n    'Close': np.float64,\n    'Target': np.float64,\n}\ncrypto_df = pd.read_csv(data_path, dtype=dtypes, usecols=list(dtypes.keys()))\n\nasset_details_path = '../input/g-research-crypto-forecasting/asset_details.csv'\nasset_details = pd.read_csv(asset_details_path)\n\ncrypto_df = crypto_df.merge(asset_details, on='Asset_ID')","metadata":{"execution":{"iopub.status.busy":"2021-12-09T14:44:42.732409Z","iopub.execute_input":"2021-12-09T14:44:42.733674Z","iopub.status.idle":"2021-12-09T14:45:14.398498Z","shell.execute_reply.started":"2021-12-09T14:44:42.733619Z","shell.execute_reply":"2021-12-09T14:45:14.397555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from https://www.kaggle.com/alexfir/recreating-target/\ndef calculate_target(data: pd.DataFrame, details: pd.DataFrame, price_column: str):\n    ids = list(details.Asset_ID)\n    asset_names = list(details.Asset_Name)\n    weights = np.array(list(details.Weight))\n\n    all_timestamps = np.sort(data['timestamp'].unique())\n    targets = pd.DataFrame(index=all_timestamps)\n\n    for i, id in enumerate(ids):\n        asset = data[data.Asset_ID == id].set_index(keys='timestamp')\n        price = pd.Series(index=all_timestamps, data=asset[price_column])\n        targets[asset_names[i]] = (\n            price.shift(periods=-16) /\n            price.shift(periods=-1)\n        ) - 1\n    \n    targets['m'] = np.average(targets.fillna(0), axis=1, weights=weights)\n    \n    m = targets['m']\n\n    num = targets.multiply(m.values, axis=0).rolling(3750).mean().values\n    denom = m.multiply(m.values, axis=0).rolling(3750).mean().values\n    beta = np.nan_to_num(num.T / denom, nan=0., posinf=0., neginf=0.)\n\n    targets = targets - (beta * m.values).T\n    targets.drop('m', axis=1, inplace=True)\n    \n    return targets","metadata":{"execution":{"iopub.status.busy":"2021-12-09T14:45:54.279366Z","iopub.execute_input":"2021-12-09T14:45:54.27972Z","iopub.status.idle":"2021-12-09T14:45:54.291355Z","shell.execute_reply.started":"2021-12-09T14:45:54.279682Z","shell.execute_reply":"2021-12-09T14:45:54.290596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"recon_targets = calculate_target(data=crypto_df, details=asset_details, price_column='Close')\nrecon_targets = pd.melt(recon_targets.reset_index(), id_vars='index')\nrecon_targets = recon_targets.rename(columns={'index':'timestamp', 'variable':'Asset_Name', 'value':'recon_Target'})","metadata":{"execution":{"iopub.status.busy":"2021-12-09T14:45:59.2088Z","iopub.execute_input":"2021-12-09T14:45:59.209319Z","iopub.status.idle":"2021-12-09T14:46:12.360915Z","shell.execute_reply.started":"2021-12-09T14:45:59.20928Z","shell.execute_reply":"2021-12-09T14:46:12.359846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = crypto_df.merge(recon_targets, on=['Asset_Name', 'timestamp'])","metadata":{"execution":{"iopub.status.busy":"2021-12-09T14:48:06.092222Z","iopub.execute_input":"2021-12-09T14:48:06.092536Z","iopub.status.idle":"2021-12-09T14:48:24.414512Z","shell.execute_reply.started":"2021-12-09T14:48:06.092503Z","shell.execute_reply":"2021-12-09T14:48:24.413499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Checks","metadata":{}},{"cell_type":"markdown","source":"Check that all NaNs match up:","metadata":{}},{"cell_type":"code","source":"all(crypto_df.Target.isna() == crypto_df.recon_Target.isna()) ","metadata":{"execution":{"iopub.status.busy":"2021-12-09T14:49:33.51018Z","iopub.execute_input":"2021-12-09T14:49:33.510521Z","iopub.status.idle":"2021-12-09T14:49:36.598284Z","shell.execute_reply.started":"2021-12-09T14:49:33.510489Z","shell.execute_reply":"2021-12-09T14:49:36.597443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reproduce absolute error claims:","metadata":{}},{"cell_type":"code","source":"crypto_df['abs_error'] = abs(crypto_df['Target'] - crypto_df['recon_Target'])\nprint(\"abs_error statistics for the entirety of train data:\")\ncrypto_df['abs_error'].describe()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T15:22:56.70413Z","iopub.execute_input":"2021-12-09T15:22:56.705178Z","iopub.status.idle":"2021-12-09T15:22:58.028781Z","shell.execute_reply.started":"2021-12-09T15:22:56.705122Z","shell.execute_reply":"2021-12-09T15:22:58.027805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"...seems to be highly skewed. Check out the large abs_errors:","metadata":{}},{"cell_type":"code","source":"crypto_df[crypto_df.abs_error > 1e-13]","metadata":{"execution":{"iopub.status.busy":"2021-12-09T15:00:35.161543Z","iopub.execute_input":"2021-12-09T15:00:35.162017Z","iopub.status.idle":"2021-12-09T15:00:35.20062Z","shell.execute_reply.started":"2021-12-09T15:00:35.161977Z","shell.execute_reply":"2021-12-09T15:00:35.199937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"They're all in the same minute!","metadata":{}},{"cell_type":"code","source":"print(f\"Datetime of offence: {pd.to_datetime(1514989740, unit='s')}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-09T15:24:49.06189Z","iopub.execute_input":"2021-12-09T15:24:49.063423Z","iopub.status.idle":"2021-12-09T15:24:49.072021Z","shell.execute_reply.started":"2021-12-09T15:24:49.063352Z","shell.execute_reply":"2021-12-09T15:24:49.07091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"indices = crypto_df[crypto_df.abs_error > 1e-13].index\nprint(\"abs_error statistics after dropping offenders:\")\ncrypto_df.drop(indices).abs_error.describe()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T15:24:58.392579Z","iopub.execute_input":"2021-12-09T15:24:58.39317Z","iopub.status.idle":"2021-12-09T15:25:02.398808Z","shell.execute_reply.started":"2021-12-09T15:24:58.39311Z","shell.execute_reply":"2021-12-09T15:25:02.397664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After dropping the four rows above, target reconstruction is pretty much perfect.\n\n**So, what's happening in this minute...?**","metadata":{}}]}