{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Crypto-portfolio Optimization: Training-test split\n\nFirst, we will present an introduction to crypto forecasting, describing the volatile nature of the cryptocurrency market and explaining why predicting how prices will behave in the near future is a fundamental task in financial modelling. Secondly, we will describe the structure and elements of the G-Research crypto forecasting dataset, as well as some relevant statistical properties of the major crypto assets (Bitcoing and Ethereum). Finally, we will devise a splitting procedure to extract\ntraining and test sets functional to training machine learning models.","metadata":{}},{"cell_type":"markdown","source":"## Cryptocurrency market\n\nThe cryptocurrency market has established itself as an extremely popular and volatile market, delivering massive returns and losses to investors. Thousands of cryptocurrencies have been created with a few major ones you may have heard of including Bitcoin (BTC), Ethereum (ETH) or Dogecoin (DOGE).\n\nTheir assets are traded extensively across crypto-exchanges, with an average volume of $41 billion traded daily over the last year, according to [CryptoCompare](https://www.cryptocompare.com/) as of July 25, 2021. Changes in prices between different cryptocurrencies are highly interconnected. For example, Bitcoin has historically been a major driver of price changes across cryptocurrencies, but others also impact the market.","metadata":{}},{"cell_type":"markdown","source":"## Return forecasting\n\nA fundamental task in financial modeling is predicting how prices will behave in the near future. Using the time-series of historical prices as training data, we want to predict if prices will go up or down, and by how much. Namely, the asset _returns_ after some _delta t_.\n\nThe forecasting of asset returns remains an open and extremely challenging task. This is a fascinating domain for the machine learning community given 1) the extreme volatility of the assets, 2) the non-stationary nature of the data, 3) the market and meme manipulation, 4) the correlation between assets and 5) the very fast changing market conditions. Maybe you will find it as fascinating as we do! ","metadata":{}},{"cell_type":"markdown","source":"## Dataset exploration\n\nThis dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum from 01/01/2018 to 21/09/2021.\n\nThis dataset will give you access to millions of rows of minute-by-minute cryptocurrency trading data. Our goal will be to build a few machine learning models to predict the returns of 14 cryptocurrencies simultaneously, in a time scale of minutes to hours ahead.\n\n### Files\n\n__N.B.:__ All files irrelevant for our purposes have been omitted from the following description.\n\n- __train.csv__ - The raw training set. It includes the following features:\n  + `timestamp` - A UNIX timestamp of the minute-by-minute data covered by the row;\n  + `Asset_ID` - The unique asset Id corresponding to one of the crytocurrencies (e.g. `Asset_ID = 1` for Bitcoin);\n  + `Count` - The number of trades that took place in the time interval;\n  + `Open` - The opening price of the time interval in USD;\n  + `High` - The highest price reached during the time interval in USD;\n  + `Low` - The lowest price reached during the time interval in USD;\n  + `Close` - The closing price of the time interval in USD;\n  + `Volume` - The number of cryptoasset units traded during the time interval in USD;\n  + `VWAP` - The average price of the asset over the time interval weighted by `Volume`;\n  + `Target` - 15-minute-ahead residualized log-returns. \n  \n  The first two columns define the time and asset indexes for a data row. The six middle columns are feature columns with the trading data for a given asset during the time interval of the row. The last column is the 15-minute-ahead prediction target.\n  \n  See the [Prediction](#Prediction) section of this notebook for details on how the `Target` column is calculated;  \n  See the [Training-test split](#Training-test-split) section of this notebook for details on how __train.csv__ is split in training and test sets.\n\n- __asset_details.csv__ - The real cryptoasset name for each `Asset_ID` and the weight each cryptoasset has in the metric.  \n  The weight of a cryptoasset is a dynamic variable that evolves within the market itself. Nonetheless, a few cryptocurrencies have historically had a larger impact in shifting the market dynamics; thus, a larger weight in the forecasting evaluation metric.\n  \n### Data loading","metadata":{}},{"cell_type":"code","source":"import datetime\nimport pathlib\nimport time\n\nimport numpy as np\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:55:58.324872Z","iopub.execute_input":"2022-10-31T01:55:58.325270Z","iopub.status.idle":"2022-10-31T01:55:58.332198Z","shell.execute_reply.started":"2022-10-31T01:55:58.325239Z","shell.execute_reply":"2022-10-31T01:55:58.330426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATAPATH = \"../input/g-research-crypto-forecasting\"\nDATAPATH = pathlib.Path(DATAPATH).resolve()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:30:07.417150Z","iopub.execute_input":"2022-10-30T23:30:07.418237Z","iopub.status.idle":"2022-10-30T23:30:07.423043Z","shell.execute_reply.started":"2022-10-30T23:30:07.418196Z","shell.execute_reply":"2022-10-30T23:30:07.422121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FILEPATH_raw_csv = DATAPATH / \"train.csv\"\nFILEPATH_weights_csv = DATAPATH / \"asset_details.csv\"\n\n\nassert FILEPATH_raw_csv.exists()\nassert FILEPATH_weights_csv.exists()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:30:07.776266Z","iopub.execute_input":"2022-10-30T23:30:07.777337Z","iopub.status.idle":"2022-10-30T23:30:07.786801Z","shell.execute_reply.started":"2022-10-30T23:30:07.777299Z","shell.execute_reply":"2022-10-30T23:30:07.785646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df = pd.read_csv(FILEPATH_raw_csv)\n\nraw_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:30:10.707181Z","iopub.execute_input":"2022-10-30T23:30:10.708040Z","iopub.status.idle":"2022-10-30T23:31:17.636725Z","shell.execute_reply.started":"2022-10-30T23:30:10.708000Z","shell.execute_reply":"2022-10-30T23:31:17.635644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weights_df = pd.read_csv(FILEPATH_weights_csv)\n\nweights_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:31:17.638957Z","iopub.execute_input":"2022-10-30T23:31:17.639432Z","iopub.status.idle":"2022-10-30T23:31:17.656469Z","shell.execute_reply.started":"2022-10-30T23:31:17.639388Z","shell.execute_reply":"2022-10-30T23:31:17.655658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Candlestick charts\n\nThe trading data format is an aggregated form of market data including for `Open`, `High`, `Low` and `Close`. The data can be visualized through the common candlestick bar chart, which allows traders to perform technical analysis on intraday values. The bar's body length represents the price range between the `Open` and `Close` prices of that trading day. A red bar means the `Close` was lower than the `Open`. A green bar, viceversa. The wicks above and below the bars show the `High` and `Low` prices of that trading day interval.\n\nIn the plot below, we will visualize a slice of the Bitcoin prices using the [Plotly](https://plotly.com/) library. At the bottom, a rangeslider controls the zoom level.","metadata":{}},{"cell_type":"code","source":"def get_single_asset_df(asset: str, df: pd.DataFrame, info_df: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"It extracts the DataFrame of a single asset from a multi-asset DataFrame\"\"\"\n    asset_row_idx = info_df[\"Asset_Name\"] == asset\n    asset_id = info_df.loc[asset_row_idx, \"Asset_ID\"].values[0]\n    \n    return df[df[\"Asset_ID\"] == asset_id]","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:31:17.657434Z","iopub.execute_input":"2022-10-30T23:31:17.657998Z","iopub.status.idle":"2022-10-30T23:31:17.664945Z","shell.execute_reply.started":"2022-10-30T23:31:17.657966Z","shell.execute_reply":"2022-10-30T23:31:17.663451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc_raw_df = get_single_asset_df(\"Bitcoin\", raw_df, weights_df)\nbtc_raw_df = btc_raw_df.set_index(\"timestamp\")\n\nbtc_tail_200_df = btc_raw_df.iloc[-200:]","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:31:17.667775Z","iopub.execute_input":"2022-10-30T23:31:17.668151Z","iopub.status.idle":"2022-10-30T23:31:18.071888Z","shell.execute_reply.started":"2022-10-30T23:31:17.668122Z","shell.execute_reply":"2022-10-30T23:31:18.070821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us install an SVG renderer engine to avoid keeping large Plotly plots in RAM","metadata":{}},{"cell_type":"code","source":"%%capture\n\n!pip install -U kaleido","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:31:18.073251Z","iopub.execute_input":"2022-10-30T23:31:18.073718Z","iopub.status.idle":"2022-10-30T23:31:36.326112Z","shell.execute_reply.started":"2022-10-30T23:31:18.073670Z","shell.execute_reply":"2022-10-30T23:31:36.322153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as pgo\n\n\ntail_200_datetime = pd.to_datetime(btc_tail_200_df.index, unit=\"s\")\n\nfigure = pgo.Figure(\n    data=[pgo.Candlestick(\n        x=tail_200_datetime,\n        open=btc_tail_200_df['Open'], \n        high=btc_tail_200_df['High'], \n        low=btc_tail_200_df['Low'], \n        close=btc_tail_200_df['Close']\n    )])\n\nfigure.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:31:36.329076Z","iopub.execute_input":"2022-10-30T23:31:36.329573Z","iopub.status.idle":"2022-10-30T23:31:36.514394Z","shell.execute_reply.started":"2022-10-30T23:31:36.329522Z","shell.execute_reply":"2022-10-30T23:31:36.511680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del btc_raw_df, btc_tail_200_df, tail_200_datetime, figure  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:31:36.516081Z","iopub.execute_input":"2022-10-30T23:31:36.516412Z","iopub.status.idle":"2022-10-30T23:31:36.529058Z","shell.execute_reply.started":"2022-10-30T23:31:36.516383Z","shell.execute_reply":"2022-10-30T23:31:36.527785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pre-processing","metadata":{}},{"cell_type":"markdown","source":"Let us inspect the data of another important asset, Ethereum.","metadata":{}},{"cell_type":"code","source":"eth_raw_df = get_single_asset_df(\"Ethereum\", raw_df, weights_df)\neth_raw_df = eth_raw_df.set_index(\"timestamp\")\n\neth_raw_df.info(show_counts =True)","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:02.167070Z","iopub.execute_input":"2022-10-30T23:32:02.168746Z","iopub.status.idle":"2022-10-30T23:32:02.699491Z","shell.execute_reply.started":"2022-10-30T23:32:02.168694Z","shell.execute_reply":"2022-10-30T23:32:02.698193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Ethereum dataset has 1956200 rows, but the `Target` column only 1955860. Are there missing values?","metadata":{}},{"cell_type":"code","source":"eth_raw_df.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:02.701347Z","iopub.execute_input":"2022-10-30T23:32:02.701709Z","iopub.status.idle":"2022-10-30T23:32:02.941729Z","shell.execute_reply.started":"2022-10-30T23:32:02.701679Z","shell.execute_reply":"2022-10-30T23:32:02.940434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 340 actual missing values in the `Target` column, which we will address below.\n\n__N.B.:__ Missing asset data for a given time interval is not represented by NaN values, but instead by the absence of those rows. Let us check the timestamp difference between consecutive rows to see if any are over 60 (seconds), i.e., they have missing rows.","metadata":{}},{"cell_type":"code","source":"eth_raw_df.index.to_series().diff().value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:07.932117Z","iopub.execute_input":"2022-10-30T23:32:07.932547Z","iopub.status.idle":"2022-10-30T23:32:07.990107Z","shell.execute_reply.started":"2022-10-30T23:32:07.932513Z","shell.execute_reply":"2022-10-30T23:32:07.988889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del eth_raw_df  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:08.177717Z","iopub.execute_input":"2022-10-30T23:32:08.178459Z","iopub.status.idle":"2022-10-30T23:32:08.184489Z","shell.execute_reply.started":"2022-10-30T23:32:08.178420Z","shell.execute_reply":"2022-10-30T23:32:08.183354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are many gaps in the data (up to over 10 minutes). Most time series models require the data into a format without time gaps. Therefore, we should preprocess the data with the `pd.DataFrame.reindex` method to forward fill the gaps.\n\n__N.B.:__ If an asset has a ratio of target rows with NaN values higher than 0.10, **it will be removed from the dataset**. By filling the gaps of such a large amount of unknowns we would introduce a lot of noise in the original data distribution.","metadata":{}},{"cell_type":"code","source":"import typing\n\nfrom tqdm.notebook import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:11.859915Z","iopub.execute_input":"2022-10-30T23:32:11.860336Z","iopub.status.idle":"2022-10-30T23:32:11.976467Z","shell.execute_reply.started":"2022-10-30T23:32:11.860302Z","shell.execute_reply":"2022-10-30T23:32:11.975139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def impute_missing_timestamps(\n    df: pd.DataFrame, \n    delta_t: int, \n    s_col: str, \n    y_col: str, \n    weights_df: typing.Optional[pd.DataFrame] = None\n) -> typing.Tuple[pd.DataFrame, pd.DataFrame]:\n    \"\"\"It imputes the missing timestamps from a multi-time-series DataFrame\n    \n    A multi-time-series DataFrame is a table with multiple synchronized time series signals.\n    The data will be imputed with the `pd.DataFrame.reindex` method to forward fill the gaps.\n    \n    If a weights DataFrame is provided, the weight of any time series will be removed,\n    if it has a ratio of target rows with NaN values above 0.10, i.e., it is an unreliable signal.\n    \n    - The multi-time-series DataFrame is assumed to have a 'timestamp' column;\n    - The weights DataFrame is assumed to have a 'Weights' column.\n    \"\"\"\n    min_t = int(df.iloc[ 0][\"timestamp\"])\n    max_t = int(df.iloc[-1][\"timestamp\"])\n    \n    labels = list(range(min_t, max_t + delta_t, delta_t))\n    \n    n_rows = df.shape[0]\n    n_labels = len(labels)\n    n_signals = df[s_col].nunique()\n    \n    if n_rows == n_labels * n_signals:\n        return df, weights_df\n    \n    imputed_groups = []\n    grouped_df = df.groupby(s_col)\n    \n    if weights_df is not None:\n        weights_df = weights_df.set_index(s_col)\n        \n    for s, group in tqdm(grouped_df, total=grouped_df.ngroups):\n        group = group.set_index(\"timestamp\")\n        group = group.reindex(labels, method=\"pad\")\n        group = group.reset_index()\n\n        n_y_na_rows = group[y_col].isna().sum()\n        n_rows = group.shape[0]\n\n        y_na_rows_ratio = n_y_na_rows / n_rows\n\n        if y_na_rows_ratio > 0.10:\n            s_name = weights_df.loc[s][\"Asset_Name\"]\n            print(f\"Too many NaN target rows - {s_name} will be removed\")\n            \n            if weights_df is not None:\n                weights_df = weights_df.drop(s)\n            \n            continue\n\n        # It resorts to autofilling\n        group = group.bfill()\n        group = group.ffill()\n\n        imputed_groups.append(group)\n        \n    if weights_df is not None:\n        weights_df = weights_df.reset_index()\n        \n    imputed_df = pd.concat(imputed_groups)\n    imputed_df = imputed_df.sort_values(by=[\"timestamp\", s_col])\n    imputed_df = imputed_df.reset_index()\n    \n    return imputed_df, weights_df","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:12.079142Z","iopub.execute_input":"2022-10-30T23:32:12.079573Z","iopub.status.idle":"2022-10-30T23:32:12.094259Z","shell.execute_reply.started":"2022-10-30T23:32:12.079535Z","shell.execute_reply":"2022-10-30T23:32:12.093264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imputed_df, weights_df = impute_missing_timestamps(\n    df=raw_df,\n    delta_t=60,\n    s_col=\"Asset_ID\",\n    y_col=\"Target\",\n    weights_df=weights_df\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:16.829230Z","iopub.execute_input":"2022-10-30T23:32:16.829749Z","iopub.status.idle":"2022-10-30T23:32:52.446942Z","shell.execute_reply.started":"2022-10-30T23:32:16.829646Z","shell.execute_reply":"2022-10-30T23:32:52.445695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see from the above logs, **five cryptocurrencies have been removed from the dataset** because unreliable:\n- Dogecoin\n- IOTA\n- Maker\n- Monero\n- Stellar\n\nFor the Dogecoin fans out there, I'm sorry, better luck next time.\n\nLet us check if we have filled all the gaps:","metadata":{}},{"cell_type":"code","source":"imputed_df.isna().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:52.449798Z","iopub.execute_input":"2022-10-30T23:32:52.450618Z","iopub.status.idle":"2022-10-30T23:32:52.831975Z","shell.execute_reply.started":"2022-10-30T23:32:52.450551Z","shell.execute_reply":"2022-10-30T23:32:52.830798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth_imputed_df = get_single_asset_df(\"Ethereum\", imputed_df, weights_df)\neth_imputed_df = eth_imputed_df.set_index(\"timestamp\")\n\neth_imputed_df.index.to_series().diff().value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:52.833222Z","iopub.execute_input":"2022-10-30T23:32:52.833563Z","iopub.status.idle":"2022-10-30T23:32:53.463274Z","shell.execute_reply.started":"2022-10-30T23:32:52.833532Z","shell.execute_reply":"2022-10-30T23:32:53.462021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del raw_df, eth_imputed_df  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:32:53.465433Z","iopub.execute_input":"2022-10-30T23:32:53.465835Z","iopub.status.idle":"2022-10-30T23:32:53.471141Z","shell.execute_reply.started":"2022-10-30T23:32:53.465801Z","shell.execute_reply":"2022-10-30T23:32:53.470354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hooray! It seems that no more NaN values or missing rows are present in the dataset.","metadata":{}},{"cell_type":"markdown","source":"## Assets correlation\n\nAt the beginning of the notebook, we stated that the tight correlation between cryptocurrencies makes crypto forecasting a fascinating challenge. Let us investigate exactly how tight this correlation is, by first looking at the `Close` prices history.\n\nTo make things simple, we will focus on the two cryptocurrencies we introduced earlier, Bitcoin and Ethereum.","metadata":{}},{"cell_type":"code","source":"def max_norm(series: pd.Series) -> pd.Series:\n    \"\"\"It normalizes a numeric Series by its maximum value\"\"\"\n    return series / series.max()","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:33:00.098907Z","iopub.execute_input":"2022-10-30T23:33:00.099315Z","iopub.status.idle":"2022-10-30T23:33:00.104808Z","shell.execute_reply.started":"2022-10-30T23:33:00.099281Z","shell.execute_reply":"2022-10-30T23:33:00.103845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"figure = pgo.Figure()\n\nx = imputed_df[\"timestamp\"].unique()\nx = pd.to_datetime(x, unit=\"s\")\n\nfor asset in [\"Bitcoin\", \"Ethereum\"]:\n    asset_imputed_df = get_single_asset_df(asset, imputed_df, weights_df)\n    asset_imputed_df = asset_imputed_df.set_index(\"timestamp\")\n    \n    y = max_norm(asset_imputed_df[\"Close\"])\n    \n    del asset_imputed_df\n            \n    figure = figure.add_trace(\n        pgo.Scatter(x=x, y=y, name=asset)\n    )\n    \nfigure.show(renderer=\"svg\")\n\ndel x, y, figure  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:33:02.499895Z","iopub.execute_input":"2022-10-30T23:33:02.500527Z","iopub.status.idle":"2022-10-30T23:35:13.032035Z","shell.execute_reply.started":"2022-10-30T23:33:02.500493Z","shell.execute_reply":"2022-10-30T23:35:13.030432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The assets have a quite different history, but maybe they have a higher correlation in recent times:","metadata":{}},{"cell_type":"code","source":"def totimestamp(datestr: str) -> int:\n    \"\"\"It converts a local datestr to a UTC timestamp\"\"\"\n    time_struct = datetime.datetime.strptime(datestr, \"%d/%m/%Y\").timetuple()\n    return np.int32(time.mktime(time_struct))","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:35:13.034822Z","iopub.execute_input":"2022-10-30T23:35:13.035938Z","iopub.status.idle":"2022-10-30T23:35:13.042029Z","shell.execute_reply.started":"2022-10-30T23:35:13.035898Z","shell.execute_reply":"2022-10-30T23:35:13.041128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"figure = pgo.Figure()\n\nstart = totimestamp(\"01/06/2021\")\nend = totimestamp(\"01/07/2021\")\n\ntimestamp_series = imputed_df[\"timestamp\"]\ntimestamp_series = timestamp_series.loc[lambda t: (t >= start) & (t <= end)]\n\nx = timestamp_series.unique()\nx = pd.to_datetime(x, unit=\"s\")\n\nfor asset in [\"Bitcoin\", \"Ethereum\"]:\n    asset_imputed_df = get_single_asset_df(asset, imputed_df, weights_df)\n    asset_imputed_df = asset_imputed_df.set_index(\"timestamp\")\n    \n    asset_imputed_mini_2021_df = asset_imputed_df.loc[start:end]\n    \n    y = max_norm(asset_imputed_mini_2021_df[\"Close\"])\n    \n    del asset_imputed_df, asset_imputed_mini_2021_df\n            \n    figure = figure.add_trace(\n        pgo.Scatter(x=x, y=y, name=asset)\n    )\n    \nfigure.show(renderer=\"svg\")\n\ndel start, end, x, y, figure  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:35:13.043528Z","iopub.execute_input":"2022-10-30T23:35:13.044090Z","iopub.status.idle":"2022-10-30T23:35:16.613504Z","shell.execute_reply.started":"2022-10-30T23:35:13.044057Z","shell.execute_reply":"2022-10-30T23:35:16.612242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On shorter intervals we can visually see some potential correlation up to a shift between both assets, with some simultaneous ups and downs.","metadata":{}},{"cell_type":"markdown","source":"### Log-returns\n\nIn order to analyze price changes for an asset we have learned that we can deal with the price difference. However, **different assets exhibit different price scales**, so that the their returns are not readily comparable (e.g., you may have noticed that both the above plots are normalized with the `max_norm` function in a common range between 0 and 1). A standard practice to work around this problem is to compute the percentage change in price, also known as return. **The return coincides with the percentage change in our invested capital**. More formally:\n\n$$r_{a, n}(t) = \\dfrac{p_a(t + n)}{p_a(t + 1)}$$\n\nwhere $r_{a, n}(t)$ is the return of asset $a$ at time $t$ over an horizon of $n$ timesteps and $p_a(t)$ is the price of asset $a$ at time $t$.\n\nThe concept of return is widely used in finance. However, **log returns are preferred** for mathematical modelling of time series, as they are **additive across time** and are **not bounded in** $\\mathcal{R}$, unlike regular returns that cannot go below -100%.\n\nTo compute the log return, we can simply take the logarithm of the ratio between two consecutive prices:\n\n$$\\begin{align*}\\log\\left(r_{a, n}(t)\\right) &= \\log\\left(\\dfrac{p_a(t + n)}{p_a(t + 1)}\\right) \\\\ &= \\log\\left(p_a(t + n)\\right) - \\log\\left(p_a(t + 1)\\right)\\end{align*}$$\n\n**N.B.:** The first row will have an empty return as the previous price is unknown. Therefore, the empty return data point will be dropped.","metadata":{}},{"cell_type":"code","source":"def log_return(series: pd.Series, n: int = 1) -> pd.Series:\n    \"\"\"It computes the log return across time over a given horizon \"\"\"\n    return np.log(series).diff(periods=n)","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:35:44.787144Z","iopub.execute_input":"2022-10-30T23:35:44.787674Z","iopub.status.idle":"2022-10-30T23:35:44.794407Z","shell.execute_reply.started":"2022-10-30T23:35:44.787639Z","shell.execute_reply":"2022-10-30T23:35:44.793030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us visualize the distribution of log returns, focusing once again on our two assets, Bitcoin and Ethereum.","metadata":{}},{"cell_type":"code","source":"figure = pgo.Figure()\n\nstart = totimestamp(\"01/06/2021\")\nend = totimestamp(\"01/07/2021\")\n\ntimestamp_series = imputed_df[\"timestamp\"]\ntimestamp_series = timestamp_series.loc[lambda t: (t >= start) & (t <= end)]\n\nx = timestamp_series.unique()\nx = pd.to_datetime(x, unit=\"s\")\n\nfor asset in [\"Bitcoin\", \"Ethereum\"]:\n    asset_imputed_df = get_single_asset_df(asset, imputed_df, weights_df)\n    asset_imputed_df = asset_imputed_df.set_index(\"timestamp\")\n    \n    asset_imputed_mini_2021_df = asset_imputed_df.loc[start:end]\n    \n    y = log_return(asset_imputed_mini_2021_df[\"Close\"])[1:]\n    \n    del asset_imputed_df, asset_imputed_mini_2021_df\n            \n    figure = figure.add_trace(\n        pgo.Scatter(x=x, y=y, name=f\"{asset} log returns\")\n    )\n    \nfigure.show(renderer=\"svg\")\n\ndel start, end, x, y, figure  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:35:50.705414Z","iopub.execute_input":"2022-10-30T23:35:50.706416Z","iopub.status.idle":"2022-10-30T23:35:54.357359Z","shell.execute_reply.started":"2022-10-30T23:35:50.706376Z","shell.execute_reply":"2022-10-30T23:35:54.356456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the signals now look more like white noise, with less drift than the time series of prices. This result further suggests that log returns are the correct framework wherein to analyze the potential correlation between cryptocurrencies.","metadata":{}},{"cell_type":"markdown","source":"### Correlation matrix\n\nEarlier in the notebook we hypothesized before that crypto asset returns may exhibit some correlation. Let us now check this in more detail, by looking at how the correlation between our two assets, Bitcoin and Ethereum, changes over time.\n\n**N.B.:** The time series are grouped with a 7-day moving average (MA) to highlight the local changes in correlation.","metadata":{}},{"cell_type":"code","source":"figure = pgo.Figure()\n\nlog_returns_series = []\n\nfor asset in [\"Bitcoin\", \"Ethereum\"]:\n    asset_imputed_df = get_single_asset_df(asset, imputed_df, weights_df)\n    asset_imputed_df = asset_imputed_df.set_index(\"timestamp\")\n    \n    y = log_return(asset_imputed_df[\"Close\"])[1:]\n    y = y.rename(f\"{asset}_logret\")\n    \n    del asset_imputed_df\n    \n    log_returns_series.append(y)\n    \nlog_returns_df = pd.concat(log_returns_series, axis=1)\n\n# It groups log returns with a 7-day moving average (MA)\nspan_7_days = 60 * 60 * 24 * 7\n\nlog_returns_ma_df   = log_returns_df.groupby(log_returns_df.index // span_7_days)\nlog_returns_ma_corr = log_returns_ma_df.corr()\n\n# It extracts the correlation from each MA\nlog_returns_ma_corr = log_returns_ma_corr.loc[:, \"Bitcoin_logret\"]\nlog_returns_ma_corr = log_returns_ma_corr.loc[:,\"Ethereum_logret\"]\n\nn_rows = log_returns_ma_corr.shape[0]\n\nx = list(range(n_rows))\ny = log_returns_ma_corr\n\nfigure = figure.add_trace(pgo.Scatter(x=x, y=y))\nfigure.show(renderer=\"svg\")\n\ndel (\n    log_returns_series, \n    log_returns_df, \n    log_returns_ma_df, \n    log_returns_ma_corr,\n    n_rows,\n    x, \n    y, \n    figure\n)  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-30T23:36:40.278808Z","iopub.execute_input":"2022-10-30T23:36:40.279796Z","iopub.status.idle":"2022-10-30T23:36:41.426038Z","shell.execute_reply.started":"2022-10-30T23:36:40.279737Z","shell.execute_reply":"2022-10-30T23:36:41.424489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above plot we can observe a high, yet variable, correlation between the assets.\n\nThere seems to be some **changing dynamics over time**, especially in the first half of the plot, which would be critical to characterize in the context of time series forecasting, i.e., in forecasting tasks of highly **non-stationary environments**:\n- A stationary behaviour of a system is characterized by non-changing statistical properties over time, e.g., mean, variance and autocorrelation;\n- A non-stationary behaviour is characterized by a continuous change of statistical properties over time. \n\nThe stationarity property is important because many useful analytical tools and statistical tests and models rely on it (e.g., ARIMA).\n\nLet us check the correlation between **all assets over the whole 2021** visualizing the correlation matrix:","metadata":{}},{"cell_type":"code","source":"all_2021_assets_df = pd.DataFrame([])\n\nstart = totimestamp(\"01/01/2021\")\nend = totimestamp(\"31/12/2021\")\n\nfor asset in weights_df[\"Asset_Name\"]:\n    asset_imputed_df = get_single_asset_df(asset, imputed_df, weights_df)\n    asset_imputed_df = asset_imputed_df.set_index(\"timestamp\")\n    asset_imputed_df = asset_imputed_df.loc[start:end]\n        \n    y = log_return(asset_imputed_df[\"Close\"])[1:]\n    y = y.rename(asset)\n    \n    all_2021_assets_df = all_2021_assets_df.join(y, how=\"outer\")\n    \nall_2021_assets_corr_df = all_2021_assets_df.corr()\n    \nfigure = pgo.Figure()\n\nfigure.add_trace(\n    pgo.Heatmap(\n        x=all_2021_assets_corr_df.columns,\n        y=all_2021_assets_corr_df.columns,\n        z=all_2021_assets_corr_df.values,\n        text=all_2021_assets_corr_df.values,\n        texttemplate='%{text:.2f}'\n    )\n)\nfigure['layout']['yaxis']['autorange'] = \"reversed\"\n\nfigure.show()\n\ndel (\n    asset_imputed_df, \n    all_2021_assets_df, \n    all_2021_assets_corr_df,\n    y, \n    figure\n)  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-31T00:26:21.168718Z","iopub.execute_input":"2022-10-31T00:26:21.169111Z","iopub.status.idle":"2022-10-31T00:26:24.170853Z","shell.execute_reply.started":"2022-10-31T00:26:21.169079Z","shell.execute_reply":"2022-10-31T00:26:24.169656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note how some assets have much higher pairwise correlation than others, but the average is very high at 0.64!\n\nLet us now analyze the very same correlation matrix, but on the `Target` column:","metadata":{}},{"cell_type":"code","source":"all_2021_assets_df = pd.DataFrame([])\n\nstart = totimestamp(\"01/01/2021\")\nend = totimestamp(\"31/12/2021\")\n\nfor asset in weights_df[\"Asset_Name\"]:\n    asset_imputed_df = get_single_asset_df(asset, imputed_df, weights_df)\n    asset_imputed_df = asset_imputed_df.set_index(\"timestamp\")\n    asset_imputed_df = asset_imputed_df.loc[start:end]\n        \n    y = asset_imputed_df[\"Target\"][1:]\n    y = y.rename(asset)\n    \n    all_2021_assets_df = all_2021_assets_df.join(y, how=\"outer\")\n    \nall_2021_assets_corr_df = all_2021_assets_df.corr()\n    \nfigure = pgo.Figure()\n\nfigure.add_trace(\n    pgo.Heatmap(\n        x=all_2021_assets_corr_df.columns,\n        y=all_2021_assets_corr_df.columns,\n        z=all_2021_assets_corr_df.values,\n        text=all_2021_assets_corr_df.values,\n        texttemplate='%{text:.2f}'\n    )\n)\nfigure['layout']['yaxis']['autorange'] = \"reversed\"\n\nfigure.show()\n\ndel (\n    asset_imputed_df, \n    all_2021_assets_df, \n    all_2021_assets_corr_df,\n    y, \n    figure\n)  # It frees some RAM","metadata":{"execution":{"iopub.status.busy":"2022-10-31T00:07:16.697048Z","iopub.execute_input":"2022-10-31T00:07:16.697490Z","iopub.status.idle":"2022-10-31T00:07:19.917436Z","shell.execute_reply.started":"2022-10-31T00:07:16.697456Z","shell.execute_reply":"2022-10-31T00:07:19.916470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is immediately noticeable how the correlation matrix appears more representitive of quasi-independent random vaiables, as most of the energy revolves around the main diagonal with only a handful of values significantly away from 0 (all less than 0.33).\n\nThis sudden change in appearance is closely related to the **residualization of variables**. It will be apparent in the next section what residualization is, and how to compute residualized log returns from log returns, but it is clear that **reducing the correlation of assets may improve our predictive power of the market by removing the need for temporal synchronization** between time series readings.\n\nThis was just a quick glance of this dataset might offer. All of you are encouraged to perform additional statistical analysis to have a stronger grasp on the dataset, including autocorrelation, time-series decomposition and stationarity tests (ADF, KPSS, etc.).","metadata":{}},{"cell_type":"markdown","source":"## Prediction","metadata":{}},{"cell_type":"markdown","source":"The time has finally come! Let us unveil what the `Target` column stands for.\n\nThe dataset aims to predict returns in the near future for prices $p_a(t)$, for any available asset $a$. The ground truth of each row in the dataset is derived from log returns, $\\log\\left(r_{a, n}(t)\\right)$, over $n = 15$ minutes, which can be computed as:\n\n$$\\log\\left(r_{a, 15}(t)\\right) = \\log\\left(p_a(t + 15)\\right) - \\log\\left(p_a(t + 1)\\right)$$\n\nIt is derived from log returns, but not equal, as we have performed a linear residualization, removing the market signal from individual asset returns. In more detail, if $m_n(t)$ is the weighted average market returns at time $t$ over an horizon of $n$ timesteps, then:\n\n$$\n\\begin{align*}\n  m_n(t) &= \\dfrac{\\sum_a w_a \\log\\left(r_{a, n}(t)\\right)}{\\sum_a w_a} \\\\\\\\\n  \\beta_{a, n} &= \\dfrac{\\langle m_n(t) \\cdot \\log\\left(r_{a, n}(t)\\right) \\rangle}{\\langle m_n(t)^2 \\rangle} \\\\\\\\\n  y_a(t) &= \\log\\left(r_{a, 15}(t)\\right) - \\beta_{a, 15}m_{15}(t)\n\\end{align*}\n$$\n\nwhere $y_a(t)$ is the ground truth of asset $a$ at time $t$, $w_a$ is the market weight of asset $a$ and the bracket $\\langle \\cdot \\rangle$ is the rolling average over time (on a bit longer than 3500 minute windows).\n\nThe evaluation metric of the dataset is a weighted version of the Pearson correlation coefficient, with weights given by **asset_details.csv**. The chosen metric emphasizes the linear residualization function, as well as the autoregressive nature of the problem.","metadata":{}},{"cell_type":"markdown","source":"## Training-test split","metadata":{}},{"cell_type":"markdown","source":"As we will train regression parameters, we need to separate training and test sets. Since we have pre-processed the dataset to fill all the gaps evenly, we opt for a timestamp **split on 01/04/2021**, i.e., roughly a **86% split**.\n\nAs standard practice not to violate the temporal dependency, the test split represents a later unseen part of the data.  \nThe split leaves us with more than three years of training data, and about six months of test data.","metadata":{}},{"cell_type":"code","source":"def training_test_split(df: pd.DataFrame, boundary: int) -> typing.Union[pd.DataFrame, pd.DataFrame]:    \n    training_df = df.loc[df[\"timestamp\"] < boundary]\n    training_df = training_df.reset_index(drop=True)\n    \n    test_df = df.loc[df[\"timestamp\"] >= boundary]\n    test_df = test_df.reset_index(drop=True)\n    \n    return training_df, test_df","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:42:17.737114Z","iopub.execute_input":"2022-10-31T01:42:17.737533Z","iopub.status.idle":"2022-10-31T01:42:17.744287Z","shell.execute_reply.started":"2022-10-31T01:42:17.737499Z","shell.execute_reply":"2022-10-31T01:42:17.743194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boundary = totimestamp(\"01/04/2021\")","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:42:19.958976Z","iopub.execute_input":"2022-10-31T01:42:19.959362Z","iopub.status.idle":"2022-10-31T01:42:19.965427Z","shell.execute_reply.started":"2022-10-31T01:42:19.959332Z","shell.execute_reply":"2022-10-31T01:42:19.964304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_df, test_df = training_test_split(imputed_df, boundary)\n\ntraining_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:50:05.470281Z","iopub.execute_input":"2022-10-31T01:50:05.470690Z","iopub.status.idle":"2022-10-31T01:50:07.763016Z","shell.execute_reply.started":"2022-10-31T01:50:05.470659Z","shell.execute_reply":"2022-10-31T01:50:07.761738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a leftover `index` column from some previous operation. Let us drop that!","metadata":{}},{"cell_type":"code","source":"training_df = training_df.drop(\"index\", axis=1)\ntest_df = test_df.drop(\"index\", axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:50:15.255510Z","iopub.execute_input":"2022-10-31T01:50:15.255908Z","iopub.status.idle":"2022-10-31T01:50:15.729004Z","shell.execute_reply.started":"2022-10-31T01:50:15.255876Z","shell.execute_reply":"2022-10-31T01:50:15.727697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:50:18.273064Z","iopub.execute_input":"2022-10-31T01:50:18.273466Z","iopub.status.idle":"2022-10-31T01:50:18.292636Z","shell.execute_reply.started":"2022-10-31T01:50:18.273435Z","shell.execute_reply":"2022-10-31T01:50:18.291418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:50:20.782497Z","iopub.execute_input":"2022-10-31T01:50:20.783097Z","iopub.status.idle":"2022-10-31T01:50:20.801630Z","shell.execute_reply.started":"2022-10-31T01:50:20.783052Z","shell.execute_reply":"2022-10-31T01:50:20.800440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:49:40.286307Z","iopub.execute_input":"2022-10-31T01:49:40.286853Z","iopub.status.idle":"2022-10-31T01:49:40.293893Z","shell.execute_reply.started":"2022-10-31T01:49:40.286796Z","shell.execute_reply":"2022-10-31T01:49:40.292984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:49:48.337634Z","iopub.execute_input":"2022-10-31T01:49:48.338091Z","iopub.status.idle":"2022-10-31T01:49:48.346177Z","shell.execute_reply.started":"2022-10-31T01:49:48.338050Z","shell.execute_reply":"2022-10-31T01:49:48.344977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the 86% split ratio is preserved with a first axis divisible by the number of remaining assets, i.e., 9.\n\nLet us now save the data and the weights in CSV format:","metadata":{}},{"cell_type":"code","source":"OUTPUTPATH = pathlib.Path(\"/kaggle/working\").resolve()","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:56:21.146846Z","iopub.execute_input":"2022-10-31T01:56:21.147274Z","iopub.status.idle":"2022-10-31T01:56:21.153435Z","shell.execute_reply.started":"2022-10-31T01:56:21.147240Z","shell.execute_reply":"2022-10-31T01:56:21.152092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FILEPATH_training_csv = OUTPUTPATH / \"training.csv\"\nFILEPATH_test_csv = OUTPUTPATH / \"test.csv\"\nFILEPATH_weights_csv = OUTPUTPATH / \"weights.csv\"","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:57:09.147800Z","iopub.execute_input":"2022-10-31T01:57:09.148500Z","iopub.status.idle":"2022-10-31T01:57:09.153860Z","shell.execute_reply.started":"2022-10-31T01:57:09.148441Z","shell.execute_reply":"2022-10-31T01:57:09.152530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_df.to_csv(FILEPATH_training_csv, index=False)\ntest_df.to_csv(FILEPATH_test_csv, index=False)\nweights_df.to_csv(FILEPATH_weights_csv, index=False)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T01:57:09.831733Z","iopub.execute_input":"2022-10-31T01:57:09.832121Z","iopub.status.idle":"2022-10-31T02:00:12.834197Z","shell.execute_reply.started":"2022-10-31T01:57:09.832090Z","shell.execute_reply":"2022-10-31T02:00:12.833290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert all(x.exists() for x in [FILEPATH_training_csv, \n                                FILEPATH_test_csv, \n                                FILEPATH_weights_csv])","metadata":{"execution":{"iopub.status.busy":"2022-10-31T02:00:12.881492Z","iopub.execute_input":"2022-10-31T02:00:12.881832Z","iopub.status.idle":"2022-10-31T02:00:12.887255Z","shell.execute_reply.started":"2022-10-31T02:00:12.881800Z","shell.execute_reply":"2022-10-31T02:00:12.886044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}