{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import os\nimport pickle\n\nimport pandas as pd\nimport numpy as np\nfrom tqdm.auto import tqdm\nfrom matplotlib import pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-12T07:45:43.741137Z","iopub.execute_input":"2022-01-12T07:45:43.741482Z","iopub.status.idle":"2022-01-12T07:45:44.707709Z","shell.execute_reply.started":"2022-01-12T07:45:43.741395Z","shell.execute_reply":"2022-01-12T07:45:44.706892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From Wikipedia, the free encyclopedia\n\nCointegration is a statistical property of a collection (X1, X2, ..., Xk) of time series variables. First, all of the series must be integrated of order d (see Order of integration). Next, if a linear combination of this collection is integrated of order less than d, then the collection is said to be co-integrated. Formally, if (X,Y,Z) are each integrated of order d, and there exist coefficients a,b,c such that aX + bY + cZ is integrated of order less than d, then X, Y, and Z are cointegrated. \n\nA common example is where the individual series are first-order integrated but some (cointegrating) vector of coefficients exists to form a stationary linear combination of them. For instance, a stock market index and the price of its associated futures contract move through time, each roughly following a random walk. \n\nIf the prices of two assets are said to be cointegrated, then their prices can be expressed in the linear form of y = ax + b. b denotes the residual, which is stationary in time.\n\nHere we perform a cointegration test for each pair of assets in the dataset to determine whether there is a cointegration relationship between them","metadata":{}},{"cell_type":"markdown","source":"# Read Data","metadata":{}},{"cell_type":"code","source":"df_ad = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv').sort_values('Asset_ID')\nid2name = {}\n\nfor row in df_ad.itertuples():\n    id2name[row.Asset_ID] = row.Asset_Name\n    \nid2name","metadata":{"execution":{"iopub.status.busy":"2022-01-12T07:45:44.709311Z","iopub.execute_input":"2022-01-12T07:45:44.709592Z","iopub.status.idle":"2022-01-12T07:45:44.738155Z","shell.execute_reply.started":"2022-01-12T07:45:44.709554Z","shell.execute_reply":"2022-01-12T07:45:44.737579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not os.path.exists('df.p'):\n    df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\n\n    df['trade_date'] = pd.to_datetime(df['timestamp'], unit='s')\n    df.drop('timestamp', axis=1, inplace=True)\n    df = df.set_index(['Asset_ID', 'trade_date']).sort_index().astype(np.float32)\n    df.to_pickle('df.p')\nelse:\n    df = pd.read_pickle('df.p')\n\ndf","metadata":{"execution":{"iopub.status.busy":"2022-01-12T07:46:18.076486Z","iopub.execute_input":"2022-01-12T07:46:18.076760Z","iopub.status.idle":"2022-01-12T07:47:34.053646Z","shell.execute_reply.started":"2022-01-12T07:46:18.076733Z","shell.execute_reply":"2022-01-12T07:47:34.052825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we use data after 2020","metadata":{}},{"cell_type":"code","source":"df2y = df.query('trade_date > \"2020-01-01\"')\ndf2y.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prices in log form","metadata":{}},{"cell_type":"code","source":"for k, v in id2name.items():\n    _df = df2y.loc[k]\n    _df['Close'].plot(logy=True, figsize=(12, 9))\n\nplt.legend([v for k, v in id2name.items()])","metadata":{"execution":{"iopub.status.busy":"2022-01-12T09:28:49.944548Z","iopub.execute_input":"2022-01-12T09:28:49.945479Z","iopub.status.idle":"2022-01-12T09:30:48.457382Z","shell.execute_reply.started":"2022-01-12T09:28:49.945420Z","shell.execute_reply":"2022-01-12T09:30:48.456505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cointegration test of BTC and ETH classic","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nimport statsmodels.tsa.stattools as ts","metadata":{"execution":{"iopub.status.busy":"2022-01-12T07:51:41.111082Z","iopub.execute_input":"2022-01-12T07:51:41.111346Z","iopub.status.idle":"2022-01-12T07:51:42.226383Z","shell.execute_reply.started":"2022-01-12T07:51:41.111319Z","shell.execute_reply":"2022-01-12T07:51:42.225635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prices of BTC and ETH classic in log form","metadata":{}},{"cell_type":"code","source":"k1 = 1 # btc\nk2 = 7 # eth classic\ndf_price = pd.merge(df2y.loc[k1][['Close']], df2y.loc[k2][['Close']], on='trade_date', how='inner')\ndf_price.plot(logy=True, figsize=(12, 9))","metadata":{"execution":{"iopub.status.busy":"2022-01-12T08:51:44.567844Z","iopub.execute_input":"2022-01-12T08:51:44.568136Z","iopub.status.idle":"2022-01-12T08:51:53.087372Z","shell.execute_reply.started":"2022-01-12T08:51:44.568103Z","shell.execute_reply":"2022-01-12T08:51:53.086603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Engle–Granger two-step test\nIf x and y are non-stationary and Order of integration d=1, then a linear combination of them must be stationary for some value of a and b . In other words:\n\ny - ax = b\n\nwhere b is stationary.\n\nIf we knew a , we could just test it for stationarity with something like a Dickey–Fuller test, Phillips–Perron test and be done. But because we don't know a , we must estimate this first, generally by using ordinary least squares and then run our stationarity test on the estimated b series.","metadata":{}},{"cell_type":"markdown","source":"First we fit a linear regression model","metadata":{}},{"cell_type":"code","source":"x = np.log(df_price['Close_x'].values)\ny = np.log(df_price['Close_y'].values)\n\nlm_model = LinearRegression(fit_intercept=True, normalize=False, n_jobs=1)\nlm_model.fit(x.reshape(-1, 1), y)        # fit() expects 2D array\n\nlm_model.coef_, lm_model.intercept_","metadata":{"execution":{"iopub.status.busy":"2022-01-12T09:00:13.631598Z","iopub.execute_input":"2022-01-12T09:00:13.631935Z","iopub.status.idle":"2022-01-12T09:00:13.664004Z","shell.execute_reply.started":"2022-01-12T09:00:13.631903Z","shell.execute_reply":"2022-01-12T09:00:13.663103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The result above means that log(eth) = log(btc) * 0.92 - 6.6 + c, and c is a stationary random variable.","metadata":{}},{"cell_type":"markdown","source":"Then we calc the residual","metadata":{}},{"cell_type":"code","source":"yfit = lm_model.coef_ * x + lm_model.intercept_\ny_residual = y - yfit\ndf_res = df_price[[]].copy()\ndf_res['res'] = y_residual\ndf_res.plot(figsize=(12, 9))","metadata":{"execution":{"iopub.status.busy":"2022-01-12T09:03:33.155417Z","iopub.execute_input":"2022-01-12T09:03:33.155902Z","iopub.status.idle":"2022-01-12T09:03:39.649247Z","shell.execute_reply.started":"2022-01-12T09:03:33.155869Z","shell.execute_reply":"2022-01-12T09:03:39.648211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This also suggests that there may be some arbitrage opportunities between these two assets. We may be able to get some excess return by going long and short over the two assets respectively depending on the spread of the residuals","metadata":{}},{"cell_type":"markdown","source":"Here is the Dickey–Fuller test result. the t-value is -3.8 and p is 0.002, which is pretty significant. That is, there is a high probability that there is a cointegration relationship between the two assets.","metadata":{}},{"cell_type":"code","source":"rst = ts.adfuller(y_residual, 1)\nrst","metadata":{"execution":{"iopub.status.busy":"2022-01-12T09:09:46.287971Z","iopub.execute_input":"2022-01-12T09:09:46.288268Z","iopub.status.idle":"2022-01-12T09:09:47.068443Z","shell.execute_reply.started":"2022-01-12T09:09:46.288238Z","shell.execute_reply":"2022-01-12T09:09:47.067543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Pair wise cointegration test","metadata":{}},{"cell_type":"markdown","source":"Here we perform cointegration tests for all asset pairs and obtain their p-values.","metadata":{}},{"cell_type":"code","source":"def CADF(x, y):\n    lm_model = LinearRegression(fit_intercept=True, normalize=False, n_jobs=1)\n    lm_model.fit(x.reshape(-1, 1), y)        # fit() expects 2D array\n    yfit = lm_model.coef_ * x + lm_model.intercept_\n    y_residual = y - yfit\n    rst = ts.adfuller(y_residual, 1)           # lag = 1\n    \n    return rst[1] # p","metadata":{"execution":{"iopub.status.busy":"2022-01-12T07:48:08.487038Z","iopub.execute_input":"2022-01-12T07:48:08.487352Z","iopub.status.idle":"2022-01-12T07:48:08.494352Z","shell.execute_reply.started":"2022-01-12T07:48:08.487288Z","shell.execute_reply":"2022-01-12T07:48:08.493773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tmp = []\nfor k1, v1 in tqdm(id2name.items()):\n    for k2, v2 in id2name.items():\n        if k1 != k2:\n            df_price = pd.merge(df2y.loc[k1][['Close']], df2y.loc[k2][['Close']], on='trade_date', how='inner')\n\n            x = np.log(df_price['Close_x'].values)\n            y = np.log(df_price['Close_y'].values)\n            p = CADF(x, y)\n            \n            tmp.append((k1, v1, k2, v2, p))\n            \ndfp = pd.DataFrame(tmp, columns=['k1', 'v1', 'k2', 'v2', 'p'])\ndfp.sort_values('p')","metadata":{"execution":{"iopub.status.busy":"2022-01-12T08:08:51.053329Z","iopub.execute_input":"2022-01-12T08:08:51.053625Z","iopub.status.idle":"2022-01-12T08:11:45.344043Z","shell.execute_reply.started":"2022-01-12T08:08:51.053592Z","shell.execute_reply":"2022-01-12T08:11:45.343232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The we use an 'confusion matrix' to visualize the result","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import ConfusionMatrixDisplay","metadata":{"execution":{"iopub.status.busy":"2022-01-12T08:20:22.115441Z","iopub.execute_input":"2022-01-12T08:20:22.115910Z","iopub.status.idle":"2022-01-12T08:20:22.120042Z","shell.execute_reply.started":"2022-01-12T08:20:22.115867Z","shell.execute_reply":"2022-01-12T08:20:22.119328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"m = np.zeros((len(id2name), len(id2name)))\n\nfor k1, v1, k2, v2, p in tmp:\n    m[k1, k2] = p\n\nf = plt.figure(figsize=(16, 16))\nConfusionMatrixDisplay(m, display_labels=[v for k, v in id2name.items()], ).plot(include_values=False, ax=f.gca())","metadata":{"execution":{"iopub.status.busy":"2022-01-12T08:25:03.104617Z","iopub.execute_input":"2022-01-12T08:25:03.104897Z","iopub.status.idle":"2022-01-12T08:25:03.571307Z","shell.execute_reply.started":"2022-01-12T08:25:03.104869Z","shell.execute_reply":"2022-01-12T08:25:03.570434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above figure shows that there is a strong cointegration relationship between eth classic and all other assets. While BTC only has this relationship with a few.","metadata":{}}]}