{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-05T19:02:47.211569Z","iopub.execute_input":"2022-01-05T19:02:47.211750Z","iopub.status.idle":"2022-01-05T19:02:47.219188Z","shell.execute_reply.started":"2022-01-05T19:02:47.211726Z","shell.execute_reply":"2022-01-05T19:02:47.218454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nfrom datetime import datetime","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:02:50.510641Z","iopub.execute_input":"2022-01-05T19:02:50.511168Z","iopub.status.idle":"2022-01-05T19:02:51.378252Z","shell.execute_reply.started":"2022-01-05T19:02:50.511130Z","shell.execute_reply":"2022-01-05T19:02:51.377528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import datetime\nimport scipy.stats as stats\nfrom datetime import datetime","metadata":{"execution":{"iopub.status.busy":"2022-01-05T22:41:12.423410Z","iopub.execute_input":"2022-01-05T22:41:12.424162Z","iopub.status.idle":"2022-01-05T22:41:12.428442Z","shell.execute_reply.started":"2022-01-05T22:41:12.424129Z","shell.execute_reply":"2022-01-05T22:41:12.427754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = '../input/g-research-crypto-forecasting/'","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:02:53.135235Z","iopub.execute_input":"2022-01-05T19:02:53.135575Z","iopub.status.idle":"2022-01-05T19:02:53.144130Z","shell.execute_reply.started":"2022-01-05T19:02:53.135535Z","shell.execute_reply":"2022-01-05T19:02:53.143255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls {path}","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:02:53.515979Z","iopub.execute_input":"2022-01-05T19:02:53.516709Z","iopub.status.idle":"2022-01-05T19:02:54.187468Z","shell.execute_reply.started":"2022-01-05T19:02:53.516654Z","shell.execute_reply":"2022-01-05T19:02:54.186412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dimensions of the datasets","metadata":{}},{"cell_type":"markdown","source":"Let's start loading datasets and explore the dimensions !","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(path+\"train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:02:56.186190Z","iopub.execute_input":"2022-01-05T19:02:56.186819Z","iopub.status.idle":"2022-01-05T19:03:46.984597Z","shell.execute_reply.started":"2022-01-05T19:02:56.186779Z","shell.execute_reply":"2022-01-05T19:03:46.983686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_train.shape)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:46.986610Z","iopub.execute_input":"2022-01-05T19:03:46.986897Z","iopub.status.idle":"2022-01-05T19:03:47.012877Z","shell.execute_reply.started":"2022-01-05T19:03:46.986859Z","shell.execute_reply":"2022-01-05T19:03:47.011986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The meanings of the features included in the set are the following:\n* timestamp: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n* Asset_ID: The asset ID corresponding to one of the crytocurrencies (e.g. Asset_ID = 1 for Bitcoin). The mapping from Asset_ID to crypto asset is contained in asset_details.csv.\n* Count: Total number of trades in the time interval (last minute).\n* Open: Opening price of the time interval (in USD).\n* High: Highest price reached during time interval (in USD).\n* Low: Lowest price reached during time interval (in USD).\n* Close: Closing price of the time interval (in USD).\n* Volume: Quantity of asset bought or sold, displayed in base currency USD.\n* VWAP: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n* Target: Residual log-returns for the asset over a 15 minute horizon.","metadata":{}},{"cell_type":"code","source":"df_asset_details = pd.read_csv(path+\"asset_details.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:47.014120Z","iopub.execute_input":"2022-01-05T19:03:47.014438Z","iopub.status.idle":"2022-01-05T19:03:47.023180Z","shell.execute_reply.started":"2022-01-05T19:03:47.014404Z","shell.execute_reply":"2022-01-05T19:03:47.022427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_asset_details.shape)\nprint(df_asset_details.sort_values(\"Asset_ID\"))","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:47.026644Z","iopub.execute_input":"2022-01-05T19:03:47.027112Z","iopub.status.idle":"2022-01-05T19:03:47.040384Z","shell.execute_reply.started":"2022-01-05T19:03:47.027077Z","shell.execute_reply":"2022-01-05T19:03:47.039719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assets = df_asset_details['Asset_Name'].tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:47.041436Z","iopub.execute_input":"2022-01-05T19:03:47.041737Z","iopub.status.idle":"2022-01-05T19:03:47.046202Z","shell.execute_reply.started":"2022-01-05T19:03:47.041702Z","shell.execute_reply":"2022-01-05T19:03:47.045436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(assets)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:47.047769Z","iopub.execute_input":"2022-01-05T19:03:47.048299Z","iopub.status.idle":"2022-01-05T19:03:47.055891Z","shell.execute_reply.started":"2022-01-05T19:03:47.048259Z","shell.execute_reply":"2022-01-05T19:03:47.055111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_supp_train = pd.read_csv(path+\"supplemental_train.csv\")\nprint(df_supp_train.shape)\ndf_supp_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:47.057559Z","iopub.execute_input":"2022-01-05T19:03:47.057849Z","iopub.status.idle":"2022-01-05T19:03:51.518359Z","shell.execute_reply.started":"2022-01-05T19:03:47.057816Z","shell.execute_reply":"2022-01-05T19:03:51.517584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Does this dataset merely contain additional/supplemental rows for the training set ? Let's check the timestamps.","metadata":{}},{"cell_type":"code","source":"print(df_train['timestamp'].astype('datetime64[s]').max())\nprint(df_supp_train['timestamp'].astype('datetime64[s]').min())","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:51.519575Z","iopub.execute_input":"2022-01-05T19:03:51.520101Z","iopub.status.idle":"2022-01-05T19:03:53.388851Z","shell.execute_reply.started":"2022-01-05T19:03:51.520059Z","shell.execute_reply":"2022-01-05T19:03:53.388038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It does not look like the records in supplemental set come after in time, the records in the training set.\nLet's first explore what we have in the training set, then will come back to this !","metadata":{}},{"cell_type":"code","source":"df_test = pd.read_csv(path+\"example_test.csv\")\nprint(df_test.shape)\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:53.389987Z","iopub.execute_input":"2022-01-05T19:03:53.390249Z","iopub.status.idle":"2022-01-05T19:03:53.411473Z","shell.execute_reply.started":"2022-01-05T19:03:53.390203Z","shell.execute_reply":"2022-01-05T19:03:53.410783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Example test set has a column 'group_num' and an additional column in row_id, and missing target column from the training set.\nGroup_num column is a mystery to me for now. Let's take a look at the distinct values of this column.","metadata":{}},{"cell_type":"code","source":"print(set(df_test['group_num']))","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:53.414280Z","iopub.execute_input":"2022-01-05T19:03:53.414556Z","iopub.status.idle":"2022-01-05T19:03:53.419045Z","shell.execute_reply.started":"2022-01-05T19:03:53.414523Z","shell.execute_reply":"2022-01-05T19:03:53.418080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am removing the datasets other than the training set and the crypto-assets details set to avoid overusage of the memory.","metadata":{}},{"cell_type":"markdown","source":"Let us join the two datasets on the AssetID column.","metadata":{}},{"cell_type":"code","source":"df = df_train.set_index('Asset_ID').join(df_asset_details.set_index('Asset_ID'))","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:53.420497Z","iopub.execute_input":"2022-01-05T19:03:53.421510Z","iopub.status.idle":"2022-01-05T19:03:59.414744Z","shell.execute_reply.started":"2022-01-05T19:03:53.421466Z","shell.execute_reply":"2022-01-05T19:03:59.411875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#dropping this unnecessary column\ndf = df.drop(['Weight'], axis = 1)\nprint(df.shape)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:03:59.417260Z","iopub.execute_input":"2022-01-05T19:03:59.417694Z","iopub.status.idle":"2022-01-05T19:04:01.934428Z","shell.execute_reply.started":"2022-01-05T19:03:59.417654Z","shell.execute_reply":"2022-01-05T19:04:01.933674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Checking of Missing values ","metadata":{}},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:01.935828Z","iopub.execute_input":"2022-01-05T19:04:01.936308Z","iopub.status.idle":"2022-01-05T19:04:04.549994Z","shell.execute_reply.started":"2022-01-05T19:04:01.936260Z","shell.execute_reply":"2022-01-05T19:04:04.549206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems like there are no NULLs except in the Target column.","metadata":{}},{"cell_type":"code","source":"del(df_test)\ndel(df_supp_train)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:04.551272Z","iopub.execute_input":"2022-01-05T19:04:04.551539Z","iopub.status.idle":"2022-01-05T19:04:04.556179Z","shell.execute_reply.started":"2022-01-05T19:04:04.551505Z","shell.execute_reply":"2022-01-05T19:04:04.555409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Statistical Analysis","metadata":{}},{"cell_type":"markdown","source":"Let us look at the distribution of the column for each of the crypto assets individually !","metadata":{}},{"cell_type":"markdown","source":"Let's check how daywise Open-Close is distributed for each crypto-asset ?","metadata":{}},{"cell_type":"code","source":"df['Diff'] = df['Close'] - df['Open']\ndf['timestamp'] = df['timestamp'].astype('datetime64[s]')\ndf['date'] = pd.to_datetime(df['timestamp']).dt.date","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:04.557821Z","iopub.execute_input":"2022-01-05T19:04:04.558068Z","iopub.status.idle":"2022-01-05T19:04:17.459621Z","shell.execute_reply.started":"2022-01-05T19:04:04.558034Z","shell.execute_reply":"2022-01-05T19:04:17.458833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:17.461148Z","iopub.execute_input":"2022-01-05T19:04:17.461427Z","iopub.status.idle":"2022-01-05T19:04:17.479443Z","shell.execute_reply.started":"2022-01-05T19:04:17.461392Z","shell.execute_reply":"2022-01-05T19:04:17.478631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Daywise Close - Open for all assets","metadata":{}},{"cell_type":"code","source":"grouped_df = df.groupby(['Asset_Name', 'date'], as_index = False).agg({'Open': ['min'], 'Close': ['max']})\n","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:02:17.764852Z","iopub.execute_input":"2022-01-05T00:02:17.765785Z","iopub.status.idle":"2022-01-05T00:02:25.002513Z","shell.execute_reply.started":"2022-01-05T00:02:17.765728Z","shell.execute_reply":"2022-01-05T00:02:25.001666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:02:25.004396Z","iopub.execute_input":"2022-01-05T00:02:25.004631Z","iopub.status.idle":"2022-01-05T00:02:25.010189Z","shell.execute_reply.started":"2022-01-05T00:02:25.004597Z","shell.execute_reply":"2022-01-05T00:02:25.009507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_df['diff_val'] = grouped_df[('Close', 'max')] - grouped_df[('Open', 'min')]\ngrouped_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:02:25.011422Z","iopub.execute_input":"2022-01-05T00:02:25.011797Z","iopub.status.idle":"2022-01-05T00:02:25.038472Z","shell.execute_reply.started":"2022-01-05T00:02:25.011762Z","shell.execute_reply":"2022-01-05T00:02:25.037775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=14, ncols=1)\n#ax = plt.axes()\ncount = 0\n\nfor asset in assets:\n    \n    grouped_df[grouped_df['Asset_Name'] == asset]['diff_val'].plot.density(figsize = (10, 30),\n                       linewidth = 1.5, ax = ax[count], label = asset)\n    ax[count].legend()\n    ax[count].grid()\n    count = count + 1\n","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:02:25.039733Z","iopub.execute_input":"2022-01-05T00:02:25.039972Z","iopub.status.idle":"2022-01-05T00:02:27.620104Z","shell.execute_reply.started":"2022-01-05T00:02:25.039939Z","shell.execute_reply":"2022-01-05T00:02:27.619375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#To save memory\ndel(df_grouped_df)\ndel(fig, ax)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Interpretation of the above charts:\n* For Bitcoin, the Daywise average difference between Close and Open has more weight towards the the positive scale and reaches as far as the value +20,000. On the negative side of the scale, it has picked up values even lower than -5000. This range is higher than any other asset.\n* Ethereum Classic and Maker also show highe range of difference values and have considerable amount of skew towards the positive differences.","metadata":{}},{"cell_type":"markdown","source":"#### Distribution of number of trades","metadata":{}},{"cell_type":"markdown","source":"If the count column represents the trades made in a minute then let's look at the distribution of number of trades made on daily basis for all 14 cryptoassets.","metadata":{}},{"cell_type":"code","source":"grouped_df = df.groupby(['Asset_Name', 'date'], as_index = False).agg({'Count': ['sum']})","metadata":{"execution":{"iopub.status.busy":"2022-01-05T22:20:40.075144Z","iopub.execute_input":"2022-01-05T22:20:40.075437Z","iopub.status.idle":"2022-01-05T22:20:48.352775Z","shell.execute_reply.started":"2022-01-05T22:20:40.075405Z","shell.execute_reply":"2022-01-05T22:20:48.352031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:06:09.259913Z","iopub.execute_input":"2022-01-05T00:06:09.260168Z","iopub.status.idle":"2022-01-05T00:06:09.272161Z","shell.execute_reply.started":"2022-01-05T00:06:09.260140Z","shell.execute_reply":"2022-01-05T00:06:09.271384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=14, ncols=1)\n\ncount = 0\n\nfor asset in assets:\n    \n    grouped_df[grouped_df['Asset_Name'] == asset].plot(x = 'date', y = ('Count', 'sum'), figsize = (15, 50),\n                                                      linewidth = 1.5, ax = ax[count], label = asset)\n    \n    ax[count].legend(loc='upper right')\n    ax[count].grid()\n    count = count + 1\n#ax.legend(assets)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:06:11.036821Z","iopub.execute_input":"2022-01-05T00:06:11.037543Z","iopub.status.idle":"2022-01-05T00:06:14.404671Z","shell.execute_reply.started":"2022-01-05T00:06:11.037506Z","shell.execute_reply":"2022-01-05T00:06:14.403231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Intrepretation:\nNumber of trades have sharply increased at the beginning of the year 2021 compared to their corresponding levels before. Not only that, for all 14 assets the count of trades decreased in the middle of year 2021 and onwards. It showed be noted that we are talking about similarity in pattern here and not in count. The count varies from asset to asset. Bitcoin, Ethereum, and Ethereum Classic has more trades throughout the time in this set. ","metadata":{}},{"cell_type":"markdown","source":"### How Close varies in time for different assets ?","metadata":{}},{"cell_type":"markdown","source":"Let's investigate how the closed values varied in time for all 14 cryptocurrencies ?","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=14, ncols=1)\n\ncount = 0\n\nfor asset in assets:\n    \n    df[df['Asset_Name'] == asset].plot(x = 'timestamp', y = 'Close', figsize = (15, 60),\n                                                      linewidth = 1.5, ax = ax[count], label = asset)\n    \n    ax[count].legend(loc='upper right')\n    ax[count].grid()\n    count = count + 1","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:06:19.235234Z","iopub.execute_input":"2022-01-05T00:06:19.235934Z","iopub.status.idle":"2022-01-05T00:09:23.382849Z","shell.execute_reply.started":"2022-01-05T00:06:19.235897Z","shell.execute_reply":"2022-01-05T00:09:23.382149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del(fig, ax)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T22:42:18.130983Z","iopub.execute_input":"2022-01-04T22:42:18.131824Z","iopub.status.idle":"2022-01-04T22:42:18.135921Z","shell.execute_reply.started":"2022-01-04T22:42:18.131781Z","shell.execute_reply":"2022-01-04T22:42:18.13488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Interpretations:\nFirst, these results are shown regardless of what the opening value was !\nNext, notice the similarity in patterns on how the Close values for the all the assets have increased in the beginning of the year 2021 and then abated in the middle of the year. This is in line with the number of trades made for each asset. There is a strong correlation then (*we haven't looked at it yet but we should quantify the correlation !*)","metadata":{}},{"cell_type":"markdown","source":"####  Log returns for different assets","metadata":{}},{"cell_type":"code","source":"# This is the function to define compute log returns as shared in the Tutorial notebook !\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T22:34:09.430381Z","iopub.execute_input":"2022-01-05T22:34:09.431132Z","iopub.status.idle":"2022-01-05T22:34:09.437279Z","shell.execute_reply.started":"2022-01-05T22:34:09.431089Z","shell.execute_reply":"2022-01-05T22:34:09.436604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# removing asset_ID as index\ndf.reset_index(inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:53.363737Z","iopub.execute_input":"2022-01-05T19:04:53.364640Z","iopub.status.idle":"2022-01-05T19:04:53.445246Z","shell.execute_reply.started":"2022-01-05T19:04:53.364585Z","shell.execute_reply":"2022-01-05T19:04:53.444404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# setting timestamp as my new index\ndf.set_index('timestamp', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:53.702860Z","iopub.execute_input":"2022-01-05T19:04:53.703248Z","iopub.status.idle":"2022-01-05T19:04:53.712253Z","shell.execute_reply.started":"2022-01-05T19:04:53.703213Z","shell.execute_reply":"2022-01-05T19:04:53.711428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here we go !\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:04:54.092017Z","iopub.execute_input":"2022-01-05T19:04:54.092257Z","iopub.status.idle":"2022-01-05T19:04:54.110362Z","shell.execute_reply.started":"2022-01-05T19:04:54.092229Z","shell.execute_reply":"2022-01-05T19:04:54.109608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2022-01-05T19:05:37.041276Z","iopub.execute_input":"2022-01-05T19:05:37.041837Z","iopub.status.idle":"2022-01-05T19:05:37.051110Z","shell.execute_reply.started":"2022-01-05T19:05:37.041798Z","shell.execute_reply":"2022-01-05T19:05:37.050210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=14, ncols=1)\n\ncount = 0\n\nfor asset in assets:\n    \n    sample_df = df[(df['date']>datetime.date(2021,1,1)) & (df['Asset_Name'] == asset)]\n    log_sample = log_return(sample_df.Close)[1:]\n    log_sample.plot(figsize = (15, 60), linewidth = 1.5, ax = ax[count], label = asset)\n    \n    ax[count].legend(loc='upper right')\n    ax[count].grid()\n    count = count + 1\n    del(sample_df)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T00:14:38.534465Z","iopub.execute_input":"2022-01-05T00:14:38.535067Z","iopub.status.idle":"2022-01-05T00:16:32.252806Z","shell.execute_reply.started":"2022-01-05T00:14:38.535025Z","shell.execute_reply":"2022-01-05T00:16:32.250981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Interpretation:\nThe above charts show the log returns for all 14 assets, centered and fluctuating around zero all of them, across the entire time (*makes sense!*). We should have a probability distribution plot for this as well !!! might have helped more.","metadata":{}},{"cell_type":"markdown","source":"#### Correlation between Cryptocurrencies:\nWe first compute the correlation between all the assets for the samples of trade or minutes after Jan 1, 2021.","metadata":{}},{"cell_type":"code","source":"# create dataframe with returns for the above assets\nassets_samples = pd.DataFrame([])\n\nfor i in range(0, len(assets)):\n    \n    asset_name = assets[i]\n    df_1 = df[(df['date']>datetime.date(2021,1,1)) & (df['Asset_Name'] == asset_name)]\n    lret_df = log_return(df_1.Close.fillna(0))[1:]\n    lret_df.rename(asset_name)\n    \n    assets_samples = assets_samples.join(lret_df, rsuffix=asset_name, how = 'outer')\n    \n    del(df_1)\n    del(lret_df)\n    ","metadata":{"execution":{"iopub.status.busy":"2022-01-05T20:01:23.934193Z","iopub.execute_input":"2022-01-05T20:01:23.934737Z","iopub.status.idle":"2022-01-05T20:02:46.318099Z","shell.execute_reply.started":"2022-01-05T20:01:23.934698Z","shell.execute_reply":"2022-01-05T20:02:46.317350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assets_samples.columns = assets\nassets_samples.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T20:03:58.145988Z","iopub.execute_input":"2022-01-05T20:03:58.146256Z","iopub.status.idle":"2022-01-05T20:03:58.173222Z","shell.execute_reply.started":"2022-01-05T20:03:58.146220Z","shell.execute_reply":"2022-01-05T20:03:58.172414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr = assets_samples.corr()\ncorr.style.background_gradient(cmap='coolwarm')","metadata":{"execution":{"iopub.status.busy":"2022-01-05T20:06:31.149539Z","iopub.execute_input":"2022-01-05T20:06:31.149801Z","iopub.status.idle":"2022-01-05T20:06:31.460487Z","shell.execute_reply.started":"2022-01-05T20:06:31.149771Z","shell.execute_reply":"2022-01-05T20:06:31.459816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As stated above, these correlation numbers are computed over a sample of time series data after the date Jan 1, 2021. However, in such a dataset, correlation between the columns / assets can vary in time.\nLet us check the consistency in correlation in time for the assets that have high correlation between them. We set the threshold of 0.75 and shortlist the pairs that have correlation number higher than this.\n\n* Bitcoin Cash - Litecoin\n* Bitcoin - Litecoin\n* Bitcoin - Ethereum\n* Ethereum - Litecoin","metadata":{}},{"cell_type":"code","source":"pairs_assets = [('Bitcoin Cash', 'Litecoin'), ('Bitcoin', 'Litecoin'), ('Bitcoin', 'Ethereum'), ('Ethereum', 'Litecoin')]","metadata":{"execution":{"iopub.status.busy":"2022-01-05T20:34:12.362739Z","iopub.execute_input":"2022-01-05T20:34:12.362990Z","iopub.status.idle":"2022-01-05T20:34:12.367123Z","shell.execute_reply.started":"2022-01-05T20:34:12.362962Z","shell.execute_reply":"2022-01-05T20:34:12.366420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(pairs_assets)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T20:34:19.330600Z","iopub.execute_input":"2022-01-05T20:34:19.330990Z","iopub.status.idle":"2022-01-05T20:34:19.337941Z","shell.execute_reply.started":"2022-01-05T20:34:19.330953Z","shell.execute_reply":"2022-01-05T20:34:19.337072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"good_pairs = len(pairs_assets)\n\nfig, ax = plt.subplots(nrows=good_pairs, ncols=1)\n\ncount = 0\n\nfor i in range(good_pairs):\n    df_1 = df[df['Asset_Name'] == pairs_assets[i][0]]\n    lret_df_1 = log_return(df_1.Close)[1:]\n    lret_df_1.rename('lret_df_1', inplace=True)\n    \n    df_2 = df[df['Asset_Name'] == pairs_assets[i][1]]\n    lret_df_2 = log_return(df_2.Close)[1:]\n    lret_df_2.rename('lret_df_2', inplace=True)\n    \n    two_assets = pd.concat([lret_df_1, lret_df_2], axis=1)\n    two_assets.reset_index(inplace = True)\n    \n    two_assets['timestamp'] = two_assets.timestamp.astype('int64') // 10**9\n    two_assets.set_index('timestamp')\n    \n    # group consecutive rows and use .corr() for correlation between columns\n    corr_time = two_assets.groupby(two_assets.index//(1000*20)).corr().loc[:,\"lret_df_1\"].loc[:,\"lret_df_2\"]\n\n    #corr_time.plot()\n    corr_time.plot(figsize = (15, 25), linewidth = 1.5, ax = ax[count], label = str(pairs_assets[i]), color = 'r')\n    ax[count].legend(loc='upper left')\n    ax[count].set_xlabel('TimeStamp')\n    ax[count].set_ylabel(\"Correlation Coefficient\")\n    ax[count].set_title(\"Correlation between the two assets\"+str(pairs_assets[i]))\n    ax[count].grid()\n    \n    count = count + 1\n    \n    del(df_1)\n    del(df_2)\n    del(lret_df_1)\n    del(lret_df_2)\n    del(two_assets)\n    del(corr_time)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T21:50:52.697010Z","iopub.execute_input":"2022-01-05T21:50:52.697461Z","iopub.status.idle":"2022-01-05T21:51:23.012080Z","shell.execute_reply.started":"2022-01-05T21:50:52.697422Z","shell.execute_reply":"2022-01-05T21:51:23.011417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interestingly ! They have different correlation numbers in time but they have a very similar pattern. We can:\n* Make finer the granularity level by wich the timestamps are grouped and check to what extent it holds.\n* Decrease the threshold on the correlation computed on the latest sample to include more pairs in the list and see how the pattern varies.\n* In the time window where these assets-pairs have low correlations, which other assets show higher correlation to these assets.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}