{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Objective of the competition\nTo forecast the short term returns (15 minute residualized returns) in 14 popular cryptocurrencies.\n    \n## Introduction\nCryptocurrency markets are very volatile in nature and has high trading volumes. It is also open 24 * 7 unlike traditional capital markets. This means we can get a continuous flow of data which will be super helpful for time series forecasting. On the other hand, most of the cryptocurrencies do not have the so called underlying fundamentals data and so most of the price movements are driven by technical indicators and market sentiments. This makes it one of the best play grounds for traders to make huge gains (and losses!).\n\nIn this competition, we will be looking at 14 cryptocurrencies and predict the short term price movements of them. We will be using a time-series API that will help us from looking into the future data (try not to play around it :)) and make predictions.\n\n## Data\nWe are given the training file `train.csv` with the following columns \n\n* timestamp - A timestamp for the minute covered by the row.\n* Asset_ID - An ID code for the cryptoasset.\n* Count - The number of trades that took place this minute.\n* Open - The USD price at the beginning of the minute.\n* High - The highest USD price during the minute.\n* Low - The lowest USD price during the minute.\n* Close - The USD price at the end of the minute.\n* Volume - The number of cryptoasset units traded during the minute.\n* VWAP - The volume weighted average price for the minute.\n* Target - 15 minute residualized returns. \n\nWe are given only 14 cryptocurrencies / assets for this competition. Let us look at them in detail.","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nfrom datetime import datetime\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nimport plotly.express as px\n\nimport seaborn as sns\ncmap = sns.color_palette()\n\ncompetition_data_path = \"../input/g-research-crypto-forecasting/\"","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-08T09:20:29.68002Z","iopub.execute_input":"2021-11-08T09:20:29.680584Z","iopub.status.idle":"2021-11-08T09:20:31.745609Z","shell.execute_reply.started":"2021-11-08T09:20:29.680456Z","shell.execute_reply":"2021-11-08T09:20:31.744911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Assets Data Exploration","metadata":{}},{"cell_type":"code","source":"assets_df = pd.read_csv(os.path.join(competition_data_path, \"asset_details.csv\"))\nassets_df.sort_values(\"Weight\", ascending=False)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T09:21:26.905978Z","iopub.execute_input":"2021-11-08T09:21:26.906474Z","iopub.status.idle":"2021-11-08T09:21:26.946854Z","shell.execute_reply.started":"2021-11-08T09:21:26.906425Z","shell.execute_reply":"2021-11-08T09:21:26.945879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that the top cryptocurrencies are given for this competition including bitcoin, ethereum, binance coin etc. Also, we can see the weights column that will be used to get the `weighted pearson correlation coefficient` evaluation metric.\n\nLet us see the sum of the weights to see if they add up to 100.","metadata":{}},{"cell_type":"code","source":"assets_df[\"Weight\"].sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-08T09:21:28.976192Z","iopub.execute_input":"2021-11-08T09:21:28.976813Z","iopub.status.idle":"2021-11-08T09:21:28.983315Z","shell.execute_reply.started":"2021-11-08T09:21:28.976776Z","shell.execute_reply":"2021-11-08T09:21:28.982495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like the weights add up to `40.88`. Not sure of the exact reason. Let us see the percentage of weights given to each of the assets now. ","metadata":{}},{"cell_type":"code","source":"assets_df[\"weight_percentage\"] = (assets_df[\"Weight\"] / assets_df[\"Weight\"].sum()) * 100\nassets_df.sort_values(\"Weight\", ascending=False)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T09:21:28.98545Z","iopub.execute_input":"2021-11-08T09:21:28.986044Z","iopub.status.idle":"2021-11-08T09:21:29.006019Z","shell.execute_reply.started":"2021-11-08T09:21:28.985999Z","shell.execute_reply":"2021-11-08T09:21:29.005097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.pie(assets_df, values='weight_percentage', names='Asset_Name', title='Weights given to each cryptocurrency')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:21:29.007218Z","iopub.execute_input":"2021-11-08T09:21:29.007462Z","iopub.status.idle":"2021-11-08T09:21:30.020949Z","shell.execute_reply.started":"2021-11-08T09:21:29.007437Z","shell.execute_reply":"2021-11-08T09:21:30.020042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Data Exploration\n\nLet us look at the top few rows of the data to start with.","metadata":{}},{"cell_type":"code","source":"crypto_df = pd.read_csv(os.path.join(competition_data_path, \"train.csv\"))\nprint(f\"There are {crypto_df.shape[0]} rows in the given dataset\")\ncrypto_df.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:21:30.022606Z","iopub.execute_input":"2021-11-08T09:21:30.022835Z","iopub.status.idle":"2021-11-08T09:22:27.217903Z","shell.execute_reply.started":"2021-11-08T09:21:30.02281Z","shell.execute_reply":"2021-11-08T09:22:27.216872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Start & End Time\nLet us now look at the start and end time of each of the given crypto currencies.","metadata":{}},{"cell_type":"code","source":"asset_names_dict = {row[\"Asset_Name\"]:row[\"Asset_ID\"] for ind, row in assets_df.iterrows()}\n\nasset_names = [\n    'Bitcoin',\n    'Ethereum',\n    'Cardano',\n    'Binance Coin',\n    'Dogecoin',\n    'Bitcoin Cash',\n    'Litecoin',\n    'Ethereum Classic',\n    'Stellar',\n    'TRON',\n    'Monero',\n    'EOS.IO',\n    'IOTA',\n    'Maker'\n]\n\ntime_list = []\nfor coin in asset_names:\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    start_time = coin_df.index[0].astype('datetime64[s]')\n    end_time = coin_df.index[-1].astype('datetime64[s]')\n    time_list.append([coin, start_time, end_time])\ntime_df = pd.DataFrame(time_list)\ntime_df.columns = [\"Asset_Name\", \"Start_Time\", \"End_Time\"]\ntime_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:22:27.22018Z","iopub.execute_input":"2021-11-08T09:22:27.220472Z","iopub.status.idle":"2021-11-08T09:22:31.26863Z","shell.execute_reply.started":"2021-11-08T09:22:27.220435Z","shell.execute_reply":"2021-11-08T09:22:31.266874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* The earliest start date in the data is Jan 1, 2018 and most coins have that start datetime.\n* The data is available till Sep 21, 2021 and all the coins have the same end datetime.\n* Dogecoin has the least historical information available of the given coins and is captured onlt from April 2019. \n\n### Missing data\n\nNow let us check the missing data in each of these assets / coins.","metadata":{}},{"cell_type":"code","source":"missing_list = []\nfor coin in asset_names:\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    missing_list.append([coin, coin_df.shape[0]] + coin_df.isna().sum().tolist())\nmissing_df = pd.DataFrame(missing_list)\nmissing_df.columns = [\"Asset_Name\", \"TotalRows\", 'Missing_Asset_ID', 'Missing_Count', 'Missing_Open', \n                      'Missing_High', 'Missing_Low', 'Missing_Close', 'Missing_Volume', 'Missing_VWAP', 'Missing_Target']\nmissing_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:22:31.269913Z","iopub.execute_input":"2021-11-08T09:22:31.270365Z","iopub.status.idle":"2021-11-08T09:22:35.756891Z","shell.execute_reply.started":"2021-11-08T09:22:31.270331Z","shell.execute_reply":"2021-11-08T09:22:35.7562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* There is a lot of missing data in the Target column\n* All the other columns do not have missing data\n* Bitcoin has the least number of missing target values and Monero the most.\n* We need to deal with these missing target values carefully.\n\n### Gaps\n\nIt is also mentioned in the competition that missing asset data, for a given minute, is not represented by NaN's, but instead by the absence of those rows. So let us check the gaps in the data. \n\nWe know that the data is collected once every 60 seconds. If we have a gap of more than one minute between consecutive rows, then it indicates that the data is missing in between. Let us check that for each of the coins.","metadata":{}},{"cell_type":"code","source":"gap_list = []\nfor coin in asset_names:\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    gap_df = (coin_df.index[1:]-coin_df.index[:-1]).value_counts().reset_index()\n    n_gaps = gap_df[gap_df[\"index\"]!=60][\"timestamp\"].sum()\n    max_gap = gap_df[\"index\"].max()\n    gap_list.append([coin, n_gaps, max_gap])\ngap_df = pd.DataFrame(gap_list)\ngap_df.columns = [\"Asset_Name\", \"number_of_gaps\", \"max_gap_in_seconds\"]\ngap_df","metadata":{"execution":{"iopub.status.busy":"2021-11-08T09:22:35.759206Z","iopub.execute_input":"2021-11-08T09:22:35.759849Z","iopub.status.idle":"2021-11-08T09:22:40.091451Z","shell.execute_reply.started":"2021-11-08T09:22:35.759801Z","shell.execute_reply":"2021-11-08T09:22:40.090453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* There are several gaps in the dataset and we need to treat them carefully\n* Maker coin has the maximum gap of 281700 seconds or about 78 hours\n* When we have gaps in the data, we can't compute the returns for the time period that comes just after these gaps. I think that is the reason we have missing target values in our data. \n\n### Visualization\n\nNow let us visualize the training data and see how the prices of the coins varied over time.","metadata":{}},{"cell_type":"code","source":"# plot close values as time series for all the assets\nf = plt.figure(figsize=(15,30))\n\nfor ind, coin in enumerate(asset_names):\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    # fill missing values \n    coin_df = coin_df.reindex(range(coin_df.index[0],coin_df.index[-1]+60,60),method='pad')\n    ax = f.add_subplot(7,2,ind+1)\n    plt.plot(coin_df['Close'], label=coin, color=cmap[ind%10])\n    plt.legend()\n    plt.xlabel('Time')\n    plt.ylabel(coin)\n    plt.title(coin)\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:22:40.092685Z","iopub.execute_input":"2021-11-08T09:22:40.092916Z","iopub.status.idle":"2021-11-08T09:23:13.538363Z","shell.execute_reply.started":"2021-11-08T09:22:40.09289Z","shell.execute_reply":"2021-11-08T09:23:13.537451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* There seemse to be a good correlation between different crypto assets and also seems to vary in different timelines.\n* We have got the bear data from 2018 timeline, bull data from 2021 timeline, sideways data from 2019 and 2020 timeline. So this dataset captures different types of market for us to model. \n\nAlso different coins have different time lines in the above graph. So let us just take the data after July 1, 2021 and see the price movements of different assets.","metadata":{}},{"cell_type":"code","source":"import time\n# auxiliary function, from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n\n# create intervals\nf = plt.figure(figsize=(15,30))\n\nfor ind, coin in enumerate(asset_names):\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    # fill missing values \n    coin_df = coin_df.reindex(range(coin_df.index[0],coin_df.index[-1]+60,60),method='pad')\n    coin_df = coin_df.loc[totimestamp('01/07/2021'):totimestamp('21/09/2021')]\n    ax = f.add_subplot(7,2,ind+1)\n    plt.plot(coin_df['Close'], label=coin, color=cmap[ind%10])\n    plt.legend()\n    plt.xlabel('Time')\n    plt.ylabel(coin)\n    plt.title(coin)\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:23:13.540178Z","iopub.execute_input":"2021-11-08T09:23:13.540958Z","iopub.status.idle":"2021-11-08T09:23:26.143312Z","shell.execute_reply.started":"2021-11-08T09:23:13.540907Z","shell.execute_reply":"2021-11-08T09:23:26.142375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Correlation\n\n### Correlation on actual price values\n\nLet us create a correlation plot on the actual price values of these different assets for the latest time period (from July 1, 2021).","metadata":{}},{"cell_type":"code","source":"all_assets_df = pd.DataFrame([])\nfor ind, coin in enumerate(asset_names):\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    coin_df = coin_df.reindex(range(coin_df.index[0],coin_df.index[-1]+60,60),method='pad')\n    # fill missing values\n    coin_df = coin_df.loc[totimestamp('01/07/2021'):totimestamp('21/09/2021')]\n    close_values = coin_df[\"Close\"].fillna(0)\n    close_values.name = coin\n    all_assets_df = all_assets_df.join(close_values, how=\"outer\")\n\ncorrmat = all_assets_df.corr()\nfig, ax = plt.subplots(figsize=(14, 14))\nsns.heatmap(corrmat, vmax=1., square=True, cmap=\"rocket_r\")\nplt.title(\"Cryptocurrency correlation map on actual price values\", fontsize=15)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:23:26.144742Z","iopub.execute_input":"2021-11-08T09:23:26.145021Z","iopub.status.idle":"2021-11-08T09:23:33.28197Z","shell.execute_reply.started":"2021-11-08T09:23:26.144991Z","shell.execute_reply":"2021-11-08T09:23:33.281046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* We could clearly see the high correlation between different assets\n* Lowest correlation itself is in the range of 0.4x\n* There is a high correlation between\n    * Bitcoin & Ethereum\n    * Cardano & Binance coin\n    * Bitcoin Cash & Litecoin\n    \n### Correlation on Log Returns\n\nBut we got the correlation on actual values. Different coins are in different price ranges. So let us use the `log returns` as suggested by the [tutorial notebook](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition) and check the correlation for the latest time period.\n\nFrom the tutorial notebook, \n\n**Log returns**\n\nIn order to analyze price changes for an asset we can deal with the price difference. However, different assets exhibit different price scales, so that the their returns are not readily comparable. We can solve this problem by computing the percentage change in price instead, also known as the return. This return coincides with the percentage change in our invested capital.\n\nReturns are widely used in finance, however log returns are preferred for mathematical modelling of time series, as they are additive across time. Also, while regular returns cannot go below -100%, log returns are not bounded.\n\nTo compute the log return, we can simply take the logarithm of the ratio between two consecutive prices. The first row will have an empty return as the previous value is unknown, therefore the empty return data point will be dropped.","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:03:54.526283Z","iopub.execute_input":"2021-11-03T11:03:54.526631Z","iopub.status.idle":"2021-11-03T11:03:55.344318Z","shell.execute_reply.started":"2021-11-03T11:03:54.526583Z","shell.execute_reply":"2021-11-03T11:03:55.343386Z"}}},{"cell_type":"code","source":"# define function to compute log returns\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T09:23:33.283691Z","iopub.execute_input":"2021-11-08T09:23:33.284112Z","iopub.status.idle":"2021-11-08T09:23:33.289156Z","shell.execute_reply.started":"2021-11-08T09:23:33.284067Z","shell.execute_reply":"2021-11-08T09:23:33.288239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_assets_df = pd.DataFrame([])\nfor ind, coin in enumerate(asset_names):\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin]].set_index(\"timestamp\")\n    coin_df = coin_df.reindex(range(coin_df.index[0],coin_df.index[-1]+60,60),method='pad')\n    # fill missing values\n    coin_df = coin_df.loc[totimestamp('01/07/2021'):totimestamp('21/09/2021')]\n    return_values = log_return(coin_df[\"Close\"].fillna(0))[1:]\n    return_values.name = coin\n    all_assets_df = all_assets_df.join(return_values, how=\"outer\")\n\ncorrmat = all_assets_df.corr()\nfig, ax = plt.subplots(figsize=(14, 14))\nsns.heatmap(corrmat, vmax=1., square=True, cmap=\"rocket_r\")\nplt.title(\"Correlation map on log returns of the coins\", fontsize=15)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:23:33.291142Z","iopub.execute_input":"2021-11-08T09:23:33.291508Z","iopub.status.idle":"2021-11-08T09:23:40.445881Z","shell.execute_reply.started":"2021-11-08T09:23:33.291466Z","shell.execute_reply":"2021-11-08T09:23:40.445213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* In the recent months, when we look at the log returns, IOTA has a very low correlation with other coins\n* Bitcoin and Ethereum still shows a high correlation.\n* Looks like correlation between the assets change over time and so this needs to be addressed while building models\n\n### Correlation change over time\n\nNow let us see how the correlations have changed over time on the log returns. We can just look at the coins Bitcoin, Ethereum, Dogecoin and IOTA and see how the correlations have changed over time.","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:15:11.59211Z","iopub.execute_input":"2021-11-03T11:15:11.593944Z","iopub.status.idle":"2021-11-03T11:15:11.634412Z","shell.execute_reply.started":"2021-11-03T11:15:11.593874Z","shell.execute_reply":"2021-11-03T11:15:11.633393Z"}}},{"cell_type":"code","source":"from bokeh.models import Panel, Tabs\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\n\noutput_notebook()\n\ndef get_plots(df, col1, col2):\n    p = figure(plot_width=1000, plot_height=500, title=f\"{col1} & {col2} - correlation distribution over time\")\n    p.line(df['timestamp'], df[col1], alpha=0.7)\n    return p\n\nmain_tabs_list = []\nfor coin1 in ['Bitcoin', 'Ethereum', 'Dogecoin', 'IOTA']:\n    tab_list = []\n    coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin1]].set_index(\"timestamp\")\n    coin_df = coin_df.reindex(range(coin_df.index[0],coin_df.index[-1]+60,60),method='pad')\n    return1_values = log_return(coin_df[\"Close\"].fillna(0))[1:]\n    return1_values.name = coin1\n    for coin2 in asset_names:\n        if coin2 != coin1:\n            coin_df = crypto_df[crypto_df[\"Asset_ID\"]==asset_names_dict[coin2]].set_index(\"timestamp\")\n            coin_df = coin_df.reindex(range(coin_df.index[0],coin_df.index[-1]+60,60),method='pad')\n            return2_values = log_return(coin_df[\"Close\"].fillna(0))[1:]\n            return2_values.name = coin2\n            two_assets = pd.concat([return1_values, return2_values], axis=1)\n            corr_time_df = two_assets.groupby(two_assets.index//(10000*60)).corr().loc[:,coin1].loc[:,coin2].reset_index()\n            p = get_plots(corr_time_df, coin1, coin2)\n            tab = Panel(child=p, title=f\"Corr:{coin1} & {coin2}\")\n            tab_list.append(tab)\n    tabs = Tabs(tabs=tab_list)\n    panel = Panel(child=tabs, title=coin1)\n    main_tabs_list.append(panel)\n            \n\ntabs = Tabs(tabs=main_tabs_list)\nshow(tabs)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T09:40:37.483887Z","iopub.execute_input":"2021-11-08T09:40:37.48422Z","iopub.status.idle":"2021-11-08T09:41:21.589841Z","shell.execute_reply.started":"2021-11-08T09:40:37.484185Z","shell.execute_reply":"2021-11-08T09:41:21.588973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n* We could clearly see that the correlations between the assets are changing over time\n* We can see that the correlation between bitcoin and ethereum is volatile and is high for most time periods.\n* If we look at the correlation of assets with IOTA, it is not as high as other pairs.\n* Since we are looking at the residualized returns in this competition, correlation between the assets play an important role. \n","metadata":{}},{"cell_type":"markdown","source":"### Reference\n\nThanks to all the below notebooks / discussions which served as references. Kindly upvote them too.\n\n* [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n\n#### Notebook in progress. Please visit later for more updates.","metadata":{}}]}