{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom datetime import datetime as dt\nimport os\nimport plotly.express as px\nimport plotly.graph_objects as go\nsns.set_style(\"whitegrid\")\npalette = sns.color_palette(\"tab10\")\nplt.rcParams['font.size']=13\n\n\n\n#data path\ndata_path = '../input/g-research-crypto-forecasting/'","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-11-08T10:12:05.088229Z","iopub.execute_input":"2021-11-08T10:12:05.088617Z","iopub.status.idle":"2021-11-08T10:12:07.530095Z","shell.execute_reply.started":"2021-11-08T10:12:05.088507Z","shell.execute_reply":"2021-11-08T10:12:07.529089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Reading and Inspecting our Data","metadata":{}},{"cell_type":"code","source":"#Reading in data\ncrypto_trade_df = pd.read_csv(os.path.join(data_path, \"train.csv\"))\nassets_df = pd.read_csv(os.path.join(data_path, \"asset_details.csv\"))\nprint('training data has {:,} rows and {} columns'.format(crypto_trade_df.shape[0],crypto_trade_df.shape[1]))\n","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:12:07.534484Z","iopub.execute_input":"2021-11-08T10:12:07.534816Z","iopub.status.idle":"2021-11-08T10:13:09.765187Z","shell.execute_reply.started":"2021-11-08T10:12:07.534775Z","shell.execute_reply":"2021-11-08T10:13:09.764192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_trade_df.head(5)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:09.766558Z","iopub.execute_input":"2021-11-08T10:13:09.766961Z","iopub.status.idle":"2021-11-08T10:13:09.792364Z","shell.execute_reply.started":"2021-11-08T10:13:09.766925Z","shell.execute_reply":"2021-11-08T10:13:09.791527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Sorting the assets so that we can access the data for each asset in through a dictionary\nassets_df.sort_values(by='Asset_ID',inplace=True)\nassets_df","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:09.794425Z","iopub.execute_input":"2021-11-08T10:13:09.794964Z","iopub.status.idle":"2021-11-08T10:13:09.813662Z","shell.execute_reply.started":"2021-11-08T10:13:09.794903Z","shell.execute_reply":"2021-11-08T10:13:09.812858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Understanding The Data\n\n**Training & Test Data**\n1. ***timestamp*** is the unix time stamp. This will be converted to datetime.\n2. ***Asset_ID*** refers to the particular crypto asset under consideration.\n3. ***Count*** refers to the number of trades that took place in a particular minute.\n4. ***Open*** referst to the USD price at the beginning of the minute.\n5. ***High*** is the highest USD price reached during the minute.\n6. ***Low*** is the lowest USD price during the minute.\n7. ***Close*** refers to the USD price at the end of the minute.\n8. ***Volume*** refers to the amount of the crypto asset involved in the trade.\n9. ***VWAP*** is the volume-weighted-average price of the crytop asset in the minute\n10. ***Target*** according to the [G-Research tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition/notebook) is the 15 minute residualized log returns for the asset.\n\n**Asset Data**\n1. **Weight** refers to the relative importance of each asset in the evalulation metric. **Asset_ID** and **Asset_Name** are self-explanatory.\n","metadata":{}},{"cell_type":"markdown","source":"**Converting unix timestamp to datetime**\n\nWe convert the unix timestamps here to datetime, and set this as the index.","metadata":{}},{"cell_type":"code","source":"crypto_trade_df.timestamp = crypto_trade_df.timestamp.astype('datetime64[s]')\ncrypto_trade_df.set_index('timestamp', inplace=True)\ncrypto_trade_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:09.814941Z","iopub.execute_input":"2021-11-08T10:13:09.815153Z","iopub.status.idle":"2021-11-08T10:13:11.920663Z","shell.execute_reply.started":"2021-11-08T10:13:09.815128Z","shell.execute_reply":"2021-11-08T10:13:11.919813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Assets Dictionary**\n\nWe create a dictionary of *Asset_Names* and their corresponding trade data. This would allow the elegant manipulations and the use of functions down the line.","metadata":{}},{"cell_type":"code","source":"assets = {}\nasset_names = assets_df.Asset_Name\nfor i, asset in enumerate(asset_names):\n    assets[asset] = crypto_trade_df[crypto_trade_df.Asset_ID == i]","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:11.922039Z","iopub.execute_input":"2021-11-08T10:13:11.922358Z","iopub.status.idle":"2021-11-08T10:13:15.138737Z","shell.execute_reply.started":"2021-11-08T10:13:11.922316Z","shell.execute_reply":"2021-11-08T10:13:15.137840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Prepocessing","metadata":{}},{"cell_type":"markdown","source":"**Missing Timestamps**\n\nFrom the [G-Research tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition/notebook), we know that there are missing timestamps for some coins. We can understand this better by subtracting from each current index, the preceding index, which should give the same interval if there are no missing timestamps, but this is not the case as seen below for the *Bitcoin* asset.","metadata":{}},{"cell_type":"code","source":"(assets['Bitcoin'].index[1:]-assets['Bitcoin'].index[:-1]).value_counts().head(5)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:15.140069Z","iopub.execute_input":"2021-11-08T10:13:15.141134Z","iopub.status.idle":"2021-11-08T10:13:15.192089Z","shell.execute_reply.started":"2021-11-08T10:13:15.141093Z","shell.execute_reply":"2021-11-08T10:13:15.191086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We would fix this by reindexing with the missing date times, and filling up empty rows by propagating the last valid observation forward.","metadata":{}},{"cell_type":"code","source":"for asset_name in assets:\n    assets[asset_name] = assets[asset_name].reindex(pd.date_range(start=assets[asset_name].index[0],\n                                                                  end=assets[asset_name].index[-1],freq='T'),method='pad')","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:15.193686Z","iopub.execute_input":"2021-11-08T10:13:15.194007Z","iopub.status.idle":"2021-11-08T10:13:19.412563Z","shell.execute_reply.started":"2021-11-08T10:13:15.193965Z","shell.execute_reply":"2021-11-08T10:13:19.411645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Having done this step for all assets, we can see below that missing time stamps have been taken care of for *Bitcoin*, by running the earlier code again.","metadata":{}},{"cell_type":"code","source":"(assets['Bitcoin'].index[1:]-assets['Bitcoin'].index[:-1]).value_counts().head(5)","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:19.413933Z","iopub.execute_input":"2021-11-08T10:13:19.414273Z","iopub.status.idle":"2021-11-08T10:13:19.460258Z","shell.execute_reply.started":"2021-11-08T10:13:19.414231Z","shell.execute_reply":"2021-11-08T10:13:19.459321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Missing Target Variables**\n\nThe crypto trades dataset is missing so many target variables. We understand from the tutorial that these cases are due to missing values in future prices, and we can ignore this for scoring purposes in the test set ground truth.","metadata":{}},{"cell_type":"code","source":"missing_targets=np.sum([assets[asset].Target.isna().sum() for asset in assets])\nprint('Total number of missing Targets:{:,}'.format(missing_targets))","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:19.463670Z","iopub.execute_input":"2021-11-08T10:13:19.464000Z","iopub.status.idle":"2021-11-08T10:13:19.531241Z","shell.execute_reply.started":"2021-11-08T10:13:19.463962Z","shell.execute_reply":"2021-11-08T10:13:19.529922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Understanding Each Asset\nLet's explore when trade data collection for each asset began as well as the number of days of data and missing targets we have for each asset.","metadata":{}},{"cell_type":"code","source":"start_times = []\nend_times = []\nmissing_targets = []\n\nfor asset in assets:\n    start_times.append(assets[asset].index[0])\n    end_times.append(assets[asset].index[-1])\n    missing_targets.append(assets[asset].Target.isna().sum()) \n    \n    \nasset_description_df = pd.concat([pd.Series([asset for asset in assets]), pd.Series(start_times), pd.Series(end_times), pd.Series(missing_targets)], axis=1)\nasset_description_df.columns = ['asset','start_time','end_time','missing_targets']\nasset_description_df['days_of_data'] = (asset_description_df.end_time - asset_description_df.start_time)\nasset_description_df['%missing_target'] = asset_description_df['missing_targets']/((asset_description_df.end_time - asset_description_df.start_time).dt.total_seconds()/60)\nasset_description_df.set_index('asset',inplace=True)\nasset_description_df = asset_description_df.iloc[:,[0,1,3,2,4]]\nasset_description_df.sort_values(by='start_time')","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:19.532612Z","iopub.execute_input":"2021-11-08T10:13:19.532836Z","iopub.status.idle":"2021-11-08T10:13:19.619040Z","shell.execute_reply.started":"2021-11-08T10:13:19.532808Z","shell.execute_reply":"2021-11-08T10:13:19.618060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data collection stopped on the same day for all assets. We find that not all assets have the same length of data collection, with Dogecoin being the most notable with ~892 days. This is exepcted as there are thousands of crypto assets, and some of this may not have had enough prominence for [CryptoCompare](https://www.cryptocompare.com) (the likely data source for G-Research)to store minute by minute trade data on. More research is needed here.\n\nFurther more, Bitcoin has the lowest number of missing targets, with the highest being Maker with more than 1 million missing targets. Below is a plot of this.","metadata":{}},{"cell_type":"code","source":"asset_description_df.sort_values(by='%missing_target',inplace=True)\n\nfig = px.bar(asset_description_df, y=asset_description_df.index, x='%missing_target', labels = {'asset':'Assets','%missing_target':'Missing Target Percentage'},\n            color_discrete_sequence=['#3366CC'], title= 'Asset Target Missingness')\nfig.update_layout(xaxis=dict(tickformat=\".1%\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:19.620561Z","iopub.execute_input":"2021-11-08T10:13:19.620872Z","iopub.status.idle":"2021-11-08T10:13:20.887146Z","shell.execute_reply.started":"2021-11-08T10:13:19.620838Z","shell.execute_reply":"2021-11-08T10:13:20.886171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"4 assets have more than 20% of targets missing. Assets with a high number of missing targets wil no doubt affect the predictions we're able to make. In a later notebook, I would compare the accuracy of the predictions from those with more complete data, against those with a high percentage of targets missing.","metadata":{}},{"cell_type":"markdown","source":"### Trends Analysis\nLet's explore the trends we have in our data starting with volume traded\n","metadata":{}},{"cell_type":"code","source":"#Monthly Volume Trends\n    \nasset_names= [asset for asset in assets]\nindex=0\nrows=round(len(assets)/2)\ncolumns = 2\nrow_indices = range(rows)\ncolumn_indices = range(columns)\nfig, axes = plt.subplots(rows, columns, figsize=(20, 30))\nfor i in row_indices:\n    for j in column_indices:\n        axes[i,j].plot( assets[asset_names[index]]['Volume'].resample('MS').sum(),\n                       linewidth=2, color=palette[index%10],label=asset_names[index])\n        axes[i,j].legend()\n        axes[i,j].set(ylabel='Total Traded Volume', title =asset_names[index] )\n        index+=1\n        \nplt.suptitle('Monthly Volume Trends For Each Asset',fontsize='x-large')             \nfig.tight_layout(rect=[0, 0.03, 1, 0.98])        \n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-08T10:13:20.888810Z","iopub.execute_input":"2021-11-08T10:13:20.889164Z","iopub.status.idle":"2021-11-08T10:13:31.240093Z","shell.execute_reply.started":"2021-11-08T10:13:20.889113Z","shell.execute_reply":"2021-11-08T10:13:31.239262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learnings**\n\nWe observe the following:\n1. *Binance Coin*, *Bitcoin*, *Bitcoin Cash*, *Ethereum*, and *Ethereum Classic* share similar traded volume trends, as they all reached a new high around the begining of 2020, and also peaked when the traded volume of other coins peaked in January and May 2021.\n2. *Dogecoin*, *EOS.IO*, *IOTA*, *Maker*, *Litecoin*, and *Stellar* also share similar trends in that they all reached a new peak in January 2021, far higher than any peak before then.\n3. The above observations imply that Q1 2020, January 2021, and May 2021 are important dates we should pay attention to.\n\nWe study closing minute-wise prices next.","metadata":{}},{"cell_type":"code","source":"#Monthly Value/Trade\n    \nasset_names= [asset for asset in assets]\nindex=0\nrows=round(len(assets)/2)\ncolumns = 2\nrow_indices = range(rows)\ncolumn_indices = range(columns)\nfig, axes = plt.subplots(rows, columns, figsize=(20, 30))\nfor i in row_indices:\n    for j in column_indices:\n        temp = assets[asset_names[index]]\n        axes[i,j].plot( temp['Close'] ,\n                       linewidth=2, color=palette[index%10],label=asset_names[index])\n        axes[i,j].legend()\n        axes[i,j].set(ylabel='Closing Price($)', title =asset_names[index] )\n        index+=1\n        \nplt.suptitle('Minute-wise Asset Closing Price($)',fontsize='x-large')             \nfig.tight_layout(rect=[0, 0.03, 1, 0.98])        \n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:13:31.241397Z","iopub.execute_input":"2021-11-08T10:13:31.242044Z","iopub.status.idle":"2021-11-08T10:14:13.097713Z","shell.execute_reply.started":"2021-11-08T10:13:31.242002Z","shell.execute_reply":"2021-11-08T10:14:13.096770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learnings**\n\nWe observe the following:\n1. The first set of coins from the volume chart have a relatively smooth trend at the beginning of 2020. However in **January 2021 (say Q1,2021)**, the price of all coins really begins to take off with a similar trend across each coin. \n2. In Q2,2021, Paritculary in May, we have a strong bearish trend across crypto assets, which also corresponds to a spike in traded volume in the volume chart.\n3. There is a **cyclic** component as expected in a volatile time series trend. We zoom in on the final month of the Bitcoin timeseries to investigate further.\n","metadata":{}},{"cell_type":"code","source":"btc = assets['Bitcoin'].iloc[-50000:,:]\n\n\nfig = px.line(btc,x=btc.index, y='Close', color_discrete_sequence = ['#3366CC'], labels={'index':''},\n              title='Bitcoin Timeseires (Aug 17 - Sep 19 2021)')\nfig.update_layout(yaxis_tickprefix = '$', yaxis_tickformat = ',.0f')","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:14:13.099362Z","iopub.execute_input":"2021-11-08T10:14:13.099677Z","iopub.status.idle":"2021-11-08T10:14:14.879164Z","shell.execute_reply.started":"2021-11-08T10:14:13.099639Z","shell.execute_reply":"2021-11-08T10:14:14.878130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On zooming in, the trend appears very random. That's why predicting it will be some challenge 😉","metadata":{}},{"cell_type":"code","source":"#Monthly Volume Trends\n    \nasset_names= [asset for asset in assets]\nindex=0\nrows=round(len(assets)/2)\ncolumns = 2\nrow_indices = range(rows)\ncolumn_indices = range(columns)\nfig, axes = plt.subplots(rows, columns, figsize=(20, 30), sharey=True)\nfor i in row_indices:\n    for j in column_indices:\n        data = assets[asset_names[index]]\n        data['Value'] = data.Close*data.Volume\n        data = data[['Value','Count']].resample('MS').sum()\n        data['value_per_trade'] = data.Value/data.Count\n        axes[i,j].plot( data['value_per_trade'],linewidth=2, \n                       color=palette[index%10],label=asset_names[index])\n        axes[i,j].legend()\n        axes[i,j].set(ylabel='Avg Value / Trade ($)', title =asset_names[index] )\n        index+=1\n        \nplt.suptitle('Monthly Average Trade Value For Each Asset',fontsize='x-large')             \nfig.tight_layout(rect=[0, 0.03, 1, 0.98])        \n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:14:14.880615Z","iopub.execute_input":"2021-11-08T10:14:14.880907Z","iopub.status.idle":"2021-11-08T10:14:26.742375Z","shell.execute_reply.started":"2021-11-08T10:14:14.880865Z","shell.execute_reply":"2021-11-08T10:14:26.741436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learnings**\n\nWe observe the following:\n1. Even for the really expesive coins like *Bitcoin*, the highest average value/trade was just over $2000 in 2018, putting all assets in a similar range in terms of value/trade.\n2. Compared to the fluctuations in *Volume* and *Price*, fluctuations in value/trade are a lot more gentle, which implies a direct relationship between the value (product of *Price* and *Volume) and the number of trades, creating some stability.\n3. As the *Volume* and *Price* of the assets increases, we also have more trades happening (a higher *Count* representing more people hitting the exchanges). We can flip this around and say that more demand in an asset results in higer volumes being traded, and consequently creates an increase in price.\n4. *Etherum* *Cardano*, and *Dogecoin* are on an upward trajectory, which can imply the price of those assets is not purely due to an increased demand (it's due to increased demand but to a lesser extent than others). \n\n\n\n","metadata":{}},{"cell_type":"code","source":"#Monthly Value/Trade\n    \nasset_names= [asset for asset in assets]\nindex=0\nrows=round(len(assets)/2)\ncolumns = 2\nrow_indices = range(rows)\ncolumn_indices = range(columns)\nfig, axes = plt.subplots(rows, columns, figsize=(20, 30))\nfor i in row_indices:\n    for j in column_indices:\n        temp = assets[asset_names[index]]\n        axes[i,j].plot( temp['Target'] ,\n                       linewidth=2, color=palette[index%10],label=asset_names[index])\n        axes[i,j].legend()\n        axes[i,j].set(ylabel='Target', title =asset_names[index] )\n        index+=1\n        \nplt.suptitle('Minute-wise Asset Targets',fontsize='x-large')             \nfig.tight_layout(rect=[0, 0.03, 1, 0.98])        \n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-08T10:14:26.743866Z","iopub.execute_input":"2021-11-08T10:14:26.744168Z","iopub.status.idle":"2021-11-08T10:15:01.789765Z","shell.execute_reply.started":"2021-11-08T10:14:26.744131Z","shell.execute_reply":"2021-11-08T10:15:01.788713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learnings**\n\nWe observe the following:\n\n1. Contrary to what was mentioned in the tutorial about empty target observations, in the case of *Maker*, there were already some empty target observations, which only became amplifed when we added in the missing timestamps.\n","metadata":{}},{"cell_type":"markdown","source":"This notbook is still in progress, please check again for more updates.","metadata":{}}]}