{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto Forecasting\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/30894/logos/header.png?t=2021-09-14-17-32-48)","metadata":{}},{"cell_type":"markdown","source":"# Data Description\nThis dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum. Your challenge is to predict their future returns.\n\nAs historic cryptocurrency prices are not confidential this will be a forecasting competition using the time series API. Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset. Expect to see many people submitting perfect submissions for fun. Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code. The final private leaderboard will be determined using real market data gathered after the submission period closes.\n\n## Files\ntrain.csv - The training set\n\n>- timestamp - A timestamp for the minute covered by the row.\n>- Asset_ID - An ID code for the cryptoasset.\n>- Count - The number of trades that took place this minute.\n>- Open - The USD price at the beginning of the minute.\n>- High - The highest USD price during the minute.\n>- Low - The lowest USD price during the minute.\n>- Close - The USD price at the end of the minute.\n>- Volume - The number of cryptoasset units traded during the minute.\n>- VWAP - The volume weighted average price for the minute.\n>- Target - 15 minute residualized returns. See the 'Prediction and Evaluation' section of this notebook for details of how the target is calculated.\n\nexample_test.csv - An example of the data that will be delivered by the time series API.\n\nexample_sample_submission.csv - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n\nasset_details.csv - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n\ngresearch_crypto - An unoptimized version of the time series API files for offline work. You may need Python 3.7 and a Linux environment to run it without errors.\n\nsupplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\n## Time-series API Details\nRefer to the time series introduction notebook for an example of how to complete a submission. The time-series API has changed somewhat from previous competitions!\n\nExpect to see roughly three months worth of data in the test set.\n\nThe API will require 0.5 GB of memory after initialization. The initialization step (env.iter_test()) will require meaningfully more memory than that; we recommend you do not load your model until after making that call. The API will also consume less than 30 minutes of runtime for loading and serving the data.\n\nThe API loads the data using the following types: Asset_ID: int8, Count: int32, row_id: int32, Count: int32, Open: float64, High: float64, Low: float64, Close: float64, Volume: float64, VWAP: float64","metadata":{}},{"cell_type":"markdown","source":"# Notebooks referenced:\n>- <https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition>\n>- <https://www.kaggle.com/sohier/detailed-api-introduction>","metadata":{}},{"cell_type":"markdown","source":"# Import packages","metadata":{}},{"cell_type":"code","source":"# Install packages if required\n!pip install \"notebook>=5.3\" \"ipywidgets>=7.5\"","metadata":{"execution":{"iopub.status.busy":"2021-11-06T08:15:23.601650Z","iopub.execute_input":"2021-11-06T08:15:23.602518Z","iopub.status.idle":"2021-11-06T08:15:23.623807Z","shell.execute_reply.started":"2021-11-06T08:15:23.602422Z","shell.execute_reply":"2021-11-06T08:15:23.622839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom datetime import datetime\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom ipywidgets import interactive\nfrom datetime import timedelta","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:05:30.976687Z","iopub.execute_input":"2021-11-05T22:05:30.977207Z","iopub.status.idle":"2021-11-05T22:05:31.061301Z","shell.execute_reply.started":"2021-11-05T22:05:30.977160Z","shell.execute_reply":"2021-11-05T22:05:31.060416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load input files","metadata":{}},{"cell_type":"code","source":"crypto_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\nasset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nsample_submission = pd.read_csv('../input/g-research-crypto-forecasting/example_sample_submission.csv')\ntest_df = pd.read_csv('../input/g-research-crypto-forecasting/example_test.csv')\nsupplemental_train_df = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:05:31.062611Z","iopub.execute_input":"2021-11-05T22:05:31.062984Z","iopub.status.idle":"2021-11-05T22:06:38.282844Z","shell.execute_reply.started":"2021-11-05T22:05:31.062951Z","shell.execute_reply":"2021-11-05T22:06:38.281833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"markdown","source":"## Looking at the data","metadata":{}},{"cell_type":"code","source":"print(crypto_df.shape)\ncrypto_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:38.284805Z","iopub.execute_input":"2021-11-05T22:06:38.285018Z","iopub.status.idle":"2021-11-05T22:06:38.314588Z","shell.execute_reply.started":"2021-11-05T22:06:38.284992Z","shell.execute_reply":"2021-11-05T22:06:38.313778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking level of data - asset and timestamp level\ncrypto_df[['timestamp','Asset_ID']].drop_duplicates().shape[0]==crypto_df.shape[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:38.315843Z","iopub.execute_input":"2021-11-05T22:06:38.316229Z","iopub.status.idle":"2021-11-05T22:06:41.632103Z","shell.execute_reply.started":"2021-11-05T22:06:38.316180Z","shell.execute_reply":"2021-11-05T22:06:41.631280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(asset_details.shape)\nasset_details.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:41.633468Z","iopub.execute_input":"2021-11-05T22:06:41.633961Z","iopub.status.idle":"2021-11-05T22:06:41.645171Z","shell.execute_reply.started":"2021-11-05T22:06:41.633913Z","shell.execute_reply":"2021-11-05T22:06:41.644372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(crypto_df.shape)\ncrypto_df = crypto_df.merge(asset_details, how = 'left', on = 'Asset_ID')\nprint(crypto_df.shape)\ncrypto_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:41.646174Z","iopub.execute_input":"2021-11-05T22:06:41.646419Z","iopub.status.idle":"2021-11-05T22:06:44.973885Z","shell.execute_reply.started":"2021-11-05T22:06:41.646389Z","shell.execute_reply":"2021-11-05T22:06:44.973027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking nulls\ncrypto_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:44.975339Z","iopub.execute_input":"2021-11-05T22:06:44.975916Z","iopub.status.idle":"2021-11-05T22:06:48.071909Z","shell.execute_reply.started":"2021-11-05T22:06:44.975874Z","shell.execute_reply":"2021-11-05T22:06:48.070844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Target is the only column with nulls","metadata":{}},{"cell_type":"code","source":"crypto_df[crypto_df['Target'].isnull()].head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:48.073364Z","iopub.execute_input":"2021-11-05T22:06:48.073667Z","iopub.status.idle":"2021-11-05T22:06:49.852732Z","shell.execute_reply.started":"2021-11-05T22:06:48.073626Z","shell.execute_reply":"2021-11-05T22:06:49.851856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('# Assets:',crypto_df['Asset_ID'].nunique(), crypto_df['Asset_Name'].unique().tolist())","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:49.856335Z","iopub.execute_input":"2021-11-05T22:06:49.856721Z","iopub.status.idle":"2021-11-05T22:06:51.937645Z","shell.execute_reply.started":"2021-11-05T22:06:49.856689Z","shell.execute_reply":"2021-11-05T22:06:51.936700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df.describe()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:06:51.939216Z","iopub.execute_input":"2021-11-05T22:06:51.939711Z","iopub.status.idle":"2021-11-05T22:07:01.448397Z","shell.execute_reply.started":"2021-11-05T22:06:51.939669Z","shell.execute_reply":"2021-11-05T22:07:01.447490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sample_submission.shape)\nsample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.449604Z","iopub.execute_input":"2021-11-05T22:07:01.450230Z","iopub.status.idle":"2021-11-05T22:07:01.459963Z","shell.execute_reply.started":"2021-11-05T22:07:01.450186Z","shell.execute_reply":"2021-11-05T22:07:01.459389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission['group_num'].unique()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.460866Z","iopub.execute_input":"2021-11-05T22:07:01.461523Z","iopub.status.idle":"2021-11-05T22:07:01.471514Z","shell.execute_reply.started":"2021-11-05T22:07:01.461482Z","shell.execute_reply":"2021-11-05T22:07:01.470674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(test_df.shape)\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.472725Z","iopub.execute_input":"2021-11-05T22:07:01.473024Z","iopub.status.idle":"2021-11-05T22:07:01.493521Z","shell.execute_reply.started":"2021-11-05T22:07:01.472996Z","shell.execute_reply":"2021-11-05T22:07:01.492707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df[test_df['group_num']==1].head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.494667Z","iopub.execute_input":"2021-11-05T22:07:01.494878Z","iopub.status.idle":"2021-11-05T22:07:01.512038Z","shell.execute_reply.started":"2021-11-05T22:07:01.494852Z","shell.execute_reply":"2021-11-05T22:07:01.511428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Number of rows per asset\ntest_df.groupby(['Asset_ID']).size()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.513419Z","iopub.execute_input":"2021-11-05T22:07:01.513642Z","iopub.status.idle":"2021-11-05T22:07:01.522374Z","shell.execute_reply.started":"2021-11-05T22:07:01.513615Z","shell.execute_reply":"2021-11-05T22:07:01.521405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Number of assets per group\ntest_df.groupby(['group_num']).agg({'Asset_ID':['unique', 'nunique']}).reset_index()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.524016Z","iopub.execute_input":"2021-11-05T22:07:01.524912Z","iopub.status.idle":"2021-11-05T22:07:01.558146Z","shell.execute_reply.started":"2021-11-05T22:07:01.524841Z","shell.execute_reply":"2021-11-05T22:07:01.557448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(supplemental_train_df.shape)\nsupplemental_train_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.561535Z","iopub.execute_input":"2021-11-05T22:07:01.561790Z","iopub.status.idle":"2021-11-05T22:07:01.578678Z","shell.execute_reply.started":"2021-11-05T22:07:01.561757Z","shell.execute_reply":"2021-11-05T22:07:01.577994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"supplemental_train_df[supplemental_train_df['timestamp']==1623542400]","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.579912Z","iopub.execute_input":"2021-11-05T22:07:01.580960Z","iopub.status.idle":"2021-11-05T22:07:01.609167Z","shell.execute_reply.started":"2021-11-05T22:07:01.580921Z","shell.execute_reply":"2021-11-05T22:07:01.608227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df[crypto_df['timestamp']==1623542400]","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.610608Z","iopub.execute_input":"2021-11-05T22:07:01.611094Z","iopub.status.idle":"2021-11-05T22:07:01.654899Z","shell.execute_reply.started":"2021-11-05T22:07:01.611046Z","shell.execute_reply":"2021-11-05T22:07:01.653871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Candlestick chart","metadata":{}},{"cell_type":"code","source":"crypto_df['time'] = crypto_df['timestamp'].astype('datetime64[s]')","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:01.656542Z","iopub.execute_input":"2021-11-05T22:07:01.656929Z","iopub.status.idle":"2021-11-05T22:07:03.634840Z","shell.execute_reply.started":"2021-11-05T22:07:01.656881Z","shell.execute_reply":"2021-11-05T22:07:03.633830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def candlestick_chart(Asset_Name):\n    temp_data = crypto_df[crypto_df['Asset_Name'] == Asset_Name].reset_index(drop = True)\n    temp_data = temp_data.iloc[-500:]\n    fig = go.Figure(data=[go.Candlestick(x=temp_data.time, open=temp_data['Open'], high=temp_data['High'], low=temp_data['Low'], close=temp_data['Close'])])\n    fig.show()\n\nw = interactive(candlestick_chart, Asset_Name = crypto_df['Asset_Name'].unique())\ndisplay(w)","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:03.635970Z","iopub.execute_input":"2021-11-05T22:07:03.636179Z","iopub.status.idle":"2021-11-05T22:07:09.637915Z","shell.execute_reply.started":"2021-11-05T22:07:03.636154Z","shell.execute_reply":"2021-11-05T22:07:09.636999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Comparison of assets","metadata":{}},{"cell_type":"code","source":"asset_id_list = crypto_df[\"Asset_ID\"].unique().tolist()\n\nf = plt.figure(figsize=(15,30))\n\nfor i in range(0,14):\n    asset_id = asset_id_list[i]\n    btc = crypto_df[crypto_df[\"Asset_ID\"]==asset_id].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\n\n    beg_btc = btc.index[0].astype('datetime64[s]')\n    end_btc = btc.index[-1].astype('datetime64[s]')\n\n    print(btc['Asset_Name'].unique().tolist()[0] + ' data goes from ', beg_btc, 'to ', end_btc)\n\n    print('\\n--------------Checking timegaps start')\n    display((btc.index[1:]-btc.index[:-1]).value_counts().head())\n\n    btc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad') # ffill\n\n    display((btc.index[1:]-btc.index[:-1]).value_counts().head())\n    print('\\n--------------Checking timegaps end')\n\n    ax = f.add_subplot(7,2,i+1)\n    \n    ax.plot(btc['time'], btc['Close'], label=btc['Asset_Name'].unique().tolist()[0])\n    plt.legend()\n    plt.xlabel('Time')\n    plt.ylabel(btc['Asset_Name'].unique().tolist()[0])\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-05T22:07:09.639183Z","iopub.execute_input":"2021-11-05T22:07:09.639525Z","iopub.status.idle":"2021-11-05T22:07:53.286071Z","shell.execute_reply.started":"2021-11-05T22:07:09.639482Z","shell.execute_reply":"2021-11-05T22:07:53.285087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modelling","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predictions","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}