{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto forecasting competition","metadata":{"papermill":{"duration":0.022176,"end_time":"2022-01-03T13:27:50.797544","exception":false,"start_time":"2022-01-03T13:27:50.775368","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### コンペ概要\n\n目的：暗号通貨によるリターンの予測<br>\n暗号通貨に関する時系列の価格履歴を使用して、価格が上がるか下がるか、どれだけの資産がリターンするかを予測します。<br>\n本コンペでは、14種の暗号通貨についてのデータが提供されています。<br>\n\n> 暗号通貨市場の拡大：CrypoCompare社によると、2020年における一日の平均取引量は410億ドルにも及ぶとのこと。 - [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n\nデータの特性の概要\n- 時系列データ。\n- 暗号通貨の変動は、他の暗号通貨及び暗号通貨市場全体に対して大いに影響する。\n- 株価と同様に、通貨の価格はランダムウォークパターンに従っているようにも見える。","metadata":{"tags":[]}},{"cell_type":"code","source":"import pandas as pd\npd.set_option(\"display.max_columns\", None)\nimport numpy as np\nfrom datetime import datetime\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\n\nfrom plotly.subplots import make_subplots\nfrom statsmodels.graphics.tsaplots import plot_acf\nfrom statsmodels.graphics.tsaplots import plot_pacf","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":1.526291,"end_time":"2022-01-03T13:27:52.390995","exception":false,"start_time":"2022-01-03T13:27:50.864704","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:44:55.529512Z","iopub.execute_input":"2022-01-17T14:44:55.530282Z","iopub.status.idle":"2022-01-17T14:44:57.736912Z","shell.execute_reply.started":"2022-01-17T14:44:55.530178Z","shell.execute_reply":"2022-01-17T14:44:57.736054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### データ概要確認","metadata":{}},{"cell_type":"code","source":"# 学習用データの確認\ndata = pd.read_csv('../input/g-research-crypto-forecasting/train.csv') \nprint('データ数：',data.shape[0])\ndata.head() ","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:44:57.738803Z","iopub.execute_input":"2022-01-17T14:44:57.739303Z","iopub.status.idle":"2022-01-17T14:45:50.564215Z","shell.execute_reply.started":"2022-01-17T14:44:57.739255Z","shell.execute_reply":"2022-01-17T14:45:50.563335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **timestamp**：1970-01-01 00:00:00.000 UTCからの経過秒数を示す。データは分スケールのため、60の倍数になっているようです。\n- **Asset_ID**：暗号通貨を識別するID(例：ビットコインの場合はAsset_ID = 1)。 Asset_IDと暗号通貨の対応関係はasset_details.csvで参照できます。\n- **Count**：時間間隔（直前）の取引の総数。\n- **Open**：指定時間間隔における始値(USD)。\n- **High**：指定時間間隔における最高価格(USD)。\n- **Low**：指定時間間隔における最低価格(USD)。\n- **Close**：指定時間間隔における終値(USD)。\n- **Volume**：購入または販売された通貨の数量。\n- **VWAP**：指定時間間隔における通貨の重みづけ平均価格。\n- **Target**：15分間の対数収益率から算出。\ntimestamp、Asset_IDの複合キーが、データを識別するインデックになります。","metadata":{"papermill":{"duration":0.021851,"end_time":"2022-01-03T13:28:55.815829","exception":false,"start_time":"2022-01-03T13:28:55.793978","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Asset_IDと暗号通貨の対応<br>\nWeightは、目的変数となっているLogReturnを算出する際に使用します。","metadata":{}},{"cell_type":"code","source":"asset_tmp = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nasset = asset_tmp.set_index('Asset_ID').sort_index()\nasset","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:50.565707Z","iopub.execute_input":"2022-01-17T14:45:50.566111Z","iopub.status.idle":"2022-01-17T14:45:50.593428Z","shell.execute_reply.started":"2022-01-17T14:45:50.566059Z","shell.execute_reply":"2022-01-17T14:45:50.592559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 欠損値の確認\nこう見ると、ほとんどTargetの欠損だけのようです。","metadata":{"papermill":{"duration":0.02318,"end_time":"2022-01-03T13:28:55.936247","exception":false,"start_time":"2022-01-03T13:28:55.913067","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(data.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:50.595293Z","iopub.execute_input":"2022-01-17T14:45:50.596192Z","iopub.status.idle":"2022-01-17T14:45:51.069114Z","shell.execute_reply.started":"2022-01-17T14:45:50.596150Z","shell.execute_reply":"2022-01-17T14:45:51.068447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n通貨毎で欠損を確認してみます。<br>\nチュートリアルによると、データ欠損は行ごと抜け落ちているパターンも存在するとのことでしたので、行ごとの欠損数も可視化します。<br>\nビットコイン、イーサリアム、ライトコインの欠損値は比較的少ないようです。","metadata":{}},{"cell_type":"code","source":"missing_list = []\nfor asset_id, coin in zip(asset.index, asset.loc[:, 'Asset_Name']):\n    asset_df = data[data[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n    gap_df = (asset_df.index[1:]-asset_df.index[:-1]).value_counts().reset_index()\n    drop_row = int(((gap_df['index']/60-1)*gap_df['timestamp']).sum())\n    missing_list.append([coin, asset_df.shape[0]] + asset_df.isna().sum().tolist() + [drop_row])\n\nmissing_df = pd.DataFrame(missing_list)\nmissing_df.columns = [\"Asset_Name\", \"TotalRows\", 'Missing_Asset_ID', 'Missing_Count', 'Missing_Open', \n                      'Missing_High', 'Missing_Low', 'Missing_Close', 'Missing_Volume', 'Missing_VWAP', 'Missing_Target', 'Missing_Row']\nmissing_df","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:51.070423Z","iopub.execute_input":"2022-01-17T14:45:51.070643Z","iopub.status.idle":"2022-01-17T14:45:55.943249Z","shell.execute_reply.started":"2022-01-17T14:45:51.070618Z","shell.execute_reply":"2022-01-17T14:45:55.942362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"各暗号通貨のについて提供されているデータサイズ","metadata":{"papermill":{"duration":0.023238,"end_time":"2022-01-03T13:28:56.728303","exception":false,"start_time":"2022-01-03T13:28:56.705065","status":"completed"},"tags":[]}},{"cell_type":"code","source":"asset_count= []\nfor i in range(14):\n    count = (data[\"Asset_ID\"]==i).sum()\n    asset_count.append(count)\nfig = px.bar(x = asset[\"Asset_Name\"],\n             y = asset_count , \n             color = asset_count ,\n             color_continuous_scale=\"Blues\") \nfig.update_xaxes(title=\"Assets\")\nfig.update_yaxes(title = \"Number of Rows\")\nfig.update_layout(showlegend = True,\n    title = {\n        'text': 'Data Distribution ',\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'} ,\n        template=\"plotly_white\")\nfig.show()","metadata":{"_kg_hide-input":true,"papermill":{"duration":1.583166,"end_time":"2022-01-03T13:28:58.335299","exception":false,"start_time":"2022-01-03T13:28:56.752133","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:45:55.944517Z","iopub.execute_input":"2022-01-17T14:45:55.944752Z","iopub.status.idle":"2022-01-17T14:45:56.616267Z","shell.execute_reply.started":"2022-01-17T14:45:55.944724Z","shell.execute_reply":"2022-01-17T14:45:56.614021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 通貨の価格変動\n株価チャートで価格変動の様子を確認してみます。plotly.graph_objectsのCandlestickで足付のチャートグラフが一発で作成できます。<br>\n時間内で価格が下がっている(終値が始値よりも低い)場合はプロットが赤、反対の場合は緑になります。","metadata":{"papermill":{"duration":0.024941,"end_time":"2022-01-03T13:28:58.385618","exception":false,"start_time":"2022-01-03T13:28:58.360677","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def crypto_df(asset_id ,data= data ):\n    df = data[data[\"Asset_ID\"]==asset_id].reset_index(drop = True)\n    df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df.set_index('timestamp')\n    return df\n\nbtc = crypto_df(asset_id = 1) #ビットコイン\neth = crypto_df(asset_id = 6) #イーサリアム","metadata":{"_kg_hide-input":true,"papermill":{"duration":1.320145,"end_time":"2022-01-03T13:28:59.730924","exception":false,"start_time":"2022-01-03T13:28:58.410779","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:45:56.617538Z","iopub.status.idle":"2022-01-17T14:45:56.618056Z","shell.execute_reply.started":"2022-01-17T14:45:56.617773Z","shell.execute_reply":"2022-01-17T14:45:56.617800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def candelstick_chart(data,title):\n    candlestick = go.Figure(data = [go.Candlestick(x =data.index, \n                                               open = data[('Open')], \n                                               high = data[('High')], \n                                               low = data[('Low')], \n                                               close = data[('Close')])])\n    candlestick.update_xaxes(title_text = 'Time',\n                             rangeslider_visible = True)\n\n    candlestick.update_layout(\n    title = {\n        'text': '{:} Candelstick Chart'.format(title),\n        'y':0.90,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'} , \n    template=\"plotly_white\")\n\n    candlestick.update_yaxes(title_text = 'Price in USD', ticksuffix = '$')\n    return candlestick","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.036331,"end_time":"2022-01-03T13:28:59.791835","exception":false,"start_time":"2022-01-03T13:28:59.755504","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:45:56.619941Z","iopub.status.idle":"2022-01-17T14:45:56.620249Z","shell.execute_reply.started":"2022-01-17T14:45:56.620098Z","shell.execute_reply":"2022-01-17T14:45:56.620115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ビットコイン ろうそく足チャート(ラスト50サンプル)","metadata":{"papermill":{"duration":0.024611,"end_time":"2022-01-03T13:28:59.841177","exception":false,"start_time":"2022-01-03T13:28:59.816566","status":"completed"},"tags":[]}},{"cell_type":"code","source":"btc_plot = candelstick_chart(btc[-50:],title = \"Bitcoin(BTC)\")\nbtc_plot.show()","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.087459,"end_time":"2022-01-03T13:28:59.953599","exception":false,"start_time":"2022-01-03T13:28:59.866140","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:45:56.621276Z","iopub.status.idle":"2022-01-17T14:45:56.621558Z","shell.execute_reply.started":"2022-01-17T14:45:56.621415Z","shell.execute_reply":"2022-01-17T14:45:56.621430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"OLCHチャートも簡単に書けますね。見やすい方で。","metadata":{"papermill":{"duration":0.030229,"end_time":"2022-01-03T13:29:00.240398","exception":false,"start_time":"2022-01-03T13:29:00.210169","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def ohlc_chart(data,title):\n    ohlc = go.Figure(data = [go.Ohlc(x =data.index, \n                                               open = data[('Open')], \n                                               high = data[('High')], \n                                               low = data[('Low')], \n                                               close = data[('Close')])])\n    ohlc.update_xaxes(title_text = 'Time',\n                             rangeslider_visible = True)\n\n    ohlc.update_layout(\n    title = {\n        'text': '{:} OHLC Chart'.format(title),\n        'y':0.9,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n        template=\"plotly_white\")\n\n    ohlc.update_yaxes(title_text = 'Price in USD', ticksuffix = '$')\n    return ohlc","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.061902,"end_time":"2022-01-03T13:29:00.338228","exception":false,"start_time":"2022-01-03T13:29:00.276326","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:45:56.622568Z","iopub.status.idle":"2022-01-17T14:45:56.622867Z","shell.execute_reply.started":"2022-01-17T14:45:56.622699Z","shell.execute_reply":"2022-01-17T14:45:56.622715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ビットコイン OHLCチャート(50サンプル)","metadata":{"papermill":{"duration":0.030033,"end_time":"2022-01-03T13:29:00.409568","exception":false,"start_time":"2022-01-03T13:29:00.379535","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ohlc_chart(btc[-50:], title = \"Bitcoin(BTC)\")","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.110142,"end_time":"2022-01-03T13:29:00.550290","exception":false,"start_time":"2022-01-03T13:29:00.440148","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-17T14:45:56.623874Z","iopub.status.idle":"2022-01-17T14:45:56.624164Z","shell.execute_reply.started":"2022-01-17T14:45:56.624008Z","shell.execute_reply":"2022-01-17T14:45:56.624024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"データを日単位でグルーピングして、時系列全体での変動を見てみます。<br>\n参考：[to-the-moon-g-research-crypto-forecasting-eda](https://www.kaggle.com/iamleonie/to-the-moon-g-research-crypto-forecasting-eda)","metadata":{}},{"cell_type":"code","source":"# タイムスタンプの変換\ndata['timestamp'] = data['timestamp'].astype('datetime64[s]')\n\n# グルーピング\ndata_daily = pd.DataFrame()\n\nfor asset_id in asset.index:\n    data_single = data[data.Asset_ID == asset_id].copy()\n\n    data_single_new = data_single[['timestamp','Count']].resample('D', on='timestamp').sum()\n    data_single_new['Open'] = data_single[['timestamp','Open']].resample('D', on='timestamp').first()['Open']\n    data_single_new['High'] = data_single[['timestamp','High']].resample('D', on='timestamp').max()['High']\n    data_single_new['Low'] = data_single[['timestamp','Low']].resample('D', on='timestamp').min()['Low']\n    data_single_new['Close'] = data_single[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\n    data_single_new['Volume'] = data_single[['timestamp','Volume']].resample('D', on='timestamp').sum()['Volume']\n    data_single_new['Asset_ID'] = asset_id\n\n    data_daily = data_daily.append(data_single_new.reset_index(drop=False))\ndata_daily = data_daily.sort_values(by = ['timestamp', 'Asset_ID']).reset_index(drop=True)\n\ndata_daily = data_daily.pivot(index='timestamp', columns='Asset_ID')[['Count', 'Open', 'High', 'Low', 'Close', 'Volume']]\ndata_daily = data_daily.reset_index(drop=False)","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:56.625026Z","iopub.status.idle":"2022-01-17T14:45:56.625309Z","shell.execute_reply.started":"2022-01-17T14:45:56.625164Z","shell.execute_reply":"2022-01-17T14:45:56.625179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ろうそく足チャート\nfig = make_subplots(\n    rows=len(asset.index), cols=1, subplot_titles=(asset.Asset_Name)\n)\n\nfor i, asset_id in enumerate(asset.index):\n    fig.append_trace(go.Candlestick(x=data_daily.timestamp, \n                                         open=data_daily[('Open', asset_id)], \n                                         high=data_daily[('High', asset_id)], \n                                         low=data_daily[('Low', asset_id)], \n                                         close=data_daily[('Close', asset_id)]),\n                                         row=i+1, col=1,\n                    )\n\n    fig.update_xaxes(range=[data_daily.timestamp.iloc[0], data_daily.timestamp.iloc[-1]], row=i+1, col=1)\n\nfig.update_layout(xaxis_rangeslider_visible = False, \n                  xaxis2_rangeslider_visible = False, \n                  xaxis3_rangeslider_visible = False,\n                  xaxis4_rangeslider_visible = False,\n                  xaxis5_rangeslider_visible = False,\n                  xaxis6_rangeslider_visible = False,\n                  xaxis7_rangeslider_visible = False,\n                  xaxis8_rangeslider_visible = False,\n                  xaxis9_rangeslider_visible = False,\n                  xaxis10_rangeslider_visible = False,\n                  xaxis11_rangeslider_visible = False,\n                  xaxis12_rangeslider_visible = False,\n                  xaxis13_rangeslider_visible = False,\n                  xaxis14_rangeslider_visible = False,\n                  height=3000, width=1000, \n                  margin = dict(l = 0, r = 0, b = 0, t = 30, pad = 0)\n                 )\n                 \nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:56.626225Z","iopub.status.idle":"2022-01-17T14:45:56.626503Z","shell.execute_reply.started":"2022-01-17T14:45:56.626355Z","shell.execute_reply":"2022-01-17T14:45:56.626370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### LogReturn\nこのコンペではTargetの算出に対数リターン(対数収益率)が使用されます。<br>\n通貨によって価格スケールが異なるので、価格の変化率を計算することで通貨の価格変動を分析します。","metadata":{}},{"cell_type":"code","source":"def log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:56.627590Z","iopub.status.idle":"2022-01-17T14:45:56.627891Z","shell.execute_reply.started":"2022-01-17T14:45:56.627723Z","shell.execute_reply":"2022-01-17T14:45:56.627738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc_lret = log_return(btc.Close)[1000000:1500000]\neth_lret = log_return(eth.Close)[1000000:1500000]","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:56.628665Z","iopub.status.idle":"2022-01-17T14:45:56.628950Z","shell.execute_reply.started":"2022-01-17T14:45:56.628790Z","shell.execute_reply":"2022-01-17T14:45:56.628805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ビットコイン、イーサネットのLogReturn\nfig = go.Figure()\nfig.add_trace(go.Scatter(x=btc_lret.index,\n                        y=btc_lret.values,\n                        #mode='line',\n                        showlegend=True,\n                        name='Bitcoin'))\nfig.add_trace(go.Scatter(x=eth_lret.index,\n                        y=eth_lret.values,\n                        #mode='line',\n                        showlegend=True,\n                        name='Ethereum'))\n\nfig.update_xaxes(title_text='Time')\nfig.update_yaxes(title_text='Log Return')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:56.629951Z","iopub.status.idle":"2022-01-17T14:45:56.630245Z","shell.execute_reply.started":"2022-01-17T14:45:56.630096Z","shell.execute_reply":"2022-01-17T14:45:56.630112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2020年3月頃COVID-19の影響とみられる変動が確認できます。<br>\n日単位のデータについてもLogReturnを可視化してみます。<br>","metadata":{}},{"cell_type":"code","source":"# 全通貨LogReturn可視化\nfor i, asset_id in enumerate(asset.index):\n    data_daily[('lret',  asset_id)] = log_return(data_daily[( 'Close',  asset_id)])\n\n    \nfig = make_subplots(\n    rows=len(asset.index), cols=1, subplot_titles=(asset.Asset_Name)\n)\nfor i, asset_id in enumerate(asset.index):\n    #df_coin = data_daily[data_daily.loc[:, 'Asset_ID']==asset_id]\n    fig.append_trace(go.Scatter(\n                        x=data_daily.timestamp,\n                        y=data_daily[('lret', asset_id)]),\n                        row=i+1, col=1)\n\n    fig.update_xaxes(range=[data_daily.timestamp.iloc[0], data_daily.timestamp.iloc[-1]], row=i+1, col=1)\n                 \nfig.update_layout(\n                  height=3000, width=1000, \n                  margin = dict(l = 0, r = 0, b = 0, t = 30, pad = 0)\n                 )\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-17T14:45:56.631051Z","iopub.status.idle":"2022-01-17T14:45:56.631363Z","shell.execute_reply.started":"2022-01-17T14:45:56.631189Z","shell.execute_reply":"2022-01-17T14:45:56.631203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}