{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Crypto forecasting tutorial**","metadata":{"id":"8PNZS8gF3CVN"}},{"cell_type":"markdown","source":"# G-Research Crypto 預測比賽\n\n在 [G-Research Crypto 預測競賽]（鏈接）中，參與者面臨著預測一系列主要加密貨幣的價格回報的挑戰。 為了方便您參與，我們創建了本教程筆記本，涵蓋了加密預測挑戰的一些相關概念。\n\n該筆記本介紹了加密預測，描述了數據集的結構和元素、一些相關的統計屬性，以及構建了幾個 ML 基線模型並提供了示例代碼提交。\n\n","metadata":{"id":"DO0FyA3q6rTl"}},{"cell_type":"markdown","source":"## 加密貨幣市場\n\n首先，快速介紹加密世界。 加密貨幣已經成為一個非常受歡迎和動蕩的市場，為投資者帶來巨額回報（以及損失）。 已經創建了數以千計的加密貨幣，其中包括比特幣 (BTC)、以太幣 (ETH) 或狗狗幣 (DOGE) 在內的一些主要加密貨幣，你們中的許多人都聽說過。\n\n根據 CryptoCompare（截至 2021 年 7 月 25 日）的數據，加密貨幣在加密貨幣交易所廣泛交易，去年平均每天交易量為 410 億美元。\n\n不同加密貨幣之間的價格變化是高度相互關聯的。 例如，比特幣歷來是加密貨幣價格變化的主要驅動力，但其他代幣也會影響市場。\n## 加密貨幣市場\n\n首先，快速介紹加密世界。 加密貨幣已經成為一個非常受歡迎和動蕩的市場，為投資者帶來巨額回報（以及損失）。 已經創建了數以千計的加密貨幣，其中包括比特幣 (BTC)、以太幣 (ETH) 或狗狗幣 (DOGE) 在內的一些主要加密貨幣，你們中的許多人都聽說過。\n\n根據 CryptoCompare（截至 2021 年 7 月 25 日）的數據，加密貨幣在加密貨幣交易所廣泛交易，去年平均每天交易量為 410 億美元。\n\n不同加密貨幣之間的價格變化是高度相互關聯的。 例如，比特幣歷來是加密貨幣價格變化的主要驅動力，但其他代幣也會影響市場。","metadata":{"id":"x3HpXt5v-ko4"}},{"cell_type":"markdown","source":"\n## 預測回報\n\n金融建模的一項基本任務是預測價格在不久的將來會如何表現。使用歷史價格的時間序列作為訓練數據，我們想要預測價格會上漲還是下跌，以及上漲多少，即資產*回報*。\n\n在本次比賽中，Kagglers 的挑戰是建立機器學習模型來預測 14 種流行加密貨幣的回報，時間範圍從幾分鐘到幾小時。您將可以訪問數百萬行每分鐘的加密貨幣交易數據，您將使用這些數據同時為所有 14 種資產設計預測模型。您的預測將根據它們與比賽結束後三個月評估期內收集的真實市場數據的相關程度進行評估。\n\n加密貨幣收益預測仍然是一項開放且極具挑戰性的預測任務。鑑於資產的極端波動性、數據的非平穩性、市場和 meme 操縱、資產之間的相關性和快速變化的市場條件，這對於 ML 社區來說是一個引人入勝的問題領域。我們希望您和我們一樣發現它令人著迷！\n","metadata":{"id":"fDAWC9JNZITK"}},{"cell_type":"markdown","source":"# 數據集描述\n​​​\n現在，讓我們深入研究數據！ 我們首先加載比賽的數據集並檢查其基本屬性。","metadata":{"id":"4e6OA0qc6x9h"}},{"cell_type":"markdown","source":"## 加載訓練集","metadata":{"id":"OevdFjmtar0j"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime","metadata":{"id":"90fqnpNQ2wd8","execution":{"iopub.status.busy":"2022-05-25T06:37:23.355533Z","iopub.execute_input":"2022-05-25T06:37:23.356384Z","iopub.status.idle":"2022-05-25T06:37:23.383134Z","shell.execute_reply.started":"2022-05-25T06:37:23.356222Z","shell.execute_reply":"2022-05-25T06:37:23.382403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_folder = \"../input/g-research-crypto-forecasting/\"\n!ls $data_folder","metadata":{"id":"Zw_oi4NETkKv","outputId":"e6fe292d-721e-4c6a-bc70-9026cfd5b915","execution":{"iopub.status.busy":"2022-05-25T06:37:23.431180Z","iopub.execute_input":"2022-05-25T06:37:23.431789Z","iopub.status.idle":"2022-05-25T06:37:24.228653Z","shell.execute_reply.started":"2022-05-25T06:37:23.431735Z","shell.execute_reply":"2022-05-25T06:37:24.226847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = pd.read_csv(data_folder + 'train.csv')","metadata":{"id":"La0-ivRAUopA","scrolled":true,"execution":{"iopub.status.busy":"2022-05-25T06:37:24.231222Z","iopub.execute_input":"2022-05-25T06:37:24.231656Z","iopub.status.idle":"2022-05-25T06:38:32.379339Z","shell.execute_reply.started":"2022-05-25T06:37:24.231609Z","shell.execute_reply":"2022-05-25T06:38:32.377401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df.head(10)","metadata":{"id":"14ZSPE6daadW","outputId":"b6a4f262-4071-4c2f-dd42-12a253452297","execution":{"iopub.status.busy":"2022-05-25T06:38:32.382639Z","iopub.execute_input":"2022-05-25T06:38:32.383115Z","iopub.status.idle":"2022-05-25T06:38:32.427831Z","shell.execute_reply.started":"2022-05-25T06:38:32.383064Z","shell.execute_reply":"2022-05-25T06:38:32.426756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that each row of the data set has the trading data for an asset, at a given minute timestamp, described in detail below. ","metadata":{"id":"ZUPxDsW8bCh1"}},{"cell_type":"markdown","source":"## 數據特徵\n我們可以看到數據集中包含的不同特徵。具體來說，每個資產包含的功能如下：\n* **timestamp**：所有時間戳都作為第二個 Unix 時間戳返回（自 1970-01-01 00:00:00.000 UTC 以來經過的秒數）。此數據集中的時間戳是 60 的倍數，表示每分鐘的數據。\n* **Asset_ID**：對應於其中一種加密貨幣的資產 ID（例如，`Asset_ID = 1` 用於比特幣）。從“Asset_ID”到加密資產的映射包含在“asset_details.csv”中。\n* **Count**：時間間隔內的交易總數（最後一分鐘）。\n* **Open**：時間間隔的開盤價（美元）。\n* **High**：在時間間隔內達到的最高價格（以美元計）。\n* **Low**：在時間間隔內達到的最低價格（以美元計）。\n* **Close**：時間間隔的收盤價（美元）。\n* **Volume**：買入或賣出的資產數量，以基礎貨幣美元顯示。\n* **VWAP**：資產在時間間隔內的平均價格，按交易量加權。 VWAP 是貿易數據的聚合形式。\n* **Target**：資產在 15 分鐘內的剩餘對數回報。\n\n前兩列定義此數據行的時間和資產索引。中間的 6 列是特徵列，其中包含該資產的交易數據和分鐘。最後一列是預測目標，我們稍後會更詳細地介紹。\n\n我們還查看資產信息，包括所有資產的列表、“Asset_ID”到資產的映射，以及用於衡量其在評估指標中的相對重要性的每個資產的權重​​。","metadata":{"id":"YjN-TfBUx2ml"}},{"cell_type":"code","source":"asset_details = pd.read_csv(data_folder + 'asset_details.csv')\nasset_details","metadata":{"execution":{"iopub.status.busy":"2022-05-25T06:38:32.430001Z","iopub.execute_input":"2022-05-25T06:38:32.430348Z","iopub.status.idle":"2022-05-25T06:38:32.453469Z","shell.execute_reply.started":"2022-05-25T06:38:32.430314Z","shell.execute_reply":"2022-05-25T06:38:32.452806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##燭台圖\n\n交易數據格式是市場數據的聚合形式，包括開盤價、最高價、最低價和收盤價。 我們可以通過常用的燭台條形圖將這些數據可視化，允許交易者對盤中值進行技術分析。 柱體長度代表當天交易的開盤價和收盤價之間的價格範圍。 當柱為紅色時，表示收盤價低於開盤價，否則為綠色。 這些也被稱為看漲和看跌燭台。 條形圖上方和下方的燈芯顯示該區間交易的最高價和最低價。","metadata":{"id":"QNWAfs_O0hyf"}},{"cell_type":"markdown","source":"我們可以使用 `plotly` 庫可視化比特幣價格的一部分。 圖的底部顯示了一個範圍滑塊，您可以使用它來放大圖。","metadata":{"id":"570DeXQa0skY"}},{"cell_type":"code","source":"btc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\nbtc_mini = btc.iloc[-200:] # Select recent data rows","metadata":{"id":"tMxYx8ZO3fuy","execution":{"iopub.status.busy":"2022-05-25T06:38:32.454546Z","iopub.execute_input":"2022-05-25T06:38:32.455171Z","iopub.status.idle":"2022-05-25T06:38:32.887873Z","shell.execute_reply.started":"2022-05-25T06:38:32.455132Z","shell.execute_reply":"2022-05-25T06:38:32.886915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\n\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"id":"HCiq4HwD4ySs","outputId":"1d422d61-2382-41f0-ba2d-ca42946ae928","execution":{"iopub.status.busy":"2022-05-25T06:38:32.889213Z","iopub.execute_input":"2022-05-25T06:38:32.889467Z","iopub.status.idle":"2022-05-25T06:38:33.055760Z","shell.execute_reply.started":"2022-05-25T06:38:32.889436Z","shell.execute_reply":"2022-05-25T06:38:33.054764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 預處理","metadata":{"id":"S-oceNYiivUi"}},{"cell_type":"markdown","source":"## 處理缺失數據\n","metadata":{"id":"K9ce7eWddHfM"}},{"cell_type":"markdown","source":"讓我們檢查另一個重要資產以太坊的數據。","metadata":{"id":"AOpyJzKgFuYb"}},{"cell_type":"code","source":"eth = crypto_df[crypto_df[\"Asset_ID\"]==6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\neth.info(show_counts =True)","metadata":{"id":"cuWT1Suxzjvb","outputId":"04ff726c-8858-4d87-b592-ee1b036f0996","execution":{"iopub.status.busy":"2022-05-25T06:38:33.057545Z","iopub.execute_input":"2022-05-25T06:38:33.058265Z","iopub.status.idle":"2022-05-25T06:38:33.470243Z","shell.execute_reply.started":"2022-05-25T06:38:33.058214Z","shell.execute_reply":"2022-05-25T06:38:33.469113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"我們可以看到訓練集中的行數，以及目標列的缺失值，我們將在稍後解決。 讓我們確認一下：\n","metadata":{"id":"3qryFWzMgXgo"}},{"cell_type":"code","source":"eth.isna().sum()","metadata":{"id":"0hVTUf1LflPM","outputId":"4c3c1993-1594-44c2-8457-52e94c199a5f","execution":{"iopub.status.busy":"2022-05-25T06:38:33.471868Z","iopub.execute_input":"2022-05-25T06:38:33.472220Z","iopub.status.idle":"2022-05-25T06:38:33.518044Z","shell.execute_reply.started":"2022-05-25T06:38:33.472175Z","shell.execute_reply":"2022-05-25T06:38:33.516960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"讓我們使用從時間戳到“日期時間”的轉換來檢查比特幣和以太坊數據的`datetime`。.","metadata":{"id":"sJ_CF60EfpQS"}},{"cell_type":"code","source":"btc.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T06:38:33.519418Z","iopub.execute_input":"2022-05-25T06:38:33.519676Z","iopub.status.idle":"2022-05-25T06:38:33.538369Z","shell.execute_reply.started":"2022-05-25T06:38:33.519647Z","shell.execute_reply":"2022-05-25T06:38:33.537311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"beg_btc = btc.index[0].astype('datetime64[s]')\nend_btc = btc.index[-1].astype('datetime64[s]')\nbeg_eth = eth.index[0].astype('datetime64[s]')\nend_eth = eth.index[-1].astype('datetime64[s]')\n\nprint('BTC data goes from ', beg_btc, 'to ', end_btc)\nprint('Ethereum data goes from ', beg_eth, 'to ', end_eth)","metadata":{"id":"YrVaTbdtZ6Qv","outputId":"c640691c-075f-4573-e3a4-ad24ae468f72","execution":{"iopub.status.busy":"2022-05-25T06:38:33.541826Z","iopub.execute_input":"2022-05-25T06:38:33.542699Z","iopub.status.idle":"2022-05-25T06:38:33.555387Z","shell.execute_reply.started":"2022-05-25T06:38:33.542639Z","shell.execute_reply":"2022-05-25T06:38:33.554169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"對於給定的一分鐘，缺少的資產數據不是由 NaN 表示，而是由這些行的缺失表示。 我們可以檢查連續行之間的時間戳差異，看看是否有缺失數據。","metadata":{"id":"4PIgY9dtZ7BX"}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"id":"KOrD8uIked2c","outputId":"51d9c340-a88c-4476-fdca-cffa8edc5c39","execution":{"iopub.status.busy":"2022-05-25T06:38:33.556756Z","iopub.execute_input":"2022-05-25T06:38:33.557035Z","iopub.status.idle":"2022-05-25T06:38:33.589550Z","shell.execute_reply.started":"2022-05-25T06:38:33.557003Z","shell.execute_reply":"2022-05-25T06:38:33.588790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"請注意，數據中有許多空白。 要使用大多數時間序列模型，我們應該將數據預處理為沒有時間間隔的格式。 為了填補空白，我們可以使用 .reindex() 方法進行前向填充，用之前的有效值填補空白。","metadata":{"id":"GFLjIGONhN9s"}},{"cell_type":"code","source":"eth = eth.reindex(range(eth.index[0],eth.index[-1]+60,60),method='pad')","metadata":{"id":"8h29S-1Hhp3Y","execution":{"iopub.status.busy":"2022-05-25T06:38:33.590786Z","iopub.execute_input":"2022-05-25T06:38:33.591219Z","iopub.status.idle":"2022-05-25T06:38:34.038609Z","shell.execute_reply.started":"2022-05-25T06:38:33.591188Z","shell.execute_reply":"2022-05-25T06:38:34.037564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"並檢查現在沒有時間間隔。","metadata":{"id":"4sX6uvcQiCC5"}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"id":"UDUygkngiBKT","outputId":"c9f12e5c-f07d-4063-c35e-2eec6d692438","execution":{"iopub.status.busy":"2022-05-25T06:38:34.039827Z","iopub.execute_input":"2022-05-25T06:38:34.040084Z","iopub.status.idle":"2022-05-25T06:38:34.088302Z","shell.execute_reply.started":"2022-05-25T06:38:34.040055Z","shell.execute_reply":"2022-05-25T06:38:34.087367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 數據可視化\n​​​\n我們將從可視化我們選擇的兩種資產的收盤價開始。","metadata":{"id":"_TcwTqisePHB"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# 繪製兩個選定資產的 vwap 時間序列\nf = plt.figure(figsize=(15,4))\n\n# 填充 BTC 的缺失值\nbtc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\n\nax = f.add_subplot(121)\nplt.plot(btc['Close'], label='BTC')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin')\n\nax2 = f.add_subplot(122)\nax2.plot(eth['Close'], color='red', label='ETH')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum')\n\nplt.tight_layout()\nplt.show()","metadata":{"id":"PtVHC0YTxq-l","outputId":"525bc0d9-9673-42d0-ab73-aadc5e6afdad","execution":{"iopub.status.busy":"2022-05-25T06:38:34.089670Z","iopub.execute_input":"2022-05-25T06:38:34.090098Z","iopub.status.idle":"2022-05-25T06:38:37.564585Z","shell.execute_reply.started":"2022-05-25T06:38:34.090058Z","shell.execute_reply":"2022-05-25T06:38:37.563096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"這些資產的歷史完全不同，但我們可以檢查它們最近是否相關。","metadata":{"id":"ECfuzVHyzEF2"}},{"cell_type":"code","source":"import time\n\n# auxiliary function, from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n\n# create intervals\nbtc_mini_2021 = btc.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]\neth_mini_2021 = eth.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]","metadata":{"id":"mcfpDT6vNzpT","execution":{"iopub.status.busy":"2022-05-25T06:38:37.566546Z","iopub.execute_input":"2022-05-25T06:38:37.566867Z","iopub.status.idle":"2022-05-25T06:38:37.575005Z","shell.execute_reply.started":"2022-05-25T06:38:37.566835Z","shell.execute_reply":"2022-05-25T06:38:37.573608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot time series for both chosen assets\nf = plt.figure(figsize=(7,8))\n\nax = f.add_subplot(211)\nplt.plot(btc_mini_2021['Close'], label='btc')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin Close')\n\nax2 = f.add_subplot(212)\nax2.plot(eth_mini_2021['Close'], color='red', label='eth')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum Close')\n\nplt.tight_layout()\nplt.show()","metadata":{"id":"EfZiXb74zQsK","outputId":"5039e885-cb14-487f-fe06-3a30e4cf7270","execution":{"iopub.status.busy":"2022-05-25T06:38:37.576347Z","iopub.execute_input":"2022-05-25T06:38:37.577112Z","iopub.status.idle":"2022-05-25T06:38:38.138648Z","shell.execute_reply.started":"2022-05-25T06:38:37.577063Z","shell.execute_reply":"2022-05-25T06:38:38.137625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On shorter intervals we can visually see some potential correlation between both assets, with some simultaneous ups and downs. A better format for analyzing such movements is by calculating asset returns. ","metadata":{"id":"O2-jEU3wPnNV"}},{"cell_type":"markdown","source":"## 日誌返回\n​​​\n為了分析資產的價格變化，我們可以處理價格差異。 然而，不同的資產表現出不同的價格尺度，因此它們的回報不易比較。 我們可以通過計算價格的百分比變化來解決這個問題，也稱為回報。 這一回報與我們投資資本的百分比變化相吻合。\n​​​\n回報在金融中被廣泛使用，但是對數回報更適合時間序列的數學建模，因為它們是隨時間相加的。 此外，雖然常規回報率不能低於 -100%，但對數回報率不受限制。\n​​​\n為了計算對數回報，我們可以簡單地取兩個連續價格之間比率的對數。 第一行將有一個空返回，因為先前的值是未知的，因此空返回數據點將被刪除。## Log returns\n\nIn order to analyze price changes for an asset we can deal with the price difference. However, different assets exhibit different price scales, so that the their returns are not readily comparable. We can solve this problem by computing the percentage change in price instead, also known as the return. This return coincides with the percentage change in our invested capital.\n\nReturns are widely used in finance, however log returns are preferred for mathematical modelling of time series, as they are additive across time. Also, while regular returns cannot go below -100%, log returns are not bounded.\n\nTo compute the log return, we can simply take the logarithm of the ratio between two consecutive prices. The first row will have an empty return as the previous value is unknown, therefore the empty return data point will be dropped.","metadata":{"id":"6WpWqQ-EReg2"}},{"cell_type":"code","source":"# define function to compute log returns\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"id":"DBg185h-SOwI","execution":{"iopub.status.busy":"2022-05-25T06:38:38.140189Z","iopub.execute_input":"2022-05-25T06:38:38.140470Z","iopub.status.idle":"2022-05-25T06:38:38.145266Z","shell.execute_reply.started":"2022-05-25T06:38:38.140437Z","shell.execute_reply":"2022-05-25T06:38:38.144078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"我們可以可視化我們的兩個資產的日誌回報。 看看信號現在看起來更像白噪聲，與價格的時間序列相比，漂移更小。","metadata":{"id":"pdAR2fjIrCge"}},{"cell_type":"code","source":"import scipy.stats as stats\n\nlret_btc = log_return(btc_mini_2021.Close)[1:]\nlret_eth = log_return(eth_mini_2021.Close)[1:]\nlret_btc.rename('lret_btc', inplace=True)\nlret_eth.rename('lret_eth', inplace=True)\n\nplt.figure(figsize=(8,4))\nplt.plot(lret_btc);\nplt.plot(lret_eth);\nplt.show()","metadata":{"id":"9WYTbi-LrDOM","outputId":"797d73c3-a320-4376-c1ff-4607a1adea29","execution":{"iopub.status.busy":"2022-05-25T06:38:38.147339Z","iopub.execute_input":"2022-05-25T06:38:38.147705Z","iopub.status.idle":"2022-05-25T06:38:39.569628Z","shell.execute_reply.started":"2022-05-25T06:38:38.147642Z","shell.execute_reply":"2022-05-25T06:38:39.568916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  資產之間的相關性\n\n我們之前假設加密資產回報可能表現出一些相關性。 現在讓我們更詳細地檢查一下。\n\n我們可以檢查在我們選擇的 2021 年期間，比特幣和以太坊之間的相關性如何隨時間變化。","metadata":{"id":"DnAM1L8dj5ir"}},{"cell_type":"code","source":"# join two asset in single DataFrame\n\nlret_btc_long = log_return(btc.Close)[1:]\nlret_eth_long = log_return(eth.Close)[1:]\nlret_btc_long.rename('lret_btc', inplace=True)\nlret_eth_long.rename('lret_eth', inplace=True)\ntwo_assets = pd.concat([lret_btc_long, lret_eth_long], axis=1)\n\n# group consecutive rows and use .corr() for correlation between columns\ncorr_time = two_assets.groupby(two_assets.index//(10000*60)).corr().loc[:,\"lret_btc\"].loc[:,\"lret_eth\"]\n\ncorr_time.plot();\nplt.xticks([])\nplt.ylabel(\"Correlation\")\nplt.title(\"Correlation between BTC and ETH over time\");","metadata":{"id":"pDB4cVz2u_kC","outputId":"7c483562-4cbd-46a1-d16d-94b3bc69d2c3","execution":{"iopub.status.busy":"2022-05-25T06:38:39.570781Z","iopub.execute_input":"2022-05-25T06:38:39.571375Z","iopub.status.idle":"2022-05-25T06:38:40.173434Z","shell.execute_reply.started":"2022-05-25T06:38:39.571340Z","shell.execute_reply":"2022-05-25T06:38:40.172440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"請注意資產之間的高但可變的相關性。 在這裡，我們可以看到隨著時間的推移存在一些變化的動態，這對於這個時間序列挑戰至關重要，即如何在高度非平穩的環境中執行預測。\n\n系統或過程的靜止行為的特徵在於不隨時間變化的統計特性，例如均值、方差和自相關。 另一方面，非平穩行為的特徵是統計特性隨時間的連續變化。 平穩性很重要，因為許多有用的分析工具、統計測試和模型都依賴於它。\n\n我們還可以通過可視化相關矩陣來檢查所有資產之間的相關性。 請注意，某些資產的成對相關性如何高於其他資產。","metadata":{"id":"oc7qBUdszXZf"}},{"cell_type":"code","source":"# create dataframe with returns for all assets\nall_assets_2021 = pd.DataFrame([])\nfor asset_id, asset_name in zip(asset_details.Asset_ID, asset_details.Asset_Name):\n  asset = crypto_df[crypto_df[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n  asset = asset.loc[totimestamp('01/01/2021'):totimestamp('01/05/2021')]\n  asset = asset.reindex(range(asset.index[0],asset.index[-1]+60,60),method='pad')\n  lret = log_return(asset.Close.fillna(0))[1:]\n  all_assets_2021 = all_assets_2021.join(lret, rsuffix=asset_name, how=\"outer\")","metadata":{"id":"Dzv6CD3nz57z","execution":{"iopub.status.busy":"2022-05-25T06:38:40.174870Z","iopub.execute_input":"2022-05-25T06:38:40.175144Z","iopub.status.idle":"2022-05-25T06:38:45.186743Z","shell.execute_reply.started":"2022-05-25T06:38:40.175112Z","shell.execute_reply":"2022-05-25T06:38:45.185726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_details.Asset_ID, asset_details.Asset_Name","metadata":{"execution":{"iopub.status.busy":"2022-05-25T06:38:45.188382Z","iopub.execute_input":"2022-05-25T06:38:45.188618Z","iopub.status.idle":"2022-05-25T06:38:45.196570Z","shell.execute_reply.started":"2022-05-25T06:38:45.188590Z","shell.execute_reply":"2022-05-25T06:38:45.195666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(all_assets_2021.corr());\nplt.yticks(range(14), asset_details.Asset_Name.values);\nplt.xticks(range(14), asset_details.Asset_Name.values, rotation='vertical');\nplt.colorbar();","metadata":{"execution":{"iopub.status.busy":"2022-05-25T06:38:45.198332Z","iopub.execute_input":"2022-05-25T06:38:45.199128Z","iopub.status.idle":"2022-05-25T06:38:45.750133Z","shell.execute_reply.started":"2022-05-25T06:38:45.199076Z","shell.execute_reply":"2022-05-25T06:38:45.748998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We encourage participants to perform additional statistical analyses to have a stronger grasp on the dataset, including autocorrelation, time-series decomposition and stationarity tests.","metadata":{}},{"cell_type":"markdown","source":"# 構建你的預測模型","metadata":{"id":"EaIGlwUB9kPy"}},{"cell_type":"markdown","source":"## 預測目標和評估","metadata":{"id":"9Akm2RTz-rOH"}},{"cell_type":"markdown","source":"該預測競賽旨在預測每項資產 $a$ 價格 $P^a$ 的近期回報。 對於數據集中的每一行，我們都包含預測目標“Target”。 `Target` 來自 15 分鐘內的日誌返回 ($R^a$)。\n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\n加密資產回報高度相關，在很大程度上跟隨整個加密市場。 由於我們想測試您預測單個資產回報的能力，我們執行線性殘差，在創建目標時從單個資產回報中去除市場信號。 更詳細地說，如果 $M(t)$ 是加權平均市場收益，則目標是：\n\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a}  \\\\\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\\\\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$\n\n其中括號 $\\langle .\\rangle$ 表示隨時間推移的滾動平均值（3750 分鐘窗口），並且用於評估指標的相同資產權重 $w^a$。\n\n由於未來價格中的缺失值，某些行的目標值為空。 出於評分目的，測試集基本事實中具有空值的行將被忽略。\n\n在比賽中，您的預測將根據 Pearson 相關係數的加權版本進行評估，權重由資產詳細信息文件中的“權重”列給出。\n\n在本教程中，我們將簡化事物並使用相關性（無權重）進行評估，並僅考慮兩種資產，BTC 和 ETH。","metadata":{"id":"hl29F0dC9mWc"}},{"cell_type":"markdown","source":"## 功能設計\n\n我們首先設計一些相關的特徵來輸入我們的模型。","metadata":{"id":"Motsk80VF7kj"}},{"cell_type":"code","source":"# Select some input features from the trading data: \n# 5 min log return, abs(5 min log return), upper shadow, and lower shadow.\nupper_shadow = lambda asset: asset.High - np.maximum(asset.Close,asset.Open)\nlower_shadow = lambda asset: np.minimum(asset.Close,asset.Open)- asset.Low\n\nX_btc = pd.concat([log_return(btc.VWAP,periods=5), log_return(btc.VWAP,periods=1).abs(), \n               upper_shadow(btc), lower_shadow(btc)], axis=1)\ny_btc = btc.Target\n\nX_eth = pd.concat([log_return(eth.VWAP,periods=5), log_return(eth.VWAP,periods=1).abs(), \n               upper_shadow(eth), lower_shadow(eth)], axis=1)\ny_eth = eth.Target","metadata":{"id":"2kQz1YDeqsch","execution":{"iopub.status.busy":"2022-05-25T06:38:45.752073Z","iopub.execute_input":"2022-05-25T06:38:45.752745Z","iopub.status.idle":"2022-05-25T06:38:46.112219Z","shell.execute_reply.started":"2022-05-25T06:38:45.752678Z","shell.execute_reply":"2022-05-25T06:38:46.111326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 為構建預測模型準備數據\n\n由於我們將訓練線性回歸參數，我們需要將訓練集和測試集分開。 為此，我們將計算 X 和 y 並將這些數據拆分為訓練和測試拆分。 請注意，測試拆分代表數據的後面部分，因為它通常在時間序列中完成。","metadata":{"id":"26locv57nOiP"}},{"cell_type":"code","source":"# select training and test periods\ntrain_window = [totimestamp(\"01/05/2021\"), totimestamp(\"30/05/2021\")]\ntest_window = [totimestamp(\"01/06/2021\"), totimestamp(\"30/06/2021\")]\n\n# divide data into train and test, compute X and y\n# we aim to build simple regression models using a window_size of 1\nX_btc_train = X_btc.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  # filling NaN's with zeros\ny_btc_train = y_btc.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \n\nX_btc_test = X_btc.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \ny_btc_test = y_btc.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \n\nX_eth_train = X_eth.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \ny_eth_train = y_eth.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \n\nX_eth_test = X_eth.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \ny_eth_test = y_eth.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() ","metadata":{"id":"Ud_lz31L7S1x","execution":{"iopub.status.busy":"2022-05-25T06:38:46.113592Z","iopub.execute_input":"2022-05-25T06:38:46.113986Z","iopub.status.idle":"2022-05-25T06:38:46.130050Z","shell.execute_reply.started":"2022-05-25T06:38:46.113951Z","shell.execute_reply":"2022-05-25T06:38:46.129310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"我們現在標準化輸入數據。 標準化是將不同變量放在同一尺度上的過程。 在回歸分析中，標準化自變量通常至關重要，否則您可能會冒著獲得誤導性結果的風險。","metadata":{"id":"iWGNGmX16l28"}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n# simple preprocessing of the data \nscaler = StandardScaler()\n\nX_btc_train_scaled = scaler.fit_transform(X_btc_train)\nX_btc_test_scaled = scaler.transform(X_btc_test)\n\nX_eth_train_scaled = scaler.fit_transform(X_eth_train)\nX_eth_test_scaled = scaler.transform(X_eth_test)","metadata":{"id":"90LRr1HC8vSH","execution":{"iopub.status.busy":"2022-05-25T06:38:46.132216Z","iopub.execute_input":"2022-05-25T06:38:46.132826Z","iopub.status.idle":"2022-05-25T06:38:46.299196Z","shell.execute_reply.started":"2022-05-25T06:38:46.132785Z","shell.execute_reply":"2022-05-25T06:38:46.297952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 基線模型：線性回歸\n\n我們將對我們設計的特徵嘗試一個簡單的線性回歸模型。 請注意，線性回歸在時間序列分析中並不常用，特別是只有一個時間步！\n\n我們比較了兩個線性回歸基線，一個獨立考慮每個資產，一個將所有資產建模在一起的多個輸入。\n","metadata":{"id":"lWJwz4t0SqiX"}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\n\n# implement basic ML baseline (one per asset)\nlr = LinearRegression()\nlr.fit(X_btc_train_scaled,y_btc_train)\ny_pred_lr_btc = lr.predict(X_btc_test_scaled)\n\nlr.fit(X_eth_train_scaled,y_eth_train)\ny_pred_lr_eth = lr.predict(X_eth_test_scaled)","metadata":{"id":"cc-Ntnit9TH5","execution":{"iopub.status.busy":"2022-05-25T06:38:46.300966Z","iopub.execute_input":"2022-05-25T06:38:46.301354Z","iopub.status.idle":"2022-05-25T06:38:46.485219Z","shell.execute_reply.started":"2022-05-25T06:38:46.301306Z","shell.execute_reply":"2022-05-25T06:38:46.484176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 評估基線\n\n競爭性能指標是加權相關性。 但是，現在我們將使用簡單的相關性來評估所構建的兩個基線模型。","metadata":{"id":"4_LbXT3KEdv5"}},{"cell_type":"code","source":"# implement more complex baseline (multiple input/output regression model)\nfrom sklearn.multioutput import MultiOutputRegressor\n\n# we concatenate X and y for both assets\nX_both_train = np.concatenate((X_btc_train_scaled, X_eth_train_scaled), axis=1)\nX_both_test = np.concatenate((X_btc_test_scaled, X_eth_test_scaled), axis=1)\ny_both_train = np.column_stack((y_btc_train, y_eth_train))\ny_both_test = np.column_stack((y_btc_test, y_eth_test))\n\n# define the direct multioutput model and fit it\nmlr = MultiOutputRegressor(LinearRegression())\nmlr.fit(X_both_train,y_both_train)\ny_pred_lr_both = mlr.predict(X_both_test)","metadata":{"id":"3_RA5yy3-DOi","execution":{"iopub.status.busy":"2022-05-25T06:38:46.486691Z","iopub.execute_input":"2022-05-25T06:38:46.487377Z","iopub.status.idle":"2022-05-25T06:38:46.535458Z","shell.execute_reply.started":"2022-05-25T06:38:46.487326Z","shell.execute_reply":"2022-05-25T06:38:46.534450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Test score for LR baseline: BTC', f\"{np.corrcoef(y_pred_lr_btc, y_btc_test)[0,1]:.2f}\", \n                                ', ETH', f\"{np.corrcoef(y_pred_lr_eth, y_eth_test)[0,1]:.2f}\")\nprint('Test score for multiple output LR baseline: BTC', f\"{np.corrcoef(y_pred_lr_both[:,0], y_btc_test)[0,1]:.2f}\", \n                                                ', ETH', f\"{np.corrcoef(y_pred_lr_both[:,1], y_eth_test)[0,1]:.2f}\")","metadata":{"id":"UhOs6lITEuXC","outputId":"eead03ce-b36e-474a-a13a-42fffc67eac1","execution":{"iopub.status.busy":"2022-05-25T06:38:46.540351Z","iopub.execute_input":"2022-05-25T06:38:46.541625Z","iopub.status.idle":"2022-05-25T06:38:46.558176Z","shell.execute_reply.started":"2022-05-25T06:38:46.541574Z","shell.execute_reply":"2022-05-25T06:38:46.557115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"我們可以看到，對於選擇的訓練和測試期，多資產 LR 模型比簡單地單獨建模每個資產的性能更好。 請注意，由於數據非常不穩定，因此這些結果在不同時期可能會有很大差異。","metadata":{"id":"qzPBu6HUrr9-"}},{"cell_type":"markdown","source":"## 提交\n\n請注意，這是一場代碼競賽，您必須提交您的筆記本才能針對隱藏的私人數據運行。 你的 notebook 應該使用提供的 python 時間序列 API，它可以確保模型不會及時向前看。 使用API請按照[代碼競賽API詳細說明](https://www.kaggle.com/sohier/detailed-api-introduction) 和[基本提交模板](https://www.kaggle.com/sohier/basic-submission-template)。\n","metadata":{"id":"alEWivyFSY68"}}]}