{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Evaluation Metric\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.\n\n### Target\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. \n\n### Weights\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.","metadata":{}},{"cell_type":"code","source":"\n\n!cp ../input/talibinstall/ta-lib-0.4.0-src.tar.gzh  ./ta-lib-0.4.0-src.tar.gz\n!tar -xzvf ta-lib-0.4.0-src.tar.gz > null\n!cd ta-lib && ./configure --prefix=/usr > null && make  > null && make install > null\n\n\n\n!cp ../input/talibinstall/TA-Lib-0.4.21.tar.gzh TA-Lib-0.4.21.tar.gz\n!pip install TA-Lib-0.4.21.tar.gz\n!pip install ../input/talibinstall/numpy-1.21.4-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl\nimport talib as ta\n\n","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:34:30.312837Z","iopub.execute_input":"2021-12-03T10:34:30.313506Z","iopub.status.idle":"2021-12-03T10:38:23.487546Z","shell.execute_reply.started":"2021-12-03T10:34:30.313407Z","shell.execute_reply":"2021-12-03T10:38:23.486453Z"},"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !pip install ta-lib","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:38:23.489855Z","iopub.execute_input":"2021-12-03T10:38:23.490203Z","iopub.status.idle":"2021-12-03T10:38:23.497017Z","shell.execute_reply.started":"2021-12-03T10:38:23.490161Z","shell.execute_reply":"2021-12-03T10:38:23.495782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import Libraries\nimport warnings # Supress warnings\nwarnings.filterwarnings(\"ignore\")\nimport pandas as pd\nimport numpy as np\npd.set_option(\"display.max_columns\", None)\n\nfrom datetime import datetime\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 14})\n\nfrom statsmodels.graphics.tsaplots import plot_acf\nfrom statsmodels.graphics.tsaplots import plot_pacf\n\nfrom PIL import Image\nimport cv2\nimport talib as ta\n\n\nasset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv', low_memory=False)\ntrain = pd.read_csv('../input/g-research-crypto-forecasting/train.csv', low_memory=False)\n#supplemental_train = pd.read_csv('supplemental_train.csv', low_memory=False)\n#example_test = pd.read_csv('example_test.csv', low_memory=False)\n#example_sample_submission = pd.read_csv('example_sample_submission.csv', low_memory=False)\n\nrename_dict = {}\nfor a in asset_details['Asset_ID']:\n    rename_dict[a] = asset_details[asset_details.Asset_ID == a].Asset_Name.values[0]\n\ndisplay(asset_details)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-12-03T10:40:42.585754Z","iopub.execute_input":"2021-12-03T10:40:42.586154Z","iopub.status.idle":"2021-12-03T10:41:49.716077Z","shell.execute_reply.started":"2021-12-03T10:40:42.586115Z","shell.execute_reply":"2021-12-03T10:41:49.715144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Preprocessing\nFor the following EDA, resampling of the minute-wise crypto data to daily samples is applied which reduces the amount of samples from 24,236,806 to 1,360.","metadata":{}},{"cell_type":"code","source":"asset_details.Weight.sum()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:41:49.717970Z","iopub.execute_input":"2021-12-03T10:41:49.718358Z","iopub.status.idle":"2021-12-03T10:41:49.726651Z","shell.execute_reply.started":"2021-12-03T10:41:49.718328Z","shell.execute_reply":"2021-12-03T10:41:49.725564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The weight of each asset used to weigh their relative importance in the evaluation metric.","metadata":{}},{"cell_type":"markdown","source":"### Check BTC","metadata":{}},{"cell_type":"code","source":"btc =train[train.Asset_ID==1].reset_index(drop=True) \nbtc","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:41:49.728139Z","iopub.execute_input":"2021-12-03T10:41:49.729027Z","iopub.status.idle":"2021-12-03T10:41:50.194474Z","shell.execute_reply.started":"2021-12-03T10:41:49.728965Z","shell.execute_reply":"2021-12-03T10:41:50.193657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc['date'] = btc.timestamp.astype('datetime64[s]')\nbtc['date'] = btc['date'].astype(str)\nbtc['time'] = btc['date'].apply(lambda x: x[11:])\nbtc","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:41:50.196515Z","iopub.execute_input":"2021-12-03T10:41:50.196991Z","iopub.status.idle":"2021-12-03T10:41:54.434964Z","shell.execute_reply.started":"2021-12-03T10:41:50.196941Z","shell.execute_reply":"2021-12-03T10:41:54.434140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc =btc[btc.time=='00:01:00'].reset_index(drop=True) \nbtc","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:41:54.436455Z","iopub.execute_input":"2021-12-03T10:41:54.436932Z","iopub.status.idle":"2021-12-03T10:41:54.973915Z","shell.execute_reply.started":"2021-12-03T10:41:54.436873Z","shell.execute_reply":"2021-12-03T10:41:54.973112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### HEATMAP","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(10,8))\nsns.heatmap(btc[['Count','Open','High','Low','Close','Volume','VWAP','Target']].corr(), vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:42:46.039906Z","iopub.execute_input":"2021-12-03T10:42:46.040343Z","iopub.status.idle":"2021-12-03T10:42:46.628019Z","shell.execute_reply.started":"2021-12-03T10:42:46.040311Z","shell.execute_reply":"2021-12-03T10:42:46.627190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    a = str(x-1) +' ' + asset_details[asset_details.Asset_ID==i].Asset_Name.values \n    plt.title(a,fontsize=18)\n    plt.plot(money.index,money.Target)\n    x +=1\ndel money","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:42:47.526075Z","iopub.execute_input":"2021-12-03T10:42:47.526402Z","iopub.status.idle":"2021-12-03T10:43:00.589645Z","shell.execute_reply.started":"2021-12-03T10:42:47.526367Z","shell.execute_reply":"2021-12-03T10:43:00.588896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Waveforms of 0 and 5 are very similar\n<br>・ Waveforms of 1 and 7 are similar\n<br>・ Waveforms of 3,8,12,13 are a little similar\n<br>・ Waveforms of 2, 9 and 10 are a little similar\n<br>・ There are not many similarities between 6 and 11","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\ndata =train #[-10000:]\nfor i in data.Asset_ID.unique():\n    money = data[data.Asset_ID==i].reset_index(drop=True) \n    plt.plot(money.index,money.Target)\ndel data","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:00.591109Z","iopub.execute_input":"2021-12-03T10:43:00.591367Z","iopub.status.idle":"2021-12-03T10:43:13.204593Z","shell.execute_reply.started":"2021-12-03T10:43:00.591338Z","shell.execute_reply":"2021-12-03T10:43:13.203919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\ndata =train[-10000:]\nfor i in data.Asset_ID.unique():\n    money = data[data.Asset_ID==i].reset_index(drop=True) \n    plt.plot(money.index,money.Target)\ndel data","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:13.205839Z","iopub.execute_input":"2021-12-03T10:43:13.206253Z","iopub.status.idle":"2021-12-03T10:43:13.762572Z","shell.execute_reply.started":"2021-12-03T10:43:13.206221Z","shell.execute_reply":"2021-12-03T10:43:13.761811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\ndata =train[-1000:]\nfor i in data.Asset_ID.unique():\n    money = data[data.Asset_ID==i].reset_index(drop=True) \n    plt.plot(money.index,money.Target)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:13.764429Z","iopub.execute_input":"2021-12-03T10:43:13.764770Z","iopub.status.idle":"2021-12-03T10:43:14.212240Z","shell.execute_reply.started":"2021-12-03T10:43:13.764739Z","shell.execute_reply":"2021-12-03T10:43:14.211415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### HeatMap Coins\n\nSome coins have a strong correlation.","metadata":{}},{"cell_type":"code","source":"check = pd.DataFrame()\nfor i in data.Asset_ID.unique():\n    check[i] = data[data.Asset_ID==i]['Target'].reset_index(drop=True) ","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:14.213434Z","iopub.execute_input":"2021-12-03T10:43:14.213667Z","iopub.status.idle":"2021-12-03T10:43:14.242275Z","shell.execute_reply.started":"2021-12-03T10:43:14.213638Z","shell.execute_reply":"2021-12-03T10:43:14.241256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(10,8))\nsns.heatmap(check.dropna().corr(), vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()\ndel check","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:14.243731Z","iopub.execute_input":"2021-12-03T10:43:14.244128Z","iopub.status.idle":"2021-12-03T10:43:15.314975Z","shell.execute_reply.started":"2021-12-03T10:43:14.244068Z","shell.execute_reply":"2021-12-03T10:43:15.313929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in  train.Asset_ID.unique():\n    check = train[train.Asset_ID==i].reset_index(drop=True) \n    print('Asset_ID=', pd.to_datetime(check.loc[0,'timestamp'],unit=\"s\",infer_datetime_format=True))","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:15.316211Z","iopub.execute_input":"2021-12-03T10:43:15.316413Z","iopub.status.idle":"2021-12-03T10:43:20.739581Z","shell.execute_reply.started":"2021-12-03T10:43:15.316388Z","shell.execute_reply":"2021-12-03T10:43:20.738114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"check","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:20.741541Z","iopub.execute_input":"2021-12-03T10:43:20.741849Z","iopub.status.idle":"2021-12-03T10:43:20.781560Z","shell.execute_reply.started":"2021-12-03T10:43:20.741809Z","shell.execute_reply":"2021-12-03T10:43:20.780484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Open/Close","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    plt.plot(money.index,money.Open,color=\"red\")\n    plt.plot(money.index,money.Close,color=\"blue\")\n    x +=1","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:20.783143Z","iopub.execute_input":"2021-12-03T10:43:20.783541Z","iopub.status.idle":"2021-12-03T10:43:37.727761Z","shell.execute_reply.started":"2021-12-03T10:43:20.783498Z","shell.execute_reply":"2021-12-03T10:43:37.726817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### High/Low","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    plt.plot(money.index,money.High,color=\"red\")\n    plt.plot(money.index,money.Low,color=\"blue\")\n    x +=1\ndel money","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:37.730215Z","iopub.execute_input":"2021-12-03T10:43:37.730555Z","iopub.status.idle":"2021-12-03T10:43:51.603679Z","shell.execute_reply.started":"2021-12-03T10:43:37.730525Z","shell.execute_reply":"2021-12-03T10:43:51.603005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Volume","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    plt.plot(money.index,money.Volume)\n    x +=1\ndel money","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:43:51.605034Z","iopub.execute_input":"2021-12-03T10:43:51.605383Z","iopub.status.idle":"2021-12-03T10:44:02.604068Z","shell.execute_reply.started":"2021-12-03T10:43:51.605344Z","shell.execute_reply":"2021-12-03T10:44:02.602956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### VWAP","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    plt.plot(money.index,money.VWAP)\n    x +=1\n    \ndel money","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:44:02.605709Z","iopub.execute_input":"2021-12-03T10:44:02.605961Z","iopub.status.idle":"2021-12-03T10:44:12.784263Z","shell.execute_reply.started":"2021-12-03T10:44:02.605923Z","shell.execute_reply":"2021-12-03T10:44:12.782935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Count","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    plt.plot(money.index,money.Count)\n    x +=1\ndel money","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:44:12.785767Z","iopub.execute_input":"2021-12-03T10:44:12.786195Z","iopub.status.idle":"2021-12-03T10:44:22.357932Z","shell.execute_reply.started":"2021-12-03T10:44:12.786161Z","shell.execute_reply":"2021-12-03T10:44:22.356930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc['high_low'] = btc['High'] - btc['Low']\nbtc['open_close'] = btc['Open'] - btc['Close']","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:44:22.359623Z","iopub.execute_input":"2021-12-03T10:44:22.360008Z","iopub.status.idle":"2021-12-03T10:44:22.365787Z","shell.execute_reply.started":"2021-12-03T10:44:22.359977Z","shell.execute_reply":"2021-12-03T10:44:22.365214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,8))\nsns.heatmap(btc[['Count','Open','High','Low','Close','Volume','high_low','open_close','VWAP','Target']].corr(), vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:44:22.366702Z","iopub.execute_input":"2021-12-03T10:44:22.367674Z","iopub.status.idle":"2021-12-03T10:44:23.070981Z","shell.execute_reply.started":"2021-12-03T10:44:22.367639Z","shell.execute_reply":"2021-12-03T10:44:23.069956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### MACD","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    money['MACD'],_,_ = ta.MACD(money['Close'], fastperiod=12, slowperiod=26, signalperiod=9)\n    plt.plot(money.index,money.MACD)\n    x +=1","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:48:25.271066Z","iopub.execute_input":"2021-12-03T10:48:25.271421Z","iopub.status.idle":"2021-12-03T10:48:36.847081Z","shell.execute_reply.started":"2021-12-03T10:48:25.271384Z","shell.execute_reply":"2021-12-03T10:48:36.845878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### RSI","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(30,20))\nx=1\nfor i in train.Asset_ID.unique():\n    money = train[train.Asset_ID==i].reset_index(drop=True) \n    fig.add_subplot(4, 4, x)\n    plt.title(asset_details[asset_details.Asset_ID==i].Asset_Name.values,fontsize=18)\n    money[\"RSI\"]=ta.RSI(money['Close'], timeperiod=14)\n    plt.plot(money.index,money.RSI)\n    x +=1","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:48:55.248296Z","iopub.execute_input":"2021-12-03T10:48:55.249292Z","iopub.status.idle":"2021-12-03T10:49:09.252378Z","shell.execute_reply.started":"2021-12-03T10:48:55.249232Z","shell.execute_reply":"2021-12-03T10:49:09.251335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(train)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert timestamp\ntrain['timestamp'] = train['timestamp'].astype('datetime64[s]')\n\n# Resample\ntrain_daily = pd.DataFrame()\n\nfor asset_id in asset_details.Asset_ID:\n    train_single = train[train.Asset_ID == asset_id].copy()\n\n    train_single_new = train_single[['timestamp','Count']].resample('D', on='timestamp').sum()\n    train_single_new['Open'] = train_single[['timestamp','Open']].resample('D', on='timestamp').first()['Open']\n    train_single_new['High'] = train_single[['timestamp','High']].resample('D', on='timestamp').max()['High']\n    train_single_new['Low'] = train_single[['timestamp','Low']].resample('D', on='timestamp').min()['Low']\n    train_single_new['Close'] = train_single[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\n    train_single_new['Volume'] = train_single[['timestamp','Volume']].resample('D', on='timestamp').sum()['Volume']\n    train_single_new['Target'] = train_single[['timestamp','Target']].resample('D', on='timestamp').mean()['Target']\n    train_single_new['Asset_ID'] = asset_id\n    \n    train_daily = train_daily.append(train_single_new.reset_index(drop=False))\n    \ntrain_daily = train_daily.sort_values(by = ['timestamp', 'Asset_ID']).reset_index(drop=True)\n\ntrain_daily = train_daily.pivot(index='timestamp', columns='Asset_ID')[['Count', 'Open', 'High', 'Low', 'Close', 'Volume']]\ntrain_daily = train_daily.reset_index(drop=False)\n\ndisplay(train_daily.head(10))","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:49:26.694522Z","iopub.execute_input":"2021-12-03T10:49:26.695494Z","iopub.status.idle":"2021-12-03T10:49:41.005283Z","shell.execute_reply.started":"2021-12-03T10:49:26.695432Z","shell.execute_reply":"2021-12-03T10:49:41.002014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Overview\nFurthermore, we have samples from 2018-01-01 to 2021-09-21 for the majority of coins. For TRON, Stellar, Cardano, IOTA, Maker, and Dogecoin we have fewer data starting from later in 2018 or even later in 2019 in Dogecoin's case.","metadata":{}},{"cell_type":"code","source":"train_daily.info()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:49:41.012807Z","iopub.execute_input":"2021-12-03T10:49:41.013171Z","iopub.status.idle":"2021-12-03T10:49:41.058420Z","shell.execute_reply.started":"2021-12-03T10:49:41.013128Z","shell.execute_reply":"2021-12-03T10:49:41.057068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(asset_details.Asset_ID)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:49:41.061163Z","iopub.execute_input":"2021-12-03T10:49:41.061485Z","iopub.status.idle":"2021-12-03T10:49:41.069626Z","shell.execute_reply.started":"2021-12-03T10:49:41.061433Z","shell.execute_reply":"2021-12-03T10:49:41.068200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_details.Asset_Name","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:49:41.072435Z","iopub.execute_input":"2021-12-03T10:49:41.072766Z","iopub.status.idle":"2021-12-03T10:49:41.083227Z","shell.execute_reply.started":"2021-12-03T10:49:41.072706Z","shell.execute_reply":"2021-12-03T10:49:41.082319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize\nfig = make_subplots(\n    rows=len(asset_details.Asset_ID), cols=1,\n    subplot_titles=(asset_details.Asset_Name)\n)\n\nfor i, asset_id in enumerate(asset_details.Asset_ID):\n    fig.append_trace(go.Candlestick(x=train_daily.timestamp, \n                                         open=train_daily[('Open', asset_id)], \n                                         high=train_daily[('High', asset_id)], \n                                         low=train_daily[('Low', asset_id)], \n                                         close=train_daily[('Close', asset_id)]),\n                  row=i+1, col=1,\n                    )\n\n    fig.update_xaxes(range=[train_daily.timestamp.iloc[0], train_daily.timestamp.iloc[-1]], row=i+1, col=1)\n    \nfig.update_layout(xaxis_rangeslider_visible = False, \n                  xaxis2_rangeslider_visible = False, \n                  xaxis3_rangeslider_visible = False,\n                  xaxis4_rangeslider_visible = False,\n                  xaxis5_rangeslider_visible = False,\n                  xaxis6_rangeslider_visible = False,\n                  xaxis7_rangeslider_visible = False,\n                  xaxis8_rangeslider_visible = False,\n                  xaxis9_rangeslider_visible = False,\n                  xaxis10_rangeslider_visible = False,\n                  xaxis11_rangeslider_visible = False,\n                  xaxis12_rangeslider_visible = False,\n                  xaxis13_rangeslider_visible = False,\n                  xaxis14_rangeslider_visible = False,\n                  height=3000, width=800, \n                  #title_text=\"Subplots with Annotations\"\n                      margin = dict(\n        l = 0,\n        r = 0,\n        b = 0,\n        t = 30,\n        pad = 0)\n                 )\n                    \nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering 1: Time Features","metadata":{}},{"cell_type":"code","source":"train_daily['year'] = pd.DatetimeIndex(train_daily['timestamp']).year\ntrain_daily['quarter'] = pd.DatetimeIndex(train_daily['timestamp']).quarter\ntrain_daily['month'] = pd.DatetimeIndex(train_daily['timestamp']).month\ntrain_daily['weekofyear'] = pd.DatetimeIndex(train_daily['timestamp']).weekofyear\ntrain_daily['dayofyear'] = pd.DatetimeIndex(train_daily['timestamp']).dayofyear\ntrain_daily['weekday'] = pd.DatetimeIndex(train_daily['timestamp']).weekday","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:49:42.605381Z","iopub.execute_input":"2021-12-03T10:49:42.606070Z","iopub.status.idle":"2021-12-03T10:49:42.654263Z","shell.execute_reply.started":"2021-12-03T10:49:42.606029Z","shell.execute_reply":"2021-12-03T10:49:42.653286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stationarity\nThe 'Close' prices seem to be mostly non-stationary. However, Bitcoin and Ethereum seem to be stationary.","metadata":{}},{"cell_type":"code","source":"from statsmodels.tsa.stattools import adfuller\n\ndef check_stationarity(series, asset_id):\n    # Copied and edited from https://machinelearningmastery.com/time-series-data-stationary-python/\n\n    result = adfuller(series.values)\n    if (result[1] <= 0.05) & (result[4]['5%'] > result[0]):\n        print(f\"{asset_details[asset_details.Asset_ID == asset_id].Asset_Name.values[0]}: \\u001b[32mStationary\\u001b[0m\")\n    else:\n        print(f\"{asset_details[asset_details.Asset_ID == asset_id].Asset_Name.values[0]}: \\x1b[31mNon-stationary\\x1b[0m\")\n       \n    print('ADF Statistic: %f' % result[0])\n    print('p-value: %f' % result[1])\n    print('Critical Values:')\n    for key, value in result[4].items():\n        print('\\t%s: %.3f' % (key, value))\n    print('\\n')\nfor i, asset_id in enumerate(asset_details.Asset_ID):    \n    check_stationarity(train_daily[('Close', i)].fillna(0), asset_id)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:50:45.615727Z","iopub.execute_input":"2021-12-03T10:50:45.616227Z","iopub.status.idle":"2021-12-03T10:50:46.549686Z","shell.execute_reply.started":"2021-12-03T10:50:45.616183Z","shell.execute_reply":"2021-12-03T10:50:46.548663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering 2: Log Return\nTo make a time series stationary, you can try differencing it. In this case, we will use the log return instead as shown below.","metadata":{}},{"cell_type":"code","source":"# define function to compute log returns\ndef log_return(series, periods=1):\n    # Copied from https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\n    return np.log(series).diff(periods=periods)\n\n\nfor i, asset_id in enumerate(asset_details.Asset_ID):\n    train_daily[('lret',  asset_id)] = log_return(train_daily[( 'Close',  asset_id)])","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:50:52.595131Z","iopub.execute_input":"2021-12-03T10:50:52.595402Z","iopub.status.idle":"2021-12-03T10:50:52.622016Z","shell.execute_reply.started":"2021-12-03T10:50:52.595375Z","shell.execute_reply":"2021-12-03T10:50:52.620649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Determining Trend with Time-Series Decomposition","metadata":{}},{"cell_type":"code","source":"from statsmodels.tsa.seasonal import seasonal_decompose\n\nperiods = [7, 28, 365]\n   \nasset_id = 1 # Bitcoin\n# Visualize\nf, ax = plt.subplots(nrows=len(periods), ncols=1, figsize=(12, 12))\nfor i, p in enumerate(periods):\n    decomp = seasonal_decompose(train_daily[('Close',  asset_id)].fillna(0), period=p, model='additive', extrapolate_trend='freq')\n    train_daily[(f'Trend_{p}',  asset_id)] = np.where(train_daily[('Close',  asset_id)].isna(), np.NaN, decomp.trend) #decomp.trend\n    \n    \n    sns.lineplot(data=train_daily, x='timestamp', y = ('Close',  asset_id) , ax=ax[i], color='lightgrey');\n    sns.lineplot(data=train_daily, x='timestamp', y = (f'Trend_{p}',  asset_id) , ax=ax[i], color='red');\n    ax[i].set_title(f\"{asset_details[asset_details.Asset_ID == asset_id].Asset_Name.values[0]} Trend with a Period of {p} Day\")\n    ax[i].set_xlim([train_daily.timestamp.iloc[0], train_daily.timestamp.iloc[-1]])\n    #ax[i].set_ylim([-0.6,0.6])\n    ax[i].set_ylabel('Close Price [$]')\n    \n#plt.suptitle(f'Underlying Trend with {PERIOD} day period\\n')\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:55:09.282362Z","iopub.execute_input":"2021-12-03T10:55:09.283432Z","iopub.status.idle":"2021-12-03T10:55:10.587970Z","shell.execute_reply.started":"2021-12-03T10:55:09.283386Z","shell.execute_reply":"2021-12-03T10:55:10.587170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trend = train_daily[('Close',  1)].rolling(\n    window=365,       # 365-day window\n    center=True,      # puts the average at the center of the window\n    min_periods=183,  # choose about half the window size\n).mean()   \n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.lineplot(data=train_daily, x='timestamp', y = ('Close',  1) , ax=ax, color='lightgrey');\nsns.lineplot(x=train_daily['timestamp'], y = trend, ax=ax, color='red');\n#ax[i].set_title(f\"{asset_details[asset_details.Asset_ID == asset_id].Asset_Name.values[0]} Trend with a Period of {p} Day\")\nax.set_xlim([train_daily.timestamp.iloc[0], train_daily.timestamp.iloc[-1]])\n#ax[i].set_ylabel('Close Price [$]')\n\n#plt.suptitle(f'Underlying Trend with {PERIOD} day period\\n')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T10:54:16.972201Z","iopub.execute_input":"2021-12-03T10:54:16.972588Z","iopub.status.idle":"2021-12-03T10:54:17.441195Z","shell.execute_reply.started":"2021-12-03T10:54:16.972553Z","shell.execute_reply":"2021-12-03T10:54:17.440394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}