{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G- Research Crypto Forecast Introduction\n\nOver $40 billion worth of cryptocurrencies are traded every day. They are among the most popular assets for speculation and investment, yet have proven wildly volatile. Fast-fluctuating prices have made millionaires of a lucky few, and delivered crushing losses to others. Could some of these price movements have been predicted in advance?\n\nIn this competition, you'll use your machine learning expertise to forecast short term returns in 14 popular cryptocurrencies. We have amassed a dataset of millions of rows of high-frequency market data dating back to 2018 which you can use to build your model. Once the submission deadline has passed, your final score will be calculated over the following 3 months using live crypto data as it is collected.\n\nThe simultaneous activity of thousands of traders ensures that most signals will be transitory, persistent alpha will be exceptionally difficult to find, and the danger of overfitting will be considerable. In addition, since 2018, interest in the cryptomarket has exploded, so the volatility and correlation structure in our data are likely to be highly non-stationary. The successful contestant will pay careful attention to these considerations, and in the process gain valuable insight into the art and science of financial forecasting.\n\nG-Research is Europe’s leading quantitative finance research firm. We have long explored the extent of market prediction possibilities, making use of machine learning, big data, and some of the most advanced technology available. Specializing in data science and AI education for workforces, Cambridge Spark is partnering with G-Research for this competition","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-12T03:32:09.484108Z","iopub.execute_input":"2022-01-12T03:32:09.484495Z","iopub.status.idle":"2022-01-12T03:32:09.525421Z","shell.execute_reply.started":"2022-01-12T03:32:09.484398Z","shell.execute_reply":"2022-01-12T03:32:09.524471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Getting the data folder","metadata":{}},{"cell_type":"code","source":"data_folder = \"../input/g-research-crypto-forecasting/\"\n!ls $data_folder","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:32:09.528948Z","iopub.execute_input":"2022-01-12T03:32:09.529608Z","iopub.status.idle":"2022-01-12T03:32:10.284829Z","shell.execute_reply.started":"2022-01-12T03:32:09.529573Z","shell.execute_reply":"2022-01-12T03:32:10.284017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#reading the train.csv\ncrypto_df = pd.read_csv(data_folder + 'train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:32:10.286527Z","iopub.execute_input":"2022-01-12T03:32:10.286969Z","iopub.status.idle":"2022-01-12T03:33:11.544658Z","shell.execute_reply.started":"2022-01-12T03:32:10.286919Z","shell.execute_reply":"2022-01-12T03:33:11.543692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:11.546769Z","iopub.execute_input":"2022-01-12T03:33:11.547033Z","iopub.status.idle":"2022-01-12T03:33:11.579443Z","shell.execute_reply.started":"2022-01-12T03:33:11.547004Z","shell.execute_reply":"2022-01-12T03:33:11.578440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data features\nWe can see the different features included in the dataset. Specifically, the features included per asset are the following:\n*   **timestamp**: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n*   **Asset_ID**: The asset ID corresponding to one of the crytocurrencies (e.g. `Asset_ID = 1` for Bitcoin). The mapping from `Asset_ID` to crypto asset is contained in `asset_details.csv`.\n*   **Count**: Total number of trades in the time interval (last minute).\n*   **Open**:\tOpening price of the time interval (in USD).\n*   **High**:\tHighest price reached during time interval (in USD).\n*   **Low**: Lowest price reached during time interval (in USD).\n*   **Close**:\tClosing price of the time interval (in USD).\n*   **Volume**:\tQuantity of asset bought or sold, displayed in base currency USD.\n*   **VWAP**: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n*   **Target**: Residual log-returns for the asset over a 15 minute horizon. \n\nThe first two columns define the time and asset indexes for this data row. The 6 middle columns are feature columns with the trading data for this asset and minute in time. The last column is the prediction target, which we will get to later in more detail.\n\nWe also view the asset information, including the list of all assets, the `Asset_ID` to asset mapping, and the weight of each asset used to weigh their relative importance in the evaluation metric.","metadata":{}},{"cell_type":"code","source":"asset_details = pd.read_csv(data_folder + 'asset_details.csv')\nasset_details\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:11.580887Z","iopub.execute_input":"2022-01-12T03:33:11.581166Z","iopub.status.idle":"2022-01-12T03:33:11.599454Z","shell.execute_reply.started":"2022-01-12T03:33:11.581133Z","shell.execute_reply":"2022-01-12T03:33:11.598613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"creating seperate data frames for each crypto","metadata":{}},{"cell_type":"code","source":"binance = crypto_df[crypto_df[\"Asset_ID\"]==0].set_index(\"timestamp\") \nbtc_cash = crypto_df[crypto_df[\"Asset_ID\"]==2].set_index(\"timestamp\") \ncardano = crypto_df[crypto_df[\"Asset_ID\"]==3].set_index(\"timestamp\") \ndodge = crypto_df[crypto_df[\"Asset_ID\"]==4].set_index(\"timestamp\")       \neos = crypto_df[crypto_df[\"Asset_ID\"]==5].set_index(\"timestamp\")   \neth = crypto_df[crypto_df[\"Asset_ID\"]==6].set_index(\"timestamp\")  \neth_classic = crypto_df[crypto_df[\"Asset_ID\"]==7].set_index(\"timestamp\")  \niota = crypto_df[crypto_df[\"Asset_ID\"]==8].set_index(\"timestamp\")  \nlite = crypto_df[crypto_df[\"Asset_ID\"]==9].set_index(\"timestamp\")\nmaker = crypto_df[crypto_df[\"Asset_ID\"]==10].set_index(\"timestamp\")  \nmonero = crypto_df[crypto_df[\"Asset_ID\"]==11].set_index(\"timestamp\")\nstellar = crypto_df[crypto_df[\"Asset_ID\"]==12].set_index(\"timestamp\")\ntron = crypto_df[crypto_df[\"Asset_ID\"]==13].set_index(\"timestamp\")          ","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:11.600989Z","iopub.execute_input":"2022-01-12T03:33:11.601479Z","iopub.status.idle":"2022-01-12T03:33:15.251610Z","shell.execute_reply.started":"2022-01-12T03:33:11.601426Z","shell.execute_reply":"2022-01-12T03:33:15.250725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Candlestick charts\n\nThe trading data format is an aggregated form of market data including for Open, High, Low and Close. We can visualize this data through the commonly used candlestick bar chart, which allows traders to perform technical analysis on intraday values. The bar's body length represents the price range between the open and close of that day's trading. When the bar is red, it means the close was lower than the open, and green otherwise. These are also referred to as bullish and bearish candlesticks. The wicks above and below the bars show the high and low prices of that interval's trading.","metadata":{}},{"cell_type":"markdown","source":"We can visualize a slice of the Bitcoin prices using the `plotly` library. The bottom part of the plot shows a rangeslider, which you can use to zoom in the plot.","metadata":{}},{"cell_type":"markdown","source":"### Visualizing BTC","metadata":{}},{"cell_type":"code","source":"import plotly.graph_objects as go\n\nbtc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\nbtc","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:15.252875Z","iopub.execute_input":"2022-01-12T03:33:15.253124Z","iopub.status.idle":"2022-01-12T03:33:15.613805Z","shell.execute_reply.started":"2022-01-12T03:33:15.253093Z","shell.execute_reply":"2022-01-12T03:33:15.612952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc_mini = btc.iloc[-200:] # Select recent data rows\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:15.615076Z","iopub.execute_input":"2022-01-12T03:33:15.615347Z","iopub.status.idle":"2022-01-12T03:33:15.742183Z","shell.execute_reply.started":"2022-01-12T03:33:15.615315Z","shell.execute_reply":"2022-01-12T03:33:15.741372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualizing Ethernum","metadata":{}},{"cell_type":"code","source":"eth = crypto_df[crypto_df[\"Asset_ID\"]==13].set_index(\"timestamp\")\neth","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:15.743616Z","iopub.execute_input":"2022-01-12T03:33:15.744457Z","iopub.status.idle":"2022-01-12T03:33:16.077682Z","shell.execute_reply.started":"2022-01-12T03:33:15.744410Z","shell.execute_reply":"2022-01-12T03:33:16.076651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth_mini = eth.iloc[-200:] # Select recent data rows\nfig = go.Figure(data=[go.Candlestick(x=eth_mini.index, open=eth_mini['Open'], high=eth_mini['High'], low=eth_mini['Low'], close=eth_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:16.081802Z","iopub.execute_input":"2022-01-12T03:33:16.082154Z","iopub.status.idle":"2022-01-12T03:33:16.095400Z","shell.execute_reply.started":"2022-01-12T03:33:16.082108Z","shell.execute_reply":"2022-01-12T03:33:16.094475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualizing Tron","metadata":{}},{"cell_type":"code","source":"tron = crypto_df[crypto_df[\"Asset_ID\"]==8].set_index(\"timestamp\")\ntron","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:16.096731Z","iopub.execute_input":"2022-01-12T03:33:16.096994Z","iopub.status.idle":"2022-01-12T03:33:16.390040Z","shell.execute_reply.started":"2022-01-12T03:33:16.096962Z","shell.execute_reply":"2022-01-12T03:33:16.389122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tron_mini = tron.iloc[-200:] # Select recent data rows\nfig = go.Figure(data=[go.Candlestick(x=tron_mini.index, open=tron_mini['Open'], high=tron_mini['High'], low=tron_mini['Low'], close=tron_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:16.391188Z","iopub.execute_input":"2022-01-12T03:33:16.391469Z","iopub.status.idle":"2022-01-12T03:33:16.403396Z","shell.execute_reply.started":"2022-01-12T03:33:16.391436Z","shell.execute_reply":"2022-01-12T03:33:16.402798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualizing Dodge","metadata":{}},{"cell_type":"code","source":"dodge = crypto_df[crypto_df[\"Asset_ID\"]==4].set_index(\"timestamp\")\ndodge","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:16.404296Z","iopub.execute_input":"2022-01-12T03:33:16.405007Z","iopub.status.idle":"2022-01-12T03:33:16.633318Z","shell.execute_reply.started":"2022-01-12T03:33:16.404963Z","shell.execute_reply":"2022-01-12T03:33:16.632677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dodge_mini = dodge.iloc[-200:] # Select recent data rows\nfig = go.Figure(data=[go.Candlestick(x=dodge_mini.index, open=dodge_mini['Open'], high=dodge_mini['High'], low=dodge_mini['Low'], close=dodge_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:16.634299Z","iopub.execute_input":"2022-01-12T03:33:16.635003Z","iopub.status.idle":"2022-01-12T03:33:16.647269Z","shell.execute_reply.started":"2022-01-12T03:33:16.634970Z","shell.execute_reply":"2022-01-12T03:33:16.646676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"#### Dealing with missing values ","metadata":{}},{"cell_type":"markdown","source":"First we can get a coin and look more into the dataset","metadata":{}},{"cell_type":"code","source":"# getting the ethernum data\neth = crypto_df[crypto_df[\"Asset_ID\"]==6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\neth.info(show_counts =True)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:16.648154Z","iopub.execute_input":"2022-01-12T03:33:16.648903Z","iopub.status.idle":"2022-01-12T03:33:17.146374Z","shell.execute_reply.started":"2022-01-12T03:33:16.648869Z","shell.execute_reply":"2022-01-12T03:33:17.145731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As in the above table all the columns except the `target`  has same number of Non-Null records. We can confirm it by","metadata":{}},{"cell_type":"code","source":"eth.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.147435Z","iopub.execute_input":"2022-01-12T03:33:17.147762Z","iopub.status.idle":"2022-01-12T03:33:17.188198Z","shell.execute_reply.started":"2022-01-12T03:33:17.147733Z","shell.execute_reply":"2022-01-12T03:33:17.187332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Target column has 340 null values ","metadata":{}},{"cell_type":"markdown","source":"Then we check the time range for the data","metadata":{}},{"cell_type":"code","source":"# getting the first five rows\nbtc.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.189564Z","iopub.execute_input":"2022-01-12T03:33:17.190058Z","iopub.status.idle":"2022-01-12T03:33:17.204942Z","shell.execute_reply.started":"2022-01-12T03:33:17.190025Z","shell.execute_reply":"2022-01-12T03:33:17.204092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# getting the last five rows\nbtc.tail()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.206166Z","iopub.execute_input":"2022-01-12T03:33:17.206615Z","iopub.status.idle":"2022-01-12T03:33:17.227141Z","shell.execute_reply.started":"2022-01-12T03:33:17.206583Z","shell.execute_reply":"2022-01-12T03:33:17.226552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# getting start and end date of btc as datetime64 format\nstart_btc = btc.index[0].astype('datetime64[s]')\nend_btc = btc.index[-1].astype('datetime64[s]')\n\n# getting start and end date of eth as datetime64 format\nstart_eth = btc.index[0].astype('datetime64[s]')\nend_eth = btc.index[-1].astype('datetime64[s]')\n\nprint('BTC data is from ',start_btc , 'to ', end_btc)\nprint('Eth data is from ',start_eth , 'to ', end_eth)\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.228203Z","iopub.execute_input":"2022-01-12T03:33:17.228582Z","iopub.status.idle":"2022-01-12T03:33:17.236869Z","shell.execute_reply.started":"2022-01-12T03:33:17.228544Z","shell.execute_reply":"2022-01-12T03:33:17.235851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When developing a timeseries model we need to find whethere there are any missing timestamps (rows) in the dataset. If there are any missing rows we have to fill the values with an appropriate method","metadata":{}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.237964Z","iopub.execute_input":"2022-01-12T03:33:17.238197Z","iopub.status.idle":"2022-01-12T03:33:17.270735Z","shell.execute_reply.started":"2022-01-12T03:33:17.238157Z","shell.execute_reply":"2022-01-12T03:33:17.269978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Notice that there are many gaps in the data. To work with most time series models, we should preprocess our data into a format without time gaps. To fill the gaps, we can use the .reindex() method for forward filling, filling gaps with the previous valid value.","metadata":{}},{"cell_type":"code","source":"eth = eth.reindex(range(eth.index[0],eth.index[-1]+60,60),method='pad')\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.271984Z","iopub.execute_input":"2022-01-12T03:33:17.272198Z","iopub.status.idle":"2022-01-12T03:33:17.624573Z","shell.execute_reply.started":"2022-01-12T03:33:17.272171Z","shell.execute_reply":"2022-01-12T03:33:17.623655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.625886Z","iopub.execute_input":"2022-01-12T03:33:17.626316Z","iopub.status.idle":"2022-01-12T03:33:17.675938Z","shell.execute_reply.started":"2022-01-12T03:33:17.626269Z","shell.execute_reply":"2022-01-12T03:33:17.675050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data visualisation\n##### We will start by visualising the Close prices for the two assets we have selecte","metadata":{}},{"cell_type":"code","source":"# fill missing values for BTC\nbtc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\n\n#fill missing values for Tron\ntron =  tron.reindex(range(tron.index[0],tron.index[-1]+60,60),method='pad')\n\n#fill missing values for Dodge\ndodge =  dodge.reindex(range(dodge.index[0],dodge.index[-1]+60,60),method='pad')\n\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:17.677271Z","iopub.execute_input":"2022-01-12T03:33:17.677498Z","iopub.status.idle":"2022-01-12T03:33:18.424909Z","shell.execute_reply.started":"2022-01-12T03:33:17.677470Z","shell.execute_reply":"2022-01-12T03:33:18.424064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n\nf = plt.figure(figsize=(20,10))\n\nax = f.add_subplot(221)\nplt.plot(btc['Close'], label='BTC')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin')\n\nax2 = f.add_subplot(222)\nax2.plot(eth['Close'], color='red', label='ETH')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum')\n\nax2 = f.add_subplot(223)\nax2.plot(dodge['Close'], color='green', label='Dodge')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Dodge')\n\nax2 = f.add_subplot(224)\nax2.plot(tron['Close'], color='yellow', label='Tron')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Tron')\n\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:33:18.426339Z","iopub.execute_input":"2022-01-12T03:33:18.426650Z","iopub.status.idle":"2022-01-12T03:33:22.722293Z","shell.execute_reply.started":"2022-01-12T03:33:18.426609Z","shell.execute_reply":"2022-01-12T03:33:22.721300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The assets have quite different history, but we could check if they correlate in recent times.","metadata":{}},{"cell_type":"code","source":"import time\nfrom datetime import datetime\n\n# auxiliary function, from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:49:52.445536Z","iopub.execute_input":"2022-01-12T03:49:52.446186Z","iopub.status.idle":"2022-01-12T03:49:52.452285Z","shell.execute_reply.started":"2022-01-12T03:49:52.446114Z","shell.execute_reply":"2022-01-12T03:49:52.451476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create intervals\nbtc_mini_2021 = btc.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]\neth_mini_2021 = eth.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:00.304361Z","iopub.execute_input":"2022-01-12T03:50:00.304979Z","iopub.status.idle":"2022-01-12T03:50:00.310098Z","shell.execute_reply.started":"2022-01-12T03:50:00.304941Z","shell.execute_reply":"2022-01-12T03:50:00.309356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot time series for both chosen assets\nf = plt.figure(figsize=(7,8))\n\nax = f.add_subplot(211)\nplt.plot(btc_mini_2021['Close'], label='btc')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin Close')\n\nax2 = f.add_subplot(212)\nax2.plot(eth_mini_2021['Close'], color='red', label='eth')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum Close')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:25.883904Z","iopub.execute_input":"2022-01-12T03:50:25.884232Z","iopub.status.idle":"2022-01-12T03:50:26.352010Z","shell.execute_reply.started":"2022-01-12T03:50:25.884177Z","shell.execute_reply":"2022-01-12T03:50:26.351344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On shorter intervals we can visually see some potential correlation between both assets, with some simultaneous ups and downs.","metadata":{}},{"cell_type":"markdown","source":"Analysing the price changes ????","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\ndf1 = pd.DataFrame(np.random.randn(6,3),columns=['col1','col2','col3'])\ndf2 = pd.DataFrame(np.random.randn(2,3),columns=['col1','col2','col3'])\n\nprint(df1)\nprint(df2)\nprint('--------------------------------')\nprint(df2.reindex_like(df1,method='pad'))","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:31.949807Z","iopub.execute_input":"2022-01-12T03:50:31.950258Z","iopub.status.idle":"2022-01-12T03:50:31.972866Z","shell.execute_reply.started":"2022-01-12T03:50:31.950202Z","shell.execute_reply":"2022-01-12T03:50:31.972001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# define function to compute log returns\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:36.453160Z","iopub.execute_input":"2022-01-12T03:50:36.453472Z","iopub.status.idle":"2022-01-12T03:50:36.458133Z","shell.execute_reply.started":"2022-01-12T03:50:36.453433Z","shell.execute_reply":"2022-01-12T03:50:36.457209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import scipy.stats as stats\n\nlret_btc = log_return(btc_mini_2021.Close)[1:]\nlret_eth = log_return(eth_mini_2021.Close)[1:]\nlret_btc.rename('lret_btc', inplace=True)\nlret_eth.rename('lret_eth', inplace=True)\n\nplt.figure(figsize=(8,4))\nplt.plot(lret_btc);\nplt.plot(lret_eth);\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:38.828114Z","iopub.execute_input":"2022-01-12T03:50:38.828419Z","iopub.status.idle":"2022-01-12T03:50:39.361935Z","shell.execute_reply.started":"2022-01-12T03:50:38.828378Z","shell.execute_reply":"2022-01-12T03:50:39.360730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Correlation between assets\nWe hypothesized before that crypto asset returns may exhibit some correlation. Let's check this in more detail now.\nWe can check how the correlation between Bitcoin and Ethereum change over time for the 2021 period we selected.","metadata":{}},{"cell_type":"code","source":"# join two asset in single DataFrame\n\nlret_btc_long = log_return(btc.Close)[1:]\nlret_eth_long = log_return(eth.Close)[1:]\nlret_btc_long.rename('lret_btc', inplace=True)\nlret_eth_long.rename('lret_eth', inplace=True)\ntwo_assets = pd.concat([lret_btc_long, lret_eth_long], axis=1)\n\n# group consecutive rows and use .corr() for correlation between columns\ncorr_time = two_assets.groupby(two_assets.index//(10000*60)).corr().loc[:,\"lret_btc\"].loc[:,\"lret_eth\"]\n\ncorr_time.plot();\nplt.xticks([])\nplt.ylabel(\"Correlation\")\nplt.title(\"Correlation between BTC and ETH over time\");","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:42.433530Z","iopub.execute_input":"2022-01-12T03:50:42.433821Z","iopub.status.idle":"2022-01-12T03:50:42.876241Z","shell.execute_reply.started":"2022-01-12T03:50:42.433789Z","shell.execute_reply":"2022-01-12T03:50:42.875227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note the high but variable correlation between the assets. Here we can see that there is some changing dynamics over time, and this would be critical for this time series challenge, that is, how to perform forecasts in a highly non-stationary environment.\n\nA stationary behaviour of a system or a process is characterized by non-changing statistical properties over time such as the mean, variance and autocorrelation. On the other hand, a non-stationary behaviour is characterized by a continuous change of statistical properties over time. Stationarity is important because many useful analytical tools and statistical tests and models rely on it.\n\nWe can also check the correlation between all assets visualizing the correlation matrix. Note how some assets have much higher pairwise correlation than others.","metadata":{}},{"cell_type":"code","source":"all_assets_2021 = pd.DataFrame([])\nfor asset_id, asset_name in zip(asset_details.Asset_ID, asset_details.Asset_Name):\n  asset = crypto_df[crypto_df[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n  asset = asset.loc[totimestamp('01/01/2021'):totimestamp('01/05/2021')]\n  asset = asset.reindex(range(asset.index[0],asset.index[-1]+60,60),method='pad')\n  lret = log_return(asset.Close.fillna(0))[1:]\n  all_assets_2021 = all_assets_2021.join(lret, rsuffix=asset_name, how=\"outer\")","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:46.783238Z","iopub.execute_input":"2022-01-12T03:50:46.783509Z","iopub.status.idle":"2022-01-12T03:50:50.540083Z","shell.execute_reply.started":"2022-01-12T03:50:46.783479Z","shell.execute_reply":"2022-01-12T03:50:50.539284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(all_assets_2021.corr());\nplt.yticks(asset_details.Asset_ID.values, asset_details.Asset_Name.values);\nplt.xticks(asset_details.Asset_ID.values, asset_details.Asset_Name.values, rotation='vertical');\nplt.colorbar();","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:54.748086Z","iopub.execute_input":"2022-01-12T03:50:54.748677Z","iopub.status.idle":"2022-01-12T03:50:55.236636Z","shell.execute_reply.started":"2022-01-12T03:50:54.748642Z","shell.execute_reply":"2022-01-12T03:50:55.236087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select some input features from the trading data: \n# 5 min log return, abs(5 min log return), upper shadow, and lower shadow.\nupper_shadow = lambda asset: asset.High - np.maximum(asset.Close,asset.Open)\nlower_shadow = lambda asset: np.minimum(asset.Close,asset.Open)- asset.Low\n\nX_btc = pd.concat([log_return(btc.VWAP,periods=5), log_return(btc.VWAP,periods=1).abs(), \n               upper_shadow(btc), lower_shadow(btc)], axis=1)\ny_btc = btc.Target\n\nX_eth = pd.concat([log_return(eth.VWAP,periods=5), log_return(eth.VWAP,periods=1).abs(), \n               upper_shadow(eth), lower_shadow(eth)], axis=1)\ny_eth = eth.Target","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:50:58.674130Z","iopub.execute_input":"2022-01-12T03:50:58.674557Z","iopub.status.idle":"2022-01-12T03:50:58.877992Z","shell.execute_reply.started":"2022-01-12T03:50:58.674525Z","shell.execute_reply":"2022-01-12T03:50:58.877197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# select training and test periods\ntrain_window = [totimestamp(\"01/05/2021\"), totimestamp(\"30/05/2021\")]\ntest_window = [totimestamp(\"01/06/2021\"), totimestamp(\"30/06/2021\")]\n\n# divide data into train and test, compute X and y\n# we aim to build simple regression models using a window_size of 1\nX_btc_train = X_btc.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  # filling NaN's with zeros\ny_btc_train = y_btc.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \n\nX_btc_test = X_btc.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \ny_btc_test = y_btc.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \n\nX_eth_train = X_eth.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \ny_eth_train = y_eth.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \n\nX_eth_test = X_eth.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() ","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:51:13.323108Z","iopub.execute_input":"2022-01-12T03:51:13.323895Z","iopub.status.idle":"2022-01-12T03:51:13.339493Z","shell.execute_reply.started":"2022-01-12T03:51:13.323836Z","shell.execute_reply":"2022-01-12T03:51:13.338442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n# simple preprocessing of the data \nscaler = StandardScaler()\n\nX_btc_train_scaled = scaler.fit_transform(X_btc_train)\nX_btc_test_scaled = scaler.transform(X_btc_test)\n\nX_eth_train_scaled = scaler.fit_transform(X_eth_train)\nX_eth_test_scaled = scaler.transform(X_eth_test)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:51:16.965262Z","iopub.execute_input":"2022-01-12T03:51:16.966111Z","iopub.status.idle":"2022-01-12T03:51:17.118056Z","shell.execute_reply.started":"2022-01-12T03:51:16.966063Z","shell.execute_reply":"2022-01-12T03:51:17.117276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}