{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:55:41.866323Z","iopub.execute_input":"2022-01-19T21:55:41.86721Z","iopub.status.idle":"2022-01-19T21:55:41.897254Z","shell.execute_reply.started":"2022-01-19T21:55:41.867049Z","shell.execute_reply":"2022-01-19T21:55:41.896287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:55:45.449995Z","iopub.execute_input":"2022-01-19T21:55:45.450363Z","iopub.status.idle":"2022-01-19T21:56:57.605606Z","shell.execute_reply.started":"2022-01-19T21:55:45.450321Z","shell.execute_reply":"2022-01-19T21:56:57.60041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/asset_details.csv')\ndf_asset_details","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:57:22.64049Z","iopub.execute_input":"2022-01-19T21:57:22.640871Z","iopub.status.idle":"2022-01-19T21:57:22.66589Z","shell.execute_reply.started":"2022-01-19T21:57:22.640822Z","shell.execute_reply":"2022-01-19T21:57:22.665223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>DATA FEATURES</center> ","metadata":{}},{"cell_type":"markdown","source":"We can see the different features included in the dataset. Specifically, the features included per asset are the following:\n*   **timestamp**: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n*   **Asset_ID**: The asset ID corresponding to one of the crytocurrencies (e.g. `Asset_ID = 1` for Bitcoin). The mapping from `Asset_ID` to crypto asset is contained in `asset_details.csv`.\n*   **Count**: Total number of trades in the time interval (last minute).\n*   **Open**:\tOpening price of the time interval (in USD).\n*   **High**:\tHighest price reached during time interval (in USD).\n*   **Low**: Lowest price reached during time interval (in USD).\n*   **Close**:\tClosing price of the time interval (in USD).\n*   **Volume**:\tQuantity of asset bought or sold, displayed in base currency USD.\n*   **VWAP**: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n*   **Target**: Residual log-returns for the asset over a 15 minute horizon. \n\nThe first two columns define the time and asset indexes for this data row. The 6 middle columns are feature columns with the trading data for this asset and minute in time. The last column is the prediction target, which we will get to later in more detail.\n\nWe also view the asset information, including the list of all assets, the `Asset_ID` to asset mapping, and the weight of each asset used to weigh their relative importance in the evaluation metric.","metadata":{}},{"cell_type":"code","source":"btc = df_train[df_train[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\nbtc_mini = btc.iloc[-200:] # Select recent data rows","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:57:41.525218Z","iopub.execute_input":"2022-01-19T21:57:41.525578Z","iopub.status.idle":"2022-01-19T21:57:42.125396Z","shell.execute_reply.started":"2022-01-19T21:57:41.525539Z","shell.execute_reply":"2022-01-19T21:57:42.124435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\n\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:57:45.021789Z","iopub.execute_input":"2022-01-19T21:57:45.022167Z","iopub.status.idle":"2022-01-19T21:57:45.181702Z","shell.execute_reply.started":"2022-01-19T21:57:45.022131Z","shell.execute_reply":"2022-01-19T21:57:45.180703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## preprocessing","metadata":{}},{"cell_type":"code","source":"btc.info(verbose=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:57:52.042519Z","iopub.execute_input":"2022-01-19T21:57:52.043551Z","iopub.status.idle":"2022-01-19T21:57:52.068935Z","shell.execute_reply.started":"2022-01-19T21:57:52.043485Z","shell.execute_reply":"2022-01-19T21:57:52.067962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:57:55.434277Z","iopub.execute_input":"2022-01-19T21:57:55.434617Z","iopub.status.idle":"2022-01-19T21:57:55.490887Z","shell.execute_reply.started":"2022-01-19T21:57:55.434585Z","shell.execute_reply":"2022-01-19T21:57:55.490018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#check the time range for Bitcoin and Ethereum data, \n#using the coversion from timestamp to `datetime`\nbeg_btc = btc.index[0].astype('datetime64[s]')\nend_btc = btc.index[-1].astype('datetime64[s]')\n\nprint('BTC data goes from ', beg_btc, 'to ', end_btc)","metadata":{"execution":{"iopub.status.busy":"2022-01-19T21:57:56.244133Z","iopub.execute_input":"2022-01-19T21:57:56.244882Z","iopub.status.idle":"2022-01-19T21:57:56.252675Z","shell.execute_reply.started":"2022-01-19T21:57:56.244834Z","shell.execute_reply":"2022-01-19T21:57:56.251646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**.reindex() description** \n\nDataFrame.reindex(labels=None, index=None, columns=None, axis=None, method=None, copy=True, level=None, fill_value=nan, limit=None, tolerance=None)\n\nmethod{None, ‘backfill’/’bfill’, ‘pad’/’ffill’, ‘nearest’}\nMethod to use for filling holes in reindexed DataFrame. Please note: this is only applicable to DataFrames/Series with a monotonically increasing/decreasing index.\n\nNone (default): don’t fill gaps\n\n*pad / ffill: Propagate last valid observation forward to next valid.*\n\nbackfill / bfill: Use next valid observation to fill gap.\n\nnearest: Use nearest valid observations to fill gap.","metadata":{}},{"cell_type":"code","source":"from datetime import datetime\nbtc = df_train[df_train[\"Asset_ID\"]==1]\nbtc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\nbtc_index = btc\nbtc_index['timestamp'] = pd.to_datetime(btc_index['timestamp'], unit='s')","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:05:51.660588Z","iopub.execute_input":"2022-01-19T22:05:51.660977Z","iopub.status.idle":"2022-01-19T22:05:52.339214Z","shell.execute_reply.started":"2022-01-19T22:05:51.660933Z","shell.execute_reply":"2022-01-19T22:05:52.338171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>BAR PLOT</center> \n\nHere I am making a bar plot of month data for 2018 to 2021. For the index, I will use [2018:]. Because our dataset contains data until 2021. So, 2018 to end should bring 2018 to 2021.\n\nEach bar represents a month. A huge spike in April 2020. Otherwise, there is no monthly seasonality here.","metadata":{}},{"cell_type":"code","source":"import matplotlib.dates as mdates\nbtc_month = btc_index.resample(\"M\", on='timestamp').mean()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:05:57.530893Z","iopub.execute_input":"2022-01-19T22:05:57.531211Z","iopub.status.idle":"2022-01-19T22:05:57.602588Z","shell.execute_reply.started":"2022-01-19T22:05:57.53118Z","shell.execute_reply":"2022-01-19T22:05:57.60165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 6))\nax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d %H'))\n#ax.set_xticklabels(btc_hour['timestamp'], rotation=90)\nax.bar(btc_month['2018':].index, btc_month.loc['2018':, \"Volume\"], width=25, align='center')\nfig.autofmt_xdate()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:01.761252Z","iopub.execute_input":"2022-01-19T22:06:01.761543Z","iopub.status.idle":"2022-01-19T22:06:02.430896Z","shell.execute_reply.started":"2022-01-19T22:06:01.761513Z","shell.execute_reply":"2022-01-19T22:06:02.429681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>BOX PLOT</center> \n\nOne way to find seasonality is by using a set of boxplots. Here I am going to make boxplots for each hour. I will use ‘Open’, ‘Close’, ‘High’ and ‘Low’ data to make this plot.\n\nThe dataset is lack of the market between 2021-09-22 to 2022-01-01, we cut out the records between 2018-01-01 to 2018-09-20, to make sure every month has the same amount of data. ","metadata":{}},{"cell_type":"code","source":"import time\nmask = (btc['timestamp'] > '2018-09-21') & (btc['timestamp'] <= '2021-09-21')\nbtc_1821 = btc.loc[mask]","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:06.424116Z","iopub.execute_input":"2022-01-19T22:06:06.424437Z","iopub.status.idle":"2022-01-19T22:06:06.448975Z","shell.execute_reply.started":"2022-01-19T22:06:06.424405Z","shell.execute_reply":"2022-01-19T22:06:06.447832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nbtc_1821['month'] = btc_1821['timestamp'].dt.strftime('%b')\n#start, end = '2018-01-01 00', '2021-07-01 00'\nfig, axes = plt.subplots(4, 1, figsize=(10, 16), sharex=True)\nfor name, ax in zip(['Open', 'Close', 'High', 'Low'], axes):\n    sns.boxplot(data=btc_1821,x='month', y=name, ax=ax)\n    ax.set_ylabel(\"\")\n    ax.set_title(name)\n    if ax != axes[-1]:\n        ax.set_xlabel('')","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:07.776967Z","iopub.execute_input":"2022-01-19T22:06:07.777346Z","iopub.status.idle":"2022-01-19T22:06:13.612694Z","shell.execute_reply.started":"2022-01-19T22:06:07.777304Z","shell.execute_reply":"2022-01-19T22:06:13.611567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The box plots show that prices are usually at their low point in Sep, Oct, Nov, Dec, while at high point in Feb, Mar, Apr.**","metadata":{}},{"cell_type":"markdown","source":"‘Volume’ data was too busy in the original dataset. It can be fixed by resampling. Instead of plotting daily data, plotting monthly average will fix this issue to a large extent. I will use the btc_month dataset I prepared already for the bar plot and box plots above for this.","metadata":{}},{"cell_type":"code","source":"btc_month['Volume'].plot(figsize=(8, 6))","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:19.433143Z","iopub.execute_input":"2022-01-19T22:06:19.433511Z","iopub.status.idle":"2022-01-19T22:06:19.785992Z","shell.execute_reply.started":"2022-01-19T22:06:19.433469Z","shell.execute_reply":"2022-01-19T22:06:19.785006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>resample to smooth out the spikes</center> \nIn the ‘Volume’ data we are working on right now, we can observe some big spikes here and there. These types of spikes are not helpful for data analysis or for modeling. normally to smooth out the spikes, resampling to a lower frequency and rolling is very helpful.","metadata":{}},{"cell_type":"code","source":"btc_week = btc_index.resample(\"W\", on='timestamp').mean()\nbtc_day = btc_index.resample(\"D\", on='timestamp').mean()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:22.217672Z","iopub.execute_input":"2022-01-19T22:06:22.21828Z","iopub.status.idle":"2022-01-19T22:06:22.318938Z","shell.execute_reply.started":"2022-01-19T22:06:22.21822Z","shell.execute_reply":"2022-01-19T22:06:22.317968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"plot the daily and weekly data in the same plot.","metadata":{}},{"cell_type":"code","source":"start, end = '2021-01', '2021-08'\nfig, ax = plt.subplots()\nax.plot(btc_day.loc[start:end, 'Volume'], marker='.', linestyle='-', linewidth = 0.5, label='Daily', color='black')\nax.plot(btc_week.loc[start:end, 'Volume'], marker='o', markersize=8, linestyle='-', label='Weekly', color='coral')\nax.set_ylabel(\"Open\")\nax.legend()\nfig.autofmt_xdate()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:23.894323Z","iopub.execute_input":"2022-01-19T22:06:23.894613Z","iopub.status.idle":"2022-01-19T22:06:24.14401Z","shell.execute_reply.started":"2022-01-19T22:06:23.89458Z","shell.execute_reply":"2022-01-19T22:06:24.142934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>PLOT THE CHANGE</center> ","metadata":{}},{"cell_type":"markdown","source":"### method 1","metadata":{}},{"cell_type":"code","source":"btc_date = btc_index.set_index(\"timestamp\")","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:48.069156Z","iopub.execute_input":"2022-01-19T22:06:48.070264Z","iopub.status.idle":"2022-01-19T22:06:48.089239Z","shell.execute_reply.started":"2022-01-19T22:06:48.070198Z","shell.execute_reply":"2022-01-19T22:06:48.088131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The shift function shifts the data before or after the specified amount of time. If I do not specify the time it will shift the data by one day by default. That means you will get the previous day's data. In financial data like this one, it is helpful to see previous day data and today's data side by side. It only plot the previous day data:","metadata":{}},{"cell_type":"code","source":"btc_date['Change'] = btc_date.Close.div(btc_date.Close.shift())\nbtc_date['Change'].plot(figsize=(20, 8), fontsize = 16)","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:51.679843Z","iopub.execute_input":"2022-01-19T22:06:51.680223Z","iopub.status.idle":"2022-01-19T22:06:55.666132Z","shell.execute_reply.started":"2022-01-19T22:06:51.680188Z","shell.execute_reply":"2022-01-19T22:06:55.665207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the code above, .div() helps to fill up the missing data. Actually, div() means division. df. div(6) will divide each element in df by 6. But here I used ‘df.Close.shift()’. So, Each element of df will be divided by each element of ‘df.Close.shift()’. We do this to avoid the null values that are created by the ‘shift()’ operation","metadata":{}},{"cell_type":"markdown","source":"We can simply take a specific period and plot to have a clearer look. This is the plot of 2020 only.","metadata":{}},{"cell_type":"code","source":"btc_date['2020']['Change'].plot(figsize=(10, 6))","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:06:59.907812Z","iopub.execute_input":"2022-01-19T22:06:59.908431Z","iopub.status.idle":"2022-01-19T22:07:01.102866Z","shell.execute_reply.started":"2022-01-19T22:06:59.908383Z","shell.execute_reply":"2022-01-19T22:07:01.101976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### method 2","metadata":{}},{"cell_type":"markdown","source":"Another way of transformation. It keeps adding the cumulative. For example, if you add an expanding function to the ‘High’ column first element remains the same. The second element becomes cumulative of the first and second element, the third element becomes cumulative of the first, second, and third element, and so on. You can use aggregate functions like mean, median, standard deviation, etc. on it too.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nax = btc_date.High.plot(label='High')\nax = btc_date.High.expanding().mean().plot(label='High expanding mean')\nax = btc_date.High.expanding().std().plot(label='High expanding std')\nax.legend()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:07:10.637187Z","iopub.execute_input":"2022-01-19T22:07:10.637497Z","iopub.status.idle":"2022-01-19T22:07:20.201191Z","shell.execute_reply.started":"2022-01-19T22:07:10.637463Z","shell.execute_reply":"2022-01-19T22:07:20.200349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center>DECOMPESITION</center> \n\nDecomposition will show the observations and these three elements in the same plot:\n*   Trend: Consistent upward or downward slope of a time series.\n*   Seasonality: Clear periodic pattern of a time series\n*   Noise: Outliers or missing values\n\nOriginal observations = Trend + Seasonality + Residuals","metadata":{}},{"cell_type":"code","source":"from pylab import rcParams\nimport statsmodels.api as sm\nrcParams['figure.figsize'] = 11, 9\ndecomposition = sm.tsa.seasonal_decompose(btc_month['Volume'], model='Additive')\nfig = decomposition.plot()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T22:07:25.610454Z","iopub.execute_input":"2022-01-19T22:07:25.610734Z","iopub.status.idle":"2022-01-19T22:07:27.279816Z","shell.execute_reply.started":"2022-01-19T22:07:25.610702Z","shell.execute_reply":"2022-01-19T22:07:27.279117Z"},"trusted":true},"execution_count":null,"outputs":[]}]}