{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nAutocorrelation analysis is an important step in the Exploratory Data Analysis (EDA) of time series. **The autocorrelation analysis helps in detecting hidden patterns and seasonality and in checking for randomness.**\nIt is especially important when you intend to use an ARIMA model for forecasting because the autocorrelation analysis helps to identify the AR and MA parameters for the ARIMA model.\n\n**Overview**\n* [Fundamentals](#Fundamentals)<br>\n    * [Auto-Regressive and Moving Average Models](#[Auto-Regressive-and-Moving-Average-Models])<br>\n    * [Stationarity](#Stationarity)<br>\n    * [Autocorrelation Function and Partial Autocorrelation Function](#Autocorrelation-Function-and-Partial-Autocorrelation-Function)<br>\n    * [Order of AR, MA, and ARMA Model](#Order-of-AR-MA-and-ARMA-Model)\n* [Examples](#Examples)<br>\n    * [AR(1) Process](#AR(1%29-Process)<br>\n    * [AR(2) Process](#AR(2%29-Process)<br>\n    * [MA(1) Process](#MA(1%29-Process)<br>\n    * [MA(2) Process](#MA(2%29-Process)<br>\n    * [Periodical](#Periodical)<br>\n    * [Trend](#Trend)<br>\n    * [White Noise](#White-Noise)<br>\n    * [Random-Walk](#Random-Walk)<br>\n    * [Constant](#Constant)<br>\n* [🚀 Cheat Sheet](#🚀-Cheat-Sheet)<br>\n* [Case Study](#Case-Study)<br>\n    * [Bitcoin](#Bitcoin)<br>  \n    * [Ethereum](#Ethereum)<br> \n    * [Discussion on Random-Walk](#Discussion-on-Random-Walk)<br> \n    \nIf you need some introduction to or a refresher on the ACF and PACF, I recommend the following video:\n<iframe width=\"700\" height=\"400\" src=\"https://www.youtube.com/embed/DeORzP0go5I\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nfrom numpy.random import seed \nimport math \n\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom datetime import datetime, date \n\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 14})\nimport seaborn as sns\n\nimport warnings # Supress warnings \nwarnings.filterwarnings('ignore')\n\nimport statsmodels as sm\nfrom statsmodels.tsa.ar_model import AutoReg\nfrom statsmodels.tsa.arima_model import ARMA\n\n# Fix seed for reproducible results\nSEED = 42\nnp.random.seed(SEED)\n\n# Visualizations\nlag_acf = 15\nlag_pacf = 15\nheight = 4\nwidth = 12","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:09.931769Z","iopub.execute_input":"2021-11-29T13:15:09.932279Z","iopub.status.idle":"2021-11-29T13:15:11.157423Z","shell.execute_reply.started":"2021-11-29T13:15:09.93217Z","shell.execute_reply":"2021-11-29T13:15:11.1567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fundamentals\n\n## Auto-Regressive and Moving Average Models\n\n### Auto-Regressive (AR) Model\n\n$\\hat{y}_t = \\alpha_1 y_{t-1} + \\dots + {\\alpha_p}y_{t-p}$\n\nThe AR model assumes that the current value ($y_t$) is **dependent on previous values** ($y_{t-1}, y_{t-2}, y_{t-3},...$). Because of this assumption, we can build a **linear** regression model.\n\nTo figure out the order of an AR model, you would use the **PACF**.\n\n### Moving Average (MA) Model\n\n$\\hat{y}_t = \\epsilon_t + \\beta_1 \\epsilon_{t-1} + \\dots + \\beta_{q} \\epsilon_{t-q}$\n\nThe MA model assumes that the current value ($y_t$) is **dependent on the error terms** including the current error ($\\epsilon_{t}, \\epsilon_{t-1}, \\epsilon_{t-2}, \\epsilon_{t-3},...$). Because error terms are random, there is **no linear** relationship between the current value and the error terms.\n\nTo figure out the order of an MA model, you would use the **ACF**.\n\n## Stationarity\n\nACF and PACF assume stationarity of the underlying time series.\nStaionarity can be checked by performing an **Augmented Dickey-Fuller (ADF) test**:\n\n> - p-value > 0.05: Fail to reject the null hypothesis (H0), the data has a unit root and is non-stationary.\n> - p-value <= 0.05: Reject the null hypothesis (H0), the data does not have a unit root and is stationary.\n>\n> [...] We can see that our [ADF] statistic value [...] is less than the value [...] at 1%.\nThis suggests that we can reject the null hypothesis with a significance level of less than 1% (i.e. a low probability that the result is a statistical fluke).\nRejecting the null hypothesis means that the process has no unit root, and in turn that the time series is stationary or does not have time-dependent structure. - [Machine Learning Mastery: How to Check if Time Series Data is Stationary with Python](https://machinelearningmastery.com/time-series-data-stationary-python/)\n\nIf the time series is stationary, continue to the next steps.\n**If the time series is not stationary, try differencing the time series** and check its stationarity again.\n","metadata":{}},{"cell_type":"code","source":"from statsmodels.tsa.stattools import adfuller\n\ndef check_stationarity(series):\n    # Copied from https://machinelearningmastery.com/time-series-data-stationary-python/\n\n    result = adfuller(series.values)\n\n    print('ADF Statistic: %f' % result[0])\n    print('p-value: %f' % result[1])\n    print('Critical Values:')\n    for key, value in result[4].items():\n        print('\\t%s: %.3f' % (key, value))\n\n    if (result[1] <= 0.05) & (result[4]['5%'] > result[0]):\n        print(\"\\u001b[32mStationary\\u001b[0m\")\n    else:\n        print(\"\\x1b[31mNon-stationary\\x1b[0m\")","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:11.160001Z","iopub.execute_input":"2021-11-29T13:15:11.160331Z","iopub.status.idle":"2021-11-29T13:15:11.176247Z","shell.execute_reply.started":"2021-11-29T13:15:11.160304Z","shell.execute_reply":"2021-11-29T13:15:11.175535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Autocorrelation Function and Partial Autocorrelation Function\n\n### Autocorrelation Function (ACF)\n\nCorrelation between time series with a lagged version of itself. The correlation between the observation at the current time spot and the observations at previous time spots.The autocorrelation function starts a lag 0, which is the correlation of the time series with itself and therefore results in a correlation of 1.\n\nWe will be using the `plot_acf` function from the `statsmodels.graphics.tsaplots` library. (See [statsmodels.tsa.stattools.acf](https://www.statsmodels.org/dev/generated/statsmodels.tsa.stattools.acf.html))\n\nThe ACF plot can provide answers to the following questions:\n- Is the observed time series **white noise / random**?\n- Is an observation related to an adjacent observation, an observation twice-removed, and so on? \n- Can the observed time series be modeled with an **MA model**? If yes, what is the order?\n\n\n\n### Partial Autocorrelation Function (PACF)\nAdditional correlation explained by each successive lagged term. The correlation between pbservations at two time spots given that we consider both observations are correlated to observations at other time spots.\n> The partial autocorrelation at lag k is the autocorrelation between $X_t$ and $X_{t−k}$ that is not accounted for by lags 1 through $k−1$. \n\nWe will be using the `plot_pacf` function from the `statsmodels.graphics.tsaplots` library with the parameter `method = \"ols\"` (regression of time series on lags of it and on constant). (See [statsmodels.tsa.stattools.pacf](https://www.statsmodels.org/dev/generated/statsmodels.tsa.stattools.pacf.html))\n\nSidenote: The default parameter for `method` is `yw` (Yule-Walker with sample-size adjustment in denominator for acovf). However, this default value is causing some implausible autocorrelations higher than 1 on the sample data. Therefore, we change the `method` parameter to one that is not causing this issue. `ywmle` would also work fine as suggested in this [StackExchange post](https://stats.stackexchange.com/questions/380196/what-do-very-high-pacf-values-10-mean)\n\nThe PACF plot can provide answers to the following questions:\n- Can the observed time series be modeled with an **AR model**? If yes, what is the order?","metadata":{}},{"cell_type":"code","source":"from statsmodels.graphics.tsaplots import plot_acf\nfrom statsmodels.graphics.tsaplots import plot_pacf","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:11.177684Z","iopub.execute_input":"2021-11-29T13:15:11.17801Z","iopub.status.idle":"2021-11-29T13:15:11.184077Z","shell.execute_reply.started":"2021-11-29T13:15:11.177971Z","shell.execute_reply":"2021-11-29T13:15:11.183261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Both the ACF and PACF start with a **lag of 0**, which is the correlation of the time series with itself and therefore results in a **correlation of 1**.\n\nThe difference between ACF and PACF is the inclusion or exclusion of indirect correlations in the calculation.\n\nFurthermore, you will see a **blue area** in the ACF and PACF plots, which depicts the 95% confidence interval and is in indicator for the **significance threshold**. That means, anything within the blue area is statistically close to zero and anything outside the blue area is statistically non-zero.\n\n## Order of AR, MA, and ARMA Model\nTo determine the order of the model, you can use the following table:\n\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) | Significant at lag $q$ / Cuts off after lag $q$|Tails off (Geometric decay) |\n|PACF| Significant at each lag $p$ / Cuts off after lag $p$|Tails off (Geometric decay) |Tails off (Geometric decay) |\n\nTo build some intuition about how to determine the order of an AR or MA model, I recommend this video:\n<iframe width=\"700\" height=\"400\" src=\"https://www.youtube.com/embed/ZE_WGBe0_VU\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>","metadata":{}},{"cell_type":"markdown","source":"# Examples\n\n## AR(1) Process\nThe following time series is an AR(1) process with 128 timesteps and the following parameters:","metadata":{}},{"cell_type":"code","source":"alpha_1 = 0.5","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:11.18658Z","iopub.execute_input":"2021-11-29T13:15:11.187539Z","iopub.status.idle":"2021-11-29T13:15:11.192346Z","shell.execute_reply.started":"2021-11-29T13:15:11.187494Z","shell.execute_reply":"2021-11-29T13:15:11.19167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_samples =  128\n\nnp.random.seed(SEED)\nar = np.r_[1, -np.array([alpha_1])] # add zero-lag and negate\nma = np.r_[1] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:11.193977Z","iopub.execute_input":"2021-11-29T13:15:11.194586Z","iopub.status.idle":"2021-11-29T13:15:11.684977Z","shell.execute_reply.started":"2021-11-29T13:15:11.194544Z","shell.execute_reply":"2021-11-29T13:15:11.684155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is stationary. Therefore, we do not need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:11.686209Z","iopub.execute_input":"2021-11-29T13:15:11.687031Z","iopub.status.idle":"2021-11-29T13:15:11.712282Z","shell.execute_reply.started":"2021-11-29T13:15:11.686988Z","shell.execute_reply":"2021-11-29T13:15:11.711569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There are several autocorrelations that are significantly non-zero. Therefore, the time series is non-random.\n- High degree of autocorrelation between adjacent (lag = 1)\n\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) |Significant at lag $q$ / Cuts off after lag $q$  |Tails off (Geometric decay) |\n|PACF| <span style= 'background:yellow'> Significant at each lag $p$ / Cuts off after lag $p$ </span>|Tails off (Geometric decay) |Tails off (Geometric decay) |\n\n-> We can use an **AR(1) model** to model this process.\n\nSo that for AR(1), we would model the AR(p) formula\n$\\hat{y}_t = \\alpha_1 y_{t-1} + \\dots + {\\alpha_p}y_{t-p}$\nto the following:\n\n$\\hat{y}_t = \\alpha_1 y_{t-1}$\n","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n\nax[1].annotate('Strong correlation at lag = 1', xy=(1, 0.6),  xycoords='data',\n            xytext=(0.17, 0.75), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:11.714967Z","iopub.execute_input":"2021-11-29T13:15:11.715312Z","iopub.status.idle":"2021-11-29T13:15:12.224147Z","shell.execute_reply.started":"2021-11-29T13:15:11.715267Z","shell.execute_reply":"2021-11-29T13:15:12.223307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling","metadata":{}},{"cell_type":"code","source":"train_len = int(0.8* num_samples)\n\ntrain = sample['t'][:train_len]\nar_model = AutoReg(train, lags=1).fit()\n\nprint(ar_model.summary())\npred = ar_model.predict(start=train_len, end=num_samples, dynamic=False)\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=sample.t[train_len:num_samples], marker='o', label='test', color='grey')\nsns.lineplot(x=sample.timestamp[:train_len], y=train, marker='o', label='train')\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=pred, marker='o', label='pred')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:12.225436Z","iopub.execute_input":"2021-11-29T13:15:12.226297Z","iopub.status.idle":"2021-11-29T13:15:12.663515Z","shell.execute_reply.started":"2021-11-29T13:15:12.22626Z","shell.execute_reply":"2021-11-29T13:15:12.662711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the AR(1) model fits an $\\alpha_1 = 0.4710$, which is quite close to the `alpha_1 = 0.5` which we have set. However, the predicted values seem to be quite off in this case.","metadata":{}},{"cell_type":"markdown","source":"## AR(2) Process\nThe following time series is an AR(2) process with 128 timesteps and the following parameters:","metadata":{}},{"cell_type":"code","source":"alpha_1 = 0.5\nalpha_2 = -0.5","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:12.664604Z","iopub.execute_input":"2021-11-29T13:15:12.664847Z","iopub.status.idle":"2021-11-29T13:15:12.668904Z","shell.execute_reply.started":"2021-11-29T13:15:12.664805Z","shell.execute_reply":"2021-11-29T13:15:12.668106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(SEED)\nar = np.r_[1, -np.array([alpha_1, alpha_2])] # add zero-lag and negate\nma = np.r_[1] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:12.671724Z","iopub.execute_input":"2021-11-29T13:15:12.672001Z","iopub.status.idle":"2021-11-29T13:15:13.011695Z","shell.execute_reply.started":"2021-11-29T13:15:12.67197Z","shell.execute_reply":"2021-11-29T13:15:13.010854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is stationary. Therefore, we do not need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:13.013087Z","iopub.execute_input":"2021-11-29T13:15:13.013363Z","iopub.status.idle":"2021-11-29T13:15:13.029657Z","shell.execute_reply.started":"2021-11-29T13:15:13.013317Z","shell.execute_reply":"2021-11-29T13:15:13.02888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There are several autocorrelations that are significantly non-zero. Therefore, the time series is non-random.\n- High degree of autocorrelation between adjacent (lag = 1) and near-adjacent (lag = 2) observations\n\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) |Significant at lag $q$ / Cuts off after lag $q$  |Tails off (Geometric decay) |\n|PACF| <span style= 'background:yellow'> Significant at each lag $p$ / Cuts off after lag $p$ </span>|Tails off (Geometric decay) |Tails off (Geometric decay) |\n\n-> We can use an **AR(2) model** to model this process.\n\nSo that for AR(2), we would model the AR(p) formula\n$\\hat{y}_t = \\alpha_1 y_{t-1} + \\dots + {\\alpha_p}y_{t-p}$\nto the following:\n\n$\\hat{y}_t = \\alpha_1 y_{t-1} + \\alpha_2 y_{t-2} $\n","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n\nax[1].annotate('Strong correlation at lag = 1', xy=(1, 0.36),  xycoords='data',\n            xytext=(0.15, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nax[1].annotate('Strong correlation at lag = 2', xy=(2.1, -0.5),  xycoords='data',\n            xytext=(0.25, 0.1), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:13.031184Z","iopub.execute_input":"2021-11-29T13:15:13.031621Z","iopub.status.idle":"2021-11-29T13:15:13.627477Z","shell.execute_reply.started":"2021-11-29T13:15:13.031589Z","shell.execute_reply":"2021-11-29T13:15:13.626522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling","metadata":{}},{"cell_type":"code","source":"train = sample['t'][:train_len]\nar_model = AutoReg(train, lags=2).fit()\n\nprint(ar_model.summary())\npred = ar_model.predict(start=train_len, end=num_samples, dynamic=False)\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=sample.t[train_len:num_samples], marker='o', label='test', color='grey')\nsns.lineplot(x=sample.timestamp[:train_len], y=train, marker='o', label='train')\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=pred, marker='o', label='pred')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:13.628459Z","iopub.execute_input":"2021-11-29T13:15:13.628653Z","iopub.status.idle":"2021-11-29T13:15:14.019475Z","shell.execute_reply.started":"2021-11-29T13:15:13.628629Z","shell.execute_reply":"2021-11-29T13:15:14.01865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the AR(2) model fits $\\alpha_1 = 0.5191$ and $\\alpha_2 = -0.5855$, which is quite close to the `alpha_1 = 0.5` and `alpha_2 = -0.5` which we have set. However, the predicted values seem to be quite off as well in this case - similarly to the AR(1) case.","metadata":{}},{"cell_type":"markdown","source":"## MA(1) Process\nThe following time series is an MA(1) process with 128 timesteps and the following parameters:","metadata":{}},{"cell_type":"code","source":"beta_1 = 0.5","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:14.0207Z","iopub.execute_input":"2021-11-29T13:15:14.020957Z","iopub.status.idle":"2021-11-29T13:15:14.024798Z","shell.execute_reply.started":"2021-11-29T13:15:14.020927Z","shell.execute_reply":"2021-11-29T13:15:14.02404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(SEED)\nar = np.r_[1] # add zero-lag and negate\nma = np.r_[1, np.array([beta_1])] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:14.025772Z","iopub.execute_input":"2021-11-29T13:15:14.026102Z","iopub.status.idle":"2021-11-29T13:15:14.27252Z","shell.execute_reply.started":"2021-11-29T13:15:14.026075Z","shell.execute_reply":"2021-11-29T13:15:14.271689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is stationary. Therefore, we do not need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:14.273625Z","iopub.execute_input":"2021-11-29T13:15:14.273886Z","iopub.status.idle":"2021-11-29T13:15:14.288195Z","shell.execute_reply.started":"2021-11-29T13:15:14.273856Z","shell.execute_reply":"2021-11-29T13:15:14.287162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There are several autocorrelations that are significantly non-zero. Therefore, the time series is non-random.\n- High degree of autocorrelation between adjacent (lag = 1)\n\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) | <span style= 'background:yellow'> Significant at lag $q$ / Cuts off after lag $q$  </span> |Tails off (Geometric decay) |\n|PACF| Significant at each lag $p$ / Cuts off after lag $p$ |Tails off (Geometric decay) |Tails off (Geometric decay) |\n\n-> We can use an **MA(1) model** to model this process.\n\nSo that for MA(1), we would model the MA(q) formula\n$\\hat{y}_t = \\epsilon_t + \\beta_1 \\epsilon_{t-1} + \\dots + \\beta_{q} \\epsilon_{t-q}$\nto the following:\n\n$\\hat{y}_t = \\epsilon_t + \\beta_1 \\epsilon_{t-1}$\n","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n\nax[0].annotate('Strong correlation at lag = 1', xy=(1, 0.5),  xycoords='data',\n            xytext=(0.15, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:14.289126Z","iopub.execute_input":"2021-11-29T13:15:14.289878Z","iopub.status.idle":"2021-11-29T13:15:14.704449Z","shell.execute_reply.started":"2021-11-29T13:15:14.289837Z","shell.execute_reply":"2021-11-29T13:15:14.703629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling","metadata":{}},{"cell_type":"code","source":"train = sample['t'][:train_len]\nma_model = ARMA(train, order=(0,1)).fit()\n\nprint(ma_model.summary())\npred = ma_model.predict(start=train_len, end=num_samples, dynamic=False)\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=sample.t[train_len:num_samples], marker='o', label='test', color='grey')\nsns.lineplot(x=sample.timestamp[:train_len], y=train, marker='o', label='train')\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=pred, marker='o', label='pred')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:14.705631Z","iopub.execute_input":"2021-11-29T13:15:14.705877Z","iopub.status.idle":"2021-11-29T13:15:15.202919Z","shell.execute_reply.started":"2021-11-29T13:15:14.705842Z","shell.execute_reply":"2021-11-29T13:15:15.202126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the MA(1) model fits $\\beta_1 = 0.5172$, which is quite close to the `beta_1 = 0.5`. However, the predicted values seem to be quite off as well in this case - similarly to the AR(p) cases.","metadata":{}},{"cell_type":"markdown","source":"## MA(2) Process\nThe following time series is an MA(2) process with 128 timesteps and the following parameters:","metadata":{}},{"cell_type":"code","source":"beta_1 = 0.5\nbeta_2 = 0.5","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:15.204056Z","iopub.execute_input":"2021-11-29T13:15:15.204291Z","iopub.status.idle":"2021-11-29T13:15:15.208597Z","shell.execute_reply.started":"2021-11-29T13:15:15.204263Z","shell.execute_reply":"2021-11-29T13:15:15.207401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(SEED)\nar = np.r_[1] # add zero-lag and negate\nma = np.r_[1, np.array([beta_1, beta_2])] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:15.210133Z","iopub.execute_input":"2021-11-29T13:15:15.211201Z","iopub.status.idle":"2021-11-29T13:15:15.537905Z","shell.execute_reply.started":"2021-11-29T13:15:15.211164Z","shell.execute_reply":"2021-11-29T13:15:15.537073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is stationary. Therefore, we do not need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:15.538927Z","iopub.execute_input":"2021-11-29T13:15:15.539135Z","iopub.status.idle":"2021-11-29T13:15:15.552632Z","shell.execute_reply.started":"2021-11-29T13:15:15.53911Z","shell.execute_reply":"2021-11-29T13:15:15.551677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There are several autocorrelations that are significantly non-zero. Therefore, the time series is non-random.\n- High degree of autocorrelation between adjacent (lag = 1) and near-adjacent (lag = 2) observations\n\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) | <span style= 'background:yellow'> Significant at lag $q$ / Cuts off after lag $q$  </span> |Tails off (Geometric decay) |\n|PACF| Significant at each lag $p$ / Cuts off after lag $p$ |Tails off (Geometric decay) |Tails off (Geometric decay) |\n\n-> We can use an **MA(2) model** to model this process.\n\nSo that for MA(2), we would model the MA(q) formula\n$\\hat{y}_t = \\epsilon_t + \\beta_1 \\epsilon_{t-1} + \\dots + \\beta_{q} \\epsilon_{t-q}$\nto the following:\n\n$\\hat{y}_t = \\epsilon_t + \\beta_1 \\epsilon_{t-1} + \\beta_2 \\epsilon_{t-2}$","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n\nax[0].annotate('Strong correlation at lag = 1', xy=(1, 0.65),  xycoords='data',\n            xytext=(0.15, 0.8), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nax[0].annotate('Strong correlation at lag = 2', xy=(2, 0.5),  xycoords='data',\n            xytext=(0.25, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:15.554073Z","iopub.execute_input":"2021-11-29T13:15:15.554598Z","iopub.status.idle":"2021-11-29T13:15:16.133123Z","shell.execute_reply.started":"2021-11-29T13:15:15.554557Z","shell.execute_reply":"2021-11-29T13:15:16.132284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling","metadata":{}},{"cell_type":"code","source":"train = sample['t'][:train_len]\nma_model = ARMA(train, order=(0,2)).fit()\n\nprint(ma_model.summary())\npred = ma_model.predict(start=train_len, end=num_samples, dynamic=False)\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=sample.t[train_len:num_samples], marker='o', label='test', color='grey')\nsns.lineplot(x=sample.timestamp[:train_len], y=train, marker='o', label='train')\nsns.lineplot(x=sample.timestamp[train_len:num_samples], y=pred, marker='o', label='pred')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:16.134899Z","iopub.execute_input":"2021-11-29T13:15:16.135438Z","iopub.status.idle":"2021-11-29T13:15:16.670818Z","shell.execute_reply.started":"2021-11-29T13:15:16.135397Z","shell.execute_reply":"2021-11-29T13:15:16.669893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the MA(2) model fits $\\beta_1 = 0.5226$ and $\\beta_2 = -0.5843$, which is quite close to the `beta_1 = 0.5` and `beta_2 = 0.5` which we have set. However, the predicted values seem to be quite off as well in this case - similarly to the MA(1) case.","metadata":{}},{"cell_type":"markdown","source":"## Periodical\nThe following time series is periodical with T=12. It consists of 48 timesteps.","metadata":{}},{"cell_type":"code","source":"T = 12\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : [1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2]\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:16.672042Z","iopub.execute_input":"2021-11-29T13:15:16.672266Z","iopub.status.idle":"2021-11-29T13:15:17.180742Z","shell.execute_reply.started":"2021-11-29T13:15:16.672239Z","shell.execute_reply":"2021-11-29T13:15:17.17919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is stationary. Therefore, we do not need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:17.183374Z","iopub.execute_input":"2021-11-29T13:15:17.183752Z","iopub.status.idle":"2021-11-29T13:15:17.201129Z","shell.execute_reply.started":"2021-11-29T13:15:17.183716Z","shell.execute_reply":"2021-11-29T13:15:17.199962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There are several autocorrelations that are significantly non-zero. Therefore, the time series is non-random.\n- High degree of autocorrelation between adjacent (lag = 1) and near-adjacent observations\n- From both the ACF and PACF plot, we can see a strong correlation with the adjacent observation (lag = 1) and also at a lag of 12, which is the amount of T.\n\n\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) |Significant at lag $q$ / Cuts off after lag $q$  |Tails off (Geometric decay) |\n|PACF| <span style= 'background:yellow'> Significant at each lag $p$ / Cuts off after lag $p=12$ </span>|Tails off (Geometric decay) |Tails off (Geometric decay) |\n\n-> We can use an AR(12) model to model this process.","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n\nfor i in range(2):\n    ax[i].axvline(x=T, color='r', linestyle='--')\n    ax[i].set_xlim([-0.5, lag_acf+0.5])\n    ax[i].set_ylim([-1.1, 1.1])\n    \nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:17.203286Z","iopub.execute_input":"2021-11-29T13:15:17.204184Z","iopub.status.idle":"2021-11-29T13:15:17.652198Z","shell.execute_reply.started":"2021-11-29T13:15:17.204137Z","shell.execute_reply":"2021-11-29T13:15:17.651329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling","metadata":{}},{"cell_type":"code","source":"train = sample['t'][:26]\nar_model = AutoReg(train, lags=12).fit()\n\nprint(ar_model.summary())\npred = ar_model.predict(start=26, end=48, dynamic=False)","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:17.653338Z","iopub.execute_input":"2021-11-29T13:15:17.653648Z","iopub.status.idle":"2021-11-29T13:15:17.681497Z","shell.execute_reply.started":"2021-11-29T13:15:17.653616Z","shell.execute_reply":"2021-11-29T13:15:17.680647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\nsns.lineplot(x=sample.timestamp[26:48], y=sample.t[26:48], marker='o', label='test', color='grey')\nsns.lineplot(x=sample.timestamp[:26], y=train, marker='o', label='train')\nsns.lineplot(x=sample.timestamp[26:48], y=pred, marker='o', label='pred')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:17.685659Z","iopub.execute_input":"2021-11-29T13:15:17.685917Z","iopub.status.idle":"2021-11-29T13:15:18.138335Z","shell.execute_reply.started":"2021-11-29T13:15:17.685887Z","shell.execute_reply":"2021-11-29T13:15:18.13763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the AR(12) model fits the periodical time series perfectly with $\\alpha_{1...11} = -0.0004$ and $\\alpha_12 = 0.9996$. \n\nThe model $\\hat{y}_t = \\alpha_1 y_{t-1} + \\dots + {\\alpha_p}y_{t-p}$ with $p=12$ results in:\n$\\hat{y}_t = \\alpha_1 y_{t-1} + \\alpha_2 y_{t-2} + \\alpha_3 y_{t-3} + \\alpha_4 y_{t-4} + \\alpha_5 y_{t-5} + \\alpha_6 y_{t-6} + \\alpha_7 y_{t-7} + \\alpha_8 y_{t-8} + \\alpha_9 y_{t-9} + \\alpha_{10} y_{t-10} + \\alpha_{11} y_{t-11} + {\\alpha_{12}}y_{t-12}$\n\nAnd with the coefficients $\\alpha_{1...11} = -0.0004 \\approx 0 $ and $\\alpha_{12} = 0.9996 \\approx 1$, we get:\n\n$\\hat{y}_t = y_{t-12}$","metadata":{}},{"cell_type":"markdown","source":"## Trend\n\nThe following time series is the same as [Periodical](#Periodical) (periodical with T=12) with added trend. It consists of 48 timesteps.","metadata":{}},{"cell_type":"code","source":"time = np.arange(0, 48)\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : [1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2] + ((0.05*time)+20)\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:18.139594Z","iopub.execute_input":"2021-11-29T13:15:18.139836Z","iopub.status.idle":"2021-11-29T13:15:18.499428Z","shell.execute_reply.started":"2021-11-29T13:15:18.139794Z","shell.execute_reply":"2021-11-29T13:15:18.498493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is non-stationary. ","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:18.500605Z","iopub.execute_input":"2021-11-29T13:15:18.500848Z","iopub.status.idle":"2021-11-29T13:15:18.513122Z","shell.execute_reply.started":"2021-11-29T13:15:18.500801Z","shell.execute_reply":"2021-11-29T13:15:18.512101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Therefore, we need to difference the time series.","metadata":{}},{"cell_type":"code","source":"sample['t_diff'] = sample['t'].diff().fillna(0)\n\ndisplay(sample.head(5).style.set_caption('Sample Time Series'))\n\ncheck_stationarity(sample['t_diff'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:18.516251Z","iopub.execute_input":"2021-11-29T13:15:18.516503Z","iopub.status.idle":"2021-11-29T13:15:18.597165Z","shell.execute_reply.started":"2021-11-29T13:15:18.516473Z","shell.execute_reply":"2021-11-29T13:15:18.596311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(12, 8))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[0])\nax[0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\n\nax[0].set_title('Sample Time Series')\n\nsns.lineplot(x=sample.timestamp, y=sample['t_diff'], marker='o', ax=ax[1])\nax[1].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[1].set_title('Differenced Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:18.599036Z","iopub.execute_input":"2021-11-29T13:15:18.599356Z","iopub.status.idle":"2021-11-29T13:15:19.129598Z","shell.execute_reply.started":"2021-11-29T13:15:18.599314Z","shell.execute_reply":"2021-11-29T13:15:19.128778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There are several autocorrelations that are significantly non-zero. Therefore, the time series is non-random.\n- High degree of autocorrelation between adjacent (lag = 1) and near-adjacent observations","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t_diff'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t_diff'],lags=lag_pacf, ax=ax[1], method='ols')\n\nfor i in range(2):\n    ax[i].axvline(x=T, color='r', linestyle='--')\n    ax[i].set_xlim([-0.5, lag_acf+0.5])\n    ax[i].set_ylim([-1.1, 1.1])\n    \nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:19.130894Z","iopub.execute_input":"2021-11-29T13:15:19.131155Z","iopub.status.idle":"2021-11-29T13:15:19.465653Z","shell.execute_reply.started":"2021-11-29T13:15:19.131124Z","shell.execute_reply":"2021-11-29T13:15:19.464838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling","metadata":{}},{"cell_type":"markdown","source":"## White Noise\n\nThe following time series is random. It consists of 48 timesteps.","metadata":{}},{"cell_type":"code","source":"sample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : np.random.randint(1,101,len(time))\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:19.466868Z","iopub.execute_input":"2021-11-29T13:15:19.467109Z","iopub.status.idle":"2021-11-29T13:15:19.822882Z","shell.execute_reply.started":"2021-11-29T13:15:19.467081Z","shell.execute_reply":"2021-11-29T13:15:19.822091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is stationary. Therefore, we do not need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:19.823985Z","iopub.execute_input":"2021-11-29T13:15:19.824191Z","iopub.status.idle":"2021-11-29T13:15:19.836083Z","shell.execute_reply.started":"2021-11-29T13:15:19.824166Z","shell.execute_reply":"2021-11-29T13:15:19.835437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- There is only one autocorrelation that is significantly non-zero at a lag of 0. Therefore, the time series is random.","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n    \nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:19.837323Z","iopub.execute_input":"2021-11-29T13:15:19.837766Z","iopub.status.idle":"2021-11-29T13:15:20.295112Z","shell.execute_reply.started":"2021-11-29T13:15:19.837737Z","shell.execute_reply":"2021-11-29T13:15:20.2943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling\n\nModelling white noise is difficult because we cannot retrieve any parameters from the ACF and PACF plots.","metadata":{}},{"cell_type":"markdown","source":"## Random-Walk\n\nThe following time series is random like [White Noise](#White-Noise). However, the current value depends on the previous one. It consists of 48 timesteps.","metadata":{}},{"cell_type":"code","source":"# Copied from https://towardsdatascience.com/time-series-from-scratch-white-noise-and-random-walk-5c96270514d3\n# Start with a random number - let's say 0\nnp.random.seed(42)\n\nrandom_walk = [0]\n\nfor i in range(1, 48):\n    # Movement direction based on a random number\n    num = -1 if np.random.random() < 0.5 else 1\n    random_walk.append(random_walk[-1] + num)\n    \nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : random_walk\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:20.296303Z","iopub.execute_input":"2021-11-29T13:15:20.296536Z","iopub.status.idle":"2021-11-29T13:15:20.647398Z","shell.execute_reply.started":"2021-11-29T13:15:20.296506Z","shell.execute_reply":"2021-11-29T13:15:20.646544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is non-stationary. Therefore, we need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:20.648992Z","iopub.execute_input":"2021-11-29T13:15:20.649637Z","iopub.status.idle":"2021-11-29T13:15:20.662537Z","shell.execute_reply.started":"2021-11-29T13:15:20.649594Z","shell.execute_reply":"2021-11-29T13:15:20.661495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample['t_diff'] = sample['t'].diff().fillna(0)\n\ncheck_stationarity(sample['t_diff'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:20.663926Z","iopub.execute_input":"2021-11-29T13:15:20.664164Z","iopub.status.idle":"2021-11-29T13:15:20.679432Z","shell.execute_reply.started":"2021-11-29T13:15:20.664136Z","shell.execute_reply":"2021-11-29T13:15:20.678656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\nWe can make the following observations:\n- In contrast to the ACF and PACF of the [White Noise](#White-Noise) process, we can see that for a lag of 1, we have some light correlation because the current value depends on the previous one.","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n\nax[1].annotate('Correlation to adjacent observation', xy=(1, 0.9),  xycoords='data',\n            xytext=(0.15, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:20.680525Z","iopub.execute_input":"2021-11-29T13:15:20.680769Z","iopub.status.idle":"2021-11-29T13:15:21.191394Z","shell.execute_reply.started":"2021-11-29T13:15:20.680739Z","shell.execute_reply":"2021-11-29T13:15:21.190568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t_diff'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t_diff'],lags=lag_pacf, ax=ax[1], method='ols')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:21.192388Z","iopub.execute_input":"2021-11-29T13:15:21.1926Z","iopub.status.idle":"2021-11-29T13:15:21.644555Z","shell.execute_reply.started":"2021-11-29T13:15:21.192574Z","shell.execute_reply":"2021-11-29T13:15:21.643686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling\n\nSimilarly to white noise, modelling random-walk is difficult because we cannot retrieve any parameters from the ACF and PACF plots.","metadata":{}},{"cell_type":"markdown","source":"## Constant\n\nThe following time series is constant. It consists of 48 timesteps.","metadata":{}},{"cell_type":"code","source":"sample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : 5\n                      })\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o')\nax.set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax.set_title('Sample Time Series')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:21.645608Z","iopub.execute_input":"2021-11-29T13:15:21.645847Z","iopub.status.idle":"2021-11-29T13:15:21.99383Z","shell.execute_reply.started":"2021-11-29T13:15:21.645795Z","shell.execute_reply":"2021-11-29T13:15:21.992859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is non-stationary. Therefore, we need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(sample['t'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:21.99724Z","iopub.execute_input":"2021-11-29T13:15:21.997477Z","iopub.status.idle":"2021-11-29T13:15:22.01241Z","shell.execute_reply.started":"2021-11-29T13:15:21.99745Z","shell.execute_reply":"2021-11-29T13:15:22.011449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample['t_diff'] = sample['t'].diff().fillna(0)\n\ncheck_stationarity(sample['t_diff'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:22.013935Z","iopub.execute_input":"2021-11-29T13:15:22.014238Z","iopub.status.idle":"2021-11-29T13:15:22.028532Z","shell.execute_reply.started":"2021-11-29T13:15:22.014196Z","shell.execute_reply":"2021-11-29T13:15:22.027917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF\n\n- ACF/PACF was applied to non-stationary time series","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0])\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1], method='ols')\n    \nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:22.029328Z","iopub.execute_input":"2021-11-29T13:15:22.029547Z","iopub.status.idle":"2021-11-29T13:15:22.618798Z","shell.execute_reply.started":"2021-11-29T13:15:22.029512Z","shell.execute_reply":"2021-11-29T13:15:22.617837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Modelling\n\nModelling a constant as an AR or MA process is diffucult because we cannot retrieve any parameters from the ACF and PACF plots. But on the other hand, if you can retrieve that a time series is constant, it should not be too difficult to forecast it, right?","metadata":{}},{"cell_type":"markdown","source":"# 🚀 Cheat Sheet\n\n1. Check stationarity. If stationary, continue to step 2. Else use differencing until time series is stationary and then continue to step 2.\n2. Check ACF and PACF with following table\n| | AR($p$) | MA($q$) | ARMA($p$, $q$) |\n|-|-|-|-|\n|ACF|Tails off (Geometric decay) | Significant at lag $q$ / Cuts off after lag $q$|Tails off (Geometric decay) |\n|PACF| Significant at each lag $p$ / Cuts off after lag $p$|Tails off (Geometric decay) |Tails off (Geometric decay) |\n\n3. Modelling","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=10, ncols=3, figsize=(2*width, 10*height))\n\n### AR(1) ###\nnp.random.seed(SEED)\nar = np.r_[1, -np.array([alpha_1])] # add zero-lag and negate\nma = np.r_[1] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[0,0])\nax[0,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[0,0].set_title('Time Series for AR(1)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[0, 1], title='ACF for AR(1)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[0, 2], method='ols', title='PACF for AR(1)')\nax[0,2].annotate('Strong correlation at lag = 1', xy=(1, 0.6),  xycoords='data',\n            xytext=(0.17, 0.75), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\n### AR(2) ###\nnp.random.seed(SEED)\nar = np.r_[1, -np.array([alpha_1, alpha_2])] # add zero-lag and negate\nma = np.r_[1] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[1,0])\nax[1,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[1,0].set_title('Time Series for AR(2)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[1, 1], title='ACF for AR(2)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[1, 2], method='ols', title='PACF for AR(2)')\n\nax[1, 2].annotate('Strong correlation at lag = 1', xy=(1, 0.36),  xycoords='data',\n            xytext=(0.15, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nax[1, 2].annotate('Strong correlation at lag = 2', xy=(2.1, -0.5),  xycoords='data',\n            xytext=(0.25, 0.1), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\n### MA(1) ###\nnp.random.seed(SEED)\nar = np.r_[1] # add zero-lag and negate\nma = np.r_[1, np.array([beta_1])] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })    \n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[2,0])\nax[2,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[2,0].set_title('Time Series for MA(1)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[2, 1], title='ACF for MA(1)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[2, 2], method='ols', title='PACF for MA(1)')\n\nax[2,1].annotate('Strong correlation at lag = 1', xy=(1, 0.5),  xycoords='data',\n            xytext=(0.15, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\n### MA(2) ###\nnp.random.seed(SEED)\nar = np.r_[1] # add zero-lag and negate\nma = np.r_[1, np.array([beta_1, beta_2])] # add zero-lag\n\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=num_samples, freq='MS'),\n                       't' : sm.tsa.arima_process.arma_generate_sample(ar, ma, num_samples)\n                      })    \n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[3,0])\nax[3,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[3,0].set_title('Time Series for MA(2)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[3, 1], title='ACF for MA(2)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[3, 2], method='ols', title='PACF for MA(2)')\n\nax[3, 1].annotate('Strong correlation at lag = 1', xy=(1, 0.65),  xycoords='data',\n            xytext=(0.15, 0.8), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\nax[3, 1].annotate('Strong correlation at lag = 2', xy=(2, 0.5),  xycoords='data',\n            xytext=(0.25, 0.7), textcoords='axes fraction',\n            arrowprops=dict(color='red', shrink=0.05, width=1))\n\n### Periodical ###\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : [1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2, 1, 2, 3, 4, 4.5, 5, 7, 8, 6, 4, 2, 2]\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[4,0])\nax[4,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[4,0].set_title('Time Series for Periodical')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[4, 1], title='ACF for Periodical')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[4, 2], method='ols', title='PACF for Periodical')\n\nax[4,2].axvline(x=T, color='r', linestyle='--')\n\n### Trend ###\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : ((0.05*time)+20)\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[5,0])\nax[5,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[5,0].set_title('Time Series for Trend (NON-STATIONARY!)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[5, 1], title='ACF for Trend (applied to non-stationary)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[5, 2], method='ols', title='PACF for Trend (applied to non-stationary)')\n\n### White Noise ###\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : np.random.randint(1,101,len(time))\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[6,0])\nax[6,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[6,0].set_title('Time Series for White Noise')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[6, 1], title='ACF for White Noise')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[6, 2], method='ols', title='PACF for White Noise')\n\n### Random-Walk ###\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : random_walk\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[7,0])\nax[7,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[7,0].set_title('Time Series for Random-Walk (NON-STATIONARY!)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[7, 1], title='ACF for Random-Walk (applied to non-stationary)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[7, 2], method='ols', title='PACF for Random-Walk (applied to non-stationary)')\n\nsample['t_diff'] = sample['t'].diff().fillna(0)\n\nplot_acf(sample['t_diff'],lags=lag_acf, ax=ax[8, 1], title='ACF for Random-Walk (applied to differenced/stationary)')\nplot_pacf(sample['t_diff'],lags=lag_pacf, ax=ax[8, 2], method='ols', title='PACF for Random-Walk (applied to differenced/stationary)')\n\n\n### Constant ###\nsample = pd.DataFrame({'timestamp' : pd.date_range('2021-01-01', periods=48, freq='MS'),\n                       't' : 5\n                      })\n\nsns.lineplot(x=sample.timestamp, y=sample['t'], marker='o', ax=ax[9,0])\nax[9,0].set_xlim([sample.timestamp.iloc[0], sample.timestamp.iloc[-1]])\nax[9,0].set_title('Time Series for Constant (NON-STATIONARY!)')\n\nplot_acf(sample['t'],lags=lag_acf, ax=ax[9, 1], title='ACF for Constant (applied to non-stationary)')\nplot_pacf(sample['t'],lags=lag_pacf, ax=ax[9, 2], method='ols', title='PACF for Constant (applied to non-stationary)')\n\nfor i in range(9):\n    ax[i, 1].set_ylim([-1.1, 1.1])\n    ax[i, 2].set_ylim([-1.1, 1.1])\n\n    \nf.delaxes(ax[8, 0])\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:15:22.620223Z","iopub.execute_input":"2021-11-29T13:15:22.620464Z","iopub.status.idle":"2021-11-29T13:15:29.08463Z","shell.execute_reply.started":"2021-11-29T13:15:22.620435Z","shell.execute_reply":"2021-11-29T13:15:29.080798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Case Study\nNext, we will interpret the ACF and PACF plots in the [G-Research Crypto Forecasting Competition](https://www.kaggle.com/c/g-research-crypto-forecasting). To simplify this case study, we will **resample** the minute-wise data into daily samples and only have a look at the `Close` price.","metadata":{}},{"cell_type":"code","source":"data_folder = \"../input/g-research-crypto-forecasting/\"\n\ntrain = pd.read_csv(data_folder + 'train.csv', low_memory=False)\n\ntrain['timestamp'] = train['timestamp'].astype('datetime64[s]')","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:15:29.085889Z","iopub.execute_input":"2021-11-29T13:15:29.086139Z","iopub.status.idle":"2021-11-29T13:16:36.329672Z","shell.execute_reply.started":"2021-11-29T13:15:29.08611Z","shell.execute_reply":"2021-11-29T13:16:36.328786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Bitcoin","metadata":{}},{"cell_type":"code","source":"train_mini = train[train.Asset_ID == 1].reset_index()\n\ntrain_mini = train_mini[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\ntrain_mini = train_mini.to_frame().reset_index(drop=False)\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.lineplot(data=train_mini, x='timestamp', y='Close')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:16:36.33114Z","iopub.execute_input":"2021-11-29T13:16:36.331662Z","iopub.status.idle":"2021-11-29T13:16:37.155862Z","shell.execute_reply.started":"2021-11-29T13:16:36.33162Z","shell.execute_reply":"2021-11-29T13:16:37.15505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is non-stationary. Therefore, we need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(train_mini['Close'])\n\ntrain_mini['Close_diff'] = train_mini['Close'].diff().fillna(0)\n\ncheck_stationarity(train_mini['Close_diff'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:16:37.157008Z","iopub.execute_input":"2021-11-29T13:16:37.157232Z","iopub.status.idle":"2021-11-29T13:16:37.284103Z","shell.execute_reply.started":"2021-11-29T13:16:37.157203Z","shell.execute_reply":"2021-11-29T13:16:37.283126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.lineplot(data=train_mini, x='timestamp', y='Close_diff')\nplt.show()\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.histplot(train_mini['Close_diff'] )\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:16:37.285613Z","iopub.execute_input":"2021-11-29T13:16:37.286324Z","iopub.status.idle":"2021-11-29T13:16:38.258206Z","shell.execute_reply.started":"2021-11-29T13:16:37.286268Z","shell.execute_reply":"2021-11-29T13:16:38.25731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=2, figsize=(2*width, 2*height))\n\nplot_acf(train_mini['Close'], lags=20, ax=ax[0, 0], title='ACF on non-stationary')\nplot_pacf(train_mini['Close'], lags=20, ax=ax[0, 1], method='ols', title='PACF on non-stationary')\n\nplot_acf(train_mini['Close_diff'], lags=20, ax=ax[1, 0], title='ACF on differenced/stationary')\nplot_pacf(train_mini['Close_diff'], lags=20, ax=ax[1, 1], method='ols', title='PACF on differenced/stationary')\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:16:38.259655Z","iopub.execute_input":"2021-11-29T13:16:38.259965Z","iopub.status.idle":"2021-11-29T13:16:39.188618Z","shell.execute_reply.started":"2021-11-29T13:16:38.25993Z","shell.execute_reply":"2021-11-29T13:16:39.18773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ethereum","metadata":{}},{"cell_type":"code","source":"train_mini = train[train.Asset_ID == 6].reset_index()\n\ntrain_mini = train_mini[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\ntrain_mini = train_mini.to_frame().reset_index(drop=False)\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.lineplot(data=train_mini, x='timestamp', y='Close')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:16:39.190099Z","iopub.execute_input":"2021-11-29T13:16:39.190537Z","iopub.status.idle":"2021-11-29T13:16:39.905554Z","shell.execute_reply.started":"2021-11-29T13:16:39.190508Z","shell.execute_reply":"2021-11-29T13:16:39.904734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Check Stationarity\nThe sample data is non-stationary. Therefore, we need to difference the time series.","metadata":{}},{"cell_type":"code","source":"check_stationarity(train_mini['Close'])\n\ntrain_mini['Close_diff'] = train_mini['Close'].diff().fillna(0)\n\ncheck_stationarity(train_mini['Close_diff'])","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:16:39.907094Z","iopub.execute_input":"2021-11-29T13:16:39.907513Z","iopub.status.idle":"2021-11-29T13:16:40.047507Z","shell.execute_reply.started":"2021-11-29T13:16:39.907472Z","shell.execute_reply":"2021-11-29T13:16:40.046624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.lineplot(data=train_mini, x='timestamp', y='Close_diff')\nplt.show()\n\nf, ax = plt.subplots(nrows=1, ncols=1, figsize=(12,5))\n\nsns.histplot(train_mini['Close_diff'] )\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:16:40.049666Z","iopub.execute_input":"2021-11-29T13:16:40.050458Z","iopub.status.idle":"2021-11-29T13:16:41.571741Z","shell.execute_reply.started":"2021-11-29T13:16:40.050407Z","shell.execute_reply":"2021-11-29T13:16:41.570749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Check ACF and PACF","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(nrows=2, ncols=2, figsize=(2*width, 2*height))\n\nplot_acf(train_mini['Close'], lags=20, ax=ax[0, 0], title='ACF on non-stationary')\nplot_pacf(train_mini['Close'], lags=20, ax=ax[0, 1], method='ols', title='PACF on non-stationary')\n\nplot_acf(train_mini['Close_diff'], lags=20, ax=ax[1, 0], title='ACF on differenced/stationary')\nplot_pacf(train_mini['Close_diff'], lags=20, ax=ax[1, 1], method='ols', title='PACF on differenced/stationary')\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-29T13:16:41.57317Z","iopub.execute_input":"2021-11-29T13:16:41.573473Z","iopub.status.idle":"2021-11-29T13:16:42.300183Z","shell.execute_reply.started":"2021-11-29T13:16:41.573433Z","shell.execute_reply":"2021-11-29T13:16:42.29926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Discussion on Random-Walk\nFor both Bitcoin and Ethereum, we can observe a **random-walk** behavior (see [🚀 Cheat Sheet](#🚀-Cheat-Sheet)). This is fairly common in stock prices (see [Random Walk Theory](https://www.investopedia.com/terms/r/randomwalktheory.asp))\n\n> A **random walk is unpredictable**; it cannot reasonably be predicted.\n>\n> Given the way that the random walk is constructed, we can expect that the best prediction we could make would be to use the observation at the previous time step as what will happen in the next time step.\n>\n>Simply because we know that the next time step will be a function of the prior time step.\n>\n>This is often called the naive forecast, or a persistence model. - [A Gentle Introduction to the Random Walk for Times Series Forecasting with Python](https://machinelearningmastery.com/gentle-introduction-random-walk-times-series-forecasting-python/)\n\nWell, that's an unsatisfying finding. So, where do we go from here?","metadata":{}},{"cell_type":"markdown","source":"# References\n## My Time Series Forecasting Series\n\n* [Intro to Time Series Forecasting](https://www.kaggle.com/iamleonie/intro-to-time-series-forecasting)\n* [Time Series Forecasting: Building Intuition](https://www.kaggle.com/iamleonie/time-series-forecasting-building-intuition)\n* [Time Series Forecasting: Interpreting ACF and PACF](https://www.kaggle.com/iamleonie/time-series-interpreting-acf-and-pacf)\n* [Time Series Forecasting: Tips & Tricks for Training LSTMs](https://www.kaggle.com/iamleonie/time-series-tips-tricks-for-training-lstms)\n\n## Other Ressources\n- https://www.linkedin.com/pulse/reading-acf-pacf-plots-missing-manual-cheatsheet-saqib-ali/\n- https://www.itl.nist.gov/div898/handbook/eda/section3/autocopl.htm\n- https://mxplus3.medium.com/interpreting-autocorrelation-partial-autocorrelation-plots-for-time-series-analysis-23f87b102c64\n- https://towardsdatascience.com/time-series-from-scratch-white-noise-and-random-walk-5c96270514d3","metadata":{}}]}