{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Credits\n\n**This notebook is a fork of [the original notebook](https://www.kaggle.com/thebrownviking20/everything-you-can-do-with-a-time-series), hence all credits to the original author ([SIDDHARTH YADAV ](https://www.kaggle.com/thebrownviking20/) aka [@thebrownviking20](https://www.kaggle.com/thebrownviking20/)). Hence most of the content from the original author is preserved and in the same person as that written originally - great work and an inspiration to all.**\n\n# Why this notebook\n\nI noticed a few opportunities to improve the content and hence I'm forking it with the improvisations. A summary of the changes you will notice in this notebook as compared to [the original one]((https://www.kaggle.com/thebrownviking20/everything-you-can-do-with-a-time-series):\n- Using stocks data for all the examples, instead of a mix of stocks and humidity/pressure data\n- Showing this using cryptocurrencies like [BTC](https://www.coingecko.com/en/coins/bitcoin) and [ETH](https://www.coingecko.com/en/coins/ethereum)\n- Using an additional stocks and cryptocurrencies dataset\n- Made the comparisons between the two cryptocurrencies uniform across the notebook\n- Minor tweaks to fix data and visualisation issues\n- Shrink large datasets, load only sample of the full dataset using [dtype_diet](https://github.com/ianozsvald/dtype_diet) and subsetting datasets\n- Notebook updated regularly to reflect latest prices of the coins\n\n# Related notebooks\n\n- **[The original notebook](https://www.kaggle.com/thebrownviking20/everything-you-can-do-with-a-time-series)**\n- **[Everything you can do with a time series (stocks)](https://www.kaggle.com/neomatrix369/everything-you-can-do-with-a-time-series-stocks)**\n- **[Everything you can do with a time series (metals)](https://www.kaggle.com/neomatrix369/everything-you-can-do-with-a-time-series-metals)**\n\n# Aim\nSince my first week on this platform, I have been fascinated by the topic of **time series analysis.** This kernel is prepared to be a container of many broad topics in the field of time series analysis. My motive is to make this the ultimate reference to time series analysis for beginners and experienced people alike.\n\n# Some important things\n1. This kernel **is a work in progress so every time you see on your home feed and open it, you will surely find fresh content.**\n2. I am doing this only after completing various courses in this field. I continue to study more advanced concepts to provide more knowledge and content.\n3. If there is any suggestion or any specific topic you would like me to cover, kindly mention that in the comments.\n4. **If you like my work, be sure to upvote** (press the like button) this kernel so it looks more relevant and meaningful to the community.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true}},{"cell_type":"code","source":"# Importing libraries\nimport os\nimport warnings\nwarnings.filterwarnings('ignore')\nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nplt.style.use('fivethirtyeight') \n\n# Above is a special style template for matplotlib, highly useful for visualizing time series data\n%matplotlib inline\nfrom pylab import rcParams\nfrom plotly import tools\nimport plotly.plotly as py\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.figure_factory as ff\nimport statsmodels.api as sm\nfrom numpy.random import normal, seed\nfrom scipy.stats import norm\nfrom statsmodels.tsa.arima_model import ARMA\nfrom statsmodels.tsa.stattools import adfuller\nfrom statsmodels.graphics.tsaplots import plot_acf, plot_pacf\nfrom statsmodels.tsa.arima_process import ArmaProcess\nfrom statsmodels.tsa.arima_model import ARIMA\nimport math\nfrom sklearn.metrics import mean_squared_error\n\nrcParams['figure.figsize'] = 11, 9\n\nfrom datetime import datetime, timedelta\nimport gc\n\nprint(os.listdir(\"../input\"))","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_kg_hide-input":true,"_kg_hide-output":false,"scrolled":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:13.455655Z","iopub.execute_input":"2022-01-24T18:35:13.45592Z","iopub.status.idle":"2022-01-24T18:35:16.246898Z","shell.execute_reply.started":"2022-01-24T18:35:13.455885Z","shell.execute_reply":"2022-01-24T18:35:16.246089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%bash\n# https://pypi.org/project/yahoofinancials/\npip install yahoofinancials","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:16.24788Z","iopub.execute_input":"2022-01-24T18:35:16.24823Z","iopub.status.idle":"2022-01-24T18:35:30.318003Z","shell.execute_reply.started":"2022-01-24T18:35:16.248096Z","shell.execute_reply":"2022-01-24T18:35:30.317011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from yahoofinancials import YahooFinancials\nfrom joblib import Memory\n\nTMPDIR = '/tmp'\nmemory = Memory(TMPDIR, verbose=0)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:30.319684Z","iopub.execute_input":"2022-01-24T18:35:30.320052Z","iopub.status.idle":"2022-01-24T18:35:30.438847Z","shell.execute_reply.started":"2022-01-24T18:35:30.319986Z","shell.execute_reply":"2022-01-24T18:35:30.438066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"@memory.cache\ndef get_ticker_data(ticker: str, param_start_date, param_end_date) -> dict:\n    raw_data = YahooFinancials(ticker)\n    return raw_data.get_historical_price_data(param_start_date, param_end_date, \"daily\").copy()\n\ndef fetch_ticker_data(ticker: str, start_date, end_date) -> pd.DataFrame:\n    date_range = pd.bdate_range(start=start_date, end=end_date)\n    values = pd.DataFrame({'Date': date_range})\n    values['Date'] = pd.to_datetime(values['Date'])\n    raw_data = get_ticker_data(ticker, start_date, end_date)\n    return pd.DataFrame(raw_data[ticker][\"prices\"])[['date', 'open', 'high', 'low', 'adjclose', 'volume']]    ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:30.440073Z","iopub.execute_input":"2022-01-24T18:35:30.440351Z","iopub.status.idle":"2022-01-24T18:35:30.457803Z","shell.execute_reply.started":"2022-01-24T18:35:30.440309Z","shell.execute_reply":"2022-01-24T18:35:30.45686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- <a href='#1'>1. Introduction to date and time</a>\n    - <a href='#1.1'>1.1 Importing time series data</a>\n    - <a href='#1.2'>1.2 Cleaning and preparing time series data</a>\n    - <a href='#1.3'>1.3 Visualizing the datasets</a>\n    - <a href='#1.4'>1.4 Timestamps and Periods</a>\n    - <a href='#1.5'>1.5 Using date_range</a>\n    - <a href='#1.6'>1.6 Using to_datetime</a>\n    - <a href='#1.7'>1.7 Shifting and lags</a>\n    - <a href='#1.8'>1.8 Resampling</a>\n- <a href='#2'>2. Finance and Statistics</a>\n    - <a href='#2.1'>2.1 Percent change</a>\n    - <a href='#2.2'>2.2 Daily returns</a>\n    - <a href='#2.3'>2.3 Absolute change in successive rows</a>\n    - <a href='#2.4'>2.4 Comparing two or more time series</a>\n    - <a href='#2.5'>2.5 Window functions</a>\n    - <a href='#2.6'>2.6 OHLC charts</a>\n    - <a href='#2.7'>2.7 Candlestick charts</a>\n    - <a href='#2.8'>2.8 Autocorrelation and Partial Autocorrelation</a>\n- <a href='#3'>3. Time series decomposition and Random Walks</a>\n    - <a href='#3.1'>3.1 Trends, Seasonality and Noise</a>\n    - <a href='#3.2'>3.2 White Noise</a>\n    - <a href='#3.3'>3.3 Random Walk</a>\n    - <a href='#3.4'>3.4 Stationarity</a>\n- <a href='#4'>4. Modelling using statsmodels</a>\n    - <a href='#4.1'>4.1 AR models</a>\n    - <a href='#4.2'>4.2 MA models</a>\n    - <a href='#4.3'>4.3 ARMA models</a>\n    - <a href='#4.4'>4.4 ARIMA models</a>\n    - <a href='#4.5'>4.5 VAR models</a>\n    - <a href='#4.6'>4.6 State space methods</a>\n        - <a href='#4.6.1'>4.6.1 SARIMA models</a>\n        - <a href='#4.6.2'>4.6.2 Unobserved components</a>\n        - <a href='#4.6.3'>4.6.3 Dynamic Factor models</a>","metadata":{"_uuid":"3d798543e3b1a8df4989f8361f241b8f3ec00b0d"}},{"cell_type":"markdown","source":"# <a id='1'>1. Introduction to date and time</a>","metadata":{"_uuid":"c0dc336a9d4969910d85fbb9462f2dde0bb01dc3"}},{"cell_type":"markdown","source":"## <a id='1.1'>1.1 Importing time series data</a>","metadata":{"_uuid":"d8fd2907fd2d4ef86da73d65ba3d228dd56326a8","_kg_hide-input":true}},{"cell_type":"markdown","source":"### How to import data?\nFirst, we import all the datasets needed for this kernel. The required time series column is imported as a datetime column using **parse_dates** parameter and is also selected as index of the dataframe using **index_col** parameter. \n#### Data being used:-\n1. Bitcoin Cryptocurrency Data\n2. Ethereum Cryptocurrency Data","metadata":{"_uuid":"0a98331f0441b4365873a02f0c2208321ac959b0","_kg_hide-input":false}},{"cell_type":"code","source":"DATASET_SOURCE = 'LIVE' # or 'COMPETITION_DATASET'","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:30.459174Z","iopub.execute_input":"2022-01-24T18:35:30.46018Z","iopub.status.idle":"2022-01-24T18:35:30.468125Z","shell.execute_reply.started":"2022-01-24T18:35:30.459465Z","shell.execute_reply":"2022-01-24T18:35:30.467383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%bash\nrm -f dtype_diet.py\nwget https://raw.githubusercontent.com/ianozsvald/dtype_diet/master/dtype_diet.py","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:30.469437Z","iopub.execute_input":"2022-01-24T18:35:30.469889Z","iopub.status.idle":"2022-01-24T18:35:31.145528Z","shell.execute_reply.started":"2022-01-24T18:35:30.469847Z","shell.execute_reply":"2022-01-24T18:35:31.144606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import dtype_diet","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:31.147372Z","iopub.execute_input":"2022-01-24T18:35:31.147747Z","iopub.status.idle":"2022-01-24T18:35:31.155468Z","shell.execute_reply.started":"2022-01-24T18:35:31.147684Z","shell.execute_reply":"2022-01-24T18:35:31.154755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nif DATASET_SOURCE == 'LIVE':\n    pass # do othing\nelse:\n    cryptocurrencies = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\n    display(cryptocurrencies.info(memory_usage=\"deep\"))\n    dtype_diet.report_on_dataframe(cryptocurrencies)\n    cryptocurrencies['timestamp'] = cryptocurrencies['timestamp'].astype(np.int32)\n    cryptocurrencies['Asset_ID'] = cryptocurrencies['Asset_ID'].astype(np.int8)\n    cryptocurrencies['Count'] = cryptocurrencies['Count'].astype(np.float32)\n    display(cryptocurrencies.info(memory_usage=\"deep\"))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:31.1566Z","iopub.execute_input":"2022-01-24T18:35:31.156861Z","iopub.status.idle":"2022-01-24T18:35:31.174193Z","shell.execute_reply.started":"2022-01-24T18:35:31.156802Z","shell.execute_reply":"2022-01-24T18:35:31.172817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def extract_single_coin(source_dataset: pd.DataFrame, asset_id: int = 1, frequency: str = \"D\") -> pd.DataFrame:\n    filter_coin = source_dataset['Asset_ID'] == asset_id\n    coin_full = source_dataset[filter_coin].sort_values(by='timestamp').copy()\n    coin_full['DateTime'] = coin_full['timestamp'].apply(lambda x: datetime.fromtimestamp(x))\n    coin_full = coin_full.set_index('DateTime')\n    coin_full = coin_full.asfreq(frequency).interpolate().copy()\n    display(coin_full.info())\n    display(coin_full.head())\n    coin = coin_full.copy()\n    return coin_full, coin","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:31.175163Z","iopub.execute_input":"2022-01-24T18:35:31.175509Z","iopub.status.idle":"2022-01-24T18:35:31.19202Z","shell.execute_reply.started":"2022-01-24T18:35:31.175471Z","shell.execute_reply":"2022-01-24T18:35:31.191276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_date = (datetime.now() - timedelta(days=365*3)).strftime(\"%Y-%m-%d\")\nend_date = datetime.now().strftime(\"%Y-%m-%d\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:31.192941Z","iopub.execute_input":"2022-01-24T18:35:31.193317Z","iopub.status.idle":"2022-01-24T18:35:31.204528Z","shell.execute_reply.started":"2022-01-24T18:35:31.193279Z","shell.execute_reply":"2022-01-24T18:35:31.203739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nif DATASET_SOURCE == 'LIVE':\n    btc_full = fetch_ticker_data('BTC-USD', start_date, end_date)\n    btc_full.columns = ['DateTime', 'Open', 'High', 'Low', 'Close', 'Volume']\n    btc_full['DateTime'] = btc_full['DateTime'].apply(lambda x: datetime.fromtimestamp(x))\n    btc_full = btc_full.fillna(method=\"ffill\", axis=0)\n    btc_full = btc_full.fillna(method=\"bfill\", axis=0)\n    btc_full = btc_full.set_index('DateTime')\n    btc = btc_full.copy()\nelse:\n    btc_full, btc = extract_single_coin(cryptocurrencies, 1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:31.206318Z","iopub.execute_input":"2022-01-24T18:35:31.206919Z","iopub.status.idle":"2022-01-24T18:35:32.980845Z","shell.execute_reply.started":"2022-01-24T18:35:31.206615Z","shell.execute_reply":"2022-01-24T18:35:32.97994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nif DATASET_SOURCE == 'LIVE':\n    eth_full = fetch_ticker_data('ETH-USD', start_date, end_date)\n    eth_full.columns = ['DateTime', 'Open', 'High', 'Low', 'Close', 'Volume']\n    eth_full['DateTime'] = eth_full['DateTime'].apply(lambda x: datetime.fromtimestamp(x))\n    eth_full = eth_full.fillna(method=\"ffill\", axis=0)\n    eth_full = eth_full.fillna(method=\"bfill\", axis=0)    \n    eth_full = eth_full.set_index('DateTime')\n    eth = eth_full.copy()\nelse:\n    eth_full, eth = extract_single_coin(cryptocurrencies, 6)","metadata":{"_uuid":"3ded63a0ec9e973d605a44fbea7ec50bb8c85a90","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:32.981944Z","iopub.execute_input":"2022-01-24T18:35:32.982209Z","iopub.status.idle":"2022-01-24T18:35:34.667918Z","shell.execute_reply.started":"2022-01-24T18:35:32.982161Z","shell.execute_reply":"2022-01-24T18:35:34.667164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DATASET_SOURCE != 'LIVE':\n    del cryptocurrencies\n    gc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:34.669166Z","iopub.execute_input":"2022-01-24T18:35:34.669656Z","iopub.status.idle":"2022-01-24T18:35:34.674788Z","shell.execute_reply.started":"2022-01-24T18:35:34.669593Z","shell.execute_reply":"2022-01-24T18:35:34.674091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='1.2'>1.2 Cleaning and preparing time series data</a>","metadata":{"_uuid":"1dc01753c087bdb1968478ea172d3f49ded7d130","_kg_hide-input":false}},{"cell_type":"markdown","source":"### How to prepare data?\nBTC or ETH crytocurrencies data do not have any missing values ","metadata":{"_uuid":"f672f5e3ef334a129cd2d730ee1bc827f92b2e62","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='1.3'>1.3 Visualizing the datasets</a>","metadata":{"_uuid":"ed2ec5326c1771450a60efbdfc246b9f20495406","_kg_hide-input":false}},{"cell_type":"code","source":"if DATASET_SOURCE == 'LIVE':\n    selected_columns = ['Open', 'High', 'Low', 'Close', 'Volume']\nelse:\n    selected_columns = ['Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:34.676066Z","iopub.execute_input":"2022-01-24T18:35:34.676583Z","iopub.status.idle":"2022-01-24T18:35:34.68584Z","shell.execute_reply.started":"2022-01-24T18:35:34.676512Z","shell.execute_reply":"2022-01-24T18:35:34.685239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc[\"Close\"].asfreq('M').interpolate().plot() # asfreq method is used to convert a time series to a specified frequency. \n                                                 # Here it is monthly frequency. Also using interpolate() to fix the gaps between the timeseries\nplt.title('Closing price of BTC over time (Monthly frequency)')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:34.686927Z","iopub.execute_input":"2022-01-24T18:35:34.687423Z","iopub.status.idle":"2022-01-24T18:35:35.057537Z","shell.execute_reply.started":"2022-01-24T18:35:34.68738Z","shell.execute_reply":"2022-01-24T18:35:35.056566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc[selected_columns].plot(subplots=True, figsize=(10,12))\nplt.title('BTC coin attributes')\nplt.savefig('BTC.png')\nplt.show()","metadata":{"_uuid":"13599f97548a5b33fa28b016bff5152c17d49296","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:35.059309Z","iopub.execute_input":"2022-01-24T18:35:35.06Z","iopub.status.idle":"2022-01-24T18:35:36.324539Z","shell.execute_reply.started":"2022-01-24T18:35:35.059932Z","shell.execute_reply":"2022-01-24T18:35:36.323858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth[\"Close\"].asfreq('M').interpolate().plot() # asfreq method is used to convert a time series to a specified frequency. \n                                                 # Here it is monthly frequency. Also using interpolate() to fix the gaps between the timeseries\nplt.title('Closing price of ETH over time (Monthly frequency)')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:36.325408Z","iopub.execute_input":"2022-01-24T18:35:36.325608Z","iopub.status.idle":"2022-01-24T18:35:36.556037Z","shell.execute_reply.started":"2022-01-24T18:35:36.325578Z","shell.execute_reply":"2022-01-24T18:35:36.555247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth[selected_columns].plot(subplots=True, figsize=(10,12))\nplt.title('ETH coin attributes')\nplt.savefig('ETH.png')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:36.557323Z","iopub.execute_input":"2022-01-24T18:35:36.557773Z","iopub.status.idle":"2022-01-24T18:35:37.494804Z","shell.execute_reply.started":"2022-01-24T18:35:36.55771Z","shell.execute_reply":"2022-01-24T18:35:37.494039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='1.4'>1.4 Timestamps and Periods</a>","metadata":{"_uuid":"bd73a8237c04b14b3afcf1d9dc3f244753b35a8b","_kg_hide-input":false}},{"cell_type":"markdown","source":"### What are timestamps and periods and how are they useful?\nTimestamps are used to represent a point in time. Periods represent an interval in time. Periods can used to check if a specific event in the given period. They can also be converted to each other's form.","metadata":{"_uuid":"4c2ada0d73d7893f0899a8cfffcdfae75ee7b57b","_kg_hide-input":false}},{"cell_type":"code","source":"# Creating a Timestamp\ntimestamp = pd.Timestamp(2017, 1, 1, 12)\ntimestamp","metadata":{"_uuid":"cd04e48cfb1fdacb64417c73021f183b0f955402","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.496039Z","iopub.execute_input":"2022-01-24T18:35:37.496498Z","iopub.status.idle":"2022-01-24T18:35:37.503999Z","shell.execute_reply.started":"2022-01-24T18:35:37.496447Z","shell.execute_reply":"2022-01-24T18:35:37.503218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a period\nperiod = pd.Period('2017-01-01')\nperiod","metadata":{"_uuid":"b562cac2339475b197cff0c2f5a705145a930c71","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.505432Z","iopub.execute_input":"2022-01-24T18:35:37.505674Z","iopub.status.idle":"2022-01-24T18:35:37.518769Z","shell.execute_reply.started":"2022-01-24T18:35:37.505631Z","shell.execute_reply":"2022-01-24T18:35:37.517937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking if the given timestamp exists in the given period\nperiod.start_time < timestamp < period.end_time","metadata":{"_uuid":"d044ccabd1e1d1ebfd74c9732aadf080ca472f17","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.52004Z","iopub.execute_input":"2022-01-24T18:35:37.52029Z","iopub.status.idle":"2022-01-24T18:35:37.532284Z","shell.execute_reply.started":"2022-01-24T18:35:37.520251Z","shell.execute_reply":"2022-01-24T18:35:37.531568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting timestamp to period\nnew_period = timestamp.to_period(freq='H')\nnew_period","metadata":{"_uuid":"bd69b8ec26959d136a34f1e88811f2523fb2f96f","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.533807Z","iopub.execute_input":"2022-01-24T18:35:37.534271Z","iopub.status.idle":"2022-01-24T18:35:37.544078Z","shell.execute_reply.started":"2022-01-24T18:35:37.534199Z","shell.execute_reply":"2022-01-24T18:35:37.543228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting period to timestamp\nnew_timestamp = period.to_timestamp(freq='H', how='start')\nnew_timestamp","metadata":{"_uuid":"9f36e7636c6a826f19a38fc9c1956b34870d56e8","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.545421Z","iopub.execute_input":"2022-01-24T18:35:37.545902Z","iopub.status.idle":"2022-01-24T18:35:37.558946Z","shell.execute_reply.started":"2022-01-24T18:35:37.545838Z","shell.execute_reply":"2022-01-24T18:35:37.557875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='1.5'>1.5 Using date_range</a>","metadata":{"_uuid":"f87c7b3f17071c0ce6ed05ce91fe085642e5377d","_kg_hide-input":false}},{"cell_type":"markdown","source":"### What is date_range and how is it useful?\n**date_range** is a method that returns  a fixed frequency datetimeindex. It is quite useful when creating your own time series attribute for pre-existing data or arranging the whole data around the time series attribute created by you.","metadata":{"_uuid":"47cf8637f8c60e0bdbeb457f2af05aeeaaf763f9","_kg_hide-input":false}},{"cell_type":"code","source":"# Creating a datetimeindex with daily frequency\ndr1 = pd.date_range(start='1/1/18', end='1/9/18')\ndr1","metadata":{"_uuid":"922cd920cd5e85d571458010af8433b9340fad09","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.560056Z","iopub.execute_input":"2022-01-24T18:35:37.560454Z","iopub.status.idle":"2022-01-24T18:35:37.572056Z","shell.execute_reply.started":"2022-01-24T18:35:37.560398Z","shell.execute_reply":"2022-01-24T18:35:37.571215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a datetimeindex with monthly frequency\ndr2 = pd.date_range(start='1/1/18', end='1/1/19', freq='M')\ndr2","metadata":{"_uuid":"b346810a9d38a629dee9f9af9b079c4eb61ceecd","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.574048Z","iopub.execute_input":"2022-01-24T18:35:37.574448Z","iopub.status.idle":"2022-01-24T18:35:37.584067Z","shell.execute_reply.started":"2022-01-24T18:35:37.57438Z","shell.execute_reply":"2022-01-24T18:35:37.583308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a datetimeindex without specifying start date and using periods\ndr3 = pd.date_range(end='1/4/2014', periods=8)\ndr3","metadata":{"_uuid":"38263cc1ea8e719844ced552ce86056cd1a5f6b4","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.585144Z","iopub.execute_input":"2022-01-24T18:35:37.585477Z","iopub.status.idle":"2022-01-24T18:35:37.597806Z","shell.execute_reply.started":"2022-01-24T18:35:37.585435Z","shell.execute_reply":"2022-01-24T18:35:37.596948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a datetimeindex specifying start date , end date and periods\ndr4 = pd.date_range(start='2013-04-24', end='2014-11-27', periods=3)\ndr4","metadata":{"_uuid":"a07f4419201ba0b849ee5cfd097eeb6aa996beab","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.598908Z","iopub.execute_input":"2022-01-24T18:35:37.59914Z","iopub.status.idle":"2022-01-24T18:35:37.608892Z","shell.execute_reply.started":"2022-01-24T18:35:37.599091Z","shell.execute_reply":"2022-01-24T18:35:37.608268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='1.6'>1.6 Using to_datetime</a> ","metadata":{"_uuid":"c26aa5fc419a077276c3b24f52d1721245de94fe","_kg_hide-input":false}},{"cell_type":"markdown","source":"pandas.to_datetime() is used for converting arguments to datetime. Here, a DataFrame is converted to a datetime series.","metadata":{"_uuid":"821003a25d35c41cc9a33134549a1c46d6ca4b99","_kg_hide-input":false}},{"cell_type":"code","source":"df = pd.DataFrame({'year': [2015, 2016], 'month': [2, 3], 'day': [4, 5]})\ndf","metadata":{"_uuid":"20e1dcadccaf8572cf40511500ba6eb9a4e1cd7e","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.6101Z","iopub.execute_input":"2022-01-24T18:35:37.610393Z","iopub.status.idle":"2022-01-24T18:35:37.631534Z","shell.execute_reply.started":"2022-01-24T18:35:37.610342Z","shell.execute_reply":"2022-01-24T18:35:37.630759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.to_datetime(df)\ndf","metadata":{"_uuid":"caa46d29f6eebe46898524ce0afe821b58c41463","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.632545Z","iopub.execute_input":"2022-01-24T18:35:37.632935Z","iopub.status.idle":"2022-01-24T18:35:37.680641Z","shell.execute_reply.started":"2022-01-24T18:35:37.63287Z","shell.execute_reply":"2022-01-24T18:35:37.679771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.to_datetime('01-01-2017')\ndf","metadata":{"_uuid":"5efe3b52d2ac539c66f43f51e11571982f771f5b","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.682045Z","iopub.execute_input":"2022-01-24T18:35:37.682501Z","iopub.status.idle":"2022-01-24T18:35:37.689333Z","shell.execute_reply.started":"2022-01-24T18:35:37.682454Z","shell.execute_reply":"2022-01-24T18:35:37.688603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='1.7'>1.7 Shifting and lags</a>","metadata":{"_uuid":"9acb890fd5a021bc1a10ee19184425031d891f84","_kg_hide-input":false,"trusted":true}},{"cell_type":"markdown","source":"We can shift index by desired number of periods with an optional time frequency. This is useful when comparing the time series with a past of itself","metadata":{"_uuid":"16a55de3afba1b92503ff5484e45b39087e77f38","_kg_hide-input":false}},{"cell_type":"code","source":"btc[\"Close\"].asfreq('M').interpolate().plot(legend=True)\nshifted = btc[\"Close\"].asfreq('M').interpolate().shift(10).plot(legend=True)\nshifted.legend(['Close','Close_lagged'])\nplt.title('Closing price of BTC over time (Monthly frequency) - raw and lagged')\nplt.show()","metadata":{"_uuid":"8ace8c9bb3b3b57d568a78ce198f96300d66a333","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:37.69056Z","iopub.execute_input":"2022-01-24T18:35:37.691146Z","iopub.status.idle":"2022-01-24T18:35:38.08431Z","shell.execute_reply.started":"2022-01-24T18:35:37.691063Z","shell.execute_reply":"2022-01-24T18:35:38.083652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc[\"Volume\"].asfreq('M').interpolate().plot(legend=True)\nshifted = btc[\"Volume\"].asfreq('M').interpolate().shift(10).plot(legend=True)\nshifted.legend(['Volume','Volume_lagged'])\nplt.title('Volume of BTC over time (Monthly frequency) - raw and lagged')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.085292Z","iopub.execute_input":"2022-01-24T18:35:38.085675Z","iopub.status.idle":"2022-01-24T18:35:38.603546Z","shell.execute_reply.started":"2022-01-24T18:35:38.085619Z","shell.execute_reply":"2022-01-24T18:35:38.602721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='#1.8'>1.8 Resampling</a>\n**Upsampling** - Time series is resampled from low frequency to high frequency (Monthly to daily frequency). It involves filling or interpolating missing data\n\n**Downsampling** - Time series is resampled from high frequency to low frequency (Weekly to monthly frequency). It involves aggregation of existing data.\n","metadata":{"_uuid":"7848513aec3116b0b7909852f72835f0b56eceaf","_kg_hide-input":false}},{"cell_type":"markdown","source":"In case there were missing data, we would use **ffill** parameter which propagates last valid observation to fill gaps. Then we would use **bfill** to propogate next valid observation to fill gaps.\n\n_Forward filling missing data_\n\n```python\ndata = data.iloc[1:]\ndata = data.fillna(method='ffill')\n```\n\n_Back filling missing data_\n\n```python\ndata = data.fillna(method='bfill')\ndata.head()\n```","metadata":{"_uuid":"e3afdbbb50ac88e74c8d3f4666e8b1b583859046","_kg_hide-input":false}},{"cell_type":"code","source":"# Shape before resampling(downsampling)\nbtc.shape","metadata":{"_uuid":"25915cb8ab6be7fe3f5784bfff1a15c8afbe5076","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.604875Z","iopub.execute_input":"2022-01-24T18:35:38.605377Z","iopub.status.idle":"2022-01-24T18:35:38.611148Z","shell.execute_reply.started":"2022-01-24T18:35:38.605319Z","shell.execute_reply":"2022-01-24T18:35:38.610343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We downsample from hourly to 3 day frequency aggregated using mean\nbtc_resampled = btc.resample('3D').mean()\nbtc_resampled.head()","metadata":{"_uuid":"5ea60c3c26fadc2c03512be0e52dd91f4a5182a5","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.612374Z","iopub.execute_input":"2022-01-24T18:35:38.612841Z","iopub.status.idle":"2022-01-24T18:35:38.641969Z","shell.execute_reply.started":"2022-01-24T18:35:38.612778Z","shell.execute_reply":"2022-01-24T18:35:38.641286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shape after resampling(downsampling)\nbtc_resampled.shape","metadata":{"_uuid":"59c990d142724e51991985dd2377b328d2573fd8","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.643311Z","iopub.execute_input":"2022-01-24T18:35:38.64378Z","iopub.status.idle":"2022-01-24T18:35:38.649082Z","shell.execute_reply.started":"2022-01-24T18:35:38.643683Z","shell.execute_reply":"2022-01-24T18:35:38.6484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Much less rows are left. Now, we will upsample from 3 day frequency to daily frequency","metadata":{"_uuid":"1de21323b780dfc602ed1d2df04881a20a3a8cee","_kg_hide-input":false}},{"cell_type":"code","source":"btc_resampled = btc.resample('D').pad()\nbtc_resampled.head()","metadata":{"_uuid":"95f252356e497a3a456d37534dfd2bb97813b55f","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.650194Z","iopub.execute_input":"2022-01-24T18:35:38.650637Z","iopub.status.idle":"2022-01-24T18:35:38.675725Z","shell.execute_reply.started":"2022-01-24T18:35:38.650585Z","shell.execute_reply":"2022-01-24T18:35:38.674654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shape after resampling(upsampling)\nbtc_resampled.shape","metadata":{"_uuid":"83f9f0b8863ff524de1179ea77187fb4b0266154","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.676995Z","iopub.execute_input":"2022-01-24T18:35:38.677485Z","iopub.status.idle":"2022-01-24T18:35:38.683556Z","shell.execute_reply.started":"2022-01-24T18:35:38.677429Z","shell.execute_reply":"2022-01-24T18:35:38.682753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again an increase in number of rows. Resampling is cool when used properly.","metadata":{"_uuid":"18c6eb20d04c8af0a4900552dbfb18575990f310","_kg_hide-input":false}},{"cell_type":"markdown","source":"# <a id='2'>2. Finance and statistics</a>","metadata":{"_uuid":"72d0ea11c176f3731727b81269148188141c7dcc","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='2.1'>2.1 Percent change</a>","metadata":{"_uuid":"e99a77337a15dfbcc5ec97358c0d31d0ca04ab0d","_kg_hide-input":false}},{"cell_type":"code","source":"btc['Change'] = btc.High.div(btc.High.shift())\nbtc['Change'].plot(figsize=(20,8))\n_ = plt.title('Change in High price of BTC over time')","metadata":{"_uuid":"ee26cbd48ae1025b9a072a4385609d6abc0406d9","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:38.6847Z","iopub.execute_input":"2022-01-24T18:35:38.684977Z","iopub.status.idle":"2022-01-24T18:35:39.053104Z","shell.execute_reply.started":"2022-01-24T18:35:38.684923Z","shell.execute_reply":"2022-01-24T18:35:39.052209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc['Volume_Change'] = btc.Volume.div(btc.Volume.shift())\nbtc['Volume_Change'].plot(figsize=(20,8))\n_ = plt.title('Change in BTC Volume over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:39.054372Z","iopub.execute_input":"2022-01-24T18:35:39.054618Z","iopub.status.idle":"2022-01-24T18:35:39.419475Z","shell.execute_reply.started":"2022-01-24T18:35:39.054576Z","shell.execute_reply":"2022-01-24T18:35:39.418655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='2.2'>2.2 Daily returns</a> ","metadata":{"_uuid":"eed7547b522e9630ea2dc833afdee51e61ca41e6","_kg_hide-input":false}},{"cell_type":"code","source":"btc['Return'] = btc.Change.sub(1).mul(100)\nbtc['Return'].plot(figsize=(20,8))\n_ = plt.title('Return based on High price change of BTC over time')","metadata":{"_uuid":"09d5af0ce40cfdd1338433c204538fe894004b3f","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:39.420699Z","iopub.execute_input":"2022-01-24T18:35:39.42094Z","iopub.status.idle":"2022-01-24T18:35:39.790142Z","shell.execute_reply.started":"2022-01-24T18:35:39.420897Z","shell.execute_reply":"2022-01-24T18:35:39.78926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc.High.pct_change().mul(100).plot(figsize=(20,6)) # Another way to calculate returns\n_ = plt.title('Percentage change of High price of BTC over time')","metadata":{"_uuid":"f5f7a089b1ae64b4403c588f8095332bc32c0d11","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:39.79126Z","iopub.execute_input":"2022-01-24T18:35:39.791485Z","iopub.status.idle":"2022-01-24T18:35:40.125143Z","shell.execute_reply.started":"2022-01-24T18:35:39.791446Z","shell.execute_reply":"2022-01-24T18:35:40.124423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc.Volume.pct_change().mul(100).plot(figsize=(20,6)) # Another way to calculate returns\n_ = plt.title('Percentage change of BTC Volume over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:40.12608Z","iopub.execute_input":"2022-01-24T18:35:40.126449Z","iopub.status.idle":"2022-01-24T18:35:40.461446Z","shell.execute_reply.started":"2022-01-24T18:35:40.126409Z","shell.execute_reply":"2022-01-24T18:35:40.460474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='2.3'>2.3 Absolute change in successive rows</a>","metadata":{"_uuid":"1daf826b9b25c7f3c69683d9b59a7d2aa285df06","_kg_hide-input":false}},{"cell_type":"code","source":"btc.High.diff().plot(figsize=(20,6))\n_ = plt.title('Difference in High price change of BTC over time')","metadata":{"_uuid":"399bd0a9f469d69dd11523e54f1599302291be44","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:40.462893Z","iopub.execute_input":"2022-01-24T18:35:40.463263Z","iopub.status.idle":"2022-01-24T18:35:40.800514Z","shell.execute_reply.started":"2022-01-24T18:35:40.463223Z","shell.execute_reply":"2022-01-24T18:35:40.799675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc.Volume.diff().plot(figsize=(20,6))\n_ = plt.title('Difference in change in Volume of BTC over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:40.801525Z","iopub.execute_input":"2022-01-24T18:35:40.801879Z","iopub.status.idle":"2022-01-24T18:35:41.128077Z","shell.execute_reply.started":"2022-01-24T18:35:40.80184Z","shell.execute_reply":"2022-01-24T18:35:41.127125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='2.4'>2.4 Comparing two or more time series</a> ","metadata":{"_uuid":"bd5ec63ecd09543b21df15aff6432db598ee1ab5","_kg_hide-input":false,"trusted":true}},{"cell_type":"markdown","source":"We will compare 2 time series by normalizing them. This is achieved by dividing each time series element of all time series by the first element. This way both series start at the same point and can be easily compared.","metadata":{"_uuid":"7a7a4a918d1d80d19d33f71965a3ca5583a8aa2e","_kg_hide-input":false}},{"cell_type":"code","source":"# Plotting before normalization\nbtc.High.plot()\neth.High.plot()\nplt.legend(['BTC','ETH'])\n_ = plt.title('Comparing BTC and ETH over time')\nplt.show()","metadata":{"_uuid":"751ce8e4ff9bb26f106342c26d9f4893c21a7bfa","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:41.12963Z","iopub.execute_input":"2022-01-24T18:35:41.130183Z","iopub.status.idle":"2022-01-24T18:35:41.486197Z","shell.execute_reply.started":"2022-01-24T18:35:41.130102Z","shell.execute_reply":"2022-01-24T18:35:41.485103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Normalizing and comparison\n# Both coins start from 100\nnormalized_btc = btc.High.div(btc.High.iloc[0]).mul(100)\nnormalized_eth = eth.High.div(eth.High.iloc[0]).mul(100)\nnormalized_btc.plot()\nnormalized_eth.plot()\n_ = plt.title('Comparing BTC and ETH over time (normalized)')\nplt.legend(['BTC','ETH'])\nplt.show()","metadata":{"_uuid":"e9d610487415c9c65726c187909abbd8ce0f57b0","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:41.487402Z","iopub.execute_input":"2022-01-24T18:35:41.487835Z","iopub.status.idle":"2022-01-24T18:35:41.84425Z","shell.execute_reply.started":"2022-01-24T18:35:41.487772Z","shell.execute_reply":"2022-01-24T18:35:41.843509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting before normalization\nbtc.Volume.plot()\neth.Volume.plot()\nplt.legend(['BTC','ETH'])\n_ = plt.title('Comparing BTC and ETH Volumes over time')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:41.845295Z","iopub.execute_input":"2022-01-24T18:35:41.845697Z","iopub.status.idle":"2022-01-24T18:35:42.199515Z","shell.execute_reply.started":"2022-01-24T18:35:41.845638Z","shell.execute_reply":"2022-01-24T18:35:42.198785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Normalizing and comparison\n# Both coin start from 100\nnormalized_btc = btc.Volume.div(btc.Volume.iloc[0]).mul(100)\nnormalized_eth = eth.Volume.div(eth.Volume.iloc[0]).mul(100)\nnormalized_btc.plot()\nnormalized_eth.plot()\n_ = plt.title('Comparing BTC and ETH Volumes over time (normalized)')\nplt.legend(['BTC','ETH'])\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:42.200899Z","iopub.execute_input":"2022-01-24T18:35:42.201393Z","iopub.status.idle":"2022-01-24T18:35:42.569957Z","shell.execute_reply.started":"2022-01-24T18:35:42.201142Z","shell.execute_reply":"2022-01-24T18:35:42.569179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can clearly see how BTC outperforms ETH over time.","metadata":{"_uuid":"5a89d0a945a7cff7d2dbd074d437be83607c17e3","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='2.5'>2.5 Window functions</a>\nWindow functions are used to identify sub periods, calculates sub-metrics of sub-periods.\n\n**Rolling** - Same size and sliding\n\n**Expanding** - Contains all prior values","metadata":{"_uuid":"7de5aa0e16da9790177a2994e843b4bf69c82688","_kg_hide-input":false,"trusted":true}},{"cell_type":"code","source":"# Rolling window functions\nrolling_btc = btc.High.rolling('90D').mean()\nbtc.High.plot()\nrolling_btc.plot()\n_ = plt.title('Comparing High price and rolling High price of BTC over time')\nplt.legend(['High','Rolling Mean'])\n# Plotting a rolling mean of 90 day window with original High attribute of BTC\nplt.show()","metadata":{"_uuid":"2759a3894d231e62d1a868d61591a78796b428ac","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:42.571307Z","iopub.execute_input":"2022-01-24T18:35:42.571569Z","iopub.status.idle":"2022-01-24T18:35:42.927785Z","shell.execute_reply.started":"2022-01-24T18:35:42.571518Z","shell.execute_reply":"2022-01-24T18:35:42.926895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, observe that rolling mean plot is a smoother version of the original plot.","metadata":{"_uuid":"6de30c1c19b15b698c04f6328430d25e0c29e016","_kg_hide-input":false}},{"cell_type":"code","source":"# Expanding window functions\nbtc_mean = btc.High.expanding().mean()\nbtc_std = btc.High.expanding().std()\nbtc.High.plot()\nbtc_mean.plot()\nbtc_std.plot()\n_ = plt.title('Comparing High price, Expanding Mean of High Price and Expanding Std. Dev. of High Price of BTC over time')\nplt.legend(['High','Expanding Mean','Expanding Standard Deviation'])\nplt.show()","metadata":{"_uuid":"0c00473ae42582d73987233a761e0a22a9c2b9af","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:42.929291Z","iopub.execute_input":"2022-01-24T18:35:42.929611Z","iopub.status.idle":"2022-01-24T18:35:43.324181Z","shell.execute_reply.started":"2022-01-24T18:35:42.929554Z","shell.execute_reply":"2022-01-24T18:35:43.322898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Rolling window functions\nrolling_btc = btc.Volume.rolling('90D').mean()\nbtc.Volume.plot()\nrolling_btc.plot()\n_ = plt.title('Comparing Volume and rolling Volume of BTC over time')\nplt.legend(['High','Rolling Mean'])\n# Plotting a rolling mean of 90 day window with original High attribute of BTC\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:43.325518Z","iopub.execute_input":"2022-01-24T18:35:43.326011Z","iopub.status.idle":"2022-01-24T18:35:43.704975Z","shell.execute_reply.started":"2022-01-24T18:35:43.325852Z","shell.execute_reply":"2022-01-24T18:35:43.704244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Expanding window functions\nbtc_mean = btc.Volume.expanding().mean()\nbtc_std = btc.Volume.expanding().std()\nbtc.Volume.plot()\nbtc_mean.plot()\nbtc_std.plot()\n_ = plt.title('Comparing Volume, Expanding Mean of Volume and Expanding Std. Dev. of Volume of BTC over time')\nplt.legend(['High','Expanding Mean','Expanding Standard Deviation'])\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:43.706087Z","iopub.execute_input":"2022-01-24T18:35:43.706461Z","iopub.status.idle":"2022-01-24T18:35:44.092762Z","shell.execute_reply.started":"2022-01-24T18:35:43.706422Z","shell.execute_reply":"2022-01-24T18:35:44.092174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='2.6'>2.6 OHLC charts</a>\nAn OHLC chart is any type of price chart that shows the open, high, low and close price of a certain time period. Open-high-low-close Charts (or OHLC Charts) are used as a trading tool to visualise and analyse the price changes over time for securities, currencies, stocks, bonds, commodities, etc. OHLC Charts are useful for interpreting the day-to-day sentiment of the market and forecasting any future price changes through the patterns produced.\n\nThe y-axis on an OHLC Chart is used for the price scale, while the x-axis is the timescale. On each single time period, an OHLC Charts plots a symbol that represents two ranges: the highest and lowest prices traded, and also the opening and closing price on that single time period (for example in a day). On the range symbol, the high and low price ranges are represented by the length of the main vertical line. The open and close prices are represented by the vertical positioning of tick-marks that appear on the left (representing the open price) and on right (representing the close price) sides of the high-low vertical line.\n\nColour can be assigned to each OHLC Chart symbol, to distinguish whether the market is \"bullish\" (the closing price is higher then it opened) or \"bearish\" (the closing price is lower then it opened).\n\n<img src=\"https://datavizcatalogue.com/methods/images/anatomy/SVG/OHLC_chart.svg\">\n\nSource: [Datavizcatalogue](https://datavizcatalogue.com/methods/OHLC_chart.html)","metadata":{"_uuid":"468318f02e689885dd64c8c4cc30984fb91d8f3b","_kg_hide-input":false}},{"cell_type":"code","source":"# OHLC chart of June 2008\ntrace = go.Ohlc(x=btc['06-2021'].index,\n                open=btc['06-2021'].Open,\n                high=btc['06-2021'].High,\n                low=btc['06-2021'].Low,\n                close=btc['06-2021'].Close)\ndata = [trace]\nprint('OHLC price chart of BTC for June 2021')\niplot(data, filename='simple_ohlc')","metadata":{"_uuid":"cda7c35e8689a4638f0cfda0cc280bc670733993","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:44.093687Z","iopub.execute_input":"2022-01-24T18:35:44.094017Z","iopub.status.idle":"2022-01-24T18:35:45.246547Z","shell.execute_reply.started":"2022-01-24T18:35:44.09398Z","shell.execute_reply":"2022-01-24T18:35:45.245721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ## <a id='2.7'>2.7 Candlestick charts</a>\nThis type of chart is used as a trading tool to visualise and analyse the price movements over time for securities, derivatives, currencies, stocks, bonds, commodities, etc. Although the symbols used in Candlestick Charts resemble a Box Plot, they function differently and therefore, are not to be confused with one another.\n\nCandlestick Charts display multiple bits of price information such as the open price, close price, highest price and lowest price through the use of candlestick-like symbols. Each symbol represents the compressed trading activity for a single time period (a minute, hour, day, month, etc). Each Candlestick symbol is plotted along a time scale on the x-axis, to show the trading activity over time.\n\nThe main rectangle in the symbol is known as the real body, which is used to display the range between the open and close price of that time period. While the lines extending from the bottom and top of the real body is known as the lower and upper shadows (or wick). Each shadow represents the highest or lowest price traded during the time period represented. When the market is Bullish (the closing price is higher than it opened), then the body is coloured typically white or green. But when the market is Bearish (the closing price is lower than it opened), then the body is usually coloured either black or red.\n\n<img src=\"https://datavizcatalogue.com/methods/images/anatomy/SVG/candlestick_chart.svg\">\n\nCandlestick Charts are great for detecting and predicting market trends over time and are useful for interpreting the day-to-day sentiment of the market, through each candlestick symbol's colouring and shape. For example, the longer the body is, the more intense the selling or buying pressure is. While, a very short body, would indicate that there is very little price movement in that time period and represents consolidation.\n\nCandlestick Charts help reveal the market psychology (the fear and greed experienced by sellers and buyers) through the various indicators, such as shape and colour, but also by the many identifiable patterns that can be found in Candlestick Charts. In total, there are 42 recognised patterns that are divided into simple and complex patterns. These patterns found in Candlestick Charts are useful for displaying price relationships and can be used for predicting the possible future movement of the market. You can find a list and description of each pattern here.\n\nPlease bear in mind, that Candlestick Charts don't express the events taking place between the open and close price - only the relationship between the two prices. So you can't tell how volatile trading was within that single time period.\n\nSource: [Datavizcatalogue](https://datavizcatalogue.com/methods/candlestick_chart.html)","metadata":{"_uuid":"93363b9969f9ec48dd95b0a33201ea00d8f6bd0b","_kg_hide-input":false}},{"cell_type":"code","source":"# Candlestick chart of march 2008\ntrace = go.Candlestick(x=btc['06-2021'].index,\n                open=btc['06-2021'].Open,\n                high=btc['06-2021'].High,\n                low=btc['06-2021'].Low,\n                close=btc['06-2021'].Close)\ndata = [trace]\nprint('OHLC price chart (Candlestick) of BTC for June 2021')\niplot(data, filename='simple_candlestick')","metadata":{"_uuid":"ab77825a5b940f2db8795af0871c769bed6a3048","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:45.247983Z","iopub.execute_input":"2022-01-24T18:35:45.248528Z","iopub.status.idle":"2022-01-24T18:35:45.474759Z","shell.execute_reply.started":"2022-01-24T18:35:45.248324Z","shell.execute_reply":"2022-01-24T18:35:45.47403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='2.8'>2.8 Autocorrelation and Partial Autocorrelation</a>\n* Autocorrelation - The autocorrelation function (ACF) measures how a series is correlated with itself at different lags.\n* Partial Autocorrelation - The partial autocorrelation function can be interpreted as a regression of the series against its past lags.  The terms can be interpreted the same way as a standard  linear regression, that is the contribution of a change in that particular lag while holding others constant. \n\nSource: [Quora](https://www.quora.com/What-is-the-difference-among-auto-correlation-partial-auto-correlation-and-inverse-auto-correlation-while-modelling-an-ARIMA-series)","metadata":{"_uuid":"554c6559d926c57de9e75da9c908aed9843bb529","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Autocorrelation","metadata":{"_uuid":"90d9e5bb466fc9efbdb4c2cdaea14c4d30704988","_kg_hide-input":false}},{"cell_type":"code","source":"# Autocorrelation of Closing price of BTC\nplot_acf(btc[\"Close\"],lags=25,title=\"Autocorrelation chart: BTC (Close price)\")\nplt.show()","metadata":{"_uuid":"4ea5b5a0b6e36ddb76ed37ef77edcd037087d2cf","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:45.475844Z","iopub.execute_input":"2022-01-24T18:35:45.476125Z","iopub.status.idle":"2022-01-24T18:35:45.762307Z","shell.execute_reply.started":"2022-01-24T18:35:45.476058Z","shell.execute_reply":"2022-01-24T18:35:45.761279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" As all lags are either close to 1 or at least greater than the confidence interval, they are statistically significant.","metadata":{"_uuid":"752cfb34cbdf8a763126625f9a7540e1d2aef84c","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Partial Autocorrelation","metadata":{"_uuid":"31e685329d81b91e23539b3fc589d21bed946ce7","_kg_hide-input":false}},{"cell_type":"code","source":"# Autocorrelation of Volume of BTC\nplot_acf(btc[\"Volume\"],lags=25,title=\"Autocorrelation chart: BTC (Volume)\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:45.763871Z","iopub.execute_input":"2022-01-24T18:35:45.764448Z","iopub.status.idle":"2022-01-24T18:35:46.051423Z","shell.execute_reply.started":"2022-01-24T18:35:45.764382Z","shell.execute_reply":"2022-01-24T18:35:46.05046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Autocorrelation of Closing price of Ethereum\nplot_acf(eth[\"Close\"],lags=25,title=\"Autocorrelation chart: ETH (Close price)\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:46.052947Z","iopub.execute_input":"2022-01-24T18:35:46.053508Z","iopub.status.idle":"2022-01-24T18:35:46.344322Z","shell.execute_reply.started":"2022-01-24T18:35:46.053428Z","shell.execute_reply":"2022-01-24T18:35:46.343396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Autocorrelation of Volume of ETH\nplot_acf(eth[\"Volume\"],lags=25,title=\"Autocorrelation chart: ETH (Volume)\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:46.345822Z","iopub.execute_input":"2022-01-24T18:35:46.346385Z","iopub.status.idle":"2022-01-24T18:35:46.635478Z","shell.execute_reply.started":"2022-01-24T18:35:46.346315Z","shell.execute_reply":"2022-01-24T18:35:46.634464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Partial Autocorrelation of Closing price of BTC\nplot_pacf(btc[\"Close\"],lags=25, title=\"Partial autocorrelation chart: BTC (Close price)\")\nplt.show()","metadata":{"_uuid":"639a28e968448ddbd41694d5d2950afe13865e40","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:46.637049Z","iopub.execute_input":"2022-01-24T18:35:46.637627Z","iopub.status.idle":"2022-01-24T18:35:46.933532Z","shell.execute_reply.started":"2022-01-24T18:35:46.637565Z","shell.execute_reply":"2022-01-24T18:35:46.932523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Though it is statistically signficant, partial autocorrelation after first 2 lags is very low.","metadata":{"_uuid":"b471c9a601cb8478971349f3727f1f477fadf6a3","_kg_hide-input":false}},{"cell_type":"code","source":"# Partial Autocorrelation of Volume of BTC\nplot_pacf(btc[\"Volume\"],lags=25, title=\"Partial autocorrelation chart: BTC (Volume)\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:46.935073Z","iopub.execute_input":"2022-01-24T18:35:46.935643Z","iopub.status.idle":"2022-01-24T18:35:47.22673Z","shell.execute_reply.started":"2022-01-24T18:35:46.93558Z","shell.execute_reply":"2022-01-24T18:35:47.225865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Partial Autocorrelation of closing price of ETH \nplot_pacf(eth[\"Close\"],lags=25, title=\"Partial autocorrelation chart: ETH (Close price)\")\nplt.show()","metadata":{"_uuid":"d7174be0162bbb5e24b78fd184703527f19625eb","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:47.228266Z","iopub.execute_input":"2022-01-24T18:35:47.228603Z","iopub.status.idle":"2022-01-24T18:35:47.528361Z","shell.execute_reply.started":"2022-01-24T18:35:47.228539Z","shell.execute_reply":"2022-01-24T18:35:47.527276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, only 0th and 1st lag are statistically significant.","metadata":{"_uuid":"82f98f2fc35b61c45f9b65e0477c227aefdaab47","_kg_hide-input":false}},{"cell_type":"code","source":"# Partial Autocorrelation of Volume of ETH\nplot_pacf(eth[\"Volume\"],lags=25, title=\"Partial autocorrelation chart: ETH (Volume)\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:47.529973Z","iopub.execute_input":"2022-01-24T18:35:47.530612Z","iopub.status.idle":"2022-01-24T18:35:47.817629Z","shell.execute_reply.started":"2022-01-24T18:35:47.530543Z","shell.execute_reply":"2022-01-24T18:35:47.816604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a id='3'>3. Time series decomposition and Random walks</a>","metadata":{"_uuid":"61ef0d5be5a9c6ab63c0f3e432deae0166530bd0","_kg_hide-input":false,"trusted":true}},{"cell_type":"markdown","source":"## <a id='3.1'>3.1. Trends, seasonality and noise</a>\nThese are the components of a time series\n* Trend - Consistent upwards or downwards slope of a time series\n* Seasonality - Clear periodic pattern of a time series(like sine funtion)\n* Noise - Outliers or missing values","metadata":{"_uuid":"29a6016f3d75811e47db9d727497f06b5d194113","_kg_hide-input":false}},{"cell_type":"code","source":"# Let's take BTC's High for this\nbtc[\"High\"].plot(figsize=(16,8))\n_ = plt.title('Comparing High price of BTC over time')","metadata":{"_uuid":"55da1ae309826244f5aeeb18d293d2512535619d","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:47.819368Z","iopub.execute_input":"2022-01-24T18:35:47.819891Z","iopub.status.idle":"2022-01-24T18:35:48.166767Z","shell.execute_reply.started":"2022-01-24T18:35:47.819832Z","shell.execute_reply":"2022-01-24T18:35:48.166024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, for decomposition...\nprint('Decomposing closing price of BTC over time')\ndecomposed_btc_close = sm.tsa.seasonal_decompose(btc[\"Close\"],freq=360) # The frequncy is annual\nfigure = decomposed_btc_close.plot()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:48.167681Z","iopub.execute_input":"2022-01-24T18:35:48.16801Z","iopub.status.idle":"2022-01-24T18:35:48.945831Z","shell.execute_reply.started":"2022-01-24T18:35:48.167973Z","shell.execute_reply":"2022-01-24T18:35:48.944922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, for decomposition...\nprint('Decomposing volume data of BTC over time')\ndecomposed_btc_volume = sm.tsa.seasonal_decompose(btc[\"Volume\"],freq=360) # The frequncy is annual\nfigure = decomposed_btc_volume.plot()\nplt.show()","metadata":{"_uuid":"eb9a833bf7f58467c00008560509a4ce6905ea92","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:48.947393Z","iopub.execute_input":"2022-01-24T18:35:48.94774Z","iopub.status.idle":"2022-01-24T18:35:49.54028Z","shell.execute_reply.started":"2022-01-24T18:35:48.947681Z","shell.execute_reply":"2022-01-24T18:35:49.539427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There is clearly an upward trend in the above plots.\n* You can also see the uniform seasonal change.\n* Non-uniform noise that represent outliers and/or missing values","metadata":{"_uuid":"b0f37643e9f41715ec54ecca09b9c10b81fe557d","_kg_hide-input":false}},{"cell_type":"code","source":"# Let's take ETH's High for this\neth[\"High\"].plot(figsize=(16,8))\n_ = plt.title('High price of ETH over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:49.541801Z","iopub.execute_input":"2022-01-24T18:35:49.542237Z","iopub.status.idle":"2022-01-24T18:35:49.764154Z","shell.execute_reply.started":"2022-01-24T18:35:49.542071Z","shell.execute_reply":"2022-01-24T18:35:49.763278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, for decomposition...\nprint('Decomposing closing price of ETH over time')\ndecomposed_eth_close = sm.tsa.seasonal_decompose(eth[\"Close\"],freq=360) # The frequncy is annual\nfigure = decomposed_eth_close.plot()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:49.765633Z","iopub.execute_input":"2022-01-24T18:35:49.766046Z","iopub.status.idle":"2022-01-24T18:35:50.337973Z","shell.execute_reply.started":"2022-01-24T18:35:49.765891Z","shell.execute_reply":"2022-01-24T18:35:50.337285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, for decomposition...\nprint('Decomposing volume data of ETH over time')\ndecomposed_eth_volume = sm.tsa.seasonal_decompose(eth[\"Volume\"],freq=360) # The frequncy is annual\nfigure = decomposed_eth_volume.plot()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:50.339563Z","iopub.execute_input":"2022-01-24T18:35:50.340121Z","iopub.status.idle":"2022-01-24T18:35:51.138027Z","shell.execute_reply.started":"2022-01-24T18:35:50.340046Z","shell.execute_reply":"2022-01-24T18:35:51.137209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='3.2'>3.2. White noise</a>\nWhite noise has...\n* Constant mean\n* Constant variance\n* Zero auto-correlation at all lags","metadata":{"_uuid":"fee84a3db1d59f4706711f078299cdff6b4bf1dc","_kg_hide-input":false}},{"cell_type":"code","source":"# Plotting white noise\nrcParams['figure.figsize'] = 16, 6\nwhite_noise = np.random.normal(loc=0, scale=1, size=1000)\n_ = plt.title('Generate Whitenoise')\n# loc is mean, scale is variance\nplt.plot(white_noise)","metadata":{"_uuid":"01317cb2d3d8142945d94f751b60b7d59765f3df","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:51.139628Z","iopub.execute_input":"2022-01-24T18:35:51.1401Z","iopub.status.idle":"2022-01-24T18:35:51.35361Z","shell.execute_reply.started":"2022-01-24T18:35:51.140043Z","shell.execute_reply":"2022-01-24T18:35:51.352795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting autocorrelation of white noise\nplot_acf(white_noise,lags=20)\nplt.show()","metadata":{"_uuid":"4408976acde81f0281375fac4f6aa8dadd0e0c30","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:51.355075Z","iopub.execute_input":"2022-01-24T18:35:51.355556Z","iopub.status.idle":"2022-01-24T18:35:51.534931Z","shell.execute_reply.started":"2022-01-24T18:35:51.355502Z","shell.execute_reply":"2022-01-24T18:35:51.53406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"See how all lags are statistically insigficant as they lie inside the confidence interval(shaded portion).","metadata":{"_uuid":"f70b17298bc4f7cd11ebb22af3369506e83ce5f8","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='3.3'>3.3. Random Walk</a>\nA random walk is a mathematical object, known as a stochastic or random process, that describes a path that consists of a succession of random steps on some mathematical space such as the integers. \n\nIn general if we talk about stocks, Today's Price = Yesterday's Price + Noise\n\n# P<sub>t</sub> = P<sub>t-1</sub> + ε<sub>t</sub> \n\nRandom walks can't be forecasted because well, noise is random.\n\nRandom Walk with Drift(drift(μ) is zero-mean)\n#### P<sub>t</sub> - P<sub>t-1</sub> = μ + ε<sub>t</sub> \n<br>\nRegression test for random walk \n#### P<sub>t</sub> = α + βP<sub>t-1</sub> + ε<sub>t</sub>\n#### Equivalent to  P<sub>t</sub> - P<sub>t-1</sub> = α + βP<sub>t-1</sub> + ε<sub>t</sub>\n<br>\nTest:\n#### H<sub>0</sub>: β = 1 (This is a random walk)\n#### H<sub>1</sub>: β < 1 (This is not a random walk)\n<br>\nDickey-Fuller Test:\n#### H<sub>0</sub>: β = 0 (This is a random walk)\n#### H<sub>1</sub>: β < 0 (This is not a random walk)","metadata":{"_uuid":"7b5855eed1497f9aea8ebc70b1f5621a1cd8ab92","_kg_hide-input":false}},{"cell_type":"markdown","source":"### Augmented Dickey-Fuller test\nAn augmented Dickey–Fuller test (ADF) tests the null hypothesis that a unit root is present in a time series sample. It is basically Dickey-Fuller test with more lagged changes on RHS.","metadata":{"_uuid":"da19a9d1d683224c227654f28c1f53432953781c","_kg_hide-input":false}},{"cell_type":"code","source":"%%time\n# Augmented Dickey-Fuller test on volume of BTC and ETH coins \nadf = adfuller(btc[\"Volume\"])\nprint(\"p-value of BTC: {}\".format(float(adf[1])))\nadf = adfuller(eth[\"Volume\"])\nprint(\"p-value of ETH: {}\".format(float(adf[1])))","metadata":{"_uuid":"c43ff44679e19f726cbca75a99d3b0c8e828cdca","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:51.537039Z","iopub.execute_input":"2022-01-24T18:35:51.537694Z","iopub.status.idle":"2022-01-24T18:35:51.678541Z","shell.execute_reply.started":"2022-01-24T18:35:51.537631Z","shell.execute_reply":"2022-01-24T18:35:51.677548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### As BTC has p-value 0.000033662 which is less than 0.05, null hypothesis is rejected and this is not a random walk.\n##### Now ETH has p-value 0.000033662 which is also less than 0.05, null hypothesis is rejected and this is not a random walk.","metadata":{"_uuid":"d81b28d7326841f6056eac3eb1e6e26e59b3809c","_kg_hide-input":false,"trusted":true}},{"cell_type":"markdown","source":"### Generating a random walk","metadata":{"_uuid":"cc765a2077924a474a404b35d895d53e2f235ece","_kg_hide-input":false}},{"cell_type":"code","source":"seed(42)\nrcParams['figure.figsize'] = 16, 6\nrandom_walk = normal(loc=0, scale=0.01, size=1000)\n_ = plt.title('Plot random walk')\nplt.plot(random_walk)\nplt.show()","metadata":{"_uuid":"a92da067abdb8f712e11e80b774bf326bf423def","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:51.68368Z","iopub.execute_input":"2022-01-24T18:35:51.686358Z","iopub.status.idle":"2022-01-24T18:35:52.032168Z","shell.execute_reply.started":"2022-01-24T18:35:51.686263Z","shell.execute_reply":"2022-01-24T18:35:52.031306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = ff.create_distplot([random_walk],['Random Walk'],bin_size=0.001)\nprint('Random walk data distribution')\niplot(fig, filename='Basic Distplot')","metadata":{"_uuid":"b65345199e9356ac6fc0c1f91013cfd2cfab0067","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:52.03359Z","iopub.execute_input":"2022-01-24T18:35:52.033933Z","iopub.status.idle":"2022-01-24T18:35:52.68882Z","shell.execute_reply.started":"2022-01-24T18:35:52.033876Z","shell.execute_reply":"2022-01-24T18:35:52.687989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='3.4'>3.4 Stationarity</a>\nA stationary time series is one whose statistical properties such as mean, variance, autocorrelation, etc. are all constant over time.\n* Strong stationarity:  is a stochastic process whose unconditional joint probability distribution does not change when shifted in time. Consequently, parameters such as mean and variance also do not change over time.\n* Weak stationarity: is a process where mean, variance, autocorrelation are constant throughout the time\n\nStationarity is important as  non-stationary series that depend on time have too many parameters to account for when modelling the time series. diff() method can easily convert a non-stationary series to a stationary series.\n\nWe will try to decompose seasonal component of the above decomposed time series.","metadata":{"_uuid":"3503b3468028a38ac1345e62ab59996a333465b9","_kg_hide-input":false}},{"cell_type":"code","source":"# The original non-stationary plot\ndecomposed_btc_close.trend.plot()\n_ = plt.title('Decomposed BTC closing price over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:52.689941Z","iopub.execute_input":"2022-01-24T18:35:52.690398Z","iopub.status.idle":"2022-01-24T18:35:52.981062Z","shell.execute_reply.started":"2022-01-24T18:35:52.690242Z","shell.execute_reply":"2022-01-24T18:35:52.980303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The original non-stationary plot\ndecomposed_btc_close.trend.diff().plot()\n_ = plt.title('Decomposed BTC closing price difference over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:52.982279Z","iopub.execute_input":"2022-01-24T18:35:52.982629Z","iopub.status.idle":"2022-01-24T18:35:53.313262Z","shell.execute_reply.started":"2022-01-24T18:35:52.982483Z","shell.execute_reply":"2022-01-24T18:35:53.312238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The original non-stationary plot\ndecomposed_btc_volume.trend.plot()\n_ = plt.title('Decomposed BTC volume over time')","metadata":{"_uuid":"aa6661f017536cfc6c69d8b1a15316a6e37ca200","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:53.315054Z","iopub.execute_input":"2022-01-24T18:35:53.315907Z","iopub.status.idle":"2022-01-24T18:35:53.613803Z","shell.execute_reply.started":"2022-01-24T18:35:53.315704Z","shell.execute_reply":"2022-01-24T18:35:53.612798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The original non-stationary plot\ndecomposed_btc_volume.trend.diff().plot()\n_ = plt.title('Decomposed BTC volume difference over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:53.615444Z","iopub.execute_input":"2022-01-24T18:35:53.61604Z","iopub.status.idle":"2022-01-24T18:35:53.928826Z","shell.execute_reply.started":"2022-01-24T18:35:53.615975Z","shell.execute_reply":"2022-01-24T18:35:53.927862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The new stationary plot\ndecomposed_eth_close.trend.plot()\n_ = plt.title('Decomposed ETH closing price over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:53.930325Z","iopub.execute_input":"2022-01-24T18:35:53.930865Z","iopub.status.idle":"2022-01-24T18:35:54.227929Z","shell.execute_reply.started":"2022-01-24T18:35:53.930805Z","shell.execute_reply":"2022-01-24T18:35:54.226891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The new stationary plot\ndecomposed_eth_close.trend.diff().plot()\n_ = plt.title('Decomposed ETH closing price difference over time')","metadata":{"_uuid":"348f09fd4cb0c9ba057930c502786aaeda80452d","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:54.229243Z","iopub.execute_input":"2022-01-24T18:35:54.229567Z","iopub.status.idle":"2022-01-24T18:35:54.58042Z","shell.execute_reply.started":"2022-01-24T18:35:54.22951Z","shell.execute_reply":"2022-01-24T18:35:54.579363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The new stationary plot\ndecomposed_eth_volume.trend.plot()\n_ = plt.title('Decomposed ETH volume over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:54.582465Z","iopub.execute_input":"2022-01-24T18:35:54.583402Z","iopub.status.idle":"2022-01-24T18:35:54.920016Z","shell.execute_reply.started":"2022-01-24T18:35:54.583333Z","shell.execute_reply":"2022-01-24T18:35:54.919398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The new stationary plot\ndecomposed_eth_volume.trend.diff().plot()\n_ = plt.title('Decomposed ETH volume difference over time')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:54.923158Z","iopub.execute_input":"2022-01-24T18:35:54.923787Z","iopub.status.idle":"2022-01-24T18:35:55.243247Z","shell.execute_reply.started":"2022-01-24T18:35:54.923719Z","shell.execute_reply":"2022-01-24T18:35:55.242348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a id='4'>4. Modelling using statstools</a>","metadata":{"_uuid":"0bf108b3638b0aaa06b4022993b99fd5a57cecd2","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='4.1'>4.1 AR models</a>\n An autoregressive (AR) model is a representation of a type of random process; as such, it is used to describe certain time-varying processes in nature, economics, etc. The autoregressive model specifies that the output variable depends linearly on its own previous values and on a stochastic term (an imperfectly predictable term); thus the model is in the form of a stochastic difference equation.\n #### AR(1) model\n R<sub>t</sub> = μ + ϕR<sub>t-1</sub> + ε<sub>t</sub>\n ##### As RHS has only one lagged value(R<sub>t-1</sub>)this is called AR model of order 1 where μ is mean and ε is noise at time t\n If ϕ = 1, it is random walk. Else if ϕ = 0, it is white noise. Else if -1 < ϕ < 1, it is stationary. If ϕ is -ve, there is men reversion. If ϕ is +ve, there is momentum.\n #### AR(2) model\n R<sub>t</sub> = μ + ϕ<sub>1</sub>R<sub>t-1</sub> + ϕ<sub>2</sub>R<sub>t-2</sub> + ε<sub>t</sub>\n #### AR(3) model\n R<sub>t</sub> = μ + ϕ<sub>1</sub>R<sub>t-1</sub> + ϕ<sub>2</sub>R<sub>t-2</sub> + ϕ<sub>3</sub>R<sub>t-3</sub> + ε<sub>t</sub>","metadata":{"_uuid":"7901cd86bf946ff83314ead9ee074f45e1ff76f8","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Simulating AR(1) model","metadata":{"_uuid":"ead96dc18024b191be5d515516e69e3009cd209b","_kg_hide-input":false}},{"cell_type":"code","source":"# AR(1) MA(1) model:AR parameter = +0.9\nrcParams['figure.figsize'] = 16, 12\nplt.subplot(4,1,1)\nar1 = np.array([1, -0.9]) # We choose -0.9 as AR parameter is +0.9\nma1 = np.array([1])\nAR1 = ArmaProcess(ar1, ma1)\nsim1 = AR1.generate_sample(nsample=1000)\nplt.title('AR(1) model: AR parameter = +0.9')\nplt.plot(sim1)\n# We will take care of MA model later\n# AR(1) MA(1) AR parameter = -0.9\nplt.subplot(4,1,2)\nar2 = np.array([1, 0.9]) # We choose +0.9 as AR parameter is -0.9\nma2 = np.array([1])\nAR2 = ArmaProcess(ar2, ma2)\nsim2 = AR2.generate_sample(nsample=1000)\nplt.title('AR(1) model: AR parameter = -0.9')\nplt.plot(sim2)\n# AR(2) MA(1) AR parameter = 0.9\nplt.subplot(4,1,3)\nar3 = np.array([2, -0.9]) # We choose -0.9 as AR parameter is +0.9\nma3 = np.array([1])\nAR3 = ArmaProcess(ar3, ma3)\nsim3 = AR3.generate_sample(nsample=1000)\nplt.title('AR(2) model: AR parameter = +0.9')\nplt.plot(sim3)\n# AR(2) MA(1) AR parameter = -0.9\nplt.subplot(4,1,4)\nar4 = np.array([2, 0.9]) # We choose +0.9 as AR parameter is -0.9\nma4 = np.array([1])\nAR4 = ArmaProcess(ar4, ma4)\nsim4 = AR4.generate_sample(nsample=1000)\nplt.title('AR(2) model: AR parameter = -0.9')\nplt.plot(sim4)\nplt.show()","metadata":{"_uuid":"ff03dd2d04772c9da9cc1bbbe44cecbeff4386a3","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:55.244955Z","iopub.execute_input":"2022-01-24T18:35:55.24546Z","iopub.status.idle":"2022-01-24T18:35:56.00806Z","shell.execute_reply.started":"2022-01-24T18:35:55.245299Z","shell.execute_reply":"2022-01-24T18:35:56.007051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Forecasting a simulated model","metadata":{"_uuid":"8a33e37bf59ccf5d312e918b3f700b43672fa8f3","_kg_hide-input":false,"trusted":true}},{"cell_type":"code","source":"model = ARMA(sim1, order=(1,0))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={} ,ϕ={}\".format(result.params[0],result.params[1]))\n","metadata":{"_uuid":"feba20bff98203d6862a547eb5919e3e6c809c8e","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:56.009562Z","iopub.execute_input":"2022-01-24T18:35:56.010009Z","iopub.status.idle":"2022-01-24T18:35:56.116963Z","shell.execute_reply.started":"2022-01-24T18:35:56.009849Z","shell.execute_reply":"2022-01-24T18:35:56.115746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ϕ is around 0.9 which is what we chose as AR parameter in our first simulated model.","metadata":{"_uuid":"20039473ec2863639d8913e1ecde4f9803ac17bf","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Predicting the models","metadata":{"_uuid":"f50dc3fc29a2618da26ae11d9127716233ed67cb","_kg_hide-input":true}},{"cell_type":"code","source":"# Predicting simulated AR(1) model \nresult.plot_predict(start=900, end=1010)\n_ = plt.title('AR model: Actual versus predicted values (with confidence interval plotted) - simulated values')\nplt.show()","metadata":{"_uuid":"a9af97bfcdf0fd1fb5cf626a8bba0b4560fc4358","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:56.118349Z","iopub.execute_input":"2022-01-24T18:35:56.118794Z","iopub.status.idle":"2022-01-24T18:35:56.419997Z","shell.execute_reply.started":"2022-01-24T18:35:56.11874Z","shell.execute_reply":"2022-01-24T18:35:56.419207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(sim1[900:1011], result.predict(start=900,end=999)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"a1ec123d6ab0780304160f82b37f5f01fd007f84","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:56.42131Z","iopub.execute_input":"2022-01-24T18:35:56.421795Z","iopub.status.idle":"2022-01-24T18:35:56.429893Z","shell.execute_reply.started":"2022-01-24T18:35:56.421735Z","shell.execute_reply":"2022-01-24T18:35:56.429103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"y is predicted plot. Quite neat!","metadata":{"_uuid":"0ecbeedf969c4972f05b9624a847aaffcb7862a9","_kg_hide-input":false}},{"cell_type":"code","source":"# Predicting Close price of BTC\nclose_price = ARMA(btc[\"Close\"].diff().iloc[1:].values, order=(1,0))\nres = close_price.fit()\nres.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: Actual versus predicted values of Closing price of BTC')\nplt.show()","metadata":{"_uuid":"b24234f9ca3f0de65a3f75aa23caa204b21833b3","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:56.433743Z","iopub.execute_input":"2022-01-24T18:35:56.434218Z","iopub.status.idle":"2022-01-24T18:35:56.862446Z","shell.execute_reply.started":"2022-01-24T18:35:56.434162Z","shell.execute_reply":"2022-01-24T18:35:56.861642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(btc[\"Close\"].diff().iloc[900:1000].values, result.predict(start=900,end=999)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"d96b6ea96dacf1ee6fe41abd902b9184d9c9edbc","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:56.863514Z","iopub.execute_input":"2022-01-24T18:35:56.863738Z","iopub.status.idle":"2022-01-24T18:35:56.871925Z","shell.execute_reply.started":"2022-01-24T18:35:56.8637Z","shell.execute_reply":"2022-01-24T18:35:56.871031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predicting Volume of BTC\nvolume = ARMA(btc[\"Volume\"].diff().iloc[1:].values, order=(1,0))\nres = volume.fit()\nres.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: Actual versus predicted values of volume of BTC')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:56.873101Z","iopub.execute_input":"2022-01-24T18:35:56.873541Z","iopub.status.idle":"2022-01-24T18:35:57.304977Z","shell.execute_reply.started":"2022-01-24T18:35:56.87339Z","shell.execute_reply":"2022-01-24T18:35:57.304226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(btc[\"Volume\"].diff().iloc[900:1000].values, result.predict(start=900,end=999)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:57.306216Z","iopub.execute_input":"2022-01-24T18:35:57.306448Z","iopub.status.idle":"2022-01-24T18:35:57.314705Z","shell.execute_reply.started":"2022-01-24T18:35:57.306407Z","shell.execute_reply":"2022-01-24T18:35:57.313861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Not quite impressive. But let's try Ethereum. ","metadata":{"_uuid":"8987a49063cea031745bd87ec71221db473069ab","_kg_hide-input":false}},{"cell_type":"code","source":"# Predicting closing prices of ETH\nclose_price = ARMA(eth[\"Close\"].diff().iloc[1:].values, order=(1,0))\nres = close_price.fit()\nres.plot_predict(start=900, end=1010)\n_ = plt.title('ARMA model: Actual versus predicted values of Closing price of ETH')\nplt.show()","metadata":{"_uuid":"72ef1135eee34841500c55718bda48310386dceb","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:57.316208Z","iopub.execute_input":"2022-01-24T18:35:57.316516Z","iopub.status.idle":"2022-01-24T18:35:57.675099Z","shell.execute_reply.started":"2022-01-24T18:35:57.316457Z","shell.execute_reply":"2022-01-24T18:35:57.674307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(eth[\"Close\"].diff().iloc[900:1000].values, result.predict(start=900,end=999)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:57.676521Z","iopub.execute_input":"2022-01-24T18:35:57.677033Z","iopub.status.idle":"2022-01-24T18:35:57.686745Z","shell.execute_reply.started":"2022-01-24T18:35:57.676978Z","shell.execute_reply":"2022-01-24T18:35:57.685928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predicting Volume of ETH\nvolume = ARMA(eth[\"Volume\"].diff().iloc[1:].values, order=(1,0))\nres = volume.fit()\nres.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: Actual versus predicted values of volume of ETH')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:57.688152Z","iopub.execute_input":"2022-01-24T18:35:57.688685Z","iopub.status.idle":"2022-01-24T18:35:58.129704Z","shell.execute_reply.started":"2022-01-24T18:35:57.688632Z","shell.execute_reply":"2022-01-24T18:35:58.127551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(eth[\"Volume\"].diff().iloc[900:1000].values, result.predict(start=900,end=999)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:58.131061Z","iopub.execute_input":"2022-01-24T18:35:58.131329Z","iopub.status.idle":"2022-01-24T18:35:58.139292Z","shell.execute_reply.started":"2022-01-24T18:35:58.131286Z","shell.execute_reply":"2022-01-24T18:35:58.138559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are always better models.","metadata":{"_uuid":"3a01a4ff2973bd9f1a0d61d4099529d2b7852a52","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='4.2'>4.2 MA models</a>\nThe moving-average (MA) model is a common approach for modeling univariate time series. The moving-average model specifies that the output variable depends linearly on the current and various past values of a stochastic (imperfectly predictable) term.\n#### MA(1) model\nR<sub>t</sub> = μ  +  ϵ<sub>t</sub>1 + θϵ<sub>t-1</sub>\n\nIt translates to Today's returns = mean + today's noise + yesterday's noise\n##### As there is only 1 lagged value in RHS, it is an MA model of order 1","metadata":{"_uuid":"00c24abb4b8f694234352b17bf4bfafc2ebe56f6","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Simulating MA(1) model","metadata":{"_uuid":"cc7f14391e83d0b7ea17af030d91fc10b6aaf1d1","_kg_hide-input":false}},{"cell_type":"code","source":"rcParams['figure.figsize'] = 16, 6\nar1 = np.array([1])\nma1 = np.array([1, -0.5])\nMA1 = ArmaProcess(ar1, ma1)\nsim1 = MA1.generate_sample(nsample=1000)\n_ = plt.title('ARMA generated sample')\nplt.plot(sim1)","metadata":{"_uuid":"43853e28272d3949e8cb6a3d8bac0198d3de2457","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:58.140541Z","iopub.execute_input":"2022-01-24T18:35:58.140933Z","iopub.status.idle":"2022-01-24T18:35:58.457861Z","shell.execute_reply.started":"2022-01-24T18:35:58.14088Z","shell.execute_reply":"2022-01-24T18:35:58.456905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Forecasting the simulated MA model","metadata":{"_uuid":"fe571f9460704eedb7f2d13a2f08cf1332cb2b8f","_kg_hide-input":false}},{"cell_type":"code","source":"model = ARMA(sim1, order=(0,1))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={} ,θ={}\".format(result.params[0],result.params[1]))","metadata":{"_uuid":"bb579590bd40b3b31ece519dfe82f04cb85e1b09","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:58.459417Z","iopub.execute_input":"2022-01-24T18:35:58.459849Z","iopub.status.idle":"2022-01-24T18:35:58.518916Z","shell.execute_reply.started":"2022-01-24T18:35:58.459713Z","shell.execute_reply":"2022-01-24T18:35:58.518021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction using MA models","metadata":{"_uuid":"2573e04db1c1a8830830122f879e6f855bad292b","_kg_hide-input":false}},{"cell_type":"code","source":"# Forecasting and predicting Closing price of BTC\nmodel = ARMA(btc[\"Close\"].diff().iloc[1:].values, order=(0,3))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={} ,θ={}\".format(result.params[0],result.params[1]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Closing price of BTC')\nplt.show()","metadata":{"_uuid":"62c83f6a97bee5f0dff89511fbf6910f83417b0a","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:58.52027Z","iopub.execute_input":"2022-01-24T18:35:58.520753Z","iopub.status.idle":"2022-01-24T18:35:59.043146Z","shell.execute_reply.started":"2022-01-24T18:35:58.520694Z","shell.execute_reply":"2022-01-24T18:35:59.042297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count = len(btc[\"Close\"].diff().iloc[1000:1101].values)\nrmse = math.sqrt(mean_squared_error(btc[\"Close\"].diff().iloc[1000:1101].values, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"0b72fb8004bfbdcee81960089502b4c892f40458","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:59.044664Z","iopub.execute_input":"2022-01-24T18:35:59.04517Z","iopub.status.idle":"2022-01-24T18:35:59.057846Z","shell.execute_reply.started":"2022-01-24T18:35:59.045094Z","shell.execute_reply":"2022-01-24T18:35:59.05674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Forecasting and predicting Volume of BTC\nmodel = ARMA(btc[\"Volume\"].diff().iloc[1:].values, order=(0,1)) # changed q from 3 to 1 to resolve error https://stackoverflow.com/questions/52815990/valueerror-the-computed-initial-ma-coefficients-are-not-invertible-you-should-i\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={} ,θ={}\".format(result.params[0],result.params[1]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Volume of BTC')\nplt.show()","metadata":{"_uuid":"62c83f6a97bee5f0dff89511fbf6910f83417b0a","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:59.059509Z","iopub.execute_input":"2022-01-24T18:35:59.060078Z","iopub.status.idle":"2022-01-24T18:35:59.635344Z","shell.execute_reply.started":"2022-01-24T18:35:59.059859Z","shell.execute_reply":"2022-01-24T18:35:59.634396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count = len(btc[\"Volume\"].diff().iloc[1000:1101].values)\nrmse = math.sqrt(mean_squared_error(btc[\"Volume\"].diff().iloc[1000:1101].values, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"0b72fb8004bfbdcee81960089502b4c892f40458","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:59.636658Z","iopub.execute_input":"2022-01-24T18:35:59.637087Z","iopub.status.idle":"2022-01-24T18:35:59.651874Z","shell.execute_reply.started":"2022-01-24T18:35:59.637036Z","shell.execute_reply":"2022-01-24T18:35:59.650621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Forecasting and predicting Closing price of ETH\nmodel = ARMA(eth[\"Close\"].diff().iloc[1:].values, order=(0,3))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={} ,θ={}\".format(result.params[0],result.params[1]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Closing price of ETH')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:35:59.653611Z","iopub.execute_input":"2022-01-24T18:35:59.65422Z","iopub.status.idle":"2022-01-24T18:36:00.126496Z","shell.execute_reply.started":"2022-01-24T18:35:59.654157Z","shell.execute_reply":"2022-01-24T18:36:00.125447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = eth[\"Close\"].diff().iloc[1000:1101].values\ncount = len(samples)\nrmse = math.sqrt(mean_squared_error(samples, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:00.128027Z","iopub.execute_input":"2022-01-24T18:36:00.128608Z","iopub.status.idle":"2022-01-24T18:36:00.139421Z","shell.execute_reply.started":"2022-01-24T18:36:00.128538Z","shell.execute_reply":"2022-01-24T18:36:00.138626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Forecasting and predicting Volume of ETH\nmodel = ARMA(eth[\"Volume\"].diff().iloc[1:].values, order=(0,1)) # changed q from 3 to 1 to resolve error https://stackoverflow.com/questions/52815990/valueerror-the-computed-initial-ma-coefficients-are-not-invertible-you-should-i\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={} ,θ={}\".format(result.params[0],result.params[1]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Volume of ETH')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:00.140776Z","iopub.execute_input":"2022-01-24T18:36:00.141179Z","iopub.status.idle":"2022-01-24T18:36:00.54562Z","shell.execute_reply.started":"2022-01-24T18:36:00.141128Z","shell.execute_reply":"2022-01-24T18:36:00.544629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = eth[\"Volume\"].diff().iloc[1000:1101].values\ncount = len(samples)\nrmse = math.sqrt(mean_squared_error(samples, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:00.546967Z","iopub.execute_input":"2022-01-24T18:36:00.547495Z","iopub.status.idle":"2022-01-24T18:36:00.560048Z","shell.execute_reply.started":"2022-01-24T18:36:00.547429Z","shell.execute_reply":"2022-01-24T18:36:00.558982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, for ARMA models.\n\n## <a id='4.3'>4.3 ARMA models</a>\nAutoregressive–moving-average (ARMA) models provide a parsimonious description of a (weakly) stationary stochastic process in terms of two polynomials, one for the autoregression and the second for the moving average. It's the fusion of AR and MA models.\n### ARMA(1,1) model\n### R<sub>t</sub> = μ + ϕR<sub>t-1</sub> + ϵ<sub>t</sub> + θϵ<sub>t-1</sub> \nBasically, Today's return = mean + Yesterday's return + noise + yesterday's noise.","metadata":{"_uuid":"b2a42dfc2f0f4d38a95e85b6e2531314b1b2865c","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Prediction using ARMA models\nI am not simulating any model because it's quite similar to AR and MA models. Just  forecasting and predictions for this one.","metadata":{"_uuid":"0b163bbf7c903aa7f82d0dae381f019b572e27f5","_kg_hide-input":false}},{"cell_type":"code","source":"%%time\n# Forecasting and predicting Closing price of BTC \nmodel = ARMA(btc[\"Close\"].diff().iloc[1:].values, order=(3,2))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={}, ϕ={}, θ={}\".format(result.params[0],result.params[1],result.params[2]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Close price BTC')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:00.561926Z","iopub.execute_input":"2022-01-24T18:36:00.562573Z","iopub.status.idle":"2022-01-24T18:36:02.551015Z","shell.execute_reply.started":"2022-01-24T18:36:00.562262Z","shell.execute_reply":"2022-01-24T18:36:02.549943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = btc[\"Close\"].diff().iloc[1000:1101].values\ncount = len(samples)\nrmse = math.sqrt(mean_squared_error(samples, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:02.552682Z","iopub.execute_input":"2022-01-24T18:36:02.553318Z","iopub.status.idle":"2022-01-24T18:36:02.566188Z","shell.execute_reply.started":"2022-01-24T18:36:02.55325Z","shell.execute_reply":"2022-01-24T18:36:02.565155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Forecasting and predicting BTC volume\nmodel = ARMA(btc[\"Volume\"].diff().iloc[1:].values, order=(3,3))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={}, ϕ={}, θ={}\".format(result.params[0],result.params[1],result.params[2]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Volume of BTC')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:02.567359Z","iopub.execute_input":"2022-01-24T18:36:02.56778Z","iopub.status.idle":"2022-01-24T18:36:04.974389Z","shell.execute_reply.started":"2022-01-24T18:36:02.567734Z","shell.execute_reply":"2022-01-24T18:36:04.973379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = btc[\"Volume\"].diff().iloc[1000:1101].values\ncount = len(samples)\nrmse = math.sqrt(mean_squared_error(samples, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:04.975333Z","iopub.execute_input":"2022-01-24T18:36:04.975556Z","iopub.status.idle":"2022-01-24T18:36:04.986658Z","shell.execute_reply.started":"2022-01-24T18:36:04.975523Z","shell.execute_reply":"2022-01-24T18:36:04.98564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Forecasting and predicting ETH volume\nmodel = ARMA(eth[\"Volume\"].diff().iloc[1:].values, order=(3,3))\nresult = model.fit()\nprint(result.summary())\nprint(\"μ={}, ϕ={}, θ={}\".format(result.params[0],result.params[1],result.params[2]))\nresult.plot_predict(start=1000, end=1100)\n_ = plt.title('ARMA model: predicting Volume of ETH')\nplt.show()","metadata":{"_uuid":"6b9f4f218af096d41cbeafc00bf15e1d6581c6c3","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:04.987904Z","iopub.execute_input":"2022-01-24T18:36:04.988213Z","iopub.status.idle":"2022-01-24T18:36:06.617937Z","shell.execute_reply.started":"2022-01-24T18:36:04.988155Z","shell.execute_reply":"2022-01-24T18:36:06.617256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = eth[\"Volume\"].diff().iloc[1000:1101].values\ncount = len(samples)\nrmse = math.sqrt(mean_squared_error(samples, result.predict(start=1000,end=999+count)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"b6a1b9a4635ee4d6ad382010f066d229daebbf2c","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:06.618933Z","iopub.execute_input":"2022-01-24T18:36:06.619296Z","iopub.status.idle":"2022-01-24T18:36:06.628602Z","shell.execute_reply.started":"2022-01-24T18:36:06.619257Z","shell.execute_reply":"2022-01-24T18:36:06.627856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ARMA model shows much better results than AR and MA models.","metadata":{"_uuid":"1c337a7fb003b98285a0c3157c0f48742d3a1b9e","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='4.4'>4.4 ARIMA models</a>\nAn autoregressive integrated moving average (ARIMA) model is a generalization of an autoregressive moving average (ARMA) model. Both of these models are fitted to time series data either to better understand the data or to predict future points in the series (forecasting). ARIMA models are applied in some cases where data show evidence of non-stationarity, where an initial differencing step (corresponding to the \"integrated\" part of the model) can be applied one or more times to eliminate the non-stationarity.\nARIMA model is of the form: ARIMA(p,d,q): p is AR parameter, d is differential parameter, q is MA parameter\n### ARIMA(1,0,0)\n**y<sub>t</sub> = a<sub>1</sub>y<sub>t-1</sub> + ϵ<sub>t</sub>**\n### ARIMA(1,0,1)\n**y<sub>t</sub> = a<sub>1</sub>y<sub>t-1</sub> + ϵ<sub>t</sub> + b<sub>1</sub>ϵ<sub>t-1</sub>**\n### ARIMA(1,1,1)\n**Δy<sub>t</sub> = a<sub>1</sub>Δy<sub>t-1</sub> + ϵ<sub>t</sub> + b<sub>1</sub>ϵ<sub>t-1</sub>** where **Δy<sub>t</sub> = y<sub>t</sub> - y<sub>t-1</sub>**","metadata":{"_uuid":"952fcfd1498c44e4541f93dc07f32c34c37cede1","_kg_hide-input":false}},{"cell_type":"markdown","source":"## Prediction using ARIMA model","metadata":{"_uuid":"1f87feaedef7e790890e271e86c4fa2e71c05367","_kg_hide-input":false}},{"cell_type":"code","source":"%%time\n# Predicting the close price of BTC \nrcParams['figure.figsize'] = 16, 6\nmodel = ARIMA(btc[\"Close\"].diff().iloc[1:].values, order=(2,1,0))\nresult = model.fit()\nprint(result.summary())\nresult.plot_predict(start=700, end=1000)\n_ = plt.title('ARIMA model: predicting Close price of BTC')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:06.629723Z","iopub.execute_input":"2022-01-24T18:36:06.630293Z","iopub.status.idle":"2022-01-24T18:36:07.174858Z","shell.execute_reply.started":"2022-01-24T18:36:06.630009Z","shell.execute_reply":"2022-01-24T18:36:07.174252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(btc[\"Close\"].diff().iloc[700:1001].values, result.predict(start=700,end=1000)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:07.1759Z","iopub.execute_input":"2022-01-24T18:36:07.17632Z","iopub.status.idle":"2022-01-24T18:36:07.184583Z","shell.execute_reply.started":"2022-01-24T18:36:07.17627Z","shell.execute_reply":"2022-01-24T18:36:07.183541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting the BTC volume\nrcParams['figure.figsize'] = 16, 6\nmodel = ARIMA(btc[\"Volume\"].diff().iloc[1:].values, order=(2,1,0))\nresult = model.fit()\nprint(result.summary())\nresult.plot_predict(start=700, end=1000)\n_ = plt.title('ARIMA model: predicting Volume of BTC')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:07.185883Z","iopub.execute_input":"2022-01-24T18:36:07.186209Z","iopub.status.idle":"2022-01-24T18:36:07.745234Z","shell.execute_reply.started":"2022-01-24T18:36:07.186155Z","shell.execute_reply":"2022-01-24T18:36:07.744206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(btc[\"Volume\"].diff().iloc[700:1001].values, result.predict(start=700,end=1000)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:07.747133Z","iopub.execute_input":"2022-01-24T18:36:07.747878Z","iopub.status.idle":"2022-01-24T18:36:07.759839Z","shell.execute_reply.started":"2022-01-24T18:36:07.747803Z","shell.execute_reply":"2022-01-24T18:36:07.759036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting the Close price of ETH \nrcParams['figure.figsize'] = 16, 6\nmodel = ARIMA(eth[\"Close\"].diff().iloc[1:].values, order=(2,1,0))\nresult = model.fit()\nprint(result.summary())\nresult.plot_predict(start=700, end=1000)\n_ = plt.title('ARIMA model: predicting Close price of ETH')\nplt.show()","metadata":{"_uuid":"d80a45b8c911f346891534fe2f1c6d519907a86c","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:07.76108Z","iopub.execute_input":"2022-01-24T18:36:07.761552Z","iopub.status.idle":"2022-01-24T18:36:08.286573Z","shell.execute_reply.started":"2022-01-24T18:36:07.76151Z","shell.execute_reply":"2022-01-24T18:36:08.285564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(eth[\"Close\"].diff().iloc[700:1001].values, result.predict(start=700,end=1000)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"feef92dc9eabf930b947f64d46dc90b91afc49a4","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:08.288372Z","iopub.execute_input":"2022-01-24T18:36:08.289081Z","iopub.status.idle":"2022-01-24T18:36:08.297097Z","shell.execute_reply.started":"2022-01-24T18:36:08.289011Z","shell.execute_reply":"2022-01-24T18:36:08.296495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting the ETH volume\nrcParams['figure.figsize'] = 16, 6\nmodel = ARIMA(eth[\"Volume\"].diff().iloc[1:].values, order=(2,1,0))\nresult = model.fit()\nprint(result.summary())\nresult.plot_predict(start=700, end=1000)\n_ = plt.title('ARIMA model: predicting Volume of ETH')\nplt.show()","metadata":{"_uuid":"d80a45b8c911f346891534fe2f1c6d519907a86c","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:08.298198Z","iopub.execute_input":"2022-01-24T18:36:08.298605Z","iopub.status.idle":"2022-01-24T18:36:08.729168Z","shell.execute_reply.started":"2022-01-24T18:36:08.298563Z","shell.execute_reply":"2022-01-24T18:36:08.728199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse = math.sqrt(mean_squared_error(eth[\"Volume\"].diff().iloc[700:1001].values, result.predict(start=700,end=1000)))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"feef92dc9eabf930b947f64d46dc90b91afc49a4","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:08.730402Z","iopub.execute_input":"2022-01-24T18:36:08.73075Z","iopub.status.idle":"2022-01-24T18:36:08.738469Z","shell.execute_reply.started":"2022-01-24T18:36:08.730709Z","shell.execute_reply":"2022-01-24T18:36:08.737683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Taking the slight lag into account, this is a fine model.","metadata":{"_uuid":"99fe66a0b2efcc111ff3c42ef48fac0b08af4c8a","_kg_hide-input":false}},{"cell_type":"markdown","source":"## <a id='4.5'>4.5 VAR models</a>\nVector autoregression (VAR) is a stochastic process model used to capture the linear interdependencies among multiple time series. VAR models generalize the univariate autoregressive model (AR model) by allowing for more than one evolving variable. All variables in a VAR enter the model in the same way: each variable has an equation explaining its evolution based on its own lagged values, the lagged values of the other model variables, and an error term. VAR modeling does not require as much knowledge about the forces influencing a variable as do structural models with simultaneous equations: The only prior knowledge required is a list of variables which can be hypothesized to affect each other intertemporally.\n\n<img src=\"https://miro.medium.com/max/1324/1*nMfvffr_-ktXuCwMq6I78g.png\">\n<img src=\"https://espin086.files.wordpress.com/2011/01/screenhunter_03-jan-16-15-14.gif\">","metadata":{"_uuid":"1c3b1f6da7f472fcc8275a23d9f9fde20a89fc3e","_kg_hide-input":false}},{"cell_type":"code","source":"# %%time\n# # Predicting closing price of BTC and ETH\n## Encountering error: LinAlgError: Singular forecast error covariance matrix encountered at period 133\n# train_sample = pd.concat([btc[\"Close\"].diff().iloc[1:], eth[\"Close\"].diff().iloc[1:]],axis=1)\n# model = sm.tsa.VARMAX(train_sample,order=(2,1),trend='c', enforce_invertibility=False, enforce_stationarity=False) \n# result = model.fit(maxiter=1000,disp=False)\n# print(result.summary())\n# predicted_result = result.predict(start=0, end=1000)\n# result.plot_diagnostics()\n# # calculating error\n# rmse = math.sqrt(mean_squared_error(train_sample.iloc[1:1002].values, predicted_result.values))\n# print(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"c7ee4df651be1857f5eb08666ddd30a64953e20a","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:08.739595Z","iopub.execute_input":"2022-01-24T18:36:08.739853Z","iopub.status.idle":"2022-01-24T18:36:08.750634Z","shell.execute_reply.started":"2022-01-24T18:36:08.739802Z","shell.execute_reply":"2022-01-24T18:36:08.749828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting Volume of BTC and ETH\ntrain_sample = pd.concat([btc[\"Volume\"].diff().iloc[1:], eth[\"Volume\"].diff().iloc[1:]],axis=1)\nmodel = sm.tsa.VARMAX(train_sample,order=(2,1),trend='c', enforce_invertibility=False, enforce_stationarity=False)\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=1000)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample.iloc[1:1002].values, predicted_result.values))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:08.751961Z","iopub.execute_input":"2022-01-24T18:36:08.752251Z","iopub.status.idle":"2022-01-24T18:36:19.858159Z","shell.execute_reply.started":"2022-01-24T18:36:08.752202Z","shell.execute_reply":"2022-01-24T18:36:19.857296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='4.6'>4.6 State Space methods</a>\n\nA general state space model is of the form\n\ny<sub>t</sub>=Z<sub>t</sub>α<sub>t</sub>+d<sub>t</sub>+ε<sub>t</sub>\n\nα<sub>t</sub>=T<sub>t</sub>α<sub>t</sub>−1+c<sub>t</sub>+R<sub>t</sub>η<sub>t</sub>\n\nwhere y<sub>t</sub> refers to the observation vector at time t, α<sub>t</sub> refers to the (unobserved) state vector at time t, and where the irregular components are defined as\n\nε<sub>t</sub>∼N(0,H<sub>t</sub>)\n\nη<sub>t</sub>∼N(0,Q<sub>t</sub>)\n\nThe remaining variables (Z<sub>t</sub>,d<sub>t</sub>,H<sub>t</sub>,T<sub>t</sub>,c<sub>t</sub>,R<sub>t</sub>,Q<sub>t</sub>) in the equations are matrices describing the process. Their variable names and dimensions are as follows\n\nZ : design (k_endog×k_states×nobs)\n\nd : obs_intercept (k_endog×nobs)\n\nH : obs_cov (k_endog×k_endog×nobs)\n\nT : transition (k_states×k_states×nobs)\n\nc : state_intercept (k_states×nobs)\n\nR : selection (k_states×k_posdef×nobs)\n\nQ : state_cov (k_posdef×k_posdef×nobs)\n\nIn the case that one of the matrices is time-invariant (so that, for example, Z<sub>t</sub>=Z<sub>t</sub>+1 ∀ <sub>t</sub>), its last dimension may be of size 1 rather than size nobs.\n\nThis generic form encapsulates many of the most popular linear time series models (see below) and is very flexible, allowing estimation with missing observations, forecasting, impulse response functions, and much more.\n\nSource: [statsmodels](https://www.statsmodels.org/dev/statespace.html)","metadata":{"_uuid":"a20e3db98f52bb92143793d651bc280884ecd068","_kg_hide-input":false,"trusted":true}},{"cell_type":"markdown","source":"## <a id='4.6.1'>4.6.1 SARIMA models</a>\nSARIMA models are useful for modeling seasonal time series, in which the mean and other statistics for a given season are not stationary across the years. The SARIMA model defined constitutes a straightforward extension of the nonseasonal autoregressive-moving average (ARMA) and autoregressive integrated moving average (ARIMA) models presented","metadata":{"_uuid":"aa056e8b4f0c457639f5cfb01cfa6f9d6b1fb486","_kg_hide-input":false}},{"cell_type":"code","source":"%%time\n# Predicting closing price of BTC'\ntrain_sample = btc[\"Close\"].diff().iloc[1:].values\n\nmodel = sm.tsa.SARIMAX(train_sample,order=(4,0,4),trend='c', enforce_invertibility=False, enforce_stationarity=False)\nresult = model.fit(maxiter=1000, disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"8441c5af41b070d19fd711dee43b118855676862","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:19.859637Z","iopub.execute_input":"2022-01-24T18:36:19.859947Z","iopub.status.idle":"2022-01-24T18:36:22.556321Z","shell.execute_reply.started":"2022-01-24T18:36:19.859899Z","shell.execute_reply":"2022-01-24T18:36:22.555434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('SARIMA model: predicting BTC Closing prices')\nplt.show()","metadata":{"_uuid":"6d9a2a2fa5762719f770406f9f37fb04af4cb384","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:22.557558Z","iopub.execute_input":"2022-01-24T18:36:22.557849Z","iopub.status.idle":"2022-01-24T18:36:22.754655Z","shell.execute_reply.started":"2022-01-24T18:36:22.557803Z","shell.execute_reply":"2022-01-24T18:36:22.753823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting Volume of BTC'\ntrain_sample = btc[\"Volume\"].diff().iloc[1:].values\n\nmodel = sm.tsa.SARIMAX(train_sample,order=(4,0,4),trend='c', enforce_invertibility=False, enforce_stationarity=False)\nresult = model.fit(maxiter=1000, disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"8441c5af41b070d19fd711dee43b118855676862","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:22.755968Z","iopub.execute_input":"2022-01-24T18:36:22.756466Z","iopub.status.idle":"2022-01-24T18:36:25.401638Z","shell.execute_reply.started":"2022-01-24T18:36:22.756408Z","shell.execute_reply":"2022-01-24T18:36:25.400771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('SARIMA model: predicting BTC Volume')\nplt.show()","metadata":{"_uuid":"6d9a2a2fa5762719f770406f9f37fb04af4cb384","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:25.402888Z","iopub.execute_input":"2022-01-24T18:36:25.403364Z","iopub.status.idle":"2022-01-24T18:36:25.616509Z","shell.execute_reply.started":"2022-01-24T18:36:25.403303Z","shell.execute_reply":"2022-01-24T18:36:25.615561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_sample = eth[\"Close\"].diff().iloc[1:].values\nmodel = sm.tsa.SARIMAX(train_sample,order=(4,0,4),trend='c', enforce_invertibility=False, enforce_stationarity=False)\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:25.618036Z","iopub.execute_input":"2022-01-24T18:36:25.618483Z","iopub.status.idle":"2022-01-24T18:36:28.303825Z","shell.execute_reply.started":"2022-01-24T18:36:25.618328Z","shell.execute_reply":"2022-01-24T18:36:28.302849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('SARIMA model: predicting ETH Closing prices')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:28.305367Z","iopub.execute_input":"2022-01-24T18:36:28.305905Z","iopub.status.idle":"2022-01-24T18:36:28.610428Z","shell.execute_reply.started":"2022-01-24T18:36:28.305839Z","shell.execute_reply":"2022-01-24T18:36:28.609473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_sample = eth[\"Volume\"].diff().iloc[1:].values\nmodel = sm.tsa.SARIMAX(train_sample,order=(4,0,4),trend='c', enforce_invertibility=False, enforce_stationarity=False)\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:28.611893Z","iopub.execute_input":"2022-01-24T18:36:28.612467Z","iopub.status.idle":"2022-01-24T18:36:30.964063Z","shell.execute_reply.started":"2022-01-24T18:36:28.612403Z","shell.execute_reply":"2022-01-24T18:36:30.963011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('SARIMA model: predicting ETH Volume')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:30.965977Z","iopub.execute_input":"2022-01-24T18:36:30.96687Z","iopub.status.idle":"2022-01-24T18:36:31.278703Z","shell.execute_reply.started":"2022-01-24T18:36:30.96673Z","shell.execute_reply":"2022-01-24T18:36:31.277591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='4.6.2'>4.6.2 Unobserved components</a>\nA UCM decomposes the response series into components such as trend, seasons, cycles, and the regression effects due to predictor series. The following model shows a possible scenario:\n<img src=\"http://support.sas.com/documentation/cdl/en/etsug/66840/HTML/default/images/etsug_ucm0134.png\">\nSource: [http://support.sas.com/documentation/cdl/en/etsug/66840/HTML/default/viewer.htm#etsug_ucm_details01.htm](http://support.sas.com/documentation/cdl/en/etsug/66840/HTML/default/viewer.htm#etsug_ucm_details01.htm)","metadata":{"_uuid":"ba27f063b177faca83f659c91981ebbcf3d32513","_kg_hide-input":false}},{"cell_type":"code","source":"%%time\n# Predicting closing price of BTC'\ntrain_sample = btc[\"Close\"].diff().iloc[1:].values\nmodel = sm.tsa.UnobservedComponents(train_sample,'local level')\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"eccfc1ab3a1ffec0f739f09c3c52e9a4e40e93c0","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:31.280484Z","iopub.execute_input":"2022-01-24T18:36:31.280998Z","iopub.status.idle":"2022-01-24T18:36:32.416306Z","shell.execute_reply.started":"2022-01-24T18:36:31.280798Z","shell.execute_reply":"2022-01-24T18:36:32.412478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('Unobserved component: predicting BTC Closing prices')\nplt.show()","metadata":{"_uuid":"5ef27e360e92c6603a80ab4072c8101416de04a3","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:32.417717Z","iopub.execute_input":"2022-01-24T18:36:32.418044Z","iopub.status.idle":"2022-01-24T18:36:32.726458Z","shell.execute_reply.started":"2022-01-24T18:36:32.417992Z","shell.execute_reply":"2022-01-24T18:36:32.725353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting Volume of BTC'\ntrain_sample = btc[\"Volume\"].diff().iloc[1:].values\nmodel = sm.tsa.UnobservedComponents(train_sample,'local level')\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"eccfc1ab3a1ffec0f739f09c3c52e9a4e40e93c0","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:32.728265Z","iopub.execute_input":"2022-01-24T18:36:32.728774Z","iopub.status.idle":"2022-01-24T18:36:33.815024Z","shell.execute_reply.started":"2022-01-24T18:36:32.728614Z","shell.execute_reply":"2022-01-24T18:36:33.814048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('Unobserved component: predicting Volume of BTC')\nplt.show()","metadata":{"_uuid":"5ef27e360e92c6603a80ab4072c8101416de04a3","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:33.816935Z","iopub.execute_input":"2022-01-24T18:36:33.817649Z","iopub.status.idle":"2022-01-24T18:36:34.133482Z","shell.execute_reply.started":"2022-01-24T18:36:33.817584Z","shell.execute_reply":"2022-01-24T18:36:34.132449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting closing price of ETH'\ntrain_sample = eth[\"Close\"].diff().iloc[1:].values\nmodel = sm.tsa.UnobservedComponents(train_sample,'local level')\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:34.135314Z","iopub.execute_input":"2022-01-24T18:36:34.135987Z","iopub.status.idle":"2022-01-24T18:36:35.25408Z","shell.execute_reply.started":"2022-01-24T18:36:34.135922Z","shell.execute_reply":"2022-01-24T18:36:35.253141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('Unobserved component: predicting ETH Closing prices')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:35.255495Z","iopub.execute_input":"2022-01-24T18:36:35.255821Z","iopub.status.idle":"2022-01-24T18:36:35.561001Z","shell.execute_reply.started":"2022-01-24T18:36:35.255771Z","shell.execute_reply":"2022-01-24T18:36:35.560028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Predicting Volume of ETH'\ntrain_sample = eth[\"Volume\"].diff().iloc[1:].values\nmodel = sm.tsa.UnobservedComponents(train_sample,'local level')\nresult = model.fit(maxiter=1000,disp=False)\nprint(result.summary())\npredicted_result = result.predict(start=0, end=500)\nresult.plot_diagnostics()\n# calculating error\nrmse = math.sqrt(mean_squared_error(train_sample[1:502], predicted_result))\nprint(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:35.562513Z","iopub.execute_input":"2022-01-24T18:36:35.562858Z","iopub.status.idle":"2022-01-24T18:36:36.670489Z","shell.execute_reply.started":"2022-01-24T18:36:35.562794Z","shell.execute_reply":"2022-01-24T18:36:36.669515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(train_sample[1:502],color='red')\nplt.plot(predicted_result,color='blue')\nplt.legend(['Actual','Predicted'])\nplt.title('Unobserved component: predicting Volume of ETH')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:36.672083Z","iopub.execute_input":"2022-01-24T18:36:36.672638Z","iopub.status.idle":"2022-01-24T18:36:36.981136Z","shell.execute_reply.started":"2022-01-24T18:36:36.672577Z","shell.execute_reply":"2022-01-24T18:36:36.980163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='4.6.3'>4.6.3 Dynamic Factor models</a>\nDynamic-factor models are flexible models for multivariate time series in which the observed endogenous variables are linear functions of exogenous covariates and unobserved factors, which have a vector autoregressive structure. The unobserved factors may also be a function of exogenous covariates. The disturbances in the equations for the dependent variables may be autocorrelated.","metadata":{"_uuid":"4d041bf48f3f835d8b9a397f066c21577c0cd235","_kg_hide-input":false}},{"cell_type":"code","source":"def extract_subset(full_dataset: pd.DataFrame, percentage_to_extract: float = 0.10) -> pd.DataFrame:\n    print(f'Before taking a sample: {full_dataset.shape}')\n    subset_size = int(full_dataset.shape[0] * percentage_to_extract)\n    result = full_dataset[-subset_size:].copy()\n    print(f'After taking a sample: {result.shape}')\n    return result","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:36.982654Z","iopub.execute_input":"2022-01-24T18:36:36.983227Z","iopub.status.idle":"2022-01-24T18:36:36.993825Z","shell.execute_reply.started":"2022-01-24T18:36:36.983162Z","shell.execute_reply":"2022-01-24T18:36:36.992942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"percentage_to_extract = 0.005","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:36.994979Z","iopub.execute_input":"2022-01-24T18:36:36.995266Z","iopub.status.idle":"2022-01-24T18:36:37.007788Z","shell.execute_reply.started":"2022-01-24T18:36:36.995218Z","shell.execute_reply":"2022-01-24T18:36:37.00704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del btc\nbtc = extract_subset(btc_full, percentage_to_extract)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.009185Z","iopub.execute_input":"2022-01-24T18:36:37.009572Z","iopub.status.idle":"2022-01-24T18:36:37.022556Z","shell.execute_reply.started":"2022-01-24T18:36:37.009391Z","shell.execute_reply":"2022-01-24T18:36:37.021599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del eth\neth = extract_subset(eth_full, percentage_to_extract)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.023811Z","iopub.execute_input":"2022-01-24T18:36:37.024068Z","iopub.status.idle":"2022-01-24T18:36:37.035551Z","shell.execute_reply.started":"2022-01-24T18:36:37.024026Z","shell.execute_reply":"2022-01-24T18:36:37.034662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.036902Z","iopub.execute_input":"2022-01-24T18:36:37.037498Z","iopub.status.idle":"2022-01-24T18:36:37.235918Z","shell.execute_reply.started":"2022-01-24T18:36:37.037442Z","shell.execute_reply":"2022-01-24T18:36:37.235092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n##### Currently throws an error, hence disabled: LinAlgError: Singular forecast error covariance matrix encountered at period 1 or period 0\n# Predicting closing price of BTC and ETH\n# train_sample = pd.concat([btc[\"Close\"].diff().iloc[1:], eth[\"Close\"].diff().iloc[1:]],axis=1)\n# train_sample = train_sample.fillna(0.0) # Ideally we should not need to do this\n# model = sm.tsa.DynamicFactor(train_sample, k_factors=1, factor_order=2)\n# result = model.fit(maxiter=1000, disp=False)\n# print(result.summary())\n# predicted_result = result.predict(start=0, end=1000)\n# result.plot_diagnostics()\n# # calculating error\n# rmse = math.sqrt(mean_squared_error(train_sample.iloc[1:1002].values, predicted_result.values))\n# print(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"d6d0a1d158db53132b9b0406ca971da7820af338","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.237347Z","iopub.execute_input":"2022-01-24T18:36:37.237757Z","iopub.status.idle":"2022-01-24T18:36:37.251084Z","shell.execute_reply.started":"2022-01-24T18:36:37.237679Z","shell.execute_reply":"2022-01-24T18:36:37.250354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predicted_result.columns=['BTC Prediction', 'ETH Prediction']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.252403Z","iopub.execute_input":"2022-01-24T18:36:37.25285Z","iopub.status.idle":"2022-01-24T18:36:37.261932Z","shell.execute_reply.started":"2022-01-24T18:36:37.252671Z","shell.execute_reply":"2022-01-24T18:36:37.261099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plt.plot(btc[\"Close\"].diff().iloc[1:1000])\n# plt.plot(predicted_result['BTC Prediction'])\n# plt.legend(['Actual','Predicted'])\n# plt.title('Dynamic Factor models: predicting BTC Closing prices')\n# plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.263249Z","iopub.execute_input":"2022-01-24T18:36:37.263718Z","iopub.status.idle":"2022-01-24T18:36:37.27282Z","shell.execute_reply.started":"2022-01-24T18:36:37.26351Z","shell.execute_reply":"2022-01-24T18:36:37.272237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plt.plot(eth[\"Close\"].diff().iloc[1:1000])\n# plt.plot(predicted_result['ETH Prediction'])\n# plt.legend(['Actual','Predicted'])\n# plt.title('Dynamic Factor models: predicting ETH Closing prices')\n# plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.273897Z","iopub.execute_input":"2022-01-24T18:36:37.274392Z","iopub.status.idle":"2022-01-24T18:36:37.287477Z","shell.execute_reply.started":"2022-01-24T18:36:37.274338Z","shell.execute_reply":"2022-01-24T18:36:37.286779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n##### Currently throws an error, hence disabled: LinAlgError: Singular forecast error covariance matrix encountered at period 0\n# # Predicting Volume of BTC and ETH\n# train_sample = pd.concat([btc[\"Volume\"].diff().iloc[1:], eth[\"Volume\"].diff().iloc[1:]],axis=1)\n# train_sample = train_sample.fillna(0.0) # Ideally we should not need to do this\n# model = sm.tsa.DynamicFactor(train_sample, k_factors=1, factor_order=2)\n# result = model.fit(maxiter=1000, disp=False)\n# print(result.summary())\n# predicted_result = result.predict(start=0, end=1000)\n# result.plot_diagnostics()\n# # calculating error\n# rmse = math.sqrt(mean_squared_error(train_sample.iloc[1:1002].values, predicted_result.values))\n# print(\"The root mean squared error is {}.\".format(rmse))","metadata":{"_uuid":"d6d0a1d158db53132b9b0406ca971da7820af338","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.288417Z","iopub.execute_input":"2022-01-24T18:36:37.288798Z","iopub.status.idle":"2022-01-24T18:36:37.2999Z","shell.execute_reply.started":"2022-01-24T18:36:37.288716Z","shell.execute_reply":"2022-01-24T18:36:37.299253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predicted_result.columns=['BTC Prediction', 'ETH Prediction']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.300955Z","iopub.execute_input":"2022-01-24T18:36:37.301238Z","iopub.status.idle":"2022-01-24T18:36:37.317628Z","shell.execute_reply.started":"2022-01-24T18:36:37.301181Z","shell.execute_reply":"2022-01-24T18:36:37.316747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plt.plot(btc[\"Volume\"].diff().iloc[1:1000])\n# plt.plot(predicted_result['BTC Prediction'])\n# plt.legend(['Actual','Predicted'])\n# plt.title('Dynamic Factor models: predicting Volume of BTC')\n# plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.318652Z","iopub.execute_input":"2022-01-24T18:36:37.319006Z","iopub.status.idle":"2022-01-24T18:36:37.329706Z","shell.execute_reply.started":"2022-01-24T18:36:37.318969Z","shell.execute_reply":"2022-01-24T18:36:37.329066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plt.plot(eth[\"Volume\"].diff().iloc[1:1000])\n# plt.plot(predicted_result['ETH Prediction'])\n# plt.legend(['Actual','Predicted'])\n# plt.title('Dynamic Factor models: predicting Volume of ETH')\n# plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-24T18:36:37.330599Z","iopub.execute_input":"2022-01-24T18:36:37.330972Z","iopub.status.idle":"2022-01-24T18:36:37.341627Z","shell.execute_reply.started":"2022-01-24T18:36:37.330935Z","shell.execute_reply":"2022-01-24T18:36:37.340888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I may add more regression models soon and there is much more stuff to cover. **But in my experience, the best models for time series forecasting are LSTM based Recurrent Neural Networks. I have prepared a detailed tutorial for that. Here is the link: <u>https://www.kaggle.com/thebrownviking20/intro-to-recurrent-neural-networks-lstm-gru</u>**","metadata":{"_uuid":"41fef604d6c49239a262468401fb548fed77ad7b","_kg_hide-input":false}},{"cell_type":"markdown","source":"References and influences (these have more in-depth content and explanations): \n* [Manipulating Time Series Data in Python](https://www.datacamp.com/courses/manipulating-time-series-data-in-python)\n* [Introduction to Time Series Analysis in Python](https://www.datacamp.com/courses/introduction-to-time-series-analysis-in-python)\n* [Visualizing Time Series Data in Python](https://www.datacamp.com/courses/visualizing-time-series-data-in-python)\n* [VAR models and LSTM](https://www.youtube.com/watch?v=_vQ0W_qXMxk)\n* [State space models](https://www.statsmodels.org/dev/statespace.html)\n* [Awesome AI-ML-DL: Time-series / anomaly detection](https://github.com/neomatrix369/awesome-ai-ml-dl/tree/master/time-series_anomaly-detection)\n\nStay tuned for more! And don't forget to upvote and comment.","metadata":{"_uuid":"920d7a4a65cb159c6c2ef3ba9701716f4eaad548","_kg_hide-input":false}}]}