{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n# G-Research Crypto Forecasting - Prediction\n\n---","metadata":{}},{"cell_type":"markdown","source":"**Problem Statement:**\n\n* Over $40 billion worth of cryptocurrencies are traded every day. \n * They are among the most popular assets for speculation and investment, yet have proven wildly volatile. \n * Fast-fluctuating prices have made millionaires of a lucky few, and delivered crushing losses to others. \n* Could some of these price movements have been predicted in advance?\n\n* Use machine learning expertise to forecast short term returns in 14 popular cryptocurrencies. \n* Dataset contains millions of rows of high-frequency market data dating back to 2018 which can be used to build ML model. \n* Once the submission deadline has passed, final score will be calculated over the following 3 months using live crypto data as it is collected. \n \n---\n \n**Main Dataset:**\n\n* **train.csv** file contains training set.\n* **example_test.csv** file includes example of the data that will be delivered by the time series API.\n* **example_sample_submission.csv** file includes example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n* **asset_details.csv** file provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n* **supplemental_train.csv**- After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period.\n\n**gresearch_crypto** - An unoptimized version of the time series API files for offline work. You may need Python 3.7 and a Linux environment to run it without errors.\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Importing Libraries:**\n\n* To get started we will use Python for data pre-processing and data analysis.\n\n* Import python libraries as necessary to get started for data load and later import other libraries as needed\n\n---","metadata":{}},{"cell_type":"code","source":"# Custom G-Research python module needed\nimport gresearch_crypto\n\n# Pandas for data manipulation\nimport pandas as pd\nimport numpy as np\n\n# module provides a portable way of using operating system dependent functionality\nimport os \n\n # Importing pyplot interface using matplotlib\nimport matplotlib.pyplot as plt \n\n# Importing seaborn library for interactive visualization\nimport seaborn as sns \n\n# Importing WordCloud for text data visualization\nfrom wordcloud import WordCloud\n\n# Importing matplotlib for plots\nimport matplotlib as mpl\n\n#Importing datetime for using datetime\nfrom datetime import datetime\n\n#Importing plotly Express for visualization\nimport plotly.express as px\n\n# Importing statsmodel for statistical methods\nimport statsmodels.api as sm\nfrom statsmodels.tsa.stattools import adfuller\n\n#Importing for customizing properties in matplotlib\nfrom pylab import rcParams\n\nimport itertools","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:34:57.164591Z","iopub.execute_input":"2022-01-29T12:34:57.165489Z","iopub.status.idle":"2022-01-29T12:35:00.902659Z","shell.execute_reply.started":"2022-01-29T12:34:57.165351Z","shell.execute_reply":"2022-01-29T12:35:00.901966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Custom G-Research python module requires this step\n#env = gresearch_crypto.make_env()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:35:00.903982Z","iopub.execute_input":"2022-01-29T12:35:00.904564Z","iopub.status.idle":"2022-01-29T12:35:00.907168Z","shell.execute_reply.started":"2022-01-29T12:35:00.904517Z","shell.execute_reply":"2022-01-29T12:35:00.906610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display files available in this competition folder\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-29T12:35:00.908202Z","iopub.execute_input":"2022-01-29T12:35:00.908611Z","iopub.status.idle":"2022-01-29T12:35:00.927276Z","shell.execute_reply.started":"2022-01-29T12:35:00.908578Z","shell.execute_reply":"2022-01-29T12:35:00.926396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Data Definition/Description\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Data Definition:**\n\n---\n\n* **train.csv:** This dataset contains information on historic trades for several cryptoassets.","metadata":{}},{"cell_type":"code","source":"# load training data\ntrain_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv', low_memory=False)\n\n# load asset details data\nasset_details_df = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv', low_memory=False)\n\n# load example test data\ntest_df = pd.read_csv('../input/g-research-crypto-forecasting/example_test.csv', low_memory=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:35:00.929395Z","iopub.execute_input":"2022-01-29T12:35:00.929703Z","iopub.status.idle":"2022-01-29T12:36:05.352493Z","shell.execute_reply.started":"2022-01-29T12:35:00.929672Z","shell.execute_reply":"2022-01-29T12:36:05.351637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the structure of train dataset?**\n\n---","metadata":{}},{"cell_type":"markdown","source":"| No. | Feature Name | Description of the feature |\n| :-- | :--| :--| \n|01| **timestamp**   | A timestamp for the minute covered by the row |\n|02| **Asset_ID** | An ID code for the cryptoasset                 |\n|03| **Count**   | The number of trades that took place this minute                 |\n|04| **Open**   | The USD price at the beginning of the minute  |\n|05| **High**   | The highest USD price during the minute|\n|06| **Low**   | The lowest USD price during the minute  |\n|07| **Close**   | The USD price at the end of the minute|\n|08| **Volume**   | The number of cryptoasset units traded during the minute|\n|09| **VWAP**   | The volume weighted average price for the minute  |\n|10| **Target**   | 15 minute residualized returns|","metadata":{}},{"cell_type":"code","source":"# get shape of dataframe\nprint('Shape of train dataset is:', train_df.shape)\n\n# print summary of dataframe\ntrain_df.info(show_counts=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:05.353626Z","iopub.execute_input":"2022-01-29T12:36:05.353853Z","iopub.status.idle":"2022-01-29T12:36:06.023710Z","shell.execute_reply.started":"2022-01-29T12:36:05.353819Z","shell.execute_reply":"2022-01-29T12:36:06.022696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**train dataset information:**\n\n* There are 24236806 data points (rows) and 10 feature (column) in train dataset.\n* There are seven columns which are of numerical float type and three column of numerical int type.\n* There are missing values (non-null count is not same) for VWAP and Target columns.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data looks like for train dataset?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print first 10 rows of dataframe\ntrain_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:06.025058Z","iopub.execute_input":"2022-01-29T12:36:06.025358Z","iopub.status.idle":"2022-01-29T12:36:06.051916Z","shell.execute_reply.started":"2022-01-29T12:36:06.025323Z","shell.execute_reply":"2022-01-29T12:36:06.051135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does data looks like for asset_details dataset?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print summary of asset details dataframe\nasset_details_df.sort_values(by=['Weight'], ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:06.052951Z","iopub.execute_input":"2022-01-29T12:36:06.053744Z","iopub.status.idle":"2022-01-29T12:36:06.067468Z","shell.execute_reply.started":"2022-01-29T12:36:06.053707Z","shell.execute_reply":"2022-01-29T12:36:06.066631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the statistics description for train dataset?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print descriptive statistics for train dataset\ntrain_df.describe(include='all').round(1)","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:36:06.069019Z","iopub.execute_input":"2022-01-29T12:36:06.069241Z","iopub.status.idle":"2022-01-29T12:36:15.447204Z","shell.execute_reply.started":"2022-01-29T12:36:06.069215Z","shell.execute_reply":"2022-01-29T12:36:15.446248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**train dataset data description:**\n\n* There are no missing values for timestamp and appears to be having normal distribution of datapoints.\n* There are no missing values for Asset_ID and appears to be having normal distribution of datapoints.\n* There are no missing values for Count, Open, High, Low, Close and Volume but appears to be having skewed distribution of datapoints.\n* There are missing values for VWAP and Target but appears to be having normal distribution of datapoints.","metadata":{}},{"cell_type":"markdown","source":"Need to merge train_df and asset_details_df","metadata":{}},{"cell_type":"code","source":"# merge dataframe using Asset_ID as key\ngresearch_crypto_df = pd.merge(train_df,asset_details_df,on=['Asset_ID'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:15.448886Z","iopub.execute_input":"2022-01-29T12:36:15.449186Z","iopub.status.idle":"2022-01-29T12:36:21.232630Z","shell.execute_reply.started":"2022-01-29T12:36:15.449145Z","shell.execute_reply":"2022-01-29T12:36:21.231756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print first 10 rows of merged dataframe\ngresearch_crypto_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:21.236821Z","iopub.execute_input":"2022-01-29T12:36:21.237045Z","iopub.status.idle":"2022-01-29T12:36:21.257205Z","shell.execute_reply.started":"2022-01-29T12:36:21.237019Z","shell.execute_reply":"2022-01-29T12:36:21.256621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print last 10 rows of merged dataframe\ngresearch_crypto_df.tail(10)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-01-29T12:36:21.258267Z","iopub.execute_input":"2022-01-29T12:36:21.259062Z","iopub.status.idle":"2022-01-29T12:36:21.286591Z","shell.execute_reply.started":"2022-01-29T12:36:21.259018Z","shell.execute_reply":"2022-01-29T12:36:21.285882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Handle missing value, will drop them","metadata":{}},{"cell_type":"code","source":"# drop missing value from dataframe\ngresearch_crypto_df.dropna(axis=0, inplace=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:21.287885Z","iopub.execute_input":"2022-01-29T12:36:21.288094Z","iopub.status.idle":"2022-01-29T12:36:29.383933Z","shell.execute_reply.started":"2022-01-29T12:36:21.288069Z","shell.execute_reply":"2022-01-29T12:36:29.383135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print summary of dataframe\ngresearch_crypto_df.info(show_counts=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:29.385372Z","iopub.execute_input":"2022-01-29T12:36:29.385707Z","iopub.status.idle":"2022-01-29T12:36:32.530060Z","shell.execute_reply.started":"2022-01-29T12:36:29.385665Z","shell.execute_reply":"2022-01-29T12:36:32.528913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Model Building/Prediction\n\n---","metadata":{}},{"cell_type":"code","source":"# convert timestamp to Datetime for further analysis\ngresearch_crypto_df['Datetime'] = pd.to_datetime(gresearch_crypto_df['timestamp'], unit='s')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:32.531571Z","iopub.execute_input":"2022-01-29T12:36:32.531832Z","iopub.status.idle":"2022-01-29T12:36:33.018527Z","shell.execute_reply.started":"2022-01-29T12:36:32.531793Z","shell.execute_reply":"2022-01-29T12:36:33.017788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lookup asset name and return asset series\ndef asset_lookup (asset_name):\n    # check for Asset_Name\n    crypto_asset =gresearch_crypto_df.loc[gresearch_crypto_df['Asset_Name'] == asset_name]\n    # drop all other columns as of now\n    crypto_asset_df= crypto_asset.drop(columns=['timestamp','Asset_ID','Count','Open','High','Low','Close','Volume','VWAP','Weight','Asset_Name'], axis=1)\n    # sort by Datetime and set index\n    crypto_asset_df = crypto_asset_df.sort_values('Datetime')\n    crypto_asset_df = crypto_asset_df.groupby('Datetime')['Target'].sum().reset_index()\n    crypto_asset_df = crypto_asset_df.set_index('Datetime')\n    # group the data by business day\n    return crypto_asset_df['Target'].resample('B').mean()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:33.019965Z","iopub.execute_input":"2022-01-29T12:36:33.020412Z","iopub.status.idle":"2022-01-29T12:36:33.027627Z","shell.execute_reply.started":"2022-01-29T12:36:33.020380Z","shell.execute_reply":"2022-01-29T12:36:33.026473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check for stationarity using ADF Test for a given asset series\ndef asset_stationarity (asset_series):\n    result = adfuller(asset_series)\n    print('ADF Statistic: %f' % result[0])\n    print('p-value: %f' % result[1])\n    print('Critical Values:')\n    for key, value in result[4].items():\n     print('\\t%s: %.3f' % (key, value))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:33.028915Z","iopub.execute_input":"2022-01-29T12:36:33.029159Z","iopub.status.idle":"2022-01-29T12:36:33.044295Z","shell.execute_reply.started":"2022-01-29T12:36:33.029129Z","shell.execute_reply":"2022-01-29T12:36:33.043536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualization for trend,seasonality and residual for a given asset series\ndef asset_decompose (asset_series):\n    rcParams['figure.figsize'] = 18, 8\n    decomposition = sm.tsa.seasonal_decompose(asset_series, model='additive')\n    fig = decomposition.plot()\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:33.045762Z","iopub.execute_input":"2022-01-29T12:36:33.046377Z","iopub.status.idle":"2022-01-29T12:36:33.059835Z","shell.execute_reply.started":"2022-01-29T12:36:33.046330Z","shell.execute_reply":"2022-01-29T12:36:33.059148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Lets explore ARIMA model for Bitcoin**","metadata":{}},{"cell_type":"code","source":"# create series for Asset_Name Bitcoin\nbitcoin = asset_lookup('Bitcoin')\nbitcoin.isna().sum()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:33.060738Z","iopub.execute_input":"2022-01-29T12:36:33.060955Z","iopub.status.idle":"2022-01-29T12:36:37.350464Z","shell.execute_reply.started":"2022-01-29T12:36:33.060927Z","shell.execute_reply":"2022-01-29T12:36:37.349604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check asset stationarity\nasset_stationarity (bitcoin)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:37.351832Z","iopub.execute_input":"2022-01-29T12:36:37.352099Z","iopub.status.idle":"2022-01-29T12:36:37.422928Z","shell.execute_reply.started":"2022-01-29T12:36:37.352065Z","shell.execute_reply":"2022-01-29T12:36:37.422105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize trend, seasonality and residual\nasset_decompose(bitcoin)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:37.424602Z","iopub.execute_input":"2022-01-29T12:36:37.425169Z","iopub.status.idle":"2022-01-29T12:36:38.495711Z","shell.execute_reply.started":"2022-01-29T12:36:37.425113Z","shell.execute_reply":"2022-01-29T12:36:38.494734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ARIMA models are denoted by p,d,q which are seasonality, trend and residual (noise)\np = d = q = range(0, 2)\npdq = list(itertools.product(p, d, q))\nseasonal_pdq = [(x[0], x[1], x[2], 12) for x in list(itertools.product(p, d, q))]\nprint('Examples of parameter combinations for Seasonal ARIMA...')\nprint('SARIMAX: {} x {}'.format(pdq[1], seasonal_pdq[1]))\nprint('SARIMAX: {} x {}'.format(pdq[1], seasonal_pdq[2]))\nprint('SARIMAX: {} x {}'.format(pdq[2], seasonal_pdq[3]))\nprint('SARIMAX: {} x {}'.format(pdq[2], seasonal_pdq[4]))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:38.497165Z","iopub.execute_input":"2022-01-29T12:36:38.497559Z","iopub.status.idle":"2022-01-29T12:36:38.507266Z","shell.execute_reply.started":"2022-01-29T12:36:38.497503Z","shell.execute_reply":"2022-01-29T12:36:38.506670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# search for best parameter combination for ARIMA model\nfor param in pdq:\n    for param_seasonal in seasonal_pdq:\n        try:\n            mod = sm.tsa.statespace.SARIMAX(bitcoin,order=param,seasonal_order=param_seasonal,enforce_stationarity=False,enforce_invertibility=False)\n            results = mod.fit()\n            print('ARIMA{}x{}12 - AIC:{}'.format(param, param_seasonal, results.aic))\n        except:\n            continue","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-29T12:36:38.508390Z","iopub.execute_input":"2022-01-29T12:36:38.509121Z","iopub.status.idle":"2022-01-29T12:37:31.630488Z","shell.execute_reply.started":"2022-01-29T12:36:38.509087Z","shell.execute_reply":"2022-01-29T12:37:31.629617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets fit ARIMA model\nmodel = sm.tsa.statespace.SARIMAX(bitcoin,\n                                order=(1, 1, 1),\n                                seasonal_order=(1, 1, 1, 12),\n                                enforce_stationarity=False,\n                                enforce_invertibility=False)\nresults = model.fit()\nprint(results.summary().tables[1])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:31.632039Z","iopub.execute_input":"2022-01-29T12:37:31.632346Z","iopub.status.idle":"2022-01-29T12:37:33.349361Z","shell.execute_reply.started":"2022-01-29T12:37:31.632314Z","shell.execute_reply":"2022-01-29T12:37:33.348490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does model diagnostics visualization looks like for Bitcoin?**\n\n---","metadata":{}},{"cell_type":"code","source":"# model diagnostics to check for errors\nresults.plot_diagnostics(figsize=(16, 8))\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:33.350879Z","iopub.execute_input":"2022-01-29T12:37:33.351270Z","iopub.status.idle":"2022-01-29T12:37:34.152086Z","shell.execute_reply.started":"2022-01-29T12:37:33.351216Z","shell.execute_reply":"2022-01-29T12:37:34.151228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does One-step ahead forecast looks like for Bitcoin?**\n\n---","metadata":{}},{"cell_type":"code","source":"# validate forecast\npred = results.get_prediction(start=pd.to_datetime('2021-01-01'), dynamic=False)\npred_ci = pred.conf_int()\nax = bitcoin['2018':].plot(label='observed')\npred.predicted_mean.plot(ax=ax, label='One-step ahead Forecast', alpha=.7, figsize=(14, 7))\nax.fill_between(pred_ci.index,\n                pred_ci.iloc[:, 0],\n                pred_ci.iloc[:, 1], color='k', alpha=.2)\nax.set_xlabel('Date')\nax.set_ylabel('Target')\nplt.legend()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:34.153758Z","iopub.execute_input":"2022-01-29T12:37:34.154051Z","iopub.status.idle":"2022-01-29T12:37:34.594456Z","shell.execute_reply.started":"2022-01-29T12:37:34.154010Z","shell.execute_reply":"2022-01-29T12:37:34.593562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_forecasted = pred.predicted_mean\ny_truth = bitcoin['2021-01-01':]\nmse = ((y_forecasted - y_truth) ** 2).mean()\nprint('The Mean Squared Error of our forecasts is {}'.format(round(mse, 2)))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:34.595729Z","iopub.execute_input":"2022-01-29T12:37:34.595951Z","iopub.status.idle":"2022-01-29T12:37:34.608046Z","shell.execute_reply.started":"2022-01-29T12:37:34.595923Z","shell.execute_reply":"2022-01-29T12:37:34.607132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('The Root Mean Squared Error of our forecasts is {}'.format(round(np.sqrt(mse), 2)))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:34.609522Z","iopub.execute_input":"2022-01-29T12:37:34.609869Z","iopub.status.idle":"2022-01-29T12:37:34.622420Z","shell.execute_reply.started":"2022-01-29T12:37:34.609825Z","shell.execute_reply":"2022-01-29T12:37:34.621516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does visualization of forecast looks like for Bitcoin?**\n\n---","metadata":{}},{"cell_type":"code","source":"# visualization of forecast\npred_uc = results.get_forecast(steps=100)\npred_ci = pred_uc.conf_int()\nax = bitcoin.plot(label='observed', figsize=(14, 7))\npred_uc.predicted_mean.plot(ax=ax, label='Forecast')\nax.fill_between(pred_ci.index,\n                pred_ci.iloc[:, 0],\n                pred_ci.iloc[:, 1], color='k', alpha=.25)\nax.set_xlabel('Date')\nax.set_ylabel('Target')\nplt.legend()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:34.626539Z","iopub.execute_input":"2022-01-29T12:37:34.627444Z","iopub.status.idle":"2022-01-29T12:37:35.015434Z","shell.execute_reply.started":"2022-01-29T12:37:34.627393Z","shell.execute_reply":"2022-01-29T12:37:35.014520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Lets explore ARIMA model for Ethereum**","metadata":{}},{"cell_type":"code","source":"# create series for Asset_Name Ethereum\nethereum = asset_lookup('Ethereum')\nethereum.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:37:35.016891Z","iopub.execute_input":"2022-01-29T12:37:35.017112Z","iopub.status.idle":"2022-01-29T12:37:39.242485Z","shell.execute_reply.started":"2022-01-29T12:37:35.017084Z","shell.execute_reply":"2022-01-29T12:37:39.241711Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check asset stationarity\nasset_stationarity (ethereum)","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:39:40.171926Z","iopub.execute_input":"2022-01-29T12:39:40.173326Z","iopub.status.idle":"2022-01-29T12:39:40.246775Z","shell.execute_reply.started":"2022-01-29T12:39:40.173236Z","shell.execute_reply":"2022-01-29T12:39:40.245662Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize trend, seasonality and residual\nasset_decompose(ethereum)","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:40:10.452241Z","iopub.execute_input":"2022-01-29T12:40:10.453423Z","iopub.status.idle":"2022-01-29T12:40:11.717692Z","shell.execute_reply.started":"2022-01-29T12:40:10.453358Z","shell.execute_reply":"2022-01-29T12:40:11.716640Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# search for best parameter combination for ARIMA model\nfor param in pdq:\n    for param_seasonal in seasonal_pdq:\n        try:\n            mod = sm.tsa.statespace.SARIMAX(ethereum,order=param,seasonal_order=param_seasonal,enforce_stationarity=False,enforce_invertibility=False)\n            results = mod.fit()\n            print('ARIMA{}x{}12 - AIC:{}'.format(param, param_seasonal, results.aic))\n        except:\n            continue","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:41:03.849953Z","iopub.execute_input":"2022-01-29T12:41:03.850306Z","iopub.status.idle":"2022-01-29T12:41:59.778368Z","shell.execute_reply.started":"2022-01-29T12:41:03.850270Z","shell.execute_reply":"2022-01-29T12:41:59.777309Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets fit ARIMA model\nmodel = sm.tsa.statespace.SARIMAX(ethereum,\n                                order=(1, 1, 1),\n                                seasonal_order=(1, 1, 1, 12),\n                                enforce_stationarity=False,\n                                enforce_invertibility=False)\nresults = model.fit()\nprint(results.summary().tables[1])","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:42:25.283992Z","iopub.execute_input":"2022-01-29T12:42:25.284283Z","iopub.status.idle":"2022-01-29T12:42:27.724405Z","shell.execute_reply.started":"2022-01-29T12:42:25.284253Z","shell.execute_reply":"2022-01-29T12:42:27.723842Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does model diagnostics visualization looks like for Ethereum?**\n\n---","metadata":{}},{"cell_type":"code","source":"# model diagnostics to check for errors\nresults.plot_diagnostics(figsize=(16, 8))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:42:53.916914Z","iopub.execute_input":"2022-01-29T12:42:53.917245Z","iopub.status.idle":"2022-01-29T12:42:54.704572Z","shell.execute_reply.started":"2022-01-29T12:42:53.917209Z","shell.execute_reply":"2022-01-29T12:42:54.703523Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does One-step ahead forecast looks like for Ethereum?**\n\n---","metadata":{}},{"cell_type":"code","source":"# validate forecast\npred = results.get_prediction(start=pd.to_datetime('2021-01-01'), dynamic=False)\npred_ci = pred.conf_int()\nax = bitcoin['2018':].plot(label='observed')\npred.predicted_mean.plot(ax=ax, label='One-step ahead Forecast', alpha=.7, figsize=(14, 7))\nax.fill_between(pred_ci.index,\n                pred_ci.iloc[:, 0],\n                pred_ci.iloc[:, 1], color='k', alpha=.2)\nax.set_xlabel('Date')\nax.set_ylabel('Target')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:43:35.454474Z","iopub.execute_input":"2022-01-29T12:43:35.454891Z","iopub.status.idle":"2022-01-29T12:43:35.855246Z","shell.execute_reply.started":"2022-01-29T12:43:35.454848Z","shell.execute_reply":"2022-01-29T12:43:35.853851Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does visualization of forecast looks like for Ethereum?**\n\n---","metadata":{}},{"cell_type":"code","source":"# visualization of forecast\npred_uc = results.get_forecast(steps=100)\npred_ci = pred_uc.conf_int()\nax = bitcoin.plot(label='observed', figsize=(14, 7))\npred_uc.predicted_mean.plot(ax=ax, label='Forecast')\nax.fill_between(pred_ci.index,\n                pred_ci.iloc[:, 0],\n                pred_ci.iloc[:, 1], color='k', alpha=.25)\nax.set_xlabel('Date')\nax.set_ylabel('Target')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-29T12:44:03.378199Z","iopub.execute_input":"2022-01-29T12:44:03.378588Z","iopub.status.idle":"2022-01-29T12:44:03.748843Z","shell.execute_reply.started":"2022-01-29T12:44:03.378536Z","shell.execute_reply":"2022-01-29T12:44:03.747918Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Thank you and Happy Learning.**\n\n---","metadata":{}},{"cell_type":"code","source":"thank_you_str=\"Thanks,Happy Learning,Collaboration,Thankyou,Keep Learning\"\n# create WordCloud with converted string\nwordcloud = WordCloud(width = 1000, height = 500, random_state=1, background_color='white', collocations=True).generate(thank_you_str)\nplt.figure(figsize=(20, 20))\nplt.imshow(wordcloud) \nplt.axis(\"off\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-29T12:37:39.243709Z","iopub.execute_input":"2022-01-29T12:37:39.243945Z","iopub.status.idle":"2022-01-29T12:37:39.832701Z","shell.execute_reply.started":"2022-01-29T12:37:39.243915Z","shell.execute_reply":"2022-01-29T12:37:39.831834Z"},"trusted":true},"execution_count":null,"outputs":[]}]}