{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction \n`V1.0.0`\n### Who am I\nJust a fellow Kaggle learner. I was creating this Notebook as practice and thought it could be useful to some others \n### Who is this for\nThis Notebook is for people that learn from examples. Forget the boring lectures and follow along for some fun/instructive time :)\n### What can I learn here\nYou learn all the basics needed to create a rudimentary RNN/LSTM Parallel Network. I go over a multitude of steps with explanations. Hopefully with these building blocks,you can go ahead and build much more complex models.\n\n### Things to remember\n+ Please Upvote/Like the Notebook so other people can learn from it\n+ Feel free to give any recommendations/changes. \n+ I will be continuously updating the notebook. Look forward to many more upcoming changes in the future.\n\n### You can also refer to these notebooks that have helped me as well:\n+ https://www.kaggle.com/yamqwe/g-research-lstm-starter-notebook#Training-%F0%9F%8F%8B%EF%B8%8F\n\n+ https://www.kaggle.com/vmuzhichenko/g-research-parallel-lstm-training\n","metadata":{}},{"cell_type":"markdown","source":"\n# Dataset Structure \n\n> **train.csv** - The training set\n> \n> 1.  timestamp - A timestamp for the minute covered by the row.\n> 2.  Asset_ID - An ID code for the cryptoasset.\n> 3.  Count - The number of trades that took place this minute.\n> 4.  Open - The USD price at the beginning of the minute.\n> 5.  High - The highest USD price during the minute.\n> 6.  Low - The lowest USD price during the minute.\n> 7.  Close - The USD price at the end of the minute.\n> 8.  Volume - The number of cryptoasset u units traded during the minute.\n> 9.  VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - There are 14 coins in the dataset\n>\n> - There are 4 years  in the [full] dataset","metadata":{}},{"cell_type":"markdown","source":"# Imports\nFirst let us start by importing the relevant libraries that we need.","metadata":{}},{"cell_type":"code","source":"# Ignore Warnings\nimport warnings\nfrom warnings import simplefilter\nwarnings.filterwarnings(\"ignore\")\n\n# Computational imports\nimport numpy as np   # Library for n-dimensional arrays\nimport pandas as pd  # Library for dataframes (structured data)\n\n# Helper imports\nimport os \nimport re\nimport time\nimport warnings\nfrom tqdm import tqdm\nimport datetime as dt\nfrom datetime import datetime\nimport scipy.stats as stats\nfrom pathlib import Path\n\n# ML/DL imports\nfrom sklearn.preprocessing import MinMaxScaler, OrdinalEncoder, LabelEncoder, RobustScaler\nfrom sklearn.metrics import mean_squared_error, mean_squared_log_error\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_probability as tfp\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import LSTM, Dense, Dropout, RepeatVector, TimeDistributed\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n\n# Plotting imports\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as dates\nimport seaborn as sns\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nfrom plotly.offline import init_notebook_mode, iplot\n\n%matplotlib inline\ninit_notebook_mode(connected=True)\n\n# Set seeds to make the experiment more reproducible.\nfrom numpy.random import seed\nseed(1)\n\n# Allows us to see more information regarding the DataFrame\npd.set_option(\"display.max_rows\", 500)\npd.set_option(\"display.max_columns\", 500)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-15T21:00:10.309462Z","iopub.execute_input":"2022-01-15T21:00:10.310140Z","iopub.status.idle":"2022-01-15T21:00:10.329485Z","shell.execute_reply.started":"2022-01-15T21:00:10.310087Z","shell.execute_reply":"2022-01-15T21:00:10.328057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plotting\nPreparing the data and plotting different kinds of plots.","metadata":{}},{"cell_type":"markdown","source":"## Reading and Preparing the Data\nLet's start by reading our data. We will store it in many dataframes.","metadata":{}},{"cell_type":"code","source":"data_path = \"../input/g-research-crypto-forecasting/\"\n\ncrypto_df = pd.read_csv(data_path + 'train.csv')\nasset_details_df = pd.read_csv(data_path + 'asset_details.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:00:10.331230Z","iopub.execute_input":"2022-01-15T21:00:10.332040Z","iopub.status.idle":"2022-01-15T21:01:03.794577Z","shell.execute_reply.started":"2022-01-15T21:00:10.332001Z","shell.execute_reply":"2022-01-15T21:01:03.793845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us see how many timesteps are in this dataframe.","metadata":{}},{"cell_type":"code","source":"len(crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\"))","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:03.795951Z","iopub.execute_input":"2022-01-15T21:01:03.796415Z","iopub.status.idle":"2022-01-15T21:01:04.184028Z","shell.execute_reply.started":"2022-01-15T21:01:03.796373Z","shell.execute_reply":"2022-01-15T21:01:04.183336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting: Candlestick plots\nBitcoin has Asset_ID of 1. We select only the recent 200 timesteps of data (-200: refers to LAST_TIMESTEP-200:LAST_TIMESTEP). We call it btc_mini since it is a miniscule portion of the whole dataframe","metadata":{}},{"cell_type":"code","source":"btc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\nbtc_mini = btc.iloc[-200:] # Select recent data rows","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:04.185910Z","iopub.execute_input":"2022-01-15T21:01:04.187262Z","iopub.status.idle":"2022-01-15T21:01:04.505062Z","shell.execute_reply.started":"2022-01-15T21:01:04.187220Z","shell.execute_reply":"2022-01-15T21:01:04.504342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use plotly library and its CandleStick graph. ","metadata":{}},{"cell_type":"code","source":"candle_stick_graph = go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])\nfig = go.Figure(data=[candle_stick_graph])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:04.507209Z","iopub.execute_input":"2022-01-15T21:01:04.507706Z","iopub.status.idle":"2022-01-15T21:01:04.565713Z","shell.execute_reply.started":"2022-01-15T21:01:04.507667Z","shell.execute_reply":"2022-01-15T21:01:04.564935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Taking care of NaN (missing values) in the dataframe\nThis allows us to check how many null/nan values each column has.\n","metadata":{}},{"cell_type":"code","source":"crypto_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:04.567175Z","iopub.execute_input":"2022-01-15T21:01:04.567420Z","iopub.status.idle":"2022-01-15T21:01:04.977627Z","shell.execute_reply.started":"2022-01-15T21:01:04.567386Z","shell.execute_reply":"2022-01-15T21:01:04.976974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check more precisely on one crypto: Ethereum.","metadata":{}},{"cell_type":"code","source":"eth = crypto_df[crypto_df[\"Asset_ID\"] == 6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\neth.info(show_counts =True)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:04.979015Z","iopub.execute_input":"2022-01-15T21:01:04.979277Z","iopub.status.idle":"2022-01-15T21:01:05.331171Z","shell.execute_reply.started":"2022-01-15T21:01:04.979242Z","shell.execute_reply":"2022-01-15T21:01:05.329636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:05.332564Z","iopub.execute_input":"2022-01-15T21:01:05.332812Z","iopub.status.idle":"2022-01-15T21:01:05.369812Z","shell.execute_reply.started":"2022-01-15T21:01:05.332775Z","shell.execute_reply":"2022-01-15T21:01:05.368976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is indeed many missing values here, we will take care of them later.","metadata":{}},{"cell_type":"code","source":"start_btc = btc.index[0].astype('datetime64[s]')\nend_btc = btc.index[-1].astype('datetime64[s]')\nstart_eth = eth.index[0].astype('datetime64[s]')\nend_eth = eth.index[-1].astype('datetime64[s]')\n\nprint('BTC data goes from', start_btc, 'to', end_btc)\nprint('Ethereum data goes from', start_eth, 'to', end_eth)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:05.371250Z","iopub.execute_input":"2022-01-15T21:01:05.371514Z","iopub.status.idle":"2022-01-15T21:01:05.378609Z","shell.execute_reply.started":"2022-01-15T21:01:05.371469Z","shell.execute_reply":"2022-01-15T21:01:05.377752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check out the intervals between all timesteps and see if they are all homogenous. If they are not, we will have to fix them.","metadata":{}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:05.382348Z","iopub.execute_input":"2022-01-15T21:01:05.382927Z","iopub.status.idle":"2022-01-15T21:01:05.407684Z","shell.execute_reply.started":"2022-01-15T21:01:05.382889Z","shell.execute_reply":"2022-01-15T21:01:05.406957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We notice that all timesteps aren't equally spaced. That is a problem we have to solve. We solve it by reindexing the index.","metadata":{}},{"cell_type":"code","source":"eth = eth.reindex(range(eth.index[0],eth.index[-1]+60,60),method='ffill')\neth.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:05.408835Z","iopub.execute_input":"2022-01-15T21:01:05.409076Z","iopub.status.idle":"2022-01-15T21:01:05.753648Z","shell.execute_reply.started":"2022-01-15T21:01:05.409041Z","shell.execute_reply":"2022-01-15T21:01:05.752969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We solved it by reindexing the dataframe and forward filling for all the missing values. ","metadata":{}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:05.754714Z","iopub.execute_input":"2022-01-15T21:01:05.755538Z","iopub.status.idle":"2022-01-15T21:01:05.798647Z","shell.execute_reply.started":"2022-01-15T21:01:05.755498Z","shell.execute_reply":"2022-01-15T21:01:05.797941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have equally spaced data now!","metadata":{}},{"cell_type":"markdown","source":"## Plotting: BTC and ETC vs time\nWe plot both BTC and ETC coin side by side to analyze and observe if they have any similarities. To do so, we use plotly subplots.","metadata":{}},{"cell_type":"code","source":"fig  = make_subplots(rows=1, cols=2, \n                    specs=[[{\"type\": \"scatter\"}, {\"type\": \"scatter\"}]],\n                    column_widths=[0.5, 0.5], vertical_spacing=0, horizontal_spacing=0.10,\n                    subplot_titles=(\"BTC vs Time\", \"ETH vs Time\"))\n\nfig.add_trace(go.Scatter(x=btc.index,y=btc.Close,name=\"BTC\"), row=1, col=1)\nfig.add_trace(go.Scatter(x=eth.index,y=eth.Close,name=\"ETH\"), row=1, col=2)\nfig.update_layout(autosize=True, margin=dict(b=0,r=20,l=20), template=\"plotly_dark\", title_font=dict(size=25, color='#8a8d93', family=\"Lato, sans-serif\"), \n                  width=900, height=500, title_text=\"Crypto vs Time\", font=dict(color='#8a8d93'),)\nfig.update_xaxes(title_text=\"Time (seconds)\")\nfig.update_yaxes(title_text=\"Closing Price ($)\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:05.799947Z","iopub.execute_input":"2022-01-15T21:01:05.800236Z","iopub.status.idle":"2022-01-15T21:01:15.054262Z","shell.execute_reply.started":"2022-01-15T21:01:05.800197Z","shell.execute_reply":"2022-01-15T21:01:15.053429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's make an inline functions that will take care of transforming string dates into posix timestamps.","metadata":{}},{"cell_type":"code","source":"# auxiliary function, from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(dt.datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n\n# create intervals\nbtc_mini_2021 = btc.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]\neth_mini_2021 = eth.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:15.055385Z","iopub.execute_input":"2022-01-15T21:01:15.055649Z","iopub.status.idle":"2022-01-15T21:01:15.075783Z","shell.execute_reply.started":"2022-01-15T21:01:15.055610Z","shell.execute_reply":"2022-01-15T21:01:15.074956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, lets build a similar graph, but this time only observing 2021 data.","metadata":{}},{"cell_type":"code","source":"fig  = make_subplots(rows=2, cols=1, \n                    specs=[[{\"type\": \"scatter\"}], [{\"type\": \"scatter\"}]],\n                    column_widths=[0.5], vertical_spacing=0.35, horizontal_spacing=0,\n                    subplot_titles=(\"BTC vs Time (2021)\", \"ETH vs Time (2021)\"))\n\nfig.add_trace(go.Scatter(x=btc_mini_2021.index,y=btc_mini_2021.Close,name=\"BTC\"), row=1, col=1)\nfig.add_trace(go.Scatter(x=eth_mini_2021.index,y=eth_mini_2021.Close,name=\"ETH\"), row=2, col=1)\nfig.update_layout(autosize=True, margin=dict(b=0,r=20,l=20), template=\"plotly_dark\", title_font=dict(size=25, color='#8a8d93', family=\"Lato, sans-serif\"), \n                  width=900, height=500, title_text=\"Crypto vs Time\", font=dict(color='#8a8d93'),)\nfig.update_xaxes(title_text=\"Time (seconds)\")\nfig.update_yaxes(title_text=\"Closing Price ($)\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:15.077277Z","iopub.execute_input":"2022-01-15T21:01:15.077680Z","iopub.status.idle":"2022-01-15T21:01:15.537050Z","shell.execute_reply.started":"2022-01-15T21:01:15.077640Z","shell.execute_reply":"2022-01-15T21:01:15.536341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting: Log Returns \nIn order to analyze price changes for an asset we can deal with the price difference. However, different assets exhibit different price scales, so that the their returns are not readily comparable. We can solve this problem by computing the percentage change in price instead, also known as the return. This return coincides with the percentage change in our invested capital.\n\nReturns are widely used in finance, however log returns are preferred for mathematical modelling of time series, as they are additive across time. Also, while regular returns cannot go below -100%, log returns are not bounded.\n\nTo compute the log return, we can simply take the logarithm of the ratio between two consecutive prices. The first row will have an empty return as the previous value is unknown, therefore the empty return data point will be dropped.","metadata":{}},{"cell_type":"code","source":"# define function to compute log returns\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:15.538258Z","iopub.execute_input":"2022-01-15T21:01:15.538631Z","iopub.status.idle":"2022-01-15T21:01:15.542644Z","shell.execute_reply.started":"2022-01-15T21:01:15.538595Z","shell.execute_reply":"2022-01-15T21:01:15.542041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lret_btc_mini = log_return(btc_mini_2021.Close)[1:]\nlret_eth_mini  = log_return(eth_mini_2021.Close)[1:]\nlret_btc_mini.rename('lret_btc_mini', inplace=True)\nlret_eth_mini.rename('lret_eth_mini', inplace=True)\n\nplt.figure(figsize=(8,4))\nplt.plot(lret_btc_mini);\nplt.plot(lret_eth_mini);\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:15.544393Z","iopub.execute_input":"2022-01-15T21:01:15.544912Z","iopub.status.idle":"2022-01-15T21:01:15.885463Z","shell.execute_reply.started":"2022-01-15T21:01:15.544867Z","shell.execute_reply":"2022-01-15T21:01:15.884867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting: Correlations","metadata":{}},{"cell_type":"code","source":"# join two asset in single DataFrame\nlret_btc_long = log_return(btc.Close)[1:]\nlret_eth_long = log_return(eth.Close)[1:]\nlret_btc_long.rename('lret_btc', inplace=True)\nlret_eth_long.rename('lret_eth', inplace=True)\ntwo_assets = pd.concat([lret_btc_long, lret_eth_long], axis=1)\n\n# group consecutive rows and use .corr() for correlation between columns\ncorr_time = two_assets.groupby(two_assets.index//(10000*60)).corr().loc[:,\"lret_btc\"].loc[:,\"lret_eth\"]\n\ncorr_time.plot();\nplt.xticks([])\nplt.ylabel(\"Correlation\")\nplt.title(\"Correlation between BTC and ETH over time\");","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:15.886950Z","iopub.execute_input":"2022-01-15T21:01:15.887386Z","iopub.status.idle":"2022-01-15T21:01:16.945813Z","shell.execute_reply.started":"2022-01-15T21:01:15.887348Z","shell.execute_reply":"2022-01-15T21:01:16.945030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We do see a strong correlation between BTC and ETH. This is what we expect. The price of ETH is known to be strongly swayed and dependent on the price of BTC.","metadata":{}},{"cell_type":"markdown","source":"## Plotting: Heatmaps\nWe are going to plot the heatmap here with the seaborn library. The heatmap will show us visualy the correlation between different assets. The hotter (red) it is, the higher the correlation.","metadata":{}},{"cell_type":"code","source":"# create dataframe with returns for all assets\nall_assets_2021 = pd.DataFrame([])\nfor asset_id, asset_name in zip(asset_details_df.Asset_ID, asset_details_df.Asset_Name):\n  asset = crypto_df[crypto_df[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n  asset = asset.loc[totimestamp('01/01/2021'):totimestamp('01/05/2021')]\n  asset = asset.reindex(range(asset.index[0],asset.index[-1]+60,60),method='pad')\n  lret = log_return(asset.Close.fillna(0))[1:]\n  all_assets_2021 = all_assets_2021.join(lret, rsuffix=asset_name, how=\"outer\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:16.947476Z","iopub.execute_input":"2022-01-15T21:01:16.948038Z","iopub.status.idle":"2022-01-15T21:01:22.925626Z","shell.execute_reply.started":"2022-01-15T21:01:16.947972Z","shell.execute_reply":"2022-01-15T21:01:22.924911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (10,7))\nsns.heatmap(all_assets_2021.corr(), annot=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:22.927025Z","iopub.execute_input":"2022-01-15T21:01:22.927488Z","iopub.status.idle":"2022-01-15T21:01:24.440527Z","shell.execute_reply.started":"2022-01-15T21:01:22.927439Z","shell.execute_reply":"2022-01-15T21:01:24.439879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting: Trend\nSimple trend plotting with rolling average. We used a 365 day window (in minutes, thus the *24*60)","metadata":{}},{"cell_type":"code","source":"moving_average = eth.Close.rolling(\n    window=365*24*60,       # 365-day window\n    center=True,            # puts the average at the center of the window\n    min_periods=182*24*60,  # choose about half the window size\n).mean()                    # compute the mean (could also do median, std, min, max, ...)\n\nax = eth.Close.plot(style=\".\", color=\"0.5\")\nmoving_average.plot(\n    ax=ax, linewidth=3, title=\"Tunnel Traffic - 365-Day Moving Average\", legend=False,\n);","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:24.441646Z","iopub.execute_input":"2022-01-15T21:01:24.442337Z","iopub.status.idle":"2022-01-15T21:01:29.119012Z","shell.execute_reply.started":"2022-01-15T21:01:24.442299Z","shell.execute_reply":"2022-01-15T21:01:29.118342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting: Seasonal Trend\nFirst we have a couple of functions that will allows us to easily plot the seasonal trend.","metadata":{}},{"cell_type":"code","source":"simplefilter(\"ignore\")\n\n# Set Matplotlib defaults\nplt.style.use(\"seaborn-whitegrid\")\nplt.rc(\"figure\", autolayout=True, figsize=(11, 5))\nplt.rc(\n    \"axes\",\n    labelweight=\"bold\",\n    labelsize=\"large\",\n    titleweight=\"bold\",\n    titlesize=16,\n    titlepad=10,\n)\nplot_params = dict(\n    color=\"0.75\",\n    style=\".-\",\n    markeredgecolor=\"0.25\",\n    markerfacecolor=\"0.25\",\n    legend=False,\n)\n%config InlineBackend.figure_format = 'retina'\n\n\n# annotations: https://stackoverflow.com/a/49238256/5769929\ndef seasonal_plot(X, y, period, freq, ax=None):\n    if ax is None:\n        _, ax = plt.subplots()\n    palette = sns.color_palette(\"husl\", n_colors=X[period].nunique(),)\n    ax = sns.lineplot(\n        x=freq,\n        y=y,\n        hue=period,\n        data=X,\n        ci=False,\n        ax=ax,\n        palette=palette,\n        legend=False,\n    )\n    ax.set_title(f\"Seasonal Plot ({period}/{freq})\")\n    for line, name in zip(ax.lines, X[period].unique()):\n        y_ = line.get_ydata()[-1]\n        ax.annotate(\n            name,\n            xy=(1, y_),\n            xytext=(6, 0),\n            color=line.get_color(),\n            xycoords=ax.get_yaxis_transform(),\n            textcoords=\"offset points\",\n            size=14,\n            va=\"center\",\n        )\n    return ax","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:29.120554Z","iopub.execute_input":"2022-01-15T21:01:29.121050Z","iopub.status.idle":"2022-01-15T21:01:29.155840Z","shell.execute_reply.started":"2022-01-15T21:01:29.121013Z","shell.execute_reply":"2022-01-15T21:01:29.155168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before anything, we have to transform the index into a datetime format to be able to plot the seasonal plot.","metadata":{}},{"cell_type":"code","source":"eth[\"temp_date\"] = eth.index\neth[\"temp_date\"]","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:29.157349Z","iopub.execute_input":"2022-01-15T21:01:29.157835Z","iopub.status.idle":"2022-01-15T21:01:29.167027Z","shell.execute_reply.started":"2022-01-15T21:01:29.157796Z","shell.execute_reply":"2022-01-15T21:01:29.166241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for idx in eth.index[:1000]:\n    new_date = datetime.utcfromtimestamp(idx).strftime('%Y-%m-%d %H:%M:%S')\n    eth[\"temp_date\"].loc[idx] = new_date","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:29.168310Z","iopub.execute_input":"2022-01-15T21:01:29.168942Z","iopub.status.idle":"2022-01-15T21:01:37.083116Z","shell.execute_reply.started":"2022-01-15T21:01:29.168904Z","shell.execute_reply":"2022-01-15T21:01:37.082361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth.set_index(eth['temp_date'], inplace = True)  \neth.drop(labels = 'temp_date', axis = 1)\neth_seas_plot = eth.iloc[:1000, :]\neth_seas_plot.index = pd.to_datetime(eth_seas_plot.index)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:37.084340Z","iopub.execute_input":"2022-01-15T21:01:37.084611Z","iopub.status.idle":"2022-01-15T21:01:37.141254Z","shell.execute_reply.started":"2022-01-15T21:01:37.084573Z","shell.execute_reply":"2022-01-15T21:01:37.140410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have set a new datetime index, we can proceed with the plotting.","metadata":{}},{"cell_type":"code","source":"X = pd.DataFrame(eth_seas_plot.Close.copy())\n\n# days within a week\nX[\"day\"] = eth_seas_plot.index.dayofweek  # the x-axis (freq)\nX[\"week\"] = eth_seas_plot.index.week  # the seasonal period (period)\nX[\"minute\"] = eth_seas_plot.index.minute\nX[\"second\"] = eth_seas_plot.index.second\n# days within a year\nX[\"dayofyear\"] = eth_seas_plot.index.dayofyear\nX[\"year\"] = eth_seas_plot.index.year\nfig, ax0 = plt.subplots(1, 1, figsize=(11, 6))\nseasonal_plot(X, y=\"Close\", period=\"week\", freq=\"minute\", ax=ax0)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:37.142806Z","iopub.execute_input":"2022-01-15T21:01:37.143017Z","iopub.status.idle":"2022-01-15T21:01:39.030323Z","shell.execute_reply.started":"2022-01-15T21:01:37.142984Z","shell.execute_reply":"2022-01-15T21:01:39.029644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training and Predicting\nThis section contains all necessary steps to train and predict your model.","metadata":{}},{"cell_type":"markdown","source":"## Helper Functions","metadata":{}},{"cell_type":"markdown","source":"### Split Sequences\nA key component of time-series problem is splitting our input data into sequences that we can feed to our LSTM network. This sequences depend on the required timesteps and horizons. ","metadata":{}},{"cell_type":"code","source":"def split_sequences(sequences, timesteps, horizon):\n    Sequences, Targets = list(), list()\n    for i in range(len(sequences)):\n        # find the end of this pattern\n        end_ix = i + timesteps\n        out_end_ix = end_ix + horizon-1\n        # check if we are beyond the dataset\n        if out_end_ix > len(sequences):\n            break\n        # gather input and output parts of the pattern\n        seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]\n        Sequences.append(seq_x)\n        Targets.append(seq_y)\n        show_shapes()\n    return array(X), array(y)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.031553Z","iopub.execute_input":"2022-01-15T21:01:39.032348Z","iopub.status.idle":"2022-01-15T21:01:39.039642Z","shell.execute_reply.started":"2022-01-15T21:01:39.032308Z","shell.execute_reply":"2022-01-15T21:01:39.039001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Downcasting\nThis functions is used to downcast our variables to types that take less memory. This helps with model performance and speed.","metadata":{}},{"cell_type":"code","source":"def downcast_dtypes(df):\n    float_cols = [c for c in df if df[c].dtype == \"float64\"]\n    int_cols = [c for c in df if df[c].dtype in [\"int64\", \"int32\"]]\n    df[float_cols] = df[float_cols].astype(np.float32)\n    df[int_cols] = df[int_cols].astype(np.int16)\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.044308Z","iopub.execute_input":"2022-01-15T21:01:39.044720Z","iopub.status.idle":"2022-01-15T21:01:39.050862Z","shell.execute_reply.started":"2022-01-15T21:01:39.044682Z","shell.execute_reply":"2022-01-15T21:01:39.050063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Normalization \nThese functions are used to normalize our data. This aids with model performance and speed. You can also use the scikit-learn MinMaxScaler if you wish, it is up to you.","metadata":{}},{"cell_type":"code","source":"def Normalize(list):\n    list = np.array(list)\n    low, high = np.percentile(list, [0, 100])\n    delta = high - low\n    if delta != 0:\n        for i in range(0, len(list)):\n            list[i] = (list[i]-low)/delta\n    return  list,low,high\n\ndef FNoramlize(list,low,high):\n    delta = high - low\n    if delta != 0:\n        for i in range(0, len(list)):\n            list[i] = list[i]*delta + low\n    return list\n\ndef Normalize2(list,low,high):\n    list = np.array(list)\n    delta = high - low\n    if delta != 0:\n        for i in range(0, len(list)):\n            list[i] = (list[i]-low)/delta\n    return  list","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.051951Z","iopub.execute_input":"2022-01-15T21:01:39.052411Z","iopub.status.idle":"2022-01-15T21:01:39.061818Z","shell.execute_reply.started":"2022-01-15T21:01:39.052373Z","shell.execute_reply":"2022-01-15T21:01:39.061080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Show Shapes\nThis functions is used to quickly check the shapes of our numpy arrays. This is especially important to assure we have the right shape for our LSTM network.","metadata":{}},{"cell_type":"code","source":"def show_shapes(Sequences, Targets): # this'll use inputs; can make yours to use local variable values\n    print(\"Expected: (num_samples, timesteps, channels)\")\n    print(\"Sequences: {}\".format(Sequences.shape))\n    print(\"Targets:   {}\".format(Targets.shape))   ","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.062931Z","iopub.execute_input":"2022-01-15T21:01:39.063524Z","iopub.status.idle":"2022-01-15T21:01:39.071087Z","shell.execute_reply.started":"2022-01-15T21:01:39.063487Z","shell.execute_reply":"2022-01-15T21:01:39.070291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Exploraty Data Analysis for pandas\nThis functions is used to quickly check the basic attributes of our pandas DataFrame.","metadata":{}},{"cell_type":"code","source":"def basic_eda(df):\n    print(\"-------------------------------TOP 5 RECORDS-----------------------------\")\n    print(df.head(5))\n    print()\n    \n    print(\"-------------------------------INFO--------------------------------------\")\n    print(df.info())\n    print()\n    \n    print(\"-------------------------------Describe----------------------------------\")\n    print(df.describe())\n    print()\n    \n    print(\"-------------------------------Columns-----------------------------------\")\n    print(df.columns)\n    print()\n    \n    print(\"-------------------------------Data Types--------------------------------\")\n    print(df.dtypes)\n    print()\n    \n    print(\"----------------------------Missing Values-------------------------------\")\n    print(df.isnull().sum())\n    print()\n    \n    print(\"----------------------------NULL values----------------------------------\")\n    print(df.isna().sum())\n    print()\n    \n    print(\"--------------------------Shape Of Data---------------------------------\")\n    print(df.shape)\n    print()\n    \n    print(\"============================================================================ \\n\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.072267Z","iopub.execute_input":"2022-01-15T21:01:39.072819Z","iopub.status.idle":"2022-01-15T21:01:39.083233Z","shell.execute_reply.started":"2022-01-15T21:01:39.072783Z","shell.execute_reply":"2022-01-15T21:01:39.082582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This feature is specific to this competition. It automatically adds usefull features to the DataFrame.","metadata":{}},{"cell_type":"code","source":"def add_features(df):\n    df['Upper_Shadow'] = df['High'] - np.maximum(df['Close'], df['Open'])\n    df['Lower_Shadow'] = np.minimum(df['Close'], df['Open']) - df['Low']\n    \n    df['spread'] = df['High'] - df['Low']\n    df['mean_trade'] = df['Volume']/df['Count']\n    df['log_price_change'] = np.log(df['Close']/df['Open'])\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.085968Z","iopub.execute_input":"2022-01-15T21:01:39.086378Z","iopub.status.idle":"2022-01-15T21:01:39.093936Z","shell.execute_reply.started":"2022-01-15T21:01:39.086349Z","shell.execute_reply":"2022-01-15T21:01:39.093159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading and Preparing the Data\nLet's start by reading our data. We will store it in many dataframes.","metadata":{}},{"cell_type":"code","source":"data_path = \"../input/g-research-crypto-forecasting/\"\n\ntrain = pd.read_csv(data_path + 'train.csv').set_index(\"timestamp\")\nassets = pd.read_csv(data_path + 'asset_details.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:01:39.096510Z","iopub.execute_input":"2022-01-15T21:01:39.096933Z","iopub.status.idle":"2022-01-15T21:02:12.156502Z","shell.execute_reply.started":"2022-01-15T21:01:39.096904Z","shell.execute_reply":"2022-01-15T21:02:12.155768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's create a dict name assets_order which tracks the true asset order in the Dataframe (for example, in the DataFrame, asset order #3 (Cardano) starts and is followed by #2). This is important so that we can create a new feature in the training dataframe named asset order which goes from 0 -> 13 for one timestep and then resets to 0 for the next timestep.","metadata":{}},{"cell_type":"code","source":"assets_order = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv').Asset_ID[:14]\nassets_order = dict((t,i) for i,t in enumerate(assets_order))\nassets_order","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:12.157891Z","iopub.execute_input":"2022-01-15T21:02:12.158170Z","iopub.status.idle":"2022-01-15T21:02:17.405456Z","shell.execute_reply.started":"2022-01-15T21:02:12.158133Z","shell.execute_reply":"2022-01-15T21:02:17.404767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'm going to truncate a the training data just for the notebook's sake. Or else, the training would take an eternity,","metadata":{}},{"cell_type":"code","source":"train_cut = train[15000000:]\n\ntrain_cut[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP','Target']] = \\\ntrain_cut[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP','Target']].astype(np.float32)\n\ntrain_cut.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:17.409643Z","iopub.execute_input":"2022-01-15T21:02:17.410332Z","iopub.status.idle":"2022-01-15T21:02:18.630755Z","shell.execute_reply.started":"2022-01-15T21:02:17.410286Z","shell.execute_reply":"2022-01-15T21:02:18.629947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Taking care of NaN (missing values) in the dataframe\nThere is multiple ways of doing this, I have chosen do it with the following method:","metadata":{}},{"cell_type":"code","source":"print(np.sum(train_cut.isna()))\ntrain_cut['Target'] = train_cut['Target'].fillna(method = 'ffill')\nprint('\\n', np.sum(train_cut.isna()))","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:18.631892Z","iopub.execute_input":"2022-01-15T21:02:18.632229Z","iopub.status.idle":"2022-01-15T21:02:18.942534Z","shell.execute_reply.started":"2022-01-15T21:02:18.632188Z","shell.execute_reply":"2022-01-15T21:02:18.941787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The VMAP feature has some nan values. Let's take care of them by replacing by our defined VMAP_max, VMAP_min.","metadata":{}},{"cell_type":"code","source":"# VWAP column has -inf and inf values. VWAP_max and VWAP_min will be used for replacement\n    \nVWAP_max = np.max(train_cut[np.isfinite(train_cut.VWAP)].VWAP)\nVWAP_min = np.min(train_cut[np.isfinite(train_cut.VWAP)].VWAP)\nprint(VWAP_max, \"\\n\", VWAP_min)\n\ntrain_cut['VWAP'] = np.nan_to_num(train_cut.VWAP, posinf=VWAP_max, neginf=VWAP_min)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:18.943677Z","iopub.execute_input":"2022-01-15T21:02:18.944422Z","iopub.status.idle":"2022-01-15T21:02:19.891277Z","shell.execute_reply.started":"2022-01-15T21:02:18.944381Z","shell.execute_reply":"2022-01-15T21:02:19.890511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's make a new series named ids, which stores what timestep the current row is part of. For example, for timestep xxxxxx (first timestep) of Cardano (#3), the ids for this row would be 1_3 (representing first timestep and asset id 3 which is Cardano).","metadata":{}},{"cell_type":"code","source":"df = train_cut[['Asset_ID', 'Target']].copy()\ntimes = dict((t,i) for i,t in enumerate(df.index.unique()))\ndf['id'] = df.index.map(times)\ndf['id'] = df['id'].astype(str) + '_' + df['Asset_ID'].astype(str)\nids = df.id.copy()\n\ndel df","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:19.892394Z","iopub.execute_input":"2022-01-15T21:02:19.893065Z","iopub.status.idle":"2022-01-15T21:02:45.493489Z","shell.execute_reply.started":"2022-01-15T21:02:19.893023Z","shell.execute_reply":"2022-01-15T21:02:45.492708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's add some extra features that will help with the prediction. We will be using the add_features function we defined previously.","metadata":{}},{"cell_type":"code","source":"train_cut = add_features(train_cut)\ntrain_cut.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:45.494667Z","iopub.execute_input":"2022-01-15T21:02:45.494920Z","iopub.status.idle":"2022-01-15T21:02:45.702383Z","shell.execute_reply.started":"2022-01-15T21:02:45.494885Z","shell.execute_reply":"2022-01-15T21:02:45.701675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Stadardizing the training data\nHere we stadardize our training data with RobustScaler. As the anme states, robust scaler is robust to outlier data. This is good for random data points that dont' really follow the data distribution. This will increase models accuracy and speed.","metadata":{}},{"cell_type":"code","source":"scale_features = train_cut.columns.drop(['Asset_ID','Target'])\nRS = RobustScaler()\ntrain_cut[scale_features] = RS.fit_transform(train_cut[scale_features])","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:45.703505Z","iopub.execute_input":"2022-01-15T21:02:45.703919Z","iopub.status.idle":"2022-01-15T21:02:49.450944Z","shell.execute_reply.started":"2022-01-15T21:02:45.703880Z","shell.execute_reply":"2022-01-15T21:02:49.450177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fixing Timesteps\nWe have to re-index the timestep (like we did for plotting). I chose to do it in a different fashion here just to practice different methods. We will be using the reindex() function to do so.","metadata":{}},{"cell_type":"code","source":"ind = train_cut.index.unique()\ndef reindex(df):\n    df = df.reindex(range(ind[0],ind[-1]+60,60),method='nearest')\n    df = df.fillna(method=\"ffill\").fillna(method=\"bfill\")\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:49.452188Z","iopub.execute_input":"2022-01-15T21:02:49.452430Z","iopub.status.idle":"2022-01-15T21:02:49.534915Z","shell.execute_reply.started":"2022-01-15T21:02:49.452398Z","shell.execute_reply":"2022-01-15T21:02:49.534149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_cut=train_cut.groupby('Asset_ID').apply(reindex).reset_index(0, drop=True).sort_index()\ntrain_cut.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:49.536253Z","iopub.execute_input":"2022-01-15T21:02:49.536604Z","iopub.status.idle":"2022-01-15T21:02:54.823879Z","shell.execute_reply.started":"2022-01-15T21:02:49.536565Z","shell.execute_reply":"2022-01-15T21:02:54.823079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Removing fake records\nHere we find fake records. Fake records are timesteps where we do not have a record for a specific crypto. Fake records have all column values set to 0.","metadata":{}},{"cell_type":"code","source":"# Matching records and marking generated rows as 'fake'\n\ntrain_cut['group_num'] = train_cut.index.map(times)\ntrain_cut = train_cut.dropna(subset=['group_num'])\ntrain_cut['group_num'] = train_cut['group_num'].astype('int')\n\ntrain_cut['id'] = train_cut['group_num'].astype(str) + '_' + train_cut['Asset_ID'].astype(str)\n\ntrain_cut['is_real'] = train_cut.id.isin(ids)*1\ntrain_cut = train_cut.drop('id', axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:02:54.825309Z","iopub.execute_input":"2022-01-15T21:02:54.825587Z","iopub.status.idle":"2022-01-15T21:03:26.989517Z","shell.execute_reply.started":"2022-01-15T21:02:54.825551Z","shell.execute_reply":"2022-01-15T21:03:26.988735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Features values for 'non-real' rows are set to zeros\n\nfeatures = train_cut.columns.drop(['Asset_ID','group_num','is_real'])\ntrain_cut.loc[train_cut.is_real == 0, features] = 0.","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:26.990798Z","iopub.execute_input":"2022-01-15T21:03:26.991039Z","iopub.status.idle":"2022-01-15T21:03:27.483521Z","shell.execute_reply.started":"2022-01-15T21:03:26.990999Z","shell.execute_reply":"2022-01-15T21:03:27.482764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we order the dataframe using the asset_order column we will create. This is to ensure that all new timesteps start with the same token (in our case, Asset_ID of 3 which is Cardano)","metadata":{}},{"cell_type":"code","source":"# Sorting assets according to their order in the 'supplemental_train.csv'\n\ntrain_cut['asset_order'] = train_cut.Asset_ID.map(assets_order) \ntrain_cut=train_cut.sort_values(by=['group_num', 'asset_order'])\ntrain_cut.head(20)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:27.484905Z","iopub.execute_input":"2022-01-15T21:03:27.485181Z","iopub.status.idle":"2022-01-15T21:03:30.537399Z","shell.execute_reply.started":"2022-01-15T21:03:27.485145Z","shell.execute_reply":"2022-01-15T21:03:30.536000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_cut['asset_order'] = train_cut['asset_order'].astype('float64')","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:30.538752Z","iopub.execute_input":"2022-01-15T21:03:30.539039Z","iopub.status.idle":"2022-01-15T21:03:30.596957Z","shell.execute_reply.started":"2022-01-15T21:03:30.539000Z","shell.execute_reply":"2022-01-15T21:03:30.596232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_targets = train_cut['Target'].to_numpy().reshape(-1, 14)\n\nfeatures = train_cut.columns.drop(['Asset_ID', 'Target', 'group_num','is_real'])\ntrain_cut = train_cut[features]\n\ntrain_cut=np.array(train_cut)\ntrain_cut = train_cut.reshape(-1,14,train_cut.shape[-1])\ntrain_cut.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:30.598376Z","iopub.execute_input":"2022-01-15T21:03:30.598643Z","iopub.status.idle":"2022-01-15T21:03:32.221023Z","shell.execute_reply.started":"2022-01-15T21:03:30.598607Z","shell.execute_reply":"2022-01-15T21:03:32.220381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_cut = train_cut.astype('float64')\ntrain_cut.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:32.222221Z","iopub.execute_input":"2022-01-15T21:03:32.222450Z","iopub.status.idle":"2022-01-15T21:03:32.626911Z","shell.execute_reply.started":"2022-01-15T21:03:32.222416Z","shell.execute_reply":"2022-01-15T21:03:32.626055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Splitting the training data into sequences\nIn this section, we split the training data into sequences that we can further feed into LSTM network. Notice that each sequence has many variables/features making it a multivariate problem. To predict the next 1 days (our horizon), we are going to use the events that occureed 15 days ago.","metadata":{}},{"cell_type":"code","source":"# timeseriesgenerator-like class, except it using target from the last timestep insteed of last+1\nclass sample_generator(keras.utils.Sequence):\n    def __init__(self, x_set, y_set, batch_size, length):\n        self.x, self.y = x_set, y_set\n        self.batch_size = batch_size\n        self.length = length\n        self.size = len(x_set)\n\n    def __len__(self):\n        return int(np.ceil(len(self.x) / float(self.batch_size)))\n\n    def __getitem__(self, idx):\n        batch_x=[]\n        batch_y=[]\n        for i in range(self.batch_size):\n            start_ind = self.batch_size*idx + i\n            end_ind = start_ind + self.length \n            if end_ind <= self.size:\n                batch_x.append(self.x[start_ind : end_ind])\n                batch_y.append(self.y[end_ind -1])\n\n        return np.asarray(batch_x).astype(\"float32\"), np.asarray(batch_y).astype(\"float32\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:32.628056Z","iopub.execute_input":"2022-01-15T21:03:32.628330Z","iopub.status.idle":"2022-01-15T21:03:32.637612Z","shell.execute_reply.started":"2022-01-15T21:03:32.628293Z","shell.execute_reply":"2022-01-15T21:03:32.636957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Split into sequence and target\nAfter sequencing and normalzing the data, we slice the data to create the input sequences and output targets.","metadata":{}},{"cell_type":"code","source":"#last 10% of the data are used as validation set\nX_train, X_test = train_cut[:-len(train_cut)//10], train_cut[-len(train_cut)//10:]\ny_train, y_test = train_targets[:-len(train_cut)//10], train_targets[-len(train_cut)//10:]","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:32.639072Z","iopub.execute_input":"2022-01-15T21:03:32.639546Z","iopub.status.idle":"2022-01-15T21:03:32.650854Z","shell.execute_reply.started":"2022-01-15T21:03:32.639506Z","shell.execute_reply":"2022-01-15T21:03:32.650074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE=2**10\ntrain_generator = sample_generator(X_train, y_train, length=15, batch_size=BATCH_SIZE)\nval_generator = sample_generator(X_test, y_test, length=15, batch_size=BATCH_SIZE)\n\nprint(f'Sample shape: {train_generator[0][0].shape}')\nprint(f'Target shape: {train_generator[0][1].shape}')","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:32.652060Z","iopub.execute_input":"2022-01-15T21:03:32.652424Z","iopub.status.idle":"2022-01-15T21:03:32.682456Z","shell.execute_reply.started":"2022-01-15T21:03:32.652366Z","shell.execute_reply":"2022-01-15T21:03:32.681784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating the LSTM Network\nWe are going to be creating a multivariate Parallel LSTM Network with a dense layer at the end. We are using dropout as a regularisation method to combat overfitting. Notice the lambda layer which allows to add our own functional layer. In this case, the lambda layer is used to slice the array to the respective crypto we want. We then loop with the for loop and create the same layer structure for each crypto. The hidden layer output is then concatenanted for each crypto and then the prediction it obtained through the dense layer. \n\n* Build the model. Simplified  structure:\n    - Lambda layer needed for assets separation\n    - Masking layer. Generated records (filled gaps) has zeros as features values, so they are not used in the computations\n    - LSTM or GRU layer\n    - Dropout as a regularisation method to combat overfitting\n    - Concatanate layer\n    - Dense Layer for Prediction (linear activation which is the default)","metadata":{}},{"cell_type":"code","source":"#https://github.com/tensorflow/tensorflow/issues/37495\ndef MaxCorrelation(y_true,y_pred):\n    \"\"\"Goal is to maximize correlation between y_pred, y_true. Same as minimizing the negative.\"\"\"\n    mask = tf.math.not_equal(y_true, 0.)\n    y_true_masked = tf.boolean_mask(y_true, mask)\n    y_pred_masked = tf.boolean_mask(y_pred, mask)\n    return -tf.math.abs(tfp.stats.correlation(y_true_masked,y_pred_masked, sample_axis=None, event_axis=None))\n\ndef Correlation(y_true,y_pred):\n    return tf.math.abs(tfp.stats.correlation(y_pred,y_true, sample_axis=None, event_axis=None))\n\ndef masked_mse(y_true, y_pred):\n    mask = tf.math.not_equal(y_true, 0.)\n    y_true_masked = tf.boolean_mask(y_true, mask)\n    y_pred_masked = tf.boolean_mask(y_pred, mask)\n    return tf.keras.losses.mean_squared_error(y_true = y_true_masked, y_pred = y_pred_masked)\n\ndef masked_mae(y_true, y_pred):\n    mask = tf.math.not_equal(y_true, 0.)\n    y_true_masked = tf.boolean_mask(y_true, mask)\n    y_pred_masked = tf.boolean_mask(y_pred, mask)\n    return tf.keras.losses.mean_absolute_error(y_true = y_true_masked, y_pred = y_pred_masked)\n\ndef masked_cosine(y_true, y_pred):\n    mask = tf.math.not_equal(y_true, 0.)\n    y_true_masked = tf.boolean_mask(y_true, mask)\n    y_pred_masked = tf.boolean_mask(y_pred, mask)\n    return tf.keras.losses.cosine_similarity(y_true_masked, y_pred_masked)\n\ndef get_model(n_assets=14):  \n    x_input = keras.Input(shape=(train_generator[0][0].shape[1], n_assets, train_generator[0][0].shape[-1]))\n\n    branch_outputs = []\n        \n    for i in range(n_assets):\n            # Slicing the ith asset:\n        a = layers.Lambda(lambda x: x[:,:, i])(x_input)\n        a = layers.Masking(mask_value=0.,)(a)\n        a = layers.LSTM(units=32, return_sequences=True)(a)\n        a = layers.Dropout(0.2)(a)\n        a = layers.LSTM(units=16)(a)\n        a = layers.Dropout(0.2)(a)\n        branch_outputs.append(a)\n    \n    x = layers.Concatenate()(branch_outputs)\n    x = layers.Dense(units=128)(x)\n    out = layers.Dense(units=n_assets)(x)\n    \n    model = keras.Model(inputs=x_input, outputs=out)\n    model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3), \n                  loss = masked_cosine,\n                  metrics=[Correlation]\n                 )\n    \n    return model \n    \nmodel=get_model()\nmodel.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-15T21:03:32.683704Z","iopub.execute_input":"2022-01-15T21:03:32.683929Z","iopub.status.idle":"2022-01-15T21:03:55.539888Z","shell.execute_reply.started":"2022-01-15T21:03:32.683896Z","shell.execute_reply":"2022-01-15T21:03:55.539134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us now use plot model method to validate our network visually.","metadata":{}},{"cell_type":"code","source":"#example with 3 assets for visibility\ntf.keras.utils.plot_model(get_model(n_assets=3), show_shapes=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:03:55.541278Z","iopub.execute_input":"2022-01-15T21:03:55.541519Z","iopub.status.idle":"2022-01-15T21:04:00.567994Z","shell.execute_reply.started":"2022-01-15T21:03:55.541485Z","shell.execute_reply":"2022-01-15T21:04:00.567193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training/Fitting time\nWe can finally train our model with our training data. Let's see how it does.","metadata":{}},{"cell_type":"code","source":"tf.random.set_seed(10)\n\nestop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, verbose=0, mode='min',restore_best_weights=True)\nscheduler = keras.optimizers.schedules.ExponentialDecay(1e-3, (0.5*len(X_train)/BATCH_SIZE), 1e-3)\nlr = keras.callbacks.LearningRateScheduler(scheduler, verbose = 1)\n    \nhistory = model.fit(train_generator, validation_data = (val_generator), epochs = 10, callbacks = [lr])","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-15T21:04:00.569604Z","iopub.execute_input":"2022-01-15T21:04:00.571299Z","iopub.status.idle":"2022-01-15T21:27:35.084018Z","shell.execute_reply.started":"2022-01-15T21:04:00.571251Z","shell.execute_reply":"2022-01-15T21:27:35.083191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting model accuracy and loss\nThis step is very important since it allows you to see if your model is performing well as you train it. If it isn't, you will rather have to create new features, tune hyperparameters, modify the RNN network or cry.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize=(16,8))\n\nhistories = pd.DataFrame(history.history)\n\nepochs = list(range(1,len(histories)+1))\nloss = histories['loss']\nval_loss = histories['val_loss']\nCorrelation = histories['Correlation']\nval_Correlation = histories['val_Correlation']\n\nax[0].plot(epochs, loss, label = 'Train Loss')\nax[0].plot(epochs, val_loss, label = 'Val Loss')\nax[0].set_title('Losses')\nax[0].set_xlabel('Epoch')\nax[0].legend(loc='upper right')\n\nax[1].plot(epochs, Correlation, label = 'Train Correlation')\nax[1].plot(epochs, val_Correlation, label = 'Val Correlation')\nax[1].set_title('Correlations')\nax[1].set_xlabel('Epoch')\nax[1].legend(loc='upper right')\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:27:50.353203Z","iopub.execute_input":"2022-01-15T21:27:50.353460Z","iopub.status.idle":"2022-01-15T21:27:51.180710Z","shell.execute_reply.started":"2022-01-15T21:27:50.353431Z","shell.execute_reply":"2022-01-15T21:27:51.180028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style('darkgrid') # darkgrid, white grid, dark, white and ticks\ncolors = sns.color_palette('pastel') # Color palette to use\nplt.rc('axes', titlesize=18)     # fontsize of the axes title\nplt.rc('axes', labelsize=14)    # fontsize of the x and y labels\nplt.rc('xtick', labelsize=13)    # fontsize of the tick labels\nplt.rc('ytick', labelsize=13)    # fontsize of the tick labels\nplt.rc('legend', fontsize=13)    # legend fontsize\nplt.rc('font', size=13)          # controls default text sizes\n\n# # Set Matplotlib defaults\n# plt.style.use(\"seaborn-whitegrid\")\n# plt.rc(\"figure\", autolayout=True, figsize=(11, 5))\n# plt.rc(\n#     \"axes\",\n#     labelweight=\"bold\",\n#     labelsize=\"large\",\n#     titleweight=\"bold\",\n#     titlesize=16,\n#     titlepad=10,\n# )\n# plot_params = dict(\n#     color=\"0.75\",\n#     style=\".-\",\n#     markeredgecolor=\"0.25\",\n#     markerfacecolor=\"0.25\",\n#     legend=False,\n# )\n# %config InlineBackend.figure_format = 'retina'\n\n\ndef plot_loss(history):\n    fig, ax = plt.subplots(figsize=(10,6), tight_layout=True)\n    ax.plot(history.history['loss'], 'o-', color=\"#004C99\", linewidth=2)\n    ax.plot(history.history['val_loss'], 'o-', color=\"#D96552\",linewidth=2)\n    ax.set_facecolor(colors[-1])\n    plt.grid(b=True,axis = 'y')\n    ax.grid(b=True,axis = 'y')\n    plt.ylabel('Loss')\n    plt.xlabel('epoch')\n    plt.legend(['Train loss', 'Validation loss'], loc='upper right',prop={'size': 15})\n    plt.show()\n    \ndef plot_future(prediction, y_test):\n    fig, ax = plt.subplots(figsize=(10,6), tight_layout=True)\n    range_future = len(prediction)\n    ax.plot(np.arange(range_future), np.array(y_test),label='Actual',color=\"#004C99\")\n    ax.plot(np.arange(range_future),np.array(prediction),label='Prediction',color=\"#D96552\")\n    ax.set_facecolor(colors[-1])\n    plt.grid(b=True,axis = 'y')\n    ax.grid(b=True,axis = 'y')\n    plt.ylabel('USD')\n    plt.legend(loc='upper left',prop={'size': 15})\n    plt.tick_params(axis='x', which='both', bottom=False, top=False, labelbottom=False)   \n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:27:53.516819Z","iopub.execute_input":"2022-01-15T21:27:53.517245Z","iopub.status.idle":"2022-01-15T21:27:53.530874Z","shell.execute_reply.started":"2022-01-15T21:27:53.517206Z","shell.execute_reply":"2022-01-15T21:27:53.529929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_loss(history)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T21:27:54.811278Z","iopub.execute_input":"2022-01-15T21:27:54.811552Z","iopub.status.idle":"2022-01-15T21:27:55.166919Z","shell.execute_reply.started":"2022-01-15T21:27:54.811520Z","shell.execute_reply":"2022-01-15T21:27:55.166287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}