","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.036915,"end_time":"2021-10-29T13:38:47.184754","exception":false,"start_time":"2021-10-29T13:38:47.147839","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.046159,"end_time":"2021-10-29T13:38:47.267941","exception":false,"start_time":"2021-10-29T13:38:47.221782","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":">### Technical Analysis Indicators #1\n>- Here are some simple indexes to analyze the charts. some can even be used as features to a model.\n>- Ta-lib is very good and very helpful library for calculating various indexes, but kernel doesn't support.\n>- Enjoy the short scripts to obtain them! \n>\n>Based on: https://www.kaggle.com/youhanlee/simple-quant-features-using-python\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\n
\n\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | โณTarget Engineering |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | |โณQuant's Volatility Features \n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |โณFourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | โณWavelets | \n\n____","metadata":{"_cell_guid":"0d6da3cc-34e7-416a-b6cb-4ae896ec5838","_uuid":"1fd23ff9-d737-4954-a657-149fa8764d15","papermill":{"duration":0.032023,"end_time":"2021-11-14T06:48:05.26399","exception":false,"start_time":"2021-11-14T06:48:05.231967","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content ๐\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.037447,"end_time":"2021-10-29T13:38:47.950458","exception":false,"start_time":"2021-10-29T13:38:47.913011","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [๐ Introduction](#introduction)\n\n2. [๐ Table Of Content](#outline) \n\n3. [๐คฟ Diving into the Data](#diving) \n\n4. [๐ Imports](#imports) \n\n5. [๐๏ธ Data Loading](#loading) \n\n6. [๐ฌ Feature Engineering](#features) \n","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.040583,"end_time":"2021-10-29T13:38:48.028885","exception":false,"start_time":"2021-10-29T13:38:47.988302","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data ๐คฟ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.038699,"end_time":"2021-10-29T13:38:48.105343","exception":false,"start_time":"2021-10-29T13:38:48.066644","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - ๐ There are 14 coins in the dataset\n>\n> - ๐ There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.036962,"end_time":"2021-10-29T13:38:48.179784","exception":false,"start_time":"2021-10-29T13:38:48.142822","status":"completed"},"tags":[]}},{"cell_type":"code","source":"css_file = '''\ndiv #notebook {\nbackground-color: white;\nfont-family: 'Open Sans', Helvetica, sans-serif;\nline-height: 20px;\n}\n\n#notebook-container {\nmargin-top: 2em;\npadding-top: 2em;\nborder-top: 4px solid %s; /* light orange */\n-webkit-box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n}\n\ndiv .input {\nmargin-bottom: 1em;\n}\n\n.rendered_html h1, .rendered_html h2, .rendered_html h3, .rendered_html h4, .rendered_html h5, .rendered_html h6 {\ncolor: %s; /* light orange */\nfont-weight: 600;\n}\n\n.rendered_html code {\n background-color: #efefef; /* light gray */\n}\n\n.CodeMirror {\ncolor: #8c8c8c; /* dark gray */\npadding: 0.7em;\n}\n\ndiv.input_area {\nborder: none;\n background-color: %s; /* rgba(229, 143, 101, 0.1); light orange [exactly #E58F65] */\n border-top: 2px solid %s; /* light orange */\n}\n\ndiv.input_prompt {\ncolor: %s; /* light blue */\n}\n\ndiv.output_prompt {\ncolor: %s; /* strong orange */\n}\n\ndiv.cell.selected:before, div.cell.selected.jupyter-soft-selected:before {\nbackground: %s; /* light orange */\n}\n\ndiv.cell.selected, div.cell.selected.jupyter-soft-selected {\n border-color: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected:before {\nbackground: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected {\nborder-color: %s; /* light orange */\n\n}\n'''\ndef to_rgb(h): return tuple(int(h[i:i+2], 16) for i in (0, 2, 4))\nmain_color_rgba = 'rgba(%s, %s, %s, 0.1)' % (to_rgb(main_color[1:])[0], to_rgb(main_color[1:])[1], to_rgb(main_color[1:])[2])\nopen('notebook.css', 'w').write(css_file % (main_color, main_color, main_color_rgba, main_color, prompt, strong_main_color, main_color, main_color, main_color, main_color))\nfrom IPython.core.display import display, HTML, Javascript\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75405d52-594f-4791-a45a-16354bc24cfa","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"775496bb-f633-46f4-a642-10e6cd29c1bb","execution":{"iopub.execute_input":"2021-10-29T13:38:48.258919Z","iopub.status.busy":"2021-10-29T13:38:48.257892Z","iopub.status.idle":"2021-10-29T13:38:48.268957Z","shell.execute_reply":"2021-10-29T13:38:48.269468Z","shell.execute_reply.started":"2021-10-29T12:52:34.274423Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.052444,"end_time":"2021-10-29T13:38:48.269664","exception":false,"start_time":"2021-10-29T13:38:48.21722","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\n# def nb(): return HTML(\"\")\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75d3e80e-0c31-48f5-88dd-5b6a3c79cf38","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"968efaca-ede8-4817-b3aa-baedaba3205e","execution":{"iopub.execute_input":"2021-10-29T13:38:48.36471Z","iopub.status.busy":"2021-10-29T13:38:48.363774Z","iopub.status.idle":"2021-10-29T13:38:48.370008Z","shell.execute_reply":"2021-10-29T13:38:48.36937Z","shell.execute_reply.started":"2021-10-29T12:52:34.293927Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.062624,"end_time":"2021-10-29T13:38:48.370222","exception":false,"start_time":"2021-10-29T13:38:48.307598","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle's G-Research Crypto Forecasting\n\nIn this competition, we need to forecast returns of cryptocurrency assets. Full description [here][1]. This is a very challenging time series task as seen by looking at the sample data below.\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/overview","metadata":{"papermill":{"duration":0.049065,"end_time":"2021-11-16T08:32:53.065811","exception":false,"start_time":"2021-11-16T08:32:53.016746","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import gc\nimport pandas as pd\nimport numpy as np\npd.set_option(\"display.max_columns\", None)\nfrom datetime import datetime\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 14})\ndata_folder = \"../input/g-research-crypto-forecasting/\"\n# data_folder = \"/t/Datasets/kaggle_crypto/\"\nasset_details = pd.read_csv(data_folder + 'asset_details.csv', low_memory=False)\ntrain = pd.read_csv(data_folder + 'train.csv', low_memory=False)\nrename_dict = {}\nfor a in asset_details['Asset_ID']: rename_dict[a] = asset_details[asset_details.Asset_ID == a].Asset_Name.values[0]\ntrain['timestamp'] = train['timestamp'].astype('datetime64[s]')\ntrain_daily = pd.DataFrame()\nfor asset_id in asset_details.Asset_ID:\n train_single = train[train.Asset_ID == asset_id].copy()\n train_single_new = train_single[['timestamp','Count']].resample('D', on='timestamp').sum()\n train_single_new['Open'] = train_single[['timestamp','Open']].resample('D', on='timestamp').first()['Open']\n train_single_new['High'] = train_single[['timestamp','High']].resample('D', on='timestamp').max()['High']\n train_single_new['Low'] = train_single[['timestamp','Low']].resample('D', on='timestamp').min()['Low']\n train_single_new['Close'] = train_single[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\n train_single_new['Volume'] = train_single[['timestamp','Volume']].resample('D', on='timestamp').sum()['Volume']\n train_single_new['Asset_ID'] = asset_id\n train_daily = train_daily.append(train_single_new.reset_index(drop=False))\ntrain_daily = train_daily.sort_values(by = ['timestamp', 'Asset_ID']).reset_index(drop=True)\ntrain_daily = train_daily.pivot(index='timestamp', columns='Asset_ID')[['Count', 'Open', 'High', 'Low', 'Close', 'Volume']]\ntrain_daily = train_daily.reset_index(drop=False)\ntrain_daily['year'] = pd.DatetimeIndex(train_daily['timestamp']).year\nfig = make_subplots( rows=len(asset_details.Asset_ID), cols=1, subplot_titles=(asset_details.Asset_Name) )\nfor i, asset_id in enumerate(asset_details.Asset_ID):\n fig.append_trace(go.Candlestick(x=train_daily.timestamp, open=train_daily[('Open', asset_id)], high=train_daily[('High', asset_id)], low=train_daily[('Low', asset_id)], close=train_daily[('Close', asset_id)]),row=i+1, col=1,)\n fig.update_xaxes(range=[train_daily.timestamp.iloc[0], train_daily.timestamp.iloc[-1]], row=i+1, col=1)\nfig.update_layout(xaxis_rangeslider_visible = False, \n xaxis2_rangeslider_visible = False, \n xaxis3_rangeslider_visible = False,\n xaxis4_rangeslider_visible = False,\n xaxis5_rangeslider_visible = False,\n xaxis6_rangeslider_visible = False,\n xaxis7_rangeslider_visible = False,\n xaxis8_rangeslider_visible = False,\n xaxis9_rangeslider_visible = False,\n xaxis10_rangeslider_visible = False,\n xaxis11_rangeslider_visible = False,\n xaxis12_rangeslider_visible = False,\n xaxis13_rangeslider_visible = False,\n xaxis14_rangeslider_visible = False,\n height=3000, width=800, \n #title_text=\"Subplots with Annotations\"\n margin = dict(\n l = 0,\n r = 0,\n b = 0,\n t = 30,\n pad = 0)\n ) \nfig.show()\ndel train, train_daily,train_single, asset_details\ngc.collect()\nplt.close()","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-11-16T08:32:53.172558Z","iopub.status.busy":"2021-11-16T08:32:53.171458Z","iopub.status.idle":"2021-11-16T08:34:16.626911Z","shell.execute_reply":"2021-11-16T08:34:16.627477Z"},"papermill":{"duration":83.51092,"end_time":"2021-11-16T08:34:16.627686","exception":false,"start_time":"2021-11-16T08:32:53.116766","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.execute_input":"2021-11-16T08:34:16.792586Z","iopub.status.busy":"2021-11-16T08:34:16.791495Z","iopub.status.idle":"2021-11-16T08:34:16.796297Z","shell.execute_reply":"2021-11-16T08:34:16.796813Z","shell.execute_reply.started":"2021-11-15T12:22:32.681506Z"},"papermill":{"duration":0.088608,"end_time":"2021-11-16T08:34:16.797007","exception":false,"start_time":"2021-11-16T08:34:16.708399","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = pd.read_csv(\"../input/g-research-crypto-forecasting/\" + 'train.csv')\nassets = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nassets_names = dict(zip(assets.Asset_ID, assets.Asset_Name))\nassets_order = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv').Asset_ID[:14]\nassets_order = dict((t,i) for i,t in enumerate(assets_order))\nbtc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\")\nbtc_mini = btc.iloc[-200:]\n# fig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\n# fig.show()\ndel btc","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-11-16T08:34:16.976277Z","iopub.status.busy":"2021-11-16T08:34:16.974947Z","iopub.status.idle":"2021-11-16T08:35:00.185727Z","shell.execute_reply":"2021-11-16T08:35:00.184972Z"},"papermill":{"duration":43.309053,"end_time":"2021-11-16T08:35:00.185889","exception":false,"start_time":"2021-11-16T08:34:16.876836","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = crypto_df.copy().set_index(\"timestamp\")\ntrain['assets'] = 1\ntrain['assets'] = train.groupby(by = train.index)['assets'].sum()\ntrain['asset_name'] = train.Asset_ID.map(assets_names)\ntrain['asset_name'].value_counts()\nall_same_time = train[train['assets'] == 14][['Asset_ID', 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'asset_name','VWAP','Target']]\ncorr_target = all_same_time.reset_index().pivot(index = 'asset_name', columns = 'timestamp')['Target'].transpose().corr()\ncorr_matrix = all_same_time.reset_index().drop(['Target', 'Asset_ID'], axis=1).pivot(index = 'asset_name', columns = 'timestamp').transpose().corr()\nfig, ax = plt.subplots(1, 2, figsize = (20, 8))\nsns.heatmap(np.round(corr_target, 2), annot = True, ax = ax[0], square = True)\nsns.heatmap(np.round(corr_matrix, 2), annot = True, ax = ax[1], square = True)\nax[0].title.set_text('Asset Targets correlation')\nax[1].title.set_text('Asset Features correlation')","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.execute_input":"2021-11-16T08:35:00.353461Z","iopub.status.busy":"2021-11-16T08:35:00.350632Z","iopub.status.idle":"2021-11-16T08:35:58.91999Z","shell.execute_reply":"2021-11-16T08:35:58.920667Z"},"papermill":{"duration":58.6563,"end_time":"2021-11-16T08:35:58.921091","exception":false,"start_time":"2021-11-16T08:35:00.264791","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try: plt.close()\nexcept: pass\nfig = make_subplots(rows=7, cols=2,shared_xaxes=True, vertical_spacing=0.03, subplot_titles=tuple([assets_names[i] for i in range(14)]))\ndata = all_same_time[1000:2400]\ndata['time'] = [pd.to_datetime(x, unit='s') for x in data.index]\nfor i in range(14): \n coin = data[data.Asset_ID == i]\n name = assets_names[i]\n fig.add_trace(go.Scatter(x=coin['time'], y=coin['VWAP'], name = name + ', VWAP'),row=i//2+1, col= i%2 +1)\nfig.update_layout(height=1000, title_text=' Weighted average prices')\nfig.show()\ndel data, train, crypto_df, all_same_time, corr_target","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.execute_input":"2021-11-16T08:35:59.109762Z","iopub.status.busy":"2021-11-16T08:35:59.103722Z","iopub.status.idle":"2021-11-16T08:35:59.788058Z","shell.execute_reply":"2021-11-16T08:35:59.787447Z"},"papermill":{"duration":0.779451,"end_time":"2021-11-16T08:35:59.788228","exception":false,"start_time":"2021-11-16T08:35:59.008777","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Libraries ๐\n\n\n#### Code starts here โฌ","metadata":{"_cell_guid":"d849b62e-dcc7-4e5a-95c7-031dee523204","_uuid":"359fc054-a69e-481b-938b-6fedc83ddc9b","papermill":{"duration":0.032261,"end_time":"2021-11-14T06:49:06.810374","exception":false,"start_time":"2021-11-14T06:49:06.778113","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport gc\nimport traceback\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\nimport gresearch_crypto\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\ndata_path = '../input/g-research-crypto-forecasting/'\n\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nwarnings.simplefilter(action='ignore', category=pd.core.common.SettingWithCopyWarning)\n \nplt.style.use('bmh')\nplt.rcParams['figure.figsize'] = [14, 8] # width, height","metadata":{"_cell_guid":"7b701afb-9ee3-4f9e-a219-3b6206a8283d","_uuid":"35744779-02d1-445c-a8d4-dcbf1e1bdba2","execution":{"iopub.execute_input":"2021-11-14T06:49:06.883189Z","iopub.status.busy":"2021-11-14T06:49:06.882553Z","iopub.status.idle":"2021-11-14T06:49:07.039928Z","shell.execute_reply":"2021-11-14T06:49:07.039343Z","shell.execute_reply.started":"2021-11-14T06:45:02.63288Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.196788,"end_time":"2021-11-14T06:49:07.040078","exception":false,"start_time":"2021-11-14T06:49:06.84329","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loading ๐๏ธ\n\n\nIn the real competition data, the number of datapoints per day (that is per \"group\") is not constant as it was in the spoofed data. We need to confirm that the time series split respects that there are different counts of samples in the the days. We load the data and reduce memory footprint.","metadata":{"_cell_guid":"0f8bf214-4860-4c6f-97fc-e3e374d5b646","_uuid":"1034e975-839a-4c8d-96eb-c83436fb735a","papermill":{"duration":0.035041,"end_time":"2021-11-14T06:49:07.10877","exception":false,"start_time":"2021-11-14T06:49:07.073729","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n \"\"\" iterate through all the columns of a dataframe and modify the data type\n to reduce memory usage.\n \"\"\"\n start_mem = df.memory_usage().sum() / 1024**2\n print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n for col in df.columns:\n col_type = df[col].dtype.name\n\n if col_type not in ['object', 'category', 'datetime64[ns, UTC]']:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64)\n\n end_mem = df.memory_usage().sum() / 1024**2\n print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n return df","metadata":{"_cell_guid":"b56ab916-dae6-4ff7-bb2d-52398dd52e1c","_kg_hide-input":true,"_uuid":"1146dd4e-6f2e-45c4-a2fd-7fee6e751e98","execution":{"iopub.execute_input":"2021-11-14T06:49:07.181449Z","iopub.status.busy":"2021-11-14T06:49:07.180732Z","iopub.status.idle":"2021-11-14T06:49:07.195261Z","shell.execute_reply":"2021-11-14T06:49:07.195809Z","shell.execute_reply.started":"2021-11-14T06:45:02.642126Z"},"papermill":{"duration":0.052878,"end_time":"2021-11-14T06:49:07.195979","exception":false,"start_time":"2021-11-14T06:49:07.143101","status":"completed"},"tags":[],"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INC2021 = 0\nINC2020 = 0\nINC2019 = 0\nINC2018 = 0\nINC2017 = 0\nINCCOMP = 1\nINCSUPP = 0\n\norig_df_train = pd.read_csv(data_path + 'train.csv')\nsupp_df_train = pd.read_csv(data_path + 'supplemental_train.csv')\ndf_asset_details = pd.read_csv(data_path + 'asset_details.csv').sort_values(\"Asset_ID\")\n\nextra_data_files = {0: '../input/cryptocurrency-extra-data-binance-coin', 2: '../input/cryptocurrency-extra-data-bitcoin-cash', 1: '../input/cryptocurrency-extra-data-bitcoin', 3: '../input/cryptocurrency-extra-data-cardano', 4: '../input/cryptocurrency-extra-data-dogecoin', 5: '../input/cryptocurrency-extra-data-eos-io', 6: '../input/cryptocurrency-extra-data-ethereum', 7: '../input/cryptocurrency-extra-data-ethereum-classic', 8: '../input/cryptocurrency-extra-data-iota', 9: '../input/cryptocurrency-extra-data-litecoin', 11: '../input/cryptocurrency-extra-data-monero', 10: '../input/cryptocurrency-extra-data-maker', 12: '../input/cryptocurrency-extra-data-stellar', 13: '../input/cryptocurrency-extra-data-tron'}\n\ndef load_training_data_for_asset(asset_id):\n dfs = []\n if INCCOMP: dfs.append(orig_df_train[orig_df_train[\"Asset_ID\"] == asset_id].copy())\n if INCSUPP: dfs.append(supp_df_train[supp_df_train[\"Asset_ID\"] == asset_id].copy())\n if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'))\n if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'))\n if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'))\n if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'))\n if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'))\n df = pd.concat(dfs, axis = 0) if len(dfs) > 1 else dfs[0]\n df['date'] = pd.to_datetime(df['timestamp'], unit = 's')\n df = df.sort_values('date')\n return df\n\ndef load_data_for_all_assets():\n dfs = []\n for asset_id in list(extra_data_files.keys()): dfs.append(load_training_data_for_asset(asset_id))\n return pd.concat(dfs)","metadata":{"_cell_guid":"8da68356-63f7-49c7-aca2-b61019c278b6","_uuid":"aafb204a-c174-4898-af33-eede907e879b","execution":{"iopub.execute_input":"2021-11-14T06:49:07.267633Z","iopub.status.busy":"2021-11-14T06:49:07.266596Z","iopub.status.idle":"2021-11-14T06:50:00.21167Z","shell.execute_reply":"2021-11-14T06:50:00.21244Z","shell.execute_reply.started":"2021-11-14T06:45:02.661071Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":52.982521,"end_time":"2021-11-14T06:50:00.21267","exception":false,"start_time":"2021-11-14T06:49:07.230149","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = load_data_for_all_assets().sort_values('timestamp').set_index(\"timestamp\")\ntest = pd.read_csv(data_path + 'example_test.csv')\nsample_prediction_df = pd.read_csv(data_path + 'example_sample_submission.csv')\nprint(\"Loaded all data!\")","metadata":{"_cell_guid":"8da68356-63f7-49c7-aca2-b61019c278b6","_uuid":"aafb204a-c174-4898-af33-eede907e879b","execution":{"iopub.execute_input":"2021-11-14T06:49:07.267633Z","iopub.status.busy":"2021-11-14T06:49:07.266596Z","iopub.status.idle":"2021-11-14T06:50:00.21167Z","shell.execute_reply":"2021-11-14T06:50:00.21244Z","shell.execute_reply.started":"2021-11-14T06:45:02.661071Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":52.982521,"end_time":"2021-11-14T06:50:00.21267","exception":false,"start_time":"2021-11-14T06:49:07.230149","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering ๐ฌ\n","metadata":{"_cell_guid":"22bb5d25-9c27-4be9-b9ac-0701b71058a5","_uuid":"137210f4-aa91-4525-9f53-5e91f4bf9ced","papermill":{"duration":0.033621,"end_time":"2021-11-14T06:50:00.280969","exception":false,"start_time":"2021-11-14T06:50:00.247348","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport time\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport lightgbm as lgb\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nplt.style.use('seaborn')\nsns.set(font_scale=2)\nimport warnings; warnings.filterwarnings('ignore')","metadata":{"_cell_guid":"6bc533d7-8cf8-4763-9fd7-947b0c97a418","_uuid":"ba8dac38-b6bf-4fed-877e-98b44d5fd021","execution":{"iopub.execute_input":"2021-11-14T06:50:00.352937Z","iopub.status.busy":"2021-11-14T06:50:00.352241Z","iopub.status.idle":"2021-11-14T06:50:02.207171Z","shell.execute_reply":"2021-11-14T06:50:02.207672Z","shell.execute_reply.started":"2021-11-14T06:46:02.484748Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":1.892525,"end_time":"2021-11-14T06:50:02.207855","exception":false,"start_time":"2021-11-14T06:50:00.31533","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = train.copy()\ntrain_data['date'] = pd.to_datetime(train_data['date'])","metadata":{"_cell_guid":"571b1302-cafc-476d-9ce3-ccbd528a8fca","_uuid":"58eb1885-6f55-45ed-a981-30b49b4cb621","execution":{"iopub.execute_input":"2021-11-14T06:50:02.282371Z","iopub.status.busy":"2021-11-14T06:50:02.281755Z","iopub.status.idle":"2021-11-14T06:50:04.470204Z","shell.execute_reply":"2021-11-14T06:50:04.469506Z","shell.execute_reply.started":"2021-11-14T06:46:02.499304Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":2.227614,"end_time":"2021-11-14T06:50:04.470352","exception":false,"start_time":"2021-11-14T06:50:02.242738","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = train_data.loc[train_data['Asset_ID'] == 1]","metadata":{"execution":{"iopub.execute_input":"2021-11-14T06:50:04.546374Z","iopub.status.busy":"2021-11-14T06:50:04.545746Z","iopub.status.idle":"2021-11-14T06:50:04.826941Z","shell.execute_reply":"2021-11-14T06:50:04.827511Z","shell.execute_reply.started":"2021-11-14T06:46:04.555668Z"},"papermill":{"duration":0.321427,"end_time":"2021-11-14T06:50:04.827715","exception":false,"start_time":"2021-11-14T06:50:04.506288","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"N=100\n\ndf['timestamp'] = df['date']\ndf.set_index(df['timestamp'], inplace=True)\ndf.drop('timestamp', axis=1, inplace=True)\n\nconvertion={\n 'Open':'first',\n 'High':'max',\n 'Low':'min',\n 'Close':'mean',\n 'Volume':'sum', \n}\nds_df = df.resample('W').apply(convertion)","metadata":{"execution":{"iopub.execute_input":"2021-11-14T06:50:04.905758Z","iopub.status.busy":"2021-11-14T06:50:04.904846Z","iopub.status.idle":"2021-11-14T06:50:05.077055Z","shell.execute_reply":"2021-11-14T06:50:05.076465Z","shell.execute_reply.started":"2021-11-14T06:46:04.846843Z"},"papermill":{"duration":0.213173,"end_time":"2021-11-14T06:50:05.07721","exception":false,"start_time":"2021-11-14T06:50:04.864037","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Moving average","metadata":{"_cell_guid":"cae4b110-d1b4-455e-92b0-50df34d1615f","_uuid":"b75b464d-6d22-4e39-a929-35ffc2b394cf","papermill":{"duration":0.034352,"end_time":"2021-11-14T06:50:05.146141","exception":false,"start_time":"2021-11-14T06:50:05.111789","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"> An example of two moving average curves\nIn statistics, a moving average (rolling average or running average) is a calculation to analyze data points by creating series of averages of different subsets of the full data set. It is also called a moving mean (MM)[1] or rolling mean and is a type of finite impulse response filter.\n\nref. https://en.wikipedia.org/wiki/Moving_average","metadata":{"_cell_guid":"8dca4c94-c86d-46ce-8d3e-c92c1a5bebac","_uuid":"5a62a22a-fe70-4908-8c24-f1c649dc133a","papermill":{"duration":0.034639,"end_time":"2021-11-14T06:50:05.215441","exception":false,"start_time":"2021-11-14T06:50:05.180802","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Moving average","metadata":{"_cell_guid":"ef98fb9b-610c-4805-8454-1d0d8657e818","_uuid":"15465fd7-3aa2-4edf-97f8-767c31344b53","papermill":{"duration":0.034247,"end_time":"2021-11-14T06:50:05.284106","exception":false,"start_time":"2021-11-14T06:50:05.249859","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"- Moving average is simple","metadata":{"_cell_guid":"2dcde37a-656c-4e01-bf84-a30dae50839a","_uuid":"4eac8cf9-581d-4305-8b5c-f3e104e9eabc","papermill":{"duration":0.034338,"end_time":"2021-11-14T06:50:05.353128","exception":false,"start_time":"2021-11-14T06:50:05.31879","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\n\nds_df['rolling_mean' + str(N) + '_' + str(5)] = ds_df.Close.rolling(window=5).mean()\nds_df['rolling_mean' + str(N) + '_' + str(10)] = ds_df.Close.rolling(window=10).mean()\n\n\n\nfig = go.Figure(go.Candlestick(x=ds_df.index,open=ds_df['Open'],high=ds_df['High'],low=ds_df['Low'],close=ds_df['Close']))\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.update_yaxes(type=\"log\")\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close'))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['rolling_mean' + str(N) + '_' + str(5)], mode='lines', name='MEAN_5' + str(N),line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['rolling_mean' + str(N) + '_' + str(10)], mode='lines', name='MEAN_10' + str(N), line=dict(color='#555555', width=2)))\nfig.show()","metadata":{"_cell_guid":"3010207c-899f-4a8a-91c1-f0f70a3f614f","_uuid":"de2e1d5a-e16c-4393-804d-c9c30a17728b","execution":{"iopub.execute_input":"2021-11-14T06:50:05.432513Z","iopub.status.busy":"2021-11-14T06:50:05.431817Z","iopub.status.idle":"2021-11-14T06:50:05.522826Z","shell.execute_reply":"2021-11-14T06:50:05.523319Z","shell.execute_reply.started":"2021-11-14T06:46:05.021022Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.135838,"end_time":"2021-11-14T06:50:05.52351","exception":false,"start_time":"2021-11-14T06:50:05.387672","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exponential Moving Average","metadata":{"_cell_guid":"04020fd2-681c-46d4-b208-f79b81ffde77","_uuid":"1360c499-7054-4853-8f01-aa92a19fe185","papermill":{"duration":0.035624,"end_time":"2021-11-14T06:50:05.595736","exception":false,"start_time":"2021-11-14T06:50:05.560112","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"> An exponential moving average (EMA), also known as an exponentially weighted moving average (EWMA),[5] is a first-order infinite impulse response filter that applies weighting factors which decrease exponentially.\n\nref. https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average","metadata":{"_cell_guid":"1d1e5a8a-4746-4ac3-8c56-28014e215861","_uuid":"157a7e6a-8d3c-41b7-a768-8e4a7d95bb3d","papermill":{"duration":0.035433,"end_time":"2021-11-14T06:50:05.666863","exception":false,"start_time":"2021-11-14T06:50:05.63143","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ewma = pd.Series.ewm","metadata":{"_cell_guid":"1d2a707d-4f9f-41b3-b55a-0907bb8cf964","_uuid":"71b68725-33dc-4949-967b-e8e954ab6028","execution":{"iopub.execute_input":"2021-11-14T06:50:05.743912Z","iopub.status.busy":"2021-11-14T06:50:05.743136Z","iopub.status.idle":"2021-11-14T06:50:05.746035Z","shell.execute_reply":"2021-11-14T06:50:05.745414Z","shell.execute_reply.started":"2021-11-14T06:46:05.074957Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.043295,"end_time":"2021-11-14T06:50:05.746182","exception":false,"start_time":"2021-11-14T06:50:05.702887","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_df['rolling_ema_'+ str(N)] = ds_df.Close.ewm(min_periods=N, span=N).mean()","metadata":{"_cell_guid":"c90748be-d0f1-4010-a5e5-f7087b5983a1","_uuid":"562b8c20-102c-4023-babe-c1aaf16dff03","execution":{"iopub.execute_input":"2021-11-14T06:50:05.824589Z","iopub.status.busy":"2021-11-14T06:50:05.823908Z","iopub.status.idle":"2021-11-14T06:50:05.82667Z","shell.execute_reply":"2021-11-14T06:50:05.826095Z","shell.execute_reply.started":"2021-11-14T06:46:05.08038Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.044545,"end_time":"2021-11-14T06:50:05.826806","exception":false,"start_time":"2021-11-14T06:50:05.782261","status":"completed"},"scrolled":true,"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nds_df['rolling_ema_' + str(N)] = ds_df.Close.ewm(min_periods=10, span=10).mean()\n\n\n\nfig = go.Figure(go.Candlestick(x=ds_df.index,open=ds_df['Open'],high=ds_df['High'],low=ds_df['Low'],close=ds_df['Close']))\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.update_yaxes(type=\"log\")\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close'))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['rolling_ema_' + str(N)], mode='lines', name='EMA_10',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"334f4843-d876-4222-bd4c-b39afecf05d9","_uuid":"52910804-3407-4ae1-b018-9852b8e2137b","execution":{"iopub.execute_input":"2021-11-14T06:50:05.934016Z","iopub.status.busy":"2021-11-14T06:50:05.928447Z","iopub.status.idle":"2021-11-14T06:50:05.938743Z","shell.execute_reply":"2021-11-14T06:50:05.938068Z","shell.execute_reply.started":"2021-11-14T06:46:05.095952Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.075934,"end_time":"2021-11-14T06:50:05.93889","exception":false,"start_time":"2021-11-14T06:50:05.862956","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MACD\n- MACD: (12-day EMA - 26-day EMA)","metadata":{"_cell_guid":"adb574c4-4181-407f-8106-d85d7958a8aa","_uuid":"111348f2-4efd-4af2-b27b-41b2abb56093","papermill":{"duration":0.037164,"end_time":"2021-11-14T06:50:06.075317","exception":false,"start_time":"2021-11-14T06:50:06.038153","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"> Moving average convergence divergence (MACD) is a trend-following momentum indicator that shows the relationship between two moving averages of prices. The MACD is calculated by subtracting the 26-day exponential moving average (EMA) from the 12-day EMA\n\nref. https://www.investopedia.com/terms/m/macd.asp","metadata":{"_cell_guid":"39e48161-7f03-431c-8352-b5a9f31dca24","_uuid":"3fc20909-0230-4d2b-b414-db45cd85c6bc","papermill":{"duration":0.038948,"end_time":"2021-11-14T06:50:06.153801","exception":false,"start_time":"2021-11-14T06:50:06.114853","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ds_df['close_5EMA'] = ewma(ds_df[\"Close\"], span=5).mean()\nds_df['close_2EMA'] = ewma(ds_df[\"Close\"], span=2).mean()\n\nds_df['MACD'] = ds_df['close_5EMA'] - ds_df['close_2EMA']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['MACD'], mode='lines', name='MACD_26_12',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"77e0b3d8-bd6d-4dba-a94f-e701060722fb","_uuid":"9661b1aa-37b0-4fe9-ade0-09b3d89c77bd","execution":{"iopub.execute_input":"2021-11-14T06:50:06.260164Z","iopub.status.busy":"2021-11-14T06:50:06.259494Z","iopub.status.idle":"2021-11-14T06:50:06.263848Z","shell.execute_reply":"2021-11-14T06:50:06.263288Z","shell.execute_reply.started":"2021-11-14T06:46:05.13779Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.072082,"end_time":"2021-11-14T06:50:06.264","exception":false,"start_time":"2021-11-14T06:50:06.191918","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Bollinger Band","metadata":{"_cell_guid":"196a9592-3970-48c3-8f69-91d13fec6a3f","_uuid":"ba216a46-7207-40ec-bebf-efeec36fe818","papermill":{"duration":0.038264,"end_time":"2021-11-14T06:50:06.341218","exception":false,"start_time":"2021-11-14T06:50:06.302954","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"> Bollinger Bands are a type of statistical chart characterizing the prices and volatility over time of a financial instrument or commodity, using a formulaic method propounded by John Bollinger in the 1980s. Financial traders employ these charts as a methodical tool to inform trading decisions, control automated trading systems, or as a component of technical analysis. Bollinger Bands display a graphical band (the envelope maximum and minimum of moving averages, similar to Keltner or Donchian channels) and volatility (expressed by the width of the envelope) in one two-dimensional chart.\n\nref. https://en.wikipedia.org/wiki/Bollinger_Bands","metadata":{"_cell_guid":"81977b36-9e0b-4ac1-9059-90bd39e61058","_uuid":"07575636-49f6-4890-8dbb-51762c1348ea","papermill":{"duration":0.038322,"end_time":"2021-11-14T06:50:06.418271","exception":false,"start_time":"2021-11-14T06:50:06.379949","status":"completed"},"tags":[]}},{"cell_type":"code","source":"window = 7\nno_of_std = 2\n\nds_df[f'MA_{window}MA'] = ds_df['Close'].rolling(window=window).mean()\nds_df[f'MA_{window}MA_std'] = ds_df['Close'].rolling(window=window).std() \nds_df[f'MA_{window}MA_BB_high'] = ds_df[f'MA_{window}MA'] + no_of_std * ds_df[f'MA_{window}MA_std']\nds_df[f'MA_{window}MA_BB_low'] = ds_df[f'MA_{window}MA'] - no_of_std * ds_df[f'MA_{window}MA_std']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_high'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_low'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"48b0e438-6bd7-4983-9e85-f856db23a34c","_uuid":"3fa2074c-3ad4-422b-a0ab-a5bb3648b868","execution":{"iopub.execute_input":"2021-11-14T06:50:06.535046Z","iopub.status.busy":"2021-11-14T06:50:06.53432Z","iopub.status.idle":"2021-11-14T06:50:06.538155Z","shell.execute_reply":"2021-11-14T06:50:06.538661Z","shell.execute_reply.started":"2021-11-14T06:46:05.172197Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.082063,"end_time":"2021-11-14T06:50:06.538827","exception":false,"start_time":"2021-11-14T06:50:06.456764","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"window = 15\nno_of_std = 2\n\nds_df[f'MA_{window}MA'] = ds_df['Close'].rolling(window=window).mean()\nds_df[f'MA_{window}MA_std'] = ds_df['Close'].rolling(window=window).std() \nds_df[f'MA_{window}MA_BB_high'] = ds_df[f'MA_{window}MA'] + no_of_std * ds_df[f'MA_{window}MA_std']\nds_df[f'MA_{window}MA_BB_low'] = ds_df[f'MA_{window}MA'] - no_of_std * ds_df[f'MA_{window}MA_std']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_high'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_low'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"c4e48be9-ea62-4d2d-bb60-c0c7f7814472","_uuid":"86c0f705-aeca-45cc-a522-70a01c5aab80","execution":{"iopub.execute_input":"2021-11-14T06:50:06.660745Z","iopub.status.busy":"2021-11-14T06:50:06.65987Z","iopub.status.idle":"2021-11-14T06:50:06.663995Z","shell.execute_reply":"2021-11-14T06:50:06.663456Z","shell.execute_reply.started":"2021-11-14T06:46:05.21749Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.084186,"end_time":"2021-11-14T06:50:06.664139","exception":false,"start_time":"2021-11-14T06:50:06.579953","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"window = 30\nno_of_std = 2\n\nds_df[f'MA_{window}MA'] = ds_df['Close'].rolling(window=window).mean()\nds_df[f'MA_{window}MA_std'] = ds_df['Close'].rolling(window=window).std() \nds_df[f'MA_{window}MA_BB_high'] = ds_df[f'MA_{window}MA'] + no_of_std * ds_df[f'MA_{window}MA_std']\nds_df[f'MA_{window}MA_BB_low'] = ds_df[f'MA_{window}MA'] - no_of_std * ds_df[f'MA_{window}MA_std']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_high'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_low'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"1205818e-df96-425d-87a3-cada7f81dbfe","_uuid":"f2a55b8c-4d8a-47c0-b3a4-ef7367a6afeb","execution":{"iopub.execute_input":"2021-11-14T06:50:06.789937Z","iopub.status.busy":"2021-11-14T06:50:06.788841Z","iopub.status.idle":"2021-11-14T06:50:06.793061Z","shell.execute_reply":"2021-11-14T06:50:06.793567Z","shell.execute_reply.started":"2021-11-14T06:46:05.261888Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.087539,"end_time":"2021-11-14T06:50:06.793735","exception":false,"start_time":"2021-11-14T06:50:06.706196","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# RSI","metadata":{"_cell_guid":"bf689a0a-8063-4b63-9d45-c0dfa4179829","_uuid":"b3638b0e-53bd-4099-a999-c5675dc92175","papermill":{"duration":0.042516,"end_time":"2021-11-14T06:50:06.879578","exception":false,"start_time":"2021-11-14T06:50:06.837062","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"> The Relative Strength Index (RSI), developed by J. Welles Wilder, is a momentum oscillator that measures the speed and change of price movements. The RSI oscillates between zero and 100. Traditionally the RSI is considered overbought when above 70 and oversold when below 30. Signals can be generated by looking for divergences and failure swings. RSI can also be used to identify the general trend.\n\nref. https://www.fidelity.com/learning-center/trading-investing/technical-analysis/technical-indicator-guide/RSI","metadata":{"_cell_guid":"22cb2cb2-1778-4231-837e-6df3db1e9097","_uuid":"929ef4bf-1f03-4c66-9255-49ea27958f5e","papermill":{"duration":0.042814,"end_time":"2021-11-14T06:50:06.965999","exception":false,"start_time":"2021-11-14T06:50:06.923185","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def rsiFunc(prices, n=14):\n deltas = np.diff(prices)\n seed = deltas[:n+1]\n up = seed[seed>=0].sum()/n\n down = -seed[seed<0].sum()/n\n rs = up/down\n rsi = np.zeros_like(prices)\n rsi[:n] = 100. - 100./(1.+rs)\n\n for i in range(n, len(prices)):\n delta = deltas[i-1] # cause the diff is 1 shorter\n\n if delta>0:\n upval = delta\n downval = 0.\n else:\n upval = 0.\n downval = -delta\n\n up = (up*(n-1) + upval)/n\n down = (down*(n-1) + downval)/n\n\n rs = up/down\n rsi[i] = 100. - 100./(1.+rs)\n\n return rsi","metadata":{"_cell_guid":"6240c59f-fdd7-4705-8c6c-f2d94cdec3cf","_uuid":"42300a35-e987-48e1-956f-b073d8f826d5","execution":{"iopub.execute_input":"2021-11-14T06:50:07.061441Z","iopub.status.busy":"2021-11-14T06:50:07.060661Z","iopub.status.idle":"2021-11-14T06:50:07.063979Z","shell.execute_reply":"2021-11-14T06:50:07.063424Z","shell.execute_reply.started":"2021-11-14T06:46:05.309059Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.054998,"end_time":"2021-11-14T06:50:07.064124","exception":false,"start_time":"2021-11-14T06:50:07.009126","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rsi_6 = rsiFunc(ds_df['Close'].values, 6)\nrsi_14 = rsiFunc(ds_df['Close'].values, 14)\nrsi_20 = rsiFunc(ds_df['Close'].values, 20)","metadata":{"_cell_guid":"c42a8e72-6c49-4d7d-b4c2-1597ceeef555","_uuid":"1d875c6d-943c-4728-aebf-1a8a58729da3","execution":{"iopub.execute_input":"2021-11-14T06:50:07.185766Z","iopub.status.busy":"2021-11-14T06:50:07.184713Z","iopub.status.idle":"2021-11-14T06:50:07.187665Z","shell.execute_reply":"2021-11-14T06:50:07.186845Z","shell.execute_reply.started":"2021-11-14T06:46:05.320418Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.074722,"end_time":"2021-11-14T06:50:07.187867","exception":false,"start_time":"2021-11-14T06:50:07.113145","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_df['rsi_6'] = rsi_6\nds_df['rsi_14'] = rsi_14\nds_df['rsi_20'] = rsi_20\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'rsi_6'], mode='lines', name=f'rsi_6',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'rsi_14'], mode='lines', name=f'rsi_14',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'rsi_20'], mode='lines', name=f'rsi_20',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"018dfafe-1397-4a3a-bdce-177cba445cc5","_uuid":"297317e3-ffc3-4238-8714-fd007fc30c67","execution":{"iopub.execute_input":"2021-11-14T06:50:07.296548Z","iopub.status.busy":"2021-11-14T06:50:07.295819Z","iopub.status.idle":"2021-11-14T06:50:07.319875Z","shell.execute_reply":"2021-11-14T06:50:07.320463Z","shell.execute_reply.started":"2021-11-14T06:46:05.340736Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.08237,"end_time":"2021-11-14T06:50:07.320635","exception":false,"start_time":"2021-11-14T06:50:07.238265","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Volume Moving Avreage","metadata":{"_cell_guid":"1f6fff08-e246-4cad-93d7-0865ea48a9b6","_uuid":"07065536-0e08-4656-a1b5-338088e9b1dd","papermill":{"duration":0.044113,"end_time":"2021-11-14T06:50:07.409527","exception":false,"start_time":"2021-11-14T06:50:07.365414","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"> A Volume Moving Average is the simplest volume-based technical indicator. Similar to a price moving average, a VMA is an average volume of a security (stock), commodity, index or exchange over a selected period of time. Volume Moving Averages are used in charts and in technical analysis to smooth and describe a volume trend by filtering short term spikes and gaps.\n\nref. https://www.marketvolume.com/analysis/volume_ma.asp","metadata":{"_cell_guid":"49aa5df8-5bda-44c6-84f3-d27d856e8d85","_uuid":"a3951021-6027-4ebb-b633-3cfb2fc6f183","papermill":{"duration":0.043827,"end_time":"2021-11-14T06:50:07.497939","exception":false,"start_time":"2021-11-14T06:50:07.454112","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ds_df['VMA_7MA'] = ds_df['Volume'].rolling(window=7).mean()\nds_df['VMA_15MA'] = ds_df['Volume'].rolling(window=15).mean()\nds_df['VMA_30MA'] = ds_df['Volume'].rolling(window=30).mean()\nds_df['VMA_60MA'] = ds_df['Volume'].rolling(window=60).mean()","metadata":{"_cell_guid":"461014ce-158c-4c5f-8caf-6f3025945f05","_uuid":"168621f4-a0e8-47e2-bed8-285df5dceb07","execution":{"iopub.execute_input":"2021-11-14T06:50:07.591363Z","iopub.status.busy":"2021-11-14T06:50:07.590699Z","iopub.status.idle":"2021-11-14T06:50:07.600096Z","shell.execute_reply":"2021-11-14T06:50:07.600621Z","shell.execute_reply.started":"2021-11-14T06:46:05.386675Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.058504,"end_time":"2021-11-14T06:50:07.600793","exception":false,"start_time":"2021-11-14T06:50:07.542289","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_7MA'], mode='lines', name=f'VMA_7MA',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_15MA'], mode='lines', name=f'VMA_15MA',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_30MA'], mode='lines', name=f'VMA_30MA',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_60MA'], mode='lines', name=f'VMA_60MA',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_cell_guid":"32f02875-efc7-4f19-8ca3-aba2eb57301a","_uuid":"90c3b0dd-ae9f-4550-bfb2-d7d125bc06b8","execution":{"iopub.execute_input":"2021-11-14T06:50:07.700693Z","iopub.status.busy":"2021-11-14T06:50:07.699962Z","iopub.status.idle":"2021-11-14T06:50:07.735394Z","shell.execute_reply":"2021-11-14T06:50:07.735889Z","shell.execute_reply.started":"2021-11-14T06:46:05.402771Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.090197,"end_time":"2021-11-14T06:50:07.736058","exception":false,"start_time":"2021-11-14T06:50:07.645861","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# More to come..","metadata":{"_cell_guid":"1f6fff08-e246-4cad-93d7-0865ea48a9b6","_uuid":"07065536-0e08-4656-a1b5-338088e9b1dd","papermill":{"duration":0.046129,"end_time":"2021-11-14T06:50:07.83123","exception":false,"start_time":"2021-11-14T06:50:07.785101","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# References\n\n1. [Initial baseline notebook](https://www.kaggle.com/julian3833) \n2. [Competition tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition) \n3. [Competition Overview](https://www.kaggle.com/c/g-research-crypto-forecasting/overview) \n4. [My Initial Ideas for this competition](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903) \n5. [My post notebook about cross validation](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit) \n6. [Chris original notebook from SIIM ISIC](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) \n\nWORK IN PROGRESS! ๐ง","metadata":{"papermill":{"duration":0.045575,"end_time":"2021-11-14T06:50:07.922766","exception":false,"start_time":"2021-11-14T06:50:07.877191","status":"completed"},"tags":[]}}]}
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n