{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\nhtml_contents = \"\"\"\n<!DOCTYPE html>\n<html lang=\"en\">\n    <head>\n        <link rel=\"stylesheet\" href=\"https://www.w3schools.com/w3css/4/w3.css\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Raleway\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Oswald\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Open Sans\">\n        <link rel=\"stylesheet\" href=\"https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css\">\n        <style>\n        .title-section{\n            font-family: \"Oswald\", Arial, sans-serif;\n            font-weight: bold;\n            color: \"#6A8CAF\";\n            letter-spacing: 6px;\n        }\n        hr { border: 1px solid #E58F65 !important;\n             color: #E58F65 !important;\n             background: #E58F65 !important;\n           }\n        body {\n            font-family: \"Open Sans\", sans-serif;\n            }\n        </style>\n    </head>\n</html>\n\"\"\"\n\nHTML(html_contents)","metadata":{"_uuid":"ee7d593b-6eb9-471f-b3ca-ef9140138133","_cell_guid":"1422d689-e9b4-42b3-8859-4fc8864ab080","papermill":{"duration":0.138805,"end_time":"2020-12-15T20:11:57.447976","exception":false,"start_time":"2020-12-15T20:11:57.309171","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-14T06:44:13.260333Z","iopub.execute_input":"2021-11-14T06:44:13.260645Z","iopub.status.idle":"2021-11-14T06:44:13.268865Z","shell.execute_reply.started":"2021-11-14T06:44:13.260613Z","shell.execute_reply":"2021-11-14T06:44:13.268081Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Technical Analysis Indicators</span>\n\n- Here are some simple indexes to analyze the charts. some can even be used as features to a model.\n- Ta-lib is very good and very helpful library for calculating various indexes, but kernel doesn't support.\n- Enjoy the short scripts to obtain them! \n\nBased on: https://www.kaggle.com/youhanlee/simple-quant-features-using-python\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. \n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903","metadata":{"_uuid":"d27b3620-794d-4dfc-9aa3-93c82c292251","_cell_guid":"2355d578-9afb-4f67-92e6-454a9f8de893","trusted":true}},{"cell_type":"markdown","source":"____\n\n#### <center>All baselines in the series 👇</center>\n\n| CV + Model | Hyperparam Optimization  | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner)        | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | ⏳Technical Analysis |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm)     | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna)     | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | ⏳Time Series Agg | \n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost)      | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | ⏳Target Engineering |\n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data)                                            | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | |⏳Neutralization |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | |⏳Quant's Volatility Features |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer)                                     | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline)  | |⏳Fourier Analysis \n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data)                                       |  |  | ⏳Wavelets | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) |  |  \n\n____","metadata":{"_uuid":"1fd23ff9-d737-4954-a657-149fa8764d15","_cell_guid":"0d6da3cc-34e7-416a-b6cb-4ae896ec5838","trusted":true}},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Kaggle's G-Research Crypto Forecasting</span>\nIn this competition, we need to forecast returns of cryptocurrency assets. Full description [here][1]. This is a very challenging time series task as seen by looking at the sample data below.\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/overview","metadata":{"_uuid":"5b319991-064e-45b4-9d2e-9eca3729068b","_cell_guid":"1e7eb273-abfa-4702-8a75-9c0d722138d7","trusted":true}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport plotly.graph_objects as go\ndata_path = '../input/g-research-crypto-forecasting/'\ncrypto_df = pd.read_csv( data_path + 'train.csv')\nbtc = crypto_df[crypto_df[\"Asset_ID\"] == 1].set_index(\"timestamp\")\nbtc_mini = btc.iloc[-200:]\nfig = go.Figure(data = [go.Candlestick(x = btc_mini.index, open = btc_mini['Open'], high = btc_mini['High'], low = btc_mini['Low'], close = btc_mini['Close'])])\nfig.show()","metadata":{"_uuid":"56f385ae-0ff5-408e-8f0d-f5983a52e9f0","_cell_guid":"ddc3d8b5-7376-4966-b8db-e2ae49ca522c","collapsed":false,"papermill":{"duration":0.138805,"end_time":"2020-12-15T20:11:57.447976","exception":false,"start_time":"2020-12-15T20:11:57.309171","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:44:13.277664Z","iopub.execute_input":"2021-11-14T06:44:13.280267Z","iopub.status.idle":"2021-11-14T06:45:02.631081Z","shell.execute_reply.started":"2021-11-14T06:44:13.280198Z","shell.execute_reply":"2021-11-14T06:45:02.630083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Initialize Environment</span>","metadata":{"_uuid":"359fc054-a69e-481b-938b-6fedc83ddc9b","_cell_guid":"d849b62e-dcc7-4e5a-95c7-031dee523204","trusted":true}},{"cell_type":"code","source":"import os\nimport gc\nimport traceback\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\nimport gresearch_crypto\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\ndata_path = '../input/g-research-crypto-forecasting/'\n\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nwarnings.simplefilter(action='ignore', category=pd.core.common.SettingWithCopyWarning)\n    \nplt.style.use('bmh')\nplt.rcParams['figure.figsize'] = [14, 8]  # width, height","metadata":{"_uuid":"35744779-02d1-445c-a8d4-dcbf1e1bdba2","_cell_guid":"7b701afb-9ee3-4f9e-a219-3b6206a8283d","collapsed":false,"papermill":{"duration":0.138805,"end_time":"2020-12-15T20:11:57.447976","exception":false,"start_time":"2020-12-15T20:11:57.309171","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:45:02.632658Z","iopub.execute_input":"2021-11-14T06:45:02.63291Z","iopub.status.idle":"2021-11-14T06:45:02.640307Z","shell.execute_reply.started":"2021-11-14T06:45:02.63288Z","shell.execute_reply":"2021-11-14T06:45:02.639162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the Competition Data\n\nIn the real competition data, the number of datapoints per day (that is per \"group\") is not constant as it was in the spoofed data. We need to confirm that the time series split respects that there are different counts of samples in the the days. We load the data and reduce memory footprint.","metadata":{"_uuid":"1034e975-839a-4c8d-96eb-c83436fb735a","_cell_guid":"0f8bf214-4860-4c6f-97fc-e3e374d5b646","papermill":{"duration":0.030318,"end_time":"2020-12-15T20:11:59.305649","exception":false,"start_time":"2020-12-15T20:11:59.275331","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        col_type = df[col].dtype.name\n\n        if col_type not in ['object', 'category', 'datetime64[ns, UTC]']:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df","metadata":{"_uuid":"1146dd4e-6f2e-45c4-a2fd-7fee6e751e98","_cell_guid":"b56ab916-dae6-4ff7-bb2d-52398dd52e1c","_kg_hide-input":true,"papermill":{"duration":0.051755,"end_time":"2020-12-15T20:11:59.387975","exception":false,"start_time":"2020-12-15T20:11:59.33622","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-11-14T06:45:02.641848Z","iopub.execute_input":"2021-11-14T06:45:02.642179Z","iopub.status.idle":"2021-11-14T06:45:02.658643Z","shell.execute_reply.started":"2021-11-14T06:45:02.642126Z","shell.execute_reply":"2021-11-14T06:45:02.657548Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INC2021 = 0\nINC2020 = 0\nINC2019 = 0\nINC2018 = 0\nINC2017 = 0\nINCCOMP = 1\nINCSUPP = 0\n\norig_df_train = pd.read_csv(data_path + 'train.csv')\nsupp_df_train = pd.read_csv(data_path + 'supplemental_train.csv')\ndf_asset_details = pd.read_csv(data_path  + 'asset_details.csv').sort_values(\"Asset_ID\")\n\nextra_data_files = {0: '../input/cryptocurrency-extra-data-binance-coin', 2: '../input/cryptocurrency-extra-data-bitcoin-cash', 1: '../input/cryptocurrency-extra-data-bitcoin', 3: '../input/cryptocurrency-extra-data-cardano', 4: '../input/cryptocurrency-extra-data-dogecoin', 5: '../input/cryptocurrency-extra-data-eos-io', 6: '../input/cryptocurrency-extra-data-ethereum', 7: '../input/cryptocurrency-extra-data-ethereum-classic', 8: '../input/cryptocurrency-extra-data-iota', 9: '../input/cryptocurrency-extra-data-litecoin', 11: '../input/cryptocurrency-extra-data-monero', 10: '../input/cryptocurrency-extra-data-maker', 12: '../input/cryptocurrency-extra-data-stellar', 13: '../input/cryptocurrency-extra-data-tron'}\n\ndef load_training_data_for_asset(asset_id):\n    dfs = []\n    if INCCOMP: dfs.append(orig_df_train[orig_df_train[\"Asset_ID\"] == asset_id].copy())\n    if INCSUPP: dfs.append(supp_df_train[supp_df_train[\"Asset_ID\"] == asset_id].copy())\n    if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'))\n    if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'))\n    if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'))\n    if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'))\n    if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'))\n    df = pd.concat(dfs, axis = 0) if len(dfs) > 1 else dfs[0]\n    df['date'] = pd.to_datetime(df['timestamp'], unit = 's')\n    df = df.sort_values('date')\n    return df\n\ndef load_data_for_all_assets():\n    dfs = []\n    for asset_id in list(extra_data_files.keys()): dfs.append(load_training_data_for_asset(asset_id))\n    return pd.concat(dfs)\n\ntrain = load_data_for_all_assets().sort_values('timestamp').set_index(\"timestamp\")\ntest = pd.read_csv(data_path + 'example_test.csv')\nsample_prediction_df = pd.read_csv(data_path + 'example_sample_submission.csv')\nprint(\"Loaded all data!\")","metadata":{"_uuid":"aafb204a-c174-4898-af33-eede907e879b","_cell_guid":"8da68356-63f7-49c7-aca2-b61019c278b6","collapsed":false,"papermill":{"duration":109.709242,"end_time":"2020-12-15T20:13:49.127945","exception":false,"start_time":"2020-12-15T20:11:59.418703","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:45:02.660853Z","iopub.execute_input":"2021-11-14T06:45:02.661111Z","iopub.status.idle":"2021-11-14T06:46:02.478016Z","shell.execute_reply.started":"2021-11-14T06:45:02.661071Z","shell.execute_reply":"2021-11-14T06:46:02.477427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Feature Engineering</span>","metadata":{"_uuid":"137210f4-aa91-4525-9f53-5e91f4bf9ced","_cell_guid":"22bb5d25-9c27-4be9-b9ac-0701b71058a5","trusted":true}},{"cell_type":"code","source":"import os\nimport time\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport lightgbm as lgb\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nplt.style.use('seaborn')\nsns.set(font_scale=2)\nimport warnings; warnings.filterwarnings('ignore')","metadata":{"_uuid":"ba8dac38-b6bf-4fed-877e-98b44d5fd021","_cell_guid":"6bc533d7-8cf8-4763-9fd7-947b0c97a418","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:02.483918Z","iopub.execute_input":"2021-11-14T06:46:02.484796Z","iopub.status.idle":"2021-11-14T06:46:02.497635Z","shell.execute_reply.started":"2021-11-14T06:46:02.484748Z","shell.execute_reply":"2021-11-14T06:46:02.496507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = train.copy()\ntrain_data['date'] = pd.to_datetime(train_data['date'])","metadata":{"_uuid":"58eb1885-6f55-45ed-a981-30b49b4cb621","_cell_guid":"571b1302-cafc-476d-9ce3-ccbd528a8fca","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:02.498683Z","iopub.execute_input":"2021-11-14T06:46:02.499336Z","iopub.status.idle":"2021-11-14T06:46:04.554166Z","shell.execute_reply.started":"2021-11-14T06:46:02.499304Z","shell.execute_reply":"2021-11-14T06:46:04.553343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = train_data.loc[train_data['Asset_ID'] == 1]","metadata":{"execution":{"iopub.status.busy":"2021-11-14T06:46:04.555146Z","iopub.execute_input":"2021-11-14T06:46:04.55571Z","iopub.status.idle":"2021-11-14T06:46:04.845558Z","shell.execute_reply.started":"2021-11-14T06:46:04.555668Z","shell.execute_reply":"2021-11-14T06:46:04.844581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"N=100\n\ndf['timestamp'] = df['date']\ndf.set_index(df['timestamp'], inplace=True)\ndf.drop('timestamp', axis=1, inplace=True)\n\nconvertion={\n    'Open':'first',\n    'High':'max',\n    'Low':'min',\n    'Close':'mean',\n    'Volume':'sum',    \n}\nds_df = df.resample('W').apply(convertion)","metadata":{"execution":{"iopub.status.busy":"2021-11-14T06:46:04.846656Z","iopub.execute_input":"2021-11-14T06:46:04.846869Z","iopub.status.idle":"2021-11-14T06:46:05.01686Z","shell.execute_reply.started":"2021-11-14T06:46:04.846843Z","shell.execute_reply":"2021-11-14T06:46:05.016096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Moving average","metadata":{"_uuid":"b75b464d-6d22-4e39-a929-35ffc2b394cf","_cell_guid":"cae4b110-d1b4-455e-92b0-50df34d1615f","trusted":true}},{"cell_type":"markdown","source":"> An example of two moving average curves\nIn statistics, a moving average (rolling average or running average) is a calculation to analyze data points by creating series of averages of different subsets of the full data set. It is also called a moving mean (MM)[1] or rolling mean and is a type of finite impulse response filter.\n\nref. https://en.wikipedia.org/wiki/Moving_average","metadata":{"_uuid":"5a62a22a-fe70-4908-8c24-f1c649dc133a","_cell_guid":"8dca4c94-c86d-46ce-8d3e-c92c1a5bebac","trusted":true}},{"cell_type":"markdown","source":"## Moving average","metadata":{"_uuid":"15465fd7-3aa2-4edf-97f8-767c31344b53","_cell_guid":"ef98fb9b-610c-4805-8454-1d0d8657e818","trusted":true}},{"cell_type":"markdown","source":"- Moving average is simple","metadata":{"_uuid":"4eac8cf9-581d-4305-8b5c-f3e104e9eabc","_cell_guid":"2dcde37a-656c-4e01-bf84-a30dae50839a","trusted":true}},{"cell_type":"code","source":"\n\nds_df['rolling_mean' + str(N) + '_' + str(5)] = ds_df.Close.rolling(window=5).mean()\nds_df['rolling_mean' + str(N) + '_' + str(10)] = ds_df.Close.rolling(window=10).mean()\n\n\n\nfig = go.Figure(go.Candlestick(x=ds_df.index,open=ds_df['Open'],high=ds_df['High'],low=ds_df['Low'],close=ds_df['Close']))\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.update_yaxes(type=\"log\")\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close'))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['rolling_mean' + str(N) + '_' + str(5)], mode='lines', name='MEAN_5' + str(N),line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['rolling_mean' + str(N) + '_' + str(10)], mode='lines', name='MEAN_10' + str(N), line=dict(color='#555555', width=2)))\nfig.show()","metadata":{"_uuid":"de2e1d5a-e16c-4393-804d-c9c30a17728b","_cell_guid":"3010207c-899f-4a8a-91c1-f0f70a3f614f","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.020273Z","iopub.execute_input":"2021-11-14T06:46:05.021061Z","iopub.status.idle":"2021-11-14T06:46:05.073514Z","shell.execute_reply.started":"2021-11-14T06:46:05.021022Z","shell.execute_reply":"2021-11-14T06:46:05.072741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exponential Moving Average","metadata":{"_uuid":"1360c499-7054-4853-8f01-aa92a19fe185","_cell_guid":"04020fd2-681c-46d4-b208-f79b81ffde77","trusted":true}},{"cell_type":"markdown","source":"> An exponential moving average (EMA), also known as an exponentially weighted moving average (EWMA),[5] is a first-order infinite impulse response filter that applies weighting factors which decrease exponentially.\n\nref. https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average","metadata":{"_uuid":"157a7e6a-8d3c-41b7-a768-8e4a7d95bb3d","_cell_guid":"1d1e5a8a-4746-4ac3-8c56-28014e215861","trusted":true}},{"cell_type":"code","source":"ewma = pd.Series.ewm","metadata":{"_uuid":"71b68725-33dc-4949-967b-e8e954ab6028","_cell_guid":"1d2a707d-4f9f-41b3-b55a-0907bb8cf964","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.074678Z","iopub.execute_input":"2021-11-14T06:46:05.074989Z","iopub.status.idle":"2021-11-14T06:46:05.078626Z","shell.execute_reply.started":"2021-11-14T06:46:05.074957Z","shell.execute_reply":"2021-11-14T06:46:05.077916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_df['rolling_ema_'+ str(N)]  = ds_df.Close.ewm(min_periods=N, span=N).mean()","metadata":{"_uuid":"562b8c20-102c-4023-babe-c1aaf16dff03","_cell_guid":"c90748be-d0f1-4010-a5e5-f7087b5983a1","collapsed":false,"scrolled":true,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.079825Z","iopub.execute_input":"2021-11-14T06:46:05.080417Z","iopub.status.idle":"2021-11-14T06:46:05.094052Z","shell.execute_reply.started":"2021-11-14T06:46:05.08038Z","shell.execute_reply":"2021-11-14T06:46:05.093129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nds_df['rolling_ema_' + str(N)] = ds_df.Close.ewm(min_periods=10, span=10).mean()\n\n\n\nfig = go.Figure(go.Candlestick(x=ds_df.index,open=ds_df['Open'],high=ds_df['High'],low=ds_df['Low'],close=ds_df['Close']))\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.update_yaxes(type=\"log\")\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close'))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['rolling_ema_' + str(N)], mode='lines', name='EMA_10',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"52910804-3407-4ae1-b018-9852b8e2137b","_cell_guid":"334f4843-d876-4222-bd4c-b39afecf05d9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.095337Z","iopub.execute_input":"2021-11-14T06:46:05.095987Z","iopub.status.idle":"2021-11-14T06:46:05.135945Z","shell.execute_reply.started":"2021-11-14T06:46:05.095952Z","shell.execute_reply":"2021-11-14T06:46:05.135073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MACD\n- MACD: (12-day EMA - 26-day EMA)","metadata":{"_uuid":"111348f2-4efd-4af2-b27b-41b2abb56093","_cell_guid":"adb574c4-4181-407f-8106-d85d7958a8aa","trusted":true}},{"cell_type":"markdown","source":"> Moving average convergence divergence (MACD) is a trend-following momentum indicator that shows the relationship between two moving averages of prices. The MACD is calculated by subtracting the 26-day exponential moving average (EMA) from the 12-day EMA\n\nref. https://www.investopedia.com/terms/m/macd.asp","metadata":{"_uuid":"3fc20909-0230-4d2b-b414-db45cd85c6bc","_cell_guid":"39e48161-7f03-431c-8352-b5a9f31dca24","trusted":true}},{"cell_type":"code","source":"ds_df['close_5EMA'] = ewma(ds_df[\"Close\"], span=5).mean()\nds_df['close_2EMA'] = ewma(ds_df[\"Close\"], span=2).mean()\n\nds_df['MACD'] = ds_df['close_5EMA'] - ds_df['close_2EMA']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['MACD'], mode='lines', name='MACD_26_12',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"9661b1aa-37b0-4fe9-ade0-09b3d89c77bd","_cell_guid":"77e0b3d8-bd6d-4dba-a94f-e701060722fb","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.13753Z","iopub.execute_input":"2021-11-14T06:46:05.137828Z","iopub.status.idle":"2021-11-14T06:46:05.170427Z","shell.execute_reply.started":"2021-11-14T06:46:05.13779Z","shell.execute_reply":"2021-11-14T06:46:05.169752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Bollinger Band","metadata":{"_uuid":"ba216a46-7207-40ec-bebf-efeec36fe818","_cell_guid":"196a9592-3970-48c3-8f69-91d13fec6a3f","trusted":true}},{"cell_type":"markdown","source":"> Bollinger Bands are a type of statistical chart characterizing the prices and volatility over time of a financial instrument or commodity, using a formulaic method propounded by John Bollinger in the 1980s. Financial traders employ these charts as a methodical tool to inform trading decisions, control automated trading systems, or as a component of technical analysis. Bollinger Bands display a graphical band (the envelope maximum and minimum of moving averages, similar to Keltner or Donchian channels) and volatility (expressed by the width of the envelope) in one two-dimensional chart.\n\nref. https://en.wikipedia.org/wiki/Bollinger_Bands","metadata":{"_uuid":"07575636-49f6-4890-8dbb-51762c1348ea","_cell_guid":"81977b36-9e0b-4ac1-9059-90bd39e61058","trusted":true}},{"cell_type":"code","source":"window = 7\nno_of_std = 2\n\nds_df[f'MA_{window}MA'] = ds_df['Close'].rolling(window=window).mean()\nds_df[f'MA_{window}MA_std'] = ds_df['Close'].rolling(window=window).std() \nds_df[f'MA_{window}MA_BB_high'] = ds_df[f'MA_{window}MA'] + no_of_std * ds_df[f'MA_{window}MA_std']\nds_df[f'MA_{window}MA_BB_low'] = ds_df[f'MA_{window}MA'] - no_of_std * ds_df[f'MA_{window}MA_std']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_high'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_low'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"3fa2074c-3ad4-422b-a0ab-a5bb3648b868","_cell_guid":"48b0e438-6bd7-4983-9e85-f856db23a34c","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.171713Z","iopub.execute_input":"2021-11-14T06:46:05.172252Z","iopub.status.idle":"2021-11-14T06:46:05.216028Z","shell.execute_reply.started":"2021-11-14T06:46:05.172197Z","shell.execute_reply":"2021-11-14T06:46:05.215283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"window = 15\nno_of_std = 2\n\nds_df[f'MA_{window}MA'] = ds_df['Close'].rolling(window=window).mean()\nds_df[f'MA_{window}MA_std'] = ds_df['Close'].rolling(window=window).std() \nds_df[f'MA_{window}MA_BB_high'] = ds_df[f'MA_{window}MA'] + no_of_std * ds_df[f'MA_{window}MA_std']\nds_df[f'MA_{window}MA_BB_low'] = ds_df[f'MA_{window}MA'] - no_of_std * ds_df[f'MA_{window}MA_std']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_high'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_low'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"86c0f705-aeca-45cc-a522-70a01c5aab80","_cell_guid":"c4e48be9-ea62-4d2d-bb60-c0c7f7814472","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.217018Z","iopub.execute_input":"2021-11-14T06:46:05.217524Z","iopub.status.idle":"2021-11-14T06:46:05.260421Z","shell.execute_reply.started":"2021-11-14T06:46:05.21749Z","shell.execute_reply":"2021-11-14T06:46:05.259461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"window = 30\nno_of_std = 2\n\nds_df[f'MA_{window}MA'] = ds_df['Close'].rolling(window=window).mean()\nds_df[f'MA_{window}MA_std'] = ds_df['Close'].rolling(window=window).std() \nds_df[f'MA_{window}MA_BB_high'] = ds_df[f'MA_{window}MA'] + no_of_std * ds_df[f'MA_{window}MA_std']\nds_df[f'MA_{window}MA_BB_low'] = ds_df[f'MA_{window}MA'] - no_of_std * ds_df[f'MA_{window}MA_std']\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df['Close'],mode='lines',name='Close', line=dict(color='#555555', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_high'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'MA_{window}MA_BB_low'], mode='lines', name=f'BB_high',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"f2a55b8c-4d8a-47c0-b3a4-ef7367a6afeb","_cell_guid":"1205818e-df96-425d-87a3-cada7f81dbfe","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.261387Z","iopub.execute_input":"2021-11-14T06:46:05.261924Z","iopub.status.idle":"2021-11-14T06:46:05.307798Z","shell.execute_reply.started":"2021-11-14T06:46:05.261888Z","shell.execute_reply":"2021-11-14T06:46:05.307186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# RSI","metadata":{"_uuid":"b3638b0e-53bd-4099-a999-c5675dc92175","_cell_guid":"bf689a0a-8063-4b63-9d45-c0dfa4179829","trusted":true}},{"cell_type":"markdown","source":"> The Relative Strength Index (RSI), developed by J. Welles Wilder, is a momentum oscillator that measures the speed and change of price movements. The RSI oscillates between zero and 100. Traditionally the RSI is considered overbought when above 70 and oversold when below 30. Signals can be generated by looking for divergences and failure swings. RSI can also be used to identify the general trend.\n\nref. https://www.fidelity.com/learning-center/trading-investing/technical-analysis/technical-indicator-guide/RSI","metadata":{"_uuid":"929ef4bf-1f03-4c66-9255-49ea27958f5e","_cell_guid":"22cb2cb2-1778-4231-837e-6df3db1e9097","trusted":true}},{"cell_type":"code","source":"def rsiFunc(prices, n=14):\n    deltas = np.diff(prices)\n    seed = deltas[:n+1]\n    up = seed[seed>=0].sum()/n\n    down = -seed[seed<0].sum()/n\n    rs = up/down\n    rsi = np.zeros_like(prices)\n    rsi[:n] = 100. - 100./(1.+rs)\n\n    for i in range(n, len(prices)):\n        delta = deltas[i-1] # cause the diff is 1 shorter\n\n        if delta>0:\n            upval = delta\n            downval = 0.\n        else:\n            upval = 0.\n            downval = -delta\n\n        up = (up*(n-1) + upval)/n\n        down = (down*(n-1) + downval)/n\n\n        rs = up/down\n        rsi[i] = 100. - 100./(1.+rs)\n\n    return rsi","metadata":{"_uuid":"42300a35-e987-48e1-956f-b073d8f826d5","_cell_guid":"6240c59f-fdd7-4705-8c6c-f2d94cdec3cf","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.308722Z","iopub.execute_input":"2021-11-14T06:46:05.309093Z","iopub.status.idle":"2021-11-14T06:46:05.317938Z","shell.execute_reply.started":"2021-11-14T06:46:05.309059Z","shell.execute_reply":"2021-11-14T06:46:05.31695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rsi_6 = rsiFunc(ds_df['Close'].values, 6)\nrsi_14 = rsiFunc(ds_df['Close'].values, 14)\nrsi_20 = rsiFunc(ds_df['Close'].values, 20)","metadata":{"_uuid":"1d875c6d-943c-4728-aebf-1a8a58729da3","_cell_guid":"c42a8e72-6c49-4d7d-b4c2-1597ceeef555","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.319615Z","iopub.execute_input":"2021-11-14T06:46:05.32047Z","iopub.status.idle":"2021-11-14T06:46:05.339149Z","shell.execute_reply.started":"2021-11-14T06:46:05.320418Z","shell.execute_reply":"2021-11-14T06:46:05.338307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_df['rsi_6'] = rsi_6\nds_df['rsi_14'] = rsi_14\nds_df['rsi_20'] = rsi_20\n\nfig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'rsi_6'], mode='lines', name=f'rsi_6',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'rsi_14'], mode='lines', name=f'rsi_14',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'rsi_20'], mode='lines', name=f'rsi_20',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"297317e3-ffc3-4238-8714-fd007fc30c67","_cell_guid":"018dfafe-1397-4a3a-bdce-177cba445cc5","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.340326Z","iopub.execute_input":"2021-11-14T06:46:05.34077Z","iopub.status.idle":"2021-11-14T06:46:05.385097Z","shell.execute_reply.started":"2021-11-14T06:46:05.340736Z","shell.execute_reply":"2021-11-14T06:46:05.383967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Volume Moving Avreage","metadata":{"_uuid":"07065536-0e08-4656-a1b5-338088e9b1dd","_cell_guid":"1f6fff08-e246-4cad-93d7-0865ea48a9b6","trusted":true}},{"cell_type":"markdown","source":"> A Volume Moving Average is the simplest volume-based technical indicator. Similar to a price moving average, a VMA is an average volume of a security (stock), commodity, index or exchange over a selected period of time. Volume Moving Averages are used in charts and in technical analysis to smooth and describe a volume trend by filtering short term spikes and gaps.\n\nref. https://www.marketvolume.com/analysis/volume_ma.asp","metadata":{"_uuid":"a3951021-6027-4ebb-b633-3cfb2fc6f183","_cell_guid":"49aa5df8-5bda-44c6-84f3-d27d856e8d85","trusted":true}},{"cell_type":"code","source":"ds_df['VMA_7MA'] = ds_df['Volume'].rolling(window=7).mean()\nds_df['VMA_15MA'] = ds_df['Volume'].rolling(window=15).mean()\nds_df['VMA_30MA'] = ds_df['Volume'].rolling(window=30).mean()\nds_df['VMA_60MA'] = ds_df['Volume'].rolling(window=60).mean()","metadata":{"_uuid":"168621f4-a0e8-47e2-bed8-285df5dceb07","_cell_guid":"461014ce-158c-4c5f-8caf-6f3025945f05","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.386389Z","iopub.execute_input":"2021-11-14T06:46:05.386704Z","iopub.status.idle":"2021-11-14T06:46:05.401132Z","shell.execute_reply.started":"2021-11-14T06:46:05.386675Z","shell.execute_reply":"2021-11-14T06:46:05.400286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\nfig.update_layout(title='Bitcoin Price', yaxis_title='BTC')\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_7MA'], mode='lines', name=f'VMA_7MA',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_15MA'], mode='lines', name=f'VMA_15MA',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_30MA'], mode='lines', name=f'VMA_30MA',line=dict(color='royalblue', width=2)))\nfig.add_trace(go.Scatter(x=ds_df.index, y=ds_df[f'VMA_60MA'], mode='lines', name=f'VMA_60MA',line=dict(color='royalblue', width=2)))\nfig.show()","metadata":{"_uuid":"90c3b0dd-ae9f-4550-bfb2-d7d125bc06b8","_cell_guid":"32f02875-efc7-4f19-8ca3-aba2eb57301a","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-11-14T06:46:05.402455Z","iopub.execute_input":"2021-11-14T06:46:05.402804Z","iopub.status.idle":"2021-11-14T06:46:05.44638Z","shell.execute_reply.started":"2021-11-14T06:46:05.402771Z","shell.execute_reply":"2021-11-14T06:46:05.445391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# More to come..","metadata":{"_uuid":"07065536-0e08-4656-a1b5-338088e9b1dd","_cell_guid":"1f6fff08-e246-4cad-93d7-0865ea48a9b6","trusted":true}},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\">References</span>\n\n<span id=\"f1\">1.</span> [Initial baseline notebook](https://www.kaggle.com/julian3833)<br>\n<span id=\"f2\">2.</span> [Competition tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)<br>\n<span id=\"f3\">3.</span> [Competition Overview](https://www.kaggle.com/c/g-research-crypto-forecasting/overview)</span><br>\n<span id=\"f4\">4.</span> [My Initial Ideas for this competition](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903)</span><br>\n<span id=\"f5\">5.</span> [My post notebook about cross validation](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit)</span><br>\n<span id=\"f5\">6.</span> [Chris original notebook from SIIM ISIC](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)</span><br>\n\n<span class=\"title-section w3-large w3-tag\">WORK IN PROGRESS! 🚧</span>","metadata":{}}]}