","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.036915,"end_time":"2021-10-29T13:38:47.184754","exception":false,"start_time":"2021-10-29T13:38:47.147839","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.046159,"end_time":"2021-10-29T13:38:47.267941","exception":false,"start_time":"2021-10-29T13:38:47.221782","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":">### PurgedGroupTimeSeries CV with Extra Data\n>This is a simple starter notebook for Kaggle's Crypto Comp showing purged group timeseries KFold with extra data. Purged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either CPU or GPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model hyperparameters, loss, and number of seeds to ensemble. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n>\n>**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validation experiments but in that case note that the overall CV score is meaningless because LB will be much different when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\n
\n\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | โณTarget Engineering |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | |โณQuant's Volatility Features \n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |โณFourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | โณWavelets | \n\n____","metadata":{"papermill":{"duration":0.016676,"end_time":"2021-11-10T08:29:54.452085","exception":false,"start_time":"2021-11-10T08:29:54.435409","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content ๐\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.037447,"end_time":"2021-10-29T13:38:47.950458","exception":false,"start_time":"2021-10-29T13:38:47.913011","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [๐ Introduction](#introduction)\n\n2. [๐ Table Of Content](#outline) \n\n3. [๐คฟ Diving into the Data](#diving) \n\n4. [๐ Imports](#imports) \n\n5. [๐๏ธ Configurations](#config) \n\n6. [๐๏ธ Data Loading](#loading) \n\n7. [๐ฌ Feature Engineering](#features) \n\n8. [โ๏ธ Configure The Model](#modelconf)\n\n9. [โฑ๏ธ LR Scheduler](#sched)\n\n10. [๐๏ธ Training](#training)\n\n11. [๐ฐ Submit to Kaggle](#submit)","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.040583,"end_time":"2021-10-29T13:38:48.028885","exception":false,"start_time":"2021-10-29T13:38:47.988302","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data ๐คฟ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.038699,"end_time":"2021-10-29T13:38:48.105343","exception":false,"start_time":"2021-10-29T13:38:48.066644","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - ๐ There are 14 coins in the dataset\n>\n> - ๐ There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.036962,"end_time":"2021-10-29T13:38:48.179784","exception":false,"start_time":"2021-10-29T13:38:48.142822","status":"completed"},"tags":[]}},{"cell_type":"code","source":"css_file = '''\ndiv #notebook {\nbackground-color: white;\nfont-family: 'Open Sans', Helvetica, sans-serif;\nline-height: 20px;\n}\n\n#notebook-container {\nmargin-top: 2em;\npadding-top: 2em;\nborder-top: 4px solid %s; /* light orange */\n-webkit-box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n}\n\ndiv .input {\nmargin-bottom: 1em;\n}\n\n.rendered_html h1, .rendered_html h2, .rendered_html h3, .rendered_html h4, .rendered_html h5, .rendered_html h6 {\ncolor: %s; /* light orange */\nfont-weight: 600;\n}\n\n.rendered_html code {\n background-color: #efefef; /* light gray */\n}\n\n.CodeMirror {\ncolor: #8c8c8c; /* dark gray */\npadding: 0.7em;\n}\n\ndiv.input_area {\nborder: none;\n background-color: %s; /* rgba(229, 143, 101, 0.1); light orange [exactly #E58F65] */\n border-top: 2px solid %s; /* light orange */\n}\n\ndiv.input_prompt {\ncolor: %s; /* light blue */\n}\n\ndiv.output_prompt {\ncolor: %s; /* strong orange */\n}\n\ndiv.cell.selected:before, div.cell.selected.jupyter-soft-selected:before {\nbackground: %s; /* light orange */\n}\n\ndiv.cell.selected, div.cell.selected.jupyter-soft-selected {\n border-color: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected:before {\nbackground: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected {\nborder-color: %s; /* light orange */\n\n}\n'''\ndef to_rgb(h): return tuple(int(h[i:i+2], 16) for i in (0, 2, 4))\nmain_color_rgba = 'rgba(%s, %s, %s, 0.1)' % (to_rgb(main_color[1:])[0], to_rgb(main_color[1:])[1], to_rgb(main_color[1:])[2])\nopen('notebook.css', 'w').write(css_file % (main_color, main_color, main_color_rgba, main_color, prompt, strong_main_color, main_color, main_color, main_color, main_color))\nfrom IPython.core.display import display, HTML, Javascript\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75405d52-594f-4791-a45a-16354bc24cfa","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"775496bb-f633-46f4-a642-10e6cd29c1bb","execution":{"iopub.execute_input":"2021-10-29T13:38:48.258919Z","iopub.status.busy":"2021-10-29T13:38:48.257892Z","iopub.status.idle":"2021-10-29T13:38:48.268957Z","shell.execute_reply":"2021-10-29T13:38:48.269468Z","shell.execute_reply.started":"2021-10-29T12:52:34.274423Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.052444,"end_time":"2021-10-29T13:38:48.269664","exception":false,"start_time":"2021-10-29T13:38:48.21722","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\n# def nb(): return HTML(\"\")\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75d3e80e-0c31-48f5-88dd-5b6a3c79cf38","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"968efaca-ede8-4817-b3aa-baedaba3205e","execution":{"iopub.execute_input":"2021-10-29T13:38:48.36471Z","iopub.status.busy":"2021-10-29T13:38:48.363774Z","iopub.status.idle":"2021-10-29T13:38:48.370008Z","shell.execute_reply":"2021-10-29T13:38:48.36937Z","shell.execute_reply.started":"2021-10-29T12:52:34.293927Z"},"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.062624,"end_time":"2021-10-29T13:38:48.370222","exception":false,"start_time":"2021-10-29T13:38:48.307598","status":"completed"},"tags":[],"collapsed":false},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle's G-Research Crypto Forecasting\n\nIn this competition, we need to forecast returns of cryptocurrency assets. Full description [here][1]. This is a very challenging time series task as seen by looking at the sample data below.\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/overview","metadata":{"papermill":{"duration":0.049065,"end_time":"2021-11-16T08:32:53.065811","exception":false,"start_time":"2021-11-16T08:32:53.016746","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import gc\nimport pandas as pd\nimport numpy as np\npd.set_option(\"display.max_columns\", None)\nfrom datetime import datetime\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 14})\ndata_folder = \"../input/g-research-crypto-forecasting/\"\n# data_folder = \"/t/Datasets/kaggle_crypto/\"\nasset_details = pd.read_csv(data_folder + 'asset_details.csv', low_memory=False)\ntrain = pd.read_csv(data_folder + 'train.csv', low_memory=False)\nrename_dict = {}\nfor a in asset_details['Asset_ID']: rename_dict[a] = asset_details[asset_details.Asset_ID == a].Asset_Name.values[0]\ntrain['timestamp'] = train['timestamp'].astype('datetime64[s]')\ntrain_daily = pd.DataFrame()\nfor asset_id in asset_details.Asset_ID:\n train_single = train[train.Asset_ID == asset_id].copy()\n train_single_new = train_single[['timestamp','Count']].resample('D', on='timestamp').sum()\n train_single_new['Open'] = train_single[['timestamp','Open']].resample('D', on='timestamp').first()['Open']\n train_single_new['High'] = train_single[['timestamp','High']].resample('D', on='timestamp').max()['High']\n train_single_new['Low'] = train_single[['timestamp','Low']].resample('D', on='timestamp').min()['Low']\n train_single_new['Close'] = train_single[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\n train_single_new['Volume'] = train_single[['timestamp','Volume']].resample('D', on='timestamp').sum()['Volume']\n train_single_new['Asset_ID'] = asset_id\n train_daily = train_daily.append(train_single_new.reset_index(drop=False))\ntrain_daily = train_daily.sort_values(by = ['timestamp', 'Asset_ID']).reset_index(drop=True)\ntrain_daily = train_daily.pivot(index='timestamp', columns='Asset_ID')[['Count', 'Open', 'High', 'Low', 'Close', 'Volume']]\ntrain_daily = train_daily.reset_index(drop=False)\ntrain_daily['year'] = pd.DatetimeIndex(train_daily['timestamp']).year\nfig = make_subplots( rows=len(asset_details.Asset_ID), cols=1, subplot_titles=(asset_details.Asset_Name) )\nfor i, asset_id in enumerate(asset_details.Asset_ID):\n fig.append_trace(go.Candlestick(x=train_daily.timestamp, open=train_daily[('Open', asset_id)], high=train_daily[('High', asset_id)], low=train_daily[('Low', asset_id)], close=train_daily[('Close', asset_id)]),row=i+1, col=1,)\n fig.update_xaxes(range=[train_daily.timestamp.iloc[0], train_daily.timestamp.iloc[-1]], row=i+1, col=1)\nfig.update_layout(xaxis_rangeslider_visible = False, \n xaxis2_rangeslider_visible = False, \n xaxis3_rangeslider_visible = False,\n xaxis4_rangeslider_visible = False,\n xaxis5_rangeslider_visible = False,\n xaxis6_rangeslider_visible = False,\n xaxis7_rangeslider_visible = False,\n xaxis8_rangeslider_visible = False,\n xaxis9_rangeslider_visible = False,\n xaxis10_rangeslider_visible = False,\n xaxis11_rangeslider_visible = False,\n xaxis12_rangeslider_visible = False,\n xaxis13_rangeslider_visible = False,\n xaxis14_rangeslider_visible = False,\n height=3000, width=800, \n #title_text=\"Subplots with Annotations\"\n margin = dict(\n l = 0,\n r = 0,\n b = 0,\n t = 30,\n pad = 0)\n ) \nfig.show()\ndel train, train_daily,train_single, asset_details\ngc.collect()\nplt.close()","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-11-16T08:32:53.172558Z","iopub.status.busy":"2021-11-16T08:32:53.171458Z","iopub.status.idle":"2021-11-16T08:34:16.626911Z","shell.execute_reply":"2021-11-16T08:34:16.627477Z"},"papermill":{"duration":83.51092,"end_time":"2021-11-16T08:34:16.627686","exception":false,"start_time":"2021-11-16T08:32:53.116766","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.execute_input":"2021-11-16T08:34:16.792586Z","iopub.status.busy":"2021-11-16T08:34:16.791495Z","iopub.status.idle":"2021-11-16T08:34:16.796297Z","shell.execute_reply":"2021-11-16T08:34:16.796813Z","shell.execute_reply.started":"2021-11-15T12:22:32.681506Z"},"papermill":{"duration":0.088608,"end_time":"2021-11-16T08:34:16.797007","exception":false,"start_time":"2021-11-16T08:34:16.708399","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = pd.read_csv(\"../input/g-research-crypto-forecasting/\" + 'train.csv')\nassets = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nassets_names = dict(zip(assets.Asset_ID, assets.Asset_Name))\nassets_order = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv').Asset_ID[:14]\nassets_order = dict((t,i) for i,t in enumerate(assets_order))\nbtc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\")\nbtc_mini = btc.iloc[-200:]\n# fig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\n# fig.show()\ndel btc","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-11-16T08:34:16.976277Z","iopub.status.busy":"2021-11-16T08:34:16.974947Z","iopub.status.idle":"2021-11-16T08:35:00.185727Z","shell.execute_reply":"2021-11-16T08:35:00.184972Z"},"papermill":{"duration":43.309053,"end_time":"2021-11-16T08:35:00.185889","exception":false,"start_time":"2021-11-16T08:34:16.876836","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = crypto_df.copy().set_index(\"timestamp\")\ntrain['assets'] = 1\ntrain['assets'] = train.groupby(by = train.index)['assets'].sum()\ntrain['asset_name'] = train.Asset_ID.map(assets_names)\ntrain['asset_name'].value_counts()\nall_same_time = train[train['assets'] == 14][['Asset_ID', 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'asset_name','VWAP','Target']]\ncorr_target = all_same_time.reset_index().pivot(index = 'asset_name', columns = 'timestamp')['Target'].transpose().corr()\ncorr_matrix = all_same_time.reset_index().drop(['Target', 'Asset_ID'], axis=1).pivot(index = 'asset_name', columns = 'timestamp').transpose().corr()\nfig, ax = plt.subplots(1, 2, figsize = (20, 8))\nsns.heatmap(np.round(corr_target, 2), annot = True, ax = ax[0], square = True)\nsns.heatmap(np.round(corr_matrix, 2), annot = True, ax = ax[1], square = True)\nax[0].title.set_text('Asset Targets correlation')\nax[1].title.set_text('Asset Features correlation')","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.execute_input":"2021-11-16T08:35:00.353461Z","iopub.status.busy":"2021-11-16T08:35:00.350632Z","iopub.status.idle":"2021-11-16T08:35:58.91999Z","shell.execute_reply":"2021-11-16T08:35:58.920667Z"},"papermill":{"duration":58.6563,"end_time":"2021-11-16T08:35:58.921091","exception":false,"start_time":"2021-11-16T08:35:00.264791","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try: plt.close()\nexcept: pass\nfig = make_subplots(rows=7, cols=2,shared_xaxes=True, vertical_spacing=0.03, subplot_titles=tuple([assets_names[i] for i in range(14)]))\ndata = all_same_time[1000:2400]\ndata['time'] = [pd.to_datetime(x, unit='s') for x in data.index]\nfor i in range(14): \n coin = data[data.Asset_ID == i]\n name = assets_names[i]\n fig.add_trace(go.Scatter(x=coin['time'], y=coin['VWAP'], name = name + ', VWAP'),row=i//2+1, col= i%2 +1)\nfig.update_layout(height=1000, title_text=' Weighted average prices')\nfig.show()\ndel data, train, crypto_df, all_same_time, corr_target","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.execute_input":"2021-11-16T08:35:59.109762Z","iopub.status.busy":"2021-11-16T08:35:59.103722Z","iopub.status.idle":"2021-11-16T08:35:59.788058Z","shell.execute_reply":"2021-11-16T08:35:59.787447Z"},"papermill":{"duration":0.779451,"end_time":"2021-11-16T08:35:59.788228","exception":false,"start_time":"2021-11-16T08:35:59.008777","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PurgedGroupTimeSeries CV with Extra Data\nThis is a simple starter notebook for Kaggle's Crypto Comp showing purged group timeseries KFold with extra data. Purged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n\n**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validaiton experiments but in that case note that the overall CV score is meaningless because LB will be much different when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords","metadata":{"papermill":{"duration":0.029802,"end_time":"2021-11-07T07:17:16.008545","exception":false,"start_time":"2021-11-07T07:17:15.978743","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Libraries ๐\n\n\n#### Code starts here โฌ","metadata":{"papermill":{"duration":0.02168,"end_time":"2021-11-07T07:18:11.840749","exception":false,"start_time":"2021-11-07T07:18:11.819069","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport traceback\nimport gresearch_crypto\nimport tensorflow as tf\nfrom scipy.stats import pearsonr\nimport pandas as pd, numpy as np\nfrom sklearn.metrics import r2_score\nimport tensorflow.keras.backend as K\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error","metadata":{"execution":{"iopub.execute_input":"2021-11-07T07:18:11.889543Z","iopub.status.busy":"2021-11-07T07:18:11.888737Z","iopub.status.idle":"2021-11-07T07:18:18.452632Z","shell.execute_reply":"2021-11-07T07:18:18.452067Z","shell.execute_reply.started":"2021-11-07T07:09:12.547302Z"},"papermill":{"duration":6.590131,"end_time":"2021-11-07T07:18:18.452796","exception":false,"start_time":"2021-11-07T07:18:11.862665","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration ๐๏ธ\n\n\nIn order to be a proper cross validation with a meaningful overall CV score, **you need to choose the same** `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP`, and `DEPTH_NETS`, `WIDTH_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds.\n* DEVICE - is GPU or TPU\n* SEED - a different seed produces a different triple stratified kfold split.\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* INC2021 - This controls whether to include the extra historical prices during 2021.\n* INC2020 - This controls whether to include the extra historical prices during 2020.\n* INC2019 - This controls whether to include the extra historical prices during 2019.\n* INC2018 - This controls whether to include the extra historical prices during 2018.\n* INC2017 - This controls whether to include the extra historical prices during 2017.\n* INCCOMP - This controls whether to include the original data of the competition.\n* INCSUPP - This controls whether to include the supplemented train data that was released with the competition.\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* DEPTH_NETS - is a list of length FOLDS. These are the Network Depths to use each fold. The number refers to the number of layers. So a number of `1` refers to 1 layer, and `2` refers to 2 layers, etc.\n* WIDTH_NETS - is a list of length FOLDS. These are the Network Widths to use each fold. The number refers to the number of units per layer. So a number of `32` refers to 32 per layer, and `643` refers to 64 layers, etc.","metadata":{"papermill":{"duration":0.021494,"end_time":"2021-11-07T07:18:18.496727","exception":false,"start_time":"2021-11-07T07:18:18.475233","status":"completed"},"tags":[]}},{"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\nSEED = 42\n\n# NUMBER OF FOLDS. \nFOLDS = 5\n# GAP FROM SPLIT TO SPLIT [SEE DETAILS ABOUT THE CV]\nGROUP_GAP = 31\n\n# WHICH YEARS TO INCLUDE? YES=1 NO=0\nINC2021 = 0\nINC2020 = 0\nINC2019 = 0\nINC2018 = 0\nINC2017 = 0\nINCCOMP = 1\nINCSUPP = 0\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [2048] * FOLDS\nEPOCHS = [1] * FOLDS\n\n# WHICH NETWORK ARCHITECTURE TO USE?\nDEPTH_NETS = [2, 2, 2, 2, 2] \nWIDTH_NETS = [64, 64, 64, 64, 64]\n","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2021-11-07T07:18:18.546463Z","iopub.status.busy":"2021-11-07T07:18:18.545401Z","iopub.status.idle":"2021-11-07T07:18:18.548924Z","shell.execute_reply":"2021-11-07T07:18:18.548198Z","shell.execute_reply.started":"2021-11-07T07:09:20.395963Z"},"papermill":{"duration":0.031723,"end_time":"2021-11-07T07:18:18.549107","exception":false,"start_time":"2021-11-07T07:18:18.517384","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n print(\"connecting to TPU...\")\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\n except ValueError:\n tpu = None\n if tpu:\n try:\n print(\"initializing TPU ...\")\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"TPU initialized\")\n except: print(\"failed to initialize TPU\")\n else: DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\": strategy = tf.distribute.get_strategy()\nif DEVICE == \"GPU\": print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync","metadata":{"execution":{"iopub.execute_input":"2021-11-07T07:18:18.599966Z","iopub.status.busy":"2021-11-07T07:18:18.599134Z","iopub.status.idle":"2021-11-07T07:18:18.822583Z","shell.execute_reply":"2021-11-07T07:18:18.823066Z","shell.execute_reply.started":"2021-11-07T07:09:20.409591Z"},"papermill":{"duration":0.252607,"end_time":"2021-11-07T07:18:18.823264","exception":false,"start_time":"2021-11-07T07:18:18.570657","status":"completed"},"tags":[],"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loading ๐๏ธ\n\n\nThe data organisation has already been done and saved to Kaggle datasets. Here we choose which years to load. We can use either 2017, 2018, 2019, 2020, 2021, Original, Supplement by changing the `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` variables in the preceeding code section. These datasets are discussed [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726\n","metadata":{"papermill":{"duration":0.021426,"end_time":"2021-11-07T07:18:18.866485","exception":false,"start_time":"2021-11-07T07:18:18.845059","status":"completed"},"tags":[]}},{"cell_type":"code","source":"orig_df_train = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')\nsupp_df_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')\ndf_asset_details = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/asset_details.csv').sort_values(\"Asset_ID\")\n\nextra_data_files = {\n 0: '../input/cryptocurrency-extra-data-binance-coin',\n 2: '../input/cryptocurrency-extra-data-bitcoin-cash',\n 1: '../input/cryptocurrency-extra-data-bitcoin',\n 3: '../input/cryptocurrency-extra-data-cardano',\n 4: '../input/cryptocurrency-extra-data-dogecoin', \n 5: '../input/cryptocurrency-extra-data-eos-io',\n 6: '../input/cryptocurrency-extra-data-ethereum',\n 7: '../input/cryptocurrency-extra-data-ethereum-classic', \n 8: '../input/cryptocurrency-extra-data-iota',\n 9: '../input/cryptocurrency-extra-data-litecoin',\n 11: '../input/cryptocurrency-extra-data-monero',\n 10: '../input/cryptocurrency-extra-data-maker',\n 12: '../input/cryptocurrency-extra-data-stellar',\n 13: '../input/cryptocurrency-extra-data-tron'\n }\n\ndef load_training_data_for_asset(asset_id):\n dfs = [] \n if INCCOMP: dfs.append(orig_df_train[orig_df_train[\"Asset_ID\"] == asset_id].copy())\n if INCSUPP: dfs.append(supp_df_train[supp_df_train[\"Asset_ID\"] == asset_id].copy()) \n if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'))\n if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'))\n if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'))\n if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'))\n if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'))\n df = pd.concat(dfs, axis = 0) if len(dfs) > 1 else dfs[0]\n df['date'] = pd.to_datetime(df['timestamp'], unit = 's') \n df = df.sort_values('date')\n return df\n","metadata":{"execution":{"iopub.execute_input":"2021-11-07T07:18:18.927534Z","iopub.status.busy":"2021-11-07T07:18:18.926646Z","iopub.status.idle":"2021-11-07T07:18:55.897894Z","shell.execute_reply":"2021-11-07T07:18:55.897048Z","shell.execute_reply.started":"2021-11-07T07:09:20.451019Z"},"papermill":{"duration":37.009077,"end_time":"2021-11-07T07:18:55.898076","exception":false,"start_time":"2021-11-07T07:18:18.888999","status":"completed"},"tags":[],"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering ๐ฌ\n\n\nThis notebook uses upper_shadow, lower_shadow, high_div_low, open_sub_close, seasonality/datetime features first shown in this notebook [here][1] and successfully used by julian3833 [here][2].\n\nAdditionally we can decide to use external data by changing the variables `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` in the preceeding code section. These variables respectively indicate whether to load last year 2021 data and/or year 2020, 2019, 2018, 2017, the original, supplemented data. These datasets are discussed [here][3]\n\nConsider experimenting with different feature engineering and/or external data. The code to extract features out of the dataset is taken from julian3833' notebook [here][2]. Thank you julian3833, this is great work.\n\n[1]: https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\n[2]: https://www.kaggle.com/julian3833\n[3]: TBD","metadata":{"papermill":{"duration":0.02391,"end_time":"2021-11-07T07:18:55.946856","exception":false,"start_time":"2021-11-07T07:18:55.922946","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Two features from the competition tutorial\ndef upper_shadow(df): return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df): return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# A utility function to build features from the original df\ndef get_features(df):\n df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n df_feat['upper_Shadow'] = upper_shadow(df_feat)\n df_feat['lower_Shadow'] = lower_shadow(df_feat)\n df_feat[\"high_div_low\"] = df_feat[\"High\"] / df_feat[\"Low\"]\n df_feat[\"open_sub_close\"] = df_feat[\"Open\"] - df_feat[\"Close\"]\n return df_feat","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configure the model โ๏ธ\n\n\nThis is a simple model with simple set of hyperparameters. Consider experimenting with different models, parameters, ensembles and so on.","metadata":{"papermill":{"duration":0.022895,"end_time":"2021-11-07T07:18:56.050623","exception":false,"start_time":"2021-11-07T07:18:56.027728","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def build_model(fold, dim = 128):\n inp = tf.keras.layers.Input(shape=(dim))\n x = inp\n \n for i in range(DEPTH_NETS[fold]):\n x = tf.keras.layers.Dense(WIDTH_NETS[fold])(x)\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Activation('swish')(x)\n\n x = tf.keras.layers.Dense(1, activation='linear')(x)\n model = tf.keras.Model(inputs = inp, outputs = x)\n opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n model.compile(optimizer=opt, loss='mse')\n return model","metadata":{"execution":{"iopub.execute_input":"2021-11-07T07:18:56.104642Z","iopub.status.busy":"2021-11-07T07:18:56.103798Z","iopub.status.idle":"2021-11-07T07:18:56.107128Z","shell.execute_reply":"2021-11-07T07:18:56.107709Z","shell.execute_reply.started":"2021-11-07T07:10:04.516429Z"},"papermill":{"duration":0.033877,"end_time":"2021-11-07T07:18:56.107869","exception":false,"start_time":"2021-11-07T07:18:56.073992","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LR Scheduler โฑ๏ธ\n\n\nThis is a common train schedule in kaggle competitions. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow.","metadata":{"papermill":{"duration":0.023542,"end_time":"2021-11-07T07:18:56.155272","exception":false,"start_time":"2021-11-07T07:18:56.13173","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_lr_callback(batch_size = 8):\n lr_start = 0.000005\n lr_max = 0.00000125 * REPLICAS * batch_size\n lr_min = 0.000001\n lr_ramp_ep = 5\n lr_sus_ep = 0\n lr_decay = 0.8\n def lrfn(epoch):\n if epoch < lr_ramp_ep: lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n elif epoch < lr_ramp_ep + lr_sus_ep: lr = lr_max\n else: lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n return lr\n lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n return lr_callback","metadata":{"execution":{"iopub.execute_input":"2021-11-07T07:18:56.207085Z","iopub.status.busy":"2021-11-07T07:18:56.206086Z","iopub.status.idle":"2021-11-07T07:18:56.209744Z","shell.execute_reply":"2021-11-07T07:18:56.209242Z","shell.execute_reply.started":"2021-11-07T07:10:04.532853Z"},"papermill":{"duration":0.031771,"end_time":"2021-11-07T07:18:56.209896","exception":false,"start_time":"2021-11-07T07:18:56.178125","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Time Series Cross Validation\n\n> \"There are many different ways one can do cross-validation, and **it is the most critical step when building a good machine learning model** which is generalizable when it comes to unseen data.\"\n-- **Approaching (Almost) Any Machine Learning Problem**, by Abhishek Thakur\n\nCV is the **first** step, but very few notebooks are talking about this. Here we look at \"purged rolling time series CV\" and actually apply it in hyperparameter tuning for a basic estimator. This notebook owes a debt of gratitude to the notebook [\"Found the Holy Grail GroupTimeSeriesSplit\"](https://www.kaggle.com/jorijnsmit/found-the-holy-grail-grouptimeseriessplit). That notebook is excellent and this solution is an extention of the quoted pending sklearn estimator. I modify that estimator to make it more suitable for the task at hand in this competition. The changes are\n\n- you can specify a **gap** between each train and validation split. This is important because even though the **group** aspect keeps whole days together, we suspect that the anonymized features have some kind of lag or window calculations in them (which would be standard for financial features). By introducing a gap, we mitigate the risk that we leak information from train into validation\n- we can specify the size of the train and validation splits in terms of **number of days**. The ability to specify a validation set size is new and the the ability to specify days, as opposed to samples, is new.\n\nThe code for `PurgedTimeSeriesSplit` is below. I've hiden it becaused it is really meant to act as an imported class. If you want to see the code and copy for your work, click on the \"Code\" box.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass GroupTimeSeriesSplit(_BaseKFold):\n \"\"\"Time Series cross-validator variant with non-overlapping groups.\n Provides train/test indices to split time series data samples\n that are observed at fixed time intervals according to a\n third-party provided group.\n In each split, test indices must be higher than before, and thus shuffling\n in cross validator is inappropriate.\n This cross-validation object is a variation of :class:`KFold`.\n In the kth split, it returns first k folds as train set and the\n (k+1)th fold as test set.\n The same group will not appear in two different folds (the number of\n distinct groups has to be at least equal to the number of folds).\n Note that unlike standard cross-validation methods, successive\n training sets are supersets of those that come before them.\n Read more in the :ref:`User Guide `.\n Parameters\n ----------\n n_splits : int, default=5\n Number of splits. Must be at least 2.\n max_train_size : int, default=None\n Maximum size for a single training set.\n Examples\n --------\n >>> import numpy as np\n >>> from sklearn.model_selection import GroupTimeSeriesSplit\n >>> groups = np.array(['a', 'a', 'a', 'a', 'a', 'a',\\\n 'b', 'b', 'b', 'b', 'b',\\\n 'c', 'c', 'c', 'c',\\\n 'd', 'd', 'd'])\n >>> gtss = GroupTimeSeriesSplit(n_splits=3)\n >>> for train_idx, test_idx in gtss.split(groups, groups=groups):\n ... print(\"TRAIN:\", train_idx, \"TEST:\", test_idx)\n ... print(\"TRAIN GROUP:\", groups[train_idx],\\\n \"TEST GROUP:\", groups[test_idx])\n TRAIN: [0, 1, 2, 3, 4, 5] TEST: [6, 7, 8, 9, 10]\n TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a']\\\n TEST GROUP: ['b' 'b' 'b' 'b' 'b']\n TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] TEST: [11, 12, 13, 14]\n TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b']\\\n TEST GROUP: ['c' 'c' 'c' 'c']\n TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\\\n TEST: [15, 16, 17]\n TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b' 'c' 'c' 'c' 'c']\\\n TEST GROUP: ['d' 'd' 'd']\n \"\"\"\n @_deprecate_positional_args\n def __init__(self,\n n_splits=5,\n *,\n max_train_size=None\n ):\n super().__init__(n_splits, shuffle=False, random_state=None)\n self.max_train_size = max_train_size\n\n def split(self, X, y=None, groups=None):\n \"\"\"Generate indices to split data into training and test set.\n Parameters\n ----------\n X : array-like of shape (n_samples, n_features)\n Training data, where n_samples is the number of samples\n and n_features is the number of features.\n y : array-like of shape (n_samples,)\n Always ignored, exists for compatibility.\n groups : array-like of shape (n_samples,)\n Group labels for the samples used while splitting the dataset into\n train/test set.\n Yields\n ------\n train : ndarray\n The training set indices for that split.\n test : ndarray\n The testing set indices for that split.\n \"\"\"\n if groups is None:\n raise ValueError(\n \"The 'groups' parameter should not be None\")\n X, y, groups = indexable(X, y, groups)\n n_samples = _num_samples(X)\n n_splits = self.n_splits\n n_folds = n_splits + 1\n group_dict = {}\n u, ind = np.unique(groups, return_index=True)\n unique_groups = u[np.argsort(ind)]\n n_samples = _num_samples(X)\n n_groups = _num_samples(unique_groups)\n for idx in np.arange(n_samples):\n if (groups[idx] in group_dict):\n group_dict[groups[idx]].append(idx)\n else:\n group_dict[groups[idx]] = [idx]\n if n_folds > n_groups:\n raise ValueError(\n (\"Cannot have number of folds={0} greater than\"\n \" the number of groups={1}\").format(n_folds,\n n_groups))\n group_test_size = n_groups // n_folds\n group_test_starts = range(n_groups - n_splits * group_test_size,\n n_groups, group_test_size)\n for group_test_start in group_test_starts:\n train_array = []\n test_array = []\n for train_group_idx in unique_groups[:group_test_start]:\n train_array_tmp = group_dict[train_group_idx]\n train_array = np.sort(np.unique(\n np.concatenate((train_array,\n train_array_tmp)),\n axis=None), axis=None)\n train_end = train_array.size\n if self.max_train_size and self.max_train_size < train_end:\n train_array = train_array[train_end -\n self.max_train_size:train_end]\n for test_group_idx in unique_groups[group_test_start:\n group_test_start +\n group_test_size]:\n test_array_tmp = group_dict[test_group_idx]\n test_array = np.sort(np.unique(\n np.concatenate((test_array,\n test_array_tmp)),\n axis=None), axis=None)\n yield [int(i) for i in train_array], [int(i) for i in test_array]\nimport numpy as np\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# modified code for group gaps; source\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass PurgedGroupTimeSeriesSplit(_BaseKFold):\n \"\"\"Time Series cross-validator variant with non-overlapping groups.\n Allows for a gap in groups to avoid potentially leaking info from\n train into test if the model has windowed or lag features.\n Provides train/test indices to split time series data samples\n that are observed at fixed time intervals according to a\n third-party provided group.\n In each split, test indices must be higher than before, and thus shuffling\n in cross validator is inappropriate.\n This cross-validation object is a variation of :class:`KFold`.\n In the kth split, it returns first k folds as train set and the\n (k+1)th fold as test set.\n The same group will not appear in two different folds (the number of\n distinct groups has to be at least equal to the number of folds).\n Note that unlike standard cross-validation methods, successive\n training sets are supersets of those that come before them.\n Read more in the :ref:`User Guide `.\n Parameters\n ----------\n n_splits : int, default=5\n Number of splits. Must be at least 2.\n max_train_group_size : int, default=Inf\n Maximum group size for a single training set.\n group_gap : int, default=None\n Gap between train and test\n max_test_group_size : int, default=Inf\n We discard this number of groups from the end of each train split\n \"\"\"\n\n @_deprecate_positional_args\n def __init__(self,\n n_splits=5,\n *,\n max_train_group_size=np.inf,\n max_test_group_size=np.inf,\n group_gap=None,\n verbose=False\n ):\n super().__init__(n_splits, shuffle=False, random_state=None)\n self.max_train_group_size = max_train_group_size\n self.group_gap = group_gap\n self.max_test_group_size = max_test_group_size\n self.verbose = verbose\n\n def split(self, X, y=None, groups=None):\n \"\"\"Generate indices to split data into training and test set.\n Parameters\n ----------\n X : array-like of shape (n_samples, n_features)\n Training data, where n_samples is the number of samples\n and n_features is the number of features.\n y : array-like of shape (n_samples,)\n Always ignored, exists for compatibility.\n groups : array-like of shape (n_samples,)\n Group labels for the samples used while splitting the dataset into\n train/test set.\n Yields\n ------\n train : ndarray\n The training set indices for that split.\n test : ndarray\n The testing set indices for that split.\n \"\"\"\n if groups is None:\n raise ValueError(\n \"The 'groups' parameter should not be None\")\n X, y, groups = indexable(X, y, groups)\n n_samples = _num_samples(X)\n n_splits = self.n_splits\n group_gap = self.group_gap\n max_test_group_size = self.max_test_group_size\n max_train_group_size = self.max_train_group_size\n n_folds = n_splits + 1\n group_dict = {}\n u, ind = np.unique(groups, return_index=True)\n unique_groups = u[np.argsort(ind)]\n n_samples = _num_samples(X)\n n_groups = _num_samples(unique_groups)\n for idx in np.arange(n_samples):\n if (groups[idx] in group_dict):\n group_dict[groups[idx]].append(idx)\n else:\n group_dict[groups[idx]] = [idx]\n if n_folds > n_groups:\n raise ValueError(\n (\"Cannot have number of folds={0} greater than\"\n \" the number of groups={1}\").format(n_folds,\n n_groups))\n\n group_test_size = min(n_groups // n_folds, max_test_group_size)\n group_test_starts = range(n_groups - n_splits * group_test_size,\n n_groups, group_test_size)\n for group_test_start in group_test_starts:\n train_array = []\n test_array = []\n\n group_st = max(0, group_test_start - group_gap - max_train_group_size)\n for train_group_idx in unique_groups[group_st:(group_test_start - group_gap)]:\n train_array_tmp = group_dict[train_group_idx]\n\n train_array = np.sort(np.unique(\n np.concatenate((train_array,\n train_array_tmp)),\n axis=None), axis=None)\n\n train_end = train_array.size\n\n for test_group_idx in unique_groups[group_test_start:\n group_test_start +\n group_test_size]:\n test_array_tmp = group_dict[test_group_idx]\n test_array = np.sort(np.unique(\n np.concatenate((test_array,\n test_array_tmp)),\n axis=None), axis=None)\n\n test_array = test_array[group_gap:]\n\n\n if self.verbose > 0:\n pass\n\n yield [int(i) for i in train_array], [int(i) for i in test_array]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.execute_input":"2021-11-07T07:18:56.287629Z","iopub.status.busy":"2021-11-07T07:18:56.283181Z","iopub.status.idle":"2021-11-07T07:18:56.290466Z","shell.execute_reply":"2021-11-07T07:18:56.289931Z","shell.execute_reply.started":"2021-11-07T07:10:04.552637Z"},"papermill":{"duration":0.058703,"end_time":"2021-11-07T07:18:56.290589","exception":false,"start_time":"2021-11-07T07:18:56.231886","status":"completed"},"pycharm":{"name":"#%%\n"},"tags":[],"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training ๐๏ธ\n\n\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Each fold the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variable `VERBOSE`. The variable `VERBOSE=1 or 2` will display the training and validation loss for each epoch as text. ","metadata":{"papermill":{"duration":0.021963,"end_time":"2021-11-07T07:18:56.333667","exception":false,"start_time":"2021-11-07T07:18:56.311704","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 1\n\ndef get_Xy_and_model_for_asset(asset_id):\n df = load_training_data_for_asset(asset_id)\n df_proc = get_features(df)\n df_proc['date'] = df['date'].copy()\n df_proc['y'] = df['Target']\n df_proc = df_proc.dropna(how=\"any\")\n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n groups = pd.factorize(X['date'].dt.day.astype(str) + '_' + X['date'].dt.month.astype(str) + '_' + X['date'].dt.year.astype(str))[0]\n X = X.drop(columns = 'date')\n oof_preds = np.zeros(len(X))\n scores, models = [], []\n \n for fold, (train_idx, val_idx) in enumerate(PurgedGroupTimeSeriesSplit(n_splits = FOLDS, group_gap = GROUP_GAP).split(X, y, groups)):\n # GET TRAINING, VALIDATION SET\n x_train, x_val = X.iloc[train_idx], X.iloc[val_idx]\n y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]\n\n # DISPLAY FOLD INFO\n if DEVICE == 'TPU':\n if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n print('#'*25); print('#### FOLD',fold+1)\n print('#### Training WIDTH %s DEPTH %s | batch_size %s' % (WIDTH_NETS[fold], DEPTH_NETS[fold], BATCH_SIZES[fold]*REPLICAS))\n\n # BUILD MODEL\n K.clear_session()\n with strategy.scope(): model = build_model(fold, dim = x_train.shape[1])\n\n # SAVE BEST MODEL EACH FOLD\n sv = tf.keras.callbacks.ModelCheckpoint('fold-%i.h5' % fold, monitor = 'val_loss', verbose = 0, save_best_only = True, save_weights_only = True, mode = 'min', save_freq = 'epoch')\n\n # TRAIN\n print('Training...')\n history = model.fit( x_train, y_train, epochs = EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], validation_data = (x_val, y_val), verbose=VERBOSE )\n print('Loading best model...')\n model.load_weights('fold-%i.h5' % fold)\n\n # PREDICT OOF\n print('Predicting OOF...')\n pred = model.predict(x_val, verbose = VERBOSE)\n models.append(model)\n\n # REPORT RESULTS\n try: mse = mean_squared_error(np.nan_to_num(y_val), np.nan_to_num(pred))\n except: mse = 0.0\n scores.append(mse)\n oof_preds[val_idx] = pred[:,0]\n print('#### FOLD %i OOF MSE %.3f' % (fold + 1, mse))\n\n df = df_proc\n df['oof_preds'] = np.nan_to_num(oof_preds)\n print('\\n\\n' + ('-' * 80) + '\\n' + 'Finished training %s. \\nResults:\\n')\n print('r2_score:', r2_score(df['y'], df['oof_preds']))\n print('pearsonr:', pearsonr(df['y'], df['oof_preds'])[0])\n print('Model std: %s | Target std: %s' % (df['oof_preds'].std(), df['y'].std()))\n try: plt.close()\n except: pass\n df2 = df.reset_index().set_index('date')\n df2['y'].rolling(3 * 30 * 24 * 60).corr(df2['oof_preds']).iloc[::24 * 60].plot()\n plt.title('3 month rolling pearsonr for {}'.format(str(asset_id)))\n plt.show()\n \n return scores, oof_preds, models\n\nmodels = {}\nscores = {}\noof_preds = {}\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n cur_scores, cur_oof_preds, cur_models = get_Xy_and_model_for_asset(asset_id)\n scores[asset_id], oof_preds[asset_id], models[asset_id] = np.mean(cur_scores), cur_oof_preds, cur_models","metadata":{"execution":{"iopub.execute_input":"2021-11-07T07:18:56.407377Z","iopub.status.busy":"2021-11-07T07:18:56.385473Z","iopub.status.idle":"2021-11-07T10:57:45.622767Z","shell.execute_reply":"2021-11-07T10:57:45.622077Z","shell.execute_reply.started":"2021-11-07T07:10:04.596917Z"},"papermill":{"duration":13129.268152,"end_time":"2021-11-07T10:57:45.622943","exception":false,"start_time":"2021-11-07T07:18:56.354791","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Calculate OOF MSE\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions.","metadata":{"papermill":{"duration":64.121908,"end_time":"2021-11-07T10:59:53.151472","exception":false,"start_time":"2021-11-07T10:58:49.029564","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# COMPUTE OVERALL OOF MSE\nprint('Overall MEAN OOF MSE %s' % np.mean(list(scores.values())))\n\n# SAVE OOF TO DISK \nfor asset in oof_preds:\n df_oof = pd.DataFrame(dict(asset_id = asset, oof_preds=oof_preds[asset]))\n df_oof.to_csv(str(asset) + '_oof.csv',index=False)\n df_oof.head()","metadata":{"execution":{"iopub.execute_input":"2021-11-07T11:02:00.559885Z","iopub.status.busy":"2021-11-07T11:02:00.558917Z","iopub.status.idle":"2021-11-07T11:02:00.588009Z","shell.execute_reply":"2021-11-07T11:02:00.587443Z","shell.execute_reply.started":"2021-11-07T07:10:04.616706Z"},"papermill":{"duration":64.312331,"end_time":"2021-11-07T11:02:00.588152","exception":false,"start_time":"2021-11-07T11:00:56.275821","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit To Kaggle ๐ฐ\n\n","metadata":{"papermill":{"duration":64.080551,"end_time":"2021-11-07T11:04:08.479502","exception":false,"start_time":"2021-11-07T11:03:04.398951","status":"completed"},"tags":[]}},{"cell_type":"code","source":"all_df_test = []\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n try:\n if row['Asset_ID'] in models:\n cur_models = models[row['Asset_ID']]\n x_test = get_features(row)\n y_pred = np.mean(np.concatenate([np.expand_dims(model.predict(x_test.to_frame().T)[0], axis = 0) for model in cur_models], axis = 0), axis = 0)\n else: y_pred = 0.0\n except: \n y_pred = 0.0\n traceback.print_exc()\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n all_df_test.append(df_test)\n env.predict(df_pred)\n","metadata":{"execution":{"iopub.execute_input":"2021-11-07T11:06:16.43461Z","iopub.status.busy":"2021-11-07T11:06:16.433606Z","iopub.status.idle":"2021-11-07T11:06:17.163501Z","shell.execute_reply":"2021-11-07T11:06:17.164292Z","shell.execute_reply.started":"2021-11-07T07:13:26.438195Z"},"papermill":{"duration":64.22934,"end_time":"2021-11-07T11:06:17.164561","exception":false,"start_time":"2021-11-07T11:05:12.935221","status":"completed"},"pycharm":{"name":"#%%\n"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References\n\n1. [Initial baseline notebook](https://www.kaggle.com/julian3833) \n2. [Competition tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition) \n3. [Competition Overview](https://www.kaggle.com/c/g-research-crypto-forecasting/overview) \n4. [My Initial Ideas for this competition](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903) \n5. [My post notebook about cross validation](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit) \n6. [Chris original notebook from SIIM ISIC](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) \n\nWORK IN PROGRESS! ๐ง","metadata":{"papermill":{"duration":63.984601,"end_time":"2021-11-07T11:08:24.372133","exception":false,"start_time":"2021-11-07T11:07:20.387532","status":"completed"},"tags":[]}}]}
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n