","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.02926,"end_time":"2021-11-27T08:16:38.745078","exception":false,"start_time":"2021-11-27T08:16:38.715818","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.029361,"end_time":"2021-11-27T08:16:38.804043","exception":false,"start_time":"2021-11-27T08:16:38.774682","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":">### 1st Place of Jane Street π β Adapted to Crypto π\n>This is a simple starter notebook for Kaggle's Crypto Comp showing purged group timeseries KFold with extra data. Purged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n>\n>**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validaiton experiments but in that case note that the overall CV score is meaningless because LB will be much different when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n>\n>**See Original Links:**\n>- https://www.kaggle.com/gogo827jz/jane-street-supervised-autoencoder-mlp\n>- https://www.kaggle.com/c/jane-street-market-prediction/discussion/224348\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n\n
\n\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | [Quant's Volatility Features](https://www.kaggle.com/yamqwe/crypto-prediction-volatility-features) |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | | β³Target Engineering |\n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |β³Fourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | β³Wavelets | \n\n____","metadata":{"papermill":{"duration":0.030232,"end_time":"2021-11-27T08:16:38.923363","exception":false,"start_time":"2021-11-27T08:16:38.893131","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content π\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.030391,"end_time":"2021-11-27T08:16:38.983127","exception":false,"start_time":"2021-11-27T08:16:38.952736","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [π Introduction](#introduction)\n\n2. [π Table Of Content](#outline) \n\n3. [π€Ώ Diving into the Data](#diving) \n\n4. [π Imports](#imports) \n\n5. [ποΈ Configurations](#config) \n\n6. [ποΈ Data Loading](#loading) \n\n7. [π¬ Feature Engineering](#features) \n\n8. [βοΈ Configure The Model](#modelconf)\n\n9. [β±οΈ LR Scheduler](#sched)\n\n10. [ποΈ Training](#training)\n\n11. [π° Submit to Kaggle](#submit)","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.029357,"end_time":"2021-11-27T08:16:39.041965","exception":false,"start_time":"2021-11-27T08:16:39.012608","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data π€Ώ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.028816,"end_time":"2021-11-27T08:16:39.100503","exception":false,"start_time":"2021-11-27T08:16:39.071687","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - π There are 14 coins in the dataset\n>\n> - π There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.028937,"end_time":"2021-11-27T08:16:39.158735","exception":false,"start_time":"2021-11-27T08:16:39.129798","status":"completed"},"tags":[]}},{"cell_type":"code","source":"css_file = '''\ndiv #notebook {\nbackground-color: white;\nfont-family: 'Open Sans', Helvetica, sans-serif;\nline-height: 20px;\n}\n\n#notebook-container {\nmargin-top: 2em;\npadding-top: 2em;\nborder-top: 4px solid %s; /* light orange */\n-webkit-box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n}\n\ndiv .input {\nmargin-bottom: 1em;\n}\n\n.rendered_html h1, .rendered_html h2, .rendered_html h3, .rendered_html h4, .rendered_html h5, .rendered_html h6 {\ncolor: %s; /* light orange */\nfont-weight: 600;\n}\n\n.rendered_html code {\n background-color: #efefef; /* light gray */\n}\n\n.CodeMirror {\ncolor: #8c8c8c; /* dark gray */\npadding: 0.7em;\n}\n\ndiv.input_area {\nborder: none;\n background-color: %s; /* rgba(229, 143, 101, 0.1); light orange [exactly #E58F65] */\n border-top: 2px solid %s; /* light orange */\n}\n\ndiv.input_prompt {\ncolor: %s; /* light blue */\n}\n\ndiv.output_prompt {\ncolor: %s; /* strong orange */\n}\n\ndiv.cell.selected:before, div.cell.selected.jupyter-soft-selected:before {\nbackground: %s; /* light orange */\n}\n\ndiv.cell.selected, div.cell.selected.jupyter-soft-selected {\n border-color: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected:before {\nbackground: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected {\nborder-color: %s; /* light orange */\n\n}\n'''\ndef to_rgb(h): return tuple(int(h[i:i+2], 16) for i in (0, 2, 4))\nmain_color_rgba = 'rgba(%s, %s, %s, 0.1)' % (to_rgb(main_color[1:])[0], to_rgb(main_color[1:])[1], to_rgb(main_color[1:])[2])\nopen('notebook.css', 'w').write(css_file % (main_color, main_color, main_color_rgba, main_color, prompt, strong_main_color, main_color, main_color, main_color, main_color))\nfrom IPython.core.display import display, HTML, Javascript\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75405d52-594f-4791-a45a-16354bc24cfa","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"775496bb-f633-46f4-a642-10e6cd29c1bb","jupyter":{"outputs_hidden":false},"papermill":{"duration":0.04554,"end_time":"2021-11-27T08:16:39.234056","exception":false,"start_time":"2021-11-27T08:16:39.188516","status":"completed"},"tags":[],"collapsed":false,"execution":{"iopub.status.busy":"2022-01-25T19:20:43.365347Z","iopub.execute_input":"2022-01-25T19:20:43.365656Z","iopub.status.idle":"2022-01-25T19:20:43.379933Z","shell.execute_reply.started":"2022-01-25T19:20:43.36562Z","shell.execute_reply":"2022-01-25T19:20:43.379238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\n# def nb(): return HTML(\"\")\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75d3e80e-0c31-48f5-88dd-5b6a3c79cf38","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"968efaca-ede8-4817-b3aa-baedaba3205e","jupyter":{"outputs_hidden":false},"papermill":{"duration":0.040618,"end_time":"2021-11-27T08:16:39.304489","exception":false,"start_time":"2021-11-27T08:16:39.263871","status":"completed"},"tags":[],"collapsed":false,"execution":{"iopub.status.busy":"2022-01-25T19:20:43.38132Z","iopub.execute_input":"2022-01-25T19:20:43.381517Z","iopub.status.idle":"2022-01-25T19:20:43.391648Z","shell.execute_reply.started":"2022-01-25T19:20:43.381494Z","shell.execute_reply":"2022-01-25T19:20:43.390861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
1st Place of Jane Street π β Adapted to Crypto π
\n \n\nThis is a simple starter notebook for Kaggle's Crypto Comp showing purged group timeseries KFold with extra data. Purged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n\n**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validaiton experiments but in that case note that the overall CV score is meaningless because LB will be much different when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n\n**See Original Links:**\n- https://www.kaggle.com/gogo827jz/jane-street-supervised-autoencoder-mlp\n- https://www.kaggle.com/c/jane-street-market-prediction/discussion/224348\n\n**Encoder + MLP**:\nThe main idea of using an encoder is the denoise the data. After many attempts at using a unsupervised autoencoder, the choice landed on a bottleneck encoder as this will preserve the intra-feature relations.\n\n\n**Main Ideas:**\n\nIn Jane street competition, Yirun have used a supervised autoencoder MLP approach and his teammates use XGBoost. Their final submission is a simple blend of these two models. Here, We explain Yirun's approach in detail.\n\nThe supervised autoencoder approach was initially proposed in [Bottleneck encoder + MLP + Keras Tuner](https://www.kaggle.com/aimind/bottleneck-encoder-mlp-keras-tuner-8601c5) and was adapted to this competition [here](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner), where one supervised autoencoder is trained separately before cross-validation (CV) split. Yirun have realised that this training may cause label leakage because the autoencoder has seen part of the data in the validation set in each CV split and it can generate label-leakage features to overfit. So, his approach was to train the supervised autoencoder along with MLP in one model in each CV split. The training processes and explanations are given in the notebook and the following statements.\n\n**The Model:**\n\n- Use autoencoder to create new features, concatenating with the original features as the input to the downstream MLP model\n- Train autoencoder and MLP together in each CV split to prevent data leakage\n- Add target information to autoencoder (supervised learning) to force it to generate more relevant features, and to create a shortcut for backpropagation of gradient\n- Add Gaussian noise layer before encoder for data augmentation and to prevent overfitting\n- Use swish activation function instead of ReLU to prevent βdead neuronβ and smooth the gradient\n- Batch Normalisation and Dropout are used for MLP\n- Use Keras Tuner to find the optimal hyperparameter set\n\n\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords","metadata":{"papermill":{"duration":0.030175,"end_time":"2021-11-27T08:16:39.364837","exception":false,"start_time":"2021-11-27T08:16:39.334662","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"____\n**Credits:**\nThe following notebook is heavily based on multiple notebook of the past Jane street market prediction competition. If you find it useful, spare some upvotes to the originals. They earned it! \n- \"Jane Street: Supervised Autoencoder MLP\" (JaneStreet 1st π) by Yirun Zhang - https://www.kaggle.com/gogo827jz/jane-street-supervised-autoencoder-mlp\n\n____","metadata":{"papermill":{"duration":0.029987,"end_time":"2021-11-27T08:16:39.425053","exception":false,"start_time":"2021-11-27T08:16:39.395066","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Libraries π\n","metadata":{"_cell_guid":"347324aa-490a-4556-bcc4-36379bdbec7c","_uuid":"03824ce3-fc08-4d19-926d-27edbb773466","papermill":{"duration":0.030949,"end_time":"2021-11-27T08:16:39.486079","exception":false,"start_time":"2021-11-27T08:16:39.45513","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Code starts here β¬","metadata":{"_cell_guid":"f5bdcf7b-aa74-4204-9ff2-f20042b819e0","_uuid":"78d84509-539f-4a94-a546-e228b28ea70a","papermill":{"duration":0.030041,"end_time":"2021-11-27T08:16:39.54644","exception":false,"start_time":"2021-11-27T08:16:39.516399","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport traceback\nimport gresearch_crypto\nimport tensorflow as tf\nfrom scipy.stats import pearsonr\nimport pandas as pd, numpy as np\nfrom sklearn.metrics import r2_score\nimport tensorflow.keras.backend as K\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error","metadata":{"papermill":{"duration":7.079069,"end_time":"2021-11-27T08:16:46.655729","exception":false,"start_time":"2021-11-27T08:16:39.57666","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:20:43.393303Z","iopub.execute_input":"2022-01-25T19:20:43.394682Z","iopub.status.idle":"2022-01-25T19:20:48.640718Z","shell.execute_reply.started":"2022-01-25T19:20:43.394648Z","shell.execute_reply":"2022-01-25T19:20:48.639961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration ποΈ\n","metadata":{"papermill":{"duration":0.030531,"end_time":"2021-11-27T08:16:46.718422","exception":false,"start_time":"2021-11-27T08:16:46.687891","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"In order to be a proper cross validation with a meaningful overall CV score, **you need to choose the same** `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP`, and `DEPTH_NETS`, `WIDTH_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds.\n* DEVICE - is GPU or TPU\n* SEED - a different seed produces a different triple stratified kfold split.\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* LOAD_STRICT - This controls whether to load strict at proposed [here](https://www.kaggle.com/julian3833/proposal-for-a-meaningful-lb-strict-lgbm)\n* INC2021 - This controls whether to include the extra historical prices during 2021.\n* INC2020 - This controls whether to include the extra historical prices during 2020.\n* INC2019 - This controls whether to include the extra historical prices during 2019.\n* INC2018 - This controls whether to include the extra historical prices during 2018.\n* INC2017 - This controls whether to include the extra historical prices during 2017.\n* INCCOMP - This controls whether to include the original data of the competition.\n* INCSUPP - This controls whether to include the supplemented train data that was released with the competition.\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* DEPTH_NETS - is a list of length FOLDS. These are the Network Depths to use each fold. The number refers to the number of layers. So a number of `1` refers to 1 layer, and `2` refers to 2 layers, etc.\n* WIDTH_NETS - is a list of length FOLDS. These are the Network Widths to use each fold. The number refers to the number of units per layer. So a number of `32` refers to 32 per layer, and `643` refers to 64 layers, etc.","metadata":{"papermill":{"duration":0.030208,"end_time":"2021-11-27T08:16:46.779977","exception":false,"start_time":"2021-11-27T08:16:46.749769","status":"completed"},"tags":[]}},{"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\nSEED = 42\n\n# CV PARAMS\nFOLDS = 5\nGROUP_GAP = 130\nMAX_TEST_GROUP_SIZE = 180\nMAX_TRAIN_GROUP_SIZE = 280\n\n# LOAD STRICT? YES=1 NO=0 | see: https://www.kaggle.com/julian3833/proposal-for-a-meaningful-lb-strict-lgbm\nLOAD_STRICT = True\n\n# WHICH YEARS TO INCLUDE? YES=1 NO=0\nINC2021 = 0\nINC2020 = 0\nINC2019 = 0\nINC2018 = 0\nINC2017 = 0\nINCCOMP = 1\nINCSUPP = 0\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [4096] * FOLDS\nEPOCHS = [1] * FOLDS\n\n# WHICH NETWORK ARCHITECTURE TO USE?\nDEPTH_NETS = [2, 2, 2, 2, 2] \nWIDTH_NETS = [256, 128, 64, 64, 32]\n","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":0.041476,"end_time":"2021-11-27T08:16:46.851866","exception":false,"start_time":"2021-11-27T08:16:46.81039","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:20:48.643163Z","iopub.execute_input":"2022-01-25T19:20:48.643422Z","iopub.status.idle":"2022-01-25T19:20:48.652116Z","shell.execute_reply.started":"2022-01-25T19:20:48.643386Z","shell.execute_reply":"2022-01-25T19:20:48.650622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n print(\"connecting to TPU...\")\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\n except ValueError:\n tpu = None\n if tpu:\n try:\n print(\"initializing TPU ...\")\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"TPU initialized\")\n except: print(\"failed to initialize TPU\")\n else: DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\": strategy = tf.distribute.get_strategy()\nif DEVICE == \"GPU\": print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.061279,"end_time":"2021-11-27T08:16:46.943859","exception":false,"start_time":"2021-11-27T08:16:46.88258","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:20:48.654347Z","iopub.execute_input":"2022-01-25T19:20:48.654681Z","iopub.status.idle":"2022-01-25T19:20:48.86434Z","shell.execute_reply.started":"2022-01-25T19:20:48.654645Z","shell.execute_reply":"2022-01-25T19:20:48.86352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loading ποΈ\n\n\nThe data organisation has already been done and saved to Kaggle datasets. Here we choose which years to load. We can use either 2017, 2018, 2019, 2020, 2021, Original, Supplement by changing the `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` variables in the preceeding code section. These datasets are discussed [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726\n","metadata":{"papermill":{"duration":0.030375,"end_time":"2021-11-27T08:16:47.005222","exception":false,"start_time":"2021-11-27T08:16:46.974847","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import datatable as dt\nextra_data_files = {0: '../input/cryptocurrency-extra-data-binance-coin', 2: '../input/cryptocurrency-extra-data-bitcoin-cash', 1: '../input/cryptocurrency-extra-data-bitcoin', 3: '../input/cryptocurrency-extra-data-cardano', 4: '../input/cryptocurrency-extra-data-dogecoin', 5: '../input/cryptocurrency-extra-data-eos-io', 6: '../input/cryptocurrency-extra-data-ethereum', 7: '../input/cryptocurrency-extra-data-ethereum-classic', 8: '../input/cryptocurrency-extra-data-iota', 9: '../input/cryptocurrency-extra-data-litecoin', 11: '../input/cryptocurrency-extra-data-monero', 10: '../input/cryptocurrency-extra-data-maker', 12: '../input/cryptocurrency-extra-data-stellar', 13: '../input/cryptocurrency-extra-data-tron'}\n\n# Uncomment to load the original csv [slower]\n# orig_df_train = pd.read_csv(data_path + 'train.csv') \n# supp_df_train = pd.read_csv(data_path + 'supplemental_train.csv')\n# df_asset_details = pd.read_csv(data_path + 'asset_details.csv').sort_values(\"Asset_ID\")\n\norig_df_train = dt.fread('../input/cryptocurrency-extra-data-binance-coin/orig_train.jay').to_pandas()\ndf_asset_details = dt.fread('../input/cryptocurrency-extra-data-binance-coin/orig_asset_details.jay').to_pandas()\nsupp_df_train = dt.fread('../input/cryptocurrency-extra-data-binance-coin/orig_supplemental_train.jay').to_pandas()\nassets_details = dt.fread('../input/cryptocurrency-extra-data-binance-coin/orig_asset_details.jay').to_pandas()\nasset_weight_dict = {assets_details['Asset_ID'].tolist()[idx]: assets_details['Weight'].tolist()[idx] for idx in range(len(assets_details))}\nasset_name_dict = {assets_details['Asset_ID'].tolist()[idx]: assets_details['Asset_Name'].tolist()[idx] for idx in range(len(assets_details))}\n\ndef load_training_data_for_asset(asset_id, load_jay = True):\n dfs = []\n if INCCOMP: dfs.append(orig_df_train[orig_df_train[\"Asset_ID\"] == asset_id].copy())\n if INCSUPP: dfs.append(supp_df_train[supp_df_train[\"Asset_ID\"] == asset_id].copy())\n \n if load_jay:\n if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(dt.fread(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.jay').to_pandas())\n if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(dt.fread(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.jay').to_pandas())\n if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(dt.fread(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.jay').to_pandas())\n if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(dt.fread(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.jay').to_pandas())\n if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(dt.fread(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.jay').to_pandas())\n else: \n if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'))\n if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'))\n if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'))\n if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'))\n if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'))\n df = pd.concat(dfs, axis = 0) if len(dfs) > 1 else dfs[0]\n df['date'] = pd.to_datetime(df['timestamp'], unit = 's')\n if LOAD_STRICT: df = df.loc[df['date'] < \"2021-06-13 00:00:00\"] \n df = df.sort_values('date')\n return df\n\ndef load_data_for_all_assets():\n dfs = []\n for asset_id in list(extra_data_files.keys()): dfs.append(load_training_data_for_asset(asset_id))\n return pd.concat(dfs)","metadata":{"_kg_hide-input":true,"papermill":{"duration":25.330417,"end_time":"2021-11-27T08:17:12.367164","exception":false,"start_time":"2021-11-27T08:16:47.036747","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:20:48.867178Z","iopub.execute_input":"2022-01-25T19:20:48.867755Z","iopub.status.idle":"2022-01-25T19:21:15.169716Z","shell.execute_reply.started":"2022-01-25T19:20:48.867718Z","shell.execute_reply":"2022-01-25T19:21:15.168989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering π¬\n\n\nThis notebook uses upper_shadow, lower_shadow, high_div_low, open_sub_close, seasonality/datetime features first shown in this notebook [here][1] and successfully used by julian3833 [here][2].\n\nAdditionally we can decide to use external data by changing the variables `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` in the preceeding code section. These variables respectively indicate whether to load last year 2021 data and/or year 2020, 2019, 2018, 2017, the original, supplemented data. These datasets are discussed [here][3]\n\nConsider experimenting with different feature engineering and/or external data. The code to extract features out of the dataset is taken from julian3833' notebook [here][2]. Thank you julian3833, this is great work.\n\n[1]: https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\n[2]: https://www.kaggle.com/julian3833\n[3]: TBD","metadata":{"papermill":{"duration":0.030782,"end_time":"2021-11-27T08:17:12.429606","exception":false,"start_time":"2021-11-27T08:17:12.398824","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def upper_shadow(df): return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df): return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# Credit: https://www.kaggle.com/swaralipibose/64-new-features-with-autoencoders/code\ndef get_features(df, row = False):\n df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n df_feat['upper_Shadow'] = upper_shadow(df_feat)\n df_feat['lower_Shadow'] = lower_shadow(df_feat)\n df_feat[\"high_div_low\"] = df_feat[\"High\"] / df_feat[\"Low\"]\n df_feat['trade'] = df_feat['Close'] - df_feat['Open']\n df_feat['gtrade'] = df_feat['trade'] / df_feat['Count']\n df_feat['shadow1'] = df_feat['trade'] / df_feat['Volume']\n df_feat['shadow3'] = df_feat['upper_Shadow'] / df_feat['Volume']\n df_feat['shadow5'] = df_feat['lower_Shadow'] / df_feat['Volume']\n df_feat['diff1'] = df_feat['Volume'] - df_feat['Count']\n df_feat['mean1'] = (df_feat['shadow5'] + df_feat['shadow3']) / 2\n df_feat['mean2'] = (df_feat['shadow1'] + df_feat['Volume']) / 2\n df_feat['mean3'] = (df_feat['trade'] + df_feat['gtrade']) / 2\n df_feat['mean4'] = (df_feat['diff1'] + df_feat['upper_Shadow']) / 2\n df_feat['mean5'] = (df_feat['diff1'] + df_feat['lower_Shadow']) / 2\n return df_feat","metadata":{"papermill":{"duration":0.047873,"end_time":"2021-11-27T08:17:12.509409","exception":false,"start_time":"2021-11-27T08:17:12.461536","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:21:15.171103Z","iopub.execute_input":"2022-01-25T19:21:15.171345Z","iopub.status.idle":"2022-01-25T19:21:15.181332Z","shell.execute_reply.started":"2022-01-25T19:21:15.171311Z","shell.execute_reply":"2022-01-25T19:21:15.180701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the Model βοΈ\n\n\nThis is a common model architecute. Consider experimenting with different backbones, custom heads, losses, and optimizers.","metadata":{"papermill":{"duration":0.03185,"end_time":"2021-11-27T08:17:12.573351","exception":false,"start_time":"2021-11-27T08:17:12.541501","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Competition Metric**","metadata":{"papermill":{"duration":0.031589,"end_time":"2021-11-27T08:17:12.636713","exception":false,"start_time":"2021-11-27T08:17:12.605124","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Numpy Version\ndef corr(a, b, w):\n cov = lambda x, y: np.sum(w * (x - np.average(x, weights=w)) * (y - np.average(y, weights=w))) / np.sum(w)\n return cov(a, b) / np.sqrt(cov(a, a) * cov(b, b))\n\n# TF Version\ndef tf_cov(x, y, w): return (tf.reduce_sum(w * (x - tf.reduce_mean(x * w)) * (y - tf.reduce_mean(y * w))) / tf.reduce_sum(w))\ndef tf_comp_metric(a, b, w): return tf_cov(a, b, w) / tf.sqrt(tf_cov(a, a, w) * tf_cov(b, b, w))\ndef nn_comp_metric(w): \n def wcorr(x, y): return tf_comp_metric(x, y ,w)\n return wcorr","metadata":{"papermill":{"duration":0.045505,"end_time":"2021-11-27T08:17:12.713855","exception":false,"start_time":"2021-11-27T08:17:12.66835","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:21:15.182396Z","iopub.execute_input":"2022-01-25T19:21:15.182775Z","iopub.status.idle":"2022-01-25T19:21:15.205603Z","shell.execute_reply.started":"2022-01-25T19:21:15.182741Z","shell.execute_reply":"2022-01-25T19:21:15.204945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The Model**","metadata":{"papermill":{"duration":0.031629,"end_time":"2021-11-27T08:17:12.777656","exception":false,"start_time":"2021-11-27T08:17:12.746027","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def build_model(fold, dim = 128, weight = 1.0):\n \n n_layers = DEPTH_NETS[fold]\n hidden_units = WIDTH_NETS[fold]\n dropout_rates = 0.002\n \n inp = tf.keras.layers.Input(shape = (dim, ))\n x0 = tf.keras.layers.BatchNormalization()(inp)\n\n encoder = tf.keras.layers.GaussianNoise(dropout_rates)(x0)\n encoder = tf.keras.layers.Dense(hidden_units)(encoder)\n encoder = tf.keras.layers.BatchNormalization()(encoder)\n encoder = tf.keras.layers.Activation('swish')(encoder)\n\n decoder = tf.keras.layers.Dropout(dropout_rates)(encoder)\n decoder = tf.keras.layers.Dense(dim, name = 'decoder')(decoder)\n\n x_ae = tf.keras.layers.Dense(hidden_units)(decoder)\n x_ae = tf.keras.layers.BatchNormalization()(x_ae)\n x_ae = tf.keras.layers.Activation('swish')(x_ae)\n x_ae = tf.keras.layers.Dropout(dropout_rates)(x_ae)\n\n out_ae = tf.keras.layers.Dense(1, activation = 'linear', name = 'ae_action')(x_ae)\n\n x = tf.keras.layers.Concatenate()([x0, encoder])\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Dropout(dropout_rates)(x)\n\n if n_layers > 2:\n for i in range(2, n_layers):\n x = tf.keras.layers.Dense(hidden_units[i])(x)\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Activation('swish')(x)\n x = tf.keras.layers.Dropout(dropout_rates)(x)\n\n out = tf.keras.layers.Dense(1, activation = 'linear', name = 'action')(x)\n\n model = tf.keras.models.Model(inputs = inp, outputs = [decoder, out_ae, out])\n model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001),\n loss = {'decoder': tf.keras.losses.MeanSquaredError(),\n 'ae_action': tf.keras.losses.MeanSquaredError(),\n 'action': tf.keras.losses.MeanSquaredError(),\n },\n metrics = {'decoder': tf.keras.metrics.MeanAbsoluteError(),\n 'ae_action': tf.keras.metrics.MeanSquaredError(),\n 'action': nn_comp_metric(weight),\n },\n ) \n \n return model","metadata":{"papermill":{"duration":0.052809,"end_time":"2021-11-27T08:17:12.862198","exception":false,"start_time":"2021-11-27T08:17:12.809389","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:21:15.206688Z","iopub.execute_input":"2022-01-25T19:21:15.207213Z","iopub.status.idle":"2022-01-25T19:21:15.22261Z","shell.execute_reply.started":"2022-01-25T19:21:15.207168Z","shell.execute_reply":"2022-01-25T19:21:15.221844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LR Scheduler β±οΈ\n\n\nThis is a common train schedule in kaggle competitions. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow.","metadata":{"papermill":{"duration":0.032247,"end_time":"2021-11-27T08:17:12.92683","exception":false,"start_time":"2021-11-27T08:17:12.894583","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_lr_callback(batch_size = 8):\n lr_start = 0.000005\n lr_max = 0.00000125 * REPLICAS * batch_size\n lr_min = 0.000001\n lr_ramp_ep = 5\n lr_sus_ep = 0\n lr_decay = 0.8\n def lrfn(epoch):\n if epoch < lr_ramp_ep: lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n elif epoch < lr_ramp_ep + lr_sus_ep: lr = lr_max\n else: lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n return lr\n lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n return lr_callback","metadata":{"papermill":{"duration":0.042625,"end_time":"2021-11-27T08:17:13.002811","exception":false,"start_time":"2021-11-27T08:17:12.960186","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:21:15.223755Z","iopub.execute_input":"2022-01-25T19:21:15.223999Z","iopub.status.idle":"2022-01-25T19:21:15.23452Z","shell.execute_reply.started":"2022-01-25T19:21:15.223967Z","shell.execute_reply":"2022-01-25T19:21:15.233775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass GroupTimeSeriesSplit(_BaseKFold):\n \"\"\"Time Series cross-validator variant with non-overlapping groups.\n Provides train/test indices to split time series data samples\n that are observed at fixed time intervals according to a\n third-party provided group.\n In each split, test indices must be higher than before, and thus shuffling\n in cross validator is inappropriate.\n This cross-validation object is a variation of :class:`KFold`.\n In the kth split, it returns first k folds as train set and the\n (k+1)th fold as test set.\n The same group will not appear in two different folds (the number of\n distinct groups has to be at least equal to the number of folds).\n Note that unlike standard cross-validation methods, successive\n training sets are supersets of those that come before them.\n Read more in the :ref:`User Guide `.\n Parameters\n ----------\n n_splits : int, default=5\n Number of splits. Must be at least 2.\n max_train_size : int, default=None\n Maximum size for a single training set.\n Examples\n --------\n >>> import numpy as np\n >>> from sklearn.model_selection import GroupTimeSeriesSplit\n >>> groups = np.array(['a', 'a', 'a', 'a', 'a', 'a',\\\n 'b', 'b', 'b', 'b', 'b',\\\n 'c', 'c', 'c', 'c',\\\n 'd', 'd', 'd'])\n >>> gtss = GroupTimeSeriesSplit(n_splits=3)\n >>> for train_idx, test_idx in gtss.split(groups, groups=groups):\n ... print(\"TRAIN:\", train_idx, \"TEST:\", test_idx)\n ... print(\"TRAIN GROUP:\", groups[train_idx],\\\n \"TEST GROUP:\", groups[test_idx])\n TRAIN: [0, 1, 2, 3, 4, 5] TEST: [6, 7, 8, 9, 10]\n TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a']\\\n TEST GROUP: ['b' 'b' 'b' 'b' 'b']\n TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] TEST: [11, 12, 13, 14]\n TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b']\\\n TEST GROUP: ['c' 'c' 'c' 'c']\n TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\\\n TEST: [15, 16, 17]\n TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b' 'c' 'c' 'c' 'c']\\\n TEST GROUP: ['d' 'd' 'd']\n \"\"\"\n @_deprecate_positional_args\n def __init__(self,\n n_splits=5,\n *,\n max_train_size=None\n ):\n super().__init__(n_splits, shuffle=False, random_state=None)\n self.max_train_size = max_train_size\n\n def split(self, X, y=None, groups=None):\n \"\"\"Generate indices to split data into training and test set.\n Parameters\n ----------\n X : array-like of shape (n_samples, n_features)\n Training data, where n_samples is the number of samples\n and n_features is the number of features.\n y : array-like of shape (n_samples,)\n Always ignored, exists for compatibility.\n groups : array-like of shape (n_samples,)\n Group labels for the samples used while splitting the dataset into\n train/test set.\n Yields\n ------\n train : ndarray\n The training set indices for that split.\n test : ndarray\n The testing set indices for that split.\n \"\"\"\n if groups is None:\n raise ValueError(\n \"The 'groups' parameter should not be None\")\n X, y, groups = indexable(X, y, groups)\n n_samples = _num_samples(X)\n n_splits = self.n_splits\n n_folds = n_splits + 1\n group_dict = {}\n u, ind = np.unique(groups, return_index=True)\n unique_groups = u[np.argsort(ind)]\n n_samples = _num_samples(X)\n n_groups = _num_samples(unique_groups)\n for idx in np.arange(n_samples):\n if (groups[idx] in group_dict):\n group_dict[groups[idx]].append(idx)\n else:\n group_dict[groups[idx]] = [idx]\n if n_folds > n_groups:\n raise ValueError(\n (\"Cannot have number of folds={0} greater than\"\n \" the number of groups={1}\").format(n_folds,\n n_groups))\n group_test_size = n_groups // n_folds\n group_test_starts = range(n_groups - n_splits * group_test_size,\n n_groups, group_test_size)\n for group_test_start in group_test_starts:\n train_array = []\n test_array = []\n for train_group_idx in unique_groups[:group_test_start]:\n train_array_tmp = group_dict[train_group_idx]\n train_array = np.sort(np.unique(\n np.concatenate((train_array,\n train_array_tmp)),\n axis=None), axis=None)\n train_end = train_array.size\n if self.max_train_size and self.max_train_size < train_end:\n train_array = train_array[train_end -\n self.max_train_size:train_end]\n for test_group_idx in unique_groups[group_test_start:\n group_test_start +\n group_test_size]:\n test_array_tmp = group_dict[test_group_idx]\n test_array = np.sort(np.unique(\n np.concatenate((test_array,\n test_array_tmp)),\n axis=None), axis=None)\n yield [int(i) for i in train_array], [int(i) for i in test_array]\nimport numpy as np\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# modified code for group gaps; source\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass PurgedGroupTimeSeriesSplit(_BaseKFold):\n \"\"\"Time Series cross-validator variant with non-overlapping groups.\n Allows for a gap in groups to avoid potentially leaking info from\n train into test if the model has windowed or lag features.\n Provides train/test indices to split time series data samples\n that are observed at fixed time intervals according to a\n third-party provided group.\n In each split, test indices must be higher than before, and thus shuffling\n in cross validator is inappropriate.\n This cross-validation object is a variation of :class:`KFold`.\n In the kth split, it returns first k folds as train set and the\n (k+1)th fold as test set.\n The same group will not appear in two different folds (the number of\n distinct groups has to be at least equal to the number of folds).\n Note that unlike standard cross-validation methods, successive\n training sets are supersets of those that come before them.\n Read more in the :ref:`User Guide `.\n Parameters\n ----------\n n_splits : int, default=5\n Number of splits. Must be at least 2.\n max_train_group_size : int, default=Inf\n Maximum group size for a single training set.\n group_gap : int, default=None\n Gap between train and test\n max_test_group_size : int, default=Inf\n We discard this number of groups from the end of each train split\n \"\"\"\n\n @_deprecate_positional_args\n def __init__(self,\n n_splits=5,\n *,\n max_train_group_size=np.inf,\n max_test_group_size=np.inf,\n group_gap=None,\n verbose=False\n ):\n super().__init__(n_splits, shuffle=False, random_state=None)\n self.max_train_group_size = max_train_group_size\n self.group_gap = group_gap\n self.max_test_group_size = max_test_group_size\n self.verbose = verbose\n\n def split(self, X, y=None, groups=None):\n \"\"\"Generate indices to split data into training and test set.\n Parameters\n ----------\n X : array-like of shape (n_samples, n_features)\n Training data, where n_samples is the number of samples\n and n_features is the number of features.\n y : array-like of shape (n_samples,)\n Always ignored, exists for compatibility.\n groups : array-like of shape (n_samples,)\n Group labels for the samples used while splitting the dataset into\n train/test set.\n Yields\n ------\n train : ndarray\n The training set indices for that split.\n test : ndarray\n The testing set indices for that split.\n \"\"\"\n if groups is None:\n raise ValueError(\n \"The 'groups' parameter should not be None\")\n X, y, groups = indexable(X, y, groups)\n n_samples = _num_samples(X)\n n_splits = self.n_splits\n group_gap = self.group_gap\n max_test_group_size = self.max_test_group_size\n max_train_group_size = self.max_train_group_size\n n_folds = n_splits + 1\n group_dict = {}\n u, ind = np.unique(groups, return_index=True)\n unique_groups = u[np.argsort(ind)]\n n_samples = _num_samples(X)\n n_groups = _num_samples(unique_groups)\n for idx in np.arange(n_samples):\n if (groups[idx] in group_dict):\n group_dict[groups[idx]].append(idx)\n else:\n group_dict[groups[idx]] = [idx]\n if n_folds > n_groups:\n raise ValueError(\n (\"Cannot have number of folds={0} greater than\"\n \" the number of groups={1}\").format(n_folds,\n n_groups))\n\n group_test_size = min(n_groups // n_folds, max_test_group_size)\n group_test_starts = range(n_groups - n_splits * group_test_size,\n n_groups, group_test_size)\n for group_test_start in group_test_starts:\n train_array = []\n test_array = []\n\n group_st = max(0, group_test_start - group_gap - max_train_group_size)\n for train_group_idx in unique_groups[group_st:(group_test_start - group_gap)]:\n train_array_tmp = group_dict[train_group_idx]\n\n train_array = np.sort(np.unique(\n np.concatenate((train_array,\n train_array_tmp)),\n axis=None), axis=None)\n\n train_end = train_array.size\n\n for test_group_idx in unique_groups[group_test_start:\n group_test_start +\n group_test_size]:\n test_array_tmp = group_dict[test_group_idx]\n test_array = np.sort(np.unique(\n np.concatenate((test_array,\n test_array_tmp)),\n axis=None), axis=None)\n\n test_array = test_array[group_gap:]\n\n\n if self.verbose > 0:\n pass\n\n yield [int(i) for i in train_array], [int(i) for i in test_array]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"papermill":{"duration":0.07369,"end_time":"2021-11-27T08:17:13.112796","exception":false,"start_time":"2021-11-27T08:17:13.039106","status":"completed"},"pycharm":{"name":"#%%\n"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:21:15.237791Z","iopub.execute_input":"2022-01-25T19:21:15.238137Z","iopub.status.idle":"2022-01-25T19:21:15.271779Z","shell.execute_reply.started":"2022-01-25T19:21:15.238111Z","shell.execute_reply":"2022-01-25T19:21:15.271022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training ποΈ\n\n\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Each fold the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variable `VERBOSE`. The variable `VERBOSE=1 or 2` will display the training and validation loss for each epoch as text. ","metadata":{"papermill":{"duration":0.032105,"end_time":"2021-11-27T08:17:13.17731","exception":false,"start_time":"2021-11-27T08:17:13.145205","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Let's take a look at our CV**","metadata":{"papermill":{"duration":0.031822,"end_time":"2021-11-27T08:17:13.302287","exception":false,"start_time":"2021-11-27T08:17:13.270465","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom matplotlib.colors import ListedColormap\n\ndef plot_cv_indices(cv, X, y, group, ax, n_splits, lw=10):\n cmap_cv = plt.cm.coolwarm\n jet = plt.cm.get_cmap('jet', 256)\n seq = np.linspace(0, 1, 256)\n _ = np.random.shuffle(seq) # inplace\n cmap_data = ListedColormap(jet(seq)) \n for ii, (tr, tt) in enumerate(list(cv.split(X=X, y=y, groups=group))):\n indices = np.array([np.nan] * len(X))\n indices[tt] = 1\n indices[tr] = 0 \n ax.scatter(range(len(indices)), [ii + .5] * len(indices), c=indices, marker='_', lw=lw, cmap=cmap_cv, vmin=-.2, vmax=1.2)\n ax.scatter(range(len(X)), [ii + 1.5] * len(X), c=y, marker='_', lw=lw, cmap=plt.cm.Set3)\n ax.scatter(range(len(X)), [ii + 2.5] * len(X), c=group, marker='_', lw=lw, cmap=cmap_data)\n yticklabels = list(range(n_splits)) + ['target', 'day']\n ax.set(yticks=np.arange(n_splits+2) + .5, yticklabels=yticklabels, xlabel='Sample index', ylabel=\"CV iteration\", ylim=[n_splits+2.2, -.2], xlim=[0, len(y)])\n ax.set_title('{}'.format(type(cv).__name__), fontsize=15)\n return ax\n\nasset_id = 0\ndf = load_training_data_for_asset(asset_id)\ndf_proc = get_features(df)\ndf_proc['date'] = df['date'].copy()\ndf_proc['y'] = df['Target']\ndf_proc = df_proc.dropna(how=\"any\")\nX = df_proc.drop(\"y\", axis=1)\ny = df_proc[\"y\"]\ngroups = pd.factorize(X['date'].dt.day.astype(str) + '_' + X['date'].dt.month.astype(str) + '_' + X['date'].dt.year.astype(str))[0]\nX = X.drop(columns = 'date')\n\nfig, ax = plt.subplots(figsize = (12, 6))\ncv = PurgedGroupTimeSeriesSplit(n_splits = FOLDS, group_gap = GROUP_GAP, max_train_group_size=MAX_TRAIN_GROUP_SIZE, max_test_group_size=MAX_TEST_GROUP_SIZE)\nplot_cv_indices(cv, X, y, groups, ax, FOLDS, lw=20)","metadata":{"_kg_hide-input":true,"papermill":{"duration":167.596868,"end_time":"2021-11-27T08:20:00.931193","exception":false,"start_time":"2021-11-27T08:17:13.334325","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-25T19:21:15.273146Z","iopub.execute_input":"2022-01-25T19:21:15.273599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 2\n\ndef get_Xy_and_model_for_asset(asset_id):\n df = load_training_data_for_asset(asset_id)\n df_proc = get_features(df)\n df_proc['date'] = df['date'].copy()\n df_proc['y'] = df['Target']\n df_proc = df_proc.dropna(how=\"any\")\n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n groups = pd.factorize(X['date'].dt.day.astype(str) + '_' + X['date'].dt.month.astype(str) + '_' + X['date'].dt.year.astype(str))[0]\n X = X.drop(columns = 'date')\n oof_preds = np.zeros(len(X))\n scores, models = [], []\n \n for fold, (train_idx, val_idx) in enumerate(PurgedGroupTimeSeriesSplit(n_splits = FOLDS, group_gap = GROUP_GAP, max_train_group_size = MAX_TRAIN_GROUP_SIZE, max_test_group_size = MAX_TEST_GROUP_SIZE).split(X, y, groups)):\n # GET TRAINING, VALIDATION SET\n x_train, x_val = X.iloc[train_idx], X.iloc[val_idx]\n y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]\n\n # DISPLAY FOLD INFO\n if DEVICE == 'TPU':\n if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n print('#'*25); print('#### FOLD',fold+1)\n print('#### Training WIDTH %s DEPTH %s | batch_size %s' % (WIDTH_NETS[fold], DEPTH_NETS[fold], BATCH_SIZES[fold]*REPLICAS))\n\n # BUILD MODEL\n K.clear_session()\n with strategy.scope(): model = build_model(fold, dim = x_train.shape[1], weight = asset_weight_dict[asset_id])\n\n # SAVE BEST MODEL EACH FOLD\n sv = tf.keras.callbacks.ModelCheckpoint('fold-%i.h5' % fold, monitor = 'val_loss', verbose = 0, save_best_only = True, save_weights_only = True, mode = 'min', save_freq = 'epoch')\n\n # TRAIN\n history = model.fit( x_train, y_train, epochs = EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], validation_data = (x_val, y_val), verbose=VERBOSE )\n model.load_weights('fold-%i.h5' % fold)\n\n # PREDICT OOF\n pred = model.predict(x_val, verbose = VERBOSE)[2]\n models.append(model)\n\n # REPORT RESULTS\n try: mse = mean_squared_error(np.nan_to_num(y_val), np.nan_to_num(pred))\n except: mse = 0.0\n scores.append(mse)\n oof_preds[val_idx] = pred[:, 0]\n w_score = corr(np.nan_to_num(y_val), np.nan_to_num(pred.flatten()), np.array([asset_weight_dict[asset_id]] * len(y_val)))\n print('#### FOLD %i OOF MSE %.3f | WCORR: %.3f' % (fold + 1, mse, w_score))\n\n df = df_proc\n df['oof_preds'] = np.nan_to_num(oof_preds)\n print('\\n\\n' + ('-' * 80) + '\\n' + 'Finished training %s. Results:' % asset_name_dict[asset_id])\n print('Model: r2_score: %s | pearsonr: %s | wcorr: %s ' % (r2_score(df['y'], df['oof_preds']), pearsonr(df['y'], df['oof_preds'])[0], corr(df['y'].values, df['oof_preds'].values, np.array([asset_weight_dict[asset_id]] * len(df['y'].values)))))\n print('Predictions std: %s | Target std: %s' % (df['oof_preds'].std(), df['y'].std()))\n \n try: plt.close()\n except: pass \n df2 = df.reset_index().set_index('date')\n fig = plt.figure(figsize = (12, 6))\n # fig, ax_left = plt.subplots(figsize = (12, 6))\n ax_left = fig.add_subplot(111)\n ax_left.set_facecolor('azure') \n ax_right = ax_left.twinx()\n ax_left.plot(df2['y'].rolling(3 * 30 * 24 * 60).corr(df2['oof_preds']).iloc[::24 * 60], color = 'crimson', label = \"Target WCorr\")\n ax_right.plot(df2['Close'].iloc[::24 * 60], color = 'darkgrey', label = \"%s Close\" % asset_name_dict[asset_id]) \n plt.legend()\n plt.grid()\n plt.xlabel('Time')\n plt.title('3 month rolling pearsonr for %s' % (asset_name_dict[asset_id]))\n plt.show()\n \n return scores, oof_preds, models, y\n\nmodels, scores, targets, oof_preds = {}, {}, {}, {}\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n cur_scores, cur_oof_preds, cur_models, cur_targets = get_Xy_and_model_for_asset(asset_id)\n scores[asset_id], oof_preds[asset_id], models[asset_id], targets[asset_id] = np.mean(cur_scores), cur_oof_preds, cur_models, cur_targets","metadata":{"papermill":{"duration":4874.677043,"end_time":"2021-11-27T09:41:15.642029","exception":false,"start_time":"2021-11-27T08:20:00.964986","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Calculate OOF MSE\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions.","metadata":{"papermill":{"duration":0.365578,"end_time":"2021-11-27T09:41:16.169668","exception":false,"start_time":"2021-11-27T09:41:15.80409","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# COMPUTE OVERALL OOF MSE\nprint('Overall MEAN OOF MSE %s' % np.mean(list(scores.values())))\n\n# SAVE OOF TO DISK \ny_pred, y_true, weights = [], [], []\nfor asset in oof_preds:\n df_oof = pd.DataFrame(dict(asset_id = asset, oof_preds=oof_preds[asset]))\n df_oof.to_csv(str(asset) + '_oof.csv',index=False)\n y_pred += oof_preds[asset].tolist()\n y_true += targets[asset].tolist() \n weights += ([asset_weight_dict[asset]] * len(oof_preds[asset].tolist()))\n print('%s score: %s' % (asset_name_dict[asset], corr(np.nan_to_num(np.array(y_true).flatten()), np.nan_to_num(np.array(y_pred).flatten()), np.nan_to_num(np.array(weights).flatten()))))\n \nprint('Overall score %s' % corr(np.nan_to_num(np.array(y_true).flatten()), np.nan_to_num(np.array(y_pred).flatten()), np.nan_to_num(np.array(weights).flatten())))","metadata":{"papermill":{"duration":164.095221,"end_time":"2021-11-27T09:44:00.427663","exception":false,"start_time":"2021-11-27T09:41:16.332442","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit To Kaggle π°\n","metadata":{"papermill":{"duration":0.169498,"end_time":"2021-11-27T09:44:00.763711","exception":false,"start_time":"2021-11-27T09:44:00.594213","status":"completed"},"tags":[]}},{"cell_type":"code","source":"all_df_test = []\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n try:\n if row['Asset_ID'] in models:\n cur_models = models[row['Asset_ID']]\n x_test = get_features(row)\n y_pred = np.mean(np.concatenate([np.expand_dims(model.predict(x_test.to_frame().T)[2][0], axis = 0) for model in cur_models], axis = 0), axis = 0)\n else: y_pred = 0.0\n except: \n y_pred = 0.0\n traceback.print_exc()\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n all_df_test.append(df_test)\n env.predict(df_pred)\n","metadata":{"papermill":{"duration":26.926705,"end_time":"2021-11-27T09:44:27.860629","exception":false,"start_time":"2021-11-27T09:44:00.933924","status":"completed"},"pycharm":{"name":"#%%\n"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References\n\n1. [Initial baseline notebook](https://www.kaggle.com/julian3833) \n2. [Competition tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition) \n3. [Competition Overview](https://www.kaggle.com/c/g-research-crypto-forecasting/overview) \n4. [My Initial Ideas for this competition](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903) \n5. [My post notebook about cross validation](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit) \n6. [Chris original notebook from SIIM ISIC](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) \n\nWORK IN PROGRESS! π§","metadata":{"papermill":{"duration":0.1671,"end_time":"2021-11-27T09:44:28.19687","exception":false,"start_time":"2021-11-27T09:44:28.02977","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n ![](\"https://i.ibb.co/2M2MsgH/upvote7.png\")