{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"%matplotlib inline\nfrom IPython.core.display import display, HTML, Javascript\n\nhtml_contents =\"\"\"\n<!DOCTYPE html>\n<html lang=\"en\">\n    <head>\n        <link rel=\"stylesheet\" href=\"https://www.w3schools.com/w3css/4/w3.css\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Raleway\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Oswald\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Open Sans\">\n        <link rel=\"stylesheet\" href=\"https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css\">\n        <style>\n        .title-section{\n            font-family: \"Oswald\", Arial, sans-serif;\n            font-weight: bold;\n            color: \"#6A8CAF\";\n            letter-spacing: 6px;\n        }\n        hr { border: 1px solid #E58F65 !important;\n             color: #E58F65 !important;\n             background: #E58F65 !important;\n           }\n        body {\n            font-family: \"Open Sans\", sans-serif;\n            }        \n        </style>\n    </head>    \n</html>\n\"\"\"\n\nHTML(html_contents)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-11T17:32:04.523252Z","iopub.execute_input":"2021-11-11T17:32:04.524084Z","iopub.status.idle":"2021-11-11T17:32:04.543245Z","shell.execute_reply.started":"2021-11-11T17:32:04.524043Z","shell.execute_reply":"2021-11-11T17:32:04.542484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">PurgedGroupTimeSeries CV with Extra Data - Xgboost Version</span>\nThis is a simple starter notebook for Kaggle's Crypto Comp showing purged group timeseries KFold with extra data. Purged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either CPU or GPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model hyperparameters, loss, and number of seeds to ensemble. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n\n**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validation experiments but in that case note that the overall CV score is meaningless because LB will be much different when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords","metadata":{}},{"cell_type":"markdown","source":"____\n\n#### <center>All baselines in the series 👇</center>\n\n| CV + Model | Hyperparam Optimization  | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner)        | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | ⏳Technical Analysis |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm)     | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna)     | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | ⏳Time Series Agg | \n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost)      | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | ⏳Target Engineering |\n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data)                                            | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | |⏳Neutralization |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | |⏳Quant's Volatility Features |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer)                                     | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline)  | |⏳Fourier Analysis \n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data)                                       |  |  | ⏳Wavelets | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) |  |  \n\n____","metadata":{}},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Kaggle's G-Research Crypto Forecasting</span>\nIn this competition, we need to forecast returns of cryptocurrency assets. Full description [here][1]. This is a very challenging time series task as seen by looking at the sample data below.\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/overview","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.graph_objects as go\ncrypto_df = pd.read_csv(\"../input/g-research-crypto-forecasting/\" + 'train.csv')\nbtc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\")\nbtc_mini = btc.iloc[-200:]\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-11T17:32:04.546669Z","iopub.execute_input":"2021-11-11T17:32:04.549623Z","iopub.status.idle":"2021-11-11T17:32:54.415176Z","shell.execute_reply.started":"2021-11-11T17:32:04.549585Z","shell.execute_reply":"2021-11-11T17:32:54.413624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Initialize Environment</span>","metadata":{}},{"cell_type":"code","source":"import os\nimport traceback\nimport gresearch_crypto\nimport pandas as pd, numpy as np\nfrom xgboost import XGBRegressor\nfrom sklearn.metrics import mean_squared_error","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:54.416357Z","iopub.execute_input":"2021-11-11T17:32:54.416759Z","iopub.status.idle":"2021-11-11T17:32:55.354209Z","shell.execute_reply.started":"2021-11-11T17:32:54.416722Z","shell.execute_reply":"2021-11-11T17:32:55.353531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Configuration</span>\nIn order to be a proper cross validation with a meaningful overall CV score, **you need to choose the same** `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP`, and `DEPTH_NETS`, `WIDTH_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds.\n* DEVICE - is CPU or GPU\n* SEED - a different seed produces a different triple stratified kfold split.\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* INC2021 - This controls whether to include the extra historical prices during 2021.\n* INC2020 - This controls whether to include the extra historical prices during 2020.\n* INC2019 - This controls whether to include the extra historical prices during 2019.\n* INC2018 - This controls whether to include the extra historical prices during 2018.\n* INC2017 - This controls whether to include the extra historical prices during 2017.\n* INCCOMP - This controls whether to include the original data of the competition.\n* INCSUPP - This controls whether to include the supplemented train data that was released with the competition.\n* N_ESTIMATORS - is a list of length FOLDS. These are n_estimators for each fold. For maximum speed, it is best to use the smallest number of estimators as your GPU or CPU allows.\n* MAX_DEPTH - is a list of length FOLDS. These are max_depth for each fold. For maximum speed, it is best to use the smallest number of estimators as your GPU or CPU allows.\n* LEARNING_RATE - is a list of length FOLDS. These are max_depth for each fold.","metadata":{}},{"cell_type":"code","source":"DEVICE = \"GPU\" #or \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. \nFOLDS = 5\n# GAP FROM SPLIT TO SPLIT [SEE DETAILS ABOUT THE CV]\nGROUP_GAP = 31\n\n# INCLUDE OLD COMP DATA? YES=1 NO=0\nINC2021 = 0\nINC2020 = 0\nINC2019 = 0\nINC2018 = 0\nINC2017 = 0\nINCCOMP = 1\nINCSUPP = 0\n\n# HYPER PARAMETERS\nLEARNING_RATE = [0.09, 0.09, 0.09, 0.09, 0.09]\nN_ESTIMATORS = [100, 100, 100, 100, 100]\nMAX_DEPTH = [10, 10, 10, 10, 10]\n","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.status.busy":"2021-11-11T17:32:55.355413Z","iopub.execute_input":"2021-11-11T17:32:55.355659Z","iopub.status.idle":"2021-11-11T17:32:55.361415Z","shell.execute_reply.started":"2021-11-11T17:32:55.355626Z","shell.execute_reply":"2021-11-11T17:32:55.360805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 1: Data Loading</span>\nThe data organisation has already been done and saved to Kaggle datasets. Here we choose which years to load. We can use either 2017, 2018, 2019, 2020, 2021, Original, Supplement by changing the `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` variables in the preceeding code section. These datasets are discussed [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726\n","metadata":{}},{"cell_type":"code","source":"# orig_df_train = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')\n# supp_df_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')\n# df_asset_details = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/asset_details.csv').sort_values(\"Asset_ID\")\n\n# extra_data_files = {\n#                         0: '../input/cryptocurrency-extra-data-binance-coin',\n#                         2: '../input/cryptocurrency-extra-data-bitcoin-cash',\n#                         1: '../input/cryptocurrency-extra-data-bitcoin',\n#                         3: '../input/cryptocurrency-extra-data-cardano',\n#                         4: '../input/cryptocurrency-extra-data-dogecoin',                        \n#                         5: '../input/cryptocurrency-extra-data-eos-io',\n#                         6: '../input/cryptocurrency-extra-data-ethereum',\n#                         7: '../input/cryptocurrency-extra-data-ethereum-classic', \n#                         8: '../input/cryptocurrency-extra-data-iota',\n#                         9: '../input/cryptocurrency-extra-data-litecoin',\n#                         11: '../input/cryptocurrency-extra-data-monero',\n#                         10: '../input/cryptocurrency-extra-data-maker',\n#                         12: '../input/cryptocurrency-extra-data-stellar',\n#                         13: '../input/cryptocurrency-extra-data-tron'\n#                    }\n\n# def load_training_data_for_asset(asset_id):\n#     dfs = []        \n#     if INCCOMP: dfs.append(orig_df_train[orig_df_train[\"Asset_ID\"] == asset_id].copy())\n#     if INCSUPP: dfs.append(supp_df_train[supp_df_train[\"Asset_ID\"] == asset_id].copy())    \n#     if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'))\n#     if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'))\n#     if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'))\n#     if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'))\n#     if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'))\n#     df = pd.concat(dfs, axis = 0) if len(dfs) > 1 else dfs[0]\n#     df['date'] = pd.to_datetime(df['timestamp'], unit = 's')        \n#     df = df.sort_values('date')\n#     return df\n","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:55.363663Z","iopub.execute_input":"2021-11-11T17:32:55.364105Z","iopub.status.idle":"2021-11-11T17:32:55.371898Z","shell.execute_reply.started":"2021-11-11T17:32:55.364071Z","shell.execute_reply":"2021-11-11T17:32:55.371250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 2: Feature Engineering</span>\nThis notebook uses upper_shadow, lower_shadow, high_div_low, open_sub_close, seasonality/datetime features first shown in this notebook [here][1] and successfully used by julian3833 [here][2].\n\nAdditionally we can decide to use external data by changing the variables `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` in the preceeding code section. These variables respectively indicate whether to load last year 2021 data and/or year 2020, 2019, 2018, 2017, the original, supplemented data. These datasets are discussed [here][3]\n\nConsider experimenting with different feature engineering and/or external data. The code to extract features out of the dataset is taken from julian3833' notebook [here][2]. Thank you julian3833, this is great work.\n\n[1]: https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\n[2]: https://www.kaggle.com/julian3833\n[3]: TBD","metadata":{}},{"cell_type":"code","source":"# Two features from the competition tutorial\ndef upper_shadow(df): return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df): return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# A utility function to build features from the original df\ndef get_features(df):\n    df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feat['upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['lower_Shadow'] = lower_shadow(df_feat)\n    df_feat[\"high_div_low\"] = df_feat[\"High\"] / df_feat[\"Low\"]\n    df_feat[\"open_sub_close\"] = df_feat[\"Open\"] - df_feat[\"Close\"]\n    return df_feat","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:55.372758Z","iopub.execute_input":"2021-11-11T17:32:55.374860Z","iopub.status.idle":"2021-11-11T17:32:55.384388Z","shell.execute_reply.started":"2021-11-11T17:32:55.374835Z","shell.execute_reply":"2021-11-11T17:32:55.383716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 3: Configure the model</span>\nThis is a simple model with simple set of hyperparameters. Consider experimenting with different models, parameters, ensembles and so on.","metadata":{}},{"cell_type":"code","source":"def build_model(fold):\n    \n    # Do feel free to experiment with different models here!\n    model = XGBRegressor(n_estimators = N_ESTIMATORS[fold], max_depth = MAX_DEPTH[fold], learning_rate = LEARNING_RATE[fold], tree_method = 'hist' if DEVICE == 'CPU' else 'gpu_hist')\n\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:55.386382Z","iopub.execute_input":"2021-11-11T17:32:55.387060Z","iopub.status.idle":"2021-11-11T17:32:55.396439Z","shell.execute_reply.started":"2021-11-11T17:32:55.387022Z","shell.execute_reply":"2021-11-11T17:32:55.395675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass GroupTimeSeriesSplit(_BaseKFold):\n    \"\"\"Time Series cross-validator variant with non-overlapping groups.\n    Provides train/test indices to split time series data samples\n    that are observed at fixed time intervals according to a\n    third-party provided group.\n    In each split, test indices must be higher than before, and thus shuffling\n    in cross validator is inappropriate.\n    This cross-validation object is a variation of :class:`KFold`.\n    In the kth split, it returns first k folds as train set and the\n    (k+1)th fold as test set.\n    The same group will not appear in two different folds (the number of\n    distinct groups has to be at least equal to the number of folds).\n    Note that unlike standard cross-validation methods, successive\n    training sets are supersets of those that come before them.\n    Read more in the :ref:`User Guide <cross_validation>`.\n    Parameters\n    ----------\n    n_splits : int, default=5\n        Number of splits. Must be at least 2.\n    max_train_size : int, default=None\n        Maximum size for a single training set.\n    Examples\n    --------\n    >>> import numpy as np\n    >>> from sklearn.model_selection import GroupTimeSeriesSplit\n    >>> groups = np.array(['a', 'a', 'a', 'a', 'a', 'a',\\\n                           'b', 'b', 'b', 'b', 'b',\\\n                           'c', 'c', 'c', 'c',\\\n                           'd', 'd', 'd'])\n    >>> gtss = GroupTimeSeriesSplit(n_splits=3)\n    >>> for train_idx, test_idx in gtss.split(groups, groups=groups):\n    ...     print(\"TRAIN:\", train_idx, \"TEST:\", test_idx)\n    ...     print(\"TRAIN GROUP:\", groups[train_idx],\\\n                  \"TEST GROUP:\", groups[test_idx])\n    TRAIN: [0, 1, 2, 3, 4, 5] TEST: [6, 7, 8, 9, 10]\n    TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a']\\\n    TEST GROUP: ['b' 'b' 'b' 'b' 'b']\n    TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] TEST: [11, 12, 13, 14]\n    TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b']\\\n    TEST GROUP: ['c' 'c' 'c' 'c']\n    TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\\\n    TEST: [15, 16, 17]\n    TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b' 'c' 'c' 'c' 'c']\\\n    TEST GROUP: ['d' 'd' 'd']\n    \"\"\"\n    @_deprecate_positional_args\n    def __init__(self,\n                 n_splits=5,\n                 *,\n                 max_train_size=None\n                 ):\n        super().__init__(n_splits, shuffle=False, random_state=None)\n        self.max_train_size = max_train_size\n\n    def split(self, X, y=None, groups=None):\n        \"\"\"Generate indices to split data into training and test set.\n        Parameters\n        ----------\n        X : array-like of shape (n_samples, n_features)\n            Training data, where n_samples is the number of samples\n            and n_features is the number of features.\n        y : array-like of shape (n_samples,)\n            Always ignored, exists for compatibility.\n        groups : array-like of shape (n_samples,)\n            Group labels for the samples used while splitting the dataset into\n            train/test set.\n        Yields\n        ------\n        train : ndarray\n            The training set indices for that split.\n        test : ndarray\n            The testing set indices for that split.\n        \"\"\"\n        if groups is None:\n            raise ValueError(\n                \"The 'groups' parameter should not be None\")\n        X, y, groups = indexable(X, y, groups)\n        n_samples = _num_samples(X)\n        n_splits = self.n_splits\n        n_folds = n_splits + 1\n        group_dict = {}\n        u, ind = np.unique(groups, return_index=True)\n        unique_groups = u[np.argsort(ind)]\n        n_samples = _num_samples(X)\n        n_groups = _num_samples(unique_groups)\n        for idx in np.arange(n_samples):\n            if (groups[idx] in group_dict):\n                group_dict[groups[idx]].append(idx)\n            else:\n                group_dict[groups[idx]] = [idx]\n        if n_folds > n_groups:\n            raise ValueError(\n                (\"Cannot have number of folds={0} greater than\"\n                 \" the number of groups={1}\").format(n_folds,\n                                                     n_groups))\n        group_test_size = n_groups // n_folds\n        group_test_starts = range(n_groups - n_splits * group_test_size,\n                                  n_groups, group_test_size)\n        for group_test_start in group_test_starts:\n            train_array = []\n            test_array = []\n            for train_group_idx in unique_groups[:group_test_start]:\n                train_array_tmp = group_dict[train_group_idx]\n                train_array = np.sort(np.unique(\n                                      np.concatenate((train_array,\n                                                      train_array_tmp)),\n                                      axis=None), axis=None)\n            train_end = train_array.size\n            if self.max_train_size and self.max_train_size < train_end:\n                train_array = train_array[train_end -\n                                          self.max_train_size:train_end]\n            for test_group_idx in unique_groups[group_test_start:\n                                                group_test_start +\n                                                group_test_size]:\n                test_array_tmp = group_dict[test_group_idx]\n                test_array = np.sort(np.unique(\n                                              np.concatenate((test_array,\n                                                              test_array_tmp)),\n                                     axis=None), axis=None)\n            yield [int(i) for i in train_array], [int(i) for i in test_array]\nimport numpy as np\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# modified code for group gaps; source\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass PurgedGroupTimeSeriesSplit(_BaseKFold):\n    \"\"\"Time Series cross-validator variant with non-overlapping groups.\n    Allows for a gap in groups to avoid potentially leaking info from\n    train into test if the model has windowed or lag features.\n    Provides train/test indices to split time series data samples\n    that are observed at fixed time intervals according to a\n    third-party provided group.\n    In each split, test indices must be higher than before, and thus shuffling\n    in cross validator is inappropriate.\n    This cross-validation object is a variation of :class:`KFold`.\n    In the kth split, it returns first k folds as train set and the\n    (k+1)th fold as test set.\n    The same group will not appear in two different folds (the number of\n    distinct groups has to be at least equal to the number of folds).\n    Note that unlike standard cross-validation methods, successive\n    training sets are supersets of those that come before them.\n    Read more in the :ref:`User Guide <cross_validation>`.\n    Parameters\n    ----------\n    n_splits : int, default=5\n        Number of splits. Must be at least 2.\n    max_train_group_size : int, default=Inf\n        Maximum group size for a single training set.\n    group_gap : int, default=None\n        Gap between train and test\n    max_test_group_size : int, default=Inf\n        We discard this number of groups from the end of each train split\n    \"\"\"\n\n    @_deprecate_positional_args\n    def __init__(self,\n                 n_splits=5,\n                 *,\n                 max_train_group_size=np.inf,\n                 max_test_group_size=np.inf,\n                 group_gap=None,\n                 verbose=False\n                 ):\n        super().__init__(n_splits, shuffle=False, random_state=None)\n        self.max_train_group_size = max_train_group_size\n        self.group_gap = group_gap\n        self.max_test_group_size = max_test_group_size\n        self.verbose = verbose\n\n    def split(self, X, y=None, groups=None):\n        \"\"\"Generate indices to split data into training and test set.\n        Parameters\n        ----------\n        X : array-like of shape (n_samples, n_features)\n            Training data, where n_samples is the number of samples\n            and n_features is the number of features.\n        y : array-like of shape (n_samples,)\n            Always ignored, exists for compatibility.\n        groups : array-like of shape (n_samples,)\n            Group labels for the samples used while splitting the dataset into\n            train/test set.\n        Yields\n        ------\n        train : ndarray\n            The training set indices for that split.\n        test : ndarray\n            The testing set indices for that split.\n        \"\"\"\n        if groups is None:\n            raise ValueError(\n                \"The 'groups' parameter should not be None\")\n        X, y, groups = indexable(X, y, groups)\n        n_samples = _num_samples(X)\n        n_splits = self.n_splits\n        group_gap = self.group_gap\n        max_test_group_size = self.max_test_group_size\n        max_train_group_size = self.max_train_group_size\n        n_folds = n_splits + 1\n        group_dict = {}\n        u, ind = np.unique(groups, return_index=True)\n        unique_groups = u[np.argsort(ind)]\n        n_samples = _num_samples(X)\n        n_groups = _num_samples(unique_groups)\n        for idx in np.arange(n_samples):\n            if (groups[idx] in group_dict):\n                group_dict[groups[idx]].append(idx)\n            else:\n                group_dict[groups[idx]] = [idx]\n        if n_folds > n_groups:\n            raise ValueError(\n                (\"Cannot have number of folds={0} greater than\"\n                 \" the number of groups={1}\").format(n_folds,\n                                                     n_groups))\n\n        group_test_size = min(n_groups // n_folds, max_test_group_size)\n        group_test_starts = range(n_groups - n_splits * group_test_size,\n                                  n_groups, group_test_size)\n        for group_test_start in group_test_starts:\n            train_array = []\n            test_array = []\n\n            group_st = max(0, group_test_start - group_gap - max_train_group_size)\n            for train_group_idx in unique_groups[group_st:(group_test_start - group_gap)]:\n                train_array_tmp = group_dict[train_group_idx]\n\n                train_array = np.sort(np.unique(\n                                      np.concatenate((train_array,\n                                                      train_array_tmp)),\n                                      axis=None), axis=None)\n\n            train_end = train_array.size\n\n            for test_group_idx in unique_groups[group_test_start:\n                                                group_test_start +\n                                                group_test_size]:\n                test_array_tmp = group_dict[test_group_idx]\n                test_array = np.sort(np.unique(\n                                              np.concatenate((test_array,\n                                                              test_array_tmp)),\n                                     axis=None), axis=None)\n\n            test_array  = test_array[group_gap:]\n\n\n            if self.verbose > 0:\n                    pass\n\n            yield [int(i) for i in train_array], [int(i) for i in test_array]","metadata":{"pycharm":{"name":"#%%\n"},"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-11T17:32:55.399553Z","iopub.execute_input":"2021-11-11T17:32:55.399749Z","iopub.status.idle":"2021-11-11T17:32:55.433552Z","shell.execute_reply.started":"2021-11-11T17:32:55.399717Z","shell.execute_reply":"2021-11-11T17:32:55.432926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Train Model</span>\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Each fold the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variable `VERBOSE`. The variable `VERBOSE=1 or 2` will display the training and validation loss for each epoch as text. ","metadata":{}},{"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 2\n\ndef get_Xy_and_model_for_asset(asset_id):\n    df = load_training_data_for_asset(asset_id)\n    df_proc = get_features(df)\n    df_proc['date'] = df['date'].copy()\n    df_proc['y'] = df['Target']\n    df_proc = df_proc.dropna(how=\"any\")\n    X = df_proc.drop(\"y\", axis=1)\n    y = df_proc[\"y\"]\n    groups = pd.factorize(X['date'].dt.day.astype(str) + '_' + X['date'].dt.month.astype(str) + '_' + X['date'].dt.year.astype(str))[0]\n    X = X.drop(columns = 'date')\n    oof_preds = np.zeros(len(X))\n    scores, models = [], []\n\n    for fold, (train_idx, val_idx) in enumerate(PurgedGroupTimeSeriesSplit(n_splits = FOLDS, group_gap = GROUP_GAP).split(X, y, groups)):\n        # GET TRAINING, VALIDATION SET\n        x_train, x_val = X.iloc[train_idx], X.iloc[val_idx]\n        y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]\n\n        # DISPLAY FOLD INFO\n        print('#' * 25); print('#### FOLD', fold + 1)\n        print('#### Training N_ESTIMATORS %s | MAX_DEPTH %s | LEARNING_RATE %s' % (N_ESTIMATORS[fold], MAX_DEPTH[fold], LEARNING_RATE[fold]))\n\n        model = build_model(fold)\n\n        # TRAIN\n        print('Training...')\n        model.fit( x_train, y_train, eval_set = [(x_val, y_val)] )\n\n        # PREDICT OOF\n        print('Predicting OOF...')\n        pred = model.predict(x_val)\n        models.append(model)\n\n        # REPORT RESULTS\n        try: mse = mean_squared_error(np.nan_to_num(y_val), np.nan_to_num(pred))\n        except: mse = 0.0\n        scores.append(mse)\n        oof_preds[val_idx] = pred\n        print('#### FOLD %i OOF MSE %.3f' % (fold + 1, mse))\n\n    return scores, oof_preds, models\n\n# models = {}\n# scores = {}\n# oof_preds = {}\n# for asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n#     print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n#     try: \n#         cur_scores, cur_oof_preds, cur_models = get_Xy_and_model_for_asset(asset_id)\n#         scores[asset_id], oof_preds[asset_id], models[asset_id] = cur_scores, oof_preds, cur_models\n#     except: traceback.print_exc()    ","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-11-11T17:32:55.434883Z","iopub.execute_input":"2021-11-11T17:32:55.435117Z","iopub.status.idle":"2021-11-11T17:32:55.447291Z","shell.execute_reply.started":"2021-11-11T17:32:55.435085Z","shell.execute_reply":"2021-11-11T17:32:55.446090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Calculate OOF MSE</span>\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions.","metadata":{}},{"cell_type":"code","source":"# # COMPUTE OVERALL OOF MSE\n# print('Overall MEAN OOF MSE %s' % np.mean(list(scores.values())))\n\n# # SAVE OOF TO DISK \n# for asset in oof_preds:\n#     df_oof = pd.DataFrame(dict(asset_id = asset, oof_preds=oof_preds))\n#     df_oof.to_csv(str(asset) + '_oof.csv',index=False)\n#     df_oof.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:55.448713Z","iopub.execute_input":"2021-11-11T17:32:55.448968Z","iopub.status.idle":"2021-11-11T17:32:55.458788Z","shell.execute_reply.started":"2021-11-11T17:32:55.448935Z","shell.execute_reply":"2021-11-11T17:32:55.458067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df = load_training_data_for_asset(1)\n# df_proc = get_features(df)\n# df_proc['date'] = df['date'].copy()\n# df_proc['y'] = df['Target']\n# df_proc = df_proc.dropna(how=\"any\")\n# X = df_proc.drop(\"y\", axis=1)\n# y = df_proc[\"y\"]\n# groups = pd.factorize(X['date'].dt.day.astype(str) + '_' + X['date'].dt.month.astype(str) + '_' + X['date'].dt.year.astype(str))[0]\n# X = X.drop(columns = 'date')\n# oof_preds = np.zeros(len(X))\n# scores, models = [], []\n\n# for fold, (train_idx, val_idx) in enumerate(PurgedGroupTimeSeriesSplit(n_splits = FOLDS, group_gap = GROUP_GAP).split(X, y, groups)):\n# # GET TRAINING, VALIDATION SET\n#     x_train, x_val = X.iloc[train_idx], X.iloc[val_idx]\n#     y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:55.460146Z","iopub.execute_input":"2021-11-11T17:32:55.460457Z","iopub.status.idle":"2021-11-11T17:32:55.467853Z","shell.execute_reply.started":"2021-11-11T17:32:55.460368Z","shell.execute_reply":"2021-11-11T17:32:55.467100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# x_train.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:32:55.469131Z","iopub.execute_input":"2021-11-11T17:32:55.469443Z","iopub.status.idle":"2021-11-11T17:32:55.479411Z","shell.execute_reply.started":"2021-11-11T17:32:55.469406Z","shell.execute_reply":"2021-11-11T17:32:55.478572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\nfile_name = \"../input/xgb-reg-v11/xgb_reg_V1.pkl\"\n\n# load\nxgb_model_loaded = pickle.load(open(file_name, \"rb\"))\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-11T17:32:55.480648Z","iopub.execute_input":"2021-11-11T17:32:55.481049Z","iopub.status.idle":"2021-11-11T17:33:04.209186Z","shell.execute_reply.started":"2021-11-11T17:32:55.481011Z","shell.execute_reply":"2021-11-11T17:33:04.208430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Submit To Kaggle</span>","metadata":{}},{"cell_type":"code","source":"# for i, (df_test, df_pred) in enumerate(iter_test):\n#     for j , row in df_test.iterrows():\n#         x_test = row\n#         print(x_test)\n#         break\n#     break","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:33:04.212188Z","iopub.execute_input":"2021-11-11T17:33:04.212453Z","iopub.status.idle":"2021-11-11T17:33:04.215519Z","shell.execute_reply.started":"2021-11-11T17:33:04.212419Z","shell.execute_reply":"2021-11-11T17:33:04.214875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# example_test = pd.read_csv(\"../input/g-research-crypto-forecasting/\" + 'example_test.csv')\n# example_test.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:33:04.216963Z","iopub.execute_input":"2021-11-11T17:33:04.217525Z","iopub.status.idle":"2021-11-11T17:33:04.225002Z","shell.execute_reply.started":"2021-11-11T17:33:04.217485Z","shell.execute_reply":"2021-11-11T17:33:04.224314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# example_test.iloc[0,:].shape","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:33:04.226398Z","iopub.execute_input":"2021-11-11T17:33:04.226744Z","iopub.status.idle":"2021-11-11T17:33:04.232800Z","shell.execute_reply.started":"2021-11-11T17:33:04.226709Z","shell.execute_reply":"2021-11-11T17:33:04.232071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# models_[0].predict(np.array(x_train.iloc[0,:]).reshape((1,-1)))","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:33:04.234358Z","iopub.execute_input":"2021-11-11T17:33:04.234665Z","iopub.status.idle":"2021-11-11T17:33:04.240756Z","shell.execute_reply.started":"2021-11-11T17:33:04.234630Z","shell.execute_reply":"2021-11-11T17:33:04.240054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# models_ = xgb_model_loaded[example_test.iloc[0,:]['Asset_ID']]\n# y_pred_example = np.mean(np.concatenate([np.expand_dims(model.predict(np.array(example_test.iloc[0,:]).reshape((1,-1))), axis = 0) for model in models_], axis = 0), axis = 0)\n# y_pred_example","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:33:04.242221Z","iopub.execute_input":"2021-11-11T17:33:04.242552Z","iopub.status.idle":"2021-11-11T17:33:04.249087Z","shell.execute_reply.started":"2021-11-11T17:33:04.242459Z","shell.execute_reply":"2021-11-11T17:33:04.248363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_df_test = []\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():\n        try:\n            if row['Asset_ID'] in xgb_model_loaded:\n                models_ = xgb_model_loaded[row['Asset_ID']]\n                x_test = get_features(row)\n                y_pred = np.mean(np.concatenate([np.expand_dims(model.predict(np.array(x_test).reshape((1,-1))), axis = 0) for model in models_], axis = 0), axis = 0)\n            else: y_pred = 0.0\n        except: \n            y_pred = 0.0\n            traceback.print_exc()\n        df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n    all_df_test.append(df_test)\n    env.predict(df_pred)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-11-11T17:33:04.250504Z","iopub.execute_input":"2021-11-11T17:33:04.250765Z","iopub.status.idle":"2021-11-11T17:33:06.472508Z","shell.execute_reply.started":"2021-11-11T17:33:04.250730Z","shell.execute_reply":"2021-11-11T17:33:06.471802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import pickle\n# import os\n# file_name = \"xgb_reg_V1.pkl\"\n\n\n# pickle.dump(models, open(file_name, \"wb\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-11T17:33:06.476310Z","iopub.execute_input":"2021-11-11T17:33:06.476526Z","iopub.status.idle":"2021-11-11T17:33:06.482004Z","shell.execute_reply.started":"2021-11-11T17:33:06.476501Z","shell.execute_reply":"2021-11-11T17:33:06.481340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\">References</span>\n\n<span id=\"f1\">1.</span> [Initial baseline notebook](https://www.kaggle.com/julian3833)<br>\n<span id=\"f2\">2.</span> [Competition tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)<br>\n<span id=\"f3\">3.</span> [Competition Overview](https://www.kaggle.com/c/g-research-crypto-forecasting/overview)</span><br>\n<span id=\"f4\">4.</span> [My Initial Ideas for this competition](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903)</span><br>\n<span id=\"f5\">5.</span> [My post notebook about cross validation](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit)</span><br>\n<span id=\"f5\">6.</span> [Chris original notebook from SIIM ISIC](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)</span><br>\n\n<span class=\"title-section w3-large w3-tag\">WORK IN PROGRESS! 🚧</span>","metadata":{}}]}