{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"%matplotlib inline\nfrom IPython.core.display import display, HTML, Javascript\n\nhtml_contents =\"\"\"\n<!DOCTYPE html>\n<html lang=\"en\">\n    <head>\n        <link rel=\"stylesheet\" href=\"https://www.w3schools.com/w3css/4/w3.css\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Raleway\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Oswald\">\n        <link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css?family=Open Sans\">\n        <link rel=\"stylesheet\" href=\"https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css\">\n        <style>\n        .title-section{\n            font-family: \"Oswald\", Arial, sans-serif;\n            font-weight: bold;\n            color: \"#6A8CAF\";\n            letter-spacing: 6px;\n        }\n        hr { border: 1px solid #E58F65 !important;\n             color: #E58F65 !important;\n             background: #E58F65 !important;\n           }\n        body {\n            font-family: \"Open Sans\", sans-serif;\n            }        \n        </style>\n    </head>    \n</html>\n\"\"\"\n\nHTML(html_contents)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-09T09:56:28.683484Z","iopub.execute_input":"2021-11-09T09:56:28.684010Z","iopub.status.idle":"2021-11-09T09:56:28.716426Z","shell.execute_reply.started":"2021-11-09T09:56:28.683907Z","shell.execute_reply":"2021-11-09T09:56:28.715488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">PurgedGroupTimeSeries CV with Extra Data</span>\nThis is a simple starter notebook for Kaggle's Crypto Comp showing purged group timeseries KFold with extra data. Purged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets at the same format of the competition, so you can input that into your model too.\n\n**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validaiton experiments but in that case note that the overall CV score is meaningless because LB will be much different when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords","metadata":{}},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Kaggle's G-Research Crypto Forecasting</span>\nIn this competition, we need to forecast returns of cryptocurrency assets. Full description [here][1]. This is a very challenging time series task as seen by looking at the sample data below.\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/overview","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.graph_objects as go\ncrypto_df = pd.read_csv(\"../input/g-research-crypto-forecasting/\" + 'train.csv')\nbtc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\")\nbtc_mini = btc.iloc[-200:]\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-11-09T09:56:28.720611Z","iopub.execute_input":"2021-11-09T09:56:28.720889Z","iopub.status.idle":"2021-11-09T09:57:26.586764Z","shell.execute_reply.started":"2021-11-09T09:56:28.720853Z","shell.execute_reply":"2021-11-09T09:57:26.585627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Initialize Environment</span>","metadata":{}},{"cell_type":"code","source":"import os\nimport traceback\nimport gresearch_crypto\nimport tensorflow as tf\nimport pandas as pd, numpy as np\nimport tensorflow.keras.backend as K\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:57:26.589034Z","iopub.execute_input":"2021-11-09T09:57:26.589360Z","iopub.status.idle":"2021-11-09T09:57:33.198078Z","shell.execute_reply.started":"2021-11-09T09:57:26.589315Z","shell.execute_reply":"2021-11-09T09:57:33.197032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Configuration</span>\nIn order to be a proper cross validation with a meaningful overall CV score, **you need to choose the same** `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP`, and `DEPTH_NETS`, `WIDTH_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds.\n* DEVICE - is GPU or TPU\n* SEED - a different seed produces a different triple stratified kfold split.\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* INC2021 - This controls whether to include the extra historical prices during 2021.\n* INC2020 - This controls whether to include the extra historical prices during 2020.\n* INC2019 - This controls whether to include the extra historical prices during 2019.\n* INC2018 - This controls whether to include the extra historical prices during 2018.\n* INC2017 - This controls whether to include the extra historical prices during 2017.\n* INCCOMP - This controls whether to include the original data of the competition.\n* INCSUPP - This controls whether to include the supplemented train data that was released with the competition.\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* DEPTH_NETS - is a list of length FOLDS. These are the Network Depths to use each fold. The number refers to the number of layers. So a number of `1` refers to 1 layer, and `2` refers to 2 layers, etc.\n* WIDTH_NETS - is a list of length FOLDS. These are the Network Widths to use each fold. The number refers to the number of units per layer. So a number of `32` refers to 32 per layer, and `643` refers to 64 layers, etc.\n* WGTS - this should be `1/FOLDS` for each fold. This is the weight when ensembling the folds to predict the test set. If you want a weird ensemble, you can use different weights.","metadata":{}},{"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. \nFOLDS = 5\n# GAP FROM SPLIT TO SPLIT [SEE DETAILS ABOUT THE CV]\nGROUP_GAP = 31\n\n# INCLUDE OLD COMP DATA? YES=1 NO=0\nINC2021 = 0\nINC2020 = 1\nINC2019 = 0\nINC2018 = 0\nINC2017 = 0\nINCCOMP = 1\nINCSUPP = 0\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [2048] * FOLDS\nEPOCHS = [1] * FOLDS\n\n# WHICH NETWORK ARCHITECTURE TO USE?\nDEPTH_NETS = [4, 4, 4, 4 ,4] \nWIDTH_NETS = [256, 256, 256, 256, 256]\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1 / FOLDS] * FOLDS","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.status.busy":"2021-11-09T09:57:33.199968Z","iopub.execute_input":"2021-11-09T09:57:33.200291Z","iopub.status.idle":"2021-11-09T09:57:33.213971Z","shell.execute_reply.started":"2021-11-09T09:57:33.200246Z","shell.execute_reply":"2021-11-09T09:57:33.210398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        tpu = None\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except: print(\"failed to initialize TPU\")\n    else: DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\": strategy = tf.distribute.get_strategy()\nif DEVICE == \"GPU\": print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:57:33.217830Z","iopub.execute_input":"2021-11-09T09:57:33.218549Z","iopub.status.idle":"2021-11-09T09:57:33.444387Z","shell.execute_reply.started":"2021-11-09T09:57:33.218487Z","shell.execute_reply":"2021-11-09T09:57:33.443382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 1: Data Loading</span>\nThe data organisation has already been done and saved to Kaggle datasets. Here we choose which years to load. We can use either 2017, 2018, 2019, 2020, 2021, Original, Supplement by changing the `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` variables in the preceeding code section. These datasets are discussed [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726\n","metadata":{}},{"cell_type":"code","source":"orig_df_train = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv')\nsupp_df_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')\ndf_asset_details = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/asset_details.csv').sort_values(\"Asset_ID\")\n\nextra_data_files = {\n                        0: '../input/cryptocurrency-extra-data-binance-coin',\n                        2: '../input/cryptocurrency-extra-data-bitcoin-cash',\n                        1: '../input/cryptocurrency-extra-data-bitcoin',\n                        3: '../input/cryptocurrency-extra-data-cardano',\n                        4: '../input/cryptocurrency-extra-data-dogecoin',                        \n                        5: '../input/cryptocurrency-extra-data-eos-io',\n                        6: '../input/cryptocurrency-extra-data-ethereum',\n                        7: '../input/cryptocurrency-extra-data-ethereum-classic', \n                        8: '../input/cryptocurrency-extra-data-iota',\n                        9: '../input/cryptocurrency-extra-data-litecoin',\n                        11: '../input/cryptocurrency-extra-data-monero',\n                        10: '../input/cryptocurrency-extra-data-maker',\n                        12: '../input/cryptocurrency-extra-data-stellar',\n                        13: '../input/cryptocurrency-extra-data-tron'\n                   }\n\ndef load_training_data_for_asset(asset_id):\n    dfs = []        \n    if INCCOMP: dfs.append(orig_df_train[orig_df_train[\"Asset_ID\"] == asset_id].copy())\n    if INCSUPP: dfs.append(supp_df_train[supp_df_train[\"Asset_ID\"] == asset_id].copy())    \n    if INC2017 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2017) + '.csv'))\n    if INC2018 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2018) + '.csv'))\n    if INC2019 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2019) + '.csv'))\n    if INC2020 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2020) + '.csv'))\n    if INC2021 and os.path.exists(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'): dfs.append(pd.read_csv(extra_data_files[asset_id] + '/full_data__' + str(asset_id) + '__' + str(2021) + '.csv'))\n    df = pd.concat(dfs, axis = 0) if len(dfs) > 1 else dfs[0]\n    df['date'] = pd.to_datetime(df['timestamp'], unit = 's')        \n    df = df.sort_values('date')\n    return df\n","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:57:33.446067Z","iopub.execute_input":"2021-11-09T09:57:33.446583Z","iopub.status.idle":"2021-11-09T09:58:12.045873Z","shell.execute_reply.started":"2021-11-09T09:57:33.446539Z","shell.execute_reply":"2021-11-09T09:58:12.044860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 2: Feature Engineering</span>\nThis notebook uses upper_shadow, lower_shadow, high_div_low, open_sub_close, seasonality/datetime features first shown in this notebook [here][1] and successfully used by julian3833 [here][2].\n\nAdditionally we can decide to use external data by changing the variables `INC2021`, `INC2020`, `INC2019`, `INC2018`, `INC2017`, `INCCOMP`, `INCSUPP` in the preceeding code section. These variables respectively indicate whether to load last year 2021 data and/or year 2020, 2019, 2018, 2017, the original, supplemented data. These datasets are discussed [here][3]\n\nConsider experimenting with different feature engineering and/or external data. The code to extract features out of the dataset is taken from julian3833' notebook [here][2]. Thank you julian3833, this is great work.\n\n[1]: https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\n[2]: https://www.kaggle.com/julian3833\n[3]: TBD","metadata":{}},{"cell_type":"code","source":"# Two features from the competition tutorial\ndef upper_shadow(df): return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df): return np.minimum(df['Close'], df['Open']) - df['Low']\n\n# A utility function to build features from the original df\ndef get_features(df):\n    df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feat['upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['lower_Shadow'] = lower_shadow(df_feat)\n    df_feat[\"high_div_low\"] = df_feat[\"High\"] / df_feat[\"Low\"]\n    df_feat[\"open_sub_close\"] = df_feat[\"Open\"] - df_feat[\"Close\"]\n    return df_feat","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:58:12.047462Z","iopub.execute_input":"2021-11-09T09:58:12.047781Z","iopub.status.idle":"2021-11-09T09:58:12.060324Z","shell.execute_reply.started":"2021-11-09T09:58:12.047740Z","shell.execute_reply":"2021-11-09T09:58:12.059236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 3: Build Model</span>\nThis is a common model architecute. Consider experimenting with different backbones, custom heads, losses, and optimizers.","metadata":{}},{"cell_type":"code","source":"def build_model(fold, dim = 11):\n    inp = tf.keras.layers.Input(shape=(dim))\n    x = inp\n    filters = WIDTH_NETS[fold]\n    for i in range(DEPTH_NETS[fold]):\n        x = tf.keras.layers.Dense(filters)(x)\n        x = tf.keras.layers.BatchNormalization()(x)\n        x = tf.keras.layers.Activation('relu')(x)\n        filters = filters // 2\n    x = tf.keras.layers.Dense(1, activation='linear')(x)\n    model = tf.keras.Model(inputs = inp, outputs = x)\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    model.compile(optimizer=opt, loss='mse')\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:58:12.063398Z","iopub.execute_input":"2021-11-09T09:58:12.063735Z","iopub.status.idle":"2021-11-09T09:58:12.074271Z","shell.execute_reply.started":"2021-11-09T09:58:12.063610Z","shell.execute_reply":"2021-11-09T09:58:12.072814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Step 4: Train Schedule</span>\nThis is a common train schedule in kaggle competitions. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow.","metadata":{}},{"cell_type":"code","source":"def get_lr_callback(batch_size = 8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n    def lrfn(epoch):\n        if epoch < lr_ramp_ep: lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n        elif epoch < lr_ramp_ep + lr_sus_ep: lr = lr_max\n        else: lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n        return lr\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:58:12.076805Z","iopub.execute_input":"2021-11-09T09:58:12.077577Z","iopub.status.idle":"2021-11-09T09:58:12.087944Z","shell.execute_reply.started":"2021-11-09T09:58:12.077536Z","shell.execute_reply":"2021-11-09T09:58:12.086464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass GroupTimeSeriesSplit(_BaseKFold):\n    \"\"\"Time Series cross-validator variant with non-overlapping groups.\n    Provides train/test indices to split time series data samples\n    that are observed at fixed time intervals according to a\n    third-party provided group.\n    In each split, test indices must be higher than before, and thus shuffling\n    in cross validator is inappropriate.\n    This cross-validation object is a variation of :class:`KFold`.\n    In the kth split, it returns first k folds as train set and the\n    (k+1)th fold as test set.\n    The same group will not appear in two different folds (the number of\n    distinct groups has to be at least equal to the number of folds).\n    Note that unlike standard cross-validation methods, successive\n    training sets are supersets of those that come before them.\n    Read more in the :ref:`User Guide <cross_validation>`.\n    Parameters\n    ----------\n    n_splits : int, default=5\n        Number of splits. Must be at least 2.\n    max_train_size : int, default=None\n        Maximum size for a single training set.\n    Examples\n    --------\n    >>> import numpy as np\n    >>> from sklearn.model_selection import GroupTimeSeriesSplit\n    >>> groups = np.array(['a', 'a', 'a', 'a', 'a', 'a',\\\n                           'b', 'b', 'b', 'b', 'b',\\\n                           'c', 'c', 'c', 'c',\\\n                           'd', 'd', 'd'])\n    >>> gtss = GroupTimeSeriesSplit(n_splits=3)\n    >>> for train_idx, test_idx in gtss.split(groups, groups=groups):\n    ...     print(\"TRAIN:\", train_idx, \"TEST:\", test_idx)\n    ...     print(\"TRAIN GROUP:\", groups[train_idx],\\\n                  \"TEST GROUP:\", groups[test_idx])\n    TRAIN: [0, 1, 2, 3, 4, 5] TEST: [6, 7, 8, 9, 10]\n    TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a']\\\n    TEST GROUP: ['b' 'b' 'b' 'b' 'b']\n    TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] TEST: [11, 12, 13, 14]\n    TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b']\\\n    TEST GROUP: ['c' 'c' 'c' 'c']\n    TRAIN: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\\\n    TEST: [15, 16, 17]\n    TRAIN GROUP: ['a' 'a' 'a' 'a' 'a' 'a' 'b' 'b' 'b' 'b' 'b' 'c' 'c' 'c' 'c']\\\n    TEST GROUP: ['d' 'd' 'd']\n    \"\"\"\n    @_deprecate_positional_args\n    def __init__(self,\n                 n_splits=5,\n                 *,\n                 max_train_size=None\n                 ):\n        super().__init__(n_splits, shuffle=False, random_state=None)\n        self.max_train_size = max_train_size\n\n    def split(self, X, y=None, groups=None):\n        \"\"\"Generate indices to split data into training and test set.\n        Parameters\n        ----------\n        X : array-like of shape (n_samples, n_features)\n            Training data, where n_samples is the number of samples\n            and n_features is the number of features.\n        y : array-like of shape (n_samples,)\n            Always ignored, exists for compatibility.\n        groups : array-like of shape (n_samples,)\n            Group labels for the samples used while splitting the dataset into\n            train/test set.\n        Yields\n        ------\n        train : ndarray\n            The training set indices for that split.\n        test : ndarray\n            The testing set indices for that split.\n        \"\"\"\n        if groups is None:\n            raise ValueError(\n                \"The 'groups' parameter should not be None\")\n        X, y, groups = indexable(X, y, groups)\n        n_samples = _num_samples(X)\n        n_splits = self.n_splits\n        n_folds = n_splits + 1\n        group_dict = {}\n        u, ind = np.unique(groups, return_index=True)\n        unique_groups = u[np.argsort(ind)]\n        n_samples = _num_samples(X)\n        n_groups = _num_samples(unique_groups)\n        for idx in np.arange(n_samples):\n            if (groups[idx] in group_dict):\n                group_dict[groups[idx]].append(idx)\n            else:\n                group_dict[groups[idx]] = [idx]\n        if n_folds > n_groups:\n            raise ValueError(\n                (\"Cannot have number of folds={0} greater than\"\n                 \" the number of groups={1}\").format(n_folds,\n                                                     n_groups))\n        group_test_size = n_groups // n_folds\n        group_test_starts = range(n_groups - n_splits * group_test_size,\n                                  n_groups, group_test_size)\n        for group_test_start in group_test_starts:\n            train_array = []\n            test_array = []\n            for train_group_idx in unique_groups[:group_test_start]:\n                train_array_tmp = group_dict[train_group_idx]\n                train_array = np.sort(np.unique(\n                                      np.concatenate((train_array,\n                                                      train_array_tmp)),\n                                      axis=None), axis=None)\n            train_end = train_array.size\n            if self.max_train_size and self.max_train_size < train_end:\n                train_array = train_array[train_end -\n                                          self.max_train_size:train_end]\n            for test_group_idx in unique_groups[group_test_start:\n                                                group_test_start +\n                                                group_test_size]:\n                test_array_tmp = group_dict[test_group_idx]\n                test_array = np.sort(np.unique(\n                                              np.concatenate((test_array,\n                                                              test_array_tmp)),\n                                     axis=None), axis=None)\n            yield [int(i) for i in train_array], [int(i) for i in test_array]\nimport numpy as np\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection._split import _BaseKFold, indexable, _num_samples\nfrom sklearn.utils.validation import _deprecate_positional_args\n\n# modified code for group gaps; source\n# https://github.com/getgaurav2/scikit-learn/blob/d4a3af5cc9da3a76f0266932644b884c99724c57/sklearn/model_selection/_split.py#L2243\nclass PurgedGroupTimeSeriesSplit(_BaseKFold):\n    \"\"\"Time Series cross-validator variant with non-overlapping groups.\n    Allows for a gap in groups to avoid potentially leaking info from\n    train into test if the model has windowed or lag features.\n    Provides train/test indices to split time series data samples\n    that are observed at fixed time intervals according to a\n    third-party provided group.\n    In each split, test indices must be higher than before, and thus shuffling\n    in cross validator is inappropriate.\n    This cross-validation object is a variation of :class:`KFold`.\n    In the kth split, it returns first k folds as train set and the\n    (k+1)th fold as test set.\n    The same group will not appear in two different folds (the number of\n    distinct groups has to be at least equal to the number of folds).\n    Note that unlike standard cross-validation methods, successive\n    training sets are supersets of those that come before them.\n    Read more in the :ref:`User Guide <cross_validation>`.\n    Parameters\n    ----------\n    n_splits : int, default=5\n        Number of splits. Must be at least 2.\n    max_train_group_size : int, default=Inf\n        Maximum group size for a single training set.\n    group_gap : int, default=None\n        Gap between train and test\n    max_test_group_size : int, default=Inf\n        We discard this number of groups from the end of each train split\n    \"\"\"\n\n    @_deprecate_positional_args\n    def __init__(self,\n                 n_splits=5,\n                 *,\n                 max_train_group_size=np.inf,\n                 max_test_group_size=np.inf,\n                 group_gap=None,\n                 verbose=False\n                 ):\n        super().__init__(n_splits, shuffle=False, random_state=None)\n        self.max_train_group_size = max_train_group_size\n        self.group_gap = group_gap\n        self.max_test_group_size = max_test_group_size\n        self.verbose = verbose\n\n    def split(self, X, y=None, groups=None):\n        \"\"\"Generate indices to split data into training and test set.\n        Parameters\n        ----------\n        X : array-like of shape (n_samples, n_features)\n            Training data, where n_samples is the number of samples\n            and n_features is the number of features.\n        y : array-like of shape (n_samples,)\n            Always ignored, exists for compatibility.\n        groups : array-like of shape (n_samples,)\n            Group labels for the samples used while splitting the dataset into\n            train/test set.\n        Yields\n        ------\n        train : ndarray\n            The training set indices for that split.\n        test : ndarray\n            The testing set indices for that split.\n        \"\"\"\n        if groups is None:\n            raise ValueError(\n                \"The 'groups' parameter should not be None\")\n        X, y, groups = indexable(X, y, groups)\n        n_samples = _num_samples(X)\n        n_splits = self.n_splits\n        group_gap = self.group_gap\n        max_test_group_size = self.max_test_group_size\n        max_train_group_size = self.max_train_group_size\n        n_folds = n_splits + 1\n        group_dict = {}\n        u, ind = np.unique(groups, return_index=True)\n        unique_groups = u[np.argsort(ind)]\n        n_samples = _num_samples(X)\n        n_groups = _num_samples(unique_groups)\n        for idx in np.arange(n_samples):\n            if (groups[idx] in group_dict):\n                group_dict[groups[idx]].append(idx)\n            else:\n                group_dict[groups[idx]] = [idx]\n        if n_folds > n_groups:\n            raise ValueError(\n                (\"Cannot have number of folds={0} greater than\"\n                 \" the number of groups={1}\").format(n_folds,\n                                                     n_groups))\n\n        group_test_size = min(n_groups // n_folds, max_test_group_size)\n        group_test_starts = range(n_groups - n_splits * group_test_size,\n                                  n_groups, group_test_size)\n        for group_test_start in group_test_starts:\n            train_array = []\n            test_array = []\n\n            group_st = max(0, group_test_start - group_gap - max_train_group_size)\n            for train_group_idx in unique_groups[group_st:(group_test_start - group_gap)]:\n                train_array_tmp = group_dict[train_group_idx]\n\n                train_array = np.sort(np.unique(\n                                      np.concatenate((train_array,\n                                                      train_array_tmp)),\n                                      axis=None), axis=None)\n\n            train_end = train_array.size\n\n            for test_group_idx in unique_groups[group_test_start:\n                                                group_test_start +\n                                                group_test_size]:\n                test_array_tmp = group_dict[test_group_idx]\n                test_array = np.sort(np.unique(\n                                              np.concatenate((test_array,\n                                                              test_array_tmp)),\n                                     axis=None), axis=None)\n\n            test_array  = test_array[group_gap:]\n\n\n            if self.verbose > 0:\n                    pass\n\n            yield [int(i) for i in train_array], [int(i) for i in test_array]","metadata":{"pycharm":{"name":"#%%\n"},"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-09T09:58:12.090317Z","iopub.execute_input":"2021-11-09T09:58:12.090918Z","iopub.status.idle":"2021-11-09T09:58:12.128790Z","shell.execute_reply.started":"2021-11-09T09:58:12.090700Z","shell.execute_reply":"2021-11-09T09:58:12.127261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Train Model</span>\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Each fold the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variable `VERBOSE`. The variable `VERBOSE=1 or 2` will display the training and validation loss for each epoch as text. ","metadata":{}},{"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 1\n\ndef get_Xy_and_model_for_asset(asset_id):\n    df = load_training_data_for_asset(asset_id)\n    df_proc = get_features(df)\n    df_proc['date'] = df['date'].copy()\n    df_proc['y'] = df['Target']\n    df_proc = df_proc.dropna(how=\"any\")\n    X = df_proc.drop(\"y\", axis=1)\n    y = df_proc[\"y\"]\n    groups = pd.factorize(X['date'].dt.day.astype(str) + '_' + X['date'].dt.month.astype(str) + '_' + X['date'].dt.year.astype(str))[0]\n    X = X.drop(columns = 'date')\n    oof_preds = np.zeros(len(X))\n    scores, models = [], []\n    \n    for fold, (train_idx, val_idx) in enumerate(PurgedGroupTimeSeriesSplit(n_splits = FOLDS, group_gap = GROUP_GAP).split(X, y, groups)):\n        # GET TRAINING, VALIDATION SET\n        x_train, x_val = X.iloc[train_idx], X.iloc[val_idx]\n        y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]\n\n        # DISPLAY FOLD INFO\n        if DEVICE == 'TPU':\n            if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n        print('#'*25); print('#### FOLD',fold+1)\n        print('#### Training WIDTH %s DEPTH %s | batch_size %s' % (WIDTH_NETS[fold], DEPTH_NETS[fold], BATCH_SIZES[fold]*REPLICAS))\n\n        # BUILD MODEL\n        # SAVE BEST MODEL EACH FOLD\n        sv = tf.keras.callbacks.ModelCheckpoint('fold-%i.h5' % fold, monitor = 'val_loss', verbose = 0, save_best_only = True, save_weights_only = True, mode = 'min', save_freq = 'epoch')\n        model = build_model(fold)\n        # TRAIN\n        print('Training...')\n        history = model.fit( x_train, y_train, epochs = EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], validation_data = (x_val, y_val), verbose=VERBOSE )\n        print('Loading best model...')\n        model.load_weights('fold-%i.h5' % fold)\n\n        # PREDICT OOF\n        print('Predicting OOF...')\n        pred = model.predict(x_val, verbose = VERBOSE)\n        models.append(model)\n\n        # REPORT RESULTS\n        try: mse = mean_squared_error(np.nan_to_num(y_val), np.nan_to_num(pred))\n        except: mse = 0.0\n        scores.append(mse)\n        oof_preds[val_idx] = pred[:,0]\n        print('#### FOLD %i OOF MSE %.3f' % (fold + 1, mse))\n\n    return scores, oof_preds, models\n\nmodels = {}\nscores = {}\noof_preds = {}\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    cur_scores, cur_oof_preds, cur_models = get_Xy_and_model_for_asset(asset_id)\n    scores[asset_id], oof_preds[asset_id], models[asset_id] = cur_scores, oof_preds, cur_models","metadata":{"execution":{"iopub.status.busy":"2021-11-09T09:58:12.132917Z","iopub.execute_input":"2021-11-09T09:58:12.133318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Calculate OOF MSE</span>\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions.","metadata":{}},{"cell_type":"code","source":"# COMPUTE OVERALL OOF MSE\nprint('Overall MEAN OOF MSE %s' % np.mean(list(scores.values())))\n\n# SAVE OOF TO DISK \nfor asset in oof_preds:\n    df_oof = pd.DataFrame(dict(asset_id = asset, oof_preds=oof_preds))\n    df_oof.to_csv(str(asset) + '_oof.csv',index=False)\n    df_oof.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\" id=\"codebook\">Submit To Kaggle</span>","metadata":{}},{"cell_type":"code","source":"all_df_test = []\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():\n        try:\n            if row['Asset_ID'] in models:\n                models = models[row['Asset_ID']]\n                x_test = get_features(row)\n                y_pred = np.mean(np.concatenate([np.expand_dims(model.predict(x_test.to_frame().T)[0], axis = 0) for model in models], axis = 0), axis = 0)\n            else: y_pred = 0.0\n        except: \n            y_pred = 0.0\n            traceback.print_exc()\n        df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n    all_df_test.append(df_test)\n    env.predict(df_pred)\n","metadata":{"pycharm":{"name":"#%%\n"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <span class=\"title-section w3-xxlarge\">References</span>\n\n<span id=\"f1\">1.</span> [Initial baseline notebook](https://www.kaggle.com/julian3833)<br>\n<span id=\"f2\">2.</span> [Competition tutorial](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)<br>\n<span id=\"f3\">3.</span> [Competition Overview](https://www.kaggle.com/c/g-research-crypto-forecasting/overview)</span><br>\n<span id=\"f4\">4.</span> [My Initial Ideas for this competition](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903)</span><br>\n<span id=\"f5\">5.</span> [My post notebook about cross validation](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit)</span><br>\n<span id=\"f5\">6.</span> [Chris original notebook from SIIM ISIC](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)</span><br>\n\n<span class=\"title-section w3-large w3-tag\">WORK IN PROGRESS! 🚧</span>","metadata":{}}]}