{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nI implemented the new feature for scikit-learn for CPCV and related class.  \nThat is:\n- PurgedKFold\n- CombinationalKFold\n- CombinationalPurgedKFold\n\nFor CPCV, see https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/302445  \nTheare many sources such as blog, author's book and so on to learn this concept.  \nSo I don't write the explanation here.  ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## About code\nI have done implementation and am preparing pull request to scikit-learn.  \nBut I am not sure this feature is accepted from schikit-learn community.  \nI think scikit-learn is for pure science.  \nThe feature is relatively for finantial matter.  \nSo it is likely that this feature is not merged.  \n\nFor now, my pull request is draft.  \nBut the source code is available from https://github.com/scikit-learn/scikit-learn/pull/22273  \nThis is not well-documented about class and method for now.  \nI will add more documentation later.  \n\n**And I would highly appreciate if you could add star [the forked repo](https://github.com/r-matsuzaka/scikit-learn).**  \n**That's because I spent a lot of time for it and it will be my motivation for coding.**  ","metadata":{}},{"cell_type":"markdown","source":"## Install forked scikit-learn","metadata":{}},{"cell_type":"code","source":"!pip install git+https://github.com/r-matsuzaka/scikit-learn.git@feat_cpcv","metadata":{"execution":{"iopub.status.busy":"2022-01-30T02:36:32.330973Z","iopub.execute_input":"2022-01-30T02:36:32.331283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import TimeSeriesSplit,KFold,GroupKFold,CombinationalKFold,CombinationalPurgedKFold,PurgedKFold","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**CombinationalKFold, CombinationalPurgedKFold, PurgedKFold** is new feature which is implemented by me.","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.patches import Patch","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ref:https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html\n\ndef plot_cv_indices(cv, X, y, y_know_futures = np.empty(0),lw=10):\n    \"\"\"Create a sample plot for indices of a cross-validation object.\"\"\"\n    _, ax = plt.subplots()\n    n_splits = cv.n_splits\n    cmap_cv = plt.cm.coolwarm\n    \n    # Generate the training/testing visualizations for each CV split\n    xrange_max = 0\n    \n    def visualise(tr,tt):\n        xrange_max = 0\n        ax.scatter(\n            range(len(indices)),\n            [ii + 0.5] * len(indices),\n            c=indices,\n            marker=\"_\",\n            lw=lw,\n            cmap=cmap_cv,\n            vmin=-0.2,\n            vmax=1.2,\n        )\n        if xrange_max < len(tr) + len(tt):\n            xrange_max = len(tr) + len(tt)\n            \n        return xrange_max\n        \n    \n    \n    if y_know_futures.size == 0:\n        for ii, (tr, tt) in enumerate(cv.split(X=X, y=y)):\n            # Fill in indices with the training/test groups\n            indices = np.array([np.nan] * len(X))\n            indices[tt] = 1\n            indices[tr] = 0\n            \n            xrange_max = visualise(tr,tt)\n            \n    else:\n        for ii, (tr, tt,overlap_index_arrray) in enumerate(cv.split(X=X, y=y, y_know_futures = y_know_futures)):\n            # Fill in indices with the training/test groups\n            indices = np.array([np.nan] * len(X))\n            #print(tr, tt,overlap_index_arrray)\n            indices[tt] = 1\n            indices[tr] = 0\n            indices[overlap_index_arrray] = 0.5\n            \n            xrange_max = visualise(tr,tt)\n\n    yticklabels = list(range(n_splits))\n    ax.set(\n        yticks=np.arange(n_splits) + 0.5,\n        yticklabels=yticklabels,\n        xlabel=\"Sample index\",\n        ylabel=\"CV iteration\",\n        ylim=[n_splits, -0.2],\n        xlim=[0, xrange_max],\n    )\n\n    ax.set_title(\"{}\".format(type(cv).__name__), fontsize=15)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TimeSeriesSplit","metadata":{}},{"cell_type":"code","source":"X = np.array([10+i for i in range(10)])\n\ny = np.array([i for i in range(10)])\n\nn_splits=5\ntscv = TimeSeriesSplit(n_splits)\nprint(tscv)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for train_index, test_index in tscv.split(X):\n    print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n    X_train, X_test = X[train_index], X[test_index]\n    y_train, y_test = y[train_index], y[test_index]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_cv_indices(tscv, X, y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## KFold","metadata":{"execution":{"iopub.status.busy":"2022-01-23T05:04:35.376438Z","iopub.execute_input":"2022-01-23T05:04:35.377131Z","iopub.status.idle":"2022-01-23T05:04:35.380749Z","shell.execute_reply.started":"2022-01-23T05:04:35.377089Z","shell.execute_reply":"2022-01-23T05:04:35.379938Z"}}},{"cell_type":"code","source":"n_splits=5\nkfold = KFold(n_splits)\nprint(kfold)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for train_index, test_index in kfold.split(X):\n    print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n    X_train, X_test = X[train_index], X[test_index]\n    y_train, y_test = y[train_index], y[test_index]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_cv_indices(kfold, X, y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only 10 data looks sparse.  \nSo it is better to use more than 100 data for visualization.  ","metadata":{"execution":{"iopub.status.busy":"2022-01-23T05:06:48.747964Z","iopub.execute_input":"2022-01-23T05:06:48.748856Z","iopub.status.idle":"2022-01-23T05:06:48.755713Z","shell.execute_reply.started":"2022-01-23T05:06:48.748803Z","shell.execute_reply":"2022-01-23T05:06:48.754533Z"}}},{"cell_type":"code","source":"np.random.seed(1338)\n\nn_splits = 4\n\n# Generate the class/group data\nn_points = 100\nX = np.random.randn(n_points, 10)\npercentiles_classes = [0.1, 0.3, 0.6]\ny = np.hstack([[ii] * int(n_points * perc) for ii, perc in enumerate(percentiles_classes)])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv = KFold(n_splits)\nprint(cv)\nplot_cv_indices(cv, X, y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## PurgedKFold","metadata":{}},{"cell_type":"markdown","source":"Let's say if target y depends on future value or know future.  \nThen normal Kfold results in leakage.  \nSo purging should be done.  \n\n`y_know_futures` is the index of the future data that target variables knows.  \nFor example, 0 means that target only know currenat states, but 2 means that target knows 2 steps later future.  ","metadata":{}},{"cell_type":"code","source":"import random\nrandom.seed(2022)\n\ny_know_futures = np.array([random.randint(0,5) for yi in y ])\ny_know_futures","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For testing, `y_know_futures` is randomly created.  ","metadata":{}},{"cell_type":"code","source":"purged_kfold = PurgedKFold(n_splits)\nprint(purged_kfold)\n\nfor train_index, test_index, overlap_index in purged_kfold.split(X, y,y_know_futures):\n    print(\"TRAIN:\", train_index, \"TEST:\", test_index, \"OVERLAP:\", overlap_index)\n    X_train, X_test = X[train_index], X[test_index]\n    y_train, y_test = y[train_index], y[test_index]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_cv_indices(purged_kfold, X, y,y_know_futures = y_know_futures)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"gray parts is overlap data.","metadata":{}},{"cell_type":"markdown","source":"## GroupKFold","metadata":{}},{"cell_type":"code","source":"def plot_groupcv_indices(cv, X, y, y_know_futures = np.empty(0),groups = np.empty(0),lw=10):\n    \"\"\"Create a sample plot for indices of a cross-validation object.\"\"\"\n    _, ax = plt.subplots()\n    n_splits = cv.n_splits\n    cmap_cv = plt.cm.coolwarm\n    \n    # Generate the training/testing visualizations for each CV split\n    xrange_max = 0\n    \n    def visualise(tr,tt):\n        xrange_max = 0\n        ax.scatter(\n            range(len(indices)),\n            [ii + 0.5] * len(indices),\n            c=indices,\n            marker=\"_\",\n            lw=lw,\n            cmap=cmap_cv,\n            vmin=-0.2,\n            vmax=1.2,\n        )\n        if xrange_max < len(tr) + len(tt):\n            xrange_max = len(tr) + len(tt)\n            \n        return xrange_max\n        \n    \n    \n    #if y_know_futures.size == 0:\n    for ii, (tr, tt) in enumerate(cv.split(X=X, y=y,groups=groups)):\n        # Fill in indices with the training/test groups\n        indices = np.array([np.nan] * len(X))\n        indices[tt] = 1\n        indices[tr] = 0\n\n        xrange_max = visualise(tr,tt)\n\n    yticklabels = list(range(n_splits))\n    ax.set(\n        yticks=np.arange(n_splits) + 0.5,\n        yticklabels=yticklabels,\n        xlabel=\"Sample index\",\n        ylabel=\"CV iteration\",\n        ylim=[n_splits, -0.2],\n        xlim=[0, xrange_max],\n    )\n\n    ax.set_title(\"{}\".format(type(cv).__name__), fontsize=15)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evenly spaced groups repeated once\ngroups = np.hstack([[ii] * 10 for ii in range(10)])\n\ngroup_kfold = GroupKFold(n_splits)\n\nplot_groupcv_indices(group_kfold, X, y,groups = groups)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_splits = 5\ngroup_kfold = GroupKFold(n_splits)\nplot_groupcv_indices(group_kfold, X, y,groups = groups)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CombinationalKFold","metadata":{"execution":{"iopub.status.busy":"2022-01-23T05:17:46.346491Z","iopub.execute_input":"2022-01-23T05:17:46.347132Z","iopub.status.idle":"2022-01-23T05:17:46.350545Z","shell.execute_reply.started":"2022-01-23T05:17:46.347091Z","shell.execute_reply":"2022-01-23T05:17:46.349943Z"}}},{"cell_type":"code","source":"groups = np.hstack([[ii] * 20 for ii in range(5)])\ngroups","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"group_ids = np.unique(groups)\ngroup_ids","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_combcv_indices(cv, X, y,lw=10):\n    \"\"\"Create a sample plot for indices of a cross-validation object.\"\"\"\n    _, ax = plt.subplots()\n    n_splits = cv.n_splits\n    n_combs = len(cv.test_comb_list)\n    cmap_cv = plt.cm.coolwarm\n    \n    # Generate the training/testing visualizations for each CV split\n    xrange_max = 0\n    \n    def visualise(tr,tt):\n        xrange_max = 0\n        ax.scatter(\n            range(len(indices)),\n            [ii + 0.5] * len(indices),\n            c=indices,\n            marker=\"_\",\n            lw=lw,\n            cmap=cmap_cv,\n            vmin=-0.2,\n            vmax=1.2,\n        )\n        if xrange_max < len(tr) + len(tt):\n            xrange_max = len(tr) + len(tt)\n            \n        return xrange_max\n        \n    for ii, (tr, tt) in enumerate(cv.split(X=X, y=y)):\n        # Fill in indices with the training/test groups\n        indices = np.array([np.nan] * len(X))\n        indices[tt] = 1\n        indices[tr] = 0\n\n        xrange_max = visualise(tr,tt)\n\n    yticklabels = list(range(n_combs))\n    ax.set(\n        yticks=np.arange(n_combs) + 0.5,\n        yticklabels=yticklabels,\n        xlabel=\"Sample index\",\n        ylabel=\"CV iteration\",\n        ylim=[n_combs, -0.2],\n        xlim=[0, xrange_max],\n    )\n\n    ax.set_title(\"{}\".format(type(cv).__name__), fontsize=15)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"comb_kfold = CombinationalKFold(groups, n_splits=5, test_group_choice=2)\nfor train_index, test_index in comb_kfold.split(X, y):\n    print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n    X_train, X_test = X[train_index], X[test_index]\n    y_train, y_test = y[train_index], y[test_index]\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"  plot_combcv_indices(comb_kfold, X, y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CombinationalPurgedKFold","metadata":{}},{"cell_type":"markdown","source":"For now `y_know_futures` is only available, so I will add `X_know_pasts`.","metadata":{}},{"cell_type":"code","source":"y_know_futures = np.array([random.randint(0,5) for yi in y ])\ny_know_futures","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"comb_purge_kfold = CombinationalPurgedKFold(groups, n_splits=5, test_group_choice=2)\n\n\nfor train_index, test_index, overlap_index in comb_purge_kfold.split(X, y,y_know_futures):\n    print(\"TRAIN:\", train_index, \"TEST:\", test_index, \"OVERLAP:\", overlap_index)\n    X_train, X_test = X[train_index], X[test_index]\n    y_train, y_test = y[train_index], y[test_index]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_combpurgecv_indices(cv, X, y, y_know_futures = np.empty(0),lw=10):\n    \"\"\"Create a sample plot for indices of a cross-validation object.\"\"\"\n    _, ax = plt.subplots()\n    n_splits = cv.n_splits\n    n_combs = len(cv.test_comb_list)\n    cmap_cv = plt.cm.coolwarm\n    \n    # Generate the training/testing visualizations for each CV split\n    xrange_max = 0\n    \n    def visualise(tr,tt):\n        xrange_max = 0\n        ax.scatter(\n            range(len(indices)),\n            [ii + 0.5] * len(indices),\n            c=indices,\n            marker=\"_\",\n            lw=lw,\n            cmap=cmap_cv,\n            vmin=-0.2,\n            vmax=1.2,\n        )\n        if xrange_max < len(tr) + len(tt):\n            xrange_max = len(tr) + len(tt)\n            \n        return xrange_max\n    \n\n    for ii, (tr, tt,overlap_index_arrray) in enumerate(cv.split(X=X, y=y, y_know_futures = y_know_futures)):\n        # Fill in indices with the training/test groups\n        indices = np.array([np.nan] * len(X))\n        #print(tr, tt,overlap_index_arrray)\n        indices[tt] = 1\n        indices[tr] = 0\n        indices[overlap_index_arrray] = 0.5\n\n        xrange_max = visualise(tr,tt)\n\n    yticklabels = list(range(n_combs))\n    ax.set(\n        yticks=np.arange(n_combs) + 0.5,\n        yticklabels=yticklabels,\n        xlabel=\"Sample index\",\n        ylabel=\"CV iteration\",\n        ylim=[n_combs, -0.2],\n        xlim=[0, xrange_max],\n    )\n\n    ax.set_title(\"{}\".format(type(cv).__name__), fontsize=15)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_combpurgecv_indices(comb_purge_kfold, X, y, y_know_futures = y_know_futures)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's it.  \nThank you for reading.  \n\n**If you find this useful, please upvote this notebook and add star to [my forked repo](https://github.com/r-matsuzaka/scikit-learn).**","metadata":{}}]}