{ "id": 433805, "title": "MultilabelStratifiedKFold[DEMO]", "url": "/competitions/cafa-5-protein-function-prediction/discussion/433805", "author_name": "", "post_date": "2023-08-23T01:12:49.289084800Z", "votes": 3, "comment_count": 1, "views": 0, "content": "

MultilabelStratifiedKFold

\n

In this notebook, we are performing the following tasks:

\n
    \n

  1. Importing Required Libraries: Specifically, we are using IterativeStratification from the skmultilearn package.

    2. \n

  2. Loading the Dataset: Reading the Gene Ontology (GO) terms representing the functions of proteins as the target variable y (N, K).

    3. \n

  3. Implementing Multilabel Stratified K-Fold Cross-Validation: We conduct MultilabelStratifiedKFold on the multilabel targets (GO terms).

    4. \n

  4. Saving the Results of Cross-Validation: The indexes of the training and validation data for each fold are saved.

    5. \n

  5. Explanation of How to Load in Other Notebooks: Describing how to utilize the saved data in other notebooks.

    6. \n
\n
!pip install iterative-stratification\n os\n random\n pickle\n numpy  np \n pandas  pd\n matplotlib.pyplot  plt\n skmultilearn.model_selection  IterativeStratification\n\n\nseed = \naspect =  \nn_splits =  \nn_labels = \n\n ():\n    os.environ[] = (seed)\n    random.seed(seed)\n    np.random.seed(seed)\n\nset_seed(seed)\n\n ():\n    array = np.zeros((n_rows, n_cols), dtype=)\n    num_ones = (n_rows * n_cols * (percentage_ones / ))\n    indices = np.random.choice(n_rows * n_cols, num_ones, replace=)\n     index  indices:\n        row = index // n_cols\n        col = index % n_cols\n        array[row, col] = \n     array\n\n ():\n     aspect == :\n        y = np.load(, allow_pickle=)\n     aspect == :\n        y = np.load(, allow_pickle=)\n     aspect == :\n        y = np.load(, allow_pickle=)\n     aspect == :\n        y = generate_array(n_rows = , n_cols = )\n    :\n         NotImplementedError()\n     y\n\n :\n     ():\n        self.n_splits = n_splits\n        self.order = order\n        self.kfold = IterativeStratification(n_splits=n_splits, order=order)\n\n     ():\n        splits = []\n         train_index, valid_index  self.kfold.split(X, y):\n            splits.append((train_index, valid_index))\n         splits\n\n\nos.makedirs(, exist_ok=)\nos.makedirs(, exist_ok=)\n\ny = make_y(aspect)  \nX = np.zeros_like(y)  \n\n\nmlskf = MultiLabelStratifiedKFold(n_splits=n_splits, order=)\n\nfolds_indexes = []\nsplits = mlskf.split(X[:, :n_labels], y[:, :n_labels])\n train_index, valid_index  splits:\n    ()\n    ()\n    ()\n\n    fold_tuple = (train_index.tolist(), valid_index.tolist())\n    folds_indexes.append(fold_tuple)\n\n\n (, )  f:\n    pickle.dump(folds_indexes, f)\n\n\ncounts = np.zeros(y.shape[])\n tr_idx, vl_idx  folds_indexes:\n    counts[vl_idx] += \nplt.scatter(np.arange((counts)), counts)\nplt.show()\n
\n

How to load from other notebooks

\n
 pickle\n\naspect = \nn_splits = \nn_labels = \n\n\npath = \n\n\n (path, )  f:\n    loaded_folds_indexes = pickle.load(f)\n\n train_index, valid_index  loaded_folds_indexes:\n    ()\n    ()\n    ()\n
", "messages": [ { "id": "2403956", "postDate": "08/23/2023 01:12:49", "content": "

MultilabelStratifiedKFold

\n

In this notebook, we are performing the following tasks:

\n
    \n

  1. Importing Required Libraries: Specifically, we are using IterativeStratification from the skmultilearn package.

    2. \n

  2. Loading the Dataset: Reading the Gene Ontology (GO) terms representing the functions of proteins as the target variable y (N, K).

    3. \n

  3. Implementing Multilabel Stratified K-Fold Cross-Validation: We conduct MultilabelStratifiedKFold on the multilabel targets (GO terms).

    4. \n

  4. Saving the Results of Cross-Validation: The indexes of the training and validation data for each fold are saved.

    5. \n

  5. Explanation of How to Load in Other Notebooks: Describing how to utilize the saved data in other notebooks.

    6. \n
\n
!pip install iterative-stratification\n os\n random\n pickle\n numpy  np \n pandas  pd\n matplotlib.pyplot  plt\n skmultilearn.model_selection  IterativeStratification\n\n\nseed = \naspect =  \nn_splits =  \nn_labels = \n\n ():\n    os.environ[] = (seed)\n    random.seed(seed)\n    np.random.seed(seed)\n\nset_seed(seed)\n\n ():\n    array = np.zeros((n_rows, n_cols), dtype=)\n    num_ones = (n_rows * n_cols * (percentage_ones / ))\n    indices = np.random.choice(n_rows * n_cols, num_ones, replace=)\n     index  indices:\n        row = index // n_cols\n        col = index % n_cols\n        array[row, col] = \n     array\n\n ():\n     aspect == :\n        y = np.load(, allow_pickle=)\n     aspect == :\n        y = np.load(, allow_pickle=)\n     aspect == :\n        y = np.load(, allow_pickle=)\n     aspect == :\n        y = generate_array(n_rows = , n_cols = )\n    :\n         NotImplementedError()\n     y\n\n :\n     ():\n        self.n_splits = n_splits\n        self.order = order\n        self.kfold = IterativeStratification(n_splits=n_splits, order=order)\n\n     ():\n        splits = []\n         train_index, valid_index  self.kfold.split(X, y):\n            splits.append((train_index, valid_index))\n         splits\n\n\nos.makedirs(, exist_ok=)\nos.makedirs(, exist_ok=)\n\ny = make_y(aspect)  \nX = np.zeros_like(y)  \n\n\nmlskf = MultiLabelStratifiedKFold(n_splits=n_splits, order=)\n\nfolds_indexes = []\nsplits = mlskf.split(X[:, :n_labels], y[:, :n_labels])\n train_index, valid_index  splits:\n    ()\n    ()\n    ()\n\n    fold_tuple = (train_index.tolist(), valid_index.tolist())\n    folds_indexes.append(fold_tuple)\n\n\n (, )  f:\n    pickle.dump(folds_indexes, f)\n\n\ncounts = np.zeros(y.shape[])\n tr_idx, vl_idx  folds_indexes:\n    counts[vl_idx] += \nplt.scatter(np.arange((counts)), counts)\nplt.show()\n
\n

How to load from other notebooks

\n
 pickle\n\naspect = \nn_splits = \nn_labels = \n\n\npath = \n\n\n (path, )  f:\n    loaded_folds_indexes = pickle.load(f)\n\n train_index, valid_index  loaded_folds_indexes:\n    ()\n    ()\n    ()\n
", "rawMarkdown": "# MultilabelStratifiedKFold\nIn this notebook, we are performing the following tasks:\n\n1. **Importing Required Libraries**: Specifically, we are using `IterativeStratification` from the `skmultilearn` package.\n\n2. **Loading the Dataset**: Reading the Gene Ontology (GO) terms representing the functions of proteins as the target variable y (N, K).\n\n3. **Implementing Multilabel Stratified K-Fold Cross-Validation**: We conduct MultilabelStratifiedKFold on the multilabel targets (GO terms).\n\n4. **Saving the Results of Cross-Validation**: The indexes of the training and validation data for each fold are saved.\n\n5. **Explanation of How to Load in Other Notebooks**: Describing how to utilize the saved data in other notebooks.\n\n```python\n!pip install iterative-stratification\nimport os\nimport random\nimport pickle\nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom skmultilearn.model_selection import IterativeStratification\n\n# parameters\nseed = 19951015\naspect = \"DEMO\" # \"BP\", \"CC\", \"MF\"\nn_splits = 5 \nn_labels = 300\n\ndef set_seed(seed):\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n\nset_seed(seed)\n\ndef generate_array(n_rows, n_cols, percentage_ones=1):\n array = np.zeros((n_rows, n_cols), dtype=int)\n num_ones = int(n_rows * n_cols * (percentage_ones / 100))\n indices = np.random.choice(n_rows * n_cols, num_ones, replace=False)\n for index in indices:\n row = index // n_cols\n col = index % n_cols\n array[row, col] = 1\n return array\n\ndef make_y(aspect):\n if aspect == \"BP\":\n y = np.load(\"/kaggle/input/full_augumented_denoised_y/Y_BP_5000.npy\", allow_pickle=True)\n elif aspect == \"CC\":\n y = np.load(\"/kaggle/input/full_augumented_denoised_y/Y_CC_2079.npy\", allow_pickle=True)\n elif aspect == \"MF\":\n y = np.load(\"/kaggle/input/full_augumented_denoised_y/Y_MF_4609.npy\", allow_pickle=True)\n elif aspect == \"DEMO\":\n y = generate_array(n_rows = 142264, n_cols = 300)\n else:\n raise NotImplementedError('aspect is chosen from ALL / BP / CC / MF')\n return y\n\nclass MultiLabelStratifiedKFold:\n def __init__(self, n_splits, order):\n self.n_splits = n_splits\n self.order = order\n self.kfold = IterativeStratification(n_splits=n_splits, order=order)\n\n def split(self, X, y):\n splits = []\n for train_index, valid_index in self.kfold.split(X, y):\n splits.append((train_index, valid_index))\n return splits\n\n# Ensure the adjusted directory exists\nos.makedirs(f'../working/{aspect}', exist_ok=True)\nos.makedirs(f'../working/{aspect}/{n_splits}', exist_ok=True)\n\ny = make_y(aspect) \nX = np.zeros_like(y) \n\n# Perform MultilabelStratifiedKFold and save results\nmlskf = MultiLabelStratifiedKFold(n_splits=n_splits, order=1)\n\nfolds_indexes = []\nsplits = mlskf.split(X[:, :n_labels], y[:, :n_labels])\nfor train_index, valid_index in splits:\n print(f'## train_index{train_index}')\n print(f'## valid_index{valid_index}')\n print()\n\n fold_tuple = (train_index.tolist(), valid_index.tolist())\n folds_indexes.append(fold_tuple)\n\n# save indexes to pkl\nwith open(f'../working/{aspect}/{n_splits}/folds_indexes_{n_labels}.pkl', 'wb') as f:\n pickle.dump(folds_indexes, f)\n\n# Verify that every index appears exactly once\ncounts = np.zeros(y.shape[0])\nfor tr_idx, vl_idx in folds_indexes:\n counts[vl_idx] += 1\nplt.scatter(np.arange(len(counts)), counts)\nplt.show()\n```\n\n\n# How to load from other notebooks\n```python\n\nimport pickle\n\naspect = \"DEMO\"\nn_splits = 5\nn_labels = 300\n\n# Define the path and file name according to yours\npath = f'../working/{aspect}/{n_splits}/folds_indexes_{n_labels}.pkl'\n\n# Open the file in read-binary mode and load the data\nwith open(path, 'rb') as f:\n loaded_folds_indexes = pickle.load(f)\n \nfor train_index, valid_index in loaded_folds_indexes:\n print(f'## train_index{train_index[:5]}')\n print(f'## valid_index{valid_index[:5]}')\n print()\n\n```", "votes": null }, { "id": "2403972", "postDate": "08/23/2023 01:33:51", "content": "

I have finally created the folds_indexes for BP, CC, MF with k=4, 5, 6.
\nhttps://www.kaggle.com/datasets/isakatsuyoshi/folds-indexes

", "rawMarkdown": "I have finally created the folds_indexes for BP, CC, MF with k=4, 5, 6.\nhttps://www.kaggle.com/datasets/isakatsuyoshi/folds-indexes", "votes": null } ], "comments": [ { "id": 2403972, "author_name": "isakatsuyoshi", "author_url": "", "post_date": "08/23/2023 01:33:51", "content": "

I have finally created the folds_indexes for BP, CC, MF with k=4, 5, 6.
\nhttps://www.kaggle.com/datasets/isakatsuyoshi/folds-indexes

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2403956": "# MultilabelStratifiedKFold\nIn this notebook, we are performing the following tasks:\n\n1. **Importing Required Libraries**: Specifically, we are using `IterativeStratification` from the `skmultilearn` package.\n\n2. **Loading the Dataset**: Reading the Gene Ontology (GO) terms representing the functions of proteins as the target variable y (N, K).\n\n3. **Implementing Multilabel Stratified K-Fold Cross-Validation**: We conduct MultilabelStratifiedKFold on the multilabel targets (GO terms).\n\n4. **Saving the Results of Cross-Validation**: The indexes of the training and validation data for each fold are saved.\n\n5. **Explanation of How to Load in Other Notebooks**: Describing how to utilize the saved data in other notebooks.\n\n```python\n!pip install iterative-stratification\nimport os\nimport random\nimport pickle\nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom skmultilearn.model_selection import IterativeStratification\n\n# parameters\nseed = 19951015\naspect = \"DEMO\" # \"BP\", \"CC\", \"MF\"\nn_splits = 5 \nn_labels = 300\n\ndef set_seed(seed):\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n\nset_seed(seed)\n\ndef generate_array(n_rows, n_cols, percentage_ones=1):\n array = np.zeros((n_rows, n_cols), dtype=int)\n num_ones = int(n_rows * n_cols * (percentage_ones / 100))\n indices = np.random.choice(n_rows * n_cols, num_ones, replace=False)\n for index in indices:\n row = index // n_cols\n col = index % n_cols\n array[row, col] = 1\n return array\n\ndef make_y(aspect):\n if aspect == \"BP\":\n y = np.load(\"/kaggle/input/full_augumented_denoised_y/Y_BP_5000.npy\", allow_pickle=True)\n elif aspect == \"CC\":\n y = np.load(\"/kaggle/input/full_augumented_denoised_y/Y_CC_2079.npy\", allow_pickle=True)\n elif aspect == \"MF\":\n y = np.load(\"/kaggle/input/full_augumented_denoised_y/Y_MF_4609.npy\", allow_pickle=True)\n elif aspect == \"DEMO\":\n y = generate_array(n_rows = 142264, n_cols = 300)\n else:\n raise NotImplementedError('aspect is chosen from ALL / BP / CC / MF')\n return y\n\nclass MultiLabelStratifiedKFold:\n def __init__(self, n_splits, order):\n self.n_splits = n_splits\n self.order = order\n self.kfold = IterativeStratification(n_splits=n_splits, order=order)\n\n def split(self, X, y):\n splits = []\n for train_index, valid_index in self.kfold.split(X, y):\n splits.append((train_index, valid_index))\n return splits\n\n# Ensure the adjusted directory exists\nos.makedirs(f'../working/{aspect}', exist_ok=True)\nos.makedirs(f'../working/{aspect}/{n_splits}', exist_ok=True)\n\ny = make_y(aspect) \nX = np.zeros_like(y) \n\n# Perform MultilabelStratifiedKFold and save results\nmlskf = MultiLabelStratifiedKFold(n_splits=n_splits, order=1)\n\nfolds_indexes = []\nsplits = mlskf.split(X[:, :n_labels], y[:, :n_labels])\nfor train_index, valid_index in splits:\n print(f'## train_index{train_index}')\n print(f'## valid_index{valid_index}')\n print()\n\n fold_tuple = (train_index.tolist(), valid_index.tolist())\n folds_indexes.append(fold_tuple)\n\n# save indexes to pkl\nwith open(f'../working/{aspect}/{n_splits}/folds_indexes_{n_labels}.pkl', 'wb') as f:\n pickle.dump(folds_indexes, f)\n\n# Verify that every index appears exactly once\ncounts = np.zeros(y.shape[0])\nfor tr_idx, vl_idx in folds_indexes:\n counts[vl_idx] += 1\nplt.scatter(np.arange(len(counts)), counts)\nplt.show()\n```\n\n\n# How to load from other notebooks\n```python\n\nimport pickle\n\naspect = \"DEMO\"\nn_splits = 5\nn_labels = 300\n\n# Define the path and file name according to yours\npath = f'../working/{aspect}/{n_splits}/folds_indexes_{n_labels}.pkl'\n\n# Open the file in read-binary mode and load the data\nwith open(path, 'rb') as f:\n loaded_folds_indexes = pickle.load(f)\n \nfor train_index, valid_index in loaded_folds_indexes:\n print(f'## train_index{train_index[:5]}')\n print(f'## valid_index{valid_index[:5]}')\n print()\n\n```", "2403972": "I have finally created the folds_indexes for BP, CC, MF with k=4, 5, 6.\nhttps://www.kaggle.com/datasets/isakatsuyoshi/folds-indexes" }, "source": "meta" }