{
  "id": 434656,
  "title": "Sophisticated Data Preprocessing for the CAFA5 Dataset",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/434656",
  "author_name": "ISAKA Tsuyoshi",
  "post_date": "2023-08-26T01:40:34.409000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>Sophisticated Data Preprocessing for the CAFA5 Dataset</h1>\n<p>This notebook is designed to perform a comprehensive data preprocessing operation aimed at facilitating seamless analysis and modeling for the Kaggle competition focused on the CAFA5 dataset. Below is an outline of the steps involved:</p>\n<h2>Steps</h2>\n<ol>\n<li><p><strong>Changing Data Types</strong>: We load the datasets and convert the data types of IDs to Unicode strings with a maximum of 10 characters (\"&lt;U10\"). The feature vectors are converted to single-precision floating point numbers (float32).</p></li>\n<li><p><strong>Extracting Non-duplicate IDs</strong>: We remove duplicate IDs present in the test dataset.</p>\n<ul>\n<li><strong>Note</strong>: It is presumed that the test dataset contains only one pair of duplicate IDs.</li>\n<li><strong>Note</strong>: There are initially 141,865 IDs, which reduce to 141,864 after removing duplicates.</li></ul></li>\n<li><p><strong>Sorting the Data</strong>: Both the training and test datasets are sorted by their IDs. This makes subsequent analysis and modeling more efficient.</p></li>\n<li><p><strong>Data Saving</strong>: After the above preprocessing steps, the data is saved as new NumPy files for future use.</p></li>\n</ol>\n<h2>File Structure</h2>\n<ul>\n<li><strong>train_ids.npy</strong>: IDs in the training dataset</li>\n<li><strong>test_ids.npy</strong>: IDs in the test dataset</li>\n<li><strong>train_embeds.npy</strong>: Feature vectors (embeddings) for the training dataset</li>\n<li><strong>test_embeds.npy</strong>: Feature vectors (embeddings) for the test dataset</li>\n</ul>\n<p>This notebook serves as a foundation for a streamlined approach to further analysis and modeling, optimizing for the unique challenges posed by the CAFA5 dataset.</p>\n<pre><code> numpy  np\n\n :\n     ():\n        self.train_ids = np.load(train_ids_path).astype()\n        self.test_ids = np.load(test_ids_path).astype()\n        self.train_data = np.load(train_data_path).astype(np.float32)\n        self.test_data = np.load(test_data_path).astype(np.float32)\n\n     ():\n        unique_ids, indices = np.unique(self.test_ids, return_index=)\n        all_indices = np.arange(self.test_ids.shape[])\n        duplicate_indices = np.setdiff1d(all_indices, indices)\n        self.non_duplicate_indices = np.setdiff1d(all_indices, duplicate_indices)\n\n     ():\n        self.test_ids = self.test_ids[self.non_duplicate_indices]\n        self.test_data = self.test_data[self.non_duplicate_indices]\n\n        argsort_train = np.argsort(self.train_ids)\n        argsort_test = np.argsort(self.test_ids)\n\n        self.train_ids = self.train_ids[argsort_train]\n        self.test_ids = self.test_ids[argsort_test]\n        self.train_data = self.train_data[argsort_train]\n        self.test_data = self.test_data[argsort_test]\n\n     ():\n        np.save(, self.train_ids)\n        np.save(, self.test_ids)\n        np.save(, self.train_data)\n        np.save(, self.test_data)\n\n __name__ == :\n    \n    train_ids_path = \n    test_ids_path = \n    train_data_path = \n    test_data_path = \n\n    processor = DataProcessor(train_ids_path, test_ids_path, train_data_path, test_data_path)\n    processor.find_non_duplicate_ids()\n    processor.preprocess_data()\n    processor.save_data()\n</code></pre>\n<p>By executing this notebook, you will be able to obtain a dataset that has been appropriately preprocessed<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fd27a976b5de358e370b1b2805c00aa8c%2F2023-08-26%20103043.png?generation=1693013498242672&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": 2408982,
      "postDate": "2023-08-26T01:40:34.410Z",
      "content": "<h1>Sophisticated Data Preprocessing for the CAFA5 Dataset</h1>\n<p>This notebook is designed to perform a comprehensive data preprocessing operation aimed at facilitating seamless analysis and modeling for the Kaggle competition focused on the CAFA5 dataset. Below is an outline of the steps involved:</p>\n<h2>Steps</h2>\n<ol>\n<li><p><strong>Changing Data Types</strong>: We load the datasets and convert the data types of IDs to Unicode strings with a maximum of 10 characters (\"&lt;U10\"). The feature vectors are converted to single-precision floating point numbers (float32).</p></li>\n<li><p><strong>Extracting Non-duplicate IDs</strong>: We remove duplicate IDs present in the test dataset.</p>\n<ul>\n<li><strong>Note</strong>: It is presumed that the test dataset contains only one pair of duplicate IDs.</li>\n<li><strong>Note</strong>: There are initially 141,865 IDs, which reduce to 141,864 after removing duplicates.</li></ul></li>\n<li><p><strong>Sorting the Data</strong>: Both the training and test datasets are sorted by their IDs. This makes subsequent analysis and modeling more efficient.</p></li>\n<li><p><strong>Data Saving</strong>: After the above preprocessing steps, the data is saved as new NumPy files for future use.</p></li>\n</ol>\n<h2>File Structure</h2>\n<ul>\n<li><strong>train_ids.npy</strong>: IDs in the training dataset</li>\n<li><strong>test_ids.npy</strong>: IDs in the test dataset</li>\n<li><strong>train_embeds.npy</strong>: Feature vectors (embeddings) for the training dataset</li>\n<li><strong>test_embeds.npy</strong>: Feature vectors (embeddings) for the test dataset</li>\n</ul>\n<p>This notebook serves as a foundation for a streamlined approach to further analysis and modeling, optimizing for the unique challenges posed by the CAFA5 dataset.</p>\n<pre><code> numpy  np\n\n :\n     ():\n        self.train_ids = np.load(train_ids_path).astype()\n        self.test_ids = np.load(test_ids_path).astype()\n        self.train_data = np.load(train_data_path).astype(np.float32)\n        self.test_data = np.load(test_data_path).astype(np.float32)\n\n     ():\n        unique_ids, indices = np.unique(self.test_ids, return_index=)\n        all_indices = np.arange(self.test_ids.shape[])\n        duplicate_indices = np.setdiff1d(all_indices, indices)\n        self.non_duplicate_indices = np.setdiff1d(all_indices, duplicate_indices)\n\n     ():\n        self.test_ids = self.test_ids[self.non_duplicate_indices]\n        self.test_data = self.test_data[self.non_duplicate_indices]\n\n        argsort_train = np.argsort(self.train_ids)\n        argsort_test = np.argsort(self.test_ids)\n\n        self.train_ids = self.train_ids[argsort_train]\n        self.test_ids = self.test_ids[argsort_test]\n        self.train_data = self.train_data[argsort_train]\n        self.test_data = self.test_data[argsort_test]\n\n     ():\n        np.save(, self.train_ids)\n        np.save(, self.test_ids)\n        np.save(, self.train_data)\n        np.save(, self.test_data)\n\n __name__ == :\n    \n    train_ids_path = \n    test_ids_path = \n    train_data_path = \n    test_data_path = \n\n    processor = DataProcessor(train_ids_path, test_ids_path, train_data_path, test_data_path)\n    processor.find_non_duplicate_ids()\n    processor.preprocess_data()\n    processor.save_data()\n</code></pre>\n<p>By executing this notebook, you will be able to obtain a dataset that has been appropriately preprocessed<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fd27a976b5de358e370b1b2805c00aa8c%2F2023-08-26%20103043.png?generation=1693013498242672&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "# Sophisticated Data Preprocessing for the CAFA5 Dataset\n\nThis notebook is designed to perform a comprehensive data preprocessing operation aimed at facilitating seamless analysis and modeling for the Kaggle competition focused on the CAFA5 dataset. Below is an outline of the steps involved:\n\n## Steps\n1. **Changing Data Types**: We load the datasets and convert the data types of IDs to Unicode strings with a maximum of 10 characters (\"<U10\"). The feature vectors are converted to single-precision floating point numbers (float32).\n2. **Extracting Non-duplicate IDs**: We remove duplicate IDs present in the test dataset.\n    - **Note**: It is presumed that the test dataset contains only one pair of duplicate IDs.\n    - **Note**: There are initially 141,865 IDs, which reduce to 141,864 after removing duplicates.\n  \n3. **Sorting the Data**: Both the training and test datasets are sorted by their IDs. This makes subsequent analysis and modeling more efficient.\n\n4. **Data Saving**: After the above preprocessing steps, the data is saved as new NumPy files for future use.\n\n## File Structure\n- **train_ids.npy**: IDs in the training dataset\n- **test_ids.npy**: IDs in the test dataset\n- **train_embeds.npy**: Feature vectors (embeddings) for the training dataset\n- **test_embeds.npy**: Feature vectors (embeddings) for the test dataset\n\nThis notebook serves as a foundation for a streamlined approach to further analysis and modeling, optimizing for the unique challenges posed by the CAFA5 dataset.\n\n\n```\nimport numpy as np\n\nclass DataProcessor:\n    def __init__(self, train_ids_path, test_ids_path, train_data_path, test_data_path):\n        self.train_ids = np.load(train_ids_path).astype(\"<U10\")\n        self.test_ids = np.load(test_ids_path).astype(\"<U10\")\n        self.train_data = np.load(train_data_path).astype(np.float32)\n        self.test_data = np.load(test_data_path).astype(np.float32)\n        \n    def find_non_duplicate_ids(self):\n        unique_ids, indices = np.unique(self.test_ids, return_index=True)\n        all_indices = np.arange(self.test_ids.shape[0])\n        duplicate_indices = np.setdiff1d(all_indices, indices)\n        self.non_duplicate_indices = np.setdiff1d(all_indices, duplicate_indices)\n        \n    def preprocess_data(self):\n        self.test_ids = self.test_ids[self.non_duplicate_indices]\n        self.test_data = self.test_data[self.non_duplicate_indices]\n        \n        argsort_train = np.argsort(self.train_ids)\n        argsort_test = np.argsort(self.test_ids)\n        \n        self.train_ids = self.train_ids[argsort_train]\n        self.test_ids = self.test_ids[argsort_test]\n        self.train_data = self.train_data[argsort_train]\n        self.test_data = self.test_data[argsort_test]\n        \n    def save_data(self):\n        np.save(\"train_ids.npy\", self.train_ids)\n        np.save(\"test_ids.npy\", self.test_ids)\n        np.save(\"train_embeddings.npy\", self.train_data)\n        np.save(\"test_embeddings.npy\", self.test_data)\n\nif __name__ == '__main__':\n    # NOTE: Please change the file paths below according to your specific dataset (e.g., T5, ESM2, ProtBERT)\n    train_ids_path = \"/kaggle/input/t5embeds/train_ids.npy\"\n    test_ids_path = \"/kaggle/input/t5embeds/test_ids.npy\"\n    train_data_path = \"/kaggle/input/t5embeds/train_embeds.npy\"\n    test_data_path = \"/kaggle/input/t5embeds/test_embeds.npy\"\n    \n    processor = DataProcessor(train_ids_path, test_ids_path, train_data_path, test_data_path)\n    processor.find_non_duplicate_ids()\n    processor.preprocess_data()\n    processor.save_data()\n```\nBy executing this notebook, you will be able to obtain a dataset that has been appropriately preprocessed\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fd27a976b5de358e370b1b2805c00aa8c%2F2023-08-26%20103043.png?generation=1693013498242672&alt=media)",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2408982": "# Sophisticated Data Preprocessing for the CAFA5 Dataset\n\nThis notebook is designed to perform a comprehensive data preprocessing operation aimed at facilitating seamless analysis and modeling for the Kaggle competition focused on the CAFA5 dataset. Below is an outline of the steps involved:\n\n## Steps\n1. **Changing Data Types**: We load the datasets and convert the data types of IDs to Unicode strings with a maximum of 10 characters (\"<U10\"). The feature vectors are converted to single-precision floating point numbers (float32).\n2. **Extracting Non-duplicate IDs**: We remove duplicate IDs present in the test dataset.\n    - **Note**: It is presumed that the test dataset contains only one pair of duplicate IDs.\n    - **Note**: There are initially 141,865 IDs, which reduce to 141,864 after removing duplicates.\n  \n3. **Sorting the Data**: Both the training and test datasets are sorted by their IDs. This makes subsequent analysis and modeling more efficient.\n\n4. **Data Saving**: After the above preprocessing steps, the data is saved as new NumPy files for future use.\n\n## File Structure\n- **train_ids.npy**: IDs in the training dataset\n- **test_ids.npy**: IDs in the test dataset\n- **train_embeds.npy**: Feature vectors (embeddings) for the training dataset\n- **test_embeds.npy**: Feature vectors (embeddings) for the test dataset\n\nThis notebook serves as a foundation for a streamlined approach to further analysis and modeling, optimizing for the unique challenges posed by the CAFA5 dataset.\n\n\n```\nimport numpy as np\n\nclass DataProcessor:\n    def __init__(self, train_ids_path, test_ids_path, train_data_path, test_data_path):\n        self.train_ids = np.load(train_ids_path).astype(\"<U10\")\n        self.test_ids = np.load(test_ids_path).astype(\"<U10\")\n        self.train_data = np.load(train_data_path).astype(np.float32)\n        self.test_data = np.load(test_data_path).astype(np.float32)\n        \n    def find_non_duplicate_ids(self):\n        unique_ids, indices = np.unique(self.test_ids, return_index=True)\n        all_indices = np.arange(self.test_ids.shape[0])\n        duplicate_indices = np.setdiff1d(all_indices, indices)\n        self.non_duplicate_indices = np.setdiff1d(all_indices, duplicate_indices)\n        \n    def preprocess_data(self):\n        self.test_ids = self.test_ids[self.non_duplicate_indices]\n        self.test_data = self.test_data[self.non_duplicate_indices]\n        \n        argsort_train = np.argsort(self.train_ids)\n        argsort_test = np.argsort(self.test_ids)\n        \n        self.train_ids = self.train_ids[argsort_train]\n        self.test_ids = self.test_ids[argsort_test]\n        self.train_data = self.train_data[argsort_train]\n        self.test_data = self.test_data[argsort_test]\n        \n    def save_data(self):\n        np.save(\"train_ids.npy\", self.train_ids)\n        np.save(\"test_ids.npy\", self.test_ids)\n        np.save(\"train_embeddings.npy\", self.train_data)\n        np.save(\"test_embeddings.npy\", self.test_data)\n\nif __name__ == '__main__':\n    # NOTE: Please change the file paths below according to your specific dataset (e.g., T5, ESM2, ProtBERT)\n    train_ids_path = \"/kaggle/input/t5embeds/train_ids.npy\"\n    test_ids_path = \"/kaggle/input/t5embeds/test_ids.npy\"\n    train_data_path = \"/kaggle/input/t5embeds/train_embeds.npy\"\n    test_data_path = \"/kaggle/input/t5embeds/test_embeds.npy\"\n    \n    processor = DataProcessor(train_ids_path, test_ids_path, train_data_path, test_data_path)\n    processor.find_non_duplicate_ids()\n    processor.preprocess_data()\n    processor.save_data()\n```\nBy executing this notebook, you will be able to obtain a dataset that has been appropriately preprocessed\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fd27a976b5de358e370b1b2805c00aa8c%2F2023-08-26%20103043.png?generation=1693013498242672&alt=media)"
  }
}