{
  "id": 611991,
  "title": "What is the size of the test set during the forecast period?",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/611991",
  "author_name": "",
  "post_date": "2025-10-16T01:23:58.376558400Z",
  "votes": 9,
  "comment_count": 1,
  "views": 0,
  "content": "<blockquote>\n  <p>The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team, but not available in UniProtKB or other public databases. </p>\n  <p>The final test set will consist of proteins that will have accumulated functional terms after the submission deadline, and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected.</p>\n</blockquote>\n<p>Hi, CAFA team.</p>\n<p>The currently published test set only accounts for a small portion. How much test data are you planning to collect for evaluating the private leaderboard scores?</p>\n<p>Or is the specific size not yet determined? Do you have a rough estimate in mind?</p>",
  "messages": [
    {
      "id": "3302505",
      "postDate": "10/16/2025 01:23:58",
      "content": "<blockquote>\n  <p>The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team, but not available in UniProtKB or other public databases. </p>\n  <p>The final test set will consist of proteins that will have accumulated functional terms after the submission deadline, and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected.</p>\n</blockquote>\n<p>Hi, CAFA team.</p>\n<p>The currently published test set only accounts for a small portion. How much test data are you planning to collect for evaluating the private leaderboard scores?</p>\n<p>Or is the specific size not yet determined? Do you have a rough estimate in mind?</p>",
      "rawMarkdown": ">The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team, but not available in UniProtKB or other public databases. \n\n>The final test set will consist of proteins that will have accumulated functional terms after the submission deadline, and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected.\n\nHi, CAFA team.\n\nThe currently published test set only accounts for a small portion. How much test data are you planning to collect for evaluating the private leaderboard scores?\n\nOr is the specific size not yet determined? Do you have a rough estimate in mind?",
      "votes": null
    },
    {
      "id": "3302566",
      "postDate": "10/16/2025 03:57:36",
      "content": "<p>The size is not determined yet. The annotations that will be used in evaluation are specifically those that do not exist in UniProt at the time of model development. Only <em>new</em> annotations will be used in the evaluation. It's difficult to forecast how large this set will be as it depends on scientific publication and annotation curation. </p>",
      "rawMarkdown": "The size is not determined yet. The annotations that will be used in evaluation are specifically those that do not exist in UniProt at the time of model development. Only *new* annotations will be used in the evaluation. It's difficult to forecast how large this set will be as it depends on scientific publication and annotation curation.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3302566,
      "author_name": "claradepaolis",
      "author_url": "",
      "post_date": "10/16/2025 03:57:36",
      "content": "<p>The size is not determined yet. The annotations that will be used in evaluation are specifically those that do not exist in UniProt at the time of model development. Only <em>new</em> annotations will be used in the evaluation. It's difficult to forecast how large this set will be as it depends on scientific publication and annotation curation. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3302505": ">The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team, but not available in UniProtKB or other public databases. \n\n>The final test set will consist of proteins that will have accumulated functional terms after the submission deadline, and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected.\n\nHi, CAFA team.\n\nThe currently published test set only accounts for a small portion. How much test data are you planning to collect for evaluating the private leaderboard scores?\n\nOr is the specific size not yet determined? Do you have a rough estimate in mind?",
    "3302566": "The size is not determined yet. The annotations that will be used in evaluation are specifically those that do not exist in UniProt at the time of model development. Only *new* annotations will be used in the evaluation. It's difficult to forecast how large this set will be as it depends on scientific publication and annotation curation."
  },
  "source": "meta"
}