{
  "id": 615447,
  "title": "How will the test evaluation handle proteins with known GO terms which gained new terms after the final submission deadline?",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/615447",
  "author_name": "",
  "post_date": "2025-11-10T14:32:17.849793400Z",
  "votes": 4,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I have a question regarding the evaluation of proteins in the training dataset. Will the organisers only consider new Gene Ontology (GO) terms that are experimentally discovered after the submission deadline, or will the GO terms included in the training data also be taken into account?</p>\n<p>Suppose the GO terms provided in the training samples are not considered. Will the GO terms (provided in the training dataset) and their entire lineage be classified as false positives by the evaluation metric? </p>\n<p>It would be helpful if the organisers could clarify how proteins in the training data will be evaluated if they acquire new GO terms. <a href=\"https://www.kaggle.com/ahphan\" target=\"_blank\">@ahphan</a> </p>",
  "messages": [
    {
      "id": "3315257",
      "postDate": "11/10/2025 14:32:17",
      "content": "<p>I have a question regarding the evaluation of proteins in the training dataset. Will the organisers only consider new Gene Ontology (GO) terms that are experimentally discovered after the submission deadline, or will the GO terms included in the training data also be taken into account?</p>\n<p>Suppose the GO terms provided in the training samples are not considered. Will the GO terms (provided in the training dataset) and their entire lineage be classified as false positives by the evaluation metric? </p>\n<p>It would be helpful if the organisers could clarify how proteins in the training data will be evaluated if they acquire new GO terms. <a href=\"https://www.kaggle.com/ahphan\" target=\"_blank\">@ahphan</a> </p>",
      "rawMarkdown": "I have a question regarding the evaluation of proteins in the training dataset. Will the organisers only consider new Gene Ontology (GO) terms that are experimentally discovered after the submission deadline, or will the GO terms included in the training data also be taken into account?\n\nSuppose the GO terms provided in the training samples are not considered. Will the GO terms (provided in the training dataset) and their entire lineage be classified as false positives by the evaluation metric? \n\nIt would be helpful if the organisers could clarify how proteins in the training data will be evaluated if they acquire new GO terms. @ahphan",
      "votes": null
    },
    {
      "id": "3319352",
      "postDate": "11/11/2025 17:39:59",
      "content": "<p>There are three subsets of proteins that will be evaluated separately for each subontology:</p>\n<ol>\n<li>No-knowledge proteins: these proteins didn't have any GO terms in training data, so any new experimentally discovered terms will be included in the ground truth and be evaluated</li>\n<li>Limited-knowledge proteins: these proteins had some GO terms in 1 or 2 subontologies, any new experimentally discovered terms in the remaining subontology will be included in the ground truth and be evaluated</li>\n<li>Partial-knowledge proteins: these proteins had <strong>1+ subontology annotated</strong> in the training data, and gained new terms in <strong>the same subontology</strong> after submission deadline. All of the old GO terms (that appeared in the training data, as well as any publicly available GO terms before submission closes) will be excluded from the final evaluation. A 'known' annotation file will be given to the CAFA-evaluator package under <code>-known</code> argument to be excluded from the evaluation. </li>\n</ol>\n<p>Note that a no-knowledge protein <strong>cannot be</strong> limited-knowledge or partial-knowledge, while limited-knowledge proteins can still be partial-knowledge in a different subontology. All proteins in the training data can only be limited-knowledge or partial-knowledge (or gain nothing new) in the final evaluation.</p>\n<p>Toy example:\nP12345 had MFO and BPO annotations in training data, then gains CCO annotations and new BPO annotations after submission deadline. </p>\n<p>P12345 is classified as <strong>limited-knowledge in CCO</strong>, and simultaneously classified as <strong>partial-knowledge in BPO</strong>. All CCO annotations are evaluated, but only new/non-overlapping (compared to training data) BPO annotations are evaluated.</p>",
      "rawMarkdown": "There are three subsets of proteins that will be evaluated separately for each subontology:\n1. No-knowledge proteins: these proteins didn't have any GO terms in training data, so any new experimentally discovered terms will be included in the ground truth and be evaluated\n2. Limited-knowledge proteins: these proteins had some GO terms in 1 or 2 subontologies, any new experimentally discovered terms in the remaining subontology will be included in the ground truth and be evaluated\n3. Partial-knowledge proteins: these proteins had **1+ subontology annotated** in the training data, and gained new terms in **the same subontology** after submission deadline. All of the old GO terms (that appeared in the training data, as well as any publicly available GO terms before submission closes) will be excluded from the final evaluation. A 'known' annotation file will be given to the CAFA-evaluator package under `-known` argument to be excluded from the evaluation. \n\nNote that a no-knowledge protein **cannot be** limited-knowledge or partial-knowledge, while limited-knowledge proteins can still be partial-knowledge in a different subontology. All proteins in the training data can only be limited-knowledge or partial-knowledge (or gain nothing new) in the final evaluation.\n\nToy example:\nP12345 had MFO and BPO annotations in training data, then gains CCO annotations and new BPO annotations after submission deadline. \n\nP12345 is classified as **limited-knowledge in CCO**, and simultaneously classified as **partial-knowledge in BPO**. All CCO annotations are evaluated, but only new/non-overlapping (compared to training data) BPO annotations are evaluated.",
      "votes": null
    },
    {
      "id": "3320008",
      "postDate": "11/12/2025 07:02:37",
      "content": "<p>If a known GO term changes?</p>",
      "rawMarkdown": "If a known GO term changes?",
      "votes": null
    },
    {
      "id": "3320541",
      "postDate": "11/12/2025 15:55:09",
      "content": "<p>Can you elaborate your question more? </p>",
      "rawMarkdown": "Can you elaborate your question more?",
      "votes": null
    },
    {
      "id": "3333205",
      "postDate": "11/17/2025 05:31:40",
      "content": "<p>Thanks for the explanation 😀 </p>",
      "rawMarkdown": "Thanks for the explanation 😀",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3319352,
      "author_name": "ahphan",
      "author_url": "",
      "post_date": "11/11/2025 17:39:59",
      "content": "<p>There are three subsets of proteins that will be evaluated separately for each subontology:</p>\n<ol>\n<li>No-knowledge proteins: these proteins didn't have any GO terms in training data, so any new experimentally discovered terms will be included in the ground truth and be evaluated</li>\n<li>Limited-knowledge proteins: these proteins had some GO terms in 1 or 2 subontologies, any new experimentally discovered terms in the remaining subontology will be included in the ground truth and be evaluated</li>\n<li>Partial-knowledge proteins: these proteins had <strong>1+ subontology annotated</strong> in the training data, and gained new terms in <strong>the same subontology</strong> after submission deadline. All of the old GO terms (that appeared in the training data, as well as any publicly available GO terms before submission closes) will be excluded from the final evaluation. A 'known' annotation file will be given to the CAFA-evaluator package under <code>-known</code> argument to be excluded from the evaluation. </li>\n</ol>\n<p>Note that a no-knowledge protein <strong>cannot be</strong> limited-knowledge or partial-knowledge, while limited-knowledge proteins can still be partial-knowledge in a different subontology. All proteins in the training data can only be limited-knowledge or partial-knowledge (or gain nothing new) in the final evaluation.</p>\n<p>Toy example:\nP12345 had MFO and BPO annotations in training data, then gains CCO annotations and new BPO annotations after submission deadline. </p>\n<p>P12345 is classified as <strong>limited-knowledge in CCO</strong>, and simultaneously classified as <strong>partial-knowledge in BPO</strong>. All CCO annotations are evaluated, but only new/non-overlapping (compared to training data) BPO annotations are evaluated.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3320008,
          "author_name": "",
          "author_url": "",
          "post_date": "11/12/2025 07:02:37",
          "content": "<p>If a known GO term changes?</p>",
          "votes": null,
          "replies": [
            {
              "id": 3320541,
              "author_name": "ahphan",
              "author_url": "",
              "post_date": "11/12/2025 15:55:09",
              "content": "<p>Can you elaborate your question more? </p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 3333205,
          "author_name": "yuvaramsingh",
          "author_url": "",
          "post_date": "11/17/2025 05:31:40",
          "content": "<p>Thanks for the explanation 😀 </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3315257": "I have a question regarding the evaluation of proteins in the training dataset. Will the organisers only consider new Gene Ontology (GO) terms that are experimentally discovered after the submission deadline, or will the GO terms included in the training data also be taken into account?\n\nSuppose the GO terms provided in the training samples are not considered. Will the GO terms (provided in the training dataset) and their entire lineage be classified as false positives by the evaluation metric? \n\nIt would be helpful if the organisers could clarify how proteins in the training data will be evaluated if they acquire new GO terms. @ahphan",
    "3319352": "There are three subsets of proteins that will be evaluated separately for each subontology:\n1. No-knowledge proteins: these proteins didn't have any GO terms in training data, so any new experimentally discovered terms will be included in the ground truth and be evaluated\n2. Limited-knowledge proteins: these proteins had some GO terms in 1 or 2 subontologies, any new experimentally discovered terms in the remaining subontology will be included in the ground truth and be evaluated\n3. Partial-knowledge proteins: these proteins had **1+ subontology annotated** in the training data, and gained new terms in **the same subontology** after submission deadline. All of the old GO terms (that appeared in the training data, as well as any publicly available GO terms before submission closes) will be excluded from the final evaluation. A 'known' annotation file will be given to the CAFA-evaluator package under `-known` argument to be excluded from the evaluation. \n\nNote that a no-knowledge protein **cannot be** limited-knowledge or partial-knowledge, while limited-knowledge proteins can still be partial-knowledge in a different subontology. All proteins in the training data can only be limited-knowledge or partial-knowledge (or gain nothing new) in the final evaluation.\n\nToy example:\nP12345 had MFO and BPO annotations in training data, then gains CCO annotations and new BPO annotations after submission deadline. \n\nP12345 is classified as **limited-knowledge in CCO**, and simultaneously classified as **partial-knowledge in BPO**. All CCO annotations are evaluated, but only new/non-overlapping (compared to training data) BPO annotations are evaluated.",
    "3320008": "If a known GO term changes?",
    "3320541": "Can you elaborate your question more?",
    "3333205": "Thanks for the explanation 😀"
  },
  "source": "meta"
}