{
  "id": 663527,
  "title": "Penalties for predicting GO terms in categories absent from ground truth",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/663527",
  "author_name": "",
  "post_date": "2025-12-18T14:33:21.010014100Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi all,</p>\n<p>I would like to clarify a point about how predictions are evaluated in CAFA, specifically regarding GO categories (BP / MF / CC) that may be absent from the ground truth for a given protein.</p>\n<p>My understanding is the following, but I would appreciate your intuition.</p>\n<p>• If a protein has no ground-truth annotations in a given GO category (e.g. CC),\n  then predictions made in that category for that protein are ignored during evaluation\n  (i.e. they do not contribute TP, FP, or FN).</p>\n<p>• Conversely, if a category is present in the ground truth, incorrect predictions in that\n  category are counted as false positives and penalize precision.</p>\n<p>The key question is:</p>\n<p>Is it correct that predicting GO terms in a category that is absent from the ground truth\n    for that protein has no effect (neither positive nor negative) on the evaluation metrics?</p>\n<p>I am asking because this has implications for whether it is safer to remain silent in a category\nwhen there is weak signal, versus attempting low-confidence predictions to increase recall.</p>\n<p>Thanks in advance for any clarification.</p>",
  "messages": [
    {
      "id": "3378575",
      "postDate": "12/18/2025 14:33:21",
      "content": "<p>Hi all,</p>\n<p>I would like to clarify a point about how predictions are evaluated in CAFA, specifically regarding GO categories (BP / MF / CC) that may be absent from the ground truth for a given protein.</p>\n<p>My understanding is the following, but I would appreciate your intuition.</p>\n<p>• If a protein has no ground-truth annotations in a given GO category (e.g. CC),\n  then predictions made in that category for that protein are ignored during evaluation\n  (i.e. they do not contribute TP, FP, or FN).</p>\n<p>• Conversely, if a category is present in the ground truth, incorrect predictions in that\n  category are counted as false positives and penalize precision.</p>\n<p>The key question is:</p>\n<p>Is it correct that predicting GO terms in a category that is absent from the ground truth\n    for that protein has no effect (neither positive nor negative) on the evaluation metrics?</p>\n<p>I am asking because this has implications for whether it is safer to remain silent in a category\nwhen there is weak signal, versus attempting low-confidence predictions to increase recall.</p>\n<p>Thanks in advance for any clarification.</p>",
      "rawMarkdown": "Hi all,\n\nI would like to clarify a point about how predictions are evaluated in CAFA, specifically regarding GO categories (BP / MF / CC) that may be absent from the ground truth for a given protein.\n\nMy understanding is the following, but I would appreciate your intuition.\n\n• If a protein has no ground-truth annotations in a given GO category (e.g. CC),\n  then predictions made in that category for that protein are ignored during evaluation\n  (i.e. they do not contribute TP, FP, or FN).\n\n• Conversely, if a category is present in the ground truth, incorrect predictions in that\n  category are counted as false positives and penalize precision.\n\nThe key question is:\n\nIs it correct that predicting GO terms in a category that is absent from the ground truth\n    for that protein has no effect (neither positive nor negative) on the evaluation metrics?\n\nI am asking because this has implications for whether it is safer to remain silent in a category\nwhen there is weak signal, versus attempting low-confidence predictions to increase recall.\n\nThanks in advance for any clarification.",
      "votes": null
    },
    {
      "id": "3378920",
      "postDate": "12/18/2025 20:33:45",
      "content": "<blockquote>\n  <p>Is it correct that predicting GO terms in a category that is absent from the ground truth for that protein has no effect (neither positive nor negative) on the evaluation metrics?</p>\n</blockquote>\n<p>Yes, during ground truth and prediction parsing, which is done per category per knowledge type (no-knowledge, limited-knowledge, partial-knowledge), any predictions made to proteins that are not in the ground truth set will be filtered out and do not contribute to the metric calculation (see <a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK/blob/main/src/cafaeval/parser.py#L180\" target=\"_blank\">pred_parser function</a> that requires ground truth and performs filtering before any calculation.</p>",
      "rawMarkdown": ">Is it correct that predicting GO terms in a category that is absent from the ground truth for that protein has no effect (neither positive nor negative) on the evaluation metrics?\n\nYes, during ground truth and prediction parsing, which is done per category per knowledge type (no-knowledge, limited-knowledge, partial-knowledge), any predictions made to proteins that are not in the ground truth set will be filtered out and do not contribute to the metric calculation (see [pred_parser function](https://github.com/claradepaolis/CAFA-evaluator-PK/blob/main/src/cafaeval/parser.py#L180) that requires ground truth and performs filtering before any calculation.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3378920,
      "author_name": "ahphan",
      "author_url": "",
      "post_date": "12/18/2025 20:33:45",
      "content": "<blockquote>\n  <p>Is it correct that predicting GO terms in a category that is absent from the ground truth for that protein has no effect (neither positive nor negative) on the evaluation metrics?</p>\n</blockquote>\n<p>Yes, during ground truth and prediction parsing, which is done per category per knowledge type (no-knowledge, limited-knowledge, partial-knowledge), any predictions made to proteins that are not in the ground truth set will be filtered out and do not contribute to the metric calculation (see <a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK/blob/main/src/cafaeval/parser.py#L180\" target=\"_blank\">pred_parser function</a> that requires ground truth and performs filtering before any calculation.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3378575": "Hi all,\n\nI would like to clarify a point about how predictions are evaluated in CAFA, specifically regarding GO categories (BP / MF / CC) that may be absent from the ground truth for a given protein.\n\nMy understanding is the following, but I would appreciate your intuition.\n\n• If a protein has no ground-truth annotations in a given GO category (e.g. CC),\n  then predictions made in that category for that protein are ignored during evaluation\n  (i.e. they do not contribute TP, FP, or FN).\n\n• Conversely, if a category is present in the ground truth, incorrect predictions in that\n  category are counted as false positives and penalize precision.\n\nThe key question is:\n\nIs it correct that predicting GO terms in a category that is absent from the ground truth\n    for that protein has no effect (neither positive nor negative) on the evaluation metrics?\n\nI am asking because this has implications for whether it is safer to remain silent in a category\nwhen there is weak signal, versus attempting low-confidence predictions to increase recall.\n\nThanks in advance for any clarification.",
    "3378920": ">Is it correct that predicting GO terms in a category that is absent from the ground truth for that protein has no effect (neither positive nor negative) on the evaluation metrics?\n\nYes, during ground truth and prediction parsing, which is done per category per knowledge type (no-knowledge, limited-knowledge, partial-knowledge), any predictions made to proteins that are not in the ground truth set will be filtered out and do not contribute to the metric calculation (see [pred_parser function](https://github.com/claradepaolis/CAFA-evaluator-PK/blob/main/src/cafaeval/parser.py#L180) that requires ground truth and performs filtering before any calculation."
  },
  "source": "meta"
}