Hi everyone,
\nI’ve been working on generating a robust Time-Split Validation Set to mimic the CAFA6 evaluation setting, using a historical snapshot (T0) vs. the current GO release (T1).
\nI implemented a strict split based on experimental evidence codes only (excluding IEA, etc.) and propagated terms using both is_a and part_of relationships. I categorized my validation proteins into three buckets:
\nNo-Knowledge (NK): Protein had zero experimental annotations in T0.
\nLimited-Knowledge (LK): Protein had annotations in T0, but none in the specific target aspect (e.g., has MF terms, but being evaluated on BP).
\nPartial-Knowledge (PK): Protein had annotations in the target aspect at T0 and gained NEW terms in T1.
\n(Note: For PK, I strictly filtered so that a protein is only included if Gain = T1_terms - T0_terms is non-empty for that specific aspect.)
\nThe Observation\nThe organizers have mentioned that the Partial Knowledge (PK) category is expected to be the \"hardest\" because models struggle to predict specific new functions on top of known ones.
\nHowever, in my benchmarks (even using simple baselines like BLAST), PK consistently yields the highest F-scores, significantly outperforming NK and LK.
\nMy Hypothesis (The \"Memory Effect\")\nI suspect this is due to how we calculate the F-score.
\nGround Truth (T1) = T0_terms + New_Gain_Terms
\nSince the model (or BLAST) has access to T0 during training, it effectively \"memorizes\" the T0_terms.
\nEven if the model fails to predict the New_Gain_Terms (the hard part), it still racks up a high Recall/Precision score simply by retrieving the T0_terms which make up the bulk of the Ground Truth.
\nQuestions for the Community\nIs this behavior expected? When the organizers say PK is \"hard,\" do they mean it's hard to get a perfect score, or hard to improve upon the baseline?
\nAre we supposed to evaluate differently? Am I doing something wrong? Or is the standard cafaeval on the full T1 set the correct approach, and high scores are just natural for this category?
\nI’d love to hear if others are seeing similar \"high performance\" on their Partial Knowledge splits or if I’ve missed a nuance in how the validation set should be constructed.
\nThanks!
", "messages": [ { "id": 3397781, "postDate": "2026-01-28T03:58:42.383Z", "content": "Hi everyone,
\nI’ve been working on generating a robust Time-Split Validation Set to mimic the CAFA6 evaluation setting, using a historical snapshot (T0) vs. the current GO release (T1).
\nI implemented a strict split based on experimental evidence codes only (excluding IEA, etc.) and propagated terms using both is_a and part_of relationships. I categorized my validation proteins into three buckets:
\nNo-Knowledge (NK): Protein had zero experimental annotations in T0.
\nLimited-Knowledge (LK): Protein had annotations in T0, but none in the specific target aspect (e.g., has MF terms, but being evaluated on BP).
\nPartial-Knowledge (PK): Protein had annotations in the target aspect at T0 and gained NEW terms in T1.
\n(Note: For PK, I strictly filtered so that a protein is only included if Gain = T1_terms - T0_terms is non-empty for that specific aspect.)
\nThe Observation\nThe organizers have mentioned that the Partial Knowledge (PK) category is expected to be the \"hardest\" because models struggle to predict specific new functions on top of known ones.
\nHowever, in my benchmarks (even using simple baselines like BLAST), PK consistently yields the highest F-scores, significantly outperforming NK and LK.
\nMy Hypothesis (The \"Memory Effect\")\nI suspect this is due to how we calculate the F-score.
\nGround Truth (T1) = T0_terms + New_Gain_Terms
\nSince the model (or BLAST) has access to T0 during training, it effectively \"memorizes\" the T0_terms.
\nEven if the model fails to predict the New_Gain_Terms (the hard part), it still racks up a high Recall/Precision score simply by retrieving the T0_terms which make up the bulk of the Ground Truth.
\nQuestions for the Community\nIs this behavior expected? When the organizers say PK is \"hard,\" do they mean it's hard to get a perfect score, or hard to improve upon the baseline?
\nAre we supposed to evaluate differently? Am I doing something wrong? Or is the standard cafaeval on the full T1 set the correct approach, and high scores are just natural for this category?
\nI’d love to hear if others are seeing similar \"high performance\" on their Partial Knowledge splits or if I’ve missed a nuance in how the validation set should be constructed.
\nThanks!
", "rawMarkdown": "Hi everyone,\n\nI’ve been working on generating a robust Time-Split Validation Set to mimic the CAFA6 evaluation setting, using a historical snapshot (T0) vs. the current GO release (T1).\n\nI implemented a strict split based on experimental evidence codes only (excluding IEA, etc.) and propagated terms using both is_a and part_of relationships. I categorized my validation proteins into three buckets:\n\nNo-Knowledge (NK): Protein had zero experimental annotations in T0.\n\nLimited-Knowledge (LK): Protein had annotations in T0, but none in the specific target aspect (e.g., has MF terms, but being evaluated on BP).\n\nPartial-Knowledge (PK): Protein had annotations in the target aspect at T0 and gained NEW terms in T1.\n\n(Note: For PK, I strictly filtered so that a protein is only included if Gain = T1_terms - T0_terms is non-empty for that specific aspect.)\n\nThe Observation\nThe organizers have mentioned that the Partial Knowledge (PK) category is expected to be the \"hardest\" because models struggle to predict specific new functions on top of known ones.\n\nHowever, in my benchmarks (even using simple baselines like BLAST), PK consistently yields the highest F-scores, significantly outperforming NK and LK.\n\nMy Hypothesis (The \"Memory Effect\")\nI suspect this is due to how we calculate the F-score.\n\nGround Truth (T1) = T0_terms + New_Gain_Terms\n\nSince the model (or BLAST) has access to T0 during training, it effectively \"memorizes\" the T0_terms.\n\nEven if the model fails to predict the New_Gain_Terms (the hard part), it still racks up a high Recall/Precision score simply by retrieving the T0_terms which make up the bulk of the Ground Truth.\n\nQuestions for the Community\nIs this behavior expected? When the organizers say PK is \"hard,\" do they mean it's hard to get a perfect score, or hard to improve upon the baseline?\n\nAre we supposed to evaluate differently? Am I doing something wrong? Or is the standard cafaeval on the full T1 set the correct approach, and high scores are just natural for this category?\n\nI’d love to hear if others are seeing similar \"high performance\" on their Partial Knowledge splits or if I’ve missed a nuance in how the validation set should be constructed.\n\nThanks!", "votes": 3 }, { "id": 3397791, "postDate": "2026-01-28T04:09:58.480Z", "content": "Hi, did you use the -known argument when running cafaeval code (https://github.com/claradepaolis/CAFA-evaluator-PK)? The T0_terms should be passed in cafaeval with the -known argument, so that all of these terms get excluded from PK evaluation (i.e., any terms that are already known to the public are excluded from PK evaluation). Ground Truth (T1) only contains New_Gain_Terms, and the methods should not gain any score by just retrieving terms in T0_terms.
\nThis is why we say predicting GO terms for PK proteins is hard, because you have to predict deeper GO terms beyond the already known GO terms for that protein in the target aspect.
", "rawMarkdown": "Hi, did you use the -known argument when running cafaeval code (https://github.com/claradepaolis/CAFA-evaluator-PK)? The T0_terms should be passed in cafaeval with the -known argument, so that all of these terms get excluded from PK evaluation (i.e., any terms that are already known to the public are excluded from PK evaluation). Ground Truth (T1) only contains New_Gain_Terms, and the methods should not gain any score by just retrieving terms in T0_terms.\n\nThis is why we say predicting GO terms for PK proteins is hard, because you have to predict deeper GO terms beyond the already known GO terms for that protein in the target aspect. ", "votes": 1, "replies": [ { "id": 3397818, "postDate": "2026-01-28T05:01:41.673Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 3397819, "postDate": "2026-01-28T05:04:05.650Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3400141, "postDate": "2026-02-01T02:22:56.800Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3399371, "postDate": "2026-01-30T18:37:52.443Z", "content": "When organizers say that PK is the “hardest” category, I don’t think they mean that it should give the lowest F-scores. It’s more that it’s harder to really improve over a baseline, not harder to get a decent absolute score.
\nPK has prior knowledge by definition. The model already “knows something” at T0, so the distribution is very different from NK or LK. If your method (or even BLAST) can recover existing annotations well, you already get high recall and precision because most of the T1 ground truth is inherited from T0, especially if you don't use --know.
\nSo PK tends to have a kind of built-in sensitivity.
\nImprovements require finer modeling rather than brute discovery
\nIn that sense, PK is hard because predicting the new terms on top of existing ones is genuinely difficult (or easy?), not because the overall F-score should be low. High scores are pretty natural here, especially if you’re evaluating against the full T1 annotation set.
\nYou’re probably not doing anything wrong — it’s more about how this category is meant to be interpreted. PK is less about discovery and more about refinement, and that doesn’t always show up clearly in global metrics unless you explicitly isolate the “new gain” terms.
\nThis is my current hypothesis, although I have to admit I haven’t been able to fully explore it as much as I would have liked yet.
", "rawMarkdown": "When organizers say that PK is the “hardest” category, I don’t think they mean that it should give the lowest F-scores. It’s more that it’s harder to really improve over a baseline, not harder to get a decent absolute score.\n\nPK has prior knowledge by definition. The model already “knows something” at T0, so the distribution is very different from NK or LK. If your method (or even BLAST) can recover existing annotations well, you already get high recall and precision because most of the T1 ground truth is inherited from T0, especially if you don't use --know.\n\nSo PK tends to have a kind of built-in sensitivity.\n\nImprovements require finer modeling rather than brute discovery\n\nIn that sense, PK is hard because predicting the new terms on top of existing ones is genuinely difficult (or easy?), not because the overall F-score should be low. High scores are pretty natural here, especially if you’re evaluating against the full T1 annotation set.\n\nYou’re probably not doing anything wrong — it’s more about how this category is meant to be interpreted. PK is less about discovery and more about refinement, and that doesn’t always show up clearly in global metrics unless you explicitly isolate the “new gain” terms.\n\nThis is my current hypothesis, although I have to admit I haven’t been able to fully explore it as much as I would have liked yet." } ], "comments": [ { "id": 3397791, "author_name": "An Phan", "author_url": "", "post_date": "2026-01-28T04:09:58.480000", "content": "Hi, did you use the -known argument when running cafaeval code (https://github.com/claradepaolis/CAFA-evaluator-PK)? The T0_terms should be passed in cafaeval with the -known argument, so that all of these terms get excluded from PK evaluation (i.e., any terms that are already known to the public are excluded from PK evaluation). Ground Truth (T1) only contains New_Gain_Terms, and the methods should not gain any score by just retrieving terms in T0_terms.
\nThis is why we say predicting GO terms for PK proteins is hard, because you have to predict deeper GO terms beyond the already known GO terms for that protein in the target aspect.
", "votes": 1, "replies": [ { "id": 3397818, "author_name": "", "author_url": "", "post_date": "2026-01-28T05:01:41.673000", "content": "", "votes": 0, "replies": [ { "id": 3397819, "author_name": "", "author_url": "", "post_date": "2026-01-28T05:04:05.650000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3400141, "author_name": "", "author_url": "", "post_date": "2026-02-01T02:22:56.800000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3399371, "author_name": "xaxipiruli", "author_url": "", "post_date": "2026-01-30T18:37:52.443000", "content": "When organizers say that PK is the “hardest” category, I don’t think they mean that it should give the lowest F-scores. It’s more that it’s harder to really improve over a baseline, not harder to get a decent absolute score.
\nPK has prior knowledge by definition. The model already “knows something” at T0, so the distribution is very different from NK or LK. If your method (or even BLAST) can recover existing annotations well, you already get high recall and precision because most of the T1 ground truth is inherited from T0, especially if you don't use --know.
\nSo PK tends to have a kind of built-in sensitivity.
\nImprovements require finer modeling rather than brute discovery
\nIn that sense, PK is hard because predicting the new terms on top of existing ones is genuinely difficult (or easy?), not because the overall F-score should be low. High scores are pretty natural here, especially if you’re evaluating against the full T1 annotation set.
\nYou’re probably not doing anything wrong — it’s more about how this category is meant to be interpreted. PK is less about discovery and more about refinement, and that doesn’t always show up clearly in global metrics unless you explicitly isolate the “new gain” terms.
\nThis is my current hypothesis, although I have to admit I haven’t been able to fully explore it as much as I would have liked yet.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3397781": "Hi everyone,\n\nI’ve been working on generating a robust Time-Split Validation Set to mimic the CAFA6 evaluation setting, using a historical snapshot (T0) vs. the current GO release (T1).\n\nI implemented a strict split based on experimental evidence codes only (excluding IEA, etc.) and propagated terms using both is_a and part_of relationships. I categorized my validation proteins into three buckets:\n\nNo-Knowledge (NK): Protein had zero experimental annotations in T0.\n\nLimited-Knowledge (LK): Protein had annotations in T0, but none in the specific target aspect (e.g., has MF terms, but being evaluated on BP).\n\nPartial-Knowledge (PK): Protein had annotations in the target aspect at T0 and gained NEW terms in T1.\n\n(Note: For PK, I strictly filtered so that a protein is only included if Gain = T1_terms - T0_terms is non-empty for that specific aspect.)\n\nThe Observation\nThe organizers have mentioned that the Partial Knowledge (PK) category is expected to be the \"hardest\" because models struggle to predict specific new functions on top of known ones.\n\nHowever, in my benchmarks (even using simple baselines like BLAST), PK consistently yields the highest F-scores, significantly outperforming NK and LK.\n\nMy Hypothesis (The \"Memory Effect\")\nI suspect this is due to how we calculate the F-score.\n\nGround Truth (T1) = T0_terms + New_Gain_Terms\n\nSince the model (or BLAST) has access to T0 during training, it effectively \"memorizes\" the T0_terms.\n\nEven if the model fails to predict the New_Gain_Terms (the hard part), it still racks up a high Recall/Precision score simply by retrieving the T0_terms which make up the bulk of the Ground Truth.\n\nQuestions for the Community\nIs this behavior expected? When the organizers say PK is \"hard,\" do they mean it's hard to get a perfect score, or hard to improve upon the baseline?\n\nAre we supposed to evaluate differently? Am I doing something wrong? Or is the standard cafaeval on the full T1 set the correct approach, and high scores are just natural for this category?\n\nI’d love to hear if others are seeing similar \"high performance\" on their Partial Knowledge splits or if I’ve missed a nuance in how the validation set should be constructed.\n\nThanks!", "3397791": "Hi, did you use the -known argument when running cafaeval code (https://github.com/claradepaolis/CAFA-evaluator-PK)? The T0_terms should be passed in cafaeval with the -known argument, so that all of these terms get excluded from PK evaluation (i.e., any terms that are already known to the public are excluded from PK evaluation). Ground Truth (T1) only contains New_Gain_Terms, and the methods should not gain any score by just retrieving terms in T0_terms.\n\nThis is why we say predicting GO terms for PK proteins is hard, because you have to predict deeper GO terms beyond the already known GO terms for that protein in the target aspect. ", "3399371": "When organizers say that PK is the “hardest” category, I don’t think they mean that it should give the lowest F-scores. It’s more that it’s harder to really improve over a baseline, not harder to get a decent absolute score.\n\nPK has prior knowledge by definition. The model already “knows something” at T0, so the distribution is very different from NK or LK. If your method (or even BLAST) can recover existing annotations well, you already get high recall and precision because most of the T1 ground truth is inherited from T0, especially if you don't use --know.\n\nSo PK tends to have a kind of built-in sensitivity.\n\nImprovements require finer modeling rather than brute discovery\n\nIn that sense, PK is hard because predicting the new terms on top of existing ones is genuinely difficult (or easy?), not because the overall F-score should be low. High scores are pretty natural here, especially if you’re evaluating against the full T1 annotation set.\n\nYou’re probably not doing anything wrong — it’s more about how this category is meant to be interpreted. PK is less about discovery and more about refinement, and that doesn’t always show up clearly in global metrics unless you explicitly isolate the “new gain” terms.\n\nThis is my current hypothesis, although I have to admit I haven’t been able to fully explore it as much as I would have liked yet." } }