I've been confused with problem of building proper validation.\nI tried 2 types of validation casual CV on training data, validation on test data with labels from Uniprot.\nBut still both didn't work. Here is a bit from my experiments:
\nWhich validation methods are work in yours experiments? Could you share? Which metrics you look at (may be FN, not sure this is enough)?
", "messages": [ { "id": "3388351", "postDate": "01/08/2026 17:17:43", "content": "I've been confused with problem of building proper validation.\nI tried 2 types of validation casual CV on training data, validation on test data with labels from Uniprot.\nBut still both didn't work. Here is a bit from my experiments:
\nWhich validation methods are work in yours experiments? Could you share? Which metrics you look at (may be FN, not sure this is enough)?
", "rawMarkdown": "I've been confused with problem of building proper validation.\nI tried 2 types of validation casual CV on training data, validation on test data with labels from Uniprot.\nBut still both didn't work. Here is a bit from my experiments:\n1. Adding Uniprot GOs to training CAFA6 dataset increase CV but worse on LB\n2. Some of my experiments also increase CV but worse on LB\nI use cafaeval repo to compute F score.\n\nWhich validation methods are work in yours experiments? Could you share? Which metrics you look at (may be FN, not sure this is enough)?", "votes": null }, { "id": "3389222", "postDate": "01/10/2026 15:24:42", "content": "Thanks for sharing. Since this topic hasn't been discussed deeply in this competition yet, I've been scratching my head over it as well.
\nI am calculating cafaeval locally using StratifiedKFold based on taxonomy. As a result, my local F-max calculated with cafaeval is around 0.58, but on the Public LB, it drops to around 0.25. I have tried several other approaches, but I haven't seen any correlation between my local CV and the Public LB, and I haven't found a solution yet.
\nAlso, while investigating the cause, I checked the annotations in testsuperset.fasta and noticed that they appear to be better annotated (more comprehensive) than the training data. Since this likely differs from the distribution of the final evaluation, I feel that trying to fit the Public LB might not be desirable.
", "rawMarkdown": "Thanks for sharing. Since this topic hasn't been discussed deeply in this competition yet, I've been scratching my head over it as well.\n\nI am calculating cafaeval locally using StratifiedKFold based on taxonomy. As a result, my local F-max calculated with cafaeval is around 0.58, but on the Public LB, it drops to around 0.25. I have tried several other approaches, but I haven't seen any correlation between my local CV and the Public LB, and I haven't found a solution yet.\n\nAlso, while investigating the cause, I checked the annotations in testsuperset.fasta and noticed that they appear to be better annotated (more comprehensive) than the training data. Since this likely differs from the distribution of the final evaluation, I feel that trying to fit the Public LB might not be desirable.", "votes": null }, { "id": "3390743", "postDate": "01/13/2026 22:07:48", "content": "A key point is that the validation problem cannot be properly addressed using static annotations.
\nWhat works conceptually (i think) is to exploit the temporal nature of GOA. Instead of mixing annotations from the same time period, validation should be framed as a time-based prediction task.
\nConcretely, this can be done by using different GOA snapshots:
\nThe difference between two snapshots allows you to extract annotations that were not available at training time, avoiding information leakage.
\nIn your case, since the training dataset is based on GOA GAF 226, that snapshot can be treated as the training knowledge cutoff. Predictions can then be evaluated against the state of annotations in a newer GOA release, considering only:
\nThis setup is much closer to the CAFA evaluation philosophy, where models are evaluated on their ability to predict annotations that become known in the future, rather than rediscovering existing ones.
\nRegarding your observations:
\nFor metrics, we focused primarily on CAFA-style Fmax, but we also inspected:
\nSimple cross-validation on static data is useful to select and tune the best model, but it does not define real performance.
", "rawMarkdown": "A key point is that **the validation problem cannot be properly addressed using static annotations**.\n\nWhat works conceptually (i think) is to **exploit the temporal nature of GOA**. Instead of mixing annotations from the same time period, validation should be framed as a **time-based prediction task**.\n\nConcretely, this can be done by using **different GOA snapshots**:\n\n* Use an **older GOA snapshot** to define the training state of knowledge.\n* Use a **more recent snapshot** to identify **newly added annotations**, which act as a proxy for future discoveries.\n\nThe difference between two snapshots allows you to extract **annotations that were not available at training time**, avoiding information leakage.\n\nIn your case, since the training dataset is based on **GOA GAF 226**, that snapshot can be treated as the **training knowledge cutoff**. Predictions can then be evaluated against the state of annotations in a newer GOA release, considering only:\n\n* proteins already present in the older snapshot, and\n* GO terms that were added *after* GAF 226.\n\nThis setup is much closer to the **CAFA evaluation philosophy**, where models are evaluated on their ability to predict annotations that become known in the future, rather than rediscovering existing ones.\n\nRegarding your observations:\n\n* Increasing CV performance but worsening LB performance is a typical symptom of **label leakage** or **overfitting to the annotation distribution** of the training snapshot.\n* Adding UniProt GO annotations to the CAFA6 training set almost certainly introduces future information, which explains the CV boost but poorer leaderboard results.\n\nFor metrics, we focused primarily on **CAFA-style Fmax**, but we also inspected:\n\n* recall at low false positive rates,\n* the number of false negatives for experimentally supported terms,\n* and performance stratified by ontology (MF, BP, CC).\n\nSimple cross-validation on static data is useful to select and tune the best model, but it does not define real performance.", "votes": null } ], "comments": [ { "id": 3389222, "author_name": "loto610", "author_url": "", "post_date": "01/10/2026 15:24:42", "content": "Thanks for sharing. Since this topic hasn't been discussed deeply in this competition yet, I've been scratching my head over it as well.
\nI am calculating cafaeval locally using StratifiedKFold based on taxonomy. As a result, my local F-max calculated with cafaeval is around 0.58, but on the Public LB, it drops to around 0.25. I have tried several other approaches, but I haven't seen any correlation between my local CV and the Public LB, and I haven't found a solution yet.
\nAlso, while investigating the cause, I checked the annotations in testsuperset.fasta and noticed that they appear to be better annotated (more comprehensive) than the training data. Since this likely differs from the distribution of the final evaluation, I feel that trying to fit the Public LB might not be desirable.
", "votes": null, "replies": [] }, { "id": 3390743, "author_name": "xaxipirulazo", "author_url": "", "post_date": "01/13/2026 22:07:48", "content": "A key point is that the validation problem cannot be properly addressed using static annotations.
\nWhat works conceptually (i think) is to exploit the temporal nature of GOA. Instead of mixing annotations from the same time period, validation should be framed as a time-based prediction task.
\nConcretely, this can be done by using different GOA snapshots:
\nThe difference between two snapshots allows you to extract annotations that were not available at training time, avoiding information leakage.
\nIn your case, since the training dataset is based on GOA GAF 226, that snapshot can be treated as the training knowledge cutoff. Predictions can then be evaluated against the state of annotations in a newer GOA release, considering only:
\nThis setup is much closer to the CAFA evaluation philosophy, where models are evaluated on their ability to predict annotations that become known in the future, rather than rediscovering existing ones.
\nRegarding your observations:
\nFor metrics, we focused primarily on CAFA-style Fmax, but we also inspected:
\nSimple cross-validation on static data is useful to select and tune the best model, but it does not define real performance.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3388351": "I've been confused with problem of building proper validation.\nI tried 2 types of validation casual CV on training data, validation on test data with labels from Uniprot.\nBut still both didn't work. Here is a bit from my experiments:\n1. Adding Uniprot GOs to training CAFA6 dataset increase CV but worse on LB\n2. Some of my experiments also increase CV but worse on LB\nI use cafaeval repo to compute F score.\n\nWhich validation methods are work in yours experiments? Could you share? Which metrics you look at (may be FN, not sure this is enough)?", "3389222": "Thanks for sharing. Since this topic hasn't been discussed deeply in this competition yet, I've been scratching my head over it as well.\n\nI am calculating cafaeval locally using StratifiedKFold based on taxonomy. As a result, my local F-max calculated with cafaeval is around 0.58, but on the Public LB, it drops to around 0.25. I have tried several other approaches, but I haven't seen any correlation between my local CV and the Public LB, and I haven't found a solution yet.\n\nAlso, while investigating the cause, I checked the annotations in testsuperset.fasta and noticed that they appear to be better annotated (more comprehensive) than the training data. Since this likely differs from the distribution of the final evaluation, I feel that trying to fit the Public LB might not be desirable.", "3390743": "A key point is that **the validation problem cannot be properly addressed using static annotations**.\n\nWhat works conceptually (i think) is to **exploit the temporal nature of GOA**. Instead of mixing annotations from the same time period, validation should be framed as a **time-based prediction task**.\n\nConcretely, this can be done by using **different GOA snapshots**:\n\n* Use an **older GOA snapshot** to define the training state of knowledge.\n* Use a **more recent snapshot** to identify **newly added annotations**, which act as a proxy for future discoveries.\n\nThe difference between two snapshots allows you to extract **annotations that were not available at training time**, avoiding information leakage.\n\nIn your case, since the training dataset is based on **GOA GAF 226**, that snapshot can be treated as the **training knowledge cutoff**. Predictions can then be evaluated against the state of annotations in a newer GOA release, considering only:\n\n* proteins already present in the older snapshot, and\n* GO terms that were added *after* GAF 226.\n\nThis setup is much closer to the **CAFA evaluation philosophy**, where models are evaluated on their ability to predict annotations that become known in the future, rather than rediscovering existing ones.\n\nRegarding your observations:\n\n* Increasing CV performance but worsening LB performance is a typical symptom of **label leakage** or **overfitting to the annotation distribution** of the training snapshot.\n* Adding UniProt GO annotations to the CAFA6 training set almost certainly introduces future information, which explains the CV boost but poorer leaderboard results.\n\nFor metrics, we focused primarily on **CAFA-style Fmax**, but we also inspected:\n\n* recall at low false positive rates,\n* the number of false negatives for experimentally supported terms,\n* and performance stratified by ontology (MF, BP, CC).\n\nSimple cross-validation on static data is useful to select and tune the best model, but it does not define real performance." }, "source": "meta" }