Goal\nSimulate a realistic time-split for CAFA5 and use Kaggle leaderboards as a sanity check.
\nExploration notebook: https://www.kaggle.com/code/xaxipirulazo/cafa5-private-test-set-discovery
\nTimeline (for this simulation)
\nMinimal setup
\ngoa_uniprot_all.gaf.217.gz (~public cut) and ...218.gz (~private cut). Use 219/220 only for post-hoc checks.Build “truth” by window
\nQuestions
\nThank you.
", "messages": [ { "id": "3311676", "postDate": "11/05/2025 14:49:56", "content": "Goal\nSimulate a realistic time-split for CAFA5 and use Kaggle leaderboards as a sanity check.
\nExploration notebook: https://www.kaggle.com/code/xaxipirulazo/cafa5-private-test-set-discovery
\nTimeline (for this simulation)
\nMinimal setup
\ngoa_uniprot_all.gaf.217.gz (~public cut) and ...218.gz (~private cut). Use 219/220 only for post-hoc checks.Build “truth” by window
\nQuestions
\nThank you.
", "rawMarkdown": "**Goal**\nSimulate a realistic **time-split** for CAFA5 and use Kaggle leaderboards as a sanity check.\n\n**Exploration notebook:** [https://www.kaggle.com/code/xaxipirulazo/cafa5-private-test-set-discovery](https://www.kaggle.com/code/xaxipirulazo/cafa5-private-test-set-discovery)\n\n**Timeline (for this simulation)**\n\n* **Start:** Apr 18, 2023\n* **Public cut:** Aug 15, 2023\n* **Private cut:** Dec 20, 2023\n\n**Minimal setup**\n\n* **GAF snapshots:** `goa_uniprot_all.gaf.217.gz` (~public cut) and `...218.gz` (~private cut). Use `219/220` only for post-hoc checks.\n* **Evidence to keep:** experimental/curated only (EXP, IDA, IMP, IGI, IPI, IEP, TAS, …).\n* **Targets:** from the test FASTA, match accessions in GOA and submit only those with experimental annotations **within each window**.\n\n**Build “truth” by window**\n\n* **Public truth:** annotations first seen **between Start → Public cut**.\n* **Private truth:** annotations first seen **between Public cut → Private cut**.\n\n**Questions**\n\n* Is this feasible? Can simple time leakage beat the top model? I’m stuck around **~0.4**.\n* Will the CAFA5 validation sets (evaluated targets + final GO terms per window) be released? *(For reference, see CAFA-evaluator wiki for CAFA1–3: [https://github.com/BioComputingUP/CAFA-evaluator/wiki](https://github.com/BioComputingUP/CAFA-evaluator/wiki))*\n* If the holdout changed, can we reproduce the same procedure using GAF history?\n\n**Thank you.**", "votes": null }, { "id": "3311788", "postDate": "11/05/2025 19:08:16", "content": "@claradepaolis Can you help answer these questions, especially the second bullet point?
", "rawMarkdown": "claradepaolis Can you help answer these questions, especially the second bullet point?", "votes": null }, { "id": "3312004", "postDate": "11/06/2025 07:31:53", "content": "Also, I have a question, because in the evaluator provided in the github produced a score which is much higher than online platform.How to adjust it? By setting topK to the evaluator or?
", "rawMarkdown": "Also, I have a question, because in the evaluator provided in the github produced a score which is much higher than online platform.How to adjust it? By setting topK to the evaluator or?", "votes": null }, { "id": "3312155", "postDate": "11/06/2025 15:17:10", "content": "What are you using as the ground truth file? That is likely the reason why the score differs. Other arguments to reproduce the score on the leaderboard are: \n-max_terms 500 -prop fill -norm cafa -no_orphans
Oh, it is helpful, I think!
", "rawMarkdown": "Oh, it is helpful, I think!", "votes": null }, { "id": "3312420", "postDate": "11/07/2025 06:43:47", "content": "I thought max_terms would be 1500:
\nFinally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n(Overview > Evaluation > Submission File)
EDIT: Wait, -max_terms is the target limit for every namespace (where \"namespace\" refers to subontology/aspect). So -max_terms 500 means 3x500=1500, but the limit is 500 terms per subontology (i.e. 500 terms per protein and subontology).
\n\nbut the limit is 500 terms per subontology (i.e. 500 terms per protein and subontology)
\n
This 500 terms/subontology limit is imposed by -max_terms argument in CAFA-evaluator package
\n\none target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n
This 1500 terms/target limit is imposed during file parsing on Kaggle platform, before passing in the input into CAFA-evaluator
", "rawMarkdown": ">but the limit is 500 terms per subontology (i.e. 500 terms per protein and subontology) \n\nThis 500 terms/subontology limit is imposed by `-max_terms` argument in CAFA-evaluator package\n\n>one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined. \n\nThis 1500 terms/target limit is imposed during file parsing on Kaggle platform, before passing in the input into CAFA-evaluator", "votes": null }, { "id": "3312634", "postDate": "11/07/2025 15:35:17", "content": "Hi，sir, I saw that all the training data is in the test set, so, if it will be used in in the score of the public lb?
", "rawMarkdown": "Hi，sir, I saw that all the training data is in the test set, so, if it will be used in in the score of the public lb?", "votes": null }, { "id": "3312640", "postDate": "11/07/2025 16:01:09", "content": "Proteins that appear in both training data and test superset could either be limited-knowledge or partial-knowledge. If proteins had 1 or 2 subontologies annotated (in training data), and gain the remaining subontology later, they are limited-knowledge. If proteins had all 3 subontologies annotated (in training data), and gain more new annotations in any subontology, they are partial-knowledge.
\n\n\nNote that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets.
\n
This is why all proteins in the training data are in the test superset, because they can still qualify as limited-knowledge or partial-knowledge proteins.
", "rawMarkdown": "Proteins that appear in both training data and test superset could either be limited-knowledge or partial-knowledge. If proteins had 1 or 2 subontologies annotated (in training data), and gain the remaining subontology later, they are limited-knowledge. If proteins had all 3 subontologies annotated (in training data), and gain more new annotations in any subontology, they are partial-knowledge. \n\n> Note that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets. \n\nThis is why all proteins in the training data are in the test superset, because they can still qualify as limited-knowledge or partial-knowledge proteins.", "votes": null }, { "id": "3312889", "postDate": "11/08/2025 06:56:07", "content": "And to this point, I also have a question, because of the training set is included in the test set, if the annotation in the public lb is more than it in the training set. Only in this way, we could mimic the private score using the public lb. And also the public score of cafa5 was covered and could not be used to see how the shake is
", "rawMarkdown": "And to this point, I also have a question, because of the training set is included in the test set, if the annotation in the public lb is more than it in the training set. Only in this way, we could mimic the private score using the public lb. And also the public score of cafa5 was covered and could not be used to see how the shake is", "votes": null }, { "id": "3313944", "postDate": "11/10/2025 02:13:14", "content": "I'm not sure I understand what your question is. The public leaderboard currently has a small number of proteins that have hold-out experimental annotations that are not included in UniProt. If these proteins appeared in the training set, they were included in the public leaderboard as either limited-knowledge or partial-knowledge proteins. If these proteins didn't appear in the training set, they were included as no-knowledge proteins.
", "rawMarkdown": "I'm not sure I understand what your question is. The public leaderboard currently has a small number of proteins that have hold-out experimental annotations that are not included in UniProt. If these proteins appeared in the training set, they were included in the public leaderboard as either limited-knowledge or partial-knowledge proteins. If these proteins didn't appear in the training set, they were included as no-knowledge proteins.", "votes": null }, { "id": "3314622", "postDate": "11/10/2025 09:01:20", "content": "Does the 500/1500 term limit apply to ancestral terms not explicitly listed in train_terms?
\nFollow-up: If we choose to submit ancestral terms instead of letting the evaluation process infer them, does that reduce our term limit?
", "rawMarkdown": "Does the 500/1500 term limit apply to ancestral terms not explicitly listed in train_terms?\n\nFollow-up: If we choose to submit ancestral terms instead of letting the evaluation process infer them, does that reduce our term limit?", "votes": null }, { "id": "3315258", "postDate": "11/10/2025 14:33:31", "content": "The term limit applies before any propagation occurs (in the evaluation process), so it does not affect ancestral terms if you don't submit ancestral terms.
\nYes, if you submit ancestral terms, those will count towards the term limit.
", "rawMarkdown": "The term limit applies before any propagation occurs (in the evaluation process), so it does not affect ancestral terms if you don't submit ancestral terms.\n\nYes, if you submit ancestral terms, those will count towards the term limit.", "votes": null }, { "id": "3318631", "postDate": "11/11/2025 09:14:36", "content": "Hi, I use the train_terms.tsv as the ground truth file, is this the reason that this script does not take the test set into account?
", "rawMarkdown": "Hi, I use the train_terms.tsv as the ground truth file, is this the reason that this script does not take the test set into account?", "votes": null }, { "id": "3319374", "postDate": "11/11/2025 17:54:57", "content": "Ground truth file should be annotations that are made in the future, after an annotation accumulation period. If you want to mimic the evaluation process and use train_terms.tsv (collected June 2025) as ground truth file, let's say the annotation accumulation period is Feb 2025 until June 2025, then your model must predict GO terms for the proteins using only knowledge publicly available before Feb 2025 to prevent leakage.
", "rawMarkdown": "Ground truth file should be annotations that are made in the future, after an annotation accumulation period. If you want to mimic the evaluation process and use train_terms.tsv (collected June 2025) as ground truth file, let's say the annotation accumulation period is Feb 2025 until June 2025, then your model must predict GO terms for the proteins using only knowledge publicly available before Feb 2025 to prevent leakage.", "votes": null }, { "id": "3322220", "postDate": "11/13/2025 11:50:24", "content": "Hi @ahphan,
\nregarding what you said about using cafaeval:
\n-max_terms 500 -prop fill -norm cafa -no_orphans\nwhat about the options -known and -toi — will they also be used?
-toi is not currently used for the public leaderboard. -known was used for partial-knowledge proteins, where we passed in the training GO terms of those proteins into the cafaevaluator to exclude these known GO terms from the evaluation process.
@claradepaolis Can you help answer these questions, especially the second bullet point?
", "votes": null, "replies": [ { "id": 3312004, "author_name": "aizealotwu", "author_url": "", "post_date": "11/06/2025 07:31:53", "content": "Also, I have a question, because in the evaluator provided in the github produced a score which is much higher than online platform.How to adjust it? By setting topK to the evaluator or?
", "votes": null, "replies": [ { "id": 3312155, "author_name": "ahphan", "author_url": "", "post_date": "11/06/2025 15:17:10", "content": "What are you using as the ground truth file? That is likely the reason why the score differs. Other arguments to reproduce the score on the leaderboard are: \n-max_terms 500 -prop fill -norm cafa -no_orphans
Oh, it is helpful, I think!
", "votes": null, "replies": [] }, { "id": 3312420, "author_name": "diogoff", "author_url": "", "post_date": "11/07/2025 06:43:47", "content": "I thought max_terms would be 1500:
\nFinally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n(Overview > Evaluation > Submission File)
EDIT: Wait, -max_terms is the target limit for every namespace (where \"namespace\" refers to subontology/aspect). So -max_terms 500 means 3x500=1500, but the limit is 500 terms per subontology (i.e. 500 terms per protein and subontology).
\n\nbut the limit is 500 terms per subontology (i.e. 500 terms per protein and subontology)
\n
This 500 terms/subontology limit is imposed by -max_terms argument in CAFA-evaluator package
\n\none target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n
This 1500 terms/target limit is imposed during file parsing on Kaggle platform, before passing in the input into CAFA-evaluator
", "votes": null, "replies": [ { "id": 3312634, "author_name": "aizealotwu", "author_url": "", "post_date": "11/07/2025 15:35:17", "content": "Hi，sir, I saw that all the training data is in the test set, so, if it will be used in in the score of the public lb?
", "votes": null, "replies": [ { "id": 3312640, "author_name": "ahphan", "author_url": "", "post_date": "11/07/2025 16:01:09", "content": "Proteins that appear in both training data and test superset could either be limited-knowledge or partial-knowledge. If proteins had 1 or 2 subontologies annotated (in training data), and gain the remaining subontology later, they are limited-knowledge. If proteins had all 3 subontologies annotated (in training data), and gain more new annotations in any subontology, they are partial-knowledge.
\n\n\nNote that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets.
\n
This is why all proteins in the training data are in the test superset, because they can still qualify as limited-knowledge or partial-knowledge proteins.
", "votes": null, "replies": [ { "id": 3312889, "author_name": "aizealotwu", "author_url": "", "post_date": "11/08/2025 06:56:07", "content": "And to this point, I also have a question, because of the training set is included in the test set, if the annotation in the public lb is more than it in the training set. Only in this way, we could mimic the private score using the public lb. And also the public score of cafa5 was covered and could not be used to see how the shake is
", "votes": null, "replies": [ { "id": 3313944, "author_name": "ahphan", "author_url": "", "post_date": "11/10/2025 02:13:14", "content": "I'm not sure I understand what your question is. The public leaderboard currently has a small number of proteins that have hold-out experimental annotations that are not included in UniProt. If these proteins appeared in the training set, they were included in the public leaderboard as either limited-knowledge or partial-knowledge proteins. If these proteins didn't appear in the training set, they were included as no-knowledge proteins.
", "votes": null, "replies": [ { "id": 3314622, "author_name": "sroger", "author_url": "", "post_date": "11/10/2025 09:01:20", "content": "Does the 500/1500 term limit apply to ancestral terms not explicitly listed in train_terms?
\nFollow-up: If we choose to submit ancestral terms instead of letting the evaluation process infer them, does that reduce our term limit?
", "votes": null, "replies": [ { "id": 3315258, "author_name": "ahphan", "author_url": "", "post_date": "11/10/2025 14:33:31", "content": "The term limit applies before any propagation occurs (in the evaluation process), so it does not affect ancestral terms if you don't submit ancestral terms.
\nYes, if you submit ancestral terms, those will count towards the term limit.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] }, { "id": 3318631, "author_name": "aizealotwu", "author_url": "", "post_date": "11/11/2025 09:14:36", "content": "Hi, I use the train_terms.tsv as the ground truth file, is this the reason that this script does not take the test set into account?
", "votes": null, "replies": [ { "id": 3319374, "author_name": "ahphan", "author_url": "", "post_date": "11/11/2025 17:54:57", "content": "Ground truth file should be annotations that are made in the future, after an annotation accumulation period. If you want to mimic the evaluation process and use train_terms.tsv (collected June 2025) as ground truth file, let's say the annotation accumulation period is Feb 2025 until June 2025, then your model must predict GO terms for the proteins using only knowledge publicly available before Feb 2025 to prevent leakage.
", "votes": null, "replies": [ { "id": 3322220, "author_name": "xaxipirulazo", "author_url": "", "post_date": "11/13/2025 11:50:24", "content": "Hi @ahphan,
\nregarding what you said about using cafaeval:
\n-max_terms 500 -prop fill -norm cafa -no_orphans\nwhat about the options -known and -toi — will they also be used?
-toi is not currently used for the public leaderboard. -known was used for partial-knowledge proteins, where we passed in the training GO terms of those proteins into the cafaevaluator to exclude these known GO terms from the evaluation process.