What exact command is run to evaluate our code using the CAFA-6-PK-Evaluator? Is it this?
\ncafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file
Are any other flags used in this command? I'm particularly interested in what type of propagation is used (\"fill\" or \"max\")
", "messages": [ { "id": "3372239", "postDate": "12/11/2025 22:20:18", "content": "What exact command is run to evaluate our code using the CAFA-6-PK-Evaluator? Is it this?
\ncafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file
Are any other flags used in this command? I'm particularly interested in what type of propagation is used (\"fill\" or \"max\")
", "rawMarkdown": "What exact command is run to evaluate our code using the CAFA-6-PK-Evaluator? Is it this?\n\n`cafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file`\n\nAre any other flags used in this command? I'm particularly interested in what type of propagation is used (\"fill\" or \"max\")", "votes": null }, { "id": "3372280", "postDate": "12/11/2025 23:17:39", "content": "We average the evaluation over three settings: No-knowledge, limited-knowledge, and partial-knowledge. See https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview/evaluation.\nTo reproduce results, you need to construct ground truth files for each setting. That is, the subset of ground truth annotations that fall into each setting.
\nThen, the evaluation uses the following arguments:
\ncafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_setting -prop-th_step 0.01 -no_orphans -max_terms 500\nFor partial-knowledge, you need to provide the terms that will get excluded from the evaluation because they were previously know. These are captured in the train_terms.tsv file, or an equivalent if you are constructing your own time-based data splits. The file can be passed in for the other settings, it will just be ignored if the ground truth proteins don't appear in the \"known\" file. The evaluation call adds the known argument:
cafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_PK -known train_terms.tsv -prop-th_step 0.01 -no_orphans -max_terms 500\nThank you so much!
", "rawMarkdown": "Thank you so much!", "votes": null }, { "id": "3381304", "postDate": "12/24/2025 08:03:34", "content": "Hi @claradepaolis . Thanks for the provided clarification. Can you please highlight the difference between \"limited knowledge\" and \"partial knowledge\". As I understood, for each ontology we can have 2 types of proteins:
\nFinally, we have mean of 6 F-scores (3 ontologies x 2 setups).
\nBut I am still confused about what is the limited knowledge setup and how is it evaluated?
\nThanks
", "rawMarkdown": "Hi @claradepaolis . Thanks for the provided clarification. Can you please highlight the difference between \"limited knowledge\" and \"partial knowledge\". As I understood, for each ontology we can have 2 types of proteins:\n\n- test protein that occurs in train set. For this one we apply partial knowledge evaluation (second command)\n- unseen test protein. For this one we use use simple no knowledge (first command)\n\nFinally, we have mean of 6 F-scores (3 ontologies x 2 setups).\n\nBut I am still confused about what is the limited knowledge setup and how is it evaluated?\n\nThanks", "votes": null }, { "id": "3381436", "postDate": "12/24/2025 15:40:07", "content": "Proteins that appear in both training data and test superset could either be limited-knowledge or partial-knowledge.
\nExample: If protein P12345 had MFO and BPO subontologies annotated (in training data), and gain CCO subontology later, they are limited-knowledge in CCO (all CCO annotations are new for this protein). If protein P12345 had all 3 subontologies annotated (in training data), and gain more new annotations in CCO subontology, they are partial-knowledge in CCO.
\nOverall, it should be a mean of 9 F-scores (3 ontologies x 3 setups). We have 3 ground truth files that correspond to 3 setups, where only the partial-knowledge setup is passed into the cafaevaluator along with a known annotations file to exclude the pre-existing annotations.
", "rawMarkdown": "Proteins that appear in both training data and test superset could either be limited-knowledge or partial-knowledge. \n\nExample: If protein P12345 had MFO and BPO subontologies annotated (in training data), and gain CCO subontology later, they are **limited-knowledge in CCO** (all CCO annotations are new for this protein). If protein P12345 had all 3 subontologies annotated (in training data), and gain more new annotations in CCO subontology, they are **partial-knowledge in CCO**.\n\nOverall, it should be a mean of 9 F-scores (3 ontologies x 3 setups). We have 3 ground truth files that correspond to 3 setups, where only the partial-knowledge setup is passed into the cafaevaluator along with a known annotations file to exclude the pre-existing annotations.", "votes": null }, { "id": "3381450", "postDate": "12/24/2025 16:07:53", "content": "Thanks for your reply, now it is clear to me
", "rawMarkdown": "Thanks for your reply, now it is clear to me", "votes": null }, { "id": "3391327", "postDate": "01/14/2026 18:00:37", "content": "Corrected command: cafaeval instead of cafaelval
\n", "rawMarkdown": "> We average the evaluation over three settings: No-knowledge, limited-knowledge, and partial-knowledge. See [https://....\n> ```\n> cafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_PK -known train_terms.tsv -prop fill -th_step 0.01 -no_orphans -max_terms 500\n> ```\nThank you very much for that clarification. However, for the life of me I cannot find the \"-known\" flag in any of the public branches of cafaeval on github. Did I overlook it? \nIf not, could you share how \"-known\" works, for example regarding propagation of terms? \nHaving serious issues to get a proper validation metric together that can run along model training at the moment.", "votes": null }, { "id": "3395301", "postDate": "01/22/2026 17:21:25", "content": "We average the evaluation over three settings: No-knowledge, limited-knowledge, and partial-knowledge. See [https://….
\n\ncafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_PK -known train_terms.tsv -prop fill -th_step 0.01 -no_orphans -max_terms 500\nThank you very much for that clarification. However, for the life of me I cannot find the \"-known\" flag in any of the public branches of cafaeval on github. Did I overlook it? \n If not, could you share how \"-known\" works, for example regarding propagation of terms? \n Having serious issues to get a proper validation metric together that can run along model training at the moment.
\n
The github repo linked here https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview/evaluation is the one that is used for this competition. \nNamely, https://github.com/claradepaolis/CAFA-evaluator-PK.
", "rawMarkdown": "The github repo linked here https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview/evaluation is the one that is used for this competition. \nNamely, https://github.com/claradepaolis/CAFA-evaluator-PK.", "votes": null }, { "id": "3395382", "postDate": "01/22/2026 21:45:37", "content": "\n \n\n
Thank you very much, did not check the exact link and was stalking around the main repo (top google hit) like crazy.
", "rawMarkdown": "> Namely, https://github.com/claradepaolis/CAFA-evaluator-PK.\n\nThank you very much, did not check the exact link and was stalking around the main repo (top google hit) like crazy.", "votes": null } ], "comments": [ { "id": 3372280, "author_name": "claradepaolis", "author_url": "", "post_date": "12/11/2025 23:17:39", "content": "We average the evaluation over three settings: No-knowledge, limited-knowledge, and partial-knowledge. See https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview/evaluation.\nTo reproduce results, you need to construct ground truth files for each setting. That is, the subset of ground truth annotations that fall into each setting.
\nThen, the evaluation uses the following arguments:
\ncafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_setting -prop-th_step 0.01 -no_orphans -max_terms 500\nFor partial-knowledge, you need to provide the terms that will get excluded from the evaluation because they were previously know. These are captured in the train_terms.tsv file, or an equivalent if you are constructing your own time-based data splits. The file can be passed in for the other settings, it will just be ignored if the ground truth proteins don't appear in the \"known\" file. The evaluation call adds the known argument:
cafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_PK -known train_terms.tsv -prop-th_step 0.01 -no_orphans -max_terms 500\nThank you so much!
", "votes": null, "replies": [] }, { "id": 3381304, "author_name": "btbpanda", "author_url": "", "post_date": "12/24/2025 08:03:34", "content": "Hi @claradepaolis . Thanks for the provided clarification. Can you please highlight the difference between \"limited knowledge\" and \"partial knowledge\". As I understood, for each ontology we can have 2 types of proteins:
\nFinally, we have mean of 6 F-scores (3 ontologies x 2 setups).
\nBut I am still confused about what is the limited knowledge setup and how is it evaluated?
\nThanks
", "votes": null, "replies": [ { "id": 3381436, "author_name": "ahphan", "author_url": "", "post_date": "12/24/2025 15:40:07", "content": "Proteins that appear in both training data and test superset could either be limited-knowledge or partial-knowledge.
\nExample: If protein P12345 had MFO and BPO subontologies annotated (in training data), and gain CCO subontology later, they are limited-knowledge in CCO (all CCO annotations are new for this protein). If protein P12345 had all 3 subontologies annotated (in training data), and gain more new annotations in CCO subontology, they are partial-knowledge in CCO.
\nOverall, it should be a mean of 9 F-scores (3 ontologies x 3 setups). We have 3 ground truth files that correspond to 3 setups, where only the partial-knowledge setup is passed into the cafaevaluator along with a known annotations file to exclude the pre-existing annotations.
", "votes": null, "replies": [ { "id": 3381450, "author_name": "btbpanda", "author_url": "", "post_date": "12/24/2025 16:07:53", "content": "Thanks for your reply, now it is clear to me
", "votes": null, "replies": [] } ] } ] }, { "id": 3395146, "author_name": "jrsilicon", "author_url": "", "post_date": "01/22/2026 10:43:03", "content": "\n", "votes": null, "replies": [ { "id": 3395301, "author_name": "claradepaolis", "author_url": "", "post_date": "01/22/2026 17:21:25", "content": "We average the evaluation over three settings: No-knowledge, limited-knowledge, and partial-knowledge. See [https://….
\n\ncafaelval -ia IA.tsv go-basic.obo prediction_folder ground_truth_file_PK -known train_terms.tsv -prop fill -th_step 0.01 -no_orphans -max_terms 500\nThank you very much for that clarification. However, for the life of me I cannot find the \"-known\" flag in any of the public branches of cafaeval on github. Did I overlook it? \n If not, could you share how \"-known\" works, for example regarding propagation of terms? \n Having serious issues to get a proper validation metric together that can run along model training at the moment.
\n
The github repo linked here https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview/evaluation is the one that is used for this competition. \nNamely, https://github.com/claradepaolis/CAFA-evaluator-PK.
", "votes": null, "replies": [ { "id": 3395382, "author_name": "jrsilicon", "author_url": "", "post_date": "01/22/2026 21:45:37", "content": "\n \n\n
Thank you very much, did not check the exact link and was stalking around the main repo (top google hit) like crazy.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3391327, "author_name": "tdoh86", "author_url": "", "post_date": "01/14/2026 18:00:37", "content": "Corrected command: cafaeval instead of cafaelval