We're adding probabilities/scores as a column in the submission file but the metric being used is F1 and from the overview description it seems like any row of data (protein/GO-term pair) is considered a \"positive\" regardless of the probability.
\nSo is it fair to assume the probability is not used during evaluation? Or is there some threshold used during evaluation that knocks out some rows prior to computing F1?
", "messages": [ { "id": "3304115", "postDate": "10/19/2025 21:27:37", "content": "We're adding probabilities/scores as a column in the submission file but the metric being used is F1 and from the overview description it seems like any row of data (protein/GO-term pair) is considered a \"positive\" regardless of the probability.
\nSo is it fair to assume the probability is not used during evaluation? Or is there some threshold used during evaluation that knocks out some rows prior to computing F1?
", "rawMarkdown": "We're adding probabilities/scores as a column in the submission file but the metric being used is F1 and from the overview description it seems like any row of data (protein/GO-term pair) is considered a \"positive\" regardless of the probability.\n\nSo is it fair to assume the probability is not used during evaluation? Or is there some threshold used during evaluation that knocks out some rows prior to computing F1?", "votes": null }, { "id": "3304357", "postDate": "10/20/2025 11:31:24", "content": "The answer to this is given in the Evaluation Metrics section of the competition's Overview tab. Everything below here is quoted directly. As I understand it, F1 is evaluated at each possible value of p and the maximum is taken.
\n\"The maximum F1-measure based on the weighted precision and recall will be calculated on each of the three test sets, and the final performance measure will be an arithmetic mean of the three maximum F-measures (for MF, BP, and CC). The formulas for computing weighted F1-measures are provided in the supplement (page 31) of the following paper: Jiang Y, et al. An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Genome Biol. (2016) 17(1): 184, in the full evaluation mode. The weights (i.e., information content ic(f), where f is a term in any subontology) for each term f of each subontology are provided by the challenge organizers. Note that we equivalently refer to those weights as ia(f), called information accretion for the functional term f. The rationale for using weighted precision and recall is that GO is hierarchical and thus, the terms on top of the hierarchy are implied by their descendants. The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins. The root terms appear in every protein's annotation and thus, their weights are 0. Terms deep in the ontology tend to appear less frequently, be harder to predict, and thus their weights are larger (Clark & Radivojac, 2013). However, this does not always hold true, as highlighted in the following discussion.
\nUsing the terminology from Jiang et al. (2016), the evaluation will be carried out for no-knowledge and limited-knowledge protein targets combined, in the full evaluation mode, using maximum F-measures of information-accretion weighted precision and recall, one for each subontology. Note that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets. The three maximum F-measures of the three subontologies (Molecular Function, Biological Process, and Cellular Component) will be combined as an arithmetic mean for each subtype of knowledge gain. Finally, the three F-measures from the three subtypes no-knowledge, limited-knowledge, and partial-knowledge will be combined again as an arithmetic mean to compute the final performance. The evaluation code is available on this GitHub repository.\"
", "rawMarkdown": "The answer to this is given in the Evaluation Metrics section of the competition's Overview tab. Everything below here is quoted directly. As I understand it, F1 is evaluated at each possible value of p and the maximum is taken.\n\n\"The maximum F1-measure based on the weighted precision and recall will be calculated on each of the three test sets, and the final performance measure will be an arithmetic mean of the three maximum F-measures (for MF, BP, and CC). The formulas for computing weighted F1-measures are provided in the supplement (page 31) of the following paper: Jiang Y, et al. An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Genome Biol. (2016) 17(1): 184, in the full evaluation mode. The weights (i.e., information content ic(f), where f is a term in any subontology) for each term f of each subontology are provided by the challenge organizers. Note that we equivalently refer to those weights as ia(f), called information accretion for the functional term f. The rationale for using weighted precision and recall is that GO is hierarchical and thus, the terms on top of the hierarchy are implied by their descendants. The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins. The root terms appear in every protein's annotation and thus, their weights are 0. Terms deep in the ontology tend to appear less frequently, be harder to predict, and thus their weights are larger (Clark & Radivojac, 2013). However, this does not always hold true, as highlighted in the following discussion.\n\nUsing the terminology from Jiang et al. (2016), the evaluation will be carried out for no-knowledge and limited-knowledge protein targets combined, in the full evaluation mode, using maximum F-measures of information-accretion weighted precision and recall, one for each subontology. Note that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets. The three maximum F-measures of the three subontologies (Molecular Function, Biological Process, and Cellular Component) will be combined as an arithmetic mean for each subtype of knowledge gain. Finally, the three F-measures from the three subtypes no-knowledge, limited-knowledge, and partial-knowledge will be combined again as an arithmetic mean to compute the final performance. The evaluation code is available on this GitHub repository.\"", "votes": null }, { "id": "3304379", "postDate": "10/20/2025 12:49:40", "content": "Ah ok, I see the formulae in the paper using tau as a threshold. Thanks!
", "rawMarkdown": "Ah ok, I see the formulae in the paper using tau as a threshold. Thanks!", "votes": null } ], "comments": [ { "id": 3304357, "author_name": "jbomitchell", "author_url": "", "post_date": "10/20/2025 11:31:24", "content": "The answer to this is given in the Evaluation Metrics section of the competition's Overview tab. Everything below here is quoted directly. As I understand it, F1 is evaluated at each possible value of p and the maximum is taken.
\n\"The maximum F1-measure based on the weighted precision and recall will be calculated on each of the three test sets, and the final performance measure will be an arithmetic mean of the three maximum F-measures (for MF, BP, and CC). The formulas for computing weighted F1-measures are provided in the supplement (page 31) of the following paper: Jiang Y, et al. An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Genome Biol. (2016) 17(1): 184, in the full evaluation mode. The weights (i.e., information content ic(f), where f is a term in any subontology) for each term f of each subontology are provided by the challenge organizers. Note that we equivalently refer to those weights as ia(f), called information accretion for the functional term f. The rationale for using weighted precision and recall is that GO is hierarchical and thus, the terms on top of the hierarchy are implied by their descendants. The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins. The root terms appear in every protein's annotation and thus, their weights are 0. Terms deep in the ontology tend to appear less frequently, be harder to predict, and thus their weights are larger (Clark & Radivojac, 2013). However, this does not always hold true, as highlighted in the following discussion.
\nUsing the terminology from Jiang et al. (2016), the evaluation will be carried out for no-knowledge and limited-knowledge protein targets combined, in the full evaluation mode, using maximum F-measures of information-accretion weighted precision and recall, one for each subontology. Note that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets. The three maximum F-measures of the three subontologies (Molecular Function, Biological Process, and Cellular Component) will be combined as an arithmetic mean for each subtype of knowledge gain. Finally, the three F-measures from the three subtypes no-knowledge, limited-knowledge, and partial-knowledge will be combined again as an arithmetic mean to compute the final performance. The evaluation code is available on this GitHub repository.\"
", "votes": null, "replies": [ { "id": 3304379, "author_name": "bklynch", "author_url": "", "post_date": "10/20/2025 12:49:40", "content": "Ah ok, I see the formulae in the paper using tau as a threshold. Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3304115": "We're adding probabilities/scores as a column in the submission file but the metric being used is F1 and from the overview description it seems like any row of data (protein/GO-term pair) is considered a \"positive\" regardless of the probability.\n\nSo is it fair to assume the probability is not used during evaluation? Or is there some threshold used during evaluation that knocks out some rows prior to computing F1?", "3304357": "The answer to this is given in the Evaluation Metrics section of the competition's Overview tab. Everything below here is quoted directly. As I understand it, F1 is evaluated at each possible value of p and the maximum is taken.\n\n\"The maximum F1-measure based on the weighted precision and recall will be calculated on each of the three test sets, and the final performance measure will be an arithmetic mean of the three maximum F-measures (for MF, BP, and CC). The formulas for computing weighted F1-measures are provided in the supplement (page 31) of the following paper: Jiang Y, et al. An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Genome Biol. (2016) 17(1): 184, in the full evaluation mode. The weights (i.e., information content ic(f), where f is a term in any subontology) for each term f of each subontology are provided by the challenge organizers. Note that we equivalently refer to those weights as ia(f), called information accretion for the functional term f. The rationale for using weighted precision and recall is that GO is hierarchical and thus, the terms on top of the hierarchy are implied by their descendants. The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins. The root terms appear in every protein's annotation and thus, their weights are 0. Terms deep in the ontology tend to appear less frequently, be harder to predict, and thus their weights are larger (Clark & Radivojac, 2013). However, this does not always hold true, as highlighted in the following discussion.\n\nUsing the terminology from Jiang et al. (2016), the evaluation will be carried out for no-knowledge and limited-knowledge protein targets combined, in the full evaluation mode, using maximum F-measures of information-accretion weighted precision and recall, one for each subontology. Note that in this competition, we also include the evaluation of proteins that already had experimental terms in all three subontologies, and have accumulated more experimental terms after the submission deadline, this is known as partial-knowledge protein targets. The three maximum F-measures of the three subontologies (Molecular Function, Biological Process, and Cellular Component) will be combined as an arithmetic mean for each subtype of knowledge gain. Finally, the three F-measures from the three subtypes no-knowledge, limited-knowledge, and partial-knowledge will be combined again as an arithmetic mean to compute the final performance. The evaluation code is available on this GitHub repository.\"", "3304379": "Ah ok, I see the formulae in the paper using tau as a threshold. Thanks!" }, "source": "meta" }