{ "id": 421887, "title": "Still Confused about Final Test Set", "url": "/competitions/cafa-5-protein-function-prediction/discussion/421887", "author_name": "", "post_date": "2023-07-07T10:23:55.357613200Z", "votes": 15, "comment_count": 4, "views": 0, "content": "

Hi everyone,

\n

I just join this competition and still find it hard to understand which proteins will be included in the final test set (a subset of the test superset) after going through the competition description and relevant forums. My concern comes from the following two statements:

\n

1. Section Evaluation Metrics in evaluation page

\n
\n

Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation.

\n
\n

2. Section Leaderboard in evaluation page

\n
\n

The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
\n

Suppose protein \\(p_i \\) (from the test superset) accumulates experimentally-validated functional annotation (denoted as \\(mf^1 \\)) in the subontology Molecular Function (MF) after the submission deadline. Then, the question is in which of the following situations (\\(p_i \\) , \\(mf^1 \\)) will be included in the final test set:

\n\n

Following is the figure demonstrating condition 1:
\n\"2023-07-07-181418.png\"

\n

For me, I think (\\(p_i \\) , \\(mf^1 \\)) will be included in the test set only in condition 3 and condition 4. Is it correct?
\nThanks a lot!

", "messages": [ { "id": "2333968", "postDate": "07/07/2023 10:23:55", "content": "

Hi everyone,

\n

I just join this competition and still find it hard to understand which proteins will be included in the final test set (a subset of the test superset) after going through the competition description and relevant forums. My concern comes from the following two statements:

\n

1. Section Evaluation Metrics in evaluation page

\n
\n

Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation.

\n
\n

2. Section Leaderboard in evaluation page

\n
\n

The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
\n

Suppose protein \\(p_i \\) (from the test superset) accumulates experimentally-validated functional annotation (denoted as \\(mf^1 \\)) in the subontology Molecular Function (MF) after the submission deadline. Then, the question is in which of the following situations (\\(p_i \\) , \\(mf^1 \\)) will be included in the final test set:

\n\n

Following is the figure demonstrating condition 1:
\n\"2023-07-07-181418.png\"

\n

For me, I think (\\(p_i \\) , \\(mf^1 \\)) will be included in the test set only in condition 3 and condition 4. Is it correct?
\nThanks a lot!

", "rawMarkdown": "Hi everyone,\n\nI just join this competition and still find it hard to understand which proteins will be included in the final test set (a subset of the test superset) after going through the competition description and relevant forums. My concern comes from the following two statements:\n\n#### 1. Section **Evaluation Metrics** in [evaluation page](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)\n\n> Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation.\n\n#### 2. Section **Leaderboard** in [evaluation page](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)\n\n> The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.\n\nSuppose protein \\\\(p_i \\\\) (from the test superset) accumulates experimentally-validated functional annotation (denoted as \\\\(mf^1 \\\\)) in the subontology Molecular Function (MF) **after** the submission deadline. Then, the question is **in which of the following situations (\\\\(p_i \\\\) , \\\\(mf^1 \\\\)) will be included in the final test set**:\n\n* *Condition 1* - \\\\(p_i \\\\) **already has** the experimentally-validated functional annotation (denoted as \\\\(mf^0\\\\)) in the subontology MF, and (\\\\(p_i \\\\), \\\\(mf^0\\\\)) **is** used for public LB evaluation.\n* *Condition 2* - \\\\(p_i \\\\) **already has** the experimentally-validated functional annotation (denoted as \\\\(mf^0\\\\)) in the subontology MF, and (\\\\(p_i \\\\), \\\\(mf^0\\\\)) **isn't** used for public LB evaluation.\n* *Condition 3* - \\\\(p_i \\\\) **doesn't have** the experimentally-validated functional annotation in the subontology MF, but **have one in BP** (denoted as \\\\(bp^0\\\\)). Also, (\\\\(p_i \\\\), \\\\(bp^0\\\\)) **is** used for public LB evaluation.\n* *Condition 4* - \\\\(p_i \\\\) **doesn't have** the experimentally-validated functional annotation in the subontology MF, but **have one in BP** (denoted as \\\\(bp^0\\\\)). Also, (\\\\(p_i \\\\), \\\\(bp^0\\\\)) **isn't** used for public LB evaluation.\n\nFollowing is the figure demonstrating *condition 1*:\n[![2023-07-07-181418.png](https://i.postimg.cc/Xqy4RxD2/2023-07-07-181418.png)](https://postimg.cc/8fD2WdGW)\n\nFor me, I think (\\\\(p_i \\\\) , \\\\(mf^1 \\\\)) will be included in the test set only in *condition 3* and *condition 4*. Is it correct?\nThanks a lot!", "votes": null }, { "id": "2351938", "postDate": "07/20/2023 13:46:17", "content": "

Thanks for your share.

", "rawMarkdown": "Thanks for your share.", "votes": null }, { "id": "2352162", "postDate": "07/20/2023 17:11:11", "content": "

Simple answer is that \"1. Section Evaluation Metrics in evaluation page\" refers to the private leaderboard on which the real competition results will eventually be evaluated. \"2. Section Leaderboard in evaluation page\" refers to the public leaderboard which is used to calculate the provisional scores we see now. The wording \"These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation\" means simply that they are mutually exclusive. Both public and private LB sets of proteins are subsets of the given test set, the exact composition of the private test set will depend on which proteins acquire new annotations in the relevant time period.

\n

You've set my head spinning with all the mathematical notation, but IIUC then condition 4 is the only relevant one since

\n
\n

These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
", "rawMarkdown": "Simple answer is that \"1. Section Evaluation Metrics in evaluation page\" refers to the private leaderboard on which the real competition results will eventually be evaluated. \"2. Section Leaderboard in evaluation page\" refers to the public leaderboard which is used to calculate the provisional scores we see now. The wording \"These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation\" means simply that they are mutually exclusive. Both public and private LB sets of proteins are subsets of the given test set, the exact composition of the private test set will depend on which proteins acquire new annotations in the relevant time period.\n\nYou've set my head spinning with all the mathematical notation, but IIUC then condition 4 is the only relevant one since\n\n>These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.", "votes": null }, { "id": "2352525", "postDate": "07/21/2023 04:40:31", "content": "

Hi @jbomitchell,

\n

There are three test sets constructed for the final evaluation, each per namespace. Doesn't it mean that the condition 3 still holds?

\n
\n

These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
\n

If a protein is included in the public leaderboard for subontology MFO, then this protein may still be included in the final test set of BPO (if it obtains experimental annotation in BPO after submission deadline). Is it correct?

\n

Thanks for the comment!

", "rawMarkdown": "Hi @jbomitchell,\n\nThere are three test sets constructed for the final evaluation, each per namespace. Doesn't it mean that the *condition 3* still holds?\n\n> These proteins will not be included in the test set **for the subontologies** used for the leaderboard evaluation.\n\nIf a protein is included in the public leaderboard for subontology `MFO`, then this protein may still be included in the final test set of `BPO` (if it obtains experimental annotation in `BPO` after submission deadline). Is it correct?\n\nThanks for the comment!", "votes": null }, { "id": "2352645", "postDate": "07/21/2023 06:47:59", "content": "

I assume that the organisers say what they mean and mean what they say.

\n
\n

These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
\n

[EDIT] But if they were to mean \"that particular subontology\" rather than \"the subontologies\", I'm not sure it would make much practical difference. Even if a user had LB-probed the content of the public set and had their own independent validation protocol, would being able to exclude a small fraction of proteins give a worthwhile advantage? Almost all methods suitable for this competition are those that can be applied at scale to ~140k sequences/structures. So most people use methods which cover as many proteins as possible. Ultimately, none of us know exactly what proteins will be annotated in the coming weeks or months and included in the actual test set.

", "rawMarkdown": "I assume that the organisers say what they mean and mean what they say.\n\n>These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.\n\n[EDIT] But if they were to mean \"that particular subontology\" rather than \"the subontologies\", I'm not sure it would make much practical difference. Even if a user had LB-probed the content of the public set and had their own independent validation protocol, would being able to exclude a small fraction of proteins give a worthwhile advantage? Almost all methods suitable for this competition are those that can be applied at scale to ~140k sequences/structures. So most people use methods which cover as many proteins as possible. Ultimately, none of us know exactly what proteins will be annotated in the coming weeks or months and included in the actual test set.", "votes": null } ], "comments": [ { "id": 2351938, "author_name": "daishu", "author_url": "", "post_date": "07/20/2023 13:46:17", "content": "

Thanks for your share.

", "votes": null, "replies": [] }, { "id": 2352162, "author_name": "jbomitchell", "author_url": "", "post_date": "07/20/2023 17:11:11", "content": "

Simple answer is that \"1. Section Evaluation Metrics in evaluation page\" refers to the private leaderboard on which the real competition results will eventually be evaluated. \"2. Section Leaderboard in evaluation page\" refers to the public leaderboard which is used to calculate the provisional scores we see now. The wording \"These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation\" means simply that they are mutually exclusive. Both public and private LB sets of proteins are subsets of the given test set, the exact composition of the private test set will depend on which proteins acquire new annotations in the relevant time period.

\n

You've set my head spinning with all the mathematical notation, but IIUC then condition 4 is the only relevant one since

\n
\n

These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
", "votes": null, "replies": [ { "id": 2352525, "author_name": "abaojiang", "author_url": "", "post_date": "07/21/2023 04:40:31", "content": "

Hi @jbomitchell,

\n

There are three test sets constructed for the final evaluation, each per namespace. Doesn't it mean that the condition 3 still holds?

\n
\n

These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
\n

If a protein is included in the public leaderboard for subontology MFO, then this protein may still be included in the final test set of BPO (if it obtains experimental annotation in BPO after submission deadline). Is it correct?

\n

Thanks for the comment!

", "votes": null, "replies": [ { "id": 2352645, "author_name": "jbomitchell", "author_url": "", "post_date": "07/21/2023 06:47:59", "content": "

I assume that the organisers say what they mean and mean what they say.

\n
\n

These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.

\n
\n

[EDIT] But if they were to mean \"that particular subontology\" rather than \"the subontologies\", I'm not sure it would make much practical difference. Even if a user had LB-probed the content of the public set and had their own independent validation protocol, would being able to exclude a small fraction of proteins give a worthwhile advantage? Almost all methods suitable for this competition are those that can be applied at scale to ~140k sequences/structures. So most people use methods which cover as many proteins as possible. Ultimately, none of us know exactly what proteins will be annotated in the coming weeks or months and included in the actual test set.

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2333968": "Hi everyone,\n\nI just join this competition and still find it hard to understand which proteins will be included in the final test set (a subset of the test superset) after going through the competition description and relevant forums. My concern comes from the following two statements:\n\n#### 1. Section **Evaluation Metrics** in [evaluation page](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)\n\n> Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation.\n\n#### 2. Section **Leaderboard** in [evaluation page](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)\n\n> The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.\n\nSuppose protein \\\\(p_i \\\\) (from the test superset) accumulates experimentally-validated functional annotation (denoted as \\\\(mf^1 \\\\)) in the subontology Molecular Function (MF) **after** the submission deadline. Then, the question is **in which of the following situations (\\\\(p_i \\\\) , \\\\(mf^1 \\\\)) will be included in the final test set**:\n\n* *Condition 1* - \\\\(p_i \\\\) **already has** the experimentally-validated functional annotation (denoted as \\\\(mf^0\\\\)) in the subontology MF, and (\\\\(p_i \\\\), \\\\(mf^0\\\\)) **is** used for public LB evaluation.\n* *Condition 2* - \\\\(p_i \\\\) **already has** the experimentally-validated functional annotation (denoted as \\\\(mf^0\\\\)) in the subontology MF, and (\\\\(p_i \\\\), \\\\(mf^0\\\\)) **isn't** used for public LB evaluation.\n* *Condition 3* - \\\\(p_i \\\\) **doesn't have** the experimentally-validated functional annotation in the subontology MF, but **have one in BP** (denoted as \\\\(bp^0\\\\)). Also, (\\\\(p_i \\\\), \\\\(bp^0\\\\)) **is** used for public LB evaluation.\n* *Condition 4* - \\\\(p_i \\\\) **doesn't have** the experimentally-validated functional annotation in the subontology MF, but **have one in BP** (denoted as \\\\(bp^0\\\\)). Also, (\\\\(p_i \\\\), \\\\(bp^0\\\\)) **isn't** used for public LB evaluation.\n\nFollowing is the figure demonstrating *condition 1*:\n[![2023-07-07-181418.png](https://i.postimg.cc/Xqy4RxD2/2023-07-07-181418.png)](https://postimg.cc/8fD2WdGW)\n\nFor me, I think (\\\\(p_i \\\\) , \\\\(mf^1 \\\\)) will be included in the test set only in *condition 3* and *condition 4*. Is it correct?\nThanks a lot!", "2351938": "Thanks for your share.", "2352162": "Simple answer is that \"1. Section Evaluation Metrics in evaluation page\" refers to the private leaderboard on which the real competition results will eventually be evaluated. \"2. Section Leaderboard in evaluation page\" refers to the public leaderboard which is used to calculate the provisional scores we see now. The wording \"These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation\" means simply that they are mutually exclusive. Both public and private LB sets of proteins are subsets of the given test set, the exact composition of the private test set will depend on which proteins acquire new annotations in the relevant time period.\n\nYou've set my head spinning with all the mathematical notation, but IIUC then condition 4 is the only relevant one since\n\n>These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.", "2352525": "Hi @jbomitchell,\n\nThere are three test sets constructed for the final evaluation, each per namespace. Doesn't it mean that the *condition 3* still holds?\n\n> These proteins will not be included in the test set **for the subontologies** used for the leaderboard evaluation.\n\nIf a protein is included in the public leaderboard for subontology `MFO`, then this protein may still be included in the final test set of `BPO` (if it obtains experimental annotation in `BPO` after submission deadline). Is it correct?\n\nThanks for the comment!", "2352645": "I assume that the organisers say what they mean and mean what they say.\n\n>These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation.\n\n[EDIT] But if they were to mean \"that particular subontology\" rather than \"the subontologies\", I'm not sure it would make much practical difference. Even if a user had LB-probed the content of the public set and had their own independent validation protocol, would being able to exclude a small fraction of proteins give a worthwhile advantage? Almost all methods suitable for this competition are those that can be applied at scale to ~140k sequences/structures. So most people use methods which cover as many proteins as possible. Ultimately, none of us know exactly what proteins will be annotated in the coming weeks or months and included in the actual test set." }, "source": "meta" }