{ "id": 405898, "title": "What does \"published\" mean for the test superset?", "url": "/competitions/cafa-5-protein-function-prediction/discussion/405898", "author_name": "", "post_date": "2023-04-29T19:26:28.661947200Z", "votes": 5, "comment_count": 5, "views": 0, "content": "

In evaluation it says:

\n
\n

Many proteins in the Test data do not currently have any assigned functions. Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set.
\n […]
\n The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases … The final test set will consist of proteins that will have accumulated functional terms after the submission deadline

\n
\n

Does published refer to available in UniProtKB? If so, are the unpublished annotations to be included already known by UniProtKB, or can some be submitted by researchers in the meantime?

\n

It seems like more details on the publication process could be useful. Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?

\n

Thanks

", "messages": [ { "id": "2239734", "postDate": "04/29/2023 19:26:28", "content": "

In evaluation it says:

\n
\n

Many proteins in the Test data do not currently have any assigned functions. Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set.
\n […]
\n The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases … The final test set will consist of proteins that will have accumulated functional terms after the submission deadline

\n
\n

Does published refer to available in UniProtKB? If so, are the unpublished annotations to be included already known by UniProtKB, or can some be submitted by researchers in the meantime?

\n

It seems like more details on the publication process could be useful. Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?

\n

Thanks

", "rawMarkdown": "In evaluation it says:\n\n>Many proteins in the Test data do not currently have any assigned functions. Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set. \n> [...]\n>The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases ... The final test set will consist of proteins that will have accumulated functional terms after the submission deadline\n\nDoes published refer to available in UniProtKB? If so, are the unpublished annotations to be included already known by UniProtKB, or can some be submitted by researchers in the meantime? \n\nIt seems like more details on the publication process could be useful. Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?\n\nThanks", "votes": null }, { "id": "2241739", "postDate": "05/01/2023 19:00:47", "content": "
\n

Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set.

\n
\n

That's a bit of a misnomer. It's not that the GO term will be published, but rather that the protein will have a database entry associated with that GO term using an experimental validation where there wasn't such an association before. As pointed out in other discussions, there are experimental annotations out there in the literature that were not curated yet into the database.

\n
\n

Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?

\n
\n

You have just described an ongoing effort for two decades of automated biocuratrion efforts. There is a whole body of literature on that, e.g. https://pubmed.ncbi.nlm.nih.gov/23514326/

", "rawMarkdown": "> Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set. \n\nThat's a bit of a misnomer. It's not that the GO term will be published, but rather that the protein will have a database entry associated with that GO term using an experimental validation where there wasn't such an association before. As pointed out in other discussions, there are experimental annotations out there in the literature that were not curated yet into the database.\n\n\n> Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?\n\nYou have just described an ongoing effort for two decades of automated biocuratrion efforts. There is a whole body of literature on that, e.g. https://pubmed.ncbi.nlm.nih.gov/23514326/", "votes": null }, { "id": "2242106", "postDate": "05/02/2023 03:40:52", "content": "

Thanks, I can see how such a tool would still be useful. From the paper:

\n
\n

There are two main ways to use text data in the context of inferring protein function: Information Extraction and Classification. The system presented here belongs to the latter category, namely, classification. Systems that use the information extraction approach aim to identify and extract phrases or terms from text sources that directly describe the function of a protein. That is, rather than predict protein function, such systems aim to find what is already known and reported in the literature about the function or the process in which a protein is involved. In contrast, a classification system uses features that are derived from text sources in order to represent any protein - regardless of whether its function is reported in the text or not. The system uses the text-based representation of proteins whose function is already known to train machine-learning classifiers that can then assign function labels to yet-unannotated proteins (where the latter are also represented using text-based features).

\n
\n

That paper was about \"Classification\", but clearly makes the distinction I was concerned about. Is it considered valid in this competition to also do \"Information Extraction\"? Because that seems like an easier task (although not trivial, the answer already exists somewhere).

", "rawMarkdown": "Thanks, I can see how such a tool would still be useful. From the paper:\n\n>There are two main ways to use text data in the context of inferring protein function: Information Extraction and Classification. The system presented here belongs to the latter category, namely, classification. Systems that use the information extraction approach aim to identify and extract phrases or terms from text sources that directly describe the function of a protein. That is, rather than predict protein function, such systems aim to find what is already known and reported in the literature about the function or the process in which a protein is involved. In contrast, a classification system uses features that are derived from text sources in order to represent any protein - regardless of whether its function is reported in the text or not. The system uses the text-based representation of proteins whose function is already known to train machine-learning classifiers that can then assign function labels to yet-unannotated proteins (where the latter are also represented using text-based features).\n\nThat paper was about \"Classification\", but clearly makes the distinction I was concerned about. Is it considered valid in this competition to also do \"Information Extraction\"? Because that seems like an easier task (although not trivial, the answer already exists somewhere).", "votes": null }, { "id": "2243005", "postDate": "05/02/2023 16:23:15", "content": "
\n

Is it considered valid in this competition to also do \"Information Extraction\"?

\n
\n

You can use any information you like, as long as it's public, as elaborated in the Rules:

\n
\n

C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).

\n
", "rawMarkdown": "> Is it considered valid in this competition to also do \"Information Extraction\"? \n\nYou can use any information you like, as long as it's public, as elaborated in the Rules:\n\n> C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).", "votes": null }, { "id": "2247089", "postDate": "05/05/2023 17:49:16", "content": "

Thanks. In that case you might get better results if someone in this field makes a post describing the publication process in more detail.

", "rawMarkdown": "Thanks. In that case you might get better results if someone in this field makes a post describing the publication process in more detail.", "votes": null }, { "id": "2247250", "postDate": "05/05/2023 20:38:50", "content": "

I may do that. In the meantime, in the Evaluations Metrics section, you will find a more precise description of the test set:

\n
\n

Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation. For example, a protein that had no experimental terms in say Molecular Function (MF) subontology of GO and has accumulated experimental annotations in MF after the submission deadline will be included in the test set for evaluating the MF term predictions. The same holds for the Biological Process (BP) or Cellular Component (CC) subontologies of GO. The proteins that qualify will create three different test sets, one for each subontology of GO. The same protein can appear in more than one test set if it accumulates experimentally-validated annotations in more than a single subontology

\n
", "rawMarkdown": "I may do that. In the meantime, in the Evaluations Metrics section, you will find a more precise description of the test set:\n\n>Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation. For example, a protein that had no experimental terms in say Molecular Function (MF) subontology of GO and has accumulated experimental annotations in MF after the submission deadline will be included in the test set for evaluating the MF term predictions. The same holds for the Biological Process (BP) or Cellular Component (CC) subontologies of GO. The proteins that qualify will create three different test sets, one for each subontology of GO. The same protein can appear in more than one test set if it accumulates experimentally-validated annotations in more than a single subontology", "votes": null } ], "comments": [ { "id": 2241739, "author_name": "iddux1", "author_url": "", "post_date": "05/01/2023 19:00:47", "content": "
\n

Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set.

\n
\n

That's a bit of a misnomer. It's not that the GO term will be published, but rather that the protein will have a database entry associated with that GO term using an experimental validation where there wasn't such an association before. As pointed out in other discussions, there are experimental annotations out there in the literature that were not curated yet into the database.

\n
\n

Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?

\n
\n

You have just described an ongoing effort for two decades of automated biocuratrion efforts. There is a whole body of literature on that, e.g. https://pubmed.ncbi.nlm.nih.gov/23514326/

", "votes": null, "replies": [ { "id": 2242106, "author_name": "halfwit", "author_url": "", "post_date": "05/02/2023 03:40:52", "content": "

Thanks, I can see how such a tool would still be useful. From the paper:

\n
\n

There are two main ways to use text data in the context of inferring protein function: Information Extraction and Classification. The system presented here belongs to the latter category, namely, classification. Systems that use the information extraction approach aim to identify and extract phrases or terms from text sources that directly describe the function of a protein. That is, rather than predict protein function, such systems aim to find what is already known and reported in the literature about the function or the process in which a protein is involved. In contrast, a classification system uses features that are derived from text sources in order to represent any protein - regardless of whether its function is reported in the text or not. The system uses the text-based representation of proteins whose function is already known to train machine-learning classifiers that can then assign function labels to yet-unannotated proteins (where the latter are also represented using text-based features).

\n
\n

That paper was about \"Classification\", but clearly makes the distinction I was concerned about. Is it considered valid in this competition to also do \"Information Extraction\"? Because that seems like an easier task (although not trivial, the answer already exists somewhere).

", "votes": null, "replies": [ { "id": 2243005, "author_name": "iddux1", "author_url": "", "post_date": "05/02/2023 16:23:15", "content": "
\n

Is it considered valid in this competition to also do \"Information Extraction\"?

\n
\n

You can use any information you like, as long as it's public, as elaborated in the Rules:

\n
\n

C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).

\n
", "votes": null, "replies": [ { "id": 2247089, "author_name": "halfwit", "author_url": "", "post_date": "05/05/2023 17:49:16", "content": "

Thanks. In that case you might get better results if someone in this field makes a post describing the publication process in more detail.

", "votes": null, "replies": [ { "id": 2247250, "author_name": "iddux1", "author_url": "", "post_date": "05/05/2023 20:38:50", "content": "

I may do that. In the meantime, in the Evaluations Metrics section, you will find a more precise description of the test set:

\n
\n

Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation. For example, a protein that had no experimental terms in say Molecular Function (MF) subontology of GO and has accumulated experimental annotations in MF after the submission deadline will be included in the test set for evaluating the MF term predictions. The same holds for the Biological Process (BP) or Cellular Component (CC) subontologies of GO. The proteins that qualify will create three different test sets, one for each subontology of GO. The same protein can appear in more than one test set if it accumulates experimentally-validated annotations in more than a single subontology

\n
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2239734": "In evaluation it says:\n\n>Many proteins in the Test data do not currently have any assigned functions. Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set. \n> [...]\n>The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases ... The final test set will consist of proteins that will have accumulated functional terms after the submission deadline\n\nDoes published refer to available in UniProtKB? If so, are the unpublished annotations to be included already known by UniProtKB, or can some be submitted by researchers in the meantime? \n\nIt seems like more details on the publication process could be useful. Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?\n\nThanks", "2241739": "> Proteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set. \n\nThat's a bit of a misnomer. It's not that the GO term will be published, but rather that the protein will have a database entry associated with that GO term using an experimental validation where there wasn't such an association before. As pointed out in other discussions, there are experimental annotations out there in the literature that were not curated yet into the database.\n\n\n> Eg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?\n\nYou have just described an ongoing effort for two decades of automated biocuratrion efforts. There is a whole body of literature on that, e.g. https://pubmed.ncbi.nlm.nih.gov/23514326/", "2242106": "Thanks, I can see how such a tool would still be useful. From the paper:\n\n>There are two main ways to use text data in the context of inferring protein function: Information Extraction and Classification. The system presented here belongs to the latter category, namely, classification. Systems that use the information extraction approach aim to identify and extract phrases or terms from text sources that directly describe the function of a protein. That is, rather than predict protein function, such systems aim to find what is already known and reported in the literature about the function or the process in which a protein is involved. In contrast, a classification system uses features that are derived from text sources in order to represent any protein - regardless of whether its function is reported in the text or not. The system uses the text-based representation of proteins whose function is already known to train machine-learning classifiers that can then assign function labels to yet-unannotated proteins (where the latter are also represented using text-based features).\n\nThat paper was about \"Classification\", but clearly makes the distinction I was concerned about. Is it considered valid in this competition to also do \"Information Extraction\"? Because that seems like an easier task (although not trivial, the answer already exists somewhere).", "2243005": "> Is it considered valid in this competition to also do \"Information Extraction\"? \n\nYou can use any information you like, as long as it's public, as elaborated in the Rules:\n\n> C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).", "2247089": "Thanks. In that case you might get better results if someone in this field makes a post describing the publication process in more detail.", "2247250": "I may do that. In the meantime, in the Evaluations Metrics section, you will find a more precise description of the test set:\n\n>Submissions will be evaluated on proteins (test set) that did not have experimentally determined functional annotations in at least one subontology of the Gene Ontology (GO) before the submission deadline and have accumulated experimentally-validated functional annotations in that subontology between the submission deadline and the time of evaluation. For example, a protein that had no experimental terms in say Molecular Function (MF) subontology of GO and has accumulated experimental annotations in MF after the submission deadline will be included in the test set for evaluating the MF term predictions. The same holds for the Biological Process (BP) or Cellular Component (CC) subontologies of GO. The proteins that qualify will create three different test sets, one for each subontology of GO. The same protein can appear in more than one test set if it accumulates experimentally-validated annotations in more than a single subontology" }, "source": "meta" }