I'm preparing my submission, and as sanity check I decided to count the number of domains in which proteins are annotated in the latest version of GOA. To do this I filtered the GAF file to those entries with an evidence code included in the following list:
\nTAS, IPI, IDA, IMP, IC, IGI, IEP, EXP, HTP, HDA, HMP, HGI, HEP
I do find several instances where the protein already has at least one annotation in all subdomains. According to the evaluation rules, however, it seems that such proteins should not be included in any test set. Am I interpreting something wrong (perhaps the correct set of evidence codes, or am I reading the rules incorrectly)?
\nBelow are a couple of example proteins in the test superset where this happens:
\n GO ID Evidence Code domain\n GO: IDA CC\n GO: IMP BP\n GO: IPI MF\n\n GO: IPI MF\n GO: IMP BP\n GO: TAS CC\nI'm preparing my submission, and as sanity check I decided to count the number of domains in which proteins are annotated in the latest version of GOA. To do this I filtered the GAF file to those entries with an evidence code included in the following list:
\nTAS, IPI, IDA, IMP, IC, IGI, IEP, EXP, HTP, HDA, HMP, HGI, HEP
I do find several instances where the protein already has at least one annotation in all subdomains. According to the evaluation rules, however, it seems that such proteins should not be included in any test set. Am I interpreting something wrong (perhaps the correct set of evidence codes, or am I reading the rules incorrectly)?
\nBelow are a couple of example proteins in the test superset where this happens:
\n GO ID Evidence Code domain\n GO: IDA CC\n GO: IMP BP\n GO: IPI MF\n\n GO: IPI MF\n GO: IMP BP\n GO: TAS CC\nNo reason to panic or worry about this. Indeed, it isn't really surprising, as discussed some tine ago multiple proteins in the test set are also in the training set.
\nNothing demands that such proteins shouldn't be included in the larger 'test set' that we predict, only that they won't be included (at least for that subontology) in the smaller set of new annotations for which the private LB scores will be calculated.
", "rawMarkdown": "No reason to panic or worry about this. Indeed, it isn't really surprising, [as discussed some tine ago](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403200) multiple proteins in the test set are also in the training set.\n\nNothing demands that such proteins shouldn't be included in the larger 'test set' that we predict, only that they won't be included (at least for that subontology) in the smaller set of new annotations for which the private LB scores will be calculated.", "votes": null }, { "id": "2397028", "postDate": "08/18/2023 16:57:09", "content": "I understand this for proteins that already have annotations in 1 or 2 subontologies. However, if a protein has annotations in all sub-ontologies, it should not be included in any of the 3 possible test sets (one test set per subontology, as described in the \"Evaluation\" section in the \"Overview\" tab of the competition).
\nMy main concern is that it is not clear which of the currently publicly available annotations are considered to determine if a specific protein has the annotation in a sub-ontology or not. I've seen others already asking similar questions to help clarify this here (about which qualifiers apply) and here (asking about what the annotation being \"published\" means, Iddo clarifies there that this technically means that an annotation needs to be added to a database with an experimental evidence code, but it does not specify which database/s).
\nThe consequence of this is that one may choose to allocate the 1500 scores to a protein differently, depending on how many ontologies already have an annotation for it. (For example, if a protein has annotations in BP and MF, it makes sense to only output scores for CC, as it will be the only subontology that will be evaluated)
", "rawMarkdown": "I understand this for proteins that already have annotations in 1 or 2 subontologies. However, if a protein has annotations in all sub-ontologies, it should not be included in any of the 3 possible test sets (one test set per subontology, as described in the \"Evaluation\" section in the \"Overview\" tab of the competition). \n\nMy main concern is that it is not clear which of the currently publicly available annotations are considered to determine if a specific protein has the annotation in a sub-ontology or not. I've seen others already asking similar questions to help clarify this [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/415601) (about which qualifiers apply) and [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405898) (asking about what the annotation being \"published\" means, Iddo clarifies there that this technically means that an annotation needs to be added to a database with an experimental evidence code, but it does not specify which database/s).\n\nThe consequence of this is that one may choose to allocate the 1500 scores to a protein differently, depending on how many ontologies already have an annotation for it. (For example, if a protein has annotations in BP and MF, it makes sense to only output scores for CC, as it will be the only subontology that will be evaluated)", "votes": null }, { "id": "2397240", "postDate": "08/18/2023 20:56:53", "content": "Hi everyone, proteins that have annotations in public databases in all three ontologies will not be used for the test set in this competition (having said that, we are always interested if a method can successfully complete functional annotations of proteins). We think of UniProt as the main source of experimental annotations (including TAS and IC). However, there may be other sources containing experimental annotations that can later be incorporated in UniProt. If we identify proteins that are experimentally annotated in other databases (before the submission deadline) that UniProt plans to incorporate, we will exclude them from the test set.
", "rawMarkdown": "Hi everyone, proteins that have annotations in public databases in all three ontologies will not be used for the test set in this competition (having said that, we are always interested if a method can successfully complete functional annotations of proteins). We think of UniProt as the main source of experimental annotations (including TAS and IC). However, there may be other sources containing experimental annotations that can later be incorporated in UniProt. If we identify proteins that are experimentally annotated in other databases (before the submission deadline) that UniProt plans to incorporate, we will exclude them from the test set.", "votes": null }, { "id": "2397257", "postDate": "08/18/2023 21:16:29", "content": "Thanks for this explanation, this is indeed very useful.
\nI wonder if you have any advice then regarding which database to consider now as \"official\" when it comes to submitting predictions. Ideally, I would like to allocate the 1500 scores my method produces in a way that is useful. For example
\nTo make this decision, however, I would need to know what is the \"official\" source (database) of experimentally supported annotations. SO far, I had assumed it is the following GAF file: https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz . However, after filtering out all \"NOT\" qualifiers, there are several thousand proteins that are both in the test superset AND have at least one experimental annotation in ALL SUBONTOLOGIES (that is, annotated with one of the evidence codes listed above for BP, MF, and CC). For those proteins, am I safe assuming they won't be included in the test set of any of the subontologies?
\nEDIT: - I'm insisting on this, because I find several thousands of proteins that meet this conditions, not just a few
\nA related but relevant question for now is. What is the \"official\" criterion to determine if a specific protein has annotations now? (This is, how do I know BEFORE the submission if a protein already has an annotation in a specific subontology?). This is to avoid \"wasting\" scores for a protein in a subontology when it has no chance to be included in any evaluation.
\nHopefully what I described makes sense.
", "rawMarkdown": "Thanks for this explanation, this is indeed very useful. \n\nI wonder if you have any advice then regarding which database to consider now as \"official\" when it comes to submitting predictions. Ideally, I would like to allocate the 1500 scores my method produces in a way that is useful. For example\n\n* if I know a specific protein already has annotations in BP and MF -> allocate 1500 to CC\n* If I know a specific protein only has annotations in BP -> allocate 750 to MF, and 750 to CC\n\nTo make this decision, however, I would need to know what is the \"official\" source (database) of experimentally supported annotations. SO far, I had assumed it is the following GAF file: https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz . However, after filtering out all \"NOT\" qualifiers, there are several thousand proteins that are both in the test superset AND have at least one experimental annotation in ALL SUBONTOLOGIES (that is, annotated with one of the evidence codes listed above for BP, MF, and CC). For those proteins, am I safe assuming they won't be included in the test set of any of the subontologies?\n\n\n**EDIT:** - I'm insisting on this, because I find several thousands of proteins that meet this conditions, not just a few\n\nA related but relevant question for now is. What is the \"official\" criterion to determine if a specific protein has annotations now? (This is, how do I know BEFORE the submission if a protein already has an annotation in a specific subontology?). This is to avoid \"wasting\" scores for a protein in a subontology when it has no chance to be included in any evaluation.\n\nHopefully what I described makes sense.", "votes": null }, { "id": "2399623", "postDate": "08/20/2023 13:50:07", "content": "Hi @torresmateo,
\nAfaik, the maximum prediction for a single protein in one subontology is 500. Hence, even if the case you mentioned in the last paragraph holds, the evaluation code will take only the first 500 predicting terms into consideration for CC, not 1500; that is, the latter will be truncated. The code snippet is shown as follows (line 128 of parser.py),
max_terms np.count_nonzero(matrix[ns][i]) <= max_terms:\nThis clarifies the matters a lot! Thanks a lot for pointing me to it @abaojiang
", "rawMarkdown": "This clarifies the matters a lot! Thanks a lot for pointing me to it @abaojiang", "votes": null } ], "comments": [ { "id": 2396620, "author_name": "jbomitchell", "author_url": "", "post_date": "08/18/2023 11:14:37", "content": "No reason to panic or worry about this. Indeed, it isn't really surprising, as discussed some tine ago multiple proteins in the test set are also in the training set.
\nNothing demands that such proteins shouldn't be included in the larger 'test set' that we predict, only that they won't be included (at least for that subontology) in the smaller set of new annotations for which the private LB scores will be calculated.
", "votes": null, "replies": [ { "id": 2397028, "author_name": "torresmateo", "author_url": "", "post_date": "08/18/2023 16:57:09", "content": "I understand this for proteins that already have annotations in 1 or 2 subontologies. However, if a protein has annotations in all sub-ontologies, it should not be included in any of the 3 possible test sets (one test set per subontology, as described in the \"Evaluation\" section in the \"Overview\" tab of the competition).
\nMy main concern is that it is not clear which of the currently publicly available annotations are considered to determine if a specific protein has the annotation in a sub-ontology or not. I've seen others already asking similar questions to help clarify this here (about which qualifiers apply) and here (asking about what the annotation being \"published\" means, Iddo clarifies there that this technically means that an annotation needs to be added to a database with an experimental evidence code, but it does not specify which database/s).
\nThe consequence of this is that one may choose to allocate the 1500 scores to a protein differently, depending on how many ontologies already have an annotation for it. (For example, if a protein has annotations in BP and MF, it makes sense to only output scores for CC, as it will be the only subontology that will be evaluated)
", "votes": null, "replies": [ { "id": 2399623, "author_name": "abaojiang", "author_url": "", "post_date": "08/20/2023 13:50:07", "content": "Hi @torresmateo,
\nAfaik, the maximum prediction for a single protein in one subontology is 500. Hence, even if the case you mentioned in the last paragraph holds, the evaluation code will take only the first 500 predicting terms into consideration for CC, not 1500; that is, the latter will be truncated. The code snippet is shown as follows (line 128 of parser.py),
max_terms np.count_nonzero(matrix[ns][i]) <= max_terms:\nThis clarifies the matters a lot! Thanks a lot for pointing me to it @abaojiang
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2397240, "author_name": "predragradivojac", "author_url": "", "post_date": "08/18/2023 20:56:53", "content": "Hi everyone, proteins that have annotations in public databases in all three ontologies will not be used for the test set in this competition (having said that, we are always interested if a method can successfully complete functional annotations of proteins). We think of UniProt as the main source of experimental annotations (including TAS and IC). However, there may be other sources containing experimental annotations that can later be incorporated in UniProt. If we identify proteins that are experimentally annotated in other databases (before the submission deadline) that UniProt plans to incorporate, we will exclude them from the test set.
", "votes": null, "replies": [ { "id": 2397257, "author_name": "torresmateo", "author_url": "", "post_date": "08/18/2023 21:16:29", "content": "Thanks for this explanation, this is indeed very useful.
\nI wonder if you have any advice then regarding which database to consider now as \"official\" when it comes to submitting predictions. Ideally, I would like to allocate the 1500 scores my method produces in a way that is useful. For example
\nTo make this decision, however, I would need to know what is the \"official\" source (database) of experimentally supported annotations. SO far, I had assumed it is the following GAF file: https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz . However, after filtering out all \"NOT\" qualifiers, there are several thousand proteins that are both in the test superset AND have at least one experimental annotation in ALL SUBONTOLOGIES (that is, annotated with one of the evidence codes listed above for BP, MF, and CC). For those proteins, am I safe assuming they won't be included in the test set of any of the subontologies?
\nEDIT: - I'm insisting on this, because I find several thousands of proteins that meet this conditions, not just a few
\nA related but relevant question for now is. What is the \"official\" criterion to determine if a specific protein has annotations now? (This is, how do I know BEFORE the submission if a protein already has an annotation in a specific subontology?). This is to avoid \"wasting\" scores for a protein in a subontology when it has no chance to be included in any evaluation.
\nHopefully what I described makes sense.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2395805": "I'm preparing my submission, and as sanity check I decided to count the number of domains in which proteins are annotated in the latest version of GOA. To do this I filtered the GAF file to those entries with an evidence code included in the following list:\nTAS, IPI, IDA, IMP, IC, IGI, IEP, EXP, HTP, HDA, HMP, HGI, HEP\n\nI do find several instances where the protein already has at least one annotation in all subdomains. According to the evaluation rules, however, it seems that such proteins should not be included in any test set. Am I interpreting something wrong (perhaps the correct set of evidence codes, or am I reading the rules incorrectly)? \n\nBelow are a couple of example proteins in the test superset where this happens:\n```\nprotein GO ID Evidence Code domain\nP62259 GO:0005737 IDA CC\nP62259 GO:0001764 IMP BP\nP62259 GO:0005515 IPI MF\n\nO70456 GO:0019901 IPI MF\nO70456 GO:0031424 IMP BP\nO70456 GO:0005829 TAS CC\n```", "2396620": "No reason to panic or worry about this. Indeed, it isn't really surprising, [as discussed some tine ago](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403200) multiple proteins in the test set are also in the training set.\n\nNothing demands that such proteins shouldn't be included in the larger 'test set' that we predict, only that they won't be included (at least for that subontology) in the smaller set of new annotations for which the private LB scores will be calculated.", "2397028": "I understand this for proteins that already have annotations in 1 or 2 subontologies. However, if a protein has annotations in all sub-ontologies, it should not be included in any of the 3 possible test sets (one test set per subontology, as described in the \"Evaluation\" section in the \"Overview\" tab of the competition). \n\nMy main concern is that it is not clear which of the currently publicly available annotations are considered to determine if a specific protein has the annotation in a sub-ontology or not. I've seen others already asking similar questions to help clarify this [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/415601) (about which qualifiers apply) and [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405898) (asking about what the annotation being \"published\" means, Iddo clarifies there that this technically means that an annotation needs to be added to a database with an experimental evidence code, but it does not specify which database/s).\n\nThe consequence of this is that one may choose to allocate the 1500 scores to a protein differently, depending on how many ontologies already have an annotation for it. (For example, if a protein has annotations in BP and MF, it makes sense to only output scores for CC, as it will be the only subontology that will be evaluated)", "2397240": "Hi everyone, proteins that have annotations in public databases in all three ontologies will not be used for the test set in this competition (having said that, we are always interested if a method can successfully complete functional annotations of proteins). We think of UniProt as the main source of experimental annotations (including TAS and IC). However, there may be other sources containing experimental annotations that can later be incorporated in UniProt. If we identify proteins that are experimentally annotated in other databases (before the submission deadline) that UniProt plans to incorporate, we will exclude them from the test set.", "2397257": "Thanks for this explanation, this is indeed very useful. \n\nI wonder if you have any advice then regarding which database to consider now as \"official\" when it comes to submitting predictions. Ideally, I would like to allocate the 1500 scores my method produces in a way that is useful. For example\n\n* if I know a specific protein already has annotations in BP and MF -> allocate 1500 to CC\n* If I know a specific protein only has annotations in BP -> allocate 750 to MF, and 750 to CC\n\nTo make this decision, however, I would need to know what is the \"official\" source (database) of experimentally supported annotations. SO far, I had assumed it is the following GAF file: https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz . However, after filtering out all \"NOT\" qualifiers, there are several thousand proteins that are both in the test superset AND have at least one experimental annotation in ALL SUBONTOLOGIES (that is, annotated with one of the evidence codes listed above for BP, MF, and CC). For those proteins, am I safe assuming they won't be included in the test set of any of the subontologies?\n\n\n**EDIT:** - I'm insisting on this, because I find several thousands of proteins that meet this conditions, not just a few\n\nA related but relevant question for now is. What is the \"official\" criterion to determine if a specific protein has annotations now? (This is, how do I know BEFORE the submission if a protein already has an annotation in a specific subontology?). This is to avoid \"wasting\" scores for a protein in a subontology when it has no chance to be included in any evaluation.\n\nHopefully what I described makes sense.", "2399623": "Hi @torresmateo,\n\nAfaik, the maximum prediction for a single protein in one subontology is 500. Hence, even if the case you mentioned in the last paragraph holds, the evaluation code will take only **the first 500** predicting terms into consideration for CC, not 1500; that is, the latter will be truncated. The code snippet is shown as follows (line 128 of `parser.py`),\n\n```python\nif max_terms is None or np.count_nonzero(matrix[ns][i]) <= max_terms:\n```", "2400038": "This clarifies the matters a lot! Thanks a lot for pointing me to it @abaojiang" }, "source": "meta" }