I noticed that most of the proteins in the testing superset are from SwissProt. Also, many proteins in the training set are included in the testing superset. Is this deliberate?
\nAnd there any TrEMBL proteins in the testing superset?
I noticed that most of the proteins in the testing superset are from SwissProt. Also, many proteins in the training set are included in the testing superset. Is this deliberate?
\nAnd there any TrEMBL proteins in the testing superset?
Hello,
\nAll of the proteins in the training set and test superset this time are from SwissProt. There are no TrEMBL proteins in either set.
\nAll of the proteins in the training set are intentionally included in the test superset, because they might accumulate experimentally-validated functional annotations between the submission deadline and the time of evaluation.
I also realized that some proteins (for example the first one - Q5W0B1), has one DO which is not listed in ncbi (GO:0005515) and also missing one DO which is listed in ncbi (GO:0008270). Why so?
", "rawMarkdown": "I also realized that some proteins (for example the first one - Q5W0B1), has one DO which is not listed in ncbi (GO:0005515) and also missing one DO which is listed in ncbi (GO:0008270). Why so?", "votes": 1, "replies": [ { "id": 3305341, "postDate": "2025-10-22T14:33:31.537Z", "content": "For Q5W0B1, since the annotation for GO:0008270 is listed with an \"IEA\" evidence code (inferred from Electronic Annotation) which is not an experimentally verified annotation, it is removed from the train_terms.tsv file. We recommend using the UniProt-GOA as the source for GO annotations since this is where we created training data and collect evaluation data later on.
", "rawMarkdown": "For Q5W0B1, since the annotation for GO:0008270 is listed with an \"IEA\" [evidence code](https://geneontology.org/docs/guide-go-evidence-codes/) (inferred from Electronic Annotation) which is not an experimentally verified annotation, it is removed from the train_terms.tsv file. We recommend using the [UniProt-GOA](https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/) as the source for GO annotations since this is where we created training data and collect evaluation data later on. ", "votes": 2 }, { "id": 3309934, "postDate": "2025-11-01T18:21:53.167Z", "content": "Thanks for this clarification! Can I also check with you that I've got the right set of experimental evidence codes? I am trying to reproduce the label set in train_terms.tsv by selecting GO tag associations labeled by any one of 'EXP', 'IDA', 'IMP','IGI', 'HDA', 'HEP', 'HGI', 'HMP', 'HTP', 'IEP', 'IPI', or 'TAS' from one of the UniProt-GOA snapshots. With this list, and using https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.226.gz as input, I am able to get a list of labels that are very close to the contest labels in train_terms.tsv …but the agreement I'm seeing is not quite exact. (And if I use the \"current\" file, https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz, the agreement I see is not quite as good.) I'm hoping to understand the remaining differences. Is it possible I have not found the correct input file?
", "rawMarkdown": "Thanks for this clarification! Can I also check with you that I've got the right set of experimental evidence codes? I am trying to reproduce the label set in train_terms.tsv by selecting GO tag associations labeled by any one of 'EXP', 'IDA', 'IMP','IGI', 'HDA', 'HEP', 'HGI', 'HMP', 'HTP', 'IEP', 'IPI', or 'TAS' from one of the UniProt-GOA snapshots. With this list, and using https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.226.gz as input, I am able to get a list of labels that are very close to the contest labels in train_terms.tsv ...but the agreement I'm seeing is not quite exact. (And if I use the \"current\" file, https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz, the agreement I see is not quite as good.) I'm hoping to understand the remaining differences. Is it possible I have not found the correct input file? ", "votes": 1 }, { "id": 3309940, "postDate": "2025-11-01T18:42:32.740Z", "content": "The goa_uniprot_all.gaf.226.gz file is the correct version, we also used \"IC\" evidence code aside from the ones you listed and excluded annotations with \"NOT\" qualifier. Note that the training set only includes proteins from all SwissProt eukaryotes and 17 bacteria-of-interest as listed in the \"selected-taxon-list.tsv\" file.
", "rawMarkdown": "The goa_uniprot_all.gaf.226.gz file is the correct version, we also used \"IC\" evidence code aside from the ones you listed and excluded annotations with \"NOT\" qualifier. Note that the training set only includes proteins from all SwissProt eukaryotes and 17 bacteria-of-interest as listed in the \"selected-taxon-list.tsv\" file.", "votes": 2 }, { "id": 3309999, "postDate": "2025-11-01T21:06:00.603Z", "content": "These are exactly the hints I needed -- I am now able to reproduce the train labels exactly. Thanks so much!
\nAlso, for anybody else who is reading this thread trying reproduce the train labels independently, there is one more detail I don't think we've mentioned in this conversation: you need to include only those GO tags that are listed in go-basic.obo. Without this selection, I pull in about 41 extra tag associations, but they are all obsolete, e.g. https://www.ebi.ac.uk/QuickGO/term/GO:0004486
", "rawMarkdown": "These are exactly the hints I needed -- I am now able to reproduce the train labels exactly. Thanks so much! \n\nAlso, for anybody else who is reading this thread trying reproduce the train labels independently, there is one more detail I don't think we've mentioned in this conversation: you need to include only those GO tags that are listed in go-basic.obo. Without this selection, I pull in about 41 extra tag associations, but they are all obsolete, e.g. https://www.ebi.ac.uk/QuickGO/term/GO:0004486", "votes": 5 }, { "id": 3336292, "postDate": "2025-11-18T12:35:47.277Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3362293, "postDate": "2025-12-04T13:20:14.737Z", "content": "So, the extra labels are not included in the ia.tsv? Very helpful information
", "rawMarkdown": "So, the extra labels are not included in the ia.tsv? Very helpful information" } ] }, { "id": 3313606, "postDate": "2025-11-09T14:31:23.937Z", "content": "Hi， why in CAFA6, all the training data are included in test data. But in CFAF5, only few of them were included.
", "rawMarkdown": "Hi， why in CAFA6, all the training data are included in test data. But in CFAF5, only few of them were included.", "replies": [ { "id": 3313953, "postDate": "2025-11-10T02:22:01.313Z", "content": "We decided to change the scope of the target proteins.
\nIn CAFA6, we created training and test superset from all eukaryotes + 13 bacteria + 1 archaea in SwissProt (all come from the same testsuperset-taxon-list.tsv), nothing from TrEMBL. CAFA6 test superset has approximately 8,000+ species. All training data proteins are included in the test superset.
CAFA5 training set had some TrEMBL proteins that have experimentally validated annotations and maybe not limited by species. CAFA5 test superset was limited to ~90 species. Since CAFA5 test superset only has 90 species, some training data proteins were not included in the test superset.
", "rawMarkdown": "We decided to change the scope of the target proteins. \n\nIn CAFA6, we created training and test superset from all eukaryotes + 13 bacteria + 1 archaea in SwissProt (all come from the same `testsuperset-taxon-list.tsv`), nothing from TrEMBL. CAFA6 test superset has approximately 8,000+ species. All training data proteins are included in the test superset.\n\nCAFA5 training set had some TrEMBL proteins that have experimentally validated annotations and maybe not limited by species. CAFA5 test superset was limited to ~90 species. Since CAFA5 test superset only has 90 species, some training data proteins were not included in the test superset.\n\n", "votes": 1 } ] } ] } ], "comments": [ { "id": 3303251, "author_name": "An Phan", "author_url": "", "post_date": "2025-10-17T14:02:20.693000", "content": "Hello,
\nAll of the proteins in the training set and test superset this time are from SwissProt. There are no TrEMBL proteins in either set.
\nAll of the proteins in the training set are intentionally included in the test superset, because they might accumulate experimentally-validated functional annotations between the submission deadline and the time of evaluation.
I also realized that some proteins (for example the first one - Q5W0B1), has one DO which is not listed in ncbi (GO:0005515) and also missing one DO which is listed in ncbi (GO:0008270). Why so?
", "votes": 1, "replies": [ { "id": 3305341, "author_name": "An Phan", "author_url": "", "post_date": "2025-10-22T14:33:31.537000", "content": "For Q5W0B1, since the annotation for GO:0008270 is listed with an \"IEA\" evidence code (inferred from Electronic Annotation) which is not an experimentally verified annotation, it is removed from the train_terms.tsv file. We recommend using the UniProt-GOA as the source for GO annotations since this is where we created training data and collect evaluation data later on.
", "votes": 2, "replies": [] }, { "id": 3309934, "author_name": "particlebbq", "author_url": "", "post_date": "2025-11-01T18:21:53.167000", "content": "Thanks for this clarification! Can I also check with you that I've got the right set of experimental evidence codes? I am trying to reproduce the label set in train_terms.tsv by selecting GO tag associations labeled by any one of 'EXP', 'IDA', 'IMP','IGI', 'HDA', 'HEP', 'HGI', 'HMP', 'HTP', 'IEP', 'IPI', or 'TAS' from one of the UniProt-GOA snapshots. With this list, and using https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.226.gz as input, I am able to get a list of labels that are very close to the contest labels in train_terms.tsv …but the agreement I'm seeing is not quite exact. (And if I use the \"current\" file, https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz, the agreement I see is not quite as good.) I'm hoping to understand the remaining differences. Is it possible I have not found the correct input file?
", "votes": 1, "replies": [] }, { "id": 3309940, "author_name": "An Phan", "author_url": "", "post_date": "2025-11-01T18:42:32.740000", "content": "The goa_uniprot_all.gaf.226.gz file is the correct version, we also used \"IC\" evidence code aside from the ones you listed and excluded annotations with \"NOT\" qualifier. Note that the training set only includes proteins from all SwissProt eukaryotes and 17 bacteria-of-interest as listed in the \"selected-taxon-list.tsv\" file.
", "votes": 2, "replies": [] }, { "id": 3309999, "author_name": "particlebbq", "author_url": "", "post_date": "2025-11-01T21:06:00.603000", "content": "These are exactly the hints I needed -- I am now able to reproduce the train labels exactly. Thanks so much!
\nAlso, for anybody else who is reading this thread trying reproduce the train labels independently, there is one more detail I don't think we've mentioned in this conversation: you need to include only those GO tags that are listed in go-basic.obo. Without this selection, I pull in about 41 extra tag associations, but they are all obsolete, e.g. https://www.ebi.ac.uk/QuickGO/term/GO:0004486
", "votes": 5, "replies": [] }, { "id": 3336292, "author_name": "", "author_url": "", "post_date": "2025-11-18T12:35:47.277000", "content": "", "votes": 0, "replies": [] }, { "id": 3362293, "author_name": "common-people", "author_url": "", "post_date": "2025-12-04T13:20:14.737000", "content": "So, the extra labels are not included in the ia.tsv? Very helpful information
", "votes": 0, "replies": [] } ] }, { "id": 3313606, "author_name": "common-people", "author_url": "", "post_date": "2025-11-09T14:31:23.937000", "content": "Hi， why in CAFA6, all the training data are included in test data. But in CFAF5, only few of them were included.
", "votes": 0, "replies": [ { "id": 3313953, "author_name": "An Phan", "author_url": "", "post_date": "2025-11-10T02:22:01.313000", "content": "We decided to change the scope of the target proteins.
\nIn CAFA6, we created training and test superset from all eukaryotes + 13 bacteria + 1 archaea in SwissProt (all come from the same testsuperset-taxon-list.tsv), nothing from TrEMBL. CAFA6 test superset has approximately 8,000+ species. All training data proteins are included in the test superset.
CAFA5 training set had some TrEMBL proteins that have experimentally validated annotations and maybe not limited by species. CAFA5 test superset was limited to ~90 species. Since CAFA5 test superset only has 90 species, some training data proteins were not included in the test superset.
", "votes": 1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3303171": "I noticed that most of the proteins in the testing superset are from SwissProt. Also, many proteins in the training set are included in the testing superset. Is this deliberate?\nAnd there any TrEMBL proteins in the testing superset?", "3303251": "Hello,\nAll of the proteins in the training set and test superset this time are from SwissProt. There are no TrEMBL proteins in either set.\nAll of the proteins in the training set are intentionally included in the test superset, because they might accumulate experimentally-validated functional annotations between the submission deadline and the time of evaluation." } }