I've recently downloaded the QuickGO annotations for the competition proteins, one at a time, using the following API:
\nquickgo_url = f'https://www.ebi.ac.uk/QuickGO/services/annotation/search?geneProductId=UniProtKB:{uniprot_id} \nI completed this process on the evening of April 26th. Using a very crude approach, just adding 0.999 as a score for these annotations, I observed a little jump, with an increase from 0.51 to 0.5499. (our 0.51 score uses only the terms made available by the host). The dictionary of protein to quick-GOs sits here:
\nhttps://www.kaggle.com/datasets/mtinti/quick-go-annotation
If anyone has a better idea for getting these annotations, without downloading/parsing several files from GO, it would be much appreciated!
", "messages": [ { "id": "2236946", "postDate": "04/27/2023 08:32:51", "content": "I've recently downloaded the QuickGO annotations for the competition proteins, one at a time, using the following API:
\nquickgo_url = f'https://www.ebi.ac.uk/QuickGO/services/annotation/search?geneProductId=UniProtKB:{uniprot_id} \nI completed this process on the evening of April 26th. Using a very crude approach, just adding 0.999 as a score for these annotations, I observed a little jump, with an increase from 0.51 to 0.5499. (our 0.51 score uses only the terms made available by the host). The dictionary of protein to quick-GOs sits here:
\nhttps://www.kaggle.com/datasets/mtinti/quick-go-annotation
If anyone has a better idea for getting these annotations, without downloading/parsing several files from GO, it would be much appreciated!
", "rawMarkdown": "I've recently downloaded the QuickGO annotations for the competition proteins, one at a time, using the following API:\n```\nquickgo_url = f'https://www.ebi.ac.uk/QuickGO/services/annotation/search?geneProductId=UniProtKB:{uniprot_id} \n```\nI completed this process on the evening of April 26th. Using a very crude approach, just adding 0.999 as a score for these annotations, I observed a little jump, with an increase from 0.51 to 0.5499. (our 0.51 score uses only the terms made available by the host). The dictionary of protein to quick-GOs sits here: \nhttps://www.kaggle.com/datasets/mtinti/quick-go-annotation\n\nIf anyone has a better idea for getting these annotations, without downloading/parsing several files from GO, it would be much appreciated!", "votes": null }, { "id": "2237020", "postDate": "04/27/2023 10:09:19", "content": "I think this is the actual base https://zenodo.org/record/7796232, but I haven't looked yet.
\n\n\nour 0.51 score uses only the terms made available by the host
\n
Do you mean you got 0.51 by only submitting answers for the proteins that are in the training dataset?
", "rawMarkdown": "I think this is the actual base https://zenodo.org/record/7796232, but I haven't looked yet.\n\n>our 0.51 score uses only the terms made available by the host\n\nDo you mean you got 0.51 by only submitting answers for the proteins that are in the training dataset?", "votes": null }, { "id": "2237030", "postDate": "04/27/2023 10:28:51", "content": "Have you tried multi-threading the download process?
", "rawMarkdown": "Have you tried multi-threading the download process?", "votes": null }, { "id": "2237070", "postDate": "04/27/2023 10:51:51", "content": "Sorry, I meant to say model trained only with the terms made available by the host… Totaly missed the Zenodo file, thanks.
", "rawMarkdown": "Sorry, I meant to say model trained only with the terms made available by the host... Totaly missed the Zenodo file, thanks.", "votes": null }, { "id": "2237075", "postDate": "04/27/2023 10:55:18", "content": "I tried to be nice on the api. Anyway, Sergei pointed to the file that should be used. One could create a notebook that runs automatically now and then to collect the go terms. I wonder in the long term what's going to happen, this file is going to become huge.
", "rawMarkdown": "I tried to be nice on the api. Anyway, Sergei pointed to the file that should be used. One could create a notebook that runs automatically now and then to collect the go terms. I wonder in the long term what's going to happen, this file is going to become huge.", "votes": null }, { "id": "2237139", "postDate": "04/27/2023 12:01:02", "content": "https://www.kaggle.com/sergeifironov/quickgo-1000
\nI made a script to load through the interface, but after 10'000 requests it stopped responding.
", "rawMarkdown": "https://www.kaggle.com/sergeifironov/quickgo-1000\n\nI made a script to load through the interface, but after 10'000 requests it stopped responding.", "votes": null }, { "id": "2237145", "postDate": "04/27/2023 12:06:46", "content": "Maybe someone at GO / EBI might be so kind to extract this data for us for the competition time…
", "rawMarkdown": "Maybe someone at GO / EBI might be so kind to extract this data for us for the competition time...", "votes": null }, { "id": "2237168", "postDate": "04/27/2023 12:27:55", "content": "I believe everything you are looking for is available via uniprotgoa
\nhttps://www.ebi.ac.uk/GOA/downloads
\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/
", "rawMarkdown": "I believe everything you are looking for is available via uniprotgoa\n\nhttps://www.ebi.ac.uk/GOA/downloads\n\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/", "votes": null }, { "id": "2237177", "postDate": "04/27/2023 12:32:33", "content": "doesn't work actually, it seems to get only the last id.
", "rawMarkdown": "~~A friend at EBI suggested using a post request, but unsure how many at time~~ doesn't work actually, it seems to get only the last id.", "votes": null }, { "id": "2237179", "postDate": "04/27/2023 12:36:49", "content": "I believe everything you are looking for is available via uniprotgoa\n\nhttps://www.ebi.ac.uk/GOA/downloads\n\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/\n`
\nThe issue here is that files are split into organisms, and this would require quite some work to map everything..
mmh… quite big, actually…
", "rawMarkdown": "mmh... quite big, actually...", "votes": null }, { "id": "2237487", "postDate": "04/27/2023 16:55:15", "content": "\n\nThe issue here is that files are split into organisms, and this would require quite some work to map everything..
\n
Not only that, many species are omitted, the organism-specific files are only for the common model ones.
\nBut there is one file that contains all of UniproKB GOA annotations. The top one in the list. ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/
\nIt is important to note that in the evaluation we are only be using the GO terms in the .obo file we provided. GO terms are added and removed over time, so you may encounter GO terms in those files that do not exist in the .obo files we provided.
\nI will write a post on this soon, but to make things clear: creating your own training set is strongly encouraged, and is considered a strong component of success in CAFA (literature to be provided in the post). Specifically, amalgamating data from many different resources (not only those) can help generate a comprehensive and informative training set.
", "rawMarkdown": "> The issue here is that files are split into organisms, and this would require quite some work to map everything..\n\nNot only that, many species are omitted, the organism-specific files are only for the common model ones.\n\nBut there is one file that contains all of UniproKB GOA annotations. The top one in the list. ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/\n\nIt is important to note that in the evaluation we are only be using the GO terms in the .obo file we provided. GO terms are added and removed over time, so you may encounter GO terms in those files that do not exist in the .obo files we provided. \n\nI will write a post on this soon, but to make things clear: creating your own training set is strongly encouraged, and is considered a strong component of success in CAFA (literature to be provided in the post). Specifically, amalgamating data from many different [resources] (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/resources) (not only those) can help generate a comprehensive and informative training set.", "votes": null }, { "id": "2238226", "postDate": "04/28/2023 10:20:35", "content": "Thanks for downloading these @mtinti, they will be of interest to lots of us. Now for a silly question: in your dataset, you have the code snippet:
\nfrom tqdm import tqdm\nfile = open('/kaggle/input/quick-go-annotation/all_dict.pkl', 'rb')\ngo_ann = pickle.load(file)\nfor temp_id in tqdm(go_ann.keys()):\n if len(go_ann[temp_id])>0:\n for go in go_ann[temp_id]:\n line = [temp_id,go,'0.999']\n line = '\\t'.join(line)+'\\n'\n sub.write(line) \nThis didn't work for me, and leads me to ask two more questions. What type of object is sub assumed to be, and could you possibly share a notebook that implements this?
", "rawMarkdown": "Thanks for downloading these @mtinti, they will be of interest to lots of us. Now for a silly question: in your dataset, you have the code snippet:\n\n```\nfrom tqdm import tqdm\nfile = open('/kaggle/input/quick-go-annotation/all_dict.pkl', 'rb')\ngo_ann = pickle.load(file)\nfor temp_id in tqdm(go_ann.keys()):\n if len(go_ann[temp_id])>0:\n for go in go_ann[temp_id]:\n line = [temp_id,go,'0.999']\n line = '\\t'.join(line)+'\\n'\n sub.write(line) \n```\nThis didn't work for me, and leads me to ask two more questions. What type of object is sub assumed to be, and could you possibly share a notebook that implements this?", "votes": null }, { "id": "2238306", "postDate": "04/28/2023 11:55:52", "content": "Apologies, I was using a different version see this notebook https://www.kaggle.com/mtinti/test-notebook-quickgo.
\nall_dict.pkl is a dictionary in the format {PROT_ID:[GO1,GO2,GO3…], sub was my submission txt file.
@mtinti - thanks!
", "rawMarkdown": "mtinti - thanks!", "votes": null }, { "id": "2238494", "postDate": "04/28/2023 15:14:54", "content": "No problem. I will draft a mail to submit here: https://www.ebi.ac.uk/QuickGO/feedback. I wonder if they can consider assembling a dataset automatically for us while the competition is running.
", "rawMarkdown": "No problem. I will draft a mail to submit here: https://www.ebi.ac.uk/QuickGO/feedback. I wonder if they can consider assembling a dataset automatically for us while the competition is running.", "votes": null }, { "id": "2239238", "postDate": "04/29/2023 09:31:47", "content": "I recently sent a message to the QuickGO team through their feedback page (https://www.ebi.ac.uk/QuickGO/feedback). QuickGO, along with many other EBI resources, plays a crucial role in the day-to-day activities of our research labs. We are truly grateful for their invaluable contributions. In my message, I kindly inquired about the possibility of them assisting us with a data dump for the proteins relevant to this competition. I believe if more of us reach out with similar requests, it would help raise awareness and potentially facilitate a positive response from the QuickGO team.
", "rawMarkdown": "I recently sent a message to the QuickGO team through their feedback page (https://www.ebi.ac.uk/QuickGO/feedback). QuickGO, along with many other EBI resources, plays a crucial role in the day-to-day activities of our research labs. We are truly grateful for their invaluable contributions. In my message, I kindly inquired about the possibility of them assisting us with a data dump for the proteins relevant to this competition. I believe if more of us reach out with similar requests, it would help raise awareness and potentially facilitate a positive response from the QuickGO team.", "votes": null }, { "id": "2340761", "postDate": "07/11/2023 17:02:43", "content": "Hi @mtinti,
\nI'm currently trying to reproduce the dataset provided by the host, but encountering some issues described as follows,
\nI can't find all proteins of train_terms.tsv in goa_uniprot_all.gaf.gz downloaded from this site.
train_terms.tsv.Protein like A0A2R8QPZ7 seems missing in QuickGO. No result is returned when I search for the annotation of this protein.
If there's any mistake, please let me know. Thanks a lot!
", "rawMarkdown": "Hi @mtinti,\n\nI'm currently trying to reproduce the dataset provided by the host, but encountering some issues described as follows,\n\n1. I can't find all proteins of `train_terms.tsv` in `goa_uniprot_all.gaf.gz` downloaded from [this site](http://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/).\n * Number of overlapping proteins is 21245, but there are 142246 in `train_terms.tsv`.\n\n2. Protein like `A0A2R8QPZ7` seems missing in [QuickGO](https://www.ebi.ac.uk/QuickGO/). No result is returned when I search for the annotation of this protein.\n\nIf there's any mistake, please let me know. Thanks a lot!", "votes": null }, { "id": "2340800", "postDate": "07/11/2023 17:22:26", "content": "\n\nI can't find all proteins of train_terms.tsv in goa_uniprot_all.gaf.gz
\n
GOA only includes GO annotated proteins. Some in the train_terms.tsv are not. The full corpus of proteins is in Uniprot and TrEMBL available for download here: https://www.uniprot.org/help/downloads
\n\n\nProtein like A0A2R8QPZ7 seems missing in QuickGO. No result is returned when I search for the annotation of this protein.
\n
This specific entry has become obsolete shortly after we compiled the set. These things happen, although very rarely.
", "rawMarkdown": "> I can't find all proteins of train_terms.tsv in goa_uniprot_all.gaf.gz\n\nGOA only includes GO annotated proteins. Some in the train_terms.tsv are not. The full corpus of proteins is in Uniprot and TrEMBL available for download here: https://www.uniprot.org/help/downloads\n\n\n\n> Protein like A0A2R8QPZ7 seems missing in QuickGO. No result is returned when I search for the annotation of this protein.\n\nThis specific entry has become obsolete shortly after we compiled the set. These things happen, although very rarely.", "votes": null }, { "id": "2341305", "postDate": "07/12/2023 03:13:54", "content": "Hi @iddux1,
\nSorry for the silly question, but I couldn't get the first answer. As described in the data page, train_terms.tsv includes annotated terms that have been validated. As for GOA, it only includes GO annotated proteins. Then, what's the difference between these two? It looks like they both have proteins with GO annotations, but only 21245 proteins of goa_uniprot_all.gaf.gz appear in train_terms.tsv.
The motivation of reproducing the dataset is that I want to try CV scheme based on the time of annotation. Hence, I need information about the timestamp of each experimentally-validated (protein, GO term) pair (i.e., when the protein gains experimental annotation). However, I still can't find this complete information.
\nThanks for your quick reply!
", "rawMarkdown": "Hi @iddux1,\n\nSorry for the silly question, but I couldn't get the first answer. As described in the [data page](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/data), `train_terms.tsv` includes annotated terms that have been validated. As for GOA, it only includes GO annotated proteins. Then, what's the difference between these two? It looks like they both have proteins with GO annotations, but only 21245 proteins of `goa_uniprot_all.gaf.gz` appear in `train_terms.tsv`.\n\nThe motivation of reproducing the dataset is that I want to try CV scheme based on **the time of annotation**. Hence, I need information about **the timestamp** of each experimentally-validated (protein, GO term) pair (*i.e.,* **when** the protein gains experimental annotation). However, I still can't find this complete information.\n\nThanks for your quick reply!", "votes": null }, { "id": "2342145", "postDate": "07/12/2023 14:55:50", "content": "Hi @abaojiang,
\nAre you filtering by the same evidence codes? We include experimental (EXP, IDA, IPI, IMP, IGI, IEP), high-throughput (HTP,HDA, HMP,HGI,HEP) as well as TAS and IC. There should be many proteins that only have, for example, ISS evidence codes and we don't include those in our dataset.
", "rawMarkdown": "Hi @abaojiang, \n\nAre you filtering by the same evidence codes? We include experimental (EXP, IDA, IPI, IMP, IGI, IEP), high-throughput (HTP,HDA, HMP,HGI,HEP) as well as TAS and IC. There should be many proteins that only have, for example, ISS evidence codes and we don't include those in our dataset.", "votes": null }, { "id": "2342149", "postDate": "07/12/2023 14:57:07", "content": "If you're taking this approach, remember to propagate the dates to the propagated terms, too.
", "rawMarkdown": "If you're taking this approach, remember to propagate the dates to the propagated terms, too.", "votes": null }, { "id": "2342224", "postDate": "07/12/2023 15:55:26", "content": "Hi @claradepaolis,
\nBecause I want to make sure that all 142246 proteins of train_terms.tsv appear in goa_uniprot_all.gaf.gz, I retrieve all unique proteins from goa_uniprot_all.gaf.gz and get 170706901 in total. And I observe that there are only 21245 overlapping proteins. Maybe I did something wrong, but just want to make sure whether all of the 142246 proteins of train_terms.tsv exist in goa_uniprot_all.gaf.gz or not.
As for the date propagation, do you mean that if one protein gains its first experimental annotation in Jan, 2023 (for example), then all of its parents should be tagged with the exact timestamp? Also, is there any recommended way to get the timestamp of each (protein, term) pair?
\nThanks for being patient with me!
", "rawMarkdown": "Hi @claradepaolis,\n\nBecause I want to make sure that all 142246 proteins of `train_terms.tsv` appear in `goa_uniprot_all.gaf.gz`, I retrieve all unique proteins from `goa_uniprot_all.gaf.gz` and get 170706901 in total. And I observe that there are **only 21245 overlapping proteins**. Maybe I did something wrong, but just want to make sure whether all of the 142246 proteins of `train_terms.tsv` exist in `goa_uniprot_all.gaf.gz` or not.\n\nAs for the date propagation, do you mean that if one protein gains its **first** experimental annotation in Jan, 2023 (for example), then all of its parents should be tagged with the exact timestamp? Also, is there any recommended way to get the timestamp of each (protein, term) pair?\n\nThanks for being patient with me!", "votes": null }, { "id": "2342433", "postDate": "07/12/2023 19:51:54", "content": "What is the goa_uniprot_all.gaf.gz version you currently looking at?
\nThe train_terms.tsv came from 2022-11-17 release. This might be the reason why you didn't find all training terms on goa file.
Hi @evansliu,
\nI download 2023-06-11 release in this page and extract all of the proteins in this file. Then, I compare this protein set with the set containing 142246 proteins of train_terms.tsv. After comparison, I only find 21245 proteins in the intersection set.
Thanks for the reply!
", "rawMarkdown": "Hi @evansliu,\n\nI download 2023-06-11 release in [this page](http://release.geneontology.org/2023-06-11/annotations/index.html) and extract all of the proteins in this file. Then, I compare this protein set with the set containing 142246 proteins of `train_terms.tsv`. After comparison, I only find 21245 proteins in the intersection set.\n\nThanks for the reply!", "votes": null }, { "id": "2342704", "postDate": "07/13/2023 04:39:52", "content": "If you want to get the timestamp of each (protein, term) pair, I would suggest you download the goa files from multiple releases and compare the new release to the old release. Then you are able to find No knowledge proteins and limited-knowledge proteins which were used as evaluation targets for the cafa.
", "rawMarkdown": "If you want to get the timestamp of each (protein, term) pair, I would suggest you download the goa files from multiple releases and compare the new release to the old release. Then you are able to find No knowledge proteins and limited-knowledge proteins which were used as evaluation targets for the cafa.", "votes": null }, { "id": "2342711", "postDate": "07/13/2023 04:47:29", "content": "Hi @evansliu,
\nGot it, I'll give it a try. Thanks again for your quick reply!
", "rawMarkdown": "Hi @evansliu,\n\nGot it, I'll give it a try. Thanks again for your quick reply!", "votes": null } ], "comments": [ { "id": 2237020, "author_name": "sergeifironov", "author_url": "", "post_date": "04/27/2023 10:09:19", "content": "I think this is the actual base https://zenodo.org/record/7796232, but I haven't looked yet.
\n\n\nour 0.51 score uses only the terms made available by the host
\n
Do you mean you got 0.51 by only submitting answers for the proteins that are in the training dataset?
", "votes": null, "replies": [ { "id": 2237070, "author_name": "mtinti", "author_url": "", "post_date": "04/27/2023 10:51:51", "content": "Sorry, I meant to say model trained only with the terms made available by the host… Totaly missed the Zenodo file, thanks.
", "votes": null, "replies": [] }, { "id": 2237241, "author_name": "mtinti", "author_url": "", "post_date": "04/27/2023 13:26:10", "content": "mmh… quite big, actually…
", "votes": null, "replies": [] } ] }, { "id": 2237030, "author_name": "munumbutt", "author_url": "", "post_date": "04/27/2023 10:28:51", "content": "Have you tried multi-threading the download process?
", "votes": null, "replies": [ { "id": 2237075, "author_name": "mtinti", "author_url": "", "post_date": "04/27/2023 10:55:18", "content": "I tried to be nice on the api. Anyway, Sergei pointed to the file that should be used. One could create a notebook that runs automatically now and then to collect the go terms. I wonder in the long term what's going to happen, this file is going to become huge.
", "votes": null, "replies": [ { "id": 2237139, "author_name": "sergeifironov", "author_url": "", "post_date": "04/27/2023 12:01:02", "content": "https://www.kaggle.com/sergeifironov/quickgo-1000
\nI made a script to load through the interface, but after 10'000 requests it stopped responding.
", "votes": null, "replies": [ { "id": 2237145, "author_name": "mtinti", "author_url": "", "post_date": "04/27/2023 12:06:46", "content": "Maybe someone at GO / EBI might be so kind to extract this data for us for the competition time…
", "votes": null, "replies": [ { "id": 2237168, "author_name": "iddux1", "author_url": "", "post_date": "04/27/2023 12:27:55", "content": "I believe everything you are looking for is available via uniprotgoa
\nhttps://www.ebi.ac.uk/GOA/downloads
\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/
", "votes": null, "replies": [ { "id": 2237179, "author_name": "mtinti", "author_url": "", "post_date": "04/27/2023 12:36:49", "content": "I believe everything you are looking for is available via uniprotgoa\n\nhttps://www.ebi.ac.uk/GOA/downloads\n\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/\n`
\nThe issue here is that files are split into organisms, and this would require quite some work to map everything..
\n\nThe issue here is that files are split into organisms, and this would require quite some work to map everything..
\n
Not only that, many species are omitted, the organism-specific files are only for the common model ones.
\nBut there is one file that contains all of UniproKB GOA annotations. The top one in the list. ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/
\nIt is important to note that in the evaluation we are only be using the GO terms in the .obo file we provided. GO terms are added and removed over time, so you may encounter GO terms in those files that do not exist in the .obo files we provided.
\nI will write a post on this soon, but to make things clear: creating your own training set is strongly encouraged, and is considered a strong component of success in CAFA (literature to be provided in the post). Specifically, amalgamating data from many different resources (not only those) can help generate a comprehensive and informative training set.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2237177, "author_name": "mtinti", "author_url": "", "post_date": "04/27/2023 12:32:33", "content": "doesn't work actually, it seems to get only the last id.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2238226, "author_name": "jbomitchell", "author_url": "", "post_date": "04/28/2023 10:20:35", "content": "Thanks for downloading these @mtinti, they will be of interest to lots of us. Now for a silly question: in your dataset, you have the code snippet:
\nfrom tqdm import tqdm\nfile = open('/kaggle/input/quick-go-annotation/all_dict.pkl', 'rb')\ngo_ann = pickle.load(file)\nfor temp_id in tqdm(go_ann.keys()):\n if len(go_ann[temp_id])>0:\n for go in go_ann[temp_id]:\n line = [temp_id,go,'0.999']\n line = '\\t'.join(line)+'\\n'\n sub.write(line) \nThis didn't work for me, and leads me to ask two more questions. What type of object is sub assumed to be, and could you possibly share a notebook that implements this?
", "votes": null, "replies": [ { "id": 2238306, "author_name": "mtinti", "author_url": "", "post_date": "04/28/2023 11:55:52", "content": "Apologies, I was using a different version see this notebook https://www.kaggle.com/mtinti/test-notebook-quickgo.
\nall_dict.pkl is a dictionary in the format {PROT_ID:[GO1,GO2,GO3…], sub was my submission txt file.
@mtinti - thanks!
", "votes": null, "replies": [ { "id": 2238494, "author_name": "mtinti", "author_url": "", "post_date": "04/28/2023 15:14:54", "content": "No problem. I will draft a mail to submit here: https://www.ebi.ac.uk/QuickGO/feedback. I wonder if they can consider assembling a dataset automatically for us while the competition is running.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2239238, "author_name": "mtinti", "author_url": "", "post_date": "04/29/2023 09:31:47", "content": "I recently sent a message to the QuickGO team through their feedback page (https://www.ebi.ac.uk/QuickGO/feedback). QuickGO, along with many other EBI resources, plays a crucial role in the day-to-day activities of our research labs. We are truly grateful for their invaluable contributions. In my message, I kindly inquired about the possibility of them assisting us with a data dump for the proteins relevant to this competition. I believe if more of us reach out with similar requests, it would help raise awareness and potentially facilitate a positive response from the QuickGO team.
", "votes": null, "replies": [] }, { "id": 2340761, "author_name": "abaojiang", "author_url": "", "post_date": "07/11/2023 17:02:43", "content": "Hi @mtinti,
\nI'm currently trying to reproduce the dataset provided by the host, but encountering some issues described as follows,
\nI can't find all proteins of train_terms.tsv in goa_uniprot_all.gaf.gz downloaded from this site.
train_terms.tsv.Protein like A0A2R8QPZ7 seems missing in QuickGO. No result is returned when I search for the annotation of this protein.
If there's any mistake, please let me know. Thanks a lot!
", "votes": null, "replies": [ { "id": 2340800, "author_name": "iddux1", "author_url": "", "post_date": "07/11/2023 17:22:26", "content": "\n\nI can't find all proteins of train_terms.tsv in goa_uniprot_all.gaf.gz
\n
GOA only includes GO annotated proteins. Some in the train_terms.tsv are not. The full corpus of proteins is in Uniprot and TrEMBL available for download here: https://www.uniprot.org/help/downloads
\n\n\nProtein like A0A2R8QPZ7 seems missing in QuickGO. No result is returned when I search for the annotation of this protein.
\n
This specific entry has become obsolete shortly after we compiled the set. These things happen, although very rarely.
", "votes": null, "replies": [ { "id": 2341305, "author_name": "abaojiang", "author_url": "", "post_date": "07/12/2023 03:13:54", "content": "Hi @iddux1,
\nSorry for the silly question, but I couldn't get the first answer. As described in the data page, train_terms.tsv includes annotated terms that have been validated. As for GOA, it only includes GO annotated proteins. Then, what's the difference between these two? It looks like they both have proteins with GO annotations, but only 21245 proteins of goa_uniprot_all.gaf.gz appear in train_terms.tsv.
The motivation of reproducing the dataset is that I want to try CV scheme based on the time of annotation. Hence, I need information about the timestamp of each experimentally-validated (protein, GO term) pair (i.e., when the protein gains experimental annotation). However, I still can't find this complete information.
\nThanks for your quick reply!
", "votes": null, "replies": [ { "id": 2342145, "author_name": "claradepaolis", "author_url": "", "post_date": "07/12/2023 14:55:50", "content": "Hi @abaojiang,
\nAre you filtering by the same evidence codes? We include experimental (EXP, IDA, IPI, IMP, IGI, IEP), high-throughput (HTP,HDA, HMP,HGI,HEP) as well as TAS and IC. There should be many proteins that only have, for example, ISS evidence codes and we don't include those in our dataset.
", "votes": null, "replies": [ { "id": 2342224, "author_name": "abaojiang", "author_url": "", "post_date": "07/12/2023 15:55:26", "content": "Hi @claradepaolis,
\nBecause I want to make sure that all 142246 proteins of train_terms.tsv appear in goa_uniprot_all.gaf.gz, I retrieve all unique proteins from goa_uniprot_all.gaf.gz and get 170706901 in total. And I observe that there are only 21245 overlapping proteins. Maybe I did something wrong, but just want to make sure whether all of the 142246 proteins of train_terms.tsv exist in goa_uniprot_all.gaf.gz or not.
As for the date propagation, do you mean that if one protein gains its first experimental annotation in Jan, 2023 (for example), then all of its parents should be tagged with the exact timestamp? Also, is there any recommended way to get the timestamp of each (protein, term) pair?
\nThanks for being patient with me!
", "votes": null, "replies": [ { "id": 2342433, "author_name": "evansliu", "author_url": "", "post_date": "07/12/2023 19:51:54", "content": "What is the goa_uniprot_all.gaf.gz version you currently looking at?
\nThe train_terms.tsv came from 2022-11-17 release. This might be the reason why you didn't find all training terms on goa file.
Hi @evansliu,
\nI download 2023-06-11 release in this page and extract all of the proteins in this file. Then, I compare this protein set with the set containing 142246 proteins of train_terms.tsv. After comparison, I only find 21245 proteins in the intersection set.
Thanks for the reply!
", "votes": null, "replies": [ { "id": 2342704, "author_name": "evansliu", "author_url": "", "post_date": "07/13/2023 04:39:52", "content": "If you want to get the timestamp of each (protein, term) pair, I would suggest you download the goa files from multiple releases and compare the new release to the old release. Then you are able to find No knowledge proteins and limited-knowledge proteins which were used as evaluation targets for the cafa.
", "votes": null, "replies": [ { "id": 2342711, "author_name": "abaojiang", "author_url": "", "post_date": "07/13/2023 04:47:29", "content": "Hi @evansliu,
\nGot it, I'll give it a try. Thanks again for your quick reply!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2342149, "author_name": "claradepaolis", "author_url": "", "post_date": "07/12/2023 14:57:07", "content": "If you're taking this approach, remember to propagate the dates to the propagated terms, too.
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2236946": "I've recently downloaded the QuickGO annotations for the competition proteins, one at a time, using the following API:\n```\nquickgo_url = f'https://www.ebi.ac.uk/QuickGO/services/annotation/search?geneProductId=UniProtKB:{uniprot_id} \n```\nI completed this process on the evening of April 26th. Using a very crude approach, just adding 0.999 as a score for these annotations, I observed a little jump, with an increase from 0.51 to 0.5499. (our 0.51 score uses only the terms made available by the host). The dictionary of protein to quick-GOs sits here: \nhttps://www.kaggle.com/datasets/mtinti/quick-go-annotation\n\nIf anyone has a better idea for getting these annotations, without downloading/parsing several files from GO, it would be much appreciated!", "2237020": "I think this is the actual base https://zenodo.org/record/7796232, but I haven't looked yet.\n\n>our 0.51 score uses only the terms made available by the host\n\nDo you mean you got 0.51 by only submitting answers for the proteins that are in the training dataset?", "2237030": "Have you tried multi-threading the download process?", "2237070": "Sorry, I meant to say model trained only with the terms made available by the host... Totaly missed the Zenodo file, thanks.", "2237075": "I tried to be nice on the api. Anyway, Sergei pointed to the file that should be used. One could create a notebook that runs automatically now and then to collect the go terms. I wonder in the long term what's going to happen, this file is going to become huge.", "2237139": "https://www.kaggle.com/sergeifironov/quickgo-1000\n\nI made a script to load through the interface, but after 10'000 requests it stopped responding.", "2237145": "Maybe someone at GO / EBI might be so kind to extract this data for us for the competition time...", "2237168": "I believe everything you are looking for is available via uniprotgoa\n\nhttps://www.ebi.ac.uk/GOA/downloads\n\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/", "2237177": "~~A friend at EBI suggested using a post request, but unsure how many at time~~ doesn't work actually, it seems to get only the last id.", "2237179": "```\nI believe everything you are looking for is available via uniprotgoa\n\nhttps://www.ebi.ac.uk/GOA/downloads\n\nAnd GO itself: http://geneontology.org/docs/download-go-annotations/\n````\nThe issue here is that files are split into organisms, and this would require quite some work to map everything..", "2237241": "mmh... quite big, actually...", "2237487": "> The issue here is that files are split into organisms, and this would require quite some work to map everything..\n\nNot only that, many species are omitted, the organism-specific files are only for the common model ones.\n\nBut there is one file that contains all of UniproKB GOA annotations. The top one in the list. ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/\n\nIt is important to note that in the evaluation we are only be using the GO terms in the .obo file we provided. GO terms are added and removed over time, so you may encounter GO terms in those files that do not exist in the .obo files we provided. \n\nI will write a post on this soon, but to make things clear: creating your own training set is strongly encouraged, and is considered a strong component of success in CAFA (literature to be provided in the post). Specifically, amalgamating data from many different [resources] (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/resources) (not only those) can help generate a comprehensive and informative training set.", "2238226": "Thanks for downloading these @mtinti, they will be of interest to lots of us. Now for a silly question: in your dataset, you have the code snippet:\n\n```\nfrom tqdm import tqdm\nfile = open('/kaggle/input/quick-go-annotation/all_dict.pkl', 'rb')\ngo_ann = pickle.load(file)\nfor temp_id in tqdm(go_ann.keys()):\n if len(go_ann[temp_id])>0:\n for go in go_ann[temp_id]:\n line = [temp_id,go,'0.999']\n line = '\\t'.join(line)+'\\n'\n sub.write(line) \n```\nThis didn't work for me, and leads me to ask two more questions. What type of object is sub assumed to be, and could you possibly share a notebook that implements this?", "2238306": "Apologies, I was using a different version see this notebook https://www.kaggle.com/mtinti/test-notebook-quickgo. \nall_dict.pkl is a dictionary in the format {PROT_ID:[GO1,GO2,GO3...], sub was my submission txt file.", "2238478": "mtinti - thanks!", "2238494": "No problem. I will draft a mail to submit here: https://www.ebi.ac.uk/QuickGO/feedback. I wonder if they can consider assembling a dataset automatically for us while the competition is running.", "2239238": "I recently sent a message to the QuickGO team through their feedback page (https://www.ebi.ac.uk/QuickGO/feedback). QuickGO, along with many other EBI resources, plays a crucial role in the day-to-day activities of our research labs. We are truly grateful for their invaluable contributions. In my message, I kindly inquired about the possibility of them assisting us with a data dump for the proteins relevant to this competition. I believe if more of us reach out with similar requests, it would help raise awareness and potentially facilitate a positive response from the QuickGO team.", "2340761": "Hi @mtinti,\n\nI'm currently trying to reproduce the dataset provided by the host, but encountering some issues described as follows,\n\n1. I can't find all proteins of `train_terms.tsv` in `goa_uniprot_all.gaf.gz` downloaded from [this site](http://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/).\n * Number of overlapping proteins is 21245, but there are 142246 in `train_terms.tsv`.\n\n2. Protein like `A0A2R8QPZ7` seems missing in [QuickGO](https://www.ebi.ac.uk/QuickGO/). No result is returned when I search for the annotation of this protein.\n\nIf there's any mistake, please let me know. Thanks a lot!", "2340800": "> I can't find all proteins of train_terms.tsv in goa_uniprot_all.gaf.gz\n\nGOA only includes GO annotated proteins. Some in the train_terms.tsv are not. The full corpus of proteins is in Uniprot and TrEMBL available for download here: https://www.uniprot.org/help/downloads\n\n\n\n> Protein like A0A2R8QPZ7 seems missing in QuickGO. No result is returned when I search for the annotation of this protein.\n\nThis specific entry has become obsolete shortly after we compiled the set. These things happen, although very rarely.", "2341305": "Hi @iddux1,\n\nSorry for the silly question, but I couldn't get the first answer. As described in the [data page](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/data), `train_terms.tsv` includes annotated terms that have been validated. As for GOA, it only includes GO annotated proteins. Then, what's the difference between these two? It looks like they both have proteins with GO annotations, but only 21245 proteins of `goa_uniprot_all.gaf.gz` appear in `train_terms.tsv`.\n\nThe motivation of reproducing the dataset is that I want to try CV scheme based on **the time of annotation**. Hence, I need information about **the timestamp** of each experimentally-validated (protein, GO term) pair (*i.e.,* **when** the protein gains experimental annotation). However, I still can't find this complete information.\n\nThanks for your quick reply!", "2342145": "Hi @abaojiang, \n\nAre you filtering by the same evidence codes? We include experimental (EXP, IDA, IPI, IMP, IGI, IEP), high-throughput (HTP,HDA, HMP,HGI,HEP) as well as TAS and IC. There should be many proteins that only have, for example, ISS evidence codes and we don't include those in our dataset.", "2342149": "If you're taking this approach, remember to propagate the dates to the propagated terms, too.", "2342224": "Hi @claradepaolis,\n\nBecause I want to make sure that all 142246 proteins of `train_terms.tsv` appear in `goa_uniprot_all.gaf.gz`, I retrieve all unique proteins from `goa_uniprot_all.gaf.gz` and get 170706901 in total. And I observe that there are **only 21245 overlapping proteins**. Maybe I did something wrong, but just want to make sure whether all of the 142246 proteins of `train_terms.tsv` exist in `goa_uniprot_all.gaf.gz` or not.\n\nAs for the date propagation, do you mean that if one protein gains its **first** experimental annotation in Jan, 2023 (for example), then all of its parents should be tagged with the exact timestamp? Also, is there any recommended way to get the timestamp of each (protein, term) pair?\n\nThanks for being patient with me!", "2342433": "What is the goa_uniprot_all.gaf.gz version you currently looking at?\nThe train_terms.tsv came from 2022-11-17 release. This might be the reason why you didn't find all training terms on goa file.", "2342640": "Hi @evansliu,\n\nI download 2023-06-11 release in [this page](http://release.geneontology.org/2023-06-11/annotations/index.html) and extract all of the proteins in this file. Then, I compare this protein set with the set containing 142246 proteins of `train_terms.tsv`. After comparison, I only find 21245 proteins in the intersection set.\n\nThanks for the reply!", "2342704": "If you want to get the timestamp of each (protein, term) pair, I would suggest you download the goa files from multiple releases and compare the new release to the old release. Then you are able to find No knowledge proteins and limited-knowledge proteins which were used as evaluation targets for the cafa.", "2342711": "Hi @evansliu,\n\nGot it, I'll give it a try. Thanks again for your quick reply!" }, "source": "meta" }