{ "id": 419802, "title": "0.6 and beyond - a literature review for better ideas", "url": "/competitions/cafa-5-protein-function-prediction/discussion/419802", "author_name": "", "post_date": "2023-06-27T17:01:55.621035600Z", "votes": 16, "comment_count": 7, "views": 0, "content": "

Let us assume that one can create a model that has predictive power of Fmax > 0.6 on a novel set of proteins for GO terms. We already have an example on the leaderboard of a model that has achieved such a score which we can assume does not represent this phenomenon termed 'overfitting'. I would like to suggest here a few approaches that might help a model achieve such a score and leave it up to others to come up with other ideas on how to do so.

\n

What are the established methods for creating automatic function prediction software? Let us turn to the published literature to learn from the best. Firstly, let us look at the papers describing the software package that won CAFA3 and its descendants by Zhu lab:
\nGOLabeler (2018): https://academic.oup.com/bioinformatics/article/34/14/2465/4924212
\nDeepText2GO (2018): https://pubmed.ncbi.nlm.nih.gov/29883746/
\nNetGO (2019): https://academic.oup.com/nar/article/47/W1/W379/5491749
\nNetGO 2 (2021): https://academic.oup.com/nar/article/49/W1/W469/6285266
\nNetGO 3 (2022): https://www.biorxiv.org/content/10.1101/2022.12.05.519073v1
\nHere we see an ensemble model approach which has been improved with only minor revisions over the years, the latest iteration including the ESM large protein language model (LR-ESM). I include here two quotes that are relevant for thinking of this problem

\n

From Supplementary Materials of “GOLabeler: Improving Sequence-based Large-scale Protein Function Prediction by Learning to Rank” (Jan 2018)
\n“ The experimental results of GOLabeler demonstrate that, the good performance of GOLabeler is largely attributed to two component methods, BLAST-KNN and LR-Interpro. In contrast to BLAST-KNN utilizing protein homology information, LR-Interpro makes use of domain, family and motif information of protein for function prediction. This means that only sequence similarity is insufficient for the good performance of GOLabeler.”

\n

From NetGO 3.0: Protein Language Model Improves Large-scale 3 Functional Annotations (Dec 2022)
\n“In Table 8, it is clear that BLAST-KNN failed to achieve a valid result because homology-based methods were not suitable for difficult protein function prediction. LR-InterPro and LR-ESM extracted features from raw amino acid sequences and obtained better results than BLAST-KNN. In the top 20 predicted GO terms, the number of true positive samples achieved by LR-ESM is significantly larger than other methods, which predicted 14 correct function labels.”

\n

None of the code underlying these models appear to be available, and we can only attempt to recreate the models presented in these papers, although the predictions produced by NetGO3.0 are available from their web server (https://dmiip.sjtu.edu.cn/ng3.0/). I invite any enterprising individuals to extract the full predicted values from the server to integrate into their model (and to share!).

\n

The model used for NetGO2 is shown here:
\n\"\"

\n

There is one other lab whose work we must also examine - they did decently in CAFA3 and are currently atop the leaderboard with a commanding lead over all other models (Hoehndorf lab).

\n

DeepGO (2018): https://academic.oup.com/bioinformatics/article/34/4/660/4265461
\nDeepGOPlus (2020): https://academic.oup.com/bioinformatics/article/36/2/422/5539866
\nDeepGOZero (2022): https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515

\n

Here are two quotes from the discussion section of DeepGOPlus :
\n“In DeepGOPlus, we combine similarity-based search to proteins with known functions and motif-based function prediction, and this combination gives us overall the best predictive performance. However, DeepGOPlus can also be applied using only sequence motifs; in particular when annotating novel proteins for which no similar proteins with known functions exist, our motif-based model would be most suitable.”

\n

“We have already experimented with several types of neural networks such as recurrent neural networks, long-short term memory networks and autoencoders to learn sequence features. However, our attempts were unsuccessful and CNNs gave us the best results.”

\n

Finally lets look at one of their most recent pre-prints which proposes a method to integrate ESM-2 with a Transformer GCN for a protein binding prediction task.
\n\"\"

\n

(2023) https://www.biorxiv.org/content/10.1101/2023.03.14.532563v1.abstract
\nIt is not difficult to imagine integrating such an approach into the current problem of GO prediction.

\n

Other than the papers I have highlighted here, there are numerous other publications with credible and somewhat less credible claims on their accuracy. I have included here a short list of these that will make for decent reading material or for the next review article on the subject that I am sure is coming soon given the recent attention lavished on the subject. The list is incomplete, non-exhaustive, and I make no effort here to summarize the approaches that have been taken.

\n

2023
\nhttps://pubmed.ncbi.nlm.nih.gov/36919205/
\nhttps://academic.oup.com/bioinformatics/article/39/3/btad094/7043095
\nhttps://www.frontiersin.org/articles/10.3389/fgene.2022.969915/full
\nhttps://elifesciences.org/articles/80942

\n

2022
\nhttps://pubmed.ncbi.nlm.nih.gov/36421723/
\nhttps://www.nature.com/articles/s41587-021-01179-w
\nhttps://www.biorxiv.org/content/10.1101/2022.12.05.519119v1.full.pdf
\nhttps://academic.oup.com/nar/article/50/W1/W228/6576357
\nhttps://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515
\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010793
\nhttps://academic.oup.com/bioinformatics/article/38/8/2102/6502274
\nhttps://academic.oup.com/bioinformatics/article/38/19/4488/6656346
\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010075

\n

2021
\nhttps://www.nature.com/articles/s41598-020-80786-0
\nhttps://www.nature.com/articles/s41467-021-23303-9

", "messages": [ { "id": "2320292", "postDate": "06/27/2023 17:01:55", "content": "

Let us assume that one can create a model that has predictive power of Fmax > 0.6 on a novel set of proteins for GO terms. We already have an example on the leaderboard of a model that has achieved such a score which we can assume does not represent this phenomenon termed 'overfitting'. I would like to suggest here a few approaches that might help a model achieve such a score and leave it up to others to come up with other ideas on how to do so.

\n

What are the established methods for creating automatic function prediction software? Let us turn to the published literature to learn from the best. Firstly, let us look at the papers describing the software package that won CAFA3 and its descendants by Zhu lab:
\nGOLabeler (2018): https://academic.oup.com/bioinformatics/article/34/14/2465/4924212
\nDeepText2GO (2018): https://pubmed.ncbi.nlm.nih.gov/29883746/
\nNetGO (2019): https://academic.oup.com/nar/article/47/W1/W379/5491749
\nNetGO 2 (2021): https://academic.oup.com/nar/article/49/W1/W469/6285266
\nNetGO 3 (2022): https://www.biorxiv.org/content/10.1101/2022.12.05.519073v1
\nHere we see an ensemble model approach which has been improved with only minor revisions over the years, the latest iteration including the ESM large protein language model (LR-ESM). I include here two quotes that are relevant for thinking of this problem

\n

From Supplementary Materials of “GOLabeler: Improving Sequence-based Large-scale Protein Function Prediction by Learning to Rank” (Jan 2018)
\n“ The experimental results of GOLabeler demonstrate that, the good performance of GOLabeler is largely attributed to two component methods, BLAST-KNN and LR-Interpro. In contrast to BLAST-KNN utilizing protein homology information, LR-Interpro makes use of domain, family and motif information of protein for function prediction. This means that only sequence similarity is insufficient for the good performance of GOLabeler.”

\n

From NetGO 3.0: Protein Language Model Improves Large-scale 3 Functional Annotations (Dec 2022)
\n“In Table 8, it is clear that BLAST-KNN failed to achieve a valid result because homology-based methods were not suitable for difficult protein function prediction. LR-InterPro and LR-ESM extracted features from raw amino acid sequences and obtained better results than BLAST-KNN. In the top 20 predicted GO terms, the number of true positive samples achieved by LR-ESM is significantly larger than other methods, which predicted 14 correct function labels.”

\n

None of the code underlying these models appear to be available, and we can only attempt to recreate the models presented in these papers, although the predictions produced by NetGO3.0 are available from their web server (https://dmiip.sjtu.edu.cn/ng3.0/). I invite any enterprising individuals to extract the full predicted values from the server to integrate into their model (and to share!).

\n

The model used for NetGO2 is shown here:
\n\"\"

\n

There is one other lab whose work we must also examine - they did decently in CAFA3 and are currently atop the leaderboard with a commanding lead over all other models (Hoehndorf lab).

\n

DeepGO (2018): https://academic.oup.com/bioinformatics/article/34/4/660/4265461
\nDeepGOPlus (2020): https://academic.oup.com/bioinformatics/article/36/2/422/5539866
\nDeepGOZero (2022): https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515

\n

Here are two quotes from the discussion section of DeepGOPlus :
\n“In DeepGOPlus, we combine similarity-based search to proteins with known functions and motif-based function prediction, and this combination gives us overall the best predictive performance. However, DeepGOPlus can also be applied using only sequence motifs; in particular when annotating novel proteins for which no similar proteins with known functions exist, our motif-based model would be most suitable.”

\n

“We have already experimented with several types of neural networks such as recurrent neural networks, long-short term memory networks and autoencoders to learn sequence features. However, our attempts were unsuccessful and CNNs gave us the best results.”

\n

Finally lets look at one of their most recent pre-prints which proposes a method to integrate ESM-2 with a Transformer GCN for a protein binding prediction task.
\n\"\"

\n

(2023) https://www.biorxiv.org/content/10.1101/2023.03.14.532563v1.abstract
\nIt is not difficult to imagine integrating such an approach into the current problem of GO prediction.

\n

Other than the papers I have highlighted here, there are numerous other publications with credible and somewhat less credible claims on their accuracy. I have included here a short list of these that will make for decent reading material or for the next review article on the subject that I am sure is coming soon given the recent attention lavished on the subject. The list is incomplete, non-exhaustive, and I make no effort here to summarize the approaches that have been taken.

\n

2023
\nhttps://pubmed.ncbi.nlm.nih.gov/36919205/
\nhttps://academic.oup.com/bioinformatics/article/39/3/btad094/7043095
\nhttps://www.frontiersin.org/articles/10.3389/fgene.2022.969915/full
\nhttps://elifesciences.org/articles/80942

\n

2022
\nhttps://pubmed.ncbi.nlm.nih.gov/36421723/
\nhttps://www.nature.com/articles/s41587-021-01179-w
\nhttps://www.biorxiv.org/content/10.1101/2022.12.05.519119v1.full.pdf
\nhttps://academic.oup.com/nar/article/50/W1/W228/6576357
\nhttps://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515
\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010793
\nhttps://academic.oup.com/bioinformatics/article/38/8/2102/6502274
\nhttps://academic.oup.com/bioinformatics/article/38/19/4488/6656346
\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010075

\n

2021
\nhttps://www.nature.com/articles/s41598-020-80786-0
\nhttps://www.nature.com/articles/s41467-021-23303-9

", "rawMarkdown": "Let us assume that one can create a model that has predictive power of Fmax > 0.6 on a novel set of proteins for GO terms. We already have an example on the leaderboard of a model that has achieved such a score which we can assume does not represent this phenomenon termed 'overfitting'. I would like to suggest here a few approaches that might help a model achieve such a score and leave it up to others to come up with other ideas on how to do so.\n\nWhat are the established methods for creating automatic function prediction software? Let us turn to the published literature to learn from the best. Firstly, let us look at the papers describing the software package that won CAFA3 and its descendants by Zhu lab:\nGOLabeler (2018): https://academic.oup.com/bioinformatics/article/34/14/2465/4924212 \nDeepText2GO (2018): https://pubmed.ncbi.nlm.nih.gov/29883746/ \nNetGO (2019): https://academic.oup.com/nar/article/47/W1/W379/5491749 \nNetGO 2 (2021): https://academic.oup.com/nar/article/49/W1/W469/6285266 \nNetGO 3 (2022): https://www.biorxiv.org/content/10.1101/2022.12.05.519073v1 \nHere we see an ensemble model approach which has been improved with only minor revisions over the years, the latest iteration including the ESM large protein language model (LR-ESM). I include here two quotes that are relevant for thinking of this problem\n\nFrom Supplementary Materials of “GOLabeler: Improving Sequence-based Large-scale Protein Function Prediction by Learning to Rank” (Jan 2018)\n**“ The experimental results of GOLabeler demonstrate that, the good performance of GOLabeler is largely attributed to two component methods, BLAST-KNN and LR-Interpro. In contrast to BLAST-KNN utilizing protein homology information, LR-Interpro makes use of domain, family and motif information of protein for function prediction. This means that only sequence similarity is insufficient for the good performance of GOLabeler.”**\n\nFrom NetGO 3.0: Protein Language Model Improves Large-scale 3 Functional Annotations (Dec 2022)\n**“In Table 8, it is clear that BLAST-KNN failed to achieve a valid result because homology-based methods were not suitable for difficult protein function prediction. LR-InterPro and LR-ESM extracted features from raw amino acid sequences and obtained better results than BLAST-KNN. In the top 20 predicted GO terms, the number of true positive samples achieved by LR-ESM is significantly larger than other methods, which predicted 14 correct function labels.”**\n\nNone of the code underlying these models appear to be available, and we can only attempt to recreate the models presented in these papers, although the predictions produced by NetGO3.0 are available from their web server (https://dmiip.sjtu.edu.cn/ng3.0/). I invite any enterprising individuals to extract the full predicted values from the server to integrate into their model (and to share!). \n\nThe model used for NetGO2 is shown here:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4444485%2F2d2943192681bfc13481427708bb574b%2Fnetgo2.png?generation=1687886401026356&alt=media)\n\nThere is one other lab whose work we must also examine - they did decently in CAFA3 and are currently atop the leaderboard with a commanding lead over all other models (Hoehndorf lab).\n\nDeepGO (2018): https://academic.oup.com/bioinformatics/article/34/4/660/4265461 \nDeepGOPlus (2020): https://academic.oup.com/bioinformatics/article/36/2/422/5539866 \nDeepGOZero (2022): https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515\n\nHere are two quotes from the discussion section of DeepGOPlus : \n**“In DeepGOPlus, we combine similarity-based search to proteins with known functions and motif-based function prediction, and this combination gives us overall the best predictive performance. However, DeepGOPlus can also be applied using only sequence motifs; in particular when annotating novel proteins for which no similar proteins with known functions exist, our motif-based model would be most suitable.”**\n\n**“We have already experimented with several types of neural networks such as recurrent neural networks, long-short term memory networks and autoencoders to learn sequence features. However, our attempts were unsuccessful and CNNs gave us the best results.”**\n\nFinally lets look at one of their most recent pre-prints which proposes a method to integrate ESM-2 with a Transformer GCN for a protein binding prediction task. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4444485%2Fb4c1158c55c4ccceb474f9e4757d458a%2Flep-ad.png?generation=1687885967726309&alt=media)\n\n(2023) https://www.biorxiv.org/content/10.1101/2023.03.14.532563v1.abstract \nIt is not difficult to imagine integrating such an approach into the current problem of GO prediction.\n\nOther than the papers I have highlighted here, there are numerous other publications with credible and somewhat less credible claims on their accuracy. I have included here a short list of these that will make for decent reading material or for the next review article on the subject that I am sure is coming soon given the recent attention lavished on the subject. The list is incomplete, non-exhaustive, and I make no effort here to summarize the approaches that have been taken. \n\n2023\nhttps://pubmed.ncbi.nlm.nih.gov/36919205/ \nhttps://academic.oup.com/bioinformatics/article/39/3/btad094/7043095 \nhttps://www.frontiersin.org/articles/10.3389/fgene.2022.969915/full \nhttps://elifesciences.org/articles/80942 \n\n2022\nhttps://pubmed.ncbi.nlm.nih.gov/36421723/ \nhttps://www.nature.com/articles/s41587-021-01179-w \nhttps://www.biorxiv.org/content/10.1101/2022.12.05.519119v1.full.pdf \nhttps://academic.oup.com/nar/article/50/W1/W228/6576357 \nhttps://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515 \nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010793 \nhttps://academic.oup.com/bioinformatics/article/38/8/2102/6502274 \nhttps://academic.oup.com/bioinformatics/article/38/19/4488/6656346 \nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010075 \n\n2021\nhttps://www.nature.com/articles/s41598-020-80786-0 \nhttps://www.nature.com/articles/s41467-021-23303-9", "votes": null }, { "id": "2321766", "postDate": "06/28/2023 21:17:12", "content": "

These resources are a valuable source of inspiration. I also suggest that curating a comprehensive training set might be crucial in achieving scores beyond 0.6. My best guess is that scores below 0.6 in the LB are likely to come from models that leverage the provided training terms. Conversely, I bet the leading team crafted a bespoke training set. Anyway,… gut feeling for the moment…

", "rawMarkdown": "These resources are a valuable source of inspiration. I also suggest that curating a comprehensive training set might be crucial in achieving scores beyond 0.6. My best guess is that scores below 0.6 in the LB are likely to come from models that leverage the provided training terms. Conversely, I bet the leading team crafted a bespoke training set. Anyway,... gut feeling for the moment...", "votes": null }, { "id": "2321782", "postDate": "06/28/2023 21:39:22", "content": "

I can perhaps suggest some ideas in that direction. It is useful to know that of the 142247 proteins in training data, about 83883 are from swiss-prot, the rest presumably are from TrEMBL. Comparing these proteins to those from the very latest version of swiss-prot (2023-02), I could only get 77529 protein ID terms to overlap. Of the ones that overlapped, the latest version has 4129571 GO-Protein pairs (propagated) compared to 375590 in the original training data, so only a modest increase. It is not clear yet if these new terms affect model performance although I suspect it is minimal based on simple tests.

\n

This indicates to me that a bespoke training dataset might not be essential. Furthermore, my current model is very poor and unsatisfactory, so I anticipate more improvements from using the current training data. On the other hand, what would a bespoke training dataset look like / include?

", "rawMarkdown": "I can perhaps suggest some ideas in that direction. It is useful to know that of the 142247 proteins in training data, about 83883 are from swiss-prot, the rest presumably are from TrEMBL. Comparing these proteins to those from the very latest version of swiss-prot (2023-02), I could only get 77529 protein ID terms to overlap. Of the ones that overlapped, the latest version has 4129571 GO-Protein pairs (propagated) compared to 375590 in the original training data, so only a modest increase. It is not clear yet if these new terms affect model performance although I suspect it is minimal based on simple tests. \n\nThis indicates to me that a bespoke training dataset might not be essential. Furthermore, my current model is very poor and unsatisfactory, so I anticipate more improvements from using the current training data. On the other hand, what would a bespoke training dataset look like / include?", "votes": null }, { "id": "2322993", "postDate": "06/29/2023 15:59:40", "content": "

In addition to the suggestions mentioned here:
\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/417696#2305603
\nI'm exploring the use of protein families.

\n

Also this discussion on incorporating GO terms:
\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405085

\n

In that context, I'm investigating ways to include more examples of proteins annotated with rare GO terms (i.e., those associated with less than ten proteins for example). I'd like to add also GO terms that are absent in the current train_term file but used for scoring. One could use the quick-go annotations for these two points, but I don't have a clear path head.

", "rawMarkdown": "In addition to the suggestions mentioned here:\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/417696#2305603\nI'm exploring the use of protein families. \n\nAlso this discussion on incorporating GO terms:\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405085\n\nIn that context, I'm investigating ways to include more examples of proteins annotated with rare GO terms (i.e., those associated with less than ten proteins for example). I'd like to add also GO terms that are absent in the current train_term file but used for scoring. One could use the quick-go annotations for these two points, but I don't have a clear path head.", "votes": null }, { "id": "2323001", "postDate": "06/29/2023 16:04:56", "content": "

I believe one of the most interesting ways to increase training set data is the use of electronically annotated GO terms - of which there are 9 different categories with various capabilities of precision / recall.
\nInferred from Sequence or structural Similarity (ISS), (ECO:0000250)
\nInferred from Sequence Orthology (ISO), (ECO:0000266)
\nInferred from Sequence Alignment (ISA), (ECO:0000247)
\nInferred from Sequence Model (ISM), (ECO:0000255)
\nInferred from Genomic Context (IGC), (ECO_0000317)
\nInferred from Biological aspect of Ancestor (IBA), (ECO:0000318)
\nInferred from Biological aspect of Descendant (IBD), (ECO:0000319)
\nInferred from Key Residues (IKR), (ECO:0000320)
\nInferred from Rapid Divergence (IRD), (ECO:0000321)
\nI have not explored this avenue of enquiry but I think selecting a combination of these might be quite promising.

", "rawMarkdown": "I believe one of the most interesting ways to increase training set data is the use of electronically annotated GO terms - of which there are 9 different categories with various capabilities of precision / recall. \nInferred from Sequence or structural Similarity (ISS), (ECO:0000250)\nInferred from Sequence Orthology (ISO), (ECO:0000266)\nInferred from Sequence Alignment (ISA), (ECO:0000247)\nInferred from Sequence Model (ISM), (ECO:0000255)\nInferred from Genomic Context (IGC), (ECO_0000317)\nInferred from Biological aspect of Ancestor (IBA), (ECO:0000318)\nInferred from Biological aspect of Descendant (IBD), (ECO:0000319)\nInferred from Key Residues (IKR), (ECO:0000320)\nInferred from Rapid Divergence (IRD), (ECO:0000321)\nI have not explored this avenue of enquiry but I think selecting a combination of these might be quite promising.", "votes": null }, { "id": "2323037", "postDate": "06/29/2023 16:27:34", "content": "

I concur with your point, however, the term GO:0102628 serves as an exception where quickGO may not be effective, given it is only annotated in a single protein. In such scenarios, my approach would be to utilize BLAST, setting a stringent homology threshold, to identify additional proteins that may potentially share this annotation\"

", "rawMarkdown": "I concur with your point, however, the term GO:0102628 serves as an exception where quickGO may not be effective, given it is only annotated in a single protein. In such scenarios, my approach would be to utilize BLAST, setting a stringent homology threshold, to identify additional proteins that may potentially share this annotation\"", "votes": null }, { "id": "2325645", "postDate": "07/01/2023 13:41:01", "content": "

@zmcxjt would you mind explaining how you downloaded the latest version of swiss-prot and the corresponding GO terms associated?

\n

What I've been able to do:

\n\n

However when combining the annotations from MF, CC and BP from the gaf file I only end up with 41234 different proteins annotated with 51.5% of them matching an existing protein in the official trainset. What am I missing ?

", "rawMarkdown": "zmcxjt would you mind explaining how you downloaded the latest version of swiss-prot and the corresponding GO terms associated?\n\nWhat I've been able to do:\n- download `uniprot_sprot.fasta` from here (ending up with 569793 sequences) : https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/\n- download the GO annotations `filtered_goa_uniprot_all_noiea.gaf.gz` here : http://current.geneontology.org/annotations/index.html\n\nHowever when combining the annotations from MF, CC and BP from the gaf file I only end up with 41234 different proteins annotated with 51.5% of them matching an existing protein in the official trainset. What am I missing ?", "votes": null }, { "id": "2325734", "postDate": "07/01/2023 15:05:57", "content": "

I used the UniProt XML to directly obtain GO terms

", "rawMarkdown": "I used the UniProt XML to directly obtain GO terms", "votes": null } ], "comments": [ { "id": 2321766, "author_name": "mtinti", "author_url": "", "post_date": "06/28/2023 21:17:12", "content": "

These resources are a valuable source of inspiration. I also suggest that curating a comprehensive training set might be crucial in achieving scores beyond 0.6. My best guess is that scores below 0.6 in the LB are likely to come from models that leverage the provided training terms. Conversely, I bet the leading team crafted a bespoke training set. Anyway,… gut feeling for the moment…

", "votes": null, "replies": [ { "id": 2321782, "author_name": "zmcxjt", "author_url": "", "post_date": "06/28/2023 21:39:22", "content": "

I can perhaps suggest some ideas in that direction. It is useful to know that of the 142247 proteins in training data, about 83883 are from swiss-prot, the rest presumably are from TrEMBL. Comparing these proteins to those from the very latest version of swiss-prot (2023-02), I could only get 77529 protein ID terms to overlap. Of the ones that overlapped, the latest version has 4129571 GO-Protein pairs (propagated) compared to 375590 in the original training data, so only a modest increase. It is not clear yet if these new terms affect model performance although I suspect it is minimal based on simple tests.

\n

This indicates to me that a bespoke training dataset might not be essential. Furthermore, my current model is very poor and unsatisfactory, so I anticipate more improvements from using the current training data. On the other hand, what would a bespoke training dataset look like / include?

", "votes": null, "replies": [ { "id": 2322993, "author_name": "mtinti", "author_url": "", "post_date": "06/29/2023 15:59:40", "content": "

In addition to the suggestions mentioned here:
\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/417696#2305603
\nI'm exploring the use of protein families.

\n

Also this discussion on incorporating GO terms:
\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405085

\n

In that context, I'm investigating ways to include more examples of proteins annotated with rare GO terms (i.e., those associated with less than ten proteins for example). I'd like to add also GO terms that are absent in the current train_term file but used for scoring. One could use the quick-go annotations for these two points, but I don't have a clear path head.

", "votes": null, "replies": [ { "id": 2323001, "author_name": "zmcxjt", "author_url": "", "post_date": "06/29/2023 16:04:56", "content": "

I believe one of the most interesting ways to increase training set data is the use of electronically annotated GO terms - of which there are 9 different categories with various capabilities of precision / recall.
\nInferred from Sequence or structural Similarity (ISS), (ECO:0000250)
\nInferred from Sequence Orthology (ISO), (ECO:0000266)
\nInferred from Sequence Alignment (ISA), (ECO:0000247)
\nInferred from Sequence Model (ISM), (ECO:0000255)
\nInferred from Genomic Context (IGC), (ECO_0000317)
\nInferred from Biological aspect of Ancestor (IBA), (ECO:0000318)
\nInferred from Biological aspect of Descendant (IBD), (ECO:0000319)
\nInferred from Key Residues (IKR), (ECO:0000320)
\nInferred from Rapid Divergence (IRD), (ECO:0000321)
\nI have not explored this avenue of enquiry but I think selecting a combination of these might be quite promising.

", "votes": null, "replies": [ { "id": 2323037, "author_name": "mtinti", "author_url": "", "post_date": "06/29/2023 16:27:34", "content": "

I concur with your point, however, the term GO:0102628 serves as an exception where quickGO may not be effective, given it is only annotated in a single protein. In such scenarios, my approach would be to utilize BLAST, setting a stringent homology threshold, to identify additional proteins that may potentially share this annotation\"

", "votes": null, "replies": [ { "id": 2325645, "author_name": "optimo", "author_url": "", "post_date": "07/01/2023 13:41:01", "content": "

@zmcxjt would you mind explaining how you downloaded the latest version of swiss-prot and the corresponding GO terms associated?

\n

What I've been able to do:

\n\n

However when combining the annotations from MF, CC and BP from the gaf file I only end up with 41234 different proteins annotated with 51.5% of them matching an existing protein in the official trainset. What am I missing ?

", "votes": null, "replies": [ { "id": 2325734, "author_name": "zmcxjt", "author_url": "", "post_date": "07/01/2023 15:05:57", "content": "

I used the UniProt XML to directly obtain GO terms

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2320292": "Let us assume that one can create a model that has predictive power of Fmax > 0.6 on a novel set of proteins for GO terms. We already have an example on the leaderboard of a model that has achieved such a score which we can assume does not represent this phenomenon termed 'overfitting'. I would like to suggest here a few approaches that might help a model achieve such a score and leave it up to others to come up with other ideas on how to do so.\n\nWhat are the established methods for creating automatic function prediction software? Let us turn to the published literature to learn from the best. Firstly, let us look at the papers describing the software package that won CAFA3 and its descendants by Zhu lab:\nGOLabeler (2018): https://academic.oup.com/bioinformatics/article/34/14/2465/4924212 \nDeepText2GO (2018): https://pubmed.ncbi.nlm.nih.gov/29883746/ \nNetGO (2019): https://academic.oup.com/nar/article/47/W1/W379/5491749 \nNetGO 2 (2021): https://academic.oup.com/nar/article/49/W1/W469/6285266 \nNetGO 3 (2022): https://www.biorxiv.org/content/10.1101/2022.12.05.519073v1 \nHere we see an ensemble model approach which has been improved with only minor revisions over the years, the latest iteration including the ESM large protein language model (LR-ESM). I include here two quotes that are relevant for thinking of this problem\n\nFrom Supplementary Materials of “GOLabeler: Improving Sequence-based Large-scale Protein Function Prediction by Learning to Rank” (Jan 2018)\n**“ The experimental results of GOLabeler demonstrate that, the good performance of GOLabeler is largely attributed to two component methods, BLAST-KNN and LR-Interpro. In contrast to BLAST-KNN utilizing protein homology information, LR-Interpro makes use of domain, family and motif information of protein for function prediction. This means that only sequence similarity is insufficient for the good performance of GOLabeler.”**\n\nFrom NetGO 3.0: Protein Language Model Improves Large-scale 3 Functional Annotations (Dec 2022)\n**“In Table 8, it is clear that BLAST-KNN failed to achieve a valid result because homology-based methods were not suitable for difficult protein function prediction. LR-InterPro and LR-ESM extracted features from raw amino acid sequences and obtained better results than BLAST-KNN. In the top 20 predicted GO terms, the number of true positive samples achieved by LR-ESM is significantly larger than other methods, which predicted 14 correct function labels.”**\n\nNone of the code underlying these models appear to be available, and we can only attempt to recreate the models presented in these papers, although the predictions produced by NetGO3.0 are available from their web server (https://dmiip.sjtu.edu.cn/ng3.0/). I invite any enterprising individuals to extract the full predicted values from the server to integrate into their model (and to share!). \n\nThe model used for NetGO2 is shown here:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4444485%2F2d2943192681bfc13481427708bb574b%2Fnetgo2.png?generation=1687886401026356&alt=media)\n\nThere is one other lab whose work we must also examine - they did decently in CAFA3 and are currently atop the leaderboard with a commanding lead over all other models (Hoehndorf lab).\n\nDeepGO (2018): https://academic.oup.com/bioinformatics/article/34/4/660/4265461 \nDeepGOPlus (2020): https://academic.oup.com/bioinformatics/article/36/2/422/5539866 \nDeepGOZero (2022): https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515\n\nHere are two quotes from the discussion section of DeepGOPlus : \n**“In DeepGOPlus, we combine similarity-based search to proteins with known functions and motif-based function prediction, and this combination gives us overall the best predictive performance. However, DeepGOPlus can also be applied using only sequence motifs; in particular when annotating novel proteins for which no similar proteins with known functions exist, our motif-based model would be most suitable.”**\n\n**“We have already experimented with several types of neural networks such as recurrent neural networks, long-short term memory networks and autoencoders to learn sequence features. However, our attempts were unsuccessful and CNNs gave us the best results.”**\n\nFinally lets look at one of their most recent pre-prints which proposes a method to integrate ESM-2 with a Transformer GCN for a protein binding prediction task. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4444485%2Fb4c1158c55c4ccceb474f9e4757d458a%2Flep-ad.png?generation=1687885967726309&alt=media)\n\n(2023) https://www.biorxiv.org/content/10.1101/2023.03.14.532563v1.abstract \nIt is not difficult to imagine integrating such an approach into the current problem of GO prediction.\n\nOther than the papers I have highlighted here, there are numerous other publications with credible and somewhat less credible claims on their accuracy. I have included here a short list of these that will make for decent reading material or for the next review article on the subject that I am sure is coming soon given the recent attention lavished on the subject. The list is incomplete, non-exhaustive, and I make no effort here to summarize the approaches that have been taken. \n\n2023\nhttps://pubmed.ncbi.nlm.nih.gov/36919205/ \nhttps://academic.oup.com/bioinformatics/article/39/3/btad094/7043095 \nhttps://www.frontiersin.org/articles/10.3389/fgene.2022.969915/full \nhttps://elifesciences.org/articles/80942 \n\n2022\nhttps://pubmed.ncbi.nlm.nih.gov/36421723/ \nhttps://www.nature.com/articles/s41587-021-01179-w \nhttps://www.biorxiv.org/content/10.1101/2022.12.05.519119v1.full.pdf \nhttps://academic.oup.com/nar/article/50/W1/W228/6576357 \nhttps://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515 \nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010793 \nhttps://academic.oup.com/bioinformatics/article/38/8/2102/6502274 \nhttps://academic.oup.com/bioinformatics/article/38/19/4488/6656346 \nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010075 \n\n2021\nhttps://www.nature.com/articles/s41598-020-80786-0 \nhttps://www.nature.com/articles/s41467-021-23303-9", "2321766": "These resources are a valuable source of inspiration. I also suggest that curating a comprehensive training set might be crucial in achieving scores beyond 0.6. My best guess is that scores below 0.6 in the LB are likely to come from models that leverage the provided training terms. Conversely, I bet the leading team crafted a bespoke training set. Anyway,... gut feeling for the moment...", "2321782": "I can perhaps suggest some ideas in that direction. It is useful to know that of the 142247 proteins in training data, about 83883 are from swiss-prot, the rest presumably are from TrEMBL. Comparing these proteins to those from the very latest version of swiss-prot (2023-02), I could only get 77529 protein ID terms to overlap. Of the ones that overlapped, the latest version has 4129571 GO-Protein pairs (propagated) compared to 375590 in the original training data, so only a modest increase. It is not clear yet if these new terms affect model performance although I suspect it is minimal based on simple tests. \n\nThis indicates to me that a bespoke training dataset might not be essential. Furthermore, my current model is very poor and unsatisfactory, so I anticipate more improvements from using the current training data. On the other hand, what would a bespoke training dataset look like / include?", "2322993": "In addition to the suggestions mentioned here:\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/417696#2305603\nI'm exploring the use of protein families. \n\nAlso this discussion on incorporating GO terms:\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405085\n\nIn that context, I'm investigating ways to include more examples of proteins annotated with rare GO terms (i.e., those associated with less than ten proteins for example). I'd like to add also GO terms that are absent in the current train_term file but used for scoring. One could use the quick-go annotations for these two points, but I don't have a clear path head.", "2323001": "I believe one of the most interesting ways to increase training set data is the use of electronically annotated GO terms - of which there are 9 different categories with various capabilities of precision / recall. \nInferred from Sequence or structural Similarity (ISS), (ECO:0000250)\nInferred from Sequence Orthology (ISO), (ECO:0000266)\nInferred from Sequence Alignment (ISA), (ECO:0000247)\nInferred from Sequence Model (ISM), (ECO:0000255)\nInferred from Genomic Context (IGC), (ECO_0000317)\nInferred from Biological aspect of Ancestor (IBA), (ECO:0000318)\nInferred from Biological aspect of Descendant (IBD), (ECO:0000319)\nInferred from Key Residues (IKR), (ECO:0000320)\nInferred from Rapid Divergence (IRD), (ECO:0000321)\nI have not explored this avenue of enquiry but I think selecting a combination of these might be quite promising.", "2323037": "I concur with your point, however, the term GO:0102628 serves as an exception where quickGO may not be effective, given it is only annotated in a single protein. In such scenarios, my approach would be to utilize BLAST, setting a stringent homology threshold, to identify additional proteins that may potentially share this annotation\"", "2325645": "zmcxjt would you mind explaining how you downloaded the latest version of swiss-prot and the corresponding GO terms associated?\n\nWhat I've been able to do:\n- download `uniprot_sprot.fasta` from here (ending up with 569793 sequences) : https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/\n- download the GO annotations `filtered_goa_uniprot_all_noiea.gaf.gz` here : http://current.geneontology.org/annotations/index.html\n\nHowever when combining the annotations from MF, CC and BP from the gaf file I only end up with 41234 different proteins annotated with 51.5% of them matching an existing protein in the official trainset. What am I missing ?", "2325734": "I used the UniProt XML to directly obtain GO terms" }, "source": "meta" }