{ "id": 466917, "title": "1st Place Solution for the CAFA5", "url": "/competitions/cafa-5-protein-function-prediction/discussion/466917", "author_name": "王少钧", "post_date": "2024-01-10T11:55:37.361000", "votes": 34, "comment_count": 7, "views": 0, "content": "

A. MODEL SUMMARY

\n

A.1 Background on you/your team

\n

Competition Name: CAFA5 Protein Function Prediction

\n

Team Name: GOCurator

\n

Private Leaderboard Score: 0.61623

\n

Private Leaderboard Place: 1

\n

Name: Shaojun Wang

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: 20110850027@fudan.edu.cn

\n

Name: Weiqi Zhai

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: 20110850010@fudan.edu.cn

\n

Name: Wei Liu

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: 21110850007@m.fudan.edu.cn

\n

Name: Tianyang Huang

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: tyhuang22@m.fudan.edu.cn

\n

Name: Huiying Yan

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: yanhuiyingwangyi@163.com

\n

A2. Background on you/your team

\n

Our team consists of five PhD students working on bioinformatics, especially automated protein function prediction, and biomedical text mining from ZhuLab, which is led by Prof. Shanfeng Zhu at Fudan University. Our lab has participated in the CAFA3 and CAFA4 and achieved the top performance. Our team developed GOCurator by considering all kinds of information including protein sequence, network, structure, and textual information. During the competition, S.W. was responsible for collecting data, building the framework, and training the model. W.L developed a component method based on FoldSeek. T.H and H.Y were mainly engaged in biomedical text mining for protein annotation. W.Z submitted the prediction result, and performed the result analysis.

\n

A3. Summary

\n

Our pipeline, GOCurator, was developed based on NetGO 3.0. In addition, we also introduced new component methods to extract functional information from protein 3D structure, protein textual description, and scientific literature. Thanks to the learning to rank model, we eventually integrated multiple component methods effectively. Figure 1 showed the score of component methods achieved in public leaderboard.
\n\"The

\n\n

When doing ensemble learning, the input to the model contains not only the predicted scores from each method, but also a 20-dim one-hot vector to characterize the protein species information.

\n

A4. Features Selections / Engineering

\n

According to the public leaderboard scores, we believed that protein language models and textual information are useful for improving accuracy. For protein language models, we have also tried the ESM2 and ProtT5 to obtain protein embeddings. Similar to LR-ESM (based on ESM-1b), we trained logistic regression classifiers for ESM2 and ProtT5 embedding and the concatenation of all three embeddings. We found little difference in the scores achieved by these methods, but the best performance was achieved when LR-ESM was integrated as a component method into the GOCurator. Therefore, we finally used ESM-1b to generate protein embeddings. For textual information, how to generate text representations is important. Following to DeepText2GO, we used document to vector (D2V) and TF-IDF to represent scientific literature. In GORetrieval and GOXML, PubMedBERT was used to generate text embedding.

\n

A5. Training Method(s)

\n

We collected functional annotation information from three database SwissProt, GOA, and GO, according to the CAFA5 evidence code. Unlike the temporal split in NetGO, we selected proteins for validation based on the species distribution of proteins in the CAFA5 test set. Specifically, we calculate the proportion of proteins from each species in the CAFA5 test set, and sampled proteins for the corresponding species from the dataset to form a validation dataset of 1000 proteins. We have improved the performance of our methods as much as possible in this way.

\n

A6. Interesting findings

\n

We found that the Net-KNN based on protein-protein interaction performs poorly in predicting proteins from those species with less functional annotation information. When we integrated all the component methods together, we even got better results by not fusing Net-KNN into it. However, according to our previous studies (NetGO series), Net-KNN is a powerful method for BPO and CCO. Therefore, we tried to filter the prediction from Net-KNN by the species. Specifically, we considered the predictions for a target protein to be valid if the number of the annotated proteins in the target protein’s species is larger. As shown in the figure, the number of valid species has an impact on the prediction performance. Finally, we treated Net-KNN predictions for the top 15 species as effective results.
\n\"\"

\n

A7. Simple Features and Methods

\n

According to the performance of different source information, we believe that incorporating a component method based on textual or structural information into NetGO 3.0 can achieve 90% of the final performance.

\n

A8. Model Execution Time

\n

Firstly, we used InterProScan and ESM-1b to get protein domain/family/motif information and protein embeddings, which took about 1-2 days, where most of the time was spent on InterProScan. Secondly, we trained each component method separately on several servers, spending roughly 10-15 hours per method. Then, the learning to rank model used for ensemble learning could be trained within 50 mins. For inference, we first used component methods to generate initial prediction, which took about 2 hours for CAFA5 test proteins. Finally, it took about 1.5 hours for ensemble learning on all test proteins.

\n

A9. References

\n
    \n
  1. You R, Zhang Z, Xiong Y, et al. GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank. Bioinformatics, Volume 34, Issue 14, 2018, 2465–2473.
    2. \n
  2. You R, Huang X, Zhu S. DeepText2GO: Improving large-scale protein function prediction with deep semantic text representation. Methods, 2018, 145: 82-90.
    3. \n
  3. You R, Yao S, Xiong Y, et al. NetGO: improving large-scale protein function prediction with massive network information. Nucleic Acids Research, Volume 47, Issue W1, 2019, W379–W387.
    4. \n
  4. Yao S, You R, Wang S, et al. NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information. Nucleic Acids Research, Volume 49, Issue W1, 2021, W469-W475
    5. \n
  5. Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021
    6. \n
  6. Rives A, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 118(15), 2021
    7. \n
  7. You R, Yao S, Mamitsuka H, et al. DeepGraphGO: graph neural network for large-scale, multispecies protein function prediction. Bioinformatics (ISMB 2021), Volume 37, Supplement_1, 2021, i262–i271.
    8. \n
  8. Wang S, You R, Liu Y, et al. NetGO 3.0: Protein Language Model Improves Large-scale Functional Annotations. Genomics, Proteomics & Bioinformatics, Volume 21, Issue 2, 2023, 349-358.
    9. \n
", "messages": [ { "id": 2595397, "postDate": "2024-01-10T11:55:37.363Z", "content": "

A. MODEL SUMMARY

\n

A.1 Background on you/your team

\n

Competition Name: CAFA5 Protein Function Prediction

\n

Team Name: GOCurator

\n

Private Leaderboard Score: 0.61623

\n

Private Leaderboard Place: 1

\n

Name: Shaojun Wang

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: 20110850027@fudan.edu.cn

\n

Name: Weiqi Zhai

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: 20110850010@fudan.edu.cn

\n

Name: Wei Liu

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: 21110850007@m.fudan.edu.cn

\n

Name: Tianyang Huang

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: tyhuang22@m.fudan.edu.cn

\n

Name: Huiying Yan

\n

Location: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China

\n

Email: yanhuiyingwangyi@163.com

\n

A2. Background on you/your team

\n

Our team consists of five PhD students working on bioinformatics, especially automated protein function prediction, and biomedical text mining from ZhuLab, which is led by Prof. Shanfeng Zhu at Fudan University. Our lab has participated in the CAFA3 and CAFA4 and achieved the top performance. Our team developed GOCurator by considering all kinds of information including protein sequence, network, structure, and textual information. During the competition, S.W. was responsible for collecting data, building the framework, and training the model. W.L developed a component method based on FoldSeek. T.H and H.Y were mainly engaged in biomedical text mining for protein annotation. W.Z submitted the prediction result, and performed the result analysis.

\n

A3. Summary

\n

Our pipeline, GOCurator, was developed based on NetGO 3.0. In addition, we also introduced new component methods to extract functional information from protein 3D structure, protein textual description, and scientific literature. Thanks to the learning to rank model, we eventually integrated multiple component methods effectively. Figure 1 showed the score of component methods achieved in public leaderboard.
\n\"The

\n\n

When doing ensemble learning, the input to the model contains not only the predicted scores from each method, but also a 20-dim one-hot vector to characterize the protein species information.

\n

A4. Features Selections / Engineering

\n

According to the public leaderboard scores, we believed that protein language models and textual information are useful for improving accuracy. For protein language models, we have also tried the ESM2 and ProtT5 to obtain protein embeddings. Similar to LR-ESM (based on ESM-1b), we trained logistic regression classifiers for ESM2 and ProtT5 embedding and the concatenation of all three embeddings. We found little difference in the scores achieved by these methods, but the best performance was achieved when LR-ESM was integrated as a component method into the GOCurator. Therefore, we finally used ESM-1b to generate protein embeddings. For textual information, how to generate text representations is important. Following to DeepText2GO, we used document to vector (D2V) and TF-IDF to represent scientific literature. In GORetrieval and GOXML, PubMedBERT was used to generate text embedding.

\n

A5. Training Method(s)

\n

We collected functional annotation information from three database SwissProt, GOA, and GO, according to the CAFA5 evidence code. Unlike the temporal split in NetGO, we selected proteins for validation based on the species distribution of proteins in the CAFA5 test set. Specifically, we calculate the proportion of proteins from each species in the CAFA5 test set, and sampled proteins for the corresponding species from the dataset to form a validation dataset of 1000 proteins. We have improved the performance of our methods as much as possible in this way.

\n

A6. Interesting findings

\n

We found that the Net-KNN based on protein-protein interaction performs poorly in predicting proteins from those species with less functional annotation information. When we integrated all the component methods together, we even got better results by not fusing Net-KNN into it. However, according to our previous studies (NetGO series), Net-KNN is a powerful method for BPO and CCO. Therefore, we tried to filter the prediction from Net-KNN by the species. Specifically, we considered the predictions for a target protein to be valid if the number of the annotated proteins in the target protein’s species is larger. As shown in the figure, the number of valid species has an impact on the prediction performance. Finally, we treated Net-KNN predictions for the top 15 species as effective results.
\n\"\"

\n

A7. Simple Features and Methods

\n

According to the performance of different source information, we believe that incorporating a component method based on textual or structural information into NetGO 3.0 can achieve 90% of the final performance.

\n

A8. Model Execution Time

\n

Firstly, we used InterProScan and ESM-1b to get protein domain/family/motif information and protein embeddings, which took about 1-2 days, where most of the time was spent on InterProScan. Secondly, we trained each component method separately on several servers, spending roughly 10-15 hours per method. Then, the learning to rank model used for ensemble learning could be trained within 50 mins. For inference, we first used component methods to generate initial prediction, which took about 2 hours for CAFA5 test proteins. Finally, it took about 1.5 hours for ensemble learning on all test proteins.

\n

A9. References

\n
    \n
  1. You R, Zhang Z, Xiong Y, et al. GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank. Bioinformatics, Volume 34, Issue 14, 2018, 2465–2473.
    2. \n
  2. You R, Huang X, Zhu S. DeepText2GO: Improving large-scale protein function prediction with deep semantic text representation. Methods, 2018, 145: 82-90.
    3. \n
  3. You R, Yao S, Xiong Y, et al. NetGO: improving large-scale protein function prediction with massive network information. Nucleic Acids Research, Volume 47, Issue W1, 2019, W379–W387.
    4. \n
  4. Yao S, You R, Wang S, et al. NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information. Nucleic Acids Research, Volume 49, Issue W1, 2021, W469-W475
    5. \n
  5. Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021
    6. \n
  6. Rives A, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 118(15), 2021
    7. \n
  7. You R, Yao S, Mamitsuka H, et al. DeepGraphGO: graph neural network for large-scale, multispecies protein function prediction. Bioinformatics (ISMB 2021), Volume 37, Supplement_1, 2021, i262–i271.
    8. \n
  8. Wang S, You R, Liu Y, et al. NetGO 3.0: Protein Language Model Improves Large-scale Functional Annotations. Genomics, Proteomics & Bioinformatics, Volume 21, Issue 2, 2023, 349-358.
    9. \n
", "rawMarkdown": "# A.\tMODEL SUMMARY\n## A.1 Background on you/your team\nCompetition Name: CAFA5 Protein Function Prediction\n\nTeam Name: GOCurator\n\nPrivate Leaderboard Score: 0.61623\n\nPrivate Leaderboard Place: 1\n\nName: Shaojun Wang\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: 20110850027@fudan.edu.cn\n\nName: Weiqi Zhai\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: 20110850010@fudan.edu.cn\n\nName: Wei Liu\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: 21110850007@m.fudan.edu.cn\n\nName: Tianyang Huang\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: tyhuang22@m.fudan.edu.cn\n\nName: Huiying Yan\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: yanhuiyingwangyi@163.com\n\n## A2. Background on you/your team\nOur team consists of five PhD students working on bioinformatics, especially automated protein function prediction, and biomedical text mining from ZhuLab, which is led by Prof. Shanfeng Zhu at Fudan University. Our lab has participated in the CAFA3 and CAFA4 and achieved the top performance. Our team developed GOCurator by considering all kinds of information including protein sequence, network, structure, and textual information. During the competition, S.W. was responsible for collecting data, building the framework, and training the model. W.L developed a component method based on FoldSeek. T.H and H.Y were mainly engaged in biomedical text mining for protein annotation. W.Z submitted the prediction result, and performed the result analysis.\n\n## A3. Summary\nOur pipeline, GOCurator, was developed based on NetGO 3.0. In addition, we also introduced new component methods to extract functional information from protein 3D structure, protein textual description, and scientific literature. Thanks to the learning to rank model, we eventually integrated multiple component methods effectively. Figure 1 showed the score of component methods achieved in public leaderboard.\n![The performance of component methods in public leaderboard](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16029558%2F27233f10b1cbcfd1e5aec8ed8a5b85c5%2Fsolution1.png?generation=1704887698109856&alt=media )\n- LR-MEM, it combined protein description, scientific literature, and protein sequence together based on the annotation from GO and GOA. Specifically, it concatenated representations from different information source as protein representations. Then we trained logistic regression classifiers based on it. We integrated this method into NetGO 3.0 and ranked in the first place for the first time on public leaderboard. \n- FoldSeek-KNN, it made prediction according to the structure information. By comparing the structural similarities between the test proteins and the proteins in the training set, we calculated the prediction score as weighted voting by similar annotated proteins.\n- GOXML, an extreme multi-label classifier based on literature information. Similar to DeepText2GO, it used AttentionXML to achieve fast high-precision large-scale multi-label classification without training logistic regression classifiers for each GO terms.\n- GORetrieval, a novel two-stage deep learning-based framework based on the protein description and protein literature. Given a target protein, it first retrieved candidate Gene Ontology (GO) terms from annotated proteins with similar descriptions. In the second stage, the retrieved GO terms were reranked based on semantic matching between the GO descriptions and the literature information of the target protein. Moreover, the second stage can also be used to enhance the prediction from other method via reranking.\n\nWhen doing ensemble learning, the input to the model contains not only the predicted scores from each method, but also a 20-dim one-hot vector to characterize the protein species information.\n \n\n## A4. Features Selections / Engineering\nAccording to the public leaderboard scores, we believed that protein language models and textual information are useful for improving accuracy. For protein language models, we have also tried the ESM2 and ProtT5 to obtain protein embeddings. Similar to LR-ESM (based on ESM-1b), we trained logistic regression classifiers for ESM2 and ProtT5 embedding and the concatenation of all three embeddings. We found little difference in the scores achieved by these methods, but the best performance was achieved when LR-ESM was integrated as a component method into the GOCurator. Therefore, we finally used ESM-1b to generate protein embeddings. For textual information, how to generate text representations is important. Following to DeepText2GO, we used document to vector (D2V) and TF-IDF to represent scientific literature. In GORetrieval and GOXML, PubMedBERT was used to generate text embedding.\n\n## A5. Training Method(s)\nWe collected functional annotation information from three database SwissProt, GOA, and GO, according to the CAFA5 evidence code. Unlike the temporal split in NetGO, we selected proteins for validation based on the species distribution of proteins in the CAFA5 test set. Specifically, we calculate the proportion of proteins from each species in the CAFA5 test set, and sampled proteins for the corresponding species from the dataset to form a validation dataset of 1000 proteins. We have improved the performance of our methods as much as possible in this way.\n\n## A6. Interesting findings\nWe found that the Net-KNN based on protein-protein interaction performs poorly in predicting proteins from those species with less functional annotation information. When we integrated all the component methods together, we even got better results by not fusing Net-KNN into it. However, according to our previous studies (NetGO series), Net-KNN is a powerful method for BPO and CCO. Therefore, we tried to filter the prediction from Net-KNN by the species. Specifically, we considered the predictions for a target protein to be valid if the number of the annotated proteins in the target protein’s species is larger. As shown in the figure, the number of valid species has an impact on the prediction performance. Finally, we treated Net-KNN predictions for the top 15 species as effective results.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16029558%2Fc669b39b4fb0047b017c6b75a9440223%2Fsolution2.png?generation=1704888407902780&alt=media)\n\n## A7. Simple Features and Methods\nAccording to the performance of different source information, we believe that incorporating a component method based on textual or structural information into NetGO 3.0 can achieve 90% of the final performance.\n\n## A8. Model Execution Time\nFirstly, we used InterProScan and ESM-1b to get protein domain/family/motif information and protein embeddings, which took about 1-2 days, where most of the time was spent on InterProScan. Secondly, we trained each component method separately on several servers, spending roughly 10-15 hours per method. Then, the learning to rank model used for ensemble learning could be trained within 50 mins. For inference, we first used component methods to generate initial prediction, which took about 2 hours for CAFA5 test proteins. Finally, it took about 1.5 hours for ensemble learning on all test proteins.\n\n## A9. References\n\n1. You R, Zhang Z, Xiong Y, et al. GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank. Bioinformatics, Volume 34, Issue 14, 2018, 2465–2473.\n2. You R, Huang X, Zhu S. DeepText2GO: Improving large-scale protein function prediction with deep semantic text representation. Methods, 2018, 145: 82-90.\n3. You R, Yao S, Xiong Y, et al. NetGO: improving large-scale protein function prediction with massive network information. Nucleic Acids Research, Volume 47, Issue W1, 2019, W379–W387.\n4. Yao S, You R, Wang S, et al. NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information. Nucleic Acids Research, Volume 49, Issue W1, 2021, W469-W475\n5. Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021\n6. Rives A, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 118(15), 2021\n7. You R, Yao S, Mamitsuka H, et al. DeepGraphGO: graph neural network for large-scale, multispecies protein function prediction. Bioinformatics (ISMB 2021), Volume 37, Supplement_1, 2021, i262–i271.\n8. Wang S, You R, Liu Y, et al. NetGO 3.0: Protein Language Model Improves Large-scale Functional Annotations. Genomics, Proteomics & Bioinformatics, Volume 21, Issue 2, 2023, 349-358.\n", "votes": 34 }, { "id": 2638950, "postDate": "2024-02-06T15:23:17.467Z", "content": "

Is it possible to get your source code of the entire solution?

", "rawMarkdown": "Is it possible to get your source code of the entire solution?", "replies": [ { "id": 2639149, "postDate": "2024-02-06T17:34:30.887Z", "content": "

how can i download the dataset?

", "rawMarkdown": "how can i download the dataset?" } ] }, { "id": 2600363, "postDate": "2024-01-13T15:11:06.523Z", "content": "

Neat! (I'm presenting about Cafa5/your approach in our lab).

\n

Some questions:

\n
    \n
  1. How did you get predictions for the first level of the ensemble, simple CV?.
    2. \n
  2. Blast-KNN = Simple PSI-blast?
    3. \n
  3. LR-Interpro = ?
    4. \n
  4. Why extract the features from interProScan and not the uniProt records?
    5. \n
", "rawMarkdown": "Neat! (I'm presenting about Cafa5/your approach in our lab). \n\nSome questions:\n1. How did you get predictions for the first level of the ensemble, simple CV?.\n2. Blast-KNN = Simple PSI-blast? \n3. LR-Interpro = ? \n4. Why extract the features from interProScan and not the uniProt records?", "replies": [ { "id": 2601162, "postDate": "2024-01-14T07:29:14.857Z", "content": "
    \n
  1. We splited the dataset into training and validation sets, and used the training set to train component methods. Then we used the trained component methods to predict function for validation proteins. Finally, we trained the ensemble model based on the prediction on validation sets. For more details, you can find in \"GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank\".
    2. \n
  2. Yep, we used psi-blast to find similar proteins for target proteins.
    3. \n
  3. LR-Interpro is also a component method derived from GOLabeler. It used InterProScan to extract protein families, domains, and motifs. Then we built a binary feature vector for each proteins based on the InterProScan results, where the '1' in j-th element indicates that the j-th feature belongs to target protein. Finally, we used these vectors to train logistic regression classifiers and make prediction.
    4. \n
  4. InterProScan is updated irregularly and collects more functional information. So, we believe that using InterProScan directly is more helpful.
    5. \n
", "rawMarkdown": "1. We splited the dataset into training and validation sets, and used the training set to train component methods. Then we used the trained component methods to predict function for validation proteins. Finally, we trained the ensemble model based on the prediction on validation sets. For more details, you can find in \"GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank\".\n2. Yep, we used psi-blast to find similar proteins for target proteins.\n3. LR-Interpro is also a component method derived from GOLabeler. It used InterProScan to extract protein families, domains, and motifs. Then we built a binary feature vector for each proteins based on the InterProScan results, where the '1' in j-th element indicates that the j-th feature belongs to target protein. Finally, we used these vectors to train logistic regression classifiers and make prediction.\n4. InterProScan is updated irregularly and collects more functional information. So, we believe that using InterProScan directly is more helpful.", "votes": 2 } ] }, { "id": 2632074, "postDate": "2024-02-02T06:01:07.263Z", "content": "

Thanks for sharing your work. I am also working on a similar project where I have created three embeddings from three language models (LLMs) and structural features. My doubt is about how to create my output vector. How can I predict the GO terms for each protein? Can you elaborate on that part?

", "rawMarkdown": "Thanks for sharing your work. I am also working on a similar project where I have created three embeddings from three language models (LLMs) and structural features. My doubt is about how to create my output vector. How can I predict the GO terms for each protein? Can you elaborate on that part?", "isDeleted": true, "replies": [ { "id": 2632303, "postDate": "2024-02-02T08:56:56.350Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2632304, "postDate": "2024-02-02T08:57:51.817Z", "content": "
    \n
  1. For the first question, are you referring to how to obtain the representation of proteins from LLMs? Protein LLMs are multi-layered and can generate representations for each amino acid. Typically, we perform mean pooling on the output of the last layer for each amino acid, and generate the feature representation of the protein.
    2. \n
  2. To predict the GO terms for proteins, we collected large-scale functional annotation information, where each GO term is associated with numerous proteins. We treated these proteins as positive instances and considered the remaining proteins as negative instances. Then we trained a classifier for each GO term, aiming to predict the association scores between proteins and GO term.
    3. \n
", "rawMarkdown": "1. For the first question, are you referring to how to obtain the representation of proteins from LLMs? Protein LLMs are multi-layered and can generate representations for each amino acid. Typically, we perform mean pooling on the output of the last layer for each amino acid, and generate the feature representation of the protein.\n2. To predict the GO terms for proteins, we collected large-scale functional annotation information, where each GO term is associated with numerous proteins. We treated these proteins as positive instances and considered the remaining proteins as negative instances. Then we trained a classifier for each GO term, aiming to predict the association scores between proteins and GO term." } ] } ], "comments": [ { "id": 2638950, "author_name": "hermit.mb", "author_url": "", "post_date": "2024-02-06T15:23:17.467000", "content": "

Is it possible to get your source code of the entire solution?

", "votes": 0, "replies": [ { "id": 2639149, "author_name": "Moori Varun", "author_url": "", "post_date": "2024-02-06T17:34:30.887000", "content": "

how can i download the dataset?

", "votes": 0, "replies": [] } ] }, { "id": 2600363, "author_name": "Dan Ofer", "author_url": "", "post_date": "2024-01-13T15:11:06.523000", "content": "

Neat! (I'm presenting about Cafa5/your approach in our lab).

\n

Some questions:

\n
    \n
  1. How did you get predictions for the first level of the ensemble, simple CV?.
    2. \n
  2. Blast-KNN = Simple PSI-blast?
    3. \n
  3. LR-Interpro = ?
    4. \n
  4. Why extract the features from interProScan and not the uniProt records?
    5. \n
", "votes": 0, "replies": [ { "id": 2601162, "author_name": "王少钧", "author_url": "", "post_date": "2024-01-14T07:29:14.857000", "content": "
    \n
  1. We splited the dataset into training and validation sets, and used the training set to train component methods. Then we used the trained component methods to predict function for validation proteins. Finally, we trained the ensemble model based on the prediction on validation sets. For more details, you can find in \"GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank\".
    2. \n
  2. Yep, we used psi-blast to find similar proteins for target proteins.
    3. \n
  3. LR-Interpro is also a component method derived from GOLabeler. It used InterProScan to extract protein families, domains, and motifs. Then we built a binary feature vector for each proteins based on the InterProScan results, where the '1' in j-th element indicates that the j-th feature belongs to target protein. Finally, we used these vectors to train logistic regression classifiers and make prediction.
    4. \n
  4. InterProScan is updated irregularly and collects more functional information. So, we believe that using InterProScan directly is more helpful.
    5. \n
", "votes": 2, "replies": [] } ] }, { "id": 2632074, "author_name": "", "author_url": "", "post_date": "2024-02-02T06:01:07.263000", "content": "

Thanks for sharing your work. I am also working on a similar project where I have created three embeddings from three language models (LLMs) and structural features. My doubt is about how to create my output vector. How can I predict the GO terms for each protein? Can you elaborate on that part?

", "votes": 0, "replies": [ { "id": 2632303, "author_name": "", "author_url": "", "post_date": "2024-02-02T08:56:56.350000", "content": "", "votes": 0, "replies": [] }, { "id": 2632304, "author_name": "王少钧", "author_url": "", "post_date": "2024-02-02T08:57:51.817000", "content": "
    \n
  1. For the first question, are you referring to how to obtain the representation of proteins from LLMs? Protein LLMs are multi-layered and can generate representations for each amino acid. Typically, we perform mean pooling on the output of the last layer for each amino acid, and generate the feature representation of the protein.
    2. \n
  2. To predict the GO terms for proteins, we collected large-scale functional annotation information, where each GO term is associated with numerous proteins. We treated these proteins as positive instances and considered the remaining proteins as negative instances. Then we trained a classifier for each GO term, aiming to predict the association scores between proteins and GO term.
    3. \n
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2595397": "# A.\tMODEL SUMMARY\n## A.1 Background on you/your team\nCompetition Name: CAFA5 Protein Function Prediction\n\nTeam Name: GOCurator\n\nPrivate Leaderboard Score: 0.61623\n\nPrivate Leaderboard Place: 1\n\nName: Shaojun Wang\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: 20110850027@fudan.edu.cn\n\nName: Weiqi Zhai\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: 20110850010@fudan.edu.cn\n\nName: Wei Liu\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: 21110850007@m.fudan.edu.cn\n\nName: Tianyang Huang\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: tyhuang22@m.fudan.edu.cn\n\nName: Huiying Yan\n\nLocation: Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China\n\nEmail: yanhuiyingwangyi@163.com\n\n## A2. Background on you/your team\nOur team consists of five PhD students working on bioinformatics, especially automated protein function prediction, and biomedical text mining from ZhuLab, which is led by Prof. Shanfeng Zhu at Fudan University. Our lab has participated in the CAFA3 and CAFA4 and achieved the top performance. Our team developed GOCurator by considering all kinds of information including protein sequence, network, structure, and textual information. During the competition, S.W. was responsible for collecting data, building the framework, and training the model. W.L developed a component method based on FoldSeek. T.H and H.Y were mainly engaged in biomedical text mining for protein annotation. W.Z submitted the prediction result, and performed the result analysis.\n\n## A3. Summary\nOur pipeline, GOCurator, was developed based on NetGO 3.0. In addition, we also introduced new component methods to extract functional information from protein 3D structure, protein textual description, and scientific literature. Thanks to the learning to rank model, we eventually integrated multiple component methods effectively. Figure 1 showed the score of component methods achieved in public leaderboard.\n![The performance of component methods in public leaderboard](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16029558%2F27233f10b1cbcfd1e5aec8ed8a5b85c5%2Fsolution1.png?generation=1704887698109856&alt=media )\n- LR-MEM, it combined protein description, scientific literature, and protein sequence together based on the annotation from GO and GOA. Specifically, it concatenated representations from different information source as protein representations. Then we trained logistic regression classifiers based on it. We integrated this method into NetGO 3.0 and ranked in the first place for the first time on public leaderboard. \n- FoldSeek-KNN, it made prediction according to the structure information. By comparing the structural similarities between the test proteins and the proteins in the training set, we calculated the prediction score as weighted voting by similar annotated proteins.\n- GOXML, an extreme multi-label classifier based on literature information. Similar to DeepText2GO, it used AttentionXML to achieve fast high-precision large-scale multi-label classification without training logistic regression classifiers for each GO terms.\n- GORetrieval, a novel two-stage deep learning-based framework based on the protein description and protein literature. Given a target protein, it first retrieved candidate Gene Ontology (GO) terms from annotated proteins with similar descriptions. In the second stage, the retrieved GO terms were reranked based on semantic matching between the GO descriptions and the literature information of the target protein. Moreover, the second stage can also be used to enhance the prediction from other method via reranking.\n\nWhen doing ensemble learning, the input to the model contains not only the predicted scores from each method, but also a 20-dim one-hot vector to characterize the protein species information.\n \n\n## A4. Features Selections / Engineering\nAccording to the public leaderboard scores, we believed that protein language models and textual information are useful for improving accuracy. For protein language models, we have also tried the ESM2 and ProtT5 to obtain protein embeddings. Similar to LR-ESM (based on ESM-1b), we trained logistic regression classifiers for ESM2 and ProtT5 embedding and the concatenation of all three embeddings. We found little difference in the scores achieved by these methods, but the best performance was achieved when LR-ESM was integrated as a component method into the GOCurator. Therefore, we finally used ESM-1b to generate protein embeddings. For textual information, how to generate text representations is important. Following to DeepText2GO, we used document to vector (D2V) and TF-IDF to represent scientific literature. In GORetrieval and GOXML, PubMedBERT was used to generate text embedding.\n\n## A5. Training Method(s)\nWe collected functional annotation information from three database SwissProt, GOA, and GO, according to the CAFA5 evidence code. Unlike the temporal split in NetGO, we selected proteins for validation based on the species distribution of proteins in the CAFA5 test set. Specifically, we calculate the proportion of proteins from each species in the CAFA5 test set, and sampled proteins for the corresponding species from the dataset to form a validation dataset of 1000 proteins. We have improved the performance of our methods as much as possible in this way.\n\n## A6. Interesting findings\nWe found that the Net-KNN based on protein-protein interaction performs poorly in predicting proteins from those species with less functional annotation information. When we integrated all the component methods together, we even got better results by not fusing Net-KNN into it. However, according to our previous studies (NetGO series), Net-KNN is a powerful method for BPO and CCO. Therefore, we tried to filter the prediction from Net-KNN by the species. Specifically, we considered the predictions for a target protein to be valid if the number of the annotated proteins in the target protein’s species is larger. As shown in the figure, the number of valid species has an impact on the prediction performance. Finally, we treated Net-KNN predictions for the top 15 species as effective results.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16029558%2Fc669b39b4fb0047b017c6b75a9440223%2Fsolution2.png?generation=1704888407902780&alt=media)\n\n## A7. Simple Features and Methods\nAccording to the performance of different source information, we believe that incorporating a component method based on textual or structural information into NetGO 3.0 can achieve 90% of the final performance.\n\n## A8. Model Execution Time\nFirstly, we used InterProScan and ESM-1b to get protein domain/family/motif information and protein embeddings, which took about 1-2 days, where most of the time was spent on InterProScan. Secondly, we trained each component method separately on several servers, spending roughly 10-15 hours per method. Then, the learning to rank model used for ensemble learning could be trained within 50 mins. For inference, we first used component methods to generate initial prediction, which took about 2 hours for CAFA5 test proteins. Finally, it took about 1.5 hours for ensemble learning on all test proteins.\n\n## A9. References\n\n1. You R, Zhang Z, Xiong Y, et al. GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank. Bioinformatics, Volume 34, Issue 14, 2018, 2465–2473.\n2. You R, Huang X, Zhu S. DeepText2GO: Improving large-scale protein function prediction with deep semantic text representation. Methods, 2018, 145: 82-90.\n3. You R, Yao S, Xiong Y, et al. NetGO: improving large-scale protein function prediction with massive network information. Nucleic Acids Research, Volume 47, Issue W1, 2019, W379–W387.\n4. Yao S, You R, Wang S, et al. NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information. Nucleic Acids Research, Volume 49, Issue W1, 2021, W469-W475\n5. Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021\n6. Rives A, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 118(15), 2021\n7. You R, Yao S, Mamitsuka H, et al. DeepGraphGO: graph neural network for large-scale, multispecies protein function prediction. Bioinformatics (ISMB 2021), Volume 37, Supplement_1, 2021, i262–i271.\n8. Wang S, You R, Liu Y, et al. NetGO 3.0: Protein Language Model Improves Large-scale Functional Annotations. Genomics, Proteomics & Bioinformatics, Volume 21, Issue 2, 2023, 349-358.\n", "2638950": "Is it possible to get your source code of the entire solution?", "2600363": "Neat! (I'm presenting about Cafa5/your approach in our lab). \n\nSome questions:\n1. How did you get predictions for the first level of the ensemble, simple CV?.\n2. Blast-KNN = Simple PSI-blast? \n3. LR-Interpro = ? \n4. Why extract the features from interProScan and not the uniProt records?", "2632074": "Thanks for sharing your work. I am also working on a similar project where I have created three embeddings from three language models (LLMs) and structural features. My doubt is about how to create my output vector. How can I predict the GO terms for each protein? Can you elaborate on that part?" } }