A. MODEL SUMMARY
\nA1. Background on you/your team
\nCompetition Name: CAFA 5 Protein Function Prediction
\nTeam Name: hfm7zc
\nPrivate Leaderboard Score: 0.56171 (maximum weighted F-measure, wFmax)
\nPrivate Leaderboard Place: 5
Name: Chengxin Zhang
\nLocation: Department of Computational Medicine and Bioinformatics, University of Michigan – Ann Arbor, MI, USA
\nEmail: zcx@umich.edu
Name: P Lydia Freddolino
\nLocation: Department of Biological Chemistry, University of Michigan – Ann Arbor, MI, USA
\nEmail: lydsf@umich.edu
A2. Background on you/your team
\nOur team consists of two faculty members working on algorithm development of protein function annotation tools as well as their proteome-wide application to microbes and human. We have previously participated in the CAFA3 challenge as team Zhang-Freddolino lab, which was ranked first in the CAFA3 limited knowledge Biological Process category in CAFA3 [1]. We participated in CAFA5 to benchmark our latest protein function prediction methods, with a particular focus on the assessment of utility of structure templates and deep learning in function prediction. C.Z. and P.L.F. conceived the project and designed the pipeline. C.Z. developed the method, performed the CAFA5 prediction and submitted the prediction result.
A3. Summary
\nOur workflow, StarFunc, first independently generates five sets of Gene Ontology (GO) predictions for a query protein using five component methods:
A4. Features Selection / Engineering
\nBased on wFmax of the component methods in an independent test set, the features (i.e., predictions from component methods) ranked in descending order of importance are sequence, deep learning, structure, Pfam, PPI and Naïve (Figure 1).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16315279%2Ffe2b66c5b505f9bad5465314041c8a2d%2FFeatureImportance.png?generation=1703270675891555&alt=media\")
\nFigure 1. Importance of different features, as measured by the wFmax of GO prediction on independent test set.
A major reason for the high performance of the sequence-based component method is the introduction of a new scoring function that weights each BLASTp hit by both the bit-score and the sequence identity [10].
\nFor the component methods and the consensus prediction, only experimental GO terms (evidence codes EXP, IDA, IPI, IMP, IGI, IEP, HTP, HDA, HMP, HGI, HEP, TAS, and IC) are included in the training dataset. On the other hand, during training, if a protein has already has a non-experimental GO annotation in UniProt-GOA, this GO annotation is also included in the final result, where the prediction score is calculated as one minus the error rate associated with the evidence code, as determined by our previous study [11]. Note that GO annotations with IBA evidence are always excluded as they are found to have higher error rate than other types of GO annotations [11].
A5. Training Method(s)
\nWe tested traditional Gradient Boosting Decision Tree (GBDT), Random Forest (RF), and Dropouts meet Multiple Additive Regression Trees (DART) implemented by LightGBM to ensemble the predictions from the five component methods. On an independent test set based on UniProt-GOA release 2023-05-18, we found that comparable results can be achieved by GBDT with 1000 trees and RF with 4000 trees, both of which are more accurate than DART. We eventually selected RF to derives the consensus prediction.
A6. Interesting findings
\nThe treatment of term GO:0005515 “protein binding” is different from previous CAFA challenges. In previous CAFA challenges (CAFA1 to CAFA3), proteins whose only Molecular Function (MF) leaf term is \"protein binding\" will be excluded from MF evaluation. According to the CAFA3 paper, \"protein binding is a highly generalized function description, does not provide more specific information about the actual function of a protein, and in many cases may indicate a non-functional, non-specific binding. If it is the only annotation that a protein has gained, … we deleted these annotations from our benchmark set.\"
\nHowever, the CAFA5 Kaggle challenge does not treat the “protein binding” term differently, while it is unclear whether the “protein binding” term will be treated differently in the academic paper reporting CAFA5 official assessment. Due to these discrepancies, we submitted two different models using the same architecture to CAFA5, where model “zcx” exclude protein binding-only proteins when training the MF prediction model while model “hfm7zc” include these training proteins. Model “hfm7zc” seems to achieve better Leaderboard score (0.56171, ranked 5) than “zcx” (0.55539, ranked 8), showing that different treatment of “protein binding” can give a significant impact on the final Leaderboard ranking. We would like to register our opinion, however, that for the academic CAFA5 assessment, the exclusion of protein binding as a leaf term would be more appropriate and biologically useful .
\nWe also note, in contrast to the findings reported below in section A7, that based on our prior experience we expect that for more challenging annotation targets, it is likely that the non-sequence pipelines will be more important [12, 13].
A7. Simple Features and Methods
\nBased on independent test, 98.7% of the performance (in terms of wFmax) can be achieved by the sequence-based component alone.
A8. Model Execution Time
\nThe random forest models as well as the logistic regression models mentioned in section A3 can be trained within < 1 day using the CAFA5 training set. To perform inference, the overall pipeline takes a few minutes per protein, where most of the time is spent by the structure template alignment component. If only the sequence-based component is run, it takes at most a few seconds per protein in large-scale inference.
A9. Acknowledgement
\nWe thank Quancheng Liu and Dr Xiaoqiong Wei for technical supports and insightful discussions. This work used the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
A10. References
\nA. MODEL SUMMARY
\nA1. Background on you/your team
\nCompetition Name: CAFA 5 Protein Function Prediction
\nTeam Name: hfm7zc
\nPrivate Leaderboard Score: 0.56171 (maximum weighted F-measure, wFmax)
\nPrivate Leaderboard Place: 5
Name: Chengxin Zhang
\nLocation: Department of Computational Medicine and Bioinformatics, University of Michigan – Ann Arbor, MI, USA
\nEmail: zcx@umich.edu
Name: P Lydia Freddolino
\nLocation: Department of Biological Chemistry, University of Michigan – Ann Arbor, MI, USA
\nEmail: lydsf@umich.edu
A2. Background on you/your team
\nOur team consists of two faculty members working on algorithm development of protein function annotation tools as well as their proteome-wide application to microbes and human. We have previously participated in the CAFA3 challenge as team Zhang-Freddolino lab, which was ranked first in the CAFA3 limited knowledge Biological Process category in CAFA3 [1]. We participated in CAFA5 to benchmark our latest protein function prediction methods, with a particular focus on the assessment of utility of structure templates and deep learning in function prediction. C.Z. and P.L.F. conceived the project and designed the pipeline. C.Z. developed the method, performed the CAFA5 prediction and submitted the prediction result.
A3. Summary
\nOur workflow, StarFunc, first independently generates five sets of Gene Ontology (GO) predictions for a query protein using five component methods:
A4. Features Selection / Engineering
\nBased on wFmax of the component methods in an independent test set, the features (i.e., predictions from component methods) ranked in descending order of importance are sequence, deep learning, structure, Pfam, PPI and Naïve (Figure 1).
\n
\nFigure 1. Importance of different features, as measured by the wFmax of GO prediction on independent test set.
A major reason for the high performance of the sequence-based component method is the introduction of a new scoring function that weights each BLASTp hit by both the bit-score and the sequence identity [10].
\nFor the component methods and the consensus prediction, only experimental GO terms (evidence codes EXP, IDA, IPI, IMP, IGI, IEP, HTP, HDA, HMP, HGI, HEP, TAS, and IC) are included in the training dataset. On the other hand, during training, if a protein has already has a non-experimental GO annotation in UniProt-GOA, this GO annotation is also included in the final result, where the prediction score is calculated as one minus the error rate associated with the evidence code, as determined by our previous study [11]. Note that GO annotations with IBA evidence are always excluded as they are found to have higher error rate than other types of GO annotations [11].
A5. Training Method(s)
\nWe tested traditional Gradient Boosting Decision Tree (GBDT), Random Forest (RF), and Dropouts meet Multiple Additive Regression Trees (DART) implemented by LightGBM to ensemble the predictions from the five component methods. On an independent test set based on UniProt-GOA release 2023-05-18, we found that comparable results can be achieved by GBDT with 1000 trees and RF with 4000 trees, both of which are more accurate than DART. We eventually selected RF to derives the consensus prediction.
A6. Interesting findings
\nThe treatment of term GO:0005515 “protein binding” is different from previous CAFA challenges. In previous CAFA challenges (CAFA1 to CAFA3), proteins whose only Molecular Function (MF) leaf term is \"protein binding\" will be excluded from MF evaluation. According to the CAFA3 paper, \"protein binding is a highly generalized function description, does not provide more specific information about the actual function of a protein, and in many cases may indicate a non-functional, non-specific binding. If it is the only annotation that a protein has gained, … we deleted these annotations from our benchmark set.\"
\nHowever, the CAFA5 Kaggle challenge does not treat the “protein binding” term differently, while it is unclear whether the “protein binding” term will be treated differently in the academic paper reporting CAFA5 official assessment. Due to these discrepancies, we submitted two different models using the same architecture to CAFA5, where model “zcx” exclude protein binding-only proteins when training the MF prediction model while model “hfm7zc” include these training proteins. Model “hfm7zc” seems to achieve better Leaderboard score (0.56171, ranked 5) than “zcx” (0.55539, ranked 8), showing that different treatment of “protein binding” can give a significant impact on the final Leaderboard ranking. We would like to register our opinion, however, that for the academic CAFA5 assessment, the exclusion of protein binding as a leaf term would be more appropriate and biologically useful .
\nWe also note, in contrast to the findings reported below in section A7, that based on our prior experience we expect that for more challenging annotation targets, it is likely that the non-sequence pipelines will be more important [12, 13].
A7. Simple Features and Methods
\nBased on independent test, 98.7% of the performance (in terms of wFmax) can be achieved by the sequence-based component alone.
A8. Model Execution Time
\nThe random forest models as well as the logistic regression models mentioned in section A3 can be trained within < 1 day using the CAFA5 training set. To perform inference, the overall pipeline takes a few minutes per protein, where most of the time is spent by the structure template alignment component. If only the sequence-based component is run, it takes at most a few seconds per protein in large-scale inference.
A9. Acknowledgement
\nWe thank Quancheng Liu and Dr Xiaoqiong Wei for technical supports and insightful discussions. This work used the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
A10. References
\n