{
  "id": 466971,
  "title": "6th Place Solution for the CAFA 5 Protein Function Prediction Challenge",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/466971",
  "author_name": "Evans",
  "post_date": "2024-01-10T15:51:06.580000",
  "votes": 8,
  "comment_count": 0,
  "views": 0,
  "content": "<p>A. MODEL SUMMARY</p>\n<p>A1. Background on you/your team<br>\nCompetition Name: CAFA 5 Protein Function Prediction<br>\nTeam Name: Evans<br>\nPrivate Leaderboard Score: 0.56076<br>\nPrivate Leaderboard Place: 6<br>\nName: Quancheng Liu<br>\nLocation: Department of Computational Medicine and Bioinformatics, University of Michigan – Ann Arbor, MI, USA<br>\nEmail: <a href=\"mailto:quanch@umich.edu\">quanch@umich.edu</a></p>\n<p>A2. Background on you/your team<br>\nI am a PhD student currently working under the Freddolino Lab, engaged in developing algorithms for protein function annotation tools, with a specific focus on leveraging deep learning techniques. My participation in the CAFA 5 competition was motivated by the objective to benchmark my method. This competition offered a valuable opportunity to evaluate the efficacy of my approach within a competitive, real-world context.</p>\n<p>A3. Summary<br>\nInterlabelGO integrates a sequence alignment-based method with a neural network-based method.<br>\n In the neural network component, the model leverages the last three layers from the ESM model as input features. The architecture of this model includes several feedforward layers.<br>\nThe sequence alignment-based method utilizes blastp/diamond to query the UniProt-GOA database. Key metrics such as bitscore and sequence identity are used to compute a weighted average score from the search hits.<br>\nFor the neural network models in InterlabelGO, training each model takes approximately 3 minutes on an RTX 4090 GPU</p>\n<p>A4. Features Selection / Engineering<br>\nThe primary feature utilized in InterlabelGO is the embedding generated from the ESM model. These embeddings are critical as they encapsulate rich, contextual information about protein sequences, which is essential for accurate function annotation.</p>\n<p>A5. Training Method(s)<br>\nInterLabelGO used 5-fold random splits of the training data and trained 5 different models for each GO aspect; the final prediction for each GO term was mean score across 5 models.<br>\nInstead of using BCE loss as is often the case in comparable methods, InterLabelGO applied IA weighted soft f1 loss across both protein level and GO term level. Additionally, InterLabelGO also uses a ranked loss (Figure 1) which is the logsumexp of zero term minus positive terms plus negative term minus zero term. This rank loss was meant to capture the label correlation. During training, these loss functions are combined through simple multiplication, integrating their individual contributions to the model's learning process.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13769621%2Fe4d50771db7119d7105b2548b8dfd95e%2FPicture1.png?generation=1704901831021262&amp;alt=media\" alt=\"\"><br>\nFinally, the prediction from neural network method is linearly combined with the alignment-based method. The optimal weight was obtained from our validation dataset.</p>\n<p>A6. Interesting findings<br>\nOne of the most significant discoveries in the development of InterlabelGO was the application of the IA weighted soft F1 loss, which led to a substantial improvement in model performance compared to the traditional BCE loss. This finding is particularly noteworthy as it may point a direction for solving the prevalent issue of label imbalance in protein function prediction.</p>\n<p>A7. Simple Features and Methods<br>\nThe neural network models, utilizing ESM embeddings, and one model for each aspect, can achieve approximately 90-95% of the full model's performance, offering a simplified yet effective solution.</p>\n<p>A8. Model Execution Time<br>\nIt takes around 60 minutes to train the model for 3 aspects (5 model pre aspect)<br>\nIt takes less than 10 minutes to generate prediction for around 2000 target proteins for neural network method, while most of time is ESM feature extraction.<br>\nIt takes less than 2 minutes to generate predictions for around 2000 target proteins for alignment-based method using diamond.<br>\nAll above performance was achieved on Ubuntu 23.10 with RTX4090 and i9 13900k.</p>\n<p>A9. Acknowledgement<br>\nI would like to extend my sincere thanks to Dr. Chengxin Zhang and Dr. Lydia Freddolino for their insightful discussions.</p>\n<p>A10. Reference<br>\nSu, J. et al. (2022) ZLPR: A novel loss for multi-label classification, arXiv.org. Available at: <a href=\"https://arxiv.org/abs/2208.02955#:~:text=Compared%20to%20other%20rank%2Dbased,the%20label%20powerset%20(LP)\" target=\"_blank\">https://arxiv.org/abs/2208.02955#:~:text=Compared%20to%20other%20rank%2Dbased,the%20label%20powerset%20(LP)</a>. </p>",
  "messages": [
    {
      "id": 2595723,
      "postDate": "2024-01-10T15:51:06.580Z",
      "content": "<p>A. MODEL SUMMARY</p>\n<p>A1. Background on you/your team<br>\nCompetition Name: CAFA 5 Protein Function Prediction<br>\nTeam Name: Evans<br>\nPrivate Leaderboard Score: 0.56076<br>\nPrivate Leaderboard Place: 6<br>\nName: Quancheng Liu<br>\nLocation: Department of Computational Medicine and Bioinformatics, University of Michigan – Ann Arbor, MI, USA<br>\nEmail: <a href=\"mailto:quanch@umich.edu\">quanch@umich.edu</a></p>\n<p>A2. Background on you/your team<br>\nI am a PhD student currently working under the Freddolino Lab, engaged in developing algorithms for protein function annotation tools, with a specific focus on leveraging deep learning techniques. My participation in the CAFA 5 competition was motivated by the objective to benchmark my method. This competition offered a valuable opportunity to evaluate the efficacy of my approach within a competitive, real-world context.</p>\n<p>A3. Summary<br>\nInterlabelGO integrates a sequence alignment-based method with a neural network-based method.<br>\n In the neural network component, the model leverages the last three layers from the ESM model as input features. The architecture of this model includes several feedforward layers.<br>\nThe sequence alignment-based method utilizes blastp/diamond to query the UniProt-GOA database. Key metrics such as bitscore and sequence identity are used to compute a weighted average score from the search hits.<br>\nFor the neural network models in InterlabelGO, training each model takes approximately 3 minutes on an RTX 4090 GPU</p>\n<p>A4. Features Selection / Engineering<br>\nThe primary feature utilized in InterlabelGO is the embedding generated from the ESM model. These embeddings are critical as they encapsulate rich, contextual information about protein sequences, which is essential for accurate function annotation.</p>\n<p>A5. Training Method(s)<br>\nInterLabelGO used 5-fold random splits of the training data and trained 5 different models for each GO aspect; the final prediction for each GO term was mean score across 5 models.<br>\nInstead of using BCE loss as is often the case in comparable methods, InterLabelGO applied IA weighted soft f1 loss across both protein level and GO term level. Additionally, InterLabelGO also uses a ranked loss (Figure 1) which is the logsumexp of zero term minus positive terms plus negative term minus zero term. This rank loss was meant to capture the label correlation. During training, these loss functions are combined through simple multiplication, integrating their individual contributions to the model's learning process.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13769621%2Fe4d50771db7119d7105b2548b8dfd95e%2FPicture1.png?generation=1704901831021262&amp;alt=media\" alt=\"\"><br>\nFinally, the prediction from neural network method is linearly combined with the alignment-based method. The optimal weight was obtained from our validation dataset.</p>\n<p>A6. Interesting findings<br>\nOne of the most significant discoveries in the development of InterlabelGO was the application of the IA weighted soft F1 loss, which led to a substantial improvement in model performance compared to the traditional BCE loss. This finding is particularly noteworthy as it may point a direction for solving the prevalent issue of label imbalance in protein function prediction.</p>\n<p>A7. Simple Features and Methods<br>\nThe neural network models, utilizing ESM embeddings, and one model for each aspect, can achieve approximately 90-95% of the full model's performance, offering a simplified yet effective solution.</p>\n<p>A8. Model Execution Time<br>\nIt takes around 60 minutes to train the model for 3 aspects (5 model pre aspect)<br>\nIt takes less than 10 minutes to generate prediction for around 2000 target proteins for neural network method, while most of time is ESM feature extraction.<br>\nIt takes less than 2 minutes to generate predictions for around 2000 target proteins for alignment-based method using diamond.<br>\nAll above performance was achieved on Ubuntu 23.10 with RTX4090 and i9 13900k.</p>\n<p>A9. Acknowledgement<br>\nI would like to extend my sincere thanks to Dr. Chengxin Zhang and Dr. Lydia Freddolino for their insightful discussions.</p>\n<p>A10. Reference<br>\nSu, J. et al. (2022) ZLPR: A novel loss for multi-label classification, arXiv.org. Available at: <a href=\"https://arxiv.org/abs/2208.02955#:~:text=Compared%20to%20other%20rank%2Dbased,the%20label%20powerset%20(LP)\" target=\"_blank\">https://arxiv.org/abs/2208.02955#:~:text=Compared%20to%20other%20rank%2Dbased,the%20label%20powerset%20(LP)</a>. </p>",
      "rawMarkdown": "A. MODEL SUMMARY\n\nA1. Background on you/your team\nCompetition Name: CAFA 5 Protein Function Prediction\nTeam Name: Evans\nPrivate Leaderboard Score: 0.56076\nPrivate Leaderboard Place: 6\nName: Quancheng Liu\nLocation: Department of Computational Medicine and Bioinformatics, University of Michigan – Ann Arbor, MI, USA\nEmail: quanch@umich.edu\n\nA2. Background on you/your team\nI am a PhD student currently working under the Freddolino Lab, engaged in developing algorithms for protein function annotation tools, with a specific focus on leveraging deep learning techniques. My participation in the CAFA 5 competition was motivated by the objective to benchmark my method. This competition offered a valuable opportunity to evaluate the efficacy of my approach within a competitive, real-world context.\n\nA3. Summary\nInterlabelGO integrates a sequence alignment-based method with a neural network-based method.\n In the neural network component, the model leverages the last three layers from the ESM model as input features. The architecture of this model includes several feedforward layers.\nThe sequence alignment-based method utilizes blastp/diamond to query the UniProt-GOA database. Key metrics such as bitscore and sequence identity are used to compute a weighted average score from the search hits.\nFor the neural network models in InterlabelGO, training each model takes approximately 3 minutes on an RTX 4090 GPU\n\nA4. Features Selection / Engineering\nThe primary feature utilized in InterlabelGO is the embedding generated from the ESM model. These embeddings are critical as they encapsulate rich, contextual information about protein sequences, which is essential for accurate function annotation.\n\nA5. Training Method(s)\nInterLabelGO used 5-fold random splits of the training data and trained 5 different models for each GO aspect; the final prediction for each GO term was mean score across 5 models.\nInstead of using BCE loss as is often the case in comparable methods, InterLabelGO applied IA weighted soft f1 loss across both protein level and GO term level. Additionally, InterLabelGO also uses a ranked loss (Figure 1) which is the logsumexp of zero term minus positive terms plus negative term minus zero term. This rank loss was meant to capture the label correlation. During training, these loss functions are combined through simple multiplication, integrating their individual contributions to the model's learning process.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13769621%2Fe4d50771db7119d7105b2548b8dfd95e%2FPicture1.png?generation=1704901831021262&alt=media)\nFinally, the prediction from neural network method is linearly combined with the alignment-based method. The optimal weight was obtained from our validation dataset.\n\nA6. Interesting findings\nOne of the most significant discoveries in the development of InterlabelGO was the application of the IA weighted soft F1 loss, which led to a substantial improvement in model performance compared to the traditional BCE loss. This finding is particularly noteworthy as it may point a direction for solving the prevalent issue of label imbalance in protein function prediction.\n\nA7. Simple Features and Methods\nThe neural network models, utilizing ESM embeddings, and one model for each aspect, can achieve approximately 90-95% of the full model's performance, offering a simplified yet effective solution.\n\nA8. Model Execution Time\nIt takes around 60 minutes to train the model for 3 aspects (5 model pre aspect)\nIt takes less than 10 minutes to generate prediction for around 2000 target proteins for neural network method, while most of time is ESM feature extraction.\nIt takes less than 2 minutes to generate predictions for around 2000 target proteins for alignment-based method using diamond.\nAll above performance was achieved on Ubuntu 23.10 with RTX4090 and i9 13900k.\n\nA9. Acknowledgement\nI would like to extend my sincere thanks to Dr. Chengxin Zhang and Dr. Lydia Freddolino for their insightful discussions.\n\nA10. Reference\nSu, J. et al. (2022) ZLPR: A novel loss for multi-label classification, arXiv.org. Available at: https://arxiv.org/abs/2208.02955#:~:text=Compared%20to%20other%20rank%2Dbased,the%20label%20powerset%20(LP). \n\n",
      "votes": 8
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2595723": "A. MODEL SUMMARY\n\nA1. Background on you/your team\nCompetition Name: CAFA 5 Protein Function Prediction\nTeam Name: Evans\nPrivate Leaderboard Score: 0.56076\nPrivate Leaderboard Place: 6\nName: Quancheng Liu\nLocation: Department of Computational Medicine and Bioinformatics, University of Michigan – Ann Arbor, MI, USA\nEmail: quanch@umich.edu\n\nA2. Background on you/your team\nI am a PhD student currently working under the Freddolino Lab, engaged in developing algorithms for protein function annotation tools, with a specific focus on leveraging deep learning techniques. My participation in the CAFA 5 competition was motivated by the objective to benchmark my method. This competition offered a valuable opportunity to evaluate the efficacy of my approach within a competitive, real-world context.\n\nA3. Summary\nInterlabelGO integrates a sequence alignment-based method with a neural network-based method.\n In the neural network component, the model leverages the last three layers from the ESM model as input features. The architecture of this model includes several feedforward layers.\nThe sequence alignment-based method utilizes blastp/diamond to query the UniProt-GOA database. Key metrics such as bitscore and sequence identity are used to compute a weighted average score from the search hits.\nFor the neural network models in InterlabelGO, training each model takes approximately 3 minutes on an RTX 4090 GPU\n\nA4. Features Selection / Engineering\nThe primary feature utilized in InterlabelGO is the embedding generated from the ESM model. These embeddings are critical as they encapsulate rich, contextual information about protein sequences, which is essential for accurate function annotation.\n\nA5. Training Method(s)\nInterLabelGO used 5-fold random splits of the training data and trained 5 different models for each GO aspect; the final prediction for each GO term was mean score across 5 models.\nInstead of using BCE loss as is often the case in comparable methods, InterLabelGO applied IA weighted soft f1 loss across both protein level and GO term level. Additionally, InterLabelGO also uses a ranked loss (Figure 1) which is the logsumexp of zero term minus positive terms plus negative term minus zero term. This rank loss was meant to capture the label correlation. During training, these loss functions are combined through simple multiplication, integrating their individual contributions to the model's learning process.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13769621%2Fe4d50771db7119d7105b2548b8dfd95e%2FPicture1.png?generation=1704901831021262&alt=media)\nFinally, the prediction from neural network method is linearly combined with the alignment-based method. The optimal weight was obtained from our validation dataset.\n\nA6. Interesting findings\nOne of the most significant discoveries in the development of InterlabelGO was the application of the IA weighted soft F1 loss, which led to a substantial improvement in model performance compared to the traditional BCE loss. This finding is particularly noteworthy as it may point a direction for solving the prevalent issue of label imbalance in protein function prediction.\n\nA7. Simple Features and Methods\nThe neural network models, utilizing ESM embeddings, and one model for each aspect, can achieve approximately 90-95% of the full model's performance, offering a simplified yet effective solution.\n\nA8. Model Execution Time\nIt takes around 60 minutes to train the model for 3 aspects (5 model pre aspect)\nIt takes less than 10 minutes to generate prediction for around 2000 target proteins for neural network method, while most of time is ESM feature extraction.\nIt takes less than 2 minutes to generate predictions for around 2000 target proteins for alignment-based method using diamond.\nAll above performance was achieved on Ubuntu 23.10 with RTX4090 and i9 13900k.\n\nA9. Acknowledgement\nI would like to extend my sincere thanks to Dr. Chengxin Zhang and Dr. Lydia Freddolino for their insightful discussions.\n\nA10. Reference\nSu, J. et al. (2022) ZLPR: A novel loss for multi-label classification, arXiv.org. Available at: https://arxiv.org/abs/2208.02955#:~:text=Compared%20to%20other%20rank%2Dbased,the%20label%20powerset%20(LP). \n\n"
  }
}