![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1328185%2F992093af6907d659132ab600b9f5273d%2FScreenshot%202023-08-22%20at%2022.50.16.png?generation=1692737454640597&alt=media\")
Thanks to Kaggle & host for bringing CAFA to the Kaggle platform, it’s an amazing competition! Also big thanks to my teammate @samusram, it’s been a pleasure working together!
\nThis isn’t a solution writeup, but a negative result write up. We decided to stay away from tuning the PLM embeddings and explore some new directions which didn’t work out at the end (we’ve had pretty good CV/LB correlation). I think there is some further potential though, I’d love to get your feedback.
\nThe main theme for me was trying to get proteins and functions to meet in the latent space. To do this, I mainly experimented with two approaches.
\n1. Contrastive learning (CLIP)
\nThe first approach was inspired by CLIP, encoding 2 different modalities (proteins and functions) and training a model to match them with contrastive loss (multiple negatives ranking loss, MNRL). Proteins were embedded with ESM, and for GO Terms I tried both PubMedBERT embeddings (using term name, description and relationships) and Anc2Vec embeddings.
\nThe hard part of this approach is making sure that positive pairs are only on the diagonal of the illustrated matrix before calculating loss. I put some attention in data loading to ensure that, but the model still failed to converge properly. One potential improvement might be using hierarchical contrastive loss, but I got frustrated seeing no improvements and moved on to the second idea.
\n
2. Joint Sequence & Function Transformer
\nThis approach was inspired by PFresGO, which proposed an attention-based mechanism incorporating inter-relationships among annotations. The model architecture is presented in the chart below.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1328185%2F5757b4ef43dae4c24e98379c03b7f39f%2FScreenshot%202023-08-22%20at%2022.50.33.png?generation=1692737488206186&alt=media\")
In this model, the sequence and a set of GO terms is embedded and then fed into a transformer that ideally should pick up inter-relationships between specific residues and functions, as well as interactions among functions. The first experiment with this approach worked reasonably well (0.48 CV, 0.44 LB) but then every “improvement” made the results worse… Either I missed something important in the implementation, or my understanding of the problem is wrong, I’d expect this to work.
\nSome other things that didn’t work for us:
\nThe final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings. We got a last minute boost when Raman implemented propagation of predictions to parent terms.
\nI’m still bullish that the joint transformer approach should work, and I missed something there… if anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!
", "messages": [ { "id": "2403768", "postDate": "08/22/2023 20:54:37", "content": "Thanks to Kaggle & host for bringing CAFA to the Kaggle platform, it’s an amazing competition! Also big thanks to my teammate @samusram, it’s been a pleasure working together!
\nThis isn’t a solution writeup, but a negative result write up. We decided to stay away from tuning the PLM embeddings and explore some new directions which didn’t work out at the end (we’ve had pretty good CV/LB correlation). I think there is some further potential though, I’d love to get your feedback.
\nThe main theme for me was trying to get proteins and functions to meet in the latent space. To do this, I mainly experimented with two approaches.
\n1. Contrastive learning (CLIP)
\nThe first approach was inspired by CLIP, encoding 2 different modalities (proteins and functions) and training a model to match them with contrastive loss (multiple negatives ranking loss, MNRL). Proteins were embedded with ESM, and for GO Terms I tried both PubMedBERT embeddings (using term name, description and relationships) and Anc2Vec embeddings.
\nThe hard part of this approach is making sure that positive pairs are only on the diagonal of the illustrated matrix before calculating loss. I put some attention in data loading to ensure that, but the model still failed to converge properly. One potential improvement might be using hierarchical contrastive loss, but I got frustrated seeing no improvements and moved on to the second idea.
\n
2. Joint Sequence & Function Transformer
\nThis approach was inspired by PFresGO, which proposed an attention-based mechanism incorporating inter-relationships among annotations. The model architecture is presented in the chart below.
\n
In this model, the sequence and a set of GO terms is embedded and then fed into a transformer that ideally should pick up inter-relationships between specific residues and functions, as well as interactions among functions. The first experiment with this approach worked reasonably well (0.48 CV, 0.44 LB) but then every “improvement” made the results worse… Either I missed something important in the implementation, or my understanding of the problem is wrong, I’d expect this to work.
\nSome other things that didn’t work for us:
\nThe final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings. We got a last minute boost when Raman implemented propagation of predictions to parent terms.
\nI’m still bullish that the joint transformer approach should work, and I missed something there… if anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!
", "rawMarkdown": "Thanks to Kaggle & host for bringing CAFA to the Kaggle platform, it’s an amazing competition! Also big thanks to my teammate @samusram, it’s been a pleasure working together! \n\nThis isn’t a solution writeup, but a negative result write up. We decided to stay away from tuning the PLM embeddings and explore some new directions which didn’t work out at the end (we’ve had pretty good CV/LB correlation). I think there is some further potential though, I’d love to get your feedback. \n\nThe main theme for me was trying to get proteins and functions to meet in the latent space. To do this, I mainly experimented with two approaches.\n\n**1. Contrastive learning (CLIP)**\n\nThe first approach was inspired by CLIP, encoding 2 different modalities (proteins and functions) and training a model to match them with contrastive loss (multiple negatives ranking loss, MNRL). Proteins were embedded with ESM, and for GO Terms I tried both PubMedBERT embeddings (using term name, description and relationships) and Anc2Vec embeddings. \n\nThe hard part of this approach is making sure that positive pairs are only on the diagonal of the illustrated matrix before calculating loss. I put some attention in data loading to ensure that, but the model still failed to converge properly. One potential improvement might be using hierarchical contrastive loss, but I got frustrated seeing no improvements and moved on to the second idea. \n\n\n\n**2. Joint Sequence & Function Transformer**\n\nThis approach was inspired by PFresGO, which proposed an attention-based mechanism incorporating inter-relationships among annotations. The model architecture is presented in the chart below. \n\n\n\nIn this model, the sequence and a set of GO terms is embedded and then fed into a transformer that ideally should pick up inter-relationships between specific residues and functions, as well as interactions among functions. The first experiment with this approach worked reasonably well (0.48 CV, 0.44 LB) but then every “improvement” made the results worse… Either I missed something important in the implementation, or my understanding of the problem is wrong, I’d expect this to work. \n\nSome other things that didn’t work for us: \n- Interpro features\n- STRING-based prediction based on protein interactions\n- Positive unlabeled learning based on CATH domains\n\nThe final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings. We got a last minute boost when Raman implemented propagation of predictions to parent terms. \n\nI’m still bullish that the joint transformer approach should work, and I missed something there… if anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!", "votes": null }, { "id": "2403813", "postDate": "08/22/2023 21:34:04", "content": "\n\nThe final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings
\n
Did you limit the number of predicted labels for proteins? Or did you train linear models on all labels?
", "rawMarkdown": ">The final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings\n\nDid you limit the number of predicted labels for proteins? Or did you train linear models on all labels?", "votes": null }, { "id": "2403815", "postDate": "08/22/2023 21:35:35", "content": "We trained linear models for all GO terms with > 100 annotations in train, then kept top 1500 predictions per protein.
", "rawMarkdown": "We trained linear models for all GO terms with > 100 annotations in train, then kept top 1500 predictions per protein.", "votes": null }, { "id": "2403816", "postDate": "08/22/2023 21:37:12", "content": "This sounds like an excellent idea, at least conceptually.
", "rawMarkdown": "This sounds like an excellent idea, at least conceptually.", "votes": null }, { "id": "2403821", "postDate": "08/22/2023 21:46:34", "content": "Thanks Tilii, I appreciate this feedback coming from you, have been learning a lot from your posts!
", "rawMarkdown": "Thanks Tilii, I appreciate this feedback coming from you, have been learning a lot from your posts!", "votes": null }, { "id": "2403832", "postDate": "08/22/2023 22:05:03", "content": "Thank you for the writeup, we had some similar ideas with mimicking Pfresgo but was not able to get far. The idea of joint embedding I think is very interesting and seems similar conceptually to DeepGoZero.
\n\n\nif anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!
\n
I would love to explore this further, let me know if this is something you might be keen on!
", "rawMarkdown": "Thank you for the writeup, we had some similar ideas with mimicking Pfresgo but was not able to get far. The idea of joint embedding I think is very interesting and seems similar conceptually to DeepGoZero.\n\n>if anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!\n\nI would love to explore this further, let me know if this is something you might be keen on!", "votes": null }, { "id": "2403836", "postDate": "08/22/2023 22:07:27", "content": "I need to rest for a couple days - imagine you too :) - but let's chat then! I'll reach out!
", "rawMarkdown": "I need to rest for a couple days - imagine you too :) - but let's chat then! I'll reach out!", "votes": null }, { "id": "2403840", "postDate": "08/22/2023 22:12:11", "content": "of course! let's chat soon :)
", "rawMarkdown": "of course! let's chat soon :)", "votes": null }, { "id": "2403900", "postDate": "08/22/2023 23:40:50", "content": "Count me in as well, at least for an initial discussion. Whether actually I will be able to contribute will depend on the timeline.
\nI feel like there are several ideas that were left unexplored, mostly because I joined really late. Even though it is squarely my fault, I feel cheated and planned to submit several models later. Didn't know that new submissions would be disabled.
", "rawMarkdown": "Count me in as well, at least for an initial discussion. Whether actually I will be able to contribute will depend on the timeline.\n\nI feel like there are several ideas that were left unexplored, mostly because I joined really late. Even though it is squarely my fault, I feel cheated and planned to submit several models later. Didn't know that new submissions would be disabled.", "votes": null }, { "id": "2404080", "postDate": "08/23/2023 03:38:28", "content": "' Raman implemented propagation of predictions to parent terms.'
\nCan you elaborate more on this part? 😀
", "rawMarkdown": "' Raman implemented propagation of predictions to parent terms.'\n\nCan you elaborate more on this part? 😀", "votes": null }, { "id": "2404341", "postDate": "08/23/2023 07:34:22", "content": "Thanks a lot Tilii I'll reach out to as well for scheduling ❤️
", "rawMarkdown": "Thanks a lot Tilii I'll reach out to as well for scheduling ❤️", "votes": null }, { "id": "2404405", "postDate": "08/23/2023 08:28:23", "content": "Here is the original post
", "rawMarkdown": "Here is the original [post](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433779#2403944)", "votes": null }, { "id": "2404747", "postDate": "08/23/2023 13:42:18", "content": "Thank you for sharing your experience! The first part (1. Contrastive learning (CLIP)) seems to be familiar with OntoProtein approach (paper, code). But authors trained ProtBert encoder for sequences. We ve tried to use embedings from this model, but also didnt achive better results. Have you seen it or maybe tried it?
", "rawMarkdown": "Thank you for sharing your experience! The first part (1. Contrastive learning (CLIP)) seems to be familiar with OntoProtein approach ([paper](https://arxiv.org/pdf/2201.11147.pdf), [code](https://github.com/zjunlp/OntoProtein?ysclid=llnrpzukqr278905031)). But authors trained ProtBert encoder for sequences. We ve tried to use embedings from this model, but also didnt achive better results. Have you seen it or maybe tried it?", "votes": null }, { "id": "2405333", "postDate": "08/23/2023 20:25:11", "content": "Yes, I've seen the paper but haven't tried the model or the code, I've had a contrastive learning pipeline from a previous competition that I wanted to use instead. Maybe that was a mistake :)
", "rawMarkdown": "Yes, I've seen the paper but haven't tried the model or the code, I've had a contrastive learning pipeline from a previous competition that I wanted to use instead. Maybe that was a mistake :)", "votes": null }, { "id": "2412690", "postDate": "08/28/2023 12:43:32", "content": "Neat! I did something similar in my (non published) approach, and was really surprisied that it didn't improve the model or few-shot results, even with using pretrained models for both representations.
", "rawMarkdown": "Neat! I did something similar in my (non published) approach, and was really surprisied that it didn't improve the model or few-shot results, even with using pretrained models for both representations.", "votes": null } ], "comments": [ { "id": 2403813, "author_name": "manwithaflower", "author_url": "", "post_date": "08/22/2023 21:34:04", "content": "\n\nThe final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings
\n
Did you limit the number of predicted labels for proteins? Or did you train linear models on all labels?
", "votes": null, "replies": [ { "id": 2403815, "author_name": "thedrcat", "author_url": "", "post_date": "08/22/2023 21:35:35", "content": "We trained linear models for all GO terms with > 100 annotations in train, then kept top 1500 predictions per protein.
", "votes": null, "replies": [] } ] }, { "id": 2403816, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 21:37:12", "content": "This sounds like an excellent idea, at least conceptually.
", "votes": null, "replies": [ { "id": 2403821, "author_name": "thedrcat", "author_url": "", "post_date": "08/22/2023 21:46:34", "content": "Thanks Tilii, I appreciate this feedback coming from you, have been learning a lot from your posts!
", "votes": null, "replies": [] } ] }, { "id": 2403832, "author_name": "zmcxjt", "author_url": "", "post_date": "08/22/2023 22:05:03", "content": "Thank you for the writeup, we had some similar ideas with mimicking Pfresgo but was not able to get far. The idea of joint embedding I think is very interesting and seems similar conceptually to DeepGoZero.
\n\n\nif anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!
\n
I would love to explore this further, let me know if this is something you might be keen on!
", "votes": null, "replies": [ { "id": 2403836, "author_name": "thedrcat", "author_url": "", "post_date": "08/22/2023 22:07:27", "content": "I need to rest for a couple days - imagine you too :) - but let's chat then! I'll reach out!
", "votes": null, "replies": [ { "id": 2403840, "author_name": "zmcxjt", "author_url": "", "post_date": "08/22/2023 22:12:11", "content": "of course! let's chat soon :)
", "votes": null, "replies": [] }, { "id": 2403900, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 23:40:50", "content": "Count me in as well, at least for an initial discussion. Whether actually I will be able to contribute will depend on the timeline.
\nI feel like there are several ideas that were left unexplored, mostly because I joined really late. Even though it is squarely my fault, I feel cheated and planned to submit several models later. Didn't know that new submissions would be disabled.
", "votes": null, "replies": [ { "id": 2404341, "author_name": "thedrcat", "author_url": "", "post_date": "08/23/2023 07:34:22", "content": "Thanks a lot Tilii I'll reach out to as well for scheduling ❤️
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2404080, "author_name": "devchopin", "author_url": "", "post_date": "08/23/2023 03:38:28", "content": "' Raman implemented propagation of predictions to parent terms.'
\nCan you elaborate more on this part? 😀
", "votes": null, "replies": [ { "id": 2404405, "author_name": "dustinthewind", "author_url": "", "post_date": "08/23/2023 08:28:23", "content": "Here is the original post
", "votes": null, "replies": [] } ] }, { "id": 2404747, "author_name": "andreylalaley", "author_url": "", "post_date": "08/23/2023 13:42:18", "content": "Thank you for sharing your experience! The first part (1. Contrastive learning (CLIP)) seems to be familiar with OntoProtein approach (paper, code). But authors trained ProtBert encoder for sequences. We ve tried to use embedings from this model, but also didnt achive better results. Have you seen it or maybe tried it?
", "votes": null, "replies": [ { "id": 2405333, "author_name": "thedrcat", "author_url": "", "post_date": "08/23/2023 20:25:11", "content": "Yes, I've seen the paper but haven't tried the model or the code, I've had a contrastive learning pipeline from a previous competition that I wanted to use instead. Maybe that was a mistake :)
", "votes": null, "replies": [] } ] }, { "id": 2412690, "author_name": "danofer", "author_url": "", "post_date": "08/28/2023 12:43:32", "content": "Neat! I did something similar in my (non published) approach, and was really surprisied that it didn't improve the model or few-shot results, even with using pretrained models for both representations.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2403768": "Thanks to Kaggle & host for bringing CAFA to the Kaggle platform, it’s an amazing competition! Also big thanks to my teammate @samusram, it’s been a pleasure working together! \n\nThis isn’t a solution writeup, but a negative result write up. We decided to stay away from tuning the PLM embeddings and explore some new directions which didn’t work out at the end (we’ve had pretty good CV/LB correlation). I think there is some further potential though, I’d love to get your feedback. \n\nThe main theme for me was trying to get proteins and functions to meet in the latent space. To do this, I mainly experimented with two approaches.\n\n**1. Contrastive learning (CLIP)**\n\nThe first approach was inspired by CLIP, encoding 2 different modalities (proteins and functions) and training a model to match them with contrastive loss (multiple negatives ranking loss, MNRL). Proteins were embedded with ESM, and for GO Terms I tried both PubMedBERT embeddings (using term name, description and relationships) and Anc2Vec embeddings. \n\nThe hard part of this approach is making sure that positive pairs are only on the diagonal of the illustrated matrix before calculating loss. I put some attention in data loading to ensure that, but the model still failed to converge properly. One potential improvement might be using hierarchical contrastive loss, but I got frustrated seeing no improvements and moved on to the second idea. \n\n\n\n**2. Joint Sequence & Function Transformer**\n\nThis approach was inspired by PFresGO, which proposed an attention-based mechanism incorporating inter-relationships among annotations. The model architecture is presented in the chart below. \n\n\n\nIn this model, the sequence and a set of GO terms is embedded and then fed into a transformer that ideally should pick up inter-relationships between specific residues and functions, as well as interactions among functions. The first experiment with this approach worked reasonably well (0.48 CV, 0.44 LB) but then every “improvement” made the results worse… Either I missed something important in the implementation, or my understanding of the problem is wrong, I’d expect this to work. \n\nSome other things that didn’t work for us: \n- Interpro features\n- STRING-based prediction based on protein interactions\n- Positive unlabeled learning based on CATH domains\n\nThe final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings. We got a last minute boost when Raman implemented propagation of predictions to parent terms. \n\nI’m still bullish that the joint transformer approach should work, and I missed something there… if anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!", "2403813": ">The final solution is a blend of some publicly shared models and linear models on top of the ESM and Prot5 embeddings\n\nDid you limit the number of predicted labels for proteins? Or did you train linear models on all labels?", "2403815": "We trained linear models for all GO terms with > 100 annotations in train, then kept top 1500 predictions per protein.", "2403816": "This sounds like an excellent idea, at least conceptually.", "2403821": "Thanks Tilii, I appreciate this feedback coming from you, have been learning a lot from your posts!", "2403832": "Thank you for the writeup, we had some similar ideas with mimicking Pfresgo but was not able to get far. The idea of joint embedding I think is very interesting and seems similar conceptually to DeepGoZero.\n\n>if anyone working in the field would like to partner up and try to develop this method further, I’d love to collaborate!\n\nI would love to explore this further, let me know if this is something you might be keen on!", "2403836": "I need to rest for a couple days - imagine you too :) - but let's chat then! I'll reach out!", "2403840": "of course! let's chat soon :)", "2403900": "Count me in as well, at least for an initial discussion. Whether actually I will be able to contribute will depend on the timeline.\n\nI feel like there are several ideas that were left unexplored, mostly because I joined really late. Even though it is squarely my fault, I feel cheated and planned to submit several models later. Didn't know that new submissions would be disabled.", "2404080": "' Raman implemented propagation of predictions to parent terms.'\n\nCan you elaborate more on this part? 😀", "2404341": "Thanks a lot Tilii I'll reach out to as well for scheduling ❤️", "2404405": "Here is the original [post](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433779#2403944)", "2404747": "Thank you for sharing your experience! The first part (1. Contrastive learning (CLIP)) seems to be familiar with OntoProtein approach ([paper](https://arxiv.org/pdf/2201.11147.pdf), [code](https://github.com/zjunlp/OntoProtein?ysclid=llnrpzukqr278905031)). But authors trained ProtBert encoder for sequences. We ve tried to use embedings from this model, but also didnt achive better results. Have you seen it or maybe tried it?", "2405333": "Yes, I've seen the paper but haven't tried the model or the code, I've had a contrastive learning pipeline from a previous competition that I wanted to use instead. Maybe that was a mistake :)", "2412690": "Neat! I did something similar in my (non published) approach, and was really surprisied that it didn't improve the model or few-shot results, even with using pretrained models for both representations." }, "source": "meta" }