Very relevant for this competition (we worked on it initially for CAFA 4): ProteinBert - a pretrained deep learning language model that also had GO annotations as a pretraining task. Trained on >100M proteins. Training, and preprocessing code and Model are available, and fast.
\nhttps://github.com/nadavbra/protein_bert
\n\n", "messages": [ { "id": "2226730", "postDate": "04/19/2023 07:00:25", "content": "Brandes, N., Ofer, D., Peleg, Y., Rappoport, N. & Linial, M.
\n
\n ProteinBERT: A universal deep-learning model of protein sequence and function.
\n Bioinformatics (2022). https://doi.org/10.1093/bioinformatics/btac020
Very relevant for this competition (we worked on it initially for CAFA 4): ProteinBert - a pretrained deep learning language model that also had GO annotations as a pretraining task. Trained on >100M proteins. Training, and preprocessing code and Model are available, and fast.
\nhttps://github.com/nadavbra/protein_bert
\n\n", "rawMarkdown": "Very relevant for this competition (we worked on it initially for CAFA 4): ProteinBert - a pretrained deep learning language model that also had GO annotations as a pretraining task. Trained on >100M proteins. Training, and preprocessing code and Model are available, and fast.\n\n\nhttps://github.com/nadavbra/protein_bert\n\n>Brandes, N., Ofer, D., Peleg, Y., Rappoport, N. & Linial, M. \nProteinBERT: A universal deep-learning model of protein sequence and function. \nBioinformatics (2022). https://doi.org/10.1093/bioinformatics/btac020", "votes": null }, { "id": "2227164", "postDate": "04/19/2023 14:45:14", "content": "Brandes, N., Ofer, D., Peleg, Y., Rappoport, N. & Linial, M.
\n
\n ProteinBERT: A universal deep-learning model of protein sequence and function.
\n Bioinformatics (2022). https://doi.org/10.1093/bioinformatics/btac020
I'm also trying this approach and did some minimal work to prepare data for finetuning ProteinBERT. Kaggle-provided environment doesn't have enough resources, though, so you'll have to run it elsewhere.
\nSee my shared notebook in this competition (I've posted the link too many times for Kaggle's spam-limit)
", "rawMarkdown": "I'm also trying this approach and did some minimal work to prepare data for finetuning ProteinBERT. Kaggle-provided environment doesn't have enough resources, though, so you'll have to run it elsewhere.\n\nSee my shared notebook in this competition (I've posted the link too many times for Kaggle's spam-limit)", "votes": null }, { "id": "2229232", "postDate": "04/21/2023 07:14:49", "content": "Kaggle's env should easily have enough resources, it's a 16M model! People ran it in other protein competitions. Where's the bottleneck for you?
", "rawMarkdown": "Kaggle's env should easily have enough resources, it's a 16M model! People ran it in other protein competitions. Where's the bottleneck for you?", "votes": null }, { "id": "2229695", "postDate": "04/21/2023 15:56:39", "content": "Hi @danofer,
\nThanks for the link. I wonder if you have comparative data on this vs other protein embeddings. I've found one comparison that puts ProtT5 ahead of ProteinBert by ~2 percentage points for GO prediction across all of BP, MF, CC. But that comparison was done by the author of ProtT5 so perhaps it uses ProteinBert in a sub-optimal way or has other problems that make it misleading.
\nIs there anything you can point to that should lead me to question those findings or does it look right to you? (It sounds like ProtT5 is much bigger than ProteinBert, which probably makes it harder to work with. So perhaps the key advantage of ProteinBert is its small size?)
\nThanks,
\nAndrew
Cool !
\nWould it be possible to generate the embeddings for the train/test here and share them with the community in the Kaggle datasets ?
What is the average time for inference on sequences like we have here ?
", "rawMarkdown": "Cool ! \nWould it be possible to generate the embeddings for the train/test here and share them with the community in the Kaggle datasets ? \n\nWhat is the average time for inference on sequences like we have here ?", "votes": null }, { "id": "2231387", "postDate": "04/23/2023 09:01:29", "content": "They didn't compare ProteinBert there, they compared the author's own ProtBERT (which has a different architecture from us, and didn't do a GO pretraining task).
\nPS - We were very stringent in filtering out annotations in the pretraining, but that's a seperate thing :)
", "rawMarkdown": "They didn't compare ProteinBert there, they compared the author's own ProtBERT (which has a different architecture from us, and didn't do a GO pretraining task). \n\nPS - We were very stringent in filtering out annotations in the pretraining, but that's a seperate thing :)", "votes": null }, { "id": "2231515", "postDate": "04/23/2023 11:42:55", "content": "It's very fast. I'd love to share such embeddings! If you have the capacity to run them and to share as a dataset, i'd be delighted to link to them from the protein_bert repo as well!
", "rawMarkdown": "It's very fast. I'd love to share such embeddings! If you have the capacity to run them and to share as a dataset, i'd be delighted to link to them from the [protein_bert](https://github.com/nadavbra/protein_bert) repo as well!", "votes": null }, { "id": "2231527", "postDate": "04/23/2023 11:57:02", "content": "Thanks for the comment !
\nKaggle itself gives unlimited capacity to share the data - just create public dataset and put <50G there.
\nIf it is more than 50G - just create several datasets. Notebook can access any number of datasets.
\nThe only technical remark is that huge Kaggle datasets - with more than say 5G on - leads to quite long notebook initialization - when Kaggle loads the dataset to the notebook environment, so sometimes it is better to have several smaller datasets rather than one big.
Sorry, my mistake. I had assumed ProtBert = ProteinBert. Thanks for the clarification.
", "rawMarkdown": "Sorry, my mistake. I had assumed ProtBert = ProteinBert. Thanks for the clarification.", "votes": null }, { "id": "2236882", "postDate": "04/27/2023 07:00:12", "content": "Haha. It's a legit mistake. There's little in common between the two though, surprisingly ;)
", "rawMarkdown": "Haha. It's a legit mistake. There's little in common between the two though, surprisingly ;)", "votes": null }, { "id": "2239458", "postDate": "04/29/2023 14:08:29", "content": "Thanks for sending the link!
\nHas anyone given BioBERT a go on Kaggle notebooks yet?
\nI tried the protein-bert demo after installing it with pip, but it threw an error when running the finetune func.
\n\n\nAttributeError: 'Adam' object has no attribute 'get_weights'
\n
I heard the error has already been fixed, but I can't seem to install it with pip.
\nI also attempted to clone it from GitHub, but it didn't work out.
Do you have any ideas on what I should do next?
", "rawMarkdown": "Thanks for sending the link!\n\nHas anyone given BioBERT a go on Kaggle notebooks yet?\n\nI tried the [protein-bert demo ](https://github.com/nadavbra/protein_bert/blob/master/ProteinBERT%20demo.ipynb)after installing it with pip, but it threw an error when running the finetune func.\n\n>AttributeError: 'Adam' object has no attribute 'get_weights'\n\nI heard the error has [already been fixed](https://github.com/nadavbra/protein_bert/pull/44), but I can't seem to install it with pip.\nI also attempted to clone it from GitHub, but it didn't work out.\n\nDo you have any ideas on what I should do next?", "votes": null }, { "id": "2259380", "postDate": "05/15/2023 00:21:05", "content": "Could I get some help with running the inference using the pre-trained model, not sure where to start to be hones.
", "rawMarkdown": "Could I get some help with running the inference using the pre-trained model, not sure where to start to be hones.", "votes": null }, { "id": "2307093", "postDate": "06/17/2023 19:50:36", "content": "oh really thanks for sharing the github link of this great project
", "rawMarkdown": "oh really thanks for sharing the github link of this great project", "votes": null }, { "id": "2327818", "postDate": "07/03/2023 07:33:32", "content": "@danofer , hello
\nI looking for CAFA 3 4 and cannot find them..
\nCan you help me find link to CAFA 3 CAFA 4 ?
https://www.kaggle.com/datasets/alexandervc/cafa-protein-function-annotation-challenges
", "rawMarkdown": "https://www.kaggle.com/datasets/alexandervc/cafa-protein-function-annotation-challenges", "votes": null }, { "id": "2398409", "postDate": "08/19/2023 17:02:37", "content": "really cooooool
", "rawMarkdown": "really cooooool", "votes": null } ], "comments": [ { "id": 2227164, "author_name": "nhgiang", "author_url": "", "post_date": "04/19/2023 14:45:14", "content": "I'm also trying this approach and did some minimal work to prepare data for finetuning ProteinBERT. Kaggle-provided environment doesn't have enough resources, though, so you'll have to run it elsewhere.
\nSee my shared notebook in this competition (I've posted the link too many times for Kaggle's spam-limit)
", "votes": null, "replies": [ { "id": 2229232, "author_name": "danofer", "author_url": "", "post_date": "04/21/2023 07:14:49", "content": "Kaggle's env should easily have enough resources, it's a 16M model! People ran it in other protein competitions. Where's the bottleneck for you?
", "votes": null, "replies": [] } ] }, { "id": 2229695, "author_name": "andrewrrose", "author_url": "", "post_date": "04/21/2023 15:56:39", "content": "Hi @danofer,
\nThanks for the link. I wonder if you have comparative data on this vs other protein embeddings. I've found one comparison that puts ProtT5 ahead of ProteinBert by ~2 percentage points for GO prediction across all of BP, MF, CC. But that comparison was done by the author of ProtT5 so perhaps it uses ProteinBert in a sub-optimal way or has other problems that make it misleading.
\nIs there anything you can point to that should lead me to question those findings or does it look right to you? (It sounds like ProtT5 is much bigger than ProteinBert, which probably makes it harder to work with. So perhaps the key advantage of ProteinBert is its small size?)
\nThanks,
\nAndrew
They didn't compare ProteinBert there, they compared the author's own ProtBERT (which has a different architecture from us, and didn't do a GO pretraining task).
\nPS - We were very stringent in filtering out annotations in the pretraining, but that's a seperate thing :)
", "votes": null, "replies": [ { "id": 2236192, "author_name": "andrewrrose", "author_url": "", "post_date": "04/26/2023 15:52:38", "content": "Sorry, my mistake. I had assumed ProtBert = ProteinBert. Thanks for the clarification.
", "votes": null, "replies": [ { "id": 2236882, "author_name": "danofer", "author_url": "", "post_date": "04/27/2023 07:00:12", "content": "Haha. It's a legit mistake. There's little in common between the two though, surprisingly ;)
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2229759, "author_name": "alexandervc", "author_url": "", "post_date": "04/21/2023 16:49:01", "content": "Cool !
\nWould it be possible to generate the embeddings for the train/test here and share them with the community in the Kaggle datasets ?
What is the average time for inference on sequences like we have here ?
", "votes": null, "replies": [ { "id": 2231515, "author_name": "danofer", "author_url": "", "post_date": "04/23/2023 11:42:55", "content": "It's very fast. I'd love to share such embeddings! If you have the capacity to run them and to share as a dataset, i'd be delighted to link to them from the protein_bert repo as well!
", "votes": null, "replies": [ { "id": 2231527, "author_name": "alexandervc", "author_url": "", "post_date": "04/23/2023 11:57:02", "content": "Thanks for the comment !
\nKaggle itself gives unlimited capacity to share the data - just create public dataset and put <50G there.
\nIf it is more than 50G - just create several datasets. Notebook can access any number of datasets.
\nThe only technical remark is that huge Kaggle datasets - with more than say 5G on - leads to quite long notebook initialization - when Kaggle loads the dataset to the notebook environment, so sometimes it is better to have several smaller datasets rather than one big.
Thanks for sending the link!
\nHas anyone given BioBERT a go on Kaggle notebooks yet?
\nI tried the protein-bert demo after installing it with pip, but it threw an error when running the finetune func.
\n\n\nAttributeError: 'Adam' object has no attribute 'get_weights'
\n
I heard the error has already been fixed, but I can't seem to install it with pip.
\nI also attempted to clone it from GitHub, but it didn't work out.
Do you have any ideas on what I should do next?
", "votes": null, "replies": [] }, { "id": 2259380, "author_name": "abishpius", "author_url": "", "post_date": "05/15/2023 00:21:05", "content": "Could I get some help with running the inference using the pre-trained model, not sure where to start to be hones.
", "votes": null, "replies": [] }, { "id": 2307093, "author_name": "belgmohamadi", "author_url": "", "post_date": "06/17/2023 19:50:36", "content": "oh really thanks for sharing the github link of this great project
", "votes": null, "replies": [] }, { "id": 2327818, "author_name": "", "author_url": "", "post_date": "07/03/2023 07:33:32", "content": "@danofer , hello
\nI looking for CAFA 3 4 and cannot find them..
\nCan you help me find link to CAFA 3 CAFA 4 ?
https://www.kaggle.com/datasets/alexandervc/cafa-protein-function-annotation-challenges
", "votes": null, "replies": [] } ] }, { "id": 2398409, "author_name": "levijunyanzhang", "author_url": "", "post_date": "08/19/2023 17:02:37", "content": "really cooooool
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2226730": "Very relevant for this competition (we worked on it initially for CAFA 4): ProteinBert - a pretrained deep learning language model that also had GO annotations as a pretraining task. Trained on >100M proteins. Training, and preprocessing code and Model are available, and fast.\n\n\nhttps://github.com/nadavbra/protein_bert\n\n>Brandes, N., Ofer, D., Peleg, Y., Rappoport, N. & Linial, M. \nProteinBERT: A universal deep-learning model of protein sequence and function. \nBioinformatics (2022). https://doi.org/10.1093/bioinformatics/btac020", "2227164": "I'm also trying this approach and did some minimal work to prepare data for finetuning ProteinBERT. Kaggle-provided environment doesn't have enough resources, though, so you'll have to run it elsewhere.\n\nSee my shared notebook in this competition (I've posted the link too many times for Kaggle's spam-limit)", "2229232": "Kaggle's env should easily have enough resources, it's a 16M model! People ran it in other protein competitions. Where's the bottleneck for you?", "2229695": "Hi @danofer,\n\nThanks for the link. I wonder if you have comparative data on this vs other protein embeddings. I've found [one comparison](https://github.com/Rostlab/goPredSim#performance-assessment) that puts ProtT5 ahead of ProteinBert by ~2 percentage points for GO prediction across all of BP, MF, CC. But that comparison was done by the author of ProtT5 so perhaps it uses ProteinBert in a sub-optimal way or has other problems that make it misleading.\n\nIs there anything you can point to that should lead me to question those findings or does it look right to you? (It sounds like ProtT5 is much bigger than ProteinBert, which probably makes it harder to work with. So perhaps the key advantage of ProteinBert is its small size?)\n\nThanks,\nAndrew", "2229759": "Cool ! \nWould it be possible to generate the embeddings for the train/test here and share them with the community in the Kaggle datasets ? \n\nWhat is the average time for inference on sequences like we have here ?", "2231387": "They didn't compare ProteinBert there, they compared the author's own ProtBERT (which has a different architecture from us, and didn't do a GO pretraining task). \n\nPS - We were very stringent in filtering out annotations in the pretraining, but that's a seperate thing :)", "2231515": "It's very fast. I'd love to share such embeddings! If you have the capacity to run them and to share as a dataset, i'd be delighted to link to them from the [protein_bert](https://github.com/nadavbra/protein_bert) repo as well!", "2231527": "Thanks for the comment ! \nKaggle itself gives unlimited capacity to share the data - just create public dataset and put <50G there. \nIf it is more than 50G - just create several datasets. Notebook can access any number of datasets.\nThe only technical remark is that huge Kaggle datasets - with more than say 5G on - leads to quite long notebook initialization - when Kaggle loads the dataset to the notebook environment, so sometimes it is better to have several smaller datasets rather than one big.", "2236192": "Sorry, my mistake. I had assumed ProtBert = ProteinBert. Thanks for the clarification.", "2236882": "Haha. It's a legit mistake. There's little in common between the two though, surprisingly ;)", "2239458": "Thanks for sending the link!\n\nHas anyone given BioBERT a go on Kaggle notebooks yet?\n\nI tried the [protein-bert demo ](https://github.com/nadavbra/protein_bert/blob/master/ProteinBERT%20demo.ipynb)after installing it with pip, but it threw an error when running the finetune func.\n\n>AttributeError: 'Adam' object has no attribute 'get_weights'\n\nI heard the error has [already been fixed](https://github.com/nadavbra/protein_bert/pull/44), but I can't seem to install it with pip.\nI also attempted to clone it from GitHub, but it didn't work out.\n\nDo you have any ideas on what I should do next?", "2259380": "Could I get some help with running the inference using the pre-trained model, not sure where to start to be hones.", "2307093": "oh really thanks for sharing the github link of this great project", "2327818": "danofer , hello\n\nI looking for CAFA 3 4 and cannot find them..\nCan you help me find link to CAFA 3 CAFA 4 ?", "2328015": "https://www.kaggle.com/datasets/alexandervc/cafa-protein-function-annotation-challenges", "2398409": "really cooooool" }, "source": "meta" }