We are likely to use a protein language model here, like ESM or Ankh. If you'd like to learn more about protein language models, or maybe train one from scratch yourself, I published a blog and code describing the process on W&B Fully Connected.
", "messages": [ { "id": 2226364, "postDate": "2023-04-18T21:36:01.887Z", "content": "We are likely to use a protein language model here, like ESM or Ankh. If you'd like to learn more about protein language models, or maybe train one from scratch yourself, I published a blog and code describing the process on W&B Fully Connected.
", "rawMarkdown": "We are likely to use a protein language model here, like [ESM](https://github.com/facebookresearch/esm) or [Ankh](https://www.proteinea.com/our-innovations/ankh-pioneering-language-models). If you'd like to learn more about protein language models, or maybe train one from scratch yourself, I published a blog and code describing the process on [W&B Fully Connected](https://wandb.ai/darek/protobert/reports/Training-a-Protein-Language-Model-on-Cloud-TPU--VmlldzoyNzg2NTgz). ", "votes": 41 }, { "id": 2230303, "postDate": "2023-04-22T08:03:05.483Z", "content": "Cool ! Thanks for sharing !
\nSome colleagues say that ESM-2 is the most powerful model. But inference for our lengths may take about 1 minute.
\nDoes it correspond to your experience?
We have 280 000 proteins, if inference is 1 minute , that is unreal for most of us.
", "rawMarkdown": "Cool ! Thanks for sharing !\n\nSome colleagues say that ESM-2 is the most powerful model. But inference for our lengths may take about 1 minute.\nDoes it correspond to your experience?\n\nWe have 280 000 proteins, if inference is 1 minute , that is unreal for most of us.", "votes": 2, "replies": [ { "id": 2230906, "postDate": "2023-04-22T21:51:23.390Z", "content": "I think ESM-2 is trained in multiple sizes, from small to large parameter size, might be worth starting with the small and then scale up… even the larger models should be significantly faster than 1 min with sufficiently big GPU I think, but haven't tried myself yet.
", "rawMarkdown": "I think ESM-2 is trained in multiple sizes, from small to large parameter size, might be worth starting with the small and then scale up... even the larger models should be significantly faster than 1 min with sufficiently big GPU I think, but haven't tried myself yet. ", "votes": 2, "replies": [ { "id": 2269100, "postDate": "2023-05-22T08:08:17.983Z", "content": "How did you run ESM-2 models on longer amino acid sequences? Did you truncate the sequence of tokens? Even the smallest ESM-2 model can't scale up cross-attention for very long proteins.
", "rawMarkdown": "How did you run ESM-2 models on longer amino acid sequences? Did you truncate the sequence of tokens? Even the smallest ESM-2 model can't scale up cross-attention for very long proteins." }, { "id": 2361243, "postDate": "2023-07-27T09:37:01.497Z", "content": "I truncated. Tis annoying
", "rawMarkdown": "I truncated. Tis annoying", "votes": 1 } ] } ] }, { "id": 2401281, "postDate": "2023-08-21T14:40:56.200Z", "content": "I'm curious to know how others did with ESM-2.
", "rawMarkdown": "I'm curious to know how others did with ESM-2. " }, { "id": 2307094, "postDate": "2023-06-17T19:51:44.290Z", "content": "I have studied using NLP for predicting protein structure in different papers …but i am really enthusiastic to know how we can do it practically?
", "rawMarkdown": "I have studied using NLP for predicting protein structure in different papers ...but i am really enthusiastic to know how we can do it practically?" }, { "id": 2309470, "postDate": "2023-06-19T16:50:48.217Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2230303, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-04-22T08:03:05.483000", "content": "Cool ! Thanks for sharing !
\nSome colleagues say that ESM-2 is the most powerful model. But inference for our lengths may take about 1 minute.
\nDoes it correspond to your experience?
We have 280 000 proteins, if inference is 1 minute , that is unreal for most of us.
", "votes": 2, "replies": [ { "id": 2230906, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2023-04-22T21:51:23.390000", "content": "I think ESM-2 is trained in multiple sizes, from small to large parameter size, might be worth starting with the small and then scale up… even the larger models should be significantly faster than 1 min with sufficiently big GPU I think, but haven't tried myself yet.
", "votes": 2, "replies": [ { "id": 2269100, "author_name": "Kaushalya Madhawa", "author_url": "", "post_date": "2023-05-22T08:08:17.983000", "content": "How did you run ESM-2 models on longer amino acid sequences? Did you truncate the sequence of tokens? Even the smallest ESM-2 model can't scale up cross-attention for very long proteins.
", "votes": 0, "replies": [] }, { "id": 2361243, "author_name": "Dan Ofer", "author_url": "", "post_date": "2023-07-27T09:37:01.497000", "content": "I truncated. Tis annoying
", "votes": 1, "replies": [] } ] } ] }, { "id": 2401281, "author_name": "Amelie Schreiber", "author_url": "", "post_date": "2023-08-21T14:40:56.200000", "content": "I'm curious to know how others did with ESM-2.
", "votes": 0, "replies": [] }, { "id": 2307094, "author_name": "Maryam Mohammadi", "author_url": "", "post_date": "2023-06-17T19:51:44.290000", "content": "I have studied using NLP for predicting protein structure in different papers …but i am really enthusiastic to know how we can do it practically?
", "votes": 0, "replies": [] }, { "id": 2309470, "author_name": "", "author_url": "", "post_date": "2023-06-19T16:50:48.217000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2226364": "We are likely to use a protein language model here, like [ESM](https://github.com/facebookresearch/esm) or [Ankh](https://www.proteinea.com/our-innovations/ankh-pioneering-language-models). If you'd like to learn more about protein language models, or maybe train one from scratch yourself, I published a blog and code describing the process on [W&B Fully Connected](https://wandb.ai/darek/protobert/reports/Training-a-Protein-Language-Model-on-Cloud-TPU--VmlldzoyNzg2NTgz). ", "2230303": "Cool ! Thanks for sharing !\n\nSome colleagues say that ESM-2 is the most powerful model. But inference for our lengths may take about 1 minute.\nDoes it correspond to your experience?\n\nWe have 280 000 proteins, if inference is 1 minute , that is unreal for most of us.", "2401281": "I'm curious to know how others did with ESM-2. ", "2307094": "I have studied using NLP for predicting protein structure in different papers ...but i am really enthusiastic to know how we can do it practically?", "2309470": "" } }