{ "id": 406168, "title": "ESM2 final-layer embeddings", "url": "/competitions/cafa-5-protein-function-prediction/discussion/406168", "author_name": "", "post_date": "2023-05-01T07:22:19.583999900Z", "votes": 47, "comment_count": 5, "views": 0, "content": "

Hello kagglers! I want to share with you final-layer embeddings from multiple ESM2 models.

\n

ESM - Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR) link to models family github

\n

There are multiple BERT like encoders in this familiy with different layers\\ params number.

\n

Here there are links to final-layer embeddings from 5 ESM variants:
\nesm2_t6_8M, embeding size = 320
\nesm2_t12_35M,embeding size = 480
\nesm2_t30_150M, embeding size = 640
\nesm2_t33_650M, embeding size = 1280
\nesm2_t36_3B, embeding size = 2560

\n

NOTE: ESM encoders requare sequences with max length = 1024. For these datasets I had to cut all sequences to length 1022 (+ cls, eos == 1024).
\nNotebook for dataset creation is here, but it can't be run on kaggle

\n

I wish you good luck in the competition!

\n

Update 29.05.2023: Corrected the shape of test embedings in all datasets (Fisrt version is also correct but has additional zero rows, that should be cut)

", "messages": [ { "id": "2241084", "postDate": "05/01/2023 07:22:19", "content": "

Hello kagglers! I want to share with you final-layer embeddings from multiple ESM2 models.

\n

ESM - Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR) link to models family github

\n

There are multiple BERT like encoders in this familiy with different layers\\ params number.

\n

Here there are links to final-layer embeddings from 5 ESM variants:
\nesm2_t6_8M, embeding size = 320
\nesm2_t12_35M,embeding size = 480
\nesm2_t30_150M, embeding size = 640
\nesm2_t33_650M, embeding size = 1280
\nesm2_t36_3B, embeding size = 2560

\n

NOTE: ESM encoders requare sequences with max length = 1024. For these datasets I had to cut all sequences to length 1022 (+ cls, eos == 1024).
\nNotebook for dataset creation is here, but it can't be run on kaggle

\n

I wish you good luck in the competition!

\n

Update 29.05.2023: Corrected the shape of test embedings in all datasets (Fisrt version is also correct but has additional zero rows, that should be cut)

", "rawMarkdown": "Hello kagglers! I want to share with you final-layer embeddings from multiple ESM2 models. \n\nESM - Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR) [link to models family github](https://github.com/facebookresearch/esm?ysclid=lh4h7w4j2b979516370)\n\nThere are multiple BERT like encoders in this familiy with different layers\\ params number.\n\nHere there are links to final-layer embeddings from 5 ESM variants:\n[esm2_t6_8M](https://www.kaggle.com/datasets/andreylalaley/315701375), embeding size = 320\n[esm2_t12_35M](https://www.kaggle.com/datasets/andreylalaley/3023750),embeding size = 480 \n[esm2_t30_150M](https://www.kaggle.com/datasets/andreylalaley/8947923), embeding size = 640\n[esm2_t33_650M](https://www.kaggle.com/datasets/andreylalaley/23468234), embeding size = 1280\n[esm2_t36_3B](https://www.kaggle.com/datasets/andreylalaley/4637427), embeding size = 2560\n\nNOTE: ESM encoders requare sequences with max length = 1024. For these datasets I had to cut all sequences to length 1022 (+ cls, eos == 1024).\nNotebook for dataset creation is [here](https://www.kaggle.com/code/andreylalaley/esm-embeds-calculation), but it can't be run on kaggle\n\nI wish you good luck in the competition!\n\nUpdate 29.05.2023: Corrected the shape of test embedings in all datasets (Fisrt version is also correct but has additional zero rows, that should be cut)", "votes": null }, { "id": "2246037", "postDate": "05/04/2023 19:12:30", "content": "

Great job! I have created a similar notebook that runs on Kaggle.

", "rawMarkdown": "Great job! I have created a similar [notebook](https://www.kaggle.com/code/viktorfairuschin/extracting-esm-2-embeddings-from-fasta-files) that runs on Kaggle.", "votes": null }, { "id": "2271422", "postDate": "05/23/2023 22:03:57", "content": "

Hello! First of all, thank you very much for sharing this dataset, it is very helpful for those of us who do not have much computing power.

\n

I've tried using your embedding esm2_t36_3B, but got a shape error:

\n

test_ids.py has a size of (141865,) while test_embeddingsis (142246, 2560).

\n

I understand that they should share the number of rows. Any tips to fix this problem?

\n

All the best!

", "rawMarkdown": "Hello! First of all, thank you very much for sharing this dataset, it is very helpful for those of us who do not have much computing power.\n\nI've tried using your embedding ***esm2_t36_3B***, but got a shape error:\n\n`test_ids.py` has a size of `(141865,)` while `test_embeddings `is `(142246, 2560)`.\n\nI understand that they should share the number of rows. Any tips to fix this problem?\n\nAll the best!", "votes": null }, { "id": "2276780", "postDate": "05/27/2023 07:49:01", "content": "

Hello! Thank you for your comment, it is my fault during embedings creation. In embeding creation script i initially create zero matrix with needed shape and then redefine rows in loop. I created zero matrix for test with the same shape as the matrix of train. So rows from 141865 to 142246 are all zero and you can freely cut them. I will also make and update with these changes, shortly

", "rawMarkdown": "Hello! Thank you for your comment, it is my fault during embedings creation. In embeding creation script i initially create zero matrix with needed shape and then redefine rows in loop. I created zero matrix for test with the same shape as the matrix of train. So rows from 141865 to 142246 are all zero and you can freely cut them. I will also make and update with these changes, shortly", "votes": null }, { "id": "2296265", "postDate": "06/11/2023 17:19:49", "content": "

Thank you for the dataset ! We are still struggling to run our own embedding and this is gold.

", "rawMarkdown": "Thank you for the dataset ! We are still struggling to run our own embedding and this is gold.", "votes": null }, { "id": "2351169", "postDate": "07/19/2023 21:47:56", "content": "

Nice! thanks for sharing! these embeddings can take considerable time / compute to create.

", "rawMarkdown": "Nice! thanks for sharing! these embeddings can take considerable time / compute to create.", "votes": null } ], "comments": [ { "id": 2246037, "author_name": "viktorfairuschin", "author_url": "", "post_date": "05/04/2023 19:12:30", "content": "

Great job! I have created a similar notebook that runs on Kaggle.

", "votes": null, "replies": [] }, { "id": 2271422, "author_name": "ancaco12", "author_url": "", "post_date": "05/23/2023 22:03:57", "content": "

Hello! First of all, thank you very much for sharing this dataset, it is very helpful for those of us who do not have much computing power.

\n

I've tried using your embedding esm2_t36_3B, but got a shape error:

\n

test_ids.py has a size of (141865,) while test_embeddingsis (142246, 2560).

\n

I understand that they should share the number of rows. Any tips to fix this problem?

\n

All the best!

", "votes": null, "replies": [ { "id": 2276780, "author_name": "andreylalaley", "author_url": "", "post_date": "05/27/2023 07:49:01", "content": "

Hello! Thank you for your comment, it is my fault during embedings creation. In embeding creation script i initially create zero matrix with needed shape and then redefine rows in loop. I created zero matrix for test with the same shape as the matrix of train. So rows from 141865 to 142246 are all zero and you can freely cut them. I will also make and update with these changes, shortly

", "votes": null, "replies": [] } ] }, { "id": 2296265, "author_name": "alumine", "author_url": "", "post_date": "06/11/2023 17:19:49", "content": "

Thank you for the dataset ! We are still struggling to run our own embedding and this is gold.

", "votes": null, "replies": [] }, { "id": 2351169, "author_name": "jlparkinson1", "author_url": "", "post_date": "07/19/2023 21:47:56", "content": "

Nice! thanks for sharing! these embeddings can take considerable time / compute to create.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2241084": "Hello kagglers! I want to share with you final-layer embeddings from multiple ESM2 models. \n\nESM - Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR) [link to models family github](https://github.com/facebookresearch/esm?ysclid=lh4h7w4j2b979516370)\n\nThere are multiple BERT like encoders in this familiy with different layers\\ params number.\n\nHere there are links to final-layer embeddings from 5 ESM variants:\n[esm2_t6_8M](https://www.kaggle.com/datasets/andreylalaley/315701375), embeding size = 320\n[esm2_t12_35M](https://www.kaggle.com/datasets/andreylalaley/3023750),embeding size = 480 \n[esm2_t30_150M](https://www.kaggle.com/datasets/andreylalaley/8947923), embeding size = 640\n[esm2_t33_650M](https://www.kaggle.com/datasets/andreylalaley/23468234), embeding size = 1280\n[esm2_t36_3B](https://www.kaggle.com/datasets/andreylalaley/4637427), embeding size = 2560\n\nNOTE: ESM encoders requare sequences with max length = 1024. For these datasets I had to cut all sequences to length 1022 (+ cls, eos == 1024).\nNotebook for dataset creation is [here](https://www.kaggle.com/code/andreylalaley/esm-embeds-calculation), but it can't be run on kaggle\n\nI wish you good luck in the competition!\n\nUpdate 29.05.2023: Corrected the shape of test embedings in all datasets (Fisrt version is also correct but has additional zero rows, that should be cut)", "2246037": "Great job! I have created a similar [notebook](https://www.kaggle.com/code/viktorfairuschin/extracting-esm-2-embeddings-from-fasta-files) that runs on Kaggle.", "2271422": "Hello! First of all, thank you very much for sharing this dataset, it is very helpful for those of us who do not have much computing power.\n\nI've tried using your embedding ***esm2_t36_3B***, but got a shape error:\n\n`test_ids.py` has a size of `(141865,)` while `test_embeddings `is `(142246, 2560)`.\n\nI understand that they should share the number of rows. Any tips to fix this problem?\n\nAll the best!", "2276780": "Hello! Thank you for your comment, it is my fault during embedings creation. In embeding creation script i initially create zero matrix with needed shape and then redefine rows in loop. I created zero matrix for test with the same shape as the matrix of train. So rows from 141865 to 142246 are all zero and you can freely cut them. I will also make and update with these changes, shortly", "2296265": "Thank you for the dataset ! We are still struggling to run our own embedding and this is gold.", "2351169": "Nice! thanks for sharing! these embeddings can take considerable time / compute to create." }, "source": "meta" }