{
  "id": 462419,
  "title": "Share: ESM2 large (dim 5120) and ankh embeddings",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/462419",
  "author_name": "Alexander Chervov",
  "post_date": "2023-12-19T20:40:05.828000",
  "votes": 8,
  "comment_count": 6,
  "views": 0,
  "content": "<p><a href=\"https://www.kaggle.com/datasets/alexandervc/protein-embeddings-3-cafa5\" target=\"_blank\">https://www.kaggle.com/datasets/alexandervc/protein-embeddings-3-cafa5</a></p>\n<p>Thanks to Sergei Fironov.</p>\n<p>It is quite difficult even to infer esm2-large model, it required lots of skills,<br>\nunfortunately, these embeds did not show up good performance,<br>\nmost models overfit with them, only some rare cases were not so bad.<br>\nBut quite far from final models.  </p>\n<p>Ankh embeds even worse. </p>\n<p>But let us share them, may be someone will have more luck with them.</p>",
  "messages": [
    {
      "id": 2567665,
      "postDate": "2023-12-19T20:40:05.827Z",
      "content": "<p><a href=\"https://www.kaggle.com/datasets/alexandervc/protein-embeddings-3-cafa5\" target=\"_blank\">https://www.kaggle.com/datasets/alexandervc/protein-embeddings-3-cafa5</a></p>\n<p>Thanks to Sergei Fironov.</p>\n<p>It is quite difficult even to infer esm2-large model, it required lots of skills,<br>\nunfortunately, these embeds did not show up good performance,<br>\nmost models overfit with them, only some rare cases were not so bad.<br>\nBut quite far from final models.  </p>\n<p>Ankh embeds even worse. </p>\n<p>But let us share them, may be someone will have more luck with them.</p>",
      "rawMarkdown": "https://www.kaggle.com/datasets/alexandervc/protein-embeddings-3-cafa5\n\nThanks to Sergei Fironov.\n\nIt is quite difficult even to infer esm2-large model, it required lots of skills,\nunfortunately, these embeds did not show up good performance,\nmost models overfit with them, only some rare cases were not so bad.\nBut quite far from final models.  \n\nAnkh embeds even worse. \n\nBut let us share them, may be someone will have more luck with them.\n\n",
      "votes": 8
    },
    {
      "id": 2568019,
      "postDate": "2023-12-20T07:31:02.900Z",
      "content": "<blockquote>\n  <p>it required lot of skills</p>\n</blockquote>\n<p>I have found it requires very few additional skills compared to smaller models, but GPU memory amount is crucial.<br>\nThis code works on single RTX A6000 (it was rent on vast.ai): <a href=\"https://www.kaggle.com/code/ogurtsov/get-esm2-15b-embeds\" target=\"_blank\">https://www.kaggle.com/code/ogurtsov/get-esm2-15b-embeds</a><br>\nI got 20 000 embeds and also found these embeds work worse than ESM-2 3B.<br>\nBy the way, CPU offloading script from ESM-2 team runs OOM error even with crazy amount of RAM such as 256Gb.</p>",
      "rawMarkdown": ">it required lot of skills\n\nI have found it requires very few additional skills compared to smaller models, but GPU memory amount is crucial.\nThis code works on single RTX A6000 (it was rent on vast.ai): https://www.kaggle.com/code/ogurtsov/get-esm2-15b-embeds\nI got 20 000 embeds and also found these embeds work worse than ESM-2 3B.\nBy the way, CPU offloading script from ESM-2 team runs OOM error even with crazy amount of RAM such as 256Gb.",
      "votes": 4
    },
    {
      "id": 2568683,
      "postDate": "2023-12-20T18:55:51.640Z",
      "content": "<p>Congrats to the good final position :)</p>",
      "rawMarkdown": "Congrats to the good final position :)",
      "votes": 2,
      "replies": [
        {
          "id": 2569219,
          "postDate": "2023-12-21T06:36:26.233Z",
          "content": "<p>Thanks for kind words ! <br>\nAnd thanks for sharing your notebooks and ideas - was very helpful ! </p>",
          "rawMarkdown": "Thanks for kind words ! \nAnd thanks for sharing your notebooks and ideas - was very helpful ! ",
          "votes": 1
        }
      ]
    },
    {
      "id": 2568670,
      "postDate": "2023-12-20T18:35:50.263Z",
      "content": "<p>Great job and congratulations to your team! All of you shared so much throughout the competition and it is nice to see you get rewarded.</p>",
      "rawMarkdown": "Great job and congratulations to your team! All of you shared so much throughout the competition and it is nice to see you get rewarded.",
      "votes": 2,
      "replies": [
        {
          "id": 2569220,
          "postDate": "2023-12-21T06:37:29.097Z",
          "content": "<p>Thanks a lot for kind words, to hear them from expert in a field like you is great honor  !</p>",
          "rawMarkdown": "Thanks a lot for kind words, to hear them from expert in a field like you is great honor  !",
          "votes": 1
        }
      ]
    },
    {
      "id": 3312451,
      "postDate": "2025-11-07T07:49:48.917Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2568019,
      "author_name": "Ogurtsov",
      "author_url": "",
      "post_date": "2023-12-20T07:31:02.900000",
      "content": "<blockquote>\n  <p>it required lot of skills</p>\n</blockquote>\n<p>I have found it requires very few additional skills compared to smaller models, but GPU memory amount is crucial.<br>\nThis code works on single RTX A6000 (it was rent on vast.ai): <a href=\"https://www.kaggle.com/code/ogurtsov/get-esm2-15b-embeds\" target=\"_blank\">https://www.kaggle.com/code/ogurtsov/get-esm2-15b-embeds</a><br>\nI got 20 000 embeds and also found these embeds work worse than ESM-2 3B.<br>\nBy the way, CPU offloading script from ESM-2 team runs OOM error even with crazy amount of RAM such as 256Gb.</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2568683,
      "author_name": "Simon Veitner",
      "author_url": "",
      "post_date": "2023-12-20T18:55:51.640000",
      "content": "<p>Congrats to the good final position :)</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2569219,
          "author_name": "Alexander Chervov",
          "author_url": "",
          "post_date": "2023-12-21T06:36:26.233000",
          "content": "<p>Thanks for kind words ! <br>\nAnd thanks for sharing your notebooks and ideas - was very helpful ! </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2568670,
      "author_name": "Tilii",
      "author_url": "",
      "post_date": "2023-12-20T18:35:50.263000",
      "content": "<p>Great job and congratulations to your team! All of you shared so much throughout the competition and it is nice to see you get rewarded.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2569220,
          "author_name": "Alexander Chervov",
          "author_url": "",
          "post_date": "2023-12-21T06:37:29.097000",
          "content": "<p>Thanks a lot for kind words, to hear them from expert in a field like you is great honor  !</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3312451,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-11-07T07:49:48.917000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2567665": "https://www.kaggle.com/datasets/alexandervc/protein-embeddings-3-cafa5\n\nThanks to Sergei Fironov.\n\nIt is quite difficult even to infer esm2-large model, it required lots of skills,\nunfortunately, these embeds did not show up good performance,\nmost models overfit with them, only some rare cases were not so bad.\nBut quite far from final models.  \n\nAnkh embeds even worse. \n\nBut let us share them, may be someone will have more luck with them.\n\n",
    "2568019": ">it required lot of skills\n\nI have found it requires very few additional skills compared to smaller models, but GPU memory amount is crucial.\nThis code works on single RTX A6000 (it was rent on vast.ai): https://www.kaggle.com/code/ogurtsov/get-esm2-15b-embeds\nI got 20 000 embeds and also found these embeds work worse than ESM-2 3B.\nBy the way, CPU offloading script from ESM-2 team runs OOM error even with crazy amount of RAM such as 256Gb.",
    "2568683": "Congrats to the good final position :)",
    "2568670": "Great job and congratulations to your team! All of you shared so much throughout the competition and it is nice to see you get rewarded.",
    "3312451": ""
  }
}