{
  "id": 664966,
  "title": "ESM3-C benefit vs ESM2",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/664966",
  "author_name": "",
  "post_date": "2025-12-29T08:54:56.855831600Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>It seems like ESM3-C is focused mostly on parameter efficiency over ESM2. </p>\n<p>If we already can obtain pre-calculated embeddings online for ESM2 is there much point? I also noticed that it doesn't seem to appear on a lot of benchmarks (usually ESM2 is there)</p>",
  "messages": [
    {
      "id": "3383053",
      "postDate": "12/29/2025 08:54:56",
      "content": "<p>It seems like ESM3-C is focused mostly on parameter efficiency over ESM2. </p>\n<p>If we already can obtain pre-calculated embeddings online for ESM2 is there much point? I also noticed that it doesn't seem to appear on a lot of benchmarks (usually ESM2 is there)</p>",
      "rawMarkdown": "It seems like ESM3-C is focused mostly on parameter efficiency over ESM2. \n\nIf we already can obtain pre-calculated embeddings online for ESM2 is there much point? I also noticed that it doesn't seem to appear on a lot of benchmarks (usually ESM2 is there)",
      "votes": null
    },
    {
      "id": "3383071",
      "postDate": "12/29/2025 10:17:29",
      "content": "<p>In CAFA6, I wouldn’t treat <strong>“ESM3-C vs ESM2”</strong> as the thing that will make or break your results.</p>\n<p>Most CAFA pipelines do something like this: take a protein sequence → get a <strong>per-residue embedding</strong> from a PLM → compress it into <strong>one vector per protein</strong> (often with <strong>mean pooling</strong>) → run a simple predictor or retrieval step on top. Once you reduce everything to a single vector, different PLMs (and even different layers) often end up being <strong>more similar than people expect</strong> for this kind of downstream setup.</p>\n<p>Where CAFA performance usually <em>does</em> change a lot is in the <strong>pipeline details</strong>, for example:</p>\n<ul>\n<li><strong>Data leakage control</strong>: making sure very similar proteins (homologs) don’t appear in both train and test.</li>\n<li><strong>Redundancy filtering</strong>: not letting near-duplicates dominate training.</li>\n<li><strong>Score calibration / thresholding</strong>: turning model scores into GO predictions in a consistent way.</li>\n<li><strong>Rare GO terms (“long tail”)</strong>: not only predicting the few very common GO terms, but also handling the many rare ones sensibly.</li>\n<li><strong>Post-processing</strong>: enforcing GO hierarchy consistency, smoothing, etc.</li>\n</ul>\n<p>If those pieces aren’t solid yet, switching from <strong>ESM2 to ESM3-C</strong> (or to Ankh/ProST/ProtT5) usually gives <strong>small, inconsistent gains</strong> compared to fixing the fundamentals.</p>\n<p>So if you already have <strong>precomputed ESM2 embeddings</strong>, the practical advice for CAFA6 is: <strong>use them and spend your effort on building a clean, reproducible evaluation and a strong pipeline</strong>, rather than chasing a new backbone model.</p>",
      "rawMarkdown": "In CAFA6, I wouldn’t treat **“ESM3-C vs ESM2”** as the thing that will make or break your results.\n\nMost CAFA pipelines do something like this: take a protein sequence → get a **per-residue embedding** from a PLM → compress it into **one vector per protein** (often with **mean pooling**) → run a simple predictor or retrieval step on top. Once you reduce everything to a single vector, different PLMs (and even different layers) often end up being **more similar than people expect** for this kind of downstream setup.\n\nWhere CAFA performance usually *does* change a lot is in the **pipeline details**, for example:\n\n* **Data leakage control**: making sure very similar proteins (homologs) don’t appear in both train and test.\n* **Redundancy filtering**: not letting near-duplicates dominate training.\n* **Score calibration / thresholding**: turning model scores into GO predictions in a consistent way.\n* **Rare GO terms (“long tail”)**: not only predicting the few very common GO terms, but also handling the many rare ones sensibly.\n* **Post-processing**: enforcing GO hierarchy consistency, smoothing, etc.\n\nIf those pieces aren’t solid yet, switching from **ESM2 to ESM3-C** (or to Ankh/ProST/ProtT5) usually gives **small, inconsistent gains** compared to fixing the fundamentals.\n\nSo if you already have **precomputed ESM2 embeddings**, the practical advice for CAFA6 is: **use them and spend your effort on building a clean, reproducible evaluation and a strong pipeline**, rather than chasing a new backbone model.",
      "votes": null
    },
    {
      "id": "3383200",
      "postDate": "12/29/2025 16:12:25",
      "content": "<p>Helpful. Thanks.</p>",
      "rawMarkdown": "Helpful. Thanks.",
      "votes": null
    },
    {
      "id": "3388730",
      "postDate": "01/09/2026 13:34:56",
      "content": "<p>thanks, I didn't know this.</p>",
      "rawMarkdown": "thanks, I didn't know this.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3383071,
      "author_name": "xaxipirulazo",
      "author_url": "",
      "post_date": "12/29/2025 10:17:29",
      "content": "<p>In CAFA6, I wouldn’t treat <strong>“ESM3-C vs ESM2”</strong> as the thing that will make or break your results.</p>\n<p>Most CAFA pipelines do something like this: take a protein sequence → get a <strong>per-residue embedding</strong> from a PLM → compress it into <strong>one vector per protein</strong> (often with <strong>mean pooling</strong>) → run a simple predictor or retrieval step on top. Once you reduce everything to a single vector, different PLMs (and even different layers) often end up being <strong>more similar than people expect</strong> for this kind of downstream setup.</p>\n<p>Where CAFA performance usually <em>does</em> change a lot is in the <strong>pipeline details</strong>, for example:</p>\n<ul>\n<li><strong>Data leakage control</strong>: making sure very similar proteins (homologs) don’t appear in both train and test.</li>\n<li><strong>Redundancy filtering</strong>: not letting near-duplicates dominate training.</li>\n<li><strong>Score calibration / thresholding</strong>: turning model scores into GO predictions in a consistent way.</li>\n<li><strong>Rare GO terms (“long tail”)</strong>: not only predicting the few very common GO terms, but also handling the many rare ones sensibly.</li>\n<li><strong>Post-processing</strong>: enforcing GO hierarchy consistency, smoothing, etc.</li>\n</ul>\n<p>If those pieces aren’t solid yet, switching from <strong>ESM2 to ESM3-C</strong> (or to Ankh/ProST/ProtT5) usually gives <strong>small, inconsistent gains</strong> compared to fixing the fundamentals.</p>\n<p>So if you already have <strong>precomputed ESM2 embeddings</strong>, the practical advice for CAFA6 is: <strong>use them and spend your effort on building a clean, reproducible evaluation and a strong pipeline</strong>, rather than chasing a new backbone model.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3383200,
          "author_name": "glennkroegel",
          "author_url": "",
          "post_date": "12/29/2025 16:12:25",
          "content": "<p>Helpful. Thanks.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3388730,
      "author_name": "uttarashah",
      "author_url": "",
      "post_date": "01/09/2026 13:34:56",
      "content": "<p>thanks, I didn't know this.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3383053": "It seems like ESM3-C is focused mostly on parameter efficiency over ESM2. \n\nIf we already can obtain pre-calculated embeddings online for ESM2 is there much point? I also noticed that it doesn't seem to appear on a lot of benchmarks (usually ESM2 is there)",
    "3383071": "In CAFA6, I wouldn’t treat **“ESM3-C vs ESM2”** as the thing that will make or break your results.\n\nMost CAFA pipelines do something like this: take a protein sequence → get a **per-residue embedding** from a PLM → compress it into **one vector per protein** (often with **mean pooling**) → run a simple predictor or retrieval step on top. Once you reduce everything to a single vector, different PLMs (and even different layers) often end up being **more similar than people expect** for this kind of downstream setup.\n\nWhere CAFA performance usually *does* change a lot is in the **pipeline details**, for example:\n\n* **Data leakage control**: making sure very similar proteins (homologs) don’t appear in both train and test.\n* **Redundancy filtering**: not letting near-duplicates dominate training.\n* **Score calibration / thresholding**: turning model scores into GO predictions in a consistent way.\n* **Rare GO terms (“long tail”)**: not only predicting the few very common GO terms, but also handling the many rare ones sensibly.\n* **Post-processing**: enforcing GO hierarchy consistency, smoothing, etc.\n\nIf those pieces aren’t solid yet, switching from **ESM2 to ESM3-C** (or to Ankh/ProST/ProtT5) usually gives **small, inconsistent gains** compared to fixing the fundamentals.\n\nSo if you already have **precomputed ESM2 embeddings**, the practical advice for CAFA6 is: **use them and spend your effort on building a clean, reproducible evaluation and a strong pipeline**, rather than chasing a new backbone model.",
    "3383200": "Helpful. Thanks.",
    "3388730": "thanks, I didn't know this."
  },
  "source": "meta"
}