{
  "id": 421836,
  "title": "About the LLM embeddings",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/421836",
  "author_name": "",
  "post_date": "2023-07-07T04:09:15.556619500Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I see people get the same length-embeddings for each protein, by taking mean of the T5/ESM2 [:protein length] layers. </p>\n<p>Instead of taking mean and get equivalent length for each protein, can we take the mean of each amino acid in the protein? So that we can get protein-length embeddings. </p>\n<p>I'm asking because protein-length embeddings can help us interpret the model better, because we can know which domain in a protein leads to a specific function.</p>",
  "messages": [
    {
      "id": "2333560",
      "postDate": "07/07/2023 04:09:15",
      "content": "<p>I see people get the same length-embeddings for each protein, by taking mean of the T5/ESM2 [:protein length] layers. </p>\n<p>Instead of taking mean and get equivalent length for each protein, can we take the mean of each amino acid in the protein? So that we can get protein-length embeddings. </p>\n<p>I'm asking because protein-length embeddings can help us interpret the model better, because we can know which domain in a protein leads to a specific function.</p>",
      "rawMarkdown": "I see people get the same length-embeddings for each protein, by taking mean of the T5/ESM2 [:protein length] layers. \n\nInstead of taking mean and get equivalent length for each protein, can we take the mean of each amino acid in the protein? So that we can get protein-length embeddings. \n\nI'm asking because protein-length embeddings can help us interpret the model better, because we can know which domain in a protein leads to a specific function.",
      "votes": null
    },
    {
      "id": "2334570",
      "postDate": "07/07/2023 19:15:41",
      "content": "<p>This is because LLM were trained with some dimension-length. Most common used for trained this models is BERT, which you can use his pretrained-weights as starting point of the model, and this <a href=\"https://github.com/google-research/bert\" target=\"_blank\">weights typically are in the length(dimension)</a> that you use to see in the embeddings. <br>\nAnd the reason because the length is the same is because the model use to be trained with static size of data, there are other ways, to be flexible, such us masks, padding with zeros or templates(used by alphafold).</p>",
      "rawMarkdown": "This is because LLM were trained with some dimension-length. Most common used for trained this models is BERT, which you can use his pretrained-weights as starting point of the model, and this [weights typically are in the length(dimension)](https://github.com/google-research/bert) that you use to see in the embeddings. \nAnd the reason because the length is the same is because the model use to be trained with static size of data, there are other ways, to be flexible, such us masks, padding with zeros or templates(used by alphafold).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2334570,
      "author_name": "victorfernandezalbor",
      "author_url": "",
      "post_date": "07/07/2023 19:15:41",
      "content": "<p>This is because LLM were trained with some dimension-length. Most common used for trained this models is BERT, which you can use his pretrained-weights as starting point of the model, and this <a href=\"https://github.com/google-research/bert\" target=\"_blank\">weights typically are in the length(dimension)</a> that you use to see in the embeddings. <br>\nAnd the reason because the length is the same is because the model use to be trained with static size of data, there are other ways, to be flexible, such us masks, padding with zeros or templates(used by alphafold).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2333560": "I see people get the same length-embeddings for each protein, by taking mean of the T5/ESM2 [:protein length] layers. \n\nInstead of taking mean and get equivalent length for each protein, can we take the mean of each amino acid in the protein? So that we can get protein-length embeddings. \n\nI'm asking because protein-length embeddings can help us interpret the model better, because we can know which domain in a protein leads to a specific function.",
    "2334570": "This is because LLM were trained with some dimension-length. Most common used for trained this models is BERT, which you can use his pretrained-weights as starting point of the model, and this [weights typically are in the length(dimension)](https://github.com/google-research/bert) that you use to see in the embeddings. \nAnd the reason because the length is the same is because the model use to be trained with static size of data, there are other ways, to be flexible, such us masks, padding with zeros or templates(used by alphafold)."
  },
  "source": "meta"
}