{
  "id": 409952,
  "title": "How to Select the Best Protein Embedding Features in T5 and Other Feature Extraction Transformers?",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/409952",
  "author_name": "",
  "post_date": "2023-05-13T10:18:15.432387400Z",
  "votes": 5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello everyone,😊<br>\nI found that the accuracy of the model was stuck at a certain point when I used depth learning model to train the T5 embedding feature data of proteins. I tested my own transformer architecture and simple Conv1d model with the linear model and found that the effect was the same; so I want to know if the problem is with the embedding feature data? Or how should I improve it next? Thanks for your help!! At the same time, I also hope to get some more detailed content about protein embedding. Finally, let's have a happy kaggling together!🎉🎉</p>",
  "messages": [
    {
      "id": "2257397",
      "postDate": "05/13/2023 10:18:15",
      "content": "<p>Hello everyone,😊<br>\nI found that the accuracy of the model was stuck at a certain point when I used depth learning model to train the T5 embedding feature data of proteins. I tested my own transformer architecture and simple Conv1d model with the linear model and found that the effect was the same; so I want to know if the problem is with the embedding feature data? Or how should I improve it next? Thanks for your help!! At the same time, I also hope to get some more detailed content about protein embedding. Finally, let's have a happy kaggling together!🎉🎉</p>",
      "rawMarkdown": "Hello everyone,😊\nI found that the accuracy of the model was stuck at a certain point when I used depth learning model to train the T5 embedding feature data of proteins. I tested my own transformer architecture and simple Conv1d model with the linear model and found that the effect was the same; so I want to know if the problem is with the embedding feature data? Or how should I improve it next? Thanks for your help!! At the same time, I also hope to get some more detailed content about protein embedding. Finally, let's have a happy kaggling together!🎉🎉",
      "votes": null
    },
    {
      "id": "2259692",
      "postDate": "05/15/2023 07:10:51",
      "content": "<p>I think there are several reasons why performance might plateau/get stuck: </p>\n<ul>\n<li><strong>GO term inbalance</strong>: there are many labels and the target space is sparse. Therefore, you might need of ways to balance the set or reweight specific labels during training to avoid overfitting the data. </li>\n<li><strong>Label correlation</strong>: GO terms are highly correlated in specific cases among themselves. Therefore, exploring solutions that incorporate these similarities might boost performance.</li>\n<li><strong>Fine-tuning T5 to your own needs</strong>: T5 is pre-trained on a specific set of aminoacids chains. You might want to unfreeze a subset of layers of the model to make it more flexible for your needs</li>\n</ul>\n<p>Hope this helps!</p>",
      "rawMarkdown": "I think there are several reasons why performance might plateau/get stuck: \n- **GO term inbalance**: there are many labels and the target space is sparse. Therefore, you might need of ways to balance the set or reweight specific labels during training to avoid overfitting the data. \n- **Label correlation**: GO terms are highly correlated in specific cases among themselves. Therefore, exploring solutions that incorporate these similarities might boost performance.\n- **Fine-tuning T5 to your own needs**: T5 is pre-trained on a specific set of aminoacids chains. You might want to unfreeze a subset of layers of the model to make it more flexible for your needs\n\nHope this helps!",
      "votes": null
    },
    {
      "id": "2260193",
      "postDate": "05/15/2023 14:13:14",
      "content": "<p>Thx U ! this has been very useful to me!🙏</p>",
      "rawMarkdown": "Thx U ! this has been very useful to me!🙏",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2259692,
      "author_name": "wadgek",
      "author_url": "",
      "post_date": "05/15/2023 07:10:51",
      "content": "<p>I think there are several reasons why performance might plateau/get stuck: </p>\n<ul>\n<li><strong>GO term inbalance</strong>: there are many labels and the target space is sparse. Therefore, you might need of ways to balance the set or reweight specific labels during training to avoid overfitting the data. </li>\n<li><strong>Label correlation</strong>: GO terms are highly correlated in specific cases among themselves. Therefore, exploring solutions that incorporate these similarities might boost performance.</li>\n<li><strong>Fine-tuning T5 to your own needs</strong>: T5 is pre-trained on a specific set of aminoacids chains. You might want to unfreeze a subset of layers of the model to make it more flexible for your needs</li>\n</ul>\n<p>Hope this helps!</p>",
      "votes": null,
      "replies": [
        {
          "id": 2260193,
          "author_name": "lau01b",
          "author_url": "",
          "post_date": "05/15/2023 14:13:14",
          "content": "<p>Thx U ! this has been very useful to me!🙏</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2257397": "Hello everyone,😊\nI found that the accuracy of the model was stuck at a certain point when I used depth learning model to train the T5 embedding feature data of proteins. I tested my own transformer architecture and simple Conv1d model with the linear model and found that the effect was the same; so I want to know if the problem is with the embedding feature data? Or how should I improve it next? Thanks for your help!! At the same time, I also hope to get some more detailed content about protein embedding. Finally, let's have a happy kaggling together!🎉🎉",
    "2259692": "I think there are several reasons why performance might plateau/get stuck: \n- **GO term inbalance**: there are many labels and the target space is sparse. Therefore, you might need of ways to balance the set or reweight specific labels during training to avoid overfitting the data. \n- **Label correlation**: GO terms are highly correlated in specific cases among themselves. Therefore, exploring solutions that incorporate these similarities might boost performance.\n- **Fine-tuning T5 to your own needs**: T5 is pre-trained on a specific set of aminoacids chains. You might want to unfreeze a subset of layers of the model to make it more flexible for your needs\n\nHope this helps!",
    "2260193": "Thx U ! this has been very useful to me!🙏"
  },
  "source": "meta"
}