\n
![\"overview\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F548996%2Fbf4221217029bc9d2ae30b78bf0d577d%2F.png?generation=1703936744527010&alt=media\")
Thanks to Kaggle and hosts for organising such an interesting competition.
\nThis competition task:\"Predict the function of this protein using any available data\" was linked to the fundamental of biology, making it a very exciting experience for me.
\nThanks also to the community, the discussions and notebooks were very helpful.
I used the following NN models.
\n
I have extracted embeddings using protein language models from a protein sequence.
\nThe models I used include T5, ESM2t36, ESM2t48, and combinations of these.
I one-hot encoded the taxonomy data and combined it with embeddings generated by protein language models.
\nDuring this process, only the 90 taxonomy IDs included in the test data were used.
The evidence codes of UniProt GOA annotations consist of 13 experimental labels and 11 non-experimental labels.
\nExperimental labels are used as GT and non-experimental labels are not used in training data.
\nSo I extracted non-experimental labels from UniProt GOA data and used it as additional features.
\nThe non-experimental labels were one-hot encoded and transformed into a tensor of dimensions (batch size, 11, number of GT labels).
\nThen this tensor is processed using a 1D-CNN which kernel_size is 1.
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F548996%2F86368def7cbc9707ec8c8fb5c77f6217%2Fnon-exp-feat.png?generation=1704986539311314&alt=media\")
The training data was created from experimental annotations in UniProtKB as of November 17, 2022, and the test data (private leaderboard) consists of data experimentally verified after August 22, 2023.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F548996%2F6ba93a1b07cbdcdd3004f88a8ebb7952%2Fvalidation.png?generation=1703936790356554&alt=media\")
\nTherefore, I approached the validation strategy as if it were time-series data.
\nValidation data was created by extracting experimental annotations from the latest UniProtKB data at the time of the competition, and any sub-ontology data which wrere annotated in the training data was excluded from the validation data.
Thanks to Kaggle and hosts for organising such an interesting competition.
\nThis competition task:\"Predict the function of this protein using any available data\" was linked to the fundamental of biology, making it a very exciting experience for me.
\nThanks also to the community, the discussions and notebooks were very helpful.
I used the following NN models.
\n
I have extracted embeddings using protein language models from a protein sequence.
\nThe models I used include T5, ESM2t36, ESM2t48, and combinations of these.
I one-hot encoded the taxonomy data and combined it with embeddings generated by protein language models.
\nDuring this process, only the 90 taxonomy IDs included in the test data were used.
The evidence codes of UniProt GOA annotations consist of 13 experimental labels and 11 non-experimental labels.
\nExperimental labels are used as GT and non-experimental labels are not used in training data.
\nSo I extracted non-experimental labels from UniProt GOA data and used it as additional features.
\nThe non-experimental labels were one-hot encoded and transformed into a tensor of dimensions (batch size, 11, number of GT labels).
\nThen this tensor is processed using a 1D-CNN which kernel_size is 1.
The training data was created from experimental annotations in UniProtKB as of November 17, 2022, and the test data (private leaderboard) consists of data experimentally verified after August 22, 2023.
\n
\nTherefore, I approached the validation strategy as if it were time-series data.
\nValidation data was created by extracting experimental annotations from the latest UniProtKB data at the time of the competition, and any sub-ontology data which wrere annotated in the training data was excluded from the validation data.
Did you make multiple DL models, per embedding source? Or did you concatenate all the embeddings from the different PLM models together and use that as one model?
\nHi @danofer
\nI made 4 types of DL models for the different PLM. Embeddings are concatenated in type3 or type4. These 4 type of models are used for final ensemble (simple average ensemble).
\ntype1 ESM2t36\ntype2 ESM2t48\ntype3 ESM2t36+T5\ntype4 ESM2t48+T5\n\n
I conducted a holdout validation using the \"Validation Data\" described above.
", "rawMarkdown": "Hi @danofer \n\nI made 4 types of DL models for the different PLM. Embeddings are concatenated in type3 or type4. These 4 type of models are used for final ensemble (simple average ensemble).\n\ntype1 ESM2t36\ntype2 ESM2t48\ntype3 ESM2t36+T5\ntype4 ESM2t48+T5\n\n\nI conducted a holdout validation using the \"Validation Data\" described above." } ] }, { "id": 2592046, "postDate": "2024-01-08T11:27:11.497Z", "content": "
For the Non-Experimental Annotations, why did you use a 1d CNN, rather than embeddings or an MLP to transform them?
", "rawMarkdown": "For the Non-Experimental Annotations, why did you use a 1d CNN, rather than embeddings or an MLP to transform them?", "votes": 1, "replies": [ { "id": 2595763, "postDate": "2024-01-10T16:21:21.563Z", "content": "Hi @danofer
\nGood question!
\nI wanted to use MLP here.
\nBut Non-Experimental Annotations are predictions about what functions each protein has.
\nAnd when these are one-hot encoded, they will have a dimensionality of number of GT labels.
\nTo manage this tensors with size (batch size, 11, number of GT labels), I utilized the characteristic that 1D-CNN with kernel size 1 is the same as a fully connected layer.
I added a picture for this.
", "rawMarkdown": "Hi @danofer \n\nGood question!\n\nI wanted to use MLP here.\nBut Non-Experimental Annotations are predictions about what functions each protein has.\nAnd when these are one-hot encoded, they will have a dimensionality of number of GT labels.\nTo manage this tensors with size (batch size, 11, number of GT labels), I utilized the characteristic that 1D-CNN with kernel size 1 is the same as a fully connected layer.\n\nI added a picture for this.", "votes": 3, "replies": [ { "id": 2600428, "postDate": "2024-01-13T15:52:45.660Z", "content": "Ah neat, the figure explains it - you wanted to use the confidences/types of annotator-confidence types per annotation.
\nThat makes sense!
can you please provide your code? I'll learn a lot from it๐
", "rawMarkdown": "can you please provide your code? I'll learn a lot from it๐" }, { "id": 2796429, "postDate": "2024-05-06T08:20:00.993Z", "content": "Has your code/repo been made available yet?
\nThanks!
Did you make multiple DL models, per embedding source? Or did you concatenate all the embeddings from the different PLM models together and use that as one model?
\nHi @danofer
\nI made 4 types of DL models for the different PLM. Embeddings are concatenated in type3 or type4. These 4 type of models are used for final ensemble (simple average ensemble).
\ntype1 ESM2t36\ntype2 ESM2t48\ntype3 ESM2t36+T5\ntype4 ESM2t48+T5\n\n
I conducted a holdout validation using the \"Validation Data\" described above.
", "votes": 0, "replies": [] } ] }, { "id": 2592046, "author_name": "Dan Ofer", "author_url": "", "post_date": "2024-01-08T11:27:11.497000", "content": "For the Non-Experimental Annotations, why did you use a 1d CNN, rather than embeddings or an MLP to transform them?
", "votes": 1, "replies": [ { "id": 2595763, "author_name": "tito", "author_url": "", "post_date": "2024-01-10T16:21:21.563000", "content": "Hi @danofer
\nGood question!
\nI wanted to use MLP here.
\nBut Non-Experimental Annotations are predictions about what functions each protein has.
\nAnd when these are one-hot encoded, they will have a dimensionality of number of GT labels.
\nTo manage this tensors with size (batch size, 11, number of GT labels), I utilized the characteristic that 1D-CNN with kernel size 1 is the same as a fully connected layer.
I added a picture for this.
", "votes": 3, "replies": [ { "id": 2600428, "author_name": "Dan Ofer", "author_url": "", "post_date": "2024-01-13T15:52:45.660000", "content": "Ah neat, the figure explains it - you wanted to use the confidences/types of annotator-confidence types per annotation.
\nThat makes sense!
can you please provide your code? I'll learn a lot from it๐
", "votes": 0, "replies": [] }, { "id": 2796429, "author_name": "Dan Ofer", "author_url": "", "post_date": "2024-05-06T08:20:00.993000", "content": "Has your code/repo been made available yet?
\nThanks!