\n
![\"t-SNE](\"https://i.ibb.co/sJJkK5v/tSNE-01a.png\")
I do this exercise for all machine learning competitions to see whether we are operating with similar data distributions. In most cases it is easy to do as we deal with numerical data, or data that can be transformed easily into numbers.
\nHere I am using protein language model (pLM) embeddings that were kindly provided by @andreylalaley, and also discussed in this thread. Please upvote Andrey's original post as he has done most of the heavy lifting.
\nThe meaning of similarity here is meant with all the caveats discussed in this thread. Embedding similarity should be a decent proxy for functional similarity but it isn't completely identical.
As there are many data points, whichever group is plotted last will dominate the color representation. This is why I show two plots: one with train data plotted last (and thus being on top) and the other with test data being on top. Notwithstanding some differences I will point out below, data distributions appear to be very similar.
\nIn this image train data (blue) is plotted last. That means any patch of red that bleeds through that sea of blue color in in the areas where there are few or no train data points. These areas are generally on the left side, and mostly in the lower quadrant.
\n
There is a big chunk of points just below the plot center that is present almost exclusively in train data. There is another group, though less prominent, in the upper right quadrant.
\n![\"t-SNE](\"https://i.ibb.co/z5Rx3PD/tSNE-01b.png\")
EDIT: As @norixmb correctly pointed out below, there is a large number of identical sequences between the two datasets. When those are removed, they look less similar.
\n![\"t-SNE](\"https://i.ibb.co/RYSHBzP/tSNE-03.png\")
I do this exercise for all machine learning competitions to see whether we are operating with similar data distributions. In most cases it is easy to do as we deal with numerical data, or data that can be transformed easily into numbers.
\nHere I am using protein language model (pLM) embeddings that were kindly provided by @andreylalaley, and also discussed in this thread. Please upvote Andrey's original post as he has done most of the heavy lifting.
\nThe meaning of similarity here is meant with all the caveats discussed in this thread. Embedding similarity should be a decent proxy for functional similarity but it isn't completely identical.
As there are many data points, whichever group is plotted last will dominate the color representation. This is why I show two plots: one with train data plotted last (and thus being on top) and the other with test data being on top. Notwithstanding some differences I will point out below, data distributions appear to be very similar.
\nIn this image train data (blue) is plotted last. That means any patch of red that bleeds through that sea of blue color in in the areas where there are few or no train data points. These areas are generally on the left side, and mostly in the lower quadrant.
\n
There is a big chunk of points just below the plot center that is present almost exclusively in train data. There is another group, though less prominent, in the upper right quadrant.
\n
EDIT: As @norixmb correctly pointed out below, there is a large number of identical sequences between the two datasets. When those are removed, they look less similar.
\n
Just something to keep in mind when comparing the raw test and train data sets with each other: they are highly redundant.
\nAs pointed out in https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403200
The test FASTA file has 141,865 sequences, only 139,155 of them are unique (i.e. there are 2,710 duplicates).
\nThe train FASTA file has 142,246 sequences, only 138,924 of them are unique (i.e. there are 3,322 duplicates).
\nOf those unique sequences 72,964 are in both data sets (test and train); i.e. only 66,191 and 65,960 are unique to test and train.
This is going by unique sequences, not IDs, as proteins from different organisms can indeed share 100% sequence identity.
\nFurther, two identical sequences do not always share all annotated GO terms; either due to the way they are annotated in the database, or they might indeed exhibit slightly different functions in different organisms (protein moonlighting).
All good points, thank you. I didn't know that ~73 K sequences were shared. Even with all that additional info, I don't think anything changes in the main point: there is a compact chunk of a certain type of train sequences that seems to be missing almost completely in test data, but other than that their overall distributions are similar.
\n\n\nFurther, two identical sequences do not always share all annotated GO terms; either due to the way they are annotated in the database, or they might indeed exhibit slightly different functions in different oragnisms.
\n
While this is true if we want to split hairs and account for all known exceptions, it may not be very helpful to casual readers who do not have a strong background in this area. High sequence identity for proteins means high likelihood of the same function. A more accurate way of stating that is that high sequence identity and highly similar sequence length almost always mean the same function. There are exceptions to everything, but this is as solid a rule as it gets in biology. For completely identical sequences that becomes a guarantee. Now, what you are calling slightly different functions is not necessarily a different original function, but often an additional function. That can be because there are non-equivalent modifications of proteins in different organisms, or different interacting partners, or different substrates present. With all those things being comparable or the same, identical protein sequences will have the same functionality. I don't want to get into the reasons why sometimes that is not reflected in GO annotations.
Warning: the following is \"extra-credit\" stuff. In case someone is interested in going down a rabbit hole that may not have a lot of impact on the competition performance, yet is still interesting and relevant.
\nNote same gene product may have completely different functions if expressed in different tissues, intra- vs. extra-cellular expression, oligomerization of subunits, etc.
\nhttps://www.sciencedirect.com/science/article/pii/S0968000498013358
\nhttp://www.moonlightingproteins.org/
\nAs to how this is reflected in the Uniprot Database, it's a it fuzzy. Usually, all the different functions will be associated with the same protein. However, as was pointed out, one exception may be in different species.
\nThis leads us to prediction of function by orthology, here is one interesting paper:
\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7355290/
", "rawMarkdown": "Warning: the following is \"extra-credit\" stuff. In case someone is interested in going down a rabbit hole that may not have a lot of impact on the competition performance, yet is still interesting and relevant. \n\nNote same gene product may have completely different functions if expressed in different tissues, intra- vs. extra-cellular expression, oligomerization of subunits, etc. \n\nhttps://www.sciencedirect.com/science/article/pii/S0968000498013358\n\nhttp://www.moonlightingproteins.org/\n\nAs to how this is reflected in the Uniprot Database, it's a it fuzzy. Usually, all the different functions will be associated with the same protein. However, as was pointed out, one exception may be in different species. \n\n\nThis leads us to prediction of function by orthology, here is one interesting paper: \n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7355290/", "votes": null }, { "id": "2243434", "postDate": "05/02/2023 22:38:25", "content": "Sorry, I should have worded it better in my initial post.
\nWhat I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data). Thus, zooming in on the proteins unique to each data set might provide more information (by removing the parts that are identical). I never wanted to question your point about the \"outlier\" cluster of train sequences.
\nRegarding the point of identical proteins with different sets of GO terms, I primarily wanted to warn about simply dropping all duplicates as you might lose information by doing so. One protein might have a GO term with experimental evidence, while a duplicate only has it via IEA (based on the former one).
", "rawMarkdown": "Sorry, I should have worded it better in my initial post.\n\nWhat I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data). Thus, zooming in on the proteins unique to each data set might provide more information (by removing the parts that are identical). I never wanted to question your point about the \"outlier\" cluster of train sequences.\n\nRegarding the point of identical proteins with different sets of GO terms, I primarily wanted to warn about simply dropping all duplicates as you might lose information by doing so. One protein might have a GO term with experimental evidence, while a duplicate only has it via IEA (based on the former one).", "votes": null }, { "id": "2243446", "postDate": "05/02/2023 22:59:07", "content": "I have learned a long time ago that not all of these unexpected or additional functions of proteins are to be believed. I don't want to name names or cite papers, as it would be unfair to colleagues who are not reading this and therefore would not have a chance to respond. Let's just say that we worked on a protein with a well-established function, and that function has since been confirmed numerous times by us and many other labs. There are also many cryo-EM structures confirming its role structurally, on top of biochemical and genetic data. Yet there is a paper stating that the same protein has a function in a different pathway. This additional function is often cited in papers out of deference to existing literature, yet it has never been confirmed by a subsequent paper from the same group, or by any other group. We are talking about decades since the original report was published. We didn't get even a reply when asking for the molecular constructs used in their paper.
I have spent a good part of my career doing \"sequence-gazing\" and predicting protein functions from sequence/structure, and even longer doing wet-lab experiment. While I will always defend classical experiments (e.g., those that are not of a high-throughput variety) as the most reliable approach to functional annotations, there is not a single piece of evidence out there that would convince me of that additional function for the protein in question. Yet that additional function is listed out there in databases.
I will leave this link for those who have enough time to surf down the rabbit hole:
\nhttps://genomebiology.biomedcentral.com/articles/10.1186/gb-2001-2-12-research0051
", "rawMarkdown": "I have learned a long time ago that not all of these `unexpected` or `additional` functions of proteins are to be believed. I don't want to name names or cite papers, as it would be unfair to colleagues who are not reading this and therefore would not have a chance to respond. Let's just say that we worked on a protein with a well-established function, and that function has since been confirmed numerous times by us and many other labs. There are also many cryo-EM structures confirming its role structurally, on top of biochemical and genetic data. Yet there is a paper stating that the same protein has a function in a different pathway. This `additional` function is often cited in papers out of deference to existing literature, yet it has never been confirmed by a subsequent paper from the same group, or by any other group. We are talking about decades since the original report was published. We didn't get even a reply when asking for the molecular constructs used in their paper.\n\nI have spent a good part of my career doing \"sequence-gazing\" and predicting protein functions from sequence/structure, and even longer doing wet-lab experiment. While I will always defend classical experiments (*e.g.*, those that are not of a high-throughput variety) as the most reliable approach to functional annotations, there is not a single piece of evidence out there that would convince me of that `additional` function for the protein in question. Yet that additional function is listed out there in databases.\n\nI will leave this link for those who have enough time to surf down the rabbit hole:\n\nhttps://genomebiology.biomedcentral.com/articles/10.1186/gb-2001-2-12-research0051", "votes": null }, { "id": "2243458", "postDate": "05/02/2023 23:28:13", "content": "\n\nWhat I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data).
\n
I get what you meant, and it is a good point. I have no way of testing it easily as I am using ESM embedding data made by someone else, and there is no 1-to-1 correspondence between sequence IDs and embeddings in the dataset (fewer IDs than sequences). So I can't just remove the redundant sequences and do t-SNE on a reduced dataset.
", "rawMarkdown": "> What I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data).\n\nI get what you meant, and it is a good point. I have no way of testing it easily as I am using ESM embedding data made by someone else, and there is no 1-to-1 correspondence between sequence IDs and embeddings in the dataset (fewer IDs than sequences). So I can't just remove the redundant sequences and do t-SNE on a reduced dataset.", "votes": null }, { "id": "2246157", "postDate": "05/04/2023 23:38:52", "content": "@norixmb I managed to find a ProtT5 dataset by @alexandervc (please upvote his dataset) that has all IDs and sequence embeddings, and was able to remove the sequences shared by both datasets. You were right, their t-SNE plots now look less similar. This new plot is shown above in the main post.
", "rawMarkdown": "norixmb I managed to find a [**ProtT5 dataset**](https://www.kaggle.com/datasets/sergeifironov/t5embeds) by @alexandervc (please upvote his dataset) that has all IDs and sequence embeddings, and was able to remove the sequences shared by both datasets. You were right, their t-SNE plots now look less similar. This new plot is shown above in the main post.", "votes": null } ], "comments": [ { "id": 2243380, "author_name": "norixmb", "author_url": "", "post_date": "05/02/2023 21:35:56", "content": "Just something to keep in mind when comparing the raw test and train data sets with each other: they are highly redundant.
\nAs pointed out in https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403200
The test FASTA file has 141,865 sequences, only 139,155 of them are unique (i.e. there are 2,710 duplicates).
\nThe train FASTA file has 142,246 sequences, only 138,924 of them are unique (i.e. there are 3,322 duplicates).
\nOf those unique sequences 72,964 are in both data sets (test and train); i.e. only 66,191 and 65,960 are unique to test and train.
This is going by unique sequences, not IDs, as proteins from different organisms can indeed share 100% sequence identity.
\nFurther, two identical sequences do not always share all annotated GO terms; either due to the way they are annotated in the database, or they might indeed exhibit slightly different functions in different organisms (protein moonlighting).
All good points, thank you. I didn't know that ~73 K sequences were shared. Even with all that additional info, I don't think anything changes in the main point: there is a compact chunk of a certain type of train sequences that seems to be missing almost completely in test data, but other than that their overall distributions are similar.
\n\n\nFurther, two identical sequences do not always share all annotated GO terms; either due to the way they are annotated in the database, or they might indeed exhibit slightly different functions in different oragnisms.
\n
While this is true if we want to split hairs and account for all known exceptions, it may not be very helpful to casual readers who do not have a strong background in this area. High sequence identity for proteins means high likelihood of the same function. A more accurate way of stating that is that high sequence identity and highly similar sequence length almost always mean the same function. There are exceptions to everything, but this is as solid a rule as it gets in biology. For completely identical sequences that becomes a guarantee. Now, what you are calling slightly different functions is not necessarily a different original function, but often an additional function. That can be because there are non-equivalent modifications of proteins in different organisms, or different interacting partners, or different substrates present. With all those things being comparable or the same, identical protein sequences will have the same functionality. I don't want to get into the reasons why sometimes that is not reflected in GO annotations.
Warning: the following is \"extra-credit\" stuff. In case someone is interested in going down a rabbit hole that may not have a lot of impact on the competition performance, yet is still interesting and relevant.
\nNote same gene product may have completely different functions if expressed in different tissues, intra- vs. extra-cellular expression, oligomerization of subunits, etc.
\nhttps://www.sciencedirect.com/science/article/pii/S0968000498013358
\nhttp://www.moonlightingproteins.org/
\nAs to how this is reflected in the Uniprot Database, it's a it fuzzy. Usually, all the different functions will be associated with the same protein. However, as was pointed out, one exception may be in different species.
\nThis leads us to prediction of function by orthology, here is one interesting paper:
\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7355290/
", "votes": null, "replies": [ { "id": 2243446, "author_name": "tilii7", "author_url": "", "post_date": "05/02/2023 22:59:07", "content": "I have learned a long time ago that not all of these unexpected or additional functions of proteins are to be believed. I don't want to name names or cite papers, as it would be unfair to colleagues who are not reading this and therefore would not have a chance to respond. Let's just say that we worked on a protein with a well-established function, and that function has since been confirmed numerous times by us and many other labs. There are also many cryo-EM structures confirming its role structurally, on top of biochemical and genetic data. Yet there is a paper stating that the same protein has a function in a different pathway. This additional function is often cited in papers out of deference to existing literature, yet it has never been confirmed by a subsequent paper from the same group, or by any other group. We are talking about decades since the original report was published. We didn't get even a reply when asking for the molecular constructs used in their paper.
I have spent a good part of my career doing \"sequence-gazing\" and predicting protein functions from sequence/structure, and even longer doing wet-lab experiment. While I will always defend classical experiments (e.g., those that are not of a high-throughput variety) as the most reliable approach to functional annotations, there is not a single piece of evidence out there that would convince me of that additional function for the protein in question. Yet that additional function is listed out there in databases.
I will leave this link for those who have enough time to surf down the rabbit hole:
\nhttps://genomebiology.biomedcentral.com/articles/10.1186/gb-2001-2-12-research0051
", "votes": null, "replies": [] } ] }, { "id": 2243434, "author_name": "norixmb", "author_url": "", "post_date": "05/02/2023 22:38:25", "content": "Sorry, I should have worded it better in my initial post.
\nWhat I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data). Thus, zooming in on the proteins unique to each data set might provide more information (by removing the parts that are identical). I never wanted to question your point about the \"outlier\" cluster of train sequences.
\nRegarding the point of identical proteins with different sets of GO terms, I primarily wanted to warn about simply dropping all duplicates as you might lose information by doing so. One protein might have a GO term with experimental evidence, while a duplicate only has it via IEA (based on the former one).
", "votes": null, "replies": [ { "id": 2243458, "author_name": "tilii7", "author_url": "", "post_date": "05/02/2023 23:28:13", "content": "\n\nWhat I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data).
\n
I get what you meant, and it is a good point. I have no way of testing it easily as I am using ESM embedding data made by someone else, and there is no 1-to-1 correspondence between sequence IDs and embeddings in the dataset (fewer IDs than sequences). So I can't just remove the redundant sequences and do t-SNE on a reduced dataset.
", "votes": null, "replies": [] }, { "id": 2246157, "author_name": "tilii7", "author_url": "", "post_date": "05/04/2023 23:38:52", "content": "@norixmb I managed to find a ProtT5 dataset by @alexandervc (please upvote his dataset) that has all IDs and sequence embeddings, and was able to remove the sequences shared by both datasets. You were right, their t-SNE plots now look less similar. This new plot is shown above in the main post.
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2243349": "I do this exercise for all machine learning competitions to see whether we are operating with similar data distributions. In most cases it is easy to do as we deal with numerical data, or data that can be transformed easily into numbers.\n\nHere I am using protein language model (pLM) embeddings that were kindly provided by @andreylalaley, and also discussed in [**this thread**](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406336). Please upvote Andrey's [**original post**](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406168) as he has done most of the heavy lifting.\n\nThe meaning of `similarity` here is meant with all the caveats discussed in [**this thread**](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406336). Embedding similarity should be a decent proxy for `functional similarity` but it isn't completely identical.\n\nAs there are many data points, whichever group is plotted last will dominate the color representation. This is why I show two plots: one with train data plotted last (and thus being on top) and the other with test data being on top. Notwithstanding some differences I will point out below, data distributions appear to be very similar.\n\nIn this image train data (blue) is plotted last. That means any patch of red that bleeds through that sea of blue color in in the areas where there are few or no train data points. These areas are generally on the left side, and mostly in the lower quadrant.\n\n\nThere is a big chunk of points just below the plot center that is present almost exclusively in train data. There is another group, though less prominent, in the upper right quadrant.\n\n\nEDIT: As @norixmb correctly pointed out [**below**](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406573#2243380), there is a large number of identical sequences between the two datasets. When those are removed, they look less similar.\n\n", "2243380": "Just something to keep in mind when comparing the raw test and train data sets with each other: they are highly redundant.\nAs pointed out in https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403200\n\nThe test FASTA file has 141,865 sequences, only 139,155 of them are unique (i.e. there are 2,710 duplicates).\nThe train FASTA file has 142,246 sequences, only 138,924 of them are unique (i.e. there are 3,322 duplicates).\nOf those unique sequences 72,964 are in both data sets (test and train); i.e. only 66,191 and 65,960 are unique to test and train.\n\nThis is going by unique sequences, not IDs, as proteins from different organisms can indeed share 100% sequence identity.\nFurther, two identical sequences do not always share all annotated GO terms; either due to the way they are annotated in the database, or they might indeed exhibit slightly different functions in different organisms (protein moonlighting).", "2243398": "All good points, thank you. I didn't know that ~73 K sequences were shared. Even with all that additional info, I don't think anything changes in the main point: there is a compact chunk of a certain type of train sequences that seems to be missing almost completely in test data, but other than that their overall distributions are similar.\n\n> Further, two identical sequences do not always share all annotated GO terms; either due to the way they are annotated in the database, or they might indeed exhibit slightly different functions in different oragnisms.\n\nWhile this is true if we want to split hairs and account for all known exceptions, it may not be very helpful to casual readers who do not have a strong background in this area. High sequence identity for proteins means high likelihood of the same function. A more accurate way of stating that is that high sequence identity and highly similar sequence length almost always mean the same function. There are exceptions to everything, but this is as solid a rule as it gets in biology. For completely identical sequences that becomes a guarantee. Now, what you are calling `slightly different functions` is not necessarily a different original function, but often an additional function. That can be because there are non-equivalent modifications of proteins in different organisms, or different interacting partners, or different substrates present. With all those things being comparable or the same, identical protein sequences will have the same functionality. I don't want to get into the reasons why sometimes that is not reflected in GO annotations.", "2243421": "Warning: the following is \"extra-credit\" stuff. In case someone is interested in going down a rabbit hole that may not have a lot of impact on the competition performance, yet is still interesting and relevant. \n\nNote same gene product may have completely different functions if expressed in different tissues, intra- vs. extra-cellular expression, oligomerization of subunits, etc. \n\nhttps://www.sciencedirect.com/science/article/pii/S0968000498013358\n\nhttp://www.moonlightingproteins.org/\n\nAs to how this is reflected in the Uniprot Database, it's a it fuzzy. Usually, all the different functions will be associated with the same protein. However, as was pointed out, one exception may be in different species. \n\n\nThis leads us to prediction of function by orthology, here is one interesting paper: \n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7355290/", "2243434": "Sorry, I should have worded it better in my initial post.\n\nWhat I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data). Thus, zooming in on the proteins unique to each data set might provide more information (by removing the parts that are identical). I never wanted to question your point about the \"outlier\" cluster of train sequences.\n\nRegarding the point of identical proteins with different sets of GO terms, I primarily wanted to warn about simply dropping all duplicates as you might lose information by doing so. One protein might have a GO term with experimental evidence, while a duplicate only has it via IEA (based on the former one).", "2243446": "I have learned a long time ago that not all of these `unexpected` or `additional` functions of proteins are to be believed. I don't want to name names or cite papers, as it would be unfair to colleagues who are not reading this and therefore would not have a chance to respond. Let's just say that we worked on a protein with a well-established function, and that function has since been confirmed numerous times by us and many other labs. There are also many cryo-EM structures confirming its role structurally, on top of biochemical and genetic data. Yet there is a paper stating that the same protein has a function in a different pathway. This `additional` function is often cited in papers out of deference to existing literature, yet it has never been confirmed by a subsequent paper from the same group, or by any other group. We are talking about decades since the original report was published. We didn't get even a reply when asking for the molecular constructs used in their paper.\n\nI have spent a good part of my career doing \"sequence-gazing\" and predicting protein functions from sequence/structure, and even longer doing wet-lab experiment. While I will always defend classical experiments (*e.g.*, those that are not of a high-throughput variety) as the most reliable approach to functional annotations, there is not a single piece of evidence out there that would convince me of that `additional` function for the protein in question. Yet that additional function is listed out there in databases.\n\nI will leave this link for those who have enough time to surf down the rabbit hole:\n\nhttps://genomebiology.biomedcentral.com/articles/10.1186/gb-2001-2-12-research0051", "2243458": "> What I wanted to point out is that the similar distribution is in part due to the high overlap (>50% of the data).\n\nI get what you meant, and it is a good point. I have no way of testing it easily as I am using ESM embedding data made by someone else, and there is no 1-to-1 correspondence between sequence IDs and embeddings in the dataset (fewer IDs than sequences). So I can't just remove the redundant sequences and do t-SNE on a reduced dataset.", "2246157": "norixmb I managed to find a [**ProtT5 dataset**](https://www.kaggle.com/datasets/sergeifironov/t5embeds) by @alexandervc (please upvote his dataset) that has all IDs and sequence embeddings, and was able to remove the sequences shared by both datasets. You were right, their t-SNE plots now look less similar. This new plot is shown above in the main post." }, "source": "meta" }