![\"t-SNE](\"https://i.ibb.co/XbhLJTD/tSNE-09.png\")
Using esm2_t30_150M_UR50D models kindly provided by @andreylalaley in this post, I did t-SNE to see how test proteins group in 2D space. Please upvote Andrey's original post and/or the datasets.
\nData points are colored by the order in which they appear in test file. It would appear that at least partial sorting by function is present in test data, as several clusters have uniform colors.
\n
The pattern is not there for train data.
\n![\"t-SNE](\"https://i.ibb.co/VSjtBTL/tSNE-10.png\")
Using esm2_t30_150M_UR50D models kindly provided by @andreylalaley in this post, I did t-SNE to see how test proteins group in 2D space. Please upvote Andrey's original post and/or the datasets.
\nData points are colored by the order in which they appear in test file. It would appear that at least partial sorting by function is present in test data, as several clusters have uniform colors.
\n
The pattern is not there for train data.
\n
It seems the proteins in test are sorted by entry name: AAAD_MOUSE, AAAAS_MOUSE, AAAT_MOUSE
", "rawMarkdown": "It seems the proteins in test are sorted by entry name: AAAD_MOUSE, AAAAS_MOUSE, AAAT_MOUSE ", "votes": 3, "replies": [ { "id": 2242799, "postDate": "2023-05-02T14:09:35.647Z", "content": "Yes, and the plot indicates that also means they are at least somewhat sorted by function.
", "rawMarkdown": "Yes, and the plot indicates that also means they are at least somewhat sorted by function.", "replies": [ { "id": 2242857, "postDate": "2023-05-02T14:42:25.230Z", "content": "I'd guess this is a simple correlation of similar name => similar function.
\nFor example, you can have two homologs from different species, one named XYZ_MOUSE and the other XYZ_HUMAN.
\nAnother thing might be subunits of complexes: XYZa_MOUSE, XYZb_MOUSE, XYZc_MOUSE.
Those would appear close to each other in a sorted list and have a higher chance to share the same GO terms.
\nOf course, there are exceptions, like SMG_DROME (Protein Smaug), NUDEL_DROME (Serine protease nudel), and SPZ_DROME (Protein spaetzle), which kind of make the names meaningless…
Edit: It looks like the proteins are primarily sorted by taxon ID. This means that consecutive proteins are from the same organism and the ESM-2 model seems to pick up that signal (rather than or in addition to function).
", "rawMarkdown": "I'd guess this is a simple correlation of similar name => similar function.\n\nFor example, you can have two homologs from different species, one named `XYZ_MOUSE` and the other `XYZ_HUMAN`.\nAnother thing might be subunits of complexes: `XYZa_MOUSE`, `XYZb_MOUSE`, `XYZc_MOUSE`.\n\nThose would appear close to each other in a sorted list and have a higher chance to share the same GO terms.\n\nOf course, there are exceptions, like `SMG_DROME` (Protein Smaug), `NUDEL_DROME` (Serine protease nudel), and `SPZ_DROME` (Protein spaetzle), which kind of make the names meaningless...\n\n\n**Edit: It looks like the proteins are primarily sorted by taxon ID. This means that consecutive proteins are from the same organism and the ESM-2 model seems to pick up that signal (rather than or in addition to function).**", "votes": 4 }, { "id": 2243003, "postDate": "2023-05-02T16:22:55.057Z", "content": "\n\nIt looks like the proteins are primarily sorted by taxon ID. This means that consecutive proteins are from the same organism and the ESM-2 model seems to pick up that signal (rather than or in addition to function).
\n
If that were the case, all clusters would be picking up taxons and we'd have a perfect rainbow distribution of colors. Instead, we have patches of uniform color, but definitely not throughout the whole dataset. I think test data is a mixture of sorting by function and by taxon, and it seems that clusters of color are picking up the functional part of it.
", "rawMarkdown": "> It looks like the proteins are primarily sorted by taxon ID. This means that consecutive proteins are from the same organism and the ESM-2 model seems to pick up that signal (rather than or in addition to function).\n\nIf that were the case, all clusters would be picking up taxons and we'd have a perfect rainbow distribution of colors. Instead, we have patches of uniform color, but definitely not throughout the whole dataset. I think test data is a mixture of sorting by function and by taxon, and it seems that clusters of color are picking up the functional part of it." }, { "id": 2243025, "postDate": "2023-05-02T16:34:50.470Z", "content": "Are we talking about alphabetical ordering?
\nLooking at files like train_sequences.fasta or sample_submission.tsv, only sometimes do the data in this competition seem to be ordered alphabetically by sequence id.
", "rawMarkdown": "Are we talking about alphabetical ordering?\n\nLooking at files like train_sequences.fasta or sample_submission.tsv, only sometimes do the data in this competition seem to be ordered alphabetically by sequence id." }, { "id": 2243046, "postDate": "2023-05-02T16:49:50.907Z", "content": "That's why I left the \"in addition to function\" as an option. Primarily sorting by taxon and then by name would put subunits of complexes or proteins with similar naming convention (protease A, protease B) closer together. Though you'd lose the orthologs, as they are from different taxons.
\nThat being said, I'd be careful to interpret clusters as hard indicators of \"similar function\". Those embeddings can pick up various signals, such as domain or kingdom, fold (all alpha, all beta, etc.), or simply protein length, which might not neccessarily have a strong correlation with sharing the same function.
\n@jbomitchell testsuperset.fasta seems to be ordered, while train_sequences.fasta is rather random.
\n\nThat being said, I'd be careful to interpret clusters as hard indicators of \"similar function\".
\n
That is a good point. Like you said, many things contribute to the signal that ends up in embeddings, and that's not surprising given their high dimensionality. Reducing it down to two dimensions will always involve some loss of information. Even though the figure below is from an older language model, I think it is fair to assume that a similar pattern can be found in modern pLMs.
\nhttps://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3220-8/figures/3
", "rawMarkdown": "> That being said, I'd be careful to interpret clusters as hard indicators of \"similar function\".\n\nThat is a good point. Like you said, many things contribute to the signal that ends up in embeddings, and that's not surprising given their high dimensionality. Reducing it down to two dimensions will always involve some loss of information. Even though the figure below is from an older language model, I think it is fair to assume that a similar pattern can be found in modern pLMs.\n\nhttps://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3220-8/figures/3\n" }, { "id": 2243239, "postDate": "2023-05-02T18:44:17.793Z", "content": "Have you tried running CDHIT to see how much of this is replicating sequence similarity clusters?
", "rawMarkdown": "Have you tried running CDHIT to see how much of this is replicating sequence similarity clusters? ", "votes": 2 }, { "id": 2243274, "postDate": "2023-05-02T19:15:53.163Z", "content": "\n\nHave you tried running CDHIT to see how much of this is replicating sequence similarity clusters?
\n
I am using ESM embeddings as provided by @andreylalaley, and don't think any redundancy removal was done by him.
\nFor the sake of other readers who may be less familiar with this concept, CD-HIT forms clusters of proteins that share identity beyond a given threshold. If we set a threshold at 90%, all proteins that are at least 90% identical to each other will end up together, and only their largest representative will be retained. It is a way to reduce a number of sequences without losing much information, as it is very likely that proteins with 90%+ identity are functionally equivalent.
\nAnother quick point for others: sequence identiity is one of the major contributors to similarities at the embedding level. It would be great if all functionally equivalent proteins were highly identical at sequence level, but that's not always the case. Instead, biology often uses synonymous words (protein sequences) that may differ quite a bit in terms of letter order and sometimes length as well. An extreme example in spoken language would be car and automobile which are nothing alike even though they mean the same thing. Protein words can often be only 10-20% identical yet still mean the same thing (that is, perform the same function).
\n\nThe pattern is not there for train data.
\n
@tilii7 nice visualization anyway🙂😉
", "rawMarkdown": "> The pattern is not there for train data.\n\n@tilii7 nice visualization anyway🙂😉 ", "votes": 2 } ], "comments": [ { "id": 2242698, "author_name": "MT", "author_url": "", "post_date": "2023-05-02T13:06:38.137000", "content": "It seems the proteins in test are sorted by entry name: AAAD_MOUSE, AAAAS_MOUSE, AAAT_MOUSE
", "votes": 3, "replies": [ { "id": 2242799, "author_name": "Tilii", "author_url": "", "post_date": "2023-05-02T14:09:35.647000", "content": "Yes, and the plot indicates that also means they are at least somewhat sorted by function.
", "votes": 0, "replies": [ { "id": 2242857, "author_name": "Norix", "author_url": "", "post_date": "2023-05-02T14:42:25.230000", "content": "I'd guess this is a simple correlation of similar name => similar function.
\nFor example, you can have two homologs from different species, one named XYZ_MOUSE and the other XYZ_HUMAN.
\nAnother thing might be subunits of complexes: XYZa_MOUSE, XYZb_MOUSE, XYZc_MOUSE.
Those would appear close to each other in a sorted list and have a higher chance to share the same GO terms.
\nOf course, there are exceptions, like SMG_DROME (Protein Smaug), NUDEL_DROME (Serine protease nudel), and SPZ_DROME (Protein spaetzle), which kind of make the names meaningless…
Edit: It looks like the proteins are primarily sorted by taxon ID. This means that consecutive proteins are from the same organism and the ESM-2 model seems to pick up that signal (rather than or in addition to function).
", "votes": 4, "replies": [] }, { "id": 2243003, "author_name": "Tilii", "author_url": "", "post_date": "2023-05-02T16:22:55.057000", "content": "\n\nIt looks like the proteins are primarily sorted by taxon ID. This means that consecutive proteins are from the same organism and the ESM-2 model seems to pick up that signal (rather than or in addition to function).
\n
If that were the case, all clusters would be picking up taxons and we'd have a perfect rainbow distribution of colors. Instead, we have patches of uniform color, but definitely not throughout the whole dataset. I think test data is a mixture of sorting by function and by taxon, and it seems that clusters of color are picking up the functional part of it.
", "votes": 0, "replies": [] }, { "id": 2243025, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-05-02T16:34:50.470000", "content": "Are we talking about alphabetical ordering?
\nLooking at files like train_sequences.fasta or sample_submission.tsv, only sometimes do the data in this competition seem to be ordered alphabetically by sequence id.
", "votes": 0, "replies": [] }, { "id": 2243046, "author_name": "Norix", "author_url": "", "post_date": "2023-05-02T16:49:50.907000", "content": "That's why I left the \"in addition to function\" as an option. Primarily sorting by taxon and then by name would put subunits of complexes or proteins with similar naming convention (protease A, protease B) closer together. Though you'd lose the orthologs, as they are from different taxons.
\nThat being said, I'd be careful to interpret clusters as hard indicators of \"similar function\". Those embeddings can pick up various signals, such as domain or kingdom, fold (all alpha, all beta, etc.), or simply protein length, which might not neccessarily have a strong correlation with sharing the same function.
\n@jbomitchell testsuperset.fasta seems to be ordered, while train_sequences.fasta is rather random.
\n\nThat being said, I'd be careful to interpret clusters as hard indicators of \"similar function\".
\n
That is a good point. Like you said, many things contribute to the signal that ends up in embeddings, and that's not surprising given their high dimensionality. Reducing it down to two dimensions will always involve some loss of information. Even though the figure below is from an older language model, I think it is fair to assume that a similar pattern can be found in modern pLMs.
\nhttps://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3220-8/figures/3
", "votes": 0, "replies": [] }, { "id": 2243239, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-05-02T18:44:17.793000", "content": "Have you tried running CDHIT to see how much of this is replicating sequence similarity clusters?
", "votes": 2, "replies": [] }, { "id": 2243274, "author_name": "Tilii", "author_url": "", "post_date": "2023-05-02T19:15:53.163000", "content": "\n\nHave you tried running CDHIT to see how much of this is replicating sequence similarity clusters?
\n
I am using ESM embeddings as provided by @andreylalaley, and don't think any redundancy removal was done by him.
\nFor the sake of other readers who may be less familiar with this concept, CD-HIT forms clusters of proteins that share identity beyond a given threshold. If we set a threshold at 90%, all proteins that are at least 90% identical to each other will end up together, and only their largest representative will be retained. It is a way to reduce a number of sequences without losing much information, as it is very likely that proteins with 90%+ identity are functionally equivalent.
\nAnother quick point for others: sequence identiity is one of the major contributors to similarities at the embedding level. It would be great if all functionally equivalent proteins were highly identical at sequence level, but that's not always the case. Instead, biology often uses synonymous words (protein sequences) that may differ quite a bit in terms of letter order and sometimes length as well. An extreme example in spoken language would be car and automobile which are nothing alike even though they mean the same thing. Protein words can often be only 10-20% identical yet still mean the same thing (that is, perform the same function).
\n\nThe pattern is not there for train data.
\n
@tilii7 nice visualization anyway🙂😉
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2242035": "Using esm2_t30_150M_UR50D models kindly provided by @andreylalaley in [**this post**](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406168), I did t-SNE to see how test proteins group in 2D space. Please upvote Andrey's original post and/or the datasets.\n\nData points are colored by the order in which they appear in test file. It would appear that at least partial sorting by function is present in test data, as several clusters have uniform colors.\n\n\n\nThe pattern is not there for train data.\n\n", "2242698": "It seems the proteins in test are sorted by entry name: AAAD_MOUSE, AAAAS_MOUSE, AAAT_MOUSE ", "2243105": "> The pattern is not there for train data.\n\n@tilii7 nice visualization anyway🙂😉 " } }