{ "id": 421189, "title": "Lots of sequences with 100% similarity", "url": "/competitions/cafa-5-protein-function-prediction/discussion/421189", "author_name": "", "post_date": "2023-07-04T10:07:35.507335600Z", "votes": 11, "comment_count": 12, "views": 0, "content": "

Hello All.
\nI used diamond to get pairwise sequence similarity for the fasta file which is our training set.
\n\"\"
\nThis looks very strange to me.
\nI filtered out all the duplicates (using df = df.filter(pl.col(\"Query Accession\") < pl.col(\"Target Accession\"))) before plotting the above but still so many sequences seem to have unreasonable high agreement.
\nHere a sample of 20 sequences
\n\"\"
\nMy questions to the people who know more about biology: Is this something we expect? Do i interpret something wrong? What (if there is) is an intuition behind it?
\nAnyway when building our models it is something (as @leechuck pointed out yesterday) we need to take care of when building our models. What do you think?

", "messages": [ { "id": "2329515", "postDate": "07/04/2023 10:07:35", "content": "

Hello All.
\nI used diamond to get pairwise sequence similarity for the fasta file which is our training set.
\n\"\"
\nThis looks very strange to me.
\nI filtered out all the duplicates (using df = df.filter(pl.col(\"Query Accession\") < pl.col(\"Target Accession\"))) before plotting the above but still so many sequences seem to have unreasonable high agreement.
\nHere a sample of 20 sequences
\n\"\"
\nMy questions to the people who know more about biology: Is this something we expect? Do i interpret something wrong? What (if there is) is an intuition behind it?
\nAnyway when building our models it is something (as @leechuck pointed out yesterday) we need to take care of when building our models. What do you think?

", "rawMarkdown": "Hello All. \nI used diamond to get pairwise sequence similarity for the fasta file which is our training set.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F8c28d8b3a4ee439275548349fb8fa3f7%2Fsequence%20similar.png?generation=1688464958940597&alt=media)\nThis looks very strange to me.\nI filtered out all the duplicates (using `df = df.filter(pl.col(\"Query Accession\") < pl.col(\"Target Accession\"))`) before plotting the above but still so many sequences seem to have unreasonable high agreement.\nHere a sample of 20 sequences\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F2d5312586fb930e2f21b265b5df87756%2Fsequence%20similar%20sample.png?generation=1688465084060430&alt=media)\nMy questions to the people who know more about biology: Is this something we expect? Do i interpret something wrong? What (if there is) is an intuition behind it?\nAnyway when building our models it is something (as @leechuck pointed out yesterday) we need to take care of when building our models. What do you think?", "votes": null }, { "id": "2331482", "postDate": "07/05/2023 15:21:32", "content": "

Checking a few of these entries. (1) some are possible isoforms. Isoforms are protein sequences encoded by the same gene, but different parts are expressed as proteins. (Think: photocopying different pages from the same book chapter). Those have a local sequence identity, but note that the start and end of the query: they are of different lengths, and overlap partially.
\n(2) some are completely identical and are obviously fully encoded by the same gene in the same organism once you read teh uniprotkb entry. The duplication is probably because one or both not curated, and could have entered the unreviewed part of Uniprot from multiple sources. I emailed the people at EBI-EMBL, and will post here once I have more information. Some seem to be duplicate entries, but one is in SwissProt and one is in TrEMBL.

", "rawMarkdown": "Checking a few of these entries. (1) some are possible isoforms. *Isoforms* are protein sequences encoded by the same gene, but different parts are expressed as proteins. (Think: photocopying different pages from the same book chapter). Those have a *local* sequence identity, but note that the start and end of the query: they are of different lengths, and overlap partially. \n(2) some are completely identical and are obviously fully encoded by the same gene in the same organism once you read teh uniprotkb entry. The duplication is probably because one or both not curated, and could have entered the unreviewed part of Uniprot from multiple sources. I emailed the people at EBI-EMBL, and will post here once I have more information. Some seem to be duplicate entries, but one is in SwissProt and one is in TrEMBL.", "votes": null }, { "id": "2331519", "postDate": "07/05/2023 15:49:37", "content": "

thanks for the quick response.

", "rawMarkdown": "thanks for the quick response.", "votes": null }, { "id": "2332083", "postDate": "07/06/2023 01:00:28", "content": "

I'll add a comment about duplicates to bring some biology in play. An example: as far as I know, all eukaryotes have calmodulin (CALM), an important gene that encodes a protein involved in signaling. Humans have 3 CALM genes, CALM1, CALM2, and CALM3, which are identical in the human reference genome. A typical reason for such gene duplication events to be retained is for a cell to be able to produce more protein copies fast. You can see these entries in UniProt as CALM1_HUMAN, CALM2_HUMAN, and CALM3_HUMAN. CALM1 is in chromosome 14, CALM2 is in chromosome 2, and CALM3 is in chromosome 19. Of course, when you look at the differences between human CALM and that of primates, mammals, etc. there won't be many differences. Etc. Technically speaking, the data that evolution presented to us is not IID, even if we like to treat it as such to make machine learning manageable. Many of these proteins (data points) are very much related (e.g., sometimes genes are duplicated, but evolve in different directions, so they will have similar function).

", "rawMarkdown": "I'll add a comment about duplicates to bring some biology in play. An example: as far as I know, all eukaryotes have calmodulin (CALM), an important gene that encodes a protein involved in signaling. Humans have 3 CALM genes, CALM1, CALM2, and CALM3, which are identical in the human reference genome. A typical reason for such gene duplication events to be retained is for a cell to be able to produce more protein copies fast. You can see these entries in UniProt as CALM1_HUMAN, CALM2_HUMAN, and CALM3_HUMAN. CALM1 is in chromosome 14, CALM2 is in chromosome 2, and CALM3 is in chromosome 19. Of course, when you look at the differences between human CALM and that of primates, mammals, etc. there won't be many differences. Etc. Technically speaking, the data that evolution presented to us is not IID, even if we like to treat it as such to make machine learning manageable. Many of these proteins (data points) are very much related (e.g., sometimes genes are duplicated, but evolve in different directions, so they will have similar function).", "votes": null }, { "id": "2332962", "postDate": "07/06/2023 14:58:44", "content": "

@simonveitner
\nTo recap:

\n

1) Many sequences with 100% identity are 100% identical over part of either sequences' length: what we call a \"local identity\". Most of those are probably isoforms: the different proteins coded by the same gene. This can happen because of alternative splicing.

\n

2) Some may be 100% identical over the entire length of both sequences, and both sequences have an equal length: \"global identity\". As @predragradivojac pointed out, those would be homologs, sequences that were duplicated at some point in evolution, coded by genes that sit on different parts of the same chromosome, or on different chromosomes. One way to see if they are local or global identical is to add 2 columns to the table with the length of both sequences, and check of the 100% identity occurs in sequences of identical length.

\n

These data you generated are actually quite interesting to our friends at EBI-EMBL, BTW. (These are the people providing the annotated data we are testing on, among other things). If you have the inclination, it would be nice to see the csv file you generated in this competition's datasets? This can also help, as you noted, with training models.

", "rawMarkdown": "simonveitner \nTo recap:\n\n1) Many sequences with 100% identity are 100% identical over part of either sequences' length: what we call a \"local identity\". Most of those are probably isoforms: the different proteins coded by the same gene. This can happen because of [alternative splicing](https://en.wikipedia.org/wiki/Alternative_splicing). \n\n\n2) Some may be 100% identical over the entire length of both sequences, and both sequences have an equal length: \"global identity\". As @predragradivojac pointed out, those would be homologs, sequences that were duplicated at some point in evolution, coded by genes that sit on different parts of the same chromosome, or on different chromosomes. One way to see if they are local or global identical is to add 2 columns to the table with the length of both sequences, and check of the 100% identity occurs in sequences of identical length.\n\n\n\nThese data you generated are actually quite interesting to our friends at EBI-EMBL, BTW. (These are the people providing the annotated data we are testing on, among other things). If you have the inclination, it would be nice to see the csv file you generated in this competition's datasets? This can also help, as you noted, with training models.", "votes": null }, { "id": "2333018", "postDate": "07/06/2023 15:41:18", "content": "

https://github.com/bbuchfink/diamond/wiki/1.-Tutorial

\n

This was essentially what I did. Exchange astral for our train fasta and setting to ultra-sensitive.
\nTook about 20 min on my regular laptop to run :-)

", "rawMarkdown": "https://github.com/bbuchfink/diamond/wiki/1.-Tutorial\n\nThis was essentially what I did. Exchange astral for our train fasta and setting to ultra-sensitive.\nTook about 20 min on my regular laptop to run :-)", "votes": null }, { "id": "2333026", "postDate": "07/06/2023 15:47:43", "content": "

To your point 2: I did this already and it turned out that about 4000 sequences are \"real duplicates\" meaning they agree from first to last letter.

", "rawMarkdown": "To your point 2: I did this already and it turned out that about 4000 sequences are \"real duplicates\" meaning they agree from first to last letter.", "votes": null }, { "id": "2333351", "postDate": "07/06/2023 22:04:58", "content": "

As a remark we can find duplicates just by pandas:

\n
\n

v = pd.Series(list_seqs).value_counts()# .head(10)
\n print('total duplicated sequences:', v[v!=1].sum())

\n
\n

See some analysis of duplicates here:
\nhttps://www.kaggle.com/code/alexandervc/cafa5-towards-eda?scriptVersionId=135448102&cellId=99

", "rawMarkdown": "As a remark we can find duplicates just by pandas:\n\n>v = pd.Series(list_seqs).value_counts()# .head(10)\nprint('total duplicated sequences:', v[v!=1].sum())\n\nSee some analysis of duplicates here:\nhttps://www.kaggle.com/code/alexandervc/cafa5-towards-eda?scriptVersionId=135448102&cellId=99", "votes": null }, { "id": "2333679", "postDate": "07/07/2023 05:39:37", "content": "

I also found much Duplicate Values. When I ran the Sequence from Fasta Sequences, into Prot-BERT BDF Encoder. I found that for the first 10,000 terms, around 40% were identical. The encoder is taking a lot of time and thus I am trying to reduce the time to get more in depth results.

", "rawMarkdown": "I also found much Duplicate Values. When I ran the Sequence from Fasta Sequences, into `Prot-BERT BDF` Encoder. I found that for the first `10,000` terms, around 40% were identical. The encoder is taking a lot of time and thus I am trying to reduce the time to get more in depth results.", "votes": null }, { "id": "2333686", "postDate": "07/07/2023 05:45:00", "content": "

Yes. To finetune longer and cost effektive it would be nice to reduce size of dataset.

", "rawMarkdown": "Yes. To finetune longer and cost effektive it would be nice to reduce size of dataset.", "votes": null }, { "id": "2333835", "postDate": "07/07/2023 07:53:54", "content": "

I found that Henri Upton=>ProtBERT Embeddings for CAFA5 did a similar thing before.

\n

When I type-casted it to Pandas.DataFrame, I found that there are only 35,113 Unique Values. Even if we consider some error from Prot-BERT BFD, and think that there were 50,000 Unique Values in the Original Sequence. That is still more than 50% of the total values

\n

This will not only lead to longer training times and require more effective management of resources, but will also add overfitting to any model.

\n

I think we should remove as much duplicates as we can and then train the Model

", "rawMarkdown": "I found that [Henri Upton](https://www.kaggle.com/henriupton)=>[ProtBERT Embeddings for CAFA5](https://www.kaggle.com/datasets/henriupton/protbert-embeddings-for-cafa5) did a similar thing before. \n\nWhen I `type-casted` it to `Pandas.DataFrame`, I found that there are only `35,113` `Unique Values`. Even if we `consider` some `error` from [Prot-BERT BFD](https://huggingface.co/Rostlab/prot_bert_bfd), and think that there were `50,000` `Unique Values` in the `Original Sequence`. That is still more than `50`% of the total values\n\nThis will not only lead to `longer training times` and require `more effective management of resources`, but will also add `overfitting` to any model. \n\nI think we should `remove` as much `duplicates` as we can and then train the Model", "votes": null }, { "id": "2335592", "postDate": "07/08/2023 16:32:02", "content": "

another insight:
\nMFO seems to contain by far the most \"duplicates\" (in proportion to the 100% alligned sequences).
\nIs it in terms of biology reasonable that this is the case?

\n

\"\"

\n

Also another interesting plot:

\n

\"\"

\n

Combining the two it seems that in MFO ontology there is a high proportional of \"global identites\" whereas in the other two \"local identity\" seems to be a more common thing. Also the distributon looks quiet different. (Much more weight for the other 2 onots in the tail)

", "rawMarkdown": "another insight:\nMFO seems to contain by far the most \"duplicates\" (in proportion to the 100% alligned sequences).\nIs it in terms of biology reasonable that this is the case?\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F59b7e4cbf9be42d95bb7ab58a26b3416%2FScreenshot_20230708_182854.png?generation=1688833865445550&alt=media)\n\nAlso another interesting plot:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F5167bbed2209e23d0b6e57ec9200cb06%2FScreenshot_20230708_183348.png?generation=1688834076855476&alt=media)\n\nCombining the two it seems that in MFO ontology there is a high proportional of \"global identites\" whereas in the other two \"local identity\" seems to be a more common thing. Also the distributon looks quiet different. (Much more weight for the other 2 onots in the tail)", "votes": null }, { "id": "2336030", "postDate": "07/09/2023 04:02:56", "content": "

Question

\n

I am not able to understand one thing.

\n

Even if the there are a lot of duplicates in the data. Then the mode should overfit on the training data. which is not actually happening(in my case). Results are not like what is expected from a Overfit Model. Why is the model not overfitting even after lot of duplicates...?

\n

\"\"
\nI tried using different Embedments (T5/Prot-BERT BFD/T5 + Prot-BERT BFD (Better Results)) for different models (Single Layer Perceptron (TEMPROT Paper)/Multi Layer Perceptron/Convolution), but there was not much change in the results.
\nCan anyone explain this...?

", "rawMarkdown": "# Question\n\nI am not able to understand one thing. \n\nEven if the there are a lot of `duplicates in the data`. Then the mode should `overfit on the training data`. which is not actually happening(in my case). `Results` are `not` like what is `expected from a Overfit Model`. Why is the model `not overfitting` even after lot of `duplicates...?`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9711370%2F76e8ceb3b244d3d9e1bf3652cf175892%2FLoss.png?generation=1688874820791814&alt=media)\nI tried using different Embedments (`T5`/`Prot-BERT BFD`/`T5 + Prot-BERT BFD (Better Results)`) for different models (`Single Layer Perceptron (TEMPROT Paper)`/`Multi Layer Perceptron`/`Convolution`), but there was not much change in the results.\n`Can anyone explain this...?`", "votes": null } ], "comments": [ { "id": 2331482, "author_name": "iddux1", "author_url": "", "post_date": "07/05/2023 15:21:32", "content": "

Checking a few of these entries. (1) some are possible isoforms. Isoforms are protein sequences encoded by the same gene, but different parts are expressed as proteins. (Think: photocopying different pages from the same book chapter). Those have a local sequence identity, but note that the start and end of the query: they are of different lengths, and overlap partially.
\n(2) some are completely identical and are obviously fully encoded by the same gene in the same organism once you read teh uniprotkb entry. The duplication is probably because one or both not curated, and could have entered the unreviewed part of Uniprot from multiple sources. I emailed the people at EBI-EMBL, and will post here once I have more information. Some seem to be duplicate entries, but one is in SwissProt and one is in TrEMBL.

", "votes": null, "replies": [ { "id": 2331519, "author_name": "simonveitner", "author_url": "", "post_date": "07/05/2023 15:49:37", "content": "

thanks for the quick response.

", "votes": null, "replies": [] }, { "id": 2332083, "author_name": "predragradivojac", "author_url": "", "post_date": "07/06/2023 01:00:28", "content": "

I'll add a comment about duplicates to bring some biology in play. An example: as far as I know, all eukaryotes have calmodulin (CALM), an important gene that encodes a protein involved in signaling. Humans have 3 CALM genes, CALM1, CALM2, and CALM3, which are identical in the human reference genome. A typical reason for such gene duplication events to be retained is for a cell to be able to produce more protein copies fast. You can see these entries in UniProt as CALM1_HUMAN, CALM2_HUMAN, and CALM3_HUMAN. CALM1 is in chromosome 14, CALM2 is in chromosome 2, and CALM3 is in chromosome 19. Of course, when you look at the differences between human CALM and that of primates, mammals, etc. there won't be many differences. Etc. Technically speaking, the data that evolution presented to us is not IID, even if we like to treat it as such to make machine learning manageable. Many of these proteins (data points) are very much related (e.g., sometimes genes are duplicated, but evolve in different directions, so they will have similar function).

", "votes": null, "replies": [ { "id": 2332962, "author_name": "iddux1", "author_url": "", "post_date": "07/06/2023 14:58:44", "content": "

@simonveitner
\nTo recap:

\n

1) Many sequences with 100% identity are 100% identical over part of either sequences' length: what we call a \"local identity\". Most of those are probably isoforms: the different proteins coded by the same gene. This can happen because of alternative splicing.

\n

2) Some may be 100% identical over the entire length of both sequences, and both sequences have an equal length: \"global identity\". As @predragradivojac pointed out, those would be homologs, sequences that were duplicated at some point in evolution, coded by genes that sit on different parts of the same chromosome, or on different chromosomes. One way to see if they are local or global identical is to add 2 columns to the table with the length of both sequences, and check of the 100% identity occurs in sequences of identical length.

\n

These data you generated are actually quite interesting to our friends at EBI-EMBL, BTW. (These are the people providing the annotated data we are testing on, among other things). If you have the inclination, it would be nice to see the csv file you generated in this competition's datasets? This can also help, as you noted, with training models.

", "votes": null, "replies": [ { "id": 2333018, "author_name": "simonveitner", "author_url": "", "post_date": "07/06/2023 15:41:18", "content": "

https://github.com/bbuchfink/diamond/wiki/1.-Tutorial

\n

This was essentially what I did. Exchange astral for our train fasta and setting to ultra-sensitive.
\nTook about 20 min on my regular laptop to run :-)

", "votes": null, "replies": [ { "id": 2333026, "author_name": "simonveitner", "author_url": "", "post_date": "07/06/2023 15:47:43", "content": "

To your point 2: I did this already and it turned out that about 4000 sequences are \"real duplicates\" meaning they agree from first to last letter.

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2333351, "author_name": "alexandervc", "author_url": "", "post_date": "07/06/2023 22:04:58", "content": "

As a remark we can find duplicates just by pandas:

\n
\n

v = pd.Series(list_seqs).value_counts()# .head(10)
\n print('total duplicated sequences:', v[v!=1].sum())

\n
\n

See some analysis of duplicates here:
\nhttps://www.kaggle.com/code/alexandervc/cafa5-towards-eda?scriptVersionId=135448102&cellId=99

", "votes": null, "replies": [] }, { "id": 2333679, "author_name": "ayushs9020", "author_url": "", "post_date": "07/07/2023 05:39:37", "content": "

I also found much Duplicate Values. When I ran the Sequence from Fasta Sequences, into Prot-BERT BDF Encoder. I found that for the first 10,000 terms, around 40% were identical. The encoder is taking a lot of time and thus I am trying to reduce the time to get more in depth results.

", "votes": null, "replies": [ { "id": 2333686, "author_name": "simonveitner", "author_url": "", "post_date": "07/07/2023 05:45:00", "content": "

Yes. To finetune longer and cost effektive it would be nice to reduce size of dataset.

", "votes": null, "replies": [ { "id": 2333835, "author_name": "ayushs9020", "author_url": "", "post_date": "07/07/2023 07:53:54", "content": "

I found that Henri Upton=>ProtBERT Embeddings for CAFA5 did a similar thing before.

\n

When I type-casted it to Pandas.DataFrame, I found that there are only 35,113 Unique Values. Even if we consider some error from Prot-BERT BFD, and think that there were 50,000 Unique Values in the Original Sequence. That is still more than 50% of the total values

\n

This will not only lead to longer training times and require more effective management of resources, but will also add overfitting to any model.

\n

I think we should remove as much duplicates as we can and then train the Model

", "votes": null, "replies": [] } ] } ] }, { "id": 2335592, "author_name": "simonveitner", "author_url": "", "post_date": "07/08/2023 16:32:02", "content": "

another insight:
\nMFO seems to contain by far the most \"duplicates\" (in proportion to the 100% alligned sequences).
\nIs it in terms of biology reasonable that this is the case?

\n

\"\"

\n

Also another interesting plot:

\n

\"\"

\n

Combining the two it seems that in MFO ontology there is a high proportional of \"global identites\" whereas in the other two \"local identity\" seems to be a more common thing. Also the distributon looks quiet different. (Much more weight for the other 2 onots in the tail)

", "votes": null, "replies": [] }, { "id": 2336030, "author_name": "ayushs9020", "author_url": "", "post_date": "07/09/2023 04:02:56", "content": "

Question

\n

I am not able to understand one thing.

\n

Even if the there are a lot of duplicates in the data. Then the mode should overfit on the training data. which is not actually happening(in my case). Results are not like what is expected from a Overfit Model. Why is the model not overfitting even after lot of duplicates...?

\n

\"\"
\nI tried using different Embedments (T5/Prot-BERT BFD/T5 + Prot-BERT BFD (Better Results)) for different models (Single Layer Perceptron (TEMPROT Paper)/Multi Layer Perceptron/Convolution), but there was not much change in the results.
\nCan anyone explain this...?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2329515": "Hello All. \nI used diamond to get pairwise sequence similarity for the fasta file which is our training set.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F8c28d8b3a4ee439275548349fb8fa3f7%2Fsequence%20similar.png?generation=1688464958940597&alt=media)\nThis looks very strange to me.\nI filtered out all the duplicates (using `df = df.filter(pl.col(\"Query Accession\") < pl.col(\"Target Accession\"))`) before plotting the above but still so many sequences seem to have unreasonable high agreement.\nHere a sample of 20 sequences\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F2d5312586fb930e2f21b265b5df87756%2Fsequence%20similar%20sample.png?generation=1688465084060430&alt=media)\nMy questions to the people who know more about biology: Is this something we expect? Do i interpret something wrong? What (if there is) is an intuition behind it?\nAnyway when building our models it is something (as @leechuck pointed out yesterday) we need to take care of when building our models. What do you think?", "2331482": "Checking a few of these entries. (1) some are possible isoforms. *Isoforms* are protein sequences encoded by the same gene, but different parts are expressed as proteins. (Think: photocopying different pages from the same book chapter). Those have a *local* sequence identity, but note that the start and end of the query: they are of different lengths, and overlap partially. \n(2) some are completely identical and are obviously fully encoded by the same gene in the same organism once you read teh uniprotkb entry. The duplication is probably because one or both not curated, and could have entered the unreviewed part of Uniprot from multiple sources. I emailed the people at EBI-EMBL, and will post here once I have more information. Some seem to be duplicate entries, but one is in SwissProt and one is in TrEMBL.", "2331519": "thanks for the quick response.", "2332083": "I'll add a comment about duplicates to bring some biology in play. An example: as far as I know, all eukaryotes have calmodulin (CALM), an important gene that encodes a protein involved in signaling. Humans have 3 CALM genes, CALM1, CALM2, and CALM3, which are identical in the human reference genome. A typical reason for such gene duplication events to be retained is for a cell to be able to produce more protein copies fast. You can see these entries in UniProt as CALM1_HUMAN, CALM2_HUMAN, and CALM3_HUMAN. CALM1 is in chromosome 14, CALM2 is in chromosome 2, and CALM3 is in chromosome 19. Of course, when you look at the differences between human CALM and that of primates, mammals, etc. there won't be many differences. Etc. Technically speaking, the data that evolution presented to us is not IID, even if we like to treat it as such to make machine learning manageable. Many of these proteins (data points) are very much related (e.g., sometimes genes are duplicated, but evolve in different directions, so they will have similar function).", "2332962": "simonveitner \nTo recap:\n\n1) Many sequences with 100% identity are 100% identical over part of either sequences' length: what we call a \"local identity\". Most of those are probably isoforms: the different proteins coded by the same gene. This can happen because of [alternative splicing](https://en.wikipedia.org/wiki/Alternative_splicing). \n\n\n2) Some may be 100% identical over the entire length of both sequences, and both sequences have an equal length: \"global identity\". As @predragradivojac pointed out, those would be homologs, sequences that were duplicated at some point in evolution, coded by genes that sit on different parts of the same chromosome, or on different chromosomes. One way to see if they are local or global identical is to add 2 columns to the table with the length of both sequences, and check of the 100% identity occurs in sequences of identical length.\n\n\n\nThese data you generated are actually quite interesting to our friends at EBI-EMBL, BTW. (These are the people providing the annotated data we are testing on, among other things). If you have the inclination, it would be nice to see the csv file you generated in this competition's datasets? This can also help, as you noted, with training models.", "2333018": "https://github.com/bbuchfink/diamond/wiki/1.-Tutorial\n\nThis was essentially what I did. Exchange astral for our train fasta and setting to ultra-sensitive.\nTook about 20 min on my regular laptop to run :-)", "2333026": "To your point 2: I did this already and it turned out that about 4000 sequences are \"real duplicates\" meaning they agree from first to last letter.", "2333351": "As a remark we can find duplicates just by pandas:\n\n>v = pd.Series(list_seqs).value_counts()# .head(10)\nprint('total duplicated sequences:', v[v!=1].sum())\n\nSee some analysis of duplicates here:\nhttps://www.kaggle.com/code/alexandervc/cafa5-towards-eda?scriptVersionId=135448102&cellId=99", "2333679": "I also found much Duplicate Values. When I ran the Sequence from Fasta Sequences, into `Prot-BERT BDF` Encoder. I found that for the first `10,000` terms, around 40% were identical. The encoder is taking a lot of time and thus I am trying to reduce the time to get more in depth results.", "2333686": "Yes. To finetune longer and cost effektive it would be nice to reduce size of dataset.", "2333835": "I found that [Henri Upton](https://www.kaggle.com/henriupton)=>[ProtBERT Embeddings for CAFA5](https://www.kaggle.com/datasets/henriupton/protbert-embeddings-for-cafa5) did a similar thing before. \n\nWhen I `type-casted` it to `Pandas.DataFrame`, I found that there are only `35,113` `Unique Values`. Even if we `consider` some `error` from [Prot-BERT BFD](https://huggingface.co/Rostlab/prot_bert_bfd), and think that there were `50,000` `Unique Values` in the `Original Sequence`. That is still more than `50`% of the total values\n\nThis will not only lead to `longer training times` and require `more effective management of resources`, but will also add `overfitting` to any model. \n\nI think we should `remove` as much `duplicates` as we can and then train the Model", "2335592": "another insight:\nMFO seems to contain by far the most \"duplicates\" (in proportion to the 100% alligned sequences).\nIs it in terms of biology reasonable that this is the case?\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F59b7e4cbf9be42d95bb7ab58a26b3416%2FScreenshot_20230708_182854.png?generation=1688833865445550&alt=media)\n\nAlso another interesting plot:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11646918%2F5167bbed2209e23d0b6e57ec9200cb06%2FScreenshot_20230708_183348.png?generation=1688834076855476&alt=media)\n\nCombining the two it seems that in MFO ontology there is a high proportional of \"global identites\" whereas in the other two \"local identity\" seems to be a more common thing. Also the distributon looks quiet different. (Much more weight for the other 2 onots in the tail)", "2336030": "# Question\n\nI am not able to understand one thing. \n\nEven if the there are a lot of `duplicates in the data`. Then the mode should `overfit on the training data`. which is not actually happening(in my case). `Results` are `not` like what is `expected from a Overfit Model`. Why is the model `not overfitting` even after lot of `duplicates...?`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9711370%2F76e8ceb3b244d3d9e1bf3652cf175892%2FLoss.png?generation=1688874820791814&alt=media)\nI tried using different Embedments (`T5`/`Prot-BERT BFD`/`T5 + Prot-BERT BFD (Better Results)`) for different models (`Single Layer Perceptron (TEMPROT Paper)`/`Multi Layer Perceptron`/`Convolution`), but there was not much change in the results.\n`Can anyone explain this...?`" }, "source": "meta" }