![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11298796%2F11bc4782c751744c88bdfbeec3d8ea9f%2FScreenshot%202023-05-12%20at%208.11.53%20AM.png?generation=1683904343421260&alt=media\")
I've noticed that the current popular notebooks split their CV randomly. I believe this can lead to an overconfident estimate of model performance since proteins that are very similar can end up being in both the train and validation set.
\nI'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set. That being said it may be valid to group by embedding as well.
\nWhat are your CV grouping strategies? Or do y'all think it's unnecessary?
", "messages": [ { "id": "2253986", "postDate": "05/10/2023 15:15:43", "content": "I've noticed that the current popular notebooks split their CV randomly. I believe this can lead to an overconfident estimate of model performance since proteins that are very similar can end up being in both the train and validation set.
\nI'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set. That being said it may be valid to group by embedding as well.
\nWhat are your CV grouping strategies? Or do y'all think it's unnecessary?
", "rawMarkdown": "I've noticed that the current popular notebooks split their CV randomly. I believe this can lead to an overconfident estimate of model performance since proteins that are very similar can end up being in both the train and validation set. \n\nI'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set. That being said it may be valid to group by embedding as well.\n\nWhat are your CV grouping strategies? Or do y'all think it's unnecessary?", "votes": null }, { "id": "2254152", "postDate": "05/10/2023 17:23:33", "content": "Preventing similar proteins (by sequence similarity/identity) between training and test data is definitely a good thing. After all, inference through homology (i.e., searching for similar proteins) is a popular method to predict/transfer annotations.
\n\n\nI'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set.
\n
One thing to keep in mind is that most clustering algorithms only care about the \"uniqueness\" of the cluster representatives. This means that members of two different clusters might still be more similar than whatever threshold was used for the clustering of the representatives. An exception would be MMseqs2 with --cluster-mode 1 (https://github.com/soedinglab/MMseqs2/wiki#clustering-modes).
Another way to reduce similarity between training and test sets would be to randomly split your data for the CV and then resample the corresponding training splits by removing any proteins that are similar to those in the current test split (or vice versa). A useful tool for this might be CD-Hit, which has a \"cd-hit-2d\" script to remove protein sequences from a set A based on their similarity to proteins from a set B.
", "rawMarkdown": "Preventing similar proteins (by sequence similarity/identity) between training and test data is definitely a good thing. After all, inference through homology (i.e., searching for similar proteins) is a popular method to predict/transfer annotations.\n\n> I'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set.\n\nOne thing to keep in mind is that most clustering algorithms only care about the \"uniqueness\" of the cluster representatives. This means that members of two different clusters might still be more similar than whatever threshold was used for the clustering of the representatives. An exception would be MMseqs2 with `--cluster-mode 1` (https://github.com/soedinglab/MMseqs2/wiki#clustering-modes).\n\nAnother way to reduce similarity between training and test sets would be to randomly split your data for the CV and then resample the corresponding training splits by removing any proteins that are similar to those in the current test split (or vice versa). A useful tool for this might be CD-Hit, which has a \"cd-hit-2d\" script to remove protein sequences from a set A based on their similarity to proteins from a set B.", "votes": null }, { "id": "2254295", "postDate": "05/10/2023 20:04:30", "content": "Your idea is conceptually sound. The problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none. See a couple of non-exhaustive examples below:
\n>Cluster 25\n0 1159aa, >Q9NCP8... at 97.67%\n1 2404aa, >Q7KU95... at 99.88%\n2 1716aa, >Q9I7Q8... at 97.38%\n3 11634aa, >X2JCM1... at 99.96%\n4 2566aa, >X2JGD3... at 94.54%\n5 2532aa, >A8JNM4... at 95.62%\n6 547aa, >M9PBV0... at 94.70%\n7 1009aa, >Q8SWY2... at 97.32%\n8 13559aa, >M9MRX4... *\n\n>Cluster 50\n0 202aa, >Q9NBC1... at 100.00%\n1 126aa, >Q9NBB8... at 93.65%\n2 786aa, >Q9GV79... at 99.11%\n3 201aa, >Q9NBA9... at 93.03%\n4 223aa, >Q9NBC0... at 99.55%\n5 231aa, >Q9NBB9... at 99.13%\n6 8255aa, >A0A0B4KG35... *\n7 929aa, >A1ZA69... at 99.68%\n8 536aa, >Q9GV82... at 100.00%\n9 90aa, >Q9NBC5... at 100.00%\n10 577aa, >Q9GV81... at 93.59%\n11 157aa, >Q9NBB4... at 93.63%\n12 215aa, >Q9N630... at 98.60%\n13 140aa, >Q9NBB6... at 95.71%\n14 623aa, >Q9GV80... at 99.68%\n15 7905aa, >A0A0B4JD82... at 99.27%\n16 119aa, >Q9NBC6... at 100.00%\n17 929aa, >O01651... at 99.03%\n18 832aa, >O01653... at 99.64%\n19 576aa, >Q8MYR8... at 98.44%\n20 212aa, >Q9NBB0... at 91.98%\n21 913aa, >A0A0B4JD64... at 100.00%\n22 151aa, >Q9NBB7... at 99.34%\n23 128aa, >Q9NBC4... at 99.22%\n24 8216aa, >A0A0B4KF84... at 99.66%\n25 602aa, >A0A0B4JCU1... at 94.52%\n26 281aa, >Q9NBC2... at 98.22%\n27 569aa, >O01652... at 93.50%\n28 209aa, >Q9GTV0... at 96.65%\n29 210aa, >Q9N6N9... at 99.52%\n30 7944aa, >A1ZA72... at 99.61%\n\n>Cluster 100453\n0 85aa, >Q1PEF6... *\nAll of the above sequences have >90% identity to at least one member of the cluster. A cluster with 30 members has strong representation of different lengths (~100 to >8000 residues), and more likely of several domains / functionalities as well. The very last cluster shown has a single member - and there are ~93 K others like it.
", "rawMarkdown": "Your idea is conceptually sound. The problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none. See a couple of non-exhaustive examples below:\n\n```\n>Cluster 25\n0 1159aa, >Q9NCP8... at 97.67%\n1 2404aa, >Q7KU95... at 99.88%\n2 1716aa, >Q9I7Q8... at 97.38%\n3 11634aa, >X2JCM1... at 99.96%\n4 2566aa, >X2JGD3... at 94.54%\n5 2532aa, >A8JNM4... at 95.62%\n6 547aa, >M9PBV0... at 94.70%\n7 1009aa, >Q8SWY2... at 97.32%\n8 13559aa, >M9MRX4... *\n\n>Cluster 50\n0 202aa, >Q9NBC1... at 100.00%\n1 126aa, >Q9NBB8... at 93.65%\n2 786aa, >Q9GV79... at 99.11%\n3 201aa, >Q9NBA9... at 93.03%\n4 223aa, >Q9NBC0... at 99.55%\n5 231aa, >Q9NBB9... at 99.13%\n6 8255aa, >A0A0B4KG35... *\n7 929aa, >A1ZA69... at 99.68%\n8 536aa, >Q9GV82... at 100.00%\n9 90aa, >Q9NBC5... at 100.00%\n10 577aa, >Q9GV81... at 93.59%\n11 157aa, >Q9NBB4... at 93.63%\n12 215aa, >Q9N630... at 98.60%\n13 140aa, >Q9NBB6... at 95.71%\n14 623aa, >Q9GV80... at 99.68%\n15 7905aa, >A0A0B4JD82... at 99.27%\n16 119aa, >Q9NBC6... at 100.00%\n17 929aa, >O01651... at 99.03%\n18 832aa, >O01653... at 99.64%\n19 576aa, >Q8MYR8... at 98.44%\n20 212aa, >Q9NBB0... at 91.98%\n21 913aa, >A0A0B4JD64... at 100.00%\n22 151aa, >Q9NBB7... at 99.34%\n23 128aa, >Q9NBC4... at 99.22%\n24 8216aa, >A0A0B4KF84... at 99.66%\n25 602aa, >A0A0B4JCU1... at 94.52%\n26 281aa, >Q9NBC2... at 98.22%\n27 569aa, >O01652... at 93.50%\n28 209aa, >Q9GTV0... at 96.65%\n29 210aa, >Q9N6N9... at 99.52%\n30 7944aa, >A1ZA72... at 99.61%\n\n>Cluster 100453\n0 85aa, >Q1PEF6... *\n\n```\n\nAll of the above sequences have >90% identity to at least one member of the cluster. A cluster with 30 members has strong representation of different lengths (~100 to >8000 residues), and more likely of several domains / functionalities as well. The very last cluster shown has a single member - and there are ~93 K others like it.", "votes": null }, { "id": "2254588", "postDate": "05/11/2023 05:21:14", "content": "In this example, how can a sequence of 2404aa and another of 547aa both have > 90% similarity? If I had to guess these are the alignment scores against a reference for the cluster and there is a really small gap penalty? My knowledge on alignment is pretty limited to 2 sequence alignment so sorry if that was a silly question.
\n\n\nThe problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none.
\n
Is the problem here that we're splitting into the train/val set by group which won't take into account their size? Thus potentially leading to a different size diff between the train/val than expected?
\nThanks!
", "rawMarkdown": "In this example, how can a sequence of 2404aa and another of 547aa both have > 90% similarity? If I had to guess these are the alignment scores against a reference for the cluster and there is a really small gap penalty? My knowledge on alignment is pretty limited to 2 sequence alignment so sorry if that was a silly question.\n\n> The problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none.\n\nIs the problem here that we're splitting into the train/val set by group which won't take into account their size? Thus potentially leading to a different size diff between the train/val than expected?\n\nThanks!", "votes": null }, { "id": "2254594", "postDate": "05/11/2023 05:25:32", "content": "I ran the command listed in the README:
mmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\nFrom a quick look I believe --cov-mode 1 will also set --cluster-mode 1 but I'll take a deeper look tomorrow and into CD-Hit.
Thanks!
", "rawMarkdown": "I ran the command listed in the `README`:\n\n```bash\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\n```\n\nFrom a quick look I believe `--cov-mode 1` will also set `--cluster-mode 1` but I'll take a deeper look tomorrow and into CD-Hit. \n\nThanks!", "votes": null }, { "id": "2255569", "postDate": "05/11/2023 20:00:12", "content": "Yes, these are all identity scores versus the reference sequence which has an asterisk (#8 in cluster 25, #6 in cluster 50). You may be mistaking sequence identity/similarity for overlap. Two sequences can be 100% identical even if one of them is 500 residues long and the other is 2000. The meaning of identity is not global, so gap penalties do not enter the picture here. They can align over 500 residues (100% overlap for the shorter sequence, 25% for the longer) and have 100% identity in that region.
\nIf you take percent overlap into account when clustering (which mmseqs does), than a 500-residues sequence is not going to end up in the same cluster/group as a 2000-residue sequence. So when you split by groups, one of these groups may end up in train data, and the other in validation data. Yet if they are 100% identical, even over a limited length, chances are very good that at least some of their functionality will be shared. So there will be a data leak because some of train data will be in validation as well.
If we cluster just by identity and ignore the overlap, as I did, we may end up with some clusters that have many members, while the majority will have only one member. Splitting both categories uniformly between train and validation data will also be a non-trivial task.
", "rawMarkdown": "Yes, these are all identity scores versus the reference sequence which has an asterisk (#8 in cluster 25, #6 in cluster 50). You may be mistaking sequence identity/similarity for overlap. Two sequences can be 100% identical even if one of them is 500 residues long and the other is 2000. The meaning of identity is not global, so gap penalties do not enter the picture here. They can align over 500 residues (100% overlap for the shorter sequence, 25% for the longer) and have 100% identity in that region.\n\nIf you take percent overlap into account when clustering (which `mmseqs` does), than a 500-residues sequence is not going to end up in the same cluster/group as a 2000-residue sequence. So when you split by groups, one of these groups may end up in train data, and the other in validation data. Yet if they are 100% identical, even over a limited length, chances are very good that at least some of their functionality will be shared. So there will be a data leak because some of train data will be in validation as well.\n\nIf we cluster just by identity and ignore the overlap, as I did, we may end up with some clusters that have many members, while the majority will have only one member. Splitting both categories uniformly between train and validation data will also be a non-trivial task.", "votes": null }, { "id": "2256553", "postDate": "05/12/2023 15:13:07", "content": "I ran an experiment re splitting uniformly. Using the suggested clustering command on the mmseqs README:
\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\n--cov-mode 1 should do clustering similarly to how you described above, which if I understood correctly should keep similar proteins together regardless of length.
Once that's done, I appended the representative protein ID to each row
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11298796%2F2d097412f4a54e82735a27f6149dc4bc%2FScreenshot%202023-05-12%20at%208.05.36%20AM.png?generation=1683903958336528&alt=media\")
I then shuffled it based on the cluster 10000 times and plotted the ratio of rows between train/test (or val, rather):
\ngss = GroupShuffleSplit(test_size=, n_splits=, random_state=).split(\n train_terms_df, groups=train_terms_df[]\n)\n\ntrain_test_ratios = []\n train_idx, test_idx tqdm(gss):\n train_test_ratios.append((train_idx) / (test_idx))\n\nplt.figure(figsize=(, ))\nplt.hist(train_test_ratios, bins=)\nplt.xlabel()\nplt.ylabel()\nplt.title()\nplt.show()\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11298796%2F8543c69d21fc7e75f90a029e8f2c0ef8%2Foutput.png?generation=1683904109408009&alt=media\")
Seems like an attempted split of 20% val results in an actual split of (Edit: ~17%-21%, I did the math wrong), which seems pretty good!
\nHowever, I'm using the suggested mmseqs settings. I don't really have enough domain knowledge to know which coverage/identity values are biologically significant. I'm guessing your -min-seq-id is lower than my 0.5.
If you could provide the command you used to generate your clusters I can give it a try with those as well. Thanks!
\nEDIT: I tried with mmseqs easy-cluster {sequences_path} clusterRes tmp --min-seq-id 0.05 -c 0.9 --cov-mode 1 and got a similar result in terms of actual data split.
I suspect it may also be important to think about how the competition will actually be scored. AFAIK, many of the experimental annotations that are validated in the relevant time period may well be 'easy' to predict - that is, experimental confirmations of functions that are exactly what you'd expect by simple inference from sequence similarity. I'd hesitate to have a validation protocol that concentrated too much on making 'difficult' inferences, when getting a high recall on the 'easy' ones may be just as important to the final score.
\nI think @tilii7 is quite right to say that it's not really possible to construct non-homologous clusters of sequences when many of the proteins are multi-domain, there will be just too many different domain-sharing combinations. In contrast, it's entirely feasible to construct non-homologous families of domains, as various databases have done.
", "rawMarkdown": "I suspect it may also be important to think about how the competition will actually be scored. AFAIK, many of the experimental annotations that are validated in the relevant time period may well be 'easy' to predict - that is, experimental confirmations of functions that are exactly what you'd expect by simple inference from sequence similarity. I'd hesitate to have a validation protocol that concentrated too much on making 'difficult' inferences, when getting a high recall on the 'easy' ones may be just as important to the final score.\n\nI think @tilii7 is quite right to say that it's not really possible to construct non-homologous clusters of sequences when many of the proteins are multi-domain, there will be just too many different domain-sharing combinations. In contrast, it's entirely feasible to construct non-homologous families of domains, as various databases have done.", "votes": null }, { "id": "2256771", "postDate": "05/12/2023 17:51:16", "content": "Part of the clustering solution I showed above was done with cd-hit.
cd-hit -i train_sequences.fasta -o train_sequences.90 -c 0.9\nIn this example, cluster summary would end up in a file train_sequences.90.clstr.
Here is some proposal on groups and folds.
\nComments are welcome:
\nhttps://www.kaggle.com/code/alexandervc/cafa5-23-groups-and-folds-diamond-igraph
Great discussion! I decided to try replicate the evaluation set-up: for each protein, I train on all labels from one subontology and predict all the labels in another subontology https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up
", "rawMarkdown": "Great discussion! I decided to try replicate the evaluation set-up: for each protein, I train on all labels from one subontology and predict all the labels in another subontology https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up", "votes": null } ], "comments": [ { "id": 2254152, "author_name": "norixmb", "author_url": "", "post_date": "05/10/2023 17:23:33", "content": "Preventing similar proteins (by sequence similarity/identity) between training and test data is definitely a good thing. After all, inference through homology (i.e., searching for similar proteins) is a popular method to predict/transfer annotations.
\n\n\nI'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set.
\n
One thing to keep in mind is that most clustering algorithms only care about the \"uniqueness\" of the cluster representatives. This means that members of two different clusters might still be more similar than whatever threshold was used for the clustering of the representatives. An exception would be MMseqs2 with --cluster-mode 1 (https://github.com/soedinglab/MMseqs2/wiki#clustering-modes).
Another way to reduce similarity between training and test sets would be to randomly split your data for the CV and then resample the corresponding training splits by removing any proteins that are similar to those in the current test split (or vice versa). A useful tool for this might be CD-Hit, which has a \"cd-hit-2d\" script to remove protein sequences from a set A based on their similarity to proteins from a set B.
", "votes": null, "replies": [ { "id": 2254594, "author_name": "subachev", "author_url": "", "post_date": "05/11/2023 05:25:32", "content": "I ran the command listed in the README:
mmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\nFrom a quick look I believe --cov-mode 1 will also set --cluster-mode 1 but I'll take a deeper look tomorrow and into CD-Hit.
Thanks!
", "votes": null, "replies": [] } ] }, { "id": 2254295, "author_name": "tilii7", "author_url": "", "post_date": "05/10/2023 20:04:30", "content": "Your idea is conceptually sound. The problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none. See a couple of non-exhaustive examples below:
\n>Cluster 25\n0 1159aa, >Q9NCP8... at 97.67%\n1 2404aa, >Q7KU95... at 99.88%\n2 1716aa, >Q9I7Q8... at 97.38%\n3 11634aa, >X2JCM1... at 99.96%\n4 2566aa, >X2JGD3... at 94.54%\n5 2532aa, >A8JNM4... at 95.62%\n6 547aa, >M9PBV0... at 94.70%\n7 1009aa, >Q8SWY2... at 97.32%\n8 13559aa, >M9MRX4... *\n\n>Cluster 50\n0 202aa, >Q9NBC1... at 100.00%\n1 126aa, >Q9NBB8... at 93.65%\n2 786aa, >Q9GV79... at 99.11%\n3 201aa, >Q9NBA9... at 93.03%\n4 223aa, >Q9NBC0... at 99.55%\n5 231aa, >Q9NBB9... at 99.13%\n6 8255aa, >A0A0B4KG35... *\n7 929aa, >A1ZA69... at 99.68%\n8 536aa, >Q9GV82... at 100.00%\n9 90aa, >Q9NBC5... at 100.00%\n10 577aa, >Q9GV81... at 93.59%\n11 157aa, >Q9NBB4... at 93.63%\n12 215aa, >Q9N630... at 98.60%\n13 140aa, >Q9NBB6... at 95.71%\n14 623aa, >Q9GV80... at 99.68%\n15 7905aa, >A0A0B4JD82... at 99.27%\n16 119aa, >Q9NBC6... at 100.00%\n17 929aa, >O01651... at 99.03%\n18 832aa, >O01653... at 99.64%\n19 576aa, >Q8MYR8... at 98.44%\n20 212aa, >Q9NBB0... at 91.98%\n21 913aa, >A0A0B4JD64... at 100.00%\n22 151aa, >Q9NBB7... at 99.34%\n23 128aa, >Q9NBC4... at 99.22%\n24 8216aa, >A0A0B4KF84... at 99.66%\n25 602aa, >A0A0B4JCU1... at 94.52%\n26 281aa, >Q9NBC2... at 98.22%\n27 569aa, >O01652... at 93.50%\n28 209aa, >Q9GTV0... at 96.65%\n29 210aa, >Q9N6N9... at 99.52%\n30 7944aa, >A1ZA72... at 99.61%\n\n>Cluster 100453\n0 85aa, >Q1PEF6... *\nAll of the above sequences have >90% identity to at least one member of the cluster. A cluster with 30 members has strong representation of different lengths (~100 to >8000 residues), and more likely of several domains / functionalities as well. The very last cluster shown has a single member - and there are ~93 K others like it.
", "votes": null, "replies": [ { "id": 2254588, "author_name": "subachev", "author_url": "", "post_date": "05/11/2023 05:21:14", "content": "In this example, how can a sequence of 2404aa and another of 547aa both have > 90% similarity? If I had to guess these are the alignment scores against a reference for the cluster and there is a really small gap penalty? My knowledge on alignment is pretty limited to 2 sequence alignment so sorry if that was a silly question.
\n\n\nThe problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none.
\n
Is the problem here that we're splitting into the train/val set by group which won't take into account their size? Thus potentially leading to a different size diff between the train/val than expected?
\nThanks!
", "votes": null, "replies": [ { "id": 2255569, "author_name": "tilii7", "author_url": "", "post_date": "05/11/2023 20:00:12", "content": "Yes, these are all identity scores versus the reference sequence which has an asterisk (#8 in cluster 25, #6 in cluster 50). You may be mistaking sequence identity/similarity for overlap. Two sequences can be 100% identical even if one of them is 500 residues long and the other is 2000. The meaning of identity is not global, so gap penalties do not enter the picture here. They can align over 500 residues (100% overlap for the shorter sequence, 25% for the longer) and have 100% identity in that region.
\nIf you take percent overlap into account when clustering (which mmseqs does), than a 500-residues sequence is not going to end up in the same cluster/group as a 2000-residue sequence. So when you split by groups, one of these groups may end up in train data, and the other in validation data. Yet if they are 100% identical, even over a limited length, chances are very good that at least some of their functionality will be shared. So there will be a data leak because some of train data will be in validation as well.
If we cluster just by identity and ignore the overlap, as I did, we may end up with some clusters that have many members, while the majority will have only one member. Splitting both categories uniformly between train and validation data will also be a non-trivial task.
", "votes": null, "replies": [ { "id": 2256553, "author_name": "subachev", "author_url": "", "post_date": "05/12/2023 15:13:07", "content": "I ran an experiment re splitting uniformly. Using the suggested clustering command on the mmseqs README:
\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\n--cov-mode 1 should do clustering similarly to how you described above, which if I understood correctly should keep similar proteins together regardless of length.
Once that's done, I appended the representative protein ID to each row
\n
I then shuffled it based on the cluster 10000 times and plotted the ratio of rows between train/test (or val, rather):
\ngss = GroupShuffleSplit(test_size=, n_splits=, random_state=).split(\n train_terms_df, groups=train_terms_df[]\n)\n\ntrain_test_ratios = []\n train_idx, test_idx tqdm(gss):\n train_test_ratios.append((train_idx) / (test_idx))\n\nplt.figure(figsize=(, ))\nplt.hist(train_test_ratios, bins=)\nplt.xlabel()\nplt.ylabel()\nplt.title()\nplt.show()\n
Seems like an attempted split of 20% val results in an actual split of (Edit: ~17%-21%, I did the math wrong), which seems pretty good!
\nHowever, I'm using the suggested mmseqs settings. I don't really have enough domain knowledge to know which coverage/identity values are biologically significant. I'm guessing your -min-seq-id is lower than my 0.5.
If you could provide the command you used to generate your clusters I can give it a try with those as well. Thanks!
\nEDIT: I tried with mmseqs easy-cluster {sequences_path} clusterRes tmp --min-seq-id 0.05 -c 0.9 --cov-mode 1 and got a similar result in terms of actual data split.
Part of the clustering solution I showed above was done with cd-hit.
cd-hit -i train_sequences.fasta -o train_sequences.90 -c 0.9\nIn this example, cluster summary would end up in a file train_sequences.90.clstr.
I suspect it may also be important to think about how the competition will actually be scored. AFAIK, many of the experimental annotations that are validated in the relevant time period may well be 'easy' to predict - that is, experimental confirmations of functions that are exactly what you'd expect by simple inference from sequence similarity. I'd hesitate to have a validation protocol that concentrated too much on making 'difficult' inferences, when getting a high recall on the 'easy' ones may be just as important to the final score.
\nI think @tilii7 is quite right to say that it's not really possible to construct non-homologous clusters of sequences when many of the proteins are multi-domain, there will be just too many different domain-sharing combinations. In contrast, it's entirely feasible to construct non-homologous families of domains, as various databases have done.
", "votes": null, "replies": [] }, { "id": 2310792, "author_name": "alexandervc", "author_url": "", "post_date": "06/20/2023 15:54:38", "content": "Here is some proposal on groups and folds.
\nComments are welcome:
\nhttps://www.kaggle.com/code/alexandervc/cafa5-23-groups-and-folds-diamond-igraph
Great discussion! I decided to try replicate the evaluation set-up: for each protein, I train on all labels from one subontology and predict all the labels in another subontology https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2253986": "I've noticed that the current popular notebooks split their CV randomly. I believe this can lead to an overconfident estimate of model performance since proteins that are very similar can end up being in both the train and validation set. \n\nI'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set. That being said it may be valid to group by embedding as well.\n\nWhat are your CV grouping strategies? Or do y'all think it's unnecessary?", "2254152": "Preventing similar proteins (by sequence similarity/identity) between training and test data is definitely a good thing. After all, inference through homology (i.e., searching for similar proteins) is a popular method to predict/transfer annotations.\n\n> I'm currently splitting my CV set by keeping groups clustered together by mmseqs within the same set.\n\nOne thing to keep in mind is that most clustering algorithms only care about the \"uniqueness\" of the cluster representatives. This means that members of two different clusters might still be more similar than whatever threshold was used for the clustering of the representatives. An exception would be MMseqs2 with `--cluster-mode 1` (https://github.com/soedinglab/MMseqs2/wiki#clustering-modes).\n\nAnother way to reduce similarity between training and test sets would be to randomly split your data for the CV and then resample the corresponding training splits by removing any proteins that are similar to those in the current test split (or vice versa). A useful tool for this might be CD-Hit, which has a \"cd-hit-2d\" script to remove protein sequences from a set A based on their similarity to proteins from a set B.", "2254295": "Your idea is conceptually sound. The problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none. See a couple of non-exhaustive examples below:\n\n```\n>Cluster 25\n0 1159aa, >Q9NCP8... at 97.67%\n1 2404aa, >Q7KU95... at 99.88%\n2 1716aa, >Q9I7Q8... at 97.38%\n3 11634aa, >X2JCM1... at 99.96%\n4 2566aa, >X2JGD3... at 94.54%\n5 2532aa, >A8JNM4... at 95.62%\n6 547aa, >M9PBV0... at 94.70%\n7 1009aa, >Q8SWY2... at 97.32%\n8 13559aa, >M9MRX4... *\n\n>Cluster 50\n0 202aa, >Q9NBC1... at 100.00%\n1 126aa, >Q9NBB8... at 93.65%\n2 786aa, >Q9GV79... at 99.11%\n3 201aa, >Q9NBA9... at 93.03%\n4 223aa, >Q9NBC0... at 99.55%\n5 231aa, >Q9NBB9... at 99.13%\n6 8255aa, >A0A0B4KG35... *\n7 929aa, >A1ZA69... at 99.68%\n8 536aa, >Q9GV82... at 100.00%\n9 90aa, >Q9NBC5... at 100.00%\n10 577aa, >Q9GV81... at 93.59%\n11 157aa, >Q9NBB4... at 93.63%\n12 215aa, >Q9N630... at 98.60%\n13 140aa, >Q9NBB6... at 95.71%\n14 623aa, >Q9GV80... at 99.68%\n15 7905aa, >A0A0B4JD82... at 99.27%\n16 119aa, >Q9NBC6... at 100.00%\n17 929aa, >O01651... at 99.03%\n18 832aa, >O01653... at 99.64%\n19 576aa, >Q8MYR8... at 98.44%\n20 212aa, >Q9NBB0... at 91.98%\n21 913aa, >A0A0B4JD64... at 100.00%\n22 151aa, >Q9NBB7... at 99.34%\n23 128aa, >Q9NBC4... at 99.22%\n24 8216aa, >A0A0B4KF84... at 99.66%\n25 602aa, >A0A0B4JCU1... at 94.52%\n26 281aa, >Q9NBC2... at 98.22%\n27 569aa, >O01652... at 93.50%\n28 209aa, >Q9GTV0... at 96.65%\n29 210aa, >Q9N6N9... at 99.52%\n30 7944aa, >A1ZA72... at 99.61%\n\n>Cluster 100453\n0 85aa, >Q1PEF6... *\n\n```\n\nAll of the above sequences have >90% identity to at least one member of the cluster. A cluster with 30 members has strong representation of different lengths (~100 to >8000 residues), and more likely of several domains / functionalities as well. The very last cluster shown has a single member - and there are ~93 K others like it.", "2254588": "In this example, how can a sequence of 2404aa and another of 547aa both have > 90% similarity? If I had to guess these are the alignment scores against a reference for the cluster and there is a really small gap penalty? My knowledge on alignment is pretty limited to 2 sequence alignment so sorry if that was a silly question.\n\n> The problem will likely arise in execution. some proteins will have many homologous sequences, and others will have none.\n\nIs the problem here that we're splitting into the train/val set by group which won't take into account their size? Thus potentially leading to a different size diff between the train/val than expected?\n\nThanks!", "2254594": "I ran the command listed in the `README`:\n\n```bash\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\n```\n\nFrom a quick look I believe `--cov-mode 1` will also set `--cluster-mode 1` but I'll take a deeper look tomorrow and into CD-Hit. \n\nThanks!", "2255569": "Yes, these are all identity scores versus the reference sequence which has an asterisk (#8 in cluster 25, #6 in cluster 50). You may be mistaking sequence identity/similarity for overlap. Two sequences can be 100% identical even if one of them is 500 residues long and the other is 2000. The meaning of identity is not global, so gap penalties do not enter the picture here. They can align over 500 residues (100% overlap for the shorter sequence, 25% for the longer) and have 100% identity in that region.\n\nIf you take percent overlap into account when clustering (which `mmseqs` does), than a 500-residues sequence is not going to end up in the same cluster/group as a 2000-residue sequence. So when you split by groups, one of these groups may end up in train data, and the other in validation data. Yet if they are 100% identical, even over a limited length, chances are very good that at least some of their functionality will be shared. So there will be a data leak because some of train data will be in validation as well.\n\nIf we cluster just by identity and ignore the overlap, as I did, we may end up with some clusters that have many members, while the majority will have only one member. Splitting both categories uniformly between train and validation data will also be a non-trivial task.", "2256553": "I ran an experiment re splitting uniformly. Using the suggested clustering command on the mmseqs `README`:\n\n```sh\n# Assume the fasta file is the one from competition. This is a copy/paste from their docs\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1\n```\n\n`--cov-mode 1` should do clustering similarly to how you described above, which if I understood correctly should keep similar proteins together regardless of length.\n\n\n\nOnce that's done, I appended the representative protein ID to each row \n\n\n\nI then shuffled it based on the cluster 10000 times and plotted the ratio of rows between train/test (or val, rather):\n\n```py\ngss = GroupShuffleSplit(test_size=0.20, n_splits=10000, random_state=0).split(\n train_terms_df, groups=train_terms_df[\"cluster\"]\n)\n\ntrain_test_ratios = []\nfor train_idx, test_idx in tqdm(gss):\n train_test_ratios.append(len(train_idx) / len(test_idx))\n\nplt.figure(figsize=(20, 10))\nplt.hist(train_test_ratios, bins=400)\nplt.xlabel(\"Train to test ratio\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Distribution of train to test ratios\")\nplt.show()\n```\n\n\n\nSeems like an attempted split of 20% val results in an actual split of ~~23%-26%~~ (*Edit*: ~17%-21%, I did the math wrong), which seems pretty good!\n\nHowever, I'm using the suggested mmseqs settings. I don't really have enough domain knowledge to know which coverage/identity values are biologically significant. I'm guessing your `-min-seq-id` is lower than my `0.5`.\n\nIf you could provide the command you used to generate your clusters I can give it a try with those as well. Thanks!\n\n\n\n**EDIT:** I tried with `mmseqs easy-cluster {sequences_path} clusterRes tmp --min-seq-id 0.05 -c 0.9 --cov-mode 1` and got a similar result in terms of actual data split.", "2256596": "I suspect it may also be important to think about how the competition will actually be scored. AFAIK, many of the experimental annotations that are validated in the relevant time period may well be 'easy' to predict - that is, experimental confirmations of functions that are exactly what you'd expect by simple inference from sequence similarity. I'd hesitate to have a validation protocol that concentrated too much on making 'difficult' inferences, when getting a high recall on the 'easy' ones may be just as important to the final score.\n\nI think @tilii7 is quite right to say that it's not really possible to construct non-homologous clusters of sequences when many of the proteins are multi-domain, there will be just too many different domain-sharing combinations. In contrast, it's entirely feasible to construct non-homologous families of domains, as various databases have done.", "2256771": "Part of the clustering solution I showed above was done with `cd-hit`.\n\n cd-hit -i train_sequences.fasta -o train_sequences.90 -c 0.9\n\nIn this example, cluster summary would end up in a file `train_sequences.90.clstr`.", "2310792": "Here is some proposal on groups and folds.\nComments are welcome:\nhttps://www.kaggle.com/code/alexandervc/cafa5-23-groups-and-folds-diamond-igraph", "2373609": "Great discussion! I decided to try replicate the evaluation set-up: for each protein, I train on all labels from one subontology and predict all the labels in another subontology https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up" }, "source": "meta" }