![\"An](\"https://i.ibb.co/y40qxc4/train-CAFA-sparse-3-UMAP.png\")
There have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions. I think the use of profiles for protein families has also been discussed, and I'd be surprised if nobody actually used them in their models even though I can't find a public notebook that does so.
\nI decided to search train and test proteins against ~30,000 protein profiles (COG, KOG, SMART, TIGRFams v15 and Pfam v35 databases). Meant to include ~70,000 PDB profiles, but ran out of time. The idea is that each protein is compared with these profiles, and their significant e-values (expressed as -log10[evalue]) were entered into a corresponding spot in the matrix. The bad news is that this matrix's dimensions are ~140,000 x 30,000 which is a lot for any method. The good news is that most proteins match 10 or fewer profiles, so the matrix has ~0.02% sparsity (similar to recommender matrices). As such, it can readily be fed into Tensorflow without blowing out the GPU memory (but still requires a solid size GPU). This works reasonably well: without any tuning, this approach gives ~0.48 on public LB. I think it could be pushed to 0.5 by some tuning, which is worse than 0.545 obtained by ProtT5 embedding but not crazy worse.
\nSignificantly, this gigantic sparse matrix clearly creates clusters, which can be seen below from 2D UMAP embedding for CAFA5 train data. Data points are colored by order in which they appear in train data, and this plot is comparable to the second plot from top in this thread.
\nHopefully I am not the only one who sees an eyeball in this plot, minus the optic nerve. There are many small clusters on the plot's outskirts. Even in the central part (the eye's \"body\") there is some kind of structure in data point organization, even though they are not separated cleanly enough to see cluster borders.
\n
EDIT: I decided to add the same plot for test data, also colored by the order of data points. This can be compared to the top plot in the same post I referenced above.
\n![\"An](\"https://i.ibb.co/THWVTjD/test-CAFA-sparse-3-UMAP.png\")
There have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions. I think the use of profiles for protein families has also been discussed, and I'd be surprised if nobody actually used them in their models even though I can't find a public notebook that does so.
\nI decided to search train and test proteins against ~30,000 protein profiles (COG, KOG, SMART, TIGRFams v15 and Pfam v35 databases). Meant to include ~70,000 PDB profiles, but ran out of time. The idea is that each protein is compared with these profiles, and their significant e-values (expressed as -log10[evalue]) were entered into a corresponding spot in the matrix. The bad news is that this matrix's dimensions are ~140,000 x 30,000 which is a lot for any method. The good news is that most proteins match 10 or fewer profiles, so the matrix has ~0.02% sparsity (similar to recommender matrices). As such, it can readily be fed into Tensorflow without blowing out the GPU memory (but still requires a solid size GPU). This works reasonably well: without any tuning, this approach gives ~0.48 on public LB. I think it could be pushed to 0.5 by some tuning, which is worse than 0.545 obtained by ProtT5 embedding but not crazy worse.
\nSignificantly, this gigantic sparse matrix clearly creates clusters, which can be seen below from 2D UMAP embedding for CAFA5 train data. Data points are colored by order in which they appear in train data, and this plot is comparable to the second plot from top in this thread.
\nHopefully I am not the only one who sees an eyeball in this plot, minus the optic nerve. There are many small clusters on the plot's outskirts. Even in the central part (the eye's \"body\") there is some kind of structure in data point organization, even though they are not separated cleanly enough to see cluster borders.
\n
EDIT: I decided to add the same plot for test data, also colored by the order of data points. This can be compared to the top plot in the same post I referenced above.
\n
Not sure I would agree that it's commonly accepted that pLMs are better than expert (which admittedly may be the important word) use of curated profiles/HMMs. Better than automated BLAST/PSI-BLAST searches - yes, probably. One thing that pLMs do not understand well is the fact that proteins are often divided into distinct functional domains, and that even the order of those domains can influence overall function. That may well be something we learn from this experiment - though it's clear that many many more people here have used pLMs, so that could bias the results somewhat in favor of pLMs. Pre-trained pLMs were actually used in the last CAFA, and did OK, but did not rank that highly in the final assessment. Of course the pLMs have improved since then, and people have clearly been trying many different ways of combining them etc., so who knows what we'll see this time.
", "rawMarkdown": "Not sure I would agree that it's commonly accepted that pLMs are better than expert (which admittedly may be the important word) use of curated profiles/HMMs. Better than automated BLAST/PSI-BLAST searches - yes, probably. One thing that pLMs do not understand well is the fact that proteins are often divided into distinct functional domains, and that even the order of those domains can influence overall function. That may well be something we learn from this experiment - though it's clear that many many more people here have used pLMs, so that could bias the results somewhat in favor of pLMs. Pre-trained pLMs were actually used in the last CAFA, and did OK, but did not rank that highly in the final assessment. Of course the pLMs have improved since then, and people have clearly been trying many different ways of combining them etc., so who knows what we'll see this time.", "votes": null }, { "id": "2408262", "postDate": "08/25/2023 14:29:41", "content": "You bring up many good points, especially about domains. My main argument was about most common homology methods, as stated in the first sentence.
\n\n\nThere have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions.
\n
In many applications - SS prediction, PTMs, signal peptides - pLMs have proven better than sequence profile methods as well. One would very likely do better by combining the two categories of models than with either one individually.
", "rawMarkdown": "You bring up many good points, especially about domains. My main argument was about most common homology methods, as stated in the first sentence.\n\n> There have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions.\n\nIn many applications - SS prediction, PTMs, signal peptides - pLMs have proven better than sequence profile methods as well. One would very likely do better by combining the two categories of models than with either one individually.", "votes": null } ], "comments": [ { "id": 2408053, "author_name": "biodav", "author_url": "", "post_date": "08/25/2023 12:08:09", "content": "Not sure I would agree that it's commonly accepted that pLMs are better than expert (which admittedly may be the important word) use of curated profiles/HMMs. Better than automated BLAST/PSI-BLAST searches - yes, probably. One thing that pLMs do not understand well is the fact that proteins are often divided into distinct functional domains, and that even the order of those domains can influence overall function. That may well be something we learn from this experiment - though it's clear that many many more people here have used pLMs, so that could bias the results somewhat in favor of pLMs. Pre-trained pLMs were actually used in the last CAFA, and did OK, but did not rank that highly in the final assessment. Of course the pLMs have improved since then, and people have clearly been trying many different ways of combining them etc., so who knows what we'll see this time.
", "votes": null, "replies": [ { "id": 2408262, "author_name": "tilii7", "author_url": "", "post_date": "08/25/2023 14:29:41", "content": "You bring up many good points, especially about domains. My main argument was about most common homology methods, as stated in the first sentence.
\n\n\nThere have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions.
\n
In many applications - SS prediction, PTMs, signal peptides - pLMs have proven better than sequence profile methods as well. One would very likely do better by combining the two categories of models than with either one individually.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2403948": "There have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions. I think the use of profiles for protein families has also been discussed, and I'd be surprised if nobody actually used them in their models even though I can't find a public notebook that does so.\n\nI decided to search train and test proteins against ~30,000 protein profiles (COG, KOG, SMART, TIGRFams v15 and Pfam v35 databases). Meant to include ~70,000 PDB profiles, but ran out of time. The idea is that each protein is compared with these profiles, and their significant e-values (expressed as -log10[evalue]) were entered into a corresponding spot in the matrix. The bad news is that this matrix's dimensions are ~140,000 x 30,000 which is a lot for any method. The good news is that most proteins match 10 or fewer profiles, so the matrix has ~0.02% sparsity (similar to recommender matrices). As such, it can readily be fed into Tensorflow without blowing out the GPU memory (but still requires a solid size GPU). This works reasonably well: without any tuning, this approach gives ~0.48 on public LB. I think it could be pushed to 0.5 by some tuning, which is worse than 0.545 obtained by ProtT5 embedding but not crazy worse.\n\nSignificantly, this gigantic sparse matrix clearly creates clusters, which can be seen below from 2D UMAP embedding for CAFA5 train data. Data points are colored by order in which they appear in train data, and this plot is comparable to the second plot from top in [**this thread**](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406336).\n\nHopefully I am not the only one who sees an eyeball in this plot, minus the optic nerve. There are many small clusters on the plot's outskirts. Even in the central part (the eye's \"body\") there is some kind of structure in data point organization, even though they are not separated cleanly enough to see cluster borders.\n\n\n\n**EDIT:** I decided to add the same plot for test data, also colored by the order of data points. This can be compared to the top plot in the same post I referenced above.\n\n", "2408053": "Not sure I would agree that it's commonly accepted that pLMs are better than expert (which admittedly may be the important word) use of curated profiles/HMMs. Better than automated BLAST/PSI-BLAST searches - yes, probably. One thing that pLMs do not understand well is the fact that proteins are often divided into distinct functional domains, and that even the order of those domains can influence overall function. That may well be something we learn from this experiment - though it's clear that many many more people here have used pLMs, so that could bias the results somewhat in favor of pLMs. Pre-trained pLMs were actually used in the last CAFA, and did OK, but did not rank that highly in the final assessment. Of course the pLMs have improved since then, and people have clearly been trying many different ways of combining them etc., so who knows what we'll see this time.", "2408262": "You bring up many good points, especially about domains. My main argument was about most common homology methods, as stated in the first sentence.\n\n> There have been discussions using BLAST and Diamond to augment predictions, but it is commonly accepted that protein language model embeddings have surpassed them for functional predictions.\n\nIn many applications - SS prediction, PTMs, signal peptides - pLMs have proven better than sequence profile methods as well. One would very likely do better by combining the two categories of models than with either one individually." }, "source": "meta" }