I haven't had a chance to properly test the question posed in the title, and would like to hear opinions.
\nI have tried ProtT5, ESM2 650 million and ProtBert. Of the three tested language models, in my hands ProtT5 works best individually, followed by ESM2 and ProtBert. Yet adding them all together brings only a miniscule improvement in the local and public scores, while requiring considerably more resources to build a model. It is quite likely that this is due to my haste in the past week to test many things rather than testing any one of them properly.
\nMy best individual model (0.545) is ProtT5 embeddings + BLAST-like scores. Would love to hear the range of your best single models before ensembling.
", "messages": [ { "id": "2401817", "postDate": "08/21/2023 19:55:42", "content": "I haven't had a chance to properly test the question posed in the title, and would like to hear opinions.
\nI have tried ProtT5, ESM2 650 million and ProtBert. Of the three tested language models, in my hands ProtT5 works best individually, followed by ESM2 and ProtBert. Yet adding them all together brings only a miniscule improvement in the local and public scores, while requiring considerably more resources to build a model. It is quite likely that this is due to my haste in the past week to test many things rather than testing any one of them properly.
\nMy best individual model (0.545) is ProtT5 embeddings + BLAST-like scores. Would love to hear the range of your best single models before ensembling.
", "rawMarkdown": "I haven't had a chance to properly test the question posed in the title, and would like to hear opinions.\n\nI have tried ProtT5, ESM2 650 million and ProtBert. Of the three tested language models, in my hands ProtT5 works best individually, followed by ESM2 and ProtBert. Yet adding them all together brings only a miniscule improvement in the local and public scores, while requiring considerably more resources to build a model. It is quite likely that this is due to my haste in the past week to test many things rather than testing any one of them properly.\n\nMy best individual model (0.545) is ProtT5 embeddings + BLAST-like scores. Would love to hear the range of your best single models before ensembling.", "votes": null }, { "id": "2401822", "postDate": "08/21/2023 20:11:29", "content": "For me T5 and ESM 650M are best, they work fine separately, but it's a little bit better to combine them. Also I used information about aminos in each sequence, and bioencoder from hugging face.
\nESM2 3B always overfitting.
This is consistent with what I am seeing on this problem, and not necessarily with other applications of pLLMs. For example, ProtT5 and ESM 650M do combine well in secondary structure predictions. I guess it all boils down to whether a little bit better is worth the extra time to build a model. I would have probably pursued ProtT5+ESM2 had I had more time.
My best solution was a ESM2_3B based model: CV-0.563 and LB-0.553. Some dimensional reduction tecniques gave me good results.
", "rawMarkdown": "My best solution was a ESM2_3B based model: CV-0.563 and LB-0.553. Some dimensional reduction tecniques gave me good results.", "votes": null }, { "id": "2401870", "postDate": "08/21/2023 21:00:20", "content": "Im getting similar scores with t5 and ESM 3B separately (~0.53 in LB and ~0.54 in ~CV without duplicated proteins, 138_000 total samples in CV). I tried to combine them in a single model but didnt get any improvement.
", "rawMarkdown": "Im getting similar scores with t5 and ESM 3B separately (~0.53 in LB and ~0.54 in ~CV without duplicated proteins, 138_000 total samples in CV). I tried to combine them in a single model but didnt get any improvement.", "votes": null }, { "id": "2401884", "postDate": "08/21/2023 21:28:59", "content": "I think this could be due to the really dense nature of language vectors, and the fact that we have a large multi-class prediction. It is possible that for some classes pLLMs work well together, but for the others they may be fighting each other.
", "rawMarkdown": "I think this could be due to the really dense nature of language vectors, and the fact that we have a large multi-class prediction. It is possible that for some classes pLLMs work well together, but for the others they may be fighting each other.", "votes": null }, { "id": "2401887", "postDate": "08/21/2023 21:30:44", "content": "\n\nSome dimensional reduction tecniques gave me good results.
\n
This was something I thought about, but didn't have time to explore. It is unlikely that all 1000+ vectors are useful for this particular purpose, but one really has to spend time in order to find what features work.
", "rawMarkdown": "> Some dimensional reduction tecniques gave me good results.\n\nThis was something I thought about, but didn't have time to explore. It is unlikely that all 1000+ vectors are useful for this particular purpose, but one really has to spend time in order to find what features work.", "votes": null }, { "id": "2401962", "postDate": "08/21/2023 23:33:05", "content": "Three of my top models have come in the past 24 hours, which is expected given my late start. I have a pretty good idea among the unscored models which ones would have done well, but no more submissions. At least I will have them scored after we close for business.
\nGood luck to all of you. May you live to be in triple digits, but always rank in single or double digits on private LB!
", "rawMarkdown": "Three of my top models have come in the past 24 hours, which is expected given my late start. I have a pretty good idea among the unscored models which ones would have done well, but no more submissions. At least I will have them scored after we close for business.\n\nGood luck to all of you. May you live to be in triple digits, but always rank in single or double digits on private LB!", "votes": null }, { "id": "2401983", "postDate": "08/22/2023 00:15:24", "content": "I have had the best performance with T5 out of all 3, and didn't get any improvement with fusing them together, on my offline evaluation. Given the large number of GO terms to predict, the problem seems very sensitive to the data I choose to train on.
", "rawMarkdown": "I have had the best performance with T5 out of all 3, and didn't get any improvement with fusing them together, on my offline evaluation. Given the large number of GO terms to predict, the problem seems very sensitive to the data I choose to train on.", "votes": null }, { "id": "2402006", "postDate": "08/22/2023 00:34:29", "content": "Yes, I can vouch for that . T5 gave me the best results …all other individual and combinations just didnt match up to it. As someone already mentioned, ESM2,ProtBert are very dense and as well as normalized..
\nI have couple of questions/observations to validate ..
\na)Have you all thresholded the predictions ?
\nIn my case, I did an experiment with thresholding at .4 and .1
\n while threshold .4 gave me an LB of .32x but .1 bumped up the LB to something like .48x..does it mean that 'confidence' of the
\n model is not really playing much role in comparison to getting the 'GO Term' included in the prediction ?
(b)selection of 'label' terms to start is key
\n tried various techniques like top 1500,top 2000, sub-ont wise tops, sub-ont wise depth/levels etc.
\n onehot as well sparse categorical (entire GO, sub-ont wise etc)
c) though the competition ended 'just now', will it still be open and we can continue to submit for 'off the competition' evaluations ?
\n as I see it , its not actually over yet..we started with 'primary' and we got secondary/tertiary/quartenary to explore and pursue..
d) MLP models seem to have an upper hand than CNNs/LSTMs etc..may be due to my inability in terms of 'computing resources' to deploy larger/deeper CNNs/LSTMS like.
\nAny thoughts you can share ?
", "rawMarkdown": "Yes, I can vouch for that . T5 gave me the best results ...all other individual and combinations just didnt match up to it. As someone already mentioned, ESM2,ProtBert are very dense and as well as normalized..\n\nI have couple of questions/observations to validate ..\n\na)Have you all thresholded the predictions ? \n\n In my case, I did an experiment with thresholding at .4 and .1\n while threshold .4 gave me an LB of .32x but .1 bumped up the LB to something like .48x..does it mean that 'confidence' of the \n model is not really playing much role in comparison to getting the 'GO Term' included in the prediction ? \n\n(b)selection of 'label' terms to start is key\n tried various techniques like top 1500,top 2000, sub-ont wise tops, sub-ont wise depth/levels etc. \n onehot as well sparse categorical (entire GO, sub-ont wise etc)\n\nc) though the competition ended 'just now', will it still be open and we can continue to submit for 'off the competition' evaluations ?\n as I see it , its not actually over yet..we started with 'primary' and we got secondary/tertiary/quartenary to explore and pursue..\n \nd) MLP models seem to have an upper hand than CNNs/LSTMs etc..may be due to my inability in terms of 'computing resources' to deploy larger/deeper CNNs/LSTMS like.\n\nAny thoughts you can share ?", "votes": null }, { "id": "2402309", "postDate": "08/22/2023 05:11:48", "content": "When I was getting more than 10 million predictions, I thresholded them to remove p<0.1. Otherwise, I was using a p<0.03 threshold to discard predictions.
\nI was also hoping to submit some of my models that didn't make it, but it seems that submissions are disabled.
", "rawMarkdown": "When I was getting more than 10 million predictions, I thresholded them to remove p<0.1. Otherwise, I was using a p<0.03 threshold to discard predictions.\n\nI was also hoping to submit some of my models that didn't make it, but it seems that submissions are disabled.", "votes": null }, { "id": "2402739", "postDate": "08/22/2023 10:01:33", "content": "I have first reduced the GO terms by ont, by frequency, for training.
\nThen, when predicting, I found that better public scores are obtained when I choose top most frequent terms and just predict on those, rather than predicting where the model is most confident. Choice and ordering of what to predict on have vast influence on public score. (e.g. do we filter 0.0 IA weight, and similar)
I didn't have enough time to play with the different number of submissions per protein, always predicting 500/ont/protein. I later realized that might be too much and potentially reducing the public score.
", "rawMarkdown": "I have first reduced the GO terms by ont, by frequency, for training.\nThen, when predicting, I found that better public scores are obtained when I choose top most frequent terms and just predict on those, rather than predicting where the model is most confident. Choice and ordering of what to predict on have vast influence on public score. (e.g. do we filter 0.0 IA weight, and similar)\n\nI didn't have enough time to play with the different number of submissions per protein, always predicting 500/ont/protein. I later realized that might be too much and potentially reducing the public score.", "votes": null }, { "id": "2402988", "postDate": "08/22/2023 12:49:36", "content": "Hi @tilii7,
\nFrom my experiments, I also observe T5 consistently performs better than ESM2. As for ProtBert, I failed to make it work like the other two. Also, the combination of T5 and ESM2 gives a little boost to the overall performance for both CV and LB. My approach is simply project them to the same dimension and add them up. One of the example is shown as follows:
\n![\"AUROC.png\"](\"https://i.postimg.cc/Z5z0whr5/AUROC.png\")
The figure shows the AUROC curve during training, red for ESM2, green for T5 and black for the simple combination. I also find that AUROC has a great correlation with the official metric, F1-Max. My best experiment yields CV 0.57558 and LB 0.52832.
As can be seen, there's always a large gap between my local CV score and LB score (though, they seem to sync well). Both the random K-Fold and GroupKFold based on sequence similarity suffer. If you don't mind me asking, which CV scheme did you choose in your experiments?
\nThanks a lot for all your insightful sharings!
", "rawMarkdown": "Hi @tilii7,\n\nFrom my experiments, I also observe `T5` consistently performs better than `ESM2`. As for `ProtBert`, I failed to make it work like the other two. Also, the combination of `T5` and `ESM2` gives a little boost to the overall performance for both CV and LB. My approach is simply project them to the same dimension and add them up. One of the example is shown as follows:\n[](https://postimg.cc/LnvHswCc)\n\nThe figure shows the AUROC curve during training, red for `ESM2`, green for `T5` and black for the simple combination. I also find that AUROC has a great correlation with the official metric, F1-Max. My best experiment yields CV 0.57558 and LB 0.52832.\n \nAs can be seen, there's always a large gap between my local CV score and LB score (though, they seem to sync well). Both the random K-Fold and GroupKFold based on sequence similarity suffer. If you don't mind me asking, which CV scheme did you choose in your experiments? \n\nThanks a lot for all your insightful sharings!", "votes": null }, { "id": "2403732", "postDate": "08/22/2023 20:21:00", "content": "Here I already shared https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491 some results that combining t5+esm2 shows quite an uplift for Ridge.
\nAnd some similar effects I observed for nn-models, however later looking more closely,
\nit seems situation is more complicated - for some models local CV score a bit DECREASES from concationation of t5 to esm2-size-1280
Here are some quotes of the results:
\nversion-of-NB Model/EmbdesID CV
\n 214 2pmlp t5 0.4941
\n215 2pmlp ['t5' , 'esm2S1280' ] 0.4906 --- decrease from concat of t5 to esm2
208 2skMLP' t5 only 0.4956
\n224 2skMLP 't5', 'esm2S1280' 0.4947 --- decrease from concat of t5 to esm2
pMLP - pytorch 2-layer MLP with Droupouts and BatchNormalizations
\nskmlp - [500,1000], 'activation':'relu - sklearn MLP-regressor
I will make notebook public later.
", "rawMarkdown": "Here I already shared https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491 some results that combining t5+esm2 shows quite an uplift for Ridge.\nAnd some similar effects I observed for nn-models, however later looking more closely,\nit seems situation is more complicated - for some models local CV score a bit DECREASES from concationation of t5 to esm2-size-1280\n\nHere are some quotes of the results: \n\nversion-of-NB Model/EmbdesID CV \n 214 2pmlp t5\t 0.4941\n215 2pmlp ['t5' , 'esm2S1280' ] \t0.4906 --- decrease from concat of t5 to esm2\n\n208 2skMLP' t5 only\t 0.4956\n224 2skMLP 't5', 'esm2S1280'\t 0.4947 --- decrease from concat of t5 to esm2\n\n\npMLP - pytorch 2-layer MLP with Droupouts and BatchNormalizations\nskmlp - [500,1000], 'activation':'relu - sklearn MLP-regressor\n\nI will make notebook public later.", "votes": null }, { "id": "2403812", "postDate": "08/22/2023 21:33:06", "content": "This is interesting, but here is something to consider. With a neural network I get public LB 0.515 from ProtT5 alone, and 0.519 when adding ESM2 embeddings. That means the prediction is already maxed out from a single embedding. If Ridge can't get that level of prediction from a single embedding, which seems to be the case, it could mean that there is enough room for improvement when adding subsequent embeddings. It doesn't necessarily mean that combining embeddings works as a general rule.
", "rawMarkdown": "This is interesting, but here is something to consider. With a neural network I get public LB 0.515 from ProtT5 alone, and 0.519 when adding ESM2 embeddings. That means the prediction is already maxed out from a single embedding. If Ridge can't get that level of prediction from a single embedding, which seems to be the case, it could mean that there is enough room for improvement when adding subsequent embeddings. It doesn't necessarily mean that combining embeddings works as a general rule.", "votes": null }, { "id": "2403864", "postDate": "08/22/2023 22:42:03", "content": "Yes, using IA was also on the pipeline.. as far as I understood, deeper you go in ontology the 'term' will be more specific and thus should carry more weight.
\nBut my certain experiments(selecting terms based on depth/level from ont) revealed that it wont affect the score much(positively) as those terms might not have been experimentally validated at this point in time..
\nso predictions remain just that, something not validated..
Wish I had time to try IA weights, but was short on time and didn't have a quick idea how to code it.
", "rawMarkdown": "Wish I had time to try IA weights, but was short on time and didn't have a quick idea how to code it.", "votes": null }, { "id": "2403871", "postDate": "08/22/2023 22:47:46", "content": "is getting hold of whole test data set an option, so that we can continue to experiment locally ? I guess , it is dynamic as annotations will get added over time as they are experimentally validated..
", "rawMarkdown": "is getting hold of whole test data set an option, so that we can continue to experiment locally ? I guess , it is dynamic as annotations will get added over time as they are experimentally validated..", "votes": null }, { "id": "2403901", "postDate": "08/22/2023 23:49:25", "content": "That looks very clear, and more convincing than my empirical observations. Well done!
", "rawMarkdown": "That looks very clear, and more convincing than my empirical observations. Well done!", "votes": null }, { "id": "2404476", "postDate": "08/23/2023 09:20:42", "content": "Yes, that is my impression also, i.e.
\nRidge by itself is a weak model , and so adding more features makes it better both on CV and LB.
While for NN models are stronger and they may get good results from T5 only, and adding esm2 not always improves the score. I mean sometimes it may improve, sometimes not - depending on NN or CV scheme.
", "rawMarkdown": "Yes, that is my impression also, i.e.\nRidge by itself is a weak model , and so adding more features makes it better both on CV and LB.\n\nWhile for NN models are stronger and they may get good results from T5 only, and adding esm2 not always improves the score. I mean sometimes it may improve, sometimes not - depending on NN or CV scheme.", "votes": null } ], "comments": [ { "id": 2401822, "author_name": "bejeweled", "author_url": "", "post_date": "08/21/2023 20:11:29", "content": "For me T5 and ESM 650M are best, they work fine separately, but it's a little bit better to combine them. Also I used information about aminos in each sequence, and bioencoder from hugging face.
\nESM2 3B always overfitting.
This is consistent with what I am seeing on this problem, and not necessarily with other applications of pLLMs. For example, ProtT5 and ESM 650M do combine well in secondary structure predictions. I guess it all boils down to whether a little bit better is worth the extra time to build a model. I would have probably pursued ProtT5+ESM2 had I had more time.
My best solution was a ESM2_3B based model: CV-0.563 and LB-0.553. Some dimensional reduction tecniques gave me good results.
", "votes": null, "replies": [ { "id": 2401887, "author_name": "tilii7", "author_url": "", "post_date": "08/21/2023 21:30:44", "content": "\n\nSome dimensional reduction tecniques gave me good results.
\n
This was something I thought about, but didn't have time to explore. It is unlikely that all 1000+ vectors are useful for this particular purpose, but one really has to spend time in order to find what features work.
", "votes": null, "replies": [] } ] }, { "id": 2401870, "author_name": "mpp1998", "author_url": "", "post_date": "08/21/2023 21:00:20", "content": "Im getting similar scores with t5 and ESM 3B separately (~0.53 in LB and ~0.54 in ~CV without duplicated proteins, 138_000 total samples in CV). I tried to combine them in a single model but didnt get any improvement.
", "votes": null, "replies": [ { "id": 2401884, "author_name": "tilii7", "author_url": "", "post_date": "08/21/2023 21:28:59", "content": "I think this could be due to the really dense nature of language vectors, and the fact that we have a large multi-class prediction. It is possible that for some classes pLLMs work well together, but for the others they may be fighting each other.
", "votes": null, "replies": [] } ] }, { "id": 2401962, "author_name": "tilii7", "author_url": "", "post_date": "08/21/2023 23:33:05", "content": "Three of my top models have come in the past 24 hours, which is expected given my late start. I have a pretty good idea among the unscored models which ones would have done well, but no more submissions. At least I will have them scored after we close for business.
\nGood luck to all of you. May you live to be in triple digits, but always rank in single or double digits on private LB!
", "votes": null, "replies": [] }, { "id": 2401983, "author_name": "jedannijenula", "author_url": "", "post_date": "08/22/2023 00:15:24", "content": "I have had the best performance with T5 out of all 3, and didn't get any improvement with fusing them together, on my offline evaluation. Given the large number of GO terms to predict, the problem seems very sensitive to the data I choose to train on.
", "votes": null, "replies": [] }, { "id": 2402006, "author_name": "sunilkrishnan", "author_url": "", "post_date": "08/22/2023 00:34:29", "content": "Yes, I can vouch for that . T5 gave me the best results …all other individual and combinations just didnt match up to it. As someone already mentioned, ESM2,ProtBert are very dense and as well as normalized..
\nI have couple of questions/observations to validate ..
\na)Have you all thresholded the predictions ?
\nIn my case, I did an experiment with thresholding at .4 and .1
\n while threshold .4 gave me an LB of .32x but .1 bumped up the LB to something like .48x..does it mean that 'confidence' of the
\n model is not really playing much role in comparison to getting the 'GO Term' included in the prediction ?
(b)selection of 'label' terms to start is key
\n tried various techniques like top 1500,top 2000, sub-ont wise tops, sub-ont wise depth/levels etc.
\n onehot as well sparse categorical (entire GO, sub-ont wise etc)
c) though the competition ended 'just now', will it still be open and we can continue to submit for 'off the competition' evaluations ?
\n as I see it , its not actually over yet..we started with 'primary' and we got secondary/tertiary/quartenary to explore and pursue..
d) MLP models seem to have an upper hand than CNNs/LSTMs etc..may be due to my inability in terms of 'computing resources' to deploy larger/deeper CNNs/LSTMS like.
\nAny thoughts you can share ?
", "votes": null, "replies": [ { "id": 2402309, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 05:11:48", "content": "When I was getting more than 10 million predictions, I thresholded them to remove p<0.1. Otherwise, I was using a p<0.03 threshold to discard predictions.
\nI was also hoping to submit some of my models that didn't make it, but it seems that submissions are disabled.
", "votes": null, "replies": [ { "id": 2402739, "author_name": "jedannijenula", "author_url": "", "post_date": "08/22/2023 10:01:33", "content": "I have first reduced the GO terms by ont, by frequency, for training.
\nThen, when predicting, I found that better public scores are obtained when I choose top most frequent terms and just predict on those, rather than predicting where the model is most confident. Choice and ordering of what to predict on have vast influence on public score. (e.g. do we filter 0.0 IA weight, and similar)
I didn't have enough time to play with the different number of submissions per protein, always predicting 500/ont/protein. I later realized that might be too much and potentially reducing the public score.
", "votes": null, "replies": [ { "id": 2403864, "author_name": "sunilkrishnan", "author_url": "", "post_date": "08/22/2023 22:42:03", "content": "Yes, using IA was also on the pipeline.. as far as I understood, deeper you go in ontology the 'term' will be more specific and thus should carry more weight.
\nBut my certain experiments(selecting terms based on depth/level from ont) revealed that it wont affect the score much(positively) as those terms might not have been experimentally validated at this point in time..
\nso predictions remain just that, something not validated..
Wish I had time to try IA weights, but was short on time and didn't have a quick idea how to code it.
", "votes": null, "replies": [] } ] }, { "id": 2403871, "author_name": "sunilkrishnan", "author_url": "", "post_date": "08/22/2023 22:47:46", "content": "is getting hold of whole test data set an option, so that we can continue to experiment locally ? I guess , it is dynamic as annotations will get added over time as they are experimentally validated..
", "votes": null, "replies": [] } ] } ] }, { "id": 2402988, "author_name": "abaojiang", "author_url": "", "post_date": "08/22/2023 12:49:36", "content": "Hi @tilii7,
\nFrom my experiments, I also observe T5 consistently performs better than ESM2. As for ProtBert, I failed to make it work like the other two. Also, the combination of T5 and ESM2 gives a little boost to the overall performance for both CV and LB. My approach is simply project them to the same dimension and add them up. One of the example is shown as follows:
\n
The figure shows the AUROC curve during training, red for ESM2, green for T5 and black for the simple combination. I also find that AUROC has a great correlation with the official metric, F1-Max. My best experiment yields CV 0.57558 and LB 0.52832.
As can be seen, there's always a large gap between my local CV score and LB score (though, they seem to sync well). Both the random K-Fold and GroupKFold based on sequence similarity suffer. If you don't mind me asking, which CV scheme did you choose in your experiments?
\nThanks a lot for all your insightful sharings!
", "votes": null, "replies": [ { "id": 2403901, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 23:49:25", "content": "That looks very clear, and more convincing than my empirical observations. Well done!
", "votes": null, "replies": [] } ] }, { "id": 2403732, "author_name": "alexandervc", "author_url": "", "post_date": "08/22/2023 20:21:00", "content": "Here I already shared https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491 some results that combining t5+esm2 shows quite an uplift for Ridge.
\nAnd some similar effects I observed for nn-models, however later looking more closely,
\nit seems situation is more complicated - for some models local CV score a bit DECREASES from concationation of t5 to esm2-size-1280
Here are some quotes of the results:
\nversion-of-NB Model/EmbdesID CV
\n 214 2pmlp t5 0.4941
\n215 2pmlp ['t5' , 'esm2S1280' ] 0.4906 --- decrease from concat of t5 to esm2
208 2skMLP' t5 only 0.4956
\n224 2skMLP 't5', 'esm2S1280' 0.4947 --- decrease from concat of t5 to esm2
pMLP - pytorch 2-layer MLP with Droupouts and BatchNormalizations
\nskmlp - [500,1000], 'activation':'relu - sklearn MLP-regressor
I will make notebook public later.
", "votes": null, "replies": [ { "id": 2403812, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 21:33:06", "content": "This is interesting, but here is something to consider. With a neural network I get public LB 0.515 from ProtT5 alone, and 0.519 when adding ESM2 embeddings. That means the prediction is already maxed out from a single embedding. If Ridge can't get that level of prediction from a single embedding, which seems to be the case, it could mean that there is enough room for improvement when adding subsequent embeddings. It doesn't necessarily mean that combining embeddings works as a general rule.
", "votes": null, "replies": [ { "id": 2404476, "author_name": "alexandervc", "author_url": "", "post_date": "08/23/2023 09:20:42", "content": "Yes, that is my impression also, i.e.
\nRidge by itself is a weak model , and so adding more features makes it better both on CV and LB.
While for NN models are stronger and they may get good results from T5 only, and adding esm2 not always improves the score. I mean sometimes it may improve, sometimes not - depending on NN or CV scheme.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2401817": "I haven't had a chance to properly test the question posed in the title, and would like to hear opinions.\n\nI have tried ProtT5, ESM2 650 million and ProtBert. Of the three tested language models, in my hands ProtT5 works best individually, followed by ESM2 and ProtBert. Yet adding them all together brings only a miniscule improvement in the local and public scores, while requiring considerably more resources to build a model. It is quite likely that this is due to my haste in the past week to test many things rather than testing any one of them properly.\n\nMy best individual model (0.545) is ProtT5 embeddings + BLAST-like scores. Would love to hear the range of your best single models before ensembling.", "2401822": "For me T5 and ESM 650M are best, they work fine separately, but it's a little bit better to combine them. Also I used information about aminos in each sequence, and bioencoder from hugging face.\nESM2 3B always overfitting.", "2401836": "This is consistent with what I am seeing on this problem, and not necessarily with other applications of pLLMs. For example, ProtT5 and ESM 650M do combine well in secondary structure predictions. I guess it all boils down to whether `a little bit better` is worth the extra time to build a model. I would have probably pursued ProtT5+ESM2 had I had more time.", "2401869": "My best solution was a ESM2_3B based model: CV-0.563 and LB-0.553. Some dimensional reduction tecniques gave me good results.", "2401870": "Im getting similar scores with t5 and ESM 3B separately (~0.53 in LB and ~0.54 in ~CV without duplicated proteins, 138_000 total samples in CV). I tried to combine them in a single model but didnt get any improvement.", "2401884": "I think this could be due to the really dense nature of language vectors, and the fact that we have a large multi-class prediction. It is possible that for some classes pLLMs work well together, but for the others they may be fighting each other.", "2401887": "> Some dimensional reduction tecniques gave me good results.\n\nThis was something I thought about, but didn't have time to explore. It is unlikely that all 1000+ vectors are useful for this particular purpose, but one really has to spend time in order to find what features work.", "2401962": "Three of my top models have come in the past 24 hours, which is expected given my late start. I have a pretty good idea among the unscored models which ones would have done well, but no more submissions. At least I will have them scored after we close for business.\n\nGood luck to all of you. May you live to be in triple digits, but always rank in single or double digits on private LB!", "2401983": "I have had the best performance with T5 out of all 3, and didn't get any improvement with fusing them together, on my offline evaluation. Given the large number of GO terms to predict, the problem seems very sensitive to the data I choose to train on.", "2402006": "Yes, I can vouch for that . T5 gave me the best results ...all other individual and combinations just didnt match up to it. As someone already mentioned, ESM2,ProtBert are very dense and as well as normalized..\n\nI have couple of questions/observations to validate ..\n\na)Have you all thresholded the predictions ? \n\n In my case, I did an experiment with thresholding at .4 and .1\n while threshold .4 gave me an LB of .32x but .1 bumped up the LB to something like .48x..does it mean that 'confidence' of the \n model is not really playing much role in comparison to getting the 'GO Term' included in the prediction ? \n\n(b)selection of 'label' terms to start is key\n tried various techniques like top 1500,top 2000, sub-ont wise tops, sub-ont wise depth/levels etc. \n onehot as well sparse categorical (entire GO, sub-ont wise etc)\n\nc) though the competition ended 'just now', will it still be open and we can continue to submit for 'off the competition' evaluations ?\n as I see it , its not actually over yet..we started with 'primary' and we got secondary/tertiary/quartenary to explore and pursue..\n \nd) MLP models seem to have an upper hand than CNNs/LSTMs etc..may be due to my inability in terms of 'computing resources' to deploy larger/deeper CNNs/LSTMS like.\n\nAny thoughts you can share ?", "2402309": "When I was getting more than 10 million predictions, I thresholded them to remove p<0.1. Otherwise, I was using a p<0.03 threshold to discard predictions.\n\nI was also hoping to submit some of my models that didn't make it, but it seems that submissions are disabled.", "2402739": "I have first reduced the GO terms by ont, by frequency, for training.\nThen, when predicting, I found that better public scores are obtained when I choose top most frequent terms and just predict on those, rather than predicting where the model is most confident. Choice and ordering of what to predict on have vast influence on public score. (e.g. do we filter 0.0 IA weight, and similar)\n\nI didn't have enough time to play with the different number of submissions per protein, always predicting 500/ont/protein. I later realized that might be too much and potentially reducing the public score.", "2402988": "Hi @tilii7,\n\nFrom my experiments, I also observe `T5` consistently performs better than `ESM2`. As for `ProtBert`, I failed to make it work like the other two. Also, the combination of `T5` and `ESM2` gives a little boost to the overall performance for both CV and LB. My approach is simply project them to the same dimension and add them up. One of the example is shown as follows:\n[](https://postimg.cc/LnvHswCc)\n\nThe figure shows the AUROC curve during training, red for `ESM2`, green for `T5` and black for the simple combination. I also find that AUROC has a great correlation with the official metric, F1-Max. My best experiment yields CV 0.57558 and LB 0.52832.\n \nAs can be seen, there's always a large gap between my local CV score and LB score (though, they seem to sync well). Both the random K-Fold and GroupKFold based on sequence similarity suffer. If you don't mind me asking, which CV scheme did you choose in your experiments? \n\nThanks a lot for all your insightful sharings!", "2403732": "Here I already shared https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491 some results that combining t5+esm2 shows quite an uplift for Ridge.\nAnd some similar effects I observed for nn-models, however later looking more closely,\nit seems situation is more complicated - for some models local CV score a bit DECREASES from concationation of t5 to esm2-size-1280\n\nHere are some quotes of the results: \n\nversion-of-NB Model/EmbdesID CV \n 214 2pmlp t5\t 0.4941\n215 2pmlp ['t5' , 'esm2S1280' ] \t0.4906 --- decrease from concat of t5 to esm2\n\n208 2skMLP' t5 only\t 0.4956\n224 2skMLP 't5', 'esm2S1280'\t 0.4947 --- decrease from concat of t5 to esm2\n\n\npMLP - pytorch 2-layer MLP with Droupouts and BatchNormalizations\nskmlp - [500,1000], 'activation':'relu - sklearn MLP-regressor\n\nI will make notebook public later.", "2403812": "This is interesting, but here is something to consider. With a neural network I get public LB 0.515 from ProtT5 alone, and 0.519 when adding ESM2 embeddings. That means the prediction is already maxed out from a single embedding. If Ridge can't get that level of prediction from a single embedding, which seems to be the case, it could mean that there is enough room for improvement when adding subsequent embeddings. It doesn't necessarily mean that combining embeddings works as a general rule.", "2403864": "Yes, using IA was also on the pipeline.. as far as I understood, deeper you go in ontology the 'term' will be more specific and thus should carry more weight.\nBut my certain experiments(selecting terms based on depth/level from ont) revealed that it wont affect the score much(positively) as those terms might not have been experimentally validated at this point in time..\nso predictions remain just that, something not validated..", "2403870": "Wish I had time to try IA weights, but was short on time and didn't have a quick idea how to code it.", "2403871": "is getting hold of whole test data set an option, so that we can continue to experiment locally ? I guess , it is dynamic as annotations will get added over time as they are experimentally validated..", "2403901": "That looks very clear, and more convincing than my empirical observations. Well done!", "2404476": "Yes, that is my impression also, i.e.\nRidge by itself is a weak model , and so adding more features makes it better both on CV and LB.\n\nWhile for NN models are stronger and they may get good results from T5 only, and adding esm2 not always improves the score. I mean sometimes it may improve, sometimes not - depending on NN or CV scheme." }, "source": "meta" }