Firstly I want to thank the organizers for a very nice competition - having it based on Kaggle opened up submissions to a wider range of people compared to previous iterations when mainly specialist labs showed results so the wider net of people being exposed to CAFA is really gratifying to see. My team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made. Both of us are molecular biologists with only limited bioinformatics and machine learning experience - so this competition was something of a trial by fire and we learned a lot from taking part.
\nWe used these datasets in our model:
\n1) Prot-T5, ESM2, and Ankh Protein Language Model (PLM) embeddings. We carried out no further modifications or finetuning on the output of PLMs, only conversion to float32 to save memory.
\n2) A single binary matrix representing species taxonomy for each protein.
\n3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein
For GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model to classify by taking the top N labels sorted by IA*frequency.
\nDatasets were sorted in alphabetical order to ensure identical ordering across datasets as well as removal of single duplicate row in test data.
\nWe made a simple dense neural network in keras that takes these as separate inputs before concatenating them into a final layer that attempts to predict every label from a single GO ontology/domain. We used binary cross entropy (BCE) as our loss function and Adam optimizer with lr=0.0003 and a simple lr scheduler. We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.
\nOur model was trained with KFold = 5 across 5 different random splits of the training data before a simple average of predictions was made for each ontology/domain and the results concatenated.
\nOne interesting thing we observed is that Ridge regression sometimes performed better on PLM embeddings compared to a neural network for some ontologies, which was unexpected and might indicate underfitting for the neural network.
\nProtBert embeddings performed poorly and were not utilized at all.
\nData from STRING did not prove helpful in the final model.
\nWe attempted to use a separate transformer layer that takes in label embeddings created with Anc2Vec (https://academic.oup.com/bib/article/23/2/bbac003/6523148) but it did not improve model performance.
\nWe attempted to use a hierarchy aware loss function (https://papers.nips.cc/paper/2020/file/6dd4e10e3296fa63738371ec0d5df818-Paper.pdf) but it did not perform better than BCE.
Using merely the PLM embeddings along with taxonomic information we obtained a public leaderboard (PL) score of about 0.55. Including text information increases the PL score to ~0.58.
\nEnsembling with a public ensemble notebook (https://www.kaggle.com/code/adaluodao/merge-datasets) increases PL score to ~0.62.
4th place overall!
\nCode used to generate our models can be found here:
\nhttps://github.com/zongmingchua/cafa5
\nFinal ensembling is shown here:
\nhttps://www.kaggle.com/code/zmcxjt/merge-datasets
Firstly I want to thank the organizers for a very nice competition - having it based on Kaggle opened up submissions to a wider range of people compared to previous iterations when mainly specialist labs showed results so the wider net of people being exposed to CAFA is really gratifying to see. My team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made. Both of us are molecular biologists with only limited bioinformatics and machine learning experience - so this competition was something of a trial by fire and we learned a lot from taking part.
\nWe used these datasets in our model:
\n1) Prot-T5, ESM2, and Ankh Protein Language Model (PLM) embeddings. We carried out no further modifications or finetuning on the output of PLMs, only conversion to float32 to save memory.
\n2) A single binary matrix representing species taxonomy for each protein.
\n3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein
For GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model to classify by taking the top N labels sorted by IA*frequency.
\nDatasets were sorted in alphabetical order to ensure identical ordering across datasets as well as removal of single duplicate row in test data.
\nWe made a simple dense neural network in keras that takes these as separate inputs before concatenating them into a final layer that attempts to predict every label from a single GO ontology/domain. We used binary cross entropy (BCE) as our loss function and Adam optimizer with lr=0.0003 and a simple lr scheduler. We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.
\nOur model was trained with KFold = 5 across 5 different random splits of the training data before a simple average of predictions was made for each ontology/domain and the results concatenated.
\nOne interesting thing we observed is that Ridge regression sometimes performed better on PLM embeddings compared to a neural network for some ontologies, which was unexpected and might indicate underfitting for the neural network.
\nProtBert embeddings performed poorly and were not utilized at all.
\nData from STRING did not prove helpful in the final model.
\nWe attempted to use a separate transformer layer that takes in label embeddings created with Anc2Vec (https://academic.oup.com/bib/article/23/2/bbac003/6523148) but it did not improve model performance.
\nWe attempted to use a hierarchy aware loss function (https://papers.nips.cc/paper/2020/file/6dd4e10e3296fa63738371ec0d5df818-Paper.pdf) but it did not perform better than BCE.
Using merely the PLM embeddings along with taxonomic information we obtained a public leaderboard (PL) score of about 0.55. Including text information increases the PL score to ~0.58.
\nEnsembling with a public ensemble notebook (https://www.kaggle.com/code/adaluodao/merge-datasets) increases PL score to ~0.62.
4th place overall!
\nCode used to generate our models can be found here:
\nhttps://github.com/zongmingchua/cafa5
\nFinal ensembling is shown here:
\nhttps://www.kaggle.com/code/zmcxjt/merge-datasets
\n\nMy team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made.
\n
Given your background, I would hardly consider that you fell short.
\nThere are many nice ideas in your solution. I haven't thought about adding text-based information, though it makes sense in retrospect. I don't think that would necessarily work as a general solution because many proteins are not correctly annotated. Still, for relatively well studied organisms that are targeted in this competition it should work.
", "rawMarkdown": "> My team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made.\n\nGiven your background, I would hardly consider that you fell short.\n\nThere are many nice ideas in your solution. I haven't thought about adding text-based information, though it makes sense in retrospect. I don't think that would necessarily work as a general solution because many proteins are not correctly annotated. Still, for relatively well studied organisms that are targeted in this competition it should work.", "votes": null }, { "id": "2403619", "postDate": "08/22/2023 18:40:57", "content": "Thank you for the kind words. Using text based information from paper abstracts is a method we borrowed directly from that of Zhu lab's NetGO model but it really only helps you with proteins that are adequately described in the literature!
", "rawMarkdown": "Thank you for the kind words. Using text based information from paper abstracts is a method we borrowed directly from that of Zhu lab's NetGO model but it really only helps you with proteins that are adequately described in the literature!", "votes": null }, { "id": "2403662", "postDate": "08/22/2023 19:23:10", "content": "Hi @zmcxjt, thank you for sharing details on your approach. I'm new to these kind of competitions and don't have possess any domain knowledge, so may I ask a few questions:
\n\n\nFor GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model
\n
Were you submitting all 1500+800+800 for each protein or did further pre-selection?
\nI was banging my head a lot about how to do the final selection for submission due to this line, which I probably misunderstood
\n\n\nFinally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n
I read it as: for each protein, the evaluation code will take up to 500 labels from each ontology, so there is no point to provide more than 1,500 per protein and one needs to come up with a ranking scheme.
\n\n\nWe further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.
\n
Did you experiment with other schemes to weigh positive labels? (maybe IA*frequency that you mentioned above)? In my not-so-ideal validation scheme, I was comparing the inverse frequency scheme with IA*frequencyfor an MLP but my validation scheme is not ideal, and I forgot to test the simpler IA weights.
\n\nWe further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well
\n
I've seen people arguing against combining batch norm and dropout citing this paper but your stellar results suggest that potential drop in performance is limited after all!
", "rawMarkdown": "Hi @zmcxjt, thank you for sharing details on your approach. I'm new to these kind of competitions and don't have possess any domain knowledge, so may I ask a few questions:\n\n>For GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model \n\nWere you submitting all 1500+800+800 for each protein or did further pre-selection?\n\nI was banging my head a lot about how to do the final selection for submission due to this [line](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation), which I probably misunderstood\n\n>Finally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined. \n\nI read it as: for each protein, the evaluation code will take up to 500 labels from each ontology, so there is no point to provide more than 1,500 per protein and one needs to come up with a ranking scheme.\n\n>We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.\n\nDid you experiment with other schemes to weigh positive labels? (maybe `IA*frequency` that you mentioned above)? In my not-so-ideal validation scheme, I was comparing the inverse frequency scheme with ` IA*frequency `for an MLP but my validation scheme is not ideal, and I forgot to test the simpler IA weights.\n\n\n>We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well\n\nI've seen people [arguing](http://karpathy.github.io/2019/04/25/recipe/) against combining batch norm and dropout citing this [paper](https://arxiv.org/abs/1801.05134) but your stellar results suggest that potential drop in performance is limited after all!", "votes": null }, { "id": "2403713", "postDate": "08/22/2023 19:57:23", "content": "\n\nWere you submitting all 1500+800+800 for each protein or did further pre-selection?
\n
In order to minimize our submission size, we actually only selected the top 5% of predictions based on probability assigned by our model. The final submission file ends up in the range of 500mb in size.
\nDue to the rarity and low IA weight of many GO terms, most proteins will have drastically lower than 1500 GO terms predicted for them.
\n\n\nDid you experiment with other schemes to weigh positive labels? (maybe IA*frequency that you mentioned above)?
\n
Unfortunately we did not! But I think the effects would be subtle.
\n\n\nI've seen people arguing against combining batch norm and dropout citing this paper but your stellar results suggest that potential drop in performance is limited after all!
\n
Due to the large number of input data being simultaneously fed in, the model quite rapidly begins to overfit which was not something we ever managed to completely overcome. We attempted many ways to regularize but were not too successful in doing so.
", "rawMarkdown": ">Were you submitting all 1500+800+800 for each protein or did further pre-selection?\n\nIn order to minimize our submission size, we actually only selected the top 5% of predictions based on probability assigned by our model. The final submission file ends up in the range of 500mb in size. \n\nDue to the rarity and low IA weight of many GO terms, most proteins will have drastically lower than 1500 GO terms predicted for them.\n\n>Did you experiment with other schemes to weigh positive labels? (maybe IA*frequency that you mentioned above)?\n\nUnfortunately we did not! But I think the effects would be subtle. \n\n>I've seen people arguing against combining batch norm and dropout citing this paper but your stellar results suggest that potential drop in performance is limited after all!\n\nDue to the large number of input data being simultaneously fed in, the model quite rapidly begins to overfit which was not something we ever managed to completely overcome. We attempted many ways to regularize but were not too successful in doing so.", "votes": null }, { "id": "2403720", "postDate": "08/22/2023 20:03:51", "content": "Understood, thank you for the prompt reply!
", "rawMarkdown": "Understood, thank you for the prompt reply!", "votes": null }, { "id": "2403791", "postDate": "08/22/2023 21:17:49", "content": "Congratulations and thanks for a detailed write up! STRING and Anc2Vec didn't work for us either, glad to see your experience is consistent here.
", "rawMarkdown": "Congratulations and thanks for a detailed write up! STRING and Anc2Vec didn't work for us either, glad to see your experience is consistent here.", "votes": null }, { "id": "2404307", "postDate": "08/23/2023 07:13:09", "content": "Congratulations, @zmcxjt ! Although this is your first time participating, you have achieved such a good ranking! Your plan is very helpful, especially using \"2) A single binary matrix representing species taxonomy for each protein\" and \"3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein\".
", "rawMarkdown": "Congratulations, @zmcxjt ! Although this is your first time participating, you have achieved such a good ranking! Your plan is very helpful, especially using \"2) A single binary matrix representing species taxonomy for each protein\" and \"3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein\".", "votes": null }, { "id": "2404438", "postDate": "08/23/2023 08:59:35", "content": "Congratulations on achieving a high ranking in your first participation! Thank you for always contributing to the discussions with valuable insights.
\n\n\n\"3) Information about the academic abstracts related to each protein, obtained through tf-idf processing of the text.\"
\n
This idea is absolutely brilliant! I never thought of that.
\n\n\n\"ProtBert embedding performance was low, and it was not used at all.\"
\n
In my experiments as well, concatenating ProtBERT rather led to a decline in score. I suspect this might be due to a lower information density compared to others like T5. However, when I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.
", "rawMarkdown": "Congratulations on achieving a high ranking in your first participation! Thank you for always contributing to the discussions with valuable insights.\n\n\n> \"3) Information about the academic abstracts related to each protein, obtained through tf-idf processing of the text.\"\n\nThis idea is absolutely brilliant! I never thought of that.\n\n\n> \"ProtBert embedding performance was low, and it was not used at all.\"\n\nIn my experiments as well, concatenating ProtBERT rather led to a decline in score. I suspect this might be due to a lower information density compared to others like T5. However, when I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.", "votes": null }, { "id": "2404844", "postDate": "08/23/2023 14:38:38", "content": "Congratulations on your impressive 10th place finish on the public leaderboard!!!
", "rawMarkdown": "Congratulations on your impressive 10th place finish on the public leaderboard!!!", "votes": null }, { "id": "2404938", "postDate": "08/23/2023 15:38:04", "content": "\n\nwhen I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.
\n
That's a very nice approach, something I should definitely have experimented more with!
", "rawMarkdown": ">when I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.\n\nThat's a very nice approach, something I should definitely have experimented more with!", "votes": null }, { "id": "2405529", "postDate": "08/24/2023 00:44:15", "content": "Congratulations on your achievement!
", "rawMarkdown": "Congratulations on your achievement!", "votes": null }, { "id": "2412704", "postDate": "08/28/2023 12:55:24", "content": "Nice and simple with literature aug!
", "rawMarkdown": "Nice and simple with literature aug!", "votes": null }, { "id": "2568669", "postDate": "12/20/2023 18:34:33", "content": "Congratulations to your team!
", "rawMarkdown": "Congratulations to your team!", "votes": null }, { "id": "2591933", "postDate": "01/08/2024 09:40:04", "content": "Re \"ProtBERT to 3dims using UMAP/tSNE\" - Dang, there goes my standing argument that PCA/compression of embeddings never improved a representation ðŸ˜
\nNice work! Sounds like a really nice, surprisingly simple rep! How much did the free text/tf-idf contrib? Did you try processing that further (e.g. embeddings from BioBERT models)?
", "rawMarkdown": "Re \"ProtBERT to 3dims using UMAP/tSNE\" - Dang, there goes my standing argument that PCA/compression of embeddings *never* improved a representation ðŸ˜\n\nNice work! Sounds like a really nice, surprisingly simple rep! How much did the free text/tf-idf contrib? Did you try processing that further (e.g. embeddings from BioBERT models)?", "votes": null }, { "id": "2591936", "postDate": "01/08/2024 09:41:52", "content": "Didja try ProteinBert (not ProtBert!) embeddings? (e.g. Global layer rep)?
\nhttps://github.com/nadavbra/protein_bert
Hello， sir， did you have the esemble notebook offline, because the link could not be used now
", "rawMarkdown": "Hello， sir， did you have the esemble notebook offline, because the link could not be used now", "votes": null }, { "id": "3312014", "postDate": "11/06/2025 07:57:36", "content": "hello, sir, I am wondering if the 3 specialists is better than a generalist model, did you do annealing experiment in this side. Then I also have a question, how you choose the topk label for 3 specialist, if it is IA * frequency?Then the loss should also be multiply with the weight of IA, simply multiply with IA?Im I correct, because I tried this in CAFA6 and found it did not work in my experiment
", "rawMarkdown": "hello, sir, I am wondering if the 3 specialists is better than a generalist model, did you do annealing experiment in this side. Then I also have a question, how you choose the topk label for 3 specialist, if it is IA * frequency?Then the loss should also be multiply with the weight of IA, simply multiply with IA?Im I correct, because I tried this in CAFA6 and found it did not work in my experiment", "votes": null } ], "comments": [ { "id": 2403600, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 18:26:36", "content": "\n\nMy team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made.
\n
Given your background, I would hardly consider that you fell short.
\nThere are many nice ideas in your solution. I haven't thought about adding text-based information, though it makes sense in retrospect. I don't think that would necessarily work as a general solution because many proteins are not correctly annotated. Still, for relatively well studied organisms that are targeted in this competition it should work.
", "votes": null, "replies": [ { "id": 2403619, "author_name": "zmcxjt", "author_url": "", "post_date": "08/22/2023 18:40:57", "content": "Thank you for the kind words. Using text based information from paper abstracts is a method we borrowed directly from that of Zhu lab's NetGO model but it really only helps you with proteins that are adequately described in the literature!
", "votes": null, "replies": [] } ] }, { "id": 2403662, "author_name": "dustinthewind", "author_url": "", "post_date": "08/22/2023 19:23:10", "content": "Hi @zmcxjt, thank you for sharing details on your approach. I'm new to these kind of competitions and don't have possess any domain knowledge, so may I ask a few questions:
\n\n\nFor GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model
\n
Were you submitting all 1500+800+800 for each protein or did further pre-selection?
\nI was banging my head a lot about how to do the final selection for submission due to this line, which I probably misunderstood
\n\n\nFinally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n
I read it as: for each protein, the evaluation code will take up to 500 labels from each ontology, so there is no point to provide more than 1,500 per protein and one needs to come up with a ranking scheme.
\n\n\nWe further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.
\n
Did you experiment with other schemes to weigh positive labels? (maybe IA*frequency that you mentioned above)? In my not-so-ideal validation scheme, I was comparing the inverse frequency scheme with IA*frequencyfor an MLP but my validation scheme is not ideal, and I forgot to test the simpler IA weights.
\n\nWe further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well
\n
I've seen people arguing against combining batch norm and dropout citing this paper but your stellar results suggest that potential drop in performance is limited after all!
", "votes": null, "replies": [ { "id": 2403713, "author_name": "zmcxjt", "author_url": "", "post_date": "08/22/2023 19:57:23", "content": "\n\nWere you submitting all 1500+800+800 for each protein or did further pre-selection?
\n
In order to minimize our submission size, we actually only selected the top 5% of predictions based on probability assigned by our model. The final submission file ends up in the range of 500mb in size.
\nDue to the rarity and low IA weight of many GO terms, most proteins will have drastically lower than 1500 GO terms predicted for them.
\n\n\nDid you experiment with other schemes to weigh positive labels? (maybe IA*frequency that you mentioned above)?
\n
Unfortunately we did not! But I think the effects would be subtle.
\n\n\nI've seen people arguing against combining batch norm and dropout citing this paper but your stellar results suggest that potential drop in performance is limited after all!
\n
Due to the large number of input data being simultaneously fed in, the model quite rapidly begins to overfit which was not something we ever managed to completely overcome. We attempted many ways to regularize but were not too successful in doing so.
", "votes": null, "replies": [ { "id": 2403720, "author_name": "dustinthewind", "author_url": "", "post_date": "08/22/2023 20:03:51", "content": "Understood, thank you for the prompt reply!
", "votes": null, "replies": [] } ] } ] }, { "id": 2403791, "author_name": "thedrcat", "author_url": "", "post_date": "08/22/2023 21:17:49", "content": "Congratulations and thanks for a detailed write up! STRING and Anc2Vec didn't work for us either, glad to see your experience is consistent here.
", "votes": null, "replies": [] }, { "id": 2404307, "author_name": "songqizhou", "author_url": "", "post_date": "08/23/2023 07:13:09", "content": "Congratulations, @zmcxjt ! Although this is your first time participating, you have achieved such a good ranking! Your plan is very helpful, especially using \"2) A single binary matrix representing species taxonomy for each protein\" and \"3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein\".
", "votes": null, "replies": [] }, { "id": 2404438, "author_name": "isakatsuyoshi", "author_url": "", "post_date": "08/23/2023 08:59:35", "content": "Congratulations on achieving a high ranking in your first participation! Thank you for always contributing to the discussions with valuable insights.
\n\n\n\"3) Information about the academic abstracts related to each protein, obtained through tf-idf processing of the text.\"
\n
This idea is absolutely brilliant! I never thought of that.
\n\n\n\"ProtBert embedding performance was low, and it was not used at all.\"
\n
In my experiments as well, concatenating ProtBERT rather led to a decline in score. I suspect this might be due to a lower information density compared to others like T5. However, when I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.
", "votes": null, "replies": [ { "id": 2404938, "author_name": "zmcxjt", "author_url": "", "post_date": "08/23/2023 15:38:04", "content": "\n\nwhen I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.
\n
That's a very nice approach, something I should definitely have experimented more with!
", "votes": null, "replies": [] }, { "id": 2591933, "author_name": "danofer", "author_url": "", "post_date": "01/08/2024 09:40:04", "content": "Re \"ProtBERT to 3dims using UMAP/tSNE\" - Dang, there goes my standing argument that PCA/compression of embeddings never improved a representation ðŸ˜
\nNice work! Sounds like a really nice, surprisingly simple rep! How much did the free text/tf-idf contrib? Did you try processing that further (e.g. embeddings from BioBERT models)?
", "votes": null, "replies": [] } ] }, { "id": 2404844, "author_name": "kummarivasanthi", "author_url": "", "post_date": "08/23/2023 14:38:38", "content": "Congratulations on your impressive 10th place finish on the public leaderboard!!!
", "votes": null, "replies": [] }, { "id": 2405529, "author_name": "yashsachan4yrbtech", "author_url": "", "post_date": "08/24/2023 00:44:15", "content": "Congratulations on your achievement!
", "votes": null, "replies": [] }, { "id": 2412704, "author_name": "danofer", "author_url": "", "post_date": "08/28/2023 12:55:24", "content": "Nice and simple with literature aug!
", "votes": null, "replies": [] }, { "id": 2568669, "author_name": "tilii7", "author_url": "", "post_date": "12/20/2023 18:34:33", "content": "Congratulations to your team!
", "votes": null, "replies": [] }, { "id": 2591936, "author_name": "danofer", "author_url": "", "post_date": "01/08/2024 09:41:52", "content": "Didja try ProteinBert (not ProtBert!) embeddings? (e.g. Global layer rep)?
\nhttps://github.com/nadavbra/protein_bert
Hello， sir， did you have the esemble notebook offline, because the link could not be used now
", "votes": null, "replies": [] }, { "id": 3312014, "author_name": "aizealotwu", "author_url": "", "post_date": "11/06/2025 07:57:36", "content": "hello, sir, I am wondering if the 3 specialists is better than a generalist model, did you do annealing experiment in this side. Then I also have a question, how you choose the topk label for 3 specialist, if it is IA * frequency?Then the loss should also be multiply with the weight of IA, simply multiply with IA?Im I correct, because I tried this in CAFA6 and found it did not work in my experiment
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2403549": "## **Introduction**\nFirstly I want to thank the organizers for a very nice competition - having it based on Kaggle opened up submissions to a wider range of people compared to previous iterations when mainly specialist labs showed results so the wider net of people being exposed to CAFA is really gratifying to see. My team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made. Both of us are molecular biologists with only limited bioinformatics and machine learning experience - so this competition was something of a trial by fire and we learned a lot from taking part.\n\n##**Approach**\nWe used these datasets in our model: \n1) Prot-T5, ESM2, and Ankh Protein Language Model (PLM) embeddings. We carried out no further modifications or finetuning on the output of PLMs, only conversion to float32 to save memory. \n2) A single binary matrix representing species taxonomy for each protein.\n3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein\n\nFor GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model to classify by taking the top N labels sorted by IA*frequency. \n\nDatasets were sorted in alphabetical order to ensure identical ordering across datasets as well as removal of single duplicate row in test data. \n\nWe made a simple dense neural network in keras that takes these as separate inputs before concatenating them into a final layer that attempts to predict every label from a single GO ontology/domain. We used binary cross entropy (BCE) as our loss function and Adam optimizer with lr=0.0003 and a simple lr scheduler. We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.\n\nOur model was trained with KFold = 5 across 5 different random splits of the training data before a simple average of predictions was made for each ontology/domain and the results concatenated. \n\nOne interesting thing we observed is that Ridge regression sometimes performed better on PLM embeddings compared to a neural network for some ontologies, which was unexpected and might indicate underfitting for the neural network. \n\n##**What didn't work**\nProtBert embeddings performed poorly and were not utilized at all. \nData from STRING did not prove helpful in the final model. \nWe attempted to use a separate transformer layer that takes in label embeddings created with Anc2Vec (https://academic.oup.com/bib/article/23/2/bbac003/6523148) but it did not improve model performance.\nWe attempted to use a hierarchy aware loss function (https://papers.nips.cc/paper/2020/file/6dd4e10e3296fa63738371ec0d5df818-Paper.pdf) but it did not perform better than BCE.\n\n##**Scores**\nUsing merely the PLM embeddings along with taxonomic information we obtained a public leaderboard (PL) score of about 0.55. Including text information increases the PL score to ~0.58.\nEnsembling with a public ensemble notebook (https://www.kaggle.com/code/adaluodao/merge-datasets) increases PL score to ~0.62.\n\n\n##**EDIT**\n4th place overall! \nCode used to generate our models can be found here:\nhttps://github.com/zongmingchua/cafa5 \nFinal ensembling is shown here:\nhttps://www.kaggle.com/code/zmcxjt/merge-datasets", "2403600": "> My team worked hard on this problem and although we fell short in the rankings it is humbling to look back at the progress we made.\n\nGiven your background, I would hardly consider that you fell short.\n\nThere are many nice ideas in your solution. I haven't thought about adding text-based information, though it makes sense in retrospect. I don't think that would necessarily work as a general solution because many proteins are not correctly annotated. Still, for relatively well studied organisms that are targeted in this competition it should work.", "2403619": "Thank you for the kind words. Using text based information from paper abstracts is a method we borrowed directly from that of Zhu lab's NetGO model but it really only helps you with proteins that are adequately described in the literature!", "2403662": "Hi @zmcxjt, thank you for sharing details on your approach. I'm new to these kind of competitions and don't have possess any domain knowledge, so may I ask a few questions:\n\n>For GO labels, we separately identified 1500 BPO, 800 CCO, and 800 MFO terms from each ontology for each model \n\nWere you submitting all 1500+800+800 for each protein or did further pre-selection?\n\nI was banging my head a lot about how to do the final selection for submission due to this [line](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation), which I probably misunderstood\n\n>Finally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined. \n\nI read it as: for each protein, the evaluation code will take up to 500 labels from each ontology, so there is no point to provide more than 1,500 per protein and one needs to come up with a ranking scheme.\n\n>We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well.\n\nDid you experiment with other schemes to weigh positive labels? (maybe `IA*frequency` that you mentioned above)? In my not-so-ideal validation scheme, I was comparing the inverse frequency scheme with ` IA*frequency `for an MLP but my validation scheme is not ideal, and I forgot to test the simpler IA weights.\n\n\n>We further used the IA weights as class_weight when calling model.fit(). We added dropout and batch normalization layers to our model as well\n\nI've seen people [arguing](http://karpathy.github.io/2019/04/25/recipe/) against combining batch norm and dropout citing this [paper](https://arxiv.org/abs/1801.05134) but your stellar results suggest that potential drop in performance is limited after all!", "2403713": ">Were you submitting all 1500+800+800 for each protein or did further pre-selection?\n\nIn order to minimize our submission size, we actually only selected the top 5% of predictions based on probability assigned by our model. The final submission file ends up in the range of 500mb in size. \n\nDue to the rarity and low IA weight of many GO terms, most proteins will have drastically lower than 1500 GO terms predicted for them.\n\n>Did you experiment with other schemes to weigh positive labels? (maybe IA*frequency that you mentioned above)?\n\nUnfortunately we did not! But I think the effects would be subtle. \n\n>I've seen people arguing against combining batch norm and dropout citing this paper but your stellar results suggest that potential drop in performance is limited after all!\n\nDue to the large number of input data being simultaneously fed in, the model quite rapidly begins to overfit which was not something we ever managed to completely overcome. We attempted many ways to regularize but were not too successful in doing so.", "2403720": "Understood, thank you for the prompt reply!", "2403791": "Congratulations and thanks for a detailed write up! STRING and Anc2Vec didn't work for us either, glad to see your experience is consistent here.", "2404307": "Congratulations, @zmcxjt ! Although this is your first time participating, you have achieved such a good ranking! Your plan is very helpful, especially using \"2) A single binary matrix representing species taxonomy for each protein\" and \"3) Text information obtained by tf-idf of abstract information from academic papers associated with each protein\".", "2404438": "Congratulations on achieving a high ranking in your first participation! Thank you for always contributing to the discussions with valuable insights.\n\n\n> \"3) Information about the academic abstracts related to each protein, obtained through tf-idf processing of the text.\"\n\nThis idea is absolutely brilliant! I never thought of that.\n\n\n> \"ProtBert embedding performance was low, and it was not used at all.\"\n\nIn my experiments as well, concatenating ProtBERT rather led to a decline in score. I suspect this might be due to a lower information density compared to others like T5. However, when I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.", "2404844": "Congratulations on your impressive 10th place finish on the public leaderboard!!!", "2404938": ">when I reduced the dimensions of ProtBERT to 3dims using UMAP/tSNE and added it, the score improved. I think this was because the information was compressed.\n\nThat's a very nice approach, something I should definitely have experimented more with!", "2405529": "Congratulations on your achievement!", "2412704": "Nice and simple with literature aug!", "2568669": "Congratulations to your team!", "2591933": "Re \"ProtBERT to 3dims using UMAP/tSNE\" - Dang, there goes my standing argument that PCA/compression of embeddings *never* improved a representation ðŸ˜\n\nNice work! Sounds like a really nice, surprisingly simple rep! How much did the free text/tf-idf contrib? Did you try processing that further (e.g. embeddings from BioBERT models)?", "2591936": "Didja try [ProteinBert](https://doi.org/10.1093/bioinformatics/btac020) (not ProtBert!) embeddings? (e.g. Global layer rep)?\nhttps://github.com/nadavbra/protein_bert", "3311480": "Hello， sir， did you have the esemble notebook offline, because the link could not be used now", "3312014": "hello, sir, I am wondering if the 3 specialists is better than a generalist model, did you do annealing experiment in this side. Then I also have a question, how you choose the topk label for 3 specialist, if it is IA * frequency?Then the loss should also be multiply with the weight of IA, simply multiply with IA?Im I correct, because I tried this in CAFA6 and found it did not work in my experiment" }, "source": "meta" }