Introduction
\nAfter spending countless hours crafting sophisticated ensembles for CAFA 5, only to crash and burn in 350th place, something very different was required this time. So, this time, the core idea is to be more effective and more efficient. That is, to construct a set of predictions that will obtain a decent score in this competition, but without requiring a large team of software engineers or burning through too many weeks of GPU time.\nIn that light, I looked at the problem in a particular and perhaps slightly unusual way.
\nIf we consider the dataset of proteins whose functions we are to predict for CAFA 6, we can divide it into these two subsets:
\n• CAFA 5: Those proteins that were also part of the CAFA 5 test set;
\n• CAFA 6: The proteins newly added to the test set for CAFA 6.
\nMy central idea was to treat these two subsets quite differently.
\nProteins Previously in CAFA 5 Test Set
\nFor the CAFA 5 subset, the light blue region on the left-hand side of the Figure, we already have knowledge of existing successful models. Indeed, submission files for three of the top four teams from CAFA 5 were easily available via Kaggle:\nhttps://www.kaggle.com/code/wang0027/esm1b/output?scriptVersionId=157751726 version 111 from the GOCurator team in position 1, CAFA5 private score = 0.61623 LB, 0.61765 (code);\nhttps://www.kaggle.com/code/btbpanda/best-stack-and-leak version 11 from U900 team in position 2, private LB 0.58240 (code);\nhttps://www.kaggle.com/code/zmcxjt/merge-datasets?scriptVersionId=140608820 version 114 from the Synthetic Goose team in position 4 CAFA5 private score = 0.56245 LB, 0.56245 (code)\nWhile simply taking the top place entry for CAFA 5 to cover that set seemed superficially attractive, some reading of the arXiv preprint describing the work of @sergeifironov, @btbpanda and @alexandervc in the 2nd placed U900 team in the 2023 competition suggested that such an approach might be problematic. Evidently, the teams in 1st and 4th places had made heavy use of early reports in the literature for their predictions of protein function. That suggests that while those predictions might have been timely in 2023, their lifecycle could be relatively short. Thus, I felt that I ought also to include U900’s own predictions, expected to remain timely for longer, in my model. Therefore, I decided to cover the CAFA 5 subset of proteins with an ensemble of all three of those historical submissions.
\nProteins New to CAFA 6
\nThis left me with the question of how to treat the CAFA 6 subset, corresponding to the yellow area on the right-hand side of the Figure. Given that I expected that leaving these proteins alone would be too detrimental to the model’s recall, I would definitely need to make my own new predictions for this subset. Nonetheless, not wanting to undermine the model’s precision either, I intended to limit this subset to a relatively small selection of predictions. Hence, I chose to cover this with a simple Diamond model. For one of my submissions, I based my model on @geraseva’s CAFA 5 notebook diamond; for this competition I set the sensitivity to standard to generate a moderately small prediction set. My second submission was based on @bnhbiphc’s Diamond CAFA 6 notebook, and had the more sensitive setting in order to produce a somewhat larger prediction set.
\nGOA Uniprot All and GAF Predictions
\nFinally, from reading the discussion forum, it seemed likely that adding in the predictions from @mtinti’s model prediction-with-goa-uniprot-all across the board would add some value. Similarly, I added in the predictions from @sacuscreed ‘s GAF Submission. While only a small proportion of predictions from these sources would end up contributing to the final score, some cases such as experimental confirmation of existing electronic annotations could make a telling contribution. Further, it seemed very unlikely that these sources would bring many incorrect predictions. Thus, these predictions were considered to be an essentially free score boost, evaluated for both subsets, added as an extra layer into my predictions, and shown as a pink stripe extending across both sides of the Figure.
\nFinal Models
\nPutting all this together gave me two submissions, both with the kind of structure shown in the illustration. They differed only in the details of their Diamond models.
\nMy second submission with the larger Diamond model scored 0.38425, equivalent to 143rd place. Fortunately, my top submission with the standard Diamond sensitivity setting scored better than this, with 0.39019 and 54th place.
\nLearnings
\nLeaving out the Diamond predictions entirely would have improved my score. A model with my existing representation the proteins of CAFA 5 and with CAFA 6 covered only by the “pink stripe” would have scored 0.39693, equivalent to 38th place. So, in fact I was incorrect in my original assumption that such a model would have too low a recall to be useful.
\nHad I not included the P2 and P4 submissions from CAFA 5 in my model and instead relied on P1 only, I would have scored only 0.38208, or 209th position. This broadly validates my concerns about the timeliness of the literature-based CAFA 5 predictions.
\nThanks
\nThanks to the organisers for setting up and managing this important competition, and especially for navigating through the difficulties when the glitches occurred with the first post-deadline scoring. All now seems to be good. Thanks also to everyone who shared code and who contributed to the Discussion forum.
", "messages": [ { "id": 3469176, "postDate": "2026-06-10T19:18:35.090Z", "content": "Introduction
\nAfter spending countless hours crafting sophisticated ensembles for CAFA 5, only to crash and burn in 350th place, something very different was required this time. So, this time, the core idea is to be more effective and more efficient. That is, to construct a set of predictions that will obtain a decent score in this competition, but without requiring a large team of software engineers or burning through too many weeks of GPU time.\nIn that light, I looked at the problem in a particular and perhaps slightly unusual way.
\nIf we consider the dataset of proteins whose functions we are to predict for CAFA 6, we can divide it into these two subsets:
\n• CAFA 5: Those proteins that were also part of the CAFA 5 test set;
\n• CAFA 6: The proteins newly added to the test set for CAFA 6.
\nMy central idea was to treat these two subsets quite differently.
\nProteins Previously in CAFA 5 Test Set
\nFor the CAFA 5 subset, the light blue region on the left-hand side of the Figure, we already have knowledge of existing successful models. Indeed, submission files for three of the top four teams from CAFA 5 were easily available via Kaggle:\nhttps://www.kaggle.com/code/wang0027/esm1b/output?scriptVersionId=157751726 version 111 from the GOCurator team in position 1, CAFA5 private score = 0.61623 LB, 0.61765 (code);\nhttps://www.kaggle.com/code/btbpanda/best-stack-and-leak version 11 from U900 team in position 2, private LB 0.58240 (code);\nhttps://www.kaggle.com/code/zmcxjt/merge-datasets?scriptVersionId=140608820 version 114 from the Synthetic Goose team in position 4 CAFA5 private score = 0.56245 LB, 0.56245 (code)\nWhile simply taking the top place entry for CAFA 5 to cover that set seemed superficially attractive, some reading of the arXiv preprint describing the work of @sergeifironov, @btbpanda and @alexandervc in the 2nd placed U900 team in the 2023 competition suggested that such an approach might be problematic. Evidently, the teams in 1st and 4th places had made heavy use of early reports in the literature for their predictions of protein function. That suggests that while those predictions might have been timely in 2023, their lifecycle could be relatively short. Thus, I felt that I ought also to include U900’s own predictions, expected to remain timely for longer, in my model. Therefore, I decided to cover the CAFA 5 subset of proteins with an ensemble of all three of those historical submissions.
\nProteins New to CAFA 6
\nThis left me with the question of how to treat the CAFA 6 subset, corresponding to the yellow area on the right-hand side of the Figure. Given that I expected that leaving these proteins alone would be too detrimental to the model’s recall, I would definitely need to make my own new predictions for this subset. Nonetheless, not wanting to undermine the model’s precision either, I intended to limit this subset to a relatively small selection of predictions. Hence, I chose to cover this with a simple Diamond model. For one of my submissions, I based my model on @geraseva’s CAFA 5 notebook diamond; for this competition I set the sensitivity to standard to generate a moderately small prediction set. My second submission was based on @bnhbiphc’s Diamond CAFA 6 notebook, and had the more sensitive setting in order to produce a somewhat larger prediction set.
\nGOA Uniprot All and GAF Predictions
\nFinally, from reading the discussion forum, it seemed likely that adding in the predictions from @mtinti’s model prediction-with-goa-uniprot-all across the board would add some value. Similarly, I added in the predictions from @sacuscreed ‘s GAF Submission. While only a small proportion of predictions from these sources would end up contributing to the final score, some cases such as experimental confirmation of existing electronic annotations could make a telling contribution. Further, it seemed very unlikely that these sources would bring many incorrect predictions. Thus, these predictions were considered to be an essentially free score boost, evaluated for both subsets, added as an extra layer into my predictions, and shown as a pink stripe extending across both sides of the Figure.
\nFinal Models
\nPutting all this together gave me two submissions, both with the kind of structure shown in the illustration. They differed only in the details of their Diamond models.
\nMy second submission with the larger Diamond model scored 0.38425, equivalent to 143rd place. Fortunately, my top submission with the standard Diamond sensitivity setting scored better than this, with 0.39019 and 54th place.
\nLearnings
\nLeaving out the Diamond predictions entirely would have improved my score. A model with my existing representation the proteins of CAFA 5 and with CAFA 6 covered only by the “pink stripe” would have scored 0.39693, equivalent to 38th place. So, in fact I was incorrect in my original assumption that such a model would have too low a recall to be useful.
\nHad I not included the P2 and P4 submissions from CAFA 5 in my model and instead relied on P1 only, I would have scored only 0.38208, or 209th position. This broadly validates my concerns about the timeliness of the literature-based CAFA 5 predictions.
\nThanks
\nThanks to the organisers for setting up and managing this important competition, and especially for navigating through the difficulties when the glitches occurred with the first post-deadline scoring. All now seems to be good. Thanks also to everyone who shared code and who contributed to the Discussion forum.
", "rawMarkdown": "**Introduction**\n\nAfter spending countless hours crafting sophisticated ensembles for CAFA 5, only to crash and burn in 350th place, something very different was required this time. So, this time, the core idea is to be more effective and more efficient. That is, to construct a set of predictions that will obtain a decent score in this competition, but without requiring a large team of software engineers or burning through too many weeks of GPU time.\nIn that light, I looked at the problem in a particular and perhaps slightly unusual way. \n\nIf we consider the dataset of proteins whose functions we are to predict for CAFA 6, we can divide it into these two subsets:\n\n•\tCAFA 5: Those proteins that were also part of the CAFA 5 test set;\n\n•\tCAFA 6: The proteins newly added to the test set for CAFA 6.\n\nMy central idea was to treat these two subsets quite differently. \n\n**Proteins Previously in CAFA 5 Test Set**\n\nFor the CAFA 5 subset, the light blue region on the left-hand side of the Figure, we already have knowledge of existing successful models. Indeed, submission files for three of the top four teams from CAFA 5 were easily available via Kaggle:\n[https://www.kaggle.com/code/wang0027/esm1b/output?scriptVersionId=157751726](https://www.kaggle.com/code/wang0027/esm1b/output?scriptVersionId=157751726) version 111 from the GOCurator team in position 1, CAFA5 private score = 0.61623 LB, 0.61765 (code);\n[https://www.kaggle.com/code/btbpanda/best-stack-and-leak](https://www.kaggle.com/code/btbpanda/best-stack-and-leak) version 11 from U900 team in position 2, private LB 0.58240 (code);\n[https://www.kaggle.com/code/zmcxjt/merge-datasets?scriptVersionId=140608820](https://www.kaggle.com/code/zmcxjt/merge-datasets?scriptVersionId=140608820) version 114 from the Synthetic Goose team in position 4 CAFA5 private score = 0.56245 LB, 0.56245 (code)\nWhile simply taking the top place entry for CAFA 5 to cover that set seemed superficially attractive, some reading of the [arXiv preprint](https://arxiv.org/abs/2412.04529v1) describing the work of @sergeifironov, @btbpanda and @alexandervc in the 2nd placed U900 team in the 2023 competition suggested that such an approach might be problematic. Evidently, the teams in 1st and 4th places had made heavy use of early reports in the literature for their predictions of protein function. That suggests that while those predictions might have been timely in 2023, their lifecycle could be relatively short. Thus, I felt that I ought also to include U900’s own predictions, expected to remain timely for longer, in my model. Therefore, I decided to cover the CAFA 5 subset of proteins with an ensemble of all three of those historical submissions.\n\n**Proteins New to CAFA 6**\n\nThis left me with the question of how to treat the CAFA 6 subset, corresponding to the yellow area on the right-hand side of the Figure. Given that I expected that leaving these proteins alone would be too detrimental to the model’s recall, I would definitely need to make my own new predictions for this subset. Nonetheless, not wanting to undermine the model’s precision either, I intended to limit this subset to a relatively small selection of predictions. Hence, I chose to cover this with a simple Diamond model. For one of my submissions, I based my model on @geraseva’s CAFA 5 notebook [diamond]( https://www.kaggle.com/code/geraseva/diamond); for this competition I set the sensitivity to standard to generate a moderately small prediction set. My second submission was based on @bnhbiphc’s [Diamond](https://www.kaggle.com/code/bnhbiphc/diamond) CAFA 6 notebook, and had the more sensitive setting in order to produce a somewhat larger prediction set.\n\n**GOA Uniprot All and GAF Predictions**\n\nFinally, from reading the discussion forum, it seemed likely that adding in the predictions from @mtinti’s model [prediction-with-goa-uniprot-all]( https://www.kaggle.com/code/mtinti/prediction-with-goa-uniprot-all) across the board would add some value. Similarly, I added in the predictions from @sacuscreed ‘s [GAF Submission](https://www.kaggle.com/code/sacuscreed/gaf-submission). While only a small proportion of predictions from these sources would end up contributing to the final score, some cases such as experimental confirmation of existing electronic annotations could make a telling contribution. Further, it seemed very unlikely that these sources would bring many incorrect predictions. Thus, these predictions were considered to be an essentially free score boost, evaluated for both subsets, added as an extra layer into my predictions, and shown as a pink stripe extending across both sides of the Figure. \n\n**Final Models**\n\nPutting all this together gave me two submissions, both with the kind of structure shown in the illustration. They differed only in the details of their Diamond models.\n\nMy second submission with the larger Diamond model scored 0.38425, equivalent to 143rd place. Fortunately, my top submission with the standard Diamond sensitivity setting scored better than this, with 0.39019 and 54th place.\n\n**Learnings**\n\nLeaving out the Diamond predictions entirely would have improved my score. A model with my existing representation the proteins of CAFA 5 and with CAFA 6 covered only by the “pink stripe” would have scored 0.39693, equivalent to 38th place. So, in fact I was incorrect in my original assumption that such a model would have too low a recall to be useful.\n\nHad I not included the P2 and P4 submissions from CAFA 5 in my model and instead relied on P1 only, I would have scored only 0.38208, or 209th position. This broadly validates my concerns about the timeliness of the literature-based CAFA 5 predictions.\n\n**Thanks**\n\nThanks to the organisers for setting up and managing this important competition, and especially for navigating through the difficulties when the glitches occurred with the first post-deadline scoring. All now seems to be good. Thanks also to everyone who shared code and who contributed to the Discussion forum.\n", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "3469176": "**Introduction**\n\nAfter spending countless hours crafting sophisticated ensembles for CAFA 5, only to crash and burn in 350th place, something very different was required this time. So, this time, the core idea is to be more effective and more efficient. That is, to construct a set of predictions that will obtain a decent score in this competition, but without requiring a large team of software engineers or burning through too many weeks of GPU time.\nIn that light, I looked at the problem in a particular and perhaps slightly unusual way. \n\nIf we consider the dataset of proteins whose functions we are to predict for CAFA 6, we can divide it into these two subsets:\n\n•\tCAFA 5: Those proteins that were also part of the CAFA 5 test set;\n\n•\tCAFA 6: The proteins newly added to the test set for CAFA 6.\n\nMy central idea was to treat these two subsets quite differently. \n\n**Proteins Previously in CAFA 5 Test Set**\n\nFor the CAFA 5 subset, the light blue region on the left-hand side of the Figure, we already have knowledge of existing successful models. Indeed, submission files for three of the top four teams from CAFA 5 were easily available via Kaggle:\n[https://www.kaggle.com/code/wang0027/esm1b/output?scriptVersionId=157751726](https://www.kaggle.com/code/wang0027/esm1b/output?scriptVersionId=157751726) version 111 from the GOCurator team in position 1, CAFA5 private score = 0.61623 LB, 0.61765 (code);\n[https://www.kaggle.com/code/btbpanda/best-stack-and-leak](https://www.kaggle.com/code/btbpanda/best-stack-and-leak) version 11 from U900 team in position 2, private LB 0.58240 (code);\n[https://www.kaggle.com/code/zmcxjt/merge-datasets?scriptVersionId=140608820](https://www.kaggle.com/code/zmcxjt/merge-datasets?scriptVersionId=140608820) version 114 from the Synthetic Goose team in position 4 CAFA5 private score = 0.56245 LB, 0.56245 (code)\nWhile simply taking the top place entry for CAFA 5 to cover that set seemed superficially attractive, some reading of the [arXiv preprint](https://arxiv.org/abs/2412.04529v1) describing the work of @sergeifironov, @btbpanda and @alexandervc in the 2nd placed U900 team in the 2023 competition suggested that such an approach might be problematic. Evidently, the teams in 1st and 4th places had made heavy use of early reports in the literature for their predictions of protein function. That suggests that while those predictions might have been timely in 2023, their lifecycle could be relatively short. Thus, I felt that I ought also to include U900’s own predictions, expected to remain timely for longer, in my model. Therefore, I decided to cover the CAFA 5 subset of proteins with an ensemble of all three of those historical submissions.\n\n**Proteins New to CAFA 6**\n\nThis left me with the question of how to treat the CAFA 6 subset, corresponding to the yellow area on the right-hand side of the Figure. Given that I expected that leaving these proteins alone would be too detrimental to the model’s recall, I would definitely need to make my own new predictions for this subset. Nonetheless, not wanting to undermine the model’s precision either, I intended to limit this subset to a relatively small selection of predictions. Hence, I chose to cover this with a simple Diamond model. For one of my submissions, I based my model on @geraseva’s CAFA 5 notebook [diamond]( https://www.kaggle.com/code/geraseva/diamond); for this competition I set the sensitivity to standard to generate a moderately small prediction set. My second submission was based on @bnhbiphc’s [Diamond](https://www.kaggle.com/code/bnhbiphc/diamond) CAFA 6 notebook, and had the more sensitive setting in order to produce a somewhat larger prediction set.\n\n**GOA Uniprot All and GAF Predictions**\n\nFinally, from reading the discussion forum, it seemed likely that adding in the predictions from @mtinti’s model [prediction-with-goa-uniprot-all]( https://www.kaggle.com/code/mtinti/prediction-with-goa-uniprot-all) across the board would add some value. Similarly, I added in the predictions from @sacuscreed ‘s [GAF Submission](https://www.kaggle.com/code/sacuscreed/gaf-submission). While only a small proportion of predictions from these sources would end up contributing to the final score, some cases such as experimental confirmation of existing electronic annotations could make a telling contribution. Further, it seemed very unlikely that these sources would bring many incorrect predictions. Thus, these predictions were considered to be an essentially free score boost, evaluated for both subsets, added as an extra layer into my predictions, and shown as a pink stripe extending across both sides of the Figure. \n\n**Final Models**\n\nPutting all this together gave me two submissions, both with the kind of structure shown in the illustration. They differed only in the details of their Diamond models.\n\nMy second submission with the larger Diamond model scored 0.38425, equivalent to 143rd place. Fortunately, my top submission with the standard Diamond sensitivity setting scored better than this, with 0.39019 and 54th place.\n\n**Learnings**\n\nLeaving out the Diamond predictions entirely would have improved my score. A model with my existing representation the proteins of CAFA 5 and with CAFA 6 covered only by the “pink stripe” would have scored 0.39693, equivalent to 38th place. So, in fact I was incorrect in my original assumption that such a model would have too low a recall to be useful.\n\nHad I not included the P2 and P4 submissions from CAFA 5 in my model and instead relied on P1 only, I would have scored only 0.38208, or 209th position. This broadly validates my concerns about the timeliness of the literature-based CAFA 5 predictions.\n\n**Thanks**\n\nThanks to the organisers for setting up and managing this important competition, and especially for navigating through the difficulties when the glitches occurred with the first post-deadline scoring. All now seems to be good. Thanks also to everyone who shared code and who contributed to the Discussion forum.\n" }, "complete": true, "source": "existing_discussion", "competition_id": 116062, "competition_slug": "cafa-6-protein-function-prediction", "team_id": 14558399, "solution_url": "/competitions/cafa-6-protein-function-prediction/writeups/split-down-the-middle" }