![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F17450344%2Fe5f4d310947d2f6748c4e987f887c21c%2FGOA_overlap_CatBoostScore_vs_IA.png?generation=1774206753120300&alt=media\")
This writeup documents a systematic study of transformer-based protein function prediction that may be of methodological interest for this competition.
\nFull writeup with figures, tables, and experimental details:\nđź”— https://vsokhoyan.github.io/CAFA6-ML
\nFour aspects were studied systematically:
\nFine-tuning vs. fixed embeddings: End-to-end fine-tuning of ProtT5-XL (two unfrozen transformer blocks) and full fine-tuning of ESM2-small consistently outperforms fixed-embedding baselines (with the largest gains on BP). A general pattern confirms the statement in the literature on similar tasks: Smaller models gain more in relative terms from full fine-tuning, but a stronger pretrained model in this case retains an absolute advantage even under partial unfreezing. The experiments required notable computational resources (used various GPUs including NVIDIA A100 and AMD MI300X) and the hyperparameter comparisons documented here may reduce overhead for similar studies.
Rare-label training dynamics: Decomposing BP validation by label frequency shows the rare-label tail continues improving well after global Fmax has plateaued. Stopping criteria based on global metrics may systematically undertrain the rare-label regime - possibly relevant for applications emphasizing specific biological process studies.
Fold-sparse ensemble structure and IA-aware aggregation: Single-fold firing events in the BP ensemble occur nearly as often as full five-fold consensus, meaning standard mean aggregation can dilute a substantial fraction of valid predictions. An IA-aware pooling strategy (mean across all folds for low-IA terms, top-3 mean for mid-IA, top-2 mean for high-IA) improved leaderboard Fmax from 0.330 (single fold) to 0.348 (+5.5%), outperforming standard aggregation methods such as mean or median.
Reference-guided denoising: A UniProt/GOA-guided denoising step in the low-to-mid confidence band produced strong FP suppression on validation data without affecting TP notably. However, this did not reliably improve public leaderboard scores. Several structural causes are discussed, including reference coverage mismatch and disproportionate rejection of high-IA terms.
Selected parts of this study may be worth summarizing as a peer-reviewed article in computational biology or bioinformatics (presumably after optimizing or fine-tuning the presented methods). I would welcome any feedback or discussion in this respect (my background is in Nuclear/Particle Physics + ML). Also, would be very curious whether others obtained similar results, e.g., in fold-sparse structure, developed related aggregation strategies, or encountered analogous limitations in UniProt-based denoising.
\nI would be happy to discuss methodology or exchange ideas on any of the above.
", "messages": [ { "id": "3421505", "postDate": "03/15/2026 17:49:31", "content": "This writeup documents a systematic study of transformer-based protein function prediction that may be of methodological interest for this competition.
\nFull writeup with figures, tables, and experimental details:\nđź”— https://vsokhoyan.github.io/CAFA6-ML
\nFour aspects were studied systematically:
\nFine-tuning vs. fixed embeddings: End-to-end fine-tuning of ProtT5-XL (two unfrozen transformer blocks) and full fine-tuning of ESM2-small consistently outperforms fixed-embedding baselines (with the largest gains on BP). A general pattern confirms the statement in the literature on similar tasks: Smaller models gain more in relative terms from full fine-tuning, but a stronger pretrained model in this case retains an absolute advantage even under partial unfreezing. The experiments required notable computational resources (used various GPUs including NVIDIA A100 and AMD MI300X) and the hyperparameter comparisons documented here may reduce overhead for similar studies.
Rare-label training dynamics: Decomposing BP validation by label frequency shows the rare-label tail continues improving well after global Fmax has plateaued. Stopping criteria based on global metrics may systematically undertrain the rare-label regime - possibly relevant for applications emphasizing specific biological process studies.
Fold-sparse ensemble structure and IA-aware aggregation: Single-fold firing events in the BP ensemble occur nearly as often as full five-fold consensus, meaning standard mean aggregation can dilute a substantial fraction of valid predictions. An IA-aware pooling strategy (mean across all folds for low-IA terms, top-3 mean for mid-IA, top-2 mean for high-IA) improved leaderboard Fmax from 0.330 (single fold) to 0.348 (+5.5%), outperforming standard aggregation methods such as mean or median.
Reference-guided denoising: A UniProt/GOA-guided denoising step in the low-to-mid confidence band produced strong FP suppression on validation data without affecting TP notably. However, this did not reliably improve public leaderboard scores. Several structural causes are discussed, including reference coverage mismatch and disproportionate rejection of high-IA terms.
Selected parts of this study may be worth summarizing as a peer-reviewed article in computational biology or bioinformatics (presumably after optimizing or fine-tuning the presented methods). I would welcome any feedback or discussion in this respect (my background is in Nuclear/Particle Physics + ML). Also, would be very curious whether others obtained similar results, e.g., in fold-sparse structure, developed related aggregation strategies, or encountered analogous limitations in UniProt-based denoising.
\nI would be happy to discuss methodology or exchange ideas on any of the above.
", "rawMarkdown": "This writeup documents a systematic study of transformer-based protein function prediction that may be of methodological interest for this competition.\n\nFull writeup with figures, tables, and experimental details:\n🔗 https://vsokhoyan.github.io/CAFA6-ML\n\nFour aspects were studied systematically:\n\n1. **Fine-tuning vs. fixed embeddings**: End-to-end fine-tuning of ProtT5-XL (two unfrozen transformer blocks) and full fine-tuning of ESM2-small consistently outperforms fixed-embedding baselines (with the largest gains on BP). A general pattern confirms the statement in the literature on similar tasks: Smaller models gain more in relative terms from full fine-tuning, but a stronger pretrained model in this case retains an absolute advantage even under partial unfreezing. The experiments required notable computational resources (used various GPUs including NVIDIA A100 and AMD MI300X) and the hyperparameter comparisons documented here may reduce overhead for similar studies.\n\n2. **Rare-label training dynamics**: Decomposing BP validation by label frequency shows the rare-label tail continues improving well after global Fmax has plateaued. Stopping criteria based on global metrics may systematically undertrain the rare-label regime - possibly relevant for applications emphasizing specific biological process studies. \n\n3. **Fold-sparse ensemble structure and IA-aware aggregation:** Single-fold firing events in the BP ensemble occur nearly as often as full five-fold consensus, meaning standard mean aggregation can dilute a substantial fraction of valid predictions. An IA-aware pooling strategy (mean across all folds for low-IA terms, top-3 mean for mid-IA, top-2 mean for high-IA) improved leaderboard Fmax from 0.330 (single fold) to 0.348 (+5.5%), outperforming standard aggregation methods such as mean or median.\n\n4. **Reference-guided denoising**: A UniProt/GOA-guided denoising step in the low-to-mid confidence band produced strong FP suppression on validation data without affecting TP notably. However, this did not reliably improve public leaderboard scores. Several structural causes are discussed, including reference coverage mismatch and disproportionate rejection of high-IA terms.\n\nSelected parts of this study may be worth summarizing as a peer-reviewed article in computational biology or bioinformatics (presumably after optimizing or fine-tuning the presented methods). I would welcome any feedback or discussion in this respect (my background is in Nuclear/Particle Physics + ML). Also, would be very curious whether others obtained similar results, e.g., in fold-sparse structure, developed related aggregation strategies, or encountered analogous limitations in UniProt-based denoising. \n\nI would be happy to discuss methodology or exchange ideas on any of the above.", "votes": null }, { "id": "3426330", "postDate": "03/22/2026 11:26:10", "content": "Very interesting work. One question: I see that you stayed close to the official competition data, but since the superset contains many more proteins, wouldn’t it make sense to explore training with a larger dataset?
\nIn my case, fairly early on I focused on reviewed UniProt proteins, thinking they might offer a better balance between annotation quality and useful training signal. Did you try something along those lines, or did you prefer to stay strictly within the challenge data to avoid distribution or comparability issues?
", "rawMarkdown": "Very interesting work. One question: I see that you stayed close to the official competition data, but since the superset contains many more proteins, wouldn’t it make sense to explore training with a larger dataset?\n\nIn my case, fairly early on I focused on reviewed UniProt proteins, thinking they might offer a better balance between annotation quality and useful training signal. Did you try something along those lines, or did you prefer to stay strictly within the challenge data to avoid distribution or comparability issues?", "votes": null }, { "id": "3426539", "postDate": "03/22/2026 19:23:27", "content": "Thank you for your feedback and questions! I looked into a few UniProt-related directions, but have not used data beyond the set provided in the competition (mainly due to limited time for experiments and uncertainty on how to approach this). So this is not a direct answer related to your setup, but could be relevant for further ML vs. UniProt/GOA analyses.
\nIn this competition, I tried:
\n1.) Using external UniProt/GOA signal as a reference and attempted reference-guided denoising with GOA/UniProt-derived features. The denoising worked well on validation (strong suppression of FP, much smaller TP reduction), but did not transfer robustly to the LB. For narrow variants the results were roughly within LB noise, while broader cuts had a tendency to reduce the score. I suspected that this could be due to cutting high-IA cases, and tried to denoise only the low IA band (e.g., IA < 1) but this did not bring an improvement on LB beyond small fluctuations.
\n2.) Probably closest diagnostics to your approach: I checked the overlap between my final ProtT5-based Meta-CatBoost model (trained on ProtT5 score + sequence length + IA … with IA-aware loss) and GOA. Overall, ~52.4% of GOA protein–GO pairs were predicted by the CatBoost model. Most interestingly, this increased strongly in the high-IA regime: GOA recovery was at about 75.4% for IA > 5, 76.3% for IA > 7, and 84.7% for IA > 9 (see the plot attached). This suggests that a substantial fraction of the stronger GOA-supported signal is already captured by the model, which may explain the limited gains from direct UniProt augmentation (see point 3). Also, on the model side (model prediction as a reference), GOA overlap was much stronger in the high-score and high-IA region.
\n
3.) Augmenting the ML solution with high-quality/high-IA UniProt slices: I tried to augment the ML-predicted sample with high-score/high-IA UniProt-derived regions. This gave at most minor gains in a narrow regime, while broader inclusion notably degraded the LB score.
\n4.) Adding GOA-based signal by augmenting training data with evidence codes and other information from GOA for the available training data (or extended variant as you proposed): This still seems to be a promising and potentially robust solution (similar approaches were also mentioned in CAFA-5 summaries), however due to limited time I was not able to bring it to a well-optimized state.
\nI would be very interested in your experience as well. Especially, how you selected the reviewed UniProt subset, whether you saw consistent gains on validation sets, and what part of the gain transferred to the LB (beyond results obtained with the official CAFA-6 training data).
", "rawMarkdown": "Thank you for your feedback and questions! I looked into a few UniProt-related directions, but have not used data beyond the set provided in the competition (mainly due to limited time for experiments and uncertainty on how to approach this). So this is not a direct answer related to your setup, but could be relevant for further ML vs. UniProt/GOA analyses.\n\nIn this competition, I tried:\n\n1.) Using external UniProt/GOA signal as a reference and attempted reference-guided denoising with GOA/UniProt-derived features. The denoising worked well on validation (strong suppression of FP, much smaller TP reduction), but did not transfer robustly to the LB. For narrow variants the results were roughly within LB noise, while broader cuts had a tendency to reduce the score. I suspected that this could be due to cutting high-IA cases, and tried to denoise only the low IA band (e.g., IA < 1) but this did not bring an improvement on LB beyond small fluctuations. \n\n2.) Probably closest diagnostics to your approach: I checked the overlap between my final ProtT5-based Meta-CatBoost model (trained on ProtT5 score + sequence length + IA ... with IA-aware loss) and GOA. Overall, ~52.4% of GOA protein–GO pairs were predicted by the CatBoost model. Most interestingly, this increased strongly in the high-IA regime: GOA recovery was at about 75.4% for IA > 5, 76.3% for IA > 7, and 84.7% for IA > 9 (see the plot attached). This suggests that a substantial fraction of the stronger GOA-supported signal is already captured by the model, which may explain the limited gains from direct UniProt augmentation (see point 3). Also, on the model side (model prediction as a reference), GOA overlap was much stronger in the high-score and high-IA region.\n\n\n\n3.) Augmenting the ML solution with high-quality/high-IA UniProt slices: I tried to augment the ML-predicted sample with high-score/high-IA UniProt-derived regions. This gave at most minor gains in a narrow regime, while broader inclusion notably degraded the LB score.\n\n4.) Adding GOA-based signal by augmenting training data with evidence codes and other information from GOA for the available training data (or extended variant as you proposed): This still seems to be a promising and potentially robust solution (similar approaches were also mentioned in CAFA-5 summaries), however due to limited time I was not able to bring it to a well-optimized state.\n\nI would be very interested in your experience as well. Especially, how you selected the reviewed UniProt subset, whether you saw consistent gains on validation sets, and what part of the gain transferred to the LB (beyond results obtained with the official CAFA-6 training data).", "votes": null }, { "id": "3426822", "postDate": "03/23/2026 10:01:50", "content": "Thanks, this is very helpful.
\nIn my case, I did try external reviewed data rather than staying close to the official competition set. I worked mainly with two routes: reviewed UniProtKB entries, and experimentally supported GO annotations (for example EXP/IDA/IPI/IMP/IGI/IEP/TAS/IC-style evidence filters). I also used QuickGO annotations filtered to reviewed proteins. So far, QuickGO has given me the best results.
\nI should also say that I was already working with these external data sources before focusing on the competition setup, so I did not spend much time on the official CAFA-6 training data. Once I saw that these reviewed/QuickGO-based variants were generally performing better for my setup, I stayed with them quite early.
\nMethodologically, my setup has been much simpler than the kind of end-to-end neural modeling you describe. I have not spent much time designing new neural architectures or reranking models. Instead, I have mostly worked with KNN in embedding space and tried to transfer annotations in a balanced way, adjusting the score with additional signals such as neighbour alignment quality, taxonomic distance, evidence codes, and similar features. I have also tried to calibrate this transfer-based setup against stronger ML-style submissions and against QuickGO-derived reference baselines.
\nI also explored an additional LLM-based review layer on top of this pipeline. In practice, I deployed a Qwen-based loop (via Ollama) that reviewed candidate annotations row by row, using ontology-specific prompts and the corresponding KNN context to make a binary keep/discard decision. I found this direction interesting, but I was not able to validate or integrate it cleanly enough for the final CAFA submission.
\nIn the end, the best-performing results came from empirical ensembles built through iterative trial and error during the different stages of the work, rather than from one clearly isolated and rigorously validated methodological gain.
\nThat said, one weakness on my side is that I did not initially have a sufficiently rigorous validation framework. After reading your comments, I think I need to run those comparisons more carefully if I want to make any strong methodological claim or turn this into a solid research study. So at the moment, I would describe the result as promising, but still under validation.
\nA key reason I keep using the reviewed superset is that it has been very useful for building temporal snapshots properly. What I do is download GAF files from different releases, compute the differences between them, predict using only the earlier snapshot, and then evaluate on the later GAF that was not used during inference. For me, this is the correct way to simulate the temporal holdout setting.
\nFrom the practical side, obtaining the UniProt-based data is straightforward through the API, either by downloading the TSV directly or scripting it through the endpoint. So the main difficulty is not data access, but understanding which features are actually useful and how the dataset behaves once you impose these filters.
", "rawMarkdown": "Thanks, this is very helpful.\n\nIn my case, I did try external reviewed data rather than staying close to the official competition set. I worked mainly with two routes: [reviewed UniProtKB entries](https://www.uniprot.org/uniprotkb?query=%28reviewed%3Atrue%29), and [experimentally supported GO annotations](https://www.uniprot.org/uniprotkb?query=%28go_exp%3A*+OR+go_ida%3A*+OR+go_ipi%3A*+OR+go_imp%3A*+OR+go_igi%3A*+OR+go_iep%3A*+OR+go_tas%3A*+OR+go_ic%3A*%29) (for example EXP/IDA/IPI/IMP/IGI/IEP/TAS/IC-style evidence filters). I also used QuickGO annotations filtered to reviewed proteins. So far, QuickGO has given me the best results.\n\nI should also say that I was already working with these external data sources before focusing on the competition setup, so I did not spend much time on the official CAFA-6 training data. Once I saw that these reviewed/QuickGO-based variants were generally performing better for my setup, I stayed with them quite early.\n\nMethodologically, my setup has been much simpler than the kind of end-to-end neural modeling you describe. I have not spent much time designing new neural architectures or reranking models. Instead, I have mostly worked with KNN in embedding space and tried to transfer annotations in a balanced way, adjusting the score with additional signals such as neighbour alignment quality, taxonomic distance, evidence codes, and similar features. I have also tried to calibrate this transfer-based setup against stronger ML-style submissions and against QuickGO-derived reference baselines.\n\nI also explored an additional LLM-based review layer on top of this pipeline. In practice, I deployed a Qwen-based loop (via Ollama) that reviewed candidate annotations row by row, using ontology-specific prompts and the corresponding KNN context to make a binary keep/discard decision. I found this direction interesting, but I was not able to validate or integrate it cleanly enough for the final CAFA submission.\n\nIn the end, the best-performing results came from empirical ensembles built through iterative trial and error during the different stages of the work, rather than from one clearly isolated and rigorously validated methodological gain.\n\nThat said, one weakness on my side is that I did not initially have a sufficiently rigorous validation framework. After reading your comments, I think I need to run those comparisons more carefully if I want to make any strong methodological claim or turn this into a solid research study. So at the moment, I would describe the result as promising, but still under validation.\n\nA key reason I keep using the reviewed superset is that it has been very useful for building temporal snapshots properly. What I do is download GAF files from different releases, compute the differences between them, predict using only the earlier snapshot, and then evaluate on the later GAF that was not used during inference. For me, this is the correct way to simulate the temporal holdout setting.\n\nFrom the practical side, obtaining the UniProt-based data is straightforward through the API, either by downloading the TSV directly or scripting it through the endpoint. So the main difficulty is not data access, but understanding which features are actually useful and how the dataset behaves once you impose these filters.", "votes": null }, { "id": "3428616", "postDate": "03/25/2026 18:10:51", "content": "Thank you very much for the detailed and thoughtful response! The approaches we used are indeed rather orthogonal, relying strongly either on data curation and annotation transfer or on experiments with model architecture and end-to-end fine-tuning.
\nAn interesting question is the complementarity between the two: There should likely be an optimal range for improving data vs. model training. My model/GOA overlap analysis hinted at this from one direction - at high IA most of GOA annotations were recovered by ML, suggesting diminishing returns from annotation transfer in that regime. The symmetric view would also be interesting: how much the transfer-based pipeline recovers vs. ML-based solutions and how that depends on IA and model performance metrics. Characterizing this quantitatively could also be interesting in a broader context, as a more general question about where gains come from data quality vs. model capacity in multilabel settings with large reference databases.
\nA lightweight practical starting point could be, e.g., running my pipeline on a QuickGO-reviewed subset with fixed embeddings and comparing overlap in both directions. If you have filtered data in a directly usable form that would help, otherwise I can try to assemble it from the API when time allows. Your temporal GAF framework as held-out validation could possibly work as a natural evaluation setup for this.
", "rawMarkdown": "Thank you very much for the detailed and thoughtful response! The approaches we used are indeed rather orthogonal, relying strongly either on data curation and annotation transfer or on experiments with model architecture and end-to-end fine-tuning.\n\nAn interesting question is the complementarity between the two: There should likely be an optimal range for improving data vs. model training. My model/GOA overlap analysis hinted at this from one direction - at high IA most of GOA annotations were recovered by ML, suggesting diminishing returns from annotation transfer in that regime. The symmetric view would also be interesting: how much the transfer-based pipeline recovers vs. ML-based solutions and how that depends on IA and model performance metrics. Characterizing this quantitatively could also be interesting in a broader context, as a more general question about where gains come from data quality vs. model capacity in multilabel settings with large reference databases.\n\nA lightweight practical starting point could be, e.g., running my pipeline on a QuickGO-reviewed subset with fixed embeddings and comparing overlap in both directions. If you have filtered data in a directly usable form that would help, otherwise I can try to assemble it from the API when time allows. Your temporal GAF framework as held-out validation could possibly work as a natural evaluation setup for this.", "votes": null } ], "comments": [ { "id": 3426330, "author_name": "xaxipirulazo", "author_url": "", "post_date": "03/22/2026 11:26:10", "content": "Very interesting work. One question: I see that you stayed close to the official competition data, but since the superset contains many more proteins, wouldn’t it make sense to explore training with a larger dataset?
\nIn my case, fairly early on I focused on reviewed UniProt proteins, thinking they might offer a better balance between annotation quality and useful training signal. Did you try something along those lines, or did you prefer to stay strictly within the challenge data to avoid distribution or comparability issues?
", "votes": null, "replies": [ { "id": 3426539, "author_name": "vahesokhoyan", "author_url": "", "post_date": "03/22/2026 19:23:27", "content": "Thank you for your feedback and questions! I looked into a few UniProt-related directions, but have not used data beyond the set provided in the competition (mainly due to limited time for experiments and uncertainty on how to approach this). So this is not a direct answer related to your setup, but could be relevant for further ML vs. UniProt/GOA analyses.
\nIn this competition, I tried:
\n1.) Using external UniProt/GOA signal as a reference and attempted reference-guided denoising with GOA/UniProt-derived features. The denoising worked well on validation (strong suppression of FP, much smaller TP reduction), but did not transfer robustly to the LB. For narrow variants the results were roughly within LB noise, while broader cuts had a tendency to reduce the score. I suspected that this could be due to cutting high-IA cases, and tried to denoise only the low IA band (e.g., IA < 1) but this did not bring an improvement on LB beyond small fluctuations.
\n2.) Probably closest diagnostics to your approach: I checked the overlap between my final ProtT5-based Meta-CatBoost model (trained on ProtT5 score + sequence length + IA … with IA-aware loss) and GOA. Overall, ~52.4% of GOA protein–GO pairs were predicted by the CatBoost model. Most interestingly, this increased strongly in the high-IA regime: GOA recovery was at about 75.4% for IA > 5, 76.3% for IA > 7, and 84.7% for IA > 9 (see the plot attached). This suggests that a substantial fraction of the stronger GOA-supported signal is already captured by the model, which may explain the limited gains from direct UniProt augmentation (see point 3). Also, on the model side (model prediction as a reference), GOA overlap was much stronger in the high-score and high-IA region.
\n
3.) Augmenting the ML solution with high-quality/high-IA UniProt slices: I tried to augment the ML-predicted sample with high-score/high-IA UniProt-derived regions. This gave at most minor gains in a narrow regime, while broader inclusion notably degraded the LB score.
\n4.) Adding GOA-based signal by augmenting training data with evidence codes and other information from GOA for the available training data (or extended variant as you proposed): This still seems to be a promising and potentially robust solution (similar approaches were also mentioned in CAFA-5 summaries), however due to limited time I was not able to bring it to a well-optimized state.
\nI would be very interested in your experience as well. Especially, how you selected the reviewed UniProt subset, whether you saw consistent gains on validation sets, and what part of the gain transferred to the LB (beyond results obtained with the official CAFA-6 training data).
", "votes": null, "replies": [ { "id": 3426822, "author_name": "xaxipirulazo", "author_url": "", "post_date": "03/23/2026 10:01:50", "content": "Thanks, this is very helpful.
\nIn my case, I did try external reviewed data rather than staying close to the official competition set. I worked mainly with two routes: reviewed UniProtKB entries, and experimentally supported GO annotations (for example EXP/IDA/IPI/IMP/IGI/IEP/TAS/IC-style evidence filters). I also used QuickGO annotations filtered to reviewed proteins. So far, QuickGO has given me the best results.
\nI should also say that I was already working with these external data sources before focusing on the competition setup, so I did not spend much time on the official CAFA-6 training data. Once I saw that these reviewed/QuickGO-based variants were generally performing better for my setup, I stayed with them quite early.
\nMethodologically, my setup has been much simpler than the kind of end-to-end neural modeling you describe. I have not spent much time designing new neural architectures or reranking models. Instead, I have mostly worked with KNN in embedding space and tried to transfer annotations in a balanced way, adjusting the score with additional signals such as neighbour alignment quality, taxonomic distance, evidence codes, and similar features. I have also tried to calibrate this transfer-based setup against stronger ML-style submissions and against QuickGO-derived reference baselines.
\nI also explored an additional LLM-based review layer on top of this pipeline. In practice, I deployed a Qwen-based loop (via Ollama) that reviewed candidate annotations row by row, using ontology-specific prompts and the corresponding KNN context to make a binary keep/discard decision. I found this direction interesting, but I was not able to validate or integrate it cleanly enough for the final CAFA submission.
\nIn the end, the best-performing results came from empirical ensembles built through iterative trial and error during the different stages of the work, rather than from one clearly isolated and rigorously validated methodological gain.
\nThat said, one weakness on my side is that I did not initially have a sufficiently rigorous validation framework. After reading your comments, I think I need to run those comparisons more carefully if I want to make any strong methodological claim or turn this into a solid research study. So at the moment, I would describe the result as promising, but still under validation.
\nA key reason I keep using the reviewed superset is that it has been very useful for building temporal snapshots properly. What I do is download GAF files from different releases, compute the differences between them, predict using only the earlier snapshot, and then evaluate on the later GAF that was not used during inference. For me, this is the correct way to simulate the temporal holdout setting.
\nFrom the practical side, obtaining the UniProt-based data is straightforward through the API, either by downloading the TSV directly or scripting it through the endpoint. So the main difficulty is not data access, but understanding which features are actually useful and how the dataset behaves once you impose these filters.
", "votes": null, "replies": [ { "id": 3428616, "author_name": "vahesokhoyan", "author_url": "", "post_date": "03/25/2026 18:10:51", "content": "Thank you very much for the detailed and thoughtful response! The approaches we used are indeed rather orthogonal, relying strongly either on data curation and annotation transfer or on experiments with model architecture and end-to-end fine-tuning.
\nAn interesting question is the complementarity between the two: There should likely be an optimal range for improving data vs. model training. My model/GOA overlap analysis hinted at this from one direction - at high IA most of GOA annotations were recovered by ML, suggesting diminishing returns from annotation transfer in that regime. The symmetric view would also be interesting: how much the transfer-based pipeline recovers vs. ML-based solutions and how that depends on IA and model performance metrics. Characterizing this quantitatively could also be interesting in a broader context, as a more general question about where gains come from data quality vs. model capacity in multilabel settings with large reference databases.
\nA lightweight practical starting point could be, e.g., running my pipeline on a QuickGO-reviewed subset with fixed embeddings and comparing overlap in both directions. If you have filtered data in a directly usable form that would help, otherwise I can try to assemble it from the API when time allows. Your temporal GAF framework as held-out validation could possibly work as a natural evaluation setup for this.
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "3421505": "This writeup documents a systematic study of transformer-based protein function prediction that may be of methodological interest for this competition.\n\nFull writeup with figures, tables, and experimental details:\n🔗 https://vsokhoyan.github.io/CAFA6-ML\n\nFour aspects were studied systematically:\n\n1. **Fine-tuning vs. fixed embeddings**: End-to-end fine-tuning of ProtT5-XL (two unfrozen transformer blocks) and full fine-tuning of ESM2-small consistently outperforms fixed-embedding baselines (with the largest gains on BP). A general pattern confirms the statement in the literature on similar tasks: Smaller models gain more in relative terms from full fine-tuning, but a stronger pretrained model in this case retains an absolute advantage even under partial unfreezing. The experiments required notable computational resources (used various GPUs including NVIDIA A100 and AMD MI300X) and the hyperparameter comparisons documented here may reduce overhead for similar studies.\n\n2. **Rare-label training dynamics**: Decomposing BP validation by label frequency shows the rare-label tail continues improving well after global Fmax has plateaued. Stopping criteria based on global metrics may systematically undertrain the rare-label regime - possibly relevant for applications emphasizing specific biological process studies. \n\n3. **Fold-sparse ensemble structure and IA-aware aggregation:** Single-fold firing events in the BP ensemble occur nearly as often as full five-fold consensus, meaning standard mean aggregation can dilute a substantial fraction of valid predictions. An IA-aware pooling strategy (mean across all folds for low-IA terms, top-3 mean for mid-IA, top-2 mean for high-IA) improved leaderboard Fmax from 0.330 (single fold) to 0.348 (+5.5%), outperforming standard aggregation methods such as mean or median.\n\n4. **Reference-guided denoising**: A UniProt/GOA-guided denoising step in the low-to-mid confidence band produced strong FP suppression on validation data without affecting TP notably. However, this did not reliably improve public leaderboard scores. Several structural causes are discussed, including reference coverage mismatch and disproportionate rejection of high-IA terms.\n\nSelected parts of this study may be worth summarizing as a peer-reviewed article in computational biology or bioinformatics (presumably after optimizing or fine-tuning the presented methods). I would welcome any feedback or discussion in this respect (my background is in Nuclear/Particle Physics + ML). Also, would be very curious whether others obtained similar results, e.g., in fold-sparse structure, developed related aggregation strategies, or encountered analogous limitations in UniProt-based denoising. \n\nI would be happy to discuss methodology or exchange ideas on any of the above.", "3426330": "Very interesting work. One question: I see that you stayed close to the official competition data, but since the superset contains many more proteins, wouldn’t it make sense to explore training with a larger dataset?\n\nIn my case, fairly early on I focused on reviewed UniProt proteins, thinking they might offer a better balance between annotation quality and useful training signal. Did you try something along those lines, or did you prefer to stay strictly within the challenge data to avoid distribution or comparability issues?", "3426539": "Thank you for your feedback and questions! I looked into a few UniProt-related directions, but have not used data beyond the set provided in the competition (mainly due to limited time for experiments and uncertainty on how to approach this). So this is not a direct answer related to your setup, but could be relevant for further ML vs. UniProt/GOA analyses.\n\nIn this competition, I tried:\n\n1.) Using external UniProt/GOA signal as a reference and attempted reference-guided denoising with GOA/UniProt-derived features. The denoising worked well on validation (strong suppression of FP, much smaller TP reduction), but did not transfer robustly to the LB. For narrow variants the results were roughly within LB noise, while broader cuts had a tendency to reduce the score. I suspected that this could be due to cutting high-IA cases, and tried to denoise only the low IA band (e.g., IA < 1) but this did not bring an improvement on LB beyond small fluctuations. \n\n2.) Probably closest diagnostics to your approach: I checked the overlap between my final ProtT5-based Meta-CatBoost model (trained on ProtT5 score + sequence length + IA ... with IA-aware loss) and GOA. Overall, ~52.4% of GOA protein–GO pairs were predicted by the CatBoost model. Most interestingly, this increased strongly in the high-IA regime: GOA recovery was at about 75.4% for IA > 5, 76.3% for IA > 7, and 84.7% for IA > 9 (see the plot attached). This suggests that a substantial fraction of the stronger GOA-supported signal is already captured by the model, which may explain the limited gains from direct UniProt augmentation (see point 3). Also, on the model side (model prediction as a reference), GOA overlap was much stronger in the high-score and high-IA region.\n\n\n\n3.) Augmenting the ML solution with high-quality/high-IA UniProt slices: I tried to augment the ML-predicted sample with high-score/high-IA UniProt-derived regions. This gave at most minor gains in a narrow regime, while broader inclusion notably degraded the LB score.\n\n4.) Adding GOA-based signal by augmenting training data with evidence codes and other information from GOA for the available training data (or extended variant as you proposed): This still seems to be a promising and potentially robust solution (similar approaches were also mentioned in CAFA-5 summaries), however due to limited time I was not able to bring it to a well-optimized state.\n\nI would be very interested in your experience as well. Especially, how you selected the reviewed UniProt subset, whether you saw consistent gains on validation sets, and what part of the gain transferred to the LB (beyond results obtained with the official CAFA-6 training data).", "3426822": "Thanks, this is very helpful.\n\nIn my case, I did try external reviewed data rather than staying close to the official competition set. I worked mainly with two routes: [reviewed UniProtKB entries](https://www.uniprot.org/uniprotkb?query=%28reviewed%3Atrue%29), and [experimentally supported GO annotations](https://www.uniprot.org/uniprotkb?query=%28go_exp%3A*+OR+go_ida%3A*+OR+go_ipi%3A*+OR+go_imp%3A*+OR+go_igi%3A*+OR+go_iep%3A*+OR+go_tas%3A*+OR+go_ic%3A*%29) (for example EXP/IDA/IPI/IMP/IGI/IEP/TAS/IC-style evidence filters). I also used QuickGO annotations filtered to reviewed proteins. So far, QuickGO has given me the best results.\n\nI should also say that I was already working with these external data sources before focusing on the competition setup, so I did not spend much time on the official CAFA-6 training data. Once I saw that these reviewed/QuickGO-based variants were generally performing better for my setup, I stayed with them quite early.\n\nMethodologically, my setup has been much simpler than the kind of end-to-end neural modeling you describe. I have not spent much time designing new neural architectures or reranking models. Instead, I have mostly worked with KNN in embedding space and tried to transfer annotations in a balanced way, adjusting the score with additional signals such as neighbour alignment quality, taxonomic distance, evidence codes, and similar features. I have also tried to calibrate this transfer-based setup against stronger ML-style submissions and against QuickGO-derived reference baselines.\n\nI also explored an additional LLM-based review layer on top of this pipeline. In practice, I deployed a Qwen-based loop (via Ollama) that reviewed candidate annotations row by row, using ontology-specific prompts and the corresponding KNN context to make a binary keep/discard decision. I found this direction interesting, but I was not able to validate or integrate it cleanly enough for the final CAFA submission.\n\nIn the end, the best-performing results came from empirical ensembles built through iterative trial and error during the different stages of the work, rather than from one clearly isolated and rigorously validated methodological gain.\n\nThat said, one weakness on my side is that I did not initially have a sufficiently rigorous validation framework. After reading your comments, I think I need to run those comparisons more carefully if I want to make any strong methodological claim or turn this into a solid research study. So at the moment, I would describe the result as promising, but still under validation.\n\nA key reason I keep using the reviewed superset is that it has been very useful for building temporal snapshots properly. What I do is download GAF files from different releases, compute the differences between them, predict using only the earlier snapshot, and then evaluate on the later GAF that was not used during inference. For me, this is the correct way to simulate the temporal holdout setting.\n\nFrom the practical side, obtaining the UniProt-based data is straightforward through the API, either by downloading the TSV directly or scripting it through the endpoint. So the main difficulty is not data access, but understanding which features are actually useful and how the dataset behaves once you impose these filters.", "3428616": "Thank you very much for the detailed and thoughtful response! The approaches we used are indeed rather orthogonal, relying strongly either on data curation and annotation transfer or on experiments with model architecture and end-to-end fine-tuning.\n\nAn interesting question is the complementarity between the two: There should likely be an optimal range for improving data vs. model training. My model/GOA overlap analysis hinted at this from one direction - at high IA most of GOA annotations were recovered by ML, suggesting diminishing returns from annotation transfer in that regime. The symmetric view would also be interesting: how much the transfer-based pipeline recovers vs. ML-based solutions and how that depends on IA and model performance metrics. Characterizing this quantitatively could also be interesting in a broader context, as a more general question about where gains come from data quality vs. model capacity in multilabel settings with large reference databases.\n\nA lightweight practical starting point could be, e.g., running my pipeline on a QuickGO-reviewed subset with fixed embeddings and comparing overlap in both directions. If you have filtered data in a directly usable form that would help, otherwise I can try to assemble it from the API when time allows. Your temporal GAF framework as held-out validation could possibly work as a natural evaluation setup for this." }, "source": "meta" }