![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F18060831%2F8c0665df6c1ad29d868b4acefb93500c%2FFinal-Approach.png?generation=1781200648111893&alt=media\")
I would like to begin by saying that this was my first Silver Medal in a Kaggle competition, and I am very happy with the final solution I was able to develop. I am also grateful to the competition organizers for designing such a challenging and meaningful problem. I would also like to thank the Kaggle community particularly the discussions and public notebooks which were extremely helpful throughout the competition and contributed significantly to my understanding and final approach.
\nMy final solution combined three complementary sources of information:
\nThe overall philosophy was simple: use the neural network to learn sequence-function relationships, apply biological constraints to ensure logically consistent predictions, and then combine multiple evidence sources to improve robustness.
\nEach protein was represented using two types of input:
\nThese two inputs were concatenated to form the final representation used by the model.
\nThe predictive model followed a shared-backbone, multi-head design.
\nA lightweight convolutional backbone processed the combined input representation and learned general protein features shared across all prediction tasks.
\nThe network then branched into three independent prediction heads corresponding to the three Gene Ontology sub-ontologies:
\nThis design allowed the model to learn common representations while still specializing for the different aspects of protein function.
\nProtein function prediction is an extremely imbalanced multi-label classification problem. Some GO terms are very common and broad, while others are rare and highly informative.
\nTo address this, I used a positive-weighted binary cross-entropy loss weighted by normalized Information Accretion (IA) values provided by the competition.\n$$\\mathcal{L}_{\\text{term}}(y_j, \\hat{y}_j) = - w_j^{\\text{IA}} \\cdot \\left[ w_A^{\\text{pos}} \\cdot y_j \\cdot \\log(\\hat{y}_j) + (1 - y_j) \\cdot \\log(1 - \\hat{y}_j) \\right]$$
\nWhere:
\nThis encouraged the model to place greater emphasis on learning informative and specific annotations rather than focusing primarily on frequent, generic terms. Since the evaluation metric itself rewards more informative predictions, aligning the training objective with the competition metric proved beneficial.
\nValidation was performed using a train–test split of the training data, with Micro-F1 score used as the evaluation metric. Hyperparameters and ensemble weights were selected based on validation performance before generating the final submissions.
\nGene Ontology is organized as a directed acyclic graph (DAG), where child terms imply the presence of their ancestor terms.
\nBecause the neural network predicts each GO term independently, its raw outputs may violate these hierarchical relationships. For example, a model could assign a higher confidence to a specific child term than to its parent.
\nTo address this, I applied GO propagation using the True Path Rule. Prediction scores were propagated upward through the ontology so that every ancestor term received a confidence score at least as high as any of its descendants.
\nThis produced biologically consistent predictions and provided a noticeable improvement in performance.
\nThe final submission combined three complementary prediction sources.
\nThe tri-head deep learning model captured general sequence-function relationships from protein embeddings and taxonomy information.
\nBLAST-based predictions transferred annotations from proteins with similar sequences.
\nThese predictions are particularly reliable when close homologs exist and complement the neural network's ability to generalize beyond exact sequence matches.
\nThe final component incorporated predictions derived from UniProt Gene Ontology Annotation (GOA) resources after applying the same hierarchical propagation procedure.
\nThis provided an additional source of biological evidence independent of both the neural network and BLAST.
\nThe three prediction sources were combined using weighted averaging. The ensemble leveraged:
\nlearned sequence patterns from the neural network,\nevolutionary evidence from sequence homology,\nand existing biological knowledge from curated annotations.
\nThe combination consistently outperformed any individual component used in isolation.
\nThe most significant performance gains came from hierarchical post-processing and multi-source biological evidence integration. GO hierarchy propagation consistently improved predictions by enforcing biologically valid relationships, while ensembling further strengthened robustness. Among external sources, GOA annotations provided the largest boost, followed by BLAST homology signals, which were particularly effective for proteins with close sequence matches.
\nOn the modeling side, I experimented with multiple backbone choices, including different ESM2 variants (650M, 3B, and 15B) as well as ProtT5, with ESM2-3B performing best overall. The final tri-head Conv1D architecture consistently outperformed simpler MLPs and alternative designs.
\nTraining stability was strongly influenced by the choice of IA scaling and positive class weights, and careful tuning was necessary for stable convergence.
\nPerformance analysis showed that the model handled Molecular Function (MF) best, followed by Cellular Component (CC), while Biological Process (BP) remained the most challenging.
\nMore complex architectural variations generally did not improve validation performance. Increases in model complexity often led to marginal or inconsistent gains compared to the simpler tri-head Conv1D backbone combined with well-tuned loss weighting and biological post-processing. I also experimented with alternative predictors such as tree-based models, but these did not perform well and underperformed the neural architecture on validation.
\nFuture improvements could include multi-task loss designs that better capture dependencies between GO sub-ontologies, as well as more careful architectural modeling of hierarchical relationships directly within the network. Additionally, more robust cross-validation strategies beyond a simple train–validation split could help reduce variance in performance estimation and improve generalization, especially given the imbalance and hierarchical structure of the GO labels.
\nhttps://www.kaggle.com/datasets/kaitonmh/cafa-6-embeddings
\nhttps://www.kaggle.com/code/redberry/blast-basic-protein-function-predictions-cafa6
\nhttps://www.kaggle.com/code/yongsukprasertsuk/cafa-6-goa-propagation
", "messages": [ { "id": 3471516, "postDate": "2026-06-11T18:00:17.207Z", "content": "
I would like to begin by saying that this was my first Silver Medal in a Kaggle competition, and I am very happy with the final solution I was able to develop. I am also grateful to the competition organizers for designing such a challenging and meaningful problem. I would also like to thank the Kaggle community particularly the discussions and public notebooks which were extremely helpful throughout the competition and contributed significantly to my understanding and final approach.
\nMy final solution combined three complementary sources of information:
\nThe overall philosophy was simple: use the neural network to learn sequence-function relationships, apply biological constraints to ensure logically consistent predictions, and then combine multiple evidence sources to improve robustness.
\nEach protein was represented using two types of input:
\nThese two inputs were concatenated to form the final representation used by the model.
\nThe predictive model followed a shared-backbone, multi-head design.
\nA lightweight convolutional backbone processed the combined input representation and learned general protein features shared across all prediction tasks.
\nThe network then branched into three independent prediction heads corresponding to the three Gene Ontology sub-ontologies:
\nThis design allowed the model to learn common representations while still specializing for the different aspects of protein function.
\nProtein function prediction is an extremely imbalanced multi-label classification problem. Some GO terms are very common and broad, while others are rare and highly informative.
\nTo address this, I used a positive-weighted binary cross-entropy loss weighted by normalized Information Accretion (IA) values provided by the competition.\n$$\\mathcal{L}_{\\text{term}}(y_j, \\hat{y}_j) = - w_j^{\\text{IA}} \\cdot \\left[ w_A^{\\text{pos}} \\cdot y_j \\cdot \\log(\\hat{y}_j) + (1 - y_j) \\cdot \\log(1 - \\hat{y}_j) \\right]$$
\nWhere:
\nThis encouraged the model to place greater emphasis on learning informative and specific annotations rather than focusing primarily on frequent, generic terms. Since the evaluation metric itself rewards more informative predictions, aligning the training objective with the competition metric proved beneficial.
\nValidation was performed using a train–test split of the training data, with Micro-F1 score used as the evaluation metric. Hyperparameters and ensemble weights were selected based on validation performance before generating the final submissions.
\nGene Ontology is organized as a directed acyclic graph (DAG), where child terms imply the presence of their ancestor terms.
\nBecause the neural network predicts each GO term independently, its raw outputs may violate these hierarchical relationships. For example, a model could assign a higher confidence to a specific child term than to its parent.
\nTo address this, I applied GO propagation using the True Path Rule. Prediction scores were propagated upward through the ontology so that every ancestor term received a confidence score at least as high as any of its descendants.
\nThis produced biologically consistent predictions and provided a noticeable improvement in performance.
\nThe final submission combined three complementary prediction sources.
\nThe tri-head deep learning model captured general sequence-function relationships from protein embeddings and taxonomy information.
\nBLAST-based predictions transferred annotations from proteins with similar sequences.
\nThese predictions are particularly reliable when close homologs exist and complement the neural network's ability to generalize beyond exact sequence matches.
\nThe final component incorporated predictions derived from UniProt Gene Ontology Annotation (GOA) resources after applying the same hierarchical propagation procedure.
\nThis provided an additional source of biological evidence independent of both the neural network and BLAST.
\nThe three prediction sources were combined using weighted averaging. The ensemble leveraged:
\nlearned sequence patterns from the neural network,\nevolutionary evidence from sequence homology,\nand existing biological knowledge from curated annotations.
\nThe combination consistently outperformed any individual component used in isolation.
\nThe most significant performance gains came from hierarchical post-processing and multi-source biological evidence integration. GO hierarchy propagation consistently improved predictions by enforcing biologically valid relationships, while ensembling further strengthened robustness. Among external sources, GOA annotations provided the largest boost, followed by BLAST homology signals, which were particularly effective for proteins with close sequence matches.
\nOn the modeling side, I experimented with multiple backbone choices, including different ESM2 variants (650M, 3B, and 15B) as well as ProtT5, with ESM2-3B performing best overall. The final tri-head Conv1D architecture consistently outperformed simpler MLPs and alternative designs.
\nTraining stability was strongly influenced by the choice of IA scaling and positive class weights, and careful tuning was necessary for stable convergence.
\nPerformance analysis showed that the model handled Molecular Function (MF) best, followed by Cellular Component (CC), while Biological Process (BP) remained the most challenging.
\nMore complex architectural variations generally did not improve validation performance. Increases in model complexity often led to marginal or inconsistent gains compared to the simpler tri-head Conv1D backbone combined with well-tuned loss weighting and biological post-processing. I also experimented with alternative predictors such as tree-based models, but these did not perform well and underperformed the neural architecture on validation.
\nFuture improvements could include multi-task loss designs that better capture dependencies between GO sub-ontologies, as well as more careful architectural modeling of hierarchical relationships directly within the network. Additionally, more robust cross-validation strategies beyond a simple train–validation split could help reduce variance in performance estimation and improve generalization, especially given the imbalance and hierarchical structure of the GO labels.
\nhttps://www.kaggle.com/datasets/kaitonmh/cafa-6-embeddings
\nhttps://www.kaggle.com/code/redberry/blast-basic-protein-function-predictions-cafa6
\nhttps://www.kaggle.com/code/yongsukprasertsuk/cafa-6-goa-propagation
", "rawMarkdown": "\n\nI would like to begin by saying that this was my first Silver Medal in a Kaggle competition, and I am very happy with the final solution I was able to develop. I am also grateful to the competition organizers for designing such a challenging and meaningful problem. I would also like to thank the Kaggle community particularly the discussions and public notebooks which were extremely helpful throughout the competition and contributed significantly to my understanding and final approach.\n\n# Context\n- **Business Context**: https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview \n- **Data Context**: https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/data\n# Overview of the Approach\nMy final solution combined three complementary sources of information:\n1. A deep learning model trained on protein language model embeddings and taxonomic information.\n2. A post-processing step that enforced the hierarchical structure of the Gene Ontology (GO).\n3. An ensemble that blended neural network predictions with sequence homology and existing annotation evidence.\n\nThe overall philosophy was simple: use the neural network to learn sequence-function relationships, apply biological constraints to ensure logically consistent predictions, and then combine multiple evidence sources to improve robustness.\n## Data Representation\n\nEach protein was represented using two types of input:\n\n- ESM2 embeddings: Precomputed embeddings generated from the ESM2 protein language model, which provide rich representations of protein sequences.\n- Taxonomic information: Species identifiers converted into learned embeddings to provide biological context.\n\nThese two inputs were concatenated to form the final representation used by the model.\n\n## Model Architecture\nThe predictive model followed a shared-backbone, multi-head design.\n\nA lightweight convolutional backbone processed the combined input representation and learned general protein features shared across all prediction tasks.\n\nThe network then branched into three independent prediction heads corresponding to the three Gene Ontology sub-ontologies:\n\n- Biological Process (BP)\n- Molecular Function (MF)\n- Cellular Component (CC)\n\nThis design allowed the model to learn common representations while still specializing for the different aspects of protein function.\n\n## Training Strategy\n\nProtein function prediction is an extremely imbalanced multi-label classification problem. Some GO terms are very common and broad, while others are rare and highly informative.\n\nTo address this, I used a positive-weighted binary cross-entropy loss weighted by normalized Information Accretion (IA) values provided by the competition.\n$$\\mathcal{L}_{\\text{term}}(y_j, \\hat{y}_j) = - w_j^{\\text{IA}} \\cdot \\left[ w_A^{\\text{pos}} \\cdot y_j \\cdot \\log(\\hat{y}_j) + (1 - y_j) \\cdot \\log(1 - \\hat{y}_j) \\right]$$\n\nWhere:\n- **y_j** (which is either 0 or 1) is the ground-truth label for term j.\n- **y_hat_j** (a value between 0 and 1) is the model's predicted probability for term j.\n- **w_A^pos** is the static positive class weight multiplier specific to aspect A (like BP, MF, or CC) to counteract label sparsity.\n- **w_j^IA** is the Information Accretion (IA) value provided by the competition for term j , min-max normalized to the stable gradient range of 0.5 to 1.0.\n\nThis encouraged the model to place greater emphasis on learning informative and specific annotations rather than focusing primarily on frequent, generic terms. Since the evaluation metric itself rewards more informative predictions, aligning the training objective with the competition metric proved beneficial. \n\n## Validation Strategy\nValidation was performed using a train–test split of the training data, with Micro-F1 score used as the evaluation metric. Hyperparameters and ensemble weights were selected based on validation performance before generating the final submissions.\n\n## Hierarchical Post-Processing\n\nGene Ontology is organized as a directed acyclic graph (DAG), where child terms imply the presence of their ancestor terms.\n\nBecause the neural network predicts each GO term independently, its raw outputs may violate these hierarchical relationships. For example, a model could assign a higher confidence to a specific child term than to its parent.\n\nTo address this, I applied GO propagation using the True Path Rule. Prediction scores were propagated upward through the ontology so that every ancestor term received a confidence score at least as high as any of its descendants.\n\nThis produced biologically consistent predictions and provided a noticeable improvement in performance.\n\n## Multi-Source Ensembling\n\nThe final submission combined three complementary prediction sources.\n\n### 1. Neural Network Predictions\n\nThe tri-head deep learning model captured general sequence-function relationships from protein embeddings and taxonomy information.\n\n### 2. BLAST Predictions\n\nBLAST-based predictions transferred annotations from proteins with similar sequences.\n\nThese predictions are particularly reliable when close homologs exist and complement the neural network's ability to generalize beyond exact sequence matches.\n\n### 3. GOA Propagation Predictions\n\nThe final component incorporated predictions derived from UniProt Gene Ontology Annotation (GOA) resources after applying the same hierarchical propagation procedure.\n\nThis provided an additional source of biological evidence independent of both the neural network and BLAST.\n\n### Final Ensemble\n\nThe three prediction sources were combined using weighted averaging. The ensemble leveraged:\n\nlearned sequence patterns from the neural network,\nevolutionary evidence from sequence homology,\nand existing biological knowledge from curated annotations.\n\nThe combination consistently outperformed any individual component used in isolation.\n\n# What Was Important\n\nThe most significant performance gains came from hierarchical post-processing and multi-source biological evidence integration. GO hierarchy propagation consistently improved predictions by enforcing biologically valid relationships, while ensembling further strengthened robustness. Among external sources, GOA annotations provided the largest boost, followed by BLAST homology signals, which were particularly effective for proteins with close sequence matches.\n\nOn the modeling side, I experimented with multiple backbone choices, including different ESM2 variants (650M, 3B, and 15B) as well as ProtT5, with ESM2-3B performing best overall. The final tri-head Conv1D architecture consistently outperformed simpler MLPs and alternative designs.\n\nTraining stability was strongly influenced by the choice of IA scaling and positive class weights, and careful tuning was necessary for stable convergence.\n\nPerformance analysis showed that the model handled Molecular Function (MF) best, followed by Cellular Component (CC), while Biological Process (BP) remained the most challenging.\n\n# What I Tried That Didn't Help\n\nMore complex architectural variations generally did not improve validation performance. Increases in model complexity often led to marginal or inconsistent gains compared to the simpler tri-head Conv1D backbone combined with well-tuned loss weighting and biological post-processing. I also experimented with alternative predictors such as tree-based models, but these did not perform well and underperformed the neural architecture on validation.\n\n# Future Work\n\nFuture improvements could include multi-task loss designs that better capture dependencies between GO sub-ontologies, as well as more careful architectural modeling of hierarchical relationships directly within the network. Additionally, more robust cross-validation strategies beyond a simple train–validation split could help reduce variance in performance estimation and improve generalization, especially given the imbalance and hierarchical structure of the GO labels.\n\n# Sources\nhttps://www.kaggle.com/datasets/kaitonmh/cafa-6-embeddings\n\nhttps://www.kaggle.com/code/redberry/blast-basic-protein-function-predictions-cafa6\n\nhttps://www.kaggle.com/code/yongsukprasertsuk/cafa-6-goa-propagation\n\n", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "3471516": "\n\nI would like to begin by saying that this was my first Silver Medal in a Kaggle competition, and I am very happy with the final solution I was able to develop. I am also grateful to the competition organizers for designing such a challenging and meaningful problem. I would also like to thank the Kaggle community particularly the discussions and public notebooks which were extremely helpful throughout the competition and contributed significantly to my understanding and final approach.\n\n# Context\n- **Business Context**: https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/overview \n- **Data Context**: https://www.kaggle.com/competitions/cafa-6-protein-function-prediction/data\n# Overview of the Approach\nMy final solution combined three complementary sources of information:\n1. A deep learning model trained on protein language model embeddings and taxonomic information.\n2. A post-processing step that enforced the hierarchical structure of the Gene Ontology (GO).\n3. An ensemble that blended neural network predictions with sequence homology and existing annotation evidence.\n\nThe overall philosophy was simple: use the neural network to learn sequence-function relationships, apply biological constraints to ensure logically consistent predictions, and then combine multiple evidence sources to improve robustness.\n## Data Representation\n\nEach protein was represented using two types of input:\n\n- ESM2 embeddings: Precomputed embeddings generated from the ESM2 protein language model, which provide rich representations of protein sequences.\n- Taxonomic information: Species identifiers converted into learned embeddings to provide biological context.\n\nThese two inputs were concatenated to form the final representation used by the model.\n\n## Model Architecture\nThe predictive model followed a shared-backbone, multi-head design.\n\nA lightweight convolutional backbone processed the combined input representation and learned general protein features shared across all prediction tasks.\n\nThe network then branched into three independent prediction heads corresponding to the three Gene Ontology sub-ontologies:\n\n- Biological Process (BP)\n- Molecular Function (MF)\n- Cellular Component (CC)\n\nThis design allowed the model to learn common representations while still specializing for the different aspects of protein function.\n\n## Training Strategy\n\nProtein function prediction is an extremely imbalanced multi-label classification problem. Some GO terms are very common and broad, while others are rare and highly informative.\n\nTo address this, I used a positive-weighted binary cross-entropy loss weighted by normalized Information Accretion (IA) values provided by the competition.\n$$\\mathcal{L}_{\\text{term}}(y_j, \\hat{y}_j) = - w_j^{\\text{IA}} \\cdot \\left[ w_A^{\\text{pos}} \\cdot y_j \\cdot \\log(\\hat{y}_j) + (1 - y_j) \\cdot \\log(1 - \\hat{y}_j) \\right]$$\n\nWhere:\n- **y_j** (which is either 0 or 1) is the ground-truth label for term j.\n- **y_hat_j** (a value between 0 and 1) is the model's predicted probability for term j.\n- **w_A^pos** is the static positive class weight multiplier specific to aspect A (like BP, MF, or CC) to counteract label sparsity.\n- **w_j^IA** is the Information Accretion (IA) value provided by the competition for term j , min-max normalized to the stable gradient range of 0.5 to 1.0.\n\nThis encouraged the model to place greater emphasis on learning informative and specific annotations rather than focusing primarily on frequent, generic terms. Since the evaluation metric itself rewards more informative predictions, aligning the training objective with the competition metric proved beneficial. \n\n## Validation Strategy\nValidation was performed using a train–test split of the training data, with Micro-F1 score used as the evaluation metric. Hyperparameters and ensemble weights were selected based on validation performance before generating the final submissions.\n\n## Hierarchical Post-Processing\n\nGene Ontology is organized as a directed acyclic graph (DAG), where child terms imply the presence of their ancestor terms.\n\nBecause the neural network predicts each GO term independently, its raw outputs may violate these hierarchical relationships. For example, a model could assign a higher confidence to a specific child term than to its parent.\n\nTo address this, I applied GO propagation using the True Path Rule. Prediction scores were propagated upward through the ontology so that every ancestor term received a confidence score at least as high as any of its descendants.\n\nThis produced biologically consistent predictions and provided a noticeable improvement in performance.\n\n## Multi-Source Ensembling\n\nThe final submission combined three complementary prediction sources.\n\n### 1. Neural Network Predictions\n\nThe tri-head deep learning model captured general sequence-function relationships from protein embeddings and taxonomy information.\n\n### 2. BLAST Predictions\n\nBLAST-based predictions transferred annotations from proteins with similar sequences.\n\nThese predictions are particularly reliable when close homologs exist and complement the neural network's ability to generalize beyond exact sequence matches.\n\n### 3. GOA Propagation Predictions\n\nThe final component incorporated predictions derived from UniProt Gene Ontology Annotation (GOA) resources after applying the same hierarchical propagation procedure.\n\nThis provided an additional source of biological evidence independent of both the neural network and BLAST.\n\n### Final Ensemble\n\nThe three prediction sources were combined using weighted averaging. The ensemble leveraged:\n\nlearned sequence patterns from the neural network,\nevolutionary evidence from sequence homology,\nand existing biological knowledge from curated annotations.\n\nThe combination consistently outperformed any individual component used in isolation.\n\n# What Was Important\n\nThe most significant performance gains came from hierarchical post-processing and multi-source biological evidence integration. GO hierarchy propagation consistently improved predictions by enforcing biologically valid relationships, while ensembling further strengthened robustness. Among external sources, GOA annotations provided the largest boost, followed by BLAST homology signals, which were particularly effective for proteins with close sequence matches.\n\nOn the modeling side, I experimented with multiple backbone choices, including different ESM2 variants (650M, 3B, and 15B) as well as ProtT5, with ESM2-3B performing best overall. The final tri-head Conv1D architecture consistently outperformed simpler MLPs and alternative designs.\n\nTraining stability was strongly influenced by the choice of IA scaling and positive class weights, and careful tuning was necessary for stable convergence.\n\nPerformance analysis showed that the model handled Molecular Function (MF) best, followed by Cellular Component (CC), while Biological Process (BP) remained the most challenging.\n\n# What I Tried That Didn't Help\n\nMore complex architectural variations generally did not improve validation performance. Increases in model complexity often led to marginal or inconsistent gains compared to the simpler tri-head Conv1D backbone combined with well-tuned loss weighting and biological post-processing. I also experimented with alternative predictors such as tree-based models, but these did not perform well and underperformed the neural architecture on validation.\n\n# Future Work\n\nFuture improvements could include multi-task loss designs that better capture dependencies between GO sub-ontologies, as well as more careful architectural modeling of hierarchical relationships directly within the network. Additionally, more robust cross-validation strategies beyond a simple train–validation split could help reduce variance in performance estimation and improve generalization, especially given the imbalance and hierarchical structure of the GO labels.\n\n# Sources\nhttps://www.kaggle.com/datasets/kaitonmh/cafa-6-embeddings\n\nhttps://www.kaggle.com/code/redberry/blast-basic-protein-function-predictions-cafa6\n\nhttps://www.kaggle.com/code/yongsukprasertsuk/cafa-6-goa-propagation\n\n" } }