![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F394451%2Fc65900bcc46d07dfb24a18cc7a46885b%2Foverview.png?generation=1782060340948206&alt=media\")
Hi everyone!
\nThanks the Hosts and the Kaggle Team for providing one more round of CAFA. We enjoyed being a part of it again. Below we provided a brief overview of our solution.
\nThe key parts of our solution are inspired by the algorithm we built during the CAFA 5 competition https://github.com/btbpanda/CAFA5-protein-function-prediction-2nd-place.
\nAfter the first look at the dataset we found that CAFA 5 and CAFA 6 are highly overlapped. Simplifying a bit, we can say that CAFA 6 is the up-to-date labelled subset of CAFA 5. So, the obvious idea to train is to try to rollback the sample definition to a large CAFA 5 like sample. It is not definitely a good idea from the performance point of view but something worth testing. For easy reference we would call
\nGeneral approach and used methods are represented on the Figure below:\n
Novel parts introduced in CAFA 6 solution are marked as red. The other difference is in the number of models. We built “Train” and “Train + Old train” data variants of base models, different embedding combinations for the base models, and more seeds of the stacker model.
\nBy the end of competition we can admit that the most important new update comes from the tf-idf articles part. The rest of the improvements are minor and mostly increase the solution robustness rather than bring new knowledge. \nBelow we describe all the steps in more detail.
\nBelow we listed all the features used in base models only. Additional stacker features are provided in the corresponding section.
\nWe tried to use the following embedding list:
\nWe used pre-trained models only. To our surprise, fine-tuning was not helpful in this task
\nDownloading abstracts: For each UniProt protein accession, PubMed identifiers (PMIDs) are retrieved from UniProt using the lit_pubmed_id field via the UniProt REST API. Duplicate PMIDs are removed, and the corresponding article titles and abstracts are then downloaded in batches from PubMed using the NCBI Entrez API.
\nArticles-derived protein embeddings: Using the downloaded article titles and abstracts, we computed TF-IDF features with 5,000 dimensions for each protein across all datasets. These TF-IDF vectors were then used as articles-derived protein embeddings in the downstream models.\nThe articles-derived TF-IDF embeddings were used cautiously, as they may introduce a minor data leakage risk. Specifically, we included only one PyBoost model in the final stack that used these TF-IDF embeddings concatenated with T5.
\nIn addition, we concatenated one-hot taxon features to the protein embedding. We selected only taxons that are good enough represented in both train and test features, around 30 totally, other taxons were merged into a single group. According to our estimation, adding one-hot taxon data improves the CAFA score of base models about 0.02-0.03 (based on CAFA5, didn’t perform extra evaluation on actual data)
\nAnother innovation introduced in CAFA 6 compared to the previous competition is evaluation over 3 independent test sets: no knowledge, limited knowledge, and partial knowledge. In 2 of those test sets we aim to make a prediction for the proteins that are not completely new but already have some labelling. The obvious idea was to utilize this knowledge somehow. We put some effort on making it work, but the effect was quite limited.
\nThe following approach we find a little bit useful, at least consistently not making it worse: let’s focus on boosting the model performance in a limited knowledge setup. In this scenario if we make a prediction for BP we already have some labels in MF or CC or both. We can do one-hot label representations of MF/CC targets, compress it via SVD, and use it as a features to predict BP. We repeated this trick for each single ontology using 512 SVD of two rest ontologies as features.
\nOur validation scheme could be separated on 2 parts:
\nBase models validation: we were not able to create a validation scheme better than a simple 5 fold CV. We had some experiments on the topic but other CV schemes lead to the models with less LB score.
\nStacking models validation: we didn’t use any local validation set to fit the stacker model. Instead, we used 100% data to train and rely on Public LB scores. Generally, it is a bad practice. However, we compensate it with a strong prior knowledge of model behavior obtained during CAFA 5, and with stochastic weighted average technique to make the learning procedure more robust.
\nAll models were trained to multi-label task, where target matrix of shape n_proteins x n_terms is predicted using feature matrix n_protein x n_features using binary cross entropy loss. In addition, some of models were trained using alternative conditional approach described in the next section\nTo train each model we selected top frequent terms from each ontology, top amount depends on the model type and provided below.
\nThe best performing models on both CV and LB are from the Gradient Boosting family. We used my own GBDT implementation called py-boost. I made it a few years ago especially to deal with extreme multi-output datasets since it was my main research area at that time. It works on GPU only and it is able to train multi-label tens on even hundreds times faster than popular well known implementations. You can check py-boost on the github or read our NeurIPS paper, where we explain all the strategies to speed-up multi-output training. The following Figure illustrates the speed up of our method (SketchBoost refers to the fast algorithm implemented inside the the py-boost library).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F394451%2F3697d358c6a1e00907f14b6417ee2365%2Fpb.png?generation=1782062128348183&alt=media\")
Locally we were able to fit 4.5к output (3000/1000/500) py-boost on a single V100 32GB GPU and it takes about 2 hours for a single fold.
\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.
\nFor the neural network models, we kept the architecture and training code the same as in the public notebook Pytorch,Keras,Etc 3 Blend, CAFA metric, etc. The main difference was the best cross-validated combination of hyper parameters averaged many times. However, instead of selecting only the top 2,000 terms, we used the same 13.5K targets as in the linear regression models. T5 and ESM embeddings were used as input features, without one-hot taxon features.
\nThis approach was discovered at the beginning of the CAFA 5 competition. That's why it makes some wrong assumptions about the data. But somehow it becomes useful for both CV and LB. The main advantage of this approach is utilising the OBO graph on the inference phase and it helps to make a prediction even for terms that were not used in training. Here are the main points:
\nWe assume that term can exist for the protein only if at least one of its parents exists. This is wrong. In practice, if the term exists, all its parents exist too because of the propagation rules.
We reformulate a classic multi-label scheme where the target matrix of shape (n_protein, n_terms) consists of 0 and 1 to the new scheme. Now targets can be 0, 1 and NaN. The term for the protein will have NaN value in the matrix if there is no parent term with value 1.
During model training phase NaN cells in the target matrix are masked and ignored
Now, our model outputs the conditional probabilities of the term in case at least one of its parents exists. On the inference phase we need to transform it back to the raw probabilities
Transformation is made in the order defined by the graph. When we process the term, all its parents are already processed and have raw probabilities. All terms are included in the scheme, even if they are not used in training. For the terms that were not used for training, we used prior mean.
While processing the term, we make another wrong assumption, that parents' probabilities for the term are independent. But if we assume that, according to [1], we can calculate raw probability for term as
$$\nP(GO:N) = P_{cond}(GO:N) * \\left(1 - \\prod_{K \\in Par(N)} (1 - P(GO:K))\\right)\n$$
\nRemember that while we process the term, all its parents already have raw probabilities calculated.
\nThe following Figure represents the main difference between classic multi label and alternative conditional scheme:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F394451%2F15d4a2847aca1b41f2f319d35ff51aa3%2Fcml_scheme.png?generation=1782062874146978&alt=media\")
On the Figure term GO:003 marked red, because it represents the situation that could not be observed in the data – GO:003 could not be 0 because its child GO:006 is 1. The figure represents our view and motivation on the moment of discovering the approach, not the real situation.
\nThat approach shows well performing on CAFA 5, so, it was accepted without any additional evaluation.
\nGBDT over DL embeddings: totally, we have 12 models in this category. The models were trained over:
\nModels were trained to predict 4.5k outputs. Finally, to reduce the number of inputs to stacker we averaged models over train dataset and embedding types and keep only 2 predictions:
\nLog reg over DL embeddings: we have the the same 12 log regs as GBDT, the difference is it was trained to predict 13.5k outputs, so finally after averaging we have 2 predictors:
\nFeed forward NN over DL embeddings: there are 2 models trained on CAFA 6 data only and combined T5+ESM embeddings:
\nCross SVD embeddings: we have 12 models in this category. Models are trained over:
\nEach ontology predictor uses 4 of them - trained on 2 rest ontologies SVD, so finally we have:
\nGBDT over DL embeddings + articles: we have a single GBDT model that uses articles tf-idf embedding + T5 + taxons. Model is trained on CAFA 6 Train only
\nSummary: 11 predictors for each ontology stacker
\nWe used a graph convolution network to aggregate all the predictions. It is trained for the node classification task where each node is a term and each protein is a graph (but all the proteins share the same adjacency matrix). As the node features we used base models predictions together with node embedding trained from scratch. We also added GO annotations features described in the next section.
\nBase model predictions: 11 base model predictions. Each model prediction is transformed to 4 channels. The important assumption here: we assume all base models have predictions for all terms. If the model doesn’t have it is filled with prior values. Priors are computed over Train or Train + Old train regarding the train sample used to fit the model. Channels are:
\nMore details are provided on the Figure below
\nGO annotation features: 38 binary features are computed based on GOA electronic labelling. Details are described in the next section. Simply saying, each binary feature represents what kind of relationship between protein and term is available in GOA
\nLearnable term embedding: just learnable from scratch embedding of term of size 8
\nSummary: each protein / term pair is represented by 11 * 4 + 38 + 8 = 90 channels
\nBinary labels of protein/term pair. All terms are used in training. Models were trained using binary cross entropy loss.
\nStackers were trained independently for each ontology using only CAFA 6 train data. Multistart with 2 random seeds were used, the prediction results are averaged in the end. Another important thing we should mention is about the metric. As far as we understand, we are evaluated only on protein/ontology pairs that are experimentally found. So, if we predict a term from ontology that does not exist, we will not get any penalty at all! That means, that we need actually to estimate the conditional probability of a term in case when its ontology exists. So, the correct way to fit a final stacker model for ontology X is to truncate the sample and take only the proteins that contain the terms from ontology X. It gives some small boost to the score and speeds-up the computations.
\nThe following Figure represents the details of stacking model implementation:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F394451%2F68c0b52f2c265ea4f0ebd4e51136a48f%2Fgcn_scheme.png?generation=1782067119252744&alt=media\")
We discovered the GO annotations dataset provided by the link. They provide not only labelling but also the evidence codes of each term. We separated the codes that Kaggle suppose to be experimental from the electronic codes. So, we can use electronic labelling as the features to predict experimental labels given by Kaggle. From our analysis we discovered that about 30% of electronic labels become experimental, so using it as a model feature performs better than just adding it as is.
\nThe electronic evidence codes provided by the labelling are not equal. They differ by:
\nUsing those information we computed 38 binary features like:
\nThis is only the idea of feature generation provided. For the extensive list of features please see our code.
\nGOA annotations releases: we started our work when release 226 was available. According to our knowledge, this release was also used to build the original competition data. Finally, by the end of the competition the latest available version was 228. So, this version was used to make a final prediction for the test set
\nOne thing we need to mention here. Since the 228 data was released after competition started, some very small amounts of test proteins get experimental labels in the latest version of GOA. If we detect such cases, we just add it to the final submission file without impact on the training procedure.
\nWe also used the OBO graph to make a post processing. The problem of ML model prediction of protein terms is that it is inconsistent. Following the propagation rule that is applied to the target if the term exists, all its parents are assumed to exist too. So, a consistent model will never predict the probability for parents lower than term probability. But our models don't care about it at all. So we can manually fix this situation. Our final term prediction is the average of term probability, maximum propagated children probability, and minimum propagated parents probability.\nThe figure below illustrated the postprocessing details:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F394451%2Fba9ef803a2832200d4f1786261925fc3%2Fpp_scheme.png?generation=1782063664822404&alt=media\")
We believe, it does not make sense to choose one best and simple approach, since:
\nWe consider using all the models and data sources to evaluate all the potential protein functions. Using GO electronic labelling as a feature in the stacker model is valid knowledge to predict, which terms will be experimentally found.
\nSolution code is open source now and available at https://github.com/btbpanda/CAFA6-updates
", "messages": [ { "id": "3477561", "postDate": "06/21/2026 17:51:33", "content": "Hi everyone!
\nThanks the Hosts and the Kaggle Team for providing one more round of CAFA. We enjoyed being a part of it again. Below we provided a brief overview of our solution.
\nThe key parts of our solution are inspired by the algorithm we built during the CAFA 5 competition https://github.com/btbpanda/CAFA5-protein-function-prediction-2nd-place.
\nAfter the first look at the dataset we found that CAFA 5 and CAFA 6 are highly overlapped. Simplifying a bit, we can say that CAFA 6 is the up-to-date labelled subset of CAFA 5. So, the obvious idea to train is to try to rollback the sample definition to a large CAFA 5 like sample. It is not definitely a good idea from the performance point of view but something worth testing. For easy reference we would call
\nGeneral approach and used methods are represented on the Figure below:\n
Novel parts introduced in CAFA 6 solution are marked as red. The other difference is in the number of models. We built “Train” and “Train + Old train” data variants of base models, different embedding combinations for the base models, and more seeds of the stacker model.
\nBy the end of competition we can admit that the most important new update comes from the tf-idf articles part. The rest of the improvements are minor and mostly increase the solution robustness rather than bring new knowledge. \nBelow we describe all the steps in more detail.
\nBelow we listed all the features used in base models only. Additional stacker features are provided in the corresponding section.
\nWe tried to use the following embedding list:
\nWe used pre-trained models only. To our surprise, fine-tuning was not helpful in this task
\nDownloading abstracts: For each UniProt protein accession, PubMed identifiers (PMIDs) are retrieved from UniProt using the lit_pubmed_id field via the UniProt REST API. Duplicate PMIDs are removed, and the corresponding article titles and abstracts are then downloaded in batches from PubMed using the NCBI Entrez API.
\nArticles-derived protein embeddings: Using the downloaded article titles and abstracts, we computed TF-IDF features with 5,000 dimensions for each protein across all datasets. These TF-IDF vectors were then used as articles-derived protein embeddings in the downstream models.\nThe articles-derived TF-IDF embeddings were used cautiously, as they may introduce a minor data leakage risk. Specifically, we included only one PyBoost model in the final stack that used these TF-IDF embeddings concatenated with T5.
\nIn addition, we concatenated one-hot taxon features to the protein embedding. We selected only taxons that are good enough represented in both train and test features, around 30 totally, other taxons were merged into a single group. According to our estimation, adding one-hot taxon data improves the CAFA score of base models about 0.02-0.03 (based on CAFA5, didn’t perform extra evaluation on actual data)
\nAnother innovation introduced in CAFA 6 compared to the previous competition is evaluation over 3 independent test sets: no knowledge, limited knowledge, and partial knowledge. In 2 of those test sets we aim to make a prediction for the proteins that are not completely new but already have some labelling. The obvious idea was to utilize this knowledge somehow. We put some effort on making it work, but the effect was quite limited.
\nThe following approach we find a little bit useful, at least consistently not making it worse: let’s focus on boosting the model performance in a limited knowledge setup. In this scenario if we make a prediction for BP we already have some labels in MF or CC or both. We can do one-hot label representations of MF/CC targets, compress it via SVD, and use it as a features to predict BP. We repeated this trick for each single ontology using 512 SVD of two rest ontologies as features.
\nOur validation scheme could be separated on 2 parts:
\nBase models validation: we were not able to create a validation scheme better than a simple 5 fold CV. We had some experiments on the topic but other CV schemes lead to the models with less LB score.
\nStacking models validation: we didn’t use any local validation set to fit the stacker model. Instead, we used 100% data to train and rely on Public LB scores. Generally, it is a bad practice. However, we compensate it with a strong prior knowledge of model behavior obtained during CAFA 5, and with stochastic weighted average technique to make the learning procedure more robust.
\nAll models were trained to multi-label task, where target matrix of shape n_proteins x n_terms is predicted using feature matrix n_protein x n_features using binary cross entropy loss. In addition, some of models were trained using alternative conditional approach described in the next section\nTo train each model we selected top frequent terms from each ontology, top amount depends on the model type and provided below.
\nThe best performing models on both CV and LB are from the Gradient Boosting family. We used my own GBDT implementation called py-boost. I made it a few years ago especially to deal with extreme multi-output datasets since it was my main research area at that time. It works on GPU only and it is able to train multi-label tens on even hundreds times faster than popular well known implementations. You can check py-boost on the github or read our NeurIPS paper, where we explain all the strategies to speed-up multi-output training. The following Figure illustrates the speed up of our method (SketchBoost refers to the fast algorithm implemented inside the the py-boost library).
\n
Locally we were able to fit 4.5к output (3000/1000/500) py-boost on a single V100 32GB GPU and it takes about 2 hours for a single fold.
\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.
\nFor the neural network models, we kept the architecture and training code the same as in the public notebook Pytorch,Keras,Etc 3 Blend, CAFA metric, etc. The main difference was the best cross-validated combination of hyper parameters averaged many times. However, instead of selecting only the top 2,000 terms, we used the same 13.5K targets as in the linear regression models. T5 and ESM embeddings were used as input features, without one-hot taxon features.
\nThis approach was discovered at the beginning of the CAFA 5 competition. That's why it makes some wrong assumptions about the data. But somehow it becomes useful for both CV and LB. The main advantage of this approach is utilising the OBO graph on the inference phase and it helps to make a prediction even for terms that were not used in training. Here are the main points:
\nWe assume that term can exist for the protein only if at least one of its parents exists. This is wrong. In practice, if the term exists, all its parents exist too because of the propagation rules.
We reformulate a classic multi-label scheme where the target matrix of shape (n_protein, n_terms) consists of 0 and 1 to the new scheme. Now targets can be 0, 1 and NaN. The term for the protein will have NaN value in the matrix if there is no parent term with value 1.
During model training phase NaN cells in the target matrix are masked and ignored
Now, our model outputs the conditional probabilities of the term in case at least one of its parents exists. On the inference phase we need to transform it back to the raw probabilities
Transformation is made in the order defined by the graph. When we process the term, all its parents are already processed and have raw probabilities. All terms are included in the scheme, even if they are not used in training. For the terms that were not used for training, we used prior mean.
While processing the term, we make another wrong assumption, that parents' probabilities for the term are independent. But if we assume that, according to [1], we can calculate raw probability for term as
$$\nP(GO:N) = P_{cond}(GO:N) * \\left(1 - \\prod_{K \\in Par(N)} (1 - P(GO:K))\\right)\n$$
\nRemember that while we process the term, all its parents already have raw probabilities calculated.
\nThe following Figure represents the main difference between classic multi label and alternative conditional scheme:
\n
On the Figure term GO:003 marked red, because it represents the situation that could not be observed in the data – GO:003 could not be 0 because its child GO:006 is 1. The figure represents our view and motivation on the moment of discovering the approach, not the real situation.
\nThat approach shows well performing on CAFA 5, so, it was accepted without any additional evaluation.
\nGBDT over DL embeddings: totally, we have 12 models in this category. The models were trained over:
\nModels were trained to predict 4.5k outputs. Finally, to reduce the number of inputs to stacker we averaged models over train dataset and embedding types and keep only 2 predictions:
\nLog reg over DL embeddings: we have the the same 12 log regs as GBDT, the difference is it was trained to predict 13.5k outputs, so finally after averaging we have 2 predictors:
\nFeed forward NN over DL embeddings: there are 2 models trained on CAFA 6 data only and combined T5+ESM embeddings:
\nCross SVD embeddings: we have 12 models in this category. Models are trained over:
\nEach ontology predictor uses 4 of them - trained on 2 rest ontologies SVD, so finally we have:
\nGBDT over DL embeddings + articles: we have a single GBDT model that uses articles tf-idf embedding + T5 + taxons. Model is trained on CAFA 6 Train only
\nSummary: 11 predictors for each ontology stacker
\nWe used a graph convolution network to aggregate all the predictions. It is trained for the node classification task where each node is a term and each protein is a graph (but all the proteins share the same adjacency matrix). As the node features we used base models predictions together with node embedding trained from scratch. We also added GO annotations features described in the next section.
\nBase model predictions: 11 base model predictions. Each model prediction is transformed to 4 channels. The important assumption here: we assume all base models have predictions for all terms. If the model doesn’t have it is filled with prior values. Priors are computed over Train or Train + Old train regarding the train sample used to fit the model. Channels are:
\nMore details are provided on the Figure below
\nGO annotation features: 38 binary features are computed based on GOA electronic labelling. Details are described in the next section. Simply saying, each binary feature represents what kind of relationship between protein and term is available in GOA
\nLearnable term embedding: just learnable from scratch embedding of term of size 8
\nSummary: each protein / term pair is represented by 11 * 4 + 38 + 8 = 90 channels
\nBinary labels of protein/term pair. All terms are used in training. Models were trained using binary cross entropy loss.
\nStackers were trained independently for each ontology using only CAFA 6 train data. Multistart with 2 random seeds were used, the prediction results are averaged in the end. Another important thing we should mention is about the metric. As far as we understand, we are evaluated only on protein/ontology pairs that are experimentally found. So, if we predict a term from ontology that does not exist, we will not get any penalty at all! That means, that we need actually to estimate the conditional probability of a term in case when its ontology exists. So, the correct way to fit a final stacker model for ontology X is to truncate the sample and take only the proteins that contain the terms from ontology X. It gives some small boost to the score and speeds-up the computations.
\nThe following Figure represents the details of stacking model implementation:
\n
We discovered the GO annotations dataset provided by the link. They provide not only labelling but also the evidence codes of each term. We separated the codes that Kaggle suppose to be experimental from the electronic codes. So, we can use electronic labelling as the features to predict experimental labels given by Kaggle. From our analysis we discovered that about 30% of electronic labels become experimental, so using it as a model feature performs better than just adding it as is.
\nThe electronic evidence codes provided by the labelling are not equal. They differ by:
\nUsing those information we computed 38 binary features like:
\nThis is only the idea of feature generation provided. For the extensive list of features please see our code.
\nGOA annotations releases: we started our work when release 226 was available. According to our knowledge, this release was also used to build the original competition data. Finally, by the end of the competition the latest available version was 228. So, this version was used to make a final prediction for the test set
\nOne thing we need to mention here. Since the 228 data was released after competition started, some very small amounts of test proteins get experimental labels in the latest version of GOA. If we detect such cases, we just add it to the final submission file without impact on the training procedure.
\nWe also used the OBO graph to make a post processing. The problem of ML model prediction of protein terms is that it is inconsistent. Following the propagation rule that is applied to the target if the term exists, all its parents are assumed to exist too. So, a consistent model will never predict the probability for parents lower than term probability. But our models don't care about it at all. So we can manually fix this situation. Our final term prediction is the average of term probability, maximum propagated children probability, and minimum propagated parents probability.\nThe figure below illustrated the postprocessing details:
\n
We believe, it does not make sense to choose one best and simple approach, since:
\nWe consider using all the models and data sources to evaluate all the potential protein functions. Using GO electronic labelling as a feature in the stacker model is valid knowledge to predict, which terms will be experimentally found.
\nSolution code is open source now and available at https://github.com/btbpanda/CAFA6-updates
", "rawMarkdown": "Hi everyone!\n\nThanks the Hosts and the Kaggle Team for providing one more round of CAFA. We enjoyed being a part of it again. Below we provided a brief overview of our solution.\n\n# Solution overview and relationship with CAFA 5\n\nThe key parts of our solution are inspired by the algorithm we built during the CAFA 5 competition https://github.com/btbpanda/CAFA5-protein-function-prediction-2nd-place. \n\nAfter the first look at the dataset we found that CAFA 5 and CAFA 6 are highly overlapped. Simplifying a bit, we can say that CAFA 6 is the up-to-date labelled subset of CAFA 5. So, the obvious idea to train is to try to rollback the sample definition to a large CAFA 5 like sample. It is not definitely a good idea from the performance point of view but something worth testing. For easy reference we would call\n\n* **Train**: train dataset provided at CAFA 6 competition\n* **Old train**: the subset of CAFA 5 proteins that were not represented in Train. Up-to-date labelling was built from the actual uniprot database.\n\nGeneral approach and used methods are represented on the Figure below:\n\n\nNovel parts introduced in CAFA 6 solution are marked as red. The other difference is in the number of models. We built **“Train”** and **“Train + Old train”** data variants of base models, different embedding combinations for the base models, and more seeds of the stacker model.\n\nBy the end of competition we can admit that the most important new update comes from the tf-idf articles part. The rest of the improvements are minor and mostly increase the solution robustness rather than bring new knowledge. \nBelow we describe all the steps in more detail.\n\n# Sequence embedding and base models features\n\nBelow we listed all the features used in base models only. Additional stacker features are provided in the corresponding section.\n\n### 1. Protein DL embedding\n\nWe tried to use the following embedding list: \n\n* T5 \n* esm2-small\n\nWe used pre-trained models only. To our surprise, fine-tuning was not helpful in this task\n\n### 2. Articles TF-DF embedding\n\n**Downloading abstracts**: For each UniProt protein accession, PubMed identifiers (PMIDs) are retrieved from UniProt using the lit_pubmed_id field via the UniProt REST API. Duplicate PMIDs are removed, and the corresponding article titles and abstracts are then downloaded in batches from PubMed using the NCBI Entrez API.\n\n**Articles-derived protein embeddings**: Using the downloaded article titles and abstracts, we computed TF-IDF features with 5,000 dimensions for each protein across all datasets. These TF-IDF vectors were then used as articles-derived protein embeddings in the downstream models.\nThe articles-derived TF-IDF embeddings were used cautiously, as they may introduce a minor data leakage risk. Specifically, we included only one PyBoost model in the final stack that used these TF-IDF embeddings concatenated with T5.\n\n### 3. Taxon one-hot representation\n\nIn addition, we concatenated one-hot taxon features to the protein embedding. We selected only taxons that are good enough represented in both train and test features, around 30 totally, other taxons were merged into a single group. According to our estimation, adding one-hot taxon data improves the CAFA score of base models about 0.02-0.03 (based on CAFA5, didn’t perform extra evaluation on actual data)\n\n### 4. Cross ontology terms SVD\n\nAnother innovation introduced in CAFA 6 compared to the previous competition is evaluation over 3 independent test sets: **no knowledge**, **limited knowledge**, and **partial knowledge**. In 2 of those test sets we aim to make a prediction for the proteins that are not completely new but already have some labelling. The obvious idea was to utilize this knowledge somehow. We put some effort on making it work, but the effect was quite limited.\n\nThe following approach we find a little bit useful, at least consistently not making it worse: let’s focus on boosting the model performance in a limited knowledge setup. In this scenario if we make a prediction for BP we already have some labels in MF or CC or both. We can do one-hot label representations of MF/CC targets, compress it via SVD, and use it as a features to predict BP. We repeated this trick for each single ontology using 512 SVD of two rest ontologies as features.\n\n# Validation\n\nOur validation scheme could be separated on 2 parts:\n\n**Base models validation**: we were not able to create a validation scheme better than a simple 5 fold CV. We had some experiments on the topic but other CV schemes lead to the models with less LB score.\n\n**Stacking models validation**: we didn’t use any local validation set to fit the stacker model. Instead, we used 100% data to train and rely on Public LB scores. Generally, it is a bad practice. However, we compensate it with a strong prior knowledge of model behavior obtained during CAFA 5, and with stochastic weighted average technique to make the learning procedure more robust.\n\n# Base models\n\nAll models were trained to multi-label task, where target matrix of shape *n_proteins x n_terms* is predicted using feature matrix *n_protein x n_features* using binary cross entropy loss. In addition, some of models were trained using alternative conditional approach described in the next section\nTo train each model we selected top frequent terms from each ontology, top amount depends on the model type and provided below.\n\n### Py-Boost\n\nThe best performing models on both CV and LB are from the Gradient Boosting family. We used my own GBDT implementation called py-boost. I made it a few years ago especially to deal with extreme multi-output datasets since it was my main research area at that time. It works on GPU only and it is able to train multi-label tens on even hundreds times faster than popular well known implementations. You can check **py-boost** on the [github](https://github.com/sb-ai-lab/Py-Boost) or read our [NeurIPS paper](https://proceedings.neurips.cc/paper_files/paper/2022/file/a36c3dbe676fa8445715a31a90c66ab3-Paper-Conference.pdf), where we explain all the strategies to speed-up multi-output training. The following Figure illustrates the speed up of our method (SketchBoost refers to the fast algorithm implemented inside the the py-boost library).\n\n\n\nLocally we were able to fit 4.5к output (3000/1000/500) py-boost on a single V100 32GB GPU and it takes about 2 hours for a single fold.\n\n### Logistic Regression\n\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.\n\n### Neural Network \n\nFor the neural network models, we kept the architecture and training code the same as in the public notebook [Pytorch,Keras,Etc 3 Blend, CAFA metric, etc](https://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc#Optimizer-%2522Sophia%2522-sometimes-better-than-Adam). The main difference was the best cross-validated combination of hyper parameters averaged many times. However, instead of selecting only the top 2,000 terms, we used the same 13.5K targets as in the linear regression models. T5 and ESM embeddings were used as input features, without one-hot taxon features.\n\n# Alternative modelling approach: predicting the conditional probabilities\n\n\nThis approach was discovered at the beginning of the CAFA 5 competition. That's why it makes some wrong assumptions about the data. But somehow it becomes useful for both CV and LB. The main advantage of this approach is utilising the OBO graph on the inference phase and it helps to make a prediction even for terms that were not used in training. Here are the main points:\n\n1. We assume that term can exist for the protein only if **at least one of its parents exists**. This is wrong. In practice, if the term exists, all its parents exist too because of the propagation rules.\n\n2. We reformulate a classic multi-label scheme where the target matrix of shape (n_protein, n_terms) consists of 0 and 1 to the new scheme. Now targets can be 0, 1 and NaN. The term for the protein will have NaN value in the matrix if there is no parent term with value 1. \n\n3. During model training phase NaN cells in the target matrix are masked and ignored\n\n4. Now, our model outputs **the conditional probabilities of the term in case at least one of its parents exists**. On the inference phase we need to transform it back to the raw probabilities\n\n5. Transformation is made in the order defined by the graph. When we process the term, all its parents are already processed and have raw probabilities. All terms are included in the scheme, even if they are not used in training. For the terms that were not used for training, we used prior mean.\n\n6. While processing the term, we make another wrong assumption, that parents' probabilities for the term are independent. But if we assume that, according to [1], we can calculate raw probability for term as \n\n$$\nP(GO:N) = P_{cond}(GO:N) * \\left(1 - \\prod_{K \\in Par(N)} (1 - P(GO:K))\\right)\n$$\n\nRemember that while we process the term, all its parents already have raw probabilities calculated.\n\nThe following Figure represents the main difference between classic multi label and alternative conditional scheme:\n\n\n\nOn the Figure term GO:003 marked red, because it represents the situation that could not be observed in the data – GO:003 could not be 0 because its child GO:006 is 1. The figure represents our view and motivation on the moment of discovering the approach, not the real situation. \n\nThat approach shows well performing on CAFA 5, so, it was accepted without any additional evaluation.\n\n# Extensive models list\n\n**GBDT over DL embeddings:** totally, we have 12 models in this category. The models were trained over:\n* Train Data / Train + Old Train data\n* T5 + taxon / ESM + taxon / T5 + ESM + taxon\n* Raw / Conditional multilabel \n\nModels were trained to predict 4.5k outputs. Finally, to reduce the number of inputs to stacker we averaged models over train dataset and embedding types and keep only 2 predictions:\n* GBDT avg conditional\n* GBDT avg raw \n\n**Log reg over DL embeddings**: we have the the same 12 log regs as GBDT, the difference is it was trained to predict 13.5k outputs, so finally after averaging we have 2 predictors:\n* Log reg avg conditional\n* Log reg avg raw\n\n**Feed forward NN over DL embeddings**: there are 2 models trained on CAFA 6 data only and combined T5+ESM embeddings:\n* NN conditional\n* NN raw\n\n**Cross SVD embeddings**: we have 12 models in this category. Models are trained over: \n* GBDT / Log reg\n* Raw / conditional \n* BP / MF / CC\n\nEach ontology predictor uses 4 of them - trained on 2 rest ontologies SVD, so finally we have:\n\n* Log reg cross SVD conditional\n* Log reg cross SVD raw\n* GBDT cross SVD conditional\n* GBDT cross SVD raw\n\n**GBDT over DL embeddings + articles**: we have a single GBDT model that uses articles tf-idf embedding + T5 + taxons. Model is trained on CAFA 6 Train only\n\n**Summary**: 11 predictors for each ontology stacker \n\n# Stacking with GCN\n\n\nWe used a graph convolution network to aggregate all the predictions. It is trained for the node classification task where each node is a term and each protein is a graph (but all the proteins share the same adjacency matrix). As the node features we used base models predictions together with node embedding trained from scratch. We also added GO annotations features described in the next section.\n\n### Features are:\n\n**Base model predictions:** 11 base model predictions. Each model prediction is transformed to 4 channels. The important assumption here: **we assume all base models have predictions for all terms**. If the model doesn’t have it is filled with prior values. Priors are computed over Train or Train + Old train regarding the train sample used to fit the model. Channels are:\n* Prior prediction flag\n* Prediction logit value\n* For conditional: propagation type 1 logit, for raw just copy prediction logit\n* For conditional: propagation type 2 logit, for raw just copy prediction logit\n\nMore details are provided on the Figure below\n\n**GO annotation features:** 38 binary features are computed based on GOA electronic labelling. Details are described in the next section. Simply saying, each binary feature represents what kind of relationship between protein and term is available in GOA\n\n**Learnable term embedding**: just learnable from scratch embedding of term of size 8\n\n**Summary:** each protein / term pair is represented by 11 * 4 + 38 + 8 = 90 channels\n\n### Targets are:\nBinary labels of protein/term pair. All terms are used in training. Models were trained using binary cross entropy loss.\n\n### Train sample\nStackers were trained independently for each ontology using only CAFA 6 train data. Multistart with 2 random seeds were used, the prediction results are averaged in the end. Another important thing we should mention is about the metric. As far as we understand, we are evaluated only on protein/ontology pairs that are experimentally found. So, if we predict a term from ontology that does not exist, we will not get any penalty at all! That means, that we need actually to estimate **the conditional probability of a term in case when its ontology exists**. So, the correct way to fit a final stacker model for ontology X is to truncate the sample and take only the proteins that contain the terms from ontology X. It gives some small boost to the score and speeds-up the computations.\n\nThe following Figure represents the details of stacking model implementation:\n\n\n\n# GO Annotations\n\nWe discovered the GO annotations dataset provided by the [link](http://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/). They provide not only labelling but also the evidence codes of each term. We separated the codes that Kaggle suppose to be experimental from the electronic codes. So, we can use electronic labelling as the features to predict experimental labels given by Kaggle. From our analysis we discovered that about 30% of electronic labels become experimental, so using it as a model feature performs better than just adding it as is. \n\nThe electronic evidence codes provided by the labelling are not equal. They differ by:\n\n* The evidence code like IEA / IBA / ISO / ISS etc. Even if we don’t understand the difference, may be ML models can make it work \n* Type of relation like part_of / acts_upstream / is_active_in\n* Evidence counts. A protein could have multiple evidence for a term with different codes. Having multiple evidences highly increase the probability of transformation to experimental, for example for 3 codes it is more that 50% compared to 30% for 1 code\n\nUsing those information we computed 38 binary features like: \n\n* Protein has at least 1/2/3/4/5 evidences for term (propagated labels counts)\n* Protein has at least 1/2/3/4/5 evidences for term (within only raw labels)\n* Protein is related with term as “part_of” \n* Protein has an evidence of type IEA for term \n…\n\nThis is only the idea of feature generation provided. For the extensive list of features please see our code. \n\n**GOA annotations releases:** we started our work when release 226 was available. According to our knowledge, this release was also used to build the original competition data. Finally, by the end of the competition the latest available version was 228. So, this version was used to make a final prediction for the test set\n\nOne thing we need to mention here. Since the 228 data was released after competition started, some very small amounts of test proteins get experimental labels in the latest version of GOA. If we detect such cases, we just add it to the final submission file without impact on the training procedure. \n\n# Postprocessing\n\nWe also used the OBO graph to make a post processing. The problem of ML model prediction of protein terms is that it is inconsistent. Following the propagation rule that is applied to the target if the term exists, all its parents are assumed to exist too. So, a consistent model will never predict the probability for parents lower than term probability. But our models don't care about it at all. So we can manually fix this situation. Our final term prediction is **the average of term probability, maximum propagated children probability, and minimum propagated parents probability**.\nThe figure below illustrated the postprocessing details:\n\n\n\n# Simplified model\n\nWe believe, it does not make sense to choose one best and simple approach, since:\n1. There is no need to make fast and efficient online inference for such kind of research task\n2. Any single model scores much less that combination of multiple approaches\n\nWe consider using all the models and data sources to evaluate all the potential protein functions. Using GO electronic labelling as a feature in the stacker model is valid knowledge to predict, which terms will be experimentally found.\n\n# Source code\n\nSolution code is open source now and available at https://github.com/btbpanda/CAFA6-updates", "votes": null }, { "id": "3478655", "postDate": "06/22/2026 22:16:07", "content": "Congratulations! I think the literature-based information is an interesting advance on your already-excellent CAFA 5 model. Do you know how much your final LB was improved by the literature abstracts?
", "rawMarkdown": "Congratulations! I think the literature-based information is an interesting advance on your already-excellent CAFA 5 model. Do you know how much your final LB was improved by the literature abstracts?", "votes": null }, { "id": "3480406", "postDate": "06/24/2026 19:33:20", "content": "Hi John. Thanks for your feedback. It is difficult to estimate the impact properly since we don't have private LB for all submissions. But, as an upper bound you can consider 0.04. Before @yuanlao153 shared a very nice walkthrough on his solution, which, he claims, is very close to our CAFA5 approach. So, we can consider 3rd and 2nd place difference is the effect of all our updates. I would say, 70-80% of this effect or even more is gained from articles.
", "rawMarkdown": "Hi John. Thanks for your feedback. It is difficult to estimate the impact properly since we don't have private LB for all submissions. But, as an upper bound you can consider 0.04. Before @yuanlao153 shared a very nice walkthrough on his solution, which, he claims, is very close to our CAFA5 approach. So, we can consider 3rd and 2nd place difference is the effect of all our updates. I would say, 70-80% of this effect or even more is gained from articles.", "votes": null }, { "id": "3481858", "postDate": "06/26/2026 14:14:36", "content": "hi, sir, I am curious about the way that you only used the literature in one model. And how you get rid of the bad result of long regression for the protein which is not rare species(by gcn?)
", "rawMarkdown": "hi, sir, I am curious about the way that you only used the literature in one model. And how you get rid of the bad result of long regression for the protein which is not rare species(by gcn?)", "votes": null }, { "id": "3484826", "postDate": "06/30/2026 19:25:02", "content": "Hi! The reason why we avoid using the literature in many models is the potential data leakage risk. We don't want the final model rely too much on the literature since we don't know in advance how good private test proteins are represented in the papers
\nFor the second question, sorry, I didn't get the idea. Can you please clarify, what do you mean by \"long regression for the protein which is not rare species\"?
", "rawMarkdown": "Hi! The reason why we avoid using the literature in many models is the potential data leakage risk. We don't want the final model rely too much on the literature since we don't know in advance how good private test proteins are represented in the papers\n\nFor the second question, sorry, I didn't get the idea. Can you please clarify, what do you mean by \"long regression for the protein which is not rare species\"?", "votes": null }, { "id": "3488983", "postDate": "07/05/2026 11:12:19", "content": "Thanks for replying, I just notice that:\n\"Logistic Regression\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.\", in your note. \nHow you find this situation and get rid of the result of the poor performance result of this method or model
", "rawMarkdown": "Thanks for replying, I just notice that:\n\"Logistic Regression\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.\", in your note. \nHow you find this situation and get rid of the result of the poor performance result of this method or model", "votes": null }, { "id": "3492887", "postDate": "07/07/2026 06:12:41", "content": "Thanks for clarification. We didn't do anything special. We rely on the stacker model instead. By design, it should decide when to attend on those poor predictors and when just better to ignore and base the decision on other types of predictors
", "rawMarkdown": "Thanks for clarification. We didn't do anything special. We rely on the stacker model instead. By design, it should decide when to attend on those poor predictors and when just better to ignore and base the decision on other types of predictors", "votes": null }, { "id": "3493119", "postDate": "07/07/2026 07:41:05", "content": "It is a good design
", "rawMarkdown": "It is a good design", "votes": null } ], "comments": [ { "id": 3478655, "author_name": "jbomitchell", "author_url": "", "post_date": "06/22/2026 22:16:07", "content": "Congratulations! I think the literature-based information is an interesting advance on your already-excellent CAFA 5 model. Do you know how much your final LB was improved by the literature abstracts?
", "votes": null, "replies": [ { "id": 3480406, "author_name": "btbpanda", "author_url": "", "post_date": "06/24/2026 19:33:20", "content": "Hi John. Thanks for your feedback. It is difficult to estimate the impact properly since we don't have private LB for all submissions. But, as an upper bound you can consider 0.04. Before @yuanlao153 shared a very nice walkthrough on his solution, which, he claims, is very close to our CAFA5 approach. So, we can consider 3rd and 2nd place difference is the effect of all our updates. I would say, 70-80% of this effect or even more is gained from articles.
", "votes": null, "replies": [] } ] }, { "id": 3481858, "author_name": "aizealotwu", "author_url": "", "post_date": "06/26/2026 14:14:36", "content": "hi, sir, I am curious about the way that you only used the literature in one model. And how you get rid of the bad result of long regression for the protein which is not rare species(by gcn?)
", "votes": null, "replies": [ { "id": 3484826, "author_name": "btbpanda", "author_url": "", "post_date": "06/30/2026 19:25:02", "content": "Hi! The reason why we avoid using the literature in many models is the potential data leakage risk. We don't want the final model rely too much on the literature since we don't know in advance how good private test proteins are represented in the papers
\nFor the second question, sorry, I didn't get the idea. Can you please clarify, what do you mean by \"long regression for the protein which is not rare species\"?
", "votes": null, "replies": [ { "id": 3488983, "author_name": "aizealotwu", "author_url": "", "post_date": "07/05/2026 11:12:19", "content": "Thanks for replying, I just notice that:\n\"Logistic Regression\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.\", in your note. \nHow you find this situation and get rid of the result of the poor performance result of this method or model
", "votes": null, "replies": [ { "id": 3492887, "author_name": "btbpanda", "author_url": "", "post_date": "07/07/2026 06:12:41", "content": "Thanks for clarification. We didn't do anything special. We rely on the stacker model instead. By design, it should decide when to attend on those poor predictors and when just better to ignore and base the decision on other types of predictors
", "votes": null, "replies": [ { "id": 3493119, "author_name": "aizealotwu", "author_url": "", "post_date": "07/07/2026 07:41:05", "content": "It is a good design
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "3477561": "Hi everyone!\n\nThanks the Hosts and the Kaggle Team for providing one more round of CAFA. We enjoyed being a part of it again. Below we provided a brief overview of our solution.\n\n# Solution overview and relationship with CAFA 5\n\nThe key parts of our solution are inspired by the algorithm we built during the CAFA 5 competition https://github.com/btbpanda/CAFA5-protein-function-prediction-2nd-place. \n\nAfter the first look at the dataset we found that CAFA 5 and CAFA 6 are highly overlapped. Simplifying a bit, we can say that CAFA 6 is the up-to-date labelled subset of CAFA 5. So, the obvious idea to train is to try to rollback the sample definition to a large CAFA 5 like sample. It is not definitely a good idea from the performance point of view but something worth testing. For easy reference we would call\n\n* **Train**: train dataset provided at CAFA 6 competition\n* **Old train**: the subset of CAFA 5 proteins that were not represented in Train. Up-to-date labelling was built from the actual uniprot database.\n\nGeneral approach and used methods are represented on the Figure below:\n\n\nNovel parts introduced in CAFA 6 solution are marked as red. The other difference is in the number of models. We built **“Train”** and **“Train + Old train”** data variants of base models, different embedding combinations for the base models, and more seeds of the stacker model.\n\nBy the end of competition we can admit that the most important new update comes from the tf-idf articles part. The rest of the improvements are minor and mostly increase the solution robustness rather than bring new knowledge. \nBelow we describe all the steps in more detail.\n\n# Sequence embedding and base models features\n\nBelow we listed all the features used in base models only. Additional stacker features are provided in the corresponding section.\n\n### 1. Protein DL embedding\n\nWe tried to use the following embedding list: \n\n* T5 \n* esm2-small\n\nWe used pre-trained models only. To our surprise, fine-tuning was not helpful in this task\n\n### 2. Articles TF-DF embedding\n\n**Downloading abstracts**: For each UniProt protein accession, PubMed identifiers (PMIDs) are retrieved from UniProt using the lit_pubmed_id field via the UniProt REST API. Duplicate PMIDs are removed, and the corresponding article titles and abstracts are then downloaded in batches from PubMed using the NCBI Entrez API.\n\n**Articles-derived protein embeddings**: Using the downloaded article titles and abstracts, we computed TF-IDF features with 5,000 dimensions for each protein across all datasets. These TF-IDF vectors were then used as articles-derived protein embeddings in the downstream models.\nThe articles-derived TF-IDF embeddings were used cautiously, as they may introduce a minor data leakage risk. Specifically, we included only one PyBoost model in the final stack that used these TF-IDF embeddings concatenated with T5.\n\n### 3. Taxon one-hot representation\n\nIn addition, we concatenated one-hot taxon features to the protein embedding. We selected only taxons that are good enough represented in both train and test features, around 30 totally, other taxons were merged into a single group. According to our estimation, adding one-hot taxon data improves the CAFA score of base models about 0.02-0.03 (based on CAFA5, didn’t perform extra evaluation on actual data)\n\n### 4. Cross ontology terms SVD\n\nAnother innovation introduced in CAFA 6 compared to the previous competition is evaluation over 3 independent test sets: **no knowledge**, **limited knowledge**, and **partial knowledge**. In 2 of those test sets we aim to make a prediction for the proteins that are not completely new but already have some labelling. The obvious idea was to utilize this knowledge somehow. We put some effort on making it work, but the effect was quite limited.\n\nThe following approach we find a little bit useful, at least consistently not making it worse: let’s focus on boosting the model performance in a limited knowledge setup. In this scenario if we make a prediction for BP we already have some labels in MF or CC or both. We can do one-hot label representations of MF/CC targets, compress it via SVD, and use it as a features to predict BP. We repeated this trick for each single ontology using 512 SVD of two rest ontologies as features.\n\n# Validation\n\nOur validation scheme could be separated on 2 parts:\n\n**Base models validation**: we were not able to create a validation scheme better than a simple 5 fold CV. We had some experiments on the topic but other CV schemes lead to the models with less LB score.\n\n**Stacking models validation**: we didn’t use any local validation set to fit the stacker model. Instead, we used 100% data to train and rely on Public LB scores. Generally, it is a bad practice. However, we compensate it with a strong prior knowledge of model behavior obtained during CAFA 5, and with stochastic weighted average technique to make the learning procedure more robust.\n\n# Base models\n\nAll models were trained to multi-label task, where target matrix of shape *n_proteins x n_terms* is predicted using feature matrix *n_protein x n_features* using binary cross entropy loss. In addition, some of models were trained using alternative conditional approach described in the next section\nTo train each model we selected top frequent terms from each ontology, top amount depends on the model type and provided below.\n\n### Py-Boost\n\nThe best performing models on both CV and LB are from the Gradient Boosting family. We used my own GBDT implementation called py-boost. I made it a few years ago especially to deal with extreme multi-output datasets since it was my main research area at that time. It works on GPU only and it is able to train multi-label tens on even hundreds times faster than popular well known implementations. You can check **py-boost** on the [github](https://github.com/sb-ai-lab/Py-Boost) or read our [NeurIPS paper](https://proceedings.neurips.cc/paper_files/paper/2022/file/a36c3dbe676fa8445715a31a90c66ab3-Paper-Conference.pdf), where we explain all the strategies to speed-up multi-output training. The following Figure illustrates the speed up of our method (SketchBoost refers to the fast algorithm implemented inside the the py-boost library).\n\n\n\nLocally we were able to fit 4.5к output (3000/1000/500) py-boost on a single V100 32GB GPU and it takes about 2 hours for a single fold.\n\n### Logistic Regression\n\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.\n\n### Neural Network \n\nFor the neural network models, we kept the architecture and training code the same as in the public notebook [Pytorch,Keras,Etc 3 Blend, CAFA metric, etc](https://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc#Optimizer-%2522Sophia%2522-sometimes-better-than-Adam). The main difference was the best cross-validated combination of hyper parameters averaged many times. However, instead of selecting only the top 2,000 terms, we used the same 13.5K targets as in the linear regression models. T5 and ESM embeddings were used as input features, without one-hot taxon features.\n\n# Alternative modelling approach: predicting the conditional probabilities\n\n\nThis approach was discovered at the beginning of the CAFA 5 competition. That's why it makes some wrong assumptions about the data. But somehow it becomes useful for both CV and LB. The main advantage of this approach is utilising the OBO graph on the inference phase and it helps to make a prediction even for terms that were not used in training. Here are the main points:\n\n1. We assume that term can exist for the protein only if **at least one of its parents exists**. This is wrong. In practice, if the term exists, all its parents exist too because of the propagation rules.\n\n2. We reformulate a classic multi-label scheme where the target matrix of shape (n_protein, n_terms) consists of 0 and 1 to the new scheme. Now targets can be 0, 1 and NaN. The term for the protein will have NaN value in the matrix if there is no parent term with value 1. \n\n3. During model training phase NaN cells in the target matrix are masked and ignored\n\n4. Now, our model outputs **the conditional probabilities of the term in case at least one of its parents exists**. On the inference phase we need to transform it back to the raw probabilities\n\n5. Transformation is made in the order defined by the graph. When we process the term, all its parents are already processed and have raw probabilities. All terms are included in the scheme, even if they are not used in training. For the terms that were not used for training, we used prior mean.\n\n6. While processing the term, we make another wrong assumption, that parents' probabilities for the term are independent. But if we assume that, according to [1], we can calculate raw probability for term as \n\n$$\nP(GO:N) = P_{cond}(GO:N) * \\left(1 - \\prod_{K \\in Par(N)} (1 - P(GO:K))\\right)\n$$\n\nRemember that while we process the term, all its parents already have raw probabilities calculated.\n\nThe following Figure represents the main difference between classic multi label and alternative conditional scheme:\n\n\n\nOn the Figure term GO:003 marked red, because it represents the situation that could not be observed in the data – GO:003 could not be 0 because its child GO:006 is 1. The figure represents our view and motivation on the moment of discovering the approach, not the real situation. \n\nThat approach shows well performing on CAFA 5, so, it was accepted without any additional evaluation.\n\n# Extensive models list\n\n**GBDT over DL embeddings:** totally, we have 12 models in this category. The models were trained over:\n* Train Data / Train + Old Train data\n* T5 + taxon / ESM + taxon / T5 + ESM + taxon\n* Raw / Conditional multilabel \n\nModels were trained to predict 4.5k outputs. Finally, to reduce the number of inputs to stacker we averaged models over train dataset and embedding types and keep only 2 predictions:\n* GBDT avg conditional\n* GBDT avg raw \n\n**Log reg over DL embeddings**: we have the the same 12 log regs as GBDT, the difference is it was trained to predict 13.5k outputs, so finally after averaging we have 2 predictors:\n* Log reg avg conditional\n* Log reg avg raw\n\n**Feed forward NN over DL embeddings**: there are 2 models trained on CAFA 6 data only and combined T5+ESM embeddings:\n* NN conditional\n* NN raw\n\n**Cross SVD embeddings**: we have 12 models in this category. Models are trained over: \n* GBDT / Log reg\n* Raw / conditional \n* BP / MF / CC\n\nEach ontology predictor uses 4 of them - trained on 2 rest ontologies SVD, so finally we have:\n\n* Log reg cross SVD conditional\n* Log reg cross SVD raw\n* GBDT cross SVD conditional\n* GBDT cross SVD raw\n\n**GBDT over DL embeddings + articles**: we have a single GBDT model that uses articles tf-idf embedding + T5 + taxons. Model is trained on CAFA 6 Train only\n\n**Summary**: 11 predictors for each ontology stacker \n\n# Stacking with GCN\n\n\nWe used a graph convolution network to aggregate all the predictions. It is trained for the node classification task where each node is a term and each protein is a graph (but all the proteins share the same adjacency matrix). As the node features we used base models predictions together with node embedding trained from scratch. We also added GO annotations features described in the next section.\n\n### Features are:\n\n**Base model predictions:** 11 base model predictions. Each model prediction is transformed to 4 channels. The important assumption here: **we assume all base models have predictions for all terms**. If the model doesn’t have it is filled with prior values. Priors are computed over Train or Train + Old train regarding the train sample used to fit the model. Channels are:\n* Prior prediction flag\n* Prediction logit value\n* For conditional: propagation type 1 logit, for raw just copy prediction logit\n* For conditional: propagation type 2 logit, for raw just copy prediction logit\n\nMore details are provided on the Figure below\n\n**GO annotation features:** 38 binary features are computed based on GOA electronic labelling. Details are described in the next section. Simply saying, each binary feature represents what kind of relationship between protein and term is available in GOA\n\n**Learnable term embedding**: just learnable from scratch embedding of term of size 8\n\n**Summary:** each protein / term pair is represented by 11 * 4 + 38 + 8 = 90 channels\n\n### Targets are:\nBinary labels of protein/term pair. All terms are used in training. Models were trained using binary cross entropy loss.\n\n### Train sample\nStackers were trained independently for each ontology using only CAFA 6 train data. Multistart with 2 random seeds were used, the prediction results are averaged in the end. Another important thing we should mention is about the metric. As far as we understand, we are evaluated only on protein/ontology pairs that are experimentally found. So, if we predict a term from ontology that does not exist, we will not get any penalty at all! That means, that we need actually to estimate **the conditional probability of a term in case when its ontology exists**. So, the correct way to fit a final stacker model for ontology X is to truncate the sample and take only the proteins that contain the terms from ontology X. It gives some small boost to the score and speeds-up the computations.\n\nThe following Figure represents the details of stacking model implementation:\n\n\n\n# GO Annotations\n\nWe discovered the GO annotations dataset provided by the [link](http://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/). They provide not only labelling but also the evidence codes of each term. We separated the codes that Kaggle suppose to be experimental from the electronic codes. So, we can use electronic labelling as the features to predict experimental labels given by Kaggle. From our analysis we discovered that about 30% of electronic labels become experimental, so using it as a model feature performs better than just adding it as is. \n\nThe electronic evidence codes provided by the labelling are not equal. They differ by:\n\n* The evidence code like IEA / IBA / ISO / ISS etc. Even if we don’t understand the difference, may be ML models can make it work \n* Type of relation like part_of / acts_upstream / is_active_in\n* Evidence counts. A protein could have multiple evidence for a term with different codes. Having multiple evidences highly increase the probability of transformation to experimental, for example for 3 codes it is more that 50% compared to 30% for 1 code\n\nUsing those information we computed 38 binary features like: \n\n* Protein has at least 1/2/3/4/5 evidences for term (propagated labels counts)\n* Protein has at least 1/2/3/4/5 evidences for term (within only raw labels)\n* Protein is related with term as “part_of” \n* Protein has an evidence of type IEA for term \n…\n\nThis is only the idea of feature generation provided. For the extensive list of features please see our code. \n\n**GOA annotations releases:** we started our work when release 226 was available. According to our knowledge, this release was also used to build the original competition data. Finally, by the end of the competition the latest available version was 228. So, this version was used to make a final prediction for the test set\n\nOne thing we need to mention here. Since the 228 data was released after competition started, some very small amounts of test proteins get experimental labels in the latest version of GOA. If we detect such cases, we just add it to the final submission file without impact on the training procedure. \n\n# Postprocessing\n\nWe also used the OBO graph to make a post processing. The problem of ML model prediction of protein terms is that it is inconsistent. Following the propagation rule that is applied to the target if the term exists, all its parents are assumed to exist too. So, a consistent model will never predict the probability for parents lower than term probability. But our models don't care about it at all. So we can manually fix this situation. Our final term prediction is **the average of term probability, maximum propagated children probability, and minimum propagated parents probability**.\nThe figure below illustrated the postprocessing details:\n\n\n\n# Simplified model\n\nWe believe, it does not make sense to choose one best and simple approach, since:\n1. There is no need to make fast and efficient online inference for such kind of research task\n2. Any single model scores much less that combination of multiple approaches\n\nWe consider using all the models and data sources to evaluate all the potential protein functions. Using GO electronic labelling as a feature in the stacker model is valid knowledge to predict, which terms will be experimentally found.\n\n# Source code\n\nSolution code is open source now and available at https://github.com/btbpanda/CAFA6-updates", "3478655": "Congratulations! I think the literature-based information is an interesting advance on your already-excellent CAFA 5 model. Do you know how much your final LB was improved by the literature abstracts?", "3480406": "Hi John. Thanks for your feedback. It is difficult to estimate the impact properly since we don't have private LB for all submissions. But, as an upper bound you can consider 0.04. Before @yuanlao153 shared a very nice walkthrough on his solution, which, he claims, is very close to our CAFA5 approach. So, we can consider 3rd and 2nd place difference is the effect of all our updates. I would say, 70-80% of this effect or even more is gained from articles.", "3481858": "hi, sir, I am curious about the way that you only used the literature in one model. And how you get rid of the bad result of long regression for the protein which is not rare species(by gcn?)", "3484826": "Hi! The reason why we avoid using the literature in many models is the potential data leakage risk. We don't want the final model rely too much on the literature since we don't know in advance how good private test proteins are represented in the papers\n\nFor the second question, sorry, I didn't get the idea. Can you please clarify, what do you mean by \"long regression for the protein which is not rare species\"?", "3488983": "Thanks for replying, I just notice that:\n\"Logistic Regression\nWe also trained a simple 13.5к output Logistic Regression (10000/2000/1500). It shows much less performance than GBDT on the popular terms, but is able to perform on rare outputs.\", in your note. \nHow you find this situation and get rid of the result of the poor performance result of this method or model", "3492887": "Thanks for clarification. We didn't do anything special. We rely on the stacker model instead. By design, it should decide when to attend on those poor predictors and when just better to ignore and base the decision on other types of predictors", "3493119": "It is a good design" }, "source": "meta" }