![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1313949%2Ff9726a776920fc16813d021068259a8e%2Fex%20(3).gif?generation=1681940906956477&alt=media\")
One month left to the end of the competition!
\nA good time to look back and review everything that happened up to this point.
I went through all discussion posts (Not with ChatGPT, I actually read everything) and compiled a summary of the the most interesting bits that can be found in the different discussions.
\nEnjoy!
\n\n\n(yes.. this took a while.. ð )
\n
This post provides an explanation of the information accretion (IA) measure in protein function annotation.
\nIn short: IA measures how much information is added to an ontology annotation by a given node, considering its already annotated parents.
Some Key Points:
\nHighly suggested read for those interested in understanding the domain knowledge behind the competition.
\nImportant topic by Darek KÅeczek!
\nOn this competition we are likley to use protein language models (Examples: ESM, Ankh).
\nDarek KÅeczek shares a blog post about this subjest.
andreylalaley_NAME shares final-layer embeddings from multiple ESM2 models.
\nESM - Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR). The post includes links to final-layer embeddings from five ESM variants:
Model family on github
\nData creation notebook.
\nDarek KÅeczek raises an concern that the competition's scoring system may inadvertently favor predicting which protein functions will be reported within the evaluation period, rather than accurately predicting protein functions themselves.
\nOpinions from the comments:
\nSee more on the original post: A very interesting read.
\n------------------------ START OF EXAMPLE SUMMARIES ------------------------
\nYaroslav Isaienkov dropped an amazing plot you might want to try yourself!
\n
Code
\ndef plot_dag(graph, term, =1):\n ng_graph = networkx.ego_graph(graph, term, =radius)\n n ng_graph.nodes(=):\n n[1][] = .join(n[1][].split())\n n[1][] = n[0]\n nt = Network(=1000, =, =, =, =, =, =)\n nt.from_nx(ng_graph)\n return nt.show()\nAlexander Chervov shared a research webinar on ProDDG, a database of protein stability change upon mutation.
\nAlthough not directly related to the CAFA5 challenge, the webinar may provide insights on how to apply the ideas to the competition, particularly in terms of data augmentation.
\nLinks:
\nIn this post, Btbpanda discovered a potential error in the metric implementation for the competition.
\nComment (from the hosts): I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.. ..The Kaggle team will now take over and recalculate the dashboard with the updated code.
Tilii shared a link to an approach developed by Burkhard Rost's lab that predicts GO annotations for proteins purely from their sequences (GitHub Link).
\nThis lab also hosts a webserver that can directly predict GO annotations without the need to install language models (Webserver Link).
From the comments:
\nOn this post we get a detailed explanation of what a FASTA file is and how to read it in Python. This is a very common file format used for biological sequences.
\nExample of a FASTA file:
\n\n\nP20536 sp|P20536|UNG_VACCC Uracil-DNA glycosylase OS=Vaccinia virus (strain Copenhagen) OX=10249 GN=UNG PE=1 SV=1
\n
\n MNSVTVSHAPYTITYHDDWEPVMSQLVEFYNEVASWLLRDETSPIPDKFFIQLKQPLRNK
\n RVCVCGIDPYPKDGTGVPFESPNFTKKSIKEIASSISRLTGVIDYKGYNLNIIDGVIPWN
\n YYLSCKLGETKSHAIYWDKISKLLLQHITKHVSVLYCLGKTDFSNIRAKLESPVTTIVGY
\n HPAARDRQFEKDRSFEIINVLLELDNKVPINWAQGFIY
How to read a FASTA file in Python
\n Bio SeqIO\n\nfilename = \nfasta_sequences = SeqIO.parse((filename),)\n\n fasta fasta_sequences:\n (fasta)\n (, fasta.)\n (, (fasta))\n \nThis post explains the levels of knowledge specificity in Gene Ontology (GO) annotations and draws a parallel with taxonomic classification of organisms.
\nThe Key Points:
\nPlease refer to the post for more details and examples.
\nIn this post, Darek KÅeczek shares their findings from conducting exploratory analysis on the competition data, along with some ideas for addressing the insights:
\nSee notebook for further exploration.
\nFrom the comments:
\nComeptition: Open Problems - Multimodal Single-Cell Integration
\n1st solution: here
\n2nd solution: here
\n3rd solution: here
\n4th solution: here
\n5th solution: here
\n6th solution: here
Title: Protein Sequence Embedding Techniques (Resources)
\nIn this post, Anusha shares a list of various embedding techniques used for protein sequences that have proven to be more efficient than typical one-hot encoding or Tokenizers.
\nList of techniques mentioned:
\nSome more resources from the comments:
\nImportant
\nIn this post, MT shares their observation of a strange jump in their submission score after adding quick-go annotations to their predictions for a set of proteins.
\nFrom the comments:
\nAlexander Chervov Asks if at the final evaluation, only new annotations can be added to existing high-confidence ones, or if some high-confidence annotations can \"disappear\" if not confirmed by new experimental data.
\nIn response:
\niddux1 states that theoretically, experimentally verified predictions may be retracted if there are errors in interpretation, redaction of papers, or new evidence that contradicts previous evidence.
\nThis discussion expresses concern about the quality of existing Gene Ontology (GO) annotations and its impact on the competition.
\nThey highlight the lack of experimental annotations, even for well-known proteins like myoglobin. They question why terms like \"heme binding\" are labeled as inferred from electronic annotation (IEA) when there is ample evidence from prior research demonstrating myoglobin's heme-binding capability.
The author also raises questions about the reliability of the experimental terms, particularly those derived from high-throughput experiments, which may have significant error rates. They question the usefulness of a competition that relies on flawed training data where high-quality training data is historically not easily accessible, and the test set likely contains terms with potential errors.
Responses from other competitors members provide additional insights, including the explanation that experimental evidence might be species-specific and dependent on direct experiments conducted on particular species. They also challenge the author's assertion about error rates in high-throughput experiments and emphasize the need for competitors to construct their own training sets.
See full discussion for more details.
\nTilii explores the similarity between the train and test data in the protein function prediction competition.
\nThe post raises a concern: If a protein ID is not listed in the submitted file, it should be assumed that all predictions are 0.
\nHowever, it seems that submissions missing test rows can trigger a failure.
But after some experimentation, the solution was to fill in the missing rows with values such as \"GO:0000001\" and \"0.123\": This allowed a previously failed submission to pass.
\nComment (from the hosts):
\nConfirming that if none of the targets overlap with the validation set, an exception is thrown by the program.
\nAlthough the notebook doesn't report the exception, you can all test it locally using the CAFA-evaluator standalone version.
In this post, an analysis showed in this post found a pattern similar to the Pareto Rule.
\nThe results indicate that focusing on the Molecular Function (MFO) category is better for evaluation. Although the number of unique MFO terms is less than others, the mean and max IA weight of MFO terms are larger.
\nTwo key results were shared in the post.
@iddux1 asks for an explanation of the rationale behind the choice of the GO termset for the training in this competition.
\ntrain_terms.tsv for the same protein.train_terms.tsv are from GeneOntology Central and have non-experimental evidence codes.From the Comments:
\nIn response, an official from the competition explains that they did not consider the IBA evidence code in the evaluation, and therefore, they omitted any annotations from train_terms.tsv that use the IBA evidence code.
\nThey state that they only included proteins with annotated terms that have been validated by experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by a curator (IC).
This discussion highlights a duplicate entry in the test data.
\nIn general it is highly advised to check for duplicates in all datasets. Just in case.
In this post there is a discussion about using augmentations to the dataset. (Specifically: MixUp)
\nFurther on in the comments there is an interesting discussion about some augmentation techniques:
\nTwo questions are raised in this post:
\n\n\nWhen making predictions, do we need to include all the parent GO terms of a term we specify or will they be automatically inferred for us from the child terms that we include?
\n
Answer (from the hosts): They will be automatically inferred (or \"propagated\").
\n\n\nIn the training/test data, if a protein is marked as having GO term X, is it guaranteed that all the parent, grandparent, etc. terms of X are also included for that protein? (If not, are we being asked to additionally predict which of the parent terms the annotator will have decided to include?)
\n
Answer (from the hosts): A term is guaranteed to have all the parents. This is known as the True Path Rule.
\nQuestion: If so, is it okay to develop/submit files from a file (uploaded to notebook, then submitted) so long as the final submission is computed by a notebook?
\nAnswer:
\n\n\nThe notebook is actually just a wrapper for the submission file. It does not have to compute anything.
\n
\n https://www.kaggle.com/code/inversion/submitting-to-the-cafa-competition
\n The important part is that the submitted notebook comes with a submission.tsv file (in the specified format).
\n Whether this file is generated by the notebook or offline and then attached to it does not matter.
In this interesting discussion, the author points out that random CV splitting in popular notebooks can lead to an overconfident estimate of model performance since similar proteins can end up in both the train and validation sets.
\nThere are serveral options discussed in the comments for CV strategy:
\n\n\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1
\n
--cov-mode 1 should do clustering similarly to how you described above, which if I understood correctly should keep similar proteins together regardless of length.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11298796%2F11bc4782c751744c88bdfbeec3d8ea9f%2FScreenshot%202023-05-12%20at%208.11.53%20AM.png?generation=1683904343421260&alt=media\")
There are two modes for the \"propagation\": one is \"fill\", another is \"max\":
\npreds = pred_parser(preds, ontologies, gt, prop_mode='fill', max_terms=500)
Question: In the current competition the default mode \"fill\" is used , is it true?
\nQuestion: Is the precision in weighted F-measures (for MF, BP, and CC) is always going to be one since there is no negative label but absent label?
\nAnswer: (Thaweewat R)
\n\n\nIn the Positive-Unlabeled learning setup for this competition, the precision will not always be one. Predicting all 1500 GO terms for each test protein sequence would increase false positives, lowering precision. Instead, focus on developing a model that accurately identifies relevant GO terms while balancing precision and recall to achieve a high F-measure score.
\n
Important note by MT regarding GO terms with high weights that do not appear in the training set.
\n\nExplanation (From the hosts):
\n>
\n\nThis is true. To explain why that happens, let's consider how IA is calculated
\nia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))}
\n
\n We use the training set (train_terms.tsv) as the empirical distribution that approximates the denominator in this equation. Since it's an empirical distribution, we regularize counts by adding a count of 1 to every term. This will avoid dividing by 0 for unseen terms.Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,
\nia(X) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } X}{\\text{number of proteins with term } X}\\right)
\n
\n For some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.For example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))
\n
MT is asking about a protein ID (A2RUG3) that is present only in the test set and now obsolete in UniProt. Can we eliminate such ids from our predictions?
\nAnswer (From the hosts):
\n\n\nWe strongly recommend not excluding any IDs while generating predictions.
\n
Answers Summary:
\nMST shares a fine-tuning baseline for ESM2 link
\nQuestion: How is the evaluation metric calculated with respect to the threshold value (tau)?
\n\n\nHow is the evaluation metric calculated with respect to tau?
\n
\n Is tau swept over a grid of vaules between 1 and 0, is tau a fixed number?
Answers (Comments):
\nInteresting insight: ZXMKCD noticed that the local evaluations of BPO and CCO terms correlate well with the test set scores, but this is not the case for MFO terms.
\nNo answers yet..
\nyunsuxiaozi asking about prior biology knowledge and how relevant it is for this competition.
\nAnswer (From the hosts):
\nThree clarifications on this post (from the hosts):
\n\n\nProteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set.
\n
That's a bit of a misnomer. It's not that the GO term will be published, but rather that the protein will have a database entry associated with that GO term using an experimental validation where there wasn't such an association before. As pointed out in other discussions, there are experimental annotations out there in the literature that were not curated yet into the database.
\n\n\nEg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?
\n
You have just described an ongoing effort for two decades of automated biocuratrion efforts. There is a whole body of literature on that, e.g. https://pubmed.ncbi.nlm.nih.gov/23514326/
\n\n\nIs it considered valid in this competition to also do \"Information Extraction\"?
\n
You can use any information you like, as long as it's public, as elaborated in the Rules.
\nThe question raised in this post is whether having a biological background or solid understanding is necessary for this competition.
\nSuggestions (In the comments):
\nCapoo Cat is facing a problem where the accuracy of the model using T5 embedding feature data for proteins is stuck.
\nIn this post, Darek KÅeczek raises a question about how the train set was selected for the competition.
\nMT response that the answer could be found here
\nFor model building we can use published tools like BLAST, protein language models, and AlphaGO2 that predicts various biochemical aspects of proteins.
\nBut does this extend to models that directly predict protein GO terms if they are publicly available? Can we use the results of such models when creating our ensemble predictions for this competition?
Answer (From the hosts):
\n\n\nIn one word: yes. Be advised of the Winner License clause in the rules though.
\n
Finetune a LLM on 2 GPUs
\nIn this post, Simon Veitner shared code for finetuning a protein language model on 2 GPUs.
\nThe LB score achieved from training on 1 fold without any finetuning is 0.329, but the post encourages others to tune hyperparameters, explore different models, and report any interesting findings or mistakes in the notebook.
\nYou can find the code here.
\nThis detailed discussion provides efficient resources for newcomers to catch up with the competition.
\nIt includes an advisory section, field knowledge, data, and a pre written DataLoader.
Highly suggested read
\nThis post raises some interesting questions:
\nAnswer (Tilii):
\nIn this post, the author seeks guidance on how to construct features optimally for protein function prediction models using non-deep learning approaches.
\nAnswers:
\nOne month left to the end of the competition!
\nA good time to look back and review everything that happened up to this point.
I went through all discussion posts (Not with ChatGPT, I actually read everything) and compiled a summary of the the most interesting bits that can be found in the different discussions.
\nEnjoy!
\n\n\n(yes.. this took a while.. ð )
\n
This post provides an explanation of the information accretion (IA) measure in protein function annotation.
\nIn short: IA measures how much information is added to an ontology annotation by a given node, considering its already annotated parents.
Some Key Points:
\nHighly suggested read for those interested in understanding the domain knowledge behind the competition.
\nImportant topic by Darek KÅeczek!
\nOn this competition we are likley to use protein language models (Examples: ESM, Ankh).
\nDarek KÅeczek shares a blog post about this subjest.
andreylalaley_NAME shares final-layer embeddings from multiple ESM2 models.
\nESM - Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR). The post includes links to final-layer embeddings from five ESM variants:
Model family on github
\nData creation notebook.
\nDarek KÅeczek raises an concern that the competition's scoring system may inadvertently favor predicting which protein functions will be reported within the evaluation period, rather than accurately predicting protein functions themselves.
\nOpinions from the comments:
\nSee more on the original post: A very interesting read.
\n------------------------ START OF EXAMPLE SUMMARIES ------------------------
\nYaroslav Isaienkov dropped an amazing plot you might want to try yourself!
\n
Code
\ndef plot_dag(graph, term, =1):\n ng_graph = networkx.ego_graph(graph, term, =radius)\n n ng_graph.nodes(=):\n n[1][] = .join(n[1][].split())\n n[1][] = n[0]\n nt = Network(=1000, =, =, =, =, =, =)\n nt.from_nx(ng_graph)\n return nt.show()\nAlexander Chervov shared a research webinar on ProDDG, a database of protein stability change upon mutation.
\nAlthough not directly related to the CAFA5 challenge, the webinar may provide insights on how to apply the ideas to the competition, particularly in terms of data augmentation.
\nLinks:
\nIn this post, Btbpanda discovered a potential error in the metric implementation for the competition.
\nComment (from the hosts): I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.. ..The Kaggle team will now take over and recalculate the dashboard with the updated code.
Tilii shared a link to an approach developed by Burkhard Rost's lab that predicts GO annotations for proteins purely from their sequences (GitHub Link).
\nThis lab also hosts a webserver that can directly predict GO annotations without the need to install language models (Webserver Link).
From the comments:
\nOn this post we get a detailed explanation of what a FASTA file is and how to read it in Python. This is a very common file format used for biological sequences.
\nExample of a FASTA file:
\n\n\nP20536 sp|P20536|UNG_VACCC Uracil-DNA glycosylase OS=Vaccinia virus (strain Copenhagen) OX=10249 GN=UNG PE=1 SV=1
\n
\n MNSVTVSHAPYTITYHDDWEPVMSQLVEFYNEVASWLLRDETSPIPDKFFIQLKQPLRNK
\n RVCVCGIDPYPKDGTGVPFESPNFTKKSIKEIASSISRLTGVIDYKGYNLNIIDGVIPWN
\n YYLSCKLGETKSHAIYWDKISKLLLQHITKHVSVLYCLGKTDFSNIRAKLESPVTTIVGY
\n HPAARDRQFEKDRSFEIINVLLELDNKVPINWAQGFIY
How to read a FASTA file in Python
\n Bio SeqIO\n\nfilename = \nfasta_sequences = SeqIO.parse((filename),)\n\n fasta fasta_sequences:\n (fasta)\n (, fasta.)\n (, (fasta))\n \nThis post explains the levels of knowledge specificity in Gene Ontology (GO) annotations and draws a parallel with taxonomic classification of organisms.
\nThe Key Points:
\nPlease refer to the post for more details and examples.
\nIn this post, Darek KÅeczek shares their findings from conducting exploratory analysis on the competition data, along with some ideas for addressing the insights:
\nSee notebook for further exploration.
\nFrom the comments:
\nComeptition: Open Problems - Multimodal Single-Cell Integration
\n1st solution: here
\n2nd solution: here
\n3rd solution: here
\n4th solution: here
\n5th solution: here
\n6th solution: here
Title: Protein Sequence Embedding Techniques (Resources)
\nIn this post, Anusha shares a list of various embedding techniques used for protein sequences that have proven to be more efficient than typical one-hot encoding or Tokenizers.
\nList of techniques mentioned:
\nSome more resources from the comments:
\nImportant
\nIn this post, MT shares their observation of a strange jump in their submission score after adding quick-go annotations to their predictions for a set of proteins.
\nFrom the comments:
\nAlexander Chervov Asks if at the final evaluation, only new annotations can be added to existing high-confidence ones, or if some high-confidence annotations can \"disappear\" if not confirmed by new experimental data.
\nIn response:
\niddux1 states that theoretically, experimentally verified predictions may be retracted if there are errors in interpretation, redaction of papers, or new evidence that contradicts previous evidence.
\nThis discussion expresses concern about the quality of existing Gene Ontology (GO) annotations and its impact on the competition.
\nThey highlight the lack of experimental annotations, even for well-known proteins like myoglobin. They question why terms like \"heme binding\" are labeled as inferred from electronic annotation (IEA) when there is ample evidence from prior research demonstrating myoglobin's heme-binding capability.
The author also raises questions about the reliability of the experimental terms, particularly those derived from high-throughput experiments, which may have significant error rates. They question the usefulness of a competition that relies on flawed training data where high-quality training data is historically not easily accessible, and the test set likely contains terms with potential errors.
Responses from other competitors members provide additional insights, including the explanation that experimental evidence might be species-specific and dependent on direct experiments conducted on particular species. They also challenge the author's assertion about error rates in high-throughput experiments and emphasize the need for competitors to construct their own training sets.
See full discussion for more details.
\nTilii explores the similarity between the train and test data in the protein function prediction competition.
\nThe post raises a concern: If a protein ID is not listed in the submitted file, it should be assumed that all predictions are 0.
\nHowever, it seems that submissions missing test rows can trigger a failure.
But after some experimentation, the solution was to fill in the missing rows with values such as \"GO:0000001\" and \"0.123\": This allowed a previously failed submission to pass.
\nComment (from the hosts):
\nConfirming that if none of the targets overlap with the validation set, an exception is thrown by the program.
\nAlthough the notebook doesn't report the exception, you can all test it locally using the CAFA-evaluator standalone version.
In this post, an analysis showed in this post found a pattern similar to the Pareto Rule.
\nThe results indicate that focusing on the Molecular Function (MFO) category is better for evaluation. Although the number of unique MFO terms is less than others, the mean and max IA weight of MFO terms are larger.
\nTwo key results were shared in the post.
@iddux1 asks for an explanation of the rationale behind the choice of the GO termset for the training in this competition.
\ntrain_terms.tsv for the same protein.train_terms.tsv are from GeneOntology Central and have non-experimental evidence codes.From the Comments:
\nIn response, an official from the competition explains that they did not consider the IBA evidence code in the evaluation, and therefore, they omitted any annotations from train_terms.tsv that use the IBA evidence code.
\nThey state that they only included proteins with annotated terms that have been validated by experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by a curator (IC).
This discussion highlights a duplicate entry in the test data.
\nIn general it is highly advised to check for duplicates in all datasets. Just in case.
In this post there is a discussion about using augmentations to the dataset. (Specifically: MixUp)
\nFurther on in the comments there is an interesting discussion about some augmentation techniques:
\nTwo questions are raised in this post:
\n\n\nWhen making predictions, do we need to include all the parent GO terms of a term we specify or will they be automatically inferred for us from the child terms that we include?
\n
Answer (from the hosts): They will be automatically inferred (or \"propagated\").
\n\n\nIn the training/test data, if a protein is marked as having GO term X, is it guaranteed that all the parent, grandparent, etc. terms of X are also included for that protein? (If not, are we being asked to additionally predict which of the parent terms the annotator will have decided to include?)
\n
Answer (from the hosts): A term is guaranteed to have all the parents. This is known as the True Path Rule.
\nQuestion: If so, is it okay to develop/submit files from a file (uploaded to notebook, then submitted) so long as the final submission is computed by a notebook?
\nAnswer:
\n\n\nThe notebook is actually just a wrapper for the submission file. It does not have to compute anything.
\n
\n https://www.kaggle.com/code/inversion/submitting-to-the-cafa-competition
\n The important part is that the submitted notebook comes with a submission.tsv file (in the specified format).
\n Whether this file is generated by the notebook or offline and then attached to it does not matter.
In this interesting discussion, the author points out that random CV splitting in popular notebooks can lead to an overconfident estimate of model performance since similar proteins can end up in both the train and validation sets.
\nThere are serveral options discussed in the comments for CV strategy:
\n\n\nmmseqs easy-cluster examples/DB.fasta clusterRes tmp --min-seq-id 0.5 -c 0.8 --cov-mode 1
\n
--cov-mode 1 should do clustering similarly to how you described above, which if I understood correctly should keep similar proteins together regardless of length.
There are two modes for the \"propagation\": one is \"fill\", another is \"max\":
\npreds = pred_parser(preds, ontologies, gt, prop_mode='fill', max_terms=500)
Question: In the current competition the default mode \"fill\" is used , is it true?
\nQuestion: Is the precision in weighted F-measures (for MF, BP, and CC) is always going to be one since there is no negative label but absent label?
\nAnswer: (Thaweewat R)
\n\n\nIn the Positive-Unlabeled learning setup for this competition, the precision will not always be one. Predicting all 1500 GO terms for each test protein sequence would increase false positives, lowering precision. Instead, focus on developing a model that accurately identifies relevant GO terms while balancing precision and recall to achieve a high F-measure score.
\n
Important note by MT regarding GO terms with high weights that do not appear in the training set.
\n\nExplanation (From the hosts):
\n>
\n\nThis is true. To explain why that happens, let's consider how IA is calculated
\nia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))}
\n
\n We use the training set (train_terms.tsv) as the empirical distribution that approximates the denominator in this equation. Since it's an empirical distribution, we regularize counts by adding a count of 1 to every term. This will avoid dividing by 0 for unseen terms.Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,
\nia(X) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } X}{\\text{number of proteins with term } X}\\right)
\n
\n For some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.For example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))
\n
MT is asking about a protein ID (A2RUG3) that is present only in the test set and now obsolete in UniProt. Can we eliminate such ids from our predictions?
\nAnswer (From the hosts):
\n\n\nWe strongly recommend not excluding any IDs while generating predictions.
\n
Answers Summary:
\nMST shares a fine-tuning baseline for ESM2 link
\nQuestion: How is the evaluation metric calculated with respect to the threshold value (tau)?
\n\n\nHow is the evaluation metric calculated with respect to tau?
\n
\n Is tau swept over a grid of vaules between 1 and 0, is tau a fixed number?
Answers (Comments):
\nInteresting insight: ZXMKCD noticed that the local evaluations of BPO and CCO terms correlate well with the test set scores, but this is not the case for MFO terms.
\nNo answers yet..
\nyunsuxiaozi asking about prior biology knowledge and how relevant it is for this competition.
\nAnswer (From the hosts):
\nThree clarifications on this post (from the hosts):
\n\n\nProteins having one or more of their functions published by researchers during the curation phase of the competition will comprise the future test set.
\n
That's a bit of a misnomer. It's not that the GO term will be published, but rather that the protein will have a database entry associated with that GO term using an experimental validation where there wasn't such an association before. As pointed out in other discussions, there are experimental annotations out there in the literature that were not curated yet into the database.
\n\n\nEg, can someone look at conference abstracts/posters, dissertations, (or perhaps NIH grants, if that is available) to guess which proteins may soon be added?
\n
You have just described an ongoing effort for two decades of automated biocuratrion efforts. There is a whole body of literature on that, e.g. https://pubmed.ncbi.nlm.nih.gov/23514326/
\n\n\nIs it considered valid in this competition to also do \"Information Extraction\"?
\n
You can use any information you like, as long as it's public, as elaborated in the Rules.
\nThe question raised in this post is whether having a biological background or solid understanding is necessary for this competition.
\nSuggestions (In the comments):
\nCapoo Cat is facing a problem where the accuracy of the model using T5 embedding feature data for proteins is stuck.
\nIn this post, Darek KÅeczek raises a question about how the train set was selected for the competition.
\nMT response that the answer could be found here
\nFor model building we can use published tools like BLAST, protein language models, and AlphaGO2 that predicts various biochemical aspects of proteins.
\nBut does this extend to models that directly predict protein GO terms if they are publicly available? Can we use the results of such models when creating our ensemble predictions for this competition?
Answer (From the hosts):
\n\n\nIn one word: yes. Be advised of the Winner License clause in the rules though.
\n
Finetune a LLM on 2 GPUs
\nIn this post, Simon Veitner shared code for finetuning a protein language model on 2 GPUs.
\nThe LB score achieved from training on 1 fold without any finetuning is 0.329, but the post encourages others to tune hyperparameters, explore different models, and report any interesting findings or mistakes in the notebook.
\nYou can find the code here.
\nThis detailed discussion provides efficient resources for newcomers to catch up with the competition.
\nIt includes an advisory section, field knowledge, data, and a pre written DataLoader.
Highly suggested read
\nThis post raises some interesting questions:
\nAnswer (Tilii):
\nIn this post, the author seeks guidance on how to construct features optimally for protein function prediction models using non-deep learning approaches.
\nAnswers:
\n