My general concern is that we need to predict the terms that will likely be experimentally annotated, not those that are actually true.
\nQ1) When an experiment is done to associate a term with a protein, do the experimenters give an equal effort to associate all terms in the relevant ontology? This way, an unannotated, but truly associated term at least has a good shot of being experimentally shown to be true.
\nQ2) Has there been protein/subontology combinations where there are some experimentally demonstrated term associations, and then at a later point in time, new terms within that ontology are experimentally shown to be associated with the protein? The time in between would be my concern since competition scoring of these labels would punish those who predict all the terms correctly since some of them are yet to be proven by experiments.
For example: consider protein A and subontology MF, with protein subontology combination A_MF.
\n1) Assume the truth is that terms t1, t2, and t3 are associated with A.
\n2) Before the submission deadline there are no terms experimentally annotated for A_MF.
\n3) After the submission deadline and before final scoring, t1 is experimentally shown to be associated with A, but t2 and t3 are not. This event determines that A_MF will be included in the scoring calculations.
\n4) My prediction file has t1, t2, and t3 all with probabilities .5.
\n5) So I have included all the correct terms, but 2 of them are not considered true since they are not experimentally proven
\nConclusion: My max precision score for A_MF will be something like 1/3, since t2 and t3 are not experimentally annotated, even though it should be 3/3 == 1 since I have perfectly predicted the terms for A.
\nIs this assessment correct?
My general concern is that we need to predict the terms that will likely be experimentally annotated, not those that are actually true.
\nQ1) When an experiment is done to associate a term with a protein, do the experimenters give an equal effort to associate all terms in the relevant ontology? This way, an unannotated, but truly associated term at least has a good shot of being experimentally shown to be true.
\nQ2) Has there been protein/subontology combinations where there are some experimentally demonstrated term associations, and then at a later point in time, new terms within that ontology are experimentally shown to be associated with the protein? The time in between would be my concern since competition scoring of these labels would punish those who predict all the terms correctly since some of them are yet to be proven by experiments.
For example: consider protein A and subontology MF, with protein subontology combination A_MF.
\n1) Assume the truth is that terms t1, t2, and t3 are associated with A.
\n2) Before the submission deadline there are no terms experimentally annotated for A_MF.
\n3) After the submission deadline and before final scoring, t1 is experimentally shown to be associated with A, but t2 and t3 are not. This event determines that A_MF will be included in the scoring calculations.
\n4) My prediction file has t1, t2, and t3 all with probabilities .5.
\n5) So I have included all the correct terms, but 2 of them are not considered true since they are not experimentally proven
\nConclusion: My max precision score for A_MF will be something like 1/3, since t2 and t3 are not experimentally annotated, even though it should be 3/3 == 1 since I have perfectly predicted the terms for A.
\nIs this assessment correct?
This might be relevant: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696
", "rawMarkdown": "This might be relevant: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696", "votes": null }, { "id": "2332092", "postDate": "07/06/2023 01:18:08", "content": "Thank you so much and I apologize for not finding that discussion before posting this one.
", "rawMarkdown": "Thank you so much and I apologize for not finding that discussion before posting this one.", "votes": null } ], "comments": [ { "id": 2331226, "author_name": "thedrcat", "author_url": "", "post_date": "07/05/2023 12:42:15", "content": "This might be relevant: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696
", "votes": null, "replies": [ { "id": 2332092, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "07/06/2023 01:18:08", "content": "Thank you so much and I apologize for not finding that discussion before posting this one.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2330248": "##### Edit ##########\nMy general concern is that we need to predict the terms that will likely be experimentally annotated, not those that are actually true. \n \nQ1) When an experiment is done to associate a term with a protein, do the experimenters give an equal effort to associate all terms in the relevant ontology? This way, an unannotated, but truly associated term at least has a good shot of being experimentally shown to be true. \n\nQ2) Has there been protein/subontology combinations where there are some experimentally demonstrated term associations, and then at a later point in time, new terms within that ontology are experimentally shown to be associated with the protein? The time in between would be my concern since competition scoring of these labels would punish those who predict all the terms correctly since some of them are yet to be proven by experiments.\n\n##### Original post #####\nFor example: consider protein A and subontology MF, with protein subontology combination A_MF.\n\n1) Assume the truth is that terms t1, t2, and t3 are associated with A. \n2) Before the submission deadline there are no terms experimentally annotated for A_MF. \n3) After the submission deadline and before final scoring, t1 is experimentally shown to be associated with A, but t2 and t3 are not. This event determines that A_MF will be included in the scoring calculations.\n4) My prediction file has t1, t2, and t3 all with probabilities .5. \n5) So I have included all the correct terms, but 2 of them are not considered true since they are not experimentally proven\n\nConclusion: My max precision score for A_MF will be something like 1/3, since t2 and t3 are not experimentally annotated, even though it should be 3/3 == 1 since I have perfectly predicted the terms for A. \n\nIs this assessment correct?", "2331226": "This might be relevant: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696", "2332092": "Thank you so much and I apologize for not finding that discussion before posting this one." }, "source": "meta" }