Quite a few of the GO terms in IA.tsv (Information Accretion = weight for evaluation metric) are not annotated to any protein, at least in the competition training dataset.\nHowever, only 5291 of those not-protein-assigned GO terms have an IA of zero. 8706 have an IA >0, in some cases quite high like 4, 7, 9 etc.
\nHow was the IA, which is based on frequency of protein annotation, computed then? Also on non-experimental annotations? Larger protein libraries than SwissProt? Using \"inherited\" annotations from GO-term-children that are not listed explicitly in the training dataset?\n(The 0 to non-0 ratio is also different from the annotated GO terms, for which it is 6537 with 0 IA vs 19588 \">0\")
\nAnd most importantly: Those non-assigned GO terms should still be predicted, right? (At least those with a IA > 0)
\nPS: thanks for the discussions and especially answers here so far. Very helpful!
", "messages": [ { "id": "3383384", "postDate": "12/30/2025 01:23:08", "content": "Quite a few of the GO terms in IA.tsv (Information Accretion = weight for evaluation metric) are not annotated to any protein, at least in the competition training dataset.\nHowever, only 5291 of those not-protein-assigned GO terms have an IA of zero. 8706 have an IA >0, in some cases quite high like 4, 7, 9 etc.
\nHow was the IA, which is based on frequency of protein annotation, computed then? Also on non-experimental annotations? Larger protein libraries than SwissProt? Using \"inherited\" annotations from GO-term-children that are not listed explicitly in the training dataset?\n(The 0 to non-0 ratio is also different from the annotated GO terms, for which it is 6537 with 0 IA vs 19588 \">0\")
\nAnd most importantly: Those non-assigned GO terms should still be predicted, right? (At least those with a IA > 0)
\nPS: thanks for the discussions and especially answers here so far. Very helpful!
", "rawMarkdown": "Quite a few of the GO terms in IA.tsv (Information Accretion = weight for evaluation metric) are not annotated to any protein, at least in the competition training dataset.\nHowever, only 5291 of those not-protein-assigned GO terms have an IA of zero. 8706 have an IA >0, in some cases quite high like 4, 7, 9 etc. \n\nHow was the IA, which is based on frequency of protein annotation, computed then? Also on non-experimental annotations? Larger protein libraries than SwissProt? Using \"inherited\" annotations from GO-term-children that are not listed explicitly in the training dataset?\n(The 0 to non-0 ratio is also different from the annotated GO terms, for which it is 6537 with 0 IA vs 19588 \">0\")\n\nAnd most importantly: Those non-assigned GO terms should still be predicted, right? (At least those with a IA > 0)\n\nPS: thanks for the discussions and especially answers here so far. Very helpful!", "votes": null }, { "id": "3383391", "postDate": "12/30/2025 01:45:51", "content": "IA = 0 does not mean that the term doesn't appear in the ground truth. Rather, IA=0 when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms of v in the ontology.
\nIA is not always intuitive. I hope this discussion thread from the CAFA 5 competition is helpful: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237
\nI'll copy here some relevant portions:
\nWhy does an unseen term have a large IA?
\n\n\nLet's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,
\n$$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$
\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.
\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))
\n
Why does a term have IA=0?
\n\n", "rawMarkdown": "IA = 0 does not mean that the term doesn't appear in the ground truth. Rather, IA=0 when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms of v in the ontology.\n\nIA is not always intuitive. I hope this discussion thread from the CAFA 5 competition is helpful: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237 \n\nI'll copy here some relevant portions:\n\n*Why does an unseen term have a large IA?*\n\n>Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,\n\n> $$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$ \n\n> For some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.\n\n> For example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1)) \n\n*Why does a term have IA=0?*\n\n> Mathematically, IA=0 when IA=log(1), thus Pr(v)=Pr(Pa(v)). Intuitively, this happens when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms Pa(v). That is, every protein in the training set that is annotated with the term(s) Pa(v) is also annotated with the term v. There is no protein that has annotations Pa(v) that doesn't also have v annotated. Having term v annotated doesn't set any protein apart from all the proteins that have term(s) Pa(v) annotated. According to the empirical counts, term v comes for \"free\" with knowing its parents. Thus, it accrues no information.", "votes": null }, { "id": "3383617", "postDate": "12/30/2025 13:57:05", "content": "Mathematically, IA=0 when IA=log(1), thus Pr(v)=Pr(Pa(v)). Intuitively, this happens when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms Pa(v). That is, every protein in the training set that is annotated with the term(s) Pa(v) is also annotated with the term v. There is no protein that has annotations Pa(v) that doesn't also have v annotated. Having term v annotated doesn't set any protein apart from all the proteins that have term(s) Pa(v) annotated. According to the empirical counts, term v comes for \"free\" with knowing its parents. Thus, it accrues no information.
\n
Thank you very much for the post and the link. That explains it very well.
\nTbh, it seems like a metric artifact to me that terms without any annotation get a high IA due to their parents having annotations and it having none. The intuitive idea I see behind IA, up-weighting terms that strongly refine their parents annotations by only having a small subset of the parent annotations, seems to be not applying here. It does not refine anything, it just has no annotations :)\nMaybe a metric with the number of actual annotations in the numerator could circumvent this in the future.
\nSure, correctly predicting no-annotation GO terms would be impressive. However, since some have a outsized weight (due to the above) and IMO a lower likelihood to be annotated within the next 6 months than \"established\" GO terms, they are likely to negatively impact the F1_max performance, since \"wrong\" predictions of those terms still \"count\". Thus it is very risky to predict them, no?
", "rawMarkdown": "Thank you very much for the post and the link. That explains it very well.\n\nTbh, it seems like a metric artifact to me that terms without any annotation get a high IA due to their parents having annotations and it having none. The intuitive idea I see behind IA, up-weighting terms that strongly refine their parents annotations by only having a small subset of the parent annotations, seems to be not applying here. It does not refine anything, it just has no annotations :)\nMaybe a metric with the number of actual annotations in the numerator could circumvent this in the future.\n\nSure, correctly predicting no-annotation GO terms would be impressive. However, since some have a outsized weight (due to the above) and IMO a lower likelihood to be annotated within the next 6 months than \"established\" GO terms, they are likely to negatively impact the F1_max performance, since \"wrong\" predictions of those terms still \"count\". Thus it is very risky to predict them, no?", "votes": null } ], "comments": [ { "id": 3383391, "author_name": "claradepaolis", "author_url": "", "post_date": "12/30/2025 01:45:51", "content": "IA = 0 does not mean that the term doesn't appear in the ground truth. Rather, IA=0 when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms of v in the ontology.
\nIA is not always intuitive. I hope this discussion thread from the CAFA 5 competition is helpful: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237
\nI'll copy here some relevant portions:
\nWhy does an unseen term have a large IA?
\n\n\nLet's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,
\n$$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$
\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.
\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))
\n
Why does a term have IA=0?
\n\n", "votes": null, "replies": [ { "id": 3383617, "author_name": "jrsilicon", "author_url": "", "post_date": "12/30/2025 13:57:05", "content": "Mathematically, IA=0 when IA=log(1), thus Pr(v)=Pr(Pa(v)). Intuitively, this happens when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms Pa(v). That is, every protein in the training set that is annotated with the term(s) Pa(v) is also annotated with the term v. There is no protein that has annotations Pa(v) that doesn't also have v annotated. Having term v annotated doesn't set any protein apart from all the proteins that have term(s) Pa(v) annotated. According to the empirical counts, term v comes for \"free\" with knowing its parents. Thus, it accrues no information.
\n
Thank you very much for the post and the link. That explains it very well.
\nTbh, it seems like a metric artifact to me that terms without any annotation get a high IA due to their parents having annotations and it having none. The intuitive idea I see behind IA, up-weighting terms that strongly refine their parents annotations by only having a small subset of the parent annotations, seems to be not applying here. It does not refine anything, it just has no annotations :)\nMaybe a metric with the number of actual annotations in the numerator could circumvent this in the future.
\nSure, correctly predicting no-annotation GO terms would be impressive. However, since some have a outsized weight (due to the above) and IMO a lower likelihood to be annotated within the next 6 months than \"established\" GO terms, they are likely to negatively impact the F1_max performance, since \"wrong\" predictions of those terms still \"count\". Thus it is very risky to predict them, no?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3383384": "Quite a few of the GO terms in IA.tsv (Information Accretion = weight for evaluation metric) are not annotated to any protein, at least in the competition training dataset.\nHowever, only 5291 of those not-protein-assigned GO terms have an IA of zero. 8706 have an IA >0, in some cases quite high like 4, 7, 9 etc. \n\nHow was the IA, which is based on frequency of protein annotation, computed then? Also on non-experimental annotations? Larger protein libraries than SwissProt? Using \"inherited\" annotations from GO-term-children that are not listed explicitly in the training dataset?\n(The 0 to non-0 ratio is also different from the annotated GO terms, for which it is 6537 with 0 IA vs 19588 \">0\")\n\nAnd most importantly: Those non-assigned GO terms should still be predicted, right? (At least those with a IA > 0)\n\nPS: thanks for the discussions and especially answers here so far. Very helpful!", "3383391": "IA = 0 does not mean that the term doesn't appear in the ground truth. Rather, IA=0 when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms of v in the ontology.\n\nIA is not always intuitive. I hope this discussion thread from the CAFA 5 competition is helpful: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237 \n\nI'll copy here some relevant portions:\n\n*Why does an unseen term have a large IA?*\n\n>Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,\n\n> $$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$ \n\n> For some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.\n\n> For example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1)) \n\n*Why does a term have IA=0?*\n\n> Mathematically, IA=0 when IA=log(1), thus Pr(v)=Pr(Pa(v)). Intuitively, this happens when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms Pa(v). That is, every protein in the training set that is annotated with the term(s) Pa(v) is also annotated with the term v. There is no protein that has annotations Pa(v) that doesn't also have v annotated. Having term v annotated doesn't set any protein apart from all the proteins that have term(s) Pa(v) annotated. According to the empirical counts, term v comes for \"free\" with knowing its parents. Thus, it accrues no information.", "3383617": "Thank you very much for the post and the link. That explains it very well.\n\nTbh, it seems like a metric artifact to me that terms without any annotation get a high IA due to their parents having annotations and it having none. The intuitive idea I see behind IA, up-weighting terms that strongly refine their parents annotations by only having a small subset of the parent annotations, seems to be not applying here. It does not refine anything, it just has no annotations :)\nMaybe a metric with the number of actual annotations in the numerator could circumvent this in the future.\n\nSure, correctly predicting no-annotation GO terms would be impressive. However, since some have a outsized weight (due to the above) and IMO a lower likelihood to be annotated within the next 6 months than \"established\" GO terms, they are likely to negatively impact the F1_max performance, since \"wrong\" predictions of those terms still \"count\". Thus it is very risky to predict them, no?" }, "source": "meta" }