I would like to take the time to highlight some great discussions that have already been posted about information accretion as well as explain a bit more about the theory and intuition of this measure.
\nBecause of the structure of this ontology and the nature of protein function annotation, sometimes the IA measure may seen unintuitive at first glance, but I hope to clear up any confusion here. Please post comments here if there are any other questions or confusion about this measure.
\nInformation accretion (ia), introduced in [Clark and Radivojac, 2013], is a measure of how much information is added to an ontology annotation by node v given that its parents Pa(v) are already annotated. Specifically,
\n$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))} = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(\\mathcal{Pa}(v)|v) \\mathrm{Pr}(v)}$$
\nSince annotation of the term corresponding to node v necessarily implies the annotation of terms Pa(v) to guarantee a consistent subgraph (valid annotations), this means that Pr(Pa(v)|v)=1 and therefore:
\n$$ia(v) = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(v)}$$
\nThe probabilities above refer to some ground-truth distribution over annotated terms over all proteins. These distributions are unknown because we do not know the functions of all proteins.
\nTo calculate these probabilities, we use the observed annotations in a dataset as the empirical distribution on which to estimate the probabilities. Thus, Pr(v) is computed as the number of examples (proteins) annotated with the term corresponding to node v (divided by the total number of proteins) and Pr(Pa(v)) as the number of examples annotated with the terms corresponding to nodes Pa(v) (divided by the total number of proteins). The division by the total number of proteins cancel in the numerator and denominator so that the empirical ia is
\n$$\\bar{ia}(v) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } v}{\\text{number of proteins with term } v}\\right)$$
Since we are using the empirical (observed) counts in the training set to estimate the true distributions, there will be terms for which Pr(v)=0 if they have not been observed in the training set. To ensure there are no terms with Pr(v)=0, counts are regularized by introducing an artificial record (protein) annotated with every terms and adding it to the counts. That is, we add 1 to the counts for all terms to avoid dividing by 0. So finally, the implemented calculation becomes:
\n$$\\bar{ia}(v) = \\log_2\\left(\\frac{1 + \\text{number of proteins with parent term(s) of } v}{1+ \\text{number of proteins with term } v}\\right)$$
I have implemented the above math here https://github.com/claradepaolis/InformationAccretion and this is the code we used to generate IA.txt from train_terms.tsv and go-basic.obo.
Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,
\n$$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$
\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.
\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))
\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403213
\nMathematically, IA=0 when IA=log(1), thus Pr(v)=Pr(Pa(v)). Intuitively, this happens when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms Pa(v). That is, every protein in the training set that is annotated with the term(s) Pa(v) is also annotated with the term v. There is no protein that has annotations Pa(v) that doesn't also have v annotated. Having term v annotated doesn't set any protein apart from all the proteins that have term(s) Pa(v) annotated. According to the empirical counts, term v comes for \"free\" with knowing its parents. Thus, it accrues no information.
\nUsually, terms deeper in the ontology are more rarely annotated in proteins because they correspond to more specialized definitions of function that may be more difficult to determine. However, this is not always the case. Some examples fall into the above discussion for a term whose IA is 0. To extend the previous intuition check, IA for term v can be a smaller value than terms in Pa(v) if in most proteins with annotations Pa(v) we also see annotations for term v.
\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405090
\nIf you have worked with the gene ontology, seen figures of it, or investigated the meaning of the edges, you may have noticed that edges point toward what we have been calling \"parents.\" That is because in the gene ontology the edges are defined for relations such as \"is a\" or \"part of\" where more specific terms point towards more general terms. This means that the most general terms, which would intuitively be the \"roots\" are actually the leaves. If you work with the graph programmatically and want to find some term's parent nodes, remember to either reverse all the edges of the graph or keep track that ancestors should be descendants and vice versa.
", "messages": [ { "id": "2236250", "postDate": "04/26/2023 16:49:08", "content": "I would like to take the time to highlight some great discussions that have already been posted about information accretion as well as explain a bit more about the theory and intuition of this measure.
\nBecause of the structure of this ontology and the nature of protein function annotation, sometimes the IA measure may seen unintuitive at first glance, but I hope to clear up any confusion here. Please post comments here if there are any other questions or confusion about this measure.
\nInformation accretion (ia), introduced in [Clark and Radivojac, 2013], is a measure of how much information is added to an ontology annotation by node v given that its parents Pa(v) are already annotated. Specifically,
\n$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))} = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(\\mathcal{Pa}(v)|v) \\mathrm{Pr}(v)}$$
\nSince annotation of the term corresponding to node v necessarily implies the annotation of terms Pa(v) to guarantee a consistent subgraph (valid annotations), this means that Pr(Pa(v)|v)=1 and therefore:
\n$$ia(v) = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(v)}$$
\nThe probabilities above refer to some ground-truth distribution over annotated terms over all proteins. These distributions are unknown because we do not know the functions of all proteins.
\nTo calculate these probabilities, we use the observed annotations in a dataset as the empirical distribution on which to estimate the probabilities. Thus, Pr(v) is computed as the number of examples (proteins) annotated with the term corresponding to node v (divided by the total number of proteins) and Pr(Pa(v)) as the number of examples annotated with the terms corresponding to nodes Pa(v) (divided by the total number of proteins). The division by the total number of proteins cancel in the numerator and denominator so that the empirical ia is
\n$$\\bar{ia}(v) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } v}{\\text{number of proteins with term } v}\\right)$$
Since we are using the empirical (observed) counts in the training set to estimate the true distributions, there will be terms for which Pr(v)=0 if they have not been observed in the training set. To ensure there are no terms with Pr(v)=0, counts are regularized by introducing an artificial record (protein) annotated with every terms and adding it to the counts. That is, we add 1 to the counts for all terms to avoid dividing by 0. So finally, the implemented calculation becomes:
\n$$\\bar{ia}(v) = \\log_2\\left(\\frac{1 + \\text{number of proteins with parent term(s) of } v}{1+ \\text{number of proteins with term } v}\\right)$$
I have implemented the above math here https://github.com/claradepaolis/InformationAccretion and this is the code we used to generate IA.txt from train_terms.tsv and go-basic.obo.
Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,
\n$$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$
\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.
\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))
\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403213
\nMathematically, IA=0 when IA=log(1), thus Pr(v)=Pr(Pa(v)). Intuitively, this happens when the count of proteins annotated with term v equals the count of proteins annotated with all parent terms Pa(v). That is, every protein in the training set that is annotated with the term(s) Pa(v) is also annotated with the term v. There is no protein that has annotations Pa(v) that doesn't also have v annotated. Having term v annotated doesn't set any protein apart from all the proteins that have term(s) Pa(v) annotated. According to the empirical counts, term v comes for \"free\" with knowing its parents. Thus, it accrues no information.
\nUsually, terms deeper in the ontology are more rarely annotated in proteins because they correspond to more specialized definitions of function that may be more difficult to determine. However, this is not always the case. Some examples fall into the above discussion for a term whose IA is 0. To extend the previous intuition check, IA for term v can be a smaller value than terms in Pa(v) if in most proteins with annotations Pa(v) we also see annotations for term v.
\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405090
\nIf you have worked with the gene ontology, seen figures of it, or investigated the meaning of the edges, you may have noticed that edges point toward what we have been calling \"parents.\" That is because in the gene ontology the edges are defined for relations such as \"is a\" or \"part of\" where more specific terms point towards more general terms. This means that the most general terms, which would intuitively be the \"roots\" are actually the leaves. If you work with the graph programmatically and want to find some term's parent nodes, remember to either reverse all the edges of the graph or keep track that ancestors should be descendants and vice versa.
", "rawMarkdown": "I would like to take the time to highlight some great discussions that have already been posted about information accretion as well as explain a bit more about the theory and intuition of this measure. \n\nBecause of the structure of this ontology and the nature of protein function annotation, sometimes the IA measure may seen unintuitive at first glance, but I hope to clear up any confusion here. Please post comments here if there are any other questions or confusion about this measure. \n\n## What is Information Accretion?\nInformation accretion (ia), introduced in [[Clark and Radivojac, 2013]](https://academic.oup.com/bioinformatics/article-pdf/29/13/i53/18535367/btt228.pdf), is a measure of how much information is added to an ontology annotation by node *v* given that its parents *Pa(v)* are already annotated. Specifically, \n\n$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))} = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(\\mathcal{Pa}(v)|v) \\mathrm{Pr}(v)}$$\n\nSince annotation of the term corresponding to node *v* necessarily implies the annotation of terms *Pa(v)* to guarantee a *consistent subgraph* (valid annotations), this means that Pr(*Pa(v)*|*v*)=1 and therefore:\n\n$$ia(v) = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(v)}$$\n\nThe probabilities above refer to some ground-truth distribution over annotated terms over all proteins. **These distributions are unknown** because we do not know the functions of all proteins. \n\n## So how can we calculate IA? \n\nTo calculate these probabilities, we use the observed annotations in a dataset as the *empirical distribution* on which to estimate the probabilities. Thus, Pr(*v*) is computed as the number of examples (proteins) annotated with the term corresponding to node *v* (divided by the total number of proteins) and Pr(*Pa(v)*) as the number of examples annotated with the terms corresponding to nodes *Pa(v)* (divided by the total number of proteins). The division by the total number of proteins cancel in the numerator and denominator so that the empirical ia is \n$$\\bar{ia}(v) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } v}{\\text{number of proteins with term } v}\\right)$$ \n\n### Regularizing the empirical counts\nSince we are using the empirical (observed) counts in the training set to estimate the true distributions, there will be terms for which Pr(*v*)=0 if they have not been observed in the training set. To ensure there are no terms with Pr(*v*)=0, counts are regularized by introducing an artificial record (protein) annotated with every terms and adding it to the counts. That is, we add 1 to the counts for all terms to avoid dividing by 0. So finally, the implemented calculation becomes: \n$$\\bar{ia}(v) = \\log_2\\left(\\frac{1 + \\text{number of proteins with parent term(s) of } v}{1+ \\text{number of proteins with term } v}\\right)$$ \n\n## How do we actually calculate this?\n\nI have implemented the above math here [https://github.com/claradepaolis/InformationAccretion](https://github.com/claradepaolis/InformationAccretion) and this is the code we used to generate `IA.txt` from `train_terms.tsv` and `go-basic.obo`. \n\n## Intuition Check: Why does an unseen term have a large IA?\n\nLet's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X, \n\n$$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$ \n\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.\n\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))\n\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403213\n\n## Intuition Check: Why does a term have IA=0?\n\nMathematically, IA=0 when IA=log(1), thus Pr(*v*)=Pr(*Pa(v)*). Intuitively, this happens when the count of proteins annotated with term *v* equals the count of proteins annotated with *all* parent terms *Pa(v)*. That is, every protein in the training set that is annotated with the term(s) *Pa(v)* is also annotated with the term *v*. There is no protein that has annotations *Pa(v)* that doesn't also have *v* annotated. Having term *v* annotated doesn't set any protein apart from all the proteins that have term(s) *Pa(v)* annotated. According to the empirical counts, term *v* comes for \"free\" with knowing its parents. Thus, it accrues no information.\n\n## Intuition Check: Why does a term deeper in the ontology have a lower IA than its predecessors?\n\nUsually, terms deeper in the ontology are more rarely annotated in proteins because they correspond to more specialized definitions of function that may be more difficult to determine. However, this is not always the case. Some examples fall into the above discussion for a term whose IA is 0. To extend the previous intuition check, IA for term *v* can be a smaller value than terms in *Pa(v)* if in most proteins with annotations *Pa(v)* we also see annotations for term *v*. \n\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405090\n\n## An important side note\nIf you have worked with the gene ontology, seen figures of it, or investigated the meaning of the edges, you may have noticed that edges point toward what we have been calling \"parents.\" That is because in the gene ontology the edges are defined for relations such as \"is a\" or \"part of\" where more specific terms point towards more general terms. This means that the most general terms, which would intuitively be the \"roots\" are actually the leaves. If you work with the graph programmatically and want to find some term's parent nodes, remember to either reverse all the edges of the graph or keep track that ancestors should be descendants and vice versa.", "votes": null }, { "id": "2236346", "postDate": "04/26/2023 19:20:02", "content": "Great explanation!
", "rawMarkdown": "Great explanation!", "votes": null }, { "id": "2236708", "postDate": "04/27/2023 04:33:22", "content": "Really helpful. Thanks 🙏
", "rawMarkdown": "Really helpful. Thanks 🙏", "votes": null }, { "id": "2237504", "postDate": "04/27/2023 17:13:47", "content": "Thanks, that was really helpful.
", "rawMarkdown": "Thanks, that was really helpful.", "votes": null }, { "id": "2238271", "postDate": "04/28/2023 11:06:27", "content": "Hi, This is great explanation as I have been trying to implement this paper in Java.
\nWhen you form the set of parent of terms Pa(v) for a given term v, do you go all the way to the root of ontology or just parent terms of v and not beyond that?
E.g. v has parents p1 and p2.
\np1 has parent p11
\np11 has parent \"root\"
\np2 has parent p21
\np21 has parent \"root\"
So, which one of the following is true?
\nPa(v) = {p1, p2, p11, p21, root}
\nor
\nPa(v) = {p1, p2}
The paper talks in section 3.3 Measuring the quality of function prediction
\n\n\nAs the decision threshold is moved from its minimum to its maximum value, the pairs of forumla.gif will result in a curve in 2D space
\n
What is min and max value of threshold?
", "rawMarkdown": "The [paper](https://watermark.silverchair.com/btt228.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAAvQwggLwBgkqhkiG9w0BBwagggLhMIIC3QIBADCCAtYGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM1pu6uvEfOsHnMnT7AgEQgIICp8w3aKuBqDjDcTnc8QVnYjkYV1nnUja-nIqKPFVDWIe-OUVgpEhTrQCVNDtd6sZi2R30dIsLsFgntSfULOffeIQc8Glk77UxxBDhXUGpj38snRlTjb6jXTBkYIod8s4-pE-3KABuS8ZddtuBsD8IJT8qEjuNXrNxW4AVxg4n7I8UBawYo7WEYEKfK2eNbv8TtsxozRP3ieZhwFzbkxrrAQnbZdfhG1uPkixCHUT08EoFvqZ4qIXQojefdhcBJmxYGsITFQ8JnWYi36XPFJh6xLzBA-rV_TNRff1WExneMwExWnTNWm8AMp07xBw2DxpkknY1N8ei2pI3r7slvxceBp789QiL1ScY_vBAMleSpf9DrZCGJH6uvJ_3Zxqq6IZMAB5SQsbxhg7xQO5qvQz567kGU7n0I4BV-ZdZTsnFLx9-W-d1VVP5tdzz07vTbWbUhSFHzfYCBj_uqvWG21mt17Sq4DIF3Xg3TLYL7ihA2xr3jP0X2smspR8qE3vaXilhYXswCEdobqRQtec2mTYOugUt8fDC5Sd-l4_Hxg8N5naEOBk9mamVDfA7rwpJ0jCRt5-q5-pRCyiR9TxnLDEXE7avDAUYVND-f9X_-6wJF9T2QMazqf2KIcRyqJJFQNHI-0YDM1m8qAALzmY7euQSJCEy95Hzr3hC5pY5srLqEd9JuQ5jQxyrJYq_aqoTRj1YX3z8U5Py-yOk2rRNtkvS6tN9rst8OMPyEtPvAZTOeVeLovJ7YwbAK2nsJ_0lPBScALh5nJeBqBNjuQ8zLPdG-yu3wgxuEF8Y2qauQT_G838c9EL-zkBaZA2n4ry3ewGFr7iT2Z7ELozPtaPlEMybNKESJTItQftFlUSRlr8XDf9tblstBUpuvexaCo6W_ym1P3ih_4eii8E) talks in section **3.3 Measuring the quality of function prediction**\n>As the decision threshold is moved from its minimum to its maximum value, the pairs of forumla.gif will result in a curve in 2D space\n\nWhat is min and max value of threshold?", "votes": null }, { "id": "2239143", "postDate": "04/29/2023 07:54:48", "content": "That was really helpful!Thank you for sharing!
", "rawMarkdown": "That was really helpful!Thank you for sharing!", "votes": null }, { "id": "2243066", "postDate": "05/02/2023 17:04:22", "content": "What is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.
", "rawMarkdown": "What is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.", "votes": null }, { "id": "2243076", "postDate": "05/02/2023 17:10:10", "content": "\n\nWhat is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.
\n
That sounds like an annotation error. Per definition, a protein with GO term X should always have all of X's parents, too.
\nHowever, keep in mind that this only works for is_a and part_of relations, not has_part or regulates.
\nhttp://geneontology.org/docs/ontology-relations/
@norixmb is correct. In addition, this can happen if you are using a different structure (ontology) than the one that generated (propagated) the original terms. If the structure has changed, the parents of nodes may have changed. The training terms and obo file we provide should match and produce no negative values of IA.
", "rawMarkdown": "norixmb is correct. In addition, this can happen if you are using a different structure (ontology) than the one that generated (propagated) the original terms. If the structure has changed, the parents of nodes may have changed. The training terms and obo file we provide should match and produce no negative values of IA.", "votes": null }, { "id": "2244227", "postDate": "05/03/2023 14:20:50", "content": "The parents are the nodes that are directly connected to node v, i.e. Pa(v) = {p1, p2} in your example.
", "rawMarkdown": "The parents are the nodes that are directly connected to node v, i.e. Pa(v) = {p1, p2} in your example.", "votes": null }, { "id": "2244243", "postDate": "05/03/2023 14:31:16", "content": "Think of it like a traditional precision-recall or ROC curve. tau is the decision threshold for predicting a positive term (class). To make a PR or ROC curve, you would compute precision and recall at many threshold (tau) values between 0 and 1 to populate every point in the curve. It's the same idea here but the curve axes are remaining uncertainty and misinformation.
", "rawMarkdown": "Think of it like a traditional precision-recall or ROC curve. tau is the decision threshold for predicting a positive term (class). To make a PR or ROC curve, you would compute precision and recall at many threshold (tau) values between 0 and 1 to populate every point in the curve. It's the same idea here but the curve axes are remaining uncertainty and misinformation.", "votes": null }, { "id": "2244448", "postDate": "05/03/2023 16:25:21", "content": "Thanks @claradepaolis for the replies!
", "rawMarkdown": "Thanks @claradepaolis for the replies!", "votes": null }, { "id": "2293265", "postDate": "06/09/2023 04:21:35", "content": "Thank you, This was a really helpful read
", "rawMarkdown": "Thank you, This was a really helpful read", "votes": null }, { "id": "2308884", "postDate": "06/19/2023 09:41:37", "content": "neat explanation, thanks!
", "rawMarkdown": "neat explanation, thanks!", "votes": null }, { "id": "2330151", "postDate": "07/04/2023 18:44:00", "content": "That's really precise and intuitive, thank you!
", "rawMarkdown": "That's really precise and intuitive, thank you!", "votes": null }, { "id": "2379530", "postDate": "08/08/2023 07:47:43", "content": "Hi, sorry for the late question.
\nWill information accretion be recalculated in the final evaluation phase? Thanks
Thanks for your explanation. It is very helpful for me.
", "rawMarkdown": "Thanks for your explanation. It is very helpful for me.", "votes": null }, { "id": "2389462", "postDate": "08/14/2023 04:43:02", "content": "Thanks, that was really helpful.
", "rawMarkdown": "Thanks, that was really helpful.", "votes": null } ], "comments": [ { "id": 2236346, "author_name": "guilhermetrajano", "author_url": "", "post_date": "04/26/2023 19:20:02", "content": "Great explanation!
", "votes": null, "replies": [] }, { "id": 2236708, "author_name": "ujjwalgrover", "author_url": "", "post_date": "04/27/2023 04:33:22", "content": "Really helpful. Thanks 🙏
", "votes": null, "replies": [] }, { "id": 2237504, "author_name": "mmhamdy", "author_url": "", "post_date": "04/27/2023 17:13:47", "content": "Thanks, that was really helpful.
", "votes": null, "replies": [] }, { "id": 2238271, "author_name": "vjoshi22", "author_url": "", "post_date": "04/28/2023 11:06:27", "content": "Hi, This is great explanation as I have been trying to implement this paper in Java.
\nWhen you form the set of parent of terms Pa(v) for a given term v, do you go all the way to the root of ontology or just parent terms of v and not beyond that?
E.g. v has parents p1 and p2.
\np1 has parent p11
\np11 has parent \"root\"
\np2 has parent p21
\np21 has parent \"root\"
So, which one of the following is true?
\nPa(v) = {p1, p2, p11, p21, root}
\nor
\nPa(v) = {p1, p2}
The parents are the nodes that are directly connected to node v, i.e. Pa(v) = {p1, p2} in your example.
", "votes": null, "replies": [] } ] }, { "id": 2238279, "author_name": "vjoshi22", "author_url": "", "post_date": "04/28/2023 11:15:24", "content": "The paper talks in section 3.3 Measuring the quality of function prediction
\n\n\nAs the decision threshold is moved from its minimum to its maximum value, the pairs of forumla.gif will result in a curve in 2D space
\n
What is min and max value of threshold?
", "votes": null, "replies": [ { "id": 2244243, "author_name": "claradepaolis", "author_url": "", "post_date": "05/03/2023 14:31:16", "content": "Think of it like a traditional precision-recall or ROC curve. tau is the decision threshold for predicting a positive term (class). To make a PR or ROC curve, you would compute precision and recall at many threshold (tau) values between 0 and 1 to populate every point in the curve. It's the same idea here but the curve axes are remaining uncertainty and misinformation.
", "votes": null, "replies": [ { "id": 2244448, "author_name": "vjoshi22", "author_url": "", "post_date": "05/03/2023 16:25:21", "content": "Thanks @claradepaolis for the replies!
", "votes": null, "replies": [] } ] } ] }, { "id": 2239143, "author_name": "ericka42", "author_url": "", "post_date": "04/29/2023 07:54:48", "content": "That was really helpful!Thank you for sharing!
", "votes": null, "replies": [] }, { "id": 2243066, "author_name": "vjoshi22", "author_url": "", "post_date": "05/02/2023 17:04:22", "content": "What is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.
", "votes": null, "replies": [ { "id": 2243076, "author_name": "norixmb", "author_url": "", "post_date": "05/02/2023 17:10:10", "content": "\n\nWhat is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.
\n
That sounds like an annotation error. Per definition, a protein with GO term X should always have all of X's parents, too.
\nHowever, keep in mind that this only works for is_a and part_of relations, not has_part or regulates.
\nhttp://geneontology.org/docs/ontology-relations/
@norixmb is correct. In addition, this can happen if you are using a different structure (ontology) than the one that generated (propagated) the original terms. If the structure has changed, the parents of nodes may have changed. The training terms and obo file we provide should match and produce no negative values of IA.
", "votes": null, "replies": [] } ] }, { "id": 2293265, "author_name": "nnilayy", "author_url": "", "post_date": "06/09/2023 04:21:35", "content": "Thank you, This was a really helpful read
", "votes": null, "replies": [] }, { "id": 2308884, "author_name": "sfilatov", "author_url": "", "post_date": "06/19/2023 09:41:37", "content": "neat explanation, thanks!
", "votes": null, "replies": [] }, { "id": 2330151, "author_name": "kaggleprollc", "author_url": "", "post_date": "07/04/2023 18:44:00", "content": "That's really precise and intuitive, thank you!
", "votes": null, "replies": [] }, { "id": 2379530, "author_name": "abaojiang", "author_url": "", "post_date": "08/08/2023 07:47:43", "content": "Hi, sorry for the late question.
\nWill information accretion be recalculated in the final evaluation phase? Thanks
Thanks for your explanation. It is very helpful for me.
", "votes": null, "replies": [] }, { "id": 2389462, "author_name": "hunterwrynn", "author_url": "", "post_date": "08/14/2023 04:43:02", "content": "Thanks, that was really helpful.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2236250": "I would like to take the time to highlight some great discussions that have already been posted about information accretion as well as explain a bit more about the theory and intuition of this measure. \n\nBecause of the structure of this ontology and the nature of protein function annotation, sometimes the IA measure may seen unintuitive at first glance, but I hope to clear up any confusion here. Please post comments here if there are any other questions or confusion about this measure. \n\n## What is Information Accretion?\nInformation accretion (ia), introduced in [[Clark and Radivojac, 2013]](https://academic.oup.com/bioinformatics/article-pdf/29/13/i53/18535367/btt228.pdf), is a measure of how much information is added to an ontology annotation by node *v* given that its parents *Pa(v)* are already annotated. Specifically, \n\n$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))} = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(\\mathcal{Pa}(v)|v) \\mathrm{Pr}(v)}$$\n\nSince annotation of the term corresponding to node *v* necessarily implies the annotation of terms *Pa(v)* to guarantee a *consistent subgraph* (valid annotations), this means that Pr(*Pa(v)*|*v*)=1 and therefore:\n\n$$ia(v) = \\log_2 \\frac{\\mathrm{Pr}(\\mathcal{Pa}(v))}{\\mathrm{Pr}(v)}$$\n\nThe probabilities above refer to some ground-truth distribution over annotated terms over all proteins. **These distributions are unknown** because we do not know the functions of all proteins. \n\n## So how can we calculate IA? \n\nTo calculate these probabilities, we use the observed annotations in a dataset as the *empirical distribution* on which to estimate the probabilities. Thus, Pr(*v*) is computed as the number of examples (proteins) annotated with the term corresponding to node *v* (divided by the total number of proteins) and Pr(*Pa(v)*) as the number of examples annotated with the terms corresponding to nodes *Pa(v)* (divided by the total number of proteins). The division by the total number of proteins cancel in the numerator and denominator so that the empirical ia is \n$$\\bar{ia}(v) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } v}{\\text{number of proteins with term } v}\\right)$$ \n\n### Regularizing the empirical counts\nSince we are using the empirical (observed) counts in the training set to estimate the true distributions, there will be terms for which Pr(*v*)=0 if they have not been observed in the training set. To ensure there are no terms with Pr(*v*)=0, counts are regularized by introducing an artificial record (protein) annotated with every terms and adding it to the counts. That is, we add 1 to the counts for all terms to avoid dividing by 0. So finally, the implemented calculation becomes: \n$$\\bar{ia}(v) = \\log_2\\left(\\frac{1 + \\text{number of proteins with parent term(s) of } v}{1+ \\text{number of proteins with term } v}\\right)$$ \n\n## How do we actually calculate this?\n\nI have implemented the above math here [https://github.com/claradepaolis/InformationAccretion](https://github.com/claradepaolis/InformationAccretion) and this is the code we used to generate `IA.txt` from `train_terms.tsv` and `go-basic.obo`. \n\n## Intuition Check: Why does an unseen term have a large IA?\n\nLet's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X, \n\n$$ia(X) = \\log_2\\left(\\frac{1+\\text{number of proteins with parent term(s) of } X}{1+\\text{number of proteins with term } X}\\right)$$ \n\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.\n\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))\n\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/403213\n\n## Intuition Check: Why does a term have IA=0?\n\nMathematically, IA=0 when IA=log(1), thus Pr(*v*)=Pr(*Pa(v)*). Intuitively, this happens when the count of proteins annotated with term *v* equals the count of proteins annotated with *all* parent terms *Pa(v)*. That is, every protein in the training set that is annotated with the term(s) *Pa(v)* is also annotated with the term *v*. There is no protein that has annotations *Pa(v)* that doesn't also have *v* annotated. Having term *v* annotated doesn't set any protein apart from all the proteins that have term(s) *Pa(v)* annotated. According to the empirical counts, term *v* comes for \"free\" with knowing its parents. Thus, it accrues no information.\n\n## Intuition Check: Why does a term deeper in the ontology have a lower IA than its predecessors?\n\nUsually, terms deeper in the ontology are more rarely annotated in proteins because they correspond to more specialized definitions of function that may be more difficult to determine. However, this is not always the case. Some examples fall into the above discussion for a term whose IA is 0. To extend the previous intuition check, IA for term *v* can be a smaller value than terms in *Pa(v)* if in most proteins with annotations *Pa(v)* we also see annotations for term *v*. \n\nSee the discussion here: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405090\n\n## An important side note\nIf you have worked with the gene ontology, seen figures of it, or investigated the meaning of the edges, you may have noticed that edges point toward what we have been calling \"parents.\" That is because in the gene ontology the edges are defined for relations such as \"is a\" or \"part of\" where more specific terms point towards more general terms. This means that the most general terms, which would intuitively be the \"roots\" are actually the leaves. If you work with the graph programmatically and want to find some term's parent nodes, remember to either reverse all the edges of the graph or keep track that ancestors should be descendants and vice versa.", "2236346": "Great explanation!", "2236708": "Really helpful. Thanks 🙏", "2237504": "Thanks, that was really helpful.", "2238271": "Hi, This is great explanation as I have been trying to implement this paper in Java.\nWhen you form the set of parent of terms Pa(v) for a given term v, do you go all the way to the root of ontology or just parent terms of v and not beyond that?\n\nE.g. v has parents p1 and p2.\np1 has parent p11 \np11 has parent \"root\"\np2 has parent p21\np21 has parent \"root\"\n\nSo, which one of the following is true? \nPa(v) = {p1, p2, p11, p21, root}\nor\nPa(v) = {p1, p2}", "2238279": "The [paper](https://watermark.silverchair.com/btt228.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAAvQwggLwBgkqhkiG9w0BBwagggLhMIIC3QIBADCCAtYGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM1pu6uvEfOsHnMnT7AgEQgIICp8w3aKuBqDjDcTnc8QVnYjkYV1nnUja-nIqKPFVDWIe-OUVgpEhTrQCVNDtd6sZi2R30dIsLsFgntSfULOffeIQc8Glk77UxxBDhXUGpj38snRlTjb6jXTBkYIod8s4-pE-3KABuS8ZddtuBsD8IJT8qEjuNXrNxW4AVxg4n7I8UBawYo7WEYEKfK2eNbv8TtsxozRP3ieZhwFzbkxrrAQnbZdfhG1uPkixCHUT08EoFvqZ4qIXQojefdhcBJmxYGsITFQ8JnWYi36XPFJh6xLzBA-rV_TNRff1WExneMwExWnTNWm8AMp07xBw2DxpkknY1N8ei2pI3r7slvxceBp789QiL1ScY_vBAMleSpf9DrZCGJH6uvJ_3Zxqq6IZMAB5SQsbxhg7xQO5qvQz567kGU7n0I4BV-ZdZTsnFLx9-W-d1VVP5tdzz07vTbWbUhSFHzfYCBj_uqvWG21mt17Sq4DIF3Xg3TLYL7ihA2xr3jP0X2smspR8qE3vaXilhYXswCEdobqRQtec2mTYOugUt8fDC5Sd-l4_Hxg8N5naEOBk9mamVDfA7rwpJ0jCRt5-q5-pRCyiR9TxnLDEXE7avDAUYVND-f9X_-6wJF9T2QMazqf2KIcRyqJJFQNHI-0YDM1m8qAALzmY7euQSJCEy95Hzr3hC5pY5srLqEd9JuQ5jQxyrJYq_aqoTRj1YX3z8U5Py-yOk2rRNtkvS6tN9rst8OMPyEtPvAZTOeVeLovJ7YwbAK2nsJ_0lPBScALh5nJeBqBNjuQ8zLPdG-yu3wgxuEF8Y2qauQT_G838c9EL-zkBaZA2n4ry3ewGFr7iT2Z7ELozPtaPlEMybNKESJTItQftFlUSRlr8XDf9tblstBUpuvexaCo6W_ym1P3ih_4eii8E) talks in section **3.3 Measuring the quality of function prediction**\n>As the decision threshold is moved from its minimum to its maximum value, the pairs of forumla.gif will result in a curve in 2D space\n\nWhat is min and max value of threshold?", "2239143": "That was really helpful!Thank you for sharing!", "2243066": "What is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.", "2243076": "> What is the intuition behind negative information accretion (ia)? It happens when the descendant term has is associated with more number of proteins than its parent.\n\nThat sounds like an annotation error. Per definition, a protein with GO term X should always have all of X's parents, too.\nHowever, keep in mind that this only works for `is_a` and `part_of` relations, not `has_part` or `regulates`.\nhttp://geneontology.org/docs/ontology-relations/", "2244221": "norixmb is correct. In addition, this can happen if you are using a different structure (ontology) than the one that generated (propagated) the original terms. If the structure has changed, the parents of nodes may have changed. The training terms and obo file we provide should match and produce no negative values of IA.", "2244227": "The parents are the nodes that are directly connected to node v, i.e. Pa(v) = {p1, p2} in your example.", "2244243": "Think of it like a traditional precision-recall or ROC curve. tau is the decision threshold for predicting a positive term (class). To make a PR or ROC curve, you would compute precision and recall at many threshold (tau) values between 0 and 1 to populate every point in the curve. It's the same idea here but the curve axes are remaining uncertainty and misinformation.", "2244448": "Thanks @claradepaolis for the replies!", "2293265": "Thank you, This was a really helpful read", "2308884": "neat explanation, thanks!", "2330151": "That's really precise and intuitive, thank you!", "2379530": "Hi, sorry for the late question.\nWill information accretion be recalculated in the final evaluation phase? Thanks", "2382037": "Thanks for your explanation. It is very helpful for me.", "2389462": "Thanks, that was really helpful." }, "source": "meta" }