{ "id": 403213, "title": "GO_terms_not_in_train", "url": "/competitions/cafa-5-protein-function-prediction/discussion/403213", "author_name": "", "post_date": "2023-04-21T20:02:35.761849400Z", "votes": 7, "comment_count": 9, "views": 0, "content": "

Something to consider is that many GO terms with high weights do not appear in the training set.
\nhttps://www.kaggle.com/code/mtinti/go-terms-not-in-train/notebook?scriptVersionId=126704504

", "messages": [ { "id": "2229905", "postDate": "04/21/2023 20:02:35", "content": "

Something to consider is that many GO terms with high weights do not appear in the training set.
\nhttps://www.kaggle.com/code/mtinti/go-terms-not-in-train/notebook?scriptVersionId=126704504

", "rawMarkdown": "Something to consider is that many GO terms with high weights do not appear in the training set. \nhttps://www.kaggle.com/code/mtinti/go-terms-not-in-train/notebook?scriptVersionId=126704504", "votes": null }, { "id": "2230059", "postDate": "04/22/2023 01:13:05", "content": "

This notebook is either not public or has been deleted.

", "rawMarkdown": "This notebook is either not public or has been deleted.", "votes": null }, { "id": "2230283", "postDate": "04/22/2023 07:25:07", "content": "

thanks, fixed now

", "rawMarkdown": "thanks, fixed now", "votes": null }, { "id": "2230356", "postDate": "04/22/2023 09:40:42", "content": "

Reading between the lines, I assume that the weights (IA.txt) have been generated from something other than the train/test data. Probably a much larger dataset containing many more proteins. Presumably most also won't appear in the future test data that solutions be evaluated on.

\n

I guess if we want to find proteins that have these other functions, we can always use other other (public) sources of data.

", "rawMarkdown": "Reading between the lines, I assume that the weights (`IA.txt`) have been generated from something other than the train/test data. Probably a much larger dataset containing many more proteins. Presumably most also won't appear in the future test data that solutions be evaluated on.\n\nI guess if we want to find proteins that have these other functions, we can always use other other (public) sources of data.", "votes": null }, { "id": "2230363", "postDate": "04/22/2023 09:49:49", "content": "

yep, the competition state: \"The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins\". What I don't get exactly is how this pool was assembled. Would not make sense to make available this dataset and use it for training?

", "rawMarkdown": "yep, the competition state: \"The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins\". What I don't get exactly is how this pool was assembled. Would not make sense to make available this dataset and use it for training?", "votes": null }, { "id": "2230919", "postDate": "04/22/2023 22:08:58", "content": "
\n

Would not make sense to make available this dataset and use it for training?

\n
\n

The dataset we used to generate the values in IA.txt is the training set :)

", "rawMarkdown": "> Would not make sense to make available this dataset and use it for training?\n\nThe dataset we used to generate the values in IA.txt is the training set :)", "votes": null }, { "id": "2230923", "postDate": "04/22/2023 22:18:07", "content": "
\n

Something to consider is that many GO terms with high weights do not appear in the training set.

\n
\n

That is a problem we have in general with protein annotation databases. The information content of most experimentally validated terms is low. (or \"low weight\" as you say). The reasons are mostly because high throughput experiments which contribute the bulk of experimental annotation increase the frequency of the terms they are designed to provide, hence lowering the information content (\"weight\") of the term.

\n

Example: You have an microscopy image analysis system, automated, that can only tell if a protein is located in the cytosol or the nucleus. The two largest and most obvious compartments int he cell. A robot reads 2000 slides a day, and categorizes the location as either \"nucleus\" or \"cytosol\". However, you need a graduate student to know if a cytosol-located protein is located somewhere more specifically: say the Golgi apparatus (look it up). That student can only do 20 slides per day. Which terms will populate the database more?

\n

See also this paper: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003063

", "rawMarkdown": "> Something to consider is that many GO terms with high weights do not appear in the training set. \n\nThat is a problem we have in general with protein annotation databases. The information content of most experimentally validated terms is low. (or \"low weight\" as you say). The reasons are mostly because high throughput experiments which contribute the bulk of experimental annotation increase the frequency of the terms they are designed to provide, hence lowering the information content (\"weight\") of the term. \n\nExample: You have an microscopy image analysis system, automated, that can only tell if a protein is located in the cytosol or the nucleus. The two largest and most obvious compartments int he cell. A robot reads 2000 slides a day, and categorizes the location as either \"nucleus\" or \"cytosol\". However, you need a graduate student to know if a cytosol-located protein is located somewhere more specifically: say the Golgi apparatus (look it up). That student can only do 20 slides per day. Which terms will populate the database more?\n\nSee also this paper: [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003063](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003063)", "votes": null }, { "id": "2231802", "postDate": "04/23/2023 17:20:36", "content": "

Thanks, so all the terms come from the train set, however only the experimental validated go terms ended in train_terms.tsv files, right?

", "rawMarkdown": "Thanks, so all the terms come from the train set, however only the experimental validated go terms ended in train_terms.tsv files, right?", "votes": null }, { "id": "2234049", "postDate": "04/24/2023 20:04:16", "content": "
\n

many GO terms with high weights do not appear in the training set.

\n
\n

This is true. To explain why that happens, let's consider how IA is calculated

\n

$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))}$$

\n

We use the training set (train_terms.tsv) as the empirical distribution that approximates the denominator in this equation. Since it's an empirical distribution, we regularize counts by adding a count of 1 to every term. This will avoid dividing by 0 for unseen terms.

\n

Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,

\n

$$ia(X) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } X}{\\text{number of proteins with term } X}\\right)$$

\n

For some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.

\n

For example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))

", "rawMarkdown": ">many GO terms with high weights do not appear in the training set. \n\nThis is true. To explain why that happens, let's consider how IA is calculated\n\n$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))}$$\n\nWe use the training set (train_terms.tsv) as the empirical distribution that approximates the denominator in this equation. Since it's an empirical distribution, we regularize counts by adding a count of 1 to every term. This will avoid dividing by 0 for unseen terms.\n\nLet's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X, \n\n$$ia(X) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } X}{\\text{number of proteins with term } X}\\right)$$ \n\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.\n\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))", "votes": null }, { "id": "2234071", "postDate": "04/24/2023 20:41:33", "content": "

Thank you, crystal clear now

", "rawMarkdown": "Thank you, crystal clear now", "votes": null } ], "comments": [ { "id": 2230059, "author_name": "tilii7", "author_url": "", "post_date": "04/22/2023 01:13:05", "content": "

This notebook is either not public or has been deleted.

", "votes": null, "replies": [ { "id": 2230283, "author_name": "mtinti", "author_url": "", "post_date": "04/22/2023 07:25:07", "content": "

thanks, fixed now

", "votes": null, "replies": [] } ] }, { "id": 2230356, "author_name": "andrewrrose", "author_url": "", "post_date": "04/22/2023 09:40:42", "content": "

Reading between the lines, I assume that the weights (IA.txt) have been generated from something other than the train/test data. Probably a much larger dataset containing many more proteins. Presumably most also won't appear in the future test data that solutions be evaluated on.

\n

I guess if we want to find proteins that have these other functions, we can always use other other (public) sources of data.

", "votes": null, "replies": [ { "id": 2230363, "author_name": "mtinti", "author_url": "", "post_date": "04/22/2023 09:49:49", "content": "

yep, the competition state: \"The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins\". What I don't get exactly is how this pool was assembled. Would not make sense to make available this dataset and use it for training?

", "votes": null, "replies": [ { "id": 2230919, "author_name": "iddux1", "author_url": "", "post_date": "04/22/2023 22:08:58", "content": "
\n

Would not make sense to make available this dataset and use it for training?

\n
\n

The dataset we used to generate the values in IA.txt is the training set :)

", "votes": null, "replies": [ { "id": 2231802, "author_name": "mtinti", "author_url": "", "post_date": "04/23/2023 17:20:36", "content": "

Thanks, so all the terms come from the train set, however only the experimental validated go terms ended in train_terms.tsv files, right?

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2230923, "author_name": "iddux1", "author_url": "", "post_date": "04/22/2023 22:18:07", "content": "
\n

Something to consider is that many GO terms with high weights do not appear in the training set.

\n
\n

That is a problem we have in general with protein annotation databases. The information content of most experimentally validated terms is low. (or \"low weight\" as you say). The reasons are mostly because high throughput experiments which contribute the bulk of experimental annotation increase the frequency of the terms they are designed to provide, hence lowering the information content (\"weight\") of the term.

\n

Example: You have an microscopy image analysis system, automated, that can only tell if a protein is located in the cytosol or the nucleus. The two largest and most obvious compartments int he cell. A robot reads 2000 slides a day, and categorizes the location as either \"nucleus\" or \"cytosol\". However, you need a graduate student to know if a cytosol-located protein is located somewhere more specifically: say the Golgi apparatus (look it up). That student can only do 20 slides per day. Which terms will populate the database more?

\n

See also this paper: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003063

", "votes": null, "replies": [] }, { "id": 2234049, "author_name": "claradepaolis", "author_url": "", "post_date": "04/24/2023 20:04:16", "content": "
\n

many GO terms with high weights do not appear in the training set.

\n
\n

This is true. To explain why that happens, let's consider how IA is calculated

\n

$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))}$$

\n

We use the training set (train_terms.tsv) as the empirical distribution that approximates the denominator in this equation. Since it's an empirical distribution, we regularize counts by adding a count of 1 to every term. This will avoid dividing by 0 for unseen terms.

\n

Let's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X,

\n

$$ia(X) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } X}{\\text{number of proteins with term } X}\\right)$$

\n

For some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.

\n

For example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))

", "votes": null, "replies": [ { "id": 2234071, "author_name": "mtinti", "author_url": "", "post_date": "04/24/2023 20:41:33", "content": "

Thank you, crystal clear now

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2229905": "Something to consider is that many GO terms with high weights do not appear in the training set. \nhttps://www.kaggle.com/code/mtinti/go-terms-not-in-train/notebook?scriptVersionId=126704504", "2230059": "This notebook is either not public or has been deleted.", "2230283": "thanks, fixed now", "2230356": "Reading between the lines, I assume that the weights (`IA.txt`) have been generated from something other than the train/test data. Probably a much larger dataset containing many more proteins. Presumably most also won't appear in the future test data that solutions be evaluated on.\n\nI guess if we want to find proteins that have these other functions, we can always use other other (public) sources of data.", "2230363": "yep, the competition state: \"The weight for a term is determined by the logarithm of the frequency of occurrence of that term in a large pool of proteins\". What I don't get exactly is how this pool was assembled. Would not make sense to make available this dataset and use it for training?", "2230919": "> Would not make sense to make available this dataset and use it for training?\n\nThe dataset we used to generate the values in IA.txt is the training set :)", "2230923": "> Something to consider is that many GO terms with high weights do not appear in the training set. \n\nThat is a problem we have in general with protein annotation databases. The information content of most experimentally validated terms is low. (or \"low weight\" as you say). The reasons are mostly because high throughput experiments which contribute the bulk of experimental annotation increase the frequency of the terms they are designed to provide, hence lowering the information content (\"weight\") of the term. \n\nExample: You have an microscopy image analysis system, automated, that can only tell if a protein is located in the cytosol or the nucleus. The two largest and most obvious compartments int he cell. A robot reads 2000 slides a day, and categorizes the location as either \"nucleus\" or \"cytosol\". However, you need a graduate student to know if a cytosol-located protein is located somewhere more specifically: say the Golgi apparatus (look it up). That student can only do 20 slides per day. Which terms will populate the database more?\n\nSee also this paper: [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003063](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003063)", "2231802": "Thanks, so all the terms come from the train set, however only the experimental validated go terms ended in train_terms.tsv files, right?", "2234049": ">many GO terms with high weights do not appear in the training set. \n\nThis is true. To explain why that happens, let's consider how IA is calculated\n\n$$ia(v) = \\log_2 \\frac{1}{\\mathrm{Pr}(v|\\mathcal{Pa}(v))}$$\n\nWe use the training set (train_terms.tsv) as the empirical distribution that approximates the denominator in this equation. Since it's an empirical distribution, we regularize counts by adding a count of 1 to every term. This will avoid dividing by 0 for unseen terms.\n\nLet's sat term X doesn't appear in the training set and its IA is large. This means that there are many proteins in the dataset that are annotated with the parent term(s) of X, \n\n$$ia(X) = \\log_2\\left(\\frac{\\text{number of proteins with parent term(s) of } X}{\\text{number of proteins with term } X}\\right)$$ \n\nFor some unseen X the denominator would be 0, but with our regularization it's 1. log_2(large number/1) = relatively large number.\n\nFor example, here is one term I just looked up. Term GO:2001081 does not appear in train_terms.tsv. It has one parent term (GO:0030247) which appears in 142 proteins in the training set. IA(GO:2001081)= 7.15987 = log_2( (142+1)/(0+1))", "2234071": "Thank you, crystal clear now" }, "source": "meta" }