Looking at the experimental vs electronic annotation situation, I'm puzzled as to why there are so few experimental annotations to train on. A protein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term - as is \"oxygen binding\". That's ridiculous, because there is probably half a public library of prior research that demonstrates with 100% confidence that myoglobin is a heme-binding protein that binds oxygen. There's even a crystal structure of the protein with heme, iron and the bound oxygen present. So, what are the GO database curators waiting for? Nobody is going to do further experiments to determine whether myoglobin binds heme and oxygen when you can just read about it in a thousand textbooks. Even a 1st year undergrad could determine that this is a completely known fact.
\nSurely doing machine learning on such a fundementally flawed training dataset is something of a waste of time. Even if we decided that the challenge was to do text mining on the literature to extract historical experimental terms, then that would still be a waste of time because the test set would only ever contain terms that have entirely new experimental validation, because the curation process seems frozen in time.
\nEven when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.
\nSo, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?
\nThis is not intended as a criticism of anyone, just an invitation to debate.
", "messages": [ { "id": "2238239", "postDate": "04/28/2023 10:40:28", "content": "Looking at the experimental vs electronic annotation situation, I'm puzzled as to why there are so few experimental annotations to train on. A protein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term - as is \"oxygen binding\". That's ridiculous, because there is probably half a public library of prior research that demonstrates with 100% confidence that myoglobin is a heme-binding protein that binds oxygen. There's even a crystal structure of the protein with heme, iron and the bound oxygen present. So, what are the GO database curators waiting for? Nobody is going to do further experiments to determine whether myoglobin binds heme and oxygen when you can just read about it in a thousand textbooks. Even a 1st year undergrad could determine that this is a completely known fact.
\nSurely doing machine learning on such a fundementally flawed training dataset is something of a waste of time. Even if we decided that the challenge was to do text mining on the literature to extract historical experimental terms, then that would still be a waste of time because the test set would only ever contain terms that have entirely new experimental validation, because the curation process seems frozen in time.
\nEven when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.
\nSo, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?
\nThis is not intended as a criticism of anyone, just an invitation to debate.
", "rawMarkdown": "Looking at the experimental vs electronic annotation situation, I'm puzzled as to why there are so few experimental annotations to train on. A protein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term - as is \"oxygen binding\". That's ridiculous, because there is probably half a public library of prior research that demonstrates with 100% confidence that myoglobin is a heme-binding protein that binds oxygen. There's even a crystal structure of the protein with heme, iron and the bound oxygen present. So, what are the GO database curators waiting for? Nobody is going to do further experiments to determine whether myoglobin binds heme and oxygen when you can just read about it in a thousand textbooks. Even a 1st year undergrad could determine that this is a completely known fact.\n\nSurely doing machine learning on such a fundementally flawed training dataset is something of a waste of time. Even if we decided that the challenge was to do text mining on the literature to extract historical experimental terms, then that would still be a waste of time because the test set would only ever contain terms that have entirely new experimental validation, because the curation process seems frozen in time.\n\nEven when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.\n\nSo, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?\n\nThis is not intended as a criticism of anyone, just an invitation to debate.", "votes": null }, { "id": "2238417", "postDate": "04/28/2023 14:03:43", "content": "\n\nprotein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term
\n
Which species is that protein from? If it is not from a species in which an experiment has been conducted directly, then there's no experimental evidence for myoglobin in that species. IEA evidence in this case would still be highly reliable due to broad knowledge of myoglobin in other species and inference by high level of sequence similarity. In any case, the assessment will probably not be conducted on this protein anyway: as you pointed out, new experiments are unlikely.
\n\n\nEven when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.
\n
I'm not sure how you made that determination regarding the quantity of the error rate, or how you measured it.
\n\n\nSo, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?
\n
As we keep saying, competitors are encouraged to construct their own training set. Historically CAFA did not provide training sets, and part of the competition was constructing one's own. The assessment will only be conducted on proteins that gain new experimentally validated. I would also welcome some evidence of the errors and fundamental flaws you claim.
", "rawMarkdown": ">protein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term\n\nWhich species is that protein from? If it is not from a species in which an experiment has been conducted directly, then there's no experimental evidence for myoglobin in that species. IEA evidence in this case would still be highly reliable due to broad knowledge of myoglobin in other species and inference by high level of sequence similarity. In any case, the assessment will probably not be conducted on this protein anyway: as you pointed out, new experiments are unlikely. \n\n> Even when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.\n\nI'm not sure how you made that determination regarding the quantity of the error rate, or how you measured it. \n\n>So, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?\n\nAs we keep saying, competitors are encouraged to construct their own training set. Historically CAFA did not provide training sets, and part of the competition was constructing one's own. The assessment will only be conducted on proteins that gain new experimentally validated. I would also welcome some evidence of the errors and fundamental flaws you claim.", "votes": null }, { "id": "2238631", "postDate": "04/28/2023 17:25:35", "content": "Sperm whale myoglobin was the one I was looking at i.e. the source of the first structure solved by Kendrew in 1958! There are hundreds of structures of it in PDB.
\nThe earliest estimates of Y2H screens were around 70% FDR - though I think that settled to around 15-20% in further studies. Admittedly, bioinformatics can be used to reduce that FDR, but still it's a well known example of high throughput data that is far from reliable. I'm not saying that it's all as bad as that, but it's not easy to ascertain how accurate all the high throughput labels are. Some certainly could be only 50% accurate.
\nI do agree that people should assemble their own training data, but it's difficult for individuals to do that on the required scale. I mean if we can't even deduce from things like QuickGO that sperm whale myoglobin IS 100% an oxgen and heme binding protein then it's difficult to know where individuals can get better data. I think this is really the thing that the field should be focussing effort on i.e. producing a really high quality training set that clearly demarkates what we definitely already know from what we don't.
", "rawMarkdown": "Sperm whale myoglobin was the one I was looking at i.e. the source of the first structure solved by Kendrew in 1958! There are hundreds of structures of it in PDB.\n\nThe earliest estimates of Y2H screens were around 70% FDR - though I think that settled to around 15-20% in further studies. Admittedly, bioinformatics can be used to reduce that FDR, but still it's a well known example of high throughput data that is far from reliable. I'm not saying that it's all as bad as that, but it's not easy to ascertain how accurate all the high throughput labels are. Some certainly could be only 50% accurate.\n\nI do agree that people should assemble their own training data, but it's difficult for individuals to do that on the required scale. I mean if we can't even deduce from things like QuickGO that sperm whale myoglobin IS 100% an oxgen and heme binding protein then it's difficult to know where individuals can get better data. I think this is really the thing that the field should be focussing effort on i.e. producing a really high quality training set that clearly demarkates what we definitely already know from what we don't.", "votes": null }, { "id": "2239078", "postDate": "04/29/2023 06:37:01", "content": "Relative to previous CAFAs, this IS a high quality training dataset - they assembled proteins and filtered annotations for a definition of experimental evidence
", "rawMarkdown": "Relative to previous CAFAs, this IS a high quality training dataset - they assembled proteins and filtered annotations for a definition of experimental evidence", "votes": null }, { "id": "2239342", "postDate": "04/29/2023 11:54:27", "content": "You are making the common mistake of equating \"experimental\" with \"reliable\". When you next listen to an experimentalist give a talk, especially when discussing new high throughput data, remember to ask them what the false discovery rate is of the approach they are using. See if you get a useful answer.
\nThe problem is that many of the non-experimental terms are as reliable or even more reliable than the experimental terms, and there are something like 100x more of them. There are therefore many times more correct non-experimental GO terms than experimental ones. Of course the same also applies to incorrect non-experimental terms than incorrect experimental terms. Even when 100% reliable experimental data is available, like for myoglobin, we are often still just given non-experimental GO terms because the curated GO annotations are so incomplete.
\nGiven that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.
", "rawMarkdown": "You are making the common mistake of equating \"experimental\" with \"reliable\". When you next listen to an experimentalist give a talk, especially when discussing new high throughput data, remember to ask them what the false discovery rate is of the approach they are using. See if you get a useful answer.\n\nThe problem is that many of the non-experimental terms are as reliable or even more reliable than the experimental terms, and there are something like 100x more of them. There are therefore many times more correct non-experimental GO terms than experimental ones. Of course the same also applies to incorrect non-experimental terms than incorrect experimental terms. Even when 100% reliable experimental data is available, like for myoglobin, we are often still just given non-experimental GO terms because the curated GO annotations are so incomplete.\n\nGiven that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.", "votes": null }, { "id": "2241818", "postDate": "05/01/2023 20:49:14", "content": "\n\nGiven that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.
\n
This is not a given. As someone who does both experiments and non-experimental predictions on a daily basis, I think that experimental data is more reliable than predictions. It is true that there is no guarantee that any given experimental data point is reliable than any given prediction, because there are poor experiments out there and there are really good non-experimental predictions. In fact, even poor predictions are sometimes \"proven\" by experiments - see here. This argues that experiments are not immune from bias or poor execution, yet the overall reliability index is clearly pointing towards experiments. My students make functional predictions about proteins all the time as part of their homework, and those are often incorrect. In some cases we don't know whether the predictions are right or wrong without experiments - and sometimes even after testing them.
\nWhile I understand your frustration and agree that many functional inferences can be made for functionally equivalent proteins, doing so is not always aligned with the spirit of GO annotations. Many non-experimental predictions are highly reliable, either for the transitivity reasons you mentioned, or because people who make them know what they are doing. Still, GO annotations are based on different rules of evidence, and that's just how it is.
", "rawMarkdown": "> Given that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.\n\nThis is not a given. As someone who does both experiments and non-experimental predictions on a daily basis, I think that experimental data is more reliable than predictions. It is true that there is no guarantee that any given experimental data point is reliable than any given prediction, because there are poor experiments out there and there are really good non-experimental predictions. In fact, even poor predictions are sometimes \"proven\" by experiments - see [**here**](https://pubmed.ncbi.nlm.nih.gov/11790254/). This argues that experiments are not immune from bias or poor execution, yet the overall reliability index is clearly pointing towards experiments. My students make functional predictions about proteins all the time as part of their homework, and those are often incorrect. In some cases we don't know whether the predictions are right or wrong without experiments - and sometimes even after testing them.\n\nWhile I understand your frustration and agree that many functional inferences can be made for functionally equivalent proteins, doing so is not always aligned with the spirit of GO annotations. Many non-experimental predictions are highly reliable, either for the transitivity reasons you mentioned, or because people who make them know what they are doing. Still, GO annotations are based on different rules of evidence, and that's just how it is.", "votes": null } ], "comments": [ { "id": 2238417, "author_name": "iddux1", "author_url": "", "post_date": "04/28/2023 14:03:43", "content": "\n\nprotein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term
\n
Which species is that protein from? If it is not from a species in which an experiment has been conducted directly, then there's no experimental evidence for myoglobin in that species. IEA evidence in this case would still be highly reliable due to broad knowledge of myoglobin in other species and inference by high level of sequence similarity. In any case, the assessment will probably not be conducted on this protein anyway: as you pointed out, new experiments are unlikely.
\n\n\nEven when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.
\n
I'm not sure how you made that determination regarding the quantity of the error rate, or how you measured it.
\n\n\nSo, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?
\n
As we keep saying, competitors are encouraged to construct their own training set. Historically CAFA did not provide training sets, and part of the competition was constructing one's own. The assessment will only be conducted on proteins that gain new experimentally validated. I would also welcome some evidence of the errors and fundamental flaws you claim.
", "votes": null, "replies": [ { "id": 2238631, "author_name": "biodav", "author_url": "", "post_date": "04/28/2023 17:25:35", "content": "Sperm whale myoglobin was the one I was looking at i.e. the source of the first structure solved by Kendrew in 1958! There are hundreds of structures of it in PDB.
\nThe earliest estimates of Y2H screens were around 70% FDR - though I think that settled to around 15-20% in further studies. Admittedly, bioinformatics can be used to reduce that FDR, but still it's a well known example of high throughput data that is far from reliable. I'm not saying that it's all as bad as that, but it's not easy to ascertain how accurate all the high throughput labels are. Some certainly could be only 50% accurate.
\nI do agree that people should assemble their own training data, but it's difficult for individuals to do that on the required scale. I mean if we can't even deduce from things like QuickGO that sperm whale myoglobin IS 100% an oxgen and heme binding protein then it's difficult to know where individuals can get better data. I think this is really the thing that the field should be focussing effort on i.e. producing a really high quality training set that clearly demarkates what we definitely already know from what we don't.
", "votes": null, "replies": [ { "id": 2239078, "author_name": "danofer", "author_url": "", "post_date": "04/29/2023 06:37:01", "content": "Relative to previous CAFAs, this IS a high quality training dataset - they assembled proteins and filtered annotations for a definition of experimental evidence
", "votes": null, "replies": [ { "id": 2239342, "author_name": "biodav", "author_url": "", "post_date": "04/29/2023 11:54:27", "content": "You are making the common mistake of equating \"experimental\" with \"reliable\". When you next listen to an experimentalist give a talk, especially when discussing new high throughput data, remember to ask them what the false discovery rate is of the approach they are using. See if you get a useful answer.
\nThe problem is that many of the non-experimental terms are as reliable or even more reliable than the experimental terms, and there are something like 100x more of them. There are therefore many times more correct non-experimental GO terms than experimental ones. Of course the same also applies to incorrect non-experimental terms than incorrect experimental terms. Even when 100% reliable experimental data is available, like for myoglobin, we are often still just given non-experimental GO terms because the curated GO annotations are so incomplete.
\nGiven that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.
", "votes": null, "replies": [ { "id": 2241818, "author_name": "tilii7", "author_url": "", "post_date": "05/01/2023 20:49:14", "content": "\n\nGiven that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.
\n
This is not a given. As someone who does both experiments and non-experimental predictions on a daily basis, I think that experimental data is more reliable than predictions. It is true that there is no guarantee that any given experimental data point is reliable than any given prediction, because there are poor experiments out there and there are really good non-experimental predictions. In fact, even poor predictions are sometimes \"proven\" by experiments - see here. This argues that experiments are not immune from bias or poor execution, yet the overall reliability index is clearly pointing towards experiments. My students make functional predictions about proteins all the time as part of their homework, and those are often incorrect. In some cases we don't know whether the predictions are right or wrong without experiments - and sometimes even after testing them.
\nWhile I understand your frustration and agree that many functional inferences can be made for functionally equivalent proteins, doing so is not always aligned with the spirit of GO annotations. Many non-experimental predictions are highly reliable, either for the transitivity reasons you mentioned, or because people who make them know what they are doing. Still, GO annotations are based on different rules of evidence, and that's just how it is.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2238239": "Looking at the experimental vs electronic annotation situation, I'm puzzled as to why there are so few experimental annotations to train on. A protein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term - as is \"oxygen binding\". That's ridiculous, because there is probably half a public library of prior research that demonstrates with 100% confidence that myoglobin is a heme-binding protein that binds oxygen. There's even a crystal structure of the protein with heme, iron and the bound oxygen present. So, what are the GO database curators waiting for? Nobody is going to do further experiments to determine whether myoglobin binds heme and oxygen when you can just read about it in a thousand textbooks. Even a 1st year undergrad could determine that this is a completely known fact.\n\nSurely doing machine learning on such a fundementally flawed training dataset is something of a waste of time. Even if we decided that the challenge was to do text mining on the literature to extract historical experimental terms, then that would still be a waste of time because the test set would only ever contain terms that have entirely new experimental validation, because the curation process seems frozen in time.\n\nEven when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.\n\nSo, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?\n\nThis is not intended as a criticism of anyone, just an invitation to debate.", "2238417": ">protein that just about anyone knows well, myoglobin, has no experimental GO terms at all. For example, \"heme binding\" is listed as an IEA term\n\nWhich species is that protein from? If it is not from a species in which an experiment has been conducted directly, then there's no experimental evidence for myoglobin in that species. IEA evidence in this case would still be highly reliable due to broad knowledge of myoglobin in other species and inference by high level of sequence similarity. In any case, the assessment will probably not be conducted on this protein anyway: as you pointed out, new experiments are unlikely. \n\n> Even when you look at the experimental terms, I'm similarly puzzled as to how some of those can be considered reliable - especially the high throughput ones, where the intrinsic error rates could well be more than 50%.\n\nI'm not sure how you made that determination regarding the quantity of the error rate, or how you measured it. \n\n>So, what's the point of a challenge like this when the highest quality training data is basically historically hidden away and the new terms that make up much of the test set are likely to have a lot of errors?\n\nAs we keep saying, competitors are encouraged to construct their own training set. Historically CAFA did not provide training sets, and part of the competition was constructing one's own. The assessment will only be conducted on proteins that gain new experimentally validated. I would also welcome some evidence of the errors and fundamental flaws you claim.", "2238631": "Sperm whale myoglobin was the one I was looking at i.e. the source of the first structure solved by Kendrew in 1958! There are hundreds of structures of it in PDB.\n\nThe earliest estimates of Y2H screens were around 70% FDR - though I think that settled to around 15-20% in further studies. Admittedly, bioinformatics can be used to reduce that FDR, but still it's a well known example of high throughput data that is far from reliable. I'm not saying that it's all as bad as that, but it's not easy to ascertain how accurate all the high throughput labels are. Some certainly could be only 50% accurate.\n\nI do agree that people should assemble their own training data, but it's difficult for individuals to do that on the required scale. I mean if we can't even deduce from things like QuickGO that sperm whale myoglobin IS 100% an oxgen and heme binding protein then it's difficult to know where individuals can get better data. I think this is really the thing that the field should be focussing effort on i.e. producing a really high quality training set that clearly demarkates what we definitely already know from what we don't.", "2239078": "Relative to previous CAFAs, this IS a high quality training dataset - they assembled proteins and filtered annotations for a definition of experimental evidence", "2239342": "You are making the common mistake of equating \"experimental\" with \"reliable\". When you next listen to an experimentalist give a talk, especially when discussing new high throughput data, remember to ask them what the false discovery rate is of the approach they are using. See if you get a useful answer.\n\nThe problem is that many of the non-experimental terms are as reliable or even more reliable than the experimental terms, and there are something like 100x more of them. There are therefore many times more correct non-experimental GO terms than experimental ones. Of course the same also applies to incorrect non-experimental terms than incorrect experimental terms. Even when 100% reliable experimental data is available, like for myoglobin, we are often still just given non-experimental GO terms because the curated GO annotations are so incomplete.\n\nGiven that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.", "2241818": "> Given that there is no guarantee that the experimental data is more reliable than the non-experimental, and given the tiny amounts of experimental data anyway, and the lack of completeness, the real challenge here is to find a way to determine which of the non-experimental terms actually can be relied upon.\n\nThis is not a given. As someone who does both experiments and non-experimental predictions on a daily basis, I think that experimental data is more reliable than predictions. It is true that there is no guarantee that any given experimental data point is reliable than any given prediction, because there are poor experiments out there and there are really good non-experimental predictions. In fact, even poor predictions are sometimes \"proven\" by experiments - see [**here**](https://pubmed.ncbi.nlm.nih.gov/11790254/). This argues that experiments are not immune from bias or poor execution, yet the overall reliability index is clearly pointing towards experiments. My students make functional predictions about proteins all the time as part of their homework, and those are often incorrect. In some cases we don't know whether the predictions are right or wrong without experiments - and sometimes even after testing them.\n\nWhile I understand your frustration and agree that many functional inferences can be made for functionally equivalent proteins, doing so is not always aligned with the spirit of GO annotations. Many non-experimental predictions are highly reliable, either for the transitivity reasons you mentioned, or because people who make them know what they are doing. Still, GO annotations are based on different rules of evidence, and that's just how it is." }, "source": "meta" }