The ontology data have changed significantly during the competition. For example, now some child terms can be propagated to >1 aspects (e.g. BPO and MFO https://www.ebi.ac.uk/QuickGO/term/GO:0016303)
\nIs it true that for final evaluation 2023-01-01 release of the GO graph will be used?
The ontology data have changed significantly during the competition. For example, now some child terms can be propagated to >1 aspects (e.g. BPO and MFO https://www.ebi.ac.uk/QuickGO/term/GO:0016303)
\nIs it true that for final evaluation 2023-01-01 release of the GO graph will be used?
I would like to add that the importance of this question lies in the substantial disparities between the provided \"go-basic.obo\" file and the current version of the ontology data. And thus, we expect that the version used will have a great impact due to the automatic propagation of predictions to the GO root.
\nThe current version of the GO graph incorporates modifications and removals of some parent terms and introduces new parents. Notably, the current ontology version reveals that certain terms, such as this one https://www.ebi.ac.uk/QuickGO/term/GO:0016303, can be propagated to both ontology roots, resulting in a considerable increase in their number of ancestors.
\nGO graph provided by the competition:
\nontology <- get_ontology(\"Train/go-basic.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))
\n[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0035004\" \"GO:0016773\" \"GO:0052742\" \"GO:0016303\"
GO graph as at 13 jul 2023:
\nontology <- get_ontology(\"datasets/go-basic_13jul23.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))
[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0001727\" \"GO:0016773\" \"GO:0052742\" \"GO:0008150\" \"GO:0008152\"
\n[11] \"GO:0009987\" \"GO:0044237\" \"GO:0006793\" \"GO:0006796\" \"GO:0071704\" \"GO:0019637\" \"GO:0044238\" \"GO:0006629\" \"GO:0044255\" \"GO:0006644\"
\n[21] \"GO:0046486\" \"GO:0006650\" \"GO:0009058\" \"GO:1901576\" \"GO:0008610\" \"GO:0090407\" \"GO:0008654\" \"GO:0044249\" \"GO:0045017\" \"GO:0046474\"
\n[31] \"GO:0046488\" \"GO:0006661\" \"GO:0046854\" \"GO:0036092\" \"GO:0016303\"
!! GO:0016303 has much more ancestors now !!
\nApart from changes in parent-child relationships, I see that a number of terms (I found approximately 300) from the training set have become obsolete. For instance, GO:0034414 (BPO) on 2023-04-24 was replaced by GO:0042780 as it represents a molecular function: https://www.ebi.ac.uk/QuickGO/term/GO:0034414.
\nThus, depending on the version used by the competition organizers for propagation predictions to the roots, different scores are likely to be obtained during the final evaluation.
\n**If the organizers utilize the outdated obo file, this presents a predicament as obsolete terms can be automatically included due to the propagation to the root, if we submit predictions of their children, but predicting obsolete terms holds no scientific value! **
\nPlease accept my apologies if I have misunderstood any aspects of the problem, looking forward to hearing what exactly version is used for propagation to the root
", "rawMarkdown": "I would like to add that the importance of this question lies in the substantial disparities between the provided \"go-basic.obo\" file and the current version of the ontology data. And thus, we expect that the version used will have a great impact due to the automatic propagation of predictions to the GO root. \n\n[The current version of the GO graph](http://geneontology.org/docs/download-ontology/ ) incorporates modifications and removals of some parent terms and introduces new parents. Notably, the current ontology version reveals that certain terms, such as this one https://www.ebi.ac.uk/QuickGO/term/GO:0016303, can be propagated to both ontology roots, resulting in a considerable increase in their number of ancestors.\n\nGO graph provided by the competition:\n\n`ontology <- get_ontology(\"Train/go-basic.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))`\n[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0035004\" \"GO:0016773\" \"GO:0052742\" \"GO:0016303\"\n\n\nGO graph as at 13 jul 2023:\n\n`ontology <- get_ontology(\"datasets/go-basic_13jul23.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))`\n\n[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0001727\" \"GO:0016773\" \"GO:0052742\" \"GO:0008150\" \"GO:0008152\"\n[11] \"GO:0009987\" \"GO:0044237\" \"GO:0006793\" \"GO:0006796\" \"GO:0071704\" \"GO:0019637\" \"GO:0044238\" \"GO:0006629\" \"GO:0044255\" \"GO:0006644\"\n[21] \"GO:0046486\" \"GO:0006650\" \"GO:0009058\" \"GO:1901576\" \"GO:0008610\" \"GO:0090407\" \"GO:0008654\" \"GO:0044249\" \"GO:0045017\" \"GO:0046474\"\n[31] \"GO:0046488\" \"GO:0006661\" \"GO:0046854\" \"GO:0036092\" \"GO:0016303\"\n\n!! GO:0016303 has much more ancestors now !!\n\nApart from changes in parent-child relationships, I see that a number of terms (I found approximately 300) from the training set have become obsolete. For instance, GO:0034414 (BPO) on 2023-04-24 was replaced by GO:0042780 as it represents a molecular function: https://www.ebi.ac.uk/QuickGO/term/GO:0034414.\n\nThus, depending on the version used by the competition organizers for propagation predictions to the roots, different scores are likely to be obtained during the final evaluation.\n\n**If the organizers utilize the outdated obo file, this presents a predicament as obsolete terms can be automatically included due to the propagation to the root, if we submit predictions of their children, but predicting obsolete terms holds no scientific value! **\n\nPlease accept my apologies if I have misunderstood any aspects of the problem, looking forward to hearing what exactly version is used for propagation to the root", "votes": null }, { "id": "2348753", "postDate": "07/18/2023 00:00:46", "content": "Good questions, all. Let me untangle them:
\nThe first question is, why does the CAFA competition use the obo file from January 1st rather than a later one. We, of course, cannot use an obo file from the future (so the current one was not available when the competition began). We also allow the use of any obo file between competition start and end since which one will we use to assess the predictions? If group A trains/predicts using a 5/2023 version and group B trains/predicts using a 7/2023 version which one would we use to assess? Or, we could use a standard obo from 12/2023 when we provide the final scores, but that would hardly be fair. We therefore use the obo file that is closest to the start time (before the start) , and which is in lockstep with the training set. Which is the obo file we will use for assessment.
\n\n\nbut predicting obsolete terms holds no scientific value!
\n
Regarding obsoletions: those do happen, but they are rare in the grand scheme of things and rarely affect predictions, since most statistically will not be represented in the test set. Furthermore, may obsoletions are synonymous (i.e. direct replacements), as on the example you gave. We can follow those easily and assess based on the older term. The obsoletions are very rarely wholesale branch removals, but rather tweaking of terms to make them more accurate. So there is quite a bit of value left.
\nRegarding propagation for terms to the root: only propagate to the relevant ontology root. So in the case of GO:0016303 which is a molecular function term, you should propagate through the MFO aspect only (grey boxes in the example).
\nLet me know if there are any other questions, or if I missed anything.
", "rawMarkdown": "Good questions, all. Let me untangle them:\n\nThe first question is, why does the CAFA competition use the obo file from January 1st rather than a later one. We, of course, cannot use an obo file from the future (so the current one was not available when the competition began). We also allow the use of any obo file between competition start and end since which one will we use to assess the predictions? If group A trains/predicts using a 5/2023 version and group B trains/predicts using a 7/2023 version which one would we use to assess? Or, we could use a standard obo from 12/2023 when we provide the final scores, but that would hardly be fair. We therefore use the obo file that is closest to the start time (before the start) , and which is in lockstep with the training set. Which is the obo file we will use for assessment. \n\n> but predicting obsolete terms holds no scientific value! \n\nRegarding obsoletions: those do happen, but they are rare in the grand scheme of things and rarely affect predictions, since most statistically will not be represented in the test set. Furthermore, may obsoletions are synonymous (i.e. direct replacements), as on the example you gave. We can follow those easily and assess based on the older term. The obsoletions are very rarely wholesale branch removals, but rather tweaking of terms to make them more accurate. So there is quite a bit of value left. \n\nRegarding propagation for terms to the root: only propagate to the relevant ontology root. So in the case of [GO:0016303](https://www.ebi.ac.uk/QuickGO/term/GO:0016303) which is a molecular function term, you should propagate through the MFO aspect only (grey boxes in the example). \n\nLet me know if there are any other questions, or if I missed anything.", "votes": null }, { "id": "2349367", "postDate": "07/18/2023 10:48:21", "content": "Now it's clear enough, thanks!
", "rawMarkdown": "Now it's clear enough, thanks!", "votes": null } ], "comments": [ { "id": 2348101, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "07/17/2023 11:43:29", "content": "I would like to add that the importance of this question lies in the substantial disparities between the provided \"go-basic.obo\" file and the current version of the ontology data. And thus, we expect that the version used will have a great impact due to the automatic propagation of predictions to the GO root.
\nThe current version of the GO graph incorporates modifications and removals of some parent terms and introduces new parents. Notably, the current ontology version reveals that certain terms, such as this one https://www.ebi.ac.uk/QuickGO/term/GO:0016303, can be propagated to both ontology roots, resulting in a considerable increase in their number of ancestors.
\nGO graph provided by the competition:
\nontology <- get_ontology(\"Train/go-basic.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))
\n[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0035004\" \"GO:0016773\" \"GO:0052742\" \"GO:0016303\"
GO graph as at 13 jul 2023:
\nontology <- get_ontology(\"datasets/go-basic_13jul23.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))
[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0001727\" \"GO:0016773\" \"GO:0052742\" \"GO:0008150\" \"GO:0008152\"
\n[11] \"GO:0009987\" \"GO:0044237\" \"GO:0006793\" \"GO:0006796\" \"GO:0071704\" \"GO:0019637\" \"GO:0044238\" \"GO:0006629\" \"GO:0044255\" \"GO:0006644\"
\n[21] \"GO:0046486\" \"GO:0006650\" \"GO:0009058\" \"GO:1901576\" \"GO:0008610\" \"GO:0090407\" \"GO:0008654\" \"GO:0044249\" \"GO:0045017\" \"GO:0046474\"
\n[31] \"GO:0046488\" \"GO:0006661\" \"GO:0046854\" \"GO:0036092\" \"GO:0016303\"
!! GO:0016303 has much more ancestors now !!
\nApart from changes in parent-child relationships, I see that a number of terms (I found approximately 300) from the training set have become obsolete. For instance, GO:0034414 (BPO) on 2023-04-24 was replaced by GO:0042780 as it represents a molecular function: https://www.ebi.ac.uk/QuickGO/term/GO:0034414.
\nThus, depending on the version used by the competition organizers for propagation predictions to the roots, different scores are likely to be obtained during the final evaluation.
\n**If the organizers utilize the outdated obo file, this presents a predicament as obsolete terms can be automatically included due to the propagation to the root, if we submit predictions of their children, but predicting obsolete terms holds no scientific value! **
\nPlease accept my apologies if I have misunderstood any aspects of the problem, looking forward to hearing what exactly version is used for propagation to the root
", "votes": null, "replies": [ { "id": 2348753, "author_name": "iddux1", "author_url": "", "post_date": "07/18/2023 00:00:46", "content": "Good questions, all. Let me untangle them:
\nThe first question is, why does the CAFA competition use the obo file from January 1st rather than a later one. We, of course, cannot use an obo file from the future (so the current one was not available when the competition began). We also allow the use of any obo file between competition start and end since which one will we use to assess the predictions? If group A trains/predicts using a 5/2023 version and group B trains/predicts using a 7/2023 version which one would we use to assess? Or, we could use a standard obo from 12/2023 when we provide the final scores, but that would hardly be fair. We therefore use the obo file that is closest to the start time (before the start) , and which is in lockstep with the training set. Which is the obo file we will use for assessment.
\n\n\nbut predicting obsolete terms holds no scientific value!
\n
Regarding obsoletions: those do happen, but they are rare in the grand scheme of things and rarely affect predictions, since most statistically will not be represented in the test set. Furthermore, may obsoletions are synonymous (i.e. direct replacements), as on the example you gave. We can follow those easily and assess based on the older term. The obsoletions are very rarely wholesale branch removals, but rather tweaking of terms to make them more accurate. So there is quite a bit of value left.
\nRegarding propagation for terms to the root: only propagate to the relevant ontology root. So in the case of GO:0016303 which is a molecular function term, you should propagate through the MFO aspect only (grey boxes in the example).
\nLet me know if there are any other questions, or if I missed anything.
", "votes": null, "replies": [ { "id": 2349367, "author_name": "ogurtsov", "author_url": "", "post_date": "07/18/2023 10:48:21", "content": "Now it's clear enough, thanks!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2346424": "The [ontology data](http://geneontology.org/docs/download-ontology/) have changed significantly during the competition. For example, now some child terms can be propagated to >1 aspects (e.g. BPO and MFO https://www.ebi.ac.uk/QuickGO/term/GO:0016303)\nIs it true that for final evaluation `2023-01-01 release of the GO graph` will be used?", "2348101": "I would like to add that the importance of this question lies in the substantial disparities between the provided \"go-basic.obo\" file and the current version of the ontology data. And thus, we expect that the version used will have a great impact due to the automatic propagation of predictions to the GO root. \n\n[The current version of the GO graph](http://geneontology.org/docs/download-ontology/ ) incorporates modifications and removals of some parent terms and introduces new parents. Notably, the current ontology version reveals that certain terms, such as this one https://www.ebi.ac.uk/QuickGO/term/GO:0016303, can be propagated to both ontology roots, resulting in a considerable increase in their number of ancestors.\n\nGO graph provided by the competition:\n\n`ontology <- get_ontology(\"Train/go-basic.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))`\n[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0035004\" \"GO:0016773\" \"GO:0052742\" \"GO:0016303\"\n\n\nGO graph as at 13 jul 2023:\n\n`ontology <- get_ontology(\"datasets/go-basic_13jul23.obo\", propagate_relationships = c(\"is_a\", \"part_of\"))\nnames(unlist(ontology$ancestors[[\"GO:0016303\"]]))`\n\n[1] \"GO:0003674\" \"GO:0003824\" \"GO:0016740\" \"GO:0016772\" \"GO:0016301\" \"GO:0001727\" \"GO:0016773\" \"GO:0052742\" \"GO:0008150\" \"GO:0008152\"\n[11] \"GO:0009987\" \"GO:0044237\" \"GO:0006793\" \"GO:0006796\" \"GO:0071704\" \"GO:0019637\" \"GO:0044238\" \"GO:0006629\" \"GO:0044255\" \"GO:0006644\"\n[21] \"GO:0046486\" \"GO:0006650\" \"GO:0009058\" \"GO:1901576\" \"GO:0008610\" \"GO:0090407\" \"GO:0008654\" \"GO:0044249\" \"GO:0045017\" \"GO:0046474\"\n[31] \"GO:0046488\" \"GO:0006661\" \"GO:0046854\" \"GO:0036092\" \"GO:0016303\"\n\n!! GO:0016303 has much more ancestors now !!\n\nApart from changes in parent-child relationships, I see that a number of terms (I found approximately 300) from the training set have become obsolete. For instance, GO:0034414 (BPO) on 2023-04-24 was replaced by GO:0042780 as it represents a molecular function: https://www.ebi.ac.uk/QuickGO/term/GO:0034414.\n\nThus, depending on the version used by the competition organizers for propagation predictions to the roots, different scores are likely to be obtained during the final evaluation.\n\n**If the organizers utilize the outdated obo file, this presents a predicament as obsolete terms can be automatically included due to the propagation to the root, if we submit predictions of their children, but predicting obsolete terms holds no scientific value! **\n\nPlease accept my apologies if I have misunderstood any aspects of the problem, looking forward to hearing what exactly version is used for propagation to the root", "2348753": "Good questions, all. Let me untangle them:\n\nThe first question is, why does the CAFA competition use the obo file from January 1st rather than a later one. We, of course, cannot use an obo file from the future (so the current one was not available when the competition began). We also allow the use of any obo file between competition start and end since which one will we use to assess the predictions? If group A trains/predicts using a 5/2023 version and group B trains/predicts using a 7/2023 version which one would we use to assess? Or, we could use a standard obo from 12/2023 when we provide the final scores, but that would hardly be fair. We therefore use the obo file that is closest to the start time (before the start) , and which is in lockstep with the training set. Which is the obo file we will use for assessment. \n\n> but predicting obsolete terms holds no scientific value! \n\nRegarding obsoletions: those do happen, but they are rare in the grand scheme of things and rarely affect predictions, since most statistically will not be represented in the test set. Furthermore, may obsoletions are synonymous (i.e. direct replacements), as on the example you gave. We can follow those easily and assess based on the older term. The obsoletions are very rarely wholesale branch removals, but rather tweaking of terms to make them more accurate. So there is quite a bit of value left. \n\nRegarding propagation for terms to the root: only propagate to the relevant ontology root. So in the case of [GO:0016303](https://www.ebi.ac.uk/QuickGO/term/GO:0016303) which is a molecular function term, you should propagate through the MFO aspect only (grey boxes in the example). \n\nLet me know if there are any other questions, or if I missed anything.", "2349367": "Now it's clear enough, thanks!" }, "source": "meta" }