The current rescoring process raises several questions.
\nIn my case, the interface shows 0/2 because I was not able to select submissions (I am not the team leader). I assumed that the two highest-scoring submissions would automatically be chosen, but there is no longer any trace of that selection. What is the rationale for evaluating all submissions instead? This appears computationally expensive, and it is not clear what benefit it provides compared to evaluating only the selected ones.
\nWas it necessary to overwrite the previous leaderboard during this debugging phase? The existing leaderboard provided a stable reference and reflected significant effort from participants. A public/private split could have preserved continuity while rescoring was in progress. At the moment, it is difficult to track performance changes.
\nThe explanation that there were not enough proteins with updated annotations is understandable. However, combining new evaluation data with previous dataset is difficult to interpret and could be seen as inconsistent with the competition description and with the decisions participants had to make during the competition. Personally, I would prefer a volatile score that follows a clear and consistent methodology, rather than changing the evaluation methodology mid-process.
\nMore generally, greater transparency about how the evaluation dataset is constructed would be very helpful. Making these details public would allow participants to audit the process and potentially help improve the design of future CAFA editions. The most recent fully available evaluation dataset I could find corresponds to CAFA3.
\nAt this point, I still have some questions. For example, was the previous evaluation set the result of a synchronized non-publication window with UniProt? Would it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?
\nClarifying these aspects would help me better understand the rationale behind the current approach.
\nThanks
", "messages": [ { "id": "3435094", "postDate": "04/03/2026 17:02:16", "content": "The current rescoring process raises several questions.
\nIn my case, the interface shows 0/2 because I was not able to select submissions (I am not the team leader). I assumed that the two highest-scoring submissions would automatically be chosen, but there is no longer any trace of that selection. What is the rationale for evaluating all submissions instead? This appears computationally expensive, and it is not clear what benefit it provides compared to evaluating only the selected ones.
\nWas it necessary to overwrite the previous leaderboard during this debugging phase? The existing leaderboard provided a stable reference and reflected significant effort from participants. A public/private split could have preserved continuity while rescoring was in progress. At the moment, it is difficult to track performance changes.
\nThe explanation that there were not enough proteins with updated annotations is understandable. However, combining new evaluation data with previous dataset is difficult to interpret and could be seen as inconsistent with the competition description and with the decisions participants had to make during the competition. Personally, I would prefer a volatile score that follows a clear and consistent methodology, rather than changing the evaluation methodology mid-process.
\nMore generally, greater transparency about how the evaluation dataset is constructed would be very helpful. Making these details public would allow participants to audit the process and potentially help improve the design of future CAFA editions. The most recent fully available evaluation dataset I could find corresponds to CAFA3.
\nAt this point, I still have some questions. For example, was the previous evaluation set the result of a synchronized non-publication window with UniProt? Would it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?
\nClarifying these aspects would help me better understand the rationale behind the current approach.
\nThanks
", "rawMarkdown": "The current rescoring process raises several questions.\n\nIn my case, the interface shows 0/2 because I was not able to select submissions (I am not the team leader). I assumed that the two highest-scoring submissions would automatically be chosen, but there is no longer any trace of that selection. What is the rationale for evaluating all submissions instead? This appears computationally expensive, and it is not clear what benefit it provides compared to evaluating only the selected ones.\n\nWas it necessary to overwrite the previous leaderboard during this debugging phase? The existing leaderboard provided a stable reference and reflected significant effort from participants. A public/private split could have preserved continuity while rescoring was in progress. At the moment, it is difficult to track performance changes.\n\nThe explanation that there were not enough proteins with updated annotations is understandable. However, combining new evaluation data with previous dataset is difficult to interpret and could be seen as inconsistent with the competition description and with the decisions participants had to make during the competition. Personally, I would prefer a volatile score that follows a clear and consistent methodology, rather than changing the evaluation methodology mid-process.\n\nMore generally, greater transparency about how the evaluation dataset is constructed would be very helpful. Making these details public would allow participants to audit the process and potentially help improve the design of future CAFA editions. The most recent fully available evaluation dataset I could find corresponds to CAFA3.\n\nAt this point, I still have some questions. For example, was the previous evaluation set the result of a synchronized non-publication window with UniProt? Would it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?\n\nClarifying these aspects would help me better understand the rationale behind the current approach.\n\nThanks", "votes": null }, { "id": "3435106", "postDate": "04/03/2026 17:22:20", "content": "\n\nwas the previous evaluation set the result of a synchronized non-publication window with UniProt?
\n
The previous evaluation set is a hold-out/secret set of GO annotations for a small number of proteins that UniProt has agreed to not include in their usual knowledgebase release, for the purpose of a live leaderboard on Kaggle CAFA. This set is currently not released to the public, hence the reason it is still included in the updated evaluation set.
\n\n\nWould it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?
\n
The new evaluation set currently has many more proteins than the previous secret evaluation set, but yes the longer we wait the more robust the evaluation set will be. CAFA4 and CAFA5 (manuscript in preparation) will have a long evaluation window that improves the robustness of the evaluation. However, within the scope of Kaggle competition, we can only evaluate on a 4-month window (between Feb 2026 and Jun 2026 for CAFA6 specifically).
", "rawMarkdown": ">was the previous evaluation set the result of a synchronized non-publication window with UniProt?\n\nThe previous evaluation set is a hold-out/secret set of GO annotations for a small number of proteins that UniProt has agreed to not include in their usual knowledgebase release, for the purpose of a live leaderboard on Kaggle CAFA. This set is currently not released to the public, hence the reason it is still included in the updated evaluation set.\n\n>Would it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?\n\nThe new evaluation set currently has many more proteins than the previous secret evaluation set, but yes the longer we wait the more robust the evaluation set will be. CAFA4 and CAFA5 (manuscript in preparation) will have a long evaluation window that improves the robustness of the evaluation. However, within the scope of Kaggle competition, we can only evaluate on a 4-month window (between Feb 2026 and Jun 2026 for CAFA6 specifically).", "votes": null }, { "id": "3435136", "postDate": "04/03/2026 17:59:23", "content": "So, just to make sure I understand: the evaluation set is large enough, but you are still adding data from the previous public leaderboard.
\nParticipants may have overfitted on that subset, but I understand it represents only a small number of additional points and are unknown.
\nMakes sense
\nMany thanks.
", "rawMarkdown": "So, just to make sure I understand: the evaluation set is large enough, but you are still adding data from the previous public leaderboard. \n\nParticipants may have overfitted on that subset, but I understand it represents only a small number of additional points and are unknown.\n\nMakes sense\n\nMany thanks.", "votes": null }, { "id": "3440061", "postDate": "04/11/2026 20:34:11", "content": "CAFA6's data preprocessing and evaluation dataset creation codes have been made public here: https://github.com/anphan0828/democafa_package
", "rawMarkdown": "CAFA6's data preprocessing and evaluation dataset creation codes have been made public here: https://github.com/anphan0828/democafa_package", "votes": null }, { "id": "3440096", "postDate": "04/11/2026 22:38:57", "content": "Thanks for release. This is exactly the kind of validation that lets people dig into the methodology and actually understand it.
\nI've been building something structurally very similar on my side, and going through democafa next to my own code has been genuinely illuminating. A few things stood out:
\nNK/LK/PK classification per (protein, namespace) rather than per protein globally — I landed on the same thing independently, and seeing an independent reference implementation confirmed a few edge cases I'd been second-guessing (e.g. a protein being simultaneously LK in CCO and PK in BPO).
NOT-qualifier handling: I ended up propagating NOT down the GO DAG via is_a / part_of (True Path Rule contrapositive), while democafa drops only the exact (protein, term) rows. I remember this being discussed during the competition and the exact-row approach was coherent in context — just noting it here for anyone comparing implementations.
The binding-only MFO exclusion (GO:0005515), the in-package Information Accretion computation, and the test-superset construction all mirror decisions I had to make independently and spent more time on than I'd like to admit — having a reference to point at cuts that learning curve enormously for the next person.
OBO reconciliation and IA computation is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference?
When I started working with this kind of data, none of it felt intuitive. Everything is rational once you get it, but building the mental model from scratch takes a while. Having this reference open makes that path much shorter.
\nOne follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?
\nExcellent work all around, thank you once more.
", "rawMarkdown": "Thanks for release. This is exactly the kind of validation that lets people dig into the methodology and actually understand it.\n\nI've been building something structurally very similar on my side, and going through democafa next to my own code has been genuinely illuminating. A few things stood out:\n\n- **NK/LK/PK classification per (protein, namespace)** rather than per protein globally — I landed on the same thing independently, and seeing an independent reference implementation confirmed a few edge cases I'd been second-guessing (e.g. a protein being simultaneously LK in CCO and PK in BPO).\n\n- **NOT-qualifier handling**: I ended up propagating NOT down the GO DAG via `is_a` / `part_of` (True Path Rule contrapositive), while democafa drops only the exact (protein, term) rows. I remember this being discussed during the competition and the exact-row approach was coherent in context — just noting it here for anyone comparing implementations.\n\n- The **binding-only MFO exclusion (`GO:0005515`)**, the **in-package Information Accretion** computation, and the **test-superset construction** all mirror decisions I had to make independently and spent more time on than I'd like to admit — having a reference to point at cuts that learning curve enormously for the next person.\n\n- **OBO reconciliation and IA computation** is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference? \n\nWhen I started working with this kind of data, none of it felt intuitive. Everything is rational once you get it, but building the mental model from scratch takes a while. Having this reference open makes that path much shorter.\n\nOne follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?\n\nExcellent work all around, thank you once more.", "votes": null }, { "id": "3440104", "postDate": "04/11/2026 23:25:29", "content": "\n\nOBO reconciliation and IA computation is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference?
\n
In the context of CAFA, the OBO release and the IA values (computed from t-1 annotation corpus) are both frozen to the t-1 time point, meaning when the competition started in Oct 2025. The reason for this is because all models that participants have trained had to depend on this frozen version, so we use the frozen t-1 OBO and IA values throughout the evaluation period and do not recalculate them.
\nWhen the DAG structure changes (e.g., at t0 and t1), the GO terms need to follow the correct DAG at that point, so any annotations that we got at t0 and t1 were propagated using its own correct version of the DAG. After that, we intersected the propagated annotations with the t-1 GO DAG to map things back to the frozen release. Finally, we propagated the remaining annotations once more using the t-1 GO DAG to ensure connectivity. This means that a GO term that is new at t0 or t1 (that hasn't existed in t-1) will be removed, but all of its ancestors (that have existed in t-1) will be retained. See wrapper_retrieve_terms function here.
\n\nOne follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?
\n
The final evaluation ground truth will most likely include a new GAF that should become available around late May 2026. The current leaderboard is not the final evaluation.
", "rawMarkdown": ">OBO reconciliation and IA computation is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference?\n\nIn the context of CAFA, the OBO release and the IA values (computed from t-1 annotation corpus) are both frozen to the t-1 time point, meaning when the competition started in Oct 2025. The reason for this is because all models that participants have trained had to depend on this frozen version, so we use the frozen t-1 OBO and IA values throughout the evaluation period and do not recalculate them. \n\nWhen the DAG structure changes (e.g., at t0 and t1), the GO terms need to follow the correct DAG at that point, so any annotations that we got at t0 and t1 were propagated using its own correct version of the DAG. After that, we intersected the propagated annotations with the t-1 GO DAG to map things back to the frozen release. Finally, we propagated the remaining annotations once more using the t-1 GO DAG to ensure connectivity. This means that a GO term that is new at t0 or t1 (that hasn't existed in t-1) will be removed, but all of its ancestors (that have existed in t-1) will be retained. See `wrapper_retrieve_terms` function [here](https://github.com/anphan0828/democafa_package/blob/master/democafa/datacollection/retrieve_terms.py).\n\n>One follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?\n\nThe final evaluation ground truth will most likely include a new GAF that should become available around late May 2026. The current leaderboard is not the final evaluation.", "votes": null }, { "id": "3456602", "postDate": "05/12/2026 08:15:36", "content": "For final judging, will the score be computed only from the two selected final submissions, or from all submissions that were rescored during the leaderboard update? Some teams now observe that unselected submissions have higher rescored scores than selected ones, so clarification would be important.
", "rawMarkdown": "For final judging, will the score be computed only from the two selected final submissions, or from all submissions that were rescored during the leaderboard update? Some teams now observe that unselected submissions have higher rescored scores than selected ones, so clarification would be important.", "votes": null }, { "id": "3456633", "postDate": "05/12/2026 10:25:04", "content": "In my case, despite the earlier glitch where submissions were showing inconsistently, the two I had originally selected eventually came back and those are the ones counting for my\n ranking.\n Just to recap the two options participants had for final selection:
\n\n\nwas the previous evaluation set the result of a synchronized non-publication window with UniProt?
\n
The previous evaluation set is a hold-out/secret set of GO annotations for a small number of proteins that UniProt has agreed to not include in their usual knowledgebase release, for the purpose of a live leaderboard on Kaggle CAFA. This set is currently not released to the public, hence the reason it is still included in the updated evaluation set.
\n\n\nWould it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?
\n
The new evaluation set currently has many more proteins than the previous secret evaluation set, but yes the longer we wait the more robust the evaluation set will be. CAFA4 and CAFA5 (manuscript in preparation) will have a long evaluation window that improves the robustness of the evaluation. However, within the scope of Kaggle competition, we can only evaluate on a 4-month window (between Feb 2026 and Jun 2026 for CAFA6 specifically).
", "votes": null, "replies": [ { "id": 3435136, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/03/2026 17:59:23", "content": "So, just to make sure I understand: the evaluation set is large enough, but you are still adding data from the previous public leaderboard.
\nParticipants may have overfitted on that subset, but I understand it represents only a small number of additional points and are unknown.
\nMakes sense
\nMany thanks.
", "votes": null, "replies": [] }, { "id": 3456602, "author_name": "skidive", "author_url": "", "post_date": "05/12/2026 08:15:36", "content": "For final judging, will the score be computed only from the two selected final submissions, or from all submissions that were rescored during the leaderboard update? Some teams now observe that unselected submissions have higher rescored scores than selected ones, so clarification would be important.
", "votes": null, "replies": [ { "id": 3456633, "author_name": "xaxipirulazo", "author_url": "", "post_date": "05/12/2026 10:25:04", "content": "In my case, despite the earlier glitch where submissions were showing inconsistently, the two I had originally selected eventually came back and those are the ones counting for my\n ranking.\n Just to recap the two options participants had for final selection:
\nCAFA6's data preprocessing and evaluation dataset creation codes have been made public here: https://github.com/anphan0828/democafa_package
", "votes": null, "replies": [ { "id": 3440096, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/11/2026 22:38:57", "content": "Thanks for release. This is exactly the kind of validation that lets people dig into the methodology and actually understand it.
\nI've been building something structurally very similar on my side, and going through democafa next to my own code has been genuinely illuminating. A few things stood out:
\nNK/LK/PK classification per (protein, namespace) rather than per protein globally — I landed on the same thing independently, and seeing an independent reference implementation confirmed a few edge cases I'd been second-guessing (e.g. a protein being simultaneously LK in CCO and PK in BPO).
NOT-qualifier handling: I ended up propagating NOT down the GO DAG via is_a / part_of (True Path Rule contrapositive), while democafa drops only the exact (protein, term) rows. I remember this being discussed during the competition and the exact-row approach was coherent in context — just noting it here for anyone comparing implementations.
The binding-only MFO exclusion (GO:0005515), the in-package Information Accretion computation, and the test-superset construction all mirror decisions I had to make independently and spent more time on than I'd like to admit — having a reference to point at cuts that learning curve enormously for the next person.
OBO reconciliation and IA computation is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference?
When I started working with this kind of data, none of it felt intuitive. Everything is rational once you get it, but building the mental model from scratch takes a while. Having this reference open makes that path much shorter.
\nOne follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?
\nExcellent work all around, thank you once more.
", "votes": null, "replies": [ { "id": 3440104, "author_name": "ahphan", "author_url": "", "post_date": "04/11/2026 23:25:29", "content": "\n\nOBO reconciliation and IA computation is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference?
\n
In the context of CAFA, the OBO release and the IA values (computed from t-1 annotation corpus) are both frozen to the t-1 time point, meaning when the competition started in Oct 2025. The reason for this is because all models that participants have trained had to depend on this frozen version, so we use the frozen t-1 OBO and IA values throughout the evaluation period and do not recalculate them.
\nWhen the DAG structure changes (e.g., at t0 and t1), the GO terms need to follow the correct DAG at that point, so any annotations that we got at t0 and t1 were propagated using its own correct version of the DAG. After that, we intersected the propagated annotations with the t-1 GO DAG to map things back to the frozen release. Finally, we propagated the remaining annotations once more using the t-1 GO DAG to ensure connectivity. This means that a GO term that is new at t0 or t1 (that hasn't existed in t-1) will be removed, but all of its ancestors (that have existed in t-1) will be retained. See wrapper_retrieve_terms function here.
\n\nOne follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?
\n
The final evaluation ground truth will most likely include a new GAF that should become available around late May 2026. The current leaderboard is not the final evaluation.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3435094": "The current rescoring process raises several questions.\n\nIn my case, the interface shows 0/2 because I was not able to select submissions (I am not the team leader). I assumed that the two highest-scoring submissions would automatically be chosen, but there is no longer any trace of that selection. What is the rationale for evaluating all submissions instead? This appears computationally expensive, and it is not clear what benefit it provides compared to evaluating only the selected ones.\n\nWas it necessary to overwrite the previous leaderboard during this debugging phase? The existing leaderboard provided a stable reference and reflected significant effort from participants. A public/private split could have preserved continuity while rescoring was in progress. At the moment, it is difficult to track performance changes.\n\nThe explanation that there were not enough proteins with updated annotations is understandable. However, combining new evaluation data with previous dataset is difficult to interpret and could be seen as inconsistent with the competition description and with the decisions participants had to make during the competition. Personally, I would prefer a volatile score that follows a clear and consistent methodology, rather than changing the evaluation methodology mid-process.\n\nMore generally, greater transparency about how the evaluation dataset is constructed would be very helpful. Making these details public would allow participants to audit the process and potentially help improve the design of future CAFA editions. The most recent fully available evaluation dataset I could find corresponds to CAFA3.\n\nAt this point, I still have some questions. For example, was the previous evaluation set the result of a synchronized non-publication window with UniProt? Would it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?\n\nClarifying these aspects would help me better understand the rationale behind the current approach.\n\nThanks", "3435106": ">was the previous evaluation set the result of a synchronized non-publication window with UniProt?\n\nThe previous evaluation set is a hold-out/secret set of GO annotations for a small number of proteins that UniProt has agreed to not include in their usual knowledgebase release, for the purpose of a live leaderboard on Kaggle CAFA. This set is currently not released to the public, hence the reason it is still included in the updated evaluation set.\n\n>Would it have been preferable to split that dataset, or simply wait additional months to obtain a more robust evaluation set?\n\nThe new evaluation set currently has many more proteins than the previous secret evaluation set, but yes the longer we wait the more robust the evaluation set will be. CAFA4 and CAFA5 (manuscript in preparation) will have a long evaluation window that improves the robustness of the evaluation. However, within the scope of Kaggle competition, we can only evaluate on a 4-month window (between Feb 2026 and Jun 2026 for CAFA6 specifically).", "3435136": "So, just to make sure I understand: the evaluation set is large enough, but you are still adding data from the previous public leaderboard. \n\nParticipants may have overfitted on that subset, but I understand it represents only a small number of additional points and are unknown.\n\nMakes sense\n\nMany thanks.", "3440061": "CAFA6's data preprocessing and evaluation dataset creation codes have been made public here: https://github.com/anphan0828/democafa_package", "3440096": "Thanks for release. This is exactly the kind of validation that lets people dig into the methodology and actually understand it.\n\nI've been building something structurally very similar on my side, and going through democafa next to my own code has been genuinely illuminating. A few things stood out:\n\n- **NK/LK/PK classification per (protein, namespace)** rather than per protein globally — I landed on the same thing independently, and seeing an independent reference implementation confirmed a few edge cases I'd been second-guessing (e.g. a protein being simultaneously LK in CCO and PK in BPO).\n\n- **NOT-qualifier handling**: I ended up propagating NOT down the GO DAG via `is_a` / `part_of` (True Path Rule contrapositive), while democafa drops only the exact (protein, term) rows. I remember this being discussed during the competition and the exact-row approach was coherent in context — just noting it here for anyone comparing implementations.\n\n- The **binding-only MFO exclusion (`GO:0005515`)**, the **in-package Information Accretion** computation, and the **test-superset construction** all mirror decisions I had to make independently and spent more time on than I'd like to admit — having a reference to point at cuts that learning curve enormously for the next person.\n\n- **OBO reconciliation and IA computation** is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference? \n\nWhen I started working with this kind of data, none of it felt intuitive. Everything is rational once you get it, but building the mental model from scratch takes a while. Having this reference open makes that path much shorter.\n\nOne follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?\n\nExcellent work all around, thank you once more.", "3440104": ">OBO reconciliation and IA computation is the one piece I'm still genuinely unsure about on my side. When t0 and t1 sit on different OBO releases, the DAG itself shifts (new terms, obsoletions, restructured relations), and that propagates into the prior probabilities used for Information Accretion. I'd love to understand how CAFA handles this in practice — is IA computed on a single pinned OBO, on the t1 release, on some merged/union view of both, and which annotation corpus is used as the frequency reference?\n\nIn the context of CAFA, the OBO release and the IA values (computed from t-1 annotation corpus) are both frozen to the t-1 time point, meaning when the competition started in Oct 2025. The reason for this is because all models that participants have trained had to depend on this frozen version, so we use the frozen t-1 OBO and IA values throughout the evaluation period and do not recalculate them. \n\nWhen the DAG structure changes (e.g., at t0 and t1), the GO terms need to follow the correct DAG at that point, so any annotations that we got at t0 and t1 were propagated using its own correct version of the DAG. After that, we intersected the propagated annotations with the t-1 GO DAG to map things back to the frozen release. Finally, we propagated the remaining annotations once more using the t-1 GO DAG to ensure connectivity. This means that a GO term that is new at t0 or t1 (that hasn't existed in t-1) will be removed, but all of its ancestors (that have existed in t-1) will be retained. See `wrapper_retrieve_terms` function [here](https://github.com/anphan0828/democafa_package/blob/master/democafa/datacollection/retrieve_terms.py).\n\n>One follow-up question on the timeline: is the evaluation ground truth locked to the current GOA release, or will it roll forward as new GAFs arrive before the Jun 2026 close? In other words — is this effectively the final evaluation, or should we expect one or two more updates as fresher releases come out?\n\nThe final evaluation ground truth will most likely include a new GAF that should become available around late May 2026. The current leaderboard is not the final evaluation.", "3456602": "For final judging, will the score be computed only from the two selected final submissions, or from all submissions that were rescored during the leaderboard update? Some teams now observe that unselected submissions have higher rescored scores than selected ones, so clarification would be important.", "3456633": "In my case, despite the earlier glitch where submissions were showing inconsistently, the two I had originally selected eventually came back and those are the ones counting for my\n ranking.\n\n Just to recap the two options participants had for final selection:\n\n 1. Don't select anything (my case): the system picked the 2 highest-scoring on the validation set over the first test split. In my case those happened to also be my best submissions\n overall.\n 2. Manually select your 2.\n\n So if a team didn't manually pick and now sees unselected submissions with higher rescored values, those weren't part of the auto-selected pair either, the auto-selection ran on the\n validation split, not on the rescore.\n\n\n I'd assume the locked-in pair is final, but I'm not on the staff." }, "source": "meta" }