Hi everyone!
\nI want to highlight the potential issue affecting the evaluation code provided in the github repo
\nSince I participated in CAFA5 I am quite familiar with the code and my goal was to check if anything changed comparing to the previous competition. As the result, I found the suspicious behavior of Graph class which I described with the examples in the kernel . Shortly, I believe there is a problem with topological sort which is required for the proper propagation of targets and predictions.
I compared the behavior with CAFA5 code and data and didn't find anything like that. Kindly ask the host to check and reply if my suggestions are correct
\n", "messages": [ { "id": "3353731", "postDate": "11/29/2025 23:14:14", "content": "Hi everyone!
\nI want to highlight the potential issue affecting the evaluation code provided in the github repo
\nSince I participated in CAFA5 I am quite familiar with the code and my goal was to check if anything changed comparing to the previous competition. As the result, I found the suspicious behavior of Graph class which I described with the examples in the kernel . Shortly, I believe there is a problem with topological sort which is required for the proper propagation of targets and predictions.
I compared the behavior with CAFA5 code and data and didn't find anything like that. Kindly ask the host to check and reply if my suggestions are correct
\n", "rawMarkdown": "Hi everyone!\n\nI want to highlight the potential issue affecting the evaluation code provided in the [github repo](https://github.com/claradepaolis/CAFA-evaluator-PK) \n\nSince I participated in CAFA5 I am quite familiar with the code and my goal was to check if anything changed comparing to the previous competition. As the result, I found the suspicious behavior of `Graph` class which I described with the examples in the [kernel](https://www.kaggle.com/code/btbpanda/metric-implementation-issue) . Shortly, I believe there is a problem with topological sort which is required for the proper propagation of targets and predictions. \n\nI compared the behavior with CAFA5 code and data and didn't find anything like that. Kindly ask the host to check and reply if my suggestions are correct\n\n@iddux1 @ahphan", "votes": null }, { "id": "3360141", "postDate": "12/01/2025 22:15:12", "content": "Hello, thanks for pointing it out. I did a quick check at the code, and I did observe that some pairs of parent-child terms are not indexed like we would expect (child terms indices should be smaller than parent terms). This is probably due to the structure of the GO DAG where a child term sometimes has a parent that are a few levels further up the DAG (but still considered direct parent due to a direct relationship), which I think led to the mix-up in the resulting topological sort (not sure, I'm checking to see if this was the cause).
\nI created a toy prediction file which only has one prediction for term GO:0006338 (at index 2116 in G.terms_list, and order 25602 in G.order), and the expected result (after propagating) should have its parent term GO:0006325 (at index 2112 in G.terms_list, and order 20878 in G.order) with the same score. During propagation, the parent term GO:0006325 at order 20878 was completely removed from order_ in the \"remove leaves\" step, leading to incorrect propagation. More details in my notebook.
The fastest way to fix this is just to comment out the \"remove leaves\" part in the function propagate and only loop through indexes in G.order, so no parent terms get removed by accident; however, this fix will slow down the evaluation code. Right now, I am figuring out a way to fix the topological sort so that all child terms get smaller indexes than their parents in G.order. It only affects a very small number of terms, and these terms might not show up in the ground truth leaderboard, so no rescoring for now until I can nail down the cause and a good solution.
Hi An Phan.
\nThanks for your reply. I checked your quick fix solution, and, unfortunately, can say it is not enough. Even if we don't remove any terms from order_, you can imagine the case:
There is a graph G with terms_list ['a', 'b', 'c']. There is a chain of terms (from parent to child) a -> b -> c,
There is a ground truth: {a: 0, b: 0, c: 1}
There is G.order, which is wrong: [0, 2, 1]. The correct order supposed to be [2, 1, 0]
Imagine, we applied your fix, and iterate through that order while propagating the ground truth:
\nWe take a as a parent and assign it with the max children value, which is 0 comes from b. The result will be {a: 0, b: 0, c: 1}.
We take c as a parent, but there is no children to propagate, so we keep the same result: {a: 0, b: 0, c: 1}
We take 'b' as a parent and assign it with the max children value, which is 1 comes from c. The result will be {a: 0, b: 1, c: 1}
After the final iteration, you can see that result is different from expected which is {a: 1, b: 1, c: 1}
The most simple, however, a bit inefficient solution that works here (only for the given Graph, not general case) is to apply your fix + add propagation twice, so just replace line 152 CAFA-evaluator-PK/src/cafaeval/parser.py together with comment remove leaves part
with
\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode=)\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode=)\nlogging.debug(.(ns, matrix))\nTo compensate the inefficiency, I suggest you to replace your propagate function with something more efficient, like I did in my CAFA5 code. To be honest, many things could be done more efficient in that code, but let's focus on propagation now😀
\n numba njit, prange\n\n\n ():\n i prange(mat.shape[]):\n mat[i, k] == :\n \n\n j adj:\n mat[i, j] == :\n mat[i, k] = \n \n\n \n\n\n ():\n i prange(mat.shape[]):\n j adj:\n mat[i, j] > mat[i, k]:\n mat[i, k] = mat[i, j] \n\n \n\n ():\n\n fn = prop_max_cpu_binary binary prop_max_cpu\n\n f G.order:\n adj = G.terms_list[f][]\n\n (adj) == :\n \n\n fn(mat, f, np.asarray(adj))\n\n \nand then replace the target propagation at line 152 CAFA-evaluator-PK/src/cafaeval/parser.py with
propagate_max(matrix, ontologies[ns], binary=) \npropagate_max(matrix, ontologies[ns], binary=) \nlogging.debug(.(ns, matrix))\nand replace prediction propagation at line 223 CAFA-evaluator-PK/src/cafaeval/parser.py with
propagate_max(matrix[ns], ontologies[ns], binary=) \npropagate_max(matrix[ns], ontologies[ns], binary=) \nlogging.debug(.(ns, matrix))\nBut of course, the most proper way for the research purposes is to fix .top_sort method of Graph
The problem occurred for 4 parent-child pairs because these child terms have both \"is_a\" and \"part_of\" relationship with their parent, leading to duplicated children in the children list for the parent term. During topological sort, the parent term reached in_degree of 0 earlier than expected, resulting in it being added to order before exhausting all of its children (and thus smaller index than its children).
Example: GO:1990700 has both \"is_a\" and \"part_of\" with parent GO:0006325
\n[Term]\n: GO:\nname: nucleolar chromatin organization\nnamespace: biological_process\n: [PMID:]\nsynonym: EXACT []\nsynonym: EXACT []\nis_a: GO:0006325 ! chromatin organization\nrelationship: part_of GO:0006325 ! chromatin organization\nrelationship: part_of GO:0007000 ! nucleolus organization\nI have tested a temporary fix for the \"Graph\" class, as detailed in my notebook. The children and adj list of each term needs to be de-duplicated after constructing adjacency matrix, before topological sorting. With the topological sort fixed, the remaining propagation process should behave correctly without modifying. We will keep testing this and eventually update the cafaevaluator code to handle this bug.
Interesting edge case, nice find! I encourage PRs to the repo for any issues found (and fixed).
", "rawMarkdown": "Interesting edge case, nice find! I encourage PRs to the repo for any issues found (and fixed).", "votes": null }, { "id": "3362938", "postDate": "12/05/2025 16:07:03", "content": "The fix to remove duplicate adjacent nodes has been integrated into the github repo. Thanks for catching this, @btbpanda!
", "rawMarkdown": "The fix to remove duplicate adjacent nodes has been integrated into the github repo. Thanks for catching this, @btbpanda!", "votes": null } ], "comments": [ { "id": 3360141, "author_name": "ahphan", "author_url": "", "post_date": "12/01/2025 22:15:12", "content": "Hello, thanks for pointing it out. I did a quick check at the code, and I did observe that some pairs of parent-child terms are not indexed like we would expect (child terms indices should be smaller than parent terms). This is probably due to the structure of the GO DAG where a child term sometimes has a parent that are a few levels further up the DAG (but still considered direct parent due to a direct relationship), which I think led to the mix-up in the resulting topological sort (not sure, I'm checking to see if this was the cause).
\nI created a toy prediction file which only has one prediction for term GO:0006338 (at index 2116 in G.terms_list, and order 25602 in G.order), and the expected result (after propagating) should have its parent term GO:0006325 (at index 2112 in G.terms_list, and order 20878 in G.order) with the same score. During propagation, the parent term GO:0006325 at order 20878 was completely removed from order_ in the \"remove leaves\" step, leading to incorrect propagation. More details in my notebook.
The fastest way to fix this is just to comment out the \"remove leaves\" part in the function propagate and only loop through indexes in G.order, so no parent terms get removed by accident; however, this fix will slow down the evaluation code. Right now, I am figuring out a way to fix the topological sort so that all child terms get smaller indexes than their parents in G.order. It only affects a very small number of terms, and these terms might not show up in the ground truth leaderboard, so no rescoring for now until I can nail down the cause and a good solution.
Hi An Phan.
\nThanks for your reply. I checked your quick fix solution, and, unfortunately, can say it is not enough. Even if we don't remove any terms from order_, you can imagine the case:
There is a graph G with terms_list ['a', 'b', 'c']. There is a chain of terms (from parent to child) a -> b -> c,
There is a ground truth: {a: 0, b: 0, c: 1}
There is G.order, which is wrong: [0, 2, 1]. The correct order supposed to be [2, 1, 0]
Imagine, we applied your fix, and iterate through that order while propagating the ground truth:
\nWe take a as a parent and assign it with the max children value, which is 0 comes from b. The result will be {a: 0, b: 0, c: 1}.
We take c as a parent, but there is no children to propagate, so we keep the same result: {a: 0, b: 0, c: 1}
We take 'b' as a parent and assign it with the max children value, which is 1 comes from c. The result will be {a: 0, b: 1, c: 1}
After the final iteration, you can see that result is different from expected which is {a: 1, b: 1, c: 1}
The most simple, however, a bit inefficient solution that works here (only for the given Graph, not general case) is to apply your fix + add propagation twice, so just replace line 152 CAFA-evaluator-PK/src/cafaeval/parser.py together with comment remove leaves part
with
\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode=)\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode=)\nlogging.debug(.(ns, matrix))\nTo compensate the inefficiency, I suggest you to replace your propagate function with something more efficient, like I did in my CAFA5 code. To be honest, many things could be done more efficient in that code, but let's focus on propagation now😀
\n numba njit, prange\n\n\n ():\n i prange(mat.shape[]):\n mat[i, k] == :\n \n\n j adj:\n mat[i, j] == :\n mat[i, k] = \n \n\n \n\n\n ():\n i prange(mat.shape[]):\n j adj:\n mat[i, j] > mat[i, k]:\n mat[i, k] = mat[i, j] \n\n \n\n ():\n\n fn = prop_max_cpu_binary binary prop_max_cpu\n\n f G.order:\n adj = G.terms_list[f][]\n\n (adj) == :\n \n\n fn(mat, f, np.asarray(adj))\n\n \nand then replace the target propagation at line 152 CAFA-evaluator-PK/src/cafaeval/parser.py with
propagate_max(matrix, ontologies[ns], binary=) \npropagate_max(matrix, ontologies[ns], binary=) \nlogging.debug(.(ns, matrix))\nand replace prediction propagation at line 223 CAFA-evaluator-PK/src/cafaeval/parser.py with
propagate_max(matrix[ns], ontologies[ns], binary=) \npropagate_max(matrix[ns], ontologies[ns], binary=) \nlogging.debug(.(ns, matrix))\nBut of course, the most proper way for the research purposes is to fix .top_sort method of Graph
The problem occurred for 4 parent-child pairs because these child terms have both \"is_a\" and \"part_of\" relationship with their parent, leading to duplicated children in the children list for the parent term. During topological sort, the parent term reached in_degree of 0 earlier than expected, resulting in it being added to order before exhausting all of its children (and thus smaller index than its children).
Example: GO:1990700 has both \"is_a\" and \"part_of\" with parent GO:0006325
\n[Term]\n: GO:\nname: nucleolar chromatin organization\nnamespace: biological_process\n: [PMID:]\nsynonym: EXACT []\nsynonym: EXACT []\nis_a: GO:0006325 ! chromatin organization\nrelationship: part_of GO:0006325 ! chromatin organization\nrelationship: part_of GO:0007000 ! nucleolus organization\nI have tested a temporary fix for the \"Graph\" class, as detailed in my notebook. The children and adj list of each term needs to be de-duplicated after constructing adjacency matrix, before topological sorting. With the topological sort fixed, the remaining propagation process should behave correctly without modifying. We will keep testing this and eventually update the cafaevaluator code to handle this bug.
Interesting edge case, nice find! I encourage PRs to the repo for any issues found (and fixed).
", "votes": null, "replies": [] } ] }, { "id": 3362938, "author_name": "claradepaolis", "author_url": "", "post_date": "12/05/2025 16:07:03", "content": "The fix to remove duplicate adjacent nodes has been integrated into the github repo. Thanks for catching this, @btbpanda!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3353731": "Hi everyone!\n\nI want to highlight the potential issue affecting the evaluation code provided in the [github repo](https://github.com/claradepaolis/CAFA-evaluator-PK) \n\nSince I participated in CAFA5 I am quite familiar with the code and my goal was to check if anything changed comparing to the previous competition. As the result, I found the suspicious behavior of `Graph` class which I described with the examples in the [kernel](https://www.kaggle.com/code/btbpanda/metric-implementation-issue) . Shortly, I believe there is a problem with topological sort which is required for the proper propagation of targets and predictions. \n\nI compared the behavior with CAFA5 code and data and didn't find anything like that. Kindly ask the host to check and reply if my suggestions are correct\n\n@iddux1 @ahphan", "3360141": "Hello, thanks for pointing it out. I did a quick check at the code, and I did observe that some pairs of parent-child terms are not indexed like we would expect (child terms indices should be smaller than parent terms). This is probably due to the structure of the GO DAG where a child term sometimes has a parent that are a few levels further up the DAG (but still considered direct parent due to a direct relationship), which I think led to the mix-up in the resulting topological sort (not sure, I'm checking to see if this was the cause).\n\nI created a toy prediction file which only has one prediction for term `GO:0006338` (at index 2116 in `G.terms_list`, and order 25602 in `G.order`), and the expected result (after propagating) should have its parent term `GO:0006325` (at index 2112 in `G.terms_list`, and order 20878 in `G.order`) with the same score. During propagation, the parent term `GO:0006325` at order 20878 was completely removed from `order_` in the \"remove leaves\" step, leading to incorrect propagation. More details in [my notebook](https://www.kaggle.com/code/ahphan/metric-implementation-issue-discussion).\n\nThe fastest way to fix this is just to comment out the \"remove leaves\" part in the function `propagate` and only loop through indexes in `G.order`, so no parent terms get removed by accident; however, this fix will slow down the evaluation code. Right now, I am figuring out a way to fix the topological sort so that all child terms get smaller indexes than their parents in `G.order`. It only affects a very small number of terms, and these terms might not show up in the ground truth leaderboard, so no rescoring for now until I can nail down the cause and a good solution.", "3360501": "Hi An Phan. \n\nThanks for your reply. I checked your quick fix solution, and, unfortunately, can say it is not enough. Even if we don't remove any terms from `order_`, you can imagine the case:\n\nThere is a graph `G` with `terms_list` `['a', 'b', 'c']`. There is a chain of terms (from parent to child) `a -> b -> c`, \n\nThere is a ground truth: `{a: 0, b: 0, c: 1}`\n\nThere is G.order, which is wrong: `[0, 2, 1]`. The correct order supposed to be `[2, 1, 0]`\n\nImagine, we applied your fix, and iterate through that order while propagating the ground truth:\n\n### Iteration 0:\n\nWe take `a` as a parent and assign it with the max children value, which is 0 comes from `b`. The result will be `{a: 0, b: 0, c: 1}`. \n\n### Iteration 1:\n\nWe take `c` as a parent, but there is no children to propagate, so we keep the same result: `{a: 0, b: 0, c: 1}`\n\n### Iteration 2:\n\nWe take 'b' as a parent and assign it with the max children value, which is 1 comes from `c`. The result will be `{a: 0, b: 1, c: 1}`\n\nAfter the final iteration, you can see that result is different from expected which is `{a: 1, b: 1, c: 1}`\n\n# Hot fix \n\nThe most simple, however, a bit inefficient solution that works here (only for the given Graph, not general case) is to apply your fix + add propagation twice, so just replace line 152 `CAFA-evaluator-PK/src/cafaeval/parser.py` together with comment remove leaves part\n\nwith \n\n```python\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode='max')\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode='max')\nlogging.debug(\"gt matrix propagated {} {} \".format(ns, matrix))\n```\n\nTo compensate the inefficiency, I suggest you to replace your propagate function with something more efficient, like I did in my CAFA5 code. To be honest, many things could be done more efficient in that code, but let's focus on propagation now😀\n\n```python\nfrom numba import njit, prange\n\n@njit\ndef prop_max_cpu_binary(mat, k, adj):\n for i in prange(mat.shape[0]):\n if mat[i, k] == 1:\n continue\n\n for j in adj:\n if mat[i, j] == 1:\n mat[i, k] = 1\n break\n\n return\n\n@njit\ndef prop_max_cpu(mat, k, adj):\n for i in prange(mat.shape[0]):\n for j in adj:\n if mat[i, j] > mat[i, k]:\n mat[i, k] = mat[i, j] \n\n return\n\ndef propagate_max(mat, G, binary=False):\n\n fn = prop_max_cpu_binary if binary else prop_max_cpu\n\n for f in G.order:\n adj = G.terms_list[f]['children']\n\n if len(adj) == 0:\n continue\n\n fn(mat, f, np.asarray(adj))\n\n return\n```\n\nand then replace the target propagation at line 152 `CAFA-evaluator-PK/src/cafaeval/parser.py` with \n\n\n ```python\n\npropagate_max(matrix, ontologies[ns], binary=True) # assumed 'mode' is always 'max'\npropagate_max(matrix, ontologies[ns], binary=True) # assumed 'mode' is always 'max'\nlogging.debug(\"gt matrix propagated {} {} \".format(ns, matrix))\n\n```\nand replace prediction propagation at line 223 `CAFA-evaluator-PK/src/cafaeval/parser.py` with\n\n```python\n\npropagate_max(matrix[ns], ontologies[ns], binary=False) # assumed 'mode' is always 'max'\npropagate_max(matrix[ns], ontologies[ns], binary=False) # assumed 'mode' is always 'max'\nlogging.debug(\"gt matrix propagated {} {} \".format(ns, matrix))\n\n```\nBut of course, the most proper way for the research purposes is to fix `.top_sort` method of `Graph`", "3360961": "The problem occurred for 4 parent-child pairs because these child terms have both \"is_a\" and \"part_of\" relationship with their parent, leading to duplicated children in the `children` list for the parent term. During topological sort, the parent term reached `in_degree` of 0 earlier than expected, resulting in it being added to `order` before exhausting all of its children (and thus smaller index than its children). \n\nExample: GO:1990700 has both \"is_a\" and \"part_of\" with parent GO:0006325\n\n\n```python\n[Term]\nid: GO:1990700\nname: nucleolar chromatin organization\nnamespace: biological_process\ndef: \"Any process that results in the specification, formation or maintenance of the physical structure of nucleolar chromatin.\" [PMID:18362178]\nsynonym: \"establishment or maintenance of nucleolar chromatin architecture\" EXACT []\nsynonym: \"nucleolar chromatin organisation\" EXACT []\nis_a: GO:0006325 ! chromatin organization\nrelationship: part_of GO:0006325 ! chromatin organization\nrelationship: part_of GO:0007000 ! nucleolus organization\n```\nI have tested a temporary fix for the \"Graph\" class, as detailed in [my notebook](https://www.kaggle.com/code/ahphan/metric-implementation-issue-discussion). The `children` and `adj` list of each term needs to be de-duplicated after constructing adjacency matrix, before topological sorting. With the topological sort fixed, the remaining propagation process should behave correctly without modifying. We will keep testing this and eventually update the cafaevaluator code to handle this bug.", "3360989": "Interesting edge case, nice find! I encourage PRs to the repo for any issues found (and fixed).", "3362938": "The fix to remove duplicate adjacent nodes has been integrated into the github repo. Thanks for catching this, @btbpanda!" }, "source": "meta" }