![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13769621%2F2a10c93cfa56e6e36ffcc96e659b755a%2FScreenshot%202023-06-30%20140952.png?generation=1688148606000433&alt=media\")
Hi everyone
\nLast few days I was exploring the metric implementation, provided by the hosts here. One of the steps of evaluation is the propagation of prediction from children to parents. The particular function that implements this steps is provided below (copied from the Github):
\n ():\n \n matrix.shape[] == :\n Exception()\n\n deepest = np.where(np.(matrix[:, order], axis=) > )[][]\n deepest.size == :\n Exception()\n\n \n order_ = np.delete(order, [(, deepest)])\n\n i order_:\n \n children = np.where(ont.dag[:, i] != )[]\n children.size > :\n cols = np.concatenate((children, [i]))\n mode == :\n matrix[:, i] = matrix[:, cols].(axis=)\n mode == :\n rows = np.where(matrix[:, i] == )[]\n rows.size:\n idx = np.ix_(rows, cols)\n matrix[rows, i] = matrix[idx].(axis=)[]\n \nHere we can observe that there are two ways to propagate - max and fill. According to the README file we can see that fill mode is actually used to evaluate our submissions. So let's pay attention to this part:
mode == :\n rows = np.where(matrix[:, i] == )[]\n rows.size:\n idx = np.ix_(rows, cols)\n matrix[rows, i] = matrix[idx].(axis=)[]\nWhat does it actually mean - we will propagate the prediction only for the parents with zero prediction. In this code matrix is the predictions of shape (n_proteins, n_terms), i is the index of parent column in this matrix, col is the list of its children indices + [i]. The propagation is made in this way: matrix[rows, i] = matrix[idx].max(axis=1)[0]. What is the suspicious part here - matrix[idx].max(axis=1)[0] is actually the constant! matrix[idx] is the matrix, after .max(axis=1) it becomes a vector, and [0] makes it a constant by taking the value as maximum of children predictions of the first protien that you can meet in the matrix. If my thoughts are correct, that means:
Can the organozers please clarify if my thought are correct? If yes, from my point of view the obvious fix is just matrix[rows, i] = matrix[idx].max(axis=1)
Hi everyone
\nLast few days I was exploring the metric implementation, provided by the hosts here. One of the steps of evaluation is the propagation of prediction from children to parents. The particular function that implements this steps is provided below (copied from the Github):
\n ():\n \n matrix.shape[] == :\n Exception()\n\n deepest = np.where(np.(matrix[:, order], axis=) > )[][]\n deepest.size == :\n Exception()\n\n \n order_ = np.delete(order, [(, deepest)])\n\n i order_:\n \n children = np.where(ont.dag[:, i] != )[]\n children.size > :\n cols = np.concatenate((children, [i]))\n mode == :\n matrix[:, i] = matrix[:, cols].(axis=)\n mode == :\n rows = np.where(matrix[:, i] == )[]\n rows.size:\n idx = np.ix_(rows, cols)\n matrix[rows, i] = matrix[idx].(axis=)[]\n \nHere we can observe that there are two ways to propagate - max and fill. According to the README file we can see that fill mode is actually used to evaluate our submissions. So let's pay attention to this part:
mode == :\n rows = np.where(matrix[:, i] == )[]\n rows.size:\n idx = np.ix_(rows, cols)\n matrix[rows, i] = matrix[idx].(axis=)[]\nWhat does it actually mean - we will propagate the prediction only for the parents with zero prediction. In this code matrix is the predictions of shape (n_proteins, n_terms), i is the index of parent column in this matrix, col is the list of its children indices + [i]. The propagation is made in this way: matrix[rows, i] = matrix[idx].max(axis=1)[0]. What is the suspicious part here - matrix[idx].max(axis=1)[0] is actually the constant! matrix[idx] is the matrix, after .max(axis=1) it becomes a vector, and [0] makes it a constant by taking the value as maximum of children predictions of the first protien that you can meet in the matrix. If my thoughts are correct, that means:
Can the organozers please clarify if my thought are correct? If yes, from my point of view the obvious fix is just matrix[rows, i] = matrix[idx].max(axis=1)
\n\nThe submission metric value depends on rows order
\n
No, LB metric is not order-sensitive.
", "rawMarkdown": ">The submission metric value depends on rows order\n\nNo, LB metric is not order-sensitive.", "votes": null }, { "id": "2324536", "postDate": "06/30/2023 17:03:36", "content": "We'll take a look at this but since we're heading into July 4th weekend I don't expect that we'll be able to get back to you until the middle of next week.
", "rawMarkdown": "We'll take a look at this but since we're heading into July 4th weekend I don't expect that we'll be able to get back to you until the middle of next week.", "votes": null }, { "id": "2324570", "postDate": "06/30/2023 17:36:26", "content": "If you mean, that shuffling the predictions doesn't affect your score, you are right. But the order of ground truth matters here. This is the part of pred_parser function:
(pred_file) f:\n line f:\n line = line.strip().split()\n line (line) > :\n p_id, term_id, prob = line[:]\n ns = ns_dict.get(term_id)\n ns gts p_id gts[ns].ids:\n i = gts[ns].ids[p_id]\n max_terms np.count_nonzero(matrix[ns][i]) <= max_terms:\n j = onts[ns].terms_dict.get(term_id)[]\n ids[ns][p_id] = i\n matrix[ns][i, j] = (matrix[ns][i, j], (prob))\nHere i is the row number in the prediction matrix. They are trying to obtain it from gts dict if the protein id p_id is there. So the order of prediction is defined by the ground truth file, which is parsed first. And I believe, if ones the organizers shuffle their targets, scores will be changed at least a little. Anyway, even if you freeze all the orders, I just follow the propagate function logic, and I suppose it is not a desired behavior.
Thanks for pointing this out.
\nI believe you are correct, this would also fill protein with max child predictions of the first row.
\nUsing the following demo as an example,
\nassuming that col 1 is the term we are doing propagation, and col 2,3,4 are col 1's children.
\nThe correct value for row1 (protein1) should be 7 and for row2 (protein2) should be 12.
\nHowever, using the provided code, both of them are 7.
\nYour solution would fix this problem.
To add more examples: ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2Fc56fd08247bc1cfeb3b3883cb1e7e91a%2Fmetric.png?generation=1688392382886578&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F6d66f303d33711db202fe24e0b2f6561%2Fmetric2.png?generation=1688392666181201&alt=media\")
The difference in scores can be quite big.
\nAs OP proposed it might depends on the order of the columns of \"Y\",
\nhere the order of Y columns is not by the number of \"1\", but different
\n(taken from Zoltan's script: \"Combine embeddings\")
This might only happen to those who do not perform parent propagation prior to the submission.
\nI just checked my propagated version results, and this correction has no effect on the evaluation scoring.
Hello everyone,
\nI want to express my gratitude for those who reported the bug and provided the fix. After investigating the issue, I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.
\nI have pushed the fix to the main branch of the CAFA-evaluator repository. The Kaggle team will now take over and recalculate the dashboard with the updated code. This process should happen as soon as possible, likely at the beginning of next week. They will communicate all the details and timeline on a pinned post.
", "rawMarkdown": "Hello everyone,\n\nI want to express my gratitude for those who reported the bug and provided the fix. After investigating the issue, I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.\n\nI have pushed the fix to the main branch of the [CAFA-evaluator repository](https://github.com/BioComputingUP/CAFA-evaluator). The Kaggle team will now take over and recalculate the dashboard with the updated code. This process should happen as soon as possible, likely at the beginning of next week. They will communicate all the details and timeline on a pinned post.", "votes": null } ], "comments": [ { "id": 2324438, "author_name": "ogurtsov", "author_url": "", "post_date": "06/30/2023 15:46:54", "content": "\n\nThe submission metric value depends on rows order
\n
No, LB metric is not order-sensitive.
", "votes": null, "replies": [ { "id": 2324570, "author_name": "btbpanda", "author_url": "", "post_date": "06/30/2023 17:36:26", "content": "If you mean, that shuffling the predictions doesn't affect your score, you are right. But the order of ground truth matters here. This is the part of pred_parser function:
(pred_file) f:\n line f:\n line = line.strip().split()\n line (line) > :\n p_id, term_id, prob = line[:]\n ns = ns_dict.get(term_id)\n ns gts p_id gts[ns].ids:\n i = gts[ns].ids[p_id]\n max_terms np.count_nonzero(matrix[ns][i]) <= max_terms:\n j = onts[ns].terms_dict.get(term_id)[]\n ids[ns][p_id] = i\n matrix[ns][i, j] = (matrix[ns][i, j], (prob))\nHere i is the row number in the prediction matrix. They are trying to obtain it from gts dict if the protein id p_id is there. So the order of prediction is defined by the ground truth file, which is parsed first. And I believe, if ones the organizers shuffle their targets, scores will be changed at least a little. Anyway, even if you freeze all the orders, I just follow the propagate function logic, and I suppose it is not a desired behavior.
We'll take a look at this but since we're heading into July 4th weekend I don't expect that we'll be able to get back to you until the middle of next week.
", "votes": null, "replies": [] }, { "id": 2324621, "author_name": "evansliu", "author_url": "", "post_date": "06/30/2023 18:14:24", "content": "Thanks for pointing this out.
\nI believe you are correct, this would also fill protein with max child predictions of the first row.
\nUsing the following demo as an example,
\nassuming that col 1 is the term we are doing propagation, and col 2,3,4 are col 1's children.
\nThe correct value for row1 (protein1) should be 7 and for row2 (protein2) should be 12.
\nHowever, using the provided code, both of them are 7.
\nYour solution would fix this problem.
To add more examples:
The difference in scores can be quite big.
\nAs OP proposed it might depends on the order of the columns of \"Y\",
\nhere the order of Y columns is not by the number of \"1\", but different
\n(taken from Zoltan's script: \"Combine embeddings\")
This might only happen to those who do not perform parent propagation prior to the submission.
\nI just checked my propagated version results, and this correction has no effect on the evaluation scoring.
Hello everyone,
\nI want to express my gratitude for those who reported the bug and provided the fix. After investigating the issue, I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.
\nI have pushed the fix to the main branch of the CAFA-evaluator repository. The Kaggle team will now take over and recalculate the dashboard with the updated code. This process should happen as soon as possible, likely at the beginning of next week. They will communicate all the details and timeline on a pinned post.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2323318": "Hi everyone\n\nLast few days I was exploring the metric implementation, provided by the hosts [here](https://github.com/BioComputingUP/CAFA-evaluator). One of the steps of evaluation is the propagation of prediction from children to parents. The particular function that implements this steps is provided below (copied from the Github):\n\n```py\ndef propagate(matrix, ont, order, mode='max'):\n \"\"\"\n Update inplace the score matrix (proteins x terms) up to the root taking the max between children and parents\n \"\"\"\n if matrix.shape[0] == 0:\n raise Exception(\"Empty matrix\")\n\n deepest = np.where(np.sum(matrix[:, order], axis=0) > 0)[0][0]\n if deepest.size == 0:\n raise Exception(\"The matrix is empty\")\n\n # Remove leaves\n order_ = np.delete(order, [range(0, deepest)])\n\n for i in order_:\n # Get direct children\n children = np.where(ont.dag[:, i] != 0)[0]\n if children.size > 0:\n cols = np.concatenate((children, [i]))\n if mode == 'max':\n matrix[:, i] = matrix[:, cols].max(axis=1)\n elif mode == 'fill':\n rows = np.where(matrix[:, i] == 0)[0]\n if rows.size:\n idx = np.ix_(rows, cols)\n matrix[rows, i] = matrix[idx].max(axis=1)[0]\n return\n\n```\n\nHere we can observe that there are two ways to propagate - `max` and `fill`. According to the `README` file we can see that `fill` mode is actually used to evaluate our submissions. So let's pay attention to this part:\n\n```py\n elif mode == 'fill':\n rows = np.where(matrix[:, i] == 0)[0]\n if rows.size:\n idx = np.ix_(rows, cols)\n matrix[rows, i] = matrix[idx].max(axis=1)[0]\n```\n\nWhat does it actually mean - we will propagate the prediction only for the parents with zero prediction. In this code `matrix` is the predictions of shape (n_proteins, n_terms), `i` is the index of parent column in this matrix, `col` is the list of its children indices + [i]. The propagation is made in this way: `matrix[rows, i] = matrix[idx].max(axis=1)[0]`. What is the suspicious part here - `matrix[idx].max(axis=1)[0]` is actually the constant! `matrix[idx]` is the matrix, after `.max(axis=1)` it becomes a vector, and `[0]` makes it a constant by taking the value as maximum of children predictions of the first protien that you can meet in the matrix. If my thoughts are correct, that means:\n\n1. Propagation value does not depend on children scores for most of the proteins\n2. The submission metric value depends on rows order\n\nCan the organozers please clarify if my thought are correct? If yes, from my point of view the obvious fix is just `matrix[rows, i] = matrix[idx].max(axis=1)`\n\n@iddux1", "2324438": ">The submission metric value depends on rows order\n\nNo, LB metric is not order-sensitive.", "2324536": "We'll take a look at this but since we're heading into July 4th weekend I don't expect that we'll be able to get back to you until the middle of next week.", "2324570": "If you mean, that shuffling the predictions doesn't affect your score, you are right. But the order of ground truth matters here. This is the part of `pred_parser` function: \n```py\n with open(pred_file) as f:\n for line in f:\n line = line.strip().split()\n if line and len(line) > 2:\n p_id, term_id, prob = line[:3]\n ns = ns_dict.get(term_id)\n if ns in gts and p_id in gts[ns].ids:\n i = gts[ns].ids[p_id]\n if max_terms is None or np.count_nonzero(matrix[ns][i]) <= max_terms:\n j = onts[ns].terms_dict.get(term_id)['index']\n ids[ns][p_id] = i\n matrix[ns][i, j] = max(matrix[ns][i, j], float(prob))\n```\n\nHere `i` is the row number in the prediction matrix. They are trying to obtain it from `gts` dict if the protein id `p_id` is there. So the order of prediction is defined by the ground truth file, which is parsed first. And I believe, if ones the organizers shuffle their targets, scores will be changed at least a little. Anyway, even if you freeze all the orders, I just follow the `propagate` function logic, and I suppose it is not a desired behavior.", "2324621": "Thanks for pointing this out.\nI believe you are correct, this would also fill protein with max child predictions of the first row.\nUsing the following demo as an example,\nassuming that col 1 is the term we are doing propagation, and col 2,3,4 are col 1's children.\nThe correct value for row1 (protein1) should be 7 and for row2 (protein2) should be 12.\nHowever, using the provided code, both of them are 7. \nYour solution would fix this problem.\n\n", "2328323": "To add more examples: \n\n\n\nThe difference in scores can be quite big. \nAs OP proposed it might depends on the order of the columns of \"Y\",\nhere the order of Y columns is not by the number of \"1\", but different\n(taken from Zoltan's script: \"Combine embeddings\")", "2328347": "This might only happen to those who do not perform parent propagation prior to the submission.\nI just checked my propagated version results, and this correction has no effect on the evaluation scoring.", "2332432": "Hello everyone,\n\nI want to express my gratitude for those who reported the bug and provided the fix. After investigating the issue, I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.\n\nI have pushed the fix to the main branch of the [CAFA-evaluator repository](https://github.com/BioComputingUP/CAFA-evaluator). The Kaggle team will now take over and recalculate the dashboard with the updated code. This process should happen as soon as possible, likely at the beginning of next week. They will communicate all the details and timeline on a pinned post." }, "source": "meta" }