\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd8b40c008a8457d4b75c63dab5f5b96e%2FSelection_753.png?generation=1758765721533376&alt=media\")
metric code:
\nhttps://www.kaggle.com/code/metric/mabe-f-beta
### key exploitation ####\ndef single_lab_f1(): \n...\n f not active_labels:\n continue # ← ignored: no TP/FP/FN\nIt is actually a segmentation metric, because it considers frame-by-frame (instead of object-by-object)
It ignores predicted behaviour not found in the ground truth
no check for multiple actions in one frame for the same mouse ????
test code
\nlab_solution = pd.DataFrame([\n {: , : , : , : , : , : ,\n : , : , },\n])\nlab_submission = pd.DataFrame([\n {: , : , : , : , : ,\n : , : , },\n])\nscore = single_lab_f1(\n pl.from_pandas(lab_solution),\n pl.from_pandas(lab_submission),\n beta = \n)\n(score) \n### why the allows multiple action per frames ????\n\nlab_solution = pd.DataFrame([\n {: , : , : , : , : , : ,\n : , : , },\n])\nlab_submission = pd.DataFrame([\n {: , : , : , : , : ,\n : , : , },\n {: , : , : , : , : ,\n : , : , }, ##bug?\n])\nscore = single_lab_f1(\n pl.from_pandas(lab_solution),\n pl.from_pandas(lab_submission),\n beta = \n)\nprint(score) #\nThe following have the same scores:
\n
1) It means you can add rubbish prediction (yellow) to maximise your score. In fact, you should add ALL
\n2) There is no incentive to group detected frames into start and end (unless that improves segmentation)
metric code:
\nhttps://www.kaggle.com/code/metric/mabe-f-beta
### key exploitation ####\ndef single_lab_f1(): \n...\n f not active_labels:\n continue # ← ignored: no TP/FP/FN\nIt is actually a segmentation metric, because it considers frame-by-frame (instead of object-by-object)
It ignores predicted behaviour not found in the ground truth
no check for multiple actions in one frame for the same mouse ????
test code
\nlab_solution = pd.DataFrame([\n {: , : , : , : , : , : ,\n : , : , },\n])\nlab_submission = pd.DataFrame([\n {: , : , : , : , : ,\n : , : , },\n])\nscore = single_lab_f1(\n pl.from_pandas(lab_solution),\n pl.from_pandas(lab_submission),\n beta = \n)\n(score) \n### why the allows multiple action per frames ????\n\nlab_solution = pd.DataFrame([\n {: , : , : , : , : , : ,\n : , : , },\n])\nlab_submission = pd.DataFrame([\n {: , : , : , : , : ,\n : , : , },\n {: , : , : , : , : ,\n : , : , }, ##bug?\n])\nscore = single_lab_f1(\n pl.from_pandas(lab_solution),\n pl.from_pandas(lab_submission),\n beta = \n)\nprint(score) #\nThe following have the same scores:
\n
1) It means you can add rubbish prediction (yellow) to maximise your score. In fact, you should add ALL
\n2) There is no incentive to group detected frames into start and end (unless that improves segmentation)
Hello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:
\nOverfragmentation: in a sense you are right, if you label a bout of sniffing from frames 10-20 and one from 21-30, we won't treat that differently from labeling one bout from frames 10-30. Evaluation is at the level of the frame, not the level of the bout.
\nNo penalty for over detection: like Rafal notes, in a lot of our datasets the annotations of animals' actions are asymmetric- eg labs annotate what mouse 1 did, but not mouse 2. Usually there was a scientific reason for this, for example they were studying specifically the actions of mouse 1, and it would not make sense for them to spend many hours annotating mouse 2 actions that they would not need in their analysis. This means: if mouse 1 was annotated for something (say sniffing), then we know the frames with no annotations are true negatives for sniffing. But we cannot draw any conclusions from these annotations about when in the video mouse 2 was sniffing, since we have no ground truth labels provided by the labs.
\nWe could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that? We wouldn't want to penalize you for providing extra information, so we decided he fairer thing to do is to only compute your score using the labels we have access to.
", "rawMarkdown": "Hello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:\n\nOverfragmentation: in a sense you are right, if you label a bout of sniffing from frames 10-20 and one from 21-30, we won't treat that differently from labeling one bout from frames 10-30. Evaluation is at the level of the frame, not the level of the bout.\n\nNo penalty for over detection: like Rafal notes, in a lot of our datasets the annotations of animals' actions are asymmetric- eg labs annotate what mouse 1 did, but not mouse 2. Usually there was a scientific reason for this, for example they were studying specifically the actions of mouse 1, and it would not make sense for them to spend many hours annotating mouse 2 actions that they would not need in their analysis. This means: if mouse 1 was annotated for something (say sniffing), then we know the frames with no annotations are true negatives for sniffing. But we cannot draw any conclusions from these annotations about when in the video mouse 2 was sniffing, since we have no ground truth labels provided by the labs.\n\nWe could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that? We wouldn't want to penalize you for providing extra information, so we decided he fairer thing to do is to only compute your score using the labels we have access to.", "votes": 13, "replies": [ { "id": 3294860, "postDate": "2025-09-27T00:19:17.190Z", "content": "\n\nHello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:
\n
\n …
@annkennedy05 Thanks for the reply and clarification.
\nI’m good with whichever host metrics are used.
\nJust want to flag possible metric exploits now, so we don’t face changes or disruptions later on.
\n\n\n\nWe could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if >> you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of
\n
\n detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that?
Sorry, I didn’t explain it clearly earlier. This isn’t about mouse 2 — it’s about mouse 1. Let’s say the ground truth only involves mouse 1. If my model predicts that mouse 1 performs action A, I should submit only “A.” But if I instead submit actions A, B, C, D and all, then the score comes from exploiting the metric, not from genuine modeling.
\nThis can be avoided by only allowing only top1 or top3 actions for each mouse at each time.
", "rawMarkdown": "> Hello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:\n> ...\n\n@annkennedy05 Thanks for the reply and clarification. \nI’m good with whichever host metrics are used. \nJust want to flag possible metric exploits now, so we don’t face changes or disruptions later on.\n\n>> We could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if >> you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of \n>> detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that?\n\nSorry, I didn’t explain it clearly earlier. This isn’t about mouse 2 — it’s about mouse 1. Let’s say the ground truth only involves mouse 1. If my model predicts that mouse 1 performs action A, I should submit only “A.” But if I instead submit actions A, B, C, D and all, then the score comes from exploiting the metric, not from genuine modeling. \n\nThis can be avoided by only allowing only top1 or top3 actions for **each mouse at each time.**\n\n\n\n\n", "replies": [ { "id": 3294942, "postDate": "2025-09-27T07:16:37.130Z", "content": "I think you have to set 'behaviors_labeled': '[\"1,2,attack\",\"1,2,mount\"]' not only [\"1,2,attack\"], and it raise HostVisibleError('Multiple predictions for the same frame from one agent/target pair'). Original one means \"1,2,mount\" wouldn't be evaluated.
I understand. But say we only evaluate on action A and B, and you label every frame as A B C D- you still will get many false positives and mislabelings for A and B that will lower your score substantially.
", "rawMarkdown": "I understand. But say we only evaluate on action A and B, and you label every frame as A B C D- you still will get many false positives and mislabelings for A and B that will lower your score substantially.", "votes": 3 } ] } ] }, { "id": 3294008, "postDate": "2025-09-25T07:30:43.900Z", "content": "I believe this is a flaw in the scoring system. The organizers could consider the following options: either not automatically skip the predictions, or count all such conflicts as FP.
", "rawMarkdown": "I believe this is a flaw in the scoring system. The organizers could consider the following options: either not automatically skip the predictions, or count all such conflicts as FP.", "votes": 1 }, { "id": 3294554, "postDate": "2025-09-26T10:27:07.977Z", "content": "CalMS21_task2 tracks only mouse with 1 ID, but there are more mouse interactions. This is why the host don't penalize FPs for another mouse. In my opinion, the host did a good job with a metric.
", "rawMarkdown": "CalMS21_task2 tracks only mouse with 1 ID, but there are more mouse interactions. This is why the host don't penalize FPs for another mouse. In my opinion, the host did a good job with a metric. " }, { "id": 3294887, "postDate": "2025-09-27T03:29:25.753Z", "content": "Only mouse with 1 ID? @annkennedy05
", "rawMarkdown": "Only mouse with 1 ID? @annkennedy05 " }, { "id": 3294602, "postDate": "2025-09-26T11:56:31.693Z", "content": "I think the metric punishes FP (even from this same mouse) if you add their labels to behaviors_labeled in a solution.
", "rawMarkdown": "I think the metric punishes FP (even from this same mouse) if you add their labels to behaviors_labeled in a solution. " }, { "id": 3294036, "postDate": "2025-09-25T08:30:05.893Z", "content": "1) It means you can add rubbish prediction (yellow) to maximise your score. In fact, you should add ALL. Not all.
\nAdding class outside of a target list is useless.
The ground truth has class A. You predicted only B. If you only submitted your prediction, you have zero score.
\nBut if submitted B, and just add A,xxx, you will have some score
", "rawMarkdown": "The ground truth has class A. You predicted only B. If you only submitted your prediction, you have zero score.\n\nBut if submitted B, and just add A,xxx, you will have some score", "votes": -1, "replies": [ { "id": 3294045, "postDate": "2025-09-25T09:01:59.040Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3294604, "postDate": "2025-09-26T11:58:19.357Z", "content": "You are right, but adding class from outside a behaviors_labeled list is useless.
", "rawMarkdown": "You are right, but adding class from outside a behaviors_labeled list is useless." } ] } ] } ], "comments": [ { "id": 3294608, "author_name": "Ann Kennedy", "author_url": "", "post_date": "2025-09-26T12:17:16.173000", "content": "Hello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:
\nOverfragmentation: in a sense you are right, if you label a bout of sniffing from frames 10-20 and one from 21-30, we won't treat that differently from labeling one bout from frames 10-30. Evaluation is at the level of the frame, not the level of the bout.
\nNo penalty for over detection: like Rafal notes, in a lot of our datasets the annotations of animals' actions are asymmetric- eg labs annotate what mouse 1 did, but not mouse 2. Usually there was a scientific reason for this, for example they were studying specifically the actions of mouse 1, and it would not make sense for them to spend many hours annotating mouse 2 actions that they would not need in their analysis. This means: if mouse 1 was annotated for something (say sniffing), then we know the frames with no annotations are true negatives for sniffing. But we cannot draw any conclusions from these annotations about when in the video mouse 2 was sniffing, since we have no ground truth labels provided by the labs.
\nWe could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that? We wouldn't want to penalize you for providing extra information, so we decided he fairer thing to do is to only compute your score using the labels we have access to.
", "votes": 13, "replies": [ { "id": 3294860, "author_name": "hengck23", "author_url": "", "post_date": "2025-09-27T00:19:17.190000", "content": "\n\nHello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:
\n
\n …
@annkennedy05 Thanks for the reply and clarification.
\nI’m good with whichever host metrics are used.
\nJust want to flag possible metric exploits now, so we don’t face changes or disruptions later on.
\n\n\n\nWe could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if >> you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of
\n
\n detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that?
Sorry, I didn’t explain it clearly earlier. This isn’t about mouse 2 — it’s about mouse 1. Let’s say the ground truth only involves mouse 1. If my model predicts that mouse 1 performs action A, I should submit only “A.” But if I instead submit actions A, B, C, D and all, then the score comes from exploiting the metric, not from genuine modeling.
\nThis can be avoided by only allowing only top1 or top3 actions for each mouse at each time.
", "votes": 0, "replies": [ { "id": 3294942, "author_name": "kyoumonemui", "author_url": "", "post_date": "2025-09-27T07:16:37.130000", "content": "I think you have to set 'behaviors_labeled': '[\"1,2,attack\",\"1,2,mount\"]' not only [\"1,2,attack\"], and it raise HostVisibleError('Multiple predictions for the same frame from one agent/target pair'). Original one means \"1,2,mount\" wouldn't be evaluated.
I understand. But say we only evaluate on action A and B, and you label every frame as A B C D- you still will get many false positives and mislabelings for A and B that will lower your score substantially.
", "votes": 3, "replies": [] } ] } ] }, { "id": 3294008, "author_name": "Reki", "author_url": "", "post_date": "2025-09-25T07:30:43.900000", "content": "I believe this is a flaw in the scoring system. The organizers could consider the following options: either not automatically skip the predictions, or count all such conflicts as FP.
", "votes": 1, "replies": [] }, { "id": 3294554, "author_name": "Rafał Pawłowski", "author_url": "", "post_date": "2025-09-26T10:27:07.977000", "content": "CalMS21_task2 tracks only mouse with 1 ID, but there are more mouse interactions. This is why the host don't penalize FPs for another mouse. In my opinion, the host did a good job with a metric.
", "votes": 0, "replies": [] }, { "id": 3294887, "author_name": "Navneet", "author_url": "", "post_date": "2025-09-27T03:29:25.753000", "content": "Only mouse with 1 ID? @annkennedy05
", "votes": 0, "replies": [] }, { "id": 3294602, "author_name": "Rafał Pawłowski", "author_url": "", "post_date": "2025-09-26T11:56:31.693000", "content": "I think the metric punishes FP (even from this same mouse) if you add their labels to behaviors_labeled in a solution.
", "votes": 0, "replies": [] }, { "id": 3294036, "author_name": "Rafał Pawłowski", "author_url": "", "post_date": "2025-09-25T08:30:05.893000", "content": "1) It means you can add rubbish prediction (yellow) to maximise your score. In fact, you should add ALL. Not all.
\nAdding class outside of a target list is useless.
The ground truth has class A. You predicted only B. If you only submitted your prediction, you have zero score.
\nBut if submitted B, and just add A,xxx, you will have some score
", "votes": -1, "replies": [ { "id": 3294045, "author_name": "", "author_url": "", "post_date": "2025-09-25T09:01:59.040000", "content": "", "votes": 0, "replies": [] }, { "id": 3294604, "author_name": "Rafał Pawłowski", "author_url": "", "post_date": "2025-09-26T11:58:19.357000", "content": "You are right, but adding class from outside a behaviors_labeled list is useless.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3293908": "metric code:\nhttps://www.kaggle.com/code/metric/mabe-f-beta\n\n```\n\n### key exploitation ####\ndef single_lab_f1(): \n...\nif f\"{agent},{target},{action}\" not in active_labels:\n continue # ← ignored: no TP/FP/FN\n\n```\n\n- It is actually a segmentation metric, because it considers frame-by-frame (instead of object-by-object)\n\n- It ignores predicted behaviour not found in the ground truth\n\n- no check for multiple actions in one frame for the same mouse ????\n\ntest code\n```\nlab_solution = pd.DataFrame([\n\t{'label_key': ' 1_1_2_attack', 'video_id': 1, 'agent_id': 1, 'target_id': 2, 'action': 'attack', 'behaviors_labeled': '[\"1,2,attack\"]',\n\t 'start_frame': 0, 'stop_frame': 10, },\n])\nlab_submission = pd.DataFrame([\n\t{'prediction_key': ' 1_1_2_attack', 'video_id': 1, 'agent_id': 1, 'target_id': 2, 'action': 'attack',\n\t 'start_frame': 0, 'stop_frame': 10, },\n])\nscore = single_lab_f1(\n\tpl.from_pandas(lab_solution),\n\tpl.from_pandas(lab_submission),\n\tbeta = 1\n)\nprint(score) #1.0\n\n```\n\n```\n### why the code allows multiple action per frames ????\n\nlab_solution = pd.DataFrame([\n\t{'label_key': ' 1_1_2_attack', 'video_id': 1, 'agent_id': 1, 'target_id': 2, 'action': 'attack', 'behaviors_labeled': '[\"1,2,attack\"]',\n\t 'start_frame': 0, 'stop_frame': 10, },\n])\nlab_submission = pd.DataFrame([\n\t{'prediction_key': ' 1_1_2_attack', 'video_id': 1, 'agent_id': 1, 'target_id': 2, 'action': 'attack',\n\t 'start_frame': 0, 'stop_frame': 10, },\n\t{'prediction_key': ' 1_1_2_mount', 'video_id': 1, 'agent_id': 1, 'target_id': 2, 'action': 'mount',\n\t 'start_frame': 0, 'stop_frame': 100, }, ##bug?\n])\nscore = single_lab_f1(\n\tpl.from_pandas(lab_solution),\n\tpl.from_pandas(lab_submission),\n\tbeta = 1\n)\nprint(score) #1.0\n\n```\n\n\nThe following have the same scores:\n\n\n1) It means you can add rubbish prediction (yellow) to maximise your score. In fact, you should add ALL\n2) There is no incentive to group detected frames into start and end (unless that improves segmentation)\n", "3294608": "Hello hengck23 et al, thanks for your thoughts on this! A few things to unpack here:\n\nOverfragmentation: in a sense you are right, if you label a bout of sniffing from frames 10-20 and one from 21-30, we won't treat that differently from labeling one bout from frames 10-30. Evaluation is at the level of the frame, not the level of the bout.\n\nNo penalty for over detection: like Rafal notes, in a lot of our datasets the annotations of animals' actions are asymmetric- eg labs annotate what mouse 1 did, but not mouse 2. Usually there was a scientific reason for this, for example they were studying specifically the actions of mouse 1, and it would not make sense for them to spend many hours annotating mouse 2 actions that they would not need in their analysis. This means: if mouse 1 was annotated for something (say sniffing), then we know the frames with no annotations are true negatives for sniffing. But we cannot draw any conclusions from these annotations about when in the video mouse 2 was sniffing, since we have no ground truth labels provided by the labs.\n\nWe could have penalized you for predicting \"unrequested\" behaviors like mouse2 sniffing- but what if you have a very good sniff-detection model and it's actually doing a great, 100% accurate job of detecting when mouse 2 sniffs, we just don't have the labels ourselves to confirm that? We wouldn't want to penalize you for providing extra information, so we decided he fairer thing to do is to only compute your score using the labels we have access to.", "3294008": "I believe this is a flaw in the scoring system. The organizers could consider the following options: either not automatically skip the predictions, or count all such conflicts as FP.", "3294554": "CalMS21_task2 tracks only mouse with 1 ID, but there are more mouse interactions. This is why the host don't penalize FPs for another mouse. In my opinion, the host did a good job with a metric. ", "3294887": "Only mouse with 1 ID? @annkennedy05 ", "3294602": "I think the metric punishes FP (even from this same mouse) if you add their labels to behaviors_labeled in a solution. ", "3294036": "1) It means you can add rubbish prediction (yellow) to maximise your score. In fact, you should add ALL. Not all.\nAdding class outside of a target list is useless.\n" } }