The competition overview states in the Evaluation section (my emphasis added):
\n\n\nThe F scores are averaged across each lab, each video, and score only the specific behaviors and mice that were annotated for a specific video.
\n
However it appears to me that the competition metric computes one F1 score per action in the solution, without regard to which video each action comes from once it's been identified as belonging to a video produced by the lab of interest:
\n () -> :\n \n (action_f1s) / (action_f1s)\nThis seems to mean that a video with a lot of actions in the solution has a greater impact on the single-lab F1 score than one with very few, and the scores are not averaged across each video.
\nCan it be confirmed if this is the intended allocation of weight within the score, and the summary on the overview therefore a subtle misstatement; or the opposite?
", "messages": [ { "id": 3298433, "postDate": "2025-10-05T14:35:08.127Z", "content": "The competition overview states in the Evaluation section (my emphasis added):
\n\n\nThe F scores are averaged across each lab, each video, and score only the specific behaviors and mice that were annotated for a specific video.
\n
However it appears to me that the competition metric computes one F1 score per action in the solution, without regard to which video each action comes from once it's been identified as belonging to a video produced by the lab of interest:
\n () -> :\n \n (action_f1s) / (action_f1s)\nThis seems to mean that a video with a lot of actions in the solution has a greater impact on the single-lab F1 score than one with very few, and the scores are not averaged across each video.
\nCan it be confirmed if this is the intended allocation of weight within the score, and the summary on the overview therefore a subtle misstatement; or the opposite?
", "rawMarkdown": "The [competition overview](https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/overview) states in the _Evaluation_ section (my emphasis added):\n\n>The F scores are averaged across each lab, **each video**, and score only the specific behaviors and mice that were annotated for a specific video.\n\nHowever it appears to me that the [competition metric](https://www.kaggle.com/code/metric/mabe-f-beta) computes one F1 score **per action** in the solution, without regard to which video each action comes from once it's been identified as belonging to a video produced by the lab of interest:\n\n```python\ndef single_lab_f1(lab_solution: pl.DataFrame, lab_submission: pl.DataFrame, beta: float = 1) -> float:\n # ...\n return sum(action_f1s) / len(action_f1s)\n```\n\nThis seems to mean that a video with a lot of actions in the solution has a greater impact on the single-lab F1 score than one with very few, and the scores are _not_ averaged across each video.\n\nCan it be confirmed if this is the intended allocation of weight within the score, and the summary on the overview therefore a subtle misstatement; or the opposite?", "votes": 2 } ], "comments": [], "raw_markdown_by_id": { "3298433": "The [competition overview](https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/overview) states in the _Evaluation_ section (my emphasis added):\n\n>The F scores are averaged across each lab, **each video**, and score only the specific behaviors and mice that were annotated for a specific video.\n\nHowever it appears to me that the [competition metric](https://www.kaggle.com/code/metric/mabe-f-beta) computes one F1 score **per action** in the solution, without regard to which video each action comes from once it's been identified as belonging to a video produced by the lab of interest:\n\n```python\ndef single_lab_f1(lab_solution: pl.DataFrame, lab_submission: pl.DataFrame, beta: float = 1) -> float:\n # ...\n return sum(action_f1s) / len(action_f1s)\n```\n\nThis seems to mean that a video with a lot of actions in the solution has a greater impact on the single-lab F1 score than one with very few, and the scores are _not_ averaged across each video.\n\nCan it be confirmed if this is the intended allocation of weight within the score, and the summary on the overview therefore a subtle misstatement; or the opposite?" } }