Hello, I have a quick question about the F-Beta score.
\nWhen my submission is 100% the same as the solution, the score should be 1.0. But in the example, the score is only 0.5.
\nThis is the example:
\nsolution = pd.DataFrame([\n # This a correct event: \n {: , : , : , : , : },\n # This another correct event: \n {: , : , : , : , : },\n])\n# My submission has events. It a perfect .\nsubmission = pd.DataFrame([\n {: , : , : , : },\n {: , : , : , : },\n])\n# Score .\n>>> mouse_fbeta(solution, submission)\n\nThe score is low because the F1 score for the 'mount' action is 0.0.
\nThis happens because the correct prediction for 'mount' is filtered out by 'behaviros_labeled', but the 'mount' event still exists in the Ground Truth, leading to a False Negative count.
\nHere is my Question:
\nPlease explain the the design philosophy behind this specific rule.
\nThank you!
", "messages": [ { "id": "3297150", "postDate": "10/02/2025 13:55:56", "content": "Hello, I have a quick question about the F-Beta score.
\nWhen my submission is 100% the same as the solution, the score should be 1.0. But in the example, the score is only 0.5.
\nThis is the example:
\nsolution = pd.DataFrame([\n # This a correct event: \n {: , : , : , : , : },\n # This another correct event: \n {: , : , : , : , : },\n])\n# My submission has events. It a perfect .\nsubmission = pd.DataFrame([\n {: , : , : , : },\n {: , : , : , : },\n])\n# Score .\n>>> mouse_fbeta(solution, submission)\n\nThe score is low because the F1 score for the 'mount' action is 0.0.
\nThis happens because the correct prediction for 'mount' is filtered out by 'behaviros_labeled', but the 'mount' event still exists in the Ground Truth, leading to a False Negative count.
\nHere is my Question:
\nPlease explain the the design philosophy behind this specific rule.
\nThank you!
", "rawMarkdown": "Hello, I have a quick question about the F-Beta score.\n\nWhen my submission is 100% the same as the solution, the score should be 1.0. But in the example, the score is only 0.5.\n\nThis is the example:\n\n```\nsolution = pd.DataFrame([\n # This is a correct event: 'attack'\n {'video_id': 123, 'action': 'attack', 'start_frame': 0, 'stop_frame': 9, 'behaviors_labeled': '[\"1,2,attack\"]'},\n # This is another correct event: 'mount'\n {'video_id': 123, 'action': 'mount', 'start_frame': 15, 'stop_frame': 24, 'behaviors_labeled': '[\"1,2,attack\"]'},\n])\n# My submission has BOTH 'attack' and 'mount' events. It is a perfect copy.\nsubmission = pd.DataFrame([\n {'video_id': 123, 'action': 'attack', 'start_frame': 0, 'stop_frame': 9},\n {'video_id': 123, 'action': 'mount', 'start_frame': 15, 'stop_frame': 24},\n])\n# Score is 0.5.\n>>> mouse_fbeta(solution, submission)\n0.5\n```\n\nThe score is low because the F1 score for the 'mount' action is 0.0.\n\nThis happens because the correct prediction for 'mount' is filtered out by 'behaviros_labeled', but the 'mount' event still exists in the Ground Truth, leading to a False Negative count.\n\nHere is my Question:\n\nPlease explain the the design philosophy behind this specific rule.\n\n Thank you!", "votes": null }, { "id": "3297287", "postDate": "10/02/2025 19:26:49", "content": "Correcting my previous misinterpretation of your question:
\nWe use \"behaviors labeled\" because different labs/videos are annotated for different actions, and we need a way to know which actions to evaluate on. It is insufficient to just take the set of actions listed in the ground truth annotations, because there is always a possibility that a lab looked for an action but it never occurred in a particular video- in which case the correct solution would be to detect no bouts of the action.
\nWhat you show (an action being annotated in the solution, but not listed in behaviors_labeled) should never be the case.
Thank you for your reply! It seems my confusion about the example and the annotated solution stemmed from the docs within the MABe F Beta notebook.
", "rawMarkdown": "Thank you for your reply! It seems my confusion about the example and the annotated solution stemmed from the docs within the [MABe F Beta](https://www.kaggle.com/code/metric/mabe-f-beta) notebook.", "votes": null }, { "id": "3300071", "postDate": "10/09/2025 12:54:24", "content": "where agent & target on submission
", "rawMarkdown": "where agent & target on submission", "votes": null } ], "comments": [ { "id": 3297287, "author_name": "annkennedy05", "author_url": "", "post_date": "10/02/2025 19:26:49", "content": "Correcting my previous misinterpretation of your question:
\nWe use \"behaviors labeled\" because different labs/videos are annotated for different actions, and we need a way to know which actions to evaluate on. It is insufficient to just take the set of actions listed in the ground truth annotations, because there is always a possibility that a lab looked for an action but it never occurred in a particular video- in which case the correct solution would be to detect no bouts of the action.
\nWhat you show (an action being annotated in the solution, but not listed in behaviors_labeled) should never be the case.
Thank you for your reply! It seems my confusion about the example and the annotated solution stemmed from the docs within the MABe F Beta notebook.
", "votes": null, "replies": [] } ] }, { "id": 3300071, "author_name": "michaelkamal92", "author_url": "", "post_date": "10/09/2025 12:54:24", "content": "where agent & target on submission
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3297150": "Hello, I have a quick question about the F-Beta score.\n\nWhen my submission is 100% the same as the solution, the score should be 1.0. But in the example, the score is only 0.5.\n\nThis is the example:\n\n```\nsolution = pd.DataFrame([\n # This is a correct event: 'attack'\n {'video_id': 123, 'action': 'attack', 'start_frame': 0, 'stop_frame': 9, 'behaviors_labeled': '[\"1,2,attack\"]'},\n # This is another correct event: 'mount'\n {'video_id': 123, 'action': 'mount', 'start_frame': 15, 'stop_frame': 24, 'behaviors_labeled': '[\"1,2,attack\"]'},\n])\n# My submission has BOTH 'attack' and 'mount' events. It is a perfect copy.\nsubmission = pd.DataFrame([\n {'video_id': 123, 'action': 'attack', 'start_frame': 0, 'stop_frame': 9},\n {'video_id': 123, 'action': 'mount', 'start_frame': 15, 'stop_frame': 24},\n])\n# Score is 0.5.\n>>> mouse_fbeta(solution, submission)\n0.5\n```\n\nThe score is low because the F1 score for the 'mount' action is 0.0.\n\nThis happens because the correct prediction for 'mount' is filtered out by 'behaviros_labeled', but the 'mount' event still exists in the Ground Truth, leading to a False Negative count.\n\nHere is my Question:\n\nPlease explain the the design philosophy behind this specific rule.\n\n Thank you!", "3297287": "Correcting my previous misinterpretation of your question:\n\nWe use \"behaviors labeled\" because different labs/videos are annotated for different actions, and we need a way to know which actions to evaluate on. It is insufficient to just take the set of actions listed in the ground truth annotations, because there is always a possibility that a lab looked for an action but it never occurred in a particular video- in which case the correct solution would be to detect no bouts of the action.\n\nWhat you show (an action being annotated in the solution, but not listed in `behaviors_labeled`) should never be the case.", "3298317": "Thank you for your reply! It seems my confusion about the example and the annotated solution stemmed from the docs within the [MABe F Beta](https://www.kaggle.com/code/metric/mabe-f-beta) notebook.", "3300071": "where agent & target on submission" }, "source": "meta" }