{
  "id": 608621,
  "title": "Clarification on the hidden test set",
  "url": "/competitions/MABe-mouse-behavior-detection/discussion/608621",
  "author_name": "",
  "post_date": "2025-09-21T11:18:34.444166300Z",
  "votes": 30,
  "comment_count": 4,
  "views": 0,
  "content": "<p>First, thank you for organizing such an interesting competition!</p>\n<p>In the public sample test (test.csv), the video include a <code>behaviors_labeled</code> field, which (as I understand), acts as a whitelist of the actions eligible to be annotated for each (agent_id, target_id) pair. </p>\n<p>It also looks like the evaluation metric (mouse_fbeta) explicitly filters predictions using this whitelist:   <br>\n<code>active_labels: str = lab_solution.filter(pl.col('video_id') == video)['behaviors_labeled'].first()</code></p>\n<p>So, my question is wether the hidden test include both - videos and the test.csv with filled <code>behaviors_labeled</code> column? This makes a substantial difference in modeling strategy: </p>\n<ul>\n<li>If it is available, we only need to predict when the specified actions occur.</li>\n<li>If it is not, we would need to predict from the full action space</li>\n</ul>",
  "messages": [
    {
      "id": "3292227",
      "postDate": "09/21/2025 11:18:34",
      "content": "<p>First, thank you for organizing such an interesting competition!</p>\n<p>In the public sample test (test.csv), the video include a <code>behaviors_labeled</code> field, which (as I understand), acts as a whitelist of the actions eligible to be annotated for each (agent_id, target_id) pair. </p>\n<p>It also looks like the evaluation metric (mouse_fbeta) explicitly filters predictions using this whitelist:   <br>\n<code>active_labels: str = lab_solution.filter(pl.col('video_id') == video)['behaviors_labeled'].first()</code></p>\n<p>So, my question is wether the hidden test include both - videos and the test.csv with filled <code>behaviors_labeled</code> column? This makes a substantial difference in modeling strategy: </p>\n<ul>\n<li>If it is available, we only need to predict when the specified actions occur.</li>\n<li>If it is not, we would need to predict from the full action space</li>\n</ul>",
      "rawMarkdown": "First, thank you for organizing such an interesting competition!\n\nIn the public sample test (test.csv), the video include a `behaviors_labeled` field, which (as I understand), acts as a whitelist of the actions eligible to be annotated for each (agent_id, target_id) pair. \n\nIt also looks like the evaluation metric (mouse_fbeta) explicitly filters predictions using this whitelist:   \n`active_labels: str = lab_solution.filter(pl.col('video_id') == video)['behaviors_labeled'].first() `\n\nSo, my question is wether the hidden test include both - videos and the test.csv with filled `behaviors_labeled` column? This makes a substantial difference in modeling strategy: \n\n- If it is available, we only need to predict when the specified actions occur.\n- If it is not, we would need to predict from the full action space",
      "votes": null
    },
    {
      "id": "3292233",
      "postDate": "09/21/2025 11:39:28",
      "content": "<p>I have additionally 2 questions.</p>\n<p>・Are there any additional lab_ids in the test set that are not included in the training set? <br>\n・Is it correct that the CalMS21, CRIM13, and MABe22 datasets are not included in the test set?</p>",
      "rawMarkdown": "I have additionally 2 questions.\n\n・Are there any additional lab_ids in the test set that are not included in the training set? \n・Is it correct that the CalMS21, CRIM13, and MABe22 datasets are not included in the test set?",
      "votes": null
    },
    {
      "id": "3292419",
      "postDate": "09/21/2025 19:48:50",
      "content": "<p>Thanks for checking! There are no new lab_ids in the test set. CalMS21, CRIM13, and MABe22 are only in the training set, as these are all previously published datasets. </p>",
      "rawMarkdown": "Thanks for checking! There are no new lab_ids in the test set. CalMS21, CRIM13, and MABe22 are only in the training set, as these are all previously published datasets.",
      "votes": null
    },
    {
      "id": "3292429",
      "postDate": "09/21/2025 20:29:53",
      "content": "<p>Very good question- yes, the <code>behaviors_labeled</code> field is included int test.csv and specifies which actions you'll need to predict/be scored on.</p>",
      "rawMarkdown": "Very good question- yes, the `behaviors_labeled` field is included int test.csv and specifies which actions you'll need to predict/be scored on.",
      "votes": null
    },
    {
      "id": "3292440",
      "postDate": "09/21/2025 21:01:53",
      "content": "<p>Thank you for replying! This competition looks very interesting and I’m really looking forward to it!!</p>",
      "rawMarkdown": "Thank you for replying! This competition looks very interesting and I’m really looking forward to it!!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3292233,
      "author_name": "kyoumonemui",
      "author_url": "",
      "post_date": "09/21/2025 11:39:28",
      "content": "<p>I have additionally 2 questions.</p>\n<p>・Are there any additional lab_ids in the test set that are not included in the training set? <br>\n・Is it correct that the CalMS21, CRIM13, and MABe22 datasets are not included in the test set?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3292419,
          "author_name": "annkennedy05",
          "author_url": "",
          "post_date": "09/21/2025 19:48:50",
          "content": "<p>Thanks for checking! There are no new lab_ids in the test set. CalMS21, CRIM13, and MABe22 are only in the training set, as these are all previously published datasets. </p>",
          "votes": null,
          "replies": [
            {
              "id": 3292440,
              "author_name": "kyoumonemui",
              "author_url": "",
              "post_date": "09/21/2025 21:01:53",
              "content": "<p>Thank you for replying! This competition looks very interesting and I’m really looking forward to it!!</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3292429,
      "author_name": "annkennedy05",
      "author_url": "",
      "post_date": "09/21/2025 20:29:53",
      "content": "<p>Very good question- yes, the <code>behaviors_labeled</code> field is included int test.csv and specifies which actions you'll need to predict/be scored on.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3292227": "First, thank you for organizing such an interesting competition!\n\nIn the public sample test (test.csv), the video include a `behaviors_labeled` field, which (as I understand), acts as a whitelist of the actions eligible to be annotated for each (agent_id, target_id) pair. \n\nIt also looks like the evaluation metric (mouse_fbeta) explicitly filters predictions using this whitelist:   \n`active_labels: str = lab_solution.filter(pl.col('video_id') == video)['behaviors_labeled'].first() `\n\nSo, my question is wether the hidden test include both - videos and the test.csv with filled `behaviors_labeled` column? This makes a substantial difference in modeling strategy: \n\n- If it is available, we only need to predict when the specified actions occur.\n- If it is not, we would need to predict from the full action space",
    "3292233": "I have additionally 2 questions.\n\n・Are there any additional lab_ids in the test set that are not included in the training set? \n・Is it correct that the CalMS21, CRIM13, and MABe22 datasets are not included in the test set?",
    "3292419": "Thanks for checking! There are no new lab_ids in the test set. CalMS21, CRIM13, and MABe22 are only in the training set, as these are all previously published datasets.",
    "3292429": "Very good question- yes, the `behaviors_labeled` field is included int test.csv and specifies which actions you'll need to predict/be scored on.",
    "3292440": "Thank you for replying! This competition looks very interesting and I’m really looking forward to it!!"
  },
  "source": "meta"
}