{
  "id": 608413,
  "title": "Are some files corrupt?",
  "url": "/competitions/MABe-mouse-behavior-detection/discussion/608413",
  "author_name": "",
  "post_date": "2025-09-19T19:20:55.079487700Z",
  "votes": 46,
  "comment_count": 7,
  "views": 0,
  "content": "<p>The mice of this competition can be visualized:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2Fa189d3f0c2c97dbdfb91a0079ec14066%2Fsniffbody.png?generation=1758309365067959&amp;alt=media\" alt=\"\"></p>\n<p>The visualization works for all lab_ids except MABe22_keypoints and MABe22_movies. The following plot is supposed to show three mice from a MABe22_keypoints video, and it is obviously wrong:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F76399718ec91109b7f22e8db3004d28d%2Fmabe.png?generation=1758309457536823&amp;alt=media\" alt=\"\"></p>\n<p>If we plot the same points without the lines and polygons, we immediately see three clusters resembling three mice. This observation leads to the conjecture that the MABe22_keypoints and MABe22_movies files somehow got garbled when they were prepared for this competition. The coordinates are correct, but they are assigned to the wrong mice and bodyparts.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F061402b395f0af79b2aeafb7334149eb%2Fmabe-clusters.png?generation=1758309511487270&amp;alt=media\" alt=\"\"></p>\n<p>Source code is <a href=\"https://www.kaggle.com/code/ambrosm/mabe-eda-which-makes-sense\" target=\"_blank\">here</a>.</p>",
  "messages": [
    {
      "id": "3291509",
      "postDate": "09/19/2025 19:20:55",
      "content": "<p>The mice of this competition can be visualized:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2Fa189d3f0c2c97dbdfb91a0079ec14066%2Fsniffbody.png?generation=1758309365067959&amp;alt=media\" alt=\"\"></p>\n<p>The visualization works for all lab_ids except MABe22_keypoints and MABe22_movies. The following plot is supposed to show three mice from a MABe22_keypoints video, and it is obviously wrong:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F76399718ec91109b7f22e8db3004d28d%2Fmabe.png?generation=1758309457536823&amp;alt=media\" alt=\"\"></p>\n<p>If we plot the same points without the lines and polygons, we immediately see three clusters resembling three mice. This observation leads to the conjecture that the MABe22_keypoints and MABe22_movies files somehow got garbled when they were prepared for this competition. The coordinates are correct, but they are assigned to the wrong mice and bodyparts.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F061402b395f0af79b2aeafb7334149eb%2Fmabe-clusters.png?generation=1758309511487270&amp;alt=media\" alt=\"\"></p>\n<p>Source code is <a href=\"https://www.kaggle.com/code/ambrosm/mabe-eda-which-makes-sense\" target=\"_blank\">here</a>.</p>",
      "rawMarkdown": "The mice of this competition can be visualized:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2Fa189d3f0c2c97dbdfb91a0079ec14066%2Fsniffbody.png?generation=1758309365067959&alt=media)\n\nThe visualization works for all lab_ids except MABe22_keypoints and MABe22_movies. The following plot is supposed to show three mice from a MABe22_keypoints video, and it is obviously wrong:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F76399718ec91109b7f22e8db3004d28d%2Fmabe.png?generation=1758309457536823&alt=media)\n\nIf we plot the same points without the lines and polygons, we immediately see three clusters resembling three mice. This observation leads to the conjecture that the MABe22_keypoints and MABe22_movies files somehow got garbled when they were prepared for this competition. The coordinates are correct, but they are assigned to the wrong mice and bodyparts.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F061402b395f0af79b2aeafb7334149eb%2Fmabe-clusters.png?generation=1758309511487270&alt=media)\n\nSource code is [here](https://www.kaggle.com/code/ambrosm/mabe-eda-which-makes-sense).",
      "votes": null
    },
    {
      "id": "3291522",
      "postDate": "09/19/2025 20:39:27",
      "content": "<p>We'll take a look at this but will most likely not have an update until early to mid next week.</p>",
      "rawMarkdown": "We'll take a look at this but will most likely not have an update until early to mid next week.",
      "votes": null
    },
    {
      "id": "3291674",
      "postDate": "09/20/2025 04:58:21",
      "content": "<p>Very nice visualizations! You are right, mouse IDs of bodyparts in the MABe22 dataset got shifted during data conversion: it seems that the concatenated vector of body parts were given mouse IDs [1,2,3,1,2,3,1,2,3…] rather than [1,1,1,…2,2,2…3,3,3…]. We've identified the culprit and will have the files updated asap next week.</p>",
      "rawMarkdown": "Very nice visualizations! You are right, mouse IDs of bodyparts in the MABe22 dataset got shifted during data conversion: it seems that the concatenated vector of body parts were given mouse IDs [1,2,3,1,2,3,1,2,3...] rather than [1,1,1,...2,2,2...3,3,3...]. We've identified the culprit and will have the files updated asap next week.",
      "votes": null
    },
    {
      "id": "3291687",
      "postDate": "09/20/2025 05:37:58",
      "content": "<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> Thanks! Is it intended that we don't have <code>behaviors_labeled</code> and <code>train_annotation</code> directory for MABe22_keypoints and MABe22_movies? The original MABe22 dataset seems to have some annotations…</p>",
      "rawMarkdown": "annkennedy05 Thanks! Is it intended that we don't have `behaviors_labeled` and `train_annotation` directory for MABe22_keypoints and MABe22_movies? The original MABe22 dataset seems to have some annotations...",
      "votes": null
    },
    {
      "id": "3292055",
      "postDate": "09/20/2025 22:27:27",
      "content": "<p>The annotations in the original dataset are generated heuristically rather than via manual annotation (we hand-annotated a subset later but it's quite small.) You could potentially regenerate the heuristic annotations using the criteria in the paper, but these actually had a LOT of false negatives, hence the later manual annotation. The \"sequence-level\" annotations, eg time-of-day information, is included in the metadata.</p>",
      "rawMarkdown": "The annotations in the original dataset are generated heuristically rather than via manual annotation (we hand-annotated a subset later but it's quite small.) You could potentially regenerate the heuristic annotations using the criteria in the paper, but these actually had a LOT of false negatives, hence the later manual annotation. The \"sequence-level\" annotations, eg time-of-day information, is included in the metadata.",
      "votes": null
    },
    {
      "id": "3292185",
      "postDate": "09/21/2025 09:10:20",
      "content": "<p>Thanks for clarifying the missing annotations for MABe22. The articles you pointed to in the competition description are very useful!</p>\n<p>But what about video 1375833299 (PleasantMeerkat)? in train.csv, it is listed with behaviors_labeled, yet I can’t find any corresponding annotation file.</p>",
      "rawMarkdown": "Thanks for clarifying the missing annotations for MABe22. The articles you pointed to in the competition description are very useful!\n\nBut what about video 1375833299 (PleasantMeerkat)? in train.csv, it is listed with behaviors_labeled, yet I can’t find any corresponding annotation file.",
      "votes": null
    },
    {
      "id": "3293833",
      "postDate": "09/24/2025 18:25:48",
      "content": "<p>Sorry I missed this post! So an important detail here: in some videos, the labs annotated for (ie looked for) a behavior, but it never actually occurred. In this case, we still list that behavior in <code>behaviors_labeled</code> because it is safe to use the video as a source of negative training data. (In contrast, if a behavior doesn't show up in the <code>behaviors_labeled</code> column that just means that researchers never checked for it, not that it never happened.)</p>\n<p>In the case of PleasantMeerkat, the annotators looked for several behaviors, but the mice in this experiment interacted so rarely that for that particular video none of them occurred, hence the annotation file for video 1375833299 is completely empty. I'll see though if we can provide an empty placeholder file just so this doesn't trip up people's analysis code.</p>\n<p>Thank you for checking this!</p>",
      "rawMarkdown": "Sorry I missed this post! So an important detail here: in some videos, the labs annotated for (ie looked for) a behavior, but it never actually occurred. In this case, we still list that behavior in `behaviors_labeled` because it is safe to use the video as a source of negative training data. (In contrast, if a behavior doesn't show up in the `behaviors_labeled` column that just means that researchers never checked for it, not that it never happened.)\n\nIn the case of PleasantMeerkat, the annotators looked for several behaviors, but the mice in this experiment interacted so rarely that for that particular video none of them occurred, hence the annotation file for video 1375833299 is completely empty. I'll see though if we can provide an empty placeholder file just so this doesn't trip up people's analysis code.\n\nThank you for checking this!",
      "votes": null
    },
    {
      "id": "3294217",
      "postDate": "09/25/2025 15:59:23",
      "content": "<p>Ahh, good to know, thank yuo!</p>",
      "rawMarkdown": "Ahh, good to know, thank yuo!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3291522,
      "author_name": "sohier",
      "author_url": "",
      "post_date": "09/19/2025 20:39:27",
      "content": "<p>We'll take a look at this but will most likely not have an update until early to mid next week.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3291674,
      "author_name": "annkennedy05",
      "author_url": "",
      "post_date": "09/20/2025 04:58:21",
      "content": "<p>Very nice visualizations! You are right, mouse IDs of bodyparts in the MABe22 dataset got shifted during data conversion: it seems that the concatenated vector of body parts were given mouse IDs [1,2,3,1,2,3,1,2,3…] rather than [1,1,1,…2,2,2…3,3,3…]. We've identified the culprit and will have the files updated asap next week.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3291687,
          "author_name": "junkoda",
          "author_url": "",
          "post_date": "09/20/2025 05:37:58",
          "content": "<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> Thanks! Is it intended that we don't have <code>behaviors_labeled</code> and <code>train_annotation</code> directory for MABe22_keypoints and MABe22_movies? The original MABe22 dataset seems to have some annotations…</p>",
          "votes": null,
          "replies": [
            {
              "id": 3292055,
              "author_name": "annkennedy05",
              "author_url": "",
              "post_date": "09/20/2025 22:27:27",
              "content": "<p>The annotations in the original dataset are generated heuristically rather than via manual annotation (we hand-annotated a subset later but it's quite small.) You could potentially regenerate the heuristic annotations using the criteria in the paper, but these actually had a LOT of false negatives, hence the later manual annotation. The \"sequence-level\" annotations, eg time-of-day information, is included in the metadata.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3292185,
                  "author_name": "antoninadolgorukova",
                  "author_url": "",
                  "post_date": "09/21/2025 09:10:20",
                  "content": "<p>Thanks for clarifying the missing annotations for MABe22. The articles you pointed to in the competition description are very useful!</p>\n<p>But what about video 1375833299 (PleasantMeerkat)? in train.csv, it is listed with behaviors_labeled, yet I can’t find any corresponding annotation file.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3293833,
                      "author_name": "annkennedy05",
                      "author_url": "",
                      "post_date": "09/24/2025 18:25:48",
                      "content": "<p>Sorry I missed this post! So an important detail here: in some videos, the labs annotated for (ie looked for) a behavior, but it never actually occurred. In this case, we still list that behavior in <code>behaviors_labeled</code> because it is safe to use the video as a source of negative training data. (In contrast, if a behavior doesn't show up in the <code>behaviors_labeled</code> column that just means that researchers never checked for it, not that it never happened.)</p>\n<p>In the case of PleasantMeerkat, the annotators looked for several behaviors, but the mice in this experiment interacted so rarely that for that particular video none of them occurred, hence the annotation file for video 1375833299 is completely empty. I'll see though if we can provide an empty placeholder file just so this doesn't trip up people's analysis code.</p>\n<p>Thank you for checking this!</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 3294217,
                          "author_name": "antoninadolgorukova",
                          "author_url": "",
                          "post_date": "09/25/2025 15:59:23",
                          "content": "<p>Ahh, good to know, thank yuo!</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3291509": "The mice of this competition can be visualized:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2Fa189d3f0c2c97dbdfb91a0079ec14066%2Fsniffbody.png?generation=1758309365067959&alt=media)\n\nThe visualization works for all lab_ids except MABe22_keypoints and MABe22_movies. The following plot is supposed to show three mice from a MABe22_keypoints video, and it is obviously wrong:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F76399718ec91109b7f22e8db3004d28d%2Fmabe.png?generation=1758309457536823&alt=media)\n\nIf we plot the same points without the lines and polygons, we immediately see three clusters resembling three mice. This observation leads to the conjecture that the MABe22_keypoints and MABe22_movies files somehow got garbled when they were prepared for this competition. The coordinates are correct, but they are assigned to the wrong mice and bodyparts.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F061402b395f0af79b2aeafb7334149eb%2Fmabe-clusters.png?generation=1758309511487270&alt=media)\n\nSource code is [here](https://www.kaggle.com/code/ambrosm/mabe-eda-which-makes-sense).",
    "3291522": "We'll take a look at this but will most likely not have an update until early to mid next week.",
    "3291674": "Very nice visualizations! You are right, mouse IDs of bodyparts in the MABe22 dataset got shifted during data conversion: it seems that the concatenated vector of body parts were given mouse IDs [1,2,3,1,2,3,1,2,3...] rather than [1,1,1,...2,2,2...3,3,3...]. We've identified the culprit and will have the files updated asap next week.",
    "3291687": "annkennedy05 Thanks! Is it intended that we don't have `behaviors_labeled` and `train_annotation` directory for MABe22_keypoints and MABe22_movies? The original MABe22 dataset seems to have some annotations...",
    "3292055": "The annotations in the original dataset are generated heuristically rather than via manual annotation (we hand-annotated a subset later but it's quite small.) You could potentially regenerate the heuristic annotations using the criteria in the paper, but these actually had a LOT of false negatives, hence the later manual annotation. The \"sequence-level\" annotations, eg time-of-day information, is included in the metadata.",
    "3292185": "Thanks for clarifying the missing annotations for MABe22. The articles you pointed to in the competition description are very useful!\n\nBut what about video 1375833299 (PleasantMeerkat)? in train.csv, it is listed with behaviors_labeled, yet I can’t find any corresponding annotation file.",
    "3293833": "Sorry I missed this post! So an important detail here: in some videos, the labs annotated for (ie looked for) a behavior, but it never actually occurred. In this case, we still list that behavior in `behaviors_labeled` because it is safe to use the video as a source of negative training data. (In contrast, if a behavior doesn't show up in the `behaviors_labeled` column that just means that researchers never checked for it, not that it never happened.)\n\nIn the case of PleasantMeerkat, the annotators looked for several behaviors, but the mice in this experiment interacted so rarely that for that particular video none of them occurred, hence the annotation file for video 1375833299 is completely empty. I'll see though if we can provide an empty placeholder file just so this doesn't trip up people's analysis code.\n\nThank you for checking this!",
    "3294217": "Ahh, good to know, thank yuo!"
  },
  "source": "meta"
}