{
  "id": 612531,
  "title": "Erroneous annotations. Case study: AdaptableSnail/1212811043 sample",
  "url": "/competitions/MABe-mouse-behavior-detection/discussion/612531",
  "author_name": "",
  "post_date": "2025-10-20T11:22:23.946844900Z",
  "votes": 28,
  "comment_count": 18,
  "views": 0,
  "content": "<p>I wrote a short script to visualize the trackers and annotations for a given lab/video. So far I've only tested AdaptableSnail (video_id: 1212811043) and it looks like the annotations are erroneous.</p>\n<p>The annotations form a title as comma-separated actions:</p>\n<pre><code>\n</code></pre>\n<p>Original annotations:</p>\n<table>\n<thead>\n<tr>\n<th>agent_id</th>\n<th>target_id</th>\n<th>action</th>\n<th>start_frame</th>\n<th>stop_frame</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>1</td>\n<td>3</td>\n<td>chase</td>\n<td>2</td>\n</tr>\n<tr>\n<td>1</td>\n<td>1</td>\n<td>3</td>\n<td>chase</td>\n<td>128</td>\n</tr>\n<tr>\n<td>2</td>\n<td>3</td>\n<td>2</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>3</td>\n<td>3</td>\n<td>1</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>4</td>\n<td>1</td>\n<td>2</td>\n<td>chase</td>\n<td>942</td>\n</tr>\n<tr>\n<td>5</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1205</td>\n</tr>\n<tr>\n<td>6</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1313</td>\n</tr>\n<tr>\n<td>7</td>\n<td>1</td>\n<td>2</td>\n<td>chase</td>\n<td>1416</td>\n</tr>\n<tr>\n<td>8</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1819</td>\n</tr>\n<tr>\n<td>9</td>\n<td>4</td>\n<td>2</td>\n<td>chase</td>\n<td>2041</td>\n</tr>\n</tbody>\n</table>\n<p>New annotations:</p>\n<table>\n<thead>\n<tr>\n<th>agent_id</th>\n<th>target_id</th>\n<th>action</th>\n<th>start_frame</th>\n<th>stop_frame</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>3</td>\n<td>2</td>\n<td>chase</td>\n<td>2</td>\n</tr>\n<tr>\n<td>1</td>\n<td>3</td>\n<td>2</td>\n<td>chase</td>\n<td>128</td>\n</tr>\n<tr>\n<td>2</td>\n<td>2</td>\n<td>3</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>3</td>\n<td>2</td>\n<td>1</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>4</td>\n<td>4</td>\n<td>2</td>\n<td>chase</td>\n<td>942</td>\n</tr>\n<tr>\n<td>5</td>\n<td>3</td>\n<td>1</td>\n<td>chase</td>\n<td>1205</td>\n</tr>\n<tr>\n<td>6</td>\n<td>3</td>\n<td>1</td>\n<td>chase</td>\n<td>1313</td>\n</tr>\n<tr>\n<td>7</td>\n<td>3</td>\n<td>1</td>\n<td>chase</td>\n<td>1416</td>\n</tr>\n<tr>\n<td>8</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1819</td>\n</tr>\n<tr>\n<td>9</td>\n<td>4</td>\n<td>3</td>\n<td>chase (chaseattack?)</td>\n<td>2041</td>\n</tr>\n</tbody>\n</table>\n<p>It doesn't look like a simple labelling error, unless it was done on the basis of some specific bodypart trackers, which sometimes get swapped.</p>\n<p>I've attached a sample MP4 showing the first 2200 frames.</p>",
  "messages": [
    {
      "id": "3304353",
      "postDate": "10/20/2025 11:22:23",
      "content": "<p>I wrote a short script to visualize the trackers and annotations for a given lab/video. So far I've only tested AdaptableSnail (video_id: 1212811043) and it looks like the annotations are erroneous.</p>\n<p>The annotations form a title as comma-separated actions:</p>\n<pre><code>\n</code></pre>\n<p>Original annotations:</p>\n<table>\n<thead>\n<tr>\n<th>agent_id</th>\n<th>target_id</th>\n<th>action</th>\n<th>start_frame</th>\n<th>stop_frame</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>1</td>\n<td>3</td>\n<td>chase</td>\n<td>2</td>\n</tr>\n<tr>\n<td>1</td>\n<td>1</td>\n<td>3</td>\n<td>chase</td>\n<td>128</td>\n</tr>\n<tr>\n<td>2</td>\n<td>3</td>\n<td>2</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>3</td>\n<td>3</td>\n<td>1</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>4</td>\n<td>1</td>\n<td>2</td>\n<td>chase</td>\n<td>942</td>\n</tr>\n<tr>\n<td>5</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1205</td>\n</tr>\n<tr>\n<td>6</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1313</td>\n</tr>\n<tr>\n<td>7</td>\n<td>1</td>\n<td>2</td>\n<td>chase</td>\n<td>1416</td>\n</tr>\n<tr>\n<td>8</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1819</td>\n</tr>\n<tr>\n<td>9</td>\n<td>4</td>\n<td>2</td>\n<td>chase</td>\n<td>2041</td>\n</tr>\n</tbody>\n</table>\n<p>New annotations:</p>\n<table>\n<thead>\n<tr>\n<th>agent_id</th>\n<th>target_id</th>\n<th>action</th>\n<th>start_frame</th>\n<th>stop_frame</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>3</td>\n<td>2</td>\n<td>chase</td>\n<td>2</td>\n</tr>\n<tr>\n<td>1</td>\n<td>3</td>\n<td>2</td>\n<td>chase</td>\n<td>128</td>\n</tr>\n<tr>\n<td>2</td>\n<td>2</td>\n<td>3</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>3</td>\n<td>2</td>\n<td>1</td>\n<td>avoid</td>\n<td>324</td>\n</tr>\n<tr>\n<td>4</td>\n<td>4</td>\n<td>2</td>\n<td>chase</td>\n<td>942</td>\n</tr>\n<tr>\n<td>5</td>\n<td>3</td>\n<td>1</td>\n<td>chase</td>\n<td>1205</td>\n</tr>\n<tr>\n<td>6</td>\n<td>3</td>\n<td>1</td>\n<td>chase</td>\n<td>1313</td>\n</tr>\n<tr>\n<td>7</td>\n<td>3</td>\n<td>1</td>\n<td>chase</td>\n<td>1416</td>\n</tr>\n<tr>\n<td>8</td>\n<td>4</td>\n<td>3</td>\n<td>chase</td>\n<td>1819</td>\n</tr>\n<tr>\n<td>9</td>\n<td>4</td>\n<td>3</td>\n<td>chase (chaseattack?)</td>\n<td>2041</td>\n</tr>\n</tbody>\n</table>\n<p>It doesn't look like a simple labelling error, unless it was done on the basis of some specific bodypart trackers, which sometimes get swapped.</p>\n<p>I've attached a sample MP4 showing the first 2200 frames.</p>",
      "rawMarkdown": "I wrote a short script to visualize the trackers and annotations for a given lab/video. So far I've only tested AdaptableSnail (video_id: 1212811043) and it looks like the annotations are erroneous.\n\nThe annotations form a title as comma-separated actions:\n```python\nf\"{agent_id} {action} {target_id}\"\n```\n\nOriginal annotations:\n| agent_id | target_id | action | start_frame | stop_frame | \n| --- | --- | --- | --- | --- |\n| 0 | 1 | 3 | chase | 2 | 54 |\n| 1 | 1 | 3 | chase | 128 | 234 |\n| 2 | 3 | 2 | avoid | 324 | 342 |\n| 3 | 3 | 1 | avoid | 324 | 342 |\n| 4 | 1 | 2 | chase | 942 | 1052 |\n| 5 | 4 | 3 | chase | 1205 | 1276 |\n| 6 | 4 | 3 | chase | 1313 | 1370 |\n| 7 | 1 | 2 | chase | 1416 | 1458 |\n| 8 | 4 | 3 | chase | 1819 | 1908 |\n| 9 | 4 | 2 | chase | 2041 | 2101 |\n\nNew annotations:\n| agent_id | target_id | action | start_frame | stop_frame | \n| --- | --- | --- | --- | --- |\n| 0 | 3 | 2 | chase | 2 | 54 |\n| 1 | 3 | 2 | chase | 128 | 234 |\n| 2 | 2 | 3 | avoid | 324 | 342 |\n| 3 | 2 | 1 | avoid | 324 | 342 |\n| 4 | 4 | 2 | chase | 942 | 1052 |\n| 5 | 3 | 1 | chase | 1205 | 1276 |\n| 6 | 3 | 1 | chase | 1313 | 1370 |\n| 7 | 3 | 1 | chase | 1416 | 1458 |\n| 8 | 4 | 3 | chase | 1819 | 1908 |\n| 9 | 4 | 3 | chase (chaseattack?) | 2041 | 2101 |\n\nIt doesn't look like a simple labelling error, unless it was done on the basis of some specific bodypart trackers, which sometimes get swapped.\n\nI've attached a sample MP4 showing the first 2200 frames.",
      "votes": null
    },
    {
      "id": "3304407",
      "postDate": "10/20/2025 13:30:37",
      "content": "<p>I checked a lot of files, this issue seems to be an anomaly. I can't spot a pattern in the labeling, so it's likely just a human mistake.</p>",
      "rawMarkdown": "I checked a lot of files, this issue seems to be an anomaly. I can't spot a pattern in the labeling, so it's likely just a human mistake.",
      "votes": null
    },
    {
      "id": "3304416",
      "postDate": "10/20/2025 13:45:38",
      "content": "<p>Have similar issues been found in other video IDs of AdaptableSnail?</p>",
      "rawMarkdown": "Have similar issues been found in other video IDs of AdaptableSnail?",
      "votes": null
    },
    {
      "id": "3304493",
      "postDate": "10/20/2025 16:15:38",
      "content": "<p>Yet to be verified. </p>",
      "rawMarkdown": "Yet to be verified.",
      "votes": null
    },
    {
      "id": "3305561",
      "postDate": "10/23/2025 01:28:13",
      "content": "<p>This makes sense; my CV For \"AdaptableSnail\" is only 0.19, and for that lab it is bad for all 7 possible actions (0.03 to 0.28). This is compared of average CV of all other labs of around 0.53. So i suspect many/most/all agent/target/action labels are incorrect for \"AdaptableSnail\" lab. Perhaps competition hosts can correct them in both train and test data. I hope <a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> can answer this question.</p>",
      "rawMarkdown": "This makes sense; my CV For \"AdaptableSnail\" is only 0.19, and for that lab it is bad for all 7 possible actions (0.03 to 0.28). This is compared of average CV of all other labs of around 0.53. So i suspect many/most/all agent/target/action labels are incorrect for \"AdaptableSnail\" lab. Perhaps competition hosts can correct them in both train and test data. I hope @annkennedy05 can answer this question.",
      "votes": null
    },
    {
      "id": "3305806",
      "postDate": "10/23/2025 13:28:44",
      "content": "<p>From what I see many trackers/annotations are often choppy and sparse too. Some labels are very precise, while others can be missing altogether.</p>\n<p>I have to sit down and do a round up of the tracker videos but <em>other stuff</em> gets a higher priority. TL;DR From what I see the erroneous labels were observed only in the AdaptableSnail/1212811043 video.</p>",
      "rawMarkdown": "From what I see many trackers/annotations are often choppy and sparse too. Some labels are very precise, while others can be missing altogether.\n\nI have to sit down and do a round up of the tracker videos but *other stuff* gets a higher priority. TL;DR From what I see the erroneous labels were observed only in the AdaptableSnail/1212811043 video.",
      "votes": null
    },
    {
      "id": "3305820",
      "postDate": "10/23/2025 13:56:24",
      "content": "<p>Looks like I should’ve treated the annotator to some coffee.</p>",
      "rawMarkdown": "Looks like I should’ve treated the annotator to some coffee.",
      "votes": null
    },
    {
      "id": "3305943",
      "postDate": "10/23/2025 18:45:00",
      "content": "<p>You are correct, animal identities were permuted in this video, and your ID-swaps correct for this. In investigating this I also found that this and 4 other videos (one in test and 4 in train) from this dataset have another issue. These 5 videos were all acquired at 25 fps (the rest are at 30), but the annotations--and not the pose--have been rescaled to be at 30fps. If you multiply all action start and stop frames by 25/30 you will see much better synchronization between annotations and behavior in these videos.</p>\n<p>I am double-checking this now and also running a test to look for other videos that have had identity permutations.</p>\n<p>There are definitely other places in the dataset where the annotator just needs coffee :) it's not the most exciting job, labeling mouse behaviors!</p>",
      "rawMarkdown": "You are correct, animal identities were permuted in this video, and your ID-swaps correct for this. In investigating this I also found that this and 4 other videos (one in test and 4 in train) from this dataset have another issue. These 5 videos were all acquired at 25 fps (the rest are at 30), but the annotations--and not the pose--have been rescaled to be at 30fps. If you multiply all action start and stop frames by 25/30 you will see much better synchronization between annotations and behavior in these videos.\n\nI am double-checking this now and also running a test to look for other videos that have had identity permutations.\n\nThere are definitely other places in the dataset where the annotator just needs coffee :) it's not the most exciting job, labeling mouse behaviors!",
      "votes": null
    },
    {
      "id": "3306497",
      "postDate": "10/24/2025 16:47:02",
      "content": "<p>Hi Ann, thank you for all the data quality insights. You mentioned there was a video in the test set with this issue and I was wondering if that issue was corrected. Just wondering if my last submission improved from making meaningful progress or if the test set was just cleaned. Thanks!</p>",
      "rawMarkdown": "Hi Ann, thank you for all the data quality insights. You mentioned there was a video in the test set with this issue and I was wondering if that issue was corrected. Just wondering if my last submission improved from making meaningful progress or if the test set was just cleaned. Thanks!",
      "votes": null
    },
    {
      "id": "3306568",
      "postDate": "10/24/2025 19:37:09",
      "content": "<p>Thanks for these investigations Ann! Since you're looking into these issues I just want to bump my <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/609470#3298960\" target=\"_blank\">observation</a> that MABe22_movies still has an issue with permutation of agent/bodypart assignment since the data patch too. Seems this may have gotten buried.</p>",
      "rawMarkdown": "Thanks for these investigations Ann! Since you're looking into these issues I just want to bump my [observation](https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/609470#3298960) that MABe22_movies still has an issue with permutation of agent/bodypart assignment since the data patch too. Seems this may have gotten buried.",
      "votes": null
    },
    {
      "id": "3306573",
      "postDate": "10/24/2025 19:44:12",
      "content": "<p>Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.</p>",
      "rawMarkdown": "Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.",
      "votes": null
    },
    {
      "id": "3306574",
      "postDate": "10/24/2025 19:45:27",
      "content": "<p>Thanks for the reminder, and I apologize for this not getting fixed earlier. Will make a patch to the dataset if Kaggle folks don't think it's too late, stay tuned.</p>",
      "rawMarkdown": "Thanks for the reminder, and I apologize for this not getting fixed earlier. Will make a patch to the dataset if Kaggle folks don't think it's too late, stay tuned.",
      "votes": null
    },
    {
      "id": "3308353",
      "postDate": "10/29/2025 07:26:21",
      "content": "<p>Gosh! i have the same problem. I also visualized it and watched all morning, that doesn't look like chasing at all lol.\nFortunately, i saw this.</p>",
      "rawMarkdown": "Gosh! i have the same problem. I also visualized it and watched all morning, that doesn't look like chasing at all lol.\nFortunately, i saw this.",
      "votes": null
    },
    {
      "id": "3308462",
      "postDate": "10/29/2025 13:40:43",
      "content": "<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> Please let us know if some fixes are upcoming (or if we should fix it all alone).</p>",
      "rawMarkdown": "annkennedy05 Please let us know if some fixes are upcoming (or if we should fix it all alone).",
      "votes": null
    },
    {
      "id": "3308472",
      "postDate": "10/29/2025 14:05:04",
      "content": "<p>Hey Thanks you for all the data quality insights.</p>",
      "rawMarkdown": "Hey Thanks you for all the data quality insights.",
      "votes": null
    },
    {
      "id": "3309595",
      "postDate": "11/01/2025 03:00:22",
      "content": "<p>Has this issue been fixed yet? I am trying to decide if I should spend time on this competition.</p>",
      "rawMarkdown": "Has this issue been fixed yet? I am trying to decide if I should spend time on this competition.",
      "votes": null
    },
    {
      "id": "3309763",
      "postDate": "11/01/2025 10:46:31",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F44a73ff7f7ca7c0a25914f8b480614a8%2Fmouse-positions.png?generation=1761993361362472&amp;alt=media\" alt=\"\"></p>\n<p>Hi <a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> - I took a look at the AdaptableSnail videos that have FPS of 25 and, as you say, modifying these by 25/30 does give the correct start and stop frames. However, as <a href=\"https://www.kaggle.com/angantyr\" target=\"_blank\">@angantyr</a> found for the 1212811043 video, all of these 25 FPS videos also seem to have the incorrect labelling for the mice. For example, in the picture (from video_id = 878123481) it can be seen that mouse 3 is chasing mouse 1, however in the annotation it has this time period as 4 chasing 3:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F008b47b4351c69ee5385485d3972a973%2Fmouse-annotation.png?generation=1761993657164397&amp;alt=media\" alt=\"\"></p>\n<p>This swapping of the mice seems to occur in all of the AdaptableSnail 25 FPS videos, and may even occur in a much wider set. Obviously I could re-label the annotations, to give the correct mice IDs, but this would only help with training - if the test data also has this mislabelling then we'd never be able to predict the correct actions. </p>\n<p>Does this mislabelling occur for all the AdaptableSnail videos and, if it does, will there be an update to fix this (or will this labs data be removed from the data set)?</p>",
      "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F44a73ff7f7ca7c0a25914f8b480614a8%2Fmouse-positions.png?generation=1761993361362472&alt=media)\n\nHi @annkennedy05 - I took a look at the AdaptableSnail videos that have FPS of 25 and, as you say, modifying these by 25/30 does give the correct start and stop frames. However, as @angantyr found for the 1212811043 video, all of these 25 FPS videos also seem to have the incorrect labelling for the mice. For example, in the picture (from video_id = 878123481) it can be seen that mouse 3 is chasing mouse 1, however in the annotation it has this time period as 4 chasing 3:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F008b47b4351c69ee5385485d3972a973%2Fmouse-annotation.png?generation=1761993657164397&alt=media)\n\nThis swapping of the mice seems to occur in all of the AdaptableSnail 25 FPS videos, and may even occur in a much wider set. Obviously I could re-label the annotations, to give the correct mice IDs, but this would only help with training - if the test data also has this mislabelling then we'd never be able to predict the correct actions. \n\nDoes this mislabelling occur for all the AdaptableSnail videos and, if it does, will there be an update to fix this (or will this labs data be removed from the data set)?",
      "votes": null
    },
    {
      "id": "3311621",
      "postDate": "11/05/2025 12:20:33",
      "content": "<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a>: is there an ETA on the data fix? We only have a little more than one month left in the competition, the fix should not be delayed too much. </p>\n<p>And the fix will apply to both test and train data, correct?</p>",
      "rawMarkdown": "annkennedy05: is there an ETA on the data fix? We only have a little more than one month left in the competition, the fix should not be delayed too much. \n\nAnd the fix will apply to both test and train data, correct?",
      "votes": null
    },
    {
      "id": "3365340",
      "postDate": "12/07/2025 05:31:23",
      "content": "<p>Hello, do ID permutations happen for other the 25 fps videos in AdaptableSnail and are the permutations identical? Also are the 25fps videos in AdaptableSnail have their annotations rescaled to 30fps?</p>",
      "rawMarkdown": "Hello, do ID permutations happen for other the 25 fps videos in AdaptableSnail and are the permutations identical? Also are the 25fps videos in AdaptableSnail have their annotations rescaled to 30fps?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3304407,
      "author_name": "milanmuntenjon",
      "author_url": "",
      "post_date": "10/20/2025 13:30:37",
      "content": "<p>I checked a lot of files, this issue seems to be an anomaly. I can't spot a pattern in the labeling, so it's likely just a human mistake.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3304416,
      "author_name": "kyoumonemui",
      "author_url": "",
      "post_date": "10/20/2025 13:45:38",
      "content": "<p>Have similar issues been found in other video IDs of AdaptableSnail?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3304493,
          "author_name": "angantyr",
          "author_url": "",
          "post_date": "10/20/2025 16:15:38",
          "content": "<p>Yet to be verified. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3305561,
      "author_name": "ymatioun",
      "author_url": "",
      "post_date": "10/23/2025 01:28:13",
      "content": "<p>This makes sense; my CV For \"AdaptableSnail\" is only 0.19, and for that lab it is bad for all 7 possible actions (0.03 to 0.28). This is compared of average CV of all other labs of around 0.53. So i suspect many/most/all agent/target/action labels are incorrect for \"AdaptableSnail\" lab. Perhaps competition hosts can correct them in both train and test data. I hope <a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> can answer this question.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3305806,
          "author_name": "angantyr",
          "author_url": "",
          "post_date": "10/23/2025 13:28:44",
          "content": "<p>From what I see many trackers/annotations are often choppy and sparse too. Some labels are very precise, while others can be missing altogether.</p>\n<p>I have to sit down and do a round up of the tracker videos but <em>other stuff</em> gets a higher priority. TL;DR From what I see the erroneous labels were observed only in the AdaptableSnail/1212811043 video.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3305820,
              "author_name": "kyoumonemui",
              "author_url": "",
              "post_date": "10/23/2025 13:56:24",
              "content": "<p>Looks like I should’ve treated the annotator to some coffee.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3305943,
      "author_name": "annkennedy05",
      "author_url": "",
      "post_date": "10/23/2025 18:45:00",
      "content": "<p>You are correct, animal identities were permuted in this video, and your ID-swaps correct for this. In investigating this I also found that this and 4 other videos (one in test and 4 in train) from this dataset have another issue. These 5 videos were all acquired at 25 fps (the rest are at 30), but the annotations--and not the pose--have been rescaled to be at 30fps. If you multiply all action start and stop frames by 25/30 you will see much better synchronization between annotations and behavior in these videos.</p>\n<p>I am double-checking this now and also running a test to look for other videos that have had identity permutations.</p>\n<p>There are definitely other places in the dataset where the annotator just needs coffee :) it's not the most exciting job, labeling mouse behaviors!</p>",
      "votes": null,
      "replies": [
        {
          "id": 3306497,
          "author_name": "jvitta",
          "author_url": "",
          "post_date": "10/24/2025 16:47:02",
          "content": "<p>Hi Ann, thank you for all the data quality insights. You mentioned there was a video in the test set with this issue and I was wondering if that issue was corrected. Just wondering if my last submission improved from making meaningful progress or if the test set was just cleaned. Thanks!</p>",
          "votes": null,
          "replies": [
            {
              "id": 3306573,
              "author_name": "annkennedy05",
              "author_url": "",
              "post_date": "10/24/2025 19:44:12",
              "content": "<p>Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3311621,
                  "author_name": "ymatioun",
                  "author_url": "",
                  "post_date": "11/05/2025 12:20:33",
                  "content": "<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a>: is there an ETA on the data fix? We only have a little more than one month left in the competition, the fix should not be delayed too much. </p>\n<p>And the fix will apply to both test and train data, correct?</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        },
        {
          "id": 3306568,
          "author_name": "willhirsch",
          "author_url": "",
          "post_date": "10/24/2025 19:37:09",
          "content": "<p>Thanks for these investigations Ann! Since you're looking into these issues I just want to bump my <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/609470#3298960\" target=\"_blank\">observation</a> that MABe22_movies still has an issue with permutation of agent/bodypart assignment since the data patch too. Seems this may have gotten buried.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3306574,
              "author_name": "annkennedy05",
              "author_url": "",
              "post_date": "10/24/2025 19:45:27",
              "content": "<p>Thanks for the reminder, and I apologize for this not getting fixed earlier. Will make a patch to the dataset if Kaggle folks don't think it's too late, stay tuned.</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 3308462,
          "author_name": "ogurtsov",
          "author_url": "",
          "post_date": "10/29/2025 13:40:43",
          "content": "<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> Please let us know if some fixes are upcoming (or if we should fix it all alone).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3365340,
          "author_name": "xiaoz259",
          "author_url": "",
          "post_date": "12/07/2025 05:31:23",
          "content": "<p>Hello, do ID permutations happen for other the 25 fps videos in AdaptableSnail and are the permutations identical? Also are the 25fps videos in AdaptableSnail have their annotations rescaled to 30fps?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3308353,
      "author_name": "pekoshda",
      "author_url": "",
      "post_date": "10/29/2025 07:26:21",
      "content": "<p>Gosh! i have the same problem. I also visualized it and watched all morning, that doesn't look like chasing at all lol.\nFortunately, i saw this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3308472,
      "author_name": "khushikyad001",
      "author_url": "",
      "post_date": "10/29/2025 14:05:04",
      "content": "<p>Hey Thanks you for all the data quality insights.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3309595,
      "author_name": "xiaoz259",
      "author_url": "",
      "post_date": "11/01/2025 03:00:22",
      "content": "<p>Has this issue been fixed yet? I am trying to decide if I should spend time on this competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3309763,
      "author_name": "steveroberts",
      "author_url": "",
      "post_date": "11/01/2025 10:46:31",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F44a73ff7f7ca7c0a25914f8b480614a8%2Fmouse-positions.png?generation=1761993361362472&amp;alt=media\" alt=\"\"></p>\n<p>Hi <a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> - I took a look at the AdaptableSnail videos that have FPS of 25 and, as you say, modifying these by 25/30 does give the correct start and stop frames. However, as <a href=\"https://www.kaggle.com/angantyr\" target=\"_blank\">@angantyr</a> found for the 1212811043 video, all of these 25 FPS videos also seem to have the incorrect labelling for the mice. For example, in the picture (from video_id = 878123481) it can be seen that mouse 3 is chasing mouse 1, however in the annotation it has this time period as 4 chasing 3:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F008b47b4351c69ee5385485d3972a973%2Fmouse-annotation.png?generation=1761993657164397&amp;alt=media\" alt=\"\"></p>\n<p>This swapping of the mice seems to occur in all of the AdaptableSnail 25 FPS videos, and may even occur in a much wider set. Obviously I could re-label the annotations, to give the correct mice IDs, but this would only help with training - if the test data also has this mislabelling then we'd never be able to predict the correct actions. </p>\n<p>Does this mislabelling occur for all the AdaptableSnail videos and, if it does, will there be an update to fix this (or will this labs data be removed from the data set)?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3304353": "I wrote a short script to visualize the trackers and annotations for a given lab/video. So far I've only tested AdaptableSnail (video_id: 1212811043) and it looks like the annotations are erroneous.\n\nThe annotations form a title as comma-separated actions:\n```python\nf\"{agent_id} {action} {target_id}\"\n```\n\nOriginal annotations:\n| agent_id | target_id | action | start_frame | stop_frame | \n| --- | --- | --- | --- | --- |\n| 0 | 1 | 3 | chase | 2 | 54 |\n| 1 | 1 | 3 | chase | 128 | 234 |\n| 2 | 3 | 2 | avoid | 324 | 342 |\n| 3 | 3 | 1 | avoid | 324 | 342 |\n| 4 | 1 | 2 | chase | 942 | 1052 |\n| 5 | 4 | 3 | chase | 1205 | 1276 |\n| 6 | 4 | 3 | chase | 1313 | 1370 |\n| 7 | 1 | 2 | chase | 1416 | 1458 |\n| 8 | 4 | 3 | chase | 1819 | 1908 |\n| 9 | 4 | 2 | chase | 2041 | 2101 |\n\nNew annotations:\n| agent_id | target_id | action | start_frame | stop_frame | \n| --- | --- | --- | --- | --- |\n| 0 | 3 | 2 | chase | 2 | 54 |\n| 1 | 3 | 2 | chase | 128 | 234 |\n| 2 | 2 | 3 | avoid | 324 | 342 |\n| 3 | 2 | 1 | avoid | 324 | 342 |\n| 4 | 4 | 2 | chase | 942 | 1052 |\n| 5 | 3 | 1 | chase | 1205 | 1276 |\n| 6 | 3 | 1 | chase | 1313 | 1370 |\n| 7 | 3 | 1 | chase | 1416 | 1458 |\n| 8 | 4 | 3 | chase | 1819 | 1908 |\n| 9 | 4 | 3 | chase (chaseattack?) | 2041 | 2101 |\n\nIt doesn't look like a simple labelling error, unless it was done on the basis of some specific bodypart trackers, which sometimes get swapped.\n\nI've attached a sample MP4 showing the first 2200 frames.",
    "3304407": "I checked a lot of files, this issue seems to be an anomaly. I can't spot a pattern in the labeling, so it's likely just a human mistake.",
    "3304416": "Have similar issues been found in other video IDs of AdaptableSnail?",
    "3304493": "Yet to be verified.",
    "3305561": "This makes sense; my CV For \"AdaptableSnail\" is only 0.19, and for that lab it is bad for all 7 possible actions (0.03 to 0.28). This is compared of average CV of all other labs of around 0.53. So i suspect many/most/all agent/target/action labels are incorrect for \"AdaptableSnail\" lab. Perhaps competition hosts can correct them in both train and test data. I hope @annkennedy05 can answer this question.",
    "3305806": "From what I see many trackers/annotations are often choppy and sparse too. Some labels are very precise, while others can be missing altogether.\n\nI have to sit down and do a round up of the tracker videos but *other stuff* gets a higher priority. TL;DR From what I see the erroneous labels were observed only in the AdaptableSnail/1212811043 video.",
    "3305820": "Looks like I should’ve treated the annotator to some coffee.",
    "3305943": "You are correct, animal identities were permuted in this video, and your ID-swaps correct for this. In investigating this I also found that this and 4 other videos (one in test and 4 in train) from this dataset have another issue. These 5 videos were all acquired at 25 fps (the rest are at 30), but the annotations--and not the pose--have been rescaled to be at 30fps. If you multiply all action start and stop frames by 25/30 you will see much better synchronization between annotations and behavior in these videos.\n\nI am double-checking this now and also running a test to look for other videos that have had identity permutations.\n\nThere are definitely other places in the dataset where the annotator just needs coffee :) it's not the most exciting job, labeling mouse behaviors!",
    "3306497": "Hi Ann, thank you for all the data quality insights. You mentioned there was a video in the test set with this issue and I was wondering if that issue was corrected. Just wondering if my last submission improved from making meaningful progress or if the test set was just cleaned. Thanks!",
    "3306568": "Thanks for these investigations Ann! Since you're looking into these issues I just want to bump my [observation](https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/609470#3298960) that MABe22_movies still has an issue with permutation of agent/bodypart assignment since the data patch too. Seems this may have gotten buried.",
    "3306573": "Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.",
    "3306574": "Thanks for the reminder, and I apologize for this not getting fixed earlier. Will make a patch to the dataset if Kaggle folks don't think it's too late, stay tuned.",
    "3308353": "Gosh! i have the same problem. I also visualized it and watched all morning, that doesn't look like chasing at all lol.\nFortunately, i saw this.",
    "3308462": "annkennedy05 Please let us know if some fixes are upcoming (or if we should fix it all alone).",
    "3308472": "Hey Thanks you for all the data quality insights.",
    "3309595": "Has this issue been fixed yet? I am trying to decide if I should spend time on this competition.",
    "3309763": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F44a73ff7f7ca7c0a25914f8b480614a8%2Fmouse-positions.png?generation=1761993361362472&alt=media)\n\nHi @annkennedy05 - I took a look at the AdaptableSnail videos that have FPS of 25 and, as you say, modifying these by 25/30 does give the correct start and stop frames. However, as @angantyr found for the 1212811043 video, all of these 25 FPS videos also seem to have the incorrect labelling for the mice. For example, in the picture (from video_id = 878123481) it can be seen that mouse 3 is chasing mouse 1, however in the annotation it has this time period as 4 chasing 3:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1650068%2F008b47b4351c69ee5385485d3972a973%2Fmouse-annotation.png?generation=1761993657164397&alt=media)\n\nThis swapping of the mice seems to occur in all of the AdaptableSnail 25 FPS videos, and may even occur in a much wider set. Obviously I could re-label the annotations, to give the correct mice IDs, but this would only help with training - if the test data also has this mislabelling then we'd never be able to predict the correct actions. \n\nDoes this mislabelling occur for all the AdaptableSnail videos and, if it does, will there be an update to fix this (or will this labs data be removed from the data set)?",
    "3311621": "annkennedy05: is there an ETA on the data fix? We only have a little more than one month left in the competition, the fix should not be delayed too much. \n\nAnd the fix will apply to both test and train data, correct?",
    "3365340": "Hello, do ID permutations happen for other the 25 fps videos in AdaptableSnail and are the permutations identical? Also are the 25fps videos in AdaptableSnail have their annotations rescaled to 30fps?"
  },
  "source": "meta"
}