![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Dec-2025/2nd_place.png\")
Wow, as a post comp experiment, I updated my final submission (code is here) to make zero predictions on AdaptableSnail private test video with frames_per_second=25. The result is Private LB = 0.535 which wins 2nd place, Wow! (This is a +0.012 improvement from 0.523)
\n
If others want to see what their Private LB score is without making predictions on AdaptableSnail test video with frames_per_second=25, add the following code to the end of your submission
\ntest = pd.read_csv(\"/kaggle/input/MABe-mouse-behavior-detection/test.csv\")\nV = test.loc[(test.lab_id==\"AdaptableSnail\")&(test.frames_per_second==25)].video_id.unique()\ndf = pd.read_csv(\"submission.csv\")\ndf = df.loc[~df.video_id.isin(V)] \ndf.to_csv(\"submission.csv\",index=False)\nPersonally, I think it is unfair that the host and Kaggle both know that the targets are wrong for private test video AdaptableSnail with frames_per_second=25 but they still left it in private LB. The host and Kaggle admits the targets are wrong in discussion here.
\nI expected that Kaggle would fix or remove this video. (So I did not do any adjusting to my final submission). What is the purpose of evaluating final scores intentionally using wrong targets? @annkennedy05 @addisonhoward @inversion
\nMore discussion here
", "messages": [ { "id": "3377738", "postDate": "12/16/2025 21:33:50", "content": "Wow, as a post comp experiment, I updated my final submission (code is here) to make zero predictions on AdaptableSnail private test video with frames_per_second=25. The result is Private LB = 0.535 which wins 2nd place, Wow! (This is a +0.012 improvement from 0.523)
\n
If others want to see what their Private LB score is without making predictions on AdaptableSnail test video with frames_per_second=25, add the following code to the end of your submission
\ntest = pd.read_csv(\"/kaggle/input/MABe-mouse-behavior-detection/test.csv\")\nV = test.loc[(test.lab_id==\"AdaptableSnail\")&(test.frames_per_second==25)].video_id.unique()\ndf = pd.read_csv(\"submission.csv\")\ndf = df.loc[~df.video_id.isin(V)] \ndf.to_csv(\"submission.csv\",index=False)\nPersonally, I think it is unfair that the host and Kaggle both know that the targets are wrong for private test video AdaptableSnail with frames_per_second=25 but they still left it in private LB. The host and Kaggle admits the targets are wrong in discussion here.
\nI expected that Kaggle would fix or remove this video. (So I did not do any adjusting to my final submission). What is the purpose of evaluating final scores intentionally using wrong targets? @annkennedy05 @addisonhoward @inversion
\nMore discussion here
", "rawMarkdown": "Wow, as a post comp experiment, I updated my final submission (code is [here][2]) to make zero predictions on AdaptableSnail private test video with frames_per_second=25. The result is Private LB = 0.535 which wins 2nd place, Wow! (This is a +0.012 improvement from 0.523)\n\n\n\nIf others want to see what their Private LB score is without making predictions on AdaptableSnail test video with frames_per_second=25, add the following code to the end of your submission\n\n test = pd.read_csv(\"/kaggle/input/MABe-mouse-behavior-detection/test.csv\")\n V = test.loc[(test.lab_id==\"AdaptableSnail\")&(test.frames_per_second==25)].video_id.unique()\n df = pd.read_csv(\"submission.csv\")\n df = df.loc[~df.video_id.isin(V)] \n df.to_csv(\"submission.csv\",index=False)\n\nPersonally, I think it is unfair that the host and Kaggle both know that the targets are wrong for private test video AdaptableSnail with frames_per_second=25 but they still left it in private LB. The host and Kaggle admits the targets are wrong in discussion [here][1].\n\nI expected that Kaggle would fix or remove this video. (So I did not do any adjusting to my final submission). What is the purpose of evaluating final scores intentionally using wrong targets? @annkennedy05 @addisonhoward @inversion \n\nMore discussion [here][3]\n\n[1]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/612531#3305943\n[2]: https://www.kaggle.com/code/cdeotte/7th-place-gold-mouse-comp-cv-0-56-lb-0-56\n[3]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663466", "votes": null }, { "id": "3377744", "postDate": "12/16/2025 21:47:01", "content": "Makes no sense to consolidate the final LB using wrong labels and is extremelly bad for Kaggle reputation.\nBest solution would be simply remove the bad data and rescored LB.
", "rawMarkdown": "Makes no sense to consolidate the final LB using wrong labels and is extremelly bad for Kaggle reputation.\nBest solution would be simply remove the bad data and rescored LB.", "votes": null }, { "id": "3377745", "postDate": "12/16/2025 21:48:07", "content": "Absolutely agree. There is no point to knowingly evaluating submissions using wrong targets. It makes absolutely no sense whatsoever.
", "rawMarkdown": "Absolutely agree. There is no point to knowingly evaluating submissions using wrong targets. It makes absolutely no sense whatsoever.", "votes": null }, { "id": "3377786", "postDate": "12/17/2025 00:04:37", "content": "there is more to it - this is from 2nd place write-up (https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/writeups/2nd-place-solution):
\nAdaptableSnail 25 FPS Adjustment:\n start_frame and stop_frame are multiplied by 30 / 25\n When mouse ID swaps are present in the tracking data, this adjustment is not meaningful\n The private dataset appears to include tracking sequences without mouse ID swaps\n For these sequences, predictions are aligned to the 30 FPS labels\n This resulted in a +0.009 improvement in the private score
\nSo, if instead of dropping those videos you would scale them up, you could bet extra 9 points.
\nI did drop those videos, but did not try to scale the prediction.
", "rawMarkdown": "there is more to it - this is from 2nd place write-up (https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/writeups/2nd-place-solution):\n\nAdaptableSnail 25 FPS Adjustment:\n start_frame and stop_frame are multiplied by 30 / 25\n When mouse ID swaps are present in the tracking data, this adjustment is not meaningful\n The private dataset appears to include tracking sequences without mouse ID swaps\n For these sequences, predictions are aligned to the 30 FPS labels\n This resulted in a +0.009 improvement in the private score\n\nSo, if instead of dropping those videos you would scale them up, you could bet extra 9 points.\n\nI did drop those videos, but did not try to scale the prediction.", "votes": null }, { "id": "3377824", "postDate": "12/17/2025 02:13:34", "content": "Skipping them was improving CV, therefore it makes sense that it improves LB too. One could also provide fps as a feature and the model could learn to reduce their probabilities. It is like any other Kaggle label noise. Rescore would be unfair as well.
\nBut I think it was bad that the host said fps25 doesn’t exist in the test set. I think hosts shouldn’t provide wrong information about the test set. Or maybe shouldn’t provide any info at all, so that we can assume test set is similar to the train set.
", "rawMarkdown": "Skipping them was improving CV, therefore it makes sense that it improves LB too. One could also provide fps as a feature and the model could learn to reduce their probabilities. It is like any other Kaggle label noise. Rescore would be unfair as well.\n\nBut I think it was bad that the host said fps25 doesn’t exist in the test set. I think hosts shouldn’t provide wrong information about the test set. Or maybe shouldn’t provide any info at all, so that we can assume test set is similar to the train set.", "votes": null }, { "id": "3377961", "postDate": "12/17/2025 08:33:08", "content": "Same for me; didn't drop those predictions and now after applying the fix private LB score would have been 0.530 (+0.014). Certainly Kaggle should have updated the datasets when these errors were found, now there is only bad decissions left.
", "rawMarkdown": "Same for me; didn't drop those predictions and now after applying the fix private LB score would have been 0.530 (+0.014). Certainly Kaggle should have updated the datasets when these errors were found, now there is only bad decissions left.", "votes": null }, { "id": "3377967", "postDate": "12/17/2025 08:43:28", "content": "I agree with you @cdeotte
", "rawMarkdown": "I agree with you @cdeotte", "votes": null }, { "id": "3377977", "postDate": "12/17/2025 09:04:37", "content": "I think the reason Adaptable Snail is very difficult to use comes from the high number of actions combined the high number of agents (three in total). This makes up a total of 74 distinct targets with 66 distinct with respect to all other targets. NiftyGoldfinch is rich in targets and actions as well.
\n![\"AdaptableSnail](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2259646%2F6a306e8d416610dae938807603b7e1c2%2FTarget-abundance-in-Adaptable-Snail.png?generation=1765961746579141&alt=media\")
In the same time the frame population is pretty fragmented. This is how it looks like for some selected labs per actions:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2259646%2F777688cef2eb14d4377fccc0b2206c38%2FSelected-frame-counts-per-action-per-lab.png?generation=1765962207959801&alt=media\")
I suspect that if you fiddle around with NiftyGoldfinch you may be able to better the score even further by decreasing the input data noise.
", "rawMarkdown": "I think the reason Adaptable Snail is very difficult to use comes from the high number of actions **combined** the high number of agents (three in total). This makes up a total of 74 distinct targets with 66 distinct with respect to all other targets. NiftyGoldfinch is rich in targets and actions as well.\n\n\n\nIn the same time the frame population is pretty fragmented. This is how it looks like for some selected labs per actions:\n\n\nI suspect that if you fiddle around with NiftyGoldfinch you may be able to better the score even further by decreasing the input data noise.", "votes": null }, { "id": "3378367", "postDate": "12/18/2025 00:38:33", "content": "Using post comp submissions, I have extracted my LB F1 score per lab (i.e. submit only 1 lab's predictions then multiply LB score by 15). We can see how the incorrectly labeled private test AdaptableSnail video is degrading the AdaptableSnail private LB F1 score:
\n| lab_id | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| AdaptableSnail | \n0.541807 | \n0.51000 | \n0.12810 | \n
| BoisterousParrot | \n0.667919 | \n0.75870 | \n0.59460 | \n
| CautiousGiraffe | \n0.601883 | \n0.63405 | \n0.64515 | \n
| DeliriousFly | \n0.589313 | \n0.58485 | \n0.47280 | \n
| ElegantMink | \n0.556870 | \n0.60000 | \n0.70815 | \n
| GroovyShrew | \n0.523861 | \n0.56550 | \n0.53835 | \n
| InvincibleJellyfish | \n0.396364 | \n0.30240 | \n0.36060 | \n
| JovialSwallow | \n0.501054 | \n0.42765 | \n0.47085 | \n
| LyricalHare | \n0.605622 | \n0.59025 | \n0.49260 | \n
| NiftyGoldfinch | \n0.468248 | \n0.48945 | \n0.49110 | \n
| PleasantMeerkat | \n0.339524 | \n0.28320 | \n0.28800 | \n
| ReflectiveManatee | \n0.877714 | \n0.91140 | \n0.89430 | \n
| SparklingTapir | \n0.754358 | \n0.54135 | \n0.60420 | \n
| TranquilPanther | \n0.500783 | \n0.58695 | \n0.52470 | \n
| UppityFerret | \n0.600439 | \n0.63210 | \n0.63270 | \n
I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.
", "rawMarkdown": "I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.", "votes": null }, { "id": "3385172", "postDate": "01/02/2026 16:52:41", "content": "For future competitions, it would be helpful for Kaggle to clarify expectations in such situations: when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?
\nMore discussion including the effect it had on private LB rankings here
", "rawMarkdown": "For future competitions, it would be helpful for Kaggle to clarify expectations in such situations: when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?\n\nMore discussion including the effect it had on private LB rankings [here][1]\n\n[1]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663466", "votes": null }, { "id": "3385249", "postDate": "01/02/2026 20:32:47", "content": "absolutely right
", "rawMarkdown": "absolutely right", "votes": null } ], "comments": [ { "id": 3377744, "author_name": "titericz", "author_url": "", "post_date": "12/16/2025 21:47:01", "content": "Makes no sense to consolidate the final LB using wrong labels and is extremelly bad for Kaggle reputation.\nBest solution would be simply remove the bad data and rescored LB.
", "votes": null, "replies": [ { "id": 3377745, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 21:48:07", "content": "Absolutely agree. There is no point to knowingly evaluating submissions using wrong targets. It makes absolutely no sense whatsoever.
", "votes": null, "replies": [ { "id": 3377786, "author_name": "ymatioun", "author_url": "", "post_date": "12/17/2025 00:04:37", "content": "there is more to it - this is from 2nd place write-up (https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/writeups/2nd-place-solution):
\nAdaptableSnail 25 FPS Adjustment:\n start_frame and stop_frame are multiplied by 30 / 25\n When mouse ID swaps are present in the tracking data, this adjustment is not meaningful\n The private dataset appears to include tracking sequences without mouse ID swaps\n For these sequences, predictions are aligned to the 30 FPS labels\n This resulted in a +0.009 improvement in the private score
\nSo, if instead of dropping those videos you would scale them up, you could bet extra 9 points.
\nI did drop those videos, but did not try to scale the prediction.
", "votes": null, "replies": [] } ] }, { "id": 3377824, "author_name": "aerdem4", "author_url": "", "post_date": "12/17/2025 02:13:34", "content": "Skipping them was improving CV, therefore it makes sense that it improves LB too. One could also provide fps as a feature and the model could learn to reduce their probabilities. It is like any other Kaggle label noise. Rescore would be unfair as well.
\nBut I think it was bad that the host said fps25 doesn’t exist in the test set. I think hosts shouldn’t provide wrong information about the test set. Or maybe shouldn’t provide any info at all, so that we can assume test set is similar to the train set.
", "votes": null, "replies": [] }, { "id": 3378870, "author_name": "sohier", "author_url": "", "post_date": "12/18/2025 18:30:37", "content": "I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.
", "votes": null, "replies": [ { "id": 3385172, "author_name": "cdeotte", "author_url": "", "post_date": "01/02/2026 16:52:41", "content": "For future competitions, it would be helpful for Kaggle to clarify expectations in such situations: when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?
\nMore discussion including the effect it had on private LB rankings here
", "votes": null, "replies": [ { "id": 3385249, "author_name": "yusufsertkayaysk", "author_url": "", "post_date": "01/02/2026 20:32:47", "content": "absolutely right
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3377961, "author_name": "herrahuu", "author_url": "", "post_date": "12/17/2025 08:33:08", "content": "Same for me; didn't drop those predictions and now after applying the fix private LB score would have been 0.530 (+0.014). Certainly Kaggle should have updated the datasets when these errors were found, now there is only bad decissions left.
", "votes": null, "replies": [] }, { "id": 3377967, "author_name": "navneetbende", "author_url": "", "post_date": "12/17/2025 08:43:28", "content": "I agree with you @cdeotte
", "votes": null, "replies": [] }, { "id": 3377977, "author_name": "angantyr", "author_url": "", "post_date": "12/17/2025 09:04:37", "content": "I think the reason Adaptable Snail is very difficult to use comes from the high number of actions combined the high number of agents (three in total). This makes up a total of 74 distinct targets with 66 distinct with respect to all other targets. NiftyGoldfinch is rich in targets and actions as well.
\n
In the same time the frame population is pretty fragmented. This is how it looks like for some selected labs per actions:\n
I suspect that if you fiddle around with NiftyGoldfinch you may be able to better the score even further by decreasing the input data noise.
", "votes": null, "replies": [] }, { "id": 3378367, "author_name": "cdeotte", "author_url": "", "post_date": "12/18/2025 00:38:33", "content": "Using post comp submissions, I have extracted my LB F1 score per lab (i.e. submit only 1 lab's predictions then multiply LB score by 15). We can see how the incorrectly labeled private test AdaptableSnail video is degrading the AdaptableSnail private LB F1 score:
\n| lab_id | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| AdaptableSnail | \n0.541807 | \n0.51000 | \n0.12810 | \n
| BoisterousParrot | \n0.667919 | \n0.75870 | \n0.59460 | \n
| CautiousGiraffe | \n0.601883 | \n0.63405 | \n0.64515 | \n
| DeliriousFly | \n0.589313 | \n0.58485 | \n0.47280 | \n
| ElegantMink | \n0.556870 | \n0.60000 | \n0.70815 | \n
| GroovyShrew | \n0.523861 | \n0.56550 | \n0.53835 | \n
| InvincibleJellyfish | \n0.396364 | \n0.30240 | \n0.36060 | \n
| JovialSwallow | \n0.501054 | \n0.42765 | \n0.47085 | \n
| LyricalHare | \n0.605622 | \n0.59025 | \n0.49260 | \n
| NiftyGoldfinch | \n0.468248 | \n0.48945 | \n0.49110 | \n
| PleasantMeerkat | \n0.339524 | \n0.28320 | \n0.28800 | \n
| ReflectiveManatee | \n0.877714 | \n0.91140 | \n0.89430 | \n
| SparklingTapir | \n0.754358 | \n0.54135 | \n0.60420 | \n
| TranquilPanther | \n0.500783 | \n0.58695 | \n0.52470 | \n
| UppityFerret | \n0.600439 | \n0.63210 | \n0.63270 | \n