{
  "id": 663466,
  "title": "Why isn't Private LB Updated?",
  "url": "/competitions/MABe-mouse-behavior-detection/discussion/663466",
  "author_name": "",
  "post_date": "2025-12-18T02:52:35.403983400Z",
  "votes": 40,
  "comment_count": 10,
  "views": 0,
  "content": "<p>By now, it appears that the private leaderboard will not be updated, despite the host stating <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/612531#3306573\" target=\"_blank\">here</a> that the private LB would be corrected:</p>\n<blockquote>\n  <p>Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.</p>\n</blockquote>\n<p>The strange decision to evaluate models with wrong labels has a huge impact on the final rankings. Kagglers who trusted the host's statement—and reasonably expected the evaluation to use corrected labels—experienced worse private leaderboard outcomes, while approaches that avoided or altered predictions on the affected data achieved better private leaderboard results. Personally I am very upset and disappointed with Kaggle.</p>\n<p>The competition metric is the average of 15 lab F1 scores. The private LB has only two AdaptableSnail videos (discovered via probing). Therefore one incorrect AdaptableSnail video has a large impact on the final private LB score and disproportionately affects final rankings. Below are my per lab LB scores (discovered via probing). We can see that the private LB did not fix the incorrect AdaptableSnail video:</p>\n<table>\n<thead>\n<tr>\n<th>lab_id</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>AdaptableSnail</td>\n<td>0.541807</td>\n<td>0.51000</td>\n<td><strong>0.12810</strong></td>\n</tr>\n<tr>\n<td>BoisterousParrot</td>\n<td>0.667919</td>\n<td>0.75870</td>\n<td>0.59460</td>\n</tr>\n<tr>\n<td>CautiousGiraffe</td>\n<td>0.601883</td>\n<td>0.63405</td>\n<td>0.64515</td>\n</tr>\n<tr>\n<td>DeliriousFly</td>\n<td>0.589313</td>\n<td>0.58485</td>\n<td>0.47280</td>\n</tr>\n<tr>\n<td>ElegantMink</td>\n<td>0.556870</td>\n<td>0.60000</td>\n<td>0.70815</td>\n</tr>\n<tr>\n<td>GroovyShrew</td>\n<td>0.523861</td>\n<td>0.56550</td>\n<td>0.53835</td>\n</tr>\n<tr>\n<td>InvincibleJellyfish</td>\n<td>0.396364</td>\n<td>0.30240</td>\n<td>0.36060</td>\n</tr>\n<tr>\n<td>JovialSwallow</td>\n<td>0.501054</td>\n<td>0.42765</td>\n<td>0.47085</td>\n</tr>\n<tr>\n<td>LyricalHare</td>\n<td>0.605622</td>\n<td>0.59025</td>\n<td>0.49260</td>\n</tr>\n<tr>\n<td>NiftyGoldfinch</td>\n<td>0.468248</td>\n<td>0.48945</td>\n<td>0.49110</td>\n</tr>\n<tr>\n<td>PleasantMeerkat</td>\n<td>0.339524</td>\n<td>0.28320</td>\n<td>0.28800</td>\n</tr>\n<tr>\n<td>ReflectiveManatee</td>\n<td>0.877714</td>\n<td>0.91140</td>\n<td>0.89430</td>\n</tr>\n<tr>\n<td>SparklingTapir</td>\n<td>0.754358</td>\n<td>0.54135</td>\n<td>0.60420</td>\n</tr>\n<tr>\n<td>TranquilPanther</td>\n<td>0.500783</td>\n<td>0.58695</td>\n<td>0.52470</td>\n</tr>\n<tr>\n<td>UppityFerret</td>\n<td>0.600439</td>\n<td>0.63210</td>\n<td>0.63270</td>\n</tr>\n</tbody>\n</table>\n<p>I am also upset that Kaggle doesn't respond to any Kaggler requests to comment. Could a Kaggle admin please clarify whether final submissions were evaluated using labels known to be incorrect, and—if so—explain why this decision was considered fair to all competitors? <a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> <a href=\"https://www.kaggle.com/addisonhoward\" target=\"_blank\">@addisonhoward</a> <a href=\"https://www.kaggle.com/inversion\" target=\"_blank\">@inversion</a> <a href=\"https://www.kaggle.com/sohiermse\" target=\"_blank\">@sohiermse</a> </p>\n<h1>UPDATE</h1>\n<p>Sohier Dane responded <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870\" target=\"_blank\">here</a></p>\n<blockquote>\n  <p>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.</p>\n</blockquote>\n<p>This confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. </p>\n<p>This sets an important precedent. In this competition, known label errors were deemed insufficient to warrant correcting the final evaluation, despite having a measurable impact on private leaderboard rankings.</p>\n<p>For future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n<strong>when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?</strong></p>\n<p>Clear guidance on this point would help competitors understand how to respond to similar issues going forward and would strengthen confidence in final leaderboard outcomes. The figure below summarizes the private leaderboard shifts associated with this issue (with more discussion <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198\" target=\"_blank\">here</a>).</p>\n<p><img src=\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Jan-2026/mouse-lb3.png\" alt=\"\"></p>",
  "messages": [
    {
      "id": "3378395",
      "postDate": "12/18/2025 02:52:35",
      "content": "<p>By now, it appears that the private leaderboard will not be updated, despite the host stating <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/612531#3306573\" target=\"_blank\">here</a> that the private LB would be corrected:</p>\n<blockquote>\n  <p>Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.</p>\n</blockquote>\n<p>The strange decision to evaluate models with wrong labels has a huge impact on the final rankings. Kagglers who trusted the host's statement—and reasonably expected the evaluation to use corrected labels—experienced worse private leaderboard outcomes, while approaches that avoided or altered predictions on the affected data achieved better private leaderboard results. Personally I am very upset and disappointed with Kaggle.</p>\n<p>The competition metric is the average of 15 lab F1 scores. The private LB has only two AdaptableSnail videos (discovered via probing). Therefore one incorrect AdaptableSnail video has a large impact on the final private LB score and disproportionately affects final rankings. Below are my per lab LB scores (discovered via probing). We can see that the private LB did not fix the incorrect AdaptableSnail video:</p>\n<table>\n<thead>\n<tr>\n<th>lab_id</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>AdaptableSnail</td>\n<td>0.541807</td>\n<td>0.51000</td>\n<td><strong>0.12810</strong></td>\n</tr>\n<tr>\n<td>BoisterousParrot</td>\n<td>0.667919</td>\n<td>0.75870</td>\n<td>0.59460</td>\n</tr>\n<tr>\n<td>CautiousGiraffe</td>\n<td>0.601883</td>\n<td>0.63405</td>\n<td>0.64515</td>\n</tr>\n<tr>\n<td>DeliriousFly</td>\n<td>0.589313</td>\n<td>0.58485</td>\n<td>0.47280</td>\n</tr>\n<tr>\n<td>ElegantMink</td>\n<td>0.556870</td>\n<td>0.60000</td>\n<td>0.70815</td>\n</tr>\n<tr>\n<td>GroovyShrew</td>\n<td>0.523861</td>\n<td>0.56550</td>\n<td>0.53835</td>\n</tr>\n<tr>\n<td>InvincibleJellyfish</td>\n<td>0.396364</td>\n<td>0.30240</td>\n<td>0.36060</td>\n</tr>\n<tr>\n<td>JovialSwallow</td>\n<td>0.501054</td>\n<td>0.42765</td>\n<td>0.47085</td>\n</tr>\n<tr>\n<td>LyricalHare</td>\n<td>0.605622</td>\n<td>0.59025</td>\n<td>0.49260</td>\n</tr>\n<tr>\n<td>NiftyGoldfinch</td>\n<td>0.468248</td>\n<td>0.48945</td>\n<td>0.49110</td>\n</tr>\n<tr>\n<td>PleasantMeerkat</td>\n<td>0.339524</td>\n<td>0.28320</td>\n<td>0.28800</td>\n</tr>\n<tr>\n<td>ReflectiveManatee</td>\n<td>0.877714</td>\n<td>0.91140</td>\n<td>0.89430</td>\n</tr>\n<tr>\n<td>SparklingTapir</td>\n<td>0.754358</td>\n<td>0.54135</td>\n<td>0.60420</td>\n</tr>\n<tr>\n<td>TranquilPanther</td>\n<td>0.500783</td>\n<td>0.58695</td>\n<td>0.52470</td>\n</tr>\n<tr>\n<td>UppityFerret</td>\n<td>0.600439</td>\n<td>0.63210</td>\n<td>0.63270</td>\n</tr>\n</tbody>\n</table>\n<p>I am also upset that Kaggle doesn't respond to any Kaggler requests to comment. Could a Kaggle admin please clarify whether final submissions were evaluated using labels known to be incorrect, and—if so—explain why this decision was considered fair to all competitors? <a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> <a href=\"https://www.kaggle.com/addisonhoward\" target=\"_blank\">@addisonhoward</a> <a href=\"https://www.kaggle.com/inversion\" target=\"_blank\">@inversion</a> <a href=\"https://www.kaggle.com/sohiermse\" target=\"_blank\">@sohiermse</a> </p>\n<h1>UPDATE</h1>\n<p>Sohier Dane responded <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870\" target=\"_blank\">here</a></p>\n<blockquote>\n  <p>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.</p>\n</blockquote>\n<p>This confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. </p>\n<p>This sets an important precedent. In this competition, known label errors were deemed insufficient to warrant correcting the final evaluation, despite having a measurable impact on private leaderboard rankings.</p>\n<p>For future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n<strong>when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?</strong></p>\n<p>Clear guidance on this point would help competitors understand how to respond to similar issues going forward and would strengthen confidence in final leaderboard outcomes. The figure below summarizes the private leaderboard shifts associated with this issue (with more discussion <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198\" target=\"_blank\">here</a>).</p>\n<p><img src=\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Jan-2026/mouse-lb3.png\" alt=\"\"></p>",
      "rawMarkdown": "By now, it appears that the private leaderboard will not be updated, despite the host stating [here][1] that the private LB would be corrected:\n> Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.\n\nThe strange decision to evaluate models with wrong labels has a huge impact on the final rankings. Kagglers who trusted the host's statement—and reasonably expected the evaluation to use corrected labels—experienced worse private leaderboard outcomes, while approaches that avoided or altered predictions on the affected data achieved better private leaderboard results. Personally I am very upset and disappointed with Kaggle.\n\nThe competition metric is the average of 15 lab F1 scores. The private LB has only two AdaptableSnail videos (discovered via probing). Therefore one incorrect AdaptableSnail video has a large impact on the final private LB score and disproportionately affects final rankings. Below are my per lab LB scores (discovered via probing). We can see that the private LB did not fix the incorrect AdaptableSnail video:\n\n| lab_id              | CV       | Public LB | Private LB |\n|---------------------|----------|-----------|------------|\n| AdaptableSnail      | 0.541807 | 0.51000   | **0.12810**    |\n| BoisterousParrot    | 0.667919 | 0.75870   | 0.59460    |\n| CautiousGiraffe     | 0.601883 | 0.63405   | 0.64515    |\n| DeliriousFly        | 0.589313 | 0.58485   | 0.47280    |\n| ElegantMink         | 0.556870 | 0.60000   | 0.70815    |\n| GroovyShrew         | 0.523861 | 0.56550   | 0.53835    |\n| InvincibleJellyfish | 0.396364 | 0.30240   | 0.36060    |\n| JovialSwallow       | 0.501054 | 0.42765   | 0.47085    |\n| LyricalHare         | 0.605622 | 0.59025   | 0.49260    |\n| NiftyGoldfinch      | 0.468248 | 0.48945   | 0.49110    |\n| PleasantMeerkat    | 0.339524 | 0.28320   | 0.28800    |\n| ReflectiveManatee  | 0.877714 | 0.91140   | 0.89430    |\n| SparklingTapir     | 0.754358 | 0.54135   | 0.60420    |\n| TranquilPanther    | 0.500783 | 0.58695   | 0.52470    |\n| UppityFerret       | 0.600439 | 0.63210   | 0.63270    |\n\nI am also upset that Kaggle doesn't respond to any Kaggler requests to comment. Could a Kaggle admin please clarify whether final submissions were evaluated using labels known to be incorrect, and—if so—explain why this decision was considered fair to all competitors? @annkennedy05 @addisonhoward @inversion @sohiermse \n\n# UPDATE\nSohier Dane responded [here][2]\n>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.\n\nThis confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. \n\nThis sets an important precedent. In this competition, known label errors were deemed insufficient to warrant correcting the final evaluation, despite having a measurable impact on private leaderboard rankings.\n\nFor future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n**when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?**\n\nClear guidance on this point would help competitors understand how to respond to similar issues going forward and would strengthen confidence in final leaderboard outcomes. The figure below summarizes the private leaderboard shifts associated with this issue (with more discussion [here][3]).\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Jan-2026/mouse-lb3.png)\n\n\n[1]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/612531#3306573\n[2]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870\n[3]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198",
      "votes": null
    },
    {
      "id": "3378481",
      "postDate": "12/18/2025 10:01:11",
      "content": "<blockquote>\n  <p>Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.</p>\n</blockquote>\n<p>I see no promise to fix anything here.</p>",
      "rawMarkdown": "> Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.\n\nI see no promise to fix anything here.",
      "votes": null
    },
    {
      "id": "3378535",
      "postDate": "12/18/2025 12:48:10",
      "content": "<p>that was my interpretation, too - when i saw no confirmation that the problem was fixed, i assumed they chose not to fix it, and acted accordingly. </p>",
      "rawMarkdown": "that was my interpretation, too - when i saw no confirmation that the problem was fixed, i assumed they chose not to fix it, and acted accordingly.",
      "votes": null
    },
    {
      "id": "3378584",
      "postDate": "12/18/2025 14:58:07",
      "content": "<p>At this rate there are enough changes to get the public xgb notebook to near gold lol We change to lab wise, add post process, and remove adaptable at 25 and get an estimated .472 public? I know we got about .01 from our post process too. </p>",
      "rawMarkdown": "At this rate there are enough changes to get the public xgb notebook to near gold lol We change to lab wise, add post process, and remove adaptable at 25 and get an estimated .472 public? I know we got about .01 from our post process too.",
      "votes": null
    },
    {
      "id": "3378654",
      "postDate": "12/18/2025 15:51:40",
      "content": "<p>Will this cause significant changes to the private LB?</p>",
      "rawMarkdown": "Will this cause significant changes to the private LB?",
      "votes": null
    },
    {
      "id": "3378852",
      "postDate": "12/18/2025 17:31:15",
      "content": "<p>To me, this sounds like an intention and a commitment to solve the problem.</p>",
      "rawMarkdown": "To me, this sounds like an intention and a commitment to solve the problem.",
      "votes": null
    },
    {
      "id": "3379598",
      "postDate": "12/20/2025 08:11:03",
      "content": "<p>Is the metric the average of 15 lab F1 scores? <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> </p>",
      "rawMarkdown": "Is the metric the average of 15 lab F1 scores? @cdeotte",
      "votes": null
    },
    {
      "id": "3381217",
      "postDate": "12/24/2025 03:36:09",
      "content": "<p>Yes. The metric is \"macro lab F1\" which is the same as computing an F1 score for each lab and then averaging the 15 numbers.</p>",
      "rawMarkdown": "Yes. The metric is \"macro lab F1\" which is the same as computing an F1 score for each lab and then averaging the 15 numbers.",
      "votes": null
    },
    {
      "id": "3381222",
      "postDate": "12/24/2025 03:46:29",
      "content": "<p>Hi xuwuu, it <strong>did</strong> (not will) cause significant changes to the private LB. Kaggle computed our private LB score with the wrong ground truths. This caused some teams to go <strong>down</strong> on the private LB and caused other teams to go <strong>up</strong> on the private LB. If the correct ground truths were used then the final results of the competition would have been different. Specifically the final top 5 cash gold positions would be different.</p>",
      "rawMarkdown": "Hi xuwuu, it **did** (not will) cause significant changes to the private LB. Kaggle computed our private LB score with the wrong ground truths. This caused some teams to go **down** on the private LB and caused other teams to go **up** on the private LB. If the correct ground truths were used then the final results of the competition would have been different. Specifically the final top 5 cash gold positions would be different.",
      "votes": null
    },
    {
      "id": "3381241",
      "postDate": "12/24/2025 04:50:45",
      "content": "<p>Got it. I tried removing it, but it had almost no impact.</p>",
      "rawMarkdown": "Got it. I tried removing it, but it had almost no impact.",
      "votes": null
    },
    {
      "id": "3385166",
      "postDate": "01/02/2026 16:43:31",
      "content": "<h1>UPDATE</h1>\n<p>Sohier Dane responded <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870\" target=\"_blank\">here</a></p>\n<blockquote>\n  <p>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.</p>\n</blockquote>\n<p>This confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. </p>\n<p>For future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n<strong>when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?</strong></p>\n<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> <a href=\"https://www.kaggle.com/addisonhoward\" target=\"_blank\">@addisonhoward</a> <a href=\"https://www.kaggle.com/inversion\" target=\"_blank\">@inversion</a> <a href=\"https://www.kaggle.com/sohiermse\" target=\"_blank\">@sohiermse</a></p>",
      "rawMarkdown": "# UPDATE\nSohier Dane responded [here][2]\n>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.\n\nThis confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. \n\nFor future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n**when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?**\n\n@annkennedy05 @addisonhoward @inversion @sohiermse\n\n[2]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3378481,
      "author_name": "ogurtsov",
      "author_url": "",
      "post_date": "12/18/2025 10:01:11",
      "content": "<blockquote>\n  <p>Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.</p>\n</blockquote>\n<p>I see no promise to fix anything here.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3378535,
          "author_name": "ymatioun",
          "author_url": "",
          "post_date": "12/18/2025 12:48:10",
          "content": "<p>that was my interpretation, too - when i saw no confirmation that the problem was fixed, i assumed they chose not to fix it, and acted accordingly. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3378852,
          "author_name": "titericz",
          "author_url": "",
          "post_date": "12/18/2025 17:31:15",
          "content": "<p>To me, this sounds like an intention and a commitment to solve the problem.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3378584,
      "author_name": "cody11null",
      "author_url": "",
      "post_date": "12/18/2025 14:58:07",
      "content": "<p>At this rate there are enough changes to get the public xgb notebook to near gold lol We change to lab wise, add post process, and remove adaptable at 25 and get an estimated .472 public? I know we got about .01 from our post process too. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3378654,
      "author_name": "lovemaid",
      "author_url": "",
      "post_date": "12/18/2025 15:51:40",
      "content": "<p>Will this cause significant changes to the private LB?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3381222,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "12/24/2025 03:46:29",
          "content": "<p>Hi xuwuu, it <strong>did</strong> (not will) cause significant changes to the private LB. Kaggle computed our private LB score with the wrong ground truths. This caused some teams to go <strong>down</strong> on the private LB and caused other teams to go <strong>up</strong> on the private LB. If the correct ground truths were used then the final results of the competition would have been different. Specifically the final top 5 cash gold positions would be different.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3381241,
              "author_name": "lovemaid",
              "author_url": "",
              "post_date": "12/24/2025 04:50:45",
              "content": "<p>Got it. I tried removing it, but it had almost no impact.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3379598,
      "author_name": "navneetbende",
      "author_url": "",
      "post_date": "12/20/2025 08:11:03",
      "content": "<p>Is the metric the average of 15 lab F1 scores? <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 3381217,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "12/24/2025 03:36:09",
          "content": "<p>Yes. The metric is \"macro lab F1\" which is the same as computing an F1 score for each lab and then averaging the 15 numbers.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3385166,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "01/02/2026 16:43:31",
      "content": "<h1>UPDATE</h1>\n<p>Sohier Dane responded <a href=\"https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870\" target=\"_blank\">here</a></p>\n<blockquote>\n  <p>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.</p>\n</blockquote>\n<p>This confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. </p>\n<p>For future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n<strong>when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?</strong></p>\n<p><a href=\"https://www.kaggle.com/annkennedy05\" target=\"_blank\">@annkennedy05</a> <a href=\"https://www.kaggle.com/addisonhoward\" target=\"_blank\">@addisonhoward</a> <a href=\"https://www.kaggle.com/inversion\" target=\"_blank\">@inversion</a> <a href=\"https://www.kaggle.com/sohiermse\" target=\"_blank\">@sohiermse</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3378395": "By now, it appears that the private leaderboard will not be updated, despite the host stating [here][1] that the private LB would be corrected:\n> Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.\n\nThe strange decision to evaluate models with wrong labels has a huge impact on the final rankings. Kagglers who trusted the host's statement—and reasonably expected the evaluation to use corrected labels—experienced worse private leaderboard outcomes, while approaches that avoided or altered predictions on the affected data achieved better private leaderboard results. Personally I am very upset and disappointed with Kaggle.\n\nThe competition metric is the average of 15 lab F1 scores. The private LB has only two AdaptableSnail videos (discovered via probing). Therefore one incorrect AdaptableSnail video has a large impact on the final private LB score and disproportionately affects final rankings. Below are my per lab LB scores (discovered via probing). We can see that the private LB did not fix the incorrect AdaptableSnail video:\n\n| lab_id              | CV       | Public LB | Private LB |\n|---------------------|----------|-----------|------------|\n| AdaptableSnail      | 0.541807 | 0.51000   | **0.12810**    |\n| BoisterousParrot    | 0.667919 | 0.75870   | 0.59460    |\n| CautiousGiraffe     | 0.601883 | 0.63405   | 0.64515    |\n| DeliriousFly        | 0.589313 | 0.58485   | 0.47280    |\n| ElegantMink         | 0.556870 | 0.60000   | 0.70815    |\n| GroovyShrew         | 0.523861 | 0.56550   | 0.53835    |\n| InvincibleJellyfish | 0.396364 | 0.30240   | 0.36060    |\n| JovialSwallow       | 0.501054 | 0.42765   | 0.47085    |\n| LyricalHare         | 0.605622 | 0.59025   | 0.49260    |\n| NiftyGoldfinch      | 0.468248 | 0.48945   | 0.49110    |\n| PleasantMeerkat    | 0.339524 | 0.28320   | 0.28800    |\n| ReflectiveManatee  | 0.877714 | 0.91140   | 0.89430    |\n| SparklingTapir     | 0.754358 | 0.54135   | 0.60420    |\n| TranquilPanther    | 0.500783 | 0.58695   | 0.52470    |\n| UppityFerret       | 0.600439 | 0.63210   | 0.63270    |\n\nI am also upset that Kaggle doesn't respond to any Kaggler requests to comment. Could a Kaggle admin please clarify whether final submissions were evaluated using labels known to be incorrect, and—if so—explain why this decision was considered fair to all competitors? @annkennedy05 @addisonhoward @inversion @sohiermse \n\n# UPDATE\nSohier Dane responded [here][2]\n>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.\n\nThis confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. \n\nThis sets an important precedent. In this competition, known label errors were deemed insufficient to warrant correcting the final evaluation, despite having a measurable impact on private leaderboard rankings.\n\nFor future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n**when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?**\n\nClear guidance on this point would help competitors understand how to respond to similar issues going forward and would strengthen confidence in final leaderboard outcomes. The figure below summarizes the private leaderboard shifts associated with this issue (with more discussion [here][3]).\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Jan-2026/mouse-lb3.png)\n\n\n[1]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/612531#3306573\n[2]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870\n[3]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198",
    "3378481": "> Not yet, I'm consulting with Kaggle on how/when to most effectively make this fix.\n\nI see no promise to fix anything here.",
    "3378535": "that was my interpretation, too - when i saw no confirmation that the problem was fixed, i assumed they chose not to fix it, and acted accordingly.",
    "3378584": "At this rate there are enough changes to get the public xgb notebook to near gold lol We change to lab wise, add post process, and remove adaptable at 25 and get an estimated .472 public? I know we got about .01 from our post process too.",
    "3378654": "Will this cause significant changes to the private LB?",
    "3378852": "To me, this sounds like an intention and a commitment to solve the problem.",
    "3379598": "Is the metric the average of 15 lab F1 scores? @cdeotte",
    "3381217": "Yes. The metric is \"macro lab F1\" which is the same as computing an F1 score for each lab and then averaging the 15 numbers.",
    "3381222": "Hi xuwuu, it **did** (not will) cause significant changes to the private LB. Kaggle computed our private LB score with the wrong ground truths. This caused some teams to go **down** on the private LB and caused other teams to go **up** on the private LB. If the correct ground truths were used then the final results of the competition would have been different. Specifically the final top 5 cash gold positions would be different.",
    "3381241": "Got it. I tried removing it, but it had almost no impact.",
    "3385166": "# UPDATE\nSohier Dane responded [here][2]\n>I made the call that this data issue was below the threshold for requiring a correction. I've heard your concerns and will take this feedback under consideration the next time a similar decision arises.\n\nThis confirms that the private leaderboard was finalized using labels known to be incorrect, and that no correction would be applied. \n\nFor future competitions, it would be helpful for Kaggle to clarify expectations in such situations:\n**when incorrect labels are known to exist in a private test set, should competitors expect final evaluations to be corrected, or should they assume results will stand as-is?**\n\n@annkennedy05 @addisonhoward @inversion @sohiermse\n\n[2]: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/discussion/663198#3378870"
  },
  "source": "meta"
}