{
  "id": 571713,
  "title": "Concerns About Leaderboard Integrity",
  "url": "/competitions/el-hackathon-2025/discussion/571713",
  "author_name": "g john rao",
  "post_date": "2025-04-05T04:52:32.006000",
  "votes": 11,
  "comment_count": 2,
  "views": 0,
  "content": "<p>The current competition structure has a serious vulnerability. Since public leaderboard coordinates are known, participants can easily probe and optimize for the private leaderboard distribution, which undermines the competition's legitimacy.</p>\n<blockquote>\n  <p>The test slide has been divided into a public test set and a private test set. The aggregated score will be computed on the two groups separately. The cellular profiles of these two groups are weakly correlated, so performance on the public set would not necessarily reflect the projected performance on the private test set. </p>\n</blockquote>\n<ul>\n<li>Test <code>slide S_7</code> has both public and private test spots clearly identified  </li>\n<li>The \"weakly correlated\" nature between sets allows for systematic exploitation  </li>\n<li>This rewards optimization for specific test points rather than developing truly generalizable solutions</li>\n</ul>\n<p>This setup effectively negates the purpose of having a private leaderboard for final evaluation. </p>\n<blockquote>\n  <p>The identity of each test spot (i.e. whether it falls in the public or private set) is provided along with the spot 2D coordinates in the Challenge data .h5 file.</p>\n</blockquote>\n<p>Competition organizers should reconsider this approach to ensure winners are selected based on scientific merit rather than leaderboard gaming techniques. </p>\n<p>One way to solve this is to use a completely separate, undisclosed slide for final evaluation without disclosing any info about the test set. </p>",
  "messages": [
    {
      "id": 3170865,
      "postDate": "2025-04-05T04:52:32.007Z",
      "content": "<p>The current competition structure has a serious vulnerability. Since public leaderboard coordinates are known, participants can easily probe and optimize for the private leaderboard distribution, which undermines the competition's legitimacy.</p>\n<blockquote>\n  <p>The test slide has been divided into a public test set and a private test set. The aggregated score will be computed on the two groups separately. The cellular profiles of these two groups are weakly correlated, so performance on the public set would not necessarily reflect the projected performance on the private test set. </p>\n</blockquote>\n<ul>\n<li>Test <code>slide S_7</code> has both public and private test spots clearly identified  </li>\n<li>The \"weakly correlated\" nature between sets allows for systematic exploitation  </li>\n<li>This rewards optimization for specific test points rather than developing truly generalizable solutions</li>\n</ul>\n<p>This setup effectively negates the purpose of having a private leaderboard for final evaluation. </p>\n<blockquote>\n  <p>The identity of each test spot (i.e. whether it falls in the public or private set) is provided along with the spot 2D coordinates in the Challenge data .h5 file.</p>\n</blockquote>\n<p>Competition organizers should reconsider this approach to ensure winners are selected based on scientific merit rather than leaderboard gaming techniques. </p>\n<p>One way to solve this is to use a completely separate, undisclosed slide for final evaluation without disclosing any info about the test set. </p>",
      "rawMarkdown": "The current competition structure has a serious vulnerability. Since public leaderboard coordinates are known, participants can easily probe and optimize for the private leaderboard distribution, which undermines the competition's legitimacy.\n\n> The test slide has been divided into a public test set and a private test set. The aggregated score will be computed on the two groups separately. The cellular profiles of these two groups are weakly correlated, so performance on the public set would not necessarily reflect the projected performance on the private test set. \n\n- Test `slide S_7` has both public and private test spots clearly identified  \n- The \"weakly correlated\" nature between sets allows for systematic exploitation  \n- This rewards optimization for specific test points rather than developing truly generalizable solutions\n\nThis setup effectively negates the purpose of having a private leaderboard for final evaluation. \n\n> The identity of each test spot (i.e. whether it falls in the public or private set) is provided along with the spot 2D coordinates in the Challenge data .h5 file.\n\n\nCompetition organizers should reconsider this approach to ensure winners are selected based on scientific merit rather than leaderboard gaming techniques. \n\nOne way to solve this is to use a completely separate, undisclosed slide for final evaluation without disclosing any info about the test set. ",
      "votes": 11
    },
    {
      "id": 3175226,
      "postDate": "2025-04-09T22:22:28.447Z",
      "content": "<p>given the weak correlation between public and private sets ,they are simply spamming submissions to fit the private test distribution.<br>\norganizer should : limit submissions(100 a day is too much)+ FULLY hide private test data +analyze submission patterns</p>",
      "rawMarkdown": "given the weak correlation between public and private sets ,they are simply spamming submissions to fit the private test distribution.\norganizer should : limit submissions(100 a day is too much)+ FULLY hide private test data +analyze submission patterns",
      "votes": 1
    },
    {
      "id": 3172038,
      "postDate": "2025-04-06T11:48:24.927Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 3175226,
      "author_name": "Nadia TRIKI",
      "author_url": "",
      "post_date": "2025-04-09T22:22:28.447000",
      "content": "<p>given the weak correlation between public and private sets ,they are simply spamming submissions to fit the private test distribution.<br>\norganizer should : limit submissions(100 a day is too much)+ FULLY hide private test data +analyze submission patterns</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3172038,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-04-06T11:48:24.927000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3170865": "The current competition structure has a serious vulnerability. Since public leaderboard coordinates are known, participants can easily probe and optimize for the private leaderboard distribution, which undermines the competition's legitimacy.\n\n> The test slide has been divided into a public test set and a private test set. The aggregated score will be computed on the two groups separately. The cellular profiles of these two groups are weakly correlated, so performance on the public set would not necessarily reflect the projected performance on the private test set. \n\n- Test `slide S_7` has both public and private test spots clearly identified  \n- The \"weakly correlated\" nature between sets allows for systematic exploitation  \n- This rewards optimization for specific test points rather than developing truly generalizable solutions\n\nThis setup effectively negates the purpose of having a private leaderboard for final evaluation. \n\n> The identity of each test spot (i.e. whether it falls in the public or private set) is provided along with the spot 2D coordinates in the Challenge data .h5 file.\n\n\nCompetition organizers should reconsider this approach to ensure winners are selected based on scientific merit rather than leaderboard gaming techniques. \n\nOne way to solve this is to use a completely separate, undisclosed slide for final evaluation without disclosing any info about the test set. ",
    "3175226": "given the weak correlation between public and private sets ,they are simply spamming submissions to fit the private test distribution.\norganizer should : limit submissions(100 a day is too much)+ FULLY hide private test data +analyze submission patterns",
    "3172038": ""
  }
}