Dear host @ozzymet
\nI think the used evaluation metric is wrong / not in concordance with the description.
\nThe description mentions the use of the macro F1 score (which would make sense due to the high class imbalance). However, after seeing that an all N submission scores 0.93840, I am certain that this is the result of (incorrectly) using the micro F1 score (which is quite susceptible to a high class imbalance).
To verify this, one can use the following code snippet and see the expected score of an all N submission for the micro or macro F1 score:
numpy np\n sklearn.metrics f1_score\n matplotlib.pyplot plt\n\nscores = []\n _ ():\n y_true = np.random.choice([, , ], size=, p=[, , ])\n y_pred = [] * (y_true)\n scores.append(f1_score(y_true, y_pred, average=))\nplt.hist(scores, bins=)\nplt.title()\nplt.xlabel()\nplt.ylabel()\nplt.show()\nThe expected micro F1 score of an all N submission is around 0.94, which is in concordance with the observed LB behaviour. A macro F1 score would be expected to be around 0.33, which is much more sensible.
Please check this and maybe change the used evaluation metric.
", "messages": [ { "id": 3295329, "postDate": "2025-09-28T12:42:11.623Z", "content": "The host should address the current problem because, as things stand, the leaderboard does not make sense: the top scores (all around 0.9384) are from submissions that predict only the 'N' label everywhere. This defeats the purpose of the competition, as it does not actually identify 'H' or 'L' points. The host should discard or penalize submissions that contain only 'N' labels, otherwise there is no meaningful competition.
", "rawMarkdown": "The host should address the current problem because, as things stand, the leaderboard does not make sense: the top scores (all around 0.9384) are from submissions that predict only the 'N' label everywhere. This defeats the purpose of the competition, as it does not actually identify 'H' or 'L' points. The host should discard or penalize submissions that contain only 'N' labels, otherwise there is no meaningful competition.", "votes": 1, "replies": [ { "id": 3295518, "postDate": "2025-09-29T02:33:29.240Z", "content": "I think the host has already abandoned this competition, or maybe they simply created it just to see if anyone could come up with a better solution than what they already had. Even in the data description, they didn’t explain anything about the features and only mentioned what the labels were, so it seems they weren’t really that invested in this competition. I also tried searching and found that there are many events with a True value, which I don’t really understand why.
", "rawMarkdown": "I think the host has already abandoned this competition, or maybe they simply created it just to see if anyone could come up with a better solution than what they already had. Even in the data description, they didn’t explain anything about the features and only mentioned what the labels were, so it seems they weren’t really that invested in this competition. I also tried searching and found that there are many events with a True value, which I don’t really understand why." } ] }, { "id": 3294110, "postDate": "2025-09-25T11:29:03.123Z", "content": "Dear host @ozzymet
\nI think the used evaluation metric is wrong / not in concordance with the description.
\nThe description mentions the use of the macro F1 score (which would make sense due to the high class imbalance). However, after seeing that an all N submission scores 0.93840, I am certain that this is the result of (incorrectly) using the micro F1 score (which is quite susceptible to a high class imbalance).
To verify this, one can use the following code snippet and see the expected score of an all N submission for the micro or macro F1 score:
numpy np\n sklearn.metrics f1_score\n matplotlib.pyplot plt\n\nscores = []\n _ ():\n y_true = np.random.choice([, , ], size=, p=[, , ])\n y_pred = [] * (y_true)\n scores.append(f1_score(y_true, y_pred, average=))\nplt.hist(scores, bins=)\nplt.title()\nplt.xlabel()\nplt.ylabel()\nplt.show()\nThe expected micro F1 score of an all N submission is around 0.94, which is in concordance with the observed LB behaviour. A macro F1 score would be expected to be around 0.33, which is much more sensible.
Please check this and maybe change the used evaluation metric.
", "rawMarkdown": "Dear host @ozzymet \n\nI think the used evaluation metric is wrong / not in concordance with the description.\nThe description mentions the use of the macro F1 score (which would make sense due to the high class imbalance). However, after seeing that an all `N` submission scores 0.93840, I am certain that this is the result of (incorrectly) using the micro F1 score (which is quite susceptible to a high class imbalance).\n\nTo verify this, one can use the following code snippet and see the expected score of an all `N` submission for the micro or macro F1 score:\n```py\nimport numpy as np\nfrom sklearn.metrics import f1_score\nimport matplotlib.pyplot as plt\n\nscores = []\nfor _ in range(10000):\n y_true = np.random.choice([0, 1, 2], size=828, p=[0.03, 0.029, 0.941])\n y_pred = [2] * len(y_true)\n scores.append(f1_score(y_true, y_pred, average=\"micro\"))\nplt.hist(scores, bins=10)\nplt.title(\"Histogram of F1 Scores\")\nplt.xlabel(\"F1 Score\")\nplt.ylabel(\"Frequency\")\nplt.show()\n```\nThe expected micro F1 score of an all `N` submission is around 0.94, which is in concordance with the observed LB behaviour. A macro F1 score would be expected to be around 0.33, which is much more sensible.\n\nPlease check this and maybe change the used evaluation metric.", "votes": 2 }, { "id": 3294234, "postDate": "2025-09-25T16:25:41.567Z", "content": "I think you are right
", "rawMarkdown": "I think you are right", "isDeleted": true } ], "comments": [ { "id": 3295329, "author_name": "Prakash Nandaniya", "author_url": "", "post_date": "2025-09-28T12:42:11.623000", "content": "The host should address the current problem because, as things stand, the leaderboard does not make sense: the top scores (all around 0.9384) are from submissions that predict only the 'N' label everywhere. This defeats the purpose of the competition, as it does not actually identify 'H' or 'L' points. The host should discard or penalize submissions that contain only 'N' labels, otherwise there is no meaningful competition.
", "votes": 1, "replies": [ { "id": 3295518, "author_name": "Sam Bui", "author_url": "", "post_date": "2025-09-29T02:33:29.240000", "content": "I think the host has already abandoned this competition, or maybe they simply created it just to see if anyone could come up with a better solution than what they already had. Even in the data description, they didn’t explain anything about the features and only mentioned what the labels were, so it seems they weren’t really that invested in this competition. I also tried searching and found that there are many events with a True value, which I don’t really understand why.
", "votes": 0, "replies": [] } ] }, { "id": 3294234, "author_name": "", "author_url": "", "post_date": "2025-09-25T16:25:41.567000", "content": "I think you are right
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3295329": "The host should address the current problem because, as things stand, the leaderboard does not make sense: the top scores (all around 0.9384) are from submissions that predict only the 'N' label everywhere. This defeats the purpose of the competition, as it does not actually identify 'H' or 'L' points. The host should discard or penalize submissions that contain only 'N' labels, otherwise there is no meaningful competition.", "3294110": "Dear host @ozzymet \n\nI think the used evaluation metric is wrong / not in concordance with the description.\nThe description mentions the use of the macro F1 score (which would make sense due to the high class imbalance). However, after seeing that an all `N` submission scores 0.93840, I am certain that this is the result of (incorrectly) using the micro F1 score (which is quite susceptible to a high class imbalance).\n\nTo verify this, one can use the following code snippet and see the expected score of an all `N` submission for the micro or macro F1 score:\n```py\nimport numpy as np\nfrom sklearn.metrics import f1_score\nimport matplotlib.pyplot as plt\n\nscores = []\nfor _ in range(10000):\n y_true = np.random.choice([0, 1, 2], size=828, p=[0.03, 0.029, 0.941])\n y_pred = [2] * len(y_true)\n scores.append(f1_score(y_true, y_pred, average=\"micro\"))\nplt.hist(scores, bins=10)\nplt.title(\"Histogram of F1 Scores\")\nplt.xlabel(\"F1 Score\")\nplt.ylabel(\"Frequency\")\nplt.show()\n```\nThe expected micro F1 score of an all `N` submission is around 0.94, which is in concordance with the observed LB behaviour. A macro F1 score would be expected to be around 0.33, which is much more sensible.\n\nPlease check this and maybe change the used evaluation metric.", "3294234": "I think you are right" } }