{ "id": 454071, "title": "Update on Evaluation Metric Bug", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/454071", "author_name": "", "post_date": "2023-11-08T19:02:57.494965100Z", "votes": 38, "comment_count": 11, "views": 0, "content": "

Hi everyone,

\n

As noted in this post there is a bug in the evaluation metric that causes it to ignore series in the test set without any annotated events.

\n

After testing a fixed version of the metric however, we've decided to leave things as they are. While the impact to the scores would have been minor, at this stage we thought it better not cause any disruption to the competition. Effectively this just means that there will continue to be a few \"eventless\" series in the test set that will continue to be ignored.

\n

Please let us know if you have any questions or concerns. And thank you to @aerdem4 for pointing out the bug.

\n", "messages": [ { "id": "2517838", "postDate": "11/08/2023 19:02:57", "content": "

Hi everyone,

\n

As noted in this post there is a bug in the evaluation metric that causes it to ignore series in the test set without any annotated events.

\n

After testing a fixed version of the metric however, we've decided to leave things as they are. While the impact to the scores would have been minor, at this stage we thought it better not cause any disruption to the competition. Effectively this just means that there will continue to be a few \"eventless\" series in the test set that will continue to be ignored.

\n

Please let us know if you have any questions or concerns. And thank you to @aerdem4 for pointing out the bug.

\n", "rawMarkdown": "Hi everyone,\n\nAs noted in [this post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/451950) there is a bug in the evaluation metric that causes it to ignore series in the test set without any annotated events.\n\nAfter testing a fixed version of the metric however, we've decided to leave things as they are. While the impact to the scores would have been minor, at this stage we thought it better not cause any disruption to the competition. Effectively this just means that there will continue to be a few \"eventless\" series in the test set that will continue to be ignored.\n\nPlease let us know if you have any questions or concerns. And thank you to @aerdem4 for pointing out the bug.\n\n- Ryan", "votes": null }, { "id": "2518108", "postDate": "11/09/2023 04:06:21", "content": "

Thank you for the update and clarity!

", "rawMarkdown": "Thank you for the update and clarity!", "votes": null }, { "id": "2522558", "postDate": "11/12/2023 19:31:10", "content": "

I think this is better for the cross validation.

", "rawMarkdown": "I think this is better for the cross validation.", "votes": null }, { "id": "2523659", "postDate": "11/13/2023 16:08:01", "content": "

Dear @ryanholbrook
\nThank you for the update! Can you clarify what is \"eventless\" series?

", "rawMarkdown": "Dear @ryanholbrook \nThank you for the update! Can you clarify what is \"eventless\" series?", "votes": null }, { "id": "2523669", "postDate": "11/13/2023 16:13:44", "content": "

Hi @luanhuynh, It just means that neither an onset nor a wakeup event occurs within the series. Such a series has nothing to detect, in other words.

", "rawMarkdown": "Hi @luanhuynh, It just means that neither an `onset` nor a `wakeup` event occurs within the series. Such a series has nothing to detect, in other words.", "votes": null }, { "id": "2524111", "postDate": "11/14/2023 02:09:18", "content": "

Hi, I hit the following error:

\n
srevent-detection-ap/metric.py in ?(solution, submission, tolerances, series_id_column_name, time_column_name, event_column_name, score_column_name, use_scoring_intervals)\n         globals()[] = event_column_name\n         globals()[] = score_column_name\n         globals()[] = use_scoring_intervals\n     \n-->       event_detection_ap(solution, submission, tolerances)\n\nsrevent-detection-ap/metric.py in ?(solution, submission, tolerances)\n                 )\n             )\n         )\n         # Average over tolerances, then over event classes\n-->      mean_ap = ap_table.groupby(event_column_name).mean().sum() / len(event_classes)\n     \n          mean_ap\n\ncondapython3.pandasframe.py in ?(self, by, axis, level, as_index, , group_keys, observed, dropna)\n             level is None and by is None:\n                raise TypeError()\n            axis = self._get_axis_number(axis)\n    \n->           DataFrameGroupBy(\n                obj=self,\n                keys=by,\n                axis=axis,\n\ncondapython3.pandasroupby/groupby.py in ?(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, , group_keys, observed, dropna)\n             self.observed = observed\n             self.dropna = dropna\n     \n              grouper is None:\n-->              grouper, exclusions, obj = get_grouper(\n                     obj,\n                     keys,\n                     axis=axis,\n\ncondapython3.pandasroupby/grouper.py in ?(obj, key, axis, level, , observed, validate, dropna)\n     \n             elif is_in_axis(gpr):  # df.groupby()\n                  obj.ndim !=  and gpr in obj:\n                      validate:\n-->                      obj._check_label_or_level_ambiguity(gpr, axis=axis)\n                     in_axis, name, gpr = , gpr, obj[gpr]\n                      gpr.ndim != :\n                         # non-unique columns; raise here to get the name in the\n\ncondapython3.pandaseneric.py in ?(self, key, axis)\n                msg = (\n                    f\n                    f\n                )\n->              raise ValueError(msg)\n\nValueError:  is both an index level and a column label, which is ambiguous.\n
", "rawMarkdown": "Hi, I hit the following error:\n\n```\n/kaggle/usr/lib/event-detection-ap/metric.py in ?(solution, submission, tolerances, series_id_column_name, time_column_name, event_column_name, score_column_name, use_scoring_intervals)\n 252 globals()['event_column_name'] = event_column_name\n 253 globals()['score_column_name'] = score_column_name\n 254 globals()['use_scoring_intervals'] = use_scoring_intervals\n 255 \n--> 256 return event_detection_ap(solution, submission, tolerances)\n\n/kaggle/usr/lib/event-detection-ap/metric.py in ?(solution, submission, tolerances)\n 442 )\n 443 )\n 444 )\n 445 # Average over tolerances, then over event classes\n--> 446 mean_ap = ap_table.groupby(event_column_name).mean().sum() / len(event_classes)\n 447 \n 448 return mean_ap\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/frame.py in ?(self, by, axis, level, as_index, sort, group_keys, observed, dropna)\n 8248 if level is None and by is None:\n 8249 raise TypeError(\"You have to supply one of 'by' and 'level'\")\n 8250 axis = self._get_axis_number(axis)\n 8251 \n-> 8252 return DataFrameGroupBy(\n 8253 obj=self,\n 8254 keys=by,\n 8255 axis=axis,\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/groupby/groupby.py in ?(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, sort, group_keys, observed, dropna)\n 927 self.observed = observed\n 928 self.dropna = dropna\n 929 \n 930 if grouper is None:\n--> 931 grouper, exclusions, obj = get_grouper(\n 932 obj,\n 933 keys,\n 934 axis=axis,\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/groupby/grouper.py in ?(obj, key, axis, level, sort, observed, validate, dropna)\n 971 \n 972 elif is_in_axis(gpr): # df.groupby('name')\n 973 if obj.ndim != 1 and gpr in obj:\n 974 if validate:\n--> 975 obj._check_label_or_level_ambiguity(gpr, axis=axis)\n 976 in_axis, name, gpr = True, gpr, obj[gpr]\n 977 if gpr.ndim != 1:\n 978 # non-unique columns; raise here to get the name in the\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/generic.py in ?(self, key, axis)\n 1731 msg = (\n 1732 f\"'{key}' is both {level_article} {level_type} level and \"\n 1733 f\"{label_article} {label_type} label, which is ambiguous.\"\n 1734 )\n-> 1735 raise ValueError(msg)\n\nValueError: 'event' is both an index level and a column label, which is ambiguous.\n```", "votes": null }, { "id": "2524143", "postDate": "11/14/2023 03:01:47", "content": "

It appears that the name of your dataframe's index (of your predictions dataframe) is the same as one of the columns. Remove the prediction dataframe's index name with

\n
 = preds.rename_axis()\n
\n

Before you input it into the metric score function.

", "rawMarkdown": "It appears that the name of your dataframe's index (of your predictions dataframe) is the same as one of the columns. Remove the prediction dataframe's index name with\n\n preds = preds.rename_axis('')\n\nBefore you input it into the metric score function.", "votes": null }, { "id": "2524158", "postDate": "11/14/2023 03:20:00", "content": "

Thanks for the reply. I only set the index to row_id, didnt set it to event

", "rawMarkdown": "Thanks for the reply. I only set the index to row_id, didnt set it to event", "votes": null }, { "id": "2524172", "postDate": "11/14/2023 03:45:30", "content": "

Then maybe it is the index of your ground truth dataframe. Try true = true.rename_axis(''). We call score by giving it two dataframes, the error your are receiving sounds like one of those dataframes has a column called event and the index named event.

", "rawMarkdown": "Then maybe it is the index of your ground truth dataframe. Try `true = true.rename_axis('')`. We call `score` by giving it two dataframes, the error your are receiving sounds like one of those dataframes has a column called `event` and the index named `event`.", "votes": null }, { "id": "2524187", "postDate": "11/14/2023 04:01:00", "content": "

Hi thanks for the reply. No I dont think i saw or set any \"event\" as index.

\n

My Submission :

\n
    submission_df = pd()\n    submission_df  = series_ids\n    submission_df  = steps\n    submission_df = submission_df()\n    submission_df  = events\n    submission_df  = scores\n    submission_df = submission_df\n    row_ids = submission_df()\n    submission_df(, , row_ids)\n
\n

My ground truth:

\n
def (series_id):\n    df = pd.().() \n    return df\n
", "rawMarkdown": "Hi thanks for the reply. No I dont think i saw or set any \"event\" as index.\n\nMy Submission :\n```\n submission_df = pd.DataFrame()\n submission_df[\"series_id\"] = series_ids\n submission_df[\"step\"] = steps\n submission_df[\"step\"] = submission_df[\"step\"].astype(\"int32\")\n submission_df[\"event\"] = events\n submission_df[\"score\"] = scores\n submission_df[\"row_id\"] = submission_df.index\n row_ids = submission_df.pop(\"row_id\")\n submission_df.insert(0, \"row_id\", row_ids)\n```\n\nMy ground truth:\n\n```\ndef get_train_events(series_id):\n df = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\").query('series_id == @series_id') \n return df\n\n```", "votes": null }, { "id": "2524708", "postDate": "11/14/2023 13:27:24", "content": "

Hi @cdeotte @ryanholbrook I fork the event metric https://www.kaggle.com/code/rubyhartono/event-detection-ap and add display before the mean_ap

\n
    \n
\n

and it displayed as per attached.

", "rawMarkdown": "Hi @cdeotte @ryanholbrook I fork the event metric https://www.kaggle.com/code/rubyhartono/event-detection-ap and add display before the mean_ap\n\n```\n display(ap_table)\n```\n\nand it displayed as per attached.", "votes": null }, { "id": "2524787", "postDate": "11/14/2023 14:30:40", "content": "

Sorry I think it works, There is a bug in my submission script causes it to return empty df

", "rawMarkdown": "Sorry I think it works, There is a bug in my submission script causes it to return empty df", "votes": null } ], "comments": [ { "id": 2518108, "author_name": "cody11null", "author_url": "", "post_date": "11/09/2023 04:06:21", "content": "

Thank you for the update and clarity!

", "votes": null, "replies": [] }, { "id": 2522558, "author_name": "tolgadincer", "author_url": "", "post_date": "11/12/2023 19:31:10", "content": "

I think this is better for the cross validation.

", "votes": null, "replies": [] }, { "id": 2523659, "author_name": "luanhuynh", "author_url": "", "post_date": "11/13/2023 16:08:01", "content": "

Dear @ryanholbrook
\nThank you for the update! Can you clarify what is \"eventless\" series?

", "votes": null, "replies": [ { "id": 2523669, "author_name": "ryanholbrook", "author_url": "", "post_date": "11/13/2023 16:13:44", "content": "

Hi @luanhuynh, It just means that neither an onset nor a wakeup event occurs within the series. Such a series has nothing to detect, in other words.

", "votes": null, "replies": [] } ] }, { "id": 2524111, "author_name": "rubyhartono", "author_url": "", "post_date": "11/14/2023 02:09:18", "content": "

Hi, I hit the following error:

\n
srevent-detection-ap/metric.py in ?(solution, submission, tolerances, series_id_column_name, time_column_name, event_column_name, score_column_name, use_scoring_intervals)\n         globals()[] = event_column_name\n         globals()[] = score_column_name\n         globals()[] = use_scoring_intervals\n     \n-->       event_detection_ap(solution, submission, tolerances)\n\nsrevent-detection-ap/metric.py in ?(solution, submission, tolerances)\n                 )\n             )\n         )\n         # Average over tolerances, then over event classes\n-->      mean_ap = ap_table.groupby(event_column_name).mean().sum() / len(event_classes)\n     \n          mean_ap\n\ncondapython3.pandasframe.py in ?(self, by, axis, level, as_index, , group_keys, observed, dropna)\n             level is None and by is None:\n                raise TypeError()\n            axis = self._get_axis_number(axis)\n    \n->           DataFrameGroupBy(\n                obj=self,\n                keys=by,\n                axis=axis,\n\ncondapython3.pandasroupby/groupby.py in ?(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, , group_keys, observed, dropna)\n             self.observed = observed\n             self.dropna = dropna\n     \n              grouper is None:\n-->              grouper, exclusions, obj = get_grouper(\n                     obj,\n                     keys,\n                     axis=axis,\n\ncondapython3.pandasroupby/grouper.py in ?(obj, key, axis, level, , observed, validate, dropna)\n     \n             elif is_in_axis(gpr):  # df.groupby()\n                  obj.ndim !=  and gpr in obj:\n                      validate:\n-->                      obj._check_label_or_level_ambiguity(gpr, axis=axis)\n                     in_axis, name, gpr = , gpr, obj[gpr]\n                      gpr.ndim != :\n                         # non-unique columns; raise here to get the name in the\n\ncondapython3.pandaseneric.py in ?(self, key, axis)\n                msg = (\n                    f\n                    f\n                )\n->              raise ValueError(msg)\n\nValueError:  is both an index level and a column label, which is ambiguous.\n
", "votes": null, "replies": [ { "id": 2524143, "author_name": "cdeotte", "author_url": "", "post_date": "11/14/2023 03:01:47", "content": "

It appears that the name of your dataframe's index (of your predictions dataframe) is the same as one of the columns. Remove the prediction dataframe's index name with

\n
 = preds.rename_axis()\n
\n

Before you input it into the metric score function.

", "votes": null, "replies": [ { "id": 2524158, "author_name": "rubyhartono", "author_url": "", "post_date": "11/14/2023 03:20:00", "content": "

Thanks for the reply. I only set the index to row_id, didnt set it to event

", "votes": null, "replies": [ { "id": 2524172, "author_name": "cdeotte", "author_url": "", "post_date": "11/14/2023 03:45:30", "content": "

Then maybe it is the index of your ground truth dataframe. Try true = true.rename_axis(''). We call score by giving it two dataframes, the error your are receiving sounds like one of those dataframes has a column called event and the index named event.

", "votes": null, "replies": [ { "id": 2524187, "author_name": "rubyhartono", "author_url": "", "post_date": "11/14/2023 04:01:00", "content": "

Hi thanks for the reply. No I dont think i saw or set any \"event\" as index.

\n

My Submission :

\n
    submission_df = pd()\n    submission_df  = series_ids\n    submission_df  = steps\n    submission_df = submission_df()\n    submission_df  = events\n    submission_df  = scores\n    submission_df = submission_df\n    row_ids = submission_df()\n    submission_df(, , row_ids)\n
\n

My ground truth:

\n
def (series_id):\n    df = pd.().() \n    return df\n
", "votes": null, "replies": [ { "id": 2524708, "author_name": "rubyhartono", "author_url": "", "post_date": "11/14/2023 13:27:24", "content": "

Hi @cdeotte @ryanholbrook I fork the event metric https://www.kaggle.com/code/rubyhartono/event-detection-ap and add display before the mean_ap

\n
    \n
\n

and it displayed as per attached.

", "votes": null, "replies": [ { "id": 2524787, "author_name": "rubyhartono", "author_url": "", "post_date": "11/14/2023 14:30:40", "content": "

Sorry I think it works, There is a bug in my submission script causes it to return empty df

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2517838": "Hi everyone,\n\nAs noted in [this post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/451950) there is a bug in the evaluation metric that causes it to ignore series in the test set without any annotated events.\n\nAfter testing a fixed version of the metric however, we've decided to leave things as they are. While the impact to the scores would have been minor, at this stage we thought it better not cause any disruption to the competition. Effectively this just means that there will continue to be a few \"eventless\" series in the test set that will continue to be ignored.\n\nPlease let us know if you have any questions or concerns. And thank you to @aerdem4 for pointing out the bug.\n\n- Ryan", "2518108": "Thank you for the update and clarity!", "2522558": "I think this is better for the cross validation.", "2523659": "Dear @ryanholbrook \nThank you for the update! Can you clarify what is \"eventless\" series?", "2523669": "Hi @luanhuynh, It just means that neither an `onset` nor a `wakeup` event occurs within the series. Such a series has nothing to detect, in other words.", "2524111": "Hi, I hit the following error:\n\n```\n/kaggle/usr/lib/event-detection-ap/metric.py in ?(solution, submission, tolerances, series_id_column_name, time_column_name, event_column_name, score_column_name, use_scoring_intervals)\n 252 globals()['event_column_name'] = event_column_name\n 253 globals()['score_column_name'] = score_column_name\n 254 globals()['use_scoring_intervals'] = use_scoring_intervals\n 255 \n--> 256 return event_detection_ap(solution, submission, tolerances)\n\n/kaggle/usr/lib/event-detection-ap/metric.py in ?(solution, submission, tolerances)\n 442 )\n 443 )\n 444 )\n 445 # Average over tolerances, then over event classes\n--> 446 mean_ap = ap_table.groupby(event_column_name).mean().sum() / len(event_classes)\n 447 \n 448 return mean_ap\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/frame.py in ?(self, by, axis, level, as_index, sort, group_keys, observed, dropna)\n 8248 if level is None and by is None:\n 8249 raise TypeError(\"You have to supply one of 'by' and 'level'\")\n 8250 axis = self._get_axis_number(axis)\n 8251 \n-> 8252 return DataFrameGroupBy(\n 8253 obj=self,\n 8254 keys=by,\n 8255 axis=axis,\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/groupby/groupby.py in ?(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, sort, group_keys, observed, dropna)\n 927 self.observed = observed\n 928 self.dropna = dropna\n 929 \n 930 if grouper is None:\n--> 931 grouper, exclusions, obj = get_grouper(\n 932 obj,\n 933 keys,\n 934 axis=axis,\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/groupby/grouper.py in ?(obj, key, axis, level, sort, observed, validate, dropna)\n 971 \n 972 elif is_in_axis(gpr): # df.groupby('name')\n 973 if obj.ndim != 1 and gpr in obj:\n 974 if validate:\n--> 975 obj._check_label_or_level_ambiguity(gpr, axis=axis)\n 976 in_axis, name, gpr = True, gpr, obj[gpr]\n 977 if gpr.ndim != 1:\n 978 # non-unique columns; raise here to get the name in the\n\n/opt/conda/lib/python3.10/site-packages/pandas/core/generic.py in ?(self, key, axis)\n 1731 msg = (\n 1732 f\"'{key}' is both {level_article} {level_type} level and \"\n 1733 f\"{label_article} {label_type} label, which is ambiguous.\"\n 1734 )\n-> 1735 raise ValueError(msg)\n\nValueError: 'event' is both an index level and a column label, which is ambiguous.\n```", "2524143": "It appears that the name of your dataframe's index (of your predictions dataframe) is the same as one of the columns. Remove the prediction dataframe's index name with\n\n preds = preds.rename_axis('')\n\nBefore you input it into the metric score function.", "2524158": "Thanks for the reply. I only set the index to row_id, didnt set it to event", "2524172": "Then maybe it is the index of your ground truth dataframe. Try `true = true.rename_axis('')`. We call `score` by giving it two dataframes, the error your are receiving sounds like one of those dataframes has a column called `event` and the index named `event`.", "2524187": "Hi thanks for the reply. No I dont think i saw or set any \"event\" as index.\n\nMy Submission :\n```\n submission_df = pd.DataFrame()\n submission_df[\"series_id\"] = series_ids\n submission_df[\"step\"] = steps\n submission_df[\"step\"] = submission_df[\"step\"].astype(\"int32\")\n submission_df[\"event\"] = events\n submission_df[\"score\"] = scores\n submission_df[\"row_id\"] = submission_df.index\n row_ids = submission_df.pop(\"row_id\")\n submission_df.insert(0, \"row_id\", row_ids)\n```\n\nMy ground truth:\n\n```\ndef get_train_events(series_id):\n df = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\").query('series_id == @series_id') \n return df\n\n```", "2524708": "Hi @cdeotte @ryanholbrook I fork the event metric https://www.kaggle.com/code/rubyhartono/event-detection-ap and add display before the mean_ap\n\n```\n display(ap_table)\n```\n\nand it displayed as per attached.", "2524787": "Sorry I think it works, There is a bug in my submission script causes it to return empty df" }, "source": "meta" }