{ "id": 448878, "title": "Is Score Calculated for Onset/Wakeup Pairs or Either Event Alone?", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/448878", "author_name": "", "post_date": "2023-10-22T03:16:20.841599900Z", "votes": 6, "comment_count": 2, "views": 0, "content": "

This discussion already has an explanation of score in submission, but I am confused about whether it applies to singular events (e.g. onset or wakeup) or event pairs (onset and wakeup)?

\n

The sample submission file seems to have scores that are paired, but many public notebooks do not follow this pattern, which is why I am confused.

", "messages": [ { "id": "2491773", "postDate": "10/22/2023 03:16:20", "content": "

This discussion already has an explanation of score in submission, but I am confused about whether it applies to singular events (e.g. onset or wakeup) or event pairs (onset and wakeup)?

\n

The sample submission file seems to have scores that are paired, but many public notebooks do not follow this pattern, which is why I am confused.

", "rawMarkdown": "[This discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438936) already has an explanation of `score` in submission, but I am confused about whether it applies to singular events (e.g. `onset` **or** `wakeup`) or event pairs (`onset` **and** `wakeup`)?\n\nThe sample submission file seems to have scores that are paired, but many public notebooks do not follow this pattern, which is why I am confused.", "votes": null }, { "id": "2492883", "postDate": "10/23/2023 02:12:16", "content": "

Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation

\n
\n

For each set of predictions and ground-truths within the same event x tolerance x series_id group, we match each ground-truth to the highest-confidence unmatched prediction occurring within the allowed tolerance.

\n
\n

My understanding is that the score is calculated event wise. Also, have seen annotated events in the train dataset that have only one event, either onset or wakeup and not have the pair of event which makes me think predicting the individual events is the ask.

", "rawMarkdown": "Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation\n\n> For each set of predictions and ground-truths within the same `event x tolerance x series_id` group, we match each ground-truth to the highest-confidence unmatched prediction occurring within the allowed tolerance.\n\nMy understanding is that the score is calculated event wise. Also, have seen annotated events in the train dataset that have only one event, either `onset` or `wakeup` and not have the pair of event which makes me think predicting the individual events is the ask.", "votes": null }, { "id": "2492906", "postDate": "10/23/2023 02:42:31", "content": "

Hi @bkowshik, that's interesting about the training data containing single events, but the data card states:

\n
\n

If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.

\n
\n

I guess there could be no contradiction, but this seems to suggest that there shouldn't be an onset event without a following wakeup event and vice versa.

\n

If events are in pairs then this problem seems to be more of a 1D signal segmentation task as opposed to a critical event detection task, which is what I thought your explanation was suggesting.

", "rawMarkdown": "Hi @bkowshik, that's interesting about the training data containing single events, but the data card states:\n\n> If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.\n\nI guess there could be no contradiction, but this seems to suggest that there shouldn't be an onset event without a following wakeup event and vice versa.\n\nIf events are in pairs then this problem seems to be more of a 1D signal segmentation task as opposed to a critical event detection task, which is what I thought [your explanation](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438936) was suggesting.", "votes": null } ], "comments": [ { "id": 2492883, "author_name": "bkowshik", "author_url": "", "post_date": "10/23/2023 02:12:16", "content": "

Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation

\n
\n

For each set of predictions and ground-truths within the same event x tolerance x series_id group, we match each ground-truth to the highest-confidence unmatched prediction occurring within the allowed tolerance.

\n
\n

My understanding is that the score is calculated event wise. Also, have seen annotated events in the train dataset that have only one event, either onset or wakeup and not have the pair of event which makes me think predicting the individual events is the ask.

", "votes": null, "replies": [ { "id": 2492906, "author_name": "jjinho", "author_url": "", "post_date": "10/23/2023 02:42:31", "content": "

Hi @bkowshik, that's interesting about the training data containing single events, but the data card states:

\n
\n

If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.

\n
\n

I guess there could be no contradiction, but this seems to suggest that there shouldn't be an onset event without a following wakeup event and vice versa.

\n

If events are in pairs then this problem seems to be more of a 1D signal segmentation task as opposed to a critical event detection task, which is what I thought your explanation was suggesting.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2491773": "[This discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438936) already has an explanation of `score` in submission, but I am confused about whether it applies to singular events (e.g. `onset` **or** `wakeup`) or event pairs (`onset` **and** `wakeup`)?\n\nThe sample submission file seems to have scores that are paired, but many public notebooks do not follow this pattern, which is why I am confused.", "2492883": "Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/overview/evaluation\n\n> For each set of predictions and ground-truths within the same `event x tolerance x series_id` group, we match each ground-truth to the highest-confidence unmatched prediction occurring within the allowed tolerance.\n\nMy understanding is that the score is calculated event wise. Also, have seen annotated events in the train dataset that have only one event, either `onset` or `wakeup` and not have the pair of event which makes me think predicting the individual events is the ask.", "2492906": "Hi @bkowshik, that's interesting about the training data containing single events, but the data card states:\n\n> If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.\n\nI guess there could be no contradiction, but this seems to suggest that there shouldn't be an onset event without a following wakeup event and vice versa.\n\nIf events are in pairs then this problem seems to be more of a 1D signal segmentation task as opposed to a critical event detection task, which is what I thought [your explanation](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438936) was suggesting." }, "source": "meta" }