\nFigure 1:
\n
![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8913321%2Fdbc4c91dfd2b5ed71f81721765bf4959%2Flabel-error-0.png?generation=1694694753598209&alt=media\")
Hi everyone,
\nWe noticed some very interesting patterns when looking in-depth at the labeling of some of the training series. We analyzed some series by hand that did not contain any NaNs in them and quickly found these 2 time periods where the onset of sleep is possibly labeled inconsistently?
\nIn both cases seen below, we suspect that sleep onset (start of awake=0) was annotated in the original training dataset around 1200 +- timesteps to late which is around 100 minutes. The sensors at that time segment already seem to match the general sleep pattern of that individual.
\nSo, it could have a great inaccurate effect on the eventual predictions even if these inaccuracies are also in the test set.
\nWe suspect that there are more, since we found these examples in the first 2 series that we have analyzed.
\nFigure 1:
\n
Figure 2:
\n![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8913321%2F18e5b99d1d34a1b29d7ab49739399ef8%2Flabel-error-1.png?generation=1694695391322921&alt=media\")
Furthermore, as a second inconsistency, there are also 3 labeling duplication events that seem to contain 2 sleep onsets and only 1 wakeup, which does not make sense:
\nFigure 3:
\n![\"Figure3\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8913321%2F00af14e2657ddddbf15456d6d7044680%2Flabeling-error.png?generation=1694702790454096&alt=media\")
Very interesting observations we can see here above, there are 2 onsets and 1 wakeup in a night where the onsets are at different steps but at the same timestamp! This seems to be a data error, and these duplicate wrong rows should eventually be reviewed. When calculated, row 3815(case 1), row 5080(case 2), row 7010 (case 3) seem to be wrong when calculating the step difference between onset and wakeup. For these cases, the timestamp would not add up.
\nTherefore I would like to ask the question of how the dataset has been retrieved? And how has it been labelled, by experts observing when people are sleeping for example? If so, might it then be possible that some expert labeling errors exist in the datasets? So also possibly in the test dataset?
\nIt would be greatly appreciated if the host of this competition could clarify the observations made above! @ryanholbrook
", "messages": [ { "id": "2438693", "postDate": "09/14/2023 12:44:06", "content": "Hi everyone,
\nWe noticed some very interesting patterns when looking in-depth at the labeling of some of the training series. We analyzed some series by hand that did not contain any NaNs in them and quickly found these 2 time periods where the onset of sleep is possibly labeled inconsistently?
\nIn both cases seen below, we suspect that sleep onset (start of awake=0) was annotated in the original training dataset around 1200 +- timesteps to late which is around 100 minutes. The sensors at that time segment already seem to match the general sleep pattern of that individual.
\nSo, it could have a great inaccurate effect on the eventual predictions even if these inaccuracies are also in the test set.
\nWe suspect that there are more, since we found these examples in the first 2 series that we have analyzed.
\nFigure 1:
\n
Figure 2:
\n
Furthermore, as a second inconsistency, there are also 3 labeling duplication events that seem to contain 2 sleep onsets and only 1 wakeup, which does not make sense:
\nFigure 3:
\n
Very interesting observations we can see here above, there are 2 onsets and 1 wakeup in a night where the onsets are at different steps but at the same timestamp! This seems to be a data error, and these duplicate wrong rows should eventually be reviewed. When calculated, row 3815(case 1), row 5080(case 2), row 7010 (case 3) seem to be wrong when calculating the step difference between onset and wakeup. For these cases, the timestamp would not add up.
\nTherefore I would like to ask the question of how the dataset has been retrieved? And how has it been labelled, by experts observing when people are sleeping for example? If so, might it then be possible that some expert labeling errors exist in the datasets? So also possibly in the test dataset?
\nIt would be greatly appreciated if the host of this competition could clarify the observations made above! @ryanholbrook
", "rawMarkdown": "Hi everyone,\n\nWe noticed some very interesting patterns when looking in-depth at the labeling of some of the training series. We analyzed some series by hand that did not contain any NaNs in them and quickly found these 2 time periods where the onset of sleep is possibly labeled inconsistently?\n\nIn both cases seen below, we suspect that sleep onset (start of awake=0) was annotated in the original training dataset around 1200 +- timesteps to late which is around 100 minutes. The sensors at that time segment already seem to match the general sleep pattern of that individual. \n\nSo, it could have a great inaccurate effect on the eventual predictions even if these inaccuracies are also in the test set. \n\nWe suspect that there are more, since we found these examples in the first 2 series that we have analyzed.\nFigure 1:\n\n\nFigure 2:\n\n\n\n**Furthermore, as a second inconsistency**, there are also 3 labeling duplication events that seem to contain 2 sleep onsets and only 1 wakeup, which does not make sense:\n\nFigure 3:\n\n\nVery interesting observations we can see here above, there are 2 onsets and 1 wakeup in a night where the onsets are at different steps but at the same timestamp! This seems to be a data error, and these duplicate wrong rows should eventually be reviewed. When calculated, row 3815(case 1), row 5080(case 2), row 7010 (case 3) seem to be wrong when calculating the step difference between onset and wakeup. For these cases, the timestamp would not add up.\n\nTherefore I would like to ask the question of how the dataset has been retrieved? And how has it been labelled, by experts observing when people are sleeping for example? If so, might it then be possible that some expert labeling errors exist in the datasets? So also possibly in the test dataset?\n\nIt would be greatly appreciated if the host of this competition could clarify the observations made above! @ryanholbrook", "votes": null }, { "id": "2438714", "postDate": "09/14/2023 12:57:32", "content": "Hi @hugodeheer. I agree with you, as I reported here. The consistency of labeling in Train-Test data is thought to have a large impact on prediction results. You may also want to ask Kaggle.
", "rawMarkdown": "Hi @hugodeheer. I agree with you, as I reported [here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438782). The consistency of labeling in Train-Test data is thought to have a large impact on prediction results. You may also want to ask Kaggle.", "votes": null }, { "id": "2438854", "postDate": "09/14/2023 14:20:57", "content": "Yeah, I think it would be nice if the host could clarify this.
", "rawMarkdown": "Yeah, I think it would be nice if the host could clarify this.", "votes": null }, { "id": "2438873", "postDate": "09/14/2023 14:33:33", "content": "Thanks @hugodeheer. I asked the question in my post. This is a very difficult question to answer…
", "rawMarkdown": "Thanks @hugodeheer. I asked the question in [my post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438782). This is a very difficult question to answer...", "votes": null }, { "id": "2439099", "postDate": "09/14/2023 16:21:24", "content": "It is not actually the same time:
\n10349 c6788e579967 20 onset 333552.0 2018-11-11T01:16:00-0400
\n10350 c6788e579967 20 onset 334272.0 2018-11-11T01:16:00-0500
\nThe first row is 4 hours behind UTC time, whereas the second row is 5 hours behind UTC time. Therefore, there is a difference of 1 hour. But it is still strange that there are 2 consecutive onset events, without a wakeup-event in between them.
Hi @mitsuyasuhoshino @hugodeheer is this problem have solved after the rescoring? just want to clarify
", "rawMarkdown": "Hi @mitsuyasuhoshino @hugodeheer is this problem have solved after the rescoring? just want to clarify", "votes": null }, { "id": "2448071", "postDate": "09/20/2023 11:57:35", "content": "It is not yet. Ryan, from Kaggle, identified problems with the duplicate onset events here and released a 'fix', which caused for a rescoring of the leaderboard.
\nNow it seems that due to the DST, all labels after the (now corrected) \"double onset nights\" appear to be incorrectly shifted forward by an hour for series 655f19eabf1e, 8a306e0890c0, and c6788e579967. Then yesterday the host responded again with the following.
\nSo today he promised to provide a fix for the shifting, which will probably result in another new update of the dataset and a rescoring of the dataset again.
\nHope this clarifies everything!
", "rawMarkdown": "It is not yet. Ryan, from Kaggle, identified problems with the duplicate onset events [here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697) and released a 'fix', which caused for a rescoring of the leaderboard.\n\nNow it seems that due to the DST, all labels after the (now corrected) \"double onset nights\" appear to be incorrectly shifted forward by an hour for series 655f19eabf1e, 8a306e0890c0, and c6788e579967. Then yesterday the host responded again with the [following](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697#2446990). \n\nSo today he promised to provide a fix for the shifting, which will probably result in another new update of the dataset and a rescoring of the dataset again.\n\nHope this clarifies everything!", "votes": null }, { "id": "2448255", "postDate": "09/20/2023 13:35:20", "content": "Ah I see, so basically another rescoring. I will keep myself posted on this. Thanks very much @hugodeheer for your answer.
", "rawMarkdown": "Ah I see, so basically another rescoring. I will keep myself posted on this. Thanks very much @hugodeheer for your answer.", "votes": null }, { "id": "2449009", "postDate": "09/20/2023 23:55:38", "content": "Just noticed that the leaderboard score shifted much, as well as my rank and submission scores changed. I think the second rescoring just happened guys @mitsuyasuhoshino @hugodeheer
", "rawMarkdown": "Just noticed that the leaderboard score shifted much, as well as my rank and submission scores changed. I think the second rescoring just happened guys @mitsuyasuhoshino @hugodeheer", "votes": null }, { "id": "2449969", "postDate": "09/21/2023 14:46:18", "content": "Hi @rimbax. I'm currently on a business trip, and I'm in a world where I only have a smartphone and no PC, so I'm planning to check the modified events next week. Sorry for the late response.
", "rawMarkdown": "Hi @rimbax. I'm currently on a business trip, and I'm in a world where I only have a smartphone and no PC, so I'm planning to check the modified events next week. Sorry for the late response.", "votes": null } ], "comments": [ { "id": 2438714, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/14/2023 12:57:32", "content": "Hi @hugodeheer. I agree with you, as I reported here. The consistency of labeling in Train-Test data is thought to have a large impact on prediction results. You may also want to ask Kaggle.
", "votes": null, "replies": [ { "id": 2438854, "author_name": "hugodeheer", "author_url": "", "post_date": "09/14/2023 14:20:57", "content": "Yeah, I think it would be nice if the host could clarify this.
", "votes": null, "replies": [ { "id": 2438873, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/14/2023 14:33:33", "content": "Thanks @hugodeheer. I asked the question in my post. This is a very difficult question to answer…
", "votes": null, "replies": [] } ] }, { "id": 2447911, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 10:04:24", "content": "Hi @mitsuyasuhoshino @hugodeheer is this problem have solved after the rescoring? just want to clarify
", "votes": null, "replies": [ { "id": 2448071, "author_name": "hugodeheer", "author_url": "", "post_date": "09/20/2023 11:57:35", "content": "It is not yet. Ryan, from Kaggle, identified problems with the duplicate onset events here and released a 'fix', which caused for a rescoring of the leaderboard.
\nNow it seems that due to the DST, all labels after the (now corrected) \"double onset nights\" appear to be incorrectly shifted forward by an hour for series 655f19eabf1e, 8a306e0890c0, and c6788e579967. Then yesterday the host responded again with the following.
\nSo today he promised to provide a fix for the shifting, which will probably result in another new update of the dataset and a rescoring of the dataset again.
\nHope this clarifies everything!
", "votes": null, "replies": [ { "id": 2448255, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 13:35:20", "content": "Ah I see, so basically another rescoring. I will keep myself posted on this. Thanks very much @hugodeheer for your answer.
", "votes": null, "replies": [] } ] }, { "id": 2449009, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 23:55:38", "content": "Just noticed that the leaderboard score shifted much, as well as my rank and submission scores changed. I think the second rescoring just happened guys @mitsuyasuhoshino @hugodeheer
", "votes": null, "replies": [ { "id": 2449969, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/21/2023 14:46:18", "content": "Hi @rimbax. I'm currently on a business trip, and I'm in a world where I only have a smartphone and no PC, so I'm planning to check the modified events next week. Sorry for the late response.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2439099, "author_name": "fritzcremer", "author_url": "", "post_date": "09/14/2023 16:21:24", "content": "It is not actually the same time:
\n10349 c6788e579967 20 onset 333552.0 2018-11-11T01:16:00-0400
\n10350 c6788e579967 20 onset 334272.0 2018-11-11T01:16:00-0500
\nThe first row is 4 hours behind UTC time, whereas the second row is 5 hours behind UTC time. Therefore, there is a difference of 1 hour. But it is still strange that there are 2 consecutive onset events, without a wakeup-event in between them.