{ "id": 438683, "title": "Cleaned training data with 269 series_id", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/438683", "author_name": "Ammar Ahmad Qazi", "post_date": "2023-09-12T09:06:14.880000", "votes": 16, "comment_count": 3, "views": 0, "content": "

This data is created by combining train_events and train_series_data.

\n

From the train_events and train_series data, the following 8 series_id out of 277 series_id have been dropped since they have no onset or wakeup information (consists of all NaN values) :

\n
\n

0f9e60a8e56d
\n 2fc653ca75c7
\n 390b487231ce
\n 89c7daa72eee
\n a3e59c2ce3f6
\n c5d08fc3e040
\n c7b1283bb7eb
\n e11b9d69f856

\n
\n

The cleaned training data consists of the remaining 269 series_id which are obtained as follows:

\n\n
\n

08db4255286f
\n 0a96f4993bd7
\n 0cfc06c129cc
\n 1087d7b0ff2e
\n 10f8bc1f7b07
\n 18b61dd5aae8
\n 29c75c018220
\n 31011ade7c0a
\n 3452b878e596
\n 349c5562ee2c
\n 3664fe9233f9
\n 483d6545417f
\n 55a47ff9dc8a
\n 5acc9d63b5fd
\n 5f94bb3e1bed
\n 655f19eabf1e
\n 67f5fc60e494
\n 72bbd1ac3edf
\n 76237b9406d5
\n 7822ee8fe3ec
\n 89bd631d1769
\n 8e32047cbc1f
\n 939932f1822d
\n 9ee455e4770d
\n a596ad0b82aa
\n a9a2f7fac455
\n a9e5f5314bcb
\n af91d9a50547
\n b364205aba43
\n c535634d7dcd
\n c6788e579967
\n c68260cc9e8f
\n ca730dbf521d
\n d150801f3145
\n d25e479ecbb7
\n d515236bdeec
\n d5e47b94477e

\n
\n\n

Example 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00
\n\"\"

\n

Example 2: Since there is only wakeup event and no onset event for night 7, so we will drop all rows for series_id efbfc4526d58 after timestamp 2018-12-18 06:44:00 in the train_series data.
\n\"\"

\n

Example 3: Only wakeup event is available in night 20 for series_id c3072a759efb, all instances in the train_series data between the timestamp 2018-03-17 08:52:00 and 2018-03-18 22:38:00 are dropped.
\n\"\"

\n\n

Example: Since there is no onset or wakeup data for first night (night 1) and last night (night 23) for series_id ‘c3072a759efb’, all rows corresponding to timestamp before 2018-02-27 19:44:00 and after 2018-03-20 08:14:00 are dropped respectively from train_series data.

\n

\"\"

\n

\"\"

\n

The dataset can be downloaded from here.

\n

It took me a couple of days to build the logic and generate this data because the data size is very large, so please do upvote the dataset if you find it useful.

", "messages": [ { "id": 2434381, "postDate": "2023-09-12T09:06:14.880Z", "content": "

This data is created by combining train_events and train_series_data.

\n

From the train_events and train_series data, the following 8 series_id out of 277 series_id have been dropped since they have no onset or wakeup information (consists of all NaN values) :

\n
\n

0f9e60a8e56d
\n 2fc653ca75c7
\n 390b487231ce
\n 89c7daa72eee
\n a3e59c2ce3f6
\n c5d08fc3e040
\n c7b1283bb7eb
\n e11b9d69f856

\n
\n

The cleaned training data consists of the remaining 269 series_id which are obtained as follows:

\n\n
\n

08db4255286f
\n 0a96f4993bd7
\n 0cfc06c129cc
\n 1087d7b0ff2e
\n 10f8bc1f7b07
\n 18b61dd5aae8
\n 29c75c018220
\n 31011ade7c0a
\n 3452b878e596
\n 349c5562ee2c
\n 3664fe9233f9
\n 483d6545417f
\n 55a47ff9dc8a
\n 5acc9d63b5fd
\n 5f94bb3e1bed
\n 655f19eabf1e
\n 67f5fc60e494
\n 72bbd1ac3edf
\n 76237b9406d5
\n 7822ee8fe3ec
\n 89bd631d1769
\n 8e32047cbc1f
\n 939932f1822d
\n 9ee455e4770d
\n a596ad0b82aa
\n a9a2f7fac455
\n a9e5f5314bcb
\n af91d9a50547
\n b364205aba43
\n c535634d7dcd
\n c6788e579967
\n c68260cc9e8f
\n ca730dbf521d
\n d150801f3145
\n d25e479ecbb7
\n d515236bdeec
\n d5e47b94477e

\n
\n\n

Example 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00
\n\"\"

\n

Example 2: Since there is only wakeup event and no onset event for night 7, so we will drop all rows for series_id efbfc4526d58 after timestamp 2018-12-18 06:44:00 in the train_series data.
\n\"\"

\n

Example 3: Only wakeup event is available in night 20 for series_id c3072a759efb, all instances in the train_series data between the timestamp 2018-03-17 08:52:00 and 2018-03-18 22:38:00 are dropped.
\n\"\"

\n\n

Example: Since there is no onset or wakeup data for first night (night 1) and last night (night 23) for series_id ‘c3072a759efb’, all rows corresponding to timestamp before 2018-02-27 19:44:00 and after 2018-03-20 08:14:00 are dropped respectively from train_series data.

\n

\"\"

\n

\"\"

\n

The dataset can be downloaded from here.

\n

It took me a couple of days to build the logic and generate this data because the data size is very large, so please do upvote the dataset if you find it useful.

", "rawMarkdown": "This [data](https://www.kaggle.com/datasets/ammarahmadqazi/cmi-cleaned-training-data) is created by combining train_events and train_series_data.\n\nFrom the train_events and train_series data, the following 8 series_id out of 277 series_id have been dropped since they have no onset or wakeup information (consists of all NaN values) : \n\n> 0f9e60a8e56d\n2fc653ca75c7\n390b487231ce\n89c7daa72eee\na3e59c2ce3f6\nc5d08fc3e040\nc7b1283bb7eb\ne11b9d69f856\n\nThe cleaned training data consists of the remaining 269 series_id which are obtained as follows:\n\n\n- 37 series_id have no missing values so we can directly combine train_events and train_series data for these to get sleep and awake cycles. They are as follows:\n> 08db4255286f\n0a96f4993bd7\n0cfc06c129cc\n1087d7b0ff2e\n10f8bc1f7b07\n18b61dd5aae8\n29c75c018220\n31011ade7c0a\n3452b878e596\n349c5562ee2c\n3664fe9233f9\n483d6545417f\n55a47ff9dc8a\n5acc9d63b5fd\n5f94bb3e1bed\n655f19eabf1e\n67f5fc60e494\n72bbd1ac3edf\n76237b9406d5\n7822ee8fe3ec\n89bd631d1769\n8e32047cbc1f\n939932f1822d\n9ee455e4770d\na596ad0b82aa\na9a2f7fac455\na9e5f5314bcb\naf91d9a50547\nb364205aba43\nc535634d7dcd\nc6788e579967\nc68260cc9e8f\nca730dbf521d\nd150801f3145\nd25e479ecbb7\nd515236bdeec\nd5e47b94477e\n\n\n\n\n- In train events data, for a particular night where we have either onset or wakeup event but not both, the rows corresponding to those timestamps have been dropped in train_series data.\n\nExample 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00 \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2Fbc0a78857e5827bd43a4688a28c24a3c%2FPicture%201.png?generation=1694494509924411&alt=media)\n\nExample 2: Since there is only wakeup event and no onset event for night 7, so we will drop all rows for series_id efbfc4526d58 after timestamp 2018-12-18 06:44:00 in the train_series data.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F81db8d66cb06b8852c738bcf4edd88e9%2FPicture%202.png?generation=1694494907342907&alt=media)\n\nExample 3: Only wakeup event is available in night 20 for series_id c3072a759efb, all instances in the train_series data between the timestamp 2018-03-17 08:52:00 and 2018-03-18 22:38:00 are dropped.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F3b883774b5f351bb9215c5a3693a60db%2FPicture%2014png.png?generation=1694495372716074&alt=media)\n\n- For duplicate onset or wakeup event in given night, we have dropped the latter one (index 5363 is dropped) for train_events data before combining with train_series data to get sleep/awake cycles.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F6d0af4b48d47409706ea3e2dc867e66b%2FPicture%203.png?generation=1694495140411894&alt=media)\n\n\n\n\n\n- If the first or last night for given series_id has either onset or wakeup or none of those events but not both events, all the rows before or after the corresponding timestamp are respectively dropped from the train_series data\n\nExample: Since there is no onset or wakeup data for first night (night 1) and last night (night 23) for series_id ‘c3072a759efb’, all rows corresponding to timestamp before 2018-02-27 19:44:00 and after 2018-03-20 08:14:00 are dropped respectively from train_series data. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F1bd480e5932ac5fd264f3a39871755c6%2FPicture%205.png?generation=1694495767661968&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2Fb1f26db444eb34c2168869487a74dbb7%2FPicture%206.png?generation=1694495771951078&alt=media)\n\n\n\n\n#### The dataset can be downloaded from [here](https://www.kaggle.com/datasets/ammarahmadqazi/cmi-cleaned-training-data). \n\n#### It took me a couple of days to build the logic and generate this data because the data size is very large, so please do upvote the dataset if you find it useful.", "votes": 16 }, { "id": 2449242, "postDate": "2023-09-21T04:55:56.027Z", "content": "

Hope we can get an update version after the event data updating. Thanks in advance.

", "rawMarkdown": "Hope we can get an update version after the event data updating. Thanks in advance.", "votes": 1 }, { "id": 2435376, "postDate": "2023-09-12T22:56:57.090Z", "content": "

Great, thank you! I was looking for something like this

", "rawMarkdown": "Great, thank you! I was looking for something like this" }, { "id": 2435703, "postDate": "2023-09-13T06:16:13.480Z", "content": "

Nice work on this @ammarahmadqazi 👏

\n

Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438563#2434966

\n
\n

The NaN cases are those when no meaningful sleep events were detected in the night, often due to non-wear (e.g., the participant took off the watch at some point). These are a common occurrence, and may exist within the test sets. The reason we coded these as NaNs rather than removing the days was so that your algorithms can also be challenged to identify when and when not to make a prediction about sleep events.

\n
\n

The 8 series IDs dropped does make sense from a prediction of the events point of view. But, given the possibility of such series occouring in the test dataset too, maybe one needs to figure out a way to identify one needs to make a prediction or not in the first place, given a series.

\n
\n

In train events data, for a particular night where we have either onset or wakeup event but not both, the rows corresponding to those timestamps have been dropped in train_series data. Example 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00

\n
\n

@ammarahmadqazi Should we keep the onset event at 2017-09-09 22:08:00 given the events are evaluated separately and there might not be a need to have a pair of onset / wakeup events?

\n
\n

For duplicate onset or wakeup event in given night, we have dropped the latter one (index 5363 is dropped) for train_events data before combining with train_series data to get sleep/awake cycles.

\n
\n

Since the wakeup event has a timezone of -05:00 would dropping the row at index=5362 be better instead?

", "rawMarkdown": "Nice work on this @ammarahmadqazi 👏\n\nRef: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438563#2434966\n\n> The NaN cases are those when no meaningful sleep events were detected in the night, often due to non-wear (e.g., the participant took off the watch at some point). These are a common occurrence, and may exist within the test sets. The reason we coded these as NaNs rather than removing the days was so that **your algorithms can also be challenged to identify when and when not to make a prediction about sleep events.**\n\nThe 8 series IDs dropped does make sense from a prediction of the events point of view. But, given the possibility of such series occouring in the `test` dataset too, maybe one needs to figure out a way to identify one needs to make a prediction or not in the first place, given a series.\n\n\n> In train events data, for a particular night where we have either onset or wakeup event but not both, the rows corresponding to those timestamps have been dropped in train_series data. Example 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00\n\n@ammarahmadqazi Should we keep the `onset` event at `2017-09-09 22:08:00` given the events are evaluated separately and there might not be a need to have a pair of `onset` / `wakeup` events?\n\n> For duplicate onset or wakeup event in given night, we have dropped the latter one (index 5363 is dropped) for train_events data before combining with train_series data to get sleep/awake cycles.\n\nSince the `wakeup` event has a timezone of `-05:00` would dropping the row at `index=5362` be better instead?", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 2449242, "author_name": "Aphysict", "author_url": "", "post_date": "2023-09-21T04:55:56.027000", "content": "

Hope we can get an update version after the event data updating. Thanks in advance.

", "votes": 1, "replies": [] }, { "id": 2435376, "author_name": "Felicius Brando", "author_url": "", "post_date": "2023-09-12T22:56:57.090000", "content": "

Great, thank you! I was looking for something like this

", "votes": 0, "replies": [] }, { "id": 2435703, "author_name": "Bhargav Kowshik", "author_url": "", "post_date": "2023-09-13T06:16:13.480000", "content": "

Nice work on this @ammarahmadqazi 👏

\n

Ref: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438563#2434966

\n
\n

The NaN cases are those when no meaningful sleep events were detected in the night, often due to non-wear (e.g., the participant took off the watch at some point). These are a common occurrence, and may exist within the test sets. The reason we coded these as NaNs rather than removing the days was so that your algorithms can also be challenged to identify when and when not to make a prediction about sleep events.

\n
\n

The 8 series IDs dropped does make sense from a prediction of the events point of view. But, given the possibility of such series occouring in the test dataset too, maybe one needs to figure out a way to identify one needs to make a prediction or not in the first place, given a series.

\n
\n

In train events data, for a particular night where we have either onset or wakeup event but not both, the rows corresponding to those timestamps have been dropped in train_series data. Example 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00

\n
\n

@ammarahmadqazi Should we keep the onset event at 2017-09-09 22:08:00 given the events are evaluated separately and there might not be a need to have a pair of onset / wakeup events?

\n
\n

For duplicate onset or wakeup event in given night, we have dropped the latter one (index 5363 is dropped) for train_events data before combining with train_series data to get sleep/awake cycles.

\n
\n

Since the wakeup event has a timezone of -05:00 would dropping the row at index=5362 be better instead?

", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2434381": "This [data](https://www.kaggle.com/datasets/ammarahmadqazi/cmi-cleaned-training-data) is created by combining train_events and train_series_data.\n\nFrom the train_events and train_series data, the following 8 series_id out of 277 series_id have been dropped since they have no onset or wakeup information (consists of all NaN values) : \n\n> 0f9e60a8e56d\n2fc653ca75c7\n390b487231ce\n89c7daa72eee\na3e59c2ce3f6\nc5d08fc3e040\nc7b1283bb7eb\ne11b9d69f856\n\nThe cleaned training data consists of the remaining 269 series_id which are obtained as follows:\n\n\n- 37 series_id have no missing values so we can directly combine train_events and train_series data for these to get sleep and awake cycles. They are as follows:\n> 08db4255286f\n0a96f4993bd7\n0cfc06c129cc\n1087d7b0ff2e\n10f8bc1f7b07\n18b61dd5aae8\n29c75c018220\n31011ade7c0a\n3452b878e596\n349c5562ee2c\n3664fe9233f9\n483d6545417f\n55a47ff9dc8a\n5acc9d63b5fd\n5f94bb3e1bed\n655f19eabf1e\n67f5fc60e494\n72bbd1ac3edf\n76237b9406d5\n7822ee8fe3ec\n89bd631d1769\n8e32047cbc1f\n939932f1822d\n9ee455e4770d\na596ad0b82aa\na9a2f7fac455\na9e5f5314bcb\naf91d9a50547\nb364205aba43\nc535634d7dcd\nc6788e579967\nc68260cc9e8f\nca730dbf521d\nd150801f3145\nd25e479ecbb7\nd515236bdeec\nd5e47b94477e\n\n\n\n\n- In train events data, for a particular night where we have either onset or wakeup event but not both, the rows corresponding to those timestamps have been dropped in train_series data.\n\nExample 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00 \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2Fbc0a78857e5827bd43a4688a28c24a3c%2FPicture%201.png?generation=1694494509924411&alt=media)\n\nExample 2: Since there is only wakeup event and no onset event for night 7, so we will drop all rows for series_id efbfc4526d58 after timestamp 2018-12-18 06:44:00 in the train_series data.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F81db8d66cb06b8852c738bcf4edd88e9%2FPicture%202.png?generation=1694494907342907&alt=media)\n\nExample 3: Only wakeup event is available in night 20 for series_id c3072a759efb, all instances in the train_series data between the timestamp 2018-03-17 08:52:00 and 2018-03-18 22:38:00 are dropped.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F3b883774b5f351bb9215c5a3693a60db%2FPicture%2014png.png?generation=1694495372716074&alt=media)\n\n- For duplicate onset or wakeup event in given night, we have dropped the latter one (index 5363 is dropped) for train_events data before combining with train_series data to get sleep/awake cycles.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F6d0af4b48d47409706ea3e2dc867e66b%2FPicture%203.png?generation=1694495140411894&alt=media)\n\n\n\n\n\n- If the first or last night for given series_id has either onset or wakeup or none of those events but not both events, all the rows before or after the corresponding timestamp are respectively dropped from the train_series data\n\nExample: Since there is no onset or wakeup data for first night (night 1) and last night (night 23) for series_id ‘c3072a759efb’, all rows corresponding to timestamp before 2018-02-27 19:44:00 and after 2018-03-20 08:14:00 are dropped respectively from train_series data. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2F1bd480e5932ac5fd264f3a39871755c6%2FPicture%205.png?generation=1694495767661968&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4472205%2Fb1f26db444eb34c2168869487a74dbb7%2FPicture%206.png?generation=1694495771951078&alt=media)\n\n\n\n\n#### The dataset can be downloaded from [here](https://www.kaggle.com/datasets/ammarahmadqazi/cmi-cleaned-training-data). \n\n#### It took me a couple of days to build the logic and generate this data because the data size is very large, so please do upvote the dataset if you find it useful.", "2449242": "Hope we can get an update version after the event data updating. Thanks in advance.", "2435376": "Great, thank you! I was looking for something like this", "2435703": "Nice work on this @ammarahmadqazi 👏\n\nRef: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438563#2434966\n\n> The NaN cases are those when no meaningful sleep events were detected in the night, often due to non-wear (e.g., the participant took off the watch at some point). These are a common occurrence, and may exist within the test sets. The reason we coded these as NaNs rather than removing the days was so that **your algorithms can also be challenged to identify when and when not to make a prediction about sleep events.**\n\nThe 8 series IDs dropped does make sense from a prediction of the events point of view. But, given the possibility of such series occouring in the `test` dataset too, maybe one needs to figure out a way to identify one needs to make a prediction or not in the first place, given a series.\n\n\n> In train events data, for a particular night where we have either onset or wakeup event but not both, the rows corresponding to those timestamps have been dropped in train_series data. Example 1: We have dropped rows corresponding to the series 0ce74d6d2106 in train_series data where the timestamp is greater than (not equal to) 2017-09-09 06:30:00 & less than (not equal to) 2017-09-10 22:00:00\n\n@ammarahmadqazi Should we keep the `onset` event at `2017-09-09 22:08:00` given the events are evaluated separately and there might not be a need to have a pair of `onset` / `wakeup` events?\n\n> For duplicate onset or wakeup event in given night, we have dropped the latter one (index 5363 is dropped) for train_events data before combining with train_series data to get sleep/awake cycles.\n\nSince the `wakeup` event has a timezone of `-05:00` would dropping the row at `index=5362` be better instead?" } }