{ "id": 441736, "title": "How do you handle irregular data or missing events in training/inference?", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/441736", "author_name": "", "post_date": "2023-09-20T01:21:04.201647100Z", "votes": 3, "comment_count": 11, "views": 0, "content": "

Based on many public discussions, many nights have no recorded events, although the recording series looks valid.
\nShould we expect the test series to also have periods with no events? In that case, how would you handle it?

", "messages": [ { "id": "2447271", "postDate": "09/20/2023 01:21:04", "content": "

Based on many public discussions, many nights have no recorded events, although the recording series looks valid.
\nShould we expect the test series to also have periods with no events? In that case, how would you handle it?

", "rawMarkdown": "Based on many public discussions, many nights have no recorded events, although the recording series looks valid. \nShould we expect the test series to also have periods with no events? In that case, how would you handle it?", "votes": null }, { "id": "2447408", "postDate": "09/20/2023 04:18:09", "content": "

Refer this discussion

", "rawMarkdown": "Refer this [discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438438#2439236)", "votes": null }, { "id": "2447414", "postDate": "09/20/2023 04:29:11", "content": "

By eyeballing, some series might have valid events yet unlabeled as a sleep period. Maybe it is because of the expert labeling procedure like at least 30mins etc?

\n

The hidden test set may or may not include series with no events. Just in case it has, there could be some false positives if our predicted events are present. I personally would assume that, or maybe the kaggle staff/competition host may elaborate more on this.

", "rawMarkdown": "By eyeballing, some series might have valid events yet unlabeled as a sleep period. Maybe it is because of the expert labeling procedure like at least 30mins etc?\n\nThe hidden test set may or may not include series with no events. Just in case it has, there could be some false positives if our predicted events are present. I personally would assume that, or maybe the kaggle staff/competition host may elaborate more on this.", "votes": null }, { "id": "2448486", "postDate": "09/20/2023 16:02:13", "content": "

Thank you! I suspect that the test data will have periods with no events as well. Other explanation is that those valid yet unlabeled events aren't the real data. In fact, it might be due to some imputation/interpolation by the host while prepping it (since most of them look repetitive to me).
\nTake this as an example: series_id= '03d92c9f6f8a'
\n\"\"

", "rawMarkdown": "Thank you! I suspect that the test data will have periods with no events as well. Other explanation is that those valid yet unlabeled events aren't the real data. In fact, it might be due to some imputation/interpolation by the host while prepping it (since most of them look repetitive to me). \nTake this as an example: `series_id= '03d92c9f6f8a'`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F6a30a6f04549e3c29d4484a90b60ce50%2Frepetitive_event.png?generation=1695226000308723&alt=media)", "votes": null }, { "id": "2448487", "postDate": "09/20/2023 16:03:00", "content": "

Thank you for the reference. That discussion was really helpful to me!

", "rawMarkdown": "Thank you for the reference. That discussion was really helpful to me!", "votes": null }, { "id": "2484276", "postDate": "10/16/2023 11:04:29", "content": "

@nghihuynh or it could be due to a trend is expected in sleep logs data, where participants might drop out of the study, making it common to see fewer series logs in the later nights

\n

\"Null

\n

What your graph is telling for series 03d92c9f6f8a, there is no label?

", "rawMarkdown": "nghihuynh or it could be due to a trend is expected in sleep logs data, where participants might drop out of the study, making it common to see fewer series logs in the later nights\n\n![Null Ratio](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213385%2F69d5758ee4032952ac0f36abfb192070%2FNull%20Ratio.JPG?generation=1697454027375769&alt=media)\n\nWhat your graph is telling for series 03d92c9f6f8a, there is no label?", "votes": null }, { "id": "2484925", "postDate": "10/16/2023 19:07:50", "content": "

Hi @oraware,
\nYes, there is no label for this period based on the given train_event data.

", "rawMarkdown": "Hi @oraware,\nYes, there is no label for this period based on the given train_event data.", "votes": null }, { "id": "2484932", "postDate": "10/16/2023 19:12:23", "content": "

Also, the recording device they're using has a battery life of 25 days. So, after this period, they may either remove the watch for recharging or simply stop using it.

", "rawMarkdown": "Also, the recording device they're using has a battery life of 25 days. So, after this period, they may either remove the watch for recharging or simply stop using it.", "votes": null }, { "id": "2484977", "postDate": "10/16/2023 20:02:51", "content": "

@nghihuynh in your graph , on x axis is timestamp and y axis is anglez for first graph? What is on for the second one?

", "rawMarkdown": "nghihuynh in your graph , on x axis is timestamp and y axis is anglez for first graph? What is on for the second one?", "votes": null }, { "id": "2485086", "postDate": "10/16/2023 23:58:14", "content": "

Hi @oraware, the y axis of the first graph is enmo, on the second graph is anglez

", "rawMarkdown": "Hi @oraware, the y axis of the first graph is enmo, on the second graph is anglez", "votes": null }, { "id": "2485112", "postDate": "10/17/2023 00:27:14", "content": "

@nghihuynh so what your red line spotting in graph?

", "rawMarkdown": "nghihuynh so what your red line spotting in graph?", "votes": null }, { "id": "2485147", "postDate": "10/17/2023 01:08:11", "content": "

Hi @oraware, red line is the onset position, green line is the wakeup position

", "rawMarkdown": "Hi @oraware, red line is the onset position, green line is the wakeup position", "votes": null } ], "comments": [ { "id": 2447408, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/20/2023 04:18:09", "content": "

Refer this discussion

", "votes": null, "replies": [ { "id": 2448487, "author_name": "nghihuynh", "author_url": "", "post_date": "09/20/2023 16:03:00", "content": "

Thank you for the reference. That discussion was really helpful to me!

", "votes": null, "replies": [] } ] }, { "id": 2447414, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 04:29:11", "content": "

By eyeballing, some series might have valid events yet unlabeled as a sleep period. Maybe it is because of the expert labeling procedure like at least 30mins etc?

\n

The hidden test set may or may not include series with no events. Just in case it has, there could be some false positives if our predicted events are present. I personally would assume that, or maybe the kaggle staff/competition host may elaborate more on this.

", "votes": null, "replies": [ { "id": 2448486, "author_name": "nghihuynh", "author_url": "", "post_date": "09/20/2023 16:02:13", "content": "

Thank you! I suspect that the test data will have periods with no events as well. Other explanation is that those valid yet unlabeled events aren't the real data. In fact, it might be due to some imputation/interpolation by the host while prepping it (since most of them look repetitive to me).
\nTake this as an example: series_id= '03d92c9f6f8a'
\n\"\"

", "votes": null, "replies": [ { "id": 2484276, "author_name": "oraware", "author_url": "", "post_date": "10/16/2023 11:04:29", "content": "

@nghihuynh or it could be due to a trend is expected in sleep logs data, where participants might drop out of the study, making it common to see fewer series logs in the later nights

\n

\"Null

\n

What your graph is telling for series 03d92c9f6f8a, there is no label?

", "votes": null, "replies": [ { "id": 2484925, "author_name": "nghihuynh", "author_url": "", "post_date": "10/16/2023 19:07:50", "content": "

Hi @oraware,
\nYes, there is no label for this period based on the given train_event data.

", "votes": null, "replies": [ { "id": 2484977, "author_name": "oraware", "author_url": "", "post_date": "10/16/2023 20:02:51", "content": "

@nghihuynh in your graph , on x axis is timestamp and y axis is anglez for first graph? What is on for the second one?

", "votes": null, "replies": [ { "id": 2485086, "author_name": "nghihuynh", "author_url": "", "post_date": "10/16/2023 23:58:14", "content": "

Hi @oraware, the y axis of the first graph is enmo, on the second graph is anglez

", "votes": null, "replies": [ { "id": 2485112, "author_name": "oraware", "author_url": "", "post_date": "10/17/2023 00:27:14", "content": "

@nghihuynh so what your red line spotting in graph?

", "votes": null, "replies": [ { "id": 2485147, "author_name": "nghihuynh", "author_url": "", "post_date": "10/17/2023 01:08:11", "content": "

Hi @oraware, red line is the onset position, green line is the wakeup position

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2484932, "author_name": "yuchen2066", "author_url": "", "post_date": "10/16/2023 19:12:23", "content": "

Also, the recording device they're using has a battery life of 25 days. So, after this period, they may either remove the watch for recharging or simply stop using it.

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2447271": "Based on many public discussions, many nights have no recorded events, although the recording series looks valid. \nShould we expect the test series to also have periods with no events? In that case, how would you handle it?", "2447408": "Refer this [discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438438#2439236)", "2447414": "By eyeballing, some series might have valid events yet unlabeled as a sleep period. Maybe it is because of the expert labeling procedure like at least 30mins etc?\n\nThe hidden test set may or may not include series with no events. Just in case it has, there could be some false positives if our predicted events are present. I personally would assume that, or maybe the kaggle staff/competition host may elaborate more on this.", "2448486": "Thank you! I suspect that the test data will have periods with no events as well. Other explanation is that those valid yet unlabeled events aren't the real data. In fact, it might be due to some imputation/interpolation by the host while prepping it (since most of them look repetitive to me). \nTake this as an example: `series_id= '03d92c9f6f8a'`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F6a30a6f04549e3c29d4484a90b60ce50%2Frepetitive_event.png?generation=1695226000308723&alt=media)", "2448487": "Thank you for the reference. That discussion was really helpful to me!", "2484276": "nghihuynh or it could be due to a trend is expected in sleep logs data, where participants might drop out of the study, making it common to see fewer series logs in the later nights\n\n![Null Ratio](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213385%2F69d5758ee4032952ac0f36abfb192070%2FNull%20Ratio.JPG?generation=1697454027375769&alt=media)\n\nWhat your graph is telling for series 03d92c9f6f8a, there is no label?", "2484925": "Hi @oraware,\nYes, there is no label for this period based on the given train_event data.", "2484932": "Also, the recording device they're using has a battery life of 25 days. So, after this period, they may either remove the watch for recharging or simply stop using it.", "2484977": "nghihuynh in your graph , on x axis is timestamp and y axis is anglez for first graph? What is on for the second one?", "2485086": "Hi @oraware, the y axis of the first graph is enmo, on the second graph is anglez", "2485112": "nghihuynh so what your red line spotting in graph?", "2485147": "Hi @oraware, red line is the onset position, green line is the wakeup position" }, "source": "meta" }