{ "id": 441638, "title": "How to deal with sleepless periods", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/441638", "author_name": "", "post_date": "2023-09-19T14:59:09.487637800Z", "votes": 36, "comment_count": 16, "views": 0, "content": "

In the data tab it states:
\nThough each series is a continuous recording, there may be periods in the series when the accelerometer device was removed. These period are determined as those where suspiciously little variation in the accelerometer signals occur over an extended period of time, which is unrealistic for typical human participants. Events are not annotated for these periods, and you should attempt to refrain from making event predictions during these periods: an event prediction will be scored as false positive.

\n

However, when looking at certain series_ids, such as 31011ade7c0a, it doesn't look like the watch was taken off, yet the second half of the data has no sleep periods. Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep.
\n\"\"

\n

Closing in on the moment from which on no further sleep events are registered:
\n\"\"

\n

With the naked eye, the later periods don't look that different and don't have \"suspiciously little variation\", as described in the data section. But it also states:
\nIf no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.

\n

Therefore I conclude that this must be the case, though it seems strange that there can't be a period of sleep detected for 10-20 nights in a row. Also unlike for some other series, there are no nan events in the df for that series. It would be great to hear from the host (@gkiar07) that this is actually intended and how to interpret this.

", "messages": [ { "id": "2446710", "postDate": "09/19/2023 14:59:09", "content": "

In the data tab it states:
\nThough each series is a continuous recording, there may be periods in the series when the accelerometer device was removed. These period are determined as those where suspiciously little variation in the accelerometer signals occur over an extended period of time, which is unrealistic for typical human participants. Events are not annotated for these periods, and you should attempt to refrain from making event predictions during these periods: an event prediction will be scored as false positive.

\n

However, when looking at certain series_ids, such as 31011ade7c0a, it doesn't look like the watch was taken off, yet the second half of the data has no sleep periods. Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep.
\n\"\"

\n

Closing in on the moment from which on no further sleep events are registered:
\n\"\"

\n

With the naked eye, the later periods don't look that different and don't have \"suspiciously little variation\", as described in the data section. But it also states:
\nIf no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.

\n

Therefore I conclude that this must be the case, though it seems strange that there can't be a period of sleep detected for 10-20 nights in a row. Also unlike for some other series, there are no nan events in the df for that series. It would be great to hear from the host (@gkiar07) that this is actually intended and how to interpret this.

", "rawMarkdown": "In the data tab it states:\n`Though each series is a continuous recording, there may be periods in the series when the accelerometer device was removed. These period are determined as those where suspiciously little variation in the accelerometer signals occur over an extended period of time, which is unrealistic for typical human participants. Events are not annotated for these periods, and you should attempt to refrain from making event predictions during these periods: an event prediction will be scored as false positive.`\n\nHowever, when looking at certain series_ids, such as 31011ade7c0a, it doesn't look like the watch was taken off, yet the second half of the data has no sleep periods. Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F39b342abc2983f4094460548fa0f4b42%2Fseries1.png?generation=1695134809724568&alt=media)\n\nClosing in on the moment from which on no further sleep events are registered:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F90af1c48e3c8067adcdf80be12a2a9df%2Fseries2.png?generation=1695134894826504&alt=media)\n\nWith the naked eye, the later periods don't look that different and don't have \"suspiciously little variation\", as described in the data section. But it also states:\n`If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.`\n\nTherefore I conclude that this must be the case, though it seems strange that there can't be a period of sleep detected for 10-20 nights in a row. Also unlike for some other series, there are no nan events in the df for that series. It would be great to hear from the host (@gkiar07) that this is actually intended and how to interpret this.", "votes": null }, { "id": "2446862", "postDate": "09/19/2023 17:07:56", "content": "

Dear @fritzcremer

\n

Indeed, series 31011ade7c0a, along with series a596ad0b82aa, were both excluded in my dataset creation notebook and called \"truncated\" events; they both seem to have reasonable looking data, but sleep event data stopped being recorded ¿perhaps it was the annotator who nodded off?

\n

All the best,
\ncarl

", "rawMarkdown": "Dear @fritzcremer \n\nIndeed, series `31011ade7c0a`, along with series `a596ad0b82aa`, [were both excluded in my dataset creation notebook](https://www.kaggle.com/code/carlmcbrideellis/zzzs-make-small-starter-datasets-target) and called \"truncated\" events; they both seem to have reasonable looking data, but sleep event data stopped being recorded ¿perhaps it was the annotator who nodded off? \n\nAll the best,\ncarl", "votes": null }, { "id": "2447082", "postDate": "09/19/2023 20:04:50", "content": "

It seems like it 😄

", "rawMarkdown": "It seems like it 😄", "votes": null }, { "id": "2447206", "postDate": "09/19/2023 23:03:28", "content": "

At some of those times, I suspect that the watches are not worn but they were somehow on a moving system (e.g. car, rocking crib, washing machine etc).

\n

I have also noticed some repeating patterns in some of the unworn times, as if the data was artificially created.

", "rawMarkdown": "At some of those times, I suspect that the watches are not worn but they were somehow on a moving system (e.g. car, rocking crib, washing machine etc).\n\nI have also noticed some repeating patterns in some of the unworn times, as if the data was artificially created.", "votes": null }, { "id": "2447215", "postDate": "09/19/2023 23:16:06", "content": "

I thought about that, but then again, it is strange how similar these periods look to the data where sleep events were detected.

", "rawMarkdown": "I thought about that, but then again, it is strange how similar these periods look to the data where sleep events were detected.", "votes": null }, { "id": "2447247", "postDate": "09/20/2023 00:14:26", "content": "

Yes, for example, the series c5d08fc3e040 looks like it has identically repeating values!?

", "rawMarkdown": "Yes, for example, the series `c5d08fc3e040` looks like it has identically repeating values!?", "votes": null }, { "id": "2448891", "postDate": "09/20/2023 21:00:19", "content": "

Hi @fritzcremer, I might have an explanation for the \"sleepless periods\" you mentioned. It might be due to some sort of imputation/interpolation (might explain why they look repetitive) by the host/organization to keep the data continuous for the experimental period. As others have mentioned briefly how they deal with it in this discussion, my approach is somewhat similar but instead of including those periods in my train data, I truncate it. For ex, I will only consider the periods until the last awake date and truncate the rest.

\n

Before truncation:
\n\"\"

\n

After truncation:
\n\"\"

\n

May I ask how would you normalize the data? Shouldn't normalization be within the range of (0,1)?

", "rawMarkdown": "Hi @fritzcremer, I might have an explanation for the \"sleepless periods\" you mentioned. It might be due to some sort of imputation/interpolation (might explain why they look repetitive) by the host/organization to keep the data continuous for the experimental period. As others have mentioned briefly how they deal with it in this [discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438438#2439236), my approach is somewhat similar but instead of including those periods in my train data, I truncate it. For ex, I will only consider the periods until the last awake date and truncate the rest. \n\nBefore truncation:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2Fd0fcc08e21e84aa60998fc764e308b03%2Fbefore.png?generation=1695243486786671&alt=media)\n\nAfter truncation:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F83fe5614d3345b9cc003a6655ce31959%2Fafter.png?generation=1695243499325011&alt=media)\n\nMay I ask how would you normalize the data? Shouldn't normalization be within the range of (0,1)?", "votes": null }, { "id": "2449034", "postDate": "09/21/2023 01:02:27", "content": "

Yes that might be a good idea.
\nI normalized so that the mean is 0 and std 1.

", "rawMarkdown": "Yes that might be a good idea.\nI normalized so that the mean is 0 and std 1.", "votes": null }, { "id": "2449038", "postDate": "09/21/2023 01:08:14", "content": "

Hi @nghihuynh , the middle area one between 2018-06-05 and 2018-06-11 seems like to be repetitive as well. Would you mind share your way to handle it?

", "rawMarkdown": "Hi @nghihuynh , the middle area one between 2018-06-05 and 2018-06-11 seems like to be repetitive as well. Would you mind share your way to handle it?", "votes": null }, { "id": "2449045", "postDate": "09/21/2023 01:21:05", "content": "

Hi @rimbax, for the one in the middle, I keep it for training purposes because I expect the test data to have similar patterns. Thus, it would be beneficial for any models developed during training learn those patterns as well.

", "rawMarkdown": "Hi @rimbax, for the one in the middle, I keep it for training purposes because I expect the test data to have similar patterns. Thus, it would be beneficial for any models developed during training learn those patterns as well.", "votes": null }, { "id": "2449046", "postDate": "09/21/2023 01:23:05", "content": "
\n

because I expect the test data might have similar patterns

\n
\n

Ah great then. Initially I want to remove the repetitive middle one but according to this I made up my mind. Thank you for sharing and good luck!

", "rawMarkdown": "> because I expect the test data might have similar patterns\n\nAh great then. Initially I want to remove the repetitive middle one but according to this I made up my mind. Thank you for sharing and good luck!", "votes": null }, { "id": "2452700", "postDate": "09/23/2023 14:14:21", "content": "

I divide periods of insomnia into 2 types:

\n
    \n
  1. Really random small values
    2. \n
  2. This is very similar to a dream, but the data is just not labeled
    3. \n
\n

The data from point 1 should be included in the training set.

\n

The data from point 2 can have 2 types of interpretation:
\nIndeed, there are problems with sleep, the child does not sleep at night, lies down, tosses and turns, and some diseases go away. Well, you never know - maybe this is not familiar to us.
\nData markup error.

\n

How can we check this?
\nCreate both datasets and look quickly. If there are significant differences, we will understand which case is ours.

\n

Has anyone already tested this hypothesis?

\n

@carlmcbrideellis @dan3dewey @fritzcremer @nghihuynh @rimbax @tolgadincer

", "rawMarkdown": "I divide periods of insomnia into 2 types:\n1. Really random small values\n2. This is very similar to a dream, but the data is just not labeled\n\nThe data from point 1 should be included in the training set.\n\nThe data from point 2 can have 2 types of interpretation:\nIndeed, there are problems with sleep, the child does not sleep at night, lies down, tosses and turns, and some diseases go away. Well, you never know - maybe this is not familiar to us.\nData markup error.\n\nHow can we check this?\nCreate both datasets and look quickly. If there are significant differences, we will understand which case is ours.\n\nHas anyone already tested this hypothesis?\n\n@carlmcbrideellis @dan3dewey @fritzcremer @nghihuynh @rimbax @tolgadincer", "votes": null }, { "id": "2474897", "postDate": "10/09/2023 14:34:26", "content": "

I believe there are artificial, \"filler\" data in regions/days of the time series. I was able to detect this by comparing the anglez value near middle-sleep with the anglez value + or - exactly 1 day away. If they are the same to 1e-6 then it's likely filler data. For example, below is my (cira 20th Century 🙃) code for checking:

\n
\n     = False\n    \n     int(middle_of_night)+** < max_step_in_series:\n         np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n                   .loc[int(middle_of_night)+**,'anglez'],\n                      =., atol=e-):\n             = True\n    \n     int(middle_of_night)-** > :\n         np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n                    .loc[int(middle_of_night)-**,'anglez'],\n                      =., atol=e-):\n             = True\n
", "rawMarkdown": "I believe there are artificial, \"filler\" data in regions/days of the time series. I was able to detect this by comparing the anglez value near middle-sleep with the anglez value + or - exactly 1 day away. If they are the same to 1e-6 then it's likely filler data. For example, below is my (cira 20th Century 🙃) code for checking:\n\n```\n# Check if the anglez value is identical 24 hours away\n night_is_filler = False\n # Check this and following night\n if int(middle_of_night)+24*60*12 < max_step_in_series:\n if np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n this_series.loc[int(middle_of_night)+24*60*12,'anglez'],\n rtol=0.0, atol=1e-6):\n night_is_filler = True\n # Check this and previous night\n if int(middle_of_night)-24*60*12 > 0:\n if np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n this_series.loc[int(middle_of_night)-24*60*12,'anglez'],\n rtol=0.0, atol=1e-6):\n night_is_filler = True\n```", "votes": null }, { "id": "2483901", "postDate": "10/16/2023 04:49:51", "content": "

I can personally testify to having no valid sleep window for 10 - 20 nights in a row 🥲

\n

\"\"

", "rawMarkdown": "I can personally testify to having no valid sleep window for 10 - 20 nights in a row 🥲\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F471225%2F8e1344f55ac35be95bdf529ab1523dbc%2Fthisisfine%20-%20Copy.png?generation=1697432103219630&alt=media)", "votes": null }, { "id": "2485302", "postDate": "10/17/2023 05:24:54", "content": "
\n

Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep

\n
\n

@fritzcremer Looking at the range of y-axis it looks to me like anglez is normalized while enmo is not normalized. What am I missing?

\n

I used sklearn.preprocessing.StandardScaler to process both the columns and we have the following summary and charts both before and after scaling. The std of the columns anglez and enmo are not one. But, after scaling they come very close to one.

\n

\"\"

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
enmo before scaling - Range: [0, 2.5]enmo after scaling. Range - [0, 30]
\"\"\"\"
\n

But, for anglez we see the following:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
anglez before scaling - Range: [-90, 90]anglez after scaling - Range: [-2, 2]
\"\"\"\"
", "rawMarkdown": "> Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep\n\n@fritzcremer Looking at the range of y-axis it looks to me like `anglez` is normalized while `enmo` is **not normalized**. What am I missing?\n\nI used `sklearn.preprocessing.StandardScaler` to process both the columns and we have the following summary and charts both before and after scaling. The std of the columns anglez and enmo are not one. But, after scaling they come very close to one.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Ff4c3ea1e5e3a21f63927a23b770ff5d9%2FScreenshot%202023-10-17%20at%2010.42.57.png?generation=1697519791540874&alt=media)\n\n| `enmo` before scaling - Range: `[0, 2.5]` | `enmo` after scaling. Range - `[0, 30]` |\n| --- | --- |\n| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fa1f1b1dea85748cb2de3c9433f81a470%2FScreenshot%202023-10-17%20at%2010.43.27.png?generation=1697519884186191&alt=media) | ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Ff131909ebec2ab7cd16abe0f3a683684%2FScreenshot%202023-10-17%20at%2010.43.14.png?generation=1697519937034112&alt=media) |\n\nBut, for `anglez` we see the following:\n\n| `anglez` before scaling - Range: `[-90, 90]` | `anglez` after scaling - Range: `[-2, 2]` |\n| --- | --- |\n| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F6825a46dd86f512a379f8347a7b5fed5%2FScreenshot%202023-10-17%20at%2010.43.37.png?generation=1697520017029002&alt=media) | ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fc37056352b0f1dd92a2d4b7deec9bab4%2FScreenshot%202023-10-17%20at%2010.43.49.png?generation=1697520032600914&alt=media) |", "votes": null }, { "id": "2485341", "postDate": "10/17/2023 06:06:03", "content": "

Ref: Jupyter notebook: Annotations per day . Detect Sleep States
\nFor every series, calculated a metric called events_per_day which is the number of non-null annotations we have divided by the range (max timestamp - min timestamp) of the series. The idea is to capture the amont of annotated duration for the different series.

\n\n
    \n
  1. As the range in days increases, the number of annotated events increases too. [This is expected ✅]
    2. \n
  2. For series with range between [20, 25] there seems to be a wide distribution in number of annotated events. [Wonder why 🤔]
    \n\"\"
    3. \n
", "rawMarkdown": "Ref: [Jupyter notebook: Annotations per day . Detect Sleep States](https://www.kaggle.com/bkowshik/annotations-per-day-detect-sleep-states)\n\nFor every series, calculated a metric called `events_per_day` which is the number of non-null annotations we have divided by the range (max timestamp - min timestamp) of the series. The idea is to capture the amont of annotated duration for the different series.\n\n- Some series with low values of `events_per_day` are: `[390b487231ce, a3e59c2ce3f6, 0f9e60a8e56d]`\n- Some series with high values of `events_per_day` are: `[d515236bdeec, a9e5f5314bcb, c535634d7dcd]`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F1eeb86f08530becb3b73c8c58ef710fb%2FScreenshot%202023-10-17%20at%2011.17.35.png?generation=1697521832974009&alt=media)\n\nFrom a histogram point of view, have both the normal as well as the cumulative version below:\n\n- Approx `30%` of the series have events per day less than one.\n- Approx `50%` of the series have events per day less than `1.5`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fe8af83c0778582a2ecea8001fe20d552%2FScreenshot%202023-10-17%20at%2011.17.23.png?generation=1697521858396965&alt=media)\n\nAlthough, I am considering the range of the timeseries and not just events per series, which means that the number of events are normalized in some fashion, I wanted to look if there was any relationship between the range of the series (short series vs long series) and the number of annotated events. Two kinds of relationships stand-out\n\n1. As the range in days increases, the number of annotated events increases too. [This is expected ✅]\n2. For series with range between `[20, 25]` there seems to be a wide distribution in number of annotated events. [Wonder why 🤔]\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fd3e61133a03e718d4bab1a3fd4e63bff%2FScreenshot%202023-10-17%20at%2011.56.26.png?generation=1697524011447223&alt=media)", "votes": null }, { "id": "2513686", "postDate": "11/05/2023 16:10:11", "content": "

@ryanholbrook @gkiar07 could you give some information about this topic? I still do not understand the reason of such series.

", "rawMarkdown": "ryanholbrook @gkiar07 could you give some information about this topic? I still do not understand the reason of such series.", "votes": null } ], "comments": [ { "id": 2446862, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/19/2023 17:07:56", "content": "

Dear @fritzcremer

\n

Indeed, series 31011ade7c0a, along with series a596ad0b82aa, were both excluded in my dataset creation notebook and called \"truncated\" events; they both seem to have reasonable looking data, but sleep event data stopped being recorded ¿perhaps it was the annotator who nodded off?

\n

All the best,
\ncarl

", "votes": null, "replies": [ { "id": 2447082, "author_name": "fritzcremer", "author_url": "", "post_date": "09/19/2023 20:04:50", "content": "

It seems like it 😄

", "votes": null, "replies": [] } ] }, { "id": 2447206, "author_name": "tolgadincer", "author_url": "", "post_date": "09/19/2023 23:03:28", "content": "

At some of those times, I suspect that the watches are not worn but they were somehow on a moving system (e.g. car, rocking crib, washing machine etc).

\n

I have also noticed some repeating patterns in some of the unworn times, as if the data was artificially created.

", "votes": null, "replies": [ { "id": 2447215, "author_name": "fritzcremer", "author_url": "", "post_date": "09/19/2023 23:16:06", "content": "

I thought about that, but then again, it is strange how similar these periods look to the data where sleep events were detected.

", "votes": null, "replies": [] }, { "id": 2447247, "author_name": "dan3dewey", "author_url": "", "post_date": "09/20/2023 00:14:26", "content": "

Yes, for example, the series c5d08fc3e040 looks like it has identically repeating values!?

", "votes": null, "replies": [] }, { "id": 2474897, "author_name": "dan3dewey", "author_url": "", "post_date": "10/09/2023 14:34:26", "content": "

I believe there are artificial, \"filler\" data in regions/days of the time series. I was able to detect this by comparing the anglez value near middle-sleep with the anglez value + or - exactly 1 day away. If they are the same to 1e-6 then it's likely filler data. For example, below is my (cira 20th Century 🙃) code for checking:

\n
\n     = False\n    \n     int(middle_of_night)+** < max_step_in_series:\n         np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n                   .loc[int(middle_of_night)+**,'anglez'],\n                      =., atol=e-):\n             = True\n    \n     int(middle_of_night)-** > :\n         np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n                    .loc[int(middle_of_night)-**,'anglez'],\n                      =., atol=e-):\n             = True\n
", "votes": null, "replies": [] } ] }, { "id": 2448891, "author_name": "nghihuynh", "author_url": "", "post_date": "09/20/2023 21:00:19", "content": "

Hi @fritzcremer, I might have an explanation for the \"sleepless periods\" you mentioned. It might be due to some sort of imputation/interpolation (might explain why they look repetitive) by the host/organization to keep the data continuous for the experimental period. As others have mentioned briefly how they deal with it in this discussion, my approach is somewhat similar but instead of including those periods in my train data, I truncate it. For ex, I will only consider the periods until the last awake date and truncate the rest.

\n

Before truncation:
\n\"\"

\n

After truncation:
\n\"\"

\n

May I ask how would you normalize the data? Shouldn't normalization be within the range of (0,1)?

", "votes": null, "replies": [ { "id": 2449034, "author_name": "fritzcremer", "author_url": "", "post_date": "09/21/2023 01:02:27", "content": "

Yes that might be a good idea.
\nI normalized so that the mean is 0 and std 1.

", "votes": null, "replies": [ { "id": 2485302, "author_name": "bkowshik", "author_url": "", "post_date": "10/17/2023 05:24:54", "content": "
\n

Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep

\n
\n

@fritzcremer Looking at the range of y-axis it looks to me like anglez is normalized while enmo is not normalized. What am I missing?

\n

I used sklearn.preprocessing.StandardScaler to process both the columns and we have the following summary and charts both before and after scaling. The std of the columns anglez and enmo are not one. But, after scaling they come very close to one.

\n

\"\"

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
enmo before scaling - Range: [0, 2.5]enmo after scaling. Range - [0, 30]
\"\"\"\"
\n

But, for anglez we see the following:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
anglez before scaling - Range: [-90, 90]anglez after scaling - Range: [-2, 2]
\"\"\"\"
", "votes": null, "replies": [] } ] }, { "id": 2449038, "author_name": "rimbax", "author_url": "", "post_date": "09/21/2023 01:08:14", "content": "

Hi @nghihuynh , the middle area one between 2018-06-05 and 2018-06-11 seems like to be repetitive as well. Would you mind share your way to handle it?

", "votes": null, "replies": [ { "id": 2449045, "author_name": "nghihuynh", "author_url": "", "post_date": "09/21/2023 01:21:05", "content": "

Hi @rimbax, for the one in the middle, I keep it for training purposes because I expect the test data to have similar patterns. Thus, it would be beneficial for any models developed during training learn those patterns as well.

", "votes": null, "replies": [ { "id": 2449046, "author_name": "rimbax", "author_url": "", "post_date": "09/21/2023 01:23:05", "content": "
\n

because I expect the test data might have similar patterns

\n
\n

Ah great then. Initially I want to remove the repetitive middle one but according to this I made up my mind. Thank you for sharing and good luck!

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2452700, "author_name": "zotovaa", "author_url": "", "post_date": "09/23/2023 14:14:21", "content": "

I divide periods of insomnia into 2 types:

\n
    \n
  1. Really random small values
    2. \n
  2. This is very similar to a dream, but the data is just not labeled
    3. \n
\n

The data from point 1 should be included in the training set.

\n

The data from point 2 can have 2 types of interpretation:
\nIndeed, there are problems with sleep, the child does not sleep at night, lies down, tosses and turns, and some diseases go away. Well, you never know - maybe this is not familiar to us.
\nData markup error.

\n

How can we check this?
\nCreate both datasets and look quickly. If there are significant differences, we will understand which case is ours.

\n

Has anyone already tested this hypothesis?

\n

@carlmcbrideellis @dan3dewey @fritzcremer @nghihuynh @rimbax @tolgadincer

", "votes": null, "replies": [] }, { "id": 2483901, "author_name": "illidan7", "author_url": "", "post_date": "10/16/2023 04:49:51", "content": "

I can personally testify to having no valid sleep window for 10 - 20 nights in a row 🥲

\n

\"\"

", "votes": null, "replies": [] }, { "id": 2485341, "author_name": "bkowshik", "author_url": "", "post_date": "10/17/2023 06:06:03", "content": "

Ref: Jupyter notebook: Annotations per day . Detect Sleep States
\nFor every series, calculated a metric called events_per_day which is the number of non-null annotations we have divided by the range (max timestamp - min timestamp) of the series. The idea is to capture the amont of annotated duration for the different series.

\n\n
    \n
  1. As the range in days increases, the number of annotated events increases too. [This is expected ✅]
    2. \n
  2. For series with range between [20, 25] there seems to be a wide distribution in number of annotated events. [Wonder why 🤔]
    \n\"\"
    3. \n
", "votes": null, "replies": [] }, { "id": 2513686, "author_name": "manwithaflower", "author_url": "", "post_date": "11/05/2023 16:10:11", "content": "

@ryanholbrook @gkiar07 could you give some information about this topic? I still do not understand the reason of such series.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2446710": "In the data tab it states:\n`Though each series is a continuous recording, there may be periods in the series when the accelerometer device was removed. These period are determined as those where suspiciously little variation in the accelerometer signals occur over an extended period of time, which is unrealistic for typical human participants. Events are not annotated for these periods, and you should attempt to refrain from making event predictions during these periods: an event prediction will be scored as false positive.`\n\nHowever, when looking at certain series_ids, such as 31011ade7c0a, it doesn't look like the watch was taken off, yet the second half of the data has no sleep periods. Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F39b342abc2983f4094460548fa0f4b42%2Fseries1.png?generation=1695134809724568&alt=media)\n\nClosing in on the moment from which on no further sleep events are registered:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6735218%2F90af1c48e3c8067adcdf80be12a2a9df%2Fseries2.png?generation=1695134894826504&alt=media)\n\nWith the naked eye, the later periods don't look that different and don't have \"suspiciously little variation\", as described in the data section. But it also states:\n`If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.`\n\nTherefore I conclude that this must be the case, though it seems strange that there can't be a period of sleep detected for 10-20 nights in a row. Also unlike for some other series, there are no nan events in the df for that series. It would be great to hear from the host (@gkiar07) that this is actually intended and how to interpret this.", "2446862": "Dear @fritzcremer \n\nIndeed, series `31011ade7c0a`, along with series `a596ad0b82aa`, [were both excluded in my dataset creation notebook](https://www.kaggle.com/code/carlmcbrideellis/zzzs-make-small-starter-datasets-target) and called \"truncated\" events; they both seem to have reasonable looking data, but sleep event data stopped being recorded ¿perhaps it was the annotator who nodded off? \n\nAll the best,\ncarl", "2447082": "It seems like it 😄", "2447206": "At some of those times, I suspect that the watches are not worn but they were somehow on a moving system (e.g. car, rocking crib, washing machine etc).\n\nI have also noticed some repeating patterns in some of the unworn times, as if the data was artificially created.", "2447215": "I thought about that, but then again, it is strange how similar these periods look to the data where sleep events were detected.", "2447247": "Yes, for example, the series `c5d08fc3e040` looks like it has identically repeating values!?", "2448891": "Hi @fritzcremer, I might have an explanation for the \"sleepless periods\" you mentioned. It might be due to some sort of imputation/interpolation (might explain why they look repetitive) by the host/organization to keep the data continuous for the experimental period. As others have mentioned briefly how they deal with it in this [discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438438#2439236), my approach is somewhat similar but instead of including those periods in my train data, I truncate it. For ex, I will only consider the periods until the last awake date and truncate the rest. \n\nBefore truncation:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2Fd0fcc08e21e84aa60998fc764e308b03%2Fbefore.png?generation=1695243486786671&alt=media)\n\nAfter truncation:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6261540%2F83fe5614d3345b9cc003a6655ce31959%2Fafter.png?generation=1695243499325011&alt=media)\n\nMay I ask how would you normalize the data? Shouldn't normalization be within the range of (0,1)?", "2449034": "Yes that might be a good idea.\nI normalized so that the mean is 0 and std 1.", "2449038": "Hi @nghihuynh , the middle area one between 2018-06-05 and 2018-06-11 seems like to be repetitive as well. Would you mind share your way to handle it?", "2449045": "Hi @rimbax, for the one in the middle, I keep it for training purposes because I expect the test data to have similar patterns. Thus, it would be beneficial for any models developed during training learn those patterns as well.", "2449046": "> because I expect the test data might have similar patterns\n\nAh great then. Initially I want to remove the repetitive middle one but according to this I made up my mind. Thank you for sharing and good luck!", "2452700": "I divide periods of insomnia into 2 types:\n1. Really random small values\n2. This is very similar to a dream, but the data is just not labeled\n\nThe data from point 1 should be included in the training set.\n\nThe data from point 2 can have 2 types of interpretation:\nIndeed, there are problems with sleep, the child does not sleep at night, lies down, tosses and turns, and some diseases go away. Well, you never know - maybe this is not familiar to us.\nData markup error.\n\nHow can we check this?\nCreate both datasets and look quickly. If there are significant differences, we will understand which case is ours.\n\nHas anyone already tested this hypothesis?\n\n@carlmcbrideellis @dan3dewey @fritzcremer @nghihuynh @rimbax @tolgadincer", "2474897": "I believe there are artificial, \"filler\" data in regions/days of the time series. I was able to detect this by comparing the anglez value near middle-sleep with the anglez value + or - exactly 1 day away. If they are the same to 1e-6 then it's likely filler data. For example, below is my (cira 20th Century 🙃) code for checking:\n\n```\n# Check if the anglez value is identical 24 hours away\n night_is_filler = False\n # Check this and following night\n if int(middle_of_night)+24*60*12 < max_step_in_series:\n if np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n this_series.loc[int(middle_of_night)+24*60*12,'anglez'],\n rtol=0.0, atol=1e-6):\n night_is_filler = True\n # Check this and previous night\n if int(middle_of_night)-24*60*12 > 0:\n if np.isclose(this_series.loc[int(middle_of_night),'anglez'],\n this_series.loc[int(middle_of_night)-24*60*12,'anglez'],\n rtol=0.0, atol=1e-6):\n night_is_filler = True\n```", "2483901": "I can personally testify to having no valid sleep window for 10 - 20 nights in a row 🥲\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F471225%2F8e1344f55ac35be95bdf529ab1523dbc%2Fthisisfine%20-%20Copy.png?generation=1697432103219630&alt=media)", "2485302": "> Red is the normalized anglez, green is the normalized enmo, and blue = 1 means awake, blue = 0 mean asleep\n\n@fritzcremer Looking at the range of y-axis it looks to me like `anglez` is normalized while `enmo` is **not normalized**. What am I missing?\n\nI used `sklearn.preprocessing.StandardScaler` to process both the columns and we have the following summary and charts both before and after scaling. The std of the columns anglez and enmo are not one. But, after scaling they come very close to one.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Ff4c3ea1e5e3a21f63927a23b770ff5d9%2FScreenshot%202023-10-17%20at%2010.42.57.png?generation=1697519791540874&alt=media)\n\n| `enmo` before scaling - Range: `[0, 2.5]` | `enmo` after scaling. Range - `[0, 30]` |\n| --- | --- |\n| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fa1f1b1dea85748cb2de3c9433f81a470%2FScreenshot%202023-10-17%20at%2010.43.27.png?generation=1697519884186191&alt=media) | ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Ff131909ebec2ab7cd16abe0f3a683684%2FScreenshot%202023-10-17%20at%2010.43.14.png?generation=1697519937034112&alt=media) |\n\nBut, for `anglez` we see the following:\n\n| `anglez` before scaling - Range: `[-90, 90]` | `anglez` after scaling - Range: `[-2, 2]` |\n| --- | --- |\n| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F6825a46dd86f512a379f8347a7b5fed5%2FScreenshot%202023-10-17%20at%2010.43.37.png?generation=1697520017029002&alt=media) | ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fc37056352b0f1dd92a2d4b7deec9bab4%2FScreenshot%202023-10-17%20at%2010.43.49.png?generation=1697520032600914&alt=media) |", "2485341": "Ref: [Jupyter notebook: Annotations per day . Detect Sleep States](https://www.kaggle.com/bkowshik/annotations-per-day-detect-sleep-states)\n\nFor every series, calculated a metric called `events_per_day` which is the number of non-null annotations we have divided by the range (max timestamp - min timestamp) of the series. The idea is to capture the amont of annotated duration for the different series.\n\n- Some series with low values of `events_per_day` are: `[390b487231ce, a3e59c2ce3f6, 0f9e60a8e56d]`\n- Some series with high values of `events_per_day` are: `[d515236bdeec, a9e5f5314bcb, c535634d7dcd]`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F1eeb86f08530becb3b73c8c58ef710fb%2FScreenshot%202023-10-17%20at%2011.17.35.png?generation=1697521832974009&alt=media)\n\nFrom a histogram point of view, have both the normal as well as the cumulative version below:\n\n- Approx `30%` of the series have events per day less than one.\n- Approx `50%` of the series have events per day less than `1.5`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fe8af83c0778582a2ecea8001fe20d552%2FScreenshot%202023-10-17%20at%2011.17.23.png?generation=1697521858396965&alt=media)\n\nAlthough, I am considering the range of the timeseries and not just events per series, which means that the number of events are normalized in some fashion, I wanted to look if there was any relationship between the range of the series (short series vs long series) and the number of annotated events. Two kinds of relationships stand-out\n\n1. As the range in days increases, the number of annotated events increases too. [This is expected ✅]\n2. For series with range between `[20, 25]` there seems to be a wide distribution in number of annotated events. [Wonder why 🤔]\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2Fd3e61133a03e718d4bab1a3fd4e63bff%2FScreenshot%202023-10-17%20at%2011.56.26.png?generation=1697524011447223&alt=media)", "2513686": "ryanholbrook @gkiar07 could you give some information about this topic? I still do not understand the reason of such series." }, "source": "meta" }