![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2Fb338eb4a93e0a38e605baba4616f14e3%2FScreen%20Shot%202023-10-20%20at%2014.26.21.png?generation=1697822938043936&alt=media\")
@ryanholbrook could you check if there are still data inconsistencies after the data update? For instance, I found for series 2f7504d0f426 that there is 1 hour missing between steps 200879 and 200880:
Thanks :)
", "messages": [ { "id": "2490458", "postDate": "10/20/2023 17:32:08", "content": "@ryanholbrook could you check if there are still data inconsistencies after the data update? For instance, I found for series 2f7504d0f426 that there is 1 hour missing between steps 200879 and 200880:
Thanks :)
", "rawMarkdown": "ryanholbrook could you check if there are still data inconsistencies after the data update? For instance, I found for series `2f7504d0f426` that there is 1 hour missing between steps `200879` and `200880`:\n\n\n\nThanks :)", "votes": null }, { "id": "2490469", "postDate": "10/20/2023 17:51:32", "content": "I was going to suggest that was when daylight saving time switched over (it's at 2am in the Spring, so losing an hour), but Google says Daylight savings time in 2018 was on March 11, not March 18. yes, odd.
", "rawMarkdown": "I was going to suggest that was when daylight saving time switched over (it's at 2am in the Spring, so losing an hour), but Google says Daylight savings time in 2018 was on March 11, not March 18. yes, odd.", "votes": null }, { "id": "2490500", "postDate": "10/20/2023 18:11:33", "content": "Some other weird series:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F0f0b1e571e2da57c74fe18537bbd726c%2FScreen%20Shot%202023-10-20%20at%2015.10.51.png?generation=1697825480730381&alt=media\")
This one goes back and forth in time.
", "rawMarkdown": "Some other weird series: \n\n\nThis one goes back and forth in time.", "votes": null }, { "id": "2490509", "postDate": "10/20/2023 18:18:01", "content": "I think that data should be checked for consecutive jumps of 5 seconds. If not, the data is wrong
", "rawMarkdown": "I think that data should be checked for consecutive jumps of 5 seconds. If not, the data is wrong", "votes": null }, { "id": "2490514", "postDate": "10/20/2023 18:24:48", "content": "So that one matches with daylight savings time ending in 2018, but is also a week off (It was Nov 4, not Nov 11 that year). So it looks like the data could be all a week off?
", "rawMarkdown": "So that one matches with daylight savings time ending in 2018, but is also a week off (It was Nov 4, not Nov 11 that year). So it looks like the data could be all a week off?", "votes": null }, { "id": "2490551", "postDate": "10/20/2023 19:17:55", "content": "I truly don't know, but I have found numerous cases where there's either data missing or these \"back in time\" cases
", "rawMarkdown": "I truly don't know, but I have found numerous cases where there's either data missing or these \"back in time\" cases", "votes": null }, { "id": "2490626", "postDate": "10/20/2023 20:56:08", "content": "There is no time missing. Row 25209359 is 1:59:55 and timezone UTC-5. Then the next row 25209360 is 3:00:00 and timezone UTC-4. If we convert both to UTC time zone, then the first is 6:59:55 and the second is 7:00:00
\nWhen converting this column from string to datetime, we need to convert everything to the same timezone. I have confirmed that the entire train data has 5 second gap between every consecutive row.
", "rawMarkdown": "There is no time missing. Row 25209359 is 1:59:55 and timezone UTC-5. Then the next row 25209360 is 3:00:00 and timezone UTC-4. If we convert both to UTC time zone, then the first is 6:59:55 and the second is 7:00:00\n\nWhen converting this column from string to datetime, we need to convert everything to the same timezone. I have confirmed that the entire train data has 5 second gap between every consecutive row.", "votes": null }, { "id": "2490690", "postDate": "10/21/2023 00:15:41", "content": "Thanks Chris, I am using datetime’s strptime method. Maybe its an issue with this conversion method. Which one are you using?
", "rawMarkdown": "Thanks Chris, I am using datetime’s strptime method. Maybe its an issue with this conversion method. Which one are you using?", "votes": null }, { "id": "2491092", "postDate": "10/21/2023 10:22:47", "content": "Perhaps they crossed a timezone border?
", "rawMarkdown": "Perhaps they crossed a timezone border?", "votes": null }, { "id": "2491341", "postDate": "10/21/2023 14:33:35", "content": "I'm using cudf.to_datetime() which converts everything to UTC timezone (and afterward every row is 5 seconds difference). I've noticed that the pd.to_datetime() does not convert everything to the same timezone. Pandas keeps the original timezones.
With Pandas we can do the following to cast all times to UTC timezone. Then we will see no gaps in df.timestamp.dt.hour:
import pytz\ndefault_datetime = \ndf = df(default_datetime)\ndf = pd(df, format=)\ndf = df(lambda x: x(pytz.utc))\ndf = pd\n@moth: There is a Daylight Time Saving (DTS) change labelling issue in the timestamps of the raw dataset. It is the root cause of what you see (and, yes, it does not manifest if you cast the timestamps to UTC). You may follow the notebook per https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for the full assessment/analysis of this problem.
", "rawMarkdown": "moth: There is a Daylight Time Saving (DTS) change labelling issue in the timestamps of the raw dataset. It is the root cause of what you see (and, yes, it does not manifest if you cast the timestamps to UTC). You may follow the notebook per https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for the full assessment/analysis of this problem.", "votes": null }, { "id": "2491348", "postDate": "10/21/2023 14:38:28", "content": "@chris62 , @alejopaullier : yes, the root cause of this issue is the wrong DST change date used in the raw dataset. You can https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for more detailed analysis of the glitch.
", "rawMarkdown": "chris62 , @alejopaullier : yes, the root cause of this issue is the wrong DST change date used in the raw dataset. You can https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for more detailed analysis of the glitch.", "votes": null }, { "id": "2491350", "postDate": "10/21/2023 14:40:39", "content": "@fritzcremer : yes, crossing the DST change dates (along with the wrong DST change dates used in the raw timestamps) is the reason for the issues observed. I posted the link to my notebook with the detailed analysis of the problem in earlier comments to this thread.
", "rawMarkdown": "fritzcremer : yes, crossing the DST change dates (along with the wrong DST change dates used in the raw timestamps) is the reason for the issues observed. I posted the link to my notebook with the detailed analysis of the problem in earlier comments to this thread.", "votes": null }, { "id": "2491379", "postDate": "10/21/2023 15:23:06", "content": "Hi @alejopaullier, not sure if this is still an issue for you. However I tried using cudf as per @cdeotte but it runs out of memory on kaggle (I assume he's using his own hardware). However when loading the data using polars, adding time_zone='UTC' fixes this problem (as shown below):
import polars pl\n\ndt_transforms = \n\ntrn_series = (\n pl.scan\n .)\n .collect\n .\n)\nAnd the results look like:
\n![\"fixed](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F557179%2F8e04b6e820806ba60fe87b1c7ac880b1%2FScreenshot%202023-10-21%20at%2011.19.32%20AM.png?generation=1697901601310187&alt=media\")
This also seems to be fairly fast and can be applied during submission.
", "rawMarkdown": "Hi @alejopaullier, not sure if this is still an issue for you. However I tried using `cudf` as per @cdeotte but it runs out of memory on kaggle (I assume he's using his own hardware). However when loading the data using polars, adding `time_zone='UTC'` fixes this problem (as shown below):\n\n```\nimport polars as pl\n\ndt_transforms = [\n pl.col('timestamp').str.to_datetime(time_zone='UTC'), \n (pl.col('timestamp').str.to_datetime().dt.date()).alias('date'), \n pl.col('timestamp').str.to_datetime().dt.time().alias('time')\n]\n\ntrn_series = (\n pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n .with_columns((dt_transforms))\n .collect()\n .to_pandas()\n)\n```\n\nAnd the results look like:\n\n\nThis also seems to be fairly fast and can be applied during submission.", "votes": null }, { "id": "2491388", "postDate": "10/21/2023 15:34:19", "content": "@jjinho The best way to use cuDF is to split the original parquet into 277 parquets (one per series_id) with code here. Afterward we load one series_id parquet at a time with cuDF, then create features, then save result to disk as NumPy, then clear memory. Then to train our model, we load all series_id NumPys and concatenate into train data.
Thanks @cdeotte, that makes total sense. I missed that. And for future reference, when using cuDF, if we use the T4x2 environment, does it automatically divide data over both GPUs?
", "rawMarkdown": "Thanks @cdeotte, that makes total sense. I missed that. And for future reference, when using cuDF, if we use the T4x2 environment, does it automatically divide data over both GPUs?", "votes": null }, { "id": "2491417", "postDate": "10/21/2023 15:50:05", "content": "No, cuDF by itself will only use 1 GPU. To use 2xGPU, we need to use DASK-cuDF (which allows us to write code same as cuDF but behind the scenes DASK will utilize all GPU).
\nOne of these days, i will publish a DASK demo. There is also DASK-XGB which is XGB that uses all GPUs.
", "rawMarkdown": "No, cuDF by itself will only use 1 GPU. To use 2xGPU, we need to use DASK-cuDF (which allows us to write code same as cuDF but behind the scenes DASK will utilize all GPU).\n\nOne of these days, i will publish a DASK demo. There is also DASK-XGB which is XGB that uses all GPUs.", "votes": null }, { "id": "2491425", "postDate": "10/21/2023 15:52:27", "content": "Thanks for clarifying that and looking forward to seeing that demo.
", "rawMarkdown": "Thanks for clarifying that and looking forward to seeing that demo.", "votes": null }, { "id": "2491487", "postDate": "10/21/2023 16:53:15", "content": "By any chance has anyone taken the time to make a train dataset with the correct timestamp or at least slices? I used Andrada’s but the timestamps are wrong. I had to refactor all my code
", "rawMarkdown": "By any chance has anyone taken the time to make a train dataset with the correct timestamp or at least slices? I used Andrada’s but the timestamps are wrong. I had to refactor all my code", "votes": null }, { "id": "2491492", "postDate": "10/21/2023 17:00:22", "content": "Thanks @cdeotte after concatenating all numpy arrays when training, do you use polars to convert the array into a dataframe?
", "rawMarkdown": "Thanks @cdeotte after concatenating all numpy arrays when training, do you use polars to convert the array into a dataframe?", "votes": null }, { "id": "2491529", "postDate": "10/21/2023 17:25:56", "content": "No, i train directly with NumPy arrays.
", "rawMarkdown": "No, i train directly with NumPy arrays.", "votes": null }, { "id": "2491555", "postDate": "10/21/2023 17:47:04", "content": "I saved the outputs here: https://www.kaggle.com/code/jjinho/simple-utc-conversion-using-polars
", "rawMarkdown": "I saved the outputs here: https://www.kaggle.com/code/jjinho/simple-utc-conversion-using-polars", "votes": null } ], "comments": [ { "id": 2490469, "author_name": "chris62", "author_url": "", "post_date": "10/20/2023 17:51:32", "content": "I was going to suggest that was when daylight saving time switched over (it's at 2am in the Spring, so losing an hour), but Google says Daylight savings time in 2018 was on March 11, not March 18. yes, odd.
", "votes": null, "replies": [ { "id": 2491092, "author_name": "fritzcremer", "author_url": "", "post_date": "10/21/2023 10:22:47", "content": "Perhaps they crossed a timezone border?
", "votes": null, "replies": [ { "id": 2491350, "author_name": "gvyshnya", "author_url": "", "post_date": "10/21/2023 14:40:39", "content": "@fritzcremer : yes, crossing the DST change dates (along with the wrong DST change dates used in the raw timestamps) is the reason for the issues observed. I posted the link to my notebook with the detailed analysis of the problem in earlier comments to this thread.
", "votes": null, "replies": [] } ] } ] }, { "id": 2490500, "author_name": "alejopaullier", "author_url": "", "post_date": "10/20/2023 18:11:33", "content": "Some other weird series:
\n
This one goes back and forth in time.
", "votes": null, "replies": [ { "id": 2490514, "author_name": "chris62", "author_url": "", "post_date": "10/20/2023 18:24:48", "content": "So that one matches with daylight savings time ending in 2018, but is also a week off (It was Nov 4, not Nov 11 that year). So it looks like the data could be all a week off?
", "votes": null, "replies": [ { "id": 2490551, "author_name": "alejopaullier", "author_url": "", "post_date": "10/20/2023 19:17:55", "content": "I truly don't know, but I have found numerous cases where there's either data missing or these \"back in time\" cases
", "votes": null, "replies": [ { "id": 2491348, "author_name": "gvyshnya", "author_url": "", "post_date": "10/21/2023 14:38:28", "content": "@chris62 , @alejopaullier : yes, the root cause of this issue is the wrong DST change date used in the raw dataset. You can https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for more detailed analysis of the glitch.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2490509, "author_name": "alejopaullier", "author_url": "", "post_date": "10/20/2023 18:18:01", "content": "I think that data should be checked for consecutive jumps of 5 seconds. If not, the data is wrong
", "votes": null, "replies": [] }, { "id": 2490626, "author_name": "cdeotte", "author_url": "", "post_date": "10/20/2023 20:56:08", "content": "There is no time missing. Row 25209359 is 1:59:55 and timezone UTC-5. Then the next row 25209360 is 3:00:00 and timezone UTC-4. If we convert both to UTC time zone, then the first is 6:59:55 and the second is 7:00:00
\nWhen converting this column from string to datetime, we need to convert everything to the same timezone. I have confirmed that the entire train data has 5 second gap between every consecutive row.
", "votes": null, "replies": [ { "id": 2490690, "author_name": "alejopaullier", "author_url": "", "post_date": "10/21/2023 00:15:41", "content": "Thanks Chris, I am using datetime’s strptime method. Maybe its an issue with this conversion method. Which one are you using?
", "votes": null, "replies": [ { "id": 2491341, "author_name": "cdeotte", "author_url": "", "post_date": "10/21/2023 14:33:35", "content": "I'm using cudf.to_datetime() which converts everything to UTC timezone (and afterward every row is 5 seconds difference). I've noticed that the pd.to_datetime() does not convert everything to the same timezone. Pandas keeps the original timezones.
With Pandas we can do the following to cast all times to UTC timezone. Then we will see no gaps in df.timestamp.dt.hour:
import pytz\ndefault_datetime = \ndf = df(default_datetime)\ndf = pd(df, format=)\ndf = df(lambda x: x(pytz.utc))\ndf = pd\n@moth: There is a Daylight Time Saving (DTS) change labelling issue in the timestamps of the raw dataset. It is the root cause of what you see (and, yes, it does not manifest if you cast the timestamps to UTC). You may follow the notebook per https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for the full assessment/analysis of this problem.
", "votes": null, "replies": [] }, { "id": 2491379, "author_name": "jjinho", "author_url": "", "post_date": "10/21/2023 15:23:06", "content": "Hi @alejopaullier, not sure if this is still an issue for you. However I tried using cudf as per @cdeotte but it runs out of memory on kaggle (I assume he's using his own hardware). However when loading the data using polars, adding time_zone='UTC' fixes this problem (as shown below):
import polars pl\n\ndt_transforms = \n\ntrn_series = (\n pl.scan\n .)\n .collect\n .\n)\nAnd the results look like:
\n
This also seems to be fairly fast and can be applied during submission.
", "votes": null, "replies": [ { "id": 2491388, "author_name": "cdeotte", "author_url": "", "post_date": "10/21/2023 15:34:19", "content": "@jjinho The best way to use cuDF is to split the original parquet into 277 parquets (one per series_id) with code here. Afterward we load one series_id parquet at a time with cuDF, then create features, then save result to disk as NumPy, then clear memory. Then to train our model, we load all series_id NumPys and concatenate into train data.
Thanks @cdeotte, that makes total sense. I missed that. And for future reference, when using cuDF, if we use the T4x2 environment, does it automatically divide data over both GPUs?
", "votes": null, "replies": [ { "id": 2491417, "author_name": "cdeotte", "author_url": "", "post_date": "10/21/2023 15:50:05", "content": "No, cuDF by itself will only use 1 GPU. To use 2xGPU, we need to use DASK-cuDF (which allows us to write code same as cuDF but behind the scenes DASK will utilize all GPU).
\nOne of these days, i will publish a DASK demo. There is also DASK-XGB which is XGB that uses all GPUs.
", "votes": null, "replies": [ { "id": 2491425, "author_name": "jjinho", "author_url": "", "post_date": "10/21/2023 15:52:27", "content": "Thanks for clarifying that and looking forward to seeing that demo.
", "votes": null, "replies": [] } ] } ] }, { "id": 2491492, "author_name": "alejopaullier", "author_url": "", "post_date": "10/21/2023 17:00:22", "content": "Thanks @cdeotte after concatenating all numpy arrays when training, do you use polars to convert the array into a dataframe?
", "votes": null, "replies": [ { "id": 2491529, "author_name": "cdeotte", "author_url": "", "post_date": "10/21/2023 17:25:56", "content": "No, i train directly with NumPy arrays.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2491487, "author_name": "alejopaullier", "author_url": "", "post_date": "10/21/2023 16:53:15", "content": "By any chance has anyone taken the time to make a train dataset with the correct timestamp or at least slices? I used Andrada’s but the timestamps are wrong. I had to refactor all my code
", "votes": null, "replies": [ { "id": 2491555, "author_name": "jjinho", "author_url": "", "post_date": "10/21/2023 17:47:04", "content": "I saved the outputs here: https://www.kaggle.com/code/jjinho/simple-utc-conversion-using-polars
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2490458": "ryanholbrook could you check if there are still data inconsistencies after the data update? For instance, I found for series `2f7504d0f426` that there is 1 hour missing between steps `200879` and `200880`:\n\n\n\nThanks :)", "2490469": "I was going to suggest that was when daylight saving time switched over (it's at 2am in the Spring, so losing an hour), but Google says Daylight savings time in 2018 was on March 11, not March 18. yes, odd.", "2490500": "Some other weird series: \n\n\nThis one goes back and forth in time.", "2490509": "I think that data should be checked for consecutive jumps of 5 seconds. If not, the data is wrong", "2490514": "So that one matches with daylight savings time ending in 2018, but is also a week off (It was Nov 4, not Nov 11 that year). So it looks like the data could be all a week off?", "2490551": "I truly don't know, but I have found numerous cases where there's either data missing or these \"back in time\" cases", "2490626": "There is no time missing. Row 25209359 is 1:59:55 and timezone UTC-5. Then the next row 25209360 is 3:00:00 and timezone UTC-4. If we convert both to UTC time zone, then the first is 6:59:55 and the second is 7:00:00\n\nWhen converting this column from string to datetime, we need to convert everything to the same timezone. I have confirmed that the entire train data has 5 second gap between every consecutive row.", "2490690": "Thanks Chris, I am using datetime’s strptime method. Maybe its an issue with this conversion method. Which one are you using?", "2491092": "Perhaps they crossed a timezone border?", "2491341": "I'm using `cudf.to_datetime()` which converts everything to UTC timezone (and afterward every row is 5 seconds difference). I've noticed that the `pd.to_datetime()` does not convert everything to the same timezone. Pandas keeps the original timezones.\n\nWith Pandas we can do the following to cast all times to UTC timezone. Then we will see no gaps in `df.timestamp.dt.hour`:\n\n import pytz\n default_datetime = \"1970-01-01T00:00:00+0000\"\n df['timestamp'] = df['timestamp'].fillna(default_datetime)\n df['timestamp'] = pd.to_datetime(df['timestamp'], format='%Y-%m-%dT%H:%M:%S%z')\n df['timestamp'] = df['timestamp'].apply(lambda x: x.astimezone(pytz.utc))\n df.loc[df.timestamp.dt.year==1970,'timestamp'] = pd.NaT", "2491345": "moth: There is a Daylight Time Saving (DTS) change labelling issue in the timestamps of the raw dataset. It is the root cause of what you see (and, yes, it does not manifest if you cast the timestamps to UTC). You may follow the notebook per https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for the full assessment/analysis of this problem.", "2491348": "chris62 , @alejopaullier : yes, the root cause of this issue is the wrong DST change date used in the raw dataset. You can https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check for more detailed analysis of the glitch.", "2491350": "fritzcremer : yes, crossing the DST change dates (along with the wrong DST change dates used in the raw timestamps) is the reason for the issues observed. I posted the link to my notebook with the detailed analysis of the problem in earlier comments to this thread.", "2491379": "Hi @alejopaullier, not sure if this is still an issue for you. However I tried using `cudf` as per @cdeotte but it runs out of memory on kaggle (I assume he's using his own hardware). However when loading the data using polars, adding `time_zone='UTC'` fixes this problem (as shown below):\n\n```\nimport polars as pl\n\ndt_transforms = [\n pl.col('timestamp').str.to_datetime(time_zone='UTC'), \n (pl.col('timestamp').str.to_datetime().dt.date()).alias('date'), \n pl.col('timestamp').str.to_datetime().dt.time().alias('time')\n]\n\ntrn_series = (\n pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n .with_columns((dt_transforms))\n .collect()\n .to_pandas()\n)\n```\n\nAnd the results look like:\n\n\nThis also seems to be fairly fast and can be applied during submission.", "2491388": "jjinho The best way to use cuDF is to split the original parquet into 277 parquets (one per `series_id`) with code [here][1]. Afterward we load one `series_id` parquet at a time with cuDF, then create features, then save result to disk as NumPy, then clear memory. Then to train our model, we load all `series_id` NumPys and concatenate into train data.\n\n[1]: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/447593", "2491393": "Thanks @cdeotte, that makes total sense. I missed that. And for future reference, when using cuDF, if we use the T4x2 environment, does it automatically divide data over both GPUs?", "2491417": "No, cuDF by itself will only use 1 GPU. To use 2xGPU, we need to use DASK-cuDF (which allows us to write code same as cuDF but behind the scenes DASK will utilize all GPU).\n\nOne of these days, i will publish a DASK demo. There is also DASK-XGB which is XGB that uses all GPUs.", "2491425": "Thanks for clarifying that and looking forward to seeing that demo.", "2491487": "By any chance has anyone taken the time to make a train dataset with the correct timestamp or at least slices? I used Andrada’s but the timestamps are wrong. I had to refactor all my code", "2491492": "Thanks @cdeotte after concatenating all numpy arrays when training, do you use polars to convert the array into a dataframe?", "2491529": "No, i train directly with NumPy arrays.", "2491555": "I saved the outputs here: https://www.kaggle.com/code/jjinho/simple-utc-conversion-using-polars" }, "source": "meta" }