![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F356605%2F5523e15009e239759537fccdbdab4dff%2Fplot_03d92c9f6f8a.png?generation=1694276111370023&alt=media\")
For series_id 03d92c9f6f8a, it seems like there is some duplicate data that must be corrupt or an error in data collection or processing. Here's a plot showing the full data series in the first panel, and two of the later dates in the lower two panels. Blue and orange lines are transformed values of anglez and enmo. In the first panel, grey shading indicates annotated events.
\n
The following code confirms that the values are indeed duplicates on those dates:
\ntrain_series = pd()\n\ntrain_series = pd(train_series,\n format=,\n utc=False)\ntrain_series = train_series(lambda x: x())\ntrain_series = train_series(lambda x: x())\n\nid = \ndates_check = \nanglez_0 = train_series == id) &\n (train_series == dates_check)]\nanglez_1 = train_series == id) &\n (train_series == dates_check)]\n\nnp(anglez_0 == anglez_1)\nNote that there are no annotated events during the days with suspicious data, so in some sense this isn't terrible -- but training on all of this duplicate data seems like a bad idea. I saw this in a spot check of 3 of the series, haven't looked at others yet.
", "messages": [ { "id": "2430848", "postDate": "09/09/2023 16:22:53", "content": "For series_id 03d92c9f6f8a, it seems like there is some duplicate data that must be corrupt or an error in data collection or processing. Here's a plot showing the full data series in the first panel, and two of the later dates in the lower two panels. Blue and orange lines are transformed values of anglez and enmo. In the first panel, grey shading indicates annotated events.
\n
The following code confirms that the values are indeed duplicates on those dates:
\ntrain_series = pd()\n\ntrain_series = pd(train_series,\n format=,\n utc=False)\ntrain_series = train_series(lambda x: x())\ntrain_series = train_series(lambda x: x())\n\nid = \ndates_check = \nanglez_0 = train_series == id) &\n (train_series == dates_check)]\nanglez_1 = train_series == id) &\n (train_series == dates_check)]\n\nnp(anglez_0 == anglez_1)\nNote that there are no annotated events during the days with suspicious data, so in some sense this isn't terrible -- but training on all of this duplicate data seems like a bad idea. I saw this in a spot check of 3 of the series, haven't looked at others yet.
", "rawMarkdown": "For series_id 03d92c9f6f8a, it seems like there is some duplicate data that must be corrupt or an error in data collection or processing. Here's a plot showing the full data series in the first panel, and two of the later dates in the lower two panels. Blue and orange lines are transformed values of anglez and enmo. In the first panel, grey shading indicates annotated events. \n\n\n\nThe following code confirms that the values are indeed duplicates on those dates:\n\n```\ntrain_series = pd.read_parquet('../child-mind-institute-detect-sleep-states/train_series.parquet')\n\ntrain_series['timestamp'] = pd.to_datetime(train_series['timestamp'],\n format='%Y-%m-%dT%H:%M:%S%z',\n utc=False)\ntrain_series['date'] = train_series['timestamp'].apply(lambda x: x.date())\ntrain_series['time'] = train_series['timestamp'].apply(lambda x: x.time())\n\nid = '03d92c9f6f8a'\ndates_check = [datetime.date(2018,6,22), datetime.date(2018,6,23)]\nanglez_0 = train_series.loc[(train_series['series_id'] == id) &\n (train_series['date'] == dates_check[0])]['anglez'].values\nanglez_1 = train_series.loc[(train_series['series_id'] == id) &\n (train_series['date'] == dates_check[1])]['anglez'].values\n\nnp.all(anglez_0 == anglez_1)\n```\n\nNote that there are no annotated events during the days with suspicious data, so in some sense this isn't terrible -- but training on all of this duplicate data seems like a bad idea. I saw this in a spot check of 3 of the series, haven't looked at others yet.", "votes": null }, { "id": "2430865", "postDate": "09/09/2023 16:29:10", "content": "if I had to guess …. they padded the data ( maybe the participant took off the device ) to keep the contest easier to manage ( i.e. length consistent time series ) .. wouldnt be surprised if it happened other places as well -- good catch 👍
", "rawMarkdown": "if I had to guess .... they padded the data ( maybe the participant took off the device ) to keep the contest easier to manage ( i.e. length consistent time series ) .. wouldnt be surprised if it happened other places as well -- good catch 👍", "votes": null }, { "id": "2431224", "postDate": "09/09/2023 22:42:55", "content": "Thanks @evanray. It doesn't matter what kind of time series data it is, as it will not be learned in my case unless it is defined as \"night\"(i.e. before and after the gray zone).
", "rawMarkdown": "Thanks @evanray. It doesn't matter what kind of time series data it is, as it will not be learned in my case unless it is defined as \"night\"(i.e. before and after the gray zone).", "votes": null } ], "comments": [ { "id": 2430865, "author_name": "scharf", "author_url": "", "post_date": "09/09/2023 16:29:10", "content": "if I had to guess …. they padded the data ( maybe the participant took off the device ) to keep the contest easier to manage ( i.e. length consistent time series ) .. wouldnt be surprised if it happened other places as well -- good catch 👍
", "votes": null, "replies": [] }, { "id": 2431224, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/09/2023 22:42:55", "content": "Thanks @evanray. It doesn't matter what kind of time series data it is, as it will not be learned in my case unless it is defined as \"night\"(i.e. before and after the gray zone).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2430848": "For series_id 03d92c9f6f8a, it seems like there is some duplicate data that must be corrupt or an error in data collection or processing. Here's a plot showing the full data series in the first panel, and two of the later dates in the lower two panels. Blue and orange lines are transformed values of anglez and enmo. In the first panel, grey shading indicates annotated events. \n\n\n\nThe following code confirms that the values are indeed duplicates on those dates:\n\n```\ntrain_series = pd.read_parquet('../child-mind-institute-detect-sleep-states/train_series.parquet')\n\ntrain_series['timestamp'] = pd.to_datetime(train_series['timestamp'],\n format='%Y-%m-%dT%H:%M:%S%z',\n utc=False)\ntrain_series['date'] = train_series['timestamp'].apply(lambda x: x.date())\ntrain_series['time'] = train_series['timestamp'].apply(lambda x: x.time())\n\nid = '03d92c9f6f8a'\ndates_check = [datetime.date(2018,6,22), datetime.date(2018,6,23)]\nanglez_0 = train_series.loc[(train_series['series_id'] == id) &\n (train_series['date'] == dates_check[0])]['anglez'].values\nanglez_1 = train_series.loc[(train_series['series_id'] == id) &\n (train_series['date'] == dates_check[1])]['anglez'].values\n\nnp.all(anglez_0 == anglez_1)\n```\n\nNote that there are no annotated events during the days with suspicious data, so in some sense this isn't terrible -- but training on all of this duplicate data seems like a bad idea. I saw this in a spot check of 3 of the series, haven't looked at others yet.", "2430865": "if I had to guess .... they padded the data ( maybe the participant took off the device ) to keep the contest easier to manage ( i.e. length consistent time series ) .. wouldnt be surprised if it happened other places as well -- good catch 👍", "2431224": "Thanks @evanray. It doesn't matter what kind of time series data it is, as it will not be learned in my case unless it is defined as \"night\"(i.e. before and after the gray zone)." }, "source": "meta" }