{ "id": 440697, "title": "Offset Events", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/440697", "author_name": "Ryan Holbrook", "post_date": "2023-09-15T20:33:59.383000", "votes": 68, "comment_count": 43, "views": 0, "content": "

Hi everyone,

\n

There were some series affected by a bug in DST handling that caused events after a time change to be incorrectly offset by an hour. Apparently, this only affected series in which a DST occurred (that is, when the time zone offset changed from \"-400\" to \"-500\"), which is 36 series in the training set (perhaps 6% of all events).

\n

I'll be updating the dataset on Monday with the correct event times and, as this also affected series in the test set, initiating a rescore. Since the portion of affected events is relatively small, I'm anticipating the effect on leaderboard scores will also likely not be large.

\n

I'll update this post once everything is complete.

", "messages": [ { "id": 2440944, "postDate": "2023-09-15T20:33:59.383Z", "content": "

Hi everyone,

\n

There were some series affected by a bug in DST handling that caused events after a time change to be incorrectly offset by an hour. Apparently, this only affected series in which a DST occurred (that is, when the time zone offset changed from \"-400\" to \"-500\"), which is 36 series in the training set (perhaps 6% of all events).

\n

I'll be updating the dataset on Monday with the correct event times and, as this also affected series in the test set, initiating a rescore. Since the portion of affected events is relatively small, I'm anticipating the effect on leaderboard scores will also likely not be large.

\n

I'll update this post once everything is complete.

", "rawMarkdown": "Hi everyone,\n\nThere were some series affected by a bug in DST handling that caused events after a time change to be incorrectly offset by an hour. Apparently, this only affected series in which a DST occurred (that is, when the time zone offset changed from \"-400\" to \"-500\"), which is 36 series in the training set (perhaps 6% of all events).\n\nI'll be updating the dataset on Monday with the correct event times and, as this also affected series in the test set, initiating a rescore. Since the portion of affected events is relatively small, I'm anticipating the effect on leaderboard scores will also likely not be large.\n\nI'll update this post once everything is complete.", "votes": 67 }, { "id": 2448756, "postDate": "2023-09-20T19:27:41.767Z", "content": "

Okay, update round two is done (data refresh and rescore). I believe this resolved the issue with the offset events and the duplicated events. The series themselves should not have changed, only the data in train_events.csv. I ran some extra tests to try to ensure all annotated events are consistent with the DST change, but please let us know if you notice anything out of sorts.

", "rawMarkdown": "Okay, update round two is done (data refresh and rescore). I believe this resolved the issue with the offset events and the duplicated events. The series themselves should not have changed, only the data in `train_events.csv`. I ran some extra tests to try to ensure all annotated events are consistent with the DST change, but please let us know if you notice anything out of sorts.", "votes": 21 }, { "id": 2445572, "postDate": "2023-09-19T00:39:21.397Z", "content": "

I think a reason for the decrease in leader board scores is that the train_events.cvs now has event step values that do not agree with the data series transition values for some series when there is a DST change.

\n

The series 655f19eabf1e had a double night 3 onset and now onsets/wakeups after that actually occur about an hour later than the training event file values.

\n

In the image below, 8a306e0890c0 (which had a double onset at night 11) now shows nights 12 and following ones as having their real transition (red) about an hour later than the events file value (orange.)

\n

\"\"

", "rawMarkdown": "I think a reason for the decrease in leader board scores is that the train_events.cvs now has event step values that do not agree with the data series transition values for some series when there is a DST change.\n\nThe series 655f19eabf1e had a double night 3 onset and now onsets/wakeups after that actually occur about an hour later than the training event file values.\n\nIn the image below, 8a306e0890c0 (which had a double onset at night 11) now shows nights 12 and following ones as having their real transition (red) about an hour later than the events file value (orange.)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Fd600d31b071a9ac93187e1925a8bada0%2Fmismatch_event_series_steps.png?generation=1695083476020478&alt=media)", "votes": 9, "replies": [ { "id": 2445595, "postDate": "2023-09-19T01:07:22.550Z", "content": "

Hi @dan3dewey good findings. What is the gray, blue, red, and orange line refer to?

", "rawMarkdown": "Hi @dan3dewey good findings. What is the gray, blue, red, and orange line refer to?" }, { "id": 2446086, "postDate": "2023-09-19T08:35:33.897Z", "content": "

Interesting, so these 3 duplicate onset events are now gone. But is the labeling now not correct after these DST events? Is that what you are saying?

", "rawMarkdown": "Interesting, so these 3 duplicate onset events are now gone. But is the labeling now not correct after these DST events? Is that what you are saying?", "votes": 2 }, { "id": 2446217, "postDate": "2023-09-19T10:18:51.710Z", "content": "

I'm seeing the same thing.

\n

All labels after the (now corrected) \"double onset nights\" appear to be incorrectly shifted forward by an hour for series 655f19eabf1e, 8a306e0890c0, and c6788e579967.

\n

Edit:
\n Upon spot checking other DST affected series, it seems like this issue is fairly widespread. Specifically, I see potential problems with the labels for:

\n\n

Anyone else want to take a look at some of these series? Because it's absolutely possible that it's just an error on my end.

", "rawMarkdown": "I'm seeing the same thing. \n\nAll labels *after* the (now corrected) \"double onset nights\" appear to be incorrectly shifted forward by an hour for series 655f19eabf1e, 8a306e0890c0, and c6788e579967. \n\n**Edit:**\n Upon spot checking [other DST affected series](https://www.kaggle.com/code/quillen/sleepstates-daylight-saving-time), it seems like this issue is fairly widespread. Specifically, I see potential problems with the labels for:\n- later nights in series where clock shifts back 1 hour: `4a31811f3558`, `a9a2f7fac455`, `416354edd92a`\n- earlier nights in series where clock shifts forward 1 hour: `bfe41e96d12f`, `2f7504d0f426`\n\nAnyone else want to take a look at some of these series? Because it's absolutely possible that it's just an error on my end.", "votes": 4, "replies": [ { "id": 2446620, "postDate": "2023-09-19T14:06:11.867Z", "content": "

I am seeing the same thing. From what I can tell the change impacted labels for ~800 events in 38 series.

\n

When I plot the old vs new labels I can see the offset looks to have made the labels less in-line with the features for most of these events.

\n

My cross validation score is also worse when using the new labels and I would think it would improve if mislabels were fixed.

\n

I didn't re-download the parquet files, just the events file- assuming that was the only file that changed- so maybe that's the cause? Could also be an error in my code.

\n

Hopefully @ryanholbrook can give us an update.

", "rawMarkdown": "I am seeing the same thing. From what I can tell the change impacted labels for ~800 events in 38 series.\n\nWhen I plot the old vs new labels I can see the offset looks to have made the labels less in-line with the features for most of these events.\n\nMy cross validation score is also worse when using the new labels and I would think it would improve if mislabels were fixed.\n\nI didn't re-download the parquet files, just the events file- assuming that was the only file that changed- so maybe that's the cause? Could also be an error in my code.\n\nHopefully @ryanholbrook can give us an update.", "votes": 7 }, { "id": 2446632, "postDate": "2023-09-19T14:13:38.753Z", "content": "

Thanks, I'll take a look.

", "rawMarkdown": "Thanks, I'll take a look.", "votes": 2 }, { "id": 2446705, "postDate": "2023-09-19T14:55:20.930Z", "content": "

I made this notebook showing the old/new events that might be helpful: https://www.kaggle.com/code/robikscube/sleep-states-events-file-update-viz/

", "rawMarkdown": "I made this notebook showing the old/new events that might be helpful: https://www.kaggle.com/code/robikscube/sleep-states-events-file-update-viz/", "votes": 3 }, { "id": 2446990, "postDate": "2023-09-19T18:47:32.567Z", "content": "

Okay, I think I've identified a fix. I'll try to have an update sometime tomorrow.

\n

And thanks again for your patience. This has been a tough bug to squash.

", "rawMarkdown": "Okay, I think I've identified a fix. I'll try to have an update sometime tomorrow.\n\nAnd thanks again for your patience. This has been a tough bug to squash.", "votes": 10 }, { "id": 2447111, "postDate": "2023-09-19T20:37:56.700Z", "content": "

i think the nature of the problem has been misunderstood here: as long as 'step' is defined consistently in 'train_series' and 'train_events', it does not matter much if time label is off (it would impact some time-based features, but this impact would be second order effect). And 'step' was defined consistently. So correction would have to impact both files, to preserve their consistency, and that is not what was done here.

", "rawMarkdown": "i think the nature of the problem has been misunderstood here: as long as 'step' is defined consistently in 'train_series' and 'train_events', it does not matter much if time label is off (it would impact some time-based features, but this impact would be second order effect). And 'step' **was** defined consistently. So correction would have to impact **both** files, to preserve their consistency, and that is not what was done here.", "votes": 3 }, { "id": 2447395, "postDate": "2023-09-20T03:54:49.823Z", "content": "

I concur with @ymatioun regarding needing to also correct the series data as well. All of the predicted events for the effected series became off by an hour (720 steps) and thus are outside of all the thresholds and led to a score of 0 for said series i.e. the event data shifted, but the actual accelerometer data did not.

\n
\n

\"it would impact some time-based features, but this impact would be second order effect\"

\n
\n

Feature importance indicates that the hour feature is very influential, and I suggest it is for this reason there was an almost universal drop of around 10% in all of the LB scores after the re-score.

\n

All the best,
\ncarl

", "rawMarkdown": "I concur with @ymatioun regarding needing to also correct the `series` data as well. All of the predicted events for the effected series became off by an hour (720 steps) and thus are outside of all the thresholds and led to a score of 0 for said series *i.e.* the event data shifted, but the actual accelerometer data did not.\n\n> \"*it would impact some time-based features, but this impact would be second order effect*\"\n\nFeature importance indicates that the `hour` feature is *very* influential, and I suggest it is for this reason there was an almost universal drop of around 10% in all of the LB scores after the re-score. \n\nAll the best,\ncarl", "votes": 2 } ] }, { "id": 2468582, "postDate": "2023-10-05T16:14:04.620Z", "content": "

THANKS !!!

", "rawMarkdown": "THANKS !!!\n" } ] }, { "id": 2445214, "postDate": "2023-09-18T16:53:27.437Z", "content": "

The update and rescore is complete. Please let us know if you have any questions or concerns.

", "rawMarkdown": "The update and rescore is complete. Please let us know if you have any questions or concerns.", "votes": 7, "replies": [ { "id": 2445376, "postDate": "2023-09-18T19:23:03.987Z", "content": "

Thanks Ryan. Can you confirm that train_events.csv was the only file in the public dataset that was impacted?

", "rawMarkdown": "Thanks Ryan. Can you confirm that `train_events.csv` was the only file in the public dataset that was impacted?", "votes": 2, "replies": [ { "id": 2445409, "postDate": "2023-09-18T19:53:54.183Z", "content": "

Hi @robikscube, that should be the case. If it seems to be otherwise, please let me know.

", "rawMarkdown": "Hi @robikscube, that should be the case. If it seems to be otherwise, please let me know.", "votes": 2 } ] } ] }, { "id": 2443400, "postDate": "2023-09-17T17:35:41.157Z", "content": "

What a relief. Looking forward to this.

\n

Will the \"onset but not wakeup\" or \"wakeup but not onset\" events like here also be fixed?

", "rawMarkdown": "What a relief. Looking forward to this.\n\nWill the \"onset but not wakeup\" or \"wakeup but not onset\" events [like here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438112) also be fixed?", "votes": 3, "replies": [ { "id": 2445299, "postDate": "2023-09-18T18:07:42.403Z", "content": "

The duplicated events should also be fixed, yes.

", "rawMarkdown": "The duplicated events should also be fixed, yes.", "votes": 1, "replies": [ { "id": 2445555, "postDate": "2023-09-19T00:01:14.080Z", "content": "

Thank you @ryanholbrook , glad to hear.

", "rawMarkdown": "Thank you @ryanholbrook , glad to hear." } ] } ] }, { "id": 2443034, "postDate": "2023-09-17T13:41:21.483Z", "content": "

Will the \"step\" values in the series always increment by 1 (5 seconds) from row to row? That is, will a DST change affect the timestamp but not cause any change to the sequential step values?

", "rawMarkdown": "Will the \"step\" values in the series always increment by 1 (5 seconds) from row to row? That is, will a DST change affect the timestamp but not cause any change to the sequential step values?", "votes": 3, "replies": [ { "id": 2443080, "postDate": "2023-09-17T14:11:06.877Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2464134, "postDate": "2023-10-02T02:12:02.093Z", "content": "

@ryanholbrook Are the switches between -0400 and -0500 happening at correct dates? The timestamp string representations are not consistent with pandas 'US/Eastern' timezone. As far as I searched, the end of daylight saving time is 2018-11-04 in US, but on 2018-11-11 in the data. The start and end of daylight saving times seem to be 1 week later than it should be.

\n

Code: https://www.kaggle.com/junkoda/checking-daylight-saving-times

", "rawMarkdown": "@ryanholbrook Are the switches between -0400 and -0500 happening at correct dates? The timestamp string representations are not consistent with pandas 'US/Eastern' timezone. As far as I searched, the end of daylight saving time is 2018-11-04 in US, but on 2018-11-11 in the data. The start and end of daylight saving times seem to be 1 week later than it should be.\n\nCode: https://www.kaggle.com/junkoda/checking-daylight-saving-times", "votes": 4 }, { "id": 2445136, "postDate": "2023-09-18T16:24:15.700Z", "content": "

Hi @ryanholbrook, request you to pin this discussion so that most teams will be aware of this

", "rawMarkdown": "Hi @ryanholbrook, request you to pin this discussion so that most teams will be aware of this", "votes": 4 }, { "id": 2441161, "postDate": "2023-09-16T04:18:49.120Z", "content": "

Is this related to duplicate onset or wakeup events in one night? For example,
\n10349 c6788e579967 20 onset 333552.0 2018-11-11T01:16:00-0400
\n10350 c6788e579967 20 onset 334272.0 2018-11-11T01:16:00-0500

", "rawMarkdown": "Is this related to duplicate onset or wakeup events in one night? For example, \n10349 c6788e579967 20 onset 333552.0 2018-11-11T01:16:00-0400\n10350 c6788e579967 20 onset 334272.0 2018-11-11T01:16:00-0500", "votes": 4, "replies": [ { "id": 2441677, "postDate": "2023-09-16T11:10:40.323Z", "content": "

Yes, this is related, and these examples helped us in recognizing the issue and identifying the cause. Thanks to you and all others for pointing this out!!

", "rawMarkdown": "Yes, this is related, and these examples helped us in recognizing the issue and identifying the cause. Thanks to you and all others for pointing this out!!", "votes": 6, "replies": [ { "id": 2441723, "postDate": "2023-09-16T11:54:26.153Z", "content": "

Also there are some ids with events starting on the second night.

", "rawMarkdown": "Also there are some ids with events starting on the second night.", "votes": 3 } ] } ] }, { "id": 2461952, "postDate": "2023-09-30T01:35:26.717Z", "content": "

@ryanholbrook , i don't know what i am missing here , still i am seeing offset changed from \"-400\" to \"-500\", though i removed and uploaded the data again

\n

\"\"

", "rawMarkdown": "@ryanholbrook , i don't know what i am missing here , still i am seeing offset changed from \"-400\" to \"-500\", though i removed and uploaded the data again\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213385%2F8c98b4282c17fcfa33720858e80348ec%2FTimezone.JPG?generation=1696037718959906&alt=media)", "votes": 1, "replies": [ { "id": 2462728, "postDate": "2023-09-30T14:53:11.507Z", "content": "

The update didn't remove times in DST, so it's OK to see -0400 and -0500. What the update did was to make sure there were not duplicate entries for an onset or wakeup that were caused by the DST change.
\nAs a check, the difference between wakeup 11 and wakeup 10 is 18360 steps which is 25.5 hours. Wakeup 10 has time 7:33 and wakeup 11 has time 8:03; the difference between these times is 24.5 hours (ignoring the offset). However, DST ended for wakeup 11 (November, so it is \"Fall back\"): the 8:03 would have been 9:03 without the change which agrees with the actual 25.5 time difference that the steps give.

", "rawMarkdown": "The update didn't remove times in DST, so it's OK to see -0400 and -0500. What the update did was to make sure there were not duplicate entries for an onset or wakeup that were caused by the DST change.\nAs a check, the difference between wakeup 11 and wakeup 10 is 18360 steps which is 25.5 hours. Wakeup 10 has time 7:33 and wakeup 11 has time 8:03; the difference between these times is 24.5 hours (ignoring the offset). However, DST ended for wakeup 11 (November, so it is \"Fall back\"): the 8:03 would have been 9:03 without the change which agrees with the actual 25.5 time difference that the steps give.", "votes": 2, "replies": [ { "id": 2463107, "postDate": "2023-09-30T23:33:17.797Z", "content": "

@dan3dewey , any reason for Daylight Saving Time (DST) change occurs in the morning at 08:03 instead of at 01:37 for night 11? It must be from midnight

", "rawMarkdown": "@dan3dewey , any reason for Daylight Saving Time (DST) change occurs in the morning at 08:03 instead of at 01:37 for night 11? It must be from midnight " }, { "id": 2463117, "postDate": "2023-10-01T00:07:50.643Z", "content": "

DST occurs at 2am in the US Eastern time zone

", "rawMarkdown": "DST occurs at 2am in the US Eastern time zone", "votes": 2 } ] } ] }, { "id": 2456306, "postDate": "2023-09-26T05:52:04.053Z", "content": "

@ryanholbrook Some “onset”-“wakeup” pairs look to be still incomplete. Would you clarify whether this is expected behavior and if the test data also has this pattern?

\n

ex.

\n
train_events = pd.read_csv()\n (sid, night), df  train_events.groupby([, ]):\n     df[].isnull().() == :\n        (df)\n
\n
          night   event      step                 timestamp\n  ce74d6d2106        onset  .  --T22::-\n  ce74d6d2106       wakeup       NaN                       NaN\n           night   event      step                 timestamp\n  fe824ed87        onset       NaN                       NaN\n  fe824ed87       wakeup  .  --T09::-\n           night   event      step                 timestamp\n  a41bba1ee7        onset       NaN                       NaN\n  a41bba1ee7       wakeup  .  --T08::-\n            night   event      step                 timestamp\n  efbfc4526d58         onset       NaN                       NaN\n  efbfc4526d58        wakeup  .  --T07::-\n            night   event      step                 timestamp\n  f8a8da8bdd00        onset       NaN                       NaN\n  f8a8da8bdd00       wakeup  .  --T12::-\n
", "rawMarkdown": "@ryanholbrook Some “onset”-“wakeup” pairs look to be still incomplete. Would you clarify whether this is expected behavior and if the test data also has this pattern?\n\nex.\n```py\ntrain_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\nfor (sid, night), df in train_events.groupby([\"series_id\", \"night\"]):\n if df[\"step\"].isnull().sum() == 1:\n print(df)\n```\n```\n series_id night event step timestamp\n558 0ce74d6d2106 20 onset 332376.0 2017-09-09T22:08:00-0400\n559 0ce74d6d2106 20 wakeup NaN NaN\n series_id night event step timestamp\n1424 154fe824ed87 30 onset NaN NaN\n1425 154fe824ed87 30 wakeup 514980.0 2019-02-25T09:45:00-0500\n series_id night event step timestamp\n3878 44a41bba1ee7 10 onset NaN NaN\n3879 44a41bba1ee7 10 wakeup 165684.0 2018-03-04T08:07:00-0500\n series_id night event step timestamp\n13552 efbfc4526d58 7 onset NaN NaN\n13553 efbfc4526d58 7 wakeup 114864.0 2018-12-21T07:47:00-0500\n series_id night event step timestamp\n14104 f8a8da8bdd00 17 onset NaN NaN\n14105 f8a8da8bdd00 17 wakeup 291384.0 2017-12-10T12:57:00-0500\n```", "votes": 1, "replies": [ { "id": 2457187, "postDate": "2023-09-26T17:29:10.930Z", "content": "

I'm seeing that there are 4,923 rows in the train_events.csv file with Null step/timestamp values.

", "rawMarkdown": "I'm seeing that there are 4,923 rows in the train_events.csv file with Null step/timestamp values.", "replies": [ { "id": 2457459, "postDate": "2023-09-26T23:45:37.910Z", "content": "

There is no problem as long as both \"onset\" and \"wakeup\" are null, but in the five cases above, only one of them is null in a single night.

\n

I think these pairs are inconsistent with the following guidelines in the Dataset description.

\n
\n

If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.

\n
", "rawMarkdown": "There is no problem as long as both \"onset\" and \"wakeup\" are null, but in the five cases above, only one of them is null in a single night.\n\nI think these pairs are inconsistent with the following guidelines in the Dataset description.\n\n> If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.", "votes": 2 }, { "id": 2457480, "postDate": "2023-09-27T00:44:18.240Z", "content": "

Got it, thanks for the clarification!

", "rawMarkdown": "Got it, thanks for the clarification!" } ] } ] }, { "id": 2513495, "postDate": "2023-11-05T13:40:34.823Z", "content": "

Great, this baseline really helped me gain a better understanding of this competition.♥️

", "rawMarkdown": "**Great, this baseline really helped me gain a better understanding of this competition.**♥️" }, { "id": 2509608, "postDate": "2023-11-02T13:27:58.863Z", "content": "

@ryanholbrook I participated this competition recently. Since only new data are available now, I can't check what is changed on the update nor validate the change is appropriate . Can you share the previous version of training data and make us for check the differences?

", "rawMarkdown": "@ryanholbrook I participated this competition recently. Since only new data are available now, I can't check what is changed on the update nor validate the change is appropriate . Can you share the previous version of training data and make us for check the differences?" }, { "id": 2469070, "postDate": "2023-10-06T06:12:38.787Z", "content": "

Hello! Series_id mentioned in the comments have duplicated timestamps (total = 1 hour for every series_id) in train_series data. Here is the list of series_id with duplicates:
\n['08db4255286f', '10469f6765bf', '16fe2798ed0f', '188d4b7cd28b', '1d4569cbac0f', '2654a87be968',
\n '2b8d87addea9', '3c336d6ba566', '416354edd92a', '44d8c02b369e', '4a31811f3558', '51fdcc8d9fe7',
\n '5e816f11f5c3', '655f19eabf1e', '67f5fc60e494', '7df249527c63', '8a306e0890c0', 'a596ad0b82aa',
\n 'a88088855de5', 'a9a2f7fac455', 'c6788e579967', 'd5e47b94477e', 'df33ae359fb5', 'e30cb792a2bc',
\n 'e4500e7e19e1', 'eec197a4bdca', 'f56824b503a0']

", "rawMarkdown": "Hello! Series_id mentioned in the comments have duplicated timestamps (total = 1 hour for every series_id) in train_series data. Here is the list of series_id with duplicates:\n['08db4255286f', '10469f6765bf', '16fe2798ed0f', '188d4b7cd28b', '1d4569cbac0f', '2654a87be968',\n '2b8d87addea9', '3c336d6ba566', '416354edd92a', '44d8c02b369e', '4a31811f3558', '51fdcc8d9fe7',\n '5e816f11f5c3', '655f19eabf1e', '67f5fc60e494', '7df249527c63', '8a306e0890c0', 'a596ad0b82aa',\n 'a88088855de5', 'a9a2f7fac455', 'c6788e579967', 'd5e47b94477e', 'df33ae359fb5', 'e30cb792a2bc',\n 'e4500e7e19e1', 'eec197a4bdca', 'f56824b503a0']", "replies": [ { "id": 2473147, "postDate": "2023-10-08T02:01:59.520Z", "content": "

Local time can be duplicated due to daylight saving time, but I don't think real time including timezone (-0400, -0500) are not duplicated in train_series.parquet. Time difference is always 5 seconds without anomaly if I checked them correctly.

\n
df = pd.read_parquet()\ndf1 = df[df[] == ]\n\n((df1[]))  \n((df1[].unique())) \n\ntimestamp_utc = pd.to_datetime(df1[], utc=)\n\n\n(np.diff(timestamp_utc) == pd.Timedelta(, )).()  \n
", "rawMarkdown": "Local time can be duplicated due to daylight saving time, but I don't think real time including timezone (-0400, -0500) are not duplicated in `train_series.parquet`. Time difference is always 5 seconds without anomaly if I checked them correctly.\n\n```python\ndf = pd.read_parquet('train_series.parquet')\ndf1 = df[df['series_id'] == '08db4255286f']\n\nprint(len(df1['timestamp'])) # => 440280\nprint(len(df1['timestamp'].unique())) # => 440280\n\ntimestamp_utc = pd.to_datetime(df1['timestamp'], utc=True)\n\n# Time difference is always 5 sec\n(np.diff(timestamp_utc) == pd.Timedelta(5, 's')).all() # => True\n``` ", "votes": 1, "replies": [ { "id": 2473275, "postDate": "2023-10-08T06:19:24.907Z", "content": "

Hi, #junKoda!

\n

When I try your code I have the following result:

\n

df = pd.read_parquet('train_series.parquet')
\ndf1 = df[df['series_id'] == '08db4255286f']

\n

print(len(df1['timestamp'])) # => 440280
\nprint(len(df1['timestamp'].unique())) # => 439560

\n

I tried to check the consistansy of timestamp in this way:

\n

1) add a new column for differences between neighboring timestamps
\ntrain_series['time_diff'] = train_series['timestamp'].diff()

\n

2) count values in the column with timestamp difference
\ntrain_series['time_diff'].value_counts()
\n=> time_diff
\n0 days 00:00:05 127946019
\n-1 days +23:00:05 27
\n0 days 01:00:05 17
\n187 days 23:30:05 1
\n-72 days +15:30:05 1
\n …
\n-98 days +10:30:05 1
\n451 days 22:00:05 1
\n-468 days +16:00:05 1
\n430 days 10:30:05 1
\n-635 days +06:30:05 1
\nName: count, Length: 279, dtype: int64

\n

If we'll make more detailed analysis for series_id with '-1 days +23:00:05 ' and '0 days 01:00:05' we'll see that series_id with '-1 days +23:00:05' have duplicated timestamps.

\n

I've updated the list of series_id with duplicates above.

", "rawMarkdown": "Hi, #junKoda!\n\nWhen I try your code I have the following result:\n\ndf = pd.read_parquet('train_series.parquet')\ndf1 = df[df['series_id'] == '08db4255286f']\n\nprint(len(df1['timestamp'])) # => 440280\nprint(len(df1['timestamp'].unique())) # => **439560**\n\nI tried to check the consistansy of timestamp in this way:\n\n1) add a new column for differences between neighboring timestamps \ntrain_series['time_diff'] = train_series['timestamp'].diff()\n\n2) count values in the column with timestamp difference\ntrain_series['time_diff'].value_counts()\n=> time_diff\n0 days 00:00:05 127946019\n-1 days +23:00:05 27\n0 days 01:00:05 17\n187 days 23:30:05 1\n-72 days +15:30:05 1\n ... \n-98 days +10:30:05 1\n451 days 22:00:05 1\n-468 days +16:00:05 1\n430 days 10:30:05 1\n-635 days +06:30:05 1\nName: count, Length: 279, dtype: int64\n\nIf we'll make more detailed analysis for series_id with '-1 days +23:00:05 ' and '0 days 01:00:05' we'll see that series_id with '-1 days +23:00:05' have duplicated timestamps. \n\nI've updated the list of series_id with duplicates above.", "votes": 1 }, { "id": 2475761, "postDate": "2023-10-10T06:28:25.783Z", "content": "

Hi Veronika,

\n

Very strange and I have no idea what is happening.
\nprint(len(df1['timestamp'].unique())) is 440280 for me and timestamp_utc.diff().value_counts() returns 0 days 00:00:05 440279\nName: count, dtype: int64

\n

My data file is Sep 20 version (+0900 is my timezone)

\n
$  -l --time-style=full-iso\n... 985815313 2023-09-20 17:57:36.000000000 +0900 train_series.parquet\n
\n

I don't think train_series data was updated, but I can only suggest to redownload the data…

", "rawMarkdown": "Hi Veronika,\n\nVery strange and I have no idea what is happening.\n`print(len(df1['timestamp'].unique()))` is `440280` for me and `timestamp_utc.diff().value_counts()` returns `0 days 00:00:05 440279\nName: count, dtype: int64`\n\nMy data file is Sep 20 version (+0900 is my timezone)\n\n```bash\n$ ls -l --time-style=full-iso\n... 985815313 2023-09-20 17:57:36.000000000 +0900 train_series.parquet\n```\n\nI don't think train_series data was updated, but I can only suggest to redownload the data...\n", "votes": 1 }, { "id": 2477410, "postDate": "2023-10-11T08:52:19.363Z", "content": "

Hi, Jun Koda!

\n

You are right! My downloading the data was not correct. After changing the code, I get the same result as you.

\n

Thanks!

", "rawMarkdown": "Hi, Jun Koda!\n\nYou are right! My downloading the data was not correct. After changing the code, I get the same result as you.\n\nThanks!", "votes": 1 } ] } ] }, { "id": 2456379, "postDate": "2023-09-26T06:38:34.863Z", "content": "

nice, this baseline help me understand this competition better

", "rawMarkdown": "nice, this baseline help me understand this competition better" }, { "id": 2445899, "postDate": "2023-09-19T06:27:45.053Z", "content": "

Shall we expect a better score after dataset is updated and model retrained?

", "rawMarkdown": "Shall we expect a better score after dataset is updated and model retrained?" }, { "id": 2667276, "postDate": "2024-02-25T03:34:51.350Z", "content": "

Thank you! I found this really helpful.

", "rawMarkdown": "Thank you! I found this really helpful." } ], "comments": [ { "id": 2448756, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-20T19:27:41.767000", "content": "

Okay, update round two is done (data refresh and rescore). I believe this resolved the issue with the offset events and the duplicated events. The series themselves should not have changed, only the data in train_events.csv. I ran some extra tests to try to ensure all annotated events are consistent with the DST change, but please let us know if you notice anything out of sorts.

", "votes": 21, "replies": [] }, { "id": 2445572, "author_name": "Daniel Dewey", "author_url": "", "post_date": "2023-09-19T00:39:21.397000", "content": "

I think a reason for the decrease in leader board scores is that the train_events.cvs now has event step values that do not agree with the data series transition values for some series when there is a DST change.

\n

The series 655f19eabf1e had a double night 3 onset and now onsets/wakeups after that actually occur about an hour later than the training event file values.

\n

In the image below, 8a306e0890c0 (which had a double onset at night 11) now shows nights 12 and following ones as having their real transition (red) about an hour later than the events file value (orange.)

\n

\"\"

", "votes": 9, "replies": [ { "id": 2445595, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-19T01:07:22.550000", "content": "

Hi @dan3dewey good findings. What is the gray, blue, red, and orange line refer to?

", "votes": 0, "replies": [] }, { "id": 2446086, "author_name": "Hugo de Heer", "author_url": "", "post_date": "2023-09-19T08:35:33.897000", "content": "

Interesting, so these 3 duplicate onset events are now gone. But is the labeling now not correct after these DST events? Is that what you are saying?

", "votes": 2, "replies": [] }, { "id": 2446217, "author_name": "Quillen", "author_url": "", "post_date": "2023-09-19T10:18:51.710000", "content": "

I'm seeing the same thing.

\n

All labels after the (now corrected) \"double onset nights\" appear to be incorrectly shifted forward by an hour for series 655f19eabf1e, 8a306e0890c0, and c6788e579967.

\n

Edit:
\n Upon spot checking other DST affected series, it seems like this issue is fairly widespread. Specifically, I see potential problems with the labels for:

\n\n

Anyone else want to take a look at some of these series? Because it's absolutely possible that it's just an error on my end.

", "votes": 4, "replies": [ { "id": 2446620, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-09-19T14:06:11.867000", "content": "

I am seeing the same thing. From what I can tell the change impacted labels for ~800 events in 38 series.

\n

When I plot the old vs new labels I can see the offset looks to have made the labels less in-line with the features for most of these events.

\n

My cross validation score is also worse when using the new labels and I would think it would improve if mislabels were fixed.

\n

I didn't re-download the parquet files, just the events file- assuming that was the only file that changed- so maybe that's the cause? Could also be an error in my code.

\n

Hopefully @ryanholbrook can give us an update.

", "votes": 7, "replies": [] }, { "id": 2446632, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-19T14:13:38.753000", "content": "

Thanks, I'll take a look.

", "votes": 2, "replies": [] }, { "id": 2446705, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-09-19T14:55:20.930000", "content": "

I made this notebook showing the old/new events that might be helpful: https://www.kaggle.com/code/robikscube/sleep-states-events-file-update-viz/

", "votes": 3, "replies": [] }, { "id": 2446990, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-19T18:47:32.567000", "content": "

Okay, I think I've identified a fix. I'll try to have an update sometime tomorrow.

\n

And thanks again for your patience. This has been a tough bug to squash.

", "votes": 10, "replies": [] }, { "id": 2447111, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2023-09-19T20:37:56.700000", "content": "

i think the nature of the problem has been misunderstood here: as long as 'step' is defined consistently in 'train_series' and 'train_events', it does not matter much if time label is off (it would impact some time-based features, but this impact would be second order effect). And 'step' was defined consistently. So correction would have to impact both files, to preserve their consistency, and that is not what was done here.

", "votes": 3, "replies": [] }, { "id": 2447395, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-20T03:54:49.823000", "content": "

I concur with @ymatioun regarding needing to also correct the series data as well. All of the predicted events for the effected series became off by an hour (720 steps) and thus are outside of all the thresholds and led to a score of 0 for said series i.e. the event data shifted, but the actual accelerometer data did not.

\n
\n

\"it would impact some time-based features, but this impact would be second order effect\"

\n
\n

Feature importance indicates that the hour feature is very influential, and I suggest it is for this reason there was an almost universal drop of around 10% in all of the LB scores after the re-score.

\n

All the best,
\ncarl

", "votes": 2, "replies": [] } ] }, { "id": 2468582, "author_name": "OlivierGuntern", "author_url": "", "post_date": "2023-10-05T16:14:04.620000", "content": "

THANKS !!!

", "votes": 0, "replies": [] } ] }, { "id": 2445214, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-18T16:53:27.437000", "content": "

The update and rescore is complete. Please let us know if you have any questions or concerns.

", "votes": 7, "replies": [ { "id": 2445376, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-09-18T19:23:03.987000", "content": "

Thanks Ryan. Can you confirm that train_events.csv was the only file in the public dataset that was impacted?

", "votes": 2, "replies": [ { "id": 2445409, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-18T19:53:54.183000", "content": "

Hi @robikscube, that should be the case. If it seems to be otherwise, please let me know.

", "votes": 2, "replies": [] } ] } ] }, { "id": 2443400, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-17T17:35:41.157000", "content": "

What a relief. Looking forward to this.

\n

Will the \"onset but not wakeup\" or \"wakeup but not onset\" events like here also be fixed?

", "votes": 3, "replies": [ { "id": 2445299, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-09-18T18:07:42.403000", "content": "

The duplicated events should also be fixed, yes.

", "votes": 1, "replies": [ { "id": 2445555, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-19T00:01:14.080000", "content": "

Thank you @ryanholbrook , glad to hear.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2443034, "author_name": "Daniel Dewey", "author_url": "", "post_date": "2023-09-17T13:41:21.483000", "content": "

Will the \"step\" values in the series always increment by 1 (5 seconds) from row to row? That is, will a DST change affect the timestamp but not cause any change to the sequential step values?

", "votes": 3, "replies": [ { "id": 2443080, "author_name": "", "author_url": "", "post_date": "2023-09-17T14:11:06.877000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2464134, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2023-10-02T02:12:02.093000", "content": "

@ryanholbrook Are the switches between -0400 and -0500 happening at correct dates? The timestamp string representations are not consistent with pandas 'US/Eastern' timezone. As far as I searched, the end of daylight saving time is 2018-11-04 in US, but on 2018-11-11 in the data. The start and end of daylight saving times seem to be 1 week later than it should be.

\n

Code: https://www.kaggle.com/junkoda/checking-daylight-saving-times

", "votes": 4, "replies": [] }, { "id": 2445136, "author_name": "Anil Reddy", "author_url": "", "post_date": "2023-09-18T16:24:15.700000", "content": "

Hi @ryanholbrook, request you to pin this discussion so that most teams will be aware of this

", "votes": 4, "replies": [] }, { "id": 2441161, "author_name": "Aphysict", "author_url": "", "post_date": "2023-09-16T04:18:49.120000", "content": "

Is this related to duplicate onset or wakeup events in one night? For example,
\n10349 c6788e579967 20 onset 333552.0 2018-11-11T01:16:00-0400
\n10350 c6788e579967 20 onset 334272.0 2018-11-11T01:16:00-0500

", "votes": 4, "replies": [ { "id": 2441677, "author_name": "Greg Kiar", "author_url": "", "post_date": "2023-09-16T11:10:40.323000", "content": "

Yes, this is related, and these examples helped us in recognizing the issue and identifying the cause. Thanks to you and all others for pointing this out!!

", "votes": 6, "replies": [ { "id": 2441723, "author_name": "Cihat Emre Çeliker", "author_url": "", "post_date": "2023-09-16T11:54:26.153000", "content": "

Also there are some ids with events starting on the second night.

", "votes": 3, "replies": [] } ] } ] }, { "id": 2461952, "author_name": "Khurram Siddiqui", "author_url": "", "post_date": "2023-09-30T01:35:26.717000", "content": "

@ryanholbrook , i don't know what i am missing here , still i am seeing offset changed from \"-400\" to \"-500\", though i removed and uploaded the data again

\n

\"\"

", "votes": 1, "replies": [ { "id": 2462728, "author_name": "Daniel Dewey", "author_url": "", "post_date": "2023-09-30T14:53:11.507000", "content": "

The update didn't remove times in DST, so it's OK to see -0400 and -0500. What the update did was to make sure there were not duplicate entries for an onset or wakeup that were caused by the DST change.
\nAs a check, the difference between wakeup 11 and wakeup 10 is 18360 steps which is 25.5 hours. Wakeup 10 has time 7:33 and wakeup 11 has time 8:03; the difference between these times is 24.5 hours (ignoring the offset). However, DST ended for wakeup 11 (November, so it is \"Fall back\"): the 8:03 would have been 9:03 without the change which agrees with the actual 25.5 time difference that the steps give.

", "votes": 2, "replies": [ { "id": 2463107, "author_name": "Khurram Siddiqui", "author_url": "", "post_date": "2023-09-30T23:33:17.797000", "content": "

@dan3dewey , any reason for Daylight Saving Time (DST) change occurs in the morning at 08:03 instead of at 01:37 for night 11? It must be from midnight

", "votes": 0, "replies": [] }, { "id": 2463117, "author_name": "Quillen", "author_url": "", "post_date": "2023-10-01T00:07:50.643000", "content": "

DST occurs at 2am in the US Eastern time zone

", "votes": 2, "replies": [] } ] } ] }, { "id": 2456306, "author_name": "i-aki.y", "author_url": "", "post_date": "2023-09-26T05:52:04.053000", "content": "

@ryanholbrook Some “onset”-“wakeup” pairs look to be still incomplete. Would you clarify whether this is expected behavior and if the test data also has this pattern?

\n

ex.

\n
train_events = pd.read_csv()\n (sid, night), df  train_events.groupby([, ]):\n     df[].isnull().() == :\n        (df)\n
\n
          night   event      step                 timestamp\n  ce74d6d2106        onset  .  --T22::-\n  ce74d6d2106       wakeup       NaN                       NaN\n           night   event      step                 timestamp\n  fe824ed87        onset       NaN                       NaN\n  fe824ed87       wakeup  .  --T09::-\n           night   event      step                 timestamp\n  a41bba1ee7        onset       NaN                       NaN\n  a41bba1ee7       wakeup  .  --T08::-\n            night   event      step                 timestamp\n  efbfc4526d58         onset       NaN                       NaN\n  efbfc4526d58        wakeup  .  --T07::-\n            night   event      step                 timestamp\n  f8a8da8bdd00        onset       NaN                       NaN\n  f8a8da8bdd00       wakeup  .  --T12::-\n
", "votes": 1, "replies": [ { "id": 2457187, "author_name": "RyanCaldwell", "author_url": "", "post_date": "2023-09-26T17:29:10.930000", "content": "

I'm seeing that there are 4,923 rows in the train_events.csv file with Null step/timestamp values.

", "votes": 0, "replies": [ { "id": 2457459, "author_name": "i-aki.y", "author_url": "", "post_date": "2023-09-26T23:45:37.910000", "content": "

There is no problem as long as both \"onset\" and \"wakeup\" are null, but in the five cases above, only one of them is null in a single night.

\n

I think these pairs are inconsistent with the following guidelines in the Dataset description.

\n
\n

If no valid sleep window is identifiable, neither an onset nor a wakeup event is recorded for that night.

\n
", "votes": 2, "replies": [] }, { "id": 2457480, "author_name": "RyanCaldwell", "author_url": "", "post_date": "2023-09-27T00:44:18.240000", "content": "

Got it, thanks for the clarification!

", "votes": 0, "replies": [] } ] } ] }, { "id": 2513495, "author_name": "zhendong niu", "author_url": "", "post_date": "2023-11-05T13:40:34.823000", "content": "

Great, this baseline really helped me gain a better understanding of this competition.♥️

", "votes": 0, "replies": [] }, { "id": 2509608, "author_name": "Bilzard", "author_url": "", "post_date": "2023-11-02T13:27:58.863000", "content": "

@ryanholbrook I participated this competition recently. Since only new data are available now, I can't check what is changed on the update nor validate the change is appropriate . Can you share the previous version of training data and make us for check the differences?

", "votes": 0, "replies": [] }, { "id": 2469070, "author_name": "Veronika Sarikaya", "author_url": "", "post_date": "2023-10-06T06:12:38.787000", "content": "

Hello! Series_id mentioned in the comments have duplicated timestamps (total = 1 hour for every series_id) in train_series data. Here is the list of series_id with duplicates:
\n['08db4255286f', '10469f6765bf', '16fe2798ed0f', '188d4b7cd28b', '1d4569cbac0f', '2654a87be968',
\n '2b8d87addea9', '3c336d6ba566', '416354edd92a', '44d8c02b369e', '4a31811f3558', '51fdcc8d9fe7',
\n '5e816f11f5c3', '655f19eabf1e', '67f5fc60e494', '7df249527c63', '8a306e0890c0', 'a596ad0b82aa',
\n 'a88088855de5', 'a9a2f7fac455', 'c6788e579967', 'd5e47b94477e', 'df33ae359fb5', 'e30cb792a2bc',
\n 'e4500e7e19e1', 'eec197a4bdca', 'f56824b503a0']

", "votes": 0, "replies": [ { "id": 2473147, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2023-10-08T02:01:59.520000", "content": "

Local time can be duplicated due to daylight saving time, but I don't think real time including timezone (-0400, -0500) are not duplicated in train_series.parquet. Time difference is always 5 seconds without anomaly if I checked them correctly.

\n
df = pd.read_parquet()\ndf1 = df[df[] == ]\n\n((df1[]))  \n((df1[].unique())) \n\ntimestamp_utc = pd.to_datetime(df1[], utc=)\n\n\n(np.diff(timestamp_utc) == pd.Timedelta(, )).()  \n
", "votes": 1, "replies": [ { "id": 2473275, "author_name": "Veronika Sarikaya", "author_url": "", "post_date": "2023-10-08T06:19:24.907000", "content": "

Hi, #junKoda!

\n

When I try your code I have the following result:

\n

df = pd.read_parquet('train_series.parquet')
\ndf1 = df[df['series_id'] == '08db4255286f']

\n

print(len(df1['timestamp'])) # => 440280
\nprint(len(df1['timestamp'].unique())) # => 439560

\n

I tried to check the consistansy of timestamp in this way:

\n

1) add a new column for differences between neighboring timestamps
\ntrain_series['time_diff'] = train_series['timestamp'].diff()

\n

2) count values in the column with timestamp difference
\ntrain_series['time_diff'].value_counts()
\n=> time_diff
\n0 days 00:00:05 127946019
\n-1 days +23:00:05 27
\n0 days 01:00:05 17
\n187 days 23:30:05 1
\n-72 days +15:30:05 1
\n …
\n-98 days +10:30:05 1
\n451 days 22:00:05 1
\n-468 days +16:00:05 1
\n430 days 10:30:05 1
\n-635 days +06:30:05 1
\nName: count, Length: 279, dtype: int64

\n

If we'll make more detailed analysis for series_id with '-1 days +23:00:05 ' and '0 days 01:00:05' we'll see that series_id with '-1 days +23:00:05' have duplicated timestamps.

\n

I've updated the list of series_id with duplicates above.

", "votes": 1, "replies": [] }, { "id": 2475761, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2023-10-10T06:28:25.783000", "content": "

Hi Veronika,

\n

Very strange and I have no idea what is happening.
\nprint(len(df1['timestamp'].unique())) is 440280 for me and timestamp_utc.diff().value_counts() returns 0 days 00:00:05 440279\nName: count, dtype: int64

\n

My data file is Sep 20 version (+0900 is my timezone)

\n
$  -l --time-style=full-iso\n... 985815313 2023-09-20 17:57:36.000000000 +0900 train_series.parquet\n
\n

I don't think train_series data was updated, but I can only suggest to redownload the data…

", "votes": 1, "replies": [] }, { "id": 2477410, "author_name": "Veronika Sarikaya", "author_url": "", "post_date": "2023-10-11T08:52:19.363000", "content": "

Hi, Jun Koda!

\n

You are right! My downloading the data was not correct. After changing the code, I get the same result as you.

\n

Thanks!

", "votes": 1, "replies": [] } ] } ] }, { "id": 2456379, "author_name": "Kaggle_fighter", "author_url": "", "post_date": "2023-09-26T06:38:34.863000", "content": "

nice, this baseline help me understand this competition better

", "votes": 0, "replies": [] }, { "id": 2445899, "author_name": "Ruby", "author_url": "", "post_date": "2023-09-19T06:27:45.053000", "content": "

Shall we expect a better score after dataset is updated and model retrained?

", "votes": 0, "replies": [] }, { "id": 2667276, "author_name": "Pratham Kadam", "author_url": "", "post_date": "2024-02-25T03:34:51.350000", "content": "

Thank you! I found this really helpful.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2440944": "Hi everyone,\n\nThere were some series affected by a bug in DST handling that caused events after a time change to be incorrectly offset by an hour. Apparently, this only affected series in which a DST occurred (that is, when the time zone offset changed from \"-400\" to \"-500\"), which is 36 series in the training set (perhaps 6% of all events).\n\nI'll be updating the dataset on Monday with the correct event times and, as this also affected series in the test set, initiating a rescore. Since the portion of affected events is relatively small, I'm anticipating the effect on leaderboard scores will also likely not be large.\n\nI'll update this post once everything is complete.", "2448756": "Okay, update round two is done (data refresh and rescore). I believe this resolved the issue with the offset events and the duplicated events. The series themselves should not have changed, only the data in `train_events.csv`. I ran some extra tests to try to ensure all annotated events are consistent with the DST change, but please let us know if you notice anything out of sorts.", "2445572": "I think a reason for the decrease in leader board scores is that the train_events.cvs now has event step values that do not agree with the data series transition values for some series when there is a DST change.\n\nThe series 655f19eabf1e had a double night 3 onset and now onsets/wakeups after that actually occur about an hour later than the training event file values.\n\nIn the image below, 8a306e0890c0 (which had a double onset at night 11) now shows nights 12 and following ones as having their real transition (red) about an hour later than the events file value (orange.)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Fd600d31b071a9ac93187e1925a8bada0%2Fmismatch_event_series_steps.png?generation=1695083476020478&alt=media)", "2445214": "The update and rescore is complete. Please let us know if you have any questions or concerns.", "2443400": "What a relief. Looking forward to this.\n\nWill the \"onset but not wakeup\" or \"wakeup but not onset\" events [like here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438112) also be fixed?", "2443034": "Will the \"step\" values in the series always increment by 1 (5 seconds) from row to row? That is, will a DST change affect the timestamp but not cause any change to the sequential step values?", "2464134": "@ryanholbrook Are the switches between -0400 and -0500 happening at correct dates? The timestamp string representations are not consistent with pandas 'US/Eastern' timezone. As far as I searched, the end of daylight saving time is 2018-11-04 in US, but on 2018-11-11 in the data. The start and end of daylight saving times seem to be 1 week later than it should be.\n\nCode: https://www.kaggle.com/junkoda/checking-daylight-saving-times", "2445136": "Hi @ryanholbrook, request you to pin this discussion so that most teams will be aware of this", "2441161": "Is this related to duplicate onset or wakeup events in one night? For example, \n10349 c6788e579967 20 onset 333552.0 2018-11-11T01:16:00-0400\n10350 c6788e579967 20 onset 334272.0 2018-11-11T01:16:00-0500", "2461952": "@ryanholbrook , i don't know what i am missing here , still i am seeing offset changed from \"-400\" to \"-500\", though i removed and uploaded the data again\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213385%2F8c98b4282c17fcfa33720858e80348ec%2FTimezone.JPG?generation=1696037718959906&alt=media)", "2456306": "@ryanholbrook Some “onset”-“wakeup” pairs look to be still incomplete. Would you clarify whether this is expected behavior and if the test data also has this pattern?\n\nex.\n```py\ntrain_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\nfor (sid, night), df in train_events.groupby([\"series_id\", \"night\"]):\n if df[\"step\"].isnull().sum() == 1:\n print(df)\n```\n```\n series_id night event step timestamp\n558 0ce74d6d2106 20 onset 332376.0 2017-09-09T22:08:00-0400\n559 0ce74d6d2106 20 wakeup NaN NaN\n series_id night event step timestamp\n1424 154fe824ed87 30 onset NaN NaN\n1425 154fe824ed87 30 wakeup 514980.0 2019-02-25T09:45:00-0500\n series_id night event step timestamp\n3878 44a41bba1ee7 10 onset NaN NaN\n3879 44a41bba1ee7 10 wakeup 165684.0 2018-03-04T08:07:00-0500\n series_id night event step timestamp\n13552 efbfc4526d58 7 onset NaN NaN\n13553 efbfc4526d58 7 wakeup 114864.0 2018-12-21T07:47:00-0500\n series_id night event step timestamp\n14104 f8a8da8bdd00 17 onset NaN NaN\n14105 f8a8da8bdd00 17 wakeup 291384.0 2017-12-10T12:57:00-0500\n```", "2513495": "**Great, this baseline really helped me gain a better understanding of this competition.**♥️", "2509608": "@ryanholbrook I participated this competition recently. Since only new data are available now, I can't check what is changed on the update nor validate the change is appropriate . Can you share the previous version of training data and make us for check the differences?", "2469070": "Hello! Series_id mentioned in the comments have duplicated timestamps (total = 1 hour for every series_id) in train_series data. Here is the list of series_id with duplicates:\n['08db4255286f', '10469f6765bf', '16fe2798ed0f', '188d4b7cd28b', '1d4569cbac0f', '2654a87be968',\n '2b8d87addea9', '3c336d6ba566', '416354edd92a', '44d8c02b369e', '4a31811f3558', '51fdcc8d9fe7',\n '5e816f11f5c3', '655f19eabf1e', '67f5fc60e494', '7df249527c63', '8a306e0890c0', 'a596ad0b82aa',\n 'a88088855de5', 'a9a2f7fac455', 'c6788e579967', 'd5e47b94477e', 'df33ae359fb5', 'e30cb792a2bc',\n 'e4500e7e19e1', 'eec197a4bdca', 'f56824b503a0']", "2456379": "nice, this baseline help me understand this competition better", "2445899": "Shall we expect a better score after dataset is updated and model retrained?", "2667276": "Thank you! I found this really helpful." } }