{
  "id": 437998,
  "title": "Timezone inconsistency and duplicated entries in train_events?",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437998",
  "author_name": "",
  "post_date": "2023-09-09T03:54:34.939201400Z",
  "votes": 19,
  "comment_count": 11,
  "views": 0,
  "content": "<p>It seems like there are some duplicated/redundant lines in the <code>train_events.csv</code> file with time zone shifts that seem wrong?</p>\n<p>The following code (disclaimer: pandas is not my strong suit, I'm sure there's a better way…):</p>\n<pre><code>train_events = pd()\ntrain_events = pd(train_events,\n                                           format=,\n                                           utc=False)\n\ntrain_events = train_events \\\n  (lambda x: x  pd(x)  x())\n\nonset_events = train_events == ) &amp;\n                                (~np(train_events.step))]\n\nonset_counts = onset_events] \\\n  () \\\n  ()\n\nonset_counts &gt; ] \\\n  (, ascending=False) \\\n  (onset_events(),\n        on = , how=)\n</code></pre>\n<p>…produces the following output:</p>\n<pre><code>        series_id        date  count  night  event      step                  timestamp\n1    8a306e0890c0  2017      3     11  onset  178464.0  2017 01:37:00:00\n1    8a306e0890c0  2017      3     11  onset  179184.0  2017 01:37:00:00\n1    8a306e0890c0  2017      3     12  onset  193584.0  2017 21:37:00:00\n0    c6788e579967  2018      3     20  onset  333552.0  2018 01:16:00:00\n0    c6788e579967  2018      3     20  onset  334272.0  2018 01:16:00:00\n..            ...         ...    ...    ...    ...       ...                        ...\n117  154fe824ed87  2019      2     23  onset  386544.0  2019 23:22:00:00\n116  72ba4a8afff4  2018      2      5  onset   74376.0  2018 00:18:00:00\n116  72ba4a8afff4  2018      2      6  onset   90960.0  2018 23:20:00:00\n127  154fe824ed87  2019      2      2  onset   24312.0  2019 00:16:00:00\n127  154fe824ed87  2019      2      3  onset   41220.0  2019 23:45:00:00\n\n[739 rows x 7 columns]\n</code></pre>\n<p>Note that many of these dates with more than one onset seem fine, but some are problematic.</p>\n<p>A spot check of the csv file itself confirms (note the two onset entries for night 11):</p>\n<pre><code>,,onset,,--T23::-\n,,wakeup,,--T07::-\n,,onset,,--T01::-\n,,onset,,--T01::-\n,,wakeup,,--T09::-\n,,onset,,--T21::-\n,,wakeup,,--T06::-\n</code></pre>\n<p>Based on plot for this particular example, it seems like the earlier onset time might be more appropriate, and in fact maybe the wakeup time should be shifted by an hour? But it'd be nice to confirm what the appropriate way to handle this is…  In the plot, orange and blue lines are enmo and anglez after some transformations, light grey region starts at the earlier onset time, 2017-11-12 01:37:00-04:00, and the darker grey region starts at the later onset time.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F356605%2F6edfd4746fc99847a19ca98406842c6d%2Fplot.png?generation=1694231333934161&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "2430065",
      "postDate": "09/09/2023 03:54:34",
      "content": "<p>It seems like there are some duplicated/redundant lines in the <code>train_events.csv</code> file with time zone shifts that seem wrong?</p>\n<p>The following code (disclaimer: pandas is not my strong suit, I'm sure there's a better way…):</p>\n<pre><code>train_events = pd()\ntrain_events = pd(train_events,\n                                           format=,\n                                           utc=False)\n\ntrain_events = train_events \\\n  (lambda x: x  pd(x)  x())\n\nonset_events = train_events == ) &amp;\n                                (~np(train_events.step))]\n\nonset_counts = onset_events] \\\n  () \\\n  ()\n\nonset_counts &gt; ] \\\n  (, ascending=False) \\\n  (onset_events(),\n        on = , how=)\n</code></pre>\n<p>…produces the following output:</p>\n<pre><code>        series_id        date  count  night  event      step                  timestamp\n1    8a306e0890c0  2017      3     11  onset  178464.0  2017 01:37:00:00\n1    8a306e0890c0  2017      3     11  onset  179184.0  2017 01:37:00:00\n1    8a306e0890c0  2017      3     12  onset  193584.0  2017 21:37:00:00\n0    c6788e579967  2018      3     20  onset  333552.0  2018 01:16:00:00\n0    c6788e579967  2018      3     20  onset  334272.0  2018 01:16:00:00\n..            ...         ...    ...    ...    ...       ...                        ...\n117  154fe824ed87  2019      2     23  onset  386544.0  2019 23:22:00:00\n116  72ba4a8afff4  2018      2      5  onset   74376.0  2018 00:18:00:00\n116  72ba4a8afff4  2018      2      6  onset   90960.0  2018 23:20:00:00\n127  154fe824ed87  2019      2      2  onset   24312.0  2019 00:16:00:00\n127  154fe824ed87  2019      2      3  onset   41220.0  2019 23:45:00:00\n\n[739 rows x 7 columns]\n</code></pre>\n<p>Note that many of these dates with more than one onset seem fine, but some are problematic.</p>\n<p>A spot check of the csv file itself confirms (note the two onset entries for night 11):</p>\n<pre><code>,,onset,,--T23::-\n,,wakeup,,--T07::-\n,,onset,,--T01::-\n,,onset,,--T01::-\n,,wakeup,,--T09::-\n,,onset,,--T21::-\n,,wakeup,,--T06::-\n</code></pre>\n<p>Based on plot for this particular example, it seems like the earlier onset time might be more appropriate, and in fact maybe the wakeup time should be shifted by an hour? But it'd be nice to confirm what the appropriate way to handle this is…  In the plot, orange and blue lines are enmo and anglez after some transformations, light grey region starts at the earlier onset time, 2017-11-12 01:37:00-04:00, and the darker grey region starts at the later onset time.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F356605%2F6edfd4746fc99847a19ca98406842c6d%2Fplot.png?generation=1694231333934161&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "It seems like there are some duplicated/redundant lines in the `train_events.csv` file with time zone shifts that seem wrong?\n\nThe following code (disclaimer: pandas is not my strong suit, I'm sure there's a better way...):\n\n```\ntrain_events = pd.read_csv('../child-mind-institute-detect-sleep-states/train_events.csv')\ntrain_events['timestamp'] = pd.to_datetime(train_events['timestamp'],\n                                           format='%Y-%m-%dT%H:%M:%S%z',\n                                           utc=False)\n\ntrain_events['date'] = train_events['timestamp'] \\\n  .apply(lambda x: x if pd.isna(x) else x.date())\n\nonset_events = train_events.loc[(train_events['event'] == 'onset') &\n                                (~np.isnan(train_events.step))]\n\nonset_counts = onset_events[['series_id', 'date']] \\\n  .value_counts() \\\n  .reset_index()\n\nonset_counts.loc[onset_counts['count'] > 1] \\\n  .sort_values('count', ascending=False) \\\n  .join(onset_events.set_index(['series_id', 'date']),\n        on = ['series_id', 'date'], how='left')\n```\n\n...produces the following output:\n```\n        series_id        date  count  night  event      step                  timestamp\n1    8a306e0890c0  2017-11-12      3     11  onset  178464.0  2017-11-12 01:37:00-04:00\n1    8a306e0890c0  2017-11-12      3     11  onset  179184.0  2017-11-12 01:37:00-05:00\n1    8a306e0890c0  2017-11-12      3     12  onset  193584.0  2017-11-12 21:37:00-05:00\n0    c6788e579967  2018-11-11      3     20  onset  333552.0  2018-11-11 01:16:00-04:00\n0    c6788e579967  2018-11-11      3     20  onset  334272.0  2018-11-11 01:16:00-05:00\n..            ...         ...    ...    ...    ...       ...                        ...\n117  154fe824ed87  2019-02-17      2     23  onset  386544.0  2019-02-17 23:22:00-05:00\n116  72ba4a8afff4  2018-09-05      2      5  onset   74376.0  2018-09-05 00:18:00-04:00\n116  72ba4a8afff4  2018-09-05      2      6  onset   90960.0  2018-09-05 23:20:00-04:00\n127  154fe824ed87  2019-01-28      2      2  onset   24312.0  2019-01-28 00:16:00-05:00\n127  154fe824ed87  2019-01-28      2      3  onset   41220.0  2019-01-28 23:45:00-05:00\n\n[739 rows x 7 columns]\n```\n\nNote that many of these dates with more than one onset seem fine, but some are problematic.\n\nA spot check of the csv file itself confirms (note the two onset entries for night 11):\n```\n8a306e0890c0,10,onset,159564,2017-11-10T23:22:00-0400\n8a306e0890c0,10,wakeup,165456,2017-11-11T07:33:00-0400\n8a306e0890c0,11,onset,178464,2017-11-12T01:37:00-0400\n8a306e0890c0,11,onset,179184,2017-11-12T01:37:00-0500\n8a306e0890c0,11,wakeup,184536,2017-11-12T09:03:00-0500\n8a306e0890c0,12,onset,193584,2017-11-12T21:37:00-0500\n8a306e0890c0,12,wakeup,199800,2017-11-13T06:15:00-0500\n```\n\nBased on plot for this particular example, it seems like the earlier onset time might be more appropriate, and in fact maybe the wakeup time should be shifted by an hour? But it'd be nice to confirm what the appropriate way to handle this is...  In the plot, orange and blue lines are enmo and anglez after some transformations, light grey region starts at the earlier onset time, 2017-11-12 01:37:00-04:00, and the darker grey region starts at the later onset time.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F356605%2F6edfd4746fc99847a19ca98406842c6d%2Fplot.png?generation=1694231333934161&alt=media)",
      "votes": null
    },
    {
      "id": "2430458",
      "postDate": "09/09/2023 09:19:45",
      "content": "<p>Thank you for sharing valuable information <a href=\"https://www.kaggle.com/evanray\" target=\"_blank\">@evanray</a>. It seems like he goes to bed twice in one night… Is it better to delete this the second time?</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa5ff636da29ce7f323113a4345c6e96d%2FID_8a306e0890c0_Night11.bmp?generation=1694251183310508&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Thank you for sharing valuable information @evanray. It seems like he goes to bed twice in one night... Is it better to delete this the second time?\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa5ff636da29ce7f323113a4345c6e96d%2FID_8a306e0890c0_Night11.bmp?generation=1694251183310508&alt=media)",
      "votes": null
    },
    {
      "id": "2430643",
      "postDate": "09/09/2023 12:52:57",
      "content": "<p>This seems to affect 3 series.  </p>\n<p>I used <a href=\"https://www.kaggle.com/mitsuyasuhoshino\" target=\"_blank\">@mitsuyasuhoshino</a>'s <a href=\"https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\" target=\"_blank\">excellent notebook</a> to visualize the 3 affected nights.  In each case, the second onset appears incorrect <strong>and</strong> the wakeup looks off by an hour.</p>\n<pre><code>train_events = (pd()\n                    (axis=)\n                    (local_time = lambda x: x(r),\n                            timestamp_utc = lambda x: pd(x, utc=True),\n                            date = lambda x: x.date))\n\nevent_counts = (train_events\n                    ()\n                    ()\n                    ()\n                    (, ascending=False)\n                    (train_events, on = )\n                    ])\n\nevent_counts\n</code></pre>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>series_id</th>\n<th>event</th>\n<th>night</th>\n<th>step</th>\n<th>date</th>\n<th>local_time</th>\n<th>timestamp_utc</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>655f19eabf1e</td>\n<td>onset</td>\n<td>3</td>\n<td>43104.0</td>\n<td>2017-11-12</td>\n<td>01:07:00</td>\n<td>2017-11-12 05:07:00+00:00</td>\n</tr>\n<tr>\n<td>1</td>\n<td>655f19eabf1e</td>\n<td>onset</td>\n<td>3</td>\n<td>43824.0</td>\n<td>2017-11-12</td>\n<td>01:07:00</td>\n<td>2017-11-12 06:07:00+00:00</td>\n</tr>\n<tr>\n<td>2</td>\n<td>8a306e0890c0</td>\n<td>onset</td>\n<td>11</td>\n<td>178464.0</td>\n<td>2017-11-12</td>\n<td>01:37:00</td>\n<td>2017-11-12 05:37:00+00:00</td>\n</tr>\n<tr>\n<td>3</td>\n<td>8a306e0890c0</td>\n<td>onset</td>\n<td>11</td>\n<td>179184.0</td>\n<td>2017-11-12</td>\n<td>01:37:00</td>\n<td>2017-11-12 06:37:00+00:00</td>\n</tr>\n<tr>\n<td>4</td>\n<td>c6788e579967</td>\n<td>onset</td>\n<td>20</td>\n<td>333552.0</td>\n<td>2018-11-11</td>\n<td>01:16:00</td>\n<td>2018-11-11 05:16:00+00:00</td>\n</tr>\n<tr>\n<td>5</td>\n<td>c6788e579967</td>\n<td>onset</td>\n<td>20</td>\n<td>334272.0</td>\n<td>2018-11-11</td>\n<td>01:16:00</td>\n<td>2018-11-11 06:16:00+00:00</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F4d03666add38e0208b2f1f9ad26adb31%2Fn3.png?generation=1694262937735236&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F69b3030b944cede707af892287df61dd%2Fn11.png?generation=1694262946831872&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F749d5d58a8d81f963147b609055d1899%2Fn20.png?generation=1694262966604202&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "This seems to affect 3 series.  \n\nI used @mitsuyasuhoshino's [excellent notebook](https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization) to visualize the 3 affected nights.  In each case, the second onset appears incorrect **and** the wakeup looks off by an hour.\n\n```\ntrain_events = (pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\n                    .dropna(axis=0)\n                    .assign(local_time = lambda x: x[\"timestamp\"].str.extract(r\"(..:..:..)\"),\n                            timestamp_utc = lambda x: pd.to_datetime(x[\"timestamp\"], utc=True),\n                            date = lambda x: x[\"timestamp_utc\"].dt.date))\n\nevent_counts = (train_events\n                    .value_counts([\"series_id\", \"night\", \"event\"])\n                    .reset_index()\n                    .query(\"count > 1\")\n                    .sort_values('count', ascending=False)\n                    .merge(train_events, on = ['series_id', 'night', 'event'])\n                    .loc[:,[\"series_id\",\"event\",\"night\",\"step\",\"date\",\"local_time\",\"timestamp_utc\"]])\n\nevent_counts\n```\n|   | series_id    | event | night | step     | date       | local_time | timestamp_utc             |\n|---|--------------|-------|-------|----------|------------|------------|---------------------------|\n| 0 | 655f19eabf1e | onset | 3     | 43104.0  | 2017-11-12 | 01:07:00   | 2017-11-12 05:07:00+00:00 |\n| 1 | 655f19eabf1e | onset | 3     | 43824.0  | 2017-11-12 | 01:07:00   | 2017-11-12 06:07:00+00:00 |\n| 2 | 8a306e0890c0 | onset | 11    | 178464.0 | 2017-11-12 | 01:37:00   | 2017-11-12 05:37:00+00:00 |\n| 3 | 8a306e0890c0 | onset | 11    | 179184.0 | 2017-11-12 | 01:37:00   | 2017-11-12 06:37:00+00:00 |\n| 4 | c6788e579967 | onset | 20    | 333552.0 | 2018-11-11 | 01:16:00   | 2018-11-11 05:16:00+00:00 |\n| 5 | c6788e579967 | onset | 20    | 334272.0 | 2018-11-11 | 01:16:00   | 2018-11-11 06:16:00+00:00 |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F4d03666add38e0208b2f1f9ad26adb31%2Fn3.png?generation=1694262937735236&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F69b3030b944cede707af892287df61dd%2Fn11.png?generation=1694262946831872&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F749d5d58a8d81f963147b609055d1899%2Fn20.png?generation=1694262966604202&alt=media)",
      "votes": null
    },
    {
      "id": "2430650",
      "postDate": "09/09/2023 12:58:41",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/quillen\" target=\"_blank\">@quillen</a>. It certainly looks like the wake-up time is an hour later. First, let's check my notebook. There's no daylight saving time in November, right !?</p>",
      "rawMarkdown": "Thanks @quillen. It certainly looks like the wake-up time is an hour later. First, let's check my notebook. There's no daylight saving time in November, right !?",
      "votes": null
    },
    {
      "id": "2430653",
      "postDate": "09/09/2023 13:02:58",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/ryanholbrook\" target=\"_blank\">@ryanholbrook</a></p>\n<p>Is the event data discussed here intentional? Thank you.</p>",
      "rawMarkdown": "Dear @ryanholbrook\n\nIs the event data discussed here intentional? Thank you.",
      "votes": null
    },
    {
      "id": "2430666",
      "postDate": "09/09/2023 13:22:51",
      "content": "<p>There is DST in November, but it happened a week before these events (2017-11-05 and 2018-11-04)</p>",
      "rawMarkdown": "There is DST in November, but it happened a week before these events (2017-11-05 and 2018-11-04)",
      "votes": null
    },
    {
      "id": "2430697",
      "postDate": "09/09/2023 13:55:05",
      "content": "<p>Some timestep is in UTC-4 and some more is in UTC-5, is this what do you mean by timezone incosistency? If so, should we force them to be similar?</p>",
      "rawMarkdown": "Some timestep is in UTC-4 and some more is in UTC-5, is this what do you mean by timezone incosistency? If so, should we force them to be similar?",
      "votes": null
    },
    {
      "id": "2431228",
      "postDate": "09/09/2023 22:52:11",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/quillen\" target=\"_blank\">@quillen</a>. I had no idea that DST was set for such a long period of time.<br>\nThis problem is probably a mistake in labeling the correct answer.</p>",
      "rawMarkdown": "Thanks @quillen. I had no idea that DST was set for such a long period of time.\nThis problem is probably a mistake in labeling the correct answer.",
      "votes": null
    },
    {
      "id": "2432342",
      "postDate": "09/10/2023 18:34:16",
      "content": "<p>DST has always ended the first weekend of November.</p>",
      "rawMarkdown": "DST has always ended the first weekend of November.",
      "votes": null
    },
    {
      "id": "2433349",
      "postDate": "09/11/2023 14:07:49",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/mitsuyasuhoshino\" target=\"_blank\">@mitsuyasuhoshino</a>, Thanks for the heads up. I'll investigate and follow up.</p>",
      "rawMarkdown": "Hi @mitsuyasuhoshino, Thanks for the heads up. I'll investigate and follow up.",
      "votes": null
    },
    {
      "id": "2434682",
      "postDate": "09/12/2023 13:00:05",
      "content": "<p>I believe the UTC-Offsets in the raw timestamps are wrong when looking at series_id that overlap periods of time that exhibit a shift from Eastern Daylight Time (EDT) to Eastern Standard Time (EST). If you reparse the the timestamps with pd.to_datetime(date_str, format=\"ISO8601\", UTC = True, yearfirst=True, errors=\"coerce\"). This will give you the Zulu Date /Time at UTC+0. With the new  Zulu Date /Time  feature,  you can create another datetime feature with tz_convert(\"us/eastern\") or .tz_convert(\"America/New_York\") . This second new feature is the local time for New York given the correct UTC-Offset. Here is a notebook on how it works <a href=\"https://www.kaggle.com/justinmustaine/utc-fix\" target=\"_blank\"><strong>UTC_FIX</strong></a> </p>",
      "rawMarkdown": "I believe the UTC-Offsets in the raw timestamps are wrong when looking at series_id that overlap periods of time that exhibit a shift from Eastern Daylight Time (EDT) to Eastern Standard Time (EST). If you reparse the the timestamps with pd.to_datetime(date_str, format=\"ISO8601\", UTC = True, yearfirst=True, errors=\"coerce\"). This will give you the Zulu Date /Time at UTC+0. With the new  Zulu Date /Time  feature,  you can create another datetime feature with tz_convert(\"us/eastern\") or .tz_convert(\"America/New_York\") . This second new feature is the local time for New York given the correct UTC-Offset. Here is a notebook on how it works [**UTC_FIX**](https://www.kaggle.com/justinmustaine/utc-fix)",
      "votes": null
    },
    {
      "id": "2436907",
      "postDate": "09/13/2023 20:48:50",
      "content": "<p>See this post and others in that discussion:<br>\n<a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768\" target=\"_blank\">https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768</a></p>",
      "rawMarkdown": "See this post and others in that discussion:\n[https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2430458,
      "author_name": "mitsuyasuhoshino",
      "author_url": "",
      "post_date": "09/09/2023 09:19:45",
      "content": "<p>Thank you for sharing valuable information <a href=\"https://www.kaggle.com/evanray\" target=\"_blank\">@evanray</a>. It seems like he goes to bed twice in one night… Is it better to delete this the second time?</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa5ff636da29ce7f323113a4345c6e96d%2FID_8a306e0890c0_Night11.bmp?generation=1694251183310508&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2430643,
      "author_name": "quillen",
      "author_url": "",
      "post_date": "09/09/2023 12:52:57",
      "content": "<p>This seems to affect 3 series.  </p>\n<p>I used <a href=\"https://www.kaggle.com/mitsuyasuhoshino\" target=\"_blank\">@mitsuyasuhoshino</a>'s <a href=\"https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\" target=\"_blank\">excellent notebook</a> to visualize the 3 affected nights.  In each case, the second onset appears incorrect <strong>and</strong> the wakeup looks off by an hour.</p>\n<pre><code>train_events = (pd()\n                    (axis=)\n                    (local_time = lambda x: x(r),\n                            timestamp_utc = lambda x: pd(x, utc=True),\n                            date = lambda x: x.date))\n\nevent_counts = (train_events\n                    ()\n                    ()\n                    ()\n                    (, ascending=False)\n                    (train_events, on = )\n                    ])\n\nevent_counts\n</code></pre>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>series_id</th>\n<th>event</th>\n<th>night</th>\n<th>step</th>\n<th>date</th>\n<th>local_time</th>\n<th>timestamp_utc</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>655f19eabf1e</td>\n<td>onset</td>\n<td>3</td>\n<td>43104.0</td>\n<td>2017-11-12</td>\n<td>01:07:00</td>\n<td>2017-11-12 05:07:00+00:00</td>\n</tr>\n<tr>\n<td>1</td>\n<td>655f19eabf1e</td>\n<td>onset</td>\n<td>3</td>\n<td>43824.0</td>\n<td>2017-11-12</td>\n<td>01:07:00</td>\n<td>2017-11-12 06:07:00+00:00</td>\n</tr>\n<tr>\n<td>2</td>\n<td>8a306e0890c0</td>\n<td>onset</td>\n<td>11</td>\n<td>178464.0</td>\n<td>2017-11-12</td>\n<td>01:37:00</td>\n<td>2017-11-12 05:37:00+00:00</td>\n</tr>\n<tr>\n<td>3</td>\n<td>8a306e0890c0</td>\n<td>onset</td>\n<td>11</td>\n<td>179184.0</td>\n<td>2017-11-12</td>\n<td>01:37:00</td>\n<td>2017-11-12 06:37:00+00:00</td>\n</tr>\n<tr>\n<td>4</td>\n<td>c6788e579967</td>\n<td>onset</td>\n<td>20</td>\n<td>333552.0</td>\n<td>2018-11-11</td>\n<td>01:16:00</td>\n<td>2018-11-11 05:16:00+00:00</td>\n</tr>\n<tr>\n<td>5</td>\n<td>c6788e579967</td>\n<td>onset</td>\n<td>20</td>\n<td>334272.0</td>\n<td>2018-11-11</td>\n<td>01:16:00</td>\n<td>2018-11-11 06:16:00+00:00</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F4d03666add38e0208b2f1f9ad26adb31%2Fn3.png?generation=1694262937735236&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F69b3030b944cede707af892287df61dd%2Fn11.png?generation=1694262946831872&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F749d5d58a8d81f963147b609055d1899%2Fn20.png?generation=1694262966604202&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 2430650,
          "author_name": "mitsuyasuhoshino",
          "author_url": "",
          "post_date": "09/09/2023 12:58:41",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/quillen\" target=\"_blank\">@quillen</a>. It certainly looks like the wake-up time is an hour later. First, let's check my notebook. There's no daylight saving time in November, right !?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2430666,
              "author_name": "quillen",
              "author_url": "",
              "post_date": "09/09/2023 13:22:51",
              "content": "<p>There is DST in November, but it happened a week before these events (2017-11-05 and 2018-11-04)</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2431228,
                  "author_name": "mitsuyasuhoshino",
                  "author_url": "",
                  "post_date": "09/09/2023 22:52:11",
                  "content": "<p>Thanks <a href=\"https://www.kaggle.com/quillen\" target=\"_blank\">@quillen</a>. I had no idea that DST was set for such a long period of time.<br>\nThis problem is probably a mistake in labeling the correct answer.</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            },
            {
              "id": 2432342,
              "author_name": "pcjimmmy",
              "author_url": "",
              "post_date": "09/10/2023 18:34:16",
              "content": "<p>DST has always ended the first weekend of November.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2430653,
      "author_name": "mitsuyasuhoshino",
      "author_url": "",
      "post_date": "09/09/2023 13:02:58",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/ryanholbrook\" target=\"_blank\">@ryanholbrook</a></p>\n<p>Is the event data discussed here intentional? Thank you.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2433349,
          "author_name": "ryanholbrook",
          "author_url": "",
          "post_date": "09/11/2023 14:07:49",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/mitsuyasuhoshino\" target=\"_blank\">@mitsuyasuhoshino</a>, Thanks for the heads up. I'll investigate and follow up.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2430697,
      "author_name": "rimbax",
      "author_url": "",
      "post_date": "09/09/2023 13:55:05",
      "content": "<p>Some timestep is in UTC-4 and some more is in UTC-5, is this what do you mean by timezone incosistency? If so, should we force them to be similar?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2434682,
      "author_name": "justinmustaine",
      "author_url": "",
      "post_date": "09/12/2023 13:00:05",
      "content": "<p>I believe the UTC-Offsets in the raw timestamps are wrong when looking at series_id that overlap periods of time that exhibit a shift from Eastern Daylight Time (EDT) to Eastern Standard Time (EST). If you reparse the the timestamps with pd.to_datetime(date_str, format=\"ISO8601\", UTC = True, yearfirst=True, errors=\"coerce\"). This will give you the Zulu Date /Time at UTC+0. With the new  Zulu Date /Time  feature,  you can create another datetime feature with tz_convert(\"us/eastern\") or .tz_convert(\"America/New_York\") . This second new feature is the local time for New York given the correct UTC-Offset. Here is a notebook on how it works <a href=\"https://www.kaggle.com/justinmustaine/utc-fix\" target=\"_blank\"><strong>UTC_FIX</strong></a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2436907,
      "author_name": "dan3dewey",
      "author_url": "",
      "post_date": "09/13/2023 20:48:50",
      "content": "<p>See this post and others in that discussion:<br>\n<a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768\" target=\"_blank\">https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2430065": "It seems like there are some duplicated/redundant lines in the `train_events.csv` file with time zone shifts that seem wrong?\n\nThe following code (disclaimer: pandas is not my strong suit, I'm sure there's a better way...):\n\n```\ntrain_events = pd.read_csv('../child-mind-institute-detect-sleep-states/train_events.csv')\ntrain_events['timestamp'] = pd.to_datetime(train_events['timestamp'],\n                                           format='%Y-%m-%dT%H:%M:%S%z',\n                                           utc=False)\n\ntrain_events['date'] = train_events['timestamp'] \\\n  .apply(lambda x: x if pd.isna(x) else x.date())\n\nonset_events = train_events.loc[(train_events['event'] == 'onset') &\n                                (~np.isnan(train_events.step))]\n\nonset_counts = onset_events[['series_id', 'date']] \\\n  .value_counts() \\\n  .reset_index()\n\nonset_counts.loc[onset_counts['count'] > 1] \\\n  .sort_values('count', ascending=False) \\\n  .join(onset_events.set_index(['series_id', 'date']),\n        on = ['series_id', 'date'], how='left')\n```\n\n...produces the following output:\n```\n        series_id        date  count  night  event      step                  timestamp\n1    8a306e0890c0  2017-11-12      3     11  onset  178464.0  2017-11-12 01:37:00-04:00\n1    8a306e0890c0  2017-11-12      3     11  onset  179184.0  2017-11-12 01:37:00-05:00\n1    8a306e0890c0  2017-11-12      3     12  onset  193584.0  2017-11-12 21:37:00-05:00\n0    c6788e579967  2018-11-11      3     20  onset  333552.0  2018-11-11 01:16:00-04:00\n0    c6788e579967  2018-11-11      3     20  onset  334272.0  2018-11-11 01:16:00-05:00\n..            ...         ...    ...    ...    ...       ...                        ...\n117  154fe824ed87  2019-02-17      2     23  onset  386544.0  2019-02-17 23:22:00-05:00\n116  72ba4a8afff4  2018-09-05      2      5  onset   74376.0  2018-09-05 00:18:00-04:00\n116  72ba4a8afff4  2018-09-05      2      6  onset   90960.0  2018-09-05 23:20:00-04:00\n127  154fe824ed87  2019-01-28      2      2  onset   24312.0  2019-01-28 00:16:00-05:00\n127  154fe824ed87  2019-01-28      2      3  onset   41220.0  2019-01-28 23:45:00-05:00\n\n[739 rows x 7 columns]\n```\n\nNote that many of these dates with more than one onset seem fine, but some are problematic.\n\nA spot check of the csv file itself confirms (note the two onset entries for night 11):\n```\n8a306e0890c0,10,onset,159564,2017-11-10T23:22:00-0400\n8a306e0890c0,10,wakeup,165456,2017-11-11T07:33:00-0400\n8a306e0890c0,11,onset,178464,2017-11-12T01:37:00-0400\n8a306e0890c0,11,onset,179184,2017-11-12T01:37:00-0500\n8a306e0890c0,11,wakeup,184536,2017-11-12T09:03:00-0500\n8a306e0890c0,12,onset,193584,2017-11-12T21:37:00-0500\n8a306e0890c0,12,wakeup,199800,2017-11-13T06:15:00-0500\n```\n\nBased on plot for this particular example, it seems like the earlier onset time might be more appropriate, and in fact maybe the wakeup time should be shifted by an hour? But it'd be nice to confirm what the appropriate way to handle this is...  In the plot, orange and blue lines are enmo and anglez after some transformations, light grey region starts at the earlier onset time, 2017-11-12 01:37:00-04:00, and the darker grey region starts at the later onset time.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F356605%2F6edfd4746fc99847a19ca98406842c6d%2Fplot.png?generation=1694231333934161&alt=media)",
    "2430458": "Thank you for sharing valuable information @evanray. It seems like he goes to bed twice in one night... Is it better to delete this the second time?\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa5ff636da29ce7f323113a4345c6e96d%2FID_8a306e0890c0_Night11.bmp?generation=1694251183310508&alt=media)",
    "2430643": "This seems to affect 3 series.  \n\nI used @mitsuyasuhoshino's [excellent notebook](https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization) to visualize the 3 affected nights.  In each case, the second onset appears incorrect **and** the wakeup looks off by an hour.\n\n```\ntrain_events = (pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\n                    .dropna(axis=0)\n                    .assign(local_time = lambda x: x[\"timestamp\"].str.extract(r\"(..:..:..)\"),\n                            timestamp_utc = lambda x: pd.to_datetime(x[\"timestamp\"], utc=True),\n                            date = lambda x: x[\"timestamp_utc\"].dt.date))\n\nevent_counts = (train_events\n                    .value_counts([\"series_id\", \"night\", \"event\"])\n                    .reset_index()\n                    .query(\"count > 1\")\n                    .sort_values('count', ascending=False)\n                    .merge(train_events, on = ['series_id', 'night', 'event'])\n                    .loc[:,[\"series_id\",\"event\",\"night\",\"step\",\"date\",\"local_time\",\"timestamp_utc\"]])\n\nevent_counts\n```\n|   | series_id    | event | night | step     | date       | local_time | timestamp_utc             |\n|---|--------------|-------|-------|----------|------------|------------|---------------------------|\n| 0 | 655f19eabf1e | onset | 3     | 43104.0  | 2017-11-12 | 01:07:00   | 2017-11-12 05:07:00+00:00 |\n| 1 | 655f19eabf1e | onset | 3     | 43824.0  | 2017-11-12 | 01:07:00   | 2017-11-12 06:07:00+00:00 |\n| 2 | 8a306e0890c0 | onset | 11    | 178464.0 | 2017-11-12 | 01:37:00   | 2017-11-12 05:37:00+00:00 |\n| 3 | 8a306e0890c0 | onset | 11    | 179184.0 | 2017-11-12 | 01:37:00   | 2017-11-12 06:37:00+00:00 |\n| 4 | c6788e579967 | onset | 20    | 333552.0 | 2018-11-11 | 01:16:00   | 2018-11-11 05:16:00+00:00 |\n| 5 | c6788e579967 | onset | 20    | 334272.0 | 2018-11-11 | 01:16:00   | 2018-11-11 06:16:00+00:00 |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F4d03666add38e0208b2f1f9ad26adb31%2Fn3.png?generation=1694262937735236&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F69b3030b944cede707af892287df61dd%2Fn11.png?generation=1694262946831872&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8650491%2F749d5d58a8d81f963147b609055d1899%2Fn20.png?generation=1694262966604202&alt=media)",
    "2430650": "Thanks @quillen. It certainly looks like the wake-up time is an hour later. First, let's check my notebook. There's no daylight saving time in November, right !?",
    "2430653": "Dear @ryanholbrook\n\nIs the event data discussed here intentional? Thank you.",
    "2430666": "There is DST in November, but it happened a week before these events (2017-11-05 and 2018-11-04)",
    "2430697": "Some timestep is in UTC-4 and some more is in UTC-5, is this what do you mean by timezone incosistency? If so, should we force them to be similar?",
    "2431228": "Thanks @quillen. I had no idea that DST was set for such a long period of time.\nThis problem is probably a mistake in labeling the correct answer.",
    "2432342": "DST has always ended the first weekend of November.",
    "2433349": "Hi @mitsuyasuhoshino, Thanks for the heads up. I'll investigate and follow up.",
    "2434682": "I believe the UTC-Offsets in the raw timestamps are wrong when looking at series_id that overlap periods of time that exhibit a shift from Eastern Daylight Time (EDT) to Eastern Standard Time (EST). If you reparse the the timestamps with pd.to_datetime(date_str, format=\"ISO8601\", UTC = True, yearfirst=True, errors=\"coerce\"). This will give you the Zulu Date /Time at UTC+0. With the new  Zulu Date /Time  feature,  you can create another datetime feature with tz_convert(\"us/eastern\") or .tz_convert(\"America/New_York\") . This second new feature is the local time for New York given the correct UTC-Offset. Here is a notebook on how it works [**UTC_FIX**](https://www.kaggle.com/justinmustaine/utc-fix)",
    "2436907": "See this post and others in that discussion:\n[https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437920#2436768)"
  },
  "source": "meta"
}