{
  "id": 437710,
  "title": "Daylight saving time",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437710",
  "author_name": "Quillen",
  "post_date": "2023-09-07T20:36:16.956000",
  "votes": 8,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Don't forget to consider <a href=\"https://en.wikipedia.org/wiki/Daylight_saving_time\" target=\"_blank\">daylight saving time</a> (DST) if you're planning on using timestamps in your analysis/models.  I found <a href=\"https://www.kaggle.com/code/quillen/sleepstates-daylight-saving-time\" target=\"_blank\">44 series</a> in the training data with timestamps affected by DST.</p>",
  "messages": [
    {
      "id": 2428399,
      "postDate": "2023-09-07T20:36:16.957Z",
      "content": "<p>Don't forget to consider <a href=\"https://en.wikipedia.org/wiki/Daylight_saving_time\" target=\"_blank\">daylight saving time</a> (DST) if you're planning on using timestamps in your analysis/models.  I found <a href=\"https://www.kaggle.com/code/quillen/sleepstates-daylight-saving-time\" target=\"_blank\">44 series</a> in the training data with timestamps affected by DST.</p>",
      "rawMarkdown": "Don't forget to consider [daylight saving time](https://en.wikipedia.org/wiki/Daylight_saving_time) (DST) if you're planning on using timestamps in your analysis/models.  I found [44 series](https://www.kaggle.com/code/quillen/sleepstates-daylight-saving-time) in the training data with timestamps affected by DST.",
      "votes": 7
    },
    {
      "id": 2428544,
      "postDate": "2023-09-08T01:55:03.333Z",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/quillen\" target=\"_blank\">@quillen</a> for this notebook; this was my first experience with Polaris and it's 🔥 Also, got to know about <code>DataFrame.assign</code> as well, super handy. Modified your code to do all the calculations in one place itself.</p>\n<pre><code>df_local = (df_max_local.to_pandas()\n    .set_index()\n    .assign(\n        duration=df_max_local[] - df_min_local[],\n        expected_steps= x: (x[].dt.seconds + (x[].dt.days***))/,\n        difference_steps= x: x[] - x[],\n        difference_hours= x: x[] / ( * ),\n        month= x: x[].dt.month,\n    )\n).sort_values([, , ])\n</code></pre>",
      "rawMarkdown": "Thank you @quillen for this notebook; this was my first experience with Polaris and it's 🔥 Also, got to know about `DataFrame.assign` as well, super handy. Modified your code to do all the calculations in one place itself.\n\n```python\ndf_local = (df_max_local.to_pandas()\n    .set_index(\"series_id\")\n    .assign(\n        duration=df_max_local[\"timestamp\"] - df_min_local[\"timestamp\"],\n        expected_steps=lambda x: (x[\"duration\"].dt.seconds + (x[\"duration\"].dt.days*24*60*60))/5,\n        difference_steps=lambda x: x['expected_steps'] - x['step'],\n        difference_hours=lambda x: x['difference_steps'] / (12 * 60),\n        month=lambda x: x[\"timestamp\"].dt.month,\n    )\n).sort_values([\"difference_hours\", \"month\", \"timestamp\"])\n```",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2428544,
      "author_name": "Bhargav Kowshik",
      "author_url": "",
      "post_date": "2023-09-08T01:55:03.333000",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/quillen\" target=\"_blank\">@quillen</a> for this notebook; this was my first experience with Polaris and it's 🔥 Also, got to know about <code>DataFrame.assign</code> as well, super handy. Modified your code to do all the calculations in one place itself.</p>\n<pre><code>df_local = (df_max_local.to_pandas()\n    .set_index()\n    .assign(\n        duration=df_max_local[] - df_min_local[],\n        expected_steps= x: (x[].dt.seconds + (x[].dt.days***))/,\n        difference_steps= x: x[] - x[],\n        difference_hours= x: x[] / ( * ),\n        month= x: x[].dt.month,\n    )\n).sort_values([, , ])\n</code></pre>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2428399": "Don't forget to consider [daylight saving time](https://en.wikipedia.org/wiki/Daylight_saving_time) (DST) if you're planning on using timestamps in your analysis/models.  I found [44 series](https://www.kaggle.com/code/quillen/sleepstates-daylight-saving-time) in the training data with timestamps affected by DST.",
    "2428544": "Thank you @quillen for this notebook; this was my first experience with Polaris and it's 🔥 Also, got to know about `DataFrame.assign` as well, super handy. Modified your code to do all the calculations in one place itself.\n\n```python\ndf_local = (df_max_local.to_pandas()\n    .set_index(\"series_id\")\n    .assign(\n        duration=df_max_local[\"timestamp\"] - df_min_local[\"timestamp\"],\n        expected_steps=lambda x: (x[\"duration\"].dt.seconds + (x[\"duration\"].dt.days*24*60*60))/5,\n        difference_steps=lambda x: x['expected_steps'] - x['step'],\n        difference_hours=lambda x: x['difference_steps'] / (12 * 60),\n        month=lambda x: x[\"timestamp\"].dt.month,\n    )\n).sort_values([\"difference_hours\", \"month\", \"timestamp\"])\n```"
  }
}