{
  "id": 438245,
  "title": "Timestamp convert using pyArrow",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/438245",
  "author_name": "cRIsPy_english",
  "post_date": "2023-09-10T06:55:40.993000",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>If anyone is having problems converting the timestamp you can read first using pyArrow then convert to Pandas. <br>\nDo this ( takes about ~50sec ) :</p>\n<pre><code>%%\n pyarrow  pa\n pyarrow.parquet  pq\n\nschema = pa.([(,pa.string()),\n          (,pa.int32()),\n          (,pa.timestamp(,tz=)),\n          (,pa.float32()),\n          (,pa.float32())\n         ])\nseries_data = pq.read_table(,schema=)\n\nseries_data = series_data.to_pandas()\n</code></pre>",
  "messages": [
    {
      "id": 2431505,
      "postDate": "2023-09-10T06:55:40.993Z",
      "content": "<p>If anyone is having problems converting the timestamp you can read first using pyArrow then convert to Pandas. <br>\nDo this ( takes about ~50sec ) :</p>\n<pre><code>%%\n pyarrow  pa\n pyarrow.parquet  pq\n\nschema = pa.([(,pa.string()),\n          (,pa.int32()),\n          (,pa.timestamp(,tz=)),\n          (,pa.float32()),\n          (,pa.float32())\n         ])\nseries_data = pq.read_table(,schema=)\n\nseries_data = series_data.to_pandas()\n</code></pre>",
      "rawMarkdown": "If anyone is having problems converting the timestamp you can read first using pyArrow then convert to Pandas. \nDo this ( takes about ~50sec ) :\n\n```\n%%time\nimport pyarrow as pa\nimport pyarrow.parquet as pq\n\nschema = pa.schema([('series_id',pa.string()),\n          ('step',pa.int32()),\n          ('timestamp',pa.timestamp('ms',tz='UTC')),\n          ('anglez',pa.float32()),\n          ('enmo',pa.float32())\n         ])\nseries_data = pq.read_table('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet',schema=schema)\n\nseries_data = series_data.to_pandas()\n```",
      "votes": 1
    },
    {
      "id": 2433501,
      "postDate": "2023-09-11T16:06:43.163Z",
      "content": "<p>Did not have any problems using the <code>format</code> specified on the data page:<br>\nRef: <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\" target=\"_blank\">https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data</a></p>\n<pre><code>train_series_df[] = pd.to_datetime(\n    train_series_df[],\n    =\n)\n</code></pre>\n<p><em>NOTE: Because of the size of the dataset, converting the <code>timestamp</code> column in the entire dataset could be super slow.</em></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F424d84558ba09dc21107206551eca054%2FScreenshot%202023-09-11%20at%2021.34.50.png?generation=1694448332386830&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Did not have any problems using the `format` specified on the data page:\nRef: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\n\n```python\ntrain_series_df['timestamp'] = pd.to_datetime(\n    train_series_df['timestamp'],\n    format='%Y-%m-%dT%H:%M:%S%z'\n)\n```\n\n_NOTE: Because of the size of the dataset, converting the `timestamp` column in the entire dataset could be super slow._\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F424d84558ba09dc21107206551eca054%2FScreenshot%202023-09-11%20at%2021.34.50.png?generation=1694448332386830&alt=media)\n",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2433501,
      "author_name": "Bhargav Kowshik",
      "author_url": "",
      "post_date": "2023-09-11T16:06:43.163000",
      "content": "<p>Did not have any problems using the <code>format</code> specified on the data page:<br>\nRef: <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\" target=\"_blank\">https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data</a></p>\n<pre><code>train_series_df[] = pd.to_datetime(\n    train_series_df[],\n    =\n)\n</code></pre>\n<p><em>NOTE: Because of the size of the dataset, converting the <code>timestamp</code> column in the entire dataset could be super slow.</em></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F424d84558ba09dc21107206551eca054%2FScreenshot%202023-09-11%20at%2021.34.50.png?generation=1694448332386830&amp;alt=media\" alt=\"\"></p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2431505": "If anyone is having problems converting the timestamp you can read first using pyArrow then convert to Pandas. \nDo this ( takes about ~50sec ) :\n\n```\n%%time\nimport pyarrow as pa\nimport pyarrow.parquet as pq\n\nschema = pa.schema([('series_id',pa.string()),\n          ('step',pa.int32()),\n          ('timestamp',pa.timestamp('ms',tz='UTC')),\n          ('anglez',pa.float32()),\n          ('enmo',pa.float32())\n         ])\nseries_data = pq.read_table('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet',schema=schema)\n\nseries_data = series_data.to_pandas()\n```",
    "2433501": "Did not have any problems using the `format` specified on the data page:\nRef: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\n\n```python\ntrain_series_df['timestamp'] = pd.to_datetime(\n    train_series_df['timestamp'],\n    format='%Y-%m-%dT%H:%M:%S%z'\n)\n```\n\n_NOTE: Because of the size of the dataset, converting the `timestamp` column in the entire dataset could be super slow._\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F424d84558ba09dc21107206551eca054%2FScreenshot%202023-09-11%20at%2021.34.50.png?generation=1694448332386830&alt=media)\n"
  }
}