{
  "id": 458546,
  "title": "Pandas to_datetime() alternative approach",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/458546",
  "author_name": "",
  "post_date": "2023-11-30T13:48:44.778437300Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>pandas to_datetime() on a large dataset is very very slow. which is taking around an hour to run for this dataset. Here is an alternative solution.</p>\n<p>convert it to numpy array<br>\ndate_time_array = np.array(train_series_df[\"timestamp\"], dtype= np.datetime64)<br>\nthen again replace the timestamp column with this numpy array.<br>\nthis solution took only 10 min to run.</p>",
  "messages": [
    {
      "id": "2543967",
      "postDate": "11/30/2023 13:48:44",
      "content": "<p>pandas to_datetime() on a large dataset is very very slow. which is taking around an hour to run for this dataset. Here is an alternative solution.</p>\n<p>convert it to numpy array<br>\ndate_time_array = np.array(train_series_df[\"timestamp\"], dtype= np.datetime64)<br>\nthen again replace the timestamp column with this numpy array.<br>\nthis solution took only 10 min to run.</p>",
      "rawMarkdown": "pandas to_datetime() on a large dataset is very very slow. which is taking around an hour to run for this dataset. Here is an alternative solution.\n\nconvert it to numpy array\ndate_time_array = np.array(train_series_df[\"timestamp\"], dtype= np.datetime64)\nthen again replace the timestamp column with this numpy array.\nthis solution took only 10 min to run.",
      "votes": null
    },
    {
      "id": "2544032",
      "postDate": "11/30/2023 14:53:50",
      "content": "<p><a href=\"https://www.kaggle.com/kesavan21u\" target=\"_blank\">@kesavan21u</a> did you explore Polars for this?</p>",
      "rawMarkdown": "kesavan21u did you explore Polars for this?",
      "votes": null
    },
    {
      "id": "2544591",
      "postDate": "12/01/2023 00:24:58",
      "content": "<p>Using polars:</p>\n<p>import polars as pl</p>\n<p>%%time<br>\ntrain_series = pl.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')</p>\n<p>CPU times: user 26.5 s, sys: 18.4 s, total: 44.9 s<br>\nWall time: 19.3 s</p>\n<p>%%time<br>\ntrain_series = train_series.with_columns(train_series['timestamp'].str.to_date('%Y-%m-%dT%H:%M:%S%z',strict=False))</p>\n<p>CPU times: user 47 s, sys: 1.27 s, total: 48.3 s<br>\nWall time: 48.3 s</p>",
      "rawMarkdown": "Using polars:\n\nimport polars as pl\n\n%%time\ntrain_series = pl.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n\nCPU times: user 26.5 s, sys: 18.4 s, total: 44.9 s\nWall time: 19.3 s\n\n\n%%time\ntrain_series = train_series.with_columns(train_series['timestamp'].str.to_date('%Y-%m-%dT%H:%M:%S%z',strict=False))\n\nCPU times: user 47 s, sys: 1.27 s, total: 48.3 s\nWall time: 48.3 s",
      "votes": null
    },
    {
      "id": "2544858",
      "postDate": "12/01/2023 06:25:27",
      "content": "<p>Oops. i haven't tried. <a href=\"https://www.kaggle.com/ravi20076\" target=\"_blank\">@ravi20076</a> Thanks for suggesting. Will explore the Polars</p>",
      "rawMarkdown": "Oops. i haven't tried. @ravi20076 Thanks for suggesting. Will explore the Polars",
      "votes": null
    },
    {
      "id": "2544859",
      "postDate": "12/01/2023 06:26:13",
      "content": "<p>Thanks for suggestion. Polar is much faster.</p>",
      "rawMarkdown": "Thanks for suggestion. Polar is much faster.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2544032,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "11/30/2023 14:53:50",
      "content": "<p><a href=\"https://www.kaggle.com/kesavan21u\" target=\"_blank\">@kesavan21u</a> did you explore Polars for this?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2544858,
          "author_name": "kesavan21u",
          "author_url": "",
          "post_date": "12/01/2023 06:25:27",
          "content": "<p>Oops. i haven't tried. <a href=\"https://www.kaggle.com/ravi20076\" target=\"_blank\">@ravi20076</a> Thanks for suggesting. Will explore the Polars</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2544591,
      "author_name": "najiama",
      "author_url": "",
      "post_date": "12/01/2023 00:24:58",
      "content": "<p>Using polars:</p>\n<p>import polars as pl</p>\n<p>%%time<br>\ntrain_series = pl.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')</p>\n<p>CPU times: user 26.5 s, sys: 18.4 s, total: 44.9 s<br>\nWall time: 19.3 s</p>\n<p>%%time<br>\ntrain_series = train_series.with_columns(train_series['timestamp'].str.to_date('%Y-%m-%dT%H:%M:%S%z',strict=False))</p>\n<p>CPU times: user 47 s, sys: 1.27 s, total: 48.3 s<br>\nWall time: 48.3 s</p>",
      "votes": null,
      "replies": [
        {
          "id": 2544859,
          "author_name": "kesavan21u",
          "author_url": "",
          "post_date": "12/01/2023 06:26:13",
          "content": "<p>Thanks for suggestion. Polar is much faster.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2543967": "pandas to_datetime() on a large dataset is very very slow. which is taking around an hour to run for this dataset. Here is an alternative solution.\n\nconvert it to numpy array\ndate_time_array = np.array(train_series_df[\"timestamp\"], dtype= np.datetime64)\nthen again replace the timestamp column with this numpy array.\nthis solution took only 10 min to run.",
    "2544032": "kesavan21u did you explore Polars for this?",
    "2544591": "Using polars:\n\nimport polars as pl\n\n%%time\ntrain_series = pl.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n\nCPU times: user 26.5 s, sys: 18.4 s, total: 44.9 s\nWall time: 19.3 s\n\n\n%%time\ntrain_series = train_series.with_columns(train_series['timestamp'].str.to_date('%Y-%m-%dT%H:%M:%S%z',strict=False))\n\nCPU times: user 47 s, sys: 1.27 s, total: 48.3 s\nWall time: 48.3 s",
    "2544858": "Oops. i haven't tried. @ravi20076 Thanks for suggesting. Will explore the Polars",
    "2544859": "Thanks for suggestion. Polar is much faster."
  },
  "source": "meta"
}