{ "id": 438165, "title": "[advice?] timestamp column does not convert", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/438165", "author_name": "Andrada", "post_date": "2023-09-09T19:27:47.770000", "votes": 10, "comment_count": 17, "views": 0, "content": "

Hi all!

\n

I have a weird problem and I do not understand what am I missing.

\n

Within the train_series dataset I would like to convert the timestamp variable from dtype object to dtype datetime.

\n

If I test my function on a small piece of data, it returns something like this:
\n

\n

All looks good. But when I try to apply the same code on the entire 140mil rows, the timestamp column does not convert. Moreover, it shows that the allocated memory for this column is bigger.

\n

I tried randomly sampling the data and take 10k rows and it's the same story, the column does not convert.
\n

\n

Can anybody explain me why is this happening? And maybe if there is a fix to this?

\n

Thank you and happy day ^^

", "messages": [ { "id": 2431100, "postDate": "2023-09-09T19:27:47.770Z", "content": "

Hi all!

\n

I have a weird problem and I do not understand what am I missing.

\n

Within the train_series dataset I would like to convert the timestamp variable from dtype object to dtype datetime.

\n

If I test my function on a small piece of data, it returns something like this:
\n

\n

All looks good. But when I try to apply the same code on the entire 140mil rows, the timestamp column does not convert. Moreover, it shows that the allocated memory for this column is bigger.

\n

I tried randomly sampling the data and take 10k rows and it's the same story, the column does not convert.
\n

\n

Can anybody explain me why is this happening? And maybe if there is a fix to this?

\n

Thank you and happy day ^^

", "rawMarkdown": "Hi all!\n\nI have a weird problem and I do not understand what am I missing.\n\nWithin the `train_series` dataset I would like to convert the `timestamp` variable from dtype object to dtype datetime.\n\nIf I test my function on a small piece of data, it returns something like this:\n\n\nAll looks good. But when I try to apply the same code on the entire 140mil rows, the `timestamp` column does not convert. Moreover, it shows that the allocated memory for this column is bigger.\n\nI tried randomly sampling the data and take 10k rows and it's the same story, the column does not convert.\n\n\nCan anybody explain me why is this happening? And maybe if there is a fix to this?\n\nThank you and happy day ^^", "votes": 10 }, { "id": 2431106, "postDate": "2023-09-09T19:33:18.903Z", "content": "

Dear @andradaolteanu

\n

Perhaps try

\n
df[] = pd.to_datetime(df[],utc=)\n
\n

All the best,
\ncarl

", "rawMarkdown": "Dear @andradaolteanu \n\nPerhaps try \n\n```python\ndf[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"],utc=True)\n```\n\nAll the best,\ncarl", "votes": 8, "replies": [ { "id": 2431755, "postDate": "2023-09-10T11:29:13.610Z", "content": "

YES IT WORKED! Thank you! ❤️

", "rawMarkdown": "YES IT WORKED! Thank you! ❤️", "votes": 2 }, { "id": 2432927, "postDate": "2023-09-11T08:03:40.287Z", "content": "

@carlmcbrideellis - your suggestion works, but it will give UTC, which may give an inconsistent picture across different geographical locations.

\n

To get local time one can do:

\n
df[] = pd.to_datetime(df[]).apply( t: t.tz_localize())\n
\n

Note that one needs to apply localisation, because that function cannot vectorise across different timezones.

", "rawMarkdown": "@carlmcbrideellis - your suggestion works, but it will give UTC, which may give an inconsistent picture across different geographical locations.\n\nTo get local time one can do:\n\n```python\ndf['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n```\n\nNote that one needs to `apply` localisation, because that function cannot vectorise across different timezones.", "votes": 7, "replies": [ { "id": 2433594, "postDate": "2023-09-11T17:10:40.577Z", "content": "

Dear @jagofc

\n

Good point!
\n(Shame that using an apply is sooo slowwww!)

\n

All the best,
\ncarl

", "rawMarkdown": "Dear @jagofc \n\nGood point! \n(Shame that using an `apply` is sooo slowwww!)\n\nAll the best,\ncarl", "votes": 1 }, { "id": 2433844, "postDate": "2023-09-11T22:08:29.913Z", "rawMarkdown": "", "votes": 4, "isDeleted": true }, { "id": 2434272, "postDate": "2023-09-12T07:42:17.347Z", "content": "

@weka511 - it's true. Just my take - I'm using step as the unique indexer, and local time just as a source of features. Also nothing wrong with preserving timestamp (utc as per @carlmcbrideellis) as an indexer and defining a new column timestamp_local for local time features instead of overwriting timestamp.

", "rawMarkdown": "@weka511 - it's true. Just my take - I'm using `step` as the unique indexer, and local time just as a source of features. Also nothing wrong with preserving `timestamp` (utc as per @carlmcbrideellis) as an indexer and defining a new column `timestamp_local` for local time features instead of overwriting `timestamp`.", "votes": 2 }, { "id": 2434380, "postDate": "2023-09-12T09:06:12.340Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 2434438, "postDate": "2023-09-12T09:47:23.117Z", "content": "

Dear @jagofc and @weka511

\n

Solved:

\n
df[] = pd.to_datetime(df[]).apply( t: t.tz_localize())\n
\n

is indeed the way too go; see my trivial test here: \"Zzzs: EDA: UTC or no UTC\"

\n

Using UTC:

\n

Wake in the morning: 3925
\nGo to sleep in the morning: 4693
\nSleep in the afternoon: 100
\nWake in the afternoon: 869

\n

Using local

\n

Wake in the morning: 4760
\nGo to sleep in the morning: 956
\nSleep in the afternoon: 3837
\nWake in the afternoon: 34

\n

All the best!
\ncarl

", "rawMarkdown": "Dear @jagofc and @weka511 \n\nSolved: \n\n```python\ndf['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n```\n\nis indeed the way too go; see my trivial test here: [\"Zzzs: EDA: UTC or no UTC\"](https://www.kaggle.com/code/carlmcbrideellis/zzzs-eda-utc-or-no-utc)\n\nUsing UTC:\n\nWake in the morning: 3925\nGo to sleep in the morning: 4693\nSleep in the afternoon: 100\nWake in the afternoon: 869\n\nUsing local\n\nWake in the morning: 4760\nGo to sleep in the morning: 956\nSleep in the afternoon: 3837\nWake in the afternoon: 34\n\n\nAll the best!\ncarl", "votes": 3 } ] } ] }, { "id": 2431139, "postDate": "2023-09-09T20:32:04.227Z", "content": "

There is a timezone at the end of the timestamp in string format, that doesn't seem to be recognized by pd.todatetime.
\nOne work around seems to be to cut the string to remove the timezone. (maybe it should be kept elsewhere for modelling).

\n
 datetime\ntargets = pd.read_csv()\ntargets.timestamp = targets.timestamp.[:-]\ntargets.timestamp = pd.to_datetime(targets.timestamp)\n
", "rawMarkdown": "There is a timezone at the end of the timestamp in string format, that doesn't seem to be recognized by pd.todatetime.\nOne work around seems to be to cut the string to remove the timezone. (maybe it should be kept elsewhere for modelling).\n\n```python\nimport datetime\ntargets = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\ntargets.timestamp = targets.timestamp.str[:-8]\ntargets.timestamp = pd.to_datetime(targets.timestamp)\n```", "votes": 6, "replies": [ { "id": 2433428, "postDate": "2023-09-11T15:28:28.623Z", "content": "

I second this. Since this has to deal with sleep cycles, the timezone (while might be significant for regional grouping in case a timezone has different sleeping habits; which might be novel to investigate), it seems like the hour relative to daylight is a more important feature than normalizing the times.

\n

Think of it this way, for those with a sleep cycle based on day-night, it's more important to know the relative hour in terms of daylight (for example 8pm when truncating the timezone should have relatively similar amounts of daylight, so it's more of an apples-to-apples comparison, whereas a UTC conversion loses this detail).

", "rawMarkdown": "I second this. Since this has to deal with sleep cycles, the timezone (while might be significant for regional grouping in case a timezone has different sleeping habits; which might be novel to investigate), it seems like the hour relative to daylight is a more important feature than normalizing the times.\n\nThink of it this way, for those with a sleep cycle based on day-night, it's more important to know the relative hour in terms of daylight (for example 8pm when truncating the timezone should have relatively similar amounts of daylight, so it's more of an apples-to-apples comparison, whereas a UTC conversion loses this detail).", "votes": 2, "replies": [ { "id": 2433492, "postDate": "2023-09-11T16:03:01.440Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2453438, "postDate": "2023-09-24T03:41:09.693Z", "content": "

Hi @rknuuds ,

\n

I also agree with you.
\nBut in new data update they have removed the time zone information completely.

", "rawMarkdown": "Hi @rknuuds ,\n\nI also agree with you.\nBut in new data update they have removed the time zone information completely." } ] } ] }, { "id": 2432028, "postDate": "2023-09-10T15:00:22.043Z", "content": "

Hi Andrada,

\n

Though Carl and Lucas already answered your issue (Parsing Timestamp).

\n

It's always nice to have a plan B (sometimes C, D too : )
\nThat's what I found on this competition:

\n

Viz of sleeping time series By Chun Fu
\nhttps://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series

\n

CMI Sleep Visualization - By Connor
\nhttps://www.kaggle.com/code/clane9/cmi-sleep-visualization

\n

Sleeping patterns - EDA - By Robert Kwiatkowski
\nhttps://www.kaggle.com/code/datark1/sleeping-patterns-eda

\n

Xgboost Notebook Updated - By Anmar Ahmadi Qazi
\nhttps://www.kaggle.com/code/ammarahmadqazi/xgboost-notebook-updated

\n

204_Preprocessing_Data - By Alvaro Rodriguez
\nhttps://www.kaggle.com/code/alvrod/204-preprocessing-data

\n

Kind Regards,
\nMarília.

", "rawMarkdown": "Hi Andrada,\n\nThough Carl and Lucas already answered your issue (Parsing Timestamp). \n\nIt's always nice to have a plan B (sometimes C, D too : )\nThat's what I found on this competition: \n\nViz of sleeping time series By Chun Fu\nhttps://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series\n\nCMI Sleep Visualization - By Connor\nhttps://www.kaggle.com/code/clane9/cmi-sleep-visualization\n\nSleeping patterns - EDA - By Robert Kwiatkowski\nhttps://www.kaggle.com/code/datark1/sleeping-patterns-eda\n\nXgboost Notebook Updated - By Anmar Ahmadi Qazi\nhttps://www.kaggle.com/code/ammarahmadqazi/xgboost-notebook-updated\n\n204_Preprocessing_Data - By Alvaro Rodriguez\nhttps://www.kaggle.com/code/alvrod/204-preprocessing-data\n\nKind Regards,\nMarília.", "votes": 3 }, { "id": 2431135, "postDate": "2023-09-09T20:05:56.247Z", "content": "

The mix of timezones (due to DST) causes problems for pandas. Small samples are less likely to encounter this problem

\n

From the Pandas documentation:

\n
\n

The following causes are responsible for datetime.datetime objects being returned (possibly inside an Index or a Series with object dtype) instead of a proper pandas designated type (Timestamp, DatetimeIndex or Series with datetime64 dtype):
\n …

\n \n
", "rawMarkdown": "The mix of timezones (due to DST) causes problems for pandas. Small samples are less likely to encounter this problem\n\nFrom the [Pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html):\n>The following causes are responsible for datetime.datetime **objects being returned** (possibly inside an Index or a Series with object dtype) **instead of a proper pandas designated type** (Timestamp, DatetimeIndex or Series with datetime64 dtype):\n>...\n>- when utc=False (default) and the input is an array-like or Series containing mixed naive/aware datetime, or aware with mixed time offsets. *Note that this happens in the (quite frequent) situation when the timezone has a daylight savings policy.* In that case you may wish to use utc=True.", "votes": 4 }, { "id": 2453442, "postDate": "2023-09-24T03:51:29.340Z", "content": "

Hi @andradaolteanu ,
\nIts little late but if it helps, I am sharing my observations.

\n

I found that operating on whole column is faster that pandarellel parallel_apply

\n

using function once returned column with dtype object.
\nSomehow applying the same function twice worked for me

\n

train_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')
\ntrain_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')

\n

this changed column dtype to datetime

\n

I don't know, if this is some kind of bug or some intricate pandas thing.

\n

the quoted code lines are from following notebook by me :

\n

CMI-DSS-23 94%-memory reduction | 20-lines of code

", "rawMarkdown": "Hi @andradaolteanu ,\nIts little late but if it helps, I am sharing my observations.\n\nI found that operating on whole column is faster that pandarellel parallel_apply\n\nusing function once returned column with dtype object.\nSomehow applying the same function twice worked for me\n\ntrain_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')\ntrain_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')\n\nthis changed column dtype to datetime \n\nI don't know, if this is some kind of bug or some intricate pandas thing.\n\nthe quoted code lines are from following notebook by me :\n\n[CMI-DSS-23 94%-memory reduction | 20-lines of code](https://www.kaggle.com/code/rohitchaudhari25/cmi-dss-23-94-memory-reduction-20-lines-of-code)" }, { "id": 2433555, "postDate": "2023-09-11T16:49:05.700Z", "content": "

Hello @andradaolteanu

\n

I did not have any problems using the format specified on the data page: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data

\n
train_series_df[] = pd.to_datetime(\n    train_series_df[],\n    =\n)\n
\n

NOTE: Because of the size of the dataset, converting the timestamp column in the entire dataset could be super slow.

\n

\"\"

", "rawMarkdown": "Hello @andradaolteanu\n\nI did not have any problems using the format specified on the data page: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\n\n```python\ntrain_series_df['timestamp'] = pd.to_datetime(\n train_series_df['timestamp'],\n format='%Y-%m-%dT%H:%M:%S%z'\n)\n```\n\n_NOTE: Because of the size of the dataset, converting the timestamp column in the entire dataset could be super slow._\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F8b35f0c752582ffef5c44dbeeef5da90%2FScreenshot%202023-09-11%20at%2021.34.50.png?generation=1694450884718531&alt=media)", "votes": 2, "isDeleted": true }, { "id": 2431333, "postDate": "2023-09-10T03:39:52.573Z", "rawMarkdown": "", "votes": -6, "isDeleted": true } ], "comments": [ { "id": 2431106, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-09T19:33:18.903000", "content": "

Dear @andradaolteanu

\n

Perhaps try

\n
df[] = pd.to_datetime(df[],utc=)\n
\n

All the best,
\ncarl

", "votes": 8, "replies": [ { "id": 2431755, "author_name": "Andrada", "author_url": "", "post_date": "2023-09-10T11:29:13.610000", "content": "

YES IT WORKED! Thank you! ❤️

", "votes": 2, "replies": [] }, { "id": 2432927, "author_name": "jagofc", "author_url": "", "post_date": "2023-09-11T08:03:40.287000", "content": "

@carlmcbrideellis - your suggestion works, but it will give UTC, which may give an inconsistent picture across different geographical locations.

\n

To get local time one can do:

\n
df[] = pd.to_datetime(df[]).apply( t: t.tz_localize())\n
\n

Note that one needs to apply localisation, because that function cannot vectorise across different timezones.

", "votes": 7, "replies": [ { "id": 2433594, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-11T17:10:40.577000", "content": "

Dear @jagofc

\n

Good point!
\n(Shame that using an apply is sooo slowwww!)

\n

All the best,
\ncarl

", "votes": 1, "replies": [] }, { "id": 2433844, "author_name": "", "author_url": "", "post_date": "2023-09-11T22:08:29.913000", "content": "", "votes": 4, "replies": [] }, { "id": 2434272, "author_name": "jagofc", "author_url": "", "post_date": "2023-09-12T07:42:17.347000", "content": "

@weka511 - it's true. Just my take - I'm using step as the unique indexer, and local time just as a source of features. Also nothing wrong with preserving timestamp (utc as per @carlmcbrideellis) as an indexer and defining a new column timestamp_local for local time features instead of overwriting timestamp.

", "votes": 2, "replies": [] }, { "id": 2434380, "author_name": "", "author_url": "", "post_date": "2023-09-12T09:06:12.340000", "content": "", "votes": 3, "replies": [] }, { "id": 2434438, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-12T09:47:23.117000", "content": "

Dear @jagofc and @weka511

\n

Solved:

\n
df[] = pd.to_datetime(df[]).apply( t: t.tz_localize())\n
\n

is indeed the way too go; see my trivial test here: \"Zzzs: EDA: UTC or no UTC\"

\n

Using UTC:

\n

Wake in the morning: 3925
\nGo to sleep in the morning: 4693
\nSleep in the afternoon: 100
\nWake in the afternoon: 869

\n

Using local

\n

Wake in the morning: 4760
\nGo to sleep in the morning: 956
\nSleep in the afternoon: 3837
\nWake in the afternoon: 34

\n

All the best!
\ncarl

", "votes": 3, "replies": [] } ] } ] }, { "id": 2431139, "author_name": "Lucas Morin", "author_url": "", "post_date": "2023-09-09T20:32:04.227000", "content": "

There is a timezone at the end of the timestamp in string format, that doesn't seem to be recognized by pd.todatetime.
\nOne work around seems to be to cut the string to remove the timezone. (maybe it should be kept elsewhere for modelling).

\n
 datetime\ntargets = pd.read_csv()\ntargets.timestamp = targets.timestamp.[:-]\ntargets.timestamp = pd.to_datetime(targets.timestamp)\n
", "votes": 6, "replies": [ { "id": 2433428, "author_name": "Rob K.", "author_url": "", "post_date": "2023-09-11T15:28:28.623000", "content": "

I second this. Since this has to deal with sleep cycles, the timezone (while might be significant for regional grouping in case a timezone has different sleeping habits; which might be novel to investigate), it seems like the hour relative to daylight is a more important feature than normalizing the times.

\n

Think of it this way, for those with a sleep cycle based on day-night, it's more important to know the relative hour in terms of daylight (for example 8pm when truncating the timezone should have relatively similar amounts of daylight, so it's more of an apples-to-apples comparison, whereas a UTC conversion loses this detail).

", "votes": 2, "replies": [ { "id": 2433492, "author_name": "", "author_url": "", "post_date": "2023-09-11T16:03:01.440000", "content": "", "votes": 0, "replies": [] }, { "id": 2453438, "author_name": "Rohit Chaudhari", "author_url": "", "post_date": "2023-09-24T03:41:09.693000", "content": "

Hi @rknuuds ,

\n

I also agree with you.
\nBut in new data update they have removed the time zone information completely.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2432028, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-09-10T15:00:22.043000", "content": "

Hi Andrada,

\n

Though Carl and Lucas already answered your issue (Parsing Timestamp).

\n

It's always nice to have a plan B (sometimes C, D too : )
\nThat's what I found on this competition:

\n

Viz of sleeping time series By Chun Fu
\nhttps://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series

\n

CMI Sleep Visualization - By Connor
\nhttps://www.kaggle.com/code/clane9/cmi-sleep-visualization

\n

Sleeping patterns - EDA - By Robert Kwiatkowski
\nhttps://www.kaggle.com/code/datark1/sleeping-patterns-eda

\n

Xgboost Notebook Updated - By Anmar Ahmadi Qazi
\nhttps://www.kaggle.com/code/ammarahmadqazi/xgboost-notebook-updated

\n

204_Preprocessing_Data - By Alvaro Rodriguez
\nhttps://www.kaggle.com/code/alvrod/204-preprocessing-data

\n

Kind Regards,
\nMarília.

", "votes": 3, "replies": [] }, { "id": 2431135, "author_name": "Quillen", "author_url": "", "post_date": "2023-09-09T20:05:56.247000", "content": "

The mix of timezones (due to DST) causes problems for pandas. Small samples are less likely to encounter this problem

\n

From the Pandas documentation:

\n
\n

The following causes are responsible for datetime.datetime objects being returned (possibly inside an Index or a Series with object dtype) instead of a proper pandas designated type (Timestamp, DatetimeIndex or Series with datetime64 dtype):
\n …

\n \n
", "votes": 4, "replies": [] }, { "id": 2453442, "author_name": "Rohit Chaudhari", "author_url": "", "post_date": "2023-09-24T03:51:29.340000", "content": "

Hi @andradaolteanu ,
\nIts little late but if it helps, I am sharing my observations.

\n

I found that operating on whole column is faster that pandarellel parallel_apply

\n

using function once returned column with dtype object.
\nSomehow applying the same function twice worked for me

\n

train_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')
\ntrain_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')

\n

this changed column dtype to datetime

\n

I don't know, if this is some kind of bug or some intricate pandas thing.

\n

the quoted code lines are from following notebook by me :

\n

CMI-DSS-23 94%-memory reduction | 20-lines of code

", "votes": 0, "replies": [] }, { "id": 2433555, "author_name": "Bhargav Kowshik", "author_url": "", "post_date": "2023-09-11T16:49:05.700000", "content": "

Hello @andradaolteanu

\n

I did not have any problems using the format specified on the data page: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data

\n
train_series_df[] = pd.to_datetime(\n    train_series_df[],\n    =\n)\n
\n

NOTE: Because of the size of the dataset, converting the timestamp column in the entire dataset could be super slow.

\n

\"\"

", "votes": 2, "replies": [] }, { "id": 2431333, "author_name": "", "author_url": "", "post_date": "2023-09-10T03:39:52.573000", "content": "", "votes": -6, "replies": [] } ], "raw_markdown_by_id": { "2431100": "Hi all!\n\nI have a weird problem and I do not understand what am I missing.\n\nWithin the `train_series` dataset I would like to convert the `timestamp` variable from dtype object to dtype datetime.\n\nIf I test my function on a small piece of data, it returns something like this:\n\n\nAll looks good. But when I try to apply the same code on the entire 140mil rows, the `timestamp` column does not convert. Moreover, it shows that the allocated memory for this column is bigger.\n\nI tried randomly sampling the data and take 10k rows and it's the same story, the column does not convert.\n\n\nCan anybody explain me why is this happening? And maybe if there is a fix to this?\n\nThank you and happy day ^^", "2431106": "Dear @andradaolteanu \n\nPerhaps try \n\n```python\ndf[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"],utc=True)\n```\n\nAll the best,\ncarl", "2431139": "There is a timezone at the end of the timestamp in string format, that doesn't seem to be recognized by pd.todatetime.\nOne work around seems to be to cut the string to remove the timezone. (maybe it should be kept elsewhere for modelling).\n\n```python\nimport datetime\ntargets = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\ntargets.timestamp = targets.timestamp.str[:-8]\ntargets.timestamp = pd.to_datetime(targets.timestamp)\n```", "2432028": "Hi Andrada,\n\nThough Carl and Lucas already answered your issue (Parsing Timestamp). \n\nIt's always nice to have a plan B (sometimes C, D too : )\nThat's what I found on this competition: \n\nViz of sleeping time series By Chun Fu\nhttps://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series\n\nCMI Sleep Visualization - By Connor\nhttps://www.kaggle.com/code/clane9/cmi-sleep-visualization\n\nSleeping patterns - EDA - By Robert Kwiatkowski\nhttps://www.kaggle.com/code/datark1/sleeping-patterns-eda\n\nXgboost Notebook Updated - By Anmar Ahmadi Qazi\nhttps://www.kaggle.com/code/ammarahmadqazi/xgboost-notebook-updated\n\n204_Preprocessing_Data - By Alvaro Rodriguez\nhttps://www.kaggle.com/code/alvrod/204-preprocessing-data\n\nKind Regards,\nMarília.", "2431135": "The mix of timezones (due to DST) causes problems for pandas. Small samples are less likely to encounter this problem\n\nFrom the [Pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html):\n>The following causes are responsible for datetime.datetime **objects being returned** (possibly inside an Index or a Series with object dtype) **instead of a proper pandas designated type** (Timestamp, DatetimeIndex or Series with datetime64 dtype):\n>...\n>- when utc=False (default) and the input is an array-like or Series containing mixed naive/aware datetime, or aware with mixed time offsets. *Note that this happens in the (quite frequent) situation when the timezone has a daylight savings policy.* In that case you may wish to use utc=True.", "2453442": "Hi @andradaolteanu ,\nIts little late but if it helps, I am sharing my observations.\n\nI found that operating on whole column is faster that pandarellel parallel_apply\n\nusing function once returned column with dtype object.\nSomehow applying the same function twice worked for me\n\ntrain_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')\ntrain_events[\"timestamp\"] = pd.to_datetime(train_events[\"timestamp\"],errors = \"coerce\",format='ISO8601')\n\nthis changed column dtype to datetime \n\nI don't know, if this is some kind of bug or some intricate pandas thing.\n\nthe quoted code lines are from following notebook by me :\n\n[CMI-DSS-23 94%-memory reduction | 20-lines of code](https://www.kaggle.com/code/rohitchaudhari25/cmi-dss-23-94-memory-reduction-20-lines-of-code)", "2433555": "Hello @andradaolteanu\n\nI did not have any problems using the format specified on the data page: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data\n\n```python\ntrain_series_df['timestamp'] = pd.to_datetime(\n train_series_df['timestamp'],\n format='%Y-%m-%dT%H:%M:%S%z'\n)\n```\n\n_NOTE: Because of the size of the dataset, converting the timestamp column in the entire dataset could be super slow._\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5210536%2F8b35f0c752582ffef5c44dbeeef5da90%2FScreenshot%202023-09-11%20at%2021.34.50.png?generation=1694450884718531&alt=media)", "2431333": "" } }