{ "id": 440170, "title": "80% reducted datasets", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/440170", "author_name": "Andrada", "post_date": "2023-09-14T03:01:53.697000", "votes": 47, "comment_count": 5, "views": 0, "content": "

Hi all!

\n

The first thing I tried to do in this comp was to upload and look at the data. The notebook kept crushing when I was performing some basic operations of the train_series.parquet file, so I tried to reduce the memory usage as much as possible.

\n

What I did:

\n\n

The main difference was made by remapping the series_id and the datetime convertion oftimestamp`.

\n

Dataset: https://www.kaggle.com/datasets/andradaolteanu/detect-sleep-states-memory-decrease

\n

Notebook on process and numbers: https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction

\n

Hope it helps, if you find any flaw please do let me know :)

", "messages": [ { "id": 2438061, "postDate": "2023-09-14T03:01:53.697Z", "content": "

Hi all!

\n

The first thing I tried to do in this comp was to upload and look at the data. The notebook kept crushing when I was performing some basic operations of the train_series.parquet file, so I tried to reduce the memory usage as much as possible.

\n

What I did:

\n\n

The main difference was made by remapping the series_id and the datetime convertion oftimestamp`.

\n

Dataset: https://www.kaggle.com/datasets/andradaolteanu/detect-sleep-states-memory-decrease

\n

Notebook on process and numbers: https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction

\n

Hope it helps, if you find any flaw please do let me know :)

", "rawMarkdown": "Hi all!\n\nThe first thing I tried to do in this comp was to upload and look at the data. The notebook kept crushing when I was performing some basic operations of the `train_series.parquet` file, so I tried to reduce the memory usage as much as possible.\n\nWhat I did:\n* `series_id`: from object -> uint16\n * there are only 277 unique ids (in both events and series datasets)\n * this one is formed by a long sequence of numbers and letters\n * hence I remapped them with a new variable called`id_map` from 0 to 277\n * faster during development, can easily switch back to original id\n* `night`: from int64 -> uint16\n* `event`: from object -> uint8\n * relabeled as follows: 'onset':'1', 'wakeup':'2'\n* `step`: from int64 -> uint32\n* `timestamp`: from object -> datetime64\n\nThe main difference was made by remapping the `series_id and the datetime convertion of `timestamp`.\n\n### Dataset: https://www.kaggle.com/datasets/andradaolteanu/detect-sleep-states-memory-decrease\n\n### Notebook on process and numbers: https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction\n\nHope it helps, if you find any flaw please do let me know :)", "votes": 45 }, { "id": 2490766, "postDate": "2023-10-21T03:44:25.137Z", "content": "

@andradaolteanu could you check if your timestamp conversion was performed correctly? I found some cases where there were problems with timezones. Thanks in advance :)

", "rawMarkdown": "@andradaolteanu could you check if your `timestamp` conversion was performed correctly? I found some [cases](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/448683) where there were problems with timezones. Thanks in advance :)" }, { "id": 2438091, "postDate": "2023-09-14T03:48:32.917Z", "content": "

@andradaolteanu I was also having trouble with data handling, so this is very helpful. The \"timestamp\" column is UTC time, right? Users will probably want to use local time, so it's a good idea to specify it somewhere.

", "rawMarkdown": "@andradaolteanu I was also having trouble with data handling, so this is very helpful. The \"timestamp\" column is UTC time, right? Users will probably want to use local time, ***so it's a good idea to specify it somewhere.***", "replies": [ { "id": 2438269, "postDate": "2023-09-14T07:06:41.290Z", "content": "

the timestamp is local :) it was UTC at some point, but I converted back to local for feature engineering purposes

", "rawMarkdown": "the timestamp is local :) it was UTC at some point, but I converted back to local for feature engineering purposes", "votes": 1, "replies": [ { "id": 2438338, "postDate": "2023-09-14T07:53:50.287Z", "content": "

Thanks, I had a misunderstanding @andradaolteanu. I was able to confirm that timestamp is local time.

", "rawMarkdown": "Thanks, I had a misunderstanding @andradaolteanu. I was able to confirm that timestamp is local time.", "votes": 1 } ] } ] }, { "id": 2438093, "postDate": "2023-09-14T03:50:48.547Z", "content": "
\n

There are only 277 unique ids (in both events and series datasets). This one is formed by a long sequence of numbers and letters. Hence I remapped them with a new variable calledid_map from 0 to 277. Faster during development, can easily switch back to original id.

\n
\n

This is an interesting and useful idea. Thank you for sharing @andradaolteanu

", "rawMarkdown": "> There are only 277 unique ids (in both events and series datasets). This one is formed by a long sequence of numbers and letters. Hence I remapped them with a new variable calledid_map from 0 to 277. Faster during development, can easily switch back to original id.\n\nThis is an interesting and useful idea. Thank you for sharing @andradaolteanu ", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 2490766, "author_name": "moth", "author_url": "", "post_date": "2023-10-21T03:44:25.137000", "content": "

@andradaolteanu could you check if your timestamp conversion was performed correctly? I found some cases where there were problems with timezones. Thanks in advance :)

", "votes": 0, "replies": [] }, { "id": 2438091, "author_name": "mh", "author_url": "", "post_date": "2023-09-14T03:48:32.917000", "content": "

@andradaolteanu I was also having trouble with data handling, so this is very helpful. The \"timestamp\" column is UTC time, right? Users will probably want to use local time, so it's a good idea to specify it somewhere.

", "votes": 0, "replies": [ { "id": 2438269, "author_name": "Andrada", "author_url": "", "post_date": "2023-09-14T07:06:41.290000", "content": "

the timestamp is local :) it was UTC at some point, but I converted back to local for feature engineering purposes

", "votes": 1, "replies": [ { "id": 2438338, "author_name": "mh", "author_url": "", "post_date": "2023-09-14T07:53:50.287000", "content": "

Thanks, I had a misunderstanding @andradaolteanu. I was able to confirm that timestamp is local time.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2438093, "author_name": "Bhargav Kowshik", "author_url": "", "post_date": "2023-09-14T03:50:48.547000", "content": "
\n

There are only 277 unique ids (in both events and series datasets). This one is formed by a long sequence of numbers and letters. Hence I remapped them with a new variable calledid_map from 0 to 277. Faster during development, can easily switch back to original id.

\n
\n

This is an interesting and useful idea. Thank you for sharing @andradaolteanu

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2438061": "Hi all!\n\nThe first thing I tried to do in this comp was to upload and look at the data. The notebook kept crushing when I was performing some basic operations of the `train_series.parquet` file, so I tried to reduce the memory usage as much as possible.\n\nWhat I did:\n* `series_id`: from object -> uint16\n * there are only 277 unique ids (in both events and series datasets)\n * this one is formed by a long sequence of numbers and letters\n * hence I remapped them with a new variable called`id_map` from 0 to 277\n * faster during development, can easily switch back to original id\n* `night`: from int64 -> uint16\n* `event`: from object -> uint8\n * relabeled as follows: 'onset':'1', 'wakeup':'2'\n* `step`: from int64 -> uint32\n* `timestamp`: from object -> datetime64\n\nThe main difference was made by remapping the `series_id and the datetime convertion of `timestamp`.\n\n### Dataset: https://www.kaggle.com/datasets/andradaolteanu/detect-sleep-states-memory-decrease\n\n### Notebook on process and numbers: https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction\n\nHope it helps, if you find any flaw please do let me know :)", "2490766": "@andradaolteanu could you check if your `timestamp` conversion was performed correctly? I found some [cases](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/448683) where there were problems with timezones. Thanks in advance :)", "2438091": "@andradaolteanu I was also having trouble with data handling, so this is very helpful. The \"timestamp\" column is UTC time, right? Users will probably want to use local time, ***so it's a good idea to specify it somewhere.***", "2438093": "> There are only 277 unique ids (in both events and series datasets). This one is formed by a long sequence of numbers and letters. Hence I remapped them with a new variable calledid_map from 0 to 277. Faster during development, can easily switch back to original id.\n\nThis is an interesting and useful idea. Thank you for sharing @andradaolteanu " } }