Currently using the environment given by default with a new Kaggle notebook (no accelerator, 30 GB RAM, 73.1 GB disk space, and 12 hrs session time)
\nI've been trying to pre-process and compress the training data read from train_series.parquet and train_events.csv. However, I've noticed that my current implementation takes up a lot of my notebook's runtime (takes ~15 min to run) and the RAM usage is p bad (spike from about 6 GB before the pre-processing stage up to about 15-17 GB)
\nOverall, the memory occupied by the processed dataframes are lower than the initial memory usage, but in converting data types there seems to be a lot of RAM used. Even after running garbage collection after this stage, memory is not freed up.
\nLooking at other discussions on memory reduction, the use of a categorical variable for the series_id might be the culprit (haven't seen anyone else use it yet). Why might this be the case?
\nIs it possible there are memory leaks?
\ntrain_events = train_events.dropna()\n\n\n\n\n\n\n\ntrain_events[] = train_events[].astype()\n\n\ntrain_events[] = train_events[].astype()\n\n\n\ntrain_series[] = pd.to_datetime(train_series[], =)\n\n\n\n\n\ntrain_events[] = train_events[].astype()\n\n\ntrain_events[] = train_events[].astype()\n\n\ntrain_series[] = train_series[].astype()\n\n\n\n\n\n\n\ntrain_events = train_events.drop(columns=)\nThanks in advance
", "messages": [ { "id": "2454699", "postDate": "09/25/2023 03:46:24", "content": "Currently using the environment given by default with a new Kaggle notebook (no accelerator, 30 GB RAM, 73.1 GB disk space, and 12 hrs session time)
\nI've been trying to pre-process and compress the training data read from train_series.parquet and train_events.csv. However, I've noticed that my current implementation takes up a lot of my notebook's runtime (takes ~15 min to run) and the RAM usage is p bad (spike from about 6 GB before the pre-processing stage up to about 15-17 GB)
\nOverall, the memory occupied by the processed dataframes are lower than the initial memory usage, but in converting data types there seems to be a lot of RAM used. Even after running garbage collection after this stage, memory is not freed up.
\nLooking at other discussions on memory reduction, the use of a categorical variable for the series_id might be the culprit (haven't seen anyone else use it yet). Why might this be the case?
\nIs it possible there are memory leaks?
\ntrain_events = train_events.dropna()\n\n\n\n\n\n\n\ntrain_events[] = train_events[].astype()\n\n\ntrain_events[] = train_events[].astype()\n\n\n\ntrain_series[] = pd.to_datetime(train_series[], =)\n\n\n\n\n\ntrain_events[] = train_events[].astype()\n\n\ntrain_events[] = train_events[].astype()\n\n\ntrain_series[] = train_series[].astype()\n\n\n\n\n\n\n\ntrain_events = train_events.drop(columns=)\nThanks in advance
", "rawMarkdown": "Currently using the environment given by default with a new Kaggle notebook (no accelerator, 30 GB RAM, 73.1 GB disk space, and 12 hrs session time)\n\nI've been trying to pre-process and compress the training data read from *train_series.parquet* and *train_events.csv*. However, I've noticed that my current implementation takes up a lot of my notebook's runtime (takes ~15 min to run) and the RAM usage is p bad (spike from about 6 GB before the pre-processing stage up to about 15-17 GB)\n\nOverall, the memory occupied by the processed dataframes are lower than the initial memory usage, but in converting data types there seems to be a lot of RAM used. Even after running garbage collection after this stage, memory is not freed up.\n\nLooking at other discussions on memory reduction, the use of a categorical variable for the series_id might be the culprit (haven't seen anyone else use it yet). Why might this be the case?\n\nIs it possible there are memory leaks?\n\n```python\ntrain_events = train_events.dropna()\n\n### Numeric type conversions ###\n\n# Compress data down to more manageable size by converting to smaller types (i.e., fewer bytes)\n# Per recommendation by: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438394\n\n# Convert train_events 'step' column into int\ntrain_events['step'] = train_events['step'].astype('uint32')\n\n# Convert train_events 'night' column into short int\ntrain_events['night'] = train_events['night'].astype('uint8')\n\n### Datetime conversions ###\n\ntrain_series['timestamp'] = pd.to_datetime(train_series['timestamp'], format='%Y-%m-%dT%H:%M:%S-%f')\n# test_series['timestamp'] = pd.to_datetime(test_series['timestamp'], format='%Y-%m-%dT%H:%M:%S-%f')\n\n### Categorical type conversions ###\n\n# Convert string-based classification of events (\"onset\" and \"wakeup\") to categories\ntrain_events['event'] = train_events['event'].astype('category')\n\n# Convert train_events string-based series IDs to categories\ntrain_events['series_id'] = train_events['series_id'].astype('category')\n\n# Convert train_series string-based series IDs to categories\ntrain_series['series_id'] = train_series['series_id'].astype('category')\n\n# Convert test_series string-based series IDs to categories\n# test_series['series_id'] = test_series['series_id'].astype('category')\n\n### Drop redundant column ###\n\n# Drop timestamp column from train_events... already in train_series\ntrain_events = train_events.drop(columns='timestamp')\n```\n\nThanks in advance", "votes": null }, { "id": "2454914", "postDate": "09/25/2023 07:11:45", "content": "Pandas is particularly slow and inefficient when converting the timestamp column of the large parquet file, so that's probably your biggest bottleneck. Try using the Polars library to quickly preprocess train_series.parquet and return a pandas DataFrame. This method uses less memory when performing the operations and it correctly frees up all unused memory without needing to manually run garbage collection.
For example, you can read train_series.parquet, convert timestamps and series_id, and return a pandas DataFrame in <1 minute with the following code:
import polars as pl\nutc = \n = \n\ntrain_series = (pl.scan_parquet(TRAIN_PATH)\n .with_columns([\n (pl.col()\n .str.strptime(pl.Datetime, utc)), \n (pl.col()\n .cast(pl.Categorical))\n ])\n .collect()\n .to_pandas()\n)\nNote that categorical data is not a numpy array, so it can be slower if you try to perform certain operations that require conversion to a np.array.
", "rawMarkdown": "Pandas is particularly slow and inefficient when converting the `timestamp` column of the large parquet file, so that's probably your biggest bottleneck. Try using the [Polars](https://www.pola.rs/) library to quickly preprocess `train_series.parquet` and return a pandas DataFrame. This method uses less memory when performing the operations and it correctly frees up all unused memory without needing to manually run garbage collection.\n\nFor example, you can read `train_series.parquet`, convert `timestamps` and `series_id`, and return a pandas DataFrame in <1 minute with the following code: \n\n```\nimport polars as pl\nutc = \"%Y-%m-%dT%H:%M:%S%z\" #convert to UTC time\nlocal = \"%Y-%m-%dT%H:%M:%S%Z\" #keep local time\n\ntrain_series = (pl.scan_parquet(TRAIN_PATH)\n .with_columns([\n (pl.col(\"timestamp\")\n .str.strptime(pl.Datetime, utc)), #choose utc or local time\n (pl.col(\"series_id\")\n .cast(pl.Categorical))\n ])\n .collect()\n .to_pandas()\n)\n```\n\nNote that categorical data [is not a numpy array](https://pandas.pydata.org/docs/user_guide/categorical.html#categorical-is-not-a-numpy-array), so it can be slower if you try to perform certain operations that require conversion to a np.array.", "votes": null }, { "id": "2454980", "postDate": "09/25/2023 08:23:16", "content": "@quillen : thanks for sharing the polars hint. It is really useful, indeed.
\nHowever, the in-memory dataframe size reduction, the initial concern of @athreyamurali , remains valid even if you read the parquet data with polars, casting the timestamps to datetime on the go. The combination of those two techniques could bring the result expected. You can refer to https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive to see the blended approach utilizing both techniques.
\nApart from that, there is a discussion thread per Best Practices of Tackling BigData-Scale Tasks where various useful performance and optimization tricks are discussed in the context of this contest.
\nI hope it is helpful.
", "rawMarkdown": "quillen : thanks for sharing the polars hint. It is really useful, indeed. \n\nHowever, the in-memory dataframe size reduction, the initial concern of @athreyamurali , remains valid even if you read the parquet data with polars, casting the timestamps to datetime on the go. The combination of those two techniques could bring the result expected. You can refer to https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive to see the blended approach utilizing both techniques.\n\nApart from that, there is a discussion thread per [Best Practices of Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) where various useful performance and optimization tricks are discussed in the context of this contest.\n\nI hope it is helpful.", "votes": null }, { "id": "2455673", "postDate": "09/25/2023 16:32:52", "content": "@quillen and @gvyshnya, thanks to you both for your quick responses. I'm going to try out polars and see how its performance compares.
\nQuillen, looking at the link from the Pandas docs you gave, I'd imagine the bottleneck occurs during any NumPy operations on the modified dataframe (since the series_id column is no longer an np.array after being converted to a categorical variable). Saw a link that supports this argument: https://pbpython.com/pandas_dtypes_cat.html
\nGeorgii, your EDA dive notebook was actually one of the first I looked at for this competition. It's been incredibly useful, so thank you for making your results readily available like this!
\nAs an aside, I have a question about how you parsed the train_series parquet file. Looking at your code, one major difference between it and Quillen's snippet above is that you parse the timestamp column into distinct year, month, and hour columns. Did you do this solely for your hourly insights later on? In an actual submission, is there any benefit to having redundant data across multiple columns?
Thanks again!
", "rawMarkdown": "quillen and @gvyshnya, thanks to you both for your quick responses. I'm going to try out polars and see how its performance compares.\n\nQuillen, looking at the link from the Pandas docs you gave, I'd imagine the bottleneck occurs during any NumPy operations on the modified dataframe (since the series_id column is no longer an np.array after being converted to a categorical variable). Saw a link that supports this argument: https://pbpython.com/pandas_dtypes_cat.html\n\n\nGeorgii, your EDA dive notebook was actually one of the first I looked at for this competition. It's been incredibly useful, so thank you for making your results readily available like this!\nAs an aside, I have a question about how you parsed the train_series parquet file. Looking at your code, one major difference between it and Quillen's snippet above is that you parse the `timestamp` column into distinct year, month, and hour columns. Did you do this solely for your hourly insights later on? In an actual submission, is there any benefit to having redundant data across multiple columns?\n\nThanks again!", "votes": null }, { "id": "2456745", "postDate": "09/26/2023 11:40:36", "content": "@athreyamurali : Hey Athreya, thanks for your question. I parsed the year, month, and hour parts of timestamp into separate columns for the EDA reasons. However, such a feature extraction can be helpful in the ML flow as well whenever you try to tackle the prediction problem of this contest in the regression setup.
Pandas is particularly slow and inefficient when converting the timestamp column of the large parquet file, so that's probably your biggest bottleneck. Try using the Polars library to quickly preprocess train_series.parquet and return a pandas DataFrame. This method uses less memory when performing the operations and it correctly frees up all unused memory without needing to manually run garbage collection.
For example, you can read train_series.parquet, convert timestamps and series_id, and return a pandas DataFrame in <1 minute with the following code:
import polars as pl\nutc = \n = \n\ntrain_series = (pl.scan_parquet(TRAIN_PATH)\n .with_columns([\n (pl.col()\n .str.strptime(pl.Datetime, utc)), \n (pl.col()\n .cast(pl.Categorical))\n ])\n .collect()\n .to_pandas()\n)\nNote that categorical data is not a numpy array, so it can be slower if you try to perform certain operations that require conversion to a np.array.
", "votes": null, "replies": [ { "id": 2454980, "author_name": "gvyshnya", "author_url": "", "post_date": "09/25/2023 08:23:16", "content": "@quillen : thanks for sharing the polars hint. It is really useful, indeed.
\nHowever, the in-memory dataframe size reduction, the initial concern of @athreyamurali , remains valid even if you read the parquet data with polars, casting the timestamps to datetime on the go. The combination of those two techniques could bring the result expected. You can refer to https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive to see the blended approach utilizing both techniques.
\nApart from that, there is a discussion thread per Best Practices of Tackling BigData-Scale Tasks where various useful performance and optimization tricks are discussed in the context of this contest.
\nI hope it is helpful.
", "votes": null, "replies": [ { "id": 2455673, "author_name": "athreyamurali", "author_url": "", "post_date": "09/25/2023 16:32:52", "content": "@quillen and @gvyshnya, thanks to you both for your quick responses. I'm going to try out polars and see how its performance compares.
\nQuillen, looking at the link from the Pandas docs you gave, I'd imagine the bottleneck occurs during any NumPy operations on the modified dataframe (since the series_id column is no longer an np.array after being converted to a categorical variable). Saw a link that supports this argument: https://pbpython.com/pandas_dtypes_cat.html
\nGeorgii, your EDA dive notebook was actually one of the first I looked at for this competition. It's been incredibly useful, so thank you for making your results readily available like this!
\nAs an aside, I have a question about how you parsed the train_series parquet file. Looking at your code, one major difference between it and Quillen's snippet above is that you parse the timestamp column into distinct year, month, and hour columns. Did you do this solely for your hourly insights later on? In an actual submission, is there any benefit to having redundant data across multiple columns?
Thanks again!
", "votes": null, "replies": [ { "id": 2456745, "author_name": "gvyshnya", "author_url": "", "post_date": "09/26/2023 11:40:36", "content": "@athreyamurali : Hey Athreya, thanks for your question. I parsed the year, month, and hour parts of timestamp into separate columns for the EDA reasons. However, such a feature extraction can be helpful in the ML flow as well whenever you try to tackle the prediction problem of this contest in the regression setup.