I am currently using the train data. Loading it in this way:
\ndata_dtype = {\n : ,\n : ,\n : ,\n : ,\n : ,\n }\n\ndata = pd.read_parquet().astype(data_dtype, copy=)\nIt works rather fast, takes about 4-5GB of the RAM.
\nThen I want to add some features, based on the timestamp, so I do this:
\ndata[] = pd.to_datetime(data[], utc=, cache=, =)\nAnd this line works really long. Any ways to deal with it somehow? I want, for example, to add the hour of the day. I tried to load the data in the pandas first, then transfer it to the CUdf library, but it crashes due to memory error. I know that I can wait for some long time, then save preprocessed data in the pickle format. But I cannot do this with the test.
", "messages": [ { "id": "2515325", "postDate": "11/06/2023 21:14:39", "content": "I am currently using the train data. Loading it in this way:
\ndata_dtype = {\n : ,\n : ,\n : ,\n : ,\n : ,\n }\n\ndata = pd.read_parquet().astype(data_dtype, copy=)\nIt works rather fast, takes about 4-5GB of the RAM.
\nThen I want to add some features, based on the timestamp, so I do this:
\ndata[] = pd.to_datetime(data[], utc=, cache=, =)\nAnd this line works really long. Any ways to deal with it somehow? I want, for example, to add the hour of the day. I tried to load the data in the pandas first, then transfer it to the CUdf library, but it crashes due to memory error. I know that I can wait for some long time, then save preprocessed data in the pickle format. But I cannot do this with the test.
", "rawMarkdown": "I am currently using the train data. Loading it in this way:\n\n```python\ndata_dtype = {\n \"series_id\": \"category\",\n \"step\": \"uint32\",\n \"timestamp\": \"str\",\n \"anglez\": \"float32\",\n \"enmo\": \"float32\",\n }\n\ndata = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\").astype(data_dtype, copy=False)\n```\nIt works rather fast, takes about 4-5GB of the RAM.\nThen I want to add some features, based on the timestamp, so I do this:\n\n```python\n# Convert the timestamp column to Timestamp objects\ndata[\"timestamp_datetime\"] = pd.to_datetime(data[\"timestamp\"], utc=False, cache=True, format=\"%Y-%m-%dT%H:%M:%S-%f\")\n```\n\nAnd this line works really long. Any ways to deal with it somehow? I want, for example, to add the hour of the day. I tried to load the data in the pandas first, then transfer it to the CUdf library, but it crashes due to memory error. I know that I can wait for some long time, then save preprocessed data in the pickle format. But I cannot do this with the test.", "votes": null }, { "id": "2515371", "postDate": "11/06/2023 22:10:23", "content": "@manwithaflower : To tell the long story short, you can improve the performance of your data processing pipelines by
\nYou can refer to Best Practices in Tackling BigData-Scale Tasks to see the references to the specific examples of how it can be implemented (this thread is also useful if you like to see other tricks to improve the performance of your pipelines - for example, when you do the model training or/and prepare your submission from the test data).
\nYou can also review a few public notebooks where per-series feature engineering / data preprocessing adds the edge in terms of performance:
\nP. S. From the code fragment you posted in your message, I noticed you attempt to convert the timestamp values to the datetime in the local time zone. You may want to refer to https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check to see the data quality-driven issues with such an approach (at the end, casting them to UTC may be a better idea to work around).
", "rawMarkdown": "manwithaflower : To tell the long story short, you can improve the performance of your data processing pipelines by \n\n- doing the feature engineering in the batch mode, in a series-by-series manner\n- reading data with some of the pandas alternatives that are more powerful when it comes to reading Big Data-scale datasets (Polar, Dask etc.)\n\nYou can refer to [Best Practices in Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) to see the references to the specific examples of how it can be implemented (this thread is also useful if you like to see other tricks to improve the performance of your pipelines - for example, when you do the model training or/and prepare your submission from the test data).\n\nYou can also review a few public notebooks where per-series feature engineering / data preprocessing adds the edge in terms of performance:\n\n- https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction\n- https://www.kaggle.com/code/siddhvr/cmi-ensembling-models-for-better-results\n- etc.\n\nP. S. From the code fragment you posted in your message, I noticed you attempt to convert the timestamp values to the datetime in the local time zone. You may want to refer to https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check to see the data quality-driven issues with such an approach (at the end, casting them to UTC may be a better idea to work around).", "votes": null }, { "id": "2515405", "postDate": "11/06/2023 23:10:05", "content": "Aha, here it is! Thank you very much, finally I found the nice solution!
\nThis notebook is really helpful. As I understood, we should load the series one by one by their id. And then we can do the feature engineering by id as well, using parallelization. Great!
", "rawMarkdown": "Aha, here it is! Thank you very much, finally I found the nice solution!\n\nThis [notebook](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction) is really helpful. As I understood, we should load the series one by one by their id. And then we can do the feature engineering by id as well, using parallelization. Great!", "votes": null } ], "comments": [ { "id": 2515371, "author_name": "gvyshnya", "author_url": "", "post_date": "11/06/2023 22:10:23", "content": "@manwithaflower : To tell the long story short, you can improve the performance of your data processing pipelines by
\nYou can refer to Best Practices in Tackling BigData-Scale Tasks to see the references to the specific examples of how it can be implemented (this thread is also useful if you like to see other tricks to improve the performance of your pipelines - for example, when you do the model training or/and prepare your submission from the test data).
\nYou can also review a few public notebooks where per-series feature engineering / data preprocessing adds the edge in terms of performance:
\nP. S. From the code fragment you posted in your message, I noticed you attempt to convert the timestamp values to the datetime in the local time zone. You may want to refer to https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check to see the data quality-driven issues with such an approach (at the end, casting them to UTC may be a better idea to work around).
", "votes": null, "replies": [ { "id": 2515405, "author_name": "manwithaflower", "author_url": "", "post_date": "11/06/2023 23:10:05", "content": "Aha, here it is! Thank you very much, finally I found the nice solution!
\nThis notebook is really helpful. As I understood, we should load the series one by one by their id. And then we can do the feature engineering by id as well, using parallelization. Great!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2515325": "I am currently using the train data. Loading it in this way:\n\n```python\ndata_dtype = {\n \"series_id\": \"category\",\n \"step\": \"uint32\",\n \"timestamp\": \"str\",\n \"anglez\": \"float32\",\n \"enmo\": \"float32\",\n }\n\ndata = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\").astype(data_dtype, copy=False)\n```\nIt works rather fast, takes about 4-5GB of the RAM.\nThen I want to add some features, based on the timestamp, so I do this:\n\n```python\n# Convert the timestamp column to Timestamp objects\ndata[\"timestamp_datetime\"] = pd.to_datetime(data[\"timestamp\"], utc=False, cache=True, format=\"%Y-%m-%dT%H:%M:%S-%f\")\n```\n\nAnd this line works really long. Any ways to deal with it somehow? I want, for example, to add the hour of the day. I tried to load the data in the pandas first, then transfer it to the CUdf library, but it crashes due to memory error. I know that I can wait for some long time, then save preprocessed data in the pickle format. But I cannot do this with the test.", "2515371": "manwithaflower : To tell the long story short, you can improve the performance of your data processing pipelines by \n\n- doing the feature engineering in the batch mode, in a series-by-series manner\n- reading data with some of the pandas alternatives that are more powerful when it comes to reading Big Data-scale datasets (Polar, Dask etc.)\n\nYou can refer to [Best Practices in Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) to see the references to the specific examples of how it can be implemented (this thread is also useful if you like to see other tricks to improve the performance of your pipelines - for example, when you do the model training or/and prepare your submission from the test data).\n\nYou can also review a few public notebooks where per-series feature engineering / data preprocessing adds the edge in terms of performance:\n\n- https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction\n- https://www.kaggle.com/code/siddhvr/cmi-ensembling-models-for-better-results\n- etc.\n\nP. S. From the code fragment you posted in your message, I noticed you attempt to convert the timestamp values to the datetime in the local time zone. You may want to refer to https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check to see the data quality-driven issues with such an approach (at the end, casting them to UTC may be a better idea to work around).", "2515405": "Aha, here it is! Thank you very much, finally I found the nice solution!\n\nThis [notebook](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction) is really helpful. As I understood, we should load the series one by one by their id. And then we can do the feature engineering by id as well, using parallelization. Great!" }, "source": "meta" }