My PC could't handle the parquet file due to memory limitations, so I tried Kaggle notebook. And even 30GB of RAM is just enough to pd.read_parquet().
\nI converted the parquet file to CSV file and the resulting file is 8GB.
\nHave to resort to file handlers just to count rows. The train dataset is 127,946,341 rows.
How should I handle 128 million rows ? Can't even do summary statistics rn
", "messages": [ { "id": 2431244, "postDate": "2023-09-09T23:56:21.280Z", "content": "I haven't tried it yet, so just sharing an idea (something that might be a solution). If its helpful then im happy, so please share if it works.
\nIn past competitions(PSP), the amount of training data was large, so we had to carefully manage memory. I'm planning to try this one out.
Watch the great Chris @cdeotte discussion
Hey, I looked at the PSP notebook you provided and saw the pd.read_csv(path, dtype=dtypes) and was wondering if this was possible with pd.read_parquet. It seems like there is no dtype param. I guess I would have to load the parquet and then set the dtypes, but that removes a lot of the utility that loading in a reduced memory dataset would have.
", "rawMarkdown": "Hey, I looked at the PSP notebook you provided and saw the pd.read_csv(path, dtype=dtypes) and was wondering if this was possible with pd.read_parquet. It seems like there is no dtype param. I guess I would have to load the parquet and then set the dtypes, but that removes a lot of the utility that loading in a reduced memory dataset would have.", "votes": 1, "replies": [ { "id": 2432506, "postDate": "2023-09-10T22:54:45.683Z", "content": "@aleksanderklimczyk Yes. These are examples of a csv file. The memory reduction notes by @andradaolteanu will be helpful for this competition.
", "rawMarkdown": "@aleksanderklimczyk Yes. These are examples of a csv file. [The memory reduction notes](https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction) by @andradaolteanu will be helpful for this competition.", "votes": 2 } ] } ] }, { "id": 2427854, "postDate": "2023-09-07T14:04:49.630Z", "content": "My PC could't handle the parquet file due to memory limitations, so I tried Kaggle notebook. And even 30GB of RAM is just enough to pd.read_parquet().
\nI converted the parquet file to CSV file and the resulting file is 8GB.
\nHave to resort to file handlers just to count rows. The train dataset is 127,946,341 rows.
How should I handle 128 million rows ? Can't even do summary statistics rn
", "rawMarkdown": "My PC could't handle the parquet file due to memory limitations, so I tried Kaggle notebook. And even 30GB of RAM is just enough to pd.read_parquet(). \nI converted the parquet file to CSV file and the resulting file is 8GB. \nHave to resort to file handlers just to count rows. The train dataset is 127,946,341 rows. \n\nHow should I handle 128 million rows ? Can't even do summary statistics rn", "votes": 7 }, { "id": 2428139, "postDate": "2023-09-07T17:02:14.343Z", "content": "Consider learning to use the polars package. There's a little bit of a learning curve but it is a fast, memory efficient tool that is ideal for handling large parquet files.
", "rawMarkdown": "Consider learning to use the [polars](https://www.pola.rs/) package. There's a little bit of a learning curve but it is a fast, memory efficient tool that is ideal for handling large parquet files. ", "votes": 3, "replies": [ { "id": 2428468, "postDate": "2023-09-07T22:29:29.723Z", "content": "It's helpful, but still can be too time-consuming when trying out simple tasks.
", "rawMarkdown": "It's helpful, but still can be too time-consuming when trying out simple tasks.", "votes": 1 } ] }, { "id": 2427996, "postDate": "2023-09-07T15:32:07.157Z", "content": "Dear @dietzschenostoevsky
\nIf it at all helps with getting started I have made a smaller dataset consisting of 13M rows (180M parquet file) which can be found here.
\nAll the best,\ncarl
", "rawMarkdown": "Dear @dietzschenostoevsky \n\nIf it at all helps with getting started I have made a smaller dataset consisting of 13M rows (180M parquet file) which can be found [here](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target).\n\nAll the best,\ncarl", "votes": 3, "replies": [ { "id": 2429965, "postDate": "2023-09-09T00:14:47.807Z", "content": "Thank you. I will try to see if my machine could handle it. Do you mind sharing how did you created it ? Because I am thinking about upgrading my PC and I am not sure how much RAM I would need to handle such files in future
", "rawMarkdown": "Thank you. I will try to see if my machine could handle it. Do you mind sharing how did you created it ? Because I am thinking about upgrading my PC and I am not sure how much RAM I would need to handle such files in future", "votes": 1, "replies": [ { "id": 2430326, "postDate": "2023-09-09T08:02:34Z", "content": "Dear @dietzschenostoevsky
\nThe notebook to create the dataset is here: \"Zzzs: Make clean starter dataset + target\".
\nHope this helps to get started!
\ncarl
Your contributions really make sense, many thanks!
", "rawMarkdown": "Your contributions really make sense, many thanks!" } ] }, { "id": 2430320, "postDate": "2023-09-09T07:58:27.980Z", "content": "Thanks again for your highly valued contributions. This helps a lot.
", "rawMarkdown": "Thanks again for your highly valued contributions. This helps a lot. " } ] }, { "id": 2431113, "postDate": "2023-09-09T19:38:17.720Z", "content": "You can try using dask or dask_cudf for handling the data
", "rawMarkdown": "You can try using dask or dask_cudf for handling the data\n", "votes": 1 }, { "id": 2430319, "postDate": "2023-09-09T07:55:35.957Z", "content": "This right here is a key limitation for those of us on a quarter of a shoestring budget. It is really difficult and frustrating to download and properly analyze the entire data to enable full feature engineering in my own case. Oh well, we will keep sampling and extrapolating. Something might stick on the wall eventually.
", "rawMarkdown": "This right here is a key limitation for those of us on a quarter of a shoestring budget. It is really difficult and frustrating to download and properly analyze the entire data to enable full feature engineering in my own case. Oh well, we will keep sampling and extrapolating. Something might stick on the wall eventually. ", "votes": 1 }, { "id": 2430323, "postDate": "2023-09-09T08:00:12.297Z", "content": "Has anyone tried brotli compression on the parquet file?
\nhttps://stackoverflow.com/questions/33813815/how-to-read-a-parquet-file-into-pandas-dataframe
", "rawMarkdown": "Has anyone tried brotli compression on the parquet file?\n\nhttps://stackoverflow.com/questions/33813815/how-to-read-a-parquet-file-into-pandas-dataframe" }, { "id": 2429986, "postDate": "2023-09-09T01:12:32.200Z", "content": "I have found a way to read parquet files a few lines at a time, using pyarrow.parquet. You can see it here: https://www.kaggle.com/code/nadereafshar/detect-sleep-states/edit
The link is not working
", "rawMarkdown": "The link is not working " }, { "id": 2432189, "postDate": "2023-09-10T16:48:48.390Z", "content": "You can load the data for one series via
\nimport pyarrow.parquet pq\n\nseries_id = 'f9e60a8e56d'\n\npq.read]).\nThe amount of chatGPT answers these days is overwhelming
", "rawMarkdown": "The amount of chatGPT answers these days is overwhelming", "votes": 5, "replies": [ { "id": 2430336, "postDate": "2023-09-09T08:11:37.647Z", "rawMarkdown": "", "votes": -5, "isDeleted": true } ] } ] } ], "comments": [ { "id": 2431244, "author_name": "mh", "author_url": "", "post_date": "2023-09-09T23:56:21.280000", "content": "I haven't tried it yet, so just sharing an idea (something that might be a solution). If its helpful then im happy, so please share if it works.
\nIn past competitions(PSP), the amount of training data was large, so we had to carefully manage memory. I'm planning to try this one out.
Watch the great Chris @cdeotte discussion
Hey, I looked at the PSP notebook you provided and saw the pd.read_csv(path, dtype=dtypes) and was wondering if this was possible with pd.read_parquet. It seems like there is no dtype param. I guess I would have to load the parquet and then set the dtypes, but that removes a lot of the utility that loading in a reduced memory dataset would have.
", "votes": 1, "replies": [ { "id": 2432506, "author_name": "mh", "author_url": "", "post_date": "2023-09-10T22:54:45.683000", "content": "@aleksanderklimczyk Yes. These are examples of a csv file. The memory reduction notes by @andradaolteanu will be helpful for this competition.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2428139, "author_name": "Quillen", "author_url": "", "post_date": "2023-09-07T17:02:14.343000", "content": "Consider learning to use the polars package. There's a little bit of a learning curve but it is a fast, memory efficient tool that is ideal for handling large parquet files.
", "votes": 3, "replies": [ { "id": 2428468, "author_name": "Yan Teixeira", "author_url": "", "post_date": "2023-09-07T22:29:29.723000", "content": "It's helpful, but still can be too time-consuming when trying out simple tasks.
", "votes": 1, "replies": [] } ] }, { "id": 2427996, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-07T15:32:07.157000", "content": "Dear @dietzschenostoevsky
\nIf it at all helps with getting started I have made a smaller dataset consisting of 13M rows (180M parquet file) which can be found here.
\nAll the best,\ncarl
", "votes": 3, "replies": [ { "id": 2429965, "author_name": "Dietzsche Nostoevsky", "author_url": "", "post_date": "2023-09-09T00:14:47.807000", "content": "Thank you. I will try to see if my machine could handle it. Do you mind sharing how did you created it ? Because I am thinking about upgrading my PC and I am not sure how much RAM I would need to handle such files in future
", "votes": 1, "replies": [ { "id": 2430326, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-09T08:02:34", "content": "Dear @dietzschenostoevsky
\nThe notebook to create the dataset is here: \"Zzzs: Make clean starter dataset + target\".
\nHope this helps to get started!
\ncarl
Your contributions really make sense, many thanks!
", "votes": 0, "replies": [] } ] }, { "id": 2430320, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-09T07:58:27.980000", "content": "Thanks again for your highly valued contributions. This helps a lot.
", "votes": 0, "replies": [] } ] }, { "id": 2431113, "author_name": "Dan Pietrow", "author_url": "", "post_date": "2023-09-09T19:38:17.720000", "content": "You can try using dask or dask_cudf for handling the data
", "votes": 1, "replies": [] }, { "id": 2430319, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-09T07:55:35.957000", "content": "This right here is a key limitation for those of us on a quarter of a shoestring budget. It is really difficult and frustrating to download and properly analyze the entire data to enable full feature engineering in my own case. Oh well, we will keep sampling and extrapolating. Something might stick on the wall eventually.
", "votes": 1, "replies": [] }, { "id": 2430323, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-09T08:00:12.297000", "content": "Has anyone tried brotli compression on the parquet file?
\nhttps://stackoverflow.com/questions/33813815/how-to-read-a-parquet-file-into-pandas-dataframe
", "votes": 0, "replies": [] }, { "id": 2429986, "author_name": "Nader E Afshar", "author_url": "", "post_date": "2023-09-09T01:12:32.200000", "content": "I have found a way to read parquet files a few lines at a time, using pyarrow.parquet. You can see it here: https://www.kaggle.com/code/nadereafshar/detect-sleep-states/edit
The link is not working
", "votes": 0, "replies": [] }, { "id": 2432189, "author_name": "eray", "author_url": "", "post_date": "2023-09-10T16:48:48.390000", "content": "You can load the data for one series via
\nimport pyarrow.parquet pq\n\nseries_id = 'f9e60a8e56d'\n\npq.read]).\nThe amount of chatGPT answers these days is overwhelming
", "votes": 5, "replies": [ { "id": 2430336, "author_name": "", "author_url": "", "post_date": "2023-09-09T08:11:37.647000", "content": "", "votes": -5, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2431244": "I haven't tried it yet, so just sharing an idea (something that might be a solution). If its helpful then im happy, so please share if it works.\n\n- In past competitions([PSP](https://www.kaggle.com/competitions/predict-student-performance-from-game-play)), the amount of training data was large, so we had to carefully manage memory. I'm planning to try [this](https://www.kaggle.com/code/gusthema/student-performance-w-tensorflow-decision-forests) one out.\n \n- Watch the great Chris @cdeotte discussion\n \n1. [How To Reduce Data Size](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054)\n2. [Memory Trick - Reduce Memory 8x or 16x!](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/308635)\n3. [How To Get 32GB RAM - Avoid Memory Error!](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396979)\n", "2427854": "My PC could't handle the parquet file due to memory limitations, so I tried Kaggle notebook. And even 30GB of RAM is just enough to pd.read_parquet(). \nI converted the parquet file to CSV file and the resulting file is 8GB. \nHave to resort to file handlers just to count rows. The train dataset is 127,946,341 rows. \n\nHow should I handle 128 million rows ? Can't even do summary statistics rn", "2428139": "Consider learning to use the [polars](https://www.pola.rs/) package. There's a little bit of a learning curve but it is a fast, memory efficient tool that is ideal for handling large parquet files. ", "2427996": "Dear @dietzschenostoevsky \n\nIf it at all helps with getting started I have made a smaller dataset consisting of 13M rows (180M parquet file) which can be found [here](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target).\n\nAll the best,\ncarl", "2431113": "You can try using dask or dask_cudf for handling the data\n", "2430319": "This right here is a key limitation for those of us on a quarter of a shoestring budget. It is really difficult and frustrating to download and properly analyze the entire data to enable full feature engineering in my own case. Oh well, we will keep sampling and extrapolating. Something might stick on the wall eventually. ", "2430323": "Has anyone tried brotli compression on the parquet file?\n\nhttps://stackoverflow.com/questions/33813815/how-to-read-a-parquet-file-into-pandas-dataframe", "2429986": "I have found a way to read parquet files a few lines at a time, using `pyarrow.parquet`. You can see it here: [https://www.kaggle.com/code/nadereafshar/detect-sleep-states/edit](url)", "2427937": "" } }