Is there any recommendation on how to get around running out of memory on Kaggle? I have garbage collected as much as possible, but when trying to generate training data with about 100,000 rows, I run out of memory.
\nI started with 500,000 rows, thinking a database with only 16 columns would not be that large, especially when most data are int16 making the entire data about 12MB. I thought that would be more than manageable, but I am running out of memory even when I try 100,000 rows.
\nAny ideas?
", "messages": [ { "id": "2451755", "postDate": "09/22/2023 18:42:38", "content": "Is there any recommendation on how to get around running out of memory on Kaggle? I have garbage collected as much as possible, but when trying to generate training data with about 100,000 rows, I run out of memory.
\nI started with 500,000 rows, thinking a database with only 16 columns would not be that large, especially when most data are int16 making the entire data about 12MB. I thought that would be more than manageable, but I am running out of memory even when I try 100,000 rows.
\nAny ideas?
", "rawMarkdown": "Is there any recommendation on how to get around running out of memory on Kaggle? I have garbage collected as much as possible, but when trying to generate training data with about 100,000 rows, I run out of memory.\n\nI started with 500,000 rows, thinking a database with only 16 columns would not be that large, especially when most data are int16 making the entire data about 12MB. I thought that would be more than manageable, but I am running out of memory even when I try 100,000 rows.\n\nAny ideas?", "votes": null }, { "id": "2451791", "postDate": "09/22/2023 19:24:01", "content": "\n\nwhen trying to generate training data with about 100,000 rows, I run out of memory.
\n
Can you be more specific or share a code snippet? There are fast, efficient ways of processing the training data, but specific advice depends on what exactly you're trying to do.
", "rawMarkdown": "> when trying to generate training data with about 100,000 rows, I run out of memory.\n\nCan you be more specific or share a code snippet? There are fast, efficient ways of processing the training data, but specific advice depends on what exactly you're trying to do.", "votes": null }, { "id": "2451808", "postDate": "09/22/2023 19:48:40", "content": "@quillen - I tried to execute the code written by Lucas Burke which you can find here I got as far as this cell when I keep running out of memory. I have decreased the number of rows to as small as 100,000 but still running out of memory.
", "rawMarkdown": "quillen - I tried to execute the code written by Lucas Burke which you can find [here](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction) I got as far as this [cell](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction?scriptVersionId=143599515&cellId=8) when I keep running out of memory. I have decreased the number of rows to as small as 100,000 but still running out of memory.", "votes": null }, { "id": "2451856", "postDate": "09/22/2023 20:53:45", "content": "I just ran that script and it used a peak of 16GB RAM. Are you using a GPU? Because the kaggle kernels without an accelerator should have 30GB.
\nI think the following block is the bottleneck:
\ntrain_data = (train_series(pl()(series_ids))\n ()\n ((e6))\n )\n\nX_train, y_train = (train_data, train_events)\nWhen passing train_data to make_train_dataset() memory usage spikes because .collect() is called before .sample().
You could decrease the memory requirement by using a lazyframe until you need to collect the rows (rather than passing the entire dataframe to make_train_dataset()).
@quillen - That is a brilliant observation and your suggestion makes intuitive sense. I am actually not familiar with the Polars package which is why I was using this code as a learning step. Would you please elaborate on how to modify the use of lazyframe in the specific way you suggested? Is this process as simple as eliminating the .collect() in this cell?
\nOn a second note, how come you were able to run this code, yet I ran out of memory. Are we not all using the same kaggle resources. Especially since I am using \"GPU T4 x2\" as accelerator. I may also add that I hardly aver see the GPU kick in. Its always, and only, the CPU that is busy.
\nThank you
", "rawMarkdown": "quillen - That is a brilliant observation and your suggestion makes intuitive sense. I am actually not familiar with the Polars package which is why I was using this code as a learning step. Would you please elaborate on how to modify the use of lazyframe in the specific way you suggested? Is this process as simple as eliminating the .collect() in this cell?\n\nOn a second note, how come you were able to run this code, yet I ran out of memory. Are we not all using the same kaggle resources. Especially since I am using \"GPU T4 x2\" as accelerator. I may also add that I hardly aver see the GPU kick in. Its always, and only, the CPU that is busy.\n\nThank you", "votes": null }, { "id": "2452275", "postDate": "09/23/2023 07:36:53", "content": "If you do select a GPU as an accelerator kaggle will limit other resources (the CPU an RAM). If you do not use the GPU explicitly (e.g. PyTorch or Tensorflow), then simple disable the accelerator.
", "rawMarkdown": "If you do select a GPU as an accelerator kaggle will limit other resources (the CPU an RAM). If you do not use the GPU explicitly (e.g. PyTorch or Tensorflow), then simple disable the accelerator.", "votes": null }, { "id": "2452574", "postDate": "09/23/2023 12:12:23", "content": "@nadereafshar - as mentioned by @patrickaiforfun, you can probably run the notebook without modifications if you select None for your accelerator… but I'll still say a bit about Polars purely for the sake of learning.
\nBrief background: When you make a call to transform, filter, group, etc. on a DataFrame, those requests are handled sequentially on all of the applicable data regardless of which subset of rows you're going to ultimately use in your sample.
\nOne reason that Polars is so efficient is because you can use \"lazy\" queries with a LazyFrame (LF). LFs don't execute any calls, they are simply query plan which is automatically optimized. The optimized query is executed once the LF is converted into a DataFrame (.collect()). Ideally, LFs can avoid loading and processing data that you don't actually care about.
Back to this notebook: .sample() cannot be called on a LF, so you have to call .collect() beforehand. Fortunately, the size of the DataFrame can be reduced by filtering the data before computing rolling_mean() and rolling_max().
This notebook requests that the rolling features be computed across the entire column. So the last rows of one series are involved in the calculations for the first rows of the next series. I'm not sure whether this behavior is desired…
\nCalling .filter() after computing the rolling mean/max has no effect on peak memory usage because rolling mean/max rely upon some values from the filtered series. So the filter can't take effect until the calculation is completed.
If we call .filter() before requesting new feature columns, the LF will understand that we want to exclude those rows from all calculations and it won't load those series at all.
So we can replace:
\ntrain_data = (train_series\n (pl()(series_ids)) \n ()\n ((e6))\n )\nX_train, y_train = (train_data, train_events)\n
\nwith:
train_data = (train_series\n .collect\n .sample(())\n )\nX_train, y_train = make\nand at the beginning of the notebook, we can replace
\ntrain_series = pl(TRAIN_PATH)(\n dt_transforms + data_transforms\n )\n\ntrain_series = train_series(~pl()(count_mismatches))\nwith
\ntrain_series = (.scan_parquet()\n .filter((.col('series_id').is_in()))\n .with_columns( + data_transforms)\n .filter(~pl.col('series_id').is_in(['series_id'])))\nYou can filter more aggressively to further decrease the memory usage. For example, updating the filter to sample from the first 10 days of each series with .filter((pl.col('series_id').is_in(series_ids)) & (pl.col(\"step\")<=172800)) keeps peak memory <7GB
@patrickaiforfun - That is such a valuable advice!! Thank you.
", "rawMarkdown": "patrickaiforfun - That is such a valuable advice!! Thank you.", "votes": null }, { "id": "2452748", "postDate": "09/23/2023 14:47:37", "content": "@quillen - Thank you so much for this quick and valuable lesson. You clearly have a phenomenal grasp of the polars package. I will try and find some in-depth tutorial about this package and see if I can gain some of the intricate insight you posses.
", "rawMarkdown": "quillen - Thank you so much for this quick and valuable lesson. You clearly have a phenomenal grasp of the polars package. I will try and find some in-depth tutorial about this package and see if I can gain some of the intricate insight you posses.", "votes": null } ], "comments": [ { "id": 2451791, "author_name": "quillen", "author_url": "", "post_date": "09/22/2023 19:24:01", "content": "\n\nwhen trying to generate training data with about 100,000 rows, I run out of memory.
\n
Can you be more specific or share a code snippet? There are fast, efficient ways of processing the training data, but specific advice depends on what exactly you're trying to do.
", "votes": null, "replies": [ { "id": 2451808, "author_name": "nadereafshar", "author_url": "", "post_date": "09/22/2023 19:48:40", "content": "@quillen - I tried to execute the code written by Lucas Burke which you can find here I got as far as this cell when I keep running out of memory. I have decreased the number of rows to as small as 100,000 but still running out of memory.
", "votes": null, "replies": [ { "id": 2451856, "author_name": "quillen", "author_url": "", "post_date": "09/22/2023 20:53:45", "content": "I just ran that script and it used a peak of 16GB RAM. Are you using a GPU? Because the kaggle kernels without an accelerator should have 30GB.
\nI think the following block is the bottleneck:
\ntrain_data = (train_series(pl()(series_ids))\n ()\n ((e6))\n )\n\nX_train, y_train = (train_data, train_events)\nWhen passing train_data to make_train_dataset() memory usage spikes because .collect() is called before .sample().
You could decrease the memory requirement by using a lazyframe until you need to collect the rows (rather than passing the entire dataframe to make_train_dataset()).
@quillen - That is a brilliant observation and your suggestion makes intuitive sense. I am actually not familiar with the Polars package which is why I was using this code as a learning step. Would you please elaborate on how to modify the use of lazyframe in the specific way you suggested? Is this process as simple as eliminating the .collect() in this cell?
\nOn a second note, how come you were able to run this code, yet I ran out of memory. Are we not all using the same kaggle resources. Especially since I am using \"GPU T4 x2\" as accelerator. I may also add that I hardly aver see the GPU kick in. Its always, and only, the CPU that is busy.
\nThank you
", "votes": null, "replies": [ { "id": 2452275, "author_name": "patrickaiforfun", "author_url": "", "post_date": "09/23/2023 07:36:53", "content": "If you do select a GPU as an accelerator kaggle will limit other resources (the CPU an RAM). If you do not use the GPU explicitly (e.g. PyTorch or Tensorflow), then simple disable the accelerator.
", "votes": null, "replies": [ { "id": 2452734, "author_name": "nadereafshar", "author_url": "", "post_date": "09/23/2023 14:36:33", "content": "@patrickaiforfun - That is such a valuable advice!! Thank you.
", "votes": null, "replies": [] } ] }, { "id": 2452574, "author_name": "quillen", "author_url": "", "post_date": "09/23/2023 12:12:23", "content": "@nadereafshar - as mentioned by @patrickaiforfun, you can probably run the notebook without modifications if you select None for your accelerator… but I'll still say a bit about Polars purely for the sake of learning.
\nBrief background: When you make a call to transform, filter, group, etc. on a DataFrame, those requests are handled sequentially on all of the applicable data regardless of which subset of rows you're going to ultimately use in your sample.
\nOne reason that Polars is so efficient is because you can use \"lazy\" queries with a LazyFrame (LF). LFs don't execute any calls, they are simply query plan which is automatically optimized. The optimized query is executed once the LF is converted into a DataFrame (.collect()). Ideally, LFs can avoid loading and processing data that you don't actually care about.
Back to this notebook: .sample() cannot be called on a LF, so you have to call .collect() beforehand. Fortunately, the size of the DataFrame can be reduced by filtering the data before computing rolling_mean() and rolling_max().
This notebook requests that the rolling features be computed across the entire column. So the last rows of one series are involved in the calculations for the first rows of the next series. I'm not sure whether this behavior is desired…
\nCalling .filter() after computing the rolling mean/max has no effect on peak memory usage because rolling mean/max rely upon some values from the filtered series. So the filter can't take effect until the calculation is completed.
If we call .filter() before requesting new feature columns, the LF will understand that we want to exclude those rows from all calculations and it won't load those series at all.
So we can replace:
\ntrain_data = (train_series\n (pl()(series_ids)) \n ()\n ((e6))\n )\nX_train, y_train = (train_data, train_events)\n
\nwith:
train_data = (train_series\n .collect\n .sample(())\n )\nX_train, y_train = make\nand at the beginning of the notebook, we can replace
\ntrain_series = pl(TRAIN_PATH)(\n dt_transforms + data_transforms\n )\n\ntrain_series = train_series(~pl()(count_mismatches))\nwith
\ntrain_series = (.scan_parquet()\n .filter((.col('series_id').is_in()))\n .with_columns( + data_transforms)\n .filter(~pl.col('series_id').is_in(['series_id'])))\nYou can filter more aggressively to further decrease the memory usage. For example, updating the filter to sample from the first 10 days of each series with .filter((pl.col('series_id').is_in(series_ids)) & (pl.col(\"step\")<=172800)) keeps peak memory <7GB
@quillen - Thank you so much for this quick and valuable lesson. You clearly have a phenomenal grasp of the polars package. I will try and find some in-depth tutorial about this package and see if I can gain some of the intricate insight you posses.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2451755": "Is there any recommendation on how to get around running out of memory on Kaggle? I have garbage collected as much as possible, but when trying to generate training data with about 100,000 rows, I run out of memory.\n\nI started with 500,000 rows, thinking a database with only 16 columns would not be that large, especially when most data are int16 making the entire data about 12MB. I thought that would be more than manageable, but I am running out of memory even when I try 100,000 rows.\n\nAny ideas?", "2451791": "> when trying to generate training data with about 100,000 rows, I run out of memory.\n\nCan you be more specific or share a code snippet? There are fast, efficient ways of processing the training data, but specific advice depends on what exactly you're trying to do.", "2451808": "quillen - I tried to execute the code written by Lucas Burke which you can find [here](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction) I got as far as this [cell](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction?scriptVersionId=143599515&cellId=8) when I keep running out of memory. I have decreased the number of rows to as small as 100,000 but still running out of memory.", "2451856": "I just ran that script and it used a peak of 16GB RAM. Are you using a GPU? Because the kaggle kernels without an accelerator should have 30GB. \n\nI think the following block is the bottleneck:\n \n```\ntrain_data = (train_series.filter(pl.col('series_id').is_in(series_ids))\n .collect()\n .sample(int(1e6))\n )\n\nX_train, y_train = make_train_dataset(train_data, train_events)\n```\nWhen passing `train_data` to `make_train_dataset()` memory usage spikes because `.collect()` is called before `.sample()`. \n\nYou could decrease the memory requirement by using a lazyframe until you need to collect the rows (rather than passing the entire dataframe to `make_train_dataset()`).", "2452014": "quillen - That is a brilliant observation and your suggestion makes intuitive sense. I am actually not familiar with the Polars package which is why I was using this code as a learning step. Would you please elaborate on how to modify the use of lazyframe in the specific way you suggested? Is this process as simple as eliminating the .collect() in this cell?\n\nOn a second note, how come you were able to run this code, yet I ran out of memory. Are we not all using the same kaggle resources. Especially since I am using \"GPU T4 x2\" as accelerator. I may also add that I hardly aver see the GPU kick in. Its always, and only, the CPU that is busy.\n\nThank you", "2452275": "If you do select a GPU as an accelerator kaggle will limit other resources (the CPU an RAM). If you do not use the GPU explicitly (e.g. PyTorch or Tensorflow), then simple disable the accelerator.", "2452574": "nadereafshar - as mentioned by @patrickaiforfun, you can probably run the notebook without modifications if you select None for your accelerator... but I'll still say a bit about Polars purely for the sake of learning. \n\n**Brief background:** When you make a call to transform, filter, group, etc. on a DataFrame, those requests are handled sequentially on all of the applicable data regardless of which subset of rows you're going to ultimately use in your sample.\n\nOne reason that Polars is so efficient is because you can use \"lazy\" queries with a LazyFrame (LF). LFs *don't* execute any calls, they are simply query *plan* which is automatically optimized. The optimized query is executed once the LF is converted into a DataFrame (`.collect()`). Ideally, LFs can avoid loading and processing data that you don't actually care about.\n\n**Back to this notebook:** `.sample()` cannot be called on a LF, so you have to call `.collect()` beforehand. Fortunately, the size of the DataFrame can be reduced by filtering the data *before* computing `rolling_mean()` and `rolling_max()`. \n\nThis notebook requests that the rolling features be computed across the entire column. So the last rows of one series are involved in the calculations for the first rows of the next series. **I'm not sure whether this behavior is desired...**\n\nCalling `.filter()` *after* computing the rolling mean/max has no effect on peak memory usage because rolling mean/max rely upon some values from the filtered series. So the filter can't take effect until the calculation is completed. \n\nIf we call `.filter()` *before* requesting new feature columns, the LF will understand that we want to exclude those rows from all calculations and it won't load those series at all.\n\nSo we can replace:\n```\ntrain_data = (train_series\n .filter(pl.col('series_id').is_in(series_ids)) \n .collect()\n .sample(int(1e6))\n )\nX_train, y_train = make_train_dataset(train_data, train_events)\n``` \nwith:\n```\ntrain_data = (train_series\n .collect()\n .sample(int(1e6))\n )\nX_train, y_train = make_train_dataset(train_data, train_events)\n```\nand at the beginning of the notebook, we can replace\n```\ntrain_series = pl.scan_parquet(TRAIN_PATH).with_columns(\n dt_transforms + data_transforms\n )\n\ntrain_series = train_series.filter(~pl.col('series_id').is_in(count_mismatches['series_id']))\n```\nwith\n```\ntrain_series = (pl.scan_parquet(TRAIN_PATH)\n .filter((pl.col('series_id').is_in(series_ids)))\n .with_columns(dt_transforms + data_transforms)\n .filter(~pl.col('series_id').is_in(count_mismatches['series_id'])))\n```\n\nYou can filter more aggressively to further decrease the memory usage. For example, updating the filter to sample from the first 10 days of each series with `.filter((pl.col('series_id').is_in(series_ids)) & (pl.col(\"step\")<=172800))` keeps peak memory <7GB", "2452734": "patrickaiforfun - That is such a valuable advice!! Thank you.", "2452748": "quillen - Thank you so much for this quick and valuable lesson. You clearly have a phenomenal grasp of the polars package. I will try and find some in-depth tutorial about this package and see if I can gain some of the intricate insight you posses." }, "source": "meta" }