{ "id": 443274, "title": "RAM out of memory error? Why? ", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/443274", "author_name": "", "post_date": "2023-09-26T10:10:05.025458700Z", "votes": 2, "comment_count": 13, "views": 0, "content": "

kaggle kernels pull alikarbala/sleep-detection-submission-with-keras

\n

Help me… Why this notebook is raising RAM out of memory error

", "messages": [ { "id": "2456606", "postDate": "09/26/2023 10:10:05", "content": "

kaggle kernels pull alikarbala/sleep-detection-submission-with-keras

\n

Help me… Why this notebook is raising RAM out of memory error

", "rawMarkdown": "kaggle kernels pull alikarbala/sleep-detection-submission-with-keras\n\nHelp me... Why this notebook is raising RAM out of memory error", "votes": null }, { "id": "2457093", "postDate": "09/26/2023 15:59:54", "content": "

Hi @alikarbala, you maybe doing data manipulation or model fitting that does not fit into the memory. Consider

\n
    \n
  1. Using optimal datatypes whenever possible
    2. \n
  2. If you are using GPU and full dataset, consider using light weight dataset
    3. \n
  3. Consider training and saving model in a kernel and then submit only inference notebook
    4. \n
  4. If you still face issue after submitting only inference, try batchwise predictions
    \nYou can refer public kernels for datasets and batch predictions
    5. \n
", "rawMarkdown": "Hi @alikarbala, you maybe doing data manipulation or model fitting that does not fit into the memory. Consider \n1. Using optimal datatypes whenever possible\n2. If you are using GPU and full dataset, consider using light weight dataset\n3. Consider training and saving model in a kernel and then submit only inference notebook\n4. If you still face issue after submitting only inference, try batchwise predictions\nYou can refer public kernels for datasets and batch predictions", "votes": null }, { "id": "2457819", "postDate": "09/27/2023 07:15:55", "content": "

@alikarbala : Hey Ali, you may want to review the parallel discussion thread per Best Practices in Tackling BigData-Scale Tasks to see the tricks to optimize the RAM/CPU utilization for the dataset of this contest.

\n

I hope it is helpful.

", "rawMarkdown": "alikarbala : Hey Ali, you may want to review the parallel discussion thread per [Best Practices in Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) to see the tricks to optimize the RAM/CPU utilization for the dataset of this contest.\n\nI hope it is helpful.", "votes": null }, { "id": "2459369", "postDate": "09/28/2023 07:16:21", "content": "

In addition to the helpful suggestions already given, you need to perform garbage collection and delete some large data you don't need before adding more data.

\n

Also, keep an eye out for the usage meter on the top right of the Notebook pane when you use CPU, GPU or TPU. You will see the RAM bar turning red (reaching or exceeding full capacity), which helps you know how your code is running on the available resources. When your code is running okay, the RAM bar would be under half and green in colour.

", "rawMarkdown": "In addition to the helpful suggestions already given, you need to perform garbage collection and delete some large data you don't need before adding more data. \n\nAlso, keep an eye out for the usage meter on the top right of the Notebook pane when you use CPU, GPU or TPU. You will see the RAM bar turning red (reaching or exceeding full capacity), which helps you know how your code is running on the available resources. When your code is running okay, the RAM bar would be under half and green in colour.", "votes": null }, { "id": "2459579", "postDate": "09/28/2023 09:32:52", "content": "

Thanks for all helpful suggestions… I tried all the ways and didnt work… The notebook is running succesfully without RAM turning red I check it before submitting, i did gc.collect and batchwise predictions, even i submit the inference on another notebook… Non of these work thats why i ask for help…

", "rawMarkdown": "Thanks for all helpful suggestions... I tried all the ways and didnt work... The notebook is running succesfully without RAM turning red I check it before submitting, i did gc.collect and batchwise predictions, even i submit the inference on another notebook... Non of these work thats why i ask for help...", "votes": null }, { "id": "2459613", "postDate": "09/28/2023 09:53:37", "content": "

How are you generating your batches?

\n

From the notebook you linked, it looks like you're loading the entire test_data.parquet file, transforming all of it with a single make_features() call, and then passing the whole thing to your model.

", "rawMarkdown": "How are you generating your batches? \n\nFrom the notebook you linked, it looks like you're loading the entire `test_data.parquet` file, transforming all of it with a single `make_features()` call, and then passing the whole thing to your model.", "votes": null }, { "id": "2459953", "postDate": "09/28/2023 13:42:33", "content": "

test = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet\")

\n

Preprocess the test data to create X_test

\n

test = make_features(test)
\nX_test = test[features]

\n

Define batch size for predictions

\n

batch_size = 16

\n

predictions = []

\n

Iterate through the test data and make batch predictions

\n

for i in range(0, len(X_test), batch_size):
\n batch = X_test.iloc[i:i + batch_size]
\n batch_predictions = model.predict(batch)
\n predictions.extend(batch_predictions)

\n

Update the test DataFrame with the predictions

\n

test[\"not_awake\"] = predictions
\ntest[\"awake\"] = predictions

\n

i did this in previous version.. It takes a long time in scoring thought that it works and after that it crashes.. RAM issue

", "rawMarkdown": "test = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet\")\n\n# Preprocess the test data to create X_test\ntest = make_features(test)\nX_test = test[features]\n\n# Define batch size for predictions\nbatch_size = 16\n\n\npredictions = []\n\n# Iterate through the test data and make batch predictions\nfor i in range(0, len(X_test), batch_size):\n batch = X_test.iloc[i:i + batch_size]\n batch_predictions = model.predict(batch)\n predictions.extend(batch_predictions)\n\n# Update the test DataFrame with the predictions\ntest[\"not_awake\"] = predictions\ntest[\"awake\"] = predictions\n\ni did this in previous version.. It takes a long time in scoring thought that it works and after that it crashes.. RAM issue", "votes": null }, { "id": "2460026", "postDate": "09/28/2023 14:32:06", "content": "

The hidden test dataset is ~200 full series, so the train series is a decent stand-in for diagnosing memory problems. Change test = pd.read_parquet(f\"{DIR}/test_series.parquet\") to load train_series.parquet instead and you'll see that a notebook with 30GB RAM crashes before it is able to use the model.

\n

Try looping through each series id one at a time. Load, transform and predict for that single series before moving on to the next one. You can make a submission dataframe for a single series on each pass, then concatenate them all together before saving it as a csv.

", "rawMarkdown": "The hidden test dataset is ~200 full series, so the train series is a decent stand-in for diagnosing memory problems. Change `test = pd.read_parquet(f\"{DIR}/test_series.parquet\")` to load `train_series.parquet` instead and you'll see that a notebook with 30GB RAM crashes before it is able to use the model.\n\nTry looping through each series id one at a time. Load, transform and predict for that single series before moving on to the next one. You can make a submission dataframe for a single series on each pass, then concatenate them all together before saving it as a csv.", "votes": null }, { "id": "2460077", "postDate": "09/28/2023 15:03:10", "content": "

I'll try my best.. Thanks a lot

", "rawMarkdown": "I'll try my best.. Thanks a lot", "votes": null }, { "id": "2463877", "postDate": "10/01/2023 16:29:43", "content": "

Hi @alikarbala.

\n

I was getting the same RAM issue and I resolved using the solution explained here.

\n

Basically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?

\n

Thanks to @abhranta for give me the solution.

\n

Best Regards!

", "rawMarkdown": "Hi @alikarbala.\n\nI was getting the same RAM issue and I resolved using the solution explained [here](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set).\n\nBasically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?\n\nThanks to @abhranta for give me the solution.\n\nBest Regards!", "votes": null }, { "id": "2463915", "postDate": "10/01/2023 17:16:43", "content": "

Thanks for your response..
\nActually I was training my keras model and I performed all previous suggestion from kagglers responded to my question.
\nBut the submission didnt work until I removed the hidden layers and the model is basic with low score..
\nI decide to change my model.
\nI will check your link and reach back.

", "rawMarkdown": "Thanks for your response..\nActually I was training my keras model and I performed all previous suggestion from kagglers responded to my question.\nBut the submission didnt work until I removed the hidden layers and the model is basic with low score..\nI decide to change my model.\nI will check your link and reach back.", "votes": null }, { "id": "2464424", "postDate": "10/02/2023 08:58:11", "content": "

Also try dask or polars libraries for reading parquet files. They seem to be faster. However, there may be some issues when trying to use pandas operations on the imported data so may need to study the documentation of these libraries.

", "rawMarkdown": "Also try dask or polars libraries for reading parquet files. They seem to be faster. However, there may be some issues when trying to use pandas operations on the imported data so may need to study the documentation of these libraries.", "votes": null }, { "id": "2464564", "postDate": "10/02/2023 10:49:54", "content": "

It works :) thanks @lvarochapresto

", "rawMarkdown": "It works :) thanks @lvarochapresto", "votes": null }, { "id": "2464573", "postDate": "10/02/2023 11:05:48", "content": "

Great!! Best Regards!

", "rawMarkdown": "Great!! Best Regards!", "votes": null } ], "comments": [ { "id": 2457093, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/26/2023 15:59:54", "content": "

Hi @alikarbala, you maybe doing data manipulation or model fitting that does not fit into the memory. Consider

\n
    \n
  1. Using optimal datatypes whenever possible
    2. \n
  2. If you are using GPU and full dataset, consider using light weight dataset
    3. \n
  3. Consider training and saving model in a kernel and then submit only inference notebook
    4. \n
  4. If you still face issue after submitting only inference, try batchwise predictions
    \nYou can refer public kernels for datasets and batch predictions
    5. \n
", "votes": null, "replies": [] }, { "id": 2457819, "author_name": "gvyshnya", "author_url": "", "post_date": "09/27/2023 07:15:55", "content": "

@alikarbala : Hey Ali, you may want to review the parallel discussion thread per Best Practices in Tackling BigData-Scale Tasks to see the tricks to optimize the RAM/CPU utilization for the dataset of this contest.

\n

I hope it is helpful.

", "votes": null, "replies": [] }, { "id": 2459369, "author_name": "aidlre001", "author_url": "", "post_date": "09/28/2023 07:16:21", "content": "

In addition to the helpful suggestions already given, you need to perform garbage collection and delete some large data you don't need before adding more data.

\n

Also, keep an eye out for the usage meter on the top right of the Notebook pane when you use CPU, GPU or TPU. You will see the RAM bar turning red (reaching or exceeding full capacity), which helps you know how your code is running on the available resources. When your code is running okay, the RAM bar would be under half and green in colour.

", "votes": null, "replies": [ { "id": 2459579, "author_name": "alikarbala", "author_url": "", "post_date": "09/28/2023 09:32:52", "content": "

Thanks for all helpful suggestions… I tried all the ways and didnt work… The notebook is running succesfully without RAM turning red I check it before submitting, i did gc.collect and batchwise predictions, even i submit the inference on another notebook… Non of these work thats why i ask for help…

", "votes": null, "replies": [ { "id": 2459613, "author_name": "quillen", "author_url": "", "post_date": "09/28/2023 09:53:37", "content": "

How are you generating your batches?

\n

From the notebook you linked, it looks like you're loading the entire test_data.parquet file, transforming all of it with a single make_features() call, and then passing the whole thing to your model.

", "votes": null, "replies": [ { "id": 2459953, "author_name": "alikarbala", "author_url": "", "post_date": "09/28/2023 13:42:33", "content": "

test = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet\")

\n

Preprocess the test data to create X_test

\n

test = make_features(test)
\nX_test = test[features]

\n

Define batch size for predictions

\n

batch_size = 16

\n

predictions = []

\n

Iterate through the test data and make batch predictions

\n

for i in range(0, len(X_test), batch_size):
\n batch = X_test.iloc[i:i + batch_size]
\n batch_predictions = model.predict(batch)
\n predictions.extend(batch_predictions)

\n

Update the test DataFrame with the predictions

\n

test[\"not_awake\"] = predictions
\ntest[\"awake\"] = predictions

\n

i did this in previous version.. It takes a long time in scoring thought that it works and after that it crashes.. RAM issue

", "votes": null, "replies": [ { "id": 2460026, "author_name": "quillen", "author_url": "", "post_date": "09/28/2023 14:32:06", "content": "

The hidden test dataset is ~200 full series, so the train series is a decent stand-in for diagnosing memory problems. Change test = pd.read_parquet(f\"{DIR}/test_series.parquet\") to load train_series.parquet instead and you'll see that a notebook with 30GB RAM crashes before it is able to use the model.

\n

Try looping through each series id one at a time. Load, transform and predict for that single series before moving on to the next one. You can make a submission dataframe for a single series on each pass, then concatenate them all together before saving it as a csv.

", "votes": null, "replies": [ { "id": 2460077, "author_name": "alikarbala", "author_url": "", "post_date": "09/28/2023 15:03:10", "content": "

I'll try my best.. Thanks a lot

", "votes": null, "replies": [ { "id": 2464424, "author_name": "aidlre001", "author_url": "", "post_date": "10/02/2023 08:58:11", "content": "

Also try dask or polars libraries for reading parquet files. They seem to be faster. However, there may be some issues when trying to use pandas operations on the imported data so may need to study the documentation of these libraries.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2463877, "author_name": "lvarochapresto", "author_url": "", "post_date": "10/01/2023 16:29:43", "content": "

Hi @alikarbala.

\n

I was getting the same RAM issue and I resolved using the solution explained here.

\n

Basically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?

\n

Thanks to @abhranta for give me the solution.

\n

Best Regards!

", "votes": null, "replies": [ { "id": 2463915, "author_name": "alikarbala", "author_url": "", "post_date": "10/01/2023 17:16:43", "content": "

Thanks for your response..
\nActually I was training my keras model and I performed all previous suggestion from kagglers responded to my question.
\nBut the submission didnt work until I removed the hidden layers and the model is basic with low score..
\nI decide to change my model.
\nI will check your link and reach back.

", "votes": null, "replies": [] }, { "id": 2464564, "author_name": "alikarbala", "author_url": "", "post_date": "10/02/2023 10:49:54", "content": "

It works :) thanks @lvarochapresto

", "votes": null, "replies": [ { "id": 2464573, "author_name": "lvarochapresto", "author_url": "", "post_date": "10/02/2023 11:05:48", "content": "

Great!! Best Regards!

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2456606": "kaggle kernels pull alikarbala/sleep-detection-submission-with-keras\n\nHelp me... Why this notebook is raising RAM out of memory error", "2457093": "Hi @alikarbala, you maybe doing data manipulation or model fitting that does not fit into the memory. Consider \n1. Using optimal datatypes whenever possible\n2. If you are using GPU and full dataset, consider using light weight dataset\n3. Consider training and saving model in a kernel and then submit only inference notebook\n4. If you still face issue after submitting only inference, try batchwise predictions\nYou can refer public kernels for datasets and batch predictions", "2457819": "alikarbala : Hey Ali, you may want to review the parallel discussion thread per [Best Practices in Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) to see the tricks to optimize the RAM/CPU utilization for the dataset of this contest.\n\nI hope it is helpful.", "2459369": "In addition to the helpful suggestions already given, you need to perform garbage collection and delete some large data you don't need before adding more data. \n\nAlso, keep an eye out for the usage meter on the top right of the Notebook pane when you use CPU, GPU or TPU. You will see the RAM bar turning red (reaching or exceeding full capacity), which helps you know how your code is running on the available resources. When your code is running okay, the RAM bar would be under half and green in colour.", "2459579": "Thanks for all helpful suggestions... I tried all the ways and didnt work... The notebook is running succesfully without RAM turning red I check it before submitting, i did gc.collect and batchwise predictions, even i submit the inference on another notebook... Non of these work thats why i ask for help...", "2459613": "How are you generating your batches? \n\nFrom the notebook you linked, it looks like you're loading the entire `test_data.parquet` file, transforming all of it with a single `make_features()` call, and then passing the whole thing to your model.", "2459953": "test = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet\")\n\n# Preprocess the test data to create X_test\ntest = make_features(test)\nX_test = test[features]\n\n# Define batch size for predictions\nbatch_size = 16\n\n\npredictions = []\n\n# Iterate through the test data and make batch predictions\nfor i in range(0, len(X_test), batch_size):\n batch = X_test.iloc[i:i + batch_size]\n batch_predictions = model.predict(batch)\n predictions.extend(batch_predictions)\n\n# Update the test DataFrame with the predictions\ntest[\"not_awake\"] = predictions\ntest[\"awake\"] = predictions\n\ni did this in previous version.. It takes a long time in scoring thought that it works and after that it crashes.. RAM issue", "2460026": "The hidden test dataset is ~200 full series, so the train series is a decent stand-in for diagnosing memory problems. Change `test = pd.read_parquet(f\"{DIR}/test_series.parquet\")` to load `train_series.parquet` instead and you'll see that a notebook with 30GB RAM crashes before it is able to use the model.\n\nTry looping through each series id one at a time. Load, transform and predict for that single series before moving on to the next one. You can make a submission dataframe for a single series on each pass, then concatenate them all together before saving it as a csv.", "2460077": "I'll try my best.. Thanks a lot", "2463877": "Hi @alikarbala.\n\nI was getting the same RAM issue and I resolved using the solution explained [here](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set).\n\nBasically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?\n\nThanks to @abhranta for give me the solution.\n\nBest Regards!", "2463915": "Thanks for your response..\nActually I was training my keras model and I performed all previous suggestion from kagglers responded to my question.\nBut the submission didnt work until I removed the hidden layers and the model is basic with low score..\nI decide to change my model.\nI will check your link and reach back.", "2464424": "Also try dask or polars libraries for reading parquet files. They seem to be faster. However, there may be some issues when trying to use pandas operations on the imported data so may need to study the documentation of these libraries.", "2464564": "It works :) thanks @lvarochapresto", "2464573": "Great!! Best Regards!" }, "source": "meta" }