{ "id": 439007, "title": "Another Submission Error Question", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/439007", "author_name": "", "post_date": "2023-09-13T10:50:18.937131600Z", "votes": null, "comment_count": 3, "views": 0, "content": "

I'm running into the following Error - \"Notebook out of Memory\":
\n\"\"
\n\"logdata\"
\nI've tried the reconmendations from this post:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437600
\nHere is my notebook:
\nhttps://www.kaggle.com/code/harriwashere/xgboost-feature-eng-selection-and-submission
\nIs there a way i can debug this? The notebook runs fine, this error only occurs when submitting?

", "messages": [ { "id": "2436098", "postDate": "09/13/2023 10:50:18", "content": "

I'm running into the following Error - \"Notebook out of Memory\":
\n\"\"
\n\"logdata\"
\nI've tried the reconmendations from this post:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437600
\nHere is my notebook:
\nhttps://www.kaggle.com/code/harriwashere/xgboost-feature-eng-selection-and-submission
\nIs there a way i can debug this? The notebook runs fine, this error only occurs when submitting?

", "rawMarkdown": "I'm running into the following Error - \"Notebook out of Memory\":\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3127267%2Fd6c92ddec79e095f7ebf22c56b841a57%2FScreenshot%202023-09-13%20204657.jpg?generation=1694602037518025&alt=media)\n![logdata](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3127267%2Fe39394d2ff816ff9b2afa1d223e857e6%2FScreenshot%202023-09-13%20204604.jpg?generation=1694601982331361&alt=media)\n\nI've tried the reconmendations from this post:\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437600\n\nHere is my notebook:\nhttps://www.kaggle.com/code/harriwashere/xgboost-feature-eng-selection-and-submission\n\nIs there a way i can debug this? The notebook runs fine, this error only occurs when submitting?", "votes": null }, { "id": "2436127", "postDate": "09/13/2023 11:22:21", "content": "

The hidden test set (~200 series) is much larger than the dummy test set that is visible. It's also much larger than the reduced training data (35 series) that you're using.

\n

You could debug it by using the ~240 series excluded from your reduced training data as a substitute for the test data. Once you can get it to run under those circumstances, your submissions shouldn't run out of memory

", "rawMarkdown": "The hidden test set (~200 series) is *much* larger than the dummy test set that is visible. It's also much larger than the reduced training data (35 series) that you're using. \n\nYou could debug it by using the ~240 series excluded from your reduced training data as a substitute for the test data. Once you can get it to run under those circumstances, your submissions shouldn't run out of memory", "votes": null }, { "id": "2436130", "postDate": "09/13/2023 11:25:22", "content": "

I think the test set is too large, so during execution of feat_eng function you might be hitting RAM limit. See if any of this helps

\n
    \n
  1. Try reducing the number of features
    2. \n
  2. Batch wise predictions
    3. \n
  3. If possible train and save the model beforehand and submit only inference notebook.
    4. \n
", "rawMarkdown": "I think the test set is too large, so during execution of `feat_eng` function you might be hitting RAM limit. See if any of this helps\n1. Try reducing the number of features\n2. Batch wise predictions\n3. If possible train and save the model beforehand and submit only inference notebook.", "votes": null }, { "id": "2441098", "postDate": "09/16/2023 03:03:31", "content": "

That is the problem, I created a loop to filter the test and run it in parts.
\nYou can see this idea in this notebook.

", "rawMarkdown": "That is the problem, I created a loop to filter the test and run it in parts.\nYou can see this idea in this [notebook](https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi).", "votes": null } ], "comments": [ { "id": 2436127, "author_name": "quillen", "author_url": "", "post_date": "09/13/2023 11:22:21", "content": "

The hidden test set (~200 series) is much larger than the dummy test set that is visible. It's also much larger than the reduced training data (35 series) that you're using.

\n

You could debug it by using the ~240 series excluded from your reduced training data as a substitute for the test data. Once you can get it to run under those circumstances, your submissions shouldn't run out of memory

", "votes": null, "replies": [ { "id": 2441098, "author_name": "renatoreggiani", "author_url": "", "post_date": "09/16/2023 03:03:31", "content": "

That is the problem, I created a loop to filter the test and run it in parts.
\nYou can see this idea in this notebook.

", "votes": null, "replies": [] } ] }, { "id": 2436130, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/13/2023 11:25:22", "content": "

I think the test set is too large, so during execution of feat_eng function you might be hitting RAM limit. See if any of this helps

\n
    \n
  1. Try reducing the number of features
    2. \n
  2. Batch wise predictions
    3. \n
  3. If possible train and save the model beforehand and submit only inference notebook.
    4. \n
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2436098": "I'm running into the following Error - \"Notebook out of Memory\":\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3127267%2Fd6c92ddec79e095f7ebf22c56b841a57%2FScreenshot%202023-09-13%20204657.jpg?generation=1694602037518025&alt=media)\n![logdata](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3127267%2Fe39394d2ff816ff9b2afa1d223e857e6%2FScreenshot%202023-09-13%20204604.jpg?generation=1694601982331361&alt=media)\n\nI've tried the reconmendations from this post:\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437600\n\nHere is my notebook:\nhttps://www.kaggle.com/code/harriwashere/xgboost-feature-eng-selection-and-submission\n\nIs there a way i can debug this? The notebook runs fine, this error only occurs when submitting?", "2436127": "The hidden test set (~200 series) is *much* larger than the dummy test set that is visible. It's also much larger than the reduced training data (35 series) that you're using. \n\nYou could debug it by using the ~240 series excluded from your reduced training data as a substitute for the test data. Once you can get it to run under those circumstances, your submissions shouldn't run out of memory", "2436130": "I think the test set is too large, so during execution of `feat_eng` function you might be hitting RAM limit. See if any of this helps\n1. Try reducing the number of features\n2. Batch wise predictions\n3. If possible train and save the model beforehand and submit only inference notebook.", "2441098": "That is the problem, I created a loop to filter the test and run it in parts.\nYou can see this idea in this [notebook](https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi)." }, "source": "meta" }