{ "id": 440523, "title": "How can I solve \"Notebook Out of Memory Error!\" ?", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/440523", "author_name": "", "post_date": "2023-09-15T08:23:13.989431500Z", "votes": 1, "comment_count": 6, "views": 0, "content": "

Hi everyone,

\n

I keep getting \"notebook out of memory\" errors.

\n

Firstly, I imported the train_series, created the feature, trained the model, and deleted the dataset and used gc.

\n

Then, I imported test_series using by polar.

\n

Despite all these attempts, I constantly encounter the same problem.

\n

Is there anyone to help me?

\n

Thanks.

", "messages": [ { "id": "2440033", "postDate": "09/15/2023 08:23:13", "content": "

Hi everyone,

\n

I keep getting \"notebook out of memory\" errors.

\n

Firstly, I imported the train_series, created the feature, trained the model, and deleted the dataset and used gc.

\n

Then, I imported test_series using by polar.

\n

Despite all these attempts, I constantly encounter the same problem.

\n

Is there anyone to help me?

\n

Thanks.

", "rawMarkdown": "Hi everyone,\n\nI keep getting \"notebook out of memory\" errors.\n\nFirstly, I imported the train_series, created the feature, trained the model, and deleted the dataset and used gc.\n\nThen, I imported test_series using by polar. \n\nDespite all these attempts, I constantly encounter the same problem.\n\nIs there anyone to help me?\n\nThanks.", "votes": null }, { "id": "2440088", "postDate": "09/15/2023 08:58:02", "content": "

If you're encountering a \"Notebook Out of Memory\" error, it means that your computing environment, such as a Jupyter Notebook, has run out of available RAM (random access memory). This issue can occur when you're working with large datasets or complex models.

", "rawMarkdown": "If you're encountering a \"Notebook Out of Memory\" error, it means that your computing environment, such as a Jupyter Notebook, has run out of available RAM (random access memory). This issue can occur when you're working with large datasets or complex models.", "votes": null }, { "id": "2440093", "postDate": "09/15/2023 09:02:02", "content": "

I didn't import different data set, i just used the competetion data set.

\n

How can i solve it? Do you have any solution advise?

", "rawMarkdown": "I didn't import different data set, i just used the competetion data set.\n\nHow can i solve it? Do you have any solution advise?", "votes": null }, { "id": "2440099", "postDate": "09/15/2023 09:08:22", "content": "

Hi @ozansabirlioglu. I'm starting the preprocessing using this lightened dataset. I'm increasing the number of features while observing memory usage.
\nAlso, looking at the published notebooks, they all use lightweight datasets, such as by narrowing down the IDs, here.

", "rawMarkdown": "Hi @ozansabirlioglu. I'm starting the preprocessing using [this lightened dataset](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440170). I'm increasing the number of features while observing memory usage.\nAlso, looking at the published notebooks, they all use lightweight datasets, such as by narrowing down the IDs, [here](https://www.kaggle.com/code/carlmcbrideellis/zzzs-random-forest-model-starter).", "votes": null }, { "id": "2440321", "postDate": "09/15/2023 12:39:56", "content": "

Hi @ozansabirlioglu

\n

I've experienced this as well. After I used the lightened dataset mentioned by @mitsuyasuhoshino and doing some memory usage reduction, I solved this problem

\n

Furthermore, if you encounter any other issue during submission, you can refer to this official guideline

\n

https://www.kaggle.com/code-competition-debugging

\n

Good luck!

", "rawMarkdown": "Hi @ozansabirlioglu \n\nI've experienced this as well. After I used the lightened dataset mentioned by @mitsuyasuhoshino and doing some memory usage reduction, I solved this problem \n\nFurthermore, if you encounter any other issue during submission, you can refer to this official guideline\n\nhttps://www.kaggle.com/code-competition-debugging\n\nGood luck!", "votes": null }, { "id": "2440473", "postDate": "09/15/2023 14:30:49", "content": "

Hi @ozansabirlioglu, try this Tackling out of memory issue on test set

", "rawMarkdown": "Hi @ozansabirlioglu, try this [Tackling out of memory issue on test set](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440592)", "votes": null }, { "id": "2441095", "postDate": "09/16/2023 02:59:44", "content": "

@ozansabirlioglu , the test_series data is larger than train_series, I created a loop to filter the test and run it in parts.
\nYou can see this idea in this notebook.
\nIt is important to take this care because data from private tests is greater than that from public tests.

", "rawMarkdown": "ozansabirlioglu , the test_series data is larger than train_series, I created a loop to filter the test and run it in parts.\nYou can see this idea in this [notebook](https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi).\nIt is important to take this care because data from private tests is greater than that from public tests.", "votes": null } ], "comments": [ { "id": 2440088, "author_name": "sheerazhussain", "author_url": "", "post_date": "09/15/2023 08:58:02", "content": "

If you're encountering a \"Notebook Out of Memory\" error, it means that your computing environment, such as a Jupyter Notebook, has run out of available RAM (random access memory). This issue can occur when you're working with large datasets or complex models.

", "votes": null, "replies": [ { "id": 2440093, "author_name": "ozansabirlioglu", "author_url": "", "post_date": "09/15/2023 09:02:02", "content": "

I didn't import different data set, i just used the competetion data set.

\n

How can i solve it? Do you have any solution advise?

", "votes": null, "replies": [ { "id": 2440099, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/15/2023 09:08:22", "content": "

Hi @ozansabirlioglu. I'm starting the preprocessing using this lightened dataset. I'm increasing the number of features while observing memory usage.
\nAlso, looking at the published notebooks, they all use lightweight datasets, such as by narrowing down the IDs, here.

", "votes": null, "replies": [] } ] } ] }, { "id": 2440321, "author_name": "rimbax", "author_url": "", "post_date": "09/15/2023 12:39:56", "content": "

Hi @ozansabirlioglu

\n

I've experienced this as well. After I used the lightened dataset mentioned by @mitsuyasuhoshino and doing some memory usage reduction, I solved this problem

\n

Furthermore, if you encounter any other issue during submission, you can refer to this official guideline

\n

https://www.kaggle.com/code-competition-debugging

\n

Good luck!

", "votes": null, "replies": [] }, { "id": 2440473, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/15/2023 14:30:49", "content": "

Hi @ozansabirlioglu, try this Tackling out of memory issue on test set

", "votes": null, "replies": [] }, { "id": 2441095, "author_name": "renatoreggiani", "author_url": "", "post_date": "09/16/2023 02:59:44", "content": "

@ozansabirlioglu , the test_series data is larger than train_series, I created a loop to filter the test and run it in parts.
\nYou can see this idea in this notebook.
\nIt is important to take this care because data from private tests is greater than that from public tests.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2440033": "Hi everyone,\n\nI keep getting \"notebook out of memory\" errors.\n\nFirstly, I imported the train_series, created the feature, trained the model, and deleted the dataset and used gc.\n\nThen, I imported test_series using by polar. \n\nDespite all these attempts, I constantly encounter the same problem.\n\nIs there anyone to help me?\n\nThanks.", "2440088": "If you're encountering a \"Notebook Out of Memory\" error, it means that your computing environment, such as a Jupyter Notebook, has run out of available RAM (random access memory). This issue can occur when you're working with large datasets or complex models.", "2440093": "I didn't import different data set, i just used the competetion data set.\n\nHow can i solve it? Do you have any solution advise?", "2440099": "Hi @ozansabirlioglu. I'm starting the preprocessing using [this lightened dataset](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440170). I'm increasing the number of features while observing memory usage.\nAlso, looking at the published notebooks, they all use lightweight datasets, such as by narrowing down the IDs, [here](https://www.kaggle.com/code/carlmcbrideellis/zzzs-random-forest-model-starter).", "2440321": "Hi @ozansabirlioglu \n\nI've experienced this as well. After I used the lightened dataset mentioned by @mitsuyasuhoshino and doing some memory usage reduction, I solved this problem \n\nFurthermore, if you encounter any other issue during submission, you can refer to this official guideline\n\nhttps://www.kaggle.com/code-competition-debugging\n\nGood luck!", "2440473": "Hi @ozansabirlioglu, try this [Tackling out of memory issue on test set](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440592)", "2441095": "ozansabirlioglu , the test_series data is larger than train_series, I created a loop to filter the test and run it in parts.\nYou can see this idea in this [notebook](https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi).\nIt is important to take this care because data from private tests is greater than that from public tests." }, "source": "meta" }