{ "id": 441888, "title": "Out Of Memory Error during submission", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/441888", "author_name": "", "post_date": "2023-09-20T13:50:57.338330700Z", "votes": 3, "comment_count": 8, "views": 0, "content": "

Hey kagglers!

\n

During this days I tried to make a baseline submission and I'm getting constant OOM Errors.

\n

In order to deal with it I created a notebook that loads the model trained previously and import the feature extraction logic in order to make the submission without any training process. However, I'm still getting OOM Error.

\n

I'm using a default LGB model with 16 basic features (moving avg, std, sin(hour) cos(hour) and so on) and my dataset is the 43 series one created by @carlmcbrideellis.

\n

Here is the link to the dataset: Dataset

\n

Can you give me some advice or suggestions?

\n

Thanks in advance :)

", "messages": [ { "id": "2448275", "postDate": "09/20/2023 13:50:57", "content": "

Hey kagglers!

\n

During this days I tried to make a baseline submission and I'm getting constant OOM Errors.

\n

In order to deal with it I created a notebook that loads the model trained previously and import the feature extraction logic in order to make the submission without any training process. However, I'm still getting OOM Error.

\n

I'm using a default LGB model with 16 basic features (moving avg, std, sin(hour) cos(hour) and so on) and my dataset is the 43 series one created by @carlmcbrideellis.

\n

Here is the link to the dataset: Dataset

\n

Can you give me some advice or suggestions?

\n

Thanks in advance :)

", "rawMarkdown": "Hey kagglers! \n\nDuring this days I tried to make a baseline submission and I'm getting constant OOM Errors. \n\nIn order to deal with it I created a notebook that loads the model trained previously and import the feature extraction logic in order to make the submission without any training process. However, I'm still getting OOM Error.\n\nI'm using a default LGB model with 16 basic features (moving avg, std, sin(hour) cos(hour) and so on) and my dataset is the 43 series one created by @carlmcbrideellis.\n\nHere is the link to the dataset: [Dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target)\n\nCan you give me some advice or suggestions? \n\nThanks in advance :)", "votes": null }, { "id": "2448280", "postDate": "09/20/2023 13:55:21", "content": "

Try batch predictions like load some series and infer them and then move to next batch. Here is reference notebook for inferencing one series at a time.

", "rawMarkdown": "Try batch predictions like load some series and infer them and then move to next batch. Here is reference [notebook](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set) for inferencing one series at a time.", "votes": null }, { "id": "2448281", "postDate": "09/20/2023 13:55:33", "content": "

Hi @jbowski , have you tried using memory usage reduction during submission? There are plenty public notebooks for that.

\n

If you still encounter OOMs after that, try using batching inference on the test set during submission, i.e. do feature engineering and prediction batch by batch, not on the whole set.

", "rawMarkdown": "Hi @jbowski , have you tried using memory usage reduction during submission? There are plenty public notebooks for that.\n\nIf you still encounter OOMs after that, try using batching inference on the test set during submission, i.e. do feature engineering and prediction batch by batch, not on the whole set.", "votes": null }, { "id": "2448291", "postDate": "09/20/2023 14:01:50", "content": "

@jbowski LightGBM is causing some OOM issues. We too faced this problem. If you wish to use lightGBM only, then I may suggest you to train the model elsewhere and purely infer in the submission kernel. You may also try and make a submission using another model too, perhaps ExtraTrees/ Random Forest.
\nYou may also use polars instead of pandas as polars offers a lazy execution framework that could prevent idle data from causing OOm issues. You may use our notebooks to curate your submission for now, and improve it as well. Links are as below-

\n
    \n
  1. Data curation - https://www.kaggle.com/code/ravi20076/sleepstate-datacuration
    2. \n
  2. Training and inference - https://www.kaggle.com/code/ravi20076/sleepstate-baseline-models
    3. \n
\n

Best regards!

", "rawMarkdown": "jbowski LightGBM is causing some OOM issues. We too faced this problem. If you wish to use lightGBM only, then I may suggest you to train the model elsewhere and purely infer in the submission kernel. You may also try and make a submission using another model too, perhaps ExtraTrees/ Random Forest. \nYou may also use polars instead of pandas as polars offers a lazy execution framework that could prevent idle data from causing OOm issues. You may use our notebooks to curate your submission for now, and improve it as well. Links are as below-\n1. Data curation - https://www.kaggle.com/code/ravi20076/sleepstate-datacuration\n2. Training and inference - https://www.kaggle.com/code/ravi20076/sleepstate-baseline-models\n\nBest regards!", "votes": null }, { "id": "2448299", "postDate": "09/20/2023 14:05:26", "content": "

Try to run the inference on the full training dataset (not the dummy test dataset) before every submission. Since when submitting, the test dataset is swapped to a larger one, similar to the size of full training dataset. Although quite time-consuming, but by doing so, at least you don't waste a submission.

\n

Also, when OOM, just delete some of the not so useful features, you can check it by your_model_name.feature_importances_ for lgb.

", "rawMarkdown": "Try to run the inference on the full training dataset (not the dummy test dataset) before every submission. Since when submitting, the test dataset is swapped to a larger one, similar to the size of full training dataset. Although quite time-consuming, but by doing so, at least you don't waste a submission. \n\nAlso, when OOM, just delete some of the not so useful features, you can check it by your_model_name.feature_importances_ for lgb.", "votes": null }, { "id": "2448511", "postDate": "09/20/2023 16:18:48", "content": "

Thanks a lot for your suggestions :)
\nI tried what you recommended me and I was able to run the baseline submission without OOM. Now time for improve everything!

\n

@anilreddyvv @ravi20076 @rimbax @yuchen2066

", "rawMarkdown": "Thanks a lot for your suggestions :)\nI tried what you recommended me and I was able to run the baseline submission without OOM. Now time for improve everything! \n\n@anilreddyvv @ravi20076 @rimbax @yuchen2066", "votes": null }, { "id": "2448738", "postDate": "09/20/2023 19:10:13", "content": "

Great going @jbowski all the best!!

", "rawMarkdown": "Great going @jbowski all the best!!", "votes": null }, { "id": "2451040", "postDate": "09/22/2023 09:28:16", "content": "

Hi @jbowski, the teste_series in submission is very huge, you could run tests slicing by series_id. Here is an example to do that notebook

", "rawMarkdown": "Hi @jbowski, the teste_series in submission is very huge, you could run tests slicing by series_id. Here is an example to do that [notebook](https://www.kaggle.com/code/renatoreggiani/xgb-blend-cmi?scriptVersionId=143807271)", "votes": null }, { "id": "2463885", "postDate": "10/01/2023 16:46:15", "content": "

Hi @jbowski.

\n

I was getting the same RAM issue and I resolved using the solution explained here.

\n

I agree with @renatoreggiani, the issue maybe related to Test series.

\n

Basically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?

\n

Thanks to @abhranta for give me the solution.

\n

Best Regards!

", "rawMarkdown": "Hi @jbowski.\n\nI was getting the same RAM issue and I resolved using the solution explained [here](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set).\n\nI agree with @renatoreggiani, the issue maybe related to Test series. \n\nBasically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?\n\nThanks to @abhranta for give me the solution.\n\nBest Regards!", "votes": null } ], "comments": [ { "id": 2448280, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/20/2023 13:55:21", "content": "

Try batch predictions like load some series and infer them and then move to next batch. Here is reference notebook for inferencing one series at a time.

", "votes": null, "replies": [] }, { "id": 2448281, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 13:55:33", "content": "

Hi @jbowski , have you tried using memory usage reduction during submission? There are plenty public notebooks for that.

\n

If you still encounter OOMs after that, try using batching inference on the test set during submission, i.e. do feature engineering and prediction batch by batch, not on the whole set.

", "votes": null, "replies": [] }, { "id": 2448291, "author_name": "ravi20076", "author_url": "", "post_date": "09/20/2023 14:01:50", "content": "

@jbowski LightGBM is causing some OOM issues. We too faced this problem. If you wish to use lightGBM only, then I may suggest you to train the model elsewhere and purely infer in the submission kernel. You may also try and make a submission using another model too, perhaps ExtraTrees/ Random Forest.
\nYou may also use polars instead of pandas as polars offers a lazy execution framework that could prevent idle data from causing OOm issues. You may use our notebooks to curate your submission for now, and improve it as well. Links are as below-

\n
    \n
  1. Data curation - https://www.kaggle.com/code/ravi20076/sleepstate-datacuration
    2. \n
  2. Training and inference - https://www.kaggle.com/code/ravi20076/sleepstate-baseline-models
    3. \n
\n

Best regards!

", "votes": null, "replies": [] }, { "id": 2448299, "author_name": "yuchen2066", "author_url": "", "post_date": "09/20/2023 14:05:26", "content": "

Try to run the inference on the full training dataset (not the dummy test dataset) before every submission. Since when submitting, the test dataset is swapped to a larger one, similar to the size of full training dataset. Although quite time-consuming, but by doing so, at least you don't waste a submission.

\n

Also, when OOM, just delete some of the not so useful features, you can check it by your_model_name.feature_importances_ for lgb.

", "votes": null, "replies": [] }, { "id": 2448511, "author_name": "jbowski", "author_url": "", "post_date": "09/20/2023 16:18:48", "content": "

Thanks a lot for your suggestions :)
\nI tried what you recommended me and I was able to run the baseline submission without OOM. Now time for improve everything!

\n

@anilreddyvv @ravi20076 @rimbax @yuchen2066

", "votes": null, "replies": [ { "id": 2448738, "author_name": "ravi20076", "author_url": "", "post_date": "09/20/2023 19:10:13", "content": "

Great going @jbowski all the best!!

", "votes": null, "replies": [] } ] }, { "id": 2451040, "author_name": "renatoreggiani", "author_url": "", "post_date": "09/22/2023 09:28:16", "content": "

Hi @jbowski, the teste_series in submission is very huge, you could run tests slicing by series_id. Here is an example to do that notebook

", "votes": null, "replies": [] }, { "id": 2463885, "author_name": "lvarochapresto", "author_url": "", "post_date": "10/01/2023 16:46:15", "content": "

Hi @jbowski.

\n

I was getting the same RAM issue and I resolved using the solution explained here.

\n

I agree with @renatoreggiani, the issue maybe related to Test series.

\n

Basically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?

\n

Thanks to @abhranta for give me the solution.

\n

Best Regards!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2448275": "Hey kagglers! \n\nDuring this days I tried to make a baseline submission and I'm getting constant OOM Errors. \n\nIn order to deal with it I created a notebook that loads the model trained previously and import the feature extraction logic in order to make the submission without any training process. However, I'm still getting OOM Error.\n\nI'm using a default LGB model with 16 basic features (moving avg, std, sin(hour) cos(hour) and so on) and my dataset is the 43 series one created by @carlmcbrideellis.\n\nHere is the link to the dataset: [Dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target)\n\nCan you give me some advice or suggestions? \n\nThanks in advance :)", "2448280": "Try batch predictions like load some series and infer them and then move to next batch. Here is reference [notebook](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set) for inferencing one series at a time.", "2448281": "Hi @jbowski , have you tried using memory usage reduction during submission? There are plenty public notebooks for that.\n\nIf you still encounter OOMs after that, try using batching inference on the test set during submission, i.e. do feature engineering and prediction batch by batch, not on the whole set.", "2448291": "jbowski LightGBM is causing some OOM issues. We too faced this problem. If you wish to use lightGBM only, then I may suggest you to train the model elsewhere and purely infer in the submission kernel. You may also try and make a submission using another model too, perhaps ExtraTrees/ Random Forest. \nYou may also use polars instead of pandas as polars offers a lazy execution framework that could prevent idle data from causing OOm issues. You may use our notebooks to curate your submission for now, and improve it as well. Links are as below-\n1. Data curation - https://www.kaggle.com/code/ravi20076/sleepstate-datacuration\n2. Training and inference - https://www.kaggle.com/code/ravi20076/sleepstate-baseline-models\n\nBest regards!", "2448299": "Try to run the inference on the full training dataset (not the dummy test dataset) before every submission. Since when submitting, the test dataset is swapped to a larger one, similar to the size of full training dataset. Although quite time-consuming, but by doing so, at least you don't waste a submission. \n\nAlso, when OOM, just delete some of the not so useful features, you can check it by your_model_name.feature_importances_ for lgb.", "2448511": "Thanks a lot for your suggestions :)\nI tried what you recommended me and I was able to run the baseline submission without OOM. Now time for improve everything! \n\n@anilreddyvv @ravi20076 @rimbax @yuchen2066", "2448738": "Great going @jbowski all the best!!", "2451040": "Hi @jbowski, the teste_series in submission is very huge, you could run tests slicing by series_id. Here is an example to do that [notebook](https://www.kaggle.com/code/renatoreggiani/xgb-blend-cmi?scriptVersionId=143807271)", "2463885": "Hi @jbowski.\n\nI was getting the same RAM issue and I resolved using the solution explained [here](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set).\n\nI agree with @renatoreggiani, the issue maybe related to Test series. \n\nBasically, loading and predicting test series iteratively could be your solution. Can you try and give us feedback?\n\nThanks to @abhranta for give me the solution.\n\nBest Regards!" }, "source": "meta" }