LightGBM based notebooks are running out of memory during submissions. I think we need better garbage collection while using LightGBM.
", "messages": [ { "id": "2444410", "postDate": "09/18/2023 09:13:09", "content": "LightGBM based notebooks are running out of memory during submissions. I think we need better garbage collection while using LightGBM.
", "rawMarkdown": "LightGBM based notebooks are running out of memory during submissions. I think we need better garbage collection while using LightGBM.", "votes": null }, { "id": "2444447", "postDate": "09/18/2023 09:20:18", "content": "Hi @siddhvr
\nCan you elaborate more on the size of your training set, number of features, and any other information that causing an OOM error on your submission?
\nRegarding why LGBM is faster but probably takes more memory, I think it is because a space-time trade-off nuance in the algorithm, CMIIW.
", "rawMarkdown": "Hi @siddhvr \n\nCan you elaborate more on the size of your training set, number of features, and any other information that causing an OOM error on your submission?\n\nRegarding why LGBM is faster but probably takes more memory, I think it is because a space-time trade-off nuance in the algorithm, CMIIW.", "votes": null }, { "id": "2444667", "postDate": "09/18/2023 11:50:09", "content": "Hey @siddhvr : RAM management issues are critical for this contest indeed.
\nForcing the regular garbage collection with gc package at the certain points of your notebook can address the problem parcially.
\nApart from that, you can take a look at the techniques to optimize the RAM usage in your notebook, as per Best Practices of Tackling BigData-Scale Tasks
\nLast but not least, there is a public kernel with lightgbm model submitting to the contest, per https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi - you can fork your own 'branch' of this notebook to start with. Then you can add your tunings/tweaks on top of it to see where the out-of-memory crashes start to manifest.
\nI hope it is helpful.
", "rawMarkdown": "Hey @siddhvr : RAM management issues are critical for this contest indeed.\n\nForcing the regular garbage collection with gc package at the certain points of your notebook can address the problem parcially.\n\nApart from that, you can take a look at the techniques to optimize the RAM usage in your notebook, as per [Best Practices of Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047)\n\nLast but not least, there is a public kernel with lightgbm model submitting to the contest, per https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi - you can fork your own 'branch' of this notebook to start with. Then you can add your tunings/tweaks on top of it to see where the out-of-memory crashes start to manifest.\n\nI hope it is helpful.", "votes": null }, { "id": "2444672", "postDate": "09/18/2023 11:53:41", "content": "Thanks for the tips Georgii.
", "rawMarkdown": "Thanks for the tips Georgii.", "votes": null }, { "id": "2444677", "postDate": "09/18/2023 11:57:59", "content": "Since, the competition is in its initial stages, I am keeping my work completely open. You can refer all of it here. I agree with your premise over the space-time tradeoff, but I have also made submissions with HistGradient Boost which works completely fine. Gradient Boosting is way too slower to use for this competition.
", "rawMarkdown": "Since, the competition is in its initial stages, I am keeping my work completely open. You can refer all of it [here](https://www.kaggle.com/code/siddhvr/cmi-gradient-boost-update-params). I agree with your premise over the space-time tradeoff, but I have also made submissions with HistGradient Boost which works completely fine. Gradient Boosting is way too slower to use for this competition.", "votes": null }, { "id": "2444678", "postDate": "09/18/2023 11:58:57", "content": "I'll surely refer to the resources you provided.
", "rawMarkdown": "I'll surely refer to the resources you provided.", "votes": null }, { "id": "2444950", "postDate": "09/18/2023 14:52:48", "content": "Good tip Thanks.
", "rawMarkdown": "Good tip Thanks.", "votes": null }, { "id": "2444967", "postDate": "09/18/2023 15:01:23", "content": "Alternatively we can use XGBoost but with tree_method='hist' as well to \"mimic\" the HistGBDT. It can also utilizes GPU if I remembered correctly.
Thanks @siddhvr. What can you present the results compared to another model?
", "rawMarkdown": "Thanks @siddhvr. What can you present the results compared to another model?", "votes": null }, { "id": "2445075", "postDate": "09/18/2023 15:57:12", "content": "Good to know. Thanks @Rimba Erlangga
", "rawMarkdown": "Good to know. Thanks @Rimba Erlangga", "votes": null }, { "id": "2445447", "postDate": "09/18/2023 20:44:35", "content": "Yes. But XGBoost requires some amount of parameter tuning before giving good results.
", "rawMarkdown": "Yes. But XGBoost requires some amount of parameter tuning before giving good results.", "votes": null }, { "id": "2550171", "postDate": "12/05/2023 20:54:02", "content": "How to outcome the notebook memory out of memory issue
", "rawMarkdown": "How to outcome the notebook memory out of memory issue", "votes": null } ], "comments": [ { "id": 2444447, "author_name": "rimbax", "author_url": "", "post_date": "09/18/2023 09:20:18", "content": "Hi @siddhvr
\nCan you elaborate more on the size of your training set, number of features, and any other information that causing an OOM error on your submission?
\nRegarding why LGBM is faster but probably takes more memory, I think it is because a space-time trade-off nuance in the algorithm, CMIIW.
", "votes": null, "replies": [ { "id": 2444677, "author_name": "siddhvr", "author_url": "", "post_date": "09/18/2023 11:57:59", "content": "Since, the competition is in its initial stages, I am keeping my work completely open. You can refer all of it here. I agree with your premise over the space-time tradeoff, but I have also made submissions with HistGradient Boost which works completely fine. Gradient Boosting is way too slower to use for this competition.
", "votes": null, "replies": [ { "id": 2444950, "author_name": "aidlre001", "author_url": "", "post_date": "09/18/2023 14:52:48", "content": "Good tip Thanks.
", "votes": null, "replies": [] }, { "id": 2444967, "author_name": "rimbax", "author_url": "", "post_date": "09/18/2023 15:01:23", "content": "Alternatively we can use XGBoost but with tree_method='hist' as well to \"mimic\" the HistGBDT. It can also utilizes GPU if I remembered correctly.
Good to know. Thanks @Rimba Erlangga
", "votes": null, "replies": [] }, { "id": 2445447, "author_name": "siddhvr", "author_url": "", "post_date": "09/18/2023 20:44:35", "content": "Yes. But XGBoost requires some amount of parameter tuning before giving good results.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2444667, "author_name": "gvyshnya", "author_url": "", "post_date": "09/18/2023 11:50:09", "content": "Hey @siddhvr : RAM management issues are critical for this contest indeed.
\nForcing the regular garbage collection with gc package at the certain points of your notebook can address the problem parcially.
\nApart from that, you can take a look at the techniques to optimize the RAM usage in your notebook, as per Best Practices of Tackling BigData-Scale Tasks
\nLast but not least, there is a public kernel with lightgbm model submitting to the contest, per https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi - you can fork your own 'branch' of this notebook to start with. Then you can add your tunings/tweaks on top of it to see where the out-of-memory crashes start to manifest.
\nI hope it is helpful.
", "votes": null, "replies": [ { "id": 2444672, "author_name": "siddhvr", "author_url": "", "post_date": "09/18/2023 11:53:41", "content": "Thanks for the tips Georgii.
", "votes": null, "replies": [] }, { "id": 2444678, "author_name": "siddhvr", "author_url": "", "post_date": "09/18/2023 11:58:57", "content": "I'll surely refer to the resources you provided.
", "votes": null, "replies": [] } ] }, { "id": 2444969, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/18/2023 15:01:46", "content": "Thanks @siddhvr. What can you present the results compared to another model?
", "votes": null, "replies": [] }, { "id": 2550171, "author_name": "kokulraj", "author_url": "", "post_date": "12/05/2023 20:54:02", "content": "How to outcome the notebook memory out of memory issue
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2444410": "LightGBM based notebooks are running out of memory during submissions. I think we need better garbage collection while using LightGBM.", "2444447": "Hi @siddhvr \n\nCan you elaborate more on the size of your training set, number of features, and any other information that causing an OOM error on your submission?\n\nRegarding why LGBM is faster but probably takes more memory, I think it is because a space-time trade-off nuance in the algorithm, CMIIW.", "2444667": "Hey @siddhvr : RAM management issues are critical for this contest indeed.\n\nForcing the regular garbage collection with gc package at the certain points of your notebook can address the problem parcially.\n\nApart from that, you can take a look at the techniques to optimize the RAM usage in your notebook, as per [Best Practices of Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047)\n\nLast but not least, there is a public kernel with lightgbm model submitting to the contest, per https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi - you can fork your own 'branch' of this notebook to start with. Then you can add your tunings/tweaks on top of it to see where the out-of-memory crashes start to manifest.\n\nI hope it is helpful.", "2444672": "Thanks for the tips Georgii.", "2444677": "Since, the competition is in its initial stages, I am keeping my work completely open. You can refer all of it [here](https://www.kaggle.com/code/siddhvr/cmi-gradient-boost-update-params). I agree with your premise over the space-time tradeoff, but I have also made submissions with HistGradient Boost which works completely fine. Gradient Boosting is way too slower to use for this competition.", "2444678": "I'll surely refer to the resources you provided.", "2444950": "Good tip Thanks.", "2444967": "Alternatively we can use XGBoost but with `tree_method='hist'` as well to \"mimic\" the HistGBDT. It can also utilizes GPU if I remembered correctly.", "2444969": "Thanks @siddhvr. What can you present the results compared to another model?", "2445075": "Good to know. Thanks @Rimba Erlangga", "2445447": "Yes. But XGBoost requires some amount of parameter tuning before giving good results.", "2550171": "How to outcome the notebook memory out of memory issue" }, "source": "meta" }