Hi,
\n\nI use XGBRegressor but when I run \"fit\" method with the whole train data, it fails on out of memory. When I take only 1M of train records, it works fine. How can overcome this memory limit?
", "messages": [ { "id": "671966", "postDate": "11/13/2019 11:34:43", "content": "Hi,
\n\nI use XGBRegressor but when I run \"fit\" method with the whole train data, it fails on out of memory. When I take only 1M of train records, it works fine. How can overcome this memory limit?
", "rawMarkdown": "Hi,\n\nI use XGBRegressor but when I run \"fit\" method with the whole train data, it fails on out of memory. When I take only 1M of train records, it works fine. How can overcome this memory limit?", "votes": null }, { "id": "672067", "postDate": "11/13/2019 13:39:23", "content": "Are you using GPU? I found using non GPU will have higher RAM available compared with GPU enabled.
", "rawMarkdown": "Are you using GPU? I found using non GPU will have higher RAM available compared with GPU enabled.", "votes": null }, { "id": "672178", "postDate": "11/13/2019 15:32:13", "content": "If your running kernel on local PC you can change virtual memory size (Windows) or change swap file size (Linux).
", "rawMarkdown": "If your running kernel on local PC you can change virtual memory size (Windows) or change swap file size (Linux).", "votes": null }, { "id": "672182", "postDate": "11/13/2019 15:38:23", "content": "I just tried CPU. Still fails.
", "rawMarkdown": "I just tried CPU. Still fails.", "votes": null }, { "id": "672192", "postDate": "11/13/2019 15:46:32", "content": "No, I run Kaggle kernel.
", "rawMarkdown": "No, I run Kaggle kernel.", "votes": null }, { "id": "672211", "postDate": "11/13/2019 16:24:19", "content": "Ok - I can't be of much help since I seldom use Kaggle kernels - but several shared kernels have described methods to reduce memory size so if you have not used any of those methods to reduce the bytes size for each of your variables they would be worth incorporating into your kernel. Probably any of the best scoring or high voted kernels will have a def for memory usage reduction.
", "rawMarkdown": "Ok - I can't be of much help since I seldom use Kaggle kernels - but several shared kernels have described methods to reduce memory size so if you have not used any of those methods to reduce the bytes size for each of your variables they would be worth incorporating into your kernel. Probably any of the best scoring or high voted kernels will have a def for memory usage reduction.", "votes": null }, { "id": "672219", "postDate": "11/13/2019 16:41:49", "content": "Please try to reduce the number of features and reduce the RAM usage of data types before calling XGB fit/train method.
", "rawMarkdown": "Please try to reduce the number of features and reduce the RAM usage of data types before calling XGB fit/train method.", "votes": null }, { "id": "672941", "postDate": "11/14/2019 10:10:23", "content": "Thanks for the answers.
\n\nI solved it by the following steps:\n1. I opened only needed dataframes and deleted them immediately after completing to use them. For example test.csv is a huge one and I don't need it until the end of training.\n2. I used reducing-dataframe-memory-size-by-65 techniques. But be careful: it's filling Null values. So make sure to fill them according to** your** logic.\n3. Even after improvements, I wasn't able to run it on Kaggle, since it fails on out of memory without proper logs. I had to run it on my laptop.
", "rawMarkdown": "Thanks for the answers.\n\nI solved it by the following steps:\n1. I opened only needed dataframes and deleted them immediately after completing to use them. For example test.csv is a huge one and I don't need it until the end of training.\n2. I used [reducing-dataframe-memory-size-by-65](https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65) techniques. But be careful: it's filling Null values. So make sure to fill them according to** your** logic.\n3. Even after improvements, I wasn't able to run it on Kaggle, since it fails on out of memory without proper logs. I had to run it on my laptop.", "votes": null }, { "id": "686404", "postDate": "12/03/2019 05:39:37", "content": "I was also facing the same issue. \nYou can delete the variables which are not in use so that it can be deleted from RAM.\n - Like after you split your training data into train and validation you can delete you full training dataset to save your RAM memory. \ndel training
First, i suggest to make etl pipeline and save train, test in pkl or hdf in another kernel.\nAfter that , load and fit and if you run out of memory again , you need to drop one or more column (ram used by xgboost depends highly on the number of column and row). Max number of columns to keep is 26/29 (i find it by some trial).\nTake in mind that xgboost leaks some memory during the fit so if you make cv or more model you need to optimize more.
", "rawMarkdown": "First, i suggest to make etl pipeline and save train, test in pkl or hdf in another kernel.\nAfter that , load and fit and if you run out of memory again , you need to drop one or more column (ram used by xgboost depends highly on the number of column and row). Max number of columns to keep is 26/29 (i find it by some trial).\nTake in mind that xgboost leaks some memory during the fit so if you make cv or more model you need to optimize more.", "votes": null }, { "id": "688873", "postDate": "12/06/2019 07:37:24", "content": "Hello! I've faced the same problem, but I could only choose to run on my laptop because I lack an GPU on my computer. Do you use CPU to run the program?\nBy the way, do you solve the problem now?
", "rawMarkdown": "Hello! I've faced the same problem, but I could only choose to run on my laptop because I lack an GPU on my computer. Do you use CPU to run the program?\nBy the way, do you solve the problem now?", "votes": null } ], "comments": [ { "id": 672067, "author_name": "klloke", "author_url": "", "post_date": "11/13/2019 13:39:23", "content": "Are you using GPU? I found using non GPU will have higher RAM available compared with GPU enabled.
", "votes": null, "replies": [ { "id": 672182, "author_name": "benzion18", "author_url": "", "post_date": "11/13/2019 15:38:23", "content": "I just tried CPU. Still fails.
", "votes": null, "replies": [] }, { "id": 672219, "author_name": "klloke", "author_url": "", "post_date": "11/13/2019 16:41:49", "content": "Please try to reduce the number of features and reduce the RAM usage of data types before calling XGB fit/train method.
", "votes": null, "replies": [] } ] }, { "id": 672178, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/13/2019 15:32:13", "content": "If your running kernel on local PC you can change virtual memory size (Windows) or change swap file size (Linux).
", "votes": null, "replies": [ { "id": 672192, "author_name": "benzion18", "author_url": "", "post_date": "11/13/2019 15:46:32", "content": "No, I run Kaggle kernel.
", "votes": null, "replies": [] }, { "id": 672211, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/13/2019 16:24:19", "content": "Ok - I can't be of much help since I seldom use Kaggle kernels - but several shared kernels have described methods to reduce memory size so if you have not used any of those methods to reduce the bytes size for each of your variables they would be worth incorporating into your kernel. Probably any of the best scoring or high voted kernels will have a def for memory usage reduction.
", "votes": null, "replies": [] } ] }, { "id": 672941, "author_name": "benzion18", "author_url": "", "post_date": "11/14/2019 10:10:23", "content": "Thanks for the answers.
\n\nI solved it by the following steps:\n1. I opened only needed dataframes and deleted them immediately after completing to use them. For example test.csv is a huge one and I don't need it until the end of training.\n2. I used reducing-dataframe-memory-size-by-65 techniques. But be careful: it's filling Null values. So make sure to fill them according to** your** logic.\n3. Even after improvements, I wasn't able to run it on Kaggle, since it fails on out of memory without proper logs. I had to run it on my laptop.
", "votes": null, "replies": [ { "id": 686456, "author_name": "stenford23", "author_url": "", "post_date": "12/03/2019 07:09:36", "content": "First, i suggest to make etl pipeline and save train, test in pkl or hdf in another kernel.\nAfter that , load and fit and if you run out of memory again , you need to drop one or more column (ram used by xgboost depends highly on the number of column and row). Max number of columns to keep is 26/29 (i find it by some trial).\nTake in mind that xgboost leaks some memory during the fit so if you make cv or more model you need to optimize more.
", "votes": null, "replies": [] }, { "id": 688873, "author_name": "laurencelin", "author_url": "", "post_date": "12/06/2019 07:37:24", "content": "Hello! I've faced the same problem, but I could only choose to run on my laptop because I lack an GPU on my computer. Do you use CPU to run the program?\nBy the way, do you solve the problem now?
", "votes": null, "replies": [] } ] }, { "id": 686404, "author_name": "prateek0x", "author_url": "", "post_date": "12/03/2019 05:39:37", "content": "I was also facing the same issue. \nYou can delete the variables which are not in use so that it can be deleted from RAM.\n - Like after you split your training data into train and validation you can delete you full training dataset to save your RAM memory. \ndel training