I just noticed that loading test set along with 5 CatBoost GPU models is almost impossible. They are extremely large... Larger than 1 GBs each.
\n\nCan anyone explain why those models are using that much space, and is there any way to make them smaller?
", "messages": [ { "id": "663315", "postDate": "11/01/2019 19:47:05", "content": "I just noticed that loading test set along with 5 CatBoost GPU models is almost impossible. They are extremely large... Larger than 1 GBs each.
\n\nCan anyone explain why those models are using that much space, and is there any way to make them smaller?
", "rawMarkdown": "I just noticed that loading test set along with 5 CatBoost GPU models is almost impossible. They are extremely large... Larger than 1 GBs each.\n\nCan anyone explain why those models are using that much space, and is there any way to make them smaller?", "votes": null }, { "id": "663323", "postDate": "11/01/2019 20:03:06", "content": "Do you have many of categorical features? The combinations that CatBoost creates greatly increases the GPU model size.
\n\nHope this helps!
\n\nMore information:\nhttps://github.com/catboost/catboost/issues/680
", "rawMarkdown": "Do you have many of categorical features? The combinations that CatBoost creates greatly increases the GPU model size.\n\nHope this helps!\n\nMore information:\nhttps://github.com/catboost/catboost/issues/680", "votes": null }, { "id": "663326", "postDate": "11/01/2019 20:08:12", "content": "Thanks, this explains a lot but I have only 6 categoricals. I don't think it is too many.
", "rawMarkdown": "Thanks, this explains a lot but I have only 6 categoricals. I don't think it is too many.", "votes": null }, { "id": "663329", "postDate": "11/01/2019 20:11:27", "content": "Hmm, strange. I've also had problems with CatBoost GPU gobbling up a lot of space, but I'm not sure why it needs that much.
", "rawMarkdown": "Hmm, strange. I've also had problems with CatBoost GPU gobbling up a lot of space, but I'm not sure why it needs that much.", "votes": null }, { "id": "663441", "postDate": "11/02/2019 03:00:27", "content": "catboost is coded to run maximum GPU size - I think the default is to use 96% of available. Not sure if this is related to save size of the model - ?? You can change the default to a lower percentage and see if it helps.
", "rawMarkdown": "catboost is coded to run maximum GPU size - I think the default is to use 96% of available. Not sure if this is related to save size of the model - ?? You can change the default to a lower percentage and see if it helps.", "votes": null }, { "id": "664148", "postDate": "11/03/2019 08:38:08", "content": "Tried lots of different parameters, but still couldn't find a way to implement a 5 fold cross validation with CatBoost GPU models without running out of memory. Any ideas?
", "rawMarkdown": "Tried lots of different parameters, but still couldn't find a way to implement a 5 fold cross validation with CatBoost GPU models without running out of memory. Any ideas?", "votes": null }, { "id": "664213", "postDate": "11/03/2019 10:56:35", "content": "There are few ways to make it smaller:\n* lower boosting rounds (higher lr)\n* less categorical features\n* less depth
", "rawMarkdown": "There are few ways to make it smaller:\n* lower boosting rounds (higher lr)\n* less categorical features\n* less depth", "votes": null }, { "id": "664417", "postDate": "11/03/2019 16:22:42", "content": "Increase size of swap file - I am using 200GB on my Ubuntu machine and 400GB on my Windows machines - needs to be on SSD to not slow things down.
\n\nIncreasing on Windows 10 very easy to do - on Ubuntu I am on my 5th attempt to change my 3rd PC - think something got broken in 19.10 - going to replace with 18.04 and see if I can get it working. Fails when I try swapon.
", "rawMarkdown": "Increase size of swap file - I am using 200GB on my Ubuntu machine and 400GB on my Windows machines - needs to be on SSD to not slow things down.\n\nIncreasing on Windows 10 very easy to do - on Ubuntu I am on my 5th attempt to change my 3rd PC - think something got broken in 19.10 - going to replace with 18.04 and see if I can get it working. Fails when I try swapon.", "votes": null }, { "id": "664607", "postDate": "11/04/2019 00:49:47", "content": "OK - 6th attempt in progress. Ubuntu 18.04 hangs as I try to build the swap file. Not sure how I got lucky on the first machines that it worked. Time to re-install and try again.
", "rawMarkdown": "OK - 6th attempt in progress. Ubuntu 18.04 hangs as I try to build the swap file. Not sure how I got lucky on the first machines that it worked. Time to re-install and try again.", "votes": null }, { "id": "664973", "postDate": "11/04/2019 13:37:49", "content": "Or more gpus, haha
", "rawMarkdown": "Or more gpus, haha", "votes": null }, { "id": "665199", "postDate": "11/04/2019 18:21:58", "content": "@longyin2 not an option for kaggle kernels :/
", "rawMarkdown": "longyin2 not an option for kaggle kernels :/", "votes": null }, { "id": "665419", "postDate": "11/05/2019 01:43:15", "content": "Yes, not an option for kaggle kernels
", "rawMarkdown": "Yes, not an option for kaggle kernels", "votes": null }, { "id": "665523", "postDate": "11/05/2019 04:40:14", "content": "I iteratively take predictions from each model and then delete the model and its training data before blending the predictions.
", "rawMarkdown": "I iteratively take predictions from each model and then delete the model and its training data before blending the predictions.", "votes": null } ], "comments": [ { "id": 663323, "author_name": "carlolepelaars", "author_url": "", "post_date": "11/01/2019 20:03:06", "content": "Do you have many of categorical features? The combinations that CatBoost creates greatly increases the GPU model size.
\n\nHope this helps!
\n\nMore information:\nhttps://github.com/catboost/catboost/issues/680
", "votes": null, "replies": [ { "id": 663326, "author_name": "gunesevitan", "author_url": "", "post_date": "11/01/2019 20:08:12", "content": "Thanks, this explains a lot but I have only 6 categoricals. I don't think it is too many.
", "votes": null, "replies": [] }, { "id": 663329, "author_name": "carlolepelaars", "author_url": "", "post_date": "11/01/2019 20:11:27", "content": "Hmm, strange. I've also had problems with CatBoost GPU gobbling up a lot of space, but I'm not sure why it needs that much.
", "votes": null, "replies": [] }, { "id": 663441, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/02/2019 03:00:27", "content": "catboost is coded to run maximum GPU size - I think the default is to use 96% of available. Not sure if this is related to save size of the model - ?? You can change the default to a lower percentage and see if it helps.
", "votes": null, "replies": [] } ] }, { "id": 664148, "author_name": "gunesevitan", "author_url": "", "post_date": "11/03/2019 08:38:08", "content": "Tried lots of different parameters, but still couldn't find a way to implement a 5 fold cross validation with CatBoost GPU models without running out of memory. Any ideas?
", "votes": null, "replies": [ { "id": 664417, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/03/2019 16:22:42", "content": "Increase size of swap file - I am using 200GB on my Ubuntu machine and 400GB on my Windows machines - needs to be on SSD to not slow things down.
\n\nIncreasing on Windows 10 very easy to do - on Ubuntu I am on my 5th attempt to change my 3rd PC - think something got broken in 19.10 - going to replace with 18.04 and see if I can get it working. Fails when I try swapon.
", "votes": null, "replies": [] }, { "id": 664607, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/04/2019 00:49:47", "content": "OK - 6th attempt in progress. Ubuntu 18.04 hangs as I try to build the swap file. Not sure how I got lucky on the first machines that it worked. Time to re-install and try again.
", "votes": null, "replies": [] } ] }, { "id": 664213, "author_name": "kyakovlev", "author_url": "", "post_date": "11/03/2019 10:56:35", "content": "There are few ways to make it smaller:\n* lower boosting rounds (higher lr)\n* less categorical features\n* less depth
", "votes": null, "replies": [ { "id": 664973, "author_name": "longyin2", "author_url": "", "post_date": "11/04/2019 13:37:49", "content": "Or more gpus, haha
", "votes": null, "replies": [] }, { "id": 665199, "author_name": "gunesevitan", "author_url": "", "post_date": "11/04/2019 18:21:58", "content": "@longyin2 not an option for kaggle kernels :/
", "votes": null, "replies": [] }, { "id": 665419, "author_name": "longyin2", "author_url": "", "post_date": "11/05/2019 01:43:15", "content": "Yes, not an option for kaggle kernels
", "votes": null, "replies": [] } ] }, { "id": 665523, "author_name": "thebrownviking20", "author_url": "", "post_date": "11/05/2019 04:40:14", "content": "I iteratively take predictions from each model and then delete the model and its training data before blending the predictions.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "663315": "I just noticed that loading test set along with 5 CatBoost GPU models is almost impossible. They are extremely large... Larger than 1 GBs each.\n\nCan anyone explain why those models are using that much space, and is there any way to make them smaller?", "663323": "Do you have many of categorical features? The combinations that CatBoost creates greatly increases the GPU model size.\n\nHope this helps!\n\nMore information:\nhttps://github.com/catboost/catboost/issues/680", "663326": "Thanks, this explains a lot but I have only 6 categoricals. I don't think it is too many.", "663329": "Hmm, strange. I've also had problems with CatBoost GPU gobbling up a lot of space, but I'm not sure why it needs that much.", "663441": "catboost is coded to run maximum GPU size - I think the default is to use 96% of available. Not sure if this is related to save size of the model - ?? You can change the default to a lower percentage and see if it helps.", "664148": "Tried lots of different parameters, but still couldn't find a way to implement a 5 fold cross validation with CatBoost GPU models without running out of memory. Any ideas?", "664213": "There are few ways to make it smaller:\n* lower boosting rounds (higher lr)\n* less categorical features\n* less depth", "664417": "Increase size of swap file - I am using 200GB on my Ubuntu machine and 400GB on my Windows machines - needs to be on SSD to not slow things down.\n\nIncreasing on Windows 10 very easy to do - on Ubuntu I am on my 5th attempt to change my 3rd PC - think something got broken in 19.10 - going to replace with 18.04 and see if I can get it working. Fails when I try swapon.", "664607": "OK - 6th attempt in progress. Ubuntu 18.04 hangs as I try to build the swap file. Not sure how I got lucky on the first machines that it worked. Time to re-install and try again.", "664973": "Or more gpus, haha", "665199": "longyin2 not an option for kaggle kernels :/", "665419": "Yes, not an option for kaggle kernels", "665523": "I iteratively take predictions from each model and then delete the model and its training data before blending the predictions." }, "source": "meta" }