![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2250277%2F0989c1f41f39c5a83b12f4258a6d8aea%2FNew%20Bitmap%20Image.jpg?generation=1575160160373650&alt=media\")
Hello Friends,
\n\nMy kernel is dying too often. \nI have changed the datatype of the numeric columns to reduce the memory footprint. \nDo we have any other options to fix the issue. I would highly appreciate any inputs.
\n\n
Regards,\nGopi
", "messages": [ { "id": "685091", "postDate": "12/01/2019 00:30:46", "content": "Hello Friends,
\n\nMy kernel is dying too often. \nI have changed the datatype of the numeric columns to reduce the memory footprint. \nDo we have any other options to fix the issue. I would highly appreciate any inputs.
\n\n
Regards,\nGopi
", "rawMarkdown": "Hello Friends,\n\nMy kernel is dying too often. \nI have changed the datatype of the numeric columns to reduce the memory footprint. \nDo we have any other options to fix the issue. I would highly appreciate any inputs.\n\n\n\nRegards,\nGopi", "votes": null }, { "id": "685127", "postDate": "12/01/2019 02:19:19", "content": "Maybe It's out of memory when running the script.
", "rawMarkdown": "Maybe It's out of memory when running the script.", "votes": null }, { "id": "685140", "postDate": "12/01/2019 02:53:30", "content": "Yes it is but how to tune it further without dropping essential features?
", "rawMarkdown": "Yes it is but how to tune it further without dropping essential features?", "votes": null }, { "id": "685150", "postDate": "12/01/2019 03:26:10", "content": "If you really want to minimize data, you can represent most of the weather measurements as 16-bit integers without loss of information by simply multiplying by 10 and then truncating. It'll gain you a few extra bytes.
\n\nIf you don't want to use \"del\" to get rid of unneeded data as soon as you are finished with it, you can put everything inside of bite-sized functions. As soon as the function exits, the local variables will fall out of scope and the memory will be released.
\n\nYou don't need to have all of the test data in memory at the same time. You can split the main test table into pieces and join each individual piece with the building and weather tables as it is needed. You also don't need to have your train and validation data in memory at the same time, though if you are doing full CV you'll have some nasty time/space trade-offs to consider.
", "rawMarkdown": "If you really want to minimize data, you can represent most of the weather measurements as 16-bit integers without loss of information by simply multiplying by 10 and then truncating. It'll gain you a few extra bytes.\n\nIf you don't want to use \"del\" to get rid of unneeded data as soon as you are finished with it, you can put everything inside of bite-sized functions. As soon as the function exits, the local variables will fall out of scope and the memory will be released.\n\nYou don't need to have all of the test data in memory at the same time. You can split the main test table into pieces and join each individual piece with the building and weather tables as it is needed. You also don't need to have your train and validation data in memory at the same time, though if you are doing full CV you'll have some nasty time/space trade-offs to consider.", "votes": null }, { "id": "685257", "postDate": "12/01/2019 09:28:23", "content": "Thank you Robert. Let me try this.
", "rawMarkdown": "Thank you Robert. Let me try this.", "votes": null }, { "id": "685530", "postDate": "12/01/2019 21:01:43", "content": "Actually to release all the memory you have also to put the function in a subprocess, after the sub process is killed all memory will be released. That's because thanks to memory fragmentation gc.collect won't release all memory.
\n\nhttps://stackoverflow.com/a/1316799
\n\nTo make a subprocess just follow that example:
\n\nhttps://stackoverflow.com/a/2046630
", "rawMarkdown": "Actually to release all the memory you have also to put the function in a subprocess, after the sub process is killed all memory will be released. That's because thanks to memory fragmentation gc.collect won't release all memory.\n\nhttps://stackoverflow.com/a/1316799\n\nTo make a subprocess just follow that example:\n\nhttps://stackoverflow.com/a/2046630", "votes": null }, { "id": "685552", "postDate": "12/01/2019 22:29:24", "content": "Another note: if you are using XGB models, they can leak memory pretty badly. I use the following wrapper to work around the issue:\n```\nclass XGBWrapper(BaseEstimator, RegressorMixin):\n def init(self, *params):\n self.model = XGBRegressor(*params)
\n\ndef fit(self, X, y, **kwargs):\n fitted = clone(self.model).fit(X, y.values, **kwargs)\n self.feature_importances_ = fitted.feature_importances_\n dumped = pickle.dumps(fitted)\n del fitted\n gc.collect()\n self.fitted = pickle.loads(dumped)\n del dumped\n return self\n\ndef predict(self, X):\n return self.fitted.predict(X)\n\ndef get_params(self, deep=True):\n return self.model.get_params(deep)\n\ndef set_params(self, **params):\n self.model.set_params(params)\n```
", "rawMarkdown": "Another note: if you are using XGB models, they can leak memory pretty badly. I use the following wrapper to work around the issue:\n```\nclass XGBWrapper(BaseEstimator, RegressorMixin):\n def __init__(self, **params):\n self.model = XGBRegressor(**params)\n\n def fit(self, X, y, **kwargs):\n fitted = clone(self.model).fit(X, y.values, **kwargs)\n self.feature_importances_ = fitted.feature_importances_\n dumped = pickle.dumps(fitted)\n del fitted\n gc.collect()\n self.fitted = pickle.loads(dumped)\n del dumped\n return self\n\n def predict(self, X):\n return self.fitted.predict(X)\n\n def get_params(self, deep=True):\n return self.model.get_params(deep)\n\n def set_params(self, **params):\n self.model.set_params(params)\n```", "votes": null }, { "id": "689542", "postDate": "12/07/2019 02:54:58", "content": "good job, I'm just confused why I call gc.collect, and nothing change. @stenford23
", "rawMarkdown": "good job, I'm just confused why I call gc.collect, and nothing change. @stenford23", "votes": null }, { "id": "689602", "postDate": "12/07/2019 06:28:27", "content": "@diegojohnson \nBecause python memory management works with threshold and generation to choose when to release some memory. \nSo if your dataset has too many (\"unused\") reference you can't free it's memory therefore deleting a variable and calling gc.collect won't do anything in that case.\nYou should check every reference of a dataset and work on it, but i think it's more simple to just make all the work on a separated subprocess and kill it after it finish (maybe saving the results on an hdf or pickle).
", "rawMarkdown": "diegojohnson \nBecause python memory management works with threshold and generation to choose when to release some memory. \nSo if your dataset has too many (\"unused\") reference you can't free it's memory therefore deleting a variable and calling gc.collect won't do anything in that case.\nYou should check every reference of a dataset and work on it, but i think it's more simple to just make all the work on a separated subprocess and kill it after it finish (maybe saving the results on an hdf or pickle).", "votes": null }, { "id": "689603", "postDate": "12/07/2019 06:31:58", "content": "thanks very much, it's useful.
", "rawMarkdown": "thanks very much, it's useful.", "votes": null } ], "comments": [ { "id": 685127, "author_name": "m10515009", "author_url": "", "post_date": "12/01/2019 02:19:19", "content": "Maybe It's out of memory when running the script.
", "votes": null, "replies": [] }, { "id": 685140, "author_name": "gopiyedla", "author_url": "", "post_date": "12/01/2019 02:53:30", "content": "Yes it is but how to tune it further without dropping essential features?
", "votes": null, "replies": [ { "id": 685150, "author_name": "purist1024", "author_url": "", "post_date": "12/01/2019 03:26:10", "content": "If you really want to minimize data, you can represent most of the weather measurements as 16-bit integers without loss of information by simply multiplying by 10 and then truncating. It'll gain you a few extra bytes.
\n\nIf you don't want to use \"del\" to get rid of unneeded data as soon as you are finished with it, you can put everything inside of bite-sized functions. As soon as the function exits, the local variables will fall out of scope and the memory will be released.
\n\nYou don't need to have all of the test data in memory at the same time. You can split the main test table into pieces and join each individual piece with the building and weather tables as it is needed. You also don't need to have your train and validation data in memory at the same time, though if you are doing full CV you'll have some nasty time/space trade-offs to consider.
", "votes": null, "replies": [] }, { "id": 685257, "author_name": "gopiyedla", "author_url": "", "post_date": "12/01/2019 09:28:23", "content": "Thank you Robert. Let me try this.
", "votes": null, "replies": [] }, { "id": 685530, "author_name": "stenford23", "author_url": "", "post_date": "12/01/2019 21:01:43", "content": "Actually to release all the memory you have also to put the function in a subprocess, after the sub process is killed all memory will be released. That's because thanks to memory fragmentation gc.collect won't release all memory.
\n\nhttps://stackoverflow.com/a/1316799
\n\nTo make a subprocess just follow that example:
\n\nhttps://stackoverflow.com/a/2046630
", "votes": null, "replies": [] }, { "id": 689542, "author_name": "diegojohnson", "author_url": "", "post_date": "12/07/2019 02:54:58", "content": "good job, I'm just confused why I call gc.collect, and nothing change. @stenford23
", "votes": null, "replies": [] }, { "id": 689602, "author_name": "stenford23", "author_url": "", "post_date": "12/07/2019 06:28:27", "content": "@diegojohnson \nBecause python memory management works with threshold and generation to choose when to release some memory. \nSo if your dataset has too many (\"unused\") reference you can't free it's memory therefore deleting a variable and calling gc.collect won't do anything in that case.\nYou should check every reference of a dataset and work on it, but i think it's more simple to just make all the work on a separated subprocess and kill it after it finish (maybe saving the results on an hdf or pickle).
", "votes": null, "replies": [] }, { "id": 689603, "author_name": "diegojohnson", "author_url": "", "post_date": "12/07/2019 06:31:58", "content": "thanks very much, it's useful.
", "votes": null, "replies": [] } ] }, { "id": 685552, "author_name": "purist1024", "author_url": "", "post_date": "12/01/2019 22:29:24", "content": "Another note: if you are using XGB models, they can leak memory pretty badly. I use the following wrapper to work around the issue:\n```\nclass XGBWrapper(BaseEstimator, RegressorMixin):\n def init(self, *params):\n self.model = XGBRegressor(*params)
\n\ndef fit(self, X, y, **kwargs):\n fitted = clone(self.model).fit(X, y.values, **kwargs)\n self.feature_importances_ = fitted.feature_importances_\n dumped = pickle.dumps(fitted)\n del fitted\n gc.collect()\n self.fitted = pickle.loads(dumped)\n del dumped\n return self\n\ndef predict(self, X):\n return self.fitted.predict(X)\n\ndef get_params(self, deep=True):\n return self.model.get_params(deep)\n\ndef set_params(self, **params):\n self.model.set_params(params)\n```
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "685091": "Hello Friends,\n\nMy kernel is dying too often. \nI have changed the datatype of the numeric columns to reduce the memory footprint. \nDo we have any other options to fix the issue. I would highly appreciate any inputs.\n\n\n\nRegards,\nGopi", "685127": "Maybe It's out of memory when running the script.", "685140": "Yes it is but how to tune it further without dropping essential features?", "685150": "If you really want to minimize data, you can represent most of the weather measurements as 16-bit integers without loss of information by simply multiplying by 10 and then truncating. It'll gain you a few extra bytes.\n\nIf you don't want to use \"del\" to get rid of unneeded data as soon as you are finished with it, you can put everything inside of bite-sized functions. As soon as the function exits, the local variables will fall out of scope and the memory will be released.\n\nYou don't need to have all of the test data in memory at the same time. You can split the main test table into pieces and join each individual piece with the building and weather tables as it is needed. You also don't need to have your train and validation data in memory at the same time, though if you are doing full CV you'll have some nasty time/space trade-offs to consider.", "685257": "Thank you Robert. Let me try this.", "685530": "Actually to release all the memory you have also to put the function in a subprocess, after the sub process is killed all memory will be released. That's because thanks to memory fragmentation gc.collect won't release all memory.\n\nhttps://stackoverflow.com/a/1316799\n\nTo make a subprocess just follow that example:\n\nhttps://stackoverflow.com/a/2046630", "685552": "Another note: if you are using XGB models, they can leak memory pretty badly. I use the following wrapper to work around the issue:\n```\nclass XGBWrapper(BaseEstimator, RegressorMixin):\n def __init__(self, **params):\n self.model = XGBRegressor(**params)\n\n def fit(self, X, y, **kwargs):\n fitted = clone(self.model).fit(X, y.values, **kwargs)\n self.feature_importances_ = fitted.feature_importances_\n dumped = pickle.dumps(fitted)\n del fitted\n gc.collect()\n self.fitted = pickle.loads(dumped)\n del dumped\n return self\n\n def predict(self, X):\n return self.fitted.predict(X)\n\n def get_params(self, deep=True):\n return self.model.get_params(deep)\n\n def set_params(self, **params):\n self.model.set_params(params)\n```", "689542": "good job, I'm just confused why I call gc.collect, and nothing change. @stenford23", "689602": "diegojohnson \nBecause python memory management works with threshold and generation to choose when to release some memory. \nSo if your dataset has too many (\"unused\") reference you can't free it's memory therefore deleting a variable and calling gc.collect won't do anything in that case.\nYou should check every reference of a dataset and work on it, but i think it's more simple to just make all the work on a separated subprocess and kill it after it finish (maybe saving the results on an hdf or pickle).", "689603": "thanks very much, it's useful." }, "source": "meta" }