Hi, I get error \"Notebook Exceeded Allowed Compute\" after I submit.\nI just try to use pseudo label by retraining my model. Any suggestion?
", "messages": [ { "id": "849792", "postDate": "05/16/2020 04:32:14", "content": "Hi, I get error \"Notebook Exceeded Allowed Compute\" after I submit.\nI just try to use pseudo label by retraining my model. Any suggestion?
", "rawMarkdown": "Hi, I get error \"Notebook Exceeded Allowed Compute\" after I submit.\nI just try to use pseudo label by retraining my model. Any suggestion?", "votes": null }, { "id": "849828", "postDate": "05/16/2020 05:29:11", "content": "GPU time limit (6 hours) or the more probable: memory issue. Try to lower the batch size during training.
", "rawMarkdown": "GPU time limit (6 hours) or the more probable: memory issue. Try to lower the batch size during training.", "votes": null }, { "id": "852149", "postDate": "05/18/2020 07:53:14", "content": "I set batch size to 1 but still get this error.\nI need 4GB GPU memory and 6GB cpu memory with batch size=1 during training my model in kaggle kernel. Moreover, I just save the latest weight file so I need 600MB disk storage.\nI do not know why I can train with 3422 images but can not finetune with about 1200 test images.
", "rawMarkdown": "I set batch size to 1 but still get this error.\nI need 4GB GPU memory and 6GB cpu memory with batch size=1 during training my model in kaggle kernel. Moreover, I just save the latest weight file so I need 600MB disk storage.\nI do not know why I can train with 3422 images but can not finetune with about 1200 test images.", "votes": null }, { "id": "852155", "postDate": "05/18/2020 07:57:06", "content": "Moreover, I got this error in 44 minutes after submit, it means that this error occured when start finetune with pseudo label.
", "rawMarkdown": "Moreover, I got this error in 44 minutes after submit, it means that this error occured when start finetune with pseudo label.", "votes": null }, { "id": "852249", "postDate": "05/18/2020 09:18:10", "content": "Take a look at this notebook, I made it for another competition, but you can use most of the tips here too.
", "rawMarkdown": "Take a look at [this notebook](https://www.kaggle.com/pestipeti/optimization-tips-faster-train-faster-inference), I made it for another competition, but you can use most of the tips here too.", "votes": null }, { "id": "852463", "postDate": "05/18/2020 13:16:39", "content": "Thanks for sharing! After I check all of your tips, I notice I follow all most your tips like del all parameteres if I do not need that any more. \nAnd I have aslo observed the resource of my kernel during training and testing. \nAs I mentioned above, I just need 4GB GPU memory and 6GB cpu memory with batch size=1 during training and 2GB GPU memory and 3GB cpu memory during testing (test with training dataset tov observe cpu memory).\nIn my pseudo kernel, I release the resource for inference berfore finetuning, and then I just use test data to finetune. Beause test images are less than train images, it should uses few memory to finetune than training with train dataset.\nFor finetuning , I just save the final weight file which need about 600MB, so I think 5GB is enough.\nNow, I am still confused for this error.
", "rawMarkdown": "Thanks for sharing! After I check all of your tips, I notice I follow all most your tips like del all parameteres if I do not need that any more. \nAnd I have aslo observed the resource of my kernel during training and testing. \nAs I mentioned above, I just need 4GB GPU memory and 6GB cpu memory with batch size=1 during training and 2GB GPU memory and 3GB cpu memory during testing (test with training dataset tov observe cpu memory).\nIn my pseudo kernel, I release the resource for inference berfore finetuning, and then I just use test data to finetune. Beause test images are less than train images, it should uses few memory to finetune than training with train dataset.\nFor finetuning , I just save the final weight file which need about 600MB, so I think 5GB is enough.\nNow, I am still confused for this error.", "votes": null }, { "id": "902405", "postDate": "06/26/2020 05:59:51", "content": "@rilllove were you able to resolve this ? i am also facing the same issue. it is throwing me an error after 39 mins
", "rawMarkdown": "rilllove were you able to resolve this ? i am also facing the same issue. it is throwing me an error after 39 mins", "votes": null } ], "comments": [ { "id": 849828, "author_name": "pestipeti", "author_url": "", "post_date": "05/16/2020 05:29:11", "content": "GPU time limit (6 hours) or the more probable: memory issue. Try to lower the batch size during training.
", "votes": null, "replies": [ { "id": 852149, "author_name": "rilllove", "author_url": "", "post_date": "05/18/2020 07:53:14", "content": "I set batch size to 1 but still get this error.\nI need 4GB GPU memory and 6GB cpu memory with batch size=1 during training my model in kaggle kernel. Moreover, I just save the latest weight file so I need 600MB disk storage.\nI do not know why I can train with 3422 images but can not finetune with about 1200 test images.
", "votes": null, "replies": [] }, { "id": 852155, "author_name": "rilllove", "author_url": "", "post_date": "05/18/2020 07:57:06", "content": "Moreover, I got this error in 44 minutes after submit, it means that this error occured when start finetune with pseudo label.
", "votes": null, "replies": [] }, { "id": 852249, "author_name": "pestipeti", "author_url": "", "post_date": "05/18/2020 09:18:10", "content": "Take a look at this notebook, I made it for another competition, but you can use most of the tips here too.
", "votes": null, "replies": [] }, { "id": 852463, "author_name": "rilllove", "author_url": "", "post_date": "05/18/2020 13:16:39", "content": "Thanks for sharing! After I check all of your tips, I notice I follow all most your tips like del all parameteres if I do not need that any more. \nAnd I have aslo observed the resource of my kernel during training and testing. \nAs I mentioned above, I just need 4GB GPU memory and 6GB cpu memory with batch size=1 during training and 2GB GPU memory and 3GB cpu memory during testing (test with training dataset tov observe cpu memory).\nIn my pseudo kernel, I release the resource for inference berfore finetuning, and then I just use test data to finetune. Beause test images are less than train images, it should uses few memory to finetune than training with train dataset.\nFor finetuning , I just save the final weight file which need about 600MB, so I think 5GB is enough.\nNow, I am still confused for this error.
", "votes": null, "replies": [] } ] }, { "id": 902405, "author_name": "rajasekharam", "author_url": "", "post_date": "06/26/2020 05:59:51", "content": "@rilllove were you able to resolve this ? i am also facing the same issue. it is throwing me an error after 39 mins
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "849792": "Hi, I get error \"Notebook Exceeded Allowed Compute\" after I submit.\nI just try to use pseudo label by retraining my model. Any suggestion?", "849828": "GPU time limit (6 hours) or the more probable: memory issue. Try to lower the batch size during training.", "852149": "I set batch size to 1 but still get this error.\nI need 4GB GPU memory and 6GB cpu memory with batch size=1 during training my model in kaggle kernel. Moreover, I just save the latest weight file so I need 600MB disk storage.\nI do not know why I can train with 3422 images but can not finetune with about 1200 test images.", "852155": "Moreover, I got this error in 44 minutes after submit, it means that this error occured when start finetune with pseudo label.", "852249": "Take a look at [this notebook](https://www.kaggle.com/pestipeti/optimization-tips-faster-train-faster-inference), I made it for another competition, but you can use most of the tips here too.", "852463": "Thanks for sharing! After I check all of your tips, I notice I follow all most your tips like del all parameteres if I do not need that any more. \nAnd I have aslo observed the resource of my kernel during training and testing. \nAs I mentioned above, I just need 4GB GPU memory and 6GB cpu memory with batch size=1 during training and 2GB GPU memory and 3GB cpu memory during testing (test with training dataset tov observe cpu memory).\nIn my pseudo kernel, I release the resource for inference berfore finetuning, and then I just use test data to finetune. Beause test images are less than train images, it should uses few memory to finetune than training with train dataset.\nFor finetuning , I just save the final weight file which need about 600MB, so I think 5GB is enough.\nNow, I am still confused for this error.", "902405": "rilllove were you able to resolve this ? i am also facing the same issue. it is throwing me an error after 39 mins" }, "source": "meta" }