Anybody knows what the correlation is between GPU quota and kernel runtime? According to my experience if kernel runtime is t then GPU quota decreases at least by 2 × t. Why?
", "messages": [ { "id": "684724", "postDate": "11/30/2019 11:16:00", "content": "Anybody knows what the correlation is between GPU quota and kernel runtime? According to my experience if kernel runtime is t then GPU quota decreases at least by 2 × t. Why?
", "rawMarkdown": "Anybody knows what the correlation is between GPU quota and kernel runtime? According to my experience if kernel runtime is t then GPU quota decreases at least by 2 × t. Why?", "votes": null }, { "id": "689044", "postDate": "12/06/2019 11:44:08", "content": "No, I don't think so, GPU quota is fixed irrespective of the kernel time. (If I find out otherwise I'll update you.)
", "rawMarkdown": "No, I don't think so, GPU quota is fixed irrespective of the kernel time. (If I find out otherwise I'll update you.)", "votes": null }, { "id": "689086", "postDate": "12/06/2019 13:01:17", "content": "OK, I try to be more specific. So when I make a kernel submission or in other words when I commit my kernel, it takes - let's say - 1800 sec. It is logged by kaggle and measured by my stopwatch being 1800 sec. That is fine. Then I check out my GPU quota and that is decreased by one hour (3600 sec). Anybody with similar experiment?
", "rawMarkdown": "OK, I try to be more specific. So when I make a kernel submission or in other words when I commit my kernel, it takes - let's say - 1800 sec. It is logged by kaggle and measured by my stopwatch being 1800 sec. That is fine. Then I check out my GPU quota and that is decreased by one hour (3600 sec). Anybody with similar experiment?", "votes": null }, { "id": "689095", "postDate": "12/06/2019 13:09:36", "content": "Kernels/notebooks on Kaggle take time to commit, Some times it happens but I haven't timed it.
", "rawMarkdown": "Kernels/notebooks on Kaggle take time to commit, Some times it happens but I haven't timed it.", "votes": null }, { "id": "689165", "postDate": "12/06/2019 15:06:34", "content": "After the commit you need to turn off the GPU in the session where you are editing the notebook. Otherwise both the commit and the edit session are using the GPU.
", "rawMarkdown": "After the commit you need to turn off the GPU in the session where you are editing the notebook. Otherwise both the commit and the edit session are using the GPU.", "votes": null }, { "id": "689184", "postDate": "12/06/2019 15:41:33", "content": "Hi MichaelP, nice works you do here at kaggle. Yes, I do switch GPU on right before committing and I do switch it off right after kernel committed. I try not to waste a minute of my GPU quota. So when you do a commit - your GPU quota decreases with the exact same time amount as of your commit runtime?
", "rawMarkdown": "Hi MichaelP, nice works you do here at kaggle. Yes, I do switch GPU on right before committing and I do switch it off right after kernel committed. I try not to waste a minute of my GPU quota. So when you do a commit - your GPU quota decreases with the exact same time amount as of your commit runtime?", "votes": null }, { "id": "689232", "postDate": "12/06/2019 17:21:32", "content": "Thanks. When I press commit there is a popup window that shows details of the commit. I wait for a few seconds until the first few lines of the log appear to show that something is happening with the commit then I close the popup and turn off the GPU. The GPU for the commit is still on. I'm sure that it doesn't use my quota at double the time.
", "rawMarkdown": "Thanks. When I press commit there is a popup window that shows details of the commit. I wait for a few seconds until the first few lines of the log appear to show that something is happening with the commit then I close the popup and turn off the GPU. The GPU for the commit is still on. I'm sure that it doesn't use my quota at double the time.", "votes": null }, { "id": "689256", "postDate": "12/06/2019 17:54:21", "content": "I thought GPU should be switched on while commit runs. I will test tomorrow what you say. Many thanks MichaelP.
", "rawMarkdown": "I thought GPU should be switched on while commit runs. I will test tomorrow what you say. Many thanks MichaelP.", "votes": null }, { "id": "689601", "postDate": "12/07/2019 06:27:43", "content": "You are welcome. Let me know if this doesn't work for you. You could test with just a few epochs of a model so the test does not take too long. The GPU definitely needs to be on when you first do the commit. At that point the commit copies the settings of the edit instance to launch a new independent instance. Once the log is being populated you know that instance is running. Then you can turn off the GPU in the edit session because the commit session is now independent. At least that is my understanding based on this thread and this more general page.
", "rawMarkdown": "You are welcome. Let me know if this doesn't work for you. You could test with just a few epochs of a model so the test does not take too long. The GPU definitely needs to be on when you first do the commit. At that point the commit copies the settings of the edit instance to launch a new independent instance. Once the log is being populated you know that instance is running. Then you can turn off the GPU in the edit session because the commit session is now independent. At least that is my understanding based on this [thread](https://www.kaggle.com/general/108481#624386) and this more general [page](https://www.kaggle.com/page/GPU-tips-and-tricks).", "votes": null }, { "id": "690095", "postDate": "12/08/2019 00:12:01", "content": "Does this have to do with the fact that it has to run once via commit, and then once via submission? I'm current trying to figure out this workaround for that problem ( https://www.kaggle.com/c/instant-gratification/discussion/94908 ). Perhaps I'm too skittish at the moment to just try it and possibly drain some resources/submissions.
", "rawMarkdown": "Does this have to do with the fact that it has to run once via commit, and then once via submission? I'm current trying to figure out this workaround for that problem ( https://www.kaggle.com/c/instant-gratification/discussion/94908 ). Perhaps I'm too skittish at the moment to just try it and possibly drain some resources/submissions.", "votes": null }, { "id": "690983", "postDate": "12/09/2019 12:49:17", "content": "Yesss, it does work. Thanks for the links, they are very useful.
", "rawMarkdown": "Yesss, it does work. Thanks for the links, they are very useful.", "votes": null }, { "id": "690991", "postDate": "12/09/2019 13:04:21", "content": "Sorry, I may was not clear. So when I did my commit I had an edit session with GPU switched on and I had a commit session with - probably another physical - GPU switched on. I did not know that GPU may be switched off in the edit session after commit session started. What I did stupidly is that I left GPU switched on in edit session and stared at the screen and waited for the end of the commit and when it was done I switched GPU off. So that is why my GPU quota decreased 2 times more than I expected.
", "rawMarkdown": "Sorry, I may was not clear. So when I did my commit I had an edit session with GPU switched on and I had a commit session with - probably another physical - GPU switched on. I did not know that GPU may be switched off in the edit session after commit session started. What I did stupidly is that I left GPU switched on in edit session and stared at the screen and waited for the end of the commit and when it was done I switched GPU off. So that is why my GPU quota decreased 2 times more than I expected.", "votes": null } ], "comments": [ { "id": 689044, "author_name": "amitjslearn", "author_url": "", "post_date": "12/06/2019 11:44:08", "content": "No, I don't think so, GPU quota is fixed irrespective of the kernel time. (If I find out otherwise I'll update you.)
", "votes": null, "replies": [ { "id": 689086, "author_name": "nandor65", "author_url": "", "post_date": "12/06/2019 13:01:17", "content": "OK, I try to be more specific. So when I make a kernel submission or in other words when I commit my kernel, it takes - let's say - 1800 sec. It is logged by kaggle and measured by my stopwatch being 1800 sec. That is fine. Then I check out my GPU quota and that is decreased by one hour (3600 sec). Anybody with similar experiment?
", "votes": null, "replies": [] }, { "id": 689095, "author_name": "amitjslearn", "author_url": "", "post_date": "12/06/2019 13:09:36", "content": "Kernels/notebooks on Kaggle take time to commit, Some times it happens but I haven't timed it.
", "votes": null, "replies": [] }, { "id": 689165, "author_name": "datahobbit", "author_url": "", "post_date": "12/06/2019 15:06:34", "content": "After the commit you need to turn off the GPU in the session where you are editing the notebook. Otherwise both the commit and the edit session are using the GPU.
", "votes": null, "replies": [] }, { "id": 689184, "author_name": "nandor65", "author_url": "", "post_date": "12/06/2019 15:41:33", "content": "Hi MichaelP, nice works you do here at kaggle. Yes, I do switch GPU on right before committing and I do switch it off right after kernel committed. I try not to waste a minute of my GPU quota. So when you do a commit - your GPU quota decreases with the exact same time amount as of your commit runtime?
", "votes": null, "replies": [] }, { "id": 689232, "author_name": "datahobbit", "author_url": "", "post_date": "12/06/2019 17:21:32", "content": "Thanks. When I press commit there is a popup window that shows details of the commit. I wait for a few seconds until the first few lines of the log appear to show that something is happening with the commit then I close the popup and turn off the GPU. The GPU for the commit is still on. I'm sure that it doesn't use my quota at double the time.
", "votes": null, "replies": [] }, { "id": 689256, "author_name": "nandor65", "author_url": "", "post_date": "12/06/2019 17:54:21", "content": "I thought GPU should be switched on while commit runs. I will test tomorrow what you say. Many thanks MichaelP.
", "votes": null, "replies": [] }, { "id": 689601, "author_name": "datahobbit", "author_url": "", "post_date": "12/07/2019 06:27:43", "content": "You are welcome. Let me know if this doesn't work for you. You could test with just a few epochs of a model so the test does not take too long. The GPU definitely needs to be on when you first do the commit. At that point the commit copies the settings of the edit instance to launch a new independent instance. Once the log is being populated you know that instance is running. Then you can turn off the GPU in the edit session because the commit session is now independent. At least that is my understanding based on this thread and this more general page.
", "votes": null, "replies": [] }, { "id": 690983, "author_name": "nandor65", "author_url": "", "post_date": "12/09/2019 12:49:17", "content": "Yesss, it does work. Thanks for the links, they are very useful.
", "votes": null, "replies": [] } ] }, { "id": 690095, "author_name": "egrimley", "author_url": "", "post_date": "12/08/2019 00:12:01", "content": "Does this have to do with the fact that it has to run once via commit, and then once via submission? I'm current trying to figure out this workaround for that problem ( https://www.kaggle.com/c/instant-gratification/discussion/94908 ). Perhaps I'm too skittish at the moment to just try it and possibly drain some resources/submissions.
", "votes": null, "replies": [ { "id": 690991, "author_name": "nandor65", "author_url": "", "post_date": "12/09/2019 13:04:21", "content": "Sorry, I may was not clear. So when I did my commit I had an edit session with GPU switched on and I had a commit session with - probably another physical - GPU switched on. I did not know that GPU may be switched off in the edit session after commit session started. What I did stupidly is that I left GPU switched on in edit session and stared at the screen and waited for the end of the commit and when it was done I switched GPU off. So that is why my GPU quota decreased 2 times more than I expected.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "684724": "Anybody knows what the correlation is between GPU quota and kernel runtime? According to my experience if kernel runtime is t then GPU quota decreases at least by 2 × t. Why?", "689044": "No, I don't think so, GPU quota is fixed irrespective of the kernel time. (If I find out otherwise I'll update you.)", "689086": "OK, I try to be more specific. So when I make a kernel submission or in other words when I commit my kernel, it takes - let's say - 1800 sec. It is logged by kaggle and measured by my stopwatch being 1800 sec. That is fine. Then I check out my GPU quota and that is decreased by one hour (3600 sec). Anybody with similar experiment?", "689095": "Kernels/notebooks on Kaggle take time to commit, Some times it happens but I haven't timed it.", "689165": "After the commit you need to turn off the GPU in the session where you are editing the notebook. Otherwise both the commit and the edit session are using the GPU.", "689184": "Hi MichaelP, nice works you do here at kaggle. Yes, I do switch GPU on right before committing and I do switch it off right after kernel committed. I try not to waste a minute of my GPU quota. So when you do a commit - your GPU quota decreases with the exact same time amount as of your commit runtime?", "689232": "Thanks. When I press commit there is a popup window that shows details of the commit. I wait for a few seconds until the first few lines of the log appear to show that something is happening with the commit then I close the popup and turn off the GPU. The GPU for the commit is still on. I'm sure that it doesn't use my quota at double the time.", "689256": "I thought GPU should be switched on while commit runs. I will test tomorrow what you say. Many thanks MichaelP.", "689601": "You are welcome. Let me know if this doesn't work for you. You could test with just a few epochs of a model so the test does not take too long. The GPU definitely needs to be on when you first do the commit. At that point the commit copies the settings of the edit instance to launch a new independent instance. Once the log is being populated you know that instance is running. Then you can turn off the GPU in the edit session because the commit session is now independent. At least that is my understanding based on this [thread](https://www.kaggle.com/general/108481#624386) and this more general [page](https://www.kaggle.com/page/GPU-tips-and-tricks).", "690095": "Does this have to do with the fact that it has to run once via commit, and then once via submission? I'm current trying to figure out this workaround for that problem ( https://www.kaggle.com/c/instant-gratification/discussion/94908 ). Perhaps I'm too skittish at the moment to just try it and possibly drain some resources/submissions.", "690983": "Yesss, it does work. Thanks for the links, they are very useful.", "690991": "Sorry, I may was not clear. So when I did my commit I had an edit session with GPU switched on and I had a commit session with - probably another physical - GPU switched on. I did not know that GPU may be switched off in the edit session after commit session started. What I did stupidly is that I left GPU switched on in edit session and stared at the screen and waited for the end of the commit and when it was done I switched GPU off. So that is why my GPU quota decreased 2 times more than I expected." }, "source": "meta" }