Hi guys\nThe discussion blow is a similar topic, but I confirm my cudnn installation is correct\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47912
\n\nMr. @MykolaSharhan says:
\n\n\n\n\nHi, using this notebook(link below) it took ~19 seconds/epoch with \n img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations \n were done on GTX1060.
\n \nhttps://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook
\n
But I need 150s to run the same code(nothing changed) with my 1080ti! Does anyone show the same situation as me?
", "messages": [ { "id": "278653", "postDate": "02/06/2018 16:57:41", "content": "Hi guys\nThe discussion blow is a similar topic, but I confirm my cudnn installation is correct\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47912
\n\nMr. @MykolaSharhan says:
\n\n\n\n\nHi, using this notebook(link below) it took ~19 seconds/epoch with \n img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations \n were done on GTX1060.
\n \nhttps://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook
\n
But I need 150s to run the same code(nothing changed) with my 1080ti! Does anyone show the same situation as me?
", "rawMarkdown": "Hi guys\nThe discussion blow is a similar topic, but I confirm my cudnn installation is correct\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47912\n\nMr. @MykolaSharhan says:\n\n> Hi, using this notebook(link below) it took ~19 seconds/epoch with \n> img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations \n> were done on GTX1060.\n> \n> https://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook\n\nBut I need 150s to run the same code(nothing changed) with my 1080ti! Does anyone show the same situation as me?", "votes": null }, { "id": "278699", "postDate": "02/06/2018 18:59:44", "content": "Do you use same versions of CUDA/cuDNN with GTX1060/GTX1080Ti? Try to upgrade them to latests versions.
", "rawMarkdown": "Do you use same versions of CUDA/cuDNN with GTX1060/GTX1080Ti? Try to upgrade them to latests versions.", "votes": null }, { "id": "278750", "postDate": "02/06/2018 21:45:08", "content": "Sounds like your CPU is running calcs. I dealt with some similar issues when I first set up my rig, there are three possible issues that may be happening.\n1.) Your PATH and LIBRARY bash profile variables(See NVIDIA doc for info) are not set correctly and Keras does not find them so reverts to CPU.(Try the Device Query CUDA sample program and make sure it is calling your 1080ti) Also make sure your IDE(if using) is using the bash profile.\n2.) CUDA did not initialize upon computer wake up, I had this issue one time and my IDE would not find CUDA thus running very slowly on the CPU. Restart you machine and check if that fixes it(you may have already)\n3.) If you have two GPUs the 1080ti may not get the default call, this can alos happen if the Driver version of your GPU doesn't match the CUDA driver version.
", "rawMarkdown": "Sounds like your CPU is running calcs. I dealt with some similar issues when I first set up my rig, there are three possible issues that may be happening.\n1.) Your PATH and LIBRARY bash profile variables(See NVIDIA doc for info) are not set correctly and Keras does not find them so reverts to CPU.(Try the Device Query CUDA sample program and make sure it is calling your 1080ti) Also make sure your IDE(if using) is using the bash profile.\n2.) CUDA did not initialize upon computer wake up, I had this issue one time and my IDE would not find CUDA thus running very slowly on the CPU. Restart you machine and check if that fixes it(you may have already)\n3.) If you have two GPUs the 1080ti may not get the default call, this can alos happen if the Driver version of your GPU doesn't match the CUDA driver version.", "votes": null }, { "id": "278951", "postDate": "02/07/2018 05:26:04", "content": "thx I will check it
", "rawMarkdown": "thx I will check it", "votes": null }, { "id": "278952", "postDate": "02/07/2018 05:27:43", "content": "thx a lot I will check them
", "rawMarkdown": "thx a lot I will check them", "votes": null }, { "id": "279116", "postDate": "02/07/2018 11:51:56", "content": "It can also be an issue with loading the data. If you're not doing this already, first load all training data into RAM. If you use a generator that needs to load the images from disk then the GPU will often be waiting for the CPU -- even worse when the images are on a HDD and not an SSD.
", "rawMarkdown": "It can also be an issue with loading the data. If you're not doing this already, first load all training data into RAM. If you use a generator that needs to load the images from disk then the GPU will often be waiting for the CPU -- even worse when the images are on a HDD and not an SSD.", "votes": null } ], "comments": [ { "id": 278699, "author_name": "alexkirnas", "author_url": "", "post_date": "02/06/2018 18:59:44", "content": "Do you use same versions of CUDA/cuDNN with GTX1060/GTX1080Ti? Try to upgrade them to latests versions.
", "votes": null, "replies": [ { "id": 278951, "author_name": "bintan", "author_url": "", "post_date": "02/07/2018 05:26:04", "content": "thx I will check it
", "votes": null, "replies": [] } ] }, { "id": 278750, "author_name": "ianrowan", "author_url": "", "post_date": "02/06/2018 21:45:08", "content": "Sounds like your CPU is running calcs. I dealt with some similar issues when I first set up my rig, there are three possible issues that may be happening.\n1.) Your PATH and LIBRARY bash profile variables(See NVIDIA doc for info) are not set correctly and Keras does not find them so reverts to CPU.(Try the Device Query CUDA sample program and make sure it is calling your 1080ti) Also make sure your IDE(if using) is using the bash profile.\n2.) CUDA did not initialize upon computer wake up, I had this issue one time and my IDE would not find CUDA thus running very slowly on the CPU. Restart you machine and check if that fixes it(you may have already)\n3.) If you have two GPUs the 1080ti may not get the default call, this can alos happen if the Driver version of your GPU doesn't match the CUDA driver version.
", "votes": null, "replies": [ { "id": 278952, "author_name": "bintan", "author_url": "", "post_date": "02/07/2018 05:27:43", "content": "thx a lot I will check them
", "votes": null, "replies": [] } ] }, { "id": 279116, "author_name": "humananalog", "author_url": "", "post_date": "02/07/2018 11:51:56", "content": "It can also be an issue with loading the data. If you're not doing this already, first load all training data into RAM. If you use a generator that needs to load the images from disk then the GPU will often be waiting for the CPU -- even worse when the images are on a HDD and not an SSD.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "278653": "Hi guys\nThe discussion blow is a similar topic, but I confirm my cudnn installation is correct\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47912\n\nMr. @MykolaSharhan says:\n\n> Hi, using this notebook(link below) it took ~19 seconds/epoch with \n> img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations \n> were done on GTX1060.\n> \n> https://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook\n\nBut I need 150s to run the same code(nothing changed) with my 1080ti! Does anyone show the same situation as me?", "278699": "Do you use same versions of CUDA/cuDNN with GTX1060/GTX1080Ti? Try to upgrade them to latests versions.", "278750": "Sounds like your CPU is running calcs. I dealt with some similar issues when I first set up my rig, there are three possible issues that may be happening.\n1.) Your PATH and LIBRARY bash profile variables(See NVIDIA doc for info) are not set correctly and Keras does not find them so reverts to CPU.(Try the Device Query CUDA sample program and make sure it is calling your 1080ti) Also make sure your IDE(if using) is using the bash profile.\n2.) CUDA did not initialize upon computer wake up, I had this issue one time and my IDE would not find CUDA thus running very slowly on the CPU. Restart you machine and check if that fixes it(you may have already)\n3.) If you have two GPUs the 1080ti may not get the default call, this can alos happen if the Driver version of your GPU doesn't match the CUDA driver version.", "278951": "thx I will check it", "278952": "thx a lot I will check them", "279116": "It can also be an issue with loading the data. If you're not doing this already, first load all training data into RAM. If you use a generator that needs to load the images from disk then the GPU will often be waiting for the CPU -- even worse when the images are on a HDD and not an SSD." }, "source": "meta" }