{
  "id": 50444,
  "title": "Help ! GPU Performance problem",
  "url": "/competitions/data-science-bowl-2018/discussion/50444",
  "author_name": "",
  "post_date": "2018-02-24T13:03:00.011554300Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi guys I met a problem with my 1080ti\nthe GPU performance is really weird GPU-Util is 100% but pwr is only 55/275w</p>\n\n<pre><code>+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |\n|  0%   43C    P5    55W / 275W |   7661MiB / 11177MiB |    100%      Default |\n+-------------------------------+----------------------+----------------------+\n|   1  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |\n|  0%   40C    P5    55W / 275W |   7479MiB / 11178MiB |    100%      Default |\n+-------------------------------+----------------------+----------------------+\n</code></pre>\n\n<p>the I check the clock by run <code>nvidia-smi -q -d CLOCK</code>, It's super low:</p>\n\n<pre><code>Clocks\n        Graphics                    : 227 MHz\n        SM                          : 227 MHz\n        Memory                      : 5005 MHz\n        Video                       : 810 MHz\n    Applications Clocks\n        Graphics                    : N/A\n        Memory                      : N/A\n    Default Applications Clocks\n        Graphics                    : N/A\n        Memory                      : N/A\n    Max Clocks\n        Graphics                    : 1999 MHz\n        SM                          : 1999 MHz\n        Memory                      : 5505 MHz\n        Video                       : 1620 MHz\n</code></pre>\n\n<p>then I run <code>nvidia-smi -q -d PERFORMANCE</code> :</p>\n\n<pre><code>Attached GPUs                       : 2\nGPU 00000000:04:00.0\n    Performance State               : P5\n    Clocks Throttle Reasons\n        Idle                        : Not Active\n        Applications Clocks Setting : Not Active\n        SW Power Cap                : Active\n        HW Slowdown                 : Not Active\n            HW Thermal Slowdown     : Not Active\n            HW Power Brake Slowdown : Not Active\n        Sync Boost                  : Not Active\n        SW Thermal Slowdown         : Not Active\n        Display Clock Setting       : Not Active\n\nGPU 00000000:83:00.0\n    Performance State               : P5\n    Clocks Throttle Reasons\n        Idle                        : Not Active\n        Applications Clocks Setting : Not Active\n        SW Power Cap                : Active\n        HW Slowdown                 : Not Active\n            HW Thermal Slowdown     : Not Active\n            HW Power Brake Slowdown : Not Active\n        Sync Boost                  : Not Active\n        SW Thermal Slowdown         : Not Active\n        Display Clock Setting       : Not Active\n</code></pre>\n\n<p>what is the <code>SW Power Cap</code> ？It means this issue limit my 1080ti's performance \nNow my traing time is super long , I just can only reach 1/3 of the speed of ohters who use 1070 when run the same code \nCould you give me some suggestions?</p>",
  "messages": [
    {
      "id": "287341",
      "postDate": "02/24/2018 13:03:00",
      "content": "<p>Hi guys I met a problem with my 1080ti\nthe GPU performance is really weird GPU-Util is 100% but pwr is only 55/275w</p>\n\n<pre><code>+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |\n|  0%   43C    P5    55W / 275W |   7661MiB / 11177MiB |    100%      Default |\n+-------------------------------+----------------------+----------------------+\n|   1  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |\n|  0%   40C    P5    55W / 275W |   7479MiB / 11178MiB |    100%      Default |\n+-------------------------------+----------------------+----------------------+\n</code></pre>\n\n<p>the I check the clock by run <code>nvidia-smi -q -d CLOCK</code>, It's super low:</p>\n\n<pre><code>Clocks\n        Graphics                    : 227 MHz\n        SM                          : 227 MHz\n        Memory                      : 5005 MHz\n        Video                       : 810 MHz\n    Applications Clocks\n        Graphics                    : N/A\n        Memory                      : N/A\n    Default Applications Clocks\n        Graphics                    : N/A\n        Memory                      : N/A\n    Max Clocks\n        Graphics                    : 1999 MHz\n        SM                          : 1999 MHz\n        Memory                      : 5505 MHz\n        Video                       : 1620 MHz\n</code></pre>\n\n<p>then I run <code>nvidia-smi -q -d PERFORMANCE</code> :</p>\n\n<pre><code>Attached GPUs                       : 2\nGPU 00000000:04:00.0\n    Performance State               : P5\n    Clocks Throttle Reasons\n        Idle                        : Not Active\n        Applications Clocks Setting : Not Active\n        SW Power Cap                : Active\n        HW Slowdown                 : Not Active\n            HW Thermal Slowdown     : Not Active\n            HW Power Brake Slowdown : Not Active\n        Sync Boost                  : Not Active\n        SW Thermal Slowdown         : Not Active\n        Display Clock Setting       : Not Active\n\nGPU 00000000:83:00.0\n    Performance State               : P5\n    Clocks Throttle Reasons\n        Idle                        : Not Active\n        Applications Clocks Setting : Not Active\n        SW Power Cap                : Active\n        HW Slowdown                 : Not Active\n            HW Thermal Slowdown     : Not Active\n            HW Power Brake Slowdown : Not Active\n        Sync Boost                  : Not Active\n        SW Thermal Slowdown         : Not Active\n        Display Clock Setting       : Not Active\n</code></pre>\n\n<p>what is the <code>SW Power Cap</code> ？It means this issue limit my 1080ti's performance \nNow my traing time is super long , I just can only reach 1/3 of the speed of ohters who use 1070 when run the same code \nCould you give me some suggestions?</p>",
      "rawMarkdown": "Hi guys I met a problem with my 1080ti\nthe GPU performance is really weird GPU-Util is 100% but pwr is only 55/275w\n\n    +-----------------------------------------------------------------------------+\n    | NVIDIA-SMI 390.25                 Driver Version: 390.25                    |\n    |-------------------------------+----------------------+----------------------+\n    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n    |===============================+======================+======================|\n    |   0  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |\n    |  0%   43C    P5    55W / 275W |   7661MiB / 11177MiB |    100%      Default |\n    +-------------------------------+----------------------+----------------------+\n    |   1  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |\n    |  0%   40C    P5    55W / 275W |   7479MiB / 11178MiB |    100%      Default |\n    +-------------------------------+----------------------+----------------------+\n\nthe I check the clock by run `nvidia-smi -q -d CLOCK`, It's super low:\n\n    Clocks\n            Graphics                    : 227 MHz\n            SM                          : 227 MHz\n            Memory                      : 5005 MHz\n            Video                       : 810 MHz\n        Applications Clocks\n            Graphics                    : N/A\n            Memory                      : N/A\n        Default Applications Clocks\n            Graphics                    : N/A\n            Memory                      : N/A\n        Max Clocks\n            Graphics                    : 1999 MHz\n            SM                          : 1999 MHz\n            Memory                      : 5505 MHz\n            Video                       : 1620 MHz\n\nthen I run `nvidia-smi -q -d PERFORMANCE` :\n\n    Attached GPUs                       : 2\n    GPU 00000000:04:00.0\n        Performance State               : P5\n        Clocks Throttle Reasons\n            Idle                        : Not Active\n            Applications Clocks Setting : Not Active\n            SW Power Cap                : Active\n            HW Slowdown                 : Not Active\n                HW Thermal Slowdown     : Not Active\n                HW Power Brake Slowdown : Not Active\n            Sync Boost                  : Not Active\n            SW Thermal Slowdown         : Not Active\n            Display Clock Setting       : Not Active\n    \n    GPU 00000000:83:00.0\n        Performance State               : P5\n        Clocks Throttle Reasons\n            Idle                        : Not Active\n            Applications Clocks Setting : Not Active\n            SW Power Cap                : Active\n            HW Slowdown                 : Not Active\n                HW Thermal Slowdown     : Not Active\n                HW Power Brake Slowdown : Not Active\n            Sync Boost                  : Not Active\n            SW Thermal Slowdown         : Not Active\n            Display Clock Setting       : Not Active\n\nwhat is the `SW Power Cap ` ？It means this issue limit my 1080ti's performance \nNow my traing time is super long , I just can only reach 1/3 of the speed of ohters who use 1070 when run the same code \nCould you give me some suggestions?",
      "votes": null
    },
    {
      "id": "287669",
      "postDate": "02/25/2018 16:22:20",
      "content": "<p>It would help if you could provide more info:  </p>\n\n<p>OS and OS version</p>\n\n<p>NVidia Driver version</p>\n\n<p>Deep Learning stack used, with version, eg Cuda Toolkit 9.1, Cudnn 7, TensorFlow 1.5, Keras xx</p>\n\n<p>Only someone with the same stack as you can help IMHO.</p>\n\n<p>I would ask the question on NVidia developer forum though, looks like you should have a look at your xorg configuration.</p>",
      "rawMarkdown": "It would help if you could provide more info:  \n\nOS and OS version\n\nNVidia Driver version\n\nDeep Learning stack used, with version, eg Cuda Toolkit 9.1, Cudnn 7, TensorFlow 1.5, Keras xx\n\nOnly someone with the same stack as you can help IMHO.\n\nI would ask the question on NVidia developer forum though, looks like you should have a look at your xorg configuration.",
      "votes": null
    },
    {
      "id": "360198",
      "postDate": "07/21/2018 20:34:43",
      "content": "<p>Hi!</p>\n\n<p>Have you been able to solve this? I have the exact same problem and tried for hours without any success.</p>\n\n<p>I use a single 1080Ti in a headless Ubuntu 16.04 server box. nvidia-smi reports performance state P5 at best. The power consumption never exceeds 42 Watts even when the GPU Utilization is 100% and SW Power Cap is active. </p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "Hi!\n\nHave you been able to solve this? I have the exact same problem and tried for hours without any success.\n\nI use a single 1080Ti in a headless Ubuntu 16.04 server box. nvidia-smi reports performance state P5 at best. The power consumption never exceeds 42 Watts even when the GPU Utilization is 100% and SW Power Cap is active. \n\nThanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 287669,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "02/25/2018 16:22:20",
      "content": "<p>It would help if you could provide more info:  </p>\n\n<p>OS and OS version</p>\n\n<p>NVidia Driver version</p>\n\n<p>Deep Learning stack used, with version, eg Cuda Toolkit 9.1, Cudnn 7, TensorFlow 1.5, Keras xx</p>\n\n<p>Only someone with the same stack as you can help IMHO.</p>\n\n<p>I would ask the question on NVidia developer forum though, looks like you should have a look at your xorg configuration.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 360198,
      "author_name": "bastiaigner",
      "author_url": "",
      "post_date": "07/21/2018 20:34:43",
      "content": "<p>Hi!</p>\n\n<p>Have you been able to solve this? I have the exact same problem and tried for hours without any success.</p>\n\n<p>I use a single 1080Ti in a headless Ubuntu 16.04 server box. nvidia-smi reports performance state P5 at best. The power consumption never exceeds 42 Watts even when the GPU Utilization is 100% and SW Power Cap is active. </p>\n\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "287341": "Hi guys I met a problem with my 1080ti\nthe GPU performance is really weird GPU-Util is 100% but pwr is only 55/275w\n\n    +-----------------------------------------------------------------------------+\n    | NVIDIA-SMI 390.25                 Driver Version: 390.25                    |\n    |-------------------------------+----------------------+----------------------+\n    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n    |===============================+======================+======================|\n    |   0  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |\n    |  0%   43C    P5    55W / 275W |   7661MiB / 11177MiB |    100%      Default |\n    +-------------------------------+----------------------+----------------------+\n    |   1  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |\n    |  0%   40C    P5    55W / 275W |   7479MiB / 11178MiB |    100%      Default |\n    +-------------------------------+----------------------+----------------------+\n\nthe I check the clock by run `nvidia-smi -q -d CLOCK`, It's super low:\n\n    Clocks\n            Graphics                    : 227 MHz\n            SM                          : 227 MHz\n            Memory                      : 5005 MHz\n            Video                       : 810 MHz\n        Applications Clocks\n            Graphics                    : N/A\n            Memory                      : N/A\n        Default Applications Clocks\n            Graphics                    : N/A\n            Memory                      : N/A\n        Max Clocks\n            Graphics                    : 1999 MHz\n            SM                          : 1999 MHz\n            Memory                      : 5505 MHz\n            Video                       : 1620 MHz\n\nthen I run `nvidia-smi -q -d PERFORMANCE` :\n\n    Attached GPUs                       : 2\n    GPU 00000000:04:00.0\n        Performance State               : P5\n        Clocks Throttle Reasons\n            Idle                        : Not Active\n            Applications Clocks Setting : Not Active\n            SW Power Cap                : Active\n            HW Slowdown                 : Not Active\n                HW Thermal Slowdown     : Not Active\n                HW Power Brake Slowdown : Not Active\n            Sync Boost                  : Not Active\n            SW Thermal Slowdown         : Not Active\n            Display Clock Setting       : Not Active\n    \n    GPU 00000000:83:00.0\n        Performance State               : P5\n        Clocks Throttle Reasons\n            Idle                        : Not Active\n            Applications Clocks Setting : Not Active\n            SW Power Cap                : Active\n            HW Slowdown                 : Not Active\n                HW Thermal Slowdown     : Not Active\n                HW Power Brake Slowdown : Not Active\n            Sync Boost                  : Not Active\n            SW Thermal Slowdown         : Not Active\n            Display Clock Setting       : Not Active\n\nwhat is the `SW Power Cap ` ？It means this issue limit my 1080ti's performance \nNow my traing time is super long , I just can only reach 1/3 of the speed of ohters who use 1070 when run the same code \nCould you give me some suggestions?",
    "287669": "It would help if you could provide more info:  \n\nOS and OS version\n\nNVidia Driver version\n\nDeep Learning stack used, with version, eg Cuda Toolkit 9.1, Cudnn 7, TensorFlow 1.5, Keras xx\n\nOnly someone with the same stack as you can help IMHO.\n\nI would ask the question on NVidia developer forum though, looks like you should have a look at your xorg configuration.",
    "360198": "Hi!\n\nHave you been able to solve this? I have the exact same problem and tried for hours without any success.\n\nI use a single 1080Ti in a headless Ubuntu 16.04 server box. nvidia-smi reports performance state P5 at best. The power consumption never exceeds 42 Watts even when the GPU Utilization is 100% and SW Power Cap is active. \n\nThanks!"
  },
  "source": "meta"
}