{
  "id": 587773,
  "title": "Why does a better graphics card actually slow down training speed?",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/587773",
  "author_name": "",
  "post_date": "2025-07-02T16:18:00.951770800Z",
  "votes": 4,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I used 3090 for training and found that it was slower than the p100 graphics card with the same parameters. Can anyone tell me why? Is it a pytorch version issue or a graphics card architecture issue?<br>\nReference baseline:<br>\n<a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/myso1987/cmi3-pyroch-baseline-model-add-aug-folds</a></p>",
  "messages": [
    {
      "id": "3239261",
      "postDate": "07/02/2025 16:18:00",
      "content": "<p>I used 3090 for training and found that it was slower than the p100 graphics card with the same parameters. Can anyone tell me why? Is it a pytorch version issue or a graphics card architecture issue?<br>\nReference baseline:<br>\n<a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/myso1987/cmi3-pyroch-baseline-model-add-aug-folds</a></p>",
      "rawMarkdown": "I used 3090 for training and found that it was slower than the p100 graphics card with the same parameters. Can anyone tell me why? Is it a pytorch version issue or a graphics card architecture issue?\nReference baseline:\n[https://www.kaggle.com/code/myso1987/cmi3-pyroch-baseline-model-add-aug-folds](url)",
      "votes": null
    },
    {
      "id": "3239314",
      "postDate": "07/02/2025 17:28:20",
      "content": "<p>Your observation makes sense. In fact, for this competition (CMI 2025), GPU performance (e.g., 3090 vs. P100) might not be the main bottleneck—instead, CPU-bound operations like data preprocessing or I/O could be the limiting factor.<br>\nMany baselines rely heavily on the CPU for tasks such as data loading, FIR filtering, and feature engineering before the data even reaches the GPU. So even with a powerful GPU like the 3090, you may not see a speed-up if:</p>\n<ul>\n<li>Your model includes complex CPU-side preprocessing.</li>\n<li>Your dataloader is slow (e.g., num_workers=0).</li>\n<li>The 3090 host machine actually has a weaker CPU.</li>\n<li>The batch size is too small to utilize the GPU fully.</li>\n</ul>",
      "rawMarkdown": "Your observation makes sense. In fact, for this competition (CMI 2025), GPU performance (e.g., 3090 vs. P100) might not be the main bottleneck—instead, CPU-bound operations like data preprocessing or I/O could be the limiting factor.\nMany baselines rely heavily on the CPU for tasks such as data loading, FIR filtering, and feature engineering before the data even reaches the GPU. So even with a powerful GPU like the 3090, you may not see a speed-up if:\n- Your model includes complex CPU-side preprocessing.\n- Your dataloader is slow (e.g., num_workers=0).\n- The 3090 host machine actually has a weaker CPU.\n- The batch size is too small to utilize the GPU fully.",
      "votes": null
    },
    {
      "id": "3239490",
      "postDate": "07/02/2025 21:24:56",
      "content": "<p>Interesting observation !!</p>\n<p>Sometimes newer GPUs like the 3090 can underperform due to mismatched CUDA/cuDNN versions, driver issues, or framework optimizations not being fully supported yet.</p>\n<p>It's not always raw power—compatibility and architecture matter too. Try checking PyTorch + CUDA versions and make sure they’re tuned for the 3090.<br>\nCurious to hear if others faced the same! 🔍</p>",
      "rawMarkdown": "Interesting observation !!\n\nSometimes newer GPUs like the 3090 can underperform due to mismatched CUDA/cuDNN versions, driver issues, or framework optimizations not being fully supported yet.\n\n It's not always raw power—compatibility and architecture matter too. Try checking PyTorch + CUDA versions and make sure they’re tuned for the 3090.\nCurious to hear if others faced the same! 🔍",
      "votes": null
    },
    {
      "id": "3239964",
      "postDate": "07/03/2025 10:21:41",
      "content": "<p>I also compared a GPU with a power consumption of 300W with a GPU with a power consumption of 575W.<br>\nThe 300W GPU was only using 100W, but the 575W GPU was using 200W, so the calculation speed was twice as fast.</p>",
      "rawMarkdown": "I also compared a GPU with a power consumption of 300W with a GPU with a power consumption of 575W.\nThe 300W GPU was only using 100W, but the 575W GPU was using 200W, so the calculation speed was twice as fast.",
      "votes": null
    },
    {
      "id": "3240552",
      "postDate": "07/04/2025 00:56:18",
      "content": "<p>I've also noticed this issue. Even the CV scores of the same model and feature engineering are much better when I run them locally offline compared to the Kaggle environment. My local CPU is a Gold Xeon with 128GB RAM.</p>",
      "rawMarkdown": "I've also noticed this issue. Even the CV scores of the same model and feature engineering are much better when I run them locally offline compared to the Kaggle environment. My local CPU is a Gold Xeon with 128GB RAM.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3239314,
      "author_name": "wayne127",
      "author_url": "",
      "post_date": "07/02/2025 17:28:20",
      "content": "<p>Your observation makes sense. In fact, for this competition (CMI 2025), GPU performance (e.g., 3090 vs. P100) might not be the main bottleneck—instead, CPU-bound operations like data preprocessing or I/O could be the limiting factor.<br>\nMany baselines rely heavily on the CPU for tasks such as data loading, FIR filtering, and feature engineering before the data even reaches the GPU. So even with a powerful GPU like the 3090, you may not see a speed-up if:</p>\n<ul>\n<li>Your model includes complex CPU-side preprocessing.</li>\n<li>Your dataloader is slow (e.g., num_workers=0).</li>\n<li>The 3090 host machine actually has a weaker CPU.</li>\n<li>The batch size is too small to utilize the GPU fully.</li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 3240552,
          "author_name": "sweetyheehee",
          "author_url": "",
          "post_date": "07/04/2025 00:56:18",
          "content": "<p>I've also noticed this issue. Even the CV scores of the same model and feature engineering are much better when I run them locally offline compared to the Kaggle environment. My local CPU is a Gold Xeon with 128GB RAM.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3239490,
      "author_name": "shakhoyatshujon",
      "author_url": "",
      "post_date": "07/02/2025 21:24:56",
      "content": "<p>Interesting observation !!</p>\n<p>Sometimes newer GPUs like the 3090 can underperform due to mismatched CUDA/cuDNN versions, driver issues, or framework optimizations not being fully supported yet.</p>\n<p>It's not always raw power—compatibility and architecture matter too. Try checking PyTorch + CUDA versions and make sure they’re tuned for the 3090.<br>\nCurious to hear if others faced the same! 🔍</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3239964,
      "author_name": "ttyn4519",
      "author_url": "",
      "post_date": "07/03/2025 10:21:41",
      "content": "<p>I also compared a GPU with a power consumption of 300W with a GPU with a power consumption of 575W.<br>\nThe 300W GPU was only using 100W, but the 575W GPU was using 200W, so the calculation speed was twice as fast.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3239261": "I used 3090 for training and found that it was slower than the p100 graphics card with the same parameters. Can anyone tell me why? Is it a pytorch version issue or a graphics card architecture issue?\nReference baseline:\n[https://www.kaggle.com/code/myso1987/cmi3-pyroch-baseline-model-add-aug-folds](url)",
    "3239314": "Your observation makes sense. In fact, for this competition (CMI 2025), GPU performance (e.g., 3090 vs. P100) might not be the main bottleneck—instead, CPU-bound operations like data preprocessing or I/O could be the limiting factor.\nMany baselines rely heavily on the CPU for tasks such as data loading, FIR filtering, and feature engineering before the data even reaches the GPU. So even with a powerful GPU like the 3090, you may not see a speed-up if:\n- Your model includes complex CPU-side preprocessing.\n- Your dataloader is slow (e.g., num_workers=0).\n- The 3090 host machine actually has a weaker CPU.\n- The batch size is too small to utilize the GPU fully.",
    "3239490": "Interesting observation !!\n\nSometimes newer GPUs like the 3090 can underperform due to mismatched CUDA/cuDNN versions, driver issues, or framework optimizations not being fully supported yet.\n\n It's not always raw power—compatibility and architecture matter too. Try checking PyTorch + CUDA versions and make sure they’re tuned for the 3090.\nCurious to hear if others faced the same! 🔍",
    "3239964": "I also compared a GPU with a power consumption of 300W with a GPU with a power consumption of 575W.\nThe 300W GPU was only using 100W, but the 575W GPU was using 200W, so the calculation speed was twice as fast.",
    "3240552": "I've also noticed this issue. Even the CV scores of the same model and feature engineering are much better when I run them locally offline compared to the Kaggle environment. My local CPU is a Gold Xeon with 128GB RAM."
  },
  "source": "meta"
}