{
  "id": 47912,
  "title": "How much time do you need for training U-net",
  "url": "/competitions/data-science-bowl-2018/discussion/47912",
  "author_name": "",
  "post_date": "2018-01-20T13:47:46.001842400Z",
  "votes": null,
  "comment_count": 9,
  "views": 0,
  "content": "<p>I used around 1 hour to train my model, u-net. And I am wondering how much time do you need for a training. </p>",
  "messages": [
    {
      "id": "271456",
      "postDate": "01/20/2018 13:47:46",
      "content": "<p>I used around 1 hour to train my model, u-net. And I am wondering how much time do you need for a training. </p>",
      "rawMarkdown": "I used around 1 hour to train my model, u-net. And I am wondering how much time do you need for a training.",
      "votes": null
    },
    {
      "id": "271463",
      "postDate": "01/20/2018 14:07:58",
      "content": "<p>it seems that this competition won't consume too much GPU resource, did you use any augmentation or CV?</p>",
      "rawMarkdown": "it seems that this competition won't consume too much GPU resource, did you use any augmentation or CV?",
      "votes": null
    },
    {
      "id": "271468",
      "postDate": "01/20/2018 14:13:57",
      "content": "<p>Yes. I didn't use any augmentation or CV. Here, I just setup the pipeline and estimate the GPU time for this competition.</p>",
      "rawMarkdown": "Yes. I didn't use any augmentation or CV. Here, I just setup the pipeline and estimate the GPU time for this competition.",
      "votes": null
    },
    {
      "id": "271884",
      "postDate": "01/21/2018 19:37:03",
      "content": "<p>Hi, using this notebook(link below) it took ~19 seconds/epoch with img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations were done on GTX1060.</p>\n\n<p><a href=\"https://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook\">https://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook</a></p>",
      "rawMarkdown": "Hi, using this notebook(link below) it took ~19 seconds/epoch with img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations were done on GTX1060.\n\nhttps://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook",
      "votes": null
    },
    {
      "id": "273557",
      "postDate": "01/24/2018 19:13:26",
      "content": "<p>Oops, with one 1080 ti the same notebook took me 1000 sec/epoch. Has anyone had the same/ similar experience or known any solutions? Pretty sure I missed something but just didn't know what it is.....</p>",
      "rawMarkdown": "Oops, with one 1080 ti the same notebook took me 1000 sec/epoch. Has anyone had the same/ similar experience or known any solutions? Pretty sure I missed something but just didn't know what it is.....",
      "votes": null
    },
    {
      "id": "273627",
      "postDate": "01/24/2018 22:02:09",
      "content": "<p>Are you sure the GPU is being used?</p>",
      "rawMarkdown": "Are you sure the GPU is being used?",
      "votes": null
    },
    {
      "id": "273669",
      "postDate": "01/25/2018 00:34:42",
      "content": "<p>Of course, Nvidia-smi shows ~90%GPU usage</p>",
      "rawMarkdown": "Of course, Nvidia-smi shows ~90%GPU usage",
      "votes": null
    },
    {
      "id": "274418",
      "postDate": "01/26/2018 15:24:15",
      "content": "<p>Wow. My Unet training times per epoch at between 10s and 45s and my GPU is older. Maybe there is a bottleneck somewhere else? Have you found the problem? What happens if you train a different neural net on other data, do you get more normal times?</p>",
      "rawMarkdown": "Wow. My Unet training times per epoch at between 10s and 45s and my GPU is older. Maybe there is a bottleneck somewhere else? Have you found the problem? What happens if you train a different neural net on other data, do you get more normal times?",
      "votes": null
    },
    {
      "id": "274611",
      "postDate": "01/26/2018 22:00:15",
      "content": "<p>Finally figured out what's been wrong here. Basically what happened was I did install cudnn but might have screwed up with cudnn. More specifically, I suspected that I accidentally skipped one manual step to copy cudnn.h to /usr/include.</p>\n\n<p>In that case, cudnn was not being used to accelerate computation for some specific layers such as conv2d but GPU was still being used for other types of units in the Unet architecture and that sort of explained why I saw 100% GPU usage but the overall speed was way slower than it's supposed to be.</p>\n\n<p>After installing cudnn and re-compile TF the speed seems to be normal: 25-35 seconds/ epoch as compared to 1000 seconds/epoch without cudnn.</p>\n\n<p>PS, I only found this when I was trying to install Caffe2 wich requires cudnn and threw an error explicitly but I doubt if the installation of TS would stop without cudnn which I can hardly call a caveat but it would be really nice to throw at least a warning during the installation imho.</p>\n\n<p>Thanks,</p>",
      "rawMarkdown": "Finally figured out what's been wrong here. Basically what happened was I did install cudnn but might have screwed up with cudnn. More specifically, I suspected that I accidentally skipped one manual step to copy cudnn.h to /usr/include.\n\nIn that case, cudnn was not being used to accelerate computation for some specific layers such as conv2d but GPU was still being used for other types of units in the Unet architecture and that sort of explained why I saw 100% GPU usage but the overall speed was way slower than it's supposed to be.\n\nAfter installing cudnn and re-compile TF the speed seems to be normal: 25-35 seconds/ epoch as compared to 1000 seconds/epoch without cudnn.\n\nPS, I only found this when I was trying to install Caffe2 wich requires cudnn and threw an error explicitly but I doubt if the installation of TS would stop without cudnn which I can hardly call a caveat but it would be really nice to throw at least a warning during the installation imho.\n\nThanks,",
      "votes": null
    },
    {
      "id": "275577",
      "postDate": "01/29/2018 14:16:18",
      "content": "<p>Glad to see you got it!</p>",
      "rawMarkdown": "Glad to see you got it!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 271463,
      "author_name": "plainsailing",
      "author_url": "",
      "post_date": "01/20/2018 14:07:58",
      "content": "<p>it seems that this competition won't consume too much GPU resource, did you use any augmentation or CV?</p>",
      "votes": null,
      "replies": [
        {
          "id": 271468,
          "author_name": "zhijianli",
          "author_url": "",
          "post_date": "01/20/2018 14:13:57",
          "content": "<p>Yes. I didn't use any augmentation or CV. Here, I just setup the pipeline and estimate the GPU time for this competition.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 271884,
      "author_name": "nhargan",
      "author_url": "",
      "post_date": "01/21/2018 19:37:03",
      "content": "<p>Hi, using this notebook(link below) it took ~19 seconds/epoch with img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations were done on GTX1060.</p>\n\n<p><a href=\"https://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook\">https://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 273557,
          "author_name": "chriscc",
          "author_url": "",
          "post_date": "01/24/2018 19:13:26",
          "content": "<p>Oops, with one 1080 ti the same notebook took me 1000 sec/epoch. Has anyone had the same/ similar experience or known any solutions? Pretty sure I missed something but just didn't know what it is.....</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 273627,
          "author_name": "malr87",
          "author_url": "",
          "post_date": "01/24/2018 22:02:09",
          "content": "<p>Are you sure the GPU is being used?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 273669,
          "author_name": "chriscc",
          "author_url": "",
          "post_date": "01/25/2018 00:34:42",
          "content": "<p>Of course, Nvidia-smi shows ~90%GPU usage</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 274418,
          "author_name": "malr87",
          "author_url": "",
          "post_date": "01/26/2018 15:24:15",
          "content": "<p>Wow. My Unet training times per epoch at between 10s and 45s and my GPU is older. Maybe there is a bottleneck somewhere else? Have you found the problem? What happens if you train a different neural net on other data, do you get more normal times?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 274611,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "01/26/2018 22:00:15",
      "content": "<p>Finally figured out what's been wrong here. Basically what happened was I did install cudnn but might have screwed up with cudnn. More specifically, I suspected that I accidentally skipped one manual step to copy cudnn.h to /usr/include.</p>\n\n<p>In that case, cudnn was not being used to accelerate computation for some specific layers such as conv2d but GPU was still being used for other types of units in the Unet architecture and that sort of explained why I saw 100% GPU usage but the overall speed was way slower than it's supposed to be.</p>\n\n<p>After installing cudnn and re-compile TF the speed seems to be normal: 25-35 seconds/ epoch as compared to 1000 seconds/epoch without cudnn.</p>\n\n<p>PS, I only found this when I was trying to install Caffe2 wich requires cudnn and threw an error explicitly but I doubt if the installation of TS would stop without cudnn which I can hardly call a caveat but it would be really nice to throw at least a warning during the installation imho.</p>\n\n<p>Thanks,</p>",
      "votes": null,
      "replies": [
        {
          "id": 275577,
          "author_name": "malr87",
          "author_url": "",
          "post_date": "01/29/2018 14:16:18",
          "content": "<p>Glad to see you got it!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "271456": "I used around 1 hour to train my model, u-net. And I am wondering how much time do you need for a training.",
    "271463": "it seems that this competition won't consume too much GPU resource, did you use any augmentation or CV?",
    "271468": "Yes. I didn't use any augmentation or CV. Here, I just setup the pipeline and estimate the GPU time for this competition.",
    "271884": "Hi, using this notebook(link below) it took ~19 seconds/epoch with img_size = 128 and ~80 seconds/epoch with img_size=512. Calculations were done on GTX1060.\n\nhttps://www.kaggle.com/takuok/keras-generator-starter-lb-0-304/notebook",
    "273557": "Oops, with one 1080 ti the same notebook took me 1000 sec/epoch. Has anyone had the same/ similar experience or known any solutions? Pretty sure I missed something but just didn't know what it is.....",
    "273627": "Are you sure the GPU is being used?",
    "273669": "Of course, Nvidia-smi shows ~90%GPU usage",
    "274418": "Wow. My Unet training times per epoch at between 10s and 45s and my GPU is older. Maybe there is a bottleneck somewhere else? Have you found the problem? What happens if you train a different neural net on other data, do you get more normal times?",
    "274611": "Finally figured out what's been wrong here. Basically what happened was I did install cudnn but might have screwed up with cudnn. More specifically, I suspected that I accidentally skipped one manual step to copy cudnn.h to /usr/include.\n\nIn that case, cudnn was not being used to accelerate computation for some specific layers such as conv2d but GPU was still being used for other types of units in the Unet architecture and that sort of explained why I saw 100% GPU usage but the overall speed was way slower than it's supposed to be.\n\nAfter installing cudnn and re-compile TF the speed seems to be normal: 25-35 seconds/ epoch as compared to 1000 seconds/epoch without cudnn.\n\nPS, I only found this when I was trying to install Caffe2 wich requires cudnn and threw an error explicitly but I doubt if the installation of TS would stop without cudnn which I can hardly call a caveat but it would be really nice to throw at least a warning during the installation imho.\n\nThanks,",
    "275577": "Glad to see you got it!"
  },
  "source": "meta"
}