{
  "id": 12872,
  "title": "DIGITS and Titan X - Nvidia key note",
  "url": "/competitions/datasciencebowl/discussion/12872",
  "author_name": "",
  "post_date": "2015-03-17T20:48:15.587Z",
  "votes": 2,
  "comment_count": 7,
  "views": 2157,
  "content": "<p>Nvidia unveiled DIGITS (a seemingly highly simple and flexible deep learning middleware) and Titan X in a GTC keynote that is all about deep learning.&nbsp;http://m.ustream.tv/recorded/60025825</p>",
  "messages": [
    {
      "id": "66816",
      "postDate": "03/17/2015 20:48:15",
      "content": "<p>Nvidia unveiled DIGITS (a seemingly highly simple and flexible deep learning middleware) and Titan X in a GTC keynote that is all about deep learning.&nbsp;http://m.ustream.tv/recorded/60025825</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66900",
      "postDate": "03/18/2015 03:37:02",
      "content": "<p>Titan x usd999 only&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66901",
      "postDate": "03/18/2015 03:39:18",
      "content": "<p>my gtx 970 is 340 usd only, it has 1600&nbsp;cuda cores and 4GB memory :D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66904",
      "postDate": "03/18/2015 03:49:12",
      "content": "<p>It's double precision computation is only 1/32 of its single precision, shouldn't affect gaming, not sure about computing. And already out of stock...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66916",
      "postDate": "03/18/2015 08:29:52",
      "content": "<p>If you're only going to train convnets with it you don't need double precision though :)</p>\n<p>They also announced some interesting things for the next generation of cards including mixed precision support. With Pascal-based cards (ETA 2016)&nbsp;it should be possible to use half precision for some layers in a network (and single or even double precision for others) to get further speedups.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67014",
      "postDate": "03/18/2015 17:20:11",
      "content": "<p>@sedielem, are there any research that&nbsp;illustrate single precision is good enough for SGD? I suspect there would be some loss.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67031",
      "postDate": "03/18/2015 18:22:00",
      "content": "<p>I don't know if there's a paper that shows that, but there is some recent work on half-precision and fixed precision representations. I can't give you a reference off the top of my head though.</p>\n<p>Consider that when training a neural network, there are various sources of noise, both unintentional (noise in the dataset) and intentional (dropout, using minibatch SGD instead of batch SGD, data augmentation, ...).</p>\n<p>Compared to all this, the quantization noise caused by limited precision is negligible. That is why half precision is actually a great idea for training neural networks - we can probably tolerate higher quantization noise levels because they are still almost unnoticeable compared to all the other noise we have to deal with. It might even help regularize the model!</p>\n<p>EDIT: actually, Jeff Dean mentioned in his GTC keynote just now that they use half-precision to reduce communication between nodes! He also referenced this paper:&nbsp;http://arxiv.org/abs/1502.02551</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67048",
      "postDate": "03/18/2015 18:58:45",
      "content": "<p>What @sedielem said makes a lot of sense.</p>\n<p>Why solve to 1e-15 when your model is only accurate to 1e-2?</p>\n<p>This is a very rational discussion by a Kaggler on GPU considerations.</p>\n<p>https://timdettmers.wordpress.com/2014/08/14/which-gpu-for-deep-learning/</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 66900,
      "author_name": "stevendu",
      "author_url": "",
      "post_date": "03/18/2015 03:37:02",
      "content": "<p>Titan x usd999 only&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66901,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "03/18/2015 03:39:18",
      "content": "<p>my gtx 970 is 340 usd only, it has 1600&nbsp;cuda cores and 4GB memory :D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66904,
      "author_name": "skylibrary",
      "author_url": "",
      "post_date": "03/18/2015 03:49:12",
      "content": "<p>It's double precision computation is only 1/32 of its single precision, shouldn't affect gaming, not sure about computing. And already out of stock...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66916,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "03/18/2015 08:29:52",
      "content": "<p>If you're only going to train convnets with it you don't need double precision though :)</p>\n<p>They also announced some interesting things for the next generation of cards including mixed precision support. With Pascal-based cards (ETA 2016)&nbsp;it should be possible to use half precision for some layers in a network (and single or even double precision for others) to get further speedups.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67014,
      "author_name": "songgc",
      "author_url": "",
      "post_date": "03/18/2015 17:20:11",
      "content": "<p>@sedielem, are there any research that&nbsp;illustrate single precision is good enough for SGD? I suspect there would be some loss.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67031,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "03/18/2015 18:22:00",
      "content": "<p>I don't know if there's a paper that shows that, but there is some recent work on half-precision and fixed precision representations. I can't give you a reference off the top of my head though.</p>\n<p>Consider that when training a neural network, there are various sources of noise, both unintentional (noise in the dataset) and intentional (dropout, using minibatch SGD instead of batch SGD, data augmentation, ...).</p>\n<p>Compared to all this, the quantization noise caused by limited precision is negligible. That is why half precision is actually a great idea for training neural networks - we can probably tolerate higher quantization noise levels because they are still almost unnoticeable compared to all the other noise we have to deal with. It might even help regularize the model!</p>\n<p>EDIT: actually, Jeff Dean mentioned in his GTC keynote just now that they use half-precision to reduce communication between nodes! He also referenced this paper:&nbsp;http://arxiv.org/abs/1502.02551</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67048,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "03/18/2015 18:58:45",
      "content": "<p>What @sedielem said makes a lot of sense.</p>\n<p>Why solve to 1e-15 when your model is only accurate to 1e-2?</p>\n<p>This is a very rational discussion by a Kaggler on GPU considerations.</p>\n<p>https://timdettmers.wordpress.com/2014/08/14/which-gpu-for-deep-learning/</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "66816": "",
    "66900": "",
    "66901": "",
    "66904": "",
    "66916": "",
    "67014": "",
    "67031": "",
    "67048": ""
  },
  "source": "meta"
}