{
  "id": 11420,
  "title": "resources for deep learning networks architecture",
  "url": "/competitions/datasciencebowl/discussion/11420",
  "author_name": "",
  "post_date": "2015-01-04T17:19:05.803Z",
  "votes": 1,
  "comment_count": 6,
  "views": 3341,
  "content": "<p>I'd like to use this competition to learn something about deep learning. I read through some of the Theano tutorials and I have some very basic appreciation for the math and the intuition behind some of these techniques.</p>\n<p>But I'm kind of lost about how to use DNN's in practice. Maybe I'm asking the wrong question, but is there any paper/tutorial that gives <strong>practical</strong> rules of thumb on how to proceed to determine better network architectures? I.e. number of layers, hidden units, convolutional layers, dense layers...</p>",
  "messages": [
    {
      "id": "62257",
      "postDate": "01/04/2015 17:19:05",
      "content": "<p>I'd like to use this competition to learn something about deep learning. I read through some of the Theano tutorials and I have some very basic appreciation for the math and the intuition behind some of these techniques.</p>\n<p>But I'm kind of lost about how to use DNN's in practice. Maybe I'm asking the wrong question, but is there any paper/tutorial that gives <strong>practical</strong> rules of thumb on how to proceed to determine better network architectures? I.e. number of layers, hidden units, convolutional layers, dense layers...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62258",
      "postDate": "01/04/2015 17:28:52",
      "content": "<p>A good place to start for rules-of-thumb is to look at the network structures used in some of the examples supplied with libraries like Caffe and Theano.&nbsp; In particular, the imagenet, cifar-10 and googlenet examples that come with Caffe use network structures that have proved very successful in some of the computer vision competitions in recent years.</p>\n<p>The papers associated with those networks will also give some insight into why those structures were chosen - although it is somewhat of a dark art...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62268",
      "postDate": "01/04/2015 21:05:26",
      "content": "<p>i'm curious... &nbsp;looking at an example like imagenet... &nbsp;they suggest resizing the images to 256x256 and the&nbsp;Krizhevsky paper they reference suggests&nbsp;500,000 neurons in 5 layers. &nbsp;If you store everything as float32, how can you fit all that into GPU's RAM? &nbsp;</p>\n<p>Let's just look at just the W in the first hidden layer if it's, say 100k neurons. &nbsp;it would an array of size = (256^2)*100k = 6.5e9. &nbsp;Stored as float32, that would require ~26 GB. &nbsp;If running on a GPU, the bottle neck is obviously getting that data on and off the GPU, so you'll want to store the training set and theta in RAM. &nbsp;How do you do that when you're looking at &gt;150GB just for theta?</p>\n<p>I'm pretty familiar with theano, but I'm maxing out 4GB of GPU RAM with just 60k neurons in 3 layers. &nbsp;That's having already&nbsp;downsized the images in the training set more than I'd like to, so the training set is already much smaller than I'd like to work with it at.</p>\n<p>Any suggestions?</p>\n\n<p>(updated) &nbsp;i see it says 500k neurons, but only 60 million parameters, which would obviously only take up ~.25GB. &nbsp;Not sure why the parameters are so sparse for for that many neurons, but I'll assume it's what I don't know about the specific architecture used. &nbsp;</p>\n<p>Still at 256x256, you could only fit ~15k training samples in 4GB of GPU RAM at once, which would be tiny. &nbsp;Are they using clusters or swapping data in and out of RAM or does Caffe have some elegant way to handle this that my scratch built python dbn can't fathom?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62272",
      "postDate": "01/04/2015 21:43:07",
      "content": "<p>I think this is the reason:</p>\n<p>Your calculations would be true if your first layer were a fully connected layer, i.e. 100k neurons each connected to every pixel in the input image.&nbsp; However, the imagenet/Krizhevsky networks are <em>convolutional</em> neural networks.&nbsp; The first layer of these networks are banks of small filters which are convolved with the input images, i.e. slid across as a sliding window. A typical convolutional layer might have 128 filters, each of size 5x5.&nbsp; The tensor that describes this filter bank is therefore of size 252x252x128 + 5x5x128 = 8128512 + 3200 = 8131712.&nbsp; I think this is only about 43.5Mb when stored as float32's.</p>\n<p>When fully connected layers are introduced into these networks it tends to be at the higher levels where there is at most 1000's of neurons connected to 1000's of neurons.</p>\n<p>The main point is that the number of neurons only tells part of the story - the network architecture, i.e. connections, defines how many parameters you need.</p>\n<p>EDIT:&nbsp; I'm not sure my calculation above is correct, but it's close, just checking my working now...</p>\n<p>EDIT: I think it's correct...</p>\n<p>EDIT: Third time's a charm.&nbsp; The number above is for a single image I think.&nbsp; Typically, training is done in minibatches, so that number gets multiplied by the number of images in a mini-batch to get the amount of memory needed on the GPU at training time.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62277",
      "postDate": "01/04/2015 23:07:21",
      "content": "<p>Thanks Senecaur. &nbsp;Good information, and I appreciate the response.</p>\n\n<p>One clarification... &nbsp;you mention only holding one minibatch of the training set in memory at a time... &nbsp;that would slow things down considerably, since moving things to/from GPU memory is the bottleneck. &nbsp;Is that really done in practice for large datasets?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62279",
      "postDate": "01/05/2015 01:03:29",
      "content": "<p>All of the data is indeed loaded into GPU memory at the start of training if possible - you are right that loading data is the major bottleneck.&nbsp; For large datasets such as imagenet this is usually possible.&nbsp; If your dataset gets huge then you may still need to load data to GPU in batches.</p>\n<p>The memory sizes I described above are just for the neural network weight matrices (i.e. filters), not the data itself.</p>\n<p>What mini-batch training does is controls the size of the tensor of parameters that describes the network architecture.&nbsp; If necessary you can even update the network using a single image at a time which minimizes that tensor size, but the actual data is still retrieved from GPU memory.</p>\n<p>Hope that helps!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62281",
      "postDate": "01/05/2015 01:27:05",
      "content": "<p>This is an excellent tutorial:</p>\n<p>http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 62258,
      "author_name": "senecaur",
      "author_url": "",
      "post_date": "01/04/2015 17:28:52",
      "content": "<p>A good place to start for rules-of-thumb is to look at the network structures used in some of the examples supplied with libraries like Caffe and Theano.&nbsp; In particular, the imagenet, cifar-10 and googlenet examples that come with Caffe use network structures that have proved very successful in some of the computer vision competitions in recent years.</p>\n<p>The papers associated with those networks will also give some insight into why those structures were chosen - although it is somewhat of a dark art...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62268,
      "author_name": "kevinhinson",
      "author_url": "",
      "post_date": "01/04/2015 21:05:26",
      "content": "<p>i'm curious... &nbsp;looking at an example like imagenet... &nbsp;they suggest resizing the images to 256x256 and the&nbsp;Krizhevsky paper they reference suggests&nbsp;500,000 neurons in 5 layers. &nbsp;If you store everything as float32, how can you fit all that into GPU's RAM? &nbsp;</p>\n<p>Let's just look at just the W in the first hidden layer if it's, say 100k neurons. &nbsp;it would an array of size = (256^2)*100k = 6.5e9. &nbsp;Stored as float32, that would require ~26 GB. &nbsp;If running on a GPU, the bottle neck is obviously getting that data on and off the GPU, so you'll want to store the training set and theta in RAM. &nbsp;How do you do that when you're looking at &gt;150GB just for theta?</p>\n<p>I'm pretty familiar with theano, but I'm maxing out 4GB of GPU RAM with just 60k neurons in 3 layers. &nbsp;That's having already&nbsp;downsized the images in the training set more than I'd like to, so the training set is already much smaller than I'd like to work with it at.</p>\n<p>Any suggestions?</p>\n\n<p>(updated) &nbsp;i see it says 500k neurons, but only 60 million parameters, which would obviously only take up ~.25GB. &nbsp;Not sure why the parameters are so sparse for for that many neurons, but I'll assume it's what I don't know about the specific architecture used. &nbsp;</p>\n<p>Still at 256x256, you could only fit ~15k training samples in 4GB of GPU RAM at once, which would be tiny. &nbsp;Are they using clusters or swapping data in and out of RAM or does Caffe have some elegant way to handle this that my scratch built python dbn can't fathom?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62272,
      "author_name": "senecaur",
      "author_url": "",
      "post_date": "01/04/2015 21:43:07",
      "content": "<p>I think this is the reason:</p>\n<p>Your calculations would be true if your first layer were a fully connected layer, i.e. 100k neurons each connected to every pixel in the input image.&nbsp; However, the imagenet/Krizhevsky networks are <em>convolutional</em> neural networks.&nbsp; The first layer of these networks are banks of small filters which are convolved with the input images, i.e. slid across as a sliding window. A typical convolutional layer might have 128 filters, each of size 5x5.&nbsp; The tensor that describes this filter bank is therefore of size 252x252x128 + 5x5x128 = 8128512 + 3200 = 8131712.&nbsp; I think this is only about 43.5Mb when stored as float32's.</p>\n<p>When fully connected layers are introduced into these networks it tends to be at the higher levels where there is at most 1000's of neurons connected to 1000's of neurons.</p>\n<p>The main point is that the number of neurons only tells part of the story - the network architecture, i.e. connections, defines how many parameters you need.</p>\n<p>EDIT:&nbsp; I'm not sure my calculation above is correct, but it's close, just checking my working now...</p>\n<p>EDIT: I think it's correct...</p>\n<p>EDIT: Third time's a charm.&nbsp; The number above is for a single image I think.&nbsp; Typically, training is done in minibatches, so that number gets multiplied by the number of images in a mini-batch to get the amount of memory needed on the GPU at training time.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62277,
      "author_name": "kevinhinson",
      "author_url": "",
      "post_date": "01/04/2015 23:07:21",
      "content": "<p>Thanks Senecaur. &nbsp;Good information, and I appreciate the response.</p>\n\n<p>One clarification... &nbsp;you mention only holding one minibatch of the training set in memory at a time... &nbsp;that would slow things down considerably, since moving things to/from GPU memory is the bottleneck. &nbsp;Is that really done in practice for large datasets?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62279,
      "author_name": "senecaur",
      "author_url": "",
      "post_date": "01/05/2015 01:03:29",
      "content": "<p>All of the data is indeed loaded into GPU memory at the start of training if possible - you are right that loading data is the major bottleneck.&nbsp; For large datasets such as imagenet this is usually possible.&nbsp; If your dataset gets huge then you may still need to load data to GPU in batches.</p>\n<p>The memory sizes I described above are just for the neural network weight matrices (i.e. filters), not the data itself.</p>\n<p>What mini-batch training does is controls the size of the tensor of parameters that describes the network architecture.&nbsp; If necessary you can even update the network using a single image at a time which minimizes that tensor size, but the actual data is still retrieved from GPU memory.</p>\n<p>Hope that helps!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62281,
      "author_name": "albeam",
      "author_url": "",
      "post_date": "01/05/2015 01:27:05",
      "content": "<p>This is an excellent tutorial:</p>\n<p>http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "62257": "",
    "62258": "",
    "62268": "",
    "62272": "",
    "62277": "",
    "62279": "",
    "62281": ""
  },
  "source": "meta"
}