{
  "id": 12634,
  "title": "Can someone explain what batch size is doing in convolutional NNs?",
  "url": "/competitions/datasciencebowl/discussion/12634",
  "author_name": "",
  "post_date": "2015-02-28T02:52:36.737Z",
  "votes": 1,
  "comment_count": 8,
  "views": 20994,
  "content": "<p>I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was simply controlling the number of images sent to the gpu / cpu at any given time, but I can't match that up with this behavior. Can someone explain what batch size controls in the model?&nbsp; Why the variation in performance? Any good resources?</p>\n<p>Sorry for the naive question. This is my first time using these sorts of models, and in the background reading I've done, I've still not fully grasped what this parameter is doing. </p>",
  "messages": [
    {
      "id": "65097",
      "postDate": "02/28/2015 02:52:36",
      "content": "<p>I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was simply controlling the number of images sent to the gpu / cpu at any given time, but I can't match that up with this behavior. Can someone explain what batch size controls in the model?&nbsp; Why the variation in performance? Any good resources?</p>\n<p>Sorry for the naive question. This is my first time using these sorts of models, and in the background reading I've done, I've still not fully grasped what this parameter is doing. </p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65102",
      "postDate": "02/28/2015 04:22:29",
      "content": "<p>With SGD, the gradient is averaged over the minibatch. The batch size will affect how 'noisy' the gradient is, and will change the path the minimization follows.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65110",
      "postDate": "02/28/2015 05:59:45",
      "content": "<p>emelson</p>\n<p>do you mean when we have big batch size mean less noisy gradient?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65127",
      "postDate": "02/28/2015 13:59:32",
      "content": "<p>[quote=Data-Lover;65110]</p>\n<p>do you mean when we have big batch size mean less noisy gradient?</p>\n<p>[/quote]</p>\n<p>Yes. However, the noise actually can&nbsp;be useful as it may&nbsp;help escape local minima. This is one reason SGD can be preferable to batch gradient descent.</p>\n<p>The best minibatch size will depend on the data and network, and as far as I know there's no real &quot;rule of thumb&quot; for choosing it. I would find a value that works efficiently with your hardware and then try a few adjustments.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65131",
      "postDate": "02/28/2015 15:20:45",
      "content": "<p>Hi Maineiac - Also, the first time using NNs. Here are some posts that I found helpful. Generally, the smaller the batch size the noisier the updates (like emolson said), so if you decrease the batch size you should probably decrease the learning rate, and train for more iterations.</p>\n<p>I found the two links below helpful. One is&nbsp;blog post from&nbsp;Ilya Sutskever, which has some practical advice at the bottom. Also, see the caffe thread. They talk a bit about how to adjust the learning rate for smaller batch sizes.&nbsp;</p>\n\n<p>http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html</p>\n<p>https://github.com/BVLC/caffe/issues/430</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65163",
      "postDate": "03/01/2015 03:42:04",
      "content": "<p>Thanks for the help. That clarifies things quite a bit.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65177",
      "postDate": "03/01/2015 14:22:31",
      "content": "<p>Try this paper, it should gives you some clues:</p>\n<p><a href=\"http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf\">http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf</a></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65237",
      "postDate": "03/02/2015 14:25:52",
      "content": "<p>[quote=Gregory Godin;65177]</p>\n<p>Try this paper, it should gives you some clues:</p>\n<p><a href=\"http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf\">http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf</a></p>\n<p>[/quote]</p>\n<p>Super interesting paper. I checked it out on Google scholar, and it doesn't seem like anyone has come up with anything to contradict their main finding, namely, that online learning follows the local gradient at each step (sometimes noisy), but allows for larger step sizes and faster convergence.&nbsp; This explains&nbsp; why a smaller batch sizes converged faster.</p>\n<p>Their research, however, is based on shallow neural network architectures. Is there any reason to think that these principles should be different for deep architectures? (My intuition says 'no', but just wanted to see if someone knew differently)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "100344",
      "postDate": "12/06/2015 01:36:46",
      "content": "<p>Just googled my way to this and found these resources super helpful. If somebody wants to write a blog post (or maybe I should) summarizing some of this, that'd be even more amazing. Cool challenge, btw. Looks like a really hard problem.</p>",
      "rawMarkdown": "Just googled my way to this and found these resources super helpful. If somebody wants to write a blog post (or maybe I should) summarizing some of this, that'd be even more amazing. Cool challenge, btw. Looks like a really hard problem.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 65102,
      "author_name": "emolson",
      "author_url": "",
      "post_date": "02/28/2015 04:22:29",
      "content": "<p>With SGD, the gradient is averaged over the minibatch. The batch size will affect how 'noisy' the gradient is, and will change the path the minimization follows.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65110,
      "author_name": "",
      "author_url": "",
      "post_date": "02/28/2015 05:59:45",
      "content": "<p>emelson</p>\n<p>do you mean when we have big batch size mean less noisy gradient?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65127,
      "author_name": "emolson",
      "author_url": "",
      "post_date": "02/28/2015 13:59:32",
      "content": "<p>[quote=Data-Lover;65110]</p>\n<p>do you mean when we have big batch size mean less noisy gradient?</p>\n<p>[/quote]</p>\n<p>Yes. However, the noise actually can&nbsp;be useful as it may&nbsp;help escape local minima. This is one reason SGD can be preferable to batch gradient descent.</p>\n<p>The best minibatch size will depend on the data and network, and as far as I know there's no real &quot;rule of thumb&quot; for choosing it. I would find a value that works efficiently with your hardware and then try a few adjustments.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65131,
      "author_name": "matthewkrump",
      "author_url": "",
      "post_date": "02/28/2015 15:20:45",
      "content": "<p>Hi Maineiac - Also, the first time using NNs. Here are some posts that I found helpful. Generally, the smaller the batch size the noisier the updates (like emolson said), so if you decrease the batch size you should probably decrease the learning rate, and train for more iterations.</p>\n<p>I found the two links below helpful. One is&nbsp;blog post from&nbsp;Ilya Sutskever, which has some practical advice at the bottom. Also, see the caffe thread. They talk a bit about how to adjust the learning rate for smaller batch sizes.&nbsp;</p>\n\n<p>http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html</p>\n<p>https://github.com/BVLC/caffe/issues/430</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65163,
      "author_name": "maineiac",
      "author_url": "",
      "post_date": "03/01/2015 03:42:04",
      "content": "<p>Thanks for the help. That clarifies things quite a bit.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65177,
      "author_name": "gregorygodin",
      "author_url": "",
      "post_date": "03/01/2015 14:22:31",
      "content": "<p>Try this paper, it should gives you some clues:</p>\n<p><a href=\"http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf\">http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65237,
      "author_name": "maineiac",
      "author_url": "",
      "post_date": "03/02/2015 14:25:52",
      "content": "<p>[quote=Gregory Godin;65177]</p>\n<p>Try this paper, it should gives you some clues:</p>\n<p><a href=\"http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf\">http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf</a></p>\n<p>[/quote]</p>\n<p>Super interesting paper. I checked it out on Google scholar, and it doesn't seem like anyone has come up with anything to contradict their main finding, namely, that online learning follows the local gradient at each step (sometimes noisy), but allows for larger step sizes and faster convergence.&nbsp; This explains&nbsp; why a smaller batch sizes converged faster.</p>\n<p>Their research, however, is based on shallow neural network architectures. Is there any reason to think that these principles should be different for deep architectures? (My intuition says 'no', but just wanted to see if someone knew differently)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 100344,
      "author_name": "",
      "author_url": "",
      "post_date": "12/06/2015 01:36:46",
      "content": "<p>Just googled my way to this and found these resources super helpful. If somebody wants to write a blog post (or maybe I should) summarizing some of this, that'd be even more amazing. Cool challenge, btw. Looks like a really hard problem.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "65097": "",
    "65102": "",
    "65110": "",
    "65127": "",
    "65131": "",
    "65163": "",
    "65177": "",
    "65237": "",
    "100344": "Just googled my way to this and found these resources super helpful. If somebody wants to write a blog post (or maybe I should) summarizing some of this, that'd be even more amazing. Cool challenge, btw. Looks like a really hard problem."
  },
  "source": "meta"
}