![\"Summary](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1935904%2F21df762053fcc8f68fc99c494852e9a5%2F2019-07-23_11h31_51.png?generation=1563856367486819&alt=media\")
\nThe graph above shows a summary of the best CIFAR-10 test performance achieved with different batch sizes, for the different network architectures Hi everyone,
\n\nI have read several kernels, and most of them are using batch size such as 64, 128 or even 256.\nSo I want to share with you this white paper: Revisiting Small Batch Training for Deep Neural Networks by Dominic Masters and Carlo Luschi.
\n\nThe small mini-batch will improve generalization performance and allows a significantly smaller memory footprint.
\n\nAccording to Dominic Masters and Carlo Luschi, batch size should be between 2 and 32. They did their tests with the on the CIFAR-10, CIFAR-100 and ImageNet datasets and compare the test accuracy depending on the batch size (and the learning rate).
\n\n\nThe graph above shows a summary of the best CIFAR-10 test performance achieved with different batch sizes, for the different network architectures
Moreover, there is no problem with the batch normalization even when batch size equals 4 or 8.
\n\nNevertheless, the use of small batch size also has the effect of reducing the computational parallelism available. So you are to choose between hardware efficiency and test performance.
\n\nIf you want more information, I kindly recommend you to read this paper: you will have more explanations about the batch in general and some beautiful charts showing the comparison between the different batch sizes.
\n\nI hope it will help you to choose the batch size of your future network.
\n\nKind regards,\nKaharon
", "messages": [ { "id": "582339", "postDate": "07/23/2019 04:40:07", "content": "Hi everyone,
\n\nI have read several kernels, and most of them are using batch size such as 64, 128 or even 256.\nSo I want to share with you this white paper: Revisiting Small Batch Training for Deep Neural Networks by Dominic Masters and Carlo Luschi.
\n\nThe small mini-batch will improve generalization performance and allows a significantly smaller memory footprint.
\n\nAccording to Dominic Masters and Carlo Luschi, batch size should be between 2 and 32. They did their tests with the on the CIFAR-10, CIFAR-100 and ImageNet datasets and compare the test accuracy depending on the batch size (and the learning rate).
\n\n\nThe graph above shows a summary of the best CIFAR-10 test performance achieved with different batch sizes, for the different network architectures
Moreover, there is no problem with the batch normalization even when batch size equals 4 or 8.
\n\nNevertheless, the use of small batch size also has the effect of reducing the computational parallelism available. So you are to choose between hardware efficiency and test performance.
\n\nIf you want more information, I kindly recommend you to read this paper: you will have more explanations about the batch in general and some beautiful charts showing the comparison between the different batch sizes.
\n\nI hope it will help you to choose the batch size of your future network.
\n\nKind regards,\nKaharon
", "rawMarkdown": "Hi everyone,\n\nI have read several kernels, and most of them are using batch size such as 64, 128 or even 256.\nSo I want to share with you this white paper: [Revisiting Small Batch Training for Deep Neural Networks](https://arxiv.org/abs/1804.07612) by Dominic Masters and Carlo Luschi.\n\nThe small mini-batch will improve generalization performance and allows a significantly smaller memory footprint.\n\nAccording to Dominic Masters and Carlo Luschi, batch size should be between 2 and 32. They did their tests with the on the CIFAR-10, CIFAR-100 and ImageNet datasets and compare the test accuracy depending on the batch size (and the learning rate).\n\n\nThe graph above shows a summary of the best CIFAR-10 test performance achieved with different batch sizes, for the different network architectures \n\n- _BN/noBN_ = with/without BN\n- _Aug/noAug_ = with/without data augmentation\n- _WU_ = with gradual warmup\n\nMoreover, there is no problem with the batch normalization even when batch size equals 4 or 8.\n\nNevertheless, the use of small batch size also has the effect of reducing the computational parallelism available. So you are to choose between hardware efficiency and test performance. \n\nIf you want more information, I kindly recommend you to read this paper: you will have more explanations about the batch in general and some beautiful charts showing the comparison between the different batch sizes. \n\nI hope it will help you to choose the batch size of your future network.\n\nKind regards,\nKaharon", "votes": null }, { "id": "582383", "postDate": "07/23/2019 05:50:57", "content": "That's really a good paper! Thanks!
", "rawMarkdown": "That's really a good paper! Thanks!", "votes": null }, { "id": "582625", "postDate": "07/23/2019 11:25:05", "content": "Thanks for sharing! it’s helpful
", "rawMarkdown": "Thanks for sharing! it’s helpful", "votes": null }, { "id": "582729", "postDate": "07/23/2019 14:00:09", "content": "Thanks for the tip. When I get time, I'll make a plot of RaLSGAN epoch versus FID score for various batch sizes. That will show how batch size affects RaLSGANs.
", "rawMarkdown": "Thanks for the tip. When I get time, I'll make a plot of RaLSGAN epoch versus FID score for various batch sizes. That will show how batch size affects RaLSGANs.", "votes": null }, { "id": "582733", "postDate": "07/23/2019 14:05:30", "content": "It's amazing. Thank you for sharing.
", "rawMarkdown": "It's amazing. Thank you for sharing.", "votes": null }, { "id": "583847", "postDate": "07/25/2019 04:48:17", "content": "https://arxiv.org/abs/1809.11096v2\naccording to this paper, large batch size boosts the score tremendously
", "rawMarkdown": "https://arxiv.org/abs/1809.11096v2\naccording to this paper, large batch size boosts the score tremendously", "votes": null }, { "id": "584652", "postDate": "07/26/2019 09:05:58", "content": "I choose large batch size mostly because of computational reasons. (Trying to fit as much samples into GPU as possible). So taking into account this idea of small batch size, do you advise to better train several networks in parallel with small batches than one network with a huge batch?
", "rawMarkdown": "I choose large batch size mostly because of computational reasons. (Trying to fit as much samples into GPU as possible). So taking into account this idea of small batch size, do you advise to better train several networks in parallel with small batches than one network with a huge batch?", "votes": null } ], "comments": [ { "id": 582383, "author_name": "rhonin0", "author_url": "", "post_date": "07/23/2019 05:50:57", "content": "That's really a good paper! Thanks!
", "votes": null, "replies": [] }, { "id": 582625, "author_name": "sairakun", "author_url": "", "post_date": "07/23/2019 11:25:05", "content": "Thanks for sharing! it’s helpful
", "votes": null, "replies": [] }, { "id": 582729, "author_name": "cdeotte", "author_url": "", "post_date": "07/23/2019 14:00:09", "content": "Thanks for the tip. When I get time, I'll make a plot of RaLSGAN epoch versus FID score for various batch sizes. That will show how batch size affects RaLSGANs.
", "votes": null, "replies": [] }, { "id": 582733, "author_name": "mochidan", "author_url": "", "post_date": "07/23/2019 14:05:30", "content": "It's amazing. Thank you for sharing.
", "votes": null, "replies": [] }, { "id": 583847, "author_name": "yangsaewon", "author_url": "", "post_date": "07/25/2019 04:48:17", "content": "https://arxiv.org/abs/1809.11096v2\naccording to this paper, large batch size boosts the score tremendously
", "votes": null, "replies": [] }, { "id": 584652, "author_name": "purplejester", "author_url": "", "post_date": "07/26/2019 09:05:58", "content": "I choose large batch size mostly because of computational reasons. (Trying to fit as much samples into GPU as possible). So taking into account this idea of small batch size, do you advise to better train several networks in parallel with small batches than one network with a huge batch?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "582339": "Hi everyone,\n\nI have read several kernels, and most of them are using batch size such as 64, 128 or even 256.\nSo I want to share with you this white paper: [Revisiting Small Batch Training for Deep Neural Networks](https://arxiv.org/abs/1804.07612) by Dominic Masters and Carlo Luschi.\n\nThe small mini-batch will improve generalization performance and allows a significantly smaller memory footprint.\n\nAccording to Dominic Masters and Carlo Luschi, batch size should be between 2 and 32. They did their tests with the on the CIFAR-10, CIFAR-100 and ImageNet datasets and compare the test accuracy depending on the batch size (and the learning rate).\n\n\nThe graph above shows a summary of the best CIFAR-10 test performance achieved with different batch sizes, for the different network architectures \n\n- _BN/noBN_ = with/without BN\n- _Aug/noAug_ = with/without data augmentation\n- _WU_ = with gradual warmup\n\nMoreover, there is no problem with the batch normalization even when batch size equals 4 or 8.\n\nNevertheless, the use of small batch size also has the effect of reducing the computational parallelism available. So you are to choose between hardware efficiency and test performance. \n\nIf you want more information, I kindly recommend you to read this paper: you will have more explanations about the batch in general and some beautiful charts showing the comparison between the different batch sizes. \n\nI hope it will help you to choose the batch size of your future network.\n\nKind regards,\nKaharon", "582383": "That's really a good paper! Thanks!", "582625": "Thanks for sharing! it’s helpful", "582729": "Thanks for the tip. When I get time, I'll make a plot of RaLSGAN epoch versus FID score for various batch sizes. That will show how batch size affects RaLSGANs.", "582733": "It's amazing. Thank you for sharing.", "583847": "https://arxiv.org/abs/1809.11096v2\naccording to this paper, large batch size boosts the score tremendously", "584652": "I choose large batch size mostly because of computational reasons. (Trying to fit as much samples into GPU as possible). So taking into account this idea of small batch size, do you advise to better train several networks in parallel with small batches than one network with a huge batch?" }, "source": "meta" }