I was looking at some works using the CycleGAN and saw that they use InstanceNormalization at their upsample and downsample blocks, so when you use InstanceNormalization it is recommended to use batch size = 1, I also saw that the reason the use InstanceNormalization is that it is supposed to give better results to image translation tasks. Learning this I have updated my introductory kernel to also use batch size = 1.
My questions are, first could we use batch size = 8 since we are using TPU and they have 8 cores?
\nSecond is it really better to use InstanceNormalization here? or could we move to BatchNorm or GroupNorm and use bigger batches to speed up the training?
I was looking at some works using the CycleGAN and saw that they use InstanceNormalization at their upsample and downsample blocks, so when you use InstanceNormalization it is recommended to use batch size = 1, I also saw that the reason the use InstanceNormalization is that it is supposed to give better results to image translation tasks. Learning this I have updated my introductory kernel to also use batch size = 1.
My questions are, first could we use batch size = 8 since we are using TPU and they have 8 cores?
\nSecond is it really better to use InstanceNormalization here? or could we move to BatchNorm or GroupNorm and use bigger batches to speed up the training?
My understanding is that InstanceNorm is equivalent to BatchNorm when batch size=1. But InstanceNorm can be used for batch size>1.
\nThere isn't anything actually preventing someone from using a higher batch size and getting decent results. I actually just looked into this today. There are many issues on the GitHub repository discussing this. See this issue or this issue or this issue for example.
", "rawMarkdown": "My understanding is that InstanceNorm is equivalent to BatchNorm when batch size=1. But InstanceNorm can be used for batch size>1. \n\nThere isn't anything actually preventing someone from using a higher batch size and getting decent results. I actually just looked into this today. There are many issues on the GitHub repository discussing this. See [this issue](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/27) or [this issue](https://github.com/junyanz/CycleGAN/issues/55) or [this issue](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/198) for example.", "votes": null }, { "id": "992129", "postDate": "08/31/2020 01:06:12", "content": "Yeah @tanlikesmath thanks for the input, I also was reading some comments about this topic, including the ones you listed here, seems that the Pytorch CycleGAN Git is very active, I ended with the same conclusion as you.
", "rawMarkdown": "Yeah @tanlikesmath thanks for the input, I also was reading some comments about this topic, including the ones you listed here, seems that the Pytorch CycleGAN Git is very active, I ended with the same conclusion as you.", "votes": null } ], "comments": [ { "id": 992119, "author_name": "tanlikesmath", "author_url": "", "post_date": "08/31/2020 00:45:34", "content": "My understanding is that InstanceNorm is equivalent to BatchNorm when batch size=1. But InstanceNorm can be used for batch size>1.
\nThere isn't anything actually preventing someone from using a higher batch size and getting decent results. I actually just looked into this today. There are many issues on the GitHub repository discussing this. See this issue or this issue or this issue for example.
", "votes": null, "replies": [ { "id": 992129, "author_name": "dimitreoliveira", "author_url": "", "post_date": "08/31/2020 01:06:12", "content": "Yeah @tanlikesmath thanks for the input, I also was reading some comments about this topic, including the ones you listed here, seems that the Pytorch CycleGAN Git is very active, I ended with the same conclusion as you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "991998": "I was looking at some works using the CycleGAN and saw that they use [InstanceNormalization](https://www.tensorflow.org/addons/api_docs/python/tfa/layers/InstanceNormalization) at their upsample and downsample blocks, so when you use `InstanceNormalization` it is recommended to use `batch size = 1`, I also saw that the reason the use `InstanceNormalization` is that it is supposed to give better results to image translation tasks. Learning this I have updated my [introductory kernel](https://www.kaggle.com/dimitreoliveira/introduction-to-cyclegan-monet-paintings) to also use `batch size = 1`.\n\nMy questions are, first could we use `batch size = 8` since we are using TPU and they have 8 cores?\nSecond is it really better to use `InstanceNormalization` here? or could we move to `BatchNorm` or `GroupNorm` and use bigger batches to speed up the training?", "992119": "My understanding is that InstanceNorm is equivalent to BatchNorm when batch size=1. But InstanceNorm can be used for batch size>1. \n\nThere isn't anything actually preventing someone from using a higher batch size and getting decent results. I actually just looked into this today. There are many issues on the GitHub repository discussing this. See [this issue](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/27) or [this issue](https://github.com/junyanz/CycleGAN/issues/55) or [this issue](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/198) for example.", "992129": "Yeah @tanlikesmath thanks for the input, I also was reading some comments about this topic, including the ones you listed here, seems that the Pytorch CycleGAN Git is very active, I ended with the same conclusion as you." }, "source": "meta" }