{ "id": 98993, "title": "Improved techniques for training GAN", "url": "/competitions/generative-dog-images/discussion/98993", "author_name": "", "post_date": "2019-07-08T05:25:58.564449900Z", "votes": 4, "comment_count": 13, "views": 0, "content": "

Had been going through GANs lately and read a few tricks to help with GAN. Did tried it locally and now look forward to implementing the same in the competition.

\n\n

1. Normalize the input:

\n\n\n\n

2. Batchnorm

\n\n\n\n

3. Avoid Sparse Gradients

\n\n\n\n

4: Use the ADAM Optimizer

\n\n\n\n

Apart from these, I also came across a few blogs that would also help:\n- Instance Noise: A trick for stabilising GAN training\n- Improved technique for training GAN

\n\n

References:\n1. Soumith GanHacks\n2. Unsupervised representational learning\n3. Stack GAN

", "messages": [ { "id": "570296", "postDate": "07/08/2019 05:25:58", "content": "

Had been going through GANs lately and read a few tricks to help with GAN. Did tried it locally and now look forward to implementing the same in the competition.

\n\n

1. Normalize the input:

\n\n\n\n

2. Batchnorm

\n\n\n\n

3. Avoid Sparse Gradients

\n\n\n\n

4: Use the ADAM Optimizer

\n\n\n\n

Apart from these, I also came across a few blogs that would also help:\n- Instance Noise: A trick for stabilising GAN training\n- Improved technique for training GAN

\n\n

References:\n1. Soumith GanHacks\n2. Unsupervised representational learning\n3. Stack GAN

", "rawMarkdown": "Had been going through GANs lately and read a few tricks to help with GAN. Did tried it locally and now look forward to implementing the same in the competition. \n\n### 1. Normalize the input:\n- normalize the images between -1 and 1\n- Tanh as the last layer of the generator output\n\n### 2. Batchnorm\n- Construct different mini-batches for real and fake, i.e. each mini-batch needs to contain only all real images or all generated images.\n- when batchnorm is not an option use instance normalization (for each sample, subtract mean and divide by standard deviation).\n\n### 3. Avoid Sparse Gradients\n- the stability of the GAN game suffers if you have sparse gradients\n- LeakyReLU = good (in both G and D)\n- For Downsampling, use: Average Pooling, Conv2d + stride\n\n### 4: Use the ADAM Optimizer\n- Perhaps the real papers might have a different opinion, this literally makes a remarkable improvement\n- Use SGD for discriminator and ADAM for generator. \n- To decrease instability issues decrease the learning rate to 0.0002 (from 0.001) and the momentum/beta1 to 0.5 (from 0.9) for Adam.\n\n\nApart from these, I also came across a few blogs that would also help:\n- [Instance Noise: A trick for stabilising GAN training](https://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/)\n- [Improved technique for training GAN](https://arxiv.org/pdf/1606.03498)\n\nReferences:\n1. [Soumith GanHacks](https://github.com/soumith/ganhacks)\n2. [Unsupervised representational learning](https://github.com/sarathknv/papers/blob/master/neural-nets/Unsupervised_Representation_Learning_with_Deep_Convolutional_Generative_Adversarial_Networks.md)\n3. [Stack GAN](https://github.com/sarathknv/papers/blob/master/neural-nets/StackGAN.md)", "votes": null }, { "id": "570312", "postDate": "07/08/2019 05:45:57", "content": "

Thanks a ton for sharing this . I read somewhere that people have got away with pooling layers in GAN since it reduces spatial features . \nAlso , I have seen in this competition that the different learning rate between generator and discriminator really helps convergence . \nHave you tried SGD and ADAM as discriminator for this competition ? I have tried AdaBound in one of the kernels and got a mode collapse .

", "rawMarkdown": "Thanks a ton for sharing this . I read somewhere that people have got away with pooling layers in GAN since it reduces spatial features . \nAlso , I have seen in this competition that the different learning rate between generator and discriminator really helps convergence . \nHave you tried SGD and ADAM as discriminator for this competition ? I have tried AdaBound in one of the kernels and got a mode collapse .", "votes": null }, { "id": "570321", "postDate": "07/08/2019 06:02:39", "content": "

Yes @phoenix9032 you are absolutely right. Differentiable learning rates really improve the model performances with training. And yes I have tried the SGD and ADAM and the results are comparitively better. Will share the generated results soon.\nI am a little eager to know how you found about the mode collapse problem in with your model?

", "rawMarkdown": "Yes @phoenix9032 you are absolutely right. Differentiable learning rates really improve the model performances with training. And yes I have tried the SGD and ADAM and the results are comparitively better. Will share the generated results soon.\nI am a little eager to know how you found about the _mode collapse_ problem in with your model?", "votes": null }, { "id": "570325", "postDate": "07/08/2019 06:15:26", "content": "

Hi @thanatoz . I was working with Adam for both discriminator and generator for DCGAN and executed it for 270 epochs or so .It generated images with FID of 134 . Later I wanted to test AdaBound and just replaced the Adam with the same . After 270 epochs when i saw the results it was only black boxes with a tinge of different colors inside that .

", "rawMarkdown": "Hi @thanatoz . I was working with Adam for both discriminator and generator for DCGAN and executed it for 270 epochs or so .It generated images with FID of 134 . Later I wanted to test AdaBound and just replaced the Adam with the same . After 270 epochs when i saw the results it was only black boxes with a tinge of different colors inside that .", "votes": null }, { "id": "570349", "postDate": "07/08/2019 07:02:43", "content": "

This is a copy paste from https://github.com/soumith/ganhacks right ?

", "rawMarkdown": "This is a copy paste from https://github.com/soumith/ganhacks right ?", "votes": null }, { "id": "570386", "postDate": "07/08/2019 08:02:25", "content": "

Not entirely a copy paste @theoviel. But I have added required references to it now. I only mentioned tricks that helped me and not complete writeup. Soumith's work is exceptional anyways.

", "rawMarkdown": "Not entirely a copy paste @theoviel. But I have added required references to it now. I only mentioned tricks that helped me and not complete writeup. Soumith's work is exceptional anyways.", "votes": null }, { "id": "570391", "postDate": "07/08/2019 08:06:32", "content": "

The default learning rate could be a reason for this. Try to lower the learning rate and refer to Fastai notebooks for the same. Jeremy howard has explained it pretty well. Also refer to this paper if you want:\nUnsupervised representational learning

", "rawMarkdown": "The default learning rate could be a reason for this. Try to lower the learning rate and refer to Fastai notebooks for the same. Jeremy howard has explained it pretty well. Also refer to this paper if you want:\n[Unsupervised representational learning](https://github.com/sarathknv/papers/blob/master/neural-nets/Unsupervised_Representation_Learning_with_Deep_Convolutional_Generative_Adversarial_Networks.md)", "votes": null }, { "id": "570394", "postDate": "07/08/2019 08:12:37", "content": "

Thank you . Very valuable suggestion . I will try out stuff what I have understood :) . I am a very novice person to machine learning itself . This competition also increasing my patience a lot .. Sometimes after 7 hours the commit just fails .. its painful :D

", "rawMarkdown": "Thank you . Very valuable suggestion . I will try out stuff what I have understood :) . I am a very novice person to machine learning itself . This competition also increasing my patience a lot .. Sometimes after 7 hours the commit just fails .. its painful :D", "votes": null }, { "id": "570419", "postDate": "07/08/2019 09:07:23", "content": "

I am glad that I was some help to you. If you want, I would suggest you to go through Jeremy Howard's explanation on GAN and his notebook. You can also look at my codes which are novice in my notebook if you haven't: Let's generate more faces

", "rawMarkdown": "I am glad that I was some help to you. If you want, I would suggest you to go through [Jeremy Howard's explanation on GAN](https://youtu.be/9spwoDYwW_I?t=4214) and [his notebook](https://github.com/fastai/fastai/blob/master/courses/dl2/wgan.ipynb). You can also look at my codes which are novice in my notebook if you haven't: [Let's generate more faces](https://www.kaggle.com/thanatoz/lets-generate-more-faces-gan)", "votes": null }, { "id": "570437", "postDate": "07/08/2019 09:45:55", "content": "

Okok, thanks for adding the ref though!

", "rawMarkdown": "Okok, thanks for adding the ref though!", "votes": null }, { "id": "570551", "postDate": "07/08/2019 12:50:26", "content": "

Good summary!

", "rawMarkdown": "Good summary!", "votes": null }, { "id": "571553", "postDate": "07/09/2019 18:14:48", "content": "

@phoenix9032 I would recommend to you before commiting the kernel commit it with all the code unchanged except for training for only one epoch, this way you will check that all phases of your pipeline are working well (reading data, training, image generation). If you have code that runs once per several epochs (checkpoint storing for example), make sure that your test commit covers all your code at least once. \nHappy kaggling!)

", "rawMarkdown": "phoenix9032 I would recommend to you before commiting the kernel commit it with all the code unchanged except for training for only one epoch, this way you will check that all phases of your pipeline are working well (reading data, training, image generation). If you have code that runs once per several epochs (checkpoint storing for example), make sure that your test commit covers all your code at least once. \nHappy kaggling!)", "votes": null }, { "id": "894485", "postDate": "06/20/2020 13:19:28", "content": "

Thank you for sharing this.\nI found that sometime the combination of several models may lead the hyper-parameter difficult to fine-tune, these tips for training GAN are worthy to follow as initialization configuration.

", "rawMarkdown": "Thank you for sharing this.\nI found that sometime the combination of several models may lead the hyper-parameter difficult to fine-tune, these tips for training GAN are worthy to follow as initialization configuration.", "votes": null }, { "id": "1109653", "postDate": "12/11/2020 23:24:37", "content": "

\"Construct different mini-batches for real and fake\"
\nAnyone know why this is? From my experience too if you try to mix them in batches it completely messes up the discriminator (when BN is used), and I've seen conference presentations saying not do that. In fact this issue with batchNorm is pretty much the only thing consistent about GANs. :) On the stuff about Adam and LeakyReLU you'll see conflicting advice/results out there, but if you mix fake/real with batchNorm it will mess things up every time. However I'm still to understand why. The purpose of the discriminator is to \"discriminate\" fake from real, and it does it per-sample not per batch, so why would we throw it a softball by only packing the batches uniformly?

", "rawMarkdown": "\"Construct different mini-batches for real and fake\"\nAnyone know why this is? From my experience too if you try to mix them in batches it completely messes up the discriminator (when BN is used), and I've seen conference presentations saying not do that. In fact this issue with batchNorm is pretty much the only thing consistent about GANs. :) On the stuff about Adam and LeakyReLU you'll see conflicting advice/results out there, but if you mix fake/real with batchNorm it will mess things up every time. However I'm still to understand why. The purpose of the discriminator is to \"discriminate\" fake from real, and it does it per-sample not per batch, so why would we throw it a softball by only packing the batches uniformly?", "votes": null } ], "comments": [ { "id": 570312, "author_name": "phoenix9032", "author_url": "", "post_date": "07/08/2019 05:45:57", "content": "

Thanks a ton for sharing this . I read somewhere that people have got away with pooling layers in GAN since it reduces spatial features . \nAlso , I have seen in this competition that the different learning rate between generator and discriminator really helps convergence . \nHave you tried SGD and ADAM as discriminator for this competition ? I have tried AdaBound in one of the kernels and got a mode collapse .

", "votes": null, "replies": [ { "id": 570321, "author_name": "thanatoz", "author_url": "", "post_date": "07/08/2019 06:02:39", "content": "

Yes @phoenix9032 you are absolutely right. Differentiable learning rates really improve the model performances with training. And yes I have tried the SGD and ADAM and the results are comparitively better. Will share the generated results soon.\nI am a little eager to know how you found about the mode collapse problem in with your model?

", "votes": null, "replies": [] }, { "id": 570325, "author_name": "phoenix9032", "author_url": "", "post_date": "07/08/2019 06:15:26", "content": "

Hi @thanatoz . I was working with Adam for both discriminator and generator for DCGAN and executed it for 270 epochs or so .It generated images with FID of 134 . Later I wanted to test AdaBound and just replaced the Adam with the same . After 270 epochs when i saw the results it was only black boxes with a tinge of different colors inside that .

", "votes": null, "replies": [] }, { "id": 570391, "author_name": "thanatoz", "author_url": "", "post_date": "07/08/2019 08:06:32", "content": "

The default learning rate could be a reason for this. Try to lower the learning rate and refer to Fastai notebooks for the same. Jeremy howard has explained it pretty well. Also refer to this paper if you want:\nUnsupervised representational learning

", "votes": null, "replies": [] }, { "id": 570394, "author_name": "phoenix9032", "author_url": "", "post_date": "07/08/2019 08:12:37", "content": "

Thank you . Very valuable suggestion . I will try out stuff what I have understood :) . I am a very novice person to machine learning itself . This competition also increasing my patience a lot .. Sometimes after 7 hours the commit just fails .. its painful :D

", "votes": null, "replies": [] }, { "id": 570419, "author_name": "thanatoz", "author_url": "", "post_date": "07/08/2019 09:07:23", "content": "

I am glad that I was some help to you. If you want, I would suggest you to go through Jeremy Howard's explanation on GAN and his notebook. You can also look at my codes which are novice in my notebook if you haven't: Let's generate more faces

", "votes": null, "replies": [] }, { "id": 571553, "author_name": "kostyaatarik", "author_url": "", "post_date": "07/09/2019 18:14:48", "content": "

@phoenix9032 I would recommend to you before commiting the kernel commit it with all the code unchanged except for training for only one epoch, this way you will check that all phases of your pipeline are working well (reading data, training, image generation). If you have code that runs once per several epochs (checkpoint storing for example), make sure that your test commit covers all your code at least once. \nHappy kaggling!)

", "votes": null, "replies": [] }, { "id": 1109653, "author_name": "hshmhashemi", "author_url": "", "post_date": "12/11/2020 23:24:37", "content": "

\"Construct different mini-batches for real and fake\"
\nAnyone know why this is? From my experience too if you try to mix them in batches it completely messes up the discriminator (when BN is used), and I've seen conference presentations saying not do that. In fact this issue with batchNorm is pretty much the only thing consistent about GANs. :) On the stuff about Adam and LeakyReLU you'll see conflicting advice/results out there, but if you mix fake/real with batchNorm it will mess things up every time. However I'm still to understand why. The purpose of the discriminator is to \"discriminate\" fake from real, and it does it per-sample not per batch, so why would we throw it a softball by only packing the batches uniformly?

", "votes": null, "replies": [] } ] }, { "id": 570349, "author_name": "theoviel", "author_url": "", "post_date": "07/08/2019 07:02:43", "content": "

This is a copy paste from https://github.com/soumith/ganhacks right ?

", "votes": null, "replies": [ { "id": 570386, "author_name": "thanatoz", "author_url": "", "post_date": "07/08/2019 08:02:25", "content": "

Not entirely a copy paste @theoviel. But I have added required references to it now. I only mentioned tricks that helped me and not complete writeup. Soumith's work is exceptional anyways.

", "votes": null, "replies": [] }, { "id": 570437, "author_name": "theoviel", "author_url": "", "post_date": "07/08/2019 09:45:55", "content": "

Okok, thanks for adding the ref though!

", "votes": null, "replies": [] } ] }, { "id": 570551, "author_name": "snakayama", "author_url": "", "post_date": "07/08/2019 12:50:26", "content": "

Good summary!

", "votes": null, "replies": [] }, { "id": 894485, "author_name": "huangchien", "author_url": "", "post_date": "06/20/2020 13:19:28", "content": "

Thank you for sharing this.\nI found that sometime the combination of several models may lead the hyper-parameter difficult to fine-tune, these tips for training GAN are worthy to follow as initialization configuration.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "570296": "Had been going through GANs lately and read a few tricks to help with GAN. Did tried it locally and now look forward to implementing the same in the competition. \n\n### 1. Normalize the input:\n- normalize the images between -1 and 1\n- Tanh as the last layer of the generator output\n\n### 2. Batchnorm\n- Construct different mini-batches for real and fake, i.e. each mini-batch needs to contain only all real images or all generated images.\n- when batchnorm is not an option use instance normalization (for each sample, subtract mean and divide by standard deviation).\n\n### 3. Avoid Sparse Gradients\n- the stability of the GAN game suffers if you have sparse gradients\n- LeakyReLU = good (in both G and D)\n- For Downsampling, use: Average Pooling, Conv2d + stride\n\n### 4: Use the ADAM Optimizer\n- Perhaps the real papers might have a different opinion, this literally makes a remarkable improvement\n- Use SGD for discriminator and ADAM for generator. \n- To decrease instability issues decrease the learning rate to 0.0002 (from 0.001) and the momentum/beta1 to 0.5 (from 0.9) for Adam.\n\n\nApart from these, I also came across a few blogs that would also help:\n- [Instance Noise: A trick for stabilising GAN training](https://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/)\n- [Improved technique for training GAN](https://arxiv.org/pdf/1606.03498)\n\nReferences:\n1. [Soumith GanHacks](https://github.com/soumith/ganhacks)\n2. [Unsupervised representational learning](https://github.com/sarathknv/papers/blob/master/neural-nets/Unsupervised_Representation_Learning_with_Deep_Convolutional_Generative_Adversarial_Networks.md)\n3. [Stack GAN](https://github.com/sarathknv/papers/blob/master/neural-nets/StackGAN.md)", "570312": "Thanks a ton for sharing this . I read somewhere that people have got away with pooling layers in GAN since it reduces spatial features . \nAlso , I have seen in this competition that the different learning rate between generator and discriminator really helps convergence . \nHave you tried SGD and ADAM as discriminator for this competition ? I have tried AdaBound in one of the kernels and got a mode collapse .", "570321": "Yes @phoenix9032 you are absolutely right. Differentiable learning rates really improve the model performances with training. And yes I have tried the SGD and ADAM and the results are comparitively better. Will share the generated results soon.\nI am a little eager to know how you found about the _mode collapse_ problem in with your model?", "570325": "Hi @thanatoz . I was working with Adam for both discriminator and generator for DCGAN and executed it for 270 epochs or so .It generated images with FID of 134 . Later I wanted to test AdaBound and just replaced the Adam with the same . After 270 epochs when i saw the results it was only black boxes with a tinge of different colors inside that .", "570349": "This is a copy paste from https://github.com/soumith/ganhacks right ?", "570386": "Not entirely a copy paste @theoviel. But I have added required references to it now. I only mentioned tricks that helped me and not complete writeup. Soumith's work is exceptional anyways.", "570391": "The default learning rate could be a reason for this. Try to lower the learning rate and refer to Fastai notebooks for the same. Jeremy howard has explained it pretty well. Also refer to this paper if you want:\n[Unsupervised representational learning](https://github.com/sarathknv/papers/blob/master/neural-nets/Unsupervised_Representation_Learning_with_Deep_Convolutional_Generative_Adversarial_Networks.md)", "570394": "Thank you . Very valuable suggestion . I will try out stuff what I have understood :) . I am a very novice person to machine learning itself . This competition also increasing my patience a lot .. Sometimes after 7 hours the commit just fails .. its painful :D", "570419": "I am glad that I was some help to you. If you want, I would suggest you to go through [Jeremy Howard's explanation on GAN](https://youtu.be/9spwoDYwW_I?t=4214) and [his notebook](https://github.com/fastai/fastai/blob/master/courses/dl2/wgan.ipynb). You can also look at my codes which are novice in my notebook if you haven't: [Let's generate more faces](https://www.kaggle.com/thanatoz/lets-generate-more-faces-gan)", "570437": "Okok, thanks for adding the ref though!", "570551": "Good summary!", "571553": "phoenix9032 I would recommend to you before commiting the kernel commit it with all the code unchanged except for training for only one epoch, this way you will check that all phases of your pipeline are working well (reading data, training, image generation). If you have code that runs once per several epochs (checkpoint storing for example), make sure that your test commit covers all your code at least once. \nHappy kaggling!)", "894485": "Thank you for sharing this.\nI found that sometime the combination of several models may lead the hyper-parameter difficult to fine-tune, these tips for training GAN are worthy to follow as initialization configuration.", "1109653": "\"Construct different mini-batches for real and fake\"\nAnyone know why this is? From my experience too if you try to mix them in batches it completely messes up the discriminator (when BN is used), and I've seen conference presentations saying not do that. In fact this issue with batchNorm is pretty much the only thing consistent about GANs. :) On the stuff about Adam and LeakyReLU you'll see conflicting advice/results out there, but if you mix fake/real with batchNorm it will mess things up every time. However I'm still to understand why. The purpose of the discriminator is to \"discriminate\" fake from real, and it does it per-sample not per batch, so why would we throw it a softball by only packing the batches uniformly?" }, "source": "meta" }