![\"image\"](\"http://playagricola.com/Kaggle/d12.png\")
Thank you Kaggle for hosting an exciting competition. It was tons of fun and a great learning experience. Thank you everyone for great discussions and code sharing.
\n\nI wish we had more time. There are many more ideas to try. My first model is an improvement from my previously posted LB 100 CGAN. I added more training examples by employing random cropping and added additional auxiliary classification loss to the discriminator. Furthermore precompiling the training loop with @tf.function allows twice as many training epochs by doubling the speed.
Kaggle notebook here. Version 16 is my old CGAN scoring LB 100. And version 17+ is my new ACGAN scoring LB 52+.
\n\nThe blue rectangles are the provided bounding boxes. I cropped all images to the yellow squares and resized them to 80x80 pixels. I chose a square that captures my best guess of where the dog head is and added padding beyond the bounding box to allow for random cropping during training.\n
Each original image was cropped to a 80x80 square that included 25% extra image beyond bounding box (yellow squares below). This extra room allows us to randomly choose a 64x64 crop within (red squares below) and mostly stay outside the bounding box (blue rectangles). \n![\"image\"](\"http://playagricola.com/Kaggle/d13.png\")
The Generator learns via RaLSGAN loss which compares the 1 output unit from real images with the 1 output unit from fake images. The Discriminator learns via basic GAN loss by comparing the 1 output neurons plus we add to that binary crossentropy of dense 121 against the class labels. This converts the CGAN into an ACGAN\n![\"image\"](\"http://playagricola.com/Kaggle/disc4.jpg\")
If I had more time I would add better regularization/normalization to my GAN. Then we could make the conv nets (gen and disc) bigger and train more without exploding. Next I would modulate batch norm with class labels and initial noise seed (like CBN-GAN, StyleGAN, and Self Mod).
\n\nWith only 2 days remaining, I didn't have time to improve my GAN further so I searched GitHub for a version of BigGAN! I found a model like BigGAN here. I copied and pasted the code into a Kaggle notebook and connected it to my training data pipeline described above. With no tuning it scored LB 47! BigGAN has the capability to score LB 12 as shown in this Kaggle notebook here. So I believe with more time, I could tune BigGAN to score under LB 30 in a 9 hour Kaggle kernel.
\n\n![\"image\"](\"http://playagricola.com/Kaggle/resgan281419.png\")
After a few experiments, my BigGAN achieves LB 42 with the following changes from default. Batch size to 32 (from 64), learning rates to Adam alpha=0.0003, beta1=0.2, beta2=0.9 (from 0.0002, 0, 0.9). Discriminator channels to 48 (from 64), generator channels to 48 (from 64), train for 40,000 iterations with exponential decay beginning at 32,000.
\n\nThe following papers helped me learn about GANs:\n* 2014 Basic GAN: Generative Adversarial Nets by Goodfellow, et al\n* 2016 GAN Tricks: Improved Techniques for Training GANs by Goodfellow, et al\n* 2016 GAN Tricks: NIPS 2016 Tutorial: Generative Adversarial Networks by Goodfellow\n* 2015 DCGAN: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks by Radford, et al.\n* 2016 LS-GAN: Least Squares Generative Adversarial Networks by Mao, et al.\n* 2018 Ra-GAN: The relativistic discriminator: a key element missing from standard GAN by Jolicoeur-Martineau\n* 2017 ProGAN: Progressive Growing of GANs for Improved Quality, Stability, and Variation by Karras, et al\n* 2017 WGAN: Wasserstein GAN by Arjovsky, et al.\n* 2017 WGAN-GP: Improved Training of Wasserstein GANs by Gulrajani, et al\n* 2018 StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks by Karras, et al\n* 2014 CGAN: Conditional Generative Adversarial Nets by Mirza, et al\n* 2016 SGAN: Semi-Supervised Learning with Generative Adversarial Networks by Odena\n* 2015 CatGAN: Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks by Springenberg\n* 2016 ACGAN: Conditional Image Synthesis With Auxiliary Classifier GANs by Odena, et al\n* 2018 CBN-GAN: cGANs with Projection Discriminator by Miyato, et al\n* 2018 SN-GAN: Spectral Normalization for Generative Adversarial Networks by Miyato, et al\n* 2018 BigGAN: Large Scale GAN Training for High Fidelity Natural Image Synthesis by Brock, et al\n* 2018 Self Mod: On Self Modulation for Generative Adversarial Networks by Chen, et al
\n\nTo build a great GAN, you first build two great convolutional networks (Gen and Disc). Use either ResNet (BigGAN) or VGGNet (DCGAN) architecture. Next choose a loss function. Hinge loss is the current favorite. (Others are GAN, RaLSGAN, WGAN, ACGAN, etc) Then to prevent training from exploding, add normalization/regularization such as weight clipping, gradient penalty (WGAN-GP), batch norm, or spectral normalization (SN-GAN), etc. For best performance add modulation via batch norm using either class labels (CGAN), style labels (StyleGAN), or random noise labels (Self Mod). Next optimize your hyperparameters; alpha, beta1 and beta2 of your Adam optimizer. Choose if the discriminator will train extra epochs versus generator. Lastly choose a dataset, add more data with data augmentation, and start training on a fast GPU !
", "messages": [ { "id": 601146, "postDate": "2019-08-17T04:57:12.910Z", "content": "Thank you Kaggle for hosting an exciting competition. It was tons of fun and a great learning experience. Thank you everyone for great discussions and code sharing.
\n\nI wish we had more time. There are many more ideas to try. My first model is an improvement from my previously posted LB 100 CGAN. I added more training examples by employing random cropping and added additional auxiliary classification loss to the discriminator. Furthermore precompiling the training loop with @tf.function allows twice as many training epochs by doubling the speed.
Kaggle notebook here. Version 16 is my old CGAN scoring LB 100. And version 17+ is my new ACGAN scoring LB 52+.
\n\nThe blue rectangles are the provided bounding boxes. I cropped all images to the yellow squares and resized them to 80x80 pixels. I chose a square that captures my best guess of where the dog head is and added padding beyond the bounding box to allow for random cropping during training.\n
Each original image was cropped to a 80x80 square that included 25% extra image beyond bounding box (yellow squares below). This extra room allows us to randomly choose a 64x64 crop within (red squares below) and mostly stay outside the bounding box (blue rectangles). \n
The Generator learns via RaLSGAN loss which compares the 1 output unit from real images with the 1 output unit from fake images. The Discriminator learns via basic GAN loss by comparing the 1 output neurons plus we add to that binary crossentropy of dense 121 against the class labels. This converts the CGAN into an ACGAN\n
If I had more time I would add better regularization/normalization to my GAN. Then we could make the conv nets (gen and disc) bigger and train more without exploding. Next I would modulate batch norm with class labels and initial noise seed (like CBN-GAN, StyleGAN, and Self Mod).
\n\nWith only 2 days remaining, I didn't have time to improve my GAN further so I searched GitHub for a version of BigGAN! I found a model like BigGAN here. I copied and pasted the code into a Kaggle notebook and connected it to my training data pipeline described above. With no tuning it scored LB 47! BigGAN has the capability to score LB 12 as shown in this Kaggle notebook here. So I believe with more time, I could tune BigGAN to score under LB 30 in a 9 hour Kaggle kernel.
\n\n
After a few experiments, my BigGAN achieves LB 42 with the following changes from default. Batch size to 32 (from 64), learning rates to Adam alpha=0.0003, beta1=0.2, beta2=0.9 (from 0.0002, 0, 0.9). Discriminator channels to 48 (from 64), generator channels to 48 (from 64), train for 40,000 iterations with exponential decay beginning at 32,000.
\n\nThe following papers helped me learn about GANs:\n* 2014 Basic GAN: Generative Adversarial Nets by Goodfellow, et al\n* 2016 GAN Tricks: Improved Techniques for Training GANs by Goodfellow, et al\n* 2016 GAN Tricks: NIPS 2016 Tutorial: Generative Adversarial Networks by Goodfellow\n* 2015 DCGAN: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks by Radford, et al.\n* 2016 LS-GAN: Least Squares Generative Adversarial Networks by Mao, et al.\n* 2018 Ra-GAN: The relativistic discriminator: a key element missing from standard GAN by Jolicoeur-Martineau\n* 2017 ProGAN: Progressive Growing of GANs for Improved Quality, Stability, and Variation by Karras, et al\n* 2017 WGAN: Wasserstein GAN by Arjovsky, et al.\n* 2017 WGAN-GP: Improved Training of Wasserstein GANs by Gulrajani, et al\n* 2018 StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks by Karras, et al\n* 2014 CGAN: Conditional Generative Adversarial Nets by Mirza, et al\n* 2016 SGAN: Semi-Supervised Learning with Generative Adversarial Networks by Odena\n* 2015 CatGAN: Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks by Springenberg\n* 2016 ACGAN: Conditional Image Synthesis With Auxiliary Classifier GANs by Odena, et al\n* 2018 CBN-GAN: cGANs with Projection Discriminator by Miyato, et al\n* 2018 SN-GAN: Spectral Normalization for Generative Adversarial Networks by Miyato, et al\n* 2018 BigGAN: Large Scale GAN Training for High Fidelity Natural Image Synthesis by Brock, et al\n* 2018 Self Mod: On Self Modulation for Generative Adversarial Networks by Chen, et al
\n\nTo build a great GAN, you first build two great convolutional networks (Gen and Disc). Use either ResNet (BigGAN) or VGGNet (DCGAN) architecture. Next choose a loss function. Hinge loss is the current favorite. (Others are GAN, RaLSGAN, WGAN, ACGAN, etc) Then to prevent training from exploding, add normalization/regularization such as weight clipping, gradient penalty (WGAN-GP), batch norm, or spectral normalization (SN-GAN), etc. For best performance add modulation via batch norm using either class labels (CGAN), style labels (StyleGAN), or random noise labels (Self Mod). Next optimize your hyperparameters; alpha, beta1 and beta2 of your Adam optimizer. Choose if the discriminator will train extra epochs versus generator. Lastly choose a dataset, add more data with data augmentation, and start training on a fast GPU !
", "rawMarkdown": "# Generative Dog Images Competition\nThank you Kaggle for hosting an exciting competition. It was tons of fun and a great learning experience. Thank you everyone for great discussions and code sharing. \n \nI wish we had more time. There are many more ideas to try. My first model is an improvement from my previously posted LB 100 CGAN. I added more training examples by employing random cropping and added additional auxiliary classification loss to the discriminator. Furthermore precompiling the training loop with `@tf.function` allows twice as many training epochs by doubling the speed.\n# Solution 1: TensorFlow AC-RaLS-DCGAN, LB 52\nKaggle notebook [here][1]. Version 16 is my old CGAN scoring LB 100. And version 17+ is my new ACGAN scoring LB 52+.\n## Image Preprocess\nThe blue rectangles are the provided bounding boxes. I cropped all images to the yellow squares and resized them to 80x80 pixels. I chose a square that captures my best guess of where the dog head is and added padding beyond the bounding box to allow for random cropping during training.\n\n## Data Augmentation, Random Cropping\nEach original image was cropped to a 80x80 square that included 25% extra image beyond bounding box (yellow squares below). This extra room allows us to randomly choose a 64x64 crop within (red squares below) and mostly stay outside the bounding box (blue rectangles). \n\n## Additional Auxiliary Classification Loss\nThe Generator learns via RaLSGAN loss which compares the 1 output unit from real images with the 1 output unit from fake images. The Discriminator learns via basic GAN loss by comparing the 1 output neurons plus we add to that binary crossentropy of dense 121 against the class labels. This converts the CGAN into an ACGAN\n\n## TO DO: Add Spectral Normalization and Batch Norm Modulation\nIf I had more time I would add better regularization/normalization to my GAN. Then we could make the conv nets (gen and disc) bigger and train more without exploding. Next I would modulate batch norm with class labels and initial noise seed (like CBN-GAN, StyleGAN, and Self Mod).\n# Solution 2: BigGAN, SN-CBN-ResNetGAN, LB 47\nWith only 2 days remaining, I didn't have time to improve my GAN further so I searched GitHub for a version of BigGAN! I found a model like BigGAN [here][3]. I copied and pasted the code into a Kaggle notebook and connected it to my training data pipeline described above. With no tuning it scored LB 47! BigGAN has the capability to score LB 12 as shown in this Kaggle notebook [here][2]. So I believe with more time, I could tune BigGAN to score under LB 30 in a 9 hour Kaggle kernel.\n## My BigGAN Dogs\n\n## Update\nAfter a few experiments, my BigGAN achieves LB 42 with the following changes from default. Batch size to 32 (from 64), learning rates to Adam alpha=0.0003, beta1=0.2, beta2=0.9 (from 0.0002, 0, 0.9). Discriminator channels to 48 (from 64), generator channels to 48 (from 64), train for 40,000 iterations with exponential decay beginning at 32,000.\n# References\nThe following papers helped me learn about GANs:\n* 2014 Basic GAN: [Generative Adversarial Nets][19] by Goodfellow, et al\n* 2016 GAN Tricks: [Improved Techniques for Training GANs][20] by Goodfellow, et al\n* 2016 GAN Tricks: [NIPS 2016 Tutorial: Generative Adversarial Networks][21] by Goodfellow\n* 2015 DCGAN: [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks][4] by Radford, et al.\n* 2016 LS-GAN: [Least Squares Generative Adversarial Networks][5] by Mao, et al.\n* 2018 Ra-GAN: [The relativistic discriminator: a key element missing from standard GAN][6] by Jolicoeur-Martineau\n* 2017 ProGAN: [Progressive Growing of GANs for Improved Quality, Stability, and Variation][7] by Karras, et al\n* 2017 WGAN: [Wasserstein GAN][8] by Arjovsky, et al.\n* 2017 WGAN-GP: [Improved Training of Wasserstein GANs][9] by Gulrajani, et al\n* 2018 StyleGAN: [A Style-Based Generator Architecture for Generative Adversarial Networks][10] by Karras, et al\n* 2014 CGAN: [Conditional Generative Adversarial Nets][11] by Mirza, et al\n* 2016 SGAN: [Semi-Supervised Learning with Generative Adversarial Networks][18] by Odena\n* 2015 CatGAN: [Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks][12] by Springenberg\n* 2016 ACGAN: [Conditional Image Synthesis With Auxiliary Classifier GANs][13] by Odena, et al\n* 2018 CBN-GAN: [cGANs with Projection Discriminator][14] by Miyato, et al\n* 2018 SN-GAN: [Spectral Normalization for Generative Adversarial Networks][15] by Miyato, et al\n* 2018 BigGAN: [Large Scale GAN Training for High Fidelity Natural Image Synthesis][16] by Brock, et al\n* 2018 Self Mod: [On Self Modulation for Generative Adversarial Networks][17] by Chen, et al\n# Summary\nTo build a great GAN, you first build two great convolutional networks (Gen and Disc). Use either ResNet (BigGAN) or VGGNet (DCGAN) architecture. Next choose a loss function. Hinge loss is the current favorite. (Others are GAN, RaLSGAN, WGAN, ACGAN, etc) Then to prevent training from exploding, add normalization/regularization such as weight clipping, gradient penalty (WGAN-GP), batch norm, or spectral normalization (SN-GAN), etc. For best performance add modulation via batch norm using either class labels (CGAN), style labels (StyleGAN), or random noise labels (Self Mod). Next optimize your hyperparameters; alpha, beta1 and beta2 of your Adam optimizer. Choose if the discriminator will train extra epochs versus generator. Lastly choose a dataset, add more data with data augmentation, and start training on a fast GPU !\n\n[19]: https://arxiv.org/abs/1406.2661\n[20]: https://arxiv.org/abs/1606.03498\n[21]: https://arxiv.org/abs/1701.00160\n[4]: https://arxiv.org/abs/1511.06434\n[5]: https://arxiv.org/abs/1611.04076\n[6]: https://arxiv.org/abs/1807.00734\n[7]: https://arxiv.org/abs/1710.10196\n[8]: https://arxiv.org/abs/1701.07875\n[9]: https://arxiv.org/abs/1704.00028\n[10]: https://arxiv.org/abs/1812.04948\n[11]: https://arxiv.org/abs/1411.1784\n[12]: https://arxiv.org/abs/1511.06390\n[13]: https://arxiv.org/abs/1610.09585\n[14]: https://arxiv.org/abs/1802.05637\n[15]: https://arxiv.org/abs/1802.05957\n[16]: https://arxiv.org/abs/1809.11096\n[17]: https://arxiv.org/abs/1810.01365\n[18]: https://arxiv.org/abs/1606.01583\n[1]: https://www.kaggle.com/cdeotte/dog-breed-acgan-lb-52\n[2]: https://www.kaggle.com/cdeotte/big-gan-256-lb-15\n[3]: https://github.com/pfnet-research/sngan_projection\n", "votes": 25 }, { "id": 601151, "postDate": "2019-08-17T05:07:33.020Z", "content": "Thank You @cdeotte for sharing your Amazing Work.......!!
", "rawMarkdown": "Thank You @cdeotte for sharing your Amazing Work.......!!", "votes": 3 }, { "id": 618938, "postDate": "2019-09-05T17:01:09.717Z", "content": "Thanks Chris. all your posts are informative and helped me a lot to grasp the concept.
", "rawMarkdown": "Thanks Chris. all your posts are informative and helped me a lot to grasp the concept.", "votes": 1 }, { "id": 603124, "postDate": "2019-08-19T22:06:48.243Z", "content": "Cool idea.
", "rawMarkdown": "Cool idea.", "votes": 1 }, { "id": 602634, "postDate": "2019-08-19T09:51:01.830Z", "content": "I admire your dogs Chris, even the one that kinda looks like a brown puddle, amazing work.
", "rawMarkdown": "I admire your dogs Chris, even the one that kinda looks like a brown puddle, amazing work.", "votes": 1 }, { "id": 602272, "postDate": "2019-08-18T21:03:57.510Z", "content": "Thanks for sharing. \nI used similar approach for random cropping amongst other augmentations for a while, but at the end left it only for wide images
", "rawMarkdown": "Thanks for sharing. \nI used similar approach for random cropping amongst other augmentations for a while, but at the end left it only for wide images", "votes": 1 }, { "id": 602048, "postDate": "2019-08-18T13:55:28.847Z", "content": "Chris! Thanks for your sharing!!\nYou are really amazing!
", "rawMarkdown": "Chris! Thanks for your sharing!!\nYou are really amazing!", "votes": 1 }, { "id": 601430, "postDate": "2019-08-17T15:53:08.010Z", "content": "Thank you for sharing your insights chris. I just have one question, how did you tune your GAN? For me it was mostly hit and trial and I never even touched beta1 and beta2 (them being 0.5 and 0.999). Why/How did you decide to use beta1=0.2, beta2=0.9?
", "rawMarkdown": "Thank you for sharing your insights chris. I just have one question, how did you tune your GAN? For me it was mostly hit and trial and I never even touched beta1 and beta2 (them being 0.5 and 0.999). Why/How did you decide to use beta1=0.2, beta2=0.9?", "votes": 1, "replies": [ { "id": 601498, "postDate": "2019-08-17T17:39:48.773Z", "content": "Typically VGGNet (DCGAN) style GAN architecture use beta1=0.5, beta2=0.999 while ResNet (BigGAN) style GAN use beta1=0, beta2=0.9.
The question is whether you would benefit from more training. When trying to train a BigGAN in under 9 hours, the issue is that it doesn't have enough time to train. Most public kernels are shallow DCGAN and seem to have no trouble training in under 9 hours.
\n\nVariable beta1 is training momentum. Without momentum (beta1=0) training is fast when the gradient is large and slower when gradient is small. With momentum, training stays at a more constant fast pace regardless of gradient size. If you set beta1=0.5 for ResNet GAN it seems to train too fast and explode. However I added a little momentum (beta1=0.2) to speed things up and it helped without exploding.
I also increased alpha=0.0002 to alpha=0.0003 which is the learning rate and that helped. I decreased the size of the ResNet by using 48 channels instead of 64. There was no decrease in image quality but this shortened the iteration train time allowing for more iterations within 9 hours. Similarly a batch size of 32 is quicker than a batch of 64 with no decrease in quality. These two changes allow for 40000 iterations in under 9 hours.
Standard BigGAN does 1 generator update followed by 5 discriminator updates. All at alpha=0.0002. I'm now trying to do 1 generator at alpha=0.0002 followed by 2 discriminator at alpha=0.0005 because 2*0.0005 = 5*0.0002. If this works I can do 80000 iterations within 9 hours instead of 40000.
Currently my BigGAN is at LB 42. I'll let you know if 80000 iterations improves that.
", "rawMarkdown": "Typically VGGNet (DCGAN) style GAN architecture use `beta1=0.5`, `beta2=0.999` while ResNet (BigGAN) style GAN use `beta1=0`, `beta2=0.9`.\n\nThe question is whether you would benefit from more training. When trying to train a BigGAN in under 9 hours, the issue is that it doesn't have enough time to train. Most public kernels are shallow DCGAN and seem to have no trouble training in under 9 hours. \n\nVariable `beta1` is training momentum. Without momentum (`beta1=0`) training is fast when the gradient is large and slower when gradient is small. With momentum, training stays at a more constant fast pace regardless of gradient size. If you set `beta1=0.5` for ResNet GAN it seems to train too fast and explode. However I added a little momentum (`beta1=0.2`) to speed things up and it helped without exploding.\n\nI also increased `alpha=0.0002` to `alpha=0.0003` which is the learning rate and that helped. I decreased the size of the ResNet by using 48 channels instead of 64. There was no decrease in image quality but this shortened the iteration train time allowing for more iterations within 9 hours. Similarly a batch size of 32 is quicker than a batch of 64 with no decrease in quality. These two changes allow for 40000 iterations in under 9 hours.\n\nStandard BigGAN does 1 generator update followed by 5 discriminator updates. All at `alpha=0.0002`. I'm now trying to do 1 generator at `alpha=0.0002` followed by 2 discriminator at `alpha=0.0005` because `2*0.0005 = 5*0.0002`. If this works I can do 80000 iterations within 9 hours instead of 40000.\n\nCurrently my BigGAN is at LB 42. I'll let you know if 80000 iterations improves that.", "votes": 2 }, { "id": 601563, "postDate": "2019-08-17T20:12:54.040Z", "content": "I see, great insights! I had similar thoughts about the adam's constants but I never tried changing them.\nThank you for such a detailed explanation!
", "rawMarkdown": "I see, great insights! I had similar thoughts about the adam's constants but I never tried changing them.\nThank you for such a detailed explanation!" } ] }, { "id": 601378, "postDate": "2019-08-17T14:24:16.187Z", "content": "I wonder how fast one can train a GAN to get images of 256^2, or bigger size? I've seen from BigGAN (and similar) papers that they spend many days on 4-8 GPUs to achieve high-resolution results, i.e., 512^2 and 1024^2. Do you think that the architectures and tricks discovered in this competition could be applied to reduce the training times?
", "rawMarkdown": "I wonder how fast one can train a GAN to get images of 256^2, or bigger size? I've seen from BigGAN (and similar) papers that they spend many days on 4-8 GPUs to achieve high-resolution results, i.e., 512^2 and 1024^2. Do you think that the architectures and tricks discovered in this competition could be applied to reduce the training times?", "votes": 1, "replies": [ { "id": 601406, "postDate": "2019-08-17T15:07:44.837Z", "content": "There are brilliant solutions here at Kaggle, so yes I believe the winners (Bestfitting, Dmitry, Yabea, MeisterMorxrc et al, Theo, Alvaro) discovered techniques that could advance state of the art GANs.
\n\nHowever if you compare the FID 12 images from BigGAN 128 here with the FID 12 images from Bestfitting's model here, you will notice that BigGAN 128 appear better to the eye which suggests that similar FID doesn't indicate that both models are now equal.
", "rawMarkdown": "There are brilliant solutions here at Kaggle, so yes I believe the winners (Bestfitting, Dmitry, Yabea, MeisterMorxrc et al, Theo, Alvaro) discovered techniques that could advance state of the art GANs.\n\nHowever if you compare the FID 12 images from BigGAN 128 [here][1] with the FID 12 images from Bestfitting's model [here][2], you will notice that BigGAN 128 appear better to the eye which suggests that similar FID doesn't indicate that both models are now equal.\n\n[1]: https://www.kaggle.com/cdeotte/big-gan-128-lb-12\n[2]: https://www.kaggle.com/c/generative-dog-images/discussion/104281#600211", "votes": 1 } ] }, { "id": 602640, "postDate": "2019-08-19T10:02:03.137Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 602752, "postDate": "2019-08-19T12:59:35.147Z", "content": "Nice insigh！thank you, Chris！
", "rawMarkdown": "Nice insigh！thank you, Chris！", "votes": 1 }, { "id": 601176, "postDate": "2019-08-17T06:12:16.283Z", "content": "Thanks, Chris. You are amazing.
", "rawMarkdown": "Thanks, Chris. You are amazing.", "votes": 1 } ], "comments": [ { "id": 601151, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-08-17T05:07:33.020000", "content": "Thank You @cdeotte for sharing your Amazing Work.......!!
", "votes": 3, "replies": [] }, { "id": 618938, "author_name": "Sara", "author_url": "", "post_date": "2019-09-05T17:01:09.717000", "content": "Thanks Chris. all your posts are informative and helped me a lot to grasp the concept.
", "votes": 1, "replies": [] }, { "id": 603124, "author_name": "Rob Reynolds", "author_url": "", "post_date": "2019-08-19T22:06:48.243000", "content": "Cool idea.
", "votes": 1, "replies": [] }, { "id": 602634, "author_name": "Ed", "author_url": "", "post_date": "2019-08-19T09:51:01.830000", "content": "I admire your dogs Chris, even the one that kinda looks like a brown puddle, amazing work.
", "votes": 1, "replies": [] }, { "id": 602272, "author_name": "Dmitry Vorobiev", "author_url": "", "post_date": "2019-08-18T21:03:57.510000", "content": "Thanks for sharing. \nI used similar approach for random cropping amongst other augmentations for a while, but at the end left it only for wide images
", "votes": 1, "replies": [] }, { "id": 602048, "author_name": "0xFunky", "author_url": "", "post_date": "2019-08-18T13:55:28.847000", "content": "Chris! Thanks for your sharing!!\nYou are really amazing!
", "votes": 1, "replies": [] }, { "id": 601430, "author_name": "timetraveller", "author_url": "", "post_date": "2019-08-17T15:53:08.010000", "content": "Thank you for sharing your insights chris. I just have one question, how did you tune your GAN? For me it was mostly hit and trial and I never even touched beta1 and beta2 (them being 0.5 and 0.999). Why/How did you decide to use beta1=0.2, beta2=0.9?
", "votes": 1, "replies": [ { "id": 601498, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-17T17:39:48.773000", "content": "Typically VGGNet (DCGAN) style GAN architecture use beta1=0.5, beta2=0.999 while ResNet (BigGAN) style GAN use beta1=0, beta2=0.9.
The question is whether you would benefit from more training. When trying to train a BigGAN in under 9 hours, the issue is that it doesn't have enough time to train. Most public kernels are shallow DCGAN and seem to have no trouble training in under 9 hours.
\n\nVariable beta1 is training momentum. Without momentum (beta1=0) training is fast when the gradient is large and slower when gradient is small. With momentum, training stays at a more constant fast pace regardless of gradient size. If you set beta1=0.5 for ResNet GAN it seems to train too fast and explode. However I added a little momentum (beta1=0.2) to speed things up and it helped without exploding.
I also increased alpha=0.0002 to alpha=0.0003 which is the learning rate and that helped. I decreased the size of the ResNet by using 48 channels instead of 64. There was no decrease in image quality but this shortened the iteration train time allowing for more iterations within 9 hours. Similarly a batch size of 32 is quicker than a batch of 64 with no decrease in quality. These two changes allow for 40000 iterations in under 9 hours.
Standard BigGAN does 1 generator update followed by 5 discriminator updates. All at alpha=0.0002. I'm now trying to do 1 generator at alpha=0.0002 followed by 2 discriminator at alpha=0.0005 because 2*0.0005 = 5*0.0002. If this works I can do 80000 iterations within 9 hours instead of 40000.
Currently my BigGAN is at LB 42. I'll let you know if 80000 iterations improves that.
", "votes": 2, "replies": [] }, { "id": 601563, "author_name": "timetraveller", "author_url": "", "post_date": "2019-08-17T20:12:54.040000", "content": "I see, great insights! I had similar thoughts about the adam's constants but I never tried changing them.\nThank you for such a detailed explanation!
", "votes": 0, "replies": [] } ] }, { "id": 601378, "author_name": "Ilia Zaitsev", "author_url": "", "post_date": "2019-08-17T14:24:16.187000", "content": "I wonder how fast one can train a GAN to get images of 256^2, or bigger size? I've seen from BigGAN (and similar) papers that they spend many days on 4-8 GPUs to achieve high-resolution results, i.e., 512^2 and 1024^2. Do you think that the architectures and tricks discovered in this competition could be applied to reduce the training times?
", "votes": 1, "replies": [ { "id": 601406, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-17T15:07:44.837000", "content": "There are brilliant solutions here at Kaggle, so yes I believe the winners (Bestfitting, Dmitry, Yabea, MeisterMorxrc et al, Theo, Alvaro) discovered techniques that could advance state of the art GANs.
\n\nHowever if you compare the FID 12 images from BigGAN 128 here with the FID 12 images from Bestfitting's model here, you will notice that BigGAN 128 appear better to the eye which suggests that similar FID doesn't indicate that both models are now equal.
", "votes": 1, "replies": [] } ] }, { "id": 602640, "author_name": "", "author_url": "", "post_date": "2019-08-19T10:02:03.137000", "content": "", "votes": 1, "replies": [] }, { "id": 602752, "author_name": "Sizhe Li", "author_url": "", "post_date": "2019-08-19T12:59:35.147000", "content": "Nice insigh！thank you, Chris！
", "votes": 1, "replies": [] }, { "id": 601176, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2019-08-17T06:12:16.283000", "content": "Thanks, Chris. You are amazing.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "601146": "# Generative Dog Images Competition\nThank you Kaggle for hosting an exciting competition. It was tons of fun and a great learning experience. Thank you everyone for great discussions and code sharing. \n \nI wish we had more time. There are many more ideas to try. My first model is an improvement from my previously posted LB 100 CGAN. I added more training examples by employing random cropping and added additional auxiliary classification loss to the discriminator. Furthermore precompiling the training loop with `@tf.function` allows twice as many training epochs by doubling the speed.\n# Solution 1: TensorFlow AC-RaLS-DCGAN, LB 52\nKaggle notebook [here][1]. Version 16 is my old CGAN scoring LB 100. And version 17+ is my new ACGAN scoring LB 52+.\n## Image Preprocess\nThe blue rectangles are the provided bounding boxes. I cropped all images to the yellow squares and resized them to 80x80 pixels. I chose a square that captures my best guess of where the dog head is and added padding beyond the bounding box to allow for random cropping during training.\n\n## Data Augmentation, Random Cropping\nEach original image was cropped to a 80x80 square that included 25% extra image beyond bounding box (yellow squares below). This extra room allows us to randomly choose a 64x64 crop within (red squares below) and mostly stay outside the bounding box (blue rectangles). \n\n## Additional Auxiliary Classification Loss\nThe Generator learns via RaLSGAN loss which compares the 1 output unit from real images with the 1 output unit from fake images. The Discriminator learns via basic GAN loss by comparing the 1 output neurons plus we add to that binary crossentropy of dense 121 against the class labels. This converts the CGAN into an ACGAN\n\n## TO DO: Add Spectral Normalization and Batch Norm Modulation\nIf I had more time I would add better regularization/normalization to my GAN. Then we could make the conv nets (gen and disc) bigger and train more without exploding. Next I would modulate batch norm with class labels and initial noise seed (like CBN-GAN, StyleGAN, and Self Mod).\n# Solution 2: BigGAN, SN-CBN-ResNetGAN, LB 47\nWith only 2 days remaining, I didn't have time to improve my GAN further so I searched GitHub for a version of BigGAN! I found a model like BigGAN [here][3]. I copied and pasted the code into a Kaggle notebook and connected it to my training data pipeline described above. With no tuning it scored LB 47! BigGAN has the capability to score LB 12 as shown in this Kaggle notebook [here][2]. So I believe with more time, I could tune BigGAN to score under LB 30 in a 9 hour Kaggle kernel.\n## My BigGAN Dogs\n\n## Update\nAfter a few experiments, my BigGAN achieves LB 42 with the following changes from default. Batch size to 32 (from 64), learning rates to Adam alpha=0.0003, beta1=0.2, beta2=0.9 (from 0.0002, 0, 0.9). Discriminator channels to 48 (from 64), generator channels to 48 (from 64), train for 40,000 iterations with exponential decay beginning at 32,000.\n# References\nThe following papers helped me learn about GANs:\n* 2014 Basic GAN: [Generative Adversarial Nets][19] by Goodfellow, et al\n* 2016 GAN Tricks: [Improved Techniques for Training GANs][20] by Goodfellow, et al\n* 2016 GAN Tricks: [NIPS 2016 Tutorial: Generative Adversarial Networks][21] by Goodfellow\n* 2015 DCGAN: [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks][4] by Radford, et al.\n* 2016 LS-GAN: [Least Squares Generative Adversarial Networks][5] by Mao, et al.\n* 2018 Ra-GAN: [The relativistic discriminator: a key element missing from standard GAN][6] by Jolicoeur-Martineau\n* 2017 ProGAN: [Progressive Growing of GANs for Improved Quality, Stability, and Variation][7] by Karras, et al\n* 2017 WGAN: [Wasserstein GAN][8] by Arjovsky, et al.\n* 2017 WGAN-GP: [Improved Training of Wasserstein GANs][9] by Gulrajani, et al\n* 2018 StyleGAN: [A Style-Based Generator Architecture for Generative Adversarial Networks][10] by Karras, et al\n* 2014 CGAN: [Conditional Generative Adversarial Nets][11] by Mirza, et al\n* 2016 SGAN: [Semi-Supervised Learning with Generative Adversarial Networks][18] by Odena\n* 2015 CatGAN: [Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks][12] by Springenberg\n* 2016 ACGAN: [Conditional Image Synthesis With Auxiliary Classifier GANs][13] by Odena, et al\n* 2018 CBN-GAN: [cGANs with Projection Discriminator][14] by Miyato, et al\n* 2018 SN-GAN: [Spectral Normalization for Generative Adversarial Networks][15] by Miyato, et al\n* 2018 BigGAN: [Large Scale GAN Training for High Fidelity Natural Image Synthesis][16] by Brock, et al\n* 2018 Self Mod: [On Self Modulation for Generative Adversarial Networks][17] by Chen, et al\n# Summary\nTo build a great GAN, you first build two great convolutional networks (Gen and Disc). Use either ResNet (BigGAN) or VGGNet (DCGAN) architecture. Next choose a loss function. Hinge loss is the current favorite. (Others are GAN, RaLSGAN, WGAN, ACGAN, etc) Then to prevent training from exploding, add normalization/regularization such as weight clipping, gradient penalty (WGAN-GP), batch norm, or spectral normalization (SN-GAN), etc. For best performance add modulation via batch norm using either class labels (CGAN), style labels (StyleGAN), or random noise labels (Self Mod). Next optimize your hyperparameters; alpha, beta1 and beta2 of your Adam optimizer. Choose if the discriminator will train extra epochs versus generator. Lastly choose a dataset, add more data with data augmentation, and start training on a fast GPU !\n\n[19]: https://arxiv.org/abs/1406.2661\n[20]: https://arxiv.org/abs/1606.03498\n[21]: https://arxiv.org/abs/1701.00160\n[4]: https://arxiv.org/abs/1511.06434\n[5]: https://arxiv.org/abs/1611.04076\n[6]: https://arxiv.org/abs/1807.00734\n[7]: https://arxiv.org/abs/1710.10196\n[8]: https://arxiv.org/abs/1701.07875\n[9]: https://arxiv.org/abs/1704.00028\n[10]: https://arxiv.org/abs/1812.04948\n[11]: https://arxiv.org/abs/1411.1784\n[12]: https://arxiv.org/abs/1511.06390\n[13]: https://arxiv.org/abs/1610.09585\n[14]: https://arxiv.org/abs/1802.05637\n[15]: https://arxiv.org/abs/1802.05957\n[16]: https://arxiv.org/abs/1809.11096\n[17]: https://arxiv.org/abs/1810.01365\n[18]: https://arxiv.org/abs/1606.01583\n[1]: https://www.kaggle.com/cdeotte/dog-breed-acgan-lb-52\n[2]: https://www.kaggle.com/cdeotte/big-gan-256-lb-15\n[3]: https://github.com/pfnet-research/sngan_projection\n", "601151": "Thank You @cdeotte for sharing your Amazing Work.......!!", "618938": "Thanks Chris. all your posts are informative and helped me a lot to grasp the concept.", "603124": "Cool idea.", "602634": "I admire your dogs Chris, even the one that kinda looks like a brown puddle, amazing work.", "602272": "Thanks for sharing. \nI used similar approach for random cropping amongst other augmentations for a while, but at the end left it only for wide images", "602048": "Chris! Thanks for your sharing!!\nYou are really amazing!", "601430": "Thank you for sharing your insights chris. I just have one question, how did you tune your GAN? For me it was mostly hit and trial and I never even touched beta1 and beta2 (them being 0.5 and 0.999). Why/How did you decide to use beta1=0.2, beta2=0.9?", "601378": "I wonder how fast one can train a GAN to get images of 256^2, or bigger size? I've seen from BigGAN (and similar) papers that they spend many days on 4-8 GPUs to achieve high-resolution results, i.e., 512^2 and 1024^2. Do you think that the architectures and tricks discovered in this competition could be applied to reduce the training times?", "602640": "", "602752": "Nice insigh！thank you, Chris！", "601176": "Thanks, Chris. You are amazing." } }