# 3rd place approach

My main goal for this competition is to get to know GAN technique. I often encountered the terminology of GAN in my readings, but did not know what GAN is about. This competition is a great opportunity for me to dive in and practice GAN technology. I want to thank Kaggle team for  hosting such an innovative competition and the great efforts to ensure fairness of the competition!
I followed this article "Generative Adversarial Networks - The Story So Far"(
[https://blog.floydhub.com/gans-story-so-far/](https://blog.floydhub.com/gans-story-so-far/)
) and tried various GAN techniques including DCGAN, CGAN, SAGAN, ProGAN, conditional ProGAN, StyleGAN and BigGAN. At last I settled down with SAGAN because it got me to the score 40+ in my first try and seemed more stable than other network structures. I mostly stick with it and tune it all the way from the score 40+ down to the public score 18. My model implementation is based on this github "self-attention-GAN-pytorch implementaion"(
[https://github.com/voletiv/self-attention-GAN-pytorch](https://github.com/voletiv/self-attention-GAN-pytorch)
)
Here is the list of things I tried in optimizing the model,
image processing
 a) bounding box cropping only;
 b) bounding box cropping for images with multiple objects,
 for images with single object and box_size / image_size >= 0.75, use original images instead of bounding box.
 c) bounding box cropping plus all original images
Among the three processing methods, b) gets me the best private score, c) gets me the best public score 18, but did not go well with the private dataset.
I learnt the image processing techniques from the kernels
[https://www.kaggle.com/cdeotte/supervised-generative-dog-net](https://www.kaggle.com/cdeotte/supervised-generative-dog-net)
and
[https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images](https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images)
. Special thanks to the kernel authors Chris Deotte and Nanashi!
losses 
I experimented with losses such as "standard-dcgan", "wgan-gp", "lsgan", "lsgan-with-sigmoid", "hinge", "relative-hinge". My experience is the best one for this problem with various models is the "standard dcgan" loss, the second best is "lsgan-with-sigmoid" loss.
Attention at different stages 
According to the paper "Self-Attention Generative Adversarial Networks"(
[assets/7da97bf4abae210f5a8b288a.pdf](assets/7da97bf4abae210f5a8b288a.pdf)
), self-attention at middle-to-high level feature maps achieve better performance because it receives more evidence and more freedom to choose conditions with larger feature maps. My experiences mostly concur with the paper findings. I have attention at 32x32 feature map for the generator, but at 16x16 feature map for the discriminator.
Training settings
 The learning rate for the discriminator is 0.0004, and the learning rate for the generator is 0.0002;
 batch size is 32(tried 64 but the result is not as stable as 32);
 label smoothing; thanks the kernel
[https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers](https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers)
and its author Nirjhar Roy.
 learning rate scheduler: CosineAnnealingWarmRestarts
 number of epoch: 170
5  the best latent space dimension size: 180
My best private score kernel(66.97):
[https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508](https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508)