# BigGAN solution - public LB 15

First of all, I would like to thank organizers for such a wonderful competition. Generative modeling is a really interesting topic, and I hope to see more such competitions in the future. Also thanks to all participants for sharing their ideas and keeping the forum active :)
Here is a quick summary of my solution...
Search for a good model
When I entered this competition, I had zero experience with GANs or other generative models. So I started with simple models like DCGAN or WGAN, but quickly switched to more complex solutions like Self-Attention GAN. The best score I could get out of SAGAN was 39-40 MiFID, and then I got stuck there. So I continued my search for more efficient models. I played with NVIDIA's StyleGAN for a few days, but didn't get any positive results using it. And then a found BigGAN. It took me a while to take the full advantage of this new model given the amount of settings it had.
Also I should mention the great pytorch repo, created by one of the authors of BigGAN, which made using this model a lot easier for me:
[https://github.com/ajbrock/BigGAN-PyTorch](https://github.com/ajbrock/BigGAN-PyTorch)
Data processing
My goal with image transformations at first was to keep as much usefull information as possible. Using simple center cropping often leaves "headless" dogs, which might be a bit less recognisable by the scoring network. So I try to expand the usefull image area as much as possible and capture the whole dog or at least the more recognisable parts like head with upper body. I ended up not using center cropping at all and padding tall or wide images most of the time. Given how the FID metric is sensitive to missing modes, it's reasonable to keep most of the images, even the unhandy ones.
I focused mostly on geometric transformation. Pixel-level augmentation didn't bring any improvements for me. Even the most gentle changes to brightness or contrast usually degraded MiFID metric's value. 
Also the choice of interpolation method matters, and it was a hard one for me. The popular and recommended way of downsizing images in OpenCV is using INTER_AREA, which gives the most visually appealing smooth images. But the MiFID value of resulting images usually is 1 point less than for other methods (like 10.x vs 9.x), and generated images I get with it are a bit worse. So I ended up using another resizing methods, which led to more sharp but noisy images. Which brought me to using denoise postprocessing algorithm on generated images :)
Hyperparameters
I tried different number of channels for D and G, different placements of attention layer. I found it a lot easier to use generator with less parameters than default settings. I also ended up using unbalanced networks with D having twice as much channels in conv layers than G (I used mostly 32-64, but tried 28-56, 36-64, 48-72, 48-96). I didn't find any benefits using more complex generator. Additional complexity always slowed training down and resulted with a worse score with no clear visual improvements. My guess is the current task is too simple for a model designed to represent complex datasets likes ImageNet one. I guess equal reduction in channels in D and G should also work fine, but I left D with additional channels because it didn't influenced iteration speed much.
Despite the benefits of using higher batch sizes, described in BigGAN paper, I ended up using only 32 :) I believe, that the total number of iterations matters too, and a limited time window presents us with the choice of whether we want to update our models more often or more precise.
Also I use shared class-embeddings layers in G with hierarchical noise injection.
Other things I tried:
1. BigGAN-deep. It never surpassed the perfomance of the original model for me.
2. truncation of z-vector. Didn't get much with this architecture, though for BigGAN-deep there were some improvements.
3. stochastic weight averaging (SWA). I usually get about +1-1.5 bigger MiFID metric value using it. But the choise of starting point matters and shifts all the time while I change data pipeline and other settings. So I decided to left EMA which came with the original code.
4. cyclic learning rates with cosine annealing and soft/hard warmups. With some settings it has a positive impact but tend to lack consistency.
5. different batch norm substitutes. Didn't spend much time on it, might be an area of potential improvements.
6. other activation functions. I remember gelu didn't perform very well, but it worked :)
7. relativistic losses. I tried it with several different architectures. For DCGAN it worked nicely, with SAGAN it didn't make much difference. I didn't managed to make RaHinge loss to work with BigGAN model. It might be a consequence of other setting I had chosen (like stronger discriminator compared to generator net).
8. other possible z distrubutions as described in the BigGAN paper (Appendix E). Bernoulli didn't work at all, censored Gaussian gave a bit worse results than usual normal distribution.
You can find full code for my solution here:
[https://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3](https://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3)
[https://www.kaggle.com/dvorobiev/doggies-biggan-sub-final](https://www.kaggle.com/dvorobiev/doggies-biggan-sub-final)
[https://github.com/dmitry-vorobiev/kaggle-generative-dog-images](https://github.com/dmitry-vorobiev/kaggle-generative-dog-images)