# When BigGAN met StyleGAN (public 12.4 solution)

It's an interesting competition, and this is the first time I write a solution without knowing private LB score. Great thanks to kaggle for the efforts to host this competition and quick response to the community by
[@juliaelliott](/juliaelliott)
,
[@wendykan](/wendykan)
.
My solution may be described as a mix of StyleGAN and BigGAN.
Respect to the authors of the  papers and these two github repositories:
[https://github.com/ajbrock/BigGAN-PyTorch](https://github.com/ajbrock/BigGAN-PyTorch)
[https://github.com/rosinality/style-based-gan-pytorch](https://github.com/rosinality/style-based-gan-pytorch)
Challenges
:
1. Limited resources and limited time  to train a GAN from scratch.
2. A new kind of challenge with new rules and settings. It took me 2 weeks to choose between this one and the Youbute 8M video challenge.
Solution
:
I tried DCGAN and ACGAN to get familiar with this competition at first , and then select papers related, I selected about 40 papers which might be useful.
I was impressed by the results of the StyleGAN and tried to make it work more efficiently so I could make it run in a kernel, but I found Progressive GAN and its followers were inefficient to train,  but I still spent about 10 days to tune it and got  FID about 35 , I guessed that this score was not enough to top 10, but I thought I should spend more time on it to get understanding of every detail of it before I stepped forward.
Then I tried VQ-VAE2 and a lot of other mechanisms and I was not satisfied with the efficiency and the scores.
BigGAN was listed on the top of my most promising solutions, but the
BIG
,
DEEP
in the model name prevent me from trying it in a hurry and the paper stated that using more channels was important, I worried that it might be inefficient in this competition, so I planed to try it after I have tested others. To my surprise, the channels was not so important, when I decreased the G_ch and D_ch from 64 to 24, the FID score did not decrease too much and the FID score can easily reach below 30.
What's more, the project structure of the
[https://github.com/ajbrock/BigGAN-PyTorch](https://github.com/ajbrock/BigGAN-PyTorch)
is very good, we can do a lot of experiments by just modifying some parameters, so I could start many experiments and went out have a bicycle driving or swimming :)  After several experiments, my score reached about 24.
After I checked and compared the architecture and mechanisms of the  Big and Style GAN, I found that Spectral normalization and Class-conditional Batch  normalization were important to BigAN, so we need not use progressive training,  and BigGAN's G_shared and skip_z mechanisms were similar to the StyleGAN's style mechanism to some extend, so I made a lot of modifications to BigGAN and tried to move almost all important  mechanisms of StyleGAN to it. After many experiments, I found: Style mechanism could improve the score significantly and coverage more quickly, the Blur was also important, Noise was not so important, to save time, I did not add it to my networks.  Since I had more efficient and powerful  tools for the BigGAN, so I did not use skip_z.  I found mix-style mechanism of StyleGAN was not important in this competition, so I did not use it in my final submission. After these efforts and some hyper-parameters tuning, the score can reach about 17.
I thought I might use large image size to train the network, so I designed a 96x96 model instead of 64x64, I found it could improve the score.
I tried to use Attention including BigGAN's attention and CBAM  in the network, it increased the training time and the improvement to the score was not so big, so I did not use them
To my surprise, although the paper of BigGAN stated that the larger batch-size, the better result, I found 32 was the best batch-size. So the two most important results--large channels and bigger batch-size of the BigGAN were not suitable of this dataset, so my solution should be called
Style-Small-GAN
??  :smile::smile:
My best parameters can be described as the following command:
python train.py --shuffle --batch_size 32 --num_G_accumulations 1 --num_D_accumulations 1 --num_epochs 200 --num_D_steps 1 --G_lr 1e-4 --D_lr 6e-4 --dataset DogOrigin96 --bottom_width 6 --G_ortho 0.0 --G_attn 0 --D_attn 0 --G_init ortho --D_init ortho --ema --use_ema --ema_start 2000 --test_every 25 --save_every 10 --num_best_copies 5 --num_save_copies 2 --G_ch 24 --D_ch 24 --seed 0 --augment 1 --add_blur --add_style --crop_mode 8  --on_kaggle --experiment_name i96_ch24_hinge_ema_dstep1_bs32_noatt_glr0001_glr0006_aug_init_ortho_blur_style_origin_crop_mode8
The score  improved to
12.4
on public LB after these steps
In the last several days of this competition, I turned to the model's generalization and the data itself, I found that if I cropped square box containing the dogs only, it performed better than adding resized origin images, but, as kaggle will use another datasets and my models were trained on all-dogs datasets,  they might over-fit to the cropped all-dogs to some extend, when my models were applied on other images, the dogs might not be cropped,  so I added resized origin images to the datasets with some probability. The score on public LB of this augmentation is
13.4
, but I guess this model may perform better than 12.4 one.
I also added all kinds of mechanisms , such as mode seeking, all kinds of loss functions such as such as GDPP, normalizations, regularizations , initializations, and most of them did not work.
Most of mechanism I talked above are kept in the kernels and can be tested by using the parameters. If you want to have a try, 100 epochs in 4.5h can reach FID 18.x, and 125 epochs can reach FID 15.x
What can we do to improve the model if there are no limits on the resources and time?
1. I designed a mechanism to add dog count in an image to the model, that's to say, dog count embedding. It works, but it took more time. 
2. We can still cluster the images to 30-40 clusters, use these cluster id as other embedding, so, the dog on the grass, floor..... ( I did not test it )
3. We can use large sized images.
So the FID less than 10 is not so difficult to achieve if we have two GPUs or double time. 
1 and 2 can be described as Multi-Classes-Conditional Style GAN,  with the mixture of the Style and BigGAN, after some well designed experiments we may make some progress to the state of the art GANs?
You can check some of images, a type of dogs per line:
[assets/51b434c19a6cef346246c24d.jpg](assets/51b434c19a6cef346246c24d.jpg)
The link to my kernels:
[https://www.kaggle.com/bestfitting/i96-1e-4-6e-4-bs32-195](https://www.kaggle.com/bestfitting/i96-1e-4-6e-4-bs32-195)
[https://www.kaggle.com/bestfitting/i96-crop-mode3-dstep1-1e-4-6e-4-bs32](https://www.kaggle.com/bestfitting/i96-crop-mode3-dstep1-1e-4-6e-4-bs32)
Updated
The source codes in the  kennels above are not easy to read, you can check the normal python project structure version here
[https://github.com/bestfitting/kaggle/tree/master/gandogs](https://github.com/bestfitting/kaggle/tree/master/gandogs)
Please note that the progresses and the scores of my models are not a formal ablation study in a paper and I don't plan to write a paper either, I modified the parameters and other setting during these progresses, they were results of a series of complex experiment setting.