{ "id": 102155, "title": "How to train your touchy GANs - Things that seem to work.", "url": "/competitions/generative-dog-images/discussion/102155", "author_name": "", "post_date": "2019-07-31T10:27:59.584631Z", "votes": 62, "comment_count": 71, "views": 0, "content": "

One thing I've learned from this competition is that GANs are very touchy. There is so much that can go wrong. In a world of million such wrong things I've decided to make a list of things that seem to go right for my traditional GAN model:

\n\n
    \n
  1. Smaller batch size (my score reduced by ~5-10 when I reduced bs from 64 to 32)
    2. \n
  2. Early stopping? (not sure if it's the right term) - I've been training my GANs by running them for ~9 hours (kernel limit) but that gives worse results. I calculated MiFID at every 100 epochs and found that my best GAN performs the best at 900 epochs rather than at kernel time limit (~1400 epochs).
    3. \n
  3. Loss function - I have only tried the traditional loss function and RaLSGAN's loss function. I didn't get much difference in scores. My current best GAN performs better with traditional loss.
    4. \n
  4. Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?). EDIT- I was doing something wrong, conditions work great!
    5. \n
  5. Bounded box cropping - Drastically improves traditional GAN score. Drastically deteriorates memorizer GAN score. I can't use both cropped images and original images together, I get CUDA memory errors. Has someone successfully done this? I think it might improve the score.
    6. \n
  6. Cosine learning rate annealing - I haven't explicitly tested it, I tested it with other changes and my score has improved.
    7. \n
  7. Label smoothing: It improves my score. I've added small randomness here too (as suggested in the GAN hacks repo by soumith chintala)
    8. \n
  8. Latent space: I get a better score for nz=128 than for nz=100. Also, truncated latent space improved my score a little.
    9. \n
  9. Spectral norm and pixel norm - I tried both of these together and it drastically improved my score.
    10. \n
  10. Transformations: Horizontal flip improves my score and [EDIT] rotation improves the score. I haven't tested other transformations.
    11. \n
\n\n

What has worked for you so far?

", "messages": [ { "id": "589002", "postDate": "07/31/2019 10:27:59", "content": "

One thing I've learned from this competition is that GANs are very touchy. There is so much that can go wrong. In a world of million such wrong things I've decided to make a list of things that seem to go right for my traditional GAN model:

\n\n
    \n
  1. Smaller batch size (my score reduced by ~5-10 when I reduced bs from 64 to 32)
    2. \n
  2. Early stopping? (not sure if it's the right term) - I've been training my GANs by running them for ~9 hours (kernel limit) but that gives worse results. I calculated MiFID at every 100 epochs and found that my best GAN performs the best at 900 epochs rather than at kernel time limit (~1400 epochs).
    3. \n
  3. Loss function - I have only tried the traditional loss function and RaLSGAN's loss function. I didn't get much difference in scores. My current best GAN performs better with traditional loss.
    4. \n
  4. Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?). EDIT- I was doing something wrong, conditions work great!
    5. \n
  5. Bounded box cropping - Drastically improves traditional GAN score. Drastically deteriorates memorizer GAN score. I can't use both cropped images and original images together, I get CUDA memory errors. Has someone successfully done this? I think it might improve the score.
    6. \n
  6. Cosine learning rate annealing - I haven't explicitly tested it, I tested it with other changes and my score has improved.
    7. \n
  7. Label smoothing: It improves my score. I've added small randomness here too (as suggested in the GAN hacks repo by soumith chintala)
    8. \n
  8. Latent space: I get a better score for nz=128 than for nz=100. Also, truncated latent space improved my score a little.
    9. \n
  9. Spectral norm and pixel norm - I tried both of these together and it drastically improved my score.
    10. \n
  10. Transformations: Horizontal flip improves my score and [EDIT] rotation improves the score. I haven't tested other transformations.
    11. \n
\n\n

What has worked for you so far?

", "rawMarkdown": "One thing I've learned from this competition is that GANs are very touchy. There is so much that can go wrong. In a world of million such wrong things I've decided to make a list of things that seem to go right for my traditional GAN model:\n\n\n1. Smaller batch size (my score reduced by ~5-10 when I reduced bs from 64 to 32)\n2. Early stopping? (not sure if it's the right term) - I've been training my GANs by running them for ~9 hours (kernel limit) but that gives worse results. I calculated MiFID at every 100 epochs and found that my best GAN performs the best at 900 epochs rather than at kernel time limit (~1400 epochs).\n3. Loss function - I have only tried the traditional loss function and RaLSGAN's loss function. I didn't get much difference in scores. My current best GAN performs better with traditional loss.\n4. Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?). EDIT- I was doing something wrong, conditions work great!\n5. Bounded box cropping - Drastically improves traditional GAN score. Drastically deteriorates memorizer GAN score. I can't use both cropped images and original images together, I get CUDA memory errors. Has someone successfully done this? I think it might improve the score.\n6. Cosine learning rate annealing - I haven't explicitly tested it, I tested it with other changes and my score has improved.\n7. Label smoothing: It improves my score. I've added small randomness here too (as suggested in the GAN hacks repo by soumith chintala)\n8. Latent space: I get a better score for nz=128 than for nz=100. Also, truncated latent space improved my score a little.\n9. Spectral norm and pixel norm - I tried both of these together and it drastically improved my score. \n10. Transformations: Horizontal flip improves my score and [EDIT] rotation improves the score. I haven't tested other transformations.\n\n\nWhat has worked for you so far?", "votes": null }, { "id": "589015", "postDate": "07/31/2019 10:45:15", "content": "

Most valuable insights in this juncture of the competition . My respect to you .\nDitto for number 2 . I didn't find an easy way to code MFID knowing that calculation takes some time . I just generate 10000 images every 200 epochs and calculate MFid post-facto .

\n\n

3,6,7,8,9 . True . \nFew additional questions :\nHave you tried to add more layers and increase filter size in discriminator ,generator? - not much improvement for me

\n\n

Adding more cycle for discriminator for every generator run . - ended up with less score .

", "rawMarkdown": "Most valuable insights in this juncture of the competition . My respect to you .\nDitto for number 2 . I didn't find an easy way to code MFID knowing that calculation takes some time . I just generate 10000 images every 200 epochs and calculate MFid post-facto .\n\n3,6,7,8,9 . True . \nFew additional questions :\nHave you tried to add more layers and increase filter size in discriminator ,generator? - not much improvement for me \n\nAdding more cycle for discriminator for every generator run . - ended up with less score .", "votes": null }, { "id": "589018", "postDate": "07/31/2019 10:50:11", "content": "

For my architecture (which I hope to make a public version today, but it basically https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers)

\n\n

scores between 60 and 90

\n\n

Batch size - 32 works better than 16\nDropout - seems to make worse results, but I haven't given up on it yet.\nEarly stopping - haven't fully tested, but I think I should implement. \nCosine annealing - trying to locate the optimal learning rate for prediction, at the end of the cycle, where the learning rate is significantly lowered than the initial learning rate, seems to perform worse.\nRandom crops (64,64) instead of resize(64,64) performs worse.

", "rawMarkdown": "For my architecture (which I hope to make a public version today, but it basically https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers)\n\nscores between 60 and 90\n\nBatch size - 32 works better than 16\nDropout - seems to make worse results, but I haven't given up on it yet.\nEarly stopping - haven't fully tested, but I think I should implement. \nCosine annealing - trying to locate the optimal learning rate for prediction, at the end of the cycle, where the learning rate is significantly lowered than the initial learning rate, seems to perform worse.\nRandom crops (64,64) instead of resize(64,64) performs worse.", "votes": null }, { "id": "589026", "postDate": "07/31/2019 11:00:47", "content": "

Glad to help :)\nI have learned a lot of things from your latest kernel. It has also given me so many more ideas to try out.

\n\n
\n

Have you tried to add more layers and increase filter size in discriminator ,generator?

\n
\n\n

Yes, I am currently running 3 kaggle kernels in parallel all identical but with different kernel sizes. I read in a blog that larger kernels work well for properly cropped images which is why I am trying larger kernel sizes right now. Will update the results here.

\n\n
\n

Adding more cycle for discriminator for every generator run

\n
\n\n

I am not familiar with this, haven't read anywhere that it might work.

", "rawMarkdown": "Glad to help :)\nI have learned a lot of things from your latest kernel. It has also given me so many more ideas to try out. \n\n>Have you tried to add more layers and increase filter size in discriminator ,generator?\n\nYes, I am currently running 3 kaggle kernels in parallel all identical but with different kernel sizes. I read in a blog that larger kernels work well for properly cropped images which is why I am trying larger kernel sizes right now. Will update the results here.\n\n>Adding more cycle for discriminator for every generator run\n\nI am not familiar with this, haven't read anywhere that it might work.", "votes": null }, { "id": "589029", "postDate": "07/31/2019 11:02:41", "content": "

Thanks for sharing\nI haven't tried dropout either, I am skeptical about them.\nI tried random cropping of all sorts only for memorizer GANs. Nothing worked.

", "rawMarkdown": "Thanks for sharing\nI haven't tried dropout either, I am skeptical about them.\nI tried random cropping of all sorts only for memorizer GANs. Nothing worked.", "votes": null }, { "id": "589062", "postDate": "07/31/2019 12:18:44", "content": "

i found loss of D going down but loss of G is always out of control , how to keep loss of G going well ? if i add dropout , the loss in both G and D seem good, but always has a lower score

", "rawMarkdown": "i found loss of D going down but loss of G is always out of control , how to keep loss of G going well ? if i add dropout , the loss in both G and D seem good, but always has a lower score", "votes": null }, { "id": "589077", "postDate": "07/31/2019 12:34:22", "content": "
\n

Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?).

\n
\n\n

It may depend on how you sample classes when generating images. For example, if for some reason some of the classes in your training set are rare, and you generate images using uniform class distribution, you may get worse results for these classes. Not sure how much it can affect overall score. But it's all just a theory, I haven't tested it.

", "rawMarkdown": ">Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?).\n\nIt may depend on how you sample classes when generating images. For example, if for some reason some of the classes in your training set are rare, and you generate images using uniform class distribution, you may get worse results for these classes. Not sure how much it can affect overall score. But it's all just a theory, I haven't tested it.", "votes": null }, { "id": "589083", "postDate": "07/31/2019 12:38:07", "content": "

I played a bit with cyclic learning rates and overall results are very random. Sometimes I saw clear gains but was unable to reproduce it later. Not sure if it's worth using on GANs since their training dynamic is somewhat unpredictable.

", "rawMarkdown": "I played a bit with cyclic learning rates and overall results are very random. Sometimes I saw clear gains but was unable to reproduce it later. Not sure if it's worth using on GANs since their training dynamic is somewhat unpredictable.", "votes": null }, { "id": "589138", "postDate": "07/31/2019 14:02:47", "content": "

I tried some different latent space distributions proposed in BigGAN paper Appendix-E, but they did not work well compared to standard normal distribution.

", "rawMarkdown": "I tried some different latent space distributions proposed in [BigGAN](https://arxiv.org/abs/1809.11096) paper Appendix-E, but they did not work well compared to standard normal distribution.", "votes": null }, { "id": "589214", "postDate": "07/31/2019 16:20:45", "content": "

when i use Spectral norm , my generators just give wrong images :( .May I ask how do u use it?

", "rawMarkdown": "when i use Spectral norm , my generators just give wrong images :( .May I ask how do u use it?", "votes": null }, { "id": "589266", "postDate": "07/31/2019 17:21:00", "content": "

Did you completely replace the BatchNorm in the Discriminator with SpectralNorm ? I have seen atleast Avg or Maxpooling layer needed in Discriminator ..

", "rawMarkdown": "Did you completely replace the BatchNorm in the Discriminator with SpectralNorm ? I have seen atleast Avg or Maxpooling layer needed in Discriminator ..", "votes": null }, { "id": "589497", "postDate": "08/01/2019 01:46:07", "content": "

No,I just connect a SN after CONV2D, and my discriminator works well,but when I began to train generator, it just return black images.

", "rawMarkdown": "No,I just connect a SN after CONV2D, and my discriminator works well,but when I began to train generator, it just return black images.", "votes": null }, { "id": "589526", "postDate": "08/01/2019 03:24:28", "content": "

Spectral Normalization cannot work well for me. I have thought that it restricts model capability heavily to train in 9 hours. I'll retry it.

", "rawMarkdown": "Spectral Normalization cannot work well for me. I have thought that it restricts model capability heavily to train in 9 hours. I'll retry it.", "votes": null }, { "id": "589574", "postDate": "08/01/2019 05:43:05", "content": "

For number 5 using both cropped and original - you could try including full image only for cases of 1 annotation and if bbox is a certain percentage of full image like 75% or whatever you decide. Including full images did give some improvement. But 'There is so much that can go wrong.'

", "rawMarkdown": "For number 5 using both cropped and original - you could try including full image only for cases of 1 annotation and if bbox is a certain percentage of full image like 75% or whatever you decide. Including full images did give some improvement. But 'There is so much that can go wrong.'", "votes": null }, { "id": "589589", "postDate": "08/01/2019 06:18:15", "content": "

keep it flowing

", "rawMarkdown": "keep it flowing", "votes": null }, { "id": "589875", "postDate": "08/01/2019 14:09:54", "content": "

@timetraveller: About 9, could you elaborate more on how pixel norm and spectral norm should be used?\n- can we use both pixel norm and batch norm in generator?\n- can we use both batch norm and spectral norm in discriminator?\n- or any other combinations?

", "rawMarkdown": "timetraveller: About 9, could you elaborate more on how pixel norm and spectral norm should be used?\n- can we use both pixel norm and batch norm in generator?\n- can we use both batch norm and spectral norm in discriminator?\n- or any other combinations?", "votes": null }, { "id": "589915", "postDate": "08/01/2019 14:56:42", "content": "

By hit and trial, I found that using only pixelnorm in the generator and both pixelnorm and batchnorm in the generator gets good results.\nSo is verified by this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers\nI am not sure why I get worse results when I removed batchnorm from the discriminator. There is still a lot left to experiment.

", "rawMarkdown": "By hit and trial, I found that using only pixelnorm in the generator and both pixelnorm and batchnorm in the generator gets good results.\nSo is verified by this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers\nI am not sure why I get worse results when I removed batchnorm from the discriminator. There is still a lot left to experiment.", "votes": null }, { "id": "589917", "postDate": "08/01/2019 14:57:00", "content": "

I used spectralnorm because of this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers

", "rawMarkdown": "I used spectralnorm because of this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers", "votes": null }, { "id": "589923", "postDate": "08/01/2019 15:02:48", "content": "

Timetraveller . If we remove batchnorm in discriminator we can try putting averagepooling layers instead . Try that out . But removing batchnorm in this current arch . Won't work .

", "rawMarkdown": "Timetraveller . If we remove batchnorm in discriminator we can try putting averagepooling layers instead . Try that out . But removing batchnorm in this current arch . Won't work .", "votes": null }, { "id": "589930", "postDate": "08/01/2019 15:18:30", "content": "

I haven't seen many architectures that use averagepooling that's why I have never tried them. But as you said, they won't get good results so I guess I won't test it for now.\nAlso, aren't pooling layers in general bad for GANs?\nEDIT - I was so fuckin wrong

", "rawMarkdown": "I haven't seen many architectures that use averagepooling that's why I have never tried them. But as you said, they won't get good results so I guess I won't test it for now.\nAlso, aren't pooling layers in general bad for GANs?\nEDIT - I was so fuckin wrong", "votes": null }, { "id": "589946", "postDate": "08/01/2019 15:38:53", "content": "

I also read so . But some architecture I saw that ,do not remember which one

", "rawMarkdown": "I also read so . But some architecture I saw that ,do not remember which one", "votes": null }, { "id": "590036", "postDate": "08/01/2019 18:12:19", "content": "

@timetraveller98 Do you mean, spectral-bn or pixel-bn, in discriminator, And similar confusion in generator comment with pixel/spectral

", "rawMarkdown": "timetraveller98 Do you mean, spectral-bn or pixel-bn, in discriminator, And similar confusion in generator comment with pixel/spectral", "votes": null }, { "id": "590909", "postDate": "08/02/2019 19:50:43", "content": "

@dhruuv111 in my best kernel (58) I haven't used spectral norm. so it's pixel-bn.\nI am having a very hard time getting my score below 50.

", "rawMarkdown": "dhruuv111 in my best kernel (58) I haven't used spectral norm. so it's pixel-bn.\nI am having a very hard time getting my score below 50.", "votes": null }, { "id": "590910", "postDate": "08/02/2019 19:53:39", "content": "

I tried a few more things:\n- Self attention: I don't understand what am I doing wrong but I either run into a mode collapse or a score of ~63\n- Conditions: I have a hunch that dog breed information can increase my score so I tried leveraging that. Still no good results.\n- Larger D and G networks: They aren't converging for me.

", "rawMarkdown": "I tried a few more things:\n- Self attention: I don't understand what am I doing wrong but I either run into a mode collapse or a score of ~63\n- Conditions: I have a hunch that dog breed information can increase my score so I tried leveraging that. Still no good results.\n- Larger D and G networks: They aren't converging for me.", "votes": null }, { "id": "591044", "postDate": "08/03/2019 03:49:04", "content": "

what is your highest score? I get around 56 using the same architecture.

", "rawMarkdown": "what is your highest score? I get around 56 using the same architecture.", "votes": null }, { "id": "591091", "postDate": "08/03/2019 06:02:18", "content": "

That's a good score!\nI am stuck at around 58 for now.

", "rawMarkdown": "That's a good score!\nI am stuck at around 58 for now.", "votes": null }, { "id": "591109", "postDate": "08/03/2019 06:56:44", "content": "

rotation doesn't help right? I'm trying to train gan without it :)

", "rawMarkdown": "rotation doesn't help right? I'm trying to train gan without it :)", "votes": null }, { "id": "591115", "postDate": "08/03/2019 07:06:18", "content": "

Yes, rotation never worked for me.\nEDIT - it just did worked for me.

", "rawMarkdown": "Yes, rotation never worked for me.\nEDIT - it just did worked for me.", "votes": null }, { "id": "591273", "postDate": "08/03/2019 12:43:36", "content": "

hm..rotation works for me haha..

", "rawMarkdown": "hm..rotation works for me haha..", "votes": null }, { "id": "591379", "postDate": "08/03/2019 15:44:06", "content": "

Wow\nHow many degrees in random have you rotated?

", "rawMarkdown": "Wow\nHow many degrees in random have you rotated?", "votes": null }, { "id": "591595", "postDate": "08/04/2019 01:31:29", "content": "

5 I used :) + randomapply

", "rawMarkdown": "5 I used :) + randomapply", "votes": null }, { "id": "591695", "postDate": "08/04/2019 06:08:54", "content": "

appreciate it:)

", "rawMarkdown": "appreciate it:)", "votes": null }, { "id": "592421", "postDate": "08/05/2019 08:57:45", "content": "

Thanks for sharing !\nSpectral norm and pixel norm truly improve my score about ~2 LB (my current best score is ~ 52)\nI will try another method, if I found something, I will share them out.

", "rawMarkdown": "Thanks for sharing !\nSpectral norm and pixel norm truly improve my score about ~2 LB (my current best score is ~ 52)\nI will try another method, if I found something, I will share them out.", "votes": null }, { "id": "593007", "postDate": "08/06/2019 05:21:24", "content": "

I can not use upscale+conv instead of convTranspose. \nMy GAN always run into mode collapse, I've tried everything I could.\nHas anyone faced a similar problem?

", "rawMarkdown": "I can not use upscale+conv instead of convTranspose. \nMy GAN always run into mode collapse, I've tried everything I could.\nHas anyone faced a similar problem?", "votes": null }, { "id": "593036", "postDate": "08/06/2019 05:45:08", "content": "", "rawMarkdown": "Parametric ReLUs worked better than Leaky ReLUs.\n- Convolution + PixelShuffle works better than Upsample + Convolution.\n- Smaller batch size.", "votes": null }, { "id": "593038", "postDate": "08/06/2019 05:47:14", "content": "

Yes, I'm in the same boat. Couldn't get Upscale + Conv to work. Using Conv + PixelShuffle instead.

", "rawMarkdown": "Yes, I'm in the same boat. Couldn't get Upscale + Conv to work. Using Conv + PixelShuffle instead.", "votes": null }, { "id": "593147", "postDate": "08/06/2019 08:16:55", "content": "

Thank you, I will try PixelShuffle and post the results here.

", "rawMarkdown": "Thank you, I will try PixelShuffle and post the results here.", "votes": null }, { "id": "593152", "postDate": "08/06/2019 08:19:14", "content": "

same for me . Very bad result with upscale and conv2D . Must be missing something while using this .

", "rawMarkdown": "same for me . Very bad result with upscale and conv2D . Must be missing something while using this .", "votes": null }, { "id": "593153", "postDate": "08/06/2019 08:21:04", "content": "

I cant get it past 57 though . I am using Pixel and spectral . How long are you training ? full 9 hours?

", "rawMarkdown": "I cant get it past 57 though . I am using Pixel and spectral . How long are you training ? full 9 hours?", "votes": null }, { "id": "593155", "postDate": "08/06/2019 08:22:53", "content": "

No, I train almost 7 hours, 300 epochs

", "rawMarkdown": "No, I train almost 7 hours, 300 epochs", "votes": null }, { "id": "593201", "postDate": "08/06/2019 09:53:00", "content": "

You're very welcome :)\nHere's my kernel.

", "rawMarkdown": "You're very welcome :)\n[Here](https://www.kaggle.com/dranzer/pixelshuffle-ralsgan)'s my kernel.", "votes": null }, { "id": "593494", "postDate": "08/06/2019 17:11:53", "content": "

Me : Let me just reduce the learning rate a little bit in my GAN , what could go wrong?\nMy GAN : \n\"\"

", "rawMarkdown": "Me : Let me just reduce the learning rate a little bit in my GAN , what could go wrong?\nMy GAN : \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2Fe7496adca457be662d5295be297f5038%2Ftuned_my_gan.gif?generation=1565111490247528&alt=media)", "votes": null }, { "id": "594086", "postDate": "08/07/2019 14:55:16", "content": "

@dranzer if I am allowed to ask you try . Can you please make Discriminator learning rate higher than Generator learning rate and rerun your kernel ? right now i believe your discriminator LR is 0.0005 and Generator Learning is 0.001 . \nTry with both as 0.0002 or something to start .

", "rawMarkdown": "dranzer if I am allowed to ask you try . Can you please make Discriminator learning rate higher than Generator learning rate and rerun your kernel ? right now i believe your discriminator LR is 0.0005 and Generator Learning is 0.001 . \nTry with both as 0.0002 or something to start .", "votes": null }, { "id": "596016", "postDate": "08/10/2019 02:43:02", "content": "

It seems to start working, thanks!

", "rawMarkdown": "It seems to start working, thanks!", "votes": null }, { "id": "596059", "postDate": "08/10/2019 05:27:09", "content": "

LB 16.9?! I am guessing that's a legit GAN\nGood luck! You're doing amazing.

", "rawMarkdown": "LB 16.9?! I am guessing that's a legit GAN\nGood luck! You're doing amazing.", "votes": null }, { "id": "596073", "postDate": "08/10/2019 06:02:36", "content": "

Some people already get LB ~15. I need to work hard.

", "rawMarkdown": "Some people already get LB ~15. I need to work hard.", "votes": null }, { "id": "596268", "postDate": "08/10/2019 11:50:25", "content": "

How do I implement pixel norm? Is that just adding BatchNorm after each convolution, or is it something else?

", "rawMarkdown": "How do I implement pixel norm? Is that just adding BatchNorm after each convolution, or is it something else?", "votes": null }, { "id": "596277", "postDate": "08/10/2019 12:15:25", "content": "

pixelnorm is just dividing every pixel in each channel by the channel's root mean square. I verified the code from a github implementation of proGAN. I have used it only in the generator and after every conv layer (like in the proGAN paper). \nAlso, If I use both pixnorm and batchnorm (kinda stupid) I run into mode collapse for some reasons.

", "rawMarkdown": "pixelnorm is just dividing every pixel in each channel by the channel's root mean square. I verified the code from a github implementation of proGAN. I have used it only in the generator and after every conv layer (like in the proGAN paper). \nAlso, If I use both pixnorm and batchnorm (kinda stupid) I run into mode collapse for some reasons.", "votes": null }, { "id": "596304", "postDate": "08/10/2019 13:00:20", "content": "

@phoenix9032 Sure, I'll try that out, thanks. And, sorry for the late reply. Some things came up at work so didn't check Kaggle the past three days.

", "rawMarkdown": "phoenix9032 Sure, I'll try that out, thanks. And, sorry for the late reply. Some things came up at work so didn't check Kaggle the past three days.", "votes": null }, { "id": "596543", "postDate": "08/10/2019 20:12:06", "content": "

Has anyone got good results by using the dog breed labels? / Are you using breed labels in your final models?

", "rawMarkdown": "Has anyone got good results by using the dog breed labels? / Are you using breed labels in your final models?", "votes": null }, { "id": "596704", "postDate": "08/11/2019 06:52:22", "content": "

@timetraveller98 Doesn't work at all for me. I haven't invested heavily on it (about 8-10 kernels that scored 270-75 on the LB), so I left it to pursue other directions.

", "rawMarkdown": "timetraveller98 Doesn't work at all for me. I haven't invested heavily on it (about 8-10 kernels that scored 270-75 on the LB), so I left it to pursue other directions.", "votes": null }, { "id": "596757", "postDate": "08/11/2019 08:13:34", "content": "

Just got into the 50's for the first time after trying for days. The key for me was the kernel sizes (for me 5x5 kernels worked the best in both G and D).

", "rawMarkdown": "Just got into the 50's for the first time after trying for days. The key for me was the kernel sizes (for me 5x5 kernels worked the best in both G and D).", "votes": null }, { "id": "597077", "postDate": "08/11/2019 19:18:54", "content": "

My current best solution is a CGAN using dog breed labels. It scores in the 50's.

", "rawMarkdown": "My current best solution is a CGAN using dog breed labels. It scores in the 50's.", "votes": null }, { "id": "597131", "postDate": "08/11/2019 22:02:14", "content": "

improved a bit, not too much.

", "rawMarkdown": "improved a bit, not too much.", "votes": null }, { "id": "597224", "postDate": "08/12/2019 03:06:37", "content": "

Breed label improve my score a lot, my current best score is 47 LB

", "rawMarkdown": "Breed label improve my score a lot, my current best score is 47 LB", "votes": null }, { "id": "597446", "postDate": "08/12/2019 11:01:48", "content": "

I think it improves the score because it focuses the attention of the discriminator on the dogs versus background. (You need to look at the dogs to classify them into breeds). Then when the generator tries to fool the discriminator it needs to make better dogs (than backgrounds) because the discriminator is focused on the dogs.

\n\n

Lastly, with conditional breeds, your Kaggle submission is guaranteed to have maximum variety because you will have certainly have white dogs and brown dogs and black dogs and short hair dogs and long hair dogs etc.

", "rawMarkdown": "I think it improves the score because it focuses the attention of the discriminator on the dogs versus background. (You need to look at the dogs to classify them into breeds). Then when the generator tries to fool the discriminator it needs to make better dogs (than backgrounds) because the discriminator is focused on the dogs.\n\nLastly, with conditional breeds, your Kaggle submission is guaranteed to have maximum variety because you will have certainly have white dogs and brown dogs and black dogs and short hair dogs and long hair dogs etc.", "votes": null }, { "id": "597469", "postDate": "08/12/2019 11:41:03", "content": "

I am sure that information about the dog breed will improve the GAN (CGAN now). However, In this competition with a limited amount of computational time, computational power and ~185 (22000 images / 120 labels) images per breed I am not sure that it will necessarily outperform GAN without labels.

\n\n

My score is also 47, the variety is very evident, black dogs, brown dogs, white dogs, two-headed dogs, two-legged dogs, grass only no dogs and so on. My intuition is that fewer conditions will create more variety, but who knows.

\n\n

I think the best solutions will be interesting and we will learn from them, but other highly scored solutions will be very similar to public solutions, just tweaked better.

", "rawMarkdown": "I am sure that information about the dog breed will improve the GAN (CGAN now). However, In this competition with a limited amount of computational time, computational power and ~185 (22000 images / 120 labels) images per breed I am not sure that it will necessarily outperform GAN without labels. \n\nMy score is also 47, the variety is very evident, black dogs, brown dogs, white dogs, two-headed dogs, two-legged dogs, grass only no dogs and so on. My intuition is that fewer conditions will create more variety, but who knows. \n\nI think the best solutions will be interesting and we will learn from them, but other highly scored solutions will be very similar to public solutions, just tweaked better.", "votes": null }, { "id": "597571", "postDate": "08/12/2019 14:15:19", "content": "

I've tried introducing PixelNorm into my models a couple of times (in the generator), replacing BatchNorm. Both times it has resulted in mode collapse even with drastically reducing the learning rate. Not sure what I'm doing wrong 🙁

", "rawMarkdown": "I've tried introducing PixelNorm into my models a couple of times (in the generator), replacing BatchNorm. Both times it has resulted in mode collapse even with drastically reducing the learning rate. Not sure what I'm doing wrong 🙁", "votes": null }, { "id": "597729", "postDate": "08/12/2019 17:44:29", "content": "

@weoflnwef09i32s \nRefer the public kernel with LB score ~60. It has implemented pixnorm correctly.

", "rawMarkdown": "weoflnwef09i32s \nRefer the public kernel with LB score ~60. It has implemented pixnorm correctly.", "votes": null }, { "id": "597822", "postDate": "08/12/2019 19:20:12", "content": "

@super13579 and @cdeotte thank you!! \nI finally got into the 40s and I am using dog breeds along with another thing which finally made it work. Almost gave up on this.

", "rawMarkdown": "super13579 and @cdeotte thank you!! \nI finally got into the 40s and I am using dog breeds along with another thing which finally made it work. Almost gave up on this.", "votes": null }, { "id": "597879", "postDate": "08/12/2019 21:09:56", "content": "

Thanks! All these points have be a great help.

", "rawMarkdown": "Thanks! All these points have be a great help.", "votes": null }, { "id": "598017", "postDate": "08/13/2019 02:34:57", "content": "

how many epoch you run to get 40s ?

", "rawMarkdown": "how many epoch you run to get 40s ?", "votes": null }, { "id": "598047", "postDate": "08/13/2019 03:35:25", "content": "

sorry, I am a new player. what is breed label?

", "rawMarkdown": "sorry, I am a new player. what is breed label?", "votes": null }, { "id": "598058", "postDate": "08/13/2019 04:07:46", "content": "

@peacemakertsp ~300\n@mellfish in the provided dataset, you have an annotations folder which has the breed label for every dog image. Look at some legit C-GAN implementation in the public kernel's section for a starter code.

", "rawMarkdown": "peacemakertsp ~300\n@mellfish in the provided dataset, you have an annotations folder which has the breed label for every dog image. Look at some legit C-GAN implementation in the public kernel's section for a starter code.", "votes": null }, { "id": "599156", "postDate": "08/14/2019 15:12:15", "content": "

Just did my last submission for this competition. \n47 was as low as I could go. \nGood luck to all!

", "rawMarkdown": "Just did my last submission for this competition. \n47 was as low as I could go. \nGood luck to all!", "votes": null }, { "id": "599173", "postDate": "08/14/2019 15:42:52", "content": "

Congrats timetraveller, 47 is awesome. I submitted a 47 this morning and I have 4 more notebooks running now. I'll make my final submission at 11pm. Hopefully one scores even better :-)

\n\n

Thanks for posting this thread. It encouraged me to try new ideas the past few days. One of these hints helped me achieve my best score.

", "rawMarkdown": "Congrats timetraveller, 47 is awesome. I submitted a 47 this morning and I have 4 more notebooks running now. I'll make my final submission at 11pm. Hopefully one scores even better :-)\n\nThanks for posting this thread. It encouraged me to try new ideas the past few days. One of these hints helped me achieve my best score.", "votes": null }, { "id": "599177", "postDate": "08/14/2019 15:51:03", "content": "

I hope you score better! Thank you for such kind words. Means a lot coming from you :)

", "rawMarkdown": "I hope you score better! Thank you for such kind words. Means a lot coming from you :)", "votes": null }, { "id": "599204", "postDate": "08/14/2019 16:49:42", "content": "

Good going guys! This discussion and @phoenix9032's kernel helped me a lot!\n@cdeotte how would you decide which of those 4 notebooks is the best for submission? Since you already made a submission this morning so you have 2 submissions left.

", "rawMarkdown": "Good going guys! This discussion and @phoenix9032's kernel helped me a lot!\n@cdeotte how would you decide which of those 4 notebooks is the best for submission? Since you already made a submission this morning so you have 2 submissions left.", "votes": null }, { "id": "599214", "postDate": "08/14/2019 17:04:06", "content": "

Well done @timetraveller98

", "rawMarkdown": "Well done @timetraveller98", "votes": null }, { "id": "599239", "postDate": "08/14/2019 17:48:01", "content": "

For those stuck in the 60s, the key for me using a DCGAN-like architecture was to use self-modulation, which is essentially batch-norm conditioned on the input noise. Once i replaced BN in my generator with this my scores dropped into the 40s when combined with spectral normalization, regardless of my loss functions. The paper also goes over the conditional case, which I did not have time to experiment with, but which probably would have been useful for this competition.

", "rawMarkdown": "For those stuck in the 60s, the key for me using a DCGAN-like architecture was to use [self-modulation](https://arxiv.org/abs/1810.01365), which is essentially batch-norm conditioned on the input noise. Once i replaced BN in my generator with this my scores dropped into the 40s when combined with spectral normalization, regardless of my loss functions. The paper also goes over the conditional case, which I did not have time to experiment with, but which probably would have been useful for this competition.", "votes": null }, { "id": "599244", "postDate": "08/14/2019 18:04:04", "content": "

@phoenix9032 Thank you :). Especially for your 60 MiFID public kernel!

", "rawMarkdown": "phoenix9032 Thank you :). Especially for your 60 MiFID public kernel!", "votes": null }, { "id": "599429", "postDate": "08/15/2019 00:27:16", "content": "

Thank you ! and good luck

", "rawMarkdown": "Thank you ! and good luck", "votes": null } ], "comments": [ { "id": 589015, "author_name": "phoenix9032", "author_url": "", "post_date": "07/31/2019 10:45:15", "content": "

Most valuable insights in this juncture of the competition . My respect to you .\nDitto for number 2 . I didn't find an easy way to code MFID knowing that calculation takes some time . I just generate 10000 images every 200 epochs and calculate MFid post-facto .

\n\n

3,6,7,8,9 . True . \nFew additional questions :\nHave you tried to add more layers and increase filter size in discriminator ,generator? - not much improvement for me

\n\n

Adding more cycle for discriminator for every generator run . - ended up with less score .

", "votes": null, "replies": [ { "id": 589026, "author_name": "timetraveller98", "author_url": "", "post_date": "07/31/2019 11:00:47", "content": "

Glad to help :)\nI have learned a lot of things from your latest kernel. It has also given me so many more ideas to try out.

\n\n
\n

Have you tried to add more layers and increase filter size in discriminator ,generator?

\n
\n\n

Yes, I am currently running 3 kaggle kernels in parallel all identical but with different kernel sizes. I read in a blog that larger kernels work well for properly cropped images which is why I am trying larger kernel sizes right now. Will update the results here.

\n\n
\n

Adding more cycle for discriminator for every generator run

\n
\n\n

I am not familiar with this, haven't read anywhere that it might work.

", "votes": null, "replies": [] } ] }, { "id": 589018, "author_name": "shaygu", "author_url": "", "post_date": "07/31/2019 10:50:11", "content": "

For my architecture (which I hope to make a public version today, but it basically https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers)

\n\n

scores between 60 and 90

\n\n

Batch size - 32 works better than 16\nDropout - seems to make worse results, but I haven't given up on it yet.\nEarly stopping - haven't fully tested, but I think I should implement. \nCosine annealing - trying to locate the optimal learning rate for prediction, at the end of the cycle, where the learning rate is significantly lowered than the initial learning rate, seems to perform worse.\nRandom crops (64,64) instead of resize(64,64) performs worse.

", "votes": null, "replies": [ { "id": 589029, "author_name": "timetraveller98", "author_url": "", "post_date": "07/31/2019 11:02:41", "content": "

Thanks for sharing\nI haven't tried dropout either, I am skeptical about them.\nI tried random cropping of all sorts only for memorizer GANs. Nothing worked.

", "votes": null, "replies": [] } ] }, { "id": 589062, "author_name": "peacemakertsp", "author_url": "", "post_date": "07/31/2019 12:18:44", "content": "

i found loss of D going down but loss of G is always out of control , how to keep loss of G going well ? if i add dropout , the loss in both G and D seem good, but always has a lower score

", "votes": null, "replies": [] }, { "id": 589077, "author_name": "dvorobiev", "author_url": "", "post_date": "07/31/2019 12:34:22", "content": "
\n

Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?).

\n
\n\n

It may depend on how you sample classes when generating images. For example, if for some reason some of the classes in your training set are rare, and you generate images using uniform class distribution, you may get worse results for these classes. Not sure how much it can affect overall score. But it's all just a theory, I haven't tested it.

", "votes": null, "replies": [] }, { "id": 589083, "author_name": "dvorobiev", "author_url": "", "post_date": "07/31/2019 12:38:07", "content": "

I played a bit with cyclic learning rates and overall results are very random. Sometimes I saw clear gains but was unable to reproduce it later. Not sure if it's worth using on GANs since their training dynamic is somewhat unpredictable.

", "votes": null, "replies": [] }, { "id": 589138, "author_name": "a2015003713", "author_url": "", "post_date": "07/31/2019 14:02:47", "content": "

I tried some different latent space distributions proposed in BigGAN paper Appendix-E, but they did not work well compared to standard normal distribution.

", "votes": null, "replies": [] }, { "id": 589214, "author_name": "takeiy", "author_url": "", "post_date": "07/31/2019 16:20:45", "content": "

when i use Spectral norm , my generators just give wrong images :( .May I ask how do u use it?

", "votes": null, "replies": [ { "id": 589266, "author_name": "phoenix9032", "author_url": "", "post_date": "07/31/2019 17:21:00", "content": "

Did you completely replace the BatchNorm in the Discriminator with SpectralNorm ? I have seen atleast Avg or Maxpooling layer needed in Discriminator ..

", "votes": null, "replies": [] }, { "id": 589497, "author_name": "takeiy", "author_url": "", "post_date": "08/01/2019 01:46:07", "content": "

No,I just connect a SN after CONV2D, and my discriminator works well,but when I began to train generator, it just return black images.

", "votes": null, "replies": [] } ] }, { "id": 589526, "author_name": "yukia18", "author_url": "", "post_date": "08/01/2019 03:24:28", "content": "

Spectral Normalization cannot work well for me. I have thought that it restricts model capability heavily to train in 9 hours. I'll retry it.

", "votes": null, "replies": [ { "id": 589917, "author_name": "timetraveller98", "author_url": "", "post_date": "08/01/2019 14:57:00", "content": "

I used spectralnorm because of this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers

", "votes": null, "replies": [] }, { "id": 596016, "author_name": "yukia18", "author_url": "", "post_date": "08/10/2019 02:43:02", "content": "

It seems to start working, thanks!

", "votes": null, "replies": [] }, { "id": 596059, "author_name": "timetraveller98", "author_url": "", "post_date": "08/10/2019 05:27:09", "content": "

LB 16.9?! I am guessing that's a legit GAN\nGood luck! You're doing amazing.

", "votes": null, "replies": [] }, { "id": 596073, "author_name": "yukia18", "author_url": "", "post_date": "08/10/2019 06:02:36", "content": "

Some people already get LB ~15. I need to work hard.

", "votes": null, "replies": [] } ] }, { "id": 589574, "author_name": "something4kag", "author_url": "", "post_date": "08/01/2019 05:43:05", "content": "

For number 5 using both cropped and original - you could try including full image only for cases of 1 annotation and if bbox is a certain percentage of full image like 75% or whatever you decide. Including full images did give some improvement. But 'There is so much that can go wrong.'

", "votes": null, "replies": [] }, { "id": 589589, "author_name": "", "author_url": "", "post_date": "08/01/2019 06:18:15", "content": "

keep it flowing

", "votes": null, "replies": [] }, { "id": 589875, "author_name": "duongnguyen1", "author_url": "", "post_date": "08/01/2019 14:09:54", "content": "

@timetraveller: About 9, could you elaborate more on how pixel norm and spectral norm should be used?\n- can we use both pixel norm and batch norm in generator?\n- can we use both batch norm and spectral norm in discriminator?\n- or any other combinations?

", "votes": null, "replies": [ { "id": 589915, "author_name": "timetraveller98", "author_url": "", "post_date": "08/01/2019 14:56:42", "content": "

By hit and trial, I found that using only pixelnorm in the generator and both pixelnorm and batchnorm in the generator gets good results.\nSo is verified by this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers\nI am not sure why I get worse results when I removed batchnorm from the discriminator. There is still a lot left to experiment.

", "votes": null, "replies": [] }, { "id": 589923, "author_name": "phoenix9032", "author_url": "", "post_date": "08/01/2019 15:02:48", "content": "

Timetraveller . If we remove batchnorm in discriminator we can try putting averagepooling layers instead . Try that out . But removing batchnorm in this current arch . Won't work .

", "votes": null, "replies": [] }, { "id": 589930, "author_name": "timetraveller98", "author_url": "", "post_date": "08/01/2019 15:18:30", "content": "

I haven't seen many architectures that use averagepooling that's why I have never tried them. But as you said, they won't get good results so I guess I won't test it for now.\nAlso, aren't pooling layers in general bad for GANs?\nEDIT - I was so fuckin wrong

", "votes": null, "replies": [] }, { "id": 589946, "author_name": "phoenix9032", "author_url": "", "post_date": "08/01/2019 15:38:53", "content": "

I also read so . But some architecture I saw that ,do not remember which one

", "votes": null, "replies": [] }, { "id": 590036, "author_name": "dhruuv111", "author_url": "", "post_date": "08/01/2019 18:12:19", "content": "

@timetraveller98 Do you mean, spectral-bn or pixel-bn, in discriminator, And similar confusion in generator comment with pixel/spectral

", "votes": null, "replies": [] }, { "id": 590909, "author_name": "timetraveller98", "author_url": "", "post_date": "08/02/2019 19:50:43", "content": "

@dhruuv111 in my best kernel (58) I haven't used spectral norm. so it's pixel-bn.\nI am having a very hard time getting my score below 50.

", "votes": null, "replies": [] } ] }, { "id": 590910, "author_name": "timetraveller98", "author_url": "", "post_date": "08/02/2019 19:53:39", "content": "

I tried a few more things:\n- Self attention: I don't understand what am I doing wrong but I either run into a mode collapse or a score of ~63\n- Conditions: I have a hunch that dog breed information can increase my score so I tried leveraging that. Still no good results.\n- Larger D and G networks: They aren't converging for me.

", "votes": null, "replies": [] }, { "id": 591044, "author_name": "yangsaewon", "author_url": "", "post_date": "08/03/2019 03:49:04", "content": "

what is your highest score? I get around 56 using the same architecture.

", "votes": null, "replies": [ { "id": 591091, "author_name": "timetraveller98", "author_url": "", "post_date": "08/03/2019 06:02:18", "content": "

That's a good score!\nI am stuck at around 58 for now.

", "votes": null, "replies": [] }, { "id": 591109, "author_name": "yangsaewon", "author_url": "", "post_date": "08/03/2019 06:56:44", "content": "

rotation doesn't help right? I'm trying to train gan without it :)

", "votes": null, "replies": [] }, { "id": 591115, "author_name": "timetraveller98", "author_url": "", "post_date": "08/03/2019 07:06:18", "content": "

Yes, rotation never worked for me.\nEDIT - it just did worked for me.

", "votes": null, "replies": [] }, { "id": 591273, "author_name": "yangsaewon", "author_url": "", "post_date": "08/03/2019 12:43:36", "content": "

hm..rotation works for me haha..

", "votes": null, "replies": [] }, { "id": 591379, "author_name": "timetraveller98", "author_url": "", "post_date": "08/03/2019 15:44:06", "content": "

Wow\nHow many degrees in random have you rotated?

", "votes": null, "replies": [] }, { "id": 591595, "author_name": "yangsaewon", "author_url": "", "post_date": "08/04/2019 01:31:29", "content": "

5 I used :) + randomapply

", "votes": null, "replies": [] } ] }, { "id": 591695, "author_name": "elsazhou", "author_url": "", "post_date": "08/04/2019 06:08:54", "content": "

appreciate it:)

", "votes": null, "replies": [] }, { "id": 592421, "author_name": "super13579", "author_url": "", "post_date": "08/05/2019 08:57:45", "content": "

Thanks for sharing !\nSpectral norm and pixel norm truly improve my score about ~2 LB (my current best score is ~ 52)\nI will try another method, if I found something, I will share them out.

", "votes": null, "replies": [ { "id": 593153, "author_name": "phoenix9032", "author_url": "", "post_date": "08/06/2019 08:21:04", "content": "

I cant get it past 57 though . I am using Pixel and spectral . How long are you training ? full 9 hours?

", "votes": null, "replies": [] }, { "id": 593155, "author_name": "super13579", "author_url": "", "post_date": "08/06/2019 08:22:53", "content": "

No, I train almost 7 hours, 300 epochs

", "votes": null, "replies": [] } ] }, { "id": 593007, "author_name": "timetraveller98", "author_url": "", "post_date": "08/06/2019 05:21:24", "content": "

I can not use upscale+conv instead of convTranspose. \nMy GAN always run into mode collapse, I've tried everything I could.\nHas anyone faced a similar problem?

", "votes": null, "replies": [ { "id": 593038, "author_name": "dranzer", "author_url": "", "post_date": "08/06/2019 05:47:14", "content": "

Yes, I'm in the same boat. Couldn't get Upscale + Conv to work. Using Conv + PixelShuffle instead.

", "votes": null, "replies": [] }, { "id": 593147, "author_name": "timetraveller98", "author_url": "", "post_date": "08/06/2019 08:16:55", "content": "

Thank you, I will try PixelShuffle and post the results here.

", "votes": null, "replies": [] }, { "id": 593152, "author_name": "phoenix9032", "author_url": "", "post_date": "08/06/2019 08:19:14", "content": "

same for me . Very bad result with upscale and conv2D . Must be missing something while using this .

", "votes": null, "replies": [] }, { "id": 593201, "author_name": "dranzer", "author_url": "", "post_date": "08/06/2019 09:53:00", "content": "

You're very welcome :)\nHere's my kernel.

", "votes": null, "replies": [] }, { "id": 594086, "author_name": "phoenix9032", "author_url": "", "post_date": "08/07/2019 14:55:16", "content": "

@dranzer if I am allowed to ask you try . Can you please make Discriminator learning rate higher than Generator learning rate and rerun your kernel ? right now i believe your discriminator LR is 0.0005 and Generator Learning is 0.001 . \nTry with both as 0.0002 or something to start .

", "votes": null, "replies": [] }, { "id": 596304, "author_name": "dranzer", "author_url": "", "post_date": "08/10/2019 13:00:20", "content": "

@phoenix9032 Sure, I'll try that out, thanks. And, sorry for the late reply. Some things came up at work so didn't check Kaggle the past three days.

", "votes": null, "replies": [] } ] }, { "id": 593036, "author_name": "dranzer", "author_url": "", "post_date": "08/06/2019 05:45:08", "content": "", "votes": null, "replies": [] }, { "id": 593494, "author_name": "phoenix9032", "author_url": "", "post_date": "08/06/2019 17:11:53", "content": "

Me : Let me just reduce the learning rate a little bit in my GAN , what could go wrong?\nMy GAN : \n\"\"

", "votes": null, "replies": [] }, { "id": 596268, "author_name": "cdeotte", "author_url": "", "post_date": "08/10/2019 11:50:25", "content": "

How do I implement pixel norm? Is that just adding BatchNorm after each convolution, or is it something else?

", "votes": null, "replies": [ { "id": 596277, "author_name": "timetraveller98", "author_url": "", "post_date": "08/10/2019 12:15:25", "content": "

pixelnorm is just dividing every pixel in each channel by the channel's root mean square. I verified the code from a github implementation of proGAN. I have used it only in the generator and after every conv layer (like in the proGAN paper). \nAlso, If I use both pixnorm and batchnorm (kinda stupid) I run into mode collapse for some reasons.

", "votes": null, "replies": [] }, { "id": 597571, "author_name": "weoflnwef09i32s", "author_url": "", "post_date": "08/12/2019 14:15:19", "content": "

I've tried introducing PixelNorm into my models a couple of times (in the generator), replacing BatchNorm. Both times it has resulted in mode collapse even with drastically reducing the learning rate. Not sure what I'm doing wrong 🙁

", "votes": null, "replies": [] }, { "id": 597729, "author_name": "timetraveller98", "author_url": "", "post_date": "08/12/2019 17:44:29", "content": "

@weoflnwef09i32s \nRefer the public kernel with LB score ~60. It has implemented pixnorm correctly.

", "votes": null, "replies": [] } ] }, { "id": 596543, "author_name": "timetraveller98", "author_url": "", "post_date": "08/10/2019 20:12:06", "content": "

Has anyone got good results by using the dog breed labels? / Are you using breed labels in your final models?

", "votes": null, "replies": [ { "id": 596704, "author_name": "shaygu", "author_url": "", "post_date": "08/11/2019 06:52:22", "content": "

@timetraveller98 Doesn't work at all for me. I haven't invested heavily on it (about 8-10 kernels that scored 270-75 on the LB), so I left it to pursue other directions.

", "votes": null, "replies": [] }, { "id": 597077, "author_name": "cdeotte", "author_url": "", "post_date": "08/11/2019 19:18:54", "content": "

My current best solution is a CGAN using dog breed labels. It scores in the 50's.

", "votes": null, "replies": [] }, { "id": 597131, "author_name": "zhangyixing0404", "author_url": "", "post_date": "08/11/2019 22:02:14", "content": "

improved a bit, not too much.

", "votes": null, "replies": [] }, { "id": 597224, "author_name": "super13579", "author_url": "", "post_date": "08/12/2019 03:06:37", "content": "

Breed label improve my score a lot, my current best score is 47 LB

", "votes": null, "replies": [] }, { "id": 597446, "author_name": "cdeotte", "author_url": "", "post_date": "08/12/2019 11:01:48", "content": "

I think it improves the score because it focuses the attention of the discriminator on the dogs versus background. (You need to look at the dogs to classify them into breeds). Then when the generator tries to fool the discriminator it needs to make better dogs (than backgrounds) because the discriminator is focused on the dogs.

\n\n

Lastly, with conditional breeds, your Kaggle submission is guaranteed to have maximum variety because you will have certainly have white dogs and brown dogs and black dogs and short hair dogs and long hair dogs etc.

", "votes": null, "replies": [] }, { "id": 597469, "author_name": "shaygu", "author_url": "", "post_date": "08/12/2019 11:41:03", "content": "

I am sure that information about the dog breed will improve the GAN (CGAN now). However, In this competition with a limited amount of computational time, computational power and ~185 (22000 images / 120 labels) images per breed I am not sure that it will necessarily outperform GAN without labels.

\n\n

My score is also 47, the variety is very evident, black dogs, brown dogs, white dogs, two-headed dogs, two-legged dogs, grass only no dogs and so on. My intuition is that fewer conditions will create more variety, but who knows.

\n\n

I think the best solutions will be interesting and we will learn from them, but other highly scored solutions will be very similar to public solutions, just tweaked better.

", "votes": null, "replies": [] }, { "id": 597822, "author_name": "timetraveller98", "author_url": "", "post_date": "08/12/2019 19:20:12", "content": "

@super13579 and @cdeotte thank you!! \nI finally got into the 40s and I am using dog breeds along with another thing which finally made it work. Almost gave up on this.

", "votes": null, "replies": [] }, { "id": 598017, "author_name": "peacemakertsp", "author_url": "", "post_date": "08/13/2019 02:34:57", "content": "

how many epoch you run to get 40s ?

", "votes": null, "replies": [] }, { "id": 598047, "author_name": "mellfish", "author_url": "", "post_date": "08/13/2019 03:35:25", "content": "

sorry, I am a new player. what is breed label?

", "votes": null, "replies": [] }, { "id": 598058, "author_name": "timetraveller98", "author_url": "", "post_date": "08/13/2019 04:07:46", "content": "

@peacemakertsp ~300\n@mellfish in the provided dataset, you have an annotations folder which has the breed label for every dog image. Look at some legit C-GAN implementation in the public kernel's section for a starter code.

", "votes": null, "replies": [] }, { "id": 599429, "author_name": "mellfish", "author_url": "", "post_date": "08/15/2019 00:27:16", "content": "

Thank you ! and good luck

", "votes": null, "replies": [] } ] }, { "id": 596757, "author_name": "jadeblue", "author_url": "", "post_date": "08/11/2019 08:13:34", "content": "

Just got into the 50's for the first time after trying for days. The key for me was the kernel sizes (for me 5x5 kernels worked the best in both G and D).

", "votes": null, "replies": [] }, { "id": 597879, "author_name": "maxgarber", "author_url": "", "post_date": "08/12/2019 21:09:56", "content": "

Thanks! All these points have be a great help.

", "votes": null, "replies": [] }, { "id": 599156, "author_name": "timetraveller98", "author_url": "", "post_date": "08/14/2019 15:12:15", "content": "

Just did my last submission for this competition. \n47 was as low as I could go. \nGood luck to all!

", "votes": null, "replies": [ { "id": 599173, "author_name": "cdeotte", "author_url": "", "post_date": "08/14/2019 15:42:52", "content": "

Congrats timetraveller, 47 is awesome. I submitted a 47 this morning and I have 4 more notebooks running now. I'll make my final submission at 11pm. Hopefully one scores even better :-)

\n\n

Thanks for posting this thread. It encouraged me to try new ideas the past few days. One of these hints helped me achieve my best score.

", "votes": null, "replies": [] }, { "id": 599177, "author_name": "timetraveller98", "author_url": "", "post_date": "08/14/2019 15:51:03", "content": "

I hope you score better! Thank you for such kind words. Means a lot coming from you :)

", "votes": null, "replies": [] }, { "id": 599204, "author_name": "mightyrains", "author_url": "", "post_date": "08/14/2019 16:49:42", "content": "

Good going guys! This discussion and @phoenix9032's kernel helped me a lot!\n@cdeotte how would you decide which of those 4 notebooks is the best for submission? Since you already made a submission this morning so you have 2 submissions left.

", "votes": null, "replies": [] }, { "id": 599214, "author_name": "phoenix9032", "author_url": "", "post_date": "08/14/2019 17:04:06", "content": "

Well done @timetraveller98

", "votes": null, "replies": [] }, { "id": 599244, "author_name": "timetraveller98", "author_url": "", "post_date": "08/14/2019 18:04:04", "content": "

@phoenix9032 Thank you :). Especially for your 60 MiFID public kernel!

", "votes": null, "replies": [] } ] }, { "id": 599239, "author_name": "johannl", "author_url": "", "post_date": "08/14/2019 17:48:01", "content": "

For those stuck in the 60s, the key for me using a DCGAN-like architecture was to use self-modulation, which is essentially batch-norm conditioned on the input noise. Once i replaced BN in my generator with this my scores dropped into the 40s when combined with spectral normalization, regardless of my loss functions. The paper also goes over the conditional case, which I did not have time to experiment with, but which probably would have been useful for this competition.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "589002": "One thing I've learned from this competition is that GANs are very touchy. There is so much that can go wrong. In a world of million such wrong things I've decided to make a list of things that seem to go right for my traditional GAN model:\n\n\n1. Smaller batch size (my score reduced by ~5-10 when I reduced bs from 64 to 32)\n2. Early stopping? (not sure if it's the right term) - I've been training my GANs by running them for ~9 hours (kernel limit) but that gives worse results. I calculated MiFID at every 100 epochs and found that my best GAN performs the best at 900 epochs rather than at kernel time limit (~1400 epochs).\n3. Loss function - I have only tried the traditional loss function and RaLSGAN's loss function. I didn't get much difference in scores. My current best GAN performs better with traditional loss.\n4. Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?). EDIT- I was doing something wrong, conditions work great!\n5. Bounded box cropping - Drastically improves traditional GAN score. Drastically deteriorates memorizer GAN score. I can't use both cropped images and original images together, I get CUDA memory errors. Has someone successfully done this? I think it might improve the score.\n6. Cosine learning rate annealing - I haven't explicitly tested it, I tested it with other changes and my score has improved.\n7. Label smoothing: It improves my score. I've added small randomness here too (as suggested in the GAN hacks repo by soumith chintala)\n8. Latent space: I get a better score for nz=128 than for nz=100. Also, truncated latent space improved my score a little.\n9. Spectral norm and pixel norm - I tried both of these together and it drastically improved my score. \n10. Transformations: Horizontal flip improves my score and [EDIT] rotation improves the score. I haven't tested other transformations.\n\n\nWhat has worked for you so far?", "589015": "Most valuable insights in this juncture of the competition . My respect to you .\nDitto for number 2 . I didn't find an easy way to code MFID knowing that calculation takes some time . I just generate 10000 images every 200 epochs and calculate MFid post-facto .\n\n3,6,7,8,9 . True . \nFew additional questions :\nHave you tried to add more layers and increase filter size in discriminator ,generator? - not much improvement for me \n\nAdding more cycle for discriminator for every generator run . - ended up with less score .", "589018": "For my architecture (which I hope to make a public version today, but it basically https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers)\n\nscores between 60 and 90\n\nBatch size - 32 works better than 16\nDropout - seems to make worse results, but I haven't given up on it yet.\nEarly stopping - haven't fully tested, but I think I should implement. \nCosine annealing - trying to locate the optimal learning rate for prediction, at the end of the cycle, where the learning rate is significantly lowered than the initial learning rate, seems to perform worse.\nRandom crops (64,64) instead of resize(64,64) performs worse.", "589026": "Glad to help :)\nI have learned a lot of things from your latest kernel. It has also given me so many more ideas to try out. \n\n>Have you tried to add more layers and increase filter size in discriminator ,generator?\n\nYes, I am currently running 3 kaggle kernels in parallel all identical but with different kernel sizes. I read in a blog that larger kernels work well for properly cropped images which is why I am trying larger kernel sizes right now. Will update the results here.\n\n>Adding more cycle for discriminator for every generator run\n\nI am not familiar with this, haven't read anywhere that it might work.", "589029": "Thanks for sharing\nI haven't tried dropout either, I am skeptical about them.\nI tried random cropping of all sorts only for memorizer GANs. Nothing worked.", "589062": "i found loss of D going down but loss of G is always out of control , how to keep loss of G going well ? if i add dropout , the loss in both G and D seem good, but always has a lower score", "589077": ">Conditions - Adding conditions (dog breed) doesn't work for me. The score always goes down for some reason. I might be doing something wrong (can you help?).\n\nIt may depend on how you sample classes when generating images. For example, if for some reason some of the classes in your training set are rare, and you generate images using uniform class distribution, you may get worse results for these classes. Not sure how much it can affect overall score. But it's all just a theory, I haven't tested it.", "589083": "I played a bit with cyclic learning rates and overall results are very random. Sometimes I saw clear gains but was unable to reproduce it later. Not sure if it's worth using on GANs since their training dynamic is somewhat unpredictable.", "589138": "I tried some different latent space distributions proposed in [BigGAN](https://arxiv.org/abs/1809.11096) paper Appendix-E, but they did not work well compared to standard normal distribution.", "589214": "when i use Spectral norm , my generators just give wrong images :( .May I ask how do u use it?", "589266": "Did you completely replace the BatchNorm in the Discriminator with SpectralNorm ? I have seen atleast Avg or Maxpooling layer needed in Discriminator ..", "589497": "No,I just connect a SN after CONV2D, and my discriminator works well,but when I began to train generator, it just return black images.", "589526": "Spectral Normalization cannot work well for me. I have thought that it restricts model capability heavily to train in 9 hours. I'll retry it.", "589574": "For number 5 using both cropped and original - you could try including full image only for cases of 1 annotation and if bbox is a certain percentage of full image like 75% or whatever you decide. Including full images did give some improvement. But 'There is so much that can go wrong.'", "589589": "keep it flowing", "589875": "timetraveller: About 9, could you elaborate more on how pixel norm and spectral norm should be used?\n- can we use both pixel norm and batch norm in generator?\n- can we use both batch norm and spectral norm in discriminator?\n- or any other combinations?", "589915": "By hit and trial, I found that using only pixelnorm in the generator and both pixelnorm and batchnorm in the generator gets good results.\nSo is verified by this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers\nI am not sure why I get worse results when I removed batchnorm from the discriminator. There is still a lot left to experiment.", "589917": "I used spectralnorm because of this kernel: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers", "589923": "Timetraveller . If we remove batchnorm in discriminator we can try putting averagepooling layers instead . Try that out . But removing batchnorm in this current arch . Won't work .", "589930": "I haven't seen many architectures that use averagepooling that's why I have never tried them. But as you said, they won't get good results so I guess I won't test it for now.\nAlso, aren't pooling layers in general bad for GANs?\nEDIT - I was so fuckin wrong", "589946": "I also read so . But some architecture I saw that ,do not remember which one", "590036": "timetraveller98 Do you mean, spectral-bn or pixel-bn, in discriminator, And similar confusion in generator comment with pixel/spectral", "590909": "dhruuv111 in my best kernel (58) I haven't used spectral norm. so it's pixel-bn.\nI am having a very hard time getting my score below 50.", "590910": "I tried a few more things:\n- Self attention: I don't understand what am I doing wrong but I either run into a mode collapse or a score of ~63\n- Conditions: I have a hunch that dog breed information can increase my score so I tried leveraging that. Still no good results.\n- Larger D and G networks: They aren't converging for me.", "591044": "what is your highest score? I get around 56 using the same architecture.", "591091": "That's a good score!\nI am stuck at around 58 for now.", "591109": "rotation doesn't help right? I'm trying to train gan without it :)", "591115": "Yes, rotation never worked for me.\nEDIT - it just did worked for me.", "591273": "hm..rotation works for me haha..", "591379": "Wow\nHow many degrees in random have you rotated?", "591595": "5 I used :) + randomapply", "591695": "appreciate it:)", "592421": "Thanks for sharing !\nSpectral norm and pixel norm truly improve my score about ~2 LB (my current best score is ~ 52)\nI will try another method, if I found something, I will share them out.", "593007": "I can not use upscale+conv instead of convTranspose. \nMy GAN always run into mode collapse, I've tried everything I could.\nHas anyone faced a similar problem?", "593036": "Parametric ReLUs worked better than Leaky ReLUs.\n- Convolution + PixelShuffle works better than Upsample + Convolution.\n- Smaller batch size.", "593038": "Yes, I'm in the same boat. Couldn't get Upscale + Conv to work. Using Conv + PixelShuffle instead.", "593147": "Thank you, I will try PixelShuffle and post the results here.", "593152": "same for me . Very bad result with upscale and conv2D . Must be missing something while using this .", "593153": "I cant get it past 57 though . I am using Pixel and spectral . How long are you training ? full 9 hours?", "593155": "No, I train almost 7 hours, 300 epochs", "593201": "You're very welcome :)\n[Here](https://www.kaggle.com/dranzer/pixelshuffle-ralsgan)'s my kernel.", "593494": "Me : Let me just reduce the learning rate a little bit in my GAN , what could go wrong?\nMy GAN : \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2Fe7496adca457be662d5295be297f5038%2Ftuned_my_gan.gif?generation=1565111490247528&alt=media)", "594086": "dranzer if I am allowed to ask you try . Can you please make Discriminator learning rate higher than Generator learning rate and rerun your kernel ? right now i believe your discriminator LR is 0.0005 and Generator Learning is 0.001 . \nTry with both as 0.0002 or something to start .", "596016": "It seems to start working, thanks!", "596059": "LB 16.9?! I am guessing that's a legit GAN\nGood luck! You're doing amazing.", "596073": "Some people already get LB ~15. I need to work hard.", "596268": "How do I implement pixel norm? Is that just adding BatchNorm after each convolution, or is it something else?", "596277": "pixelnorm is just dividing every pixel in each channel by the channel's root mean square. I verified the code from a github implementation of proGAN. I have used it only in the generator and after every conv layer (like in the proGAN paper). \nAlso, If I use both pixnorm and batchnorm (kinda stupid) I run into mode collapse for some reasons.", "596304": "phoenix9032 Sure, I'll try that out, thanks. And, sorry for the late reply. Some things came up at work so didn't check Kaggle the past three days.", "596543": "Has anyone got good results by using the dog breed labels? / Are you using breed labels in your final models?", "596704": "timetraveller98 Doesn't work at all for me. I haven't invested heavily on it (about 8-10 kernels that scored 270-75 on the LB), so I left it to pursue other directions.", "596757": "Just got into the 50's for the first time after trying for days. The key for me was the kernel sizes (for me 5x5 kernels worked the best in both G and D).", "597077": "My current best solution is a CGAN using dog breed labels. It scores in the 50's.", "597131": "improved a bit, not too much.", "597224": "Breed label improve my score a lot, my current best score is 47 LB", "597446": "I think it improves the score because it focuses the attention of the discriminator on the dogs versus background. (You need to look at the dogs to classify them into breeds). Then when the generator tries to fool the discriminator it needs to make better dogs (than backgrounds) because the discriminator is focused on the dogs.\n\nLastly, with conditional breeds, your Kaggle submission is guaranteed to have maximum variety because you will have certainly have white dogs and brown dogs and black dogs and short hair dogs and long hair dogs etc.", "597469": "I am sure that information about the dog breed will improve the GAN (CGAN now). However, In this competition with a limited amount of computational time, computational power and ~185 (22000 images / 120 labels) images per breed I am not sure that it will necessarily outperform GAN without labels. \n\nMy score is also 47, the variety is very evident, black dogs, brown dogs, white dogs, two-headed dogs, two-legged dogs, grass only no dogs and so on. My intuition is that fewer conditions will create more variety, but who knows. \n\nI think the best solutions will be interesting and we will learn from them, but other highly scored solutions will be very similar to public solutions, just tweaked better.", "597571": "I've tried introducing PixelNorm into my models a couple of times (in the generator), replacing BatchNorm. Both times it has resulted in mode collapse even with drastically reducing the learning rate. Not sure what I'm doing wrong 🙁", "597729": "weoflnwef09i32s \nRefer the public kernel with LB score ~60. It has implemented pixnorm correctly.", "597822": "super13579 and @cdeotte thank you!! \nI finally got into the 40s and I am using dog breeds along with another thing which finally made it work. Almost gave up on this.", "597879": "Thanks! All these points have be a great help.", "598017": "how many epoch you run to get 40s ?", "598047": "sorry, I am a new player. what is breed label?", "598058": "peacemakertsp ~300\n@mellfish in the provided dataset, you have an annotations folder which has the breed label for every dog image. Look at some legit C-GAN implementation in the public kernel's section for a starter code.", "599156": "Just did my last submission for this competition. \n47 was as low as I could go. \nGood luck to all!", "599173": "Congrats timetraveller, 47 is awesome. I submitted a 47 this morning and I have 4 more notebooks running now. I'll make my final submission at 11pm. Hopefully one scores even better :-)\n\nThanks for posting this thread. It encouraged me to try new ideas the past few days. One of these hints helped me achieve my best score.", "599177": "I hope you score better! Thank you for such kind words. Means a lot coming from you :)", "599204": "Good going guys! This discussion and @phoenix9032's kernel helped me a lot!\n@cdeotte how would you decide which of those 4 notebooks is the best for submission? Since you already made a submission this morning so you have 2 submissions left.", "599214": "Well done @timetraveller98", "599239": "For those stuck in the 60s, the key for me using a DCGAN-like architecture was to use [self-modulation](https://arxiv.org/abs/1810.01365), which is essentially batch-norm conditioned on the input noise. Once i replaced BN in my generator with this my scores dropped into the 40s when combined with spectral normalization, regardless of my loss functions. The paper also goes over the conditional case, which I did not have time to experiment with, but which probably would have been useful for this competition.", "599244": "phoenix9032 Thank you :). Especially for your 60 MiFID public kernel!", "599429": "Thank you ! and good luck" }, "source": "meta" }