This competition has closed! See Wendy's post for the details.
\n\nI've created this thread to be a curated list of your amazing work on this competition. As you post your solutions, top kernels, and takeaways, feel free to also comment a link in this thread. If you write a paper, thesis, or present at a conference, and would like to share your work - also add here, to share with the Kaggle community!
\n\nSome aggregated takeaways already shared:\n- Top Solutions summary post
", "messages": [ { "id": "610431", "postDate": "08/28/2019 19:39:28", "content": "This competition has closed! See Wendy's post for the details.
\n\nI've created this thread to be a curated list of your amazing work on this competition. As you post your solutions, top kernels, and takeaways, feel free to also comment a link in this thread. If you write a paper, thesis, or present at a conference, and would like to share your work - also add here, to share with the Kaggle community!
\n\nSome aggregated takeaways already shared:\n- Top Solutions summary post
", "rawMarkdown": "This competition has closed! See [Wendy's post](https://www.kaggle.com/c/generative-dog-images/discussion/106206) for the details. \n\nI've created this thread to be a curated list of your amazing work on this competition. As you post your solutions, top kernels, and takeaways, feel free to also comment a link in this thread. If you write a paper, thesis, or present at a conference, and would like to share your work - also add here, to share with the Kaggle community!\n\n**Some aggregated takeaways already shared:**\n- [Top Solutions](https://www.kaggle.com/c/generative-dog-images/discussion/106305) summary post", "votes": null }, { "id": "610464", "postDate": "08/28/2019 20:27:19", "content": "\n\nSome takeaways : \nAlthough I expected to rank around 6th, I was very surprised when I saw the private LB. Tom had a public roughly equal to mine, and end up 1st, whereas Dmitry had a way better public and got almost the same private LB as mine. The best explanation I have is that the FID can be a bit (a lot?) random, but I'd be curious to see if people that went up in the private actually tuned their models on other datasets or something else. \nI really enjoyed this competition, and I am looking forward to the next ones!
", "rawMarkdown": "[6th place kernel](https://www.kaggle.com/theoviel/conditional-progan-30-public?scriptVersionId=18822128)\n- [Associated repository](https://github.com/TheoViel/kaggle_dogs/edit/master/README.md)\n- [And the (small) writeup](https://www.kaggle.com/c/generative-dog-images/discussion/104215#latest-601889)\n\nSome takeaways : \nAlthough I expected to rank around 6th, I was very surprised when I saw the private LB. Tom had a public roughly equal to mine, and end up 1st, whereas Dmitry had a way better public and got almost the same private LB as mine. The best explanation I have is that the FID can be a bit (a lot?) random, but I'd be curious to see if people that went up in the private actually tuned their models on other datasets or something else. \nI really enjoyed this competition, and I am looking forward to the next ones!", "votes": null }, { "id": "610488", "postDate": "08/28/2019 20:41:48", "content": "It seems to me now that tuning a random seed/training for less epocns after all was better than fixing the unlucky settings and trying to improve upon it with experiments. I don't know if the instability of GAN's or the good luck does it, but it helped a lot 🤔
", "rawMarkdown": "It seems to me now that tuning a random seed/training for less epocns after all was better than fixing the unlucky settings and trying to improve upon it with experiments. I don't know if the instability of GAN's or the good luck does it, but it helped a lot 🤔", "votes": null }, { "id": "610624", "postDate": "08/28/2019 23:14:55", "content": "Bad luck :(
\n\n.bmp?generation=1567034048054033&alt=media)
I may be mistaken, but I don't think it's due to bad luck. The ~7 solution is probably due to memorizer GAN that got the penalty (9999999). Is the 24.7 solution a fork of imitation game? Because if so, submitting it would have got you the same penalty.
", "rawMarkdown": "I may be mistaken, but I don't think it's due to bad luck. The ~7 solution is probably due to memorizer GAN that got the penalty (9999999). Is the 24.7 solution a fork of imitation game? Because if so, submitting it would have got you the same penalty.", "votes": null }, { "id": "611182", "postDate": "08/29/2019 06:37:48", "content": "What worked for us?\nMaking the network larger, the Architectures that works for us has ~18 million parameters for the generator and ~17 million for the discriminator. Adding FID calculations to the training. Weight initialization At first we implemented weight initialization with nn.init.normal(m.weight.data, 0.0, 0.02) , the results were quite similar to our results without it. When we changed it to nn.init.normal(m.weight.data, 0.0, 0.15-1) the results improved by ~5 points.\nOther parameters and observations Try different seeds, some work better than others.
\n\nWhat didn't work for us?\nPost selection We've tried to use the discriminator to improve our submission. Meaning, to generate 10,000 images that the discriminator labels them close to real images of dogs. I think this approach is valid, however, I think we didn't understand the loss function good enough and didn't have enough time to explore the idea.
\n\nSome takeaways after the private leaderboard were published\nAfter reading some of the amazing solutions, looking at the shake-up and see how my different kernel scored along the way, it seems like calculating the FID from a different dataset seems to make the scoring a bit random.
\n\nAll in all, it was an excellent learning experience and a very fun competition!
", "rawMarkdown": "What worked for us?\n**Making the network larger**, the Architectures that works for us has ~18 million parameters for the generator and ~17 million for the discriminator. Adding **FID calculations** to the training. **Weight initialization** At first we implemented weight initialization with nn.init.normal(m.weight.data, 0.0, 0.02) , the results were quite similar to our results without it. When we changed it to nn.init.normal(m.weight.data, 0.0, 0.15-1) the results improved by ~5 points.\nOther parameters and observations Try **different seeds**, some work better than others. \n\nWhat didn't work for us?\nPost selection We've tried to use the discriminator to improve our submission. Meaning, to generate 10,000 images that the discriminator labels them close to real images of dogs. I think this approach is valid, however, I think we didn't understand the loss function good enough and didn't have enough time to explore the idea.\n\nSome takeaways after the private leaderboard were published\nAfter reading some of the amazing solutions, looking at the shake-up and see how my different kernel scored along the way, it seems like calculating the FID from a different dataset seems to make the scoring a bit random. \n\nAll in all, it was an excellent learning experience and a very fun competition!", "votes": null }, { "id": "611675", "postDate": "08/29/2019 11:30:38", "content": "24.7 solution is my solution simple using GAN , and the other ~7 one using pythorch a d other is memorizer gan
", "rawMarkdown": "24.7 solution is my solution simple using GAN , and the other ~7 one using pythorch a d other is memorizer gan", "votes": null }, { "id": "612399", "postDate": "08/29/2019 19:04:42", "content": "This was my first attempt at a competition so take what I say with a grain of salt. My takeaways from the competition after reading many research papers (unconditional case only):\n* It seems people spent a lot of time trying different GANs based on their loss function, i.e. LSGAN, WGAN, EBGAN, BEGAN. Research on these architectures shows, however, that they can all achieve a similar FID, but the variance in FID can be vastly different, meaning hyper-parameter tuning is more important in some architectures than others. Personally I found a non-saturating loss or a hinge-loss to be the easiest without taking too long to compute, unlike WGAN with GP\n* Use spectral normalization, always. Research shows this allows you to train longer and keep lowering FID at the cost of roughly doubling training time without really having to worry about mode collapse. \n* Do more meaningful work less often. I noticed in discussions people trying to do as many epochs as they could, i.e. 800 epochs. When you think about it, however, this means your network will see an image, or some alternation of it, 800 times, and so your network is very likely to overfit. It is better to instead create a network with larger capacity and use less epochs, thereby reducing the possibility of overfitting to training data.\n* Don't use regular batch normalization. Since we want to generate a diverse set of images, batch-normalization is counter to this and can make images appear more similar than you'd want, i.e. in colors across a batch. Instead use some kind of conditional batch normalization, self-modulation in the unconditional case, or other normalization method
", "rawMarkdown": "This was my first attempt at a competition so take what I say with a grain of salt. My takeaways from the competition after reading many research papers (unconditional case only):\n* It seems people spent a lot of time trying different GANs based on their loss function, i.e. LSGAN, WGAN, EBGAN, BEGAN. Research on these architectures shows, however, that they can all achieve a similar FID, but the variance in FID can be vastly different, meaning hyper-parameter tuning is more important in some architectures than others. Personally I found a non-saturating loss or a hinge-loss to be the easiest without taking too long to compute, unlike WGAN with GP\n* Use spectral normalization, always. Research shows this allows you to train longer and keep lowering FID at the cost of roughly doubling training time without really having to worry about mode collapse. \n* Do more meaningful work less often. I noticed in discussions people trying to do as many epochs as they could, i.e. 800 epochs. When you think about it, however, this means your network will see an image, or some alternation of it, 800 times, and so your network is very likely to overfit. It is better to instead create a network with larger capacity and use less epochs, thereby reducing the possibility of overfitting to training data.\n* Don't use regular batch normalization. Since we want to generate a diverse set of images, batch-normalization is counter to this and can make images appear more similar than you'd want, i.e. in colors across a batch. Instead use some kind of conditional batch normalization, self-modulation in the unconditional case, or other normalization method", "votes": null }, { "id": "612494", "postDate": "08/29/2019 20:25:29", "content": "\n\n\nwhereas Dmitry had a way better public and got almost the same private LB as mine.
\n
I thinks it has something to do with lots of noise in my last images. I really picked an awful ones for my final submission. Most of my other experiments displays 60-something private score. And I haven't properly tested my solution on other datasets or models. Proper validation should be a huge area of improvements for me in the next competitions :)
", "rawMarkdown": "> whereas Dmitry had a way better public and got almost the same private LB as mine.\n\nI thinks it has something to do with lots of noise in my last images. I really picked an awful ones for my final submission. Most of my other experiments displays 60-something private score. And I haven't properly tested my solution on other datasets or models. Proper validation should be a huge area of improvements for me in the next competitions :)", "votes": null }, { "id": "612525", "postDate": "08/29/2019 21:02:47", "content": "Glad to hear you had a bunch of better scoring models, it adds a bit more sense to the metric :)
", "rawMarkdown": "Glad to hear you had a bunch of better scoring models, it adds a bit more sense to the metric :)", "votes": null }, { "id": "612530", "postDate": "08/29/2019 21:07:23", "content": "I could not make spectral normalization work, not sure why. Otherwise agree with your other remarks, very relevant thoughts.
", "rawMarkdown": "I could not make spectral normalization work, not sure why. Otherwise agree with your other remarks, very relevant thoughts.", "votes": null }, { "id": "612935", "postDate": "08/30/2019 05:19:17", "content": "I found it always worked best using SN in both generator and discriminator, as in SAGAN. Every time I used it just in discriminator the results improved, but not significantly.
", "rawMarkdown": "I found it always worked best using SN in both generator and discriminator, as in SAGAN. Every time I used it just in discriminator the results improved, but not significantly.", "votes": null }, { "id": "1151054", "postDate": "01/13/2021 05:17:50", "content": "it is such a surprise to see you have done such a wonderful job, very nice to know you if possible.
", "rawMarkdown": "it is such a surprise to see you have done such a wonderful job, very nice to know you if possible.", "votes": null } ], "comments": [ { "id": 1151054, "author_name": "baikangliu", "author_url": "", "post_date": "01/13/2021 05:17:50", "content": "it is such a surprise to see you have done such a wonderful job, very nice to know you if possible.
", "votes": null, "replies": [] }, { "id": 610464, "author_name": "theoviel", "author_url": "", "post_date": "08/28/2019 20:27:19", "content": "\n\nSome takeaways : \nAlthough I expected to rank around 6th, I was very surprised when I saw the private LB. Tom had a public roughly equal to mine, and end up 1st, whereas Dmitry had a way better public and got almost the same private LB as mine. The best explanation I have is that the FID can be a bit (a lot?) random, but I'd be curious to see if people that went up in the private actually tuned their models on other datasets or something else. \nI really enjoyed this competition, and I am looking forward to the next ones!
", "votes": null, "replies": [ { "id": 612494, "author_name": "dvorobiev", "author_url": "", "post_date": "08/29/2019 20:25:29", "content": "\n\n\nwhereas Dmitry had a way better public and got almost the same private LB as mine.
\n
I thinks it has something to do with lots of noise in my last images. I really picked an awful ones for my final submission. Most of my other experiments displays 60-something private score. And I haven't properly tested my solution on other datasets or models. Proper validation should be a huge area of improvements for me in the next competitions :)
", "votes": null, "replies": [] }, { "id": 612525, "author_name": "theoviel", "author_url": "", "post_date": "08/29/2019 21:02:47", "content": "Glad to hear you had a bunch of better scoring models, it adds a bit more sense to the metric :)
", "votes": null, "replies": [] } ] }, { "id": 610488, "author_name": "devvindan", "author_url": "", "post_date": "08/28/2019 20:41:48", "content": "It seems to me now that tuning a random seed/training for less epocns after all was better than fixing the unlucky settings and trying to improve upon it with experiments. I don't know if the instability of GAN's or the good luck does it, but it helped a lot 🤔
", "votes": null, "replies": [] }, { "id": 610624, "author_name": "moussaid", "author_url": "", "post_date": "08/28/2019 23:14:55", "content": "Bad luck :(
\n\n.bmp?generation=1567034048054033&alt=media)
I may be mistaken, but I don't think it's due to bad luck. The ~7 solution is probably due to memorizer GAN that got the penalty (9999999). Is the 24.7 solution a fork of imitation game? Because if so, submitting it would have got you the same penalty.
", "votes": null, "replies": [] }, { "id": 611675, "author_name": "moussaid", "author_url": "", "post_date": "08/29/2019 11:30:38", "content": "24.7 solution is my solution simple using GAN , and the other ~7 one using pythorch a d other is memorizer gan
", "votes": null, "replies": [] } ] }, { "id": 611182, "author_name": "shaygu", "author_url": "", "post_date": "08/29/2019 06:37:48", "content": "What worked for us?\nMaking the network larger, the Architectures that works for us has ~18 million parameters for the generator and ~17 million for the discriminator. Adding FID calculations to the training. Weight initialization At first we implemented weight initialization with nn.init.normal(m.weight.data, 0.0, 0.02) , the results were quite similar to our results without it. When we changed it to nn.init.normal(m.weight.data, 0.0, 0.15-1) the results improved by ~5 points.\nOther parameters and observations Try different seeds, some work better than others.
\n\nWhat didn't work for us?\nPost selection We've tried to use the discriminator to improve our submission. Meaning, to generate 10,000 images that the discriminator labels them close to real images of dogs. I think this approach is valid, however, I think we didn't understand the loss function good enough and didn't have enough time to explore the idea.
\n\nSome takeaways after the private leaderboard were published\nAfter reading some of the amazing solutions, looking at the shake-up and see how my different kernel scored along the way, it seems like calculating the FID from a different dataset seems to make the scoring a bit random.
\n\nAll in all, it was an excellent learning experience and a very fun competition!
", "votes": null, "replies": [] }, { "id": 612399, "author_name": "johannl", "author_url": "", "post_date": "08/29/2019 19:04:42", "content": "This was my first attempt at a competition so take what I say with a grain of salt. My takeaways from the competition after reading many research papers (unconditional case only):\n* It seems people spent a lot of time trying different GANs based on their loss function, i.e. LSGAN, WGAN, EBGAN, BEGAN. Research on these architectures shows, however, that they can all achieve a similar FID, but the variance in FID can be vastly different, meaning hyper-parameter tuning is more important in some architectures than others. Personally I found a non-saturating loss or a hinge-loss to be the easiest without taking too long to compute, unlike WGAN with GP\n* Use spectral normalization, always. Research shows this allows you to train longer and keep lowering FID at the cost of roughly doubling training time without really having to worry about mode collapse. \n* Do more meaningful work less often. I noticed in discussions people trying to do as many epochs as they could, i.e. 800 epochs. When you think about it, however, this means your network will see an image, or some alternation of it, 800 times, and so your network is very likely to overfit. It is better to instead create a network with larger capacity and use less epochs, thereby reducing the possibility of overfitting to training data.\n* Don't use regular batch normalization. Since we want to generate a diverse set of images, batch-normalization is counter to this and can make images appear more similar than you'd want, i.e. in colors across a batch. Instead use some kind of conditional batch normalization, self-modulation in the unconditional case, or other normalization method
", "votes": null, "replies": [ { "id": 612530, "author_name": "theoviel", "author_url": "", "post_date": "08/29/2019 21:07:23", "content": "I could not make spectral normalization work, not sure why. Otherwise agree with your other remarks, very relevant thoughts.
", "votes": null, "replies": [] }, { "id": 612935, "author_name": "johannl", "author_url": "", "post_date": "08/30/2019 05:19:17", "content": "I found it always worked best using SN in both generator and discriminator, as in SAGAN. Every time I used it just in discriminator the results improved, but not significantly.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "610431": "This competition has closed! See [Wendy's post](https://www.kaggle.com/c/generative-dog-images/discussion/106206) for the details. \n\nI've created this thread to be a curated list of your amazing work on this competition. As you post your solutions, top kernels, and takeaways, feel free to also comment a link in this thread. If you write a paper, thesis, or present at a conference, and would like to share your work - also add here, to share with the Kaggle community!\n\n**Some aggregated takeaways already shared:**\n- [Top Solutions](https://www.kaggle.com/c/generative-dog-images/discussion/106305) summary post", "610464": "[6th place kernel](https://www.kaggle.com/theoviel/conditional-progan-30-public?scriptVersionId=18822128)\n- [Associated repository](https://github.com/TheoViel/kaggle_dogs/edit/master/README.md)\n- [And the (small) writeup](https://www.kaggle.com/c/generative-dog-images/discussion/104215#latest-601889)\n\nSome takeaways : \nAlthough I expected to rank around 6th, I was very surprised when I saw the private LB. Tom had a public roughly equal to mine, and end up 1st, whereas Dmitry had a way better public and got almost the same private LB as mine. The best explanation I have is that the FID can be a bit (a lot?) random, but I'd be curious to see if people that went up in the private actually tuned their models on other datasets or something else. \nI really enjoyed this competition, and I am looking forward to the next ones!", "610488": "It seems to me now that tuning a random seed/training for less epocns after all was better than fixing the unlucky settings and trying to improve upon it with experiments. I don't know if the instability of GAN's or the good luck does it, but it helped a lot 🤔", "610624": "Bad luck :(\n\n.bmp?generation=1567034048054033&alt=media)", "611164": "I may be mistaken, but I don't think it's due to bad luck. The ~7 solution is probably due to memorizer GAN that got the penalty (9999999). Is the 24.7 solution a fork of imitation game? Because if so, submitting it would have got you the same penalty.", "611182": "What worked for us?\n**Making the network larger**, the Architectures that works for us has ~18 million parameters for the generator and ~17 million for the discriminator. Adding **FID calculations** to the training. **Weight initialization** At first we implemented weight initialization with nn.init.normal(m.weight.data, 0.0, 0.02) , the results were quite similar to our results without it. When we changed it to nn.init.normal(m.weight.data, 0.0, 0.15-1) the results improved by ~5 points.\nOther parameters and observations Try **different seeds**, some work better than others. \n\nWhat didn't work for us?\nPost selection We've tried to use the discriminator to improve our submission. Meaning, to generate 10,000 images that the discriminator labels them close to real images of dogs. I think this approach is valid, however, I think we didn't understand the loss function good enough and didn't have enough time to explore the idea.\n\nSome takeaways after the private leaderboard were published\nAfter reading some of the amazing solutions, looking at the shake-up and see how my different kernel scored along the way, it seems like calculating the FID from a different dataset seems to make the scoring a bit random. \n\nAll in all, it was an excellent learning experience and a very fun competition!", "611675": "24.7 solution is my solution simple using GAN , and the other ~7 one using pythorch a d other is memorizer gan", "612399": "This was my first attempt at a competition so take what I say with a grain of salt. My takeaways from the competition after reading many research papers (unconditional case only):\n* It seems people spent a lot of time trying different GANs based on their loss function, i.e. LSGAN, WGAN, EBGAN, BEGAN. Research on these architectures shows, however, that they can all achieve a similar FID, but the variance in FID can be vastly different, meaning hyper-parameter tuning is more important in some architectures than others. Personally I found a non-saturating loss or a hinge-loss to be the easiest without taking too long to compute, unlike WGAN with GP\n* Use spectral normalization, always. Research shows this allows you to train longer and keep lowering FID at the cost of roughly doubling training time without really having to worry about mode collapse. \n* Do more meaningful work less often. I noticed in discussions people trying to do as many epochs as they could, i.e. 800 epochs. When you think about it, however, this means your network will see an image, or some alternation of it, 800 times, and so your network is very likely to overfit. It is better to instead create a network with larger capacity and use less epochs, thereby reducing the possibility of overfitting to training data.\n* Don't use regular batch normalization. Since we want to generate a diverse set of images, batch-normalization is counter to this and can make images appear more similar than you'd want, i.e. in colors across a batch. Instead use some kind of conditional batch normalization, self-modulation in the unconditional case, or other normalization method", "612494": "> whereas Dmitry had a way better public and got almost the same private LB as mine.\n\nI thinks it has something to do with lots of noise in my last images. I really picked an awful ones for my final submission. Most of my other experiments displays 60-something private score. And I haven't properly tested my solution on other datasets or models. Proper validation should be a huge area of improvements for me in the next competitions :)", "612525": "Glad to hear you had a bunch of better scoring models, it adds a bit more sense to the metric :)", "612530": "I could not make spectral normalization work, not sure why. Otherwise agree with your other remarks, very relevant thoughts.", "612935": "I found it always worked best using SN in both generator and discriminator, as in SAGAN. Every time I used it just in discriminator the results improved, but not significantly.", "1151054": "it is such a surprise to see you have done such a wonderful job, very nice to know you if possible." }, "source": "meta" }