![\"image\"](\"http://playagricola.com/Kaggle/dogs481419.png\")
First of all, I would like to thank organizers for such a wonderful competition. Generative modeling is a really interesting topic, and I hope to see more such competitions in the future. Also thanks to all participants for sharing their ideas and keeping the forum active :)
\n\nHere is a quick summary of my solution...
\n\nSearch for a good model\nWhen I entered this competition, I had zero experience with GANs or other generative models. So I started with simple models like DCGAN or WGAN, but quickly switched to more complex solutions like Self-Attention GAN. The best score I could get out of SAGAN was 39-40 MiFID, and then I got stuck there. So I continued my search for more efficient models. I played with NVIDIA's StyleGAN for a few days, but didn't get any positive results using it. And then a found BigGAN. It took me a while to take the full advantage of this new model given the amount of settings it had.\nAlso I should mention the great pytorch repo, created by one of the authors of BigGAN, which made using this model a lot easier for me:\nhttps://github.com/ajbrock/BigGAN-PyTorch
\n\nData processing\nMy goal with image transformations at first was to keep as much usefull information as possible. Using simple center cropping often leaves \"headless\" dogs, which might be a bit less recognisable by the scoring network. So I try to expand the usefull image area as much as possible and capture the whole dog or at least the more recognisable parts like head with upper body. I ended up not using center cropping at all and padding tall or wide images most of the time. Given how the FID metric is sensitive to missing modes, it's reasonable to keep most of the images, even the unhandy ones.\nI focused mostly on geometric transformation. Pixel-level augmentation didn't bring any improvements for me. Even the most gentle changes to brightness or contrast usually degraded MiFID metric's value. \nAlso the choice of interpolation method matters, and it was a hard one for me. The popular and recommended way of downsizing images in OpenCV is using INTER_AREA, which gives the most visually appealing smooth images. But the MiFID value of resulting images usually is 1 point less than for other methods (like 10.x vs 9.x), and generated images I get with it are a bit worse. So I ended up using another resizing methods, which led to more sharp but noisy images. Which brought me to using denoise postprocessing algorithm on generated images :)
\n\nHyperparameters\nI tried different number of channels for D and G, different placements of attention layer. I found it a lot easier to use generator with less parameters than default settings. I also ended up using unbalanced networks with D having twice as much channels in conv layers than G (I used mostly 32-64, but tried 28-56, 36-64, 48-72, 48-96). I didn't find any benefits using more complex generator. Additional complexity always slowed training down and resulted with a worse score with no clear visual improvements. My guess is the current task is too simple for a model designed to represent complex datasets likes ImageNet one. I guess equal reduction in channels in D and G should also work fine, but I left D with additional channels because it didn't influenced iteration speed much.\nDespite the benefits of using higher batch sizes, described in BigGAN paper, I ended up using only 32 :) I believe, that the total number of iterations matters too, and a limited time window presents us with the choice of whether we want to update our models more often or more precise.\nAlso I use shared class-embeddings layers in G with hierarchical noise injection.
\n\nOther things I tried:\n1. BigGAN-deep. It never surpassed the perfomance of the original model for me.\n2. truncation of z-vector. Didn't get much with this architecture, though for BigGAN-deep there were some improvements.\n3. stochastic weight averaging (SWA). I usually get about +1-1.5 bigger MiFID metric value using it. But the choise of starting point matters and shifts all the time while I change data pipeline and other settings. So I decided to left EMA which came with the original code.\n4. cyclic learning rates with cosine annealing and soft/hard warmups. With some settings it has a positive impact but tend to lack consistency.\n5. different batch norm substitutes. Didn't spend much time on it, might be an area of potential improvements.\n6. other activation functions. I remember gelu didn't perform very well, but it worked :)\n7. relativistic losses. I tried it with several different architectures. For DCGAN it worked nicely, with SAGAN it didn't make much difference. I didn't managed to make RaHinge loss to work with BigGAN model. It might be a consequence of other setting I had chosen (like stronger discriminator compared to generator net).\n8. other possible z distrubutions as described in the BigGAN paper (Appendix E). Bernoulli didn't work at all, censored Gaussian gave a bit worse results than usual normal distribution.
\n\nYou can find full code for my solution here:\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-final\nhttps://github.com/dmitry-vorobiev/kaggle-generative-dog-images
", "messages": [ { "id": 600202, "postDate": "2019-08-15T19:38:04.627Z", "content": "First of all, I would like to thank organizers for such a wonderful competition. Generative modeling is a really interesting topic, and I hope to see more such competitions in the future. Also thanks to all participants for sharing their ideas and keeping the forum active :)
\n\nHere is a quick summary of my solution...
\n\nSearch for a good model\nWhen I entered this competition, I had zero experience with GANs or other generative models. So I started with simple models like DCGAN or WGAN, but quickly switched to more complex solutions like Self-Attention GAN. The best score I could get out of SAGAN was 39-40 MiFID, and then I got stuck there. So I continued my search for more efficient models. I played with NVIDIA's StyleGAN for a few days, but didn't get any positive results using it. And then a found BigGAN. It took me a while to take the full advantage of this new model given the amount of settings it had.\nAlso I should mention the great pytorch repo, created by one of the authors of BigGAN, which made using this model a lot easier for me:\nhttps://github.com/ajbrock/BigGAN-PyTorch
\n\nData processing\nMy goal with image transformations at first was to keep as much usefull information as possible. Using simple center cropping often leaves \"headless\" dogs, which might be a bit less recognisable by the scoring network. So I try to expand the usefull image area as much as possible and capture the whole dog or at least the more recognisable parts like head with upper body. I ended up not using center cropping at all and padding tall or wide images most of the time. Given how the FID metric is sensitive to missing modes, it's reasonable to keep most of the images, even the unhandy ones.\nI focused mostly on geometric transformation. Pixel-level augmentation didn't bring any improvements for me. Even the most gentle changes to brightness or contrast usually degraded MiFID metric's value. \nAlso the choice of interpolation method matters, and it was a hard one for me. The popular and recommended way of downsizing images in OpenCV is using INTER_AREA, which gives the most visually appealing smooth images. But the MiFID value of resulting images usually is 1 point less than for other methods (like 10.x vs 9.x), and generated images I get with it are a bit worse. So I ended up using another resizing methods, which led to more sharp but noisy images. Which brought me to using denoise postprocessing algorithm on generated images :)
\n\nHyperparameters\nI tried different number of channels for D and G, different placements of attention layer. I found it a lot easier to use generator with less parameters than default settings. I also ended up using unbalanced networks with D having twice as much channels in conv layers than G (I used mostly 32-64, but tried 28-56, 36-64, 48-72, 48-96). I didn't find any benefits using more complex generator. Additional complexity always slowed training down and resulted with a worse score with no clear visual improvements. My guess is the current task is too simple for a model designed to represent complex datasets likes ImageNet one. I guess equal reduction in channels in D and G should also work fine, but I left D with additional channels because it didn't influenced iteration speed much.\nDespite the benefits of using higher batch sizes, described in BigGAN paper, I ended up using only 32 :) I believe, that the total number of iterations matters too, and a limited time window presents us with the choice of whether we want to update our models more often or more precise.\nAlso I use shared class-embeddings layers in G with hierarchical noise injection.
\n\nOther things I tried:\n1. BigGAN-deep. It never surpassed the perfomance of the original model for me.\n2. truncation of z-vector. Didn't get much with this architecture, though for BigGAN-deep there were some improvements.\n3. stochastic weight averaging (SWA). I usually get about +1-1.5 bigger MiFID metric value using it. But the choise of starting point matters and shifts all the time while I change data pipeline and other settings. So I decided to left EMA which came with the original code.\n4. cyclic learning rates with cosine annealing and soft/hard warmups. With some settings it has a positive impact but tend to lack consistency.\n5. different batch norm substitutes. Didn't spend much time on it, might be an area of potential improvements.\n6. other activation functions. I remember gelu didn't perform very well, but it worked :)\n7. relativistic losses. I tried it with several different architectures. For DCGAN it worked nicely, with SAGAN it didn't make much difference. I didn't managed to make RaHinge loss to work with BigGAN model. It might be a consequence of other setting I had chosen (like stronger discriminator compared to generator net).\n8. other possible z distrubutions as described in the BigGAN paper (Appendix E). Bernoulli didn't work at all, censored Gaussian gave a bit worse results than usual normal distribution.
\n\nYou can find full code for my solution here:\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-final\nhttps://github.com/dmitry-vorobiev/kaggle-generative-dog-images
", "rawMarkdown": "First of all, I would like to thank organizers for such a wonderful competition. Generative modeling is a really interesting topic, and I hope to see more such competitions in the future. Also thanks to all participants for sharing their ideas and keeping the forum active :)\n\nHere is a quick summary of my solution...\n\n***Search for a good model***\nWhen I entered this competition, I had zero experience with GANs or other generative models. So I started with simple models like DCGAN or WGAN, but quickly switched to more complex solutions like Self-Attention GAN. The best score I could get out of SAGAN was 39-40 MiFID, and then I got stuck there. So I continued my search for more efficient models. I played with NVIDIA's StyleGAN for a few days, but didn't get any positive results using it. And then a found BigGAN. It took me a while to take the full advantage of this new model given the amount of settings it had.\nAlso I should mention the great pytorch repo, created by one of the authors of BigGAN, which made using this model a lot easier for me:\nhttps://github.com/ajbrock/BigGAN-PyTorch\n\n***Data processing***\nMy goal with image transformations at first was to keep as much usefull information as possible. Using simple center cropping often leaves \"headless\" dogs, which might be a bit less recognisable by the scoring network. So I try to expand the usefull image area as much as possible and capture the whole dog or at least the more recognisable parts like head with upper body. I ended up not using center cropping at all and padding tall or wide images most of the time. Given how the FID metric is sensitive to missing modes, it's reasonable to keep most of the images, even the unhandy ones.\nI focused mostly on geometric transformation. Pixel-level augmentation didn't bring any improvements for me. Even the most gentle changes to brightness or contrast usually degraded MiFID metric's value. \nAlso the choice of interpolation method matters, and it was a hard one for me. The popular and recommended way of downsizing images in OpenCV is using INTER_AREA, which gives the most visually appealing smooth images. But the MiFID value of resulting images usually is 1 point less than for other methods (like 10.x vs 9.x), and generated images I get with it are a bit worse. So I ended up using another resizing methods, which led to more sharp but noisy images. Which brought me to using denoise postprocessing algorithm on generated images :) \n\n***Hyperparameters***\nI tried different number of channels for D and G, different placements of attention layer. I found it a lot easier to use generator with less parameters than default settings. I also ended up using unbalanced networks with D having twice as much channels in conv layers than G (I used mostly 32-64, but tried 28-56, 36-64, 48-72, 48-96). I didn't find any benefits using more complex generator. Additional complexity always slowed training down and resulted with a worse score with no clear visual improvements. My guess is the current task is too simple for a model designed to represent complex datasets likes ImageNet one. I guess equal reduction in channels in D and G should also work fine, but I left D with additional channels because it didn't influenced iteration speed much.\nDespite the benefits of using higher batch sizes, described in BigGAN paper, I ended up using only 32 :) I believe, that the total number of iterations matters too, and a limited time window presents us with the choice of whether we want to update our models more often or more precise.\nAlso I use shared class-embeddings layers in G with hierarchical noise injection.\n\n***Other things I tried:***\n1. BigGAN-deep. It never surpassed the perfomance of the original model for me.\n2. truncation of z-vector. Didn't get much with this architecture, though for BigGAN-deep there were some improvements.\n3. stochastic weight averaging (SWA). I usually get about +1-1.5 bigger MiFID metric value using it. But the choise of starting point matters and shifts all the time while I change data pipeline and other settings. So I decided to left EMA which came with the original code.\n4. cyclic learning rates with cosine annealing and soft/hard warmups. With some settings it has a positive impact but tend to lack consistency.\n5. different batch norm substitutes. Didn't spend much time on it, might be an area of potential improvements.\n6. other activation functions. I remember gelu didn't perform very well, but it worked :)\n7. relativistic losses. I tried it with several different architectures. For DCGAN it worked nicely, with SAGAN it didn't make much difference. I didn't managed to make RaHinge loss to work with BigGAN model. It might be a consequence of other setting I had chosen (like stronger discriminator compared to generator net).\n8. other possible z distrubutions as described in the BigGAN paper (Appendix E). Bernoulli didn't work at all, censored Gaussian gave a bit worse results than usual normal distribution.\n\nYou can find full code for my solution here:\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-final\nhttps://github.com/dmitry-vorobiev/kaggle-generative-dog-images", "votes": 30 }, { "id": 600510, "postDate": "2019-08-16T07:44:03.333Z", "content": "I think I missed the MultiEpochSampler from the original code. It takes me only 02:43 for 1000 iterations with bs = 32 compared to ~4:30 without using it. Will see how far I can get with it...
", "rawMarkdown": "I think I missed the MultiEpochSampler from the original code. It takes me only 02:43 for 1000 iterations with bs = 32 compared to ~4:30 without using it. Will see how far I can get with it...", "votes": 1 }, { "id": 600237, "postDate": "2019-08-15T20:31:17.877Z", "content": "Congratulations Dmitry on creating an awesome GAN. Thanks for sharing your solution and thanks for posting images during the competition. The pictures that you regularly posted in Dog Images Show Off Thread encouraged everyone by demonstrating what traditional GANs are capable of.
\n\nI remember during the first week, everyone said that LB 30 produced by Blender kernel couldn't be beat. You proved them wrong. Below are sample images that score LB 15 from Dmitry's traditional GAN kernel here\n
Hi, @wucf20 \ntry to change config['resolution'] = 64 in train.py at line 23\nI hard-coded several competition-specific parameters, namely the image size and the number of classes. You may want to delete these lines from run() function in train.py and describe them as configurable parameters in prepare_parser() function from utils.py
", "rawMarkdown": "Hi, @wucf20 \ntry to change *config['resolution'] = 64* in train.py at line 23\nI hard-coded several competition-specific parameters, namely the image size and the number of classes. You may want to delete these lines from run() function in train.py and describe them as configurable parameters in prepare_parser() function from utils.py", "replies": [ { "id": 679017, "postDate": "2019-11-22T07:04:29.913Z", "content": "Yes, thank you. Yesterday, I succeeded in outputting 64x64 images; just then, I found config['resolution'] = 64 in train.py and just had the code run. Actually, the naming was good, just that I was kind of not clear-minded enough, although the parameters are indeed numerous. Thank you @dvorobiev (y)
", "rawMarkdown": "Yes, thank you. Yesterday, I succeeded in outputting 64x64 images; just then, I found config['resolution'] = 64 in train.py and just had the code run. Actually, the naming was good, just that I was kind of not clear-minded enough, although the parameters are indeed numerous. Thank you @dvorobiev (y)" } ] }, { "id": 678917, "postDate": "2019-11-22T03:45:47.577Z", "content": "Hello, @dvorobiev,\nI took your github repo and trying to train images other than dogs. Could you please teach me how to have 128x128 output? I was looking into BigGAN.py, I think it's the right path but the model architeture seems complicated to me.
", "rawMarkdown": "Hello, @dvorobiev,\nI took your github repo and trying to train images other than dogs. Could you please teach me how to have 128x128 output? I was looking into BigGAN.py, I think it's the right path but the model architeture seems complicated to me." }, { "id": 600398, "postDate": "2019-08-16T04:07:30.457Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 600510, "author_name": "Dmitry Vorobiev", "author_url": "", "post_date": "2019-08-16T07:44:03.333000", "content": "I think I missed the MultiEpochSampler from the original code. It takes me only 02:43 for 1000 iterations with bs = 32 compared to ~4:30 without using it. Will see how far I can get with it...
", "votes": 1, "replies": [] }, { "id": 600237, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-15T20:31:17.877000", "content": "Congratulations Dmitry on creating an awesome GAN. Thanks for sharing your solution and thanks for posting images during the competition. The pictures that you regularly posted in Dog Images Show Off Thread encouraged everyone by demonstrating what traditional GANs are capable of.
\n\nI remember during the first week, everyone said that LB 30 produced by Blender kernel couldn't be beat. You proved them wrong. Below are sample images that score LB 15 from Dmitry's traditional GAN kernel here\n
Hi, @wucf20 \ntry to change config['resolution'] = 64 in train.py at line 23\nI hard-coded several competition-specific parameters, namely the image size and the number of classes. You may want to delete these lines from run() function in train.py and describe them as configurable parameters in prepare_parser() function from utils.py
", "votes": 0, "replies": [ { "id": 679017, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-22T07:04:29.913000", "content": "Yes, thank you. Yesterday, I succeeded in outputting 64x64 images; just then, I found config['resolution'] = 64 in train.py and just had the code run. Actually, the naming was good, just that I was kind of not clear-minded enough, although the parameters are indeed numerous. Thank you @dvorobiev (y)
", "votes": 0, "replies": [] } ] }, { "id": 678917, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-22T03:45:47.577000", "content": "Hello, @dvorobiev,\nI took your github repo and trying to train images other than dogs. Could you please teach me how to have 128x128 output? I was looking into BigGAN.py, I think it's the right path but the model architeture seems complicated to me.
", "votes": 0, "replies": [] }, { "id": 600398, "author_name": "", "author_url": "", "post_date": "2019-08-16T04:07:30.457000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "600202": "First of all, I would like to thank organizers for such a wonderful competition. Generative modeling is a really interesting topic, and I hope to see more such competitions in the future. Also thanks to all participants for sharing their ideas and keeping the forum active :)\n\nHere is a quick summary of my solution...\n\n***Search for a good model***\nWhen I entered this competition, I had zero experience with GANs or other generative models. So I started with simple models like DCGAN or WGAN, but quickly switched to more complex solutions like Self-Attention GAN. The best score I could get out of SAGAN was 39-40 MiFID, and then I got stuck there. So I continued my search for more efficient models. I played with NVIDIA's StyleGAN for a few days, but didn't get any positive results using it. And then a found BigGAN. It took me a while to take the full advantage of this new model given the amount of settings it had.\nAlso I should mention the great pytorch repo, created by one of the authors of BigGAN, which made using this model a lot easier for me:\nhttps://github.com/ajbrock/BigGAN-PyTorch\n\n***Data processing***\nMy goal with image transformations at first was to keep as much usefull information as possible. Using simple center cropping often leaves \"headless\" dogs, which might be a bit less recognisable by the scoring network. So I try to expand the usefull image area as much as possible and capture the whole dog or at least the more recognisable parts like head with upper body. I ended up not using center cropping at all and padding tall or wide images most of the time. Given how the FID metric is sensitive to missing modes, it's reasonable to keep most of the images, even the unhandy ones.\nI focused mostly on geometric transformation. Pixel-level augmentation didn't bring any improvements for me. Even the most gentle changes to brightness or contrast usually degraded MiFID metric's value. \nAlso the choice of interpolation method matters, and it was a hard one for me. The popular and recommended way of downsizing images in OpenCV is using INTER_AREA, which gives the most visually appealing smooth images. But the MiFID value of resulting images usually is 1 point less than for other methods (like 10.x vs 9.x), and generated images I get with it are a bit worse. So I ended up using another resizing methods, which led to more sharp but noisy images. Which brought me to using denoise postprocessing algorithm on generated images :) \n\n***Hyperparameters***\nI tried different number of channels for D and G, different placements of attention layer. I found it a lot easier to use generator with less parameters than default settings. I also ended up using unbalanced networks with D having twice as much channels in conv layers than G (I used mostly 32-64, but tried 28-56, 36-64, 48-72, 48-96). I didn't find any benefits using more complex generator. Additional complexity always slowed training down and resulted with a worse score with no clear visual improvements. My guess is the current task is too simple for a model designed to represent complex datasets likes ImageNet one. I guess equal reduction in channels in D and G should also work fine, but I left D with additional channels because it didn't influenced iteration speed much.\nDespite the benefits of using higher batch sizes, described in BigGAN paper, I ended up using only 32 :) I believe, that the total number of iterations matters too, and a limited time window presents us with the choice of whether we want to update our models more often or more precise.\nAlso I use shared class-embeddings layers in G with hierarchical noise injection.\n\n***Other things I tried:***\n1. BigGAN-deep. It never surpassed the perfomance of the original model for me.\n2. truncation of z-vector. Didn't get much with this architecture, though for BigGAN-deep there were some improvements.\n3. stochastic weight averaging (SWA). I usually get about +1-1.5 bigger MiFID metric value using it. But the choise of starting point matters and shifts all the time while I change data pipeline and other settings. So I decided to left EMA which came with the original code.\n4. cyclic learning rates with cosine annealing and soft/hard warmups. With some settings it has a positive impact but tend to lack consistency.\n5. different batch norm substitutes. Didn't spend much time on it, might be an area of potential improvements.\n6. other activation functions. I remember gelu didn't perform very well, but it worked :)\n7. relativistic losses. I tried it with several different architectures. For DCGAN it worked nicely, with SAGAN it didn't make much difference. I didn't managed to make RaHinge loss to work with BigGAN model. It might be a consequence of other setting I had chosen (like stronger discriminator compared to generator net).\n8. other possible z distrubutions as described in the BigGAN paper (Appendix E). Bernoulli didn't work at all, censored Gaussian gave a bit worse results than usual normal distribution.\n\nYou can find full code for my solution here:\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3\nhttps://www.kaggle.com/dvorobiev/doggies-biggan-sub-final\nhttps://github.com/dmitry-vorobiev/kaggle-generative-dog-images", "600510": "I think I missed the MultiEpochSampler from the original code. It takes me only 02:43 for 1000 iterations with bs = 32 compared to ~4:30 without using it. Will see how far I can get with it...", "600237": "Congratulations Dmitry on creating an awesome GAN. Thanks for sharing your solution and thanks for posting images during the competition. The pictures that you regularly posted in [Dog Images Show Off Thread][1] encouraged everyone by demonstrating what traditional GANs are capable of. \n\nI remember during the first week, everyone said that LB 30 produced by Blender kernel couldn't be beat. You proved them wrong. Below are sample images that score LB 15 from Dmitry's traditional GAN kernel [here][2]\n \n\n[1]: https://www.kaggle.com/c/generative-dog-images/discussion/97753\n[2]: https://www.kaggle.com/dvorobiev/doggies-biggan-sub-data-aug-3", "679004": "Hi, @wucf20 \ntry to change *config['resolution'] = 64* in train.py at line 23\nI hard-coded several competition-specific parameters, namely the image size and the number of classes. You may want to delete these lines from run() function in train.py and describe them as configurable parameters in prepare_parser() function from utils.py", "678917": "Hello, @dvorobiev,\nI took your github repo and trying to train images other than dogs. Could you please teach me how to have 128x128 output? I was looking into BigGAN.py, I think it's the right path but the model architeture seems complicated to me.", "600398": "" } }