My main goal for this competition is to get to know GAN technique. I often encountered the terminology of GAN in my readings, but did not know what GAN is about. This competition is a great opportunity for me to dive in and practice GAN technology. I want to thank Kaggle team for hosting such an innovative competition and the great efforts to ensure fairness of the competition!
\n\nI followed this article \"Generative Adversarial Networks - The Story So Far\"(https://blog.floydhub.com/gans-story-so-far/) and tried various GAN techniques including DCGAN, CGAN, SAGAN, ProGAN, conditional ProGAN, StyleGAN and BigGAN. At last I settled down with SAGAN because it got me to the score 40+ in my first try and seemed more stable than other network structures. I mostly stick with it and tune it all the way from the score 40+ down to the public score 18. My model implementation is based on this github \"self-attention-GAN-pytorch implementaion\"(https://github.com/voletiv/self-attention-GAN-pytorch)
\n\nHere is the list of things I tried in optimizing the model,
\n\nimage processing\n a) bounding box cropping only;\n b) bounding box cropping for images with multiple objects,\n for images with single object and box_size / image_size >= 0.75, use original images instead of bounding box.\n c) bounding box cropping plus all original images
\n\nAmong the three processing methods, b) gets me the best private score, c) gets me the best public score 18, but did not go well with the private dataset.
\n\nI learnt the image processing techniques from the kernels https://www.kaggle.com/cdeotte/supervised-generative-dog-net and https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images. Special thanks to the kernel authors Chris Deotte and Nanashi!
losses \nI experimented with losses such as \"standard-dcgan\", \"wgan-gp\", \"lsgan\", \"lsgan-with-sigmoid\", \"hinge\", \"relative-hinge\". My experience is the best one for this problem with various models is the \"standard dcgan\" loss, the second best is \"lsgan-with-sigmoid\" loss.
Attention at different stages \nAccording to the paper \"Self-Attention Generative Adversarial Networks\"(https://arxiv.org/pdf/1805.08318v2.pdf), self-attention at middle-to-high level feature maps achieve better performance because it receives more evidence and more freedom to choose conditions with larger feature maps. My experiences mostly concur with the paper findings. I have attention at 32x32 feature map for the generator, but at 16x16 feature map for the discriminator.
Training settings\n The learning rate for the discriminator is 0.0004, and the learning rate for the generator is 0.0002;\n batch size is 32(tried 64 but the result is not as stable as 32);\n label smoothing; thanks the kernel https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers and its author Nirjhar Roy.\n learning rate scheduler: CosineAnnealingWarmRestarts\n number of epoch: 170
5 the best latent space dimension size: 180
\n\nMy best private score kernel(66.97): https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508
", "messages": [ { "id": "612379", "postDate": "08/29/2019 18:50:31", "content": "My main goal for this competition is to get to know GAN technique. I often encountered the terminology of GAN in my readings, but did not know what GAN is about. This competition is a great opportunity for me to dive in and practice GAN technology. I want to thank Kaggle team for hosting such an innovative competition and the great efforts to ensure fairness of the competition!
\n\nI followed this article \"Generative Adversarial Networks - The Story So Far\"(https://blog.floydhub.com/gans-story-so-far/) and tried various GAN techniques including DCGAN, CGAN, SAGAN, ProGAN, conditional ProGAN, StyleGAN and BigGAN. At last I settled down with SAGAN because it got me to the score 40+ in my first try and seemed more stable than other network structures. I mostly stick with it and tune it all the way from the score 40+ down to the public score 18. My model implementation is based on this github \"self-attention-GAN-pytorch implementaion\"(https://github.com/voletiv/self-attention-GAN-pytorch)
\n\nHere is the list of things I tried in optimizing the model,
\n\nimage processing\n a) bounding box cropping only;\n b) bounding box cropping for images with multiple objects,\n for images with single object and box_size / image_size >= 0.75, use original images instead of bounding box.\n c) bounding box cropping plus all original images
\n\nAmong the three processing methods, b) gets me the best private score, c) gets me the best public score 18, but did not go well with the private dataset.
\n\nI learnt the image processing techniques from the kernels https://www.kaggle.com/cdeotte/supervised-generative-dog-net and https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images. Special thanks to the kernel authors Chris Deotte and Nanashi!
losses \nI experimented with losses such as \"standard-dcgan\", \"wgan-gp\", \"lsgan\", \"lsgan-with-sigmoid\", \"hinge\", \"relative-hinge\". My experience is the best one for this problem with various models is the \"standard dcgan\" loss, the second best is \"lsgan-with-sigmoid\" loss.
Attention at different stages \nAccording to the paper \"Self-Attention Generative Adversarial Networks\"(https://arxiv.org/pdf/1805.08318v2.pdf), self-attention at middle-to-high level feature maps achieve better performance because it receives more evidence and more freedom to choose conditions with larger feature maps. My experiences mostly concur with the paper findings. I have attention at 32x32 feature map for the generator, but at 16x16 feature map for the discriminator.
Training settings\n The learning rate for the discriminator is 0.0004, and the learning rate for the generator is 0.0002;\n batch size is 32(tried 64 but the result is not as stable as 32);\n label smoothing; thanks the kernel https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers and its author Nirjhar Roy.\n learning rate scheduler: CosineAnnealingWarmRestarts\n number of epoch: 170
5 the best latent space dimension size: 180
\n\nMy best private score kernel(66.97): https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508
", "rawMarkdown": "My main goal for this competition is to get to know GAN technique. I often encountered the terminology of GAN in my readings, but did not know what GAN is about. This competition is a great opportunity for me to dive in and practice GAN technology. I want to thank Kaggle team for hosting such an innovative competition and the great efforts to ensure fairness of the competition!\n\nI followed this article \"Generative Adversarial Networks - The Story So Far\"(https://blog.floydhub.com/gans-story-so-far/) and tried various GAN techniques including DCGAN, CGAN, SAGAN, ProGAN, conditional ProGAN, StyleGAN and BigGAN. At last I settled down with SAGAN because it got me to the score 40+ in my first try and seemed more stable than other network structures. I mostly stick with it and tune it all the way from the score 40+ down to the public score 18. My model implementation is based on this github \"self-attention-GAN-pytorch implementaion\"(https://github.com/voletiv/self-attention-GAN-pytorch)\n\nHere is the list of things I tried in optimizing the model,\n\n1. image processing\n a) bounding box cropping only;\n b) bounding box cropping for images with multiple objects,\n for images with single object and box_size / image_size >= 0.75, use original images instead of bounding box.\n c) bounding box cropping plus all original images\n\n Among the three processing methods, b) gets me the best private score, c) gets me the best public score 18, but did not go well with the private dataset. \n\n I learnt the image processing techniques from the kernels https://www.kaggle.com/cdeotte/supervised-generative-dog-net and https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images. Special thanks to the kernel authors Chris Deotte and Nanashi!\n\n2. losses \nI experimented with losses such as \"standard-dcgan\", \"wgan-gp\", \"lsgan\", \"lsgan-with-sigmoid\", \"hinge\", \"relative-hinge\". My experience is the best one for this problem with various models is the \"standard dcgan\" loss, the second best is \"lsgan-with-sigmoid\" loss. \n\n3. Attention at different stages \nAccording to the paper \"Self-Attention Generative Adversarial Networks\"(https://arxiv.org/pdf/1805.08318v2.pdf), self-attention at middle-to-high level feature maps achieve better performance because it receives more evidence and more freedom to choose conditions with larger feature maps. My experiences mostly concur with the paper findings. I have attention at 32x32 feature map for the generator, but at 16x16 feature map for the discriminator.\n\n4. Training settings\n The learning rate for the discriminator is 0.0004, and the learning rate for the generator is 0.0002;\n batch size is 32(tried 64 but the result is not as stable as 32);\n label smoothing; thanks the kernel https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers and its author Nirjhar Roy.\n learning rate scheduler: CosineAnnealingWarmRestarts\n number of epoch: 170\n\n5 the best latent space dimension size: 180 \n\nMy best private score kernel(66.97): https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508", "votes": null }, { "id": "612459", "postDate": "08/29/2019 19:56:18", "content": "Congrats. Great job!
", "rawMarkdown": "Congrats. Great job!", "votes": null }, { "id": "612523", "postDate": "08/29/2019 21:00:57", "content": "Congratz ! I read about SaGAN and even tried to add Self Attention in my ProGAN but the training was way too slow so I gave up.
", "rawMarkdown": "Congratz ! I read about SaGAN and even tried to add Self Attention in my ProGAN but the training was way too slow so I gave up.", "votes": null }, { "id": "612758", "postDate": "08/30/2019 01:35:25", "content": "The self attention GAN actually gave best scores ! I had one kernel which didn't give best public score so I didn't choose it .However it gave better private score ..
", "rawMarkdown": "The self attention GAN actually gave best scores ! I had one kernel which didn't give best public score so I didn't choose it .However it gave better private score ..", "votes": null }, { "id": "612903", "postDate": "08/30/2019 04:40:54", "content": "Congrats...!!\nGreat Work...\nThanks for Sharing...!! @lisali
", "rawMarkdown": "Congrats...!!\nGreat Work...\nThanks for Sharing...!! @lisali", "votes": null }, { "id": "614602", "postDate": "08/31/2019 18:23:19", "content": "Congrats. Great work!\nVery informative!!
", "rawMarkdown": "Congrats. Great work!\nVery informative!!", "votes": null }, { "id": "616757", "postDate": "09/03/2019 12:42:38", "content": "Congrats and thanks for sharing. Is it normal that I get a 404 error on your kernel?
", "rawMarkdown": "Congrats and thanks for sharing. Is it normal that I get a 404 error on your kernel?", "votes": null }, { "id": "620909", "postDate": "09/08/2019 04:06:03", "content": "your link https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508 is broken
", "rawMarkdown": "your link [https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508](https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508) is broken", "votes": null }, { "id": "639498", "postDate": "10/03/2019 09:33:22", "content": "Congrats!\nThanks for sharing!
", "rawMarkdown": "Congrats!\nThanks for sharing!", "votes": null }, { "id": "648008", "postDate": "10/13/2019 15:50:48", "content": "this is amazing. even if there are some pictures blurred you've got so hight
", "rawMarkdown": "this is amazing. even if there are some pictures blurred you've got so hight", "votes": null }, { "id": "720390", "postDate": "01/16/2020 11:33:06", "content": "@lisali Thanks for sharing your learning experience and writing in detail about your techniques(and @cdeotte ). I am using GAN to solve similar problem statement but there is one issue.Suppose we have generated dog images and now we want to retrieve a specific breed of dog images from this ,what could be the possible ways to attack this kind of vision problem. I was thinking may be I can experiment with segmentation,categorization and clustering.Do you think this is a good line of thinking.
", "rawMarkdown": "lisali Thanks for sharing your learning experience and writing in detail about your techniques(and @cdeotte ). I am using GAN to solve similar problem statement but there is one issue.Suppose we have generated dog images and now we want to retrieve a specific breed of dog images from this ,what could be the possible ways to attack this kind of vision problem. I was thinking may be I can experiment with segmentation,categorization and clustering.Do you think this is a good line of thinking.", "votes": null } ], "comments": [ { "id": 612459, "author_name": "cdeotte", "author_url": "", "post_date": "08/29/2019 19:56:18", "content": "Congrats. Great job!
", "votes": null, "replies": [] }, { "id": 612523, "author_name": "theoviel", "author_url": "", "post_date": "08/29/2019 21:00:57", "content": "Congratz ! I read about SaGAN and even tried to add Self Attention in my ProGAN but the training was way too slow so I gave up.
", "votes": null, "replies": [] }, { "id": 612758, "author_name": "phoenix9032", "author_url": "", "post_date": "08/30/2019 01:35:25", "content": "The self attention GAN actually gave best scores ! I had one kernel which didn't give best public score so I didn't choose it .However it gave better private score ..
", "votes": null, "replies": [] }, { "id": 612903, "author_name": "veeralakrishna", "author_url": "", "post_date": "08/30/2019 04:40:54", "content": "Congrats...!!\nGreat Work...\nThanks for Sharing...!! @lisali
", "votes": null, "replies": [] }, { "id": 614602, "author_name": "shwetagoyal4", "author_url": "", "post_date": "08/31/2019 18:23:19", "content": "Congrats. Great work!\nVery informative!!
", "votes": null, "replies": [] }, { "id": 616757, "author_name": "sdoria", "author_url": "", "post_date": "09/03/2019 12:42:38", "content": "Congrats and thanks for sharing. Is it normal that I get a 404 error on your kernel?
", "votes": null, "replies": [] }, { "id": 620909, "author_name": "forestchen", "author_url": "", "post_date": "09/08/2019 04:06:03", "content": "your link https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508 is broken
", "votes": null, "replies": [] }, { "id": 639498, "author_name": "maroberti", "author_url": "", "post_date": "10/03/2019 09:33:22", "content": "Congrats!\nThanks for sharing!
", "votes": null, "replies": [] }, { "id": 648008, "author_name": "zinovadr", "author_url": "", "post_date": "10/13/2019 15:50:48", "content": "this is amazing. even if there are some pictures blurred you've got so hight
", "votes": null, "replies": [] }, { "id": 720390, "author_name": "econdata", "author_url": "", "post_date": "01/16/2020 11:33:06", "content": "@lisali Thanks for sharing your learning experience and writing in detail about your techniques(and @cdeotte ). I am using GAN to solve similar problem statement but there is one issue.Suppose we have generated dog images and now we want to retrieve a specific breed of dog images from this ,what could be the possible ways to attack this kind of vision problem. I was thinking may be I can experiment with segmentation,categorization and clustering.Do you think this is a good line of thinking.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "612379": "My main goal for this competition is to get to know GAN technique. I often encountered the terminology of GAN in my readings, but did not know what GAN is about. This competition is a great opportunity for me to dive in and practice GAN technology. I want to thank Kaggle team for hosting such an innovative competition and the great efforts to ensure fairness of the competition!\n\nI followed this article \"Generative Adversarial Networks - The Story So Far\"(https://blog.floydhub.com/gans-story-so-far/) and tried various GAN techniques including DCGAN, CGAN, SAGAN, ProGAN, conditional ProGAN, StyleGAN and BigGAN. At last I settled down with SAGAN because it got me to the score 40+ in my first try and seemed more stable than other network structures. I mostly stick with it and tune it all the way from the score 40+ down to the public score 18. My model implementation is based on this github \"self-attention-GAN-pytorch implementaion\"(https://github.com/voletiv/self-attention-GAN-pytorch)\n\nHere is the list of things I tried in optimizing the model,\n\n1. image processing\n a) bounding box cropping only;\n b) bounding box cropping for images with multiple objects,\n for images with single object and box_size / image_size >= 0.75, use original images instead of bounding box.\n c) bounding box cropping plus all original images\n\n Among the three processing methods, b) gets me the best private score, c) gets me the best public score 18, but did not go well with the private dataset. \n\n I learnt the image processing techniques from the kernels https://www.kaggle.com/cdeotte/supervised-generative-dog-net and https://www.kaggle.com/jesucristo/introducing-dcgan-dogs-images. Special thanks to the kernel authors Chris Deotte and Nanashi!\n\n2. losses \nI experimented with losses such as \"standard-dcgan\", \"wgan-gp\", \"lsgan\", \"lsgan-with-sigmoid\", \"hinge\", \"relative-hinge\". My experience is the best one for this problem with various models is the \"standard dcgan\" loss, the second best is \"lsgan-with-sigmoid\" loss. \n\n3. Attention at different stages \nAccording to the paper \"Self-Attention Generative Adversarial Networks\"(https://arxiv.org/pdf/1805.08318v2.pdf), self-attention at middle-to-high level feature maps achieve better performance because it receives more evidence and more freedom to choose conditions with larger feature maps. My experiences mostly concur with the paper findings. I have attention at 32x32 feature map for the generator, but at 16x16 feature map for the discriminator.\n\n4. Training settings\n The learning rate for the discriminator is 0.0004, and the learning rate for the generator is 0.0002;\n batch size is 32(tried 64 but the result is not as stable as 32);\n label smoothing; thanks the kernel https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers and its author Nirjhar Roy.\n learning rate scheduler: CosineAnnealingWarmRestarts\n number of epoch: 170\n\n5 the best latent space dimension size: 180 \n\nMy best private score kernel(66.97): https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508", "612459": "Congrats. Great job!", "612523": "Congratz ! I read about SaGAN and even tried to add Self Attention in my ProGAN but the training was way too slow so I gave up.", "612758": "The self attention GAN actually gave best scores ! I had one kernel which didn't give best public score so I didn't choose it .However it gave better private score ..", "612903": "Congrats...!!\nGreat Work...\nThanks for Sharing...!! @lisali", "614602": "Congrats. Great work!\nVery informative!!", "616757": "Congrats and thanks for sharing. Is it normal that I get a 404 error on your kernel?", "620909": "your link [https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508](https://www.kaggle.com/lisali/sagan-submit-2?scriptVersionId=18714508) is broken", "639498": "Congrats!\nThanks for sharing!", "648008": "this is amazing. even if there are some pictures blurred you've got so hight", "720390": "lisali Thanks for sharing your learning experience and writing in detail about your techniques(and @cdeotte ). I am using GAN to solve similar problem statement but there is one issue.Suppose we have generated dog images and now we want to retrieve a specific breed of dog images from this ,what could be the possible ways to attack this kind of vision problem. I was thinking may be I can experiment with segmentation,categorization and clustering.Do you think this is a good line of thinking." }, "source": "meta" }