{
  "id": 98595,
  "title": "All you need is GAN Hacks",
  "url": "/competitions/generative-dog-images/discussion/98595",
  "author_name": "",
  "post_date": "2019-07-04T20:33:58.300408300Z",
  "votes": 68,
  "comment_count": 21,
  "views": 0,
  "content": "<p>Jason Brownlee (<a href=\"https://machinelearningmastery.com\">https://machinelearningmastery.com</a>) provides an impressive series of suggestions in our to stabilize your GANs and obtain the best results:</p>\n\n<ol>\n<li>Downsample Using Strided Convolutions</li>\n<li>Upsample Using Strided Convolutions</li>\n<li>Use LeakyReLU</li>\n<li>Use Batch Normalization</li>\n<li>Use Gaussian Weight Initialization</li>\n<li>Use Adam Stochastic Gradient Descent</li>\n<li>Scale Images to the Range [-1,1]</li>\n<li>Use a Gaussian Latent Space</li>\n<li>Separate Batches of Real and Fake Images</li>\n<li>Use Label Smoothing</li>\n<li>Use Noisy Labels</li>\n</ol>\n\n<p>You can read all you need to code your optimal GAN in TensorFlow and Keras reading these articles:</p>\n\n<ul>\n<li>How to Implement GAN Hacks to Train Stable Generative Adversarial Networks\n(<a href=\"https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/\">https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/</a>)</li>\n<li>Tips for Training Stable Generative Adversarial Networks\n(<a href=\"https://machinelearningmastery.com/how-to-train-stable-generative-adversarial-networks/\">https://machinelearningmastery.com/how-to-train-stable-generative-adversarial-networks/</a>)</li>\n</ul>\n\n<p>Just try yourself, quite a few work for me :-)</p>",
  "messages": [
    {
      "id": "568386",
      "postDate": "07/04/2019 20:33:58",
      "content": "<p>Jason Brownlee (<a href=\"https://machinelearningmastery.com\">https://machinelearningmastery.com</a>) provides an impressive series of suggestions in our to stabilize your GANs and obtain the best results:</p>\n\n<ol>\n<li>Downsample Using Strided Convolutions</li>\n<li>Upsample Using Strided Convolutions</li>\n<li>Use LeakyReLU</li>\n<li>Use Batch Normalization</li>\n<li>Use Gaussian Weight Initialization</li>\n<li>Use Adam Stochastic Gradient Descent</li>\n<li>Scale Images to the Range [-1,1]</li>\n<li>Use a Gaussian Latent Space</li>\n<li>Separate Batches of Real and Fake Images</li>\n<li>Use Label Smoothing</li>\n<li>Use Noisy Labels</li>\n</ol>\n\n<p>You can read all you need to code your optimal GAN in TensorFlow and Keras reading these articles:</p>\n\n<ul>\n<li>How to Implement GAN Hacks to Train Stable Generative Adversarial Networks\n(<a href=\"https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/\">https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/</a>)</li>\n<li>Tips for Training Stable Generative Adversarial Networks\n(<a href=\"https://machinelearningmastery.com/how-to-train-stable-generative-adversarial-networks/\">https://machinelearningmastery.com/how-to-train-stable-generative-adversarial-networks/</a>)</li>\n</ul>\n\n<p>Just try yourself, quite a few work for me :-)</p>",
      "rawMarkdown": "Jason Brownlee (https://machinelearningmastery.com) provides an impressive series of suggestions in our to stabilize your GANs and obtain the best results:\n\n1. Downsample Using Strided Convolutions\n2. Upsample Using Strided Convolutions\n3. Use LeakyReLU\n4. Use Batch Normalization\n5. Use Gaussian Weight Initialization\n6. Use Adam Stochastic Gradient Descent\n7. Scale Images to the Range [-1,1]\n8. Use a Gaussian Latent Space\n9. Separate Batches of Real and Fake Images\n10. Use Label Smoothing\n11. Use Noisy Labels\n\nYou can read all you need to code your optimal GAN in TensorFlow and Keras reading these articles:\n\n* How to Implement GAN Hacks to Train Stable Generative Adversarial Networks\n(https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/)\n* Tips for Training Stable Generative Adversarial Networks\n(https://machinelearningmastery.com/how-to-train-stable-generative-adversarial-networks/)\n\nJust try yourself, quite a few work for me :-)",
      "votes": null
    },
    {
      "id": "569125",
      "postDate": "07/06/2019 05:39:04",
      "content": "<p>Thank you for your suggestion!</p>",
      "rawMarkdown": "Thank you for your suggestion!",
      "votes": null
    },
    {
      "id": "569969",
      "postDate": "07/07/2019 16:00:25",
      "content": "<p>Here's more GAN hacks (Pytorch) - <a href=\"https://github.com/soumith/ganhacks\">https://github.com/soumith/ganhacks</a> </p>",
      "rawMarkdown": "Here's more GAN hacks (Pytorch) - https://github.com/soumith/ganhacks",
      "votes": null
    },
    {
      "id": "570249",
      "postDate": "07/08/2019 03:54:12",
      "content": "<p>Thanks Luca. These hacks are helping my GANs converge.</p>",
      "rawMarkdown": "Thanks Luca. These hacks are helping my GANs converge.",
      "votes": null
    },
    {
      "id": "570352",
      "postDate": "07/08/2019 07:08:18",
      "content": "<p>I am happy to hear that, Chris. These are tricks that I first learned during a Udacity course on Deep Learning and they worked great with CelebA dataset (and they do still work also for this Stanford Dogs dataset).</p>",
      "rawMarkdown": "I am happy to hear that, Chris. These are tricks that I first learned during a Udacity course on Deep Learning and they worked great with CelebA dataset (and they do still work also for this Stanford Dogs dataset).",
      "votes": null
    },
    {
      "id": "570614",
      "postDate": "07/08/2019 14:41:21",
      "content": "<p>Thanks! very helpful.</p>",
      "rawMarkdown": "Thanks! very helpful.",
      "votes": null
    },
    {
      "id": "572605",
      "postDate": "07/11/2019 07:04:55",
      "content": "<p>Thank you very much, Luca. It's very useful for beginners of GANs like me.</p>",
      "rawMarkdown": "Thank you very much, Luca. It's very useful for beginners of GANs like me.",
      "votes": null
    },
    {
      "id": "573193",
      "postDate": "07/11/2019 23:23:37",
      "content": "<p>What hack did you find most useful up to now <a href=\"/cdeotte\">@cdeotte</a>  ?</p>",
      "rawMarkdown": "What hack did you find most useful up to now @cdeotte  ?",
      "votes": null
    },
    {
      "id": "573198",
      "postDate": "07/11/2019 23:52:37",
      "content": "<p>I added them all at once so I'm not sure which helped the most. Before GAN hacks, I needed to carefully adjust learning rates to keep the Discriminator and Generator in sync. After GAN hacks, I just set the learning rates equal and everything works out. </p>\n\n<p>Now I'm building CGANs and even with GAN hacks I need to carefully monitor the Generator and Discriminator. I believe the presence of class information gives the Discriminator a huge advantage which needs to be regulated.</p>",
      "rawMarkdown": "I added them all at once so I'm not sure which helped the most. Before GAN hacks, I needed to carefully adjust learning rates to keep the Discriminator and Generator in sync. After GAN hacks, I just set the learning rates equal and everything works out. \n\nNow I'm building CGANs and even with GAN hacks I need to carefully monitor the Generator and Discriminator. I believe the presence of class information gives the Discriminator a huge advantage which needs to be regulated.",
      "votes": null
    },
    {
      "id": "581932",
      "postDate": "07/22/2019 14:53:46",
      "content": "<p>I am using LabelSmoothing, so how should I modify my loss function to adapt that? Before that I was just using 0 and 1 as my labels and binary_crossentropy as my loss function, but with smoothed labels I don't think this would be a good loss function to use. I have implemented DCGAN using Keras.</p>",
      "rawMarkdown": "I am using LabelSmoothing, so how should I modify my loss function to adapt that? Before that I was just using 0 and 1 as my labels and binary_crossentropy as my loss function, but with smoothed labels I don't think this would be a good loss function to use. I have implemented DCGAN using Keras.",
      "votes": null
    },
    {
      "id": "582620",
      "postDate": "07/23/2019 11:23:15",
      "content": "<p>using smoothed label helps training in my experience, use random values between 0.5~0.8 for real label. just try and see the result :)</p>",
      "rawMarkdown": "using smoothed label helps training in my experience, use random values between 0.5~0.8 for real label. just try and see the result :)",
      "votes": null
    },
    {
      "id": "582678",
      "postDate": "07/23/2019 12:34:56",
      "content": "<p>In my experience you don't need to change loss function. You just have to adjust the smoothing value, as <a href=\"/yangsaewon\">@yangsaewon</a> pointed out.</p>",
      "rawMarkdown": "In my experience you don't need to change loss function. You just have to adjust the smoothing value, as @yangsaewon pointed out.",
      "votes": null
    },
    {
      "id": "582680",
      "postDate": "07/23/2019 12:47:36",
      "content": "<p><a href=\"/techytushar\">@techytushar</a> If you're using TensorFlow / Keras, smoothing is built in. With <code>tf.keras.losses.BinaryCrossentropy(smoothing=0.4)</code> you get <code>0+0.4/2 = 0.2</code> and <code>1-0.4/2 = 0.8</code> labels (instead of 0 and 1).</p>",
      "rawMarkdown": "techytushar If you're using TensorFlow / Keras, smoothing is built in. With `tf.keras.losses.BinaryCrossentropy(smoothing=0.4)` you get `0+0.4/2 = 0.2` and `1-0.4/2 = 0.8` labels (instead of 0 and 1).",
      "votes": null
    },
    {
      "id": "582692",
      "postDate": "07/23/2019 13:26:41",
      "content": "<p>Thanks a lot for all your suggestions <a href=\"/cdeotte\">@cdeotte</a> <a href=\"/lucamassaron\">@lucamassaron</a> <a href=\"/yangsaewon\">@yangsaewon</a> </p>",
      "rawMarkdown": "Thanks a lot for all your suggestions @cdeotte @lucamassaron @yangsaewon",
      "votes": null
    },
    {
      "id": "582912",
      "postDate": "07/23/2019 19:07:35",
      "content": "<p>As per this video <a href=\"https://www.youtube.com/watch?v=myGAju4L7O8\">GAN, NIPS2016</a> by Soumith, instead of 0 use a value between [0.0,0.3] and instead of 1 use values between [0.7,1.2]</p>",
      "rawMarkdown": "As per this video [GAN, NIPS2016](https://www.youtube.com/watch?v=myGAju4L7O8) by Soumith, instead of 0 use a value between [0.0,0.3] and instead of 1 use values between [0.7,1.2]",
      "votes": null
    },
    {
      "id": "658828",
      "postDate": "10/26/2019 15:35:45",
      "content": "<p>Thanks for the suggestions and resources <a href=\"/lucamassaron\">@lucamassaron</a> </p>",
      "rawMarkdown": "Thanks for the suggestions and resources @lucamassaron",
      "votes": null
    },
    {
      "id": "934176",
      "postDate": "07/18/2020 09:29:39",
      "content": "<p>Thank you for your suggestions. I just wanted to know whether increasing the latent space dimention to 128 or 256 can have a positive impact or not?</p>",
      "rawMarkdown": "Thank you for your suggestions. I just wanted to know whether increasing the latent space dimention to 128 or 256 can have a positive impact or not?",
      "votes": null
    },
    {
      "id": "934697",
      "postDate": "07/18/2020 16:46:42",
      "content": "<p>for a complex problem as this one, it should help</p>",
      "rawMarkdown": "for a complex problem as this one, it should help",
      "votes": null
    },
    {
      "id": "934726",
      "postDate": "07/18/2020 17:12:41",
      "content": "<p>Thanks, I keep that in my mind</p>",
      "rawMarkdown": "Thanks, I keep that in my mind",
      "votes": null
    },
    {
      "id": "1672017",
      "postDate": "02/01/2022 21:03:08",
      "content": "<p>Considerable Approach</p>",
      "rawMarkdown": "Considerable Approach",
      "votes": null
    },
    {
      "id": "1706153",
      "postDate": "02/27/2022 08:54:07",
      "content": "<p>Thank you so much for sharing the knowledge! Really appreciate it!</p>",
      "rawMarkdown": "Thank you so much for sharing the knowledge! Really appreciate it!",
      "votes": null
    },
    {
      "id": "2222198",
      "postDate": "04/15/2023 03:27:50",
      "content": "<p>Thank you for your sharing, which is very helpful to me❤️</p>",
      "rawMarkdown": "Thank you for your sharing, which is very helpful to me❤️",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1672017,
      "author_name": "baqirn",
      "author_url": "",
      "post_date": "02/01/2022 21:03:08",
      "content": "<p>Considerable Approach</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1706153,
      "author_name": "nanditab35",
      "author_url": "",
      "post_date": "02/27/2022 08:54:07",
      "content": "<p>Thank you so much for sharing the knowledge! Really appreciate it!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2222198,
      "author_name": "zhongyanmin",
      "author_url": "",
      "post_date": "04/15/2023 03:27:50",
      "content": "<p>Thank you for your sharing, which is very helpful to me❤️</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 569125,
      "author_name": "tenffe",
      "author_url": "",
      "post_date": "07/06/2019 05:39:04",
      "content": "<p>Thank you for your suggestion!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 569969,
      "author_name": "rashmibanthia",
      "author_url": "",
      "post_date": "07/07/2019 16:00:25",
      "content": "<p>Here's more GAN hacks (Pytorch) - <a href=\"https://github.com/soumith/ganhacks\">https://github.com/soumith/ganhacks</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 570249,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "07/08/2019 03:54:12",
      "content": "<p>Thanks Luca. These hacks are helping my GANs converge.</p>",
      "votes": null,
      "replies": [
        {
          "id": 570352,
          "author_name": "lucamassaron",
          "author_url": "",
          "post_date": "07/08/2019 07:08:18",
          "content": "<p>I am happy to hear that, Chris. These are tricks that I first learned during a Udacity course on Deep Learning and they worked great with CelebA dataset (and they do still work also for this Stanford Dogs dataset).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 573193,
          "author_name": "lucamassaron",
          "author_url": "",
          "post_date": "07/11/2019 23:23:37",
          "content": "<p>What hack did you find most useful up to now <a href=\"/cdeotte\">@cdeotte</a>  ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 573198,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "07/11/2019 23:52:37",
          "content": "<p>I added them all at once so I'm not sure which helped the most. Before GAN hacks, I needed to carefully adjust learning rates to keep the Discriminator and Generator in sync. After GAN hacks, I just set the learning rates equal and everything works out. </p>\n\n<p>Now I'm building CGANs and even with GAN hacks I need to carefully monitor the Generator and Discriminator. I believe the presence of class information gives the Discriminator a huge advantage which needs to be regulated.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 570614,
      "author_name": "dhaqui",
      "author_url": "",
      "post_date": "07/08/2019 14:41:21",
      "content": "<p>Thanks! very helpful.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 572605,
      "author_name": "cifangyiquan",
      "author_url": "",
      "post_date": "07/11/2019 07:04:55",
      "content": "<p>Thank you very much, Luca. It's very useful for beginners of GANs like me.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 581932,
      "author_name": "techytushar",
      "author_url": "",
      "post_date": "07/22/2019 14:53:46",
      "content": "<p>I am using LabelSmoothing, so how should I modify my loss function to adapt that? Before that I was just using 0 and 1 as my labels and binary_crossentropy as my loss function, but with smoothed labels I don't think this would be a good loss function to use. I have implemented DCGAN using Keras.</p>",
      "votes": null,
      "replies": [
        {
          "id": 582620,
          "author_name": "yangsaewon",
          "author_url": "",
          "post_date": "07/23/2019 11:23:15",
          "content": "<p>using smoothed label helps training in my experience, use random values between 0.5~0.8 for real label. just try and see the result :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 582678,
          "author_name": "lucamassaron",
          "author_url": "",
          "post_date": "07/23/2019 12:34:56",
          "content": "<p>In my experience you don't need to change loss function. You just have to adjust the smoothing value, as <a href=\"/yangsaewon\">@yangsaewon</a> pointed out.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 582680,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "07/23/2019 12:47:36",
          "content": "<p><a href=\"/techytushar\">@techytushar</a> If you're using TensorFlow / Keras, smoothing is built in. With <code>tf.keras.losses.BinaryCrossentropy(smoothing=0.4)</code> you get <code>0+0.4/2 = 0.2</code> and <code>1-0.4/2 = 0.8</code> labels (instead of 0 and 1).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 582912,
          "author_name": "sankarshan7",
          "author_url": "",
          "post_date": "07/23/2019 19:07:35",
          "content": "<p>As per this video <a href=\"https://www.youtube.com/watch?v=myGAju4L7O8\">GAN, NIPS2016</a> by Soumith, instead of 0 use a value between [0.0,0.3] and instead of 1 use values between [0.7,1.2]</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 582692,
      "author_name": "techytushar",
      "author_url": "",
      "post_date": "07/23/2019 13:26:41",
      "content": "<p>Thanks a lot for all your suggestions <a href=\"/cdeotte\">@cdeotte</a> <a href=\"/lucamassaron\">@lucamassaron</a> <a href=\"/yangsaewon\">@yangsaewon</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 658828,
      "author_name": "niranjankumarc",
      "author_url": "",
      "post_date": "10/26/2019 15:35:45",
      "content": "<p>Thanks for the suggestions and resources <a href=\"/lucamassaron\">@lucamassaron</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 934176,
      "author_name": "shakibyz",
      "author_url": "",
      "post_date": "07/18/2020 09:29:39",
      "content": "<p>Thank you for your suggestions. I just wanted to know whether increasing the latent space dimention to 128 or 256 can have a positive impact or not?</p>",
      "votes": null,
      "replies": [
        {
          "id": 934697,
          "author_name": "lucamassaron",
          "author_url": "",
          "post_date": "07/18/2020 16:46:42",
          "content": "<p>for a complex problem as this one, it should help</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 934726,
          "author_name": "shakibyz",
          "author_url": "",
          "post_date": "07/18/2020 17:12:41",
          "content": "<p>Thanks, I keep that in my mind</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "568386": "Jason Brownlee (https://machinelearningmastery.com) provides an impressive series of suggestions in our to stabilize your GANs and obtain the best results:\n\n1. Downsample Using Strided Convolutions\n2. Upsample Using Strided Convolutions\n3. Use LeakyReLU\n4. Use Batch Normalization\n5. Use Gaussian Weight Initialization\n6. Use Adam Stochastic Gradient Descent\n7. Scale Images to the Range [-1,1]\n8. Use a Gaussian Latent Space\n9. Separate Batches of Real and Fake Images\n10. Use Label Smoothing\n11. Use Noisy Labels\n\nYou can read all you need to code your optimal GAN in TensorFlow and Keras reading these articles:\n\n* How to Implement GAN Hacks to Train Stable Generative Adversarial Networks\n(https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/)\n* Tips for Training Stable Generative Adversarial Networks\n(https://machinelearningmastery.com/how-to-train-stable-generative-adversarial-networks/)\n\nJust try yourself, quite a few work for me :-)",
    "569125": "Thank you for your suggestion!",
    "569969": "Here's more GAN hacks (Pytorch) - https://github.com/soumith/ganhacks",
    "570249": "Thanks Luca. These hacks are helping my GANs converge.",
    "570352": "I am happy to hear that, Chris. These are tricks that I first learned during a Udacity course on Deep Learning and they worked great with CelebA dataset (and they do still work also for this Stanford Dogs dataset).",
    "570614": "Thanks! very helpful.",
    "572605": "Thank you very much, Luca. It's very useful for beginners of GANs like me.",
    "573193": "What hack did you find most useful up to now @cdeotte  ?",
    "573198": "I added them all at once so I'm not sure which helped the most. Before GAN hacks, I needed to carefully adjust learning rates to keep the Discriminator and Generator in sync. After GAN hacks, I just set the learning rates equal and everything works out. \n\nNow I'm building CGANs and even with GAN hacks I need to carefully monitor the Generator and Discriminator. I believe the presence of class information gives the Discriminator a huge advantage which needs to be regulated.",
    "581932": "I am using LabelSmoothing, so how should I modify my loss function to adapt that? Before that I was just using 0 and 1 as my labels and binary_crossentropy as my loss function, but with smoothed labels I don't think this would be a good loss function to use. I have implemented DCGAN using Keras.",
    "582620": "using smoothed label helps training in my experience, use random values between 0.5~0.8 for real label. just try and see the result :)",
    "582678": "In my experience you don't need to change loss function. You just have to adjust the smoothing value, as @yangsaewon pointed out.",
    "582680": "techytushar If you're using TensorFlow / Keras, smoothing is built in. With `tf.keras.losses.BinaryCrossentropy(smoothing=0.4)` you get `0+0.4/2 = 0.2` and `1-0.4/2 = 0.8` labels (instead of 0 and 1).",
    "582692": "Thanks a lot for all your suggestions @cdeotte @lucamassaron @yangsaewon",
    "582912": "As per this video [GAN, NIPS2016](https://www.youtube.com/watch?v=myGAju4L7O8) by Soumith, instead of 0 use a value between [0.0,0.3] and instead of 1 use values between [0.7,1.2]",
    "658828": "Thanks for the suggestions and resources @lucamassaron",
    "934176": "Thank you for your suggestions. I just wanted to know whether increasing the latent space dimention to 128 or 256 can have a positive impact or not?",
    "934697": "for a complex problem as this one, it should help",
    "934726": "Thanks, I keep that in my mind",
    "1672017": "Considerable Approach",
    "1706153": "Thank you so much for sharing the knowledge! Really appreciate it!",
    "2222198": "Thank you for your sharing, which is very helpful to me❤️"
  },
  "source": "meta"
}