In several other discussion on Kaggle some websites with GAN tips, tricks and hacks are discussed. One of the sites with tips is ganhacks.
\n\nOne of the tips is to use a Dropout of 50% in both train and test phases in the Generator.
\n\nI've tried a few different types of GAN (GAN, WGAN, DCGAN..) and a couple of things I noticed are that using 50% Dropout usually does not benefit the MiFID score. I get better scores when using a Dropout somewhere between 15 - 30% in the Generator. I'am not sure if this could be related to how the MiFID score works.
\n\nAlso I notice in a lot of kernels and samples of GAN's on the internet that there does not seem to be a clear line on whether or not to use Dropout in the Discriminator.
\n\nI was wondering how other Kagglers are using Dropout in their models and what your experiences are with the amount of Dropout to use?
", "messages": [ { "id": "581805", "postDate": "07/22/2019 12:30:04", "content": "In several other discussion on Kaggle some websites with GAN tips, tricks and hacks are discussed. One of the sites with tips is ganhacks.
\n\nOne of the tips is to use a Dropout of 50% in both train and test phases in the Generator.
\n\nI've tried a few different types of GAN (GAN, WGAN, DCGAN..) and a couple of things I noticed are that using 50% Dropout usually does not benefit the MiFID score. I get better scores when using a Dropout somewhere between 15 - 30% in the Generator. I'am not sure if this could be related to how the MiFID score works.
\n\nAlso I notice in a lot of kernels and samples of GAN's on the internet that there does not seem to be a clear line on whether or not to use Dropout in the Discriminator.
\n\nI was wondering how other Kagglers are using Dropout in their models and what your experiences are with the amount of Dropout to use?
", "rawMarkdown": "In several other discussion on Kaggle some websites with GAN tips, tricks and hacks are discussed. One of the sites with tips is [ganhacks](https://github.com/soumith/ganhacks).\n\nOne of the tips is to use a Dropout of 50% in both train and test phases in the Generator.\n\nI've tried a few different types of GAN (GAN, WGAN, DCGAN..) and a couple of things I noticed are that using 50% Dropout usually does not benefit the MiFID score. I get better scores when using a Dropout somewhere between 15 - 30% in the Generator. I'am not sure if this could be related to how the MiFID score works.\n\nAlso I notice in a lot of kernels and samples of GAN's on the internet that there does not seem to be a clear line on whether or not to use Dropout in the Discriminator. \n\nI was wondering how other Kagglers are using Dropout in their models and what your experiences are with the amount of Dropout to use?", "votes": null }, { "id": "581810", "postDate": "07/22/2019 12:35:49", "content": "i didnt get any improvement using dropout(0.5) or dropout (0.3) towards the end layer of generator . However , I could not point towards right direction as there were other changes . Probably i need to try that out .
", "rawMarkdown": "i didnt get any improvement using dropout(0.5) or dropout (0.3) towards the end layer of generator . However , I could not point towards right direction as there were other changes . Probably i need to try that out .", "votes": null }, { "id": "581819", "postDate": "07/22/2019 12:49:05", "content": "You could try more layers with Dropout or see if a smaller value would provide improvements.\nCould also be you have the best results when using no Dropout..Would be interresting to see what model you would need to achieve that.
", "rawMarkdown": "You could try more layers with Dropout or see if a smaller value would provide improvements.\nCould also be you have the best results when using no Dropout..Would be interresting to see what model you would need to achieve that.", "votes": null }, { "id": "581958", "postDate": "07/22/2019 15:30:57", "content": "I also tried adding dropout but didn't improve the score.
", "rawMarkdown": "I also tried adding dropout but didn't improve the score.", "votes": null }, { "id": "581964", "postDate": "07/22/2019 15:34:03", "content": "dropout didn't improve my score :( has anyone benefited from using dropout?
", "rawMarkdown": "dropout didn't improve my score :( has anyone benefited from using dropout?", "votes": null }, { "id": "582164", "postDate": "07/22/2019 20:34:56", "content": "Did you try both Generator and/or Discriminator?
", "rawMarkdown": "Did you try both Generator and/or Discriminator?", "votes": null }, { "id": "582174", "postDate": "07/22/2019 20:56:07", "content": "I have had better success adding tf.keras.layers.GaussianNoise(). Adding this to the Discriminator was required for my CGAN here. Otherwise the Discriminator became too good too quickly and the Generator couldn't keep up.
Interresting. So using GaussianNoise gave better results than using Dropout? I will give that a try tomorrow....curious to see what the difference will be with my current setup. Thanks.
", "rawMarkdown": "Interresting. So using GaussianNoise gave better results than using Dropout? I will give that a try tomorrow....curious to see what the difference will be with my current setup. Thanks.", "votes": null }, { "id": "582324", "postDate": "07/23/2019 04:10:34", "content": "has anyone seen a kernel where gaussian noise is applied in pytorch? :)
", "rawMarkdown": "has anyone seen a kernel where gaussian noise is applied in pytorch? :)", "votes": null }, { "id": "582429", "postDate": "07/23/2019 07:09:20", "content": "I haven't seen a kernel with it but I presume it will be the same as with Keras...swap a Dropout layer with a GaussianNoise layer. I'am not sure if PyTorch already has a GaussianNoise layer as the feature request it is still open...the link however points to some custom code so it should be possible.
", "rawMarkdown": "I haven't seen a kernel with it but I presume it will be the same as with Keras...swap a Dropout layer with a GaussianNoise layer. I'am not sure if PyTorch already has a GaussianNoise layer as the [feature request](https://github.com/pytorch/pytorch/issues/2024) it is still open...the link however points to some custom code so it should be possible.", "votes": null }, { "id": "582551", "postDate": "07/23/2019 10:11:52", "content": "No only in the Generator @rsmits .
", "rawMarkdown": "No only in the Generator @rsmits .", "votes": null }, { "id": "582553", "postDate": "07/23/2019 10:14:31", "content": "@rsmits I see, thanks for the info :)
", "rawMarkdown": "rsmits I see, thanks for the info :)", "votes": null } ], "comments": [ { "id": 581810, "author_name": "phoenix9032", "author_url": "", "post_date": "07/22/2019 12:35:49", "content": "i didnt get any improvement using dropout(0.5) or dropout (0.3) towards the end layer of generator . However , I could not point towards right direction as there were other changes . Probably i need to try that out .
", "votes": null, "replies": [ { "id": 581819, "author_name": "rsmits", "author_url": "", "post_date": "07/22/2019 12:49:05", "content": "You could try more layers with Dropout or see if a smaller value would provide improvements.\nCould also be you have the best results when using no Dropout..Would be interresting to see what model you would need to achieve that.
", "votes": null, "replies": [] } ] }, { "id": 581958, "author_name": "joelhanson", "author_url": "", "post_date": "07/22/2019 15:30:57", "content": "I also tried adding dropout but didn't improve the score.
", "votes": null, "replies": [ { "id": 582164, "author_name": "rsmits", "author_url": "", "post_date": "07/22/2019 20:34:56", "content": "Did you try both Generator and/or Discriminator?
", "votes": null, "replies": [] }, { "id": 582551, "author_name": "joelhanson", "author_url": "", "post_date": "07/23/2019 10:11:52", "content": "No only in the Generator @rsmits .
", "votes": null, "replies": [] } ] }, { "id": 581964, "author_name": "yangsaewon", "author_url": "", "post_date": "07/22/2019 15:34:03", "content": "dropout didn't improve my score :( has anyone benefited from using dropout?
", "votes": null, "replies": [] }, { "id": 582174, "author_name": "cdeotte", "author_url": "", "post_date": "07/22/2019 20:56:07", "content": "I have had better success adding tf.keras.layers.GaussianNoise(). Adding this to the Discriminator was required for my CGAN here. Otherwise the Discriminator became too good too quickly and the Generator couldn't keep up.
Interresting. So using GaussianNoise gave better results than using Dropout? I will give that a try tomorrow....curious to see what the difference will be with my current setup. Thanks.
", "votes": null, "replies": [] }, { "id": 582324, "author_name": "yangsaewon", "author_url": "", "post_date": "07/23/2019 04:10:34", "content": "has anyone seen a kernel where gaussian noise is applied in pytorch? :)
", "votes": null, "replies": [] }, { "id": 582429, "author_name": "rsmits", "author_url": "", "post_date": "07/23/2019 07:09:20", "content": "I haven't seen a kernel with it but I presume it will be the same as with Keras...swap a Dropout layer with a GaussianNoise layer. I'am not sure if PyTorch already has a GaussianNoise layer as the feature request it is still open...the link however points to some custom code so it should be possible.
", "votes": null, "replies": [] }, { "id": 582553, "author_name": "yangsaewon", "author_url": "", "post_date": "07/23/2019 10:14:31", "content": "@rsmits I see, thanks for the info :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "581805": "In several other discussion on Kaggle some websites with GAN tips, tricks and hacks are discussed. One of the sites with tips is [ganhacks](https://github.com/soumith/ganhacks).\n\nOne of the tips is to use a Dropout of 50% in both train and test phases in the Generator.\n\nI've tried a few different types of GAN (GAN, WGAN, DCGAN..) and a couple of things I noticed are that using 50% Dropout usually does not benefit the MiFID score. I get better scores when using a Dropout somewhere between 15 - 30% in the Generator. I'am not sure if this could be related to how the MiFID score works.\n\nAlso I notice in a lot of kernels and samples of GAN's on the internet that there does not seem to be a clear line on whether or not to use Dropout in the Discriminator. \n\nI was wondering how other Kagglers are using Dropout in their models and what your experiences are with the amount of Dropout to use?", "581810": "i didnt get any improvement using dropout(0.5) or dropout (0.3) towards the end layer of generator . However , I could not point towards right direction as there were other changes . Probably i need to try that out .", "581819": "You could try more layers with Dropout or see if a smaller value would provide improvements.\nCould also be you have the best results when using no Dropout..Would be interresting to see what model you would need to achieve that.", "581958": "I also tried adding dropout but didn't improve the score.", "581964": "dropout didn't improve my score :( has anyone benefited from using dropout?", "582164": "Did you try both Generator and/or Discriminator?", "582174": "I have had better success adding `tf.keras.layers.GaussianNoise()`. Adding this to the Discriminator was required for my CGAN [here][1]. Otherwise the Discriminator became too good too quickly and the Generator couldn't keep up.\n\n[1]: https://www.kaggle.com/cdeotte/dog-breed-cgan", "582187": "Interresting. So using GaussianNoise gave better results than using Dropout? I will give that a try tomorrow....curious to see what the difference will be with my current setup. Thanks.", "582324": "has anyone seen a kernel where gaussian noise is applied in pytorch? :)", "582429": "I haven't seen a kernel with it but I presume it will be the same as with Keras...swap a Dropout layer with a GaussianNoise layer. I'am not sure if PyTorch already has a GaussianNoise layer as the [feature request](https://github.com/pytorch/pytorch/issues/2024) it is still open...the link however points to some custom code so it should be possible.", "582551": "No only in the Generator @rsmits .", "582553": "rsmits I see, thanks for the info :)" }, "source": "meta" }