I recently noticed a very nice paper which describes some advantages of using Weight Normalization instead of Batch Normalization. It should result to a more stable training and may'be also better LB score ;-)
\n\nI haven't been able to find an official implementation of a Keras Weight Normalization layer. I was however able to find the code at this github repository. With that code I've made an implementation with it in this notebook.
\n\nI've had a kernel with exact the same code but then using Adam and Batch Normalization. The implementation with Weight Normalization gave me a small improvement in the LB score. Also I get the impression that it allows for more changes in the hyperparameters and still remains 'stable' when training the GAN.
\n\nI'am curious to hear if anybody else is using Weight Normalization and what your impressions about it are?
", "messages": [ { "id": "584257", "postDate": "07/25/2019 16:14:26", "content": "I recently noticed a very nice paper which describes some advantages of using Weight Normalization instead of Batch Normalization. It should result to a more stable training and may'be also better LB score ;-)
\n\nI haven't been able to find an official implementation of a Keras Weight Normalization layer. I was however able to find the code at this github repository. With that code I've made an implementation with it in this notebook.
\n\nI've had a kernel with exact the same code but then using Adam and Batch Normalization. The implementation with Weight Normalization gave me a small improvement in the LB score. Also I get the impression that it allows for more changes in the hyperparameters and still remains 'stable' when training the GAN.
\n\nI'am curious to hear if anybody else is using Weight Normalization and what your impressions about it are?
", "rawMarkdown": "I recently noticed a very nice [paper](https://arxiv.org/pdf/1704.03971.pdf) which describes some advantages of using Weight Normalization instead of Batch Normalization. It should result to a more stable training and may'be also better LB score ;-)\n\nI haven't been able to find an official implementation of a Keras Weight Normalization layer. I was however able to find the code at this [github repository](https://github.com/krasserm/weightnorm/tree/master/keras_2). With that code I've made an implementation with it in this [notebook](https://www.kaggle.com/rsmits/keras-dcgan-with-weight-normalization). \n\nI've had a kernel with exact the same code but then using Adam and Batch Normalization. The implementation with Weight Normalization gave me a small improvement in the LB score. Also I get the impression that it allows for more changes in the hyperparameters and still remains 'stable' when training the GAN.\n\nI'am curious to hear if anybody else is using Weight Normalization and what your impressions about it are?", "votes": null }, { "id": "1690036", "postDate": "02/14/2022 16:48:02", "content": "Batch normalization(BN) to stabilize and improve GAN training is a common practice in many deep network models. But some experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. One of the reasons is that BN is memory-bandwidth bounded and runs into significant run-time during training.
\nWell, several research studies have recently explored weight normalization (WN) algorithms for GAN training, which have shown significant improvement in the stability and quality of the generated samples.
\nResearchers in this paper conclude that training using WN is superior to BN for GANs. Their results achieve 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN.
\nThen there’s this paper demonstrating that WN algorithms outperform BN only for moderately small networks and datasets. Although they have better training curves, WN algorithms show about 6% lower test top-1 accuracy, which cannot be restored by dropout or increasing weight decay.
\nTheir conclusion is that WN algorithms are limited in practical application to relatively shallow networks and thus cannot replace batch normalization for large-scale scenarios.
\nCheck out if you can get more insight on: https://github.com/stormraiser/gan-weightnorm-resnet
\nI have yet to try my hands with WN but my suggestion would be to experiment with the approach and find out if it gives you better results.
\nTo easily validate your data and results, I’d suggest deepchecks Good luck!
", "rawMarkdown": "Batch normalization(BN) to stabilize and improve GAN training is a common practice in many deep network models. But some experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. One of the reasons is that BN is memory-bandwidth bounded and runs into significant run-time during training. \n\nWell, several research studies have recently explored weight normalization (WN) algorithms for GAN training, which have shown significant improvement in the stability and quality of the generated samples. \n\nResearchers in this [paper](https://arxiv.org/abs/1704.03971) conclude that training using WN is superior to BN for GANs. Their results achieve 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN.\n\nThen there’s this [paper](https://www.researchgate.net/publication/320033027_Comparison_of_Batch_Normalization_and_Weight_Normalization_Algorithms_for_the_Large-scale_Image_Classification) demonstrating that WN algorithms outperform BN only for moderately small networks and datasets. Although they have better training curves, WN algorithms show about 6% lower test top-1 accuracy, which cannot be restored by dropout or increasing weight decay. \n\nTheir conclusion is that WN algorithms are limited in practical application to relatively shallow networks and thus cannot replace batch normalization for large-scale scenarios. \n\nCheck out if you can get more insight on: https://github.com/stormraiser/gan-weightnorm-resnet\n\nI have yet to try my hands with WN but my suggestion would be to experiment with the approach and find out if it gives you better results. \n\nTo easily validate your data and results, I’d suggest [deepchecks](https://docs.deepchecks.com/en/stable/) Good luck!", "votes": null } ], "comments": [ { "id": 1690036, "author_name": "erickbecker", "author_url": "", "post_date": "02/14/2022 16:48:02", "content": "Batch normalization(BN) to stabilize and improve GAN training is a common practice in many deep network models. But some experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. One of the reasons is that BN is memory-bandwidth bounded and runs into significant run-time during training.
\nWell, several research studies have recently explored weight normalization (WN) algorithms for GAN training, which have shown significant improvement in the stability and quality of the generated samples.
\nResearchers in this paper conclude that training using WN is superior to BN for GANs. Their results achieve 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN.
\nThen there’s this paper demonstrating that WN algorithms outperform BN only for moderately small networks and datasets. Although they have better training curves, WN algorithms show about 6% lower test top-1 accuracy, which cannot be restored by dropout or increasing weight decay.
\nTheir conclusion is that WN algorithms are limited in practical application to relatively shallow networks and thus cannot replace batch normalization for large-scale scenarios.
\nCheck out if you can get more insight on: https://github.com/stormraiser/gan-weightnorm-resnet
\nI have yet to try my hands with WN but my suggestion would be to experiment with the approach and find out if it gives you better results.
\nTo easily validate your data and results, I’d suggest deepchecks Good luck!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "584257": "I recently noticed a very nice [paper](https://arxiv.org/pdf/1704.03971.pdf) which describes some advantages of using Weight Normalization instead of Batch Normalization. It should result to a more stable training and may'be also better LB score ;-)\n\nI haven't been able to find an official implementation of a Keras Weight Normalization layer. I was however able to find the code at this [github repository](https://github.com/krasserm/weightnorm/tree/master/keras_2). With that code I've made an implementation with it in this [notebook](https://www.kaggle.com/rsmits/keras-dcgan-with-weight-normalization). \n\nI've had a kernel with exact the same code but then using Adam and Batch Normalization. The implementation with Weight Normalization gave me a small improvement in the LB score. Also I get the impression that it allows for more changes in the hyperparameters and still remains 'stable' when training the GAN.\n\nI'am curious to hear if anybody else is using Weight Normalization and what your impressions about it are?", "1690036": "Batch normalization(BN) to stabilize and improve GAN training is a common practice in many deep network models. But some experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. One of the reasons is that BN is memory-bandwidth bounded and runs into significant run-time during training. \n\nWell, several research studies have recently explored weight normalization (WN) algorithms for GAN training, which have shown significant improvement in the stability and quality of the generated samples. \n\nResearchers in this [paper](https://arxiv.org/abs/1704.03971) conclude that training using WN is superior to BN for GANs. Their results achieve 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN.\n\nThen there’s this [paper](https://www.researchgate.net/publication/320033027_Comparison_of_Batch_Normalization_and_Weight_Normalization_Algorithms_for_the_Large-scale_Image_Classification) demonstrating that WN algorithms outperform BN only for moderately small networks and datasets. Although they have better training curves, WN algorithms show about 6% lower test top-1 accuracy, which cannot be restored by dropout or increasing weight decay. \n\nTheir conclusion is that WN algorithms are limited in practical application to relatively shallow networks and thus cannot replace batch normalization for large-scale scenarios. \n\nCheck out if you can get more insight on: https://github.com/stormraiser/gan-weightnorm-resnet\n\nI have yet to try my hands with WN but my suggestion would be to experiment with the approach and find out if it gives you better results. \n\nTo easily validate your data and results, I’d suggest [deepchecks](https://docs.deepchecks.com/en/stable/) Good luck!" }, "source": "meta" }