![\"paper](\"https://storage.googleapis.com/kaggle-forum-message-attachments/302079/8855/paperfigure.png\")
New paper from fb AI research FAIR. https://arxiv.org/abs/1803.08494
\n\nIn my experience, Mask RCNN is a memory-consuming model. I got no more than batch size = 16 with 256*256 images on two K80 (total memory 24GB). For smaller batch size, group normalization seems to have some advantages.
\n\n
Abstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform or compete with its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.
", "messages": [ { "id": "302079", "postDate": "03/23/2018 17:10:13", "content": "New paper from fb AI research FAIR. https://arxiv.org/abs/1803.08494
\n\nIn my experience, Mask RCNN is a memory-consuming model. I got no more than batch size = 16 with 256*256 images on two K80 (total memory 24GB). For smaller batch size, group normalization seems to have some advantages.
\n\n
Abstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform or compete with its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.
", "rawMarkdown": "New paper from fb AI research FAIR. https://arxiv.org/abs/1803.08494\n\nIn my experience, Mask RCNN is a memory-consuming model. I got no more than batch size = 16 with 256*256 images on two K80 (total memory 24GB). For smaller batch size, group normalization seems to have some advantages.\n\n![paper figure][1]\n\nAbstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform or compete with its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/302079/8855/paperfigure.png", "votes": null }, { "id": "302129", "postDate": "03/23/2018 18:03:40", "content": "interesting
", "rawMarkdown": "interesting", "votes": null }, { "id": "302285", "postDate": "03/23/2018 21:30:05", "content": "Thanks for sharing. I am only able to get to a batch size of 2 (of course it depends on image size) with mask-rcnn using 1 gpu.
\n\nIIRC Heng also mentioned before a method that can be used to \"pause\" gradient updates until the batch size is filled up to some pre-determined amount so as to effectively reach a higher batch size. So that's also one alternative without using GN.
", "rawMarkdown": "Thanks for sharing. I am only able to get to a batch size of 2 (of course it depends on image size) with mask-rcnn using 1 gpu. \n\nIIRC Heng also mentioned before a method that can be used to \"pause\" gradient updates until the batch size is filled up to some pre-determined amount so as to effectively reach a higher batch size. So that's also one alternative without using GN.", "votes": null }, { "id": "302352", "postDate": "03/24/2018 00:10:03", "content": "Although the computation of gradient is paused, the computation of BN s not. The usual trick is:
\n\ntrain classification first and learn the parameters of BN (e.g. imagenet)
proceed to train detection using the classification model as initialisation , aka pretrained model.\nBN is frozen when training detection (e.g. voc,coco)
But i think methods like Group Norm may be a much better solution.
", "rawMarkdown": "Although the computation of gradient is paused, the computation of BN s not. The usual trick is:\n\n1. train classification first and learn the parameters of BN (e.g. imagenet)\n\n2. proceed to train detection using the classification model as initialisation , aka pretrained model.\n BN is frozen when training detection (e.g. voc,coco)\n\nBut i think methods like Group Norm may be a much better solution.", "votes": null }, { "id": "302357", "postDate": "03/24/2018 00:15:21", "content": "FYI, i have tried second solution : using pretrain imagenet and frozen BN. First keep the heads because we have a good weight from imagenet, train on other branch. After that, train all network. However, I got only 0.37LB on first phase ( training heads). I am running second phase using Heng code. Do you have similar result as I did, Heng?
", "rawMarkdown": "FYI, i have tried second solution : using pretrain imagenet and frozen BN. First keep the heads because we have a good weight from imagenet, train on other branch. After that, train all network. However, I got only 0.37LB on first phase ( training heads). I am running second phase using Heng code. Do you have similar result as I did, Heng?", "votes": null }, { "id": "304942", "postDate": "03/28/2018 08:55:58", "content": "I have tried Group Norm but the result is not good. It looks fail to training. I am using Heng code. Let me know if you success to train with it. I have changed all batch norm to group batch norm. This is group norm in pytorch https://github.com/taokong/group_normalization
", "rawMarkdown": "I have tried Group Norm but the result is not good. It looks fail to training. I am using Heng code. Let me know if you success to train with it. I have changed all batch norm to group batch norm. This is group norm in pytorch https://github.com/taokong/group_normalization", "votes": null } ], "comments": [ { "id": 302129, "author_name": "wayne999", "author_url": "", "post_date": "03/23/2018 18:03:40", "content": "interesting
", "votes": null, "replies": [] }, { "id": 302285, "author_name": "jamesrequa", "author_url": "", "post_date": "03/23/2018 21:30:05", "content": "Thanks for sharing. I am only able to get to a batch size of 2 (of course it depends on image size) with mask-rcnn using 1 gpu.
\n\nIIRC Heng also mentioned before a method that can be used to \"pause\" gradient updates until the batch size is filled up to some pre-determined amount so as to effectively reach a higher batch size. So that's also one alternative without using GN.
", "votes": null, "replies": [ { "id": 302352, "author_name": "hengck23", "author_url": "", "post_date": "03/24/2018 00:10:03", "content": "Although the computation of gradient is paused, the computation of BN s not. The usual trick is:
\n\ntrain classification first and learn the parameters of BN (e.g. imagenet)
proceed to train detection using the classification model as initialisation , aka pretrained model.\nBN is frozen when training detection (e.g. voc,coco)
But i think methods like Group Norm may be a much better solution.
", "votes": null, "replies": [] }, { "id": 302357, "author_name": "john1231983", "author_url": "", "post_date": "03/24/2018 00:15:21", "content": "FYI, i have tried second solution : using pretrain imagenet and frozen BN. First keep the heads because we have a good weight from imagenet, train on other branch. After that, train all network. However, I got only 0.37LB on first phase ( training heads). I am running second phase using Heng code. Do you have similar result as I did, Heng?
", "votes": null, "replies": [] }, { "id": 304942, "author_name": "john1231983", "author_url": "", "post_date": "03/28/2018 08:55:58", "content": "I have tried Group Norm but the result is not good. It looks fail to training. I am using Heng code. Let me know if you success to train with it. I have changed all batch norm to group batch norm. This is group norm in pytorch https://github.com/taokong/group_normalization
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "302079": "New paper from fb AI research FAIR. https://arxiv.org/abs/1803.08494\n\nIn my experience, Mask RCNN is a memory-consuming model. I got no more than batch size = 16 with 256*256 images on two K80 (total memory 24GB). For smaller batch size, group normalization seems to have some advantages.\n\n![paper figure][1]\n\nAbstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform or compete with its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/302079/8855/paperfigure.png", "302129": "interesting", "302285": "Thanks for sharing. I am only able to get to a batch size of 2 (of course it depends on image size) with mask-rcnn using 1 gpu. \n\nIIRC Heng also mentioned before a method that can be used to \"pause\" gradient updates until the batch size is filled up to some pre-determined amount so as to effectively reach a higher batch size. So that's also one alternative without using GN.", "302352": "Although the computation of gradient is paused, the computation of BN s not. The usual trick is:\n\n1. train classification first and learn the parameters of BN (e.g. imagenet)\n\n2. proceed to train detection using the classification model as initialisation , aka pretrained model.\n BN is frozen when training detection (e.g. voc,coco)\n\nBut i think methods like Group Norm may be a much better solution.", "302357": "FYI, i have tried second solution : using pretrain imagenet and frozen BN. First keep the heads because we have a good weight from imagenet, train on other branch. After that, train all network. However, I got only 0.37LB on first phase ( training heads). I am running second phase using Heng code. Do you have similar result as I did, Heng?", "304942": "I have tried Group Norm but the result is not good. It looks fail to training. I am using Heng code. Let me know if you success to train with it. I have changed all batch norm to group batch norm. This is group norm in pytorch https://github.com/taokong/group_normalization" }, "source": "meta" }