There's a useful new normalization technique called Group Norm (GN) that has many of the same benefits as Batch Norm (BN), but works well for small batch sizes. Of particular relevance, the authors try GN on Mask R-CNN and claim an increase of 1.4 in bounding box AP on COCO after replacing frozen BN layers in the ResNet backbone with GN and adding GN to the feature pyramid network.
\n\nhttps://arxiv.org/pdf/1803.08494.pdf
\n\nFor anyone using the Matterport code (or any keras model), there are already a couple of implementations on github that can be easily added to your model.
\n\nhttps://github.com/titu1994/Keras-Group-Normalization
\n\nhttps://github.com/shaoanlu/GroupNormalization-keras
\n\nI don't know how it will affect LB results yet, but it does make training noticeably faster and allows you to use larger learning rates with Matterport code (e.g. 1e-2).
", "messages": [ { "id": "309599", "postDate": "04/05/2018 16:51:01", "content": "There's a useful new normalization technique called Group Norm (GN) that has many of the same benefits as Batch Norm (BN), but works well for small batch sizes. Of particular relevance, the authors try GN on Mask R-CNN and claim an increase of 1.4 in bounding box AP on COCO after replacing frozen BN layers in the ResNet backbone with GN and adding GN to the feature pyramid network.
\n\nhttps://arxiv.org/pdf/1803.08494.pdf
\n\nFor anyone using the Matterport code (or any keras model), there are already a couple of implementations on github that can be easily added to your model.
\n\nhttps://github.com/titu1994/Keras-Group-Normalization
\n\nhttps://github.com/shaoanlu/GroupNormalization-keras
\n\nI don't know how it will affect LB results yet, but it does make training noticeably faster and allows you to use larger learning rates with Matterport code (e.g. 1e-2).
", "rawMarkdown": "There's a useful new normalization technique called Group Norm (GN) that has many of the same benefits as Batch Norm (BN), but works well for small batch sizes. Of particular relevance, the authors try GN on Mask R-CNN and claim an increase of 1.4 in bounding box AP on COCO after replacing frozen BN layers in the ResNet backbone with GN and adding GN to the feature pyramid network. \n\n[https://arxiv.org/pdf/1803.08494.pdf][1]\n\nFor anyone using the Matterport code (or any keras model), there are already a couple of implementations on github that can be easily added to your model.\n\n[https://github.com/titu1994/Keras-Group-Normalization][2]\n\n[https://github.com/shaoanlu/GroupNormalization-keras][3]\n\nI don't know how it will affect LB results yet, but it does make training noticeably faster and allows you to use larger learning rates with Matterport code (e.g. 1e-2).\n\n\n [1]: https://arxiv.org/pdf/1803.08494.pdf\n [2]: https://github.com/titu1994/Keras-Group-Normalization\n [3]: https://github.com/shaoanlu/GroupNormalization-keras", "votes": null }, { "id": "309768", "postDate": "04/05/2018 23:56:45", "content": "have you intergrate GN into the backbone or only in the head ?
", "rawMarkdown": "have you intergrate GN into the backbone or only in the head ?", "votes": null }, { "id": "310225", "postDate": "04/06/2018 21:09:08", "content": "In the backbone and the RCNN box and mask heads. For the box head, I used the architecture that the paper references. Instead of two 1024 FC layers, it says to use four 3x3 conv layers with 512 filters and keep the final FC that's used for regression. Using the conv layers apparently makes GN more useful by a small margin. I recommend just using your current model architecture and plugging in the GN layer in place of all BN layers.
", "rawMarkdown": "In the backbone and the RCNN box and mask heads. For the box head, I used the architecture that the paper references. Instead of two 1024 FC layers, it says to use four 3x3 conv layers with 512 filters and keep the final FC that's used for regression. Using the conv layers apparently makes GN more useful by a small margin. I recommend just using your current model architecture and plugging in the GN layer in place of all BN layers.", "votes": null }, { "id": "310667", "postDate": "04/08/2018 07:56:17", "content": "Thank you, I tried to intergrate it in to my code, and it actually degrade my performance.
", "rawMarkdown": "Thank you, I tried to intergrate it in to my code, and it actually degrade my performance.", "votes": null } ], "comments": [ { "id": 309768, "author_name": "algohunt", "author_url": "", "post_date": "04/05/2018 23:56:45", "content": "have you intergrate GN into the backbone or only in the head ?
", "votes": null, "replies": [ { "id": 310225, "author_name": "conradr71", "author_url": "", "post_date": "04/06/2018 21:09:08", "content": "In the backbone and the RCNN box and mask heads. For the box head, I used the architecture that the paper references. Instead of two 1024 FC layers, it says to use four 3x3 conv layers with 512 filters and keep the final FC that's used for regression. Using the conv layers apparently makes GN more useful by a small margin. I recommend just using your current model architecture and plugging in the GN layer in place of all BN layers.
", "votes": null, "replies": [] }, { "id": 310667, "author_name": "algohunt", "author_url": "", "post_date": "04/08/2018 07:56:17", "content": "Thank you, I tried to intergrate it in to my code, and it actually degrade my performance.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "309599": "There's a useful new normalization technique called Group Norm (GN) that has many of the same benefits as Batch Norm (BN), but works well for small batch sizes. Of particular relevance, the authors try GN on Mask R-CNN and claim an increase of 1.4 in bounding box AP on COCO after replacing frozen BN layers in the ResNet backbone with GN and adding GN to the feature pyramid network. \n\n[https://arxiv.org/pdf/1803.08494.pdf][1]\n\nFor anyone using the Matterport code (or any keras model), there are already a couple of implementations on github that can be easily added to your model.\n\n[https://github.com/titu1994/Keras-Group-Normalization][2]\n\n[https://github.com/shaoanlu/GroupNormalization-keras][3]\n\nI don't know how it will affect LB results yet, but it does make training noticeably faster and allows you to use larger learning rates with Matterport code (e.g. 1e-2).\n\n\n [1]: https://arxiv.org/pdf/1803.08494.pdf\n [2]: https://github.com/titu1994/Keras-Group-Normalization\n [3]: https://github.com/shaoanlu/GroupNormalization-keras", "309768": "have you intergrate GN into the backbone or only in the head ?", "310225": "In the backbone and the RCNN box and mask heads. For the box head, I used the architecture that the paper references. Instead of two 1024 FC layers, it says to use four 3x3 conv layers with 512 filters and keep the final FC that's used for regression. Using the conv layers apparently makes GN more useful by a small margin. I recommend just using your current model architecture and plugging in the GN layer in place of all BN layers.", "310667": "Thank you, I tried to intergrate it in to my code, and it actually degrade my performance." }, "source": "meta" }