![\"CycleGAN](\"http://i.imgur.com/vtgP8aL.jpg\")
\n![\"CycleGAN](\"http://i.imgur.com/KWPwzA4.jpg\")
Architecture
\n\nWe used a combined MaskRCNN + Unet network that jointly predicted both instance and semantic segmentation masks. MaskRCNN was based off Matterport's implementation and it shared the convolutional feature backbone (resnet50 or resnet101) with the Unet encoder. We believe jointly training semantic and instance masks better conditioned the backbone network, although we have no strong proof. We used the semantic mask output for post-processing.
\n\nData normalization & augmentation
\n\nWe converted images to grayscale and normalized images to their max value (on a per-image basis) and applied threshold based inversion. During training we did flips, 0,90,180 and 270 rotations and gaussian blurring.
\n\nComputing resources
\n\nWe used roughly 4-6 x 1080 Ti.
\n\nEnsembles / TTA
\n\nBoth ensembling different models and merging results from TTA followed the same approach: voting + dilation guided by semantic mask predictions. The steps taken were as follows:
\n\nLessons learned
\n\nCiara:
\n\nAndres:
\n\n\n
(left images are originals, right images are images generated using CycleGAN)
\n\nBig thanks to all the organizers and participants!
\n\nCiara, Ian, Javi and Andres
", "messages": [ { "id": "316072", "postDate": "04/18/2018 05:50:34", "content": "Architecture
\n\nWe used a combined MaskRCNN + Unet network that jointly predicted both instance and semantic segmentation masks. MaskRCNN was based off Matterport's implementation and it shared the convolutional feature backbone (resnet50 or resnet101) with the Unet encoder. We believe jointly training semantic and instance masks better conditioned the backbone network, although we have no strong proof. We used the semantic mask output for post-processing.
\n\nData normalization & augmentation
\n\nWe converted images to grayscale and normalized images to their max value (on a per-image basis) and applied threshold based inversion. During training we did flips, 0,90,180 and 270 rotations and gaussian blurring.
\n\nComputing resources
\n\nWe used roughly 4-6 x 1080 Ti.
\n\nEnsembles / TTA
\n\nBoth ensembling different models and merging results from TTA followed the same approach: voting + dilation guided by semantic mask predictions. The steps taken were as follows:
\n\nLessons learned
\n\nCiara:
\n\nAndres:
\n\n\n
(left images are originals, right images are images generated using CycleGAN)
\n\nBig thanks to all the organizers and participants!
\n\nCiara, Ian, Javi and Andres
", "rawMarkdown": "**Architecture**\n\nWe used a combined MaskRCNN + Unet network that jointly predicted both instance and semantic segmentation masks. MaskRCNN was based off Matterport's implementation and it shared the convolutional feature backbone (resnet50 or resnet101) with the Unet encoder. We believe jointly training semantic and instance masks better conditioned the backbone network, although we have no strong proof. We used the semantic mask output for post-processing.\n\n**Data normalization & augmentation**\n\nWe converted images to grayscale and normalized images to their max value (on a per-image basis) and applied threshold based inversion. During training we did flips, 0,90,180 and 270 rotations and gaussian blurring. \n\n**Computing resources**\n\nWe used roughly 4-6 x 1080 Ti. \n\n**Ensembles / TTA**\n\nBoth ensembling different models and merging results from TTA followed the same approach: voting + dilation guided by semantic mask predictions. The steps taken were as follows:\n\n1. Consider the boundary box for each mask. If the overlap in boundary boxes between masks is > threshold, then merge the boxes (new boundary box encloses all overlapping boxes) and sum the masks.\n2. Select masks from 1. based on a voting threshold. e.g. if more than 50% of cases designated a pixel as 1, then assign 1 to that pixel.\n3. Average the semantic mask output and select pixels based on a voting threshold.\n4. Dilate each mask within its boundary box limits using the semantic mask from 3. as a guide: if the dilated pixels fall within the boundary box and the semantic mask suggests that the pixel = 1, then assign 1 to that pixel.\n\n**Lessons learned**\n\nCiara:\n\n - Teamed up earlier. The real restriction was time: too many ideas to test out within the limits of what we had (basically, it was too much to do in just 2 weeks)\n - Focused less on trying to optimise parameters (so many moving parts!) and spend that time on different post processing / ensembling approaches\n - Spend more time on generating external datasets\n\nAndres:\n\n - Teamed up earlier to discuss and partition approaches: when we joined already each member already had some implementation of the pipeline and we were already too focused on MaskRCNN. In retrospect, we should have devoted some time to dig deeper with Unets.\n - More data: Data is the fuel of deep learning. We should have spend more time getting more diverse data for training, either synthetic datasets or generating new synthetic data. We tried with a CycleGAN to generate pairs from one domain where we did have GT to a different domain where we didn't. While it looked promising we ran out of time so we could not train with it:\n\n![CycleGAN Images 1][1]\n![CycleGAN Images 2][2]\n\n(left images are originals, right images are images generated using CycleGAN)\n\n - Systematic approach to augmentations. With so many gauges to dial, in some instances we added a few augmentations or changes at one and we were not sure which piece was contributing positively.\n\nBig thanks to all the organizers and participants! \n\nCiara, Ian, Javi and Andres\n\n [1]: http://i.imgur.com/vtgP8aL.jpg\n [2]: http://i.imgur.com/KWPwzA4.jpg", "votes": null }, { "id": "316156", "postDate": "04/18/2018 09:35:01", "content": "Code (not beautified!) for the solution above is available here: https://github.com/ciararogerson/Kaggle_Shared/tree/master/Mask_RCNN_DSB2018
", "rawMarkdown": "Code (not beautified!) for the solution above is available here: https://github.com/ciararogerson/Kaggle_Shared/tree/master/Mask_RCNN_DSB2018", "votes": null }, { "id": "316176", "postDate": "04/18/2018 10:46:35", "content": "¡Nice work! Some questions arises to me after reading that and looking quickly at the code.
\n\nThanks for sharing, I'm learning from your work :-)
", "rawMarkdown": "¡Nice work! Some questions arises to me after reading that and looking quickly at the code. \n\n- Your hypothesis is that jointly learning the semantic masks would improve the performance of Mask RCNN alone. Did you test just Mask without the semantic part? Did using the semantic mask help at testing time for TTA?\n - I see that you are using ELUs instead of RELUs at semantic mask part. Original UNet uses RELUs there. Is there a gain for using ELUs?\n- Looking at the code, I have seen that you plugged a U-Net decoder head to the Resnet backbone. Have you seen any reference of some work doing that before? \n- I also see that not all layers at the decoder head are connected symmetrically with the corresponding backbone layers and that used feature layer was C2. I see a comment in your code... probably the net would be too deep if using feature from C4/C5.\n\nThanks for sharing, I'm learning from your work :-)", "votes": null }, { "id": "316212", "postDate": "04/18/2018 12:49:33", "content": "Hi Ivan,
\n\nRe. your questions above:
\n\nWe were so rushed with everything that there were a lot of items we didn't get to definitively check. For example, using the semantic masks as part of the model fit + post processing helped, but I'm not sure how much of that was down to the post-processing and how much was down to the model fitting having been conditioned on the semantic masks as well. Ideally we would have run more thorough testing in this regard.
For the semantic masks, we didn't try out many different architectures at all because we were so time constrained. We used ELUs instead of RELUs because they had worked slightly better in other instances, but we didn't test their relative performance in this instance.
Re. plugging the Unet decoder to the resnet backbone, we didn't reference any one work in particular, but it would be typical of many different implementations like this. We did try out different options in terms of how deep the unet went (e.g. starting decoding from layer C4 rather than layer C3; adding symmetric connections with layers other than C2), but there were problems with convergence and overfitting. There are likely much better set ups than what we ended up using, but we just didn't have time to exploring anything else (a common theme in this project for us!).
Thank's for the answer!
", "rawMarkdown": "Thank's for the answer!", "votes": null }, { "id": "316308", "postDate": "04/18/2018 17:18:07", "content": "@ Andres, nice work! An interesting framework to merge semantic and instance segmentation.
\n\nI roughly looked your code. MaskRCNN has 4 scale levels and the nuclei only appear as ground truth in some scales.\nFor \"unet-head\", do you also has semantic segmentation at 4 scales? How about the ground truth for semantic segmentation?
\n\nTo ask simply, are the ground truths for mask-rcnn and unet the same at each scale?
", "rawMarkdown": "Andres, nice work! An interesting framework to merge semantic and instance segmentation. \n\nI roughly looked your code. MaskRCNN has 4 scale levels and the nuclei only appear as ground truth in some scales.\nFor \"unet-head\", do you also has semantic segmentation at 4 scales? How about the ground truth for semantic segmentation?\n\nTo ask simply, are the ground truths for mask-rcnn and unet the same at each scale?", "votes": null }, { "id": "316311", "postDate": "04/18/2018 17:30:37", "content": "For Unet head we only have pixel-wise segmentation at full scale (same as input image), we just stacked all instance masks into 512x512 together and had only one class (nuclei).
\n\nOne related problem is that we tried (see https://github.com/ciararogerson/Kaggle_Shared/commit/5fd31827cd41437b462ef34af9508856843ddae8) to give the Unet non-downsampled features and the image itself (image, preC1 =>first 2x downsample, C2, etc.) however the net didn't converge, so in essence the Unet we ended up using was upscaling 1:4 for predictions (not good).
\n\nMakes you wonder whether you can take the channels of the Unet close to the bottleneck (assuming to have same number of dimensions at that point... or do avgpooling to get there) and feed it to the RPN as well to help it propose regions (gradient backpropagation may be easier since at that point the graph is less deep).
", "rawMarkdown": "For Unet head we only have pixel-wise segmentation at full scale (same as input image), we just stacked all instance masks into 512x512 together and had only one class (nuclei).\n\nOne related problem is that we tried (see https://github.com/ciararogerson/Kaggle_Shared/commit/5fd31827cd41437b462ef34af9508856843ddae8) to give the Unet non-downsampled features and the image itself (image, preC1 =>first 2x downsample, C2, etc.) however the net didn't converge, so in essence the Unet we ended up using was upscaling 1:4 for predictions (not good).\n\nMakes you wonder whether you can take the channels of the Unet close to the bottleneck (assuming to have same number of dimensions at that point... or do avgpooling to get there) and feed it to the RPN as well to help it propose regions (gradient backpropagation may be easier since at that point the graph is less deep).", "votes": null } ], "comments": [ { "id": 316156, "author_name": "fergusoci", "author_url": "", "post_date": "04/18/2018 09:35:01", "content": "Code (not beautified!) for the solution above is available here: https://github.com/ciararogerson/Kaggle_Shared/tree/master/Mask_RCNN_DSB2018
", "votes": null, "replies": [] }, { "id": 316176, "author_name": "ivanprado", "author_url": "", "post_date": "04/18/2018 10:46:35", "content": "¡Nice work! Some questions arises to me after reading that and looking quickly at the code.
\n\nThanks for sharing, I'm learning from your work :-)
", "votes": null, "replies": [ { "id": 316212, "author_name": "fergusoci", "author_url": "", "post_date": "04/18/2018 12:49:33", "content": "Hi Ivan,
\n\nRe. your questions above:
\n\nWe were so rushed with everything that there were a lot of items we didn't get to definitively check. For example, using the semantic masks as part of the model fit + post processing helped, but I'm not sure how much of that was down to the post-processing and how much was down to the model fitting having been conditioned on the semantic masks as well. Ideally we would have run more thorough testing in this regard.
For the semantic masks, we didn't try out many different architectures at all because we were so time constrained. We used ELUs instead of RELUs because they had worked slightly better in other instances, but we didn't test their relative performance in this instance.
Re. plugging the Unet decoder to the resnet backbone, we didn't reference any one work in particular, but it would be typical of many different implementations like this. We did try out different options in terms of how deep the unet went (e.g. starting decoding from layer C4 rather than layer C3; adding symmetric connections with layers other than C2), but there were problems with convergence and overfitting. There are likely much better set ups than what we ended up using, but we just didn't have time to exploring anything else (a common theme in this project for us!).
Thank's for the answer!
", "votes": null, "replies": [] } ] }, { "id": 316308, "author_name": "hengck23", "author_url": "", "post_date": "04/18/2018 17:18:07", "content": "@ Andres, nice work! An interesting framework to merge semantic and instance segmentation.
\n\nI roughly looked your code. MaskRCNN has 4 scale levels and the nuclei only appear as ground truth in some scales.\nFor \"unet-head\", do you also has semantic segmentation at 4 scales? How about the ground truth for semantic segmentation?
\n\nTo ask simply, are the ground truths for mask-rcnn and unet the same at each scale?
", "votes": null, "replies": [ { "id": 316311, "author_name": "antorsae", "author_url": "", "post_date": "04/18/2018 17:30:37", "content": "For Unet head we only have pixel-wise segmentation at full scale (same as input image), we just stacked all instance masks into 512x512 together and had only one class (nuclei).
\n\nOne related problem is that we tried (see https://github.com/ciararogerson/Kaggle_Shared/commit/5fd31827cd41437b462ef34af9508856843ddae8) to give the Unet non-downsampled features and the image itself (image, preC1 =>first 2x downsample, C2, etc.) however the net didn't converge, so in essence the Unet we ended up using was upscaling 1:4 for predictions (not good).
\n\nMakes you wonder whether you can take the channels of the Unet close to the bottleneck (assuming to have same number of dimensions at that point... or do avgpooling to get there) and feed it to the RPN as well to help it propose regions (gradient backpropagation may be easier since at that point the graph is less deep).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "316072": "**Architecture**\n\nWe used a combined MaskRCNN + Unet network that jointly predicted both instance and semantic segmentation masks. MaskRCNN was based off Matterport's implementation and it shared the convolutional feature backbone (resnet50 or resnet101) with the Unet encoder. We believe jointly training semantic and instance masks better conditioned the backbone network, although we have no strong proof. We used the semantic mask output for post-processing.\n\n**Data normalization & augmentation**\n\nWe converted images to grayscale and normalized images to their max value (on a per-image basis) and applied threshold based inversion. During training we did flips, 0,90,180 and 270 rotations and gaussian blurring. \n\n**Computing resources**\n\nWe used roughly 4-6 x 1080 Ti. \n\n**Ensembles / TTA**\n\nBoth ensembling different models and merging results from TTA followed the same approach: voting + dilation guided by semantic mask predictions. The steps taken were as follows:\n\n1. Consider the boundary box for each mask. If the overlap in boundary boxes between masks is > threshold, then merge the boxes (new boundary box encloses all overlapping boxes) and sum the masks.\n2. Select masks from 1. based on a voting threshold. e.g. if more than 50% of cases designated a pixel as 1, then assign 1 to that pixel.\n3. Average the semantic mask output and select pixels based on a voting threshold.\n4. Dilate each mask within its boundary box limits using the semantic mask from 3. as a guide: if the dilated pixels fall within the boundary box and the semantic mask suggests that the pixel = 1, then assign 1 to that pixel.\n\n**Lessons learned**\n\nCiara:\n\n - Teamed up earlier. The real restriction was time: too many ideas to test out within the limits of what we had (basically, it was too much to do in just 2 weeks)\n - Focused less on trying to optimise parameters (so many moving parts!) and spend that time on different post processing / ensembling approaches\n - Spend more time on generating external datasets\n\nAndres:\n\n - Teamed up earlier to discuss and partition approaches: when we joined already each member already had some implementation of the pipeline and we were already too focused on MaskRCNN. In retrospect, we should have devoted some time to dig deeper with Unets.\n - More data: Data is the fuel of deep learning. We should have spend more time getting more diverse data for training, either synthetic datasets or generating new synthetic data. We tried with a CycleGAN to generate pairs from one domain where we did have GT to a different domain where we didn't. While it looked promising we ran out of time so we could not train with it:\n\n![CycleGAN Images 1][1]\n![CycleGAN Images 2][2]\n\n(left images are originals, right images are images generated using CycleGAN)\n\n - Systematic approach to augmentations. With so many gauges to dial, in some instances we added a few augmentations or changes at one and we were not sure which piece was contributing positively.\n\nBig thanks to all the organizers and participants! \n\nCiara, Ian, Javi and Andres\n\n [1]: http://i.imgur.com/vtgP8aL.jpg\n [2]: http://i.imgur.com/KWPwzA4.jpg", "316156": "Code (not beautified!) for the solution above is available here: https://github.com/ciararogerson/Kaggle_Shared/tree/master/Mask_RCNN_DSB2018", "316176": "¡Nice work! Some questions arises to me after reading that and looking quickly at the code. \n\n- Your hypothesis is that jointly learning the semantic masks would improve the performance of Mask RCNN alone. Did you test just Mask without the semantic part? Did using the semantic mask help at testing time for TTA?\n - I see that you are using ELUs instead of RELUs at semantic mask part. Original UNet uses RELUs there. Is there a gain for using ELUs?\n- Looking at the code, I have seen that you plugged a U-Net decoder head to the Resnet backbone. Have you seen any reference of some work doing that before? \n- I also see that not all layers at the decoder head are connected symmetrically with the corresponding backbone layers and that used feature layer was C2. I see a comment in your code... probably the net would be too deep if using feature from C4/C5.\n\nThanks for sharing, I'm learning from your work :-)", "316212": "Hi Ivan,\n\nRe. your questions above:\n\n- We were so rushed with everything that there were a lot of items we didn't get to definitively check. For example, using the semantic masks as part of the model fit + post processing helped, but I'm not sure how much of that was down to the post-processing and how much was down to the model fitting having been conditioned on the semantic masks as well. Ideally we would have run more thorough testing in this regard.\n\n- For the semantic masks, we didn't try out many different architectures at all because we were so time constrained. We used ELUs instead of RELUs because they had worked slightly better in other instances, but we didn't test their relative performance in this instance.\n\n- Re. plugging the Unet decoder to the resnet backbone, we didn't reference any one work in particular, but it would be typical of many different implementations like this. We did try out different options in terms of how deep the unet went (e.g. starting decoding from layer C4 rather than layer C3; adding symmetric connections with layers other than C2), but there were problems with convergence and overfitting. There are likely much better set ups than what we ended up using, but we just didn't have time to exploring anything else (a common theme in this project for us!).", "316220": "Thank's for the answer!", "316308": "Andres, nice work! An interesting framework to merge semantic and instance segmentation. \n\nI roughly looked your code. MaskRCNN has 4 scale levels and the nuclei only appear as ground truth in some scales.\nFor \"unet-head\", do you also has semantic segmentation at 4 scales? How about the ground truth for semantic segmentation?\n\nTo ask simply, are the ground truths for mask-rcnn and unet the same at each scale?", "316311": "For Unet head we only have pixel-wise segmentation at full scale (same as input image), we just stacked all instance masks into 512x512 together and had only one class (nuclei).\n\nOne related problem is that we tried (see https://github.com/ciararogerson/Kaggle_Shared/commit/5fd31827cd41437b462ef34af9508856843ddae8) to give the Unet non-downsampled features and the image itself (image, preC1 =>first 2x downsample, C2, etc.) however the net didn't converge, so in essence the Unet we ended up using was upscaling 1:4 for predictions (not good).\n\nMakes you wonder whether you can take the channels of the Unet close to the bottleneck (assuming to have same number of dimensions at that point... or do avgpooling to get there) and feed it to the RPN as well to help it propose regions (gradient backpropagation may be easier since at that point the graph is less deep)." }, "source": "existing_discussion", "complete": true, "competition_id": 8089, "competition_slug": "data-science-bowl-2018", "team_id": 1265531, "solution_url": "/competitions/data-science-bowl-2018/writeups/creepy-relu-hybrid-maskrcnn-unet-solution-lb-0-525" }