# Hybrid MaskRCNN + UNet solution LB 0.525 (including source code)

Architecture
We used a combined MaskRCNN + Unet network that jointly predicted both instance and semantic segmentation masks. MaskRCNN was based off Matterport's implementation and it shared the convolutional feature backbone (resnet50 or resnet101) with the Unet encoder. We believe jointly training semantic and instance masks better conditioned the backbone network, although we have no strong proof. We used the semantic mask output for post-processing.
Data normalization & augmentation
We converted images to grayscale and normalized images to their max value (on a per-image basis) and applied threshold based inversion. During training we did flips, 0,90,180 and 270 rotations and gaussian blurring.
Computing resources
We used roughly 4-6 x 1080 Ti.
Ensembles / TTA
Both ensembling different models and merging results from TTA followed the same approach: voting + dilation guided by semantic mask predictions. The steps taken were as follows:
Consider the boundary box for each mask. If the overlap in boundary boxes between masks is > threshold, then merge the boxes (new boundary box encloses all overlapping boxes) and sum the masks.
Select masks from 1. based on a voting threshold. e.g. if more than 50% of cases designated a pixel as 1, then assign 1 to that pixel.
Average the semantic mask output and select pixels based on a voting threshold.
Dilate each mask within its boundary box limits using the semantic mask from 3. as a guide: if the dilated pixels fall within the boundary box and the semantic mask suggests that the pixel = 1, then assign 1 to that pixel.
Lessons learned
Ciara:
Teamed up earlier. The real restriction was time: too many ideas to test out within the limits of what we had (basically, it was too much to do in just 2 weeks)
Focused less on trying to optimise parameters (so many moving parts!) and spend that time on different post processing / ensembling approaches
Spend more time on generating external datasets
Andres:
Teamed up earlier to discuss and partition approaches: when we joined already each member already had some implementation of the pipeline and we were already too focused on MaskRCNN. In retrospect, we should have devoted some time to dig deeper with Unets.
More data: Data is the fuel of deep learning. We should have spend more time getting more diverse data for training, either synthetic datasets or generating new synthetic data. We tried with a CycleGAN to generate pairs from one domain where we did have GT to a different domain where we didn't. While it looked promising we ran out of time so we could not train with it:
![CycleGAN Images 1](http://i.imgur.com/vtgP8aL.jpg)
![CycleGAN Images 2](http://i.imgur.com/KWPwzA4.jpg)
(left images are originals, right images are images generated using CycleGAN)
Systematic approach to augmentations. With so many gauges to dial, in some instances we added a few augmentations or changes at one and we were not sure which piece was contributing positively.
Big thanks to all the organizers and participants!
Ciara, Ian, Javi and Andres