# UNet Nuke Solution (#68 LB)

Some people asked for our solution and code, which surprises me a bit given there are so many people in front of us, some with way better scores than us.  Watch for their write up as you'll learn more than from this one.
First of all, let me thank my team mates, al.exe, Sven Hafeneger, Nicole Finnie, and Liam Finnie. I enjoyed the last 3 or 4 days of the competition when we tried to combine our approaches like crazy.  I also thank those who shared so much, Heng Cher Keng, Allen Goodman, Anne Carpenter, Kjetil Åmdal-Sævik who kickstarted us with his kernel, and many others.  Sorry I can't name you all.
Last but not least, I thank Kaggle, Booz Allen Hamilton, and the organizers for setting this competition.  Sure, there was some grumbling about stage 2 rules, but all in all the competition went pretty well.
I started this competition to learn about deep learning and images.  After reading few papers it was clear that there were 2 avenues for this: UNet or Mask-rcnn.  I selected the former because it was way simpler, and also because Kjetil shared an amazing starter kernel.   My final submission is a modification of his kernel.
After 3 weeks or so I was in top 50 on the public LB with a 0.457 score, to my surprise.  I interrupted my work on this to enter another competition (Toxic comment) and had a bad experience there as I got a gold medal then was removed because a team mate cheated.  This is probably why I bugged many here with my post on 'cheating'.  Net result however is that I lost motivation and did not resume work on DSB till I met some colleagues 4 days before competition end.  They also had entered DSB and had worked on train time augmentation quite a bit.  We decided to team, and spent 3 days cross pollinating each other.
Let me now describe what I did, Nicole Finnie, one of my team mates, described their side of the house
[in the comments below](https://www.kaggle.com/c/data-science-bowl-2018/discussion/54742#315594)
.
Preprocessing.
Intensity is rescaled (put min to 0 and max to 255) and transform all images into gray images.  Then I used Otsu thresholding, and inverted the intensity of images where the majority of pixels were above the threshold.  The goal was to get images as similar as possible.  I also did not use any clustering, thinking that training a single model would lead to better generalization.  I now think that clustering with a small number of clusters, as my colleagues were doing, is better.
Next step in preprocessing is to feed the model. I basically resized images to fit the CNN input size (256x256).  This works quite fine except for images with a large aspect ratio, as it distorts the shape of nuclei.  I decided to split images in 2 overlapping pieces that are as close as possible to squares.  A square image yields 2 copies of it, and a very thin rectangle leads to two, less thin, rectangles.  This led to less distortion.
Issue with splitting images was that the boundary of predictions for the overlapping part was a bit wrong.  I think it is because I used padding='same' instead of something more in line with the original UNet paper.
I briefly tried using 256x256 crops of the images as an overlapping tiling , but it did not work as well.  I must have been doing something wrong here because it worked fine for my team mates.
Model
It is very close to Kjetil model.  I only modified the output layer and the loss function, and I added a Speckle noise layer on the input in some runs, thanks to a post form Heng.  Here is the code for a Speckle noise layer.  it is a multiplicative Gaussian noise.  I merely modified the code of the built in Keras additive Gaussian noise.
from keras.engine import Layer
from keras import backend as K

class SpeckleNoise(Layer):

    def __init__(self, stddev, **kwargs):
        super(SpeckleNoise, self).__init__(**kwargs)
        self.supports_masking = True
        self.stddev = stddev

    def call(self, inputs, training=None):
        def noised():
            return K.clip(inputs * K.random_normal(shape=K.shape(inputs),
                                            mean=1.,
                                            stddev=self.stddev), 0.0, 1.0)
        return K.in_train_phase(noised, inputs, training=training)

    def get_config(self):
        config = {'stddev': self.stddev}
        base_config = super(SpeckleNoise, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

    def compute_output_shape(self, input_shape):
        return input_shape
I modified the output layer to predict 3 channels with sigmoid activation.  First channel predicts common boundary between adjacent nuclei, second channel predicts the union of nuclei, and third channel predicts background.  The second channel is the same as in the original UNet model.  In order to construct the ground truth for training, all masks are expanded by one pixel, and the union of these intersection is my first channel target.  Second channel is the union of all masks, and last one is the complement of the other two.  It is important to notice that the first two channels are not mutually exclusive, a pixel can be both on common boundary and in a mask.
The loss function is a pixelwise weighted loss function. I used weights of 30, 3 and 1 for each channel.  Weights were selected via cross validation.
import tensorflow as tf

def pixelwise_crossentropy(target, output):
    _epsilon = 10e-8
    output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)
    weight = 30 * target[:,:,:,0:1] + 3 * target[:,:,:,1:2] + 1 * target[:,:,:,2:3]
    return - tf.reduce_sum(target * weight *  tf.log(output) +
                           (1 - target)  *  tf.log(1 - output),
                           len(output.get_shape()) - 1)
Given the large weights I sometimes got exploding gradients (loss function grows to unreasonable levels).  The way I fixed it was to clip gradient norm in the Adam optimizer.  Maybe a better way would have to use SGD with proper settings, but I did not have time to try it.  Adam is just working out of the box reasonable well.
Training
I used 6 fold cross validation, with a limit of 20 epochs, and early stopping with a patience of 5.  Each of the 6 fold model is used to make predictions on the test data.  These predictions are averaged before post processing test images.  I shuffled images but made sure that all images (or crops) coming from one original image were in the same fold in order to avoid overfitting.  Training time augmentation was limited to flipping and 90° rotations.
Postprocessing
First, images or crops coming from the same original images are merged, and rescaled to the original image size.  Then the problem is how to segment the output as usual with UNet.
I started with a watershed, like many, and explored different ways to seed it with good markers.  Best results were obtained by defining markers as areas with a high probability to be in a mask (Second channel probability over 0.75), and far from being a common boundary (probability of first channel below 0.6).    That use of the common boundary prediction was key in separating adjacent nuclei.  Then watershed was expanded into the region where background probability was low (third channel probability below 0.4).
All thresholds were selected by cross validation.
Looking at results, I noticed that in some cases watershed was producing disconnected masks. I therefore tried another way to expand markers: random walker in scikit-image.  And this was really better than watershed.  When my team mates switched from watershed to random walker they also noticed a significant improvement. In some cases random walker was triggering an error.  In such case I used watershed.
Finally, I tried some ensembling.  I averaged the NN output over several variants (with or without noise, with various gradient clipping) then post processed it as before. This yields a CV improvement but didn't improve the Private LB score to my surprise.
Things I didn't have time to try
Pretrained weights. I wanted to reuse pretrained weights as the NN I use basically contains a VGG-11 model.  Reusing pretrained weights and make these layers not learnable would help fight overfitting. I wanted to reuse
[TernausNet](https://github.com/ternaus/TernausNet)
, but it was in Pytorch, and I did not had time to convert it to TF or Keras.
Mosaic.  Many images were obtained by splitting larger images in 4.  Working with the original images would help predict the nuclei that are split among two or more split images.  Fortunately, my team mates implemented it after we merged, and this proved to be very useful
More training time augmentation.  Color shift, blur, were on my plate.  Fortunately, my team mates did a lot in that area.
Combine with Mask Rcnn.  Thanks to Heng I tried using his output as markers for my postprocessing.  It improved LB score by nearly 0.001.  I wanted to include a mask rcnn output as input to my modified UNet, but did not had time.
Things I did not think of
Shrinking masks so that there are no adjacent asks, then expand predicted nuclei.  This is a brilliant idea shared by Ildoo Kim.  It makes UNet postprocessing way easier.
Test time augmentation (ensembling).  I did not find a way to merge my team mates predictions with mine, and I am eager to know what others did in that area.
I'm sure Ill get many more from top performers write ups!
Take away
I learned a lot, and I now see there is a lot I have to learn still.  One thing I had a hunch for, and that was confirmed here, is the importance of pre and post processing.  Many proponents of deep learning say that there is no need to preprocess data.  Well, in my case, stage1 score went from 0.263 to 0.457 via pre and post processing alone.
Hope the above is of interest to some.
I shared the code that led to my best stage 1 submission
[on github](https://github.com/jfpuget/DSB_2018)
.  My team mate shared the final submission code on
[github](https://github.com/nicolefinnie/kaggle-dsb2018)
too.