{ "id": 54742, "title": "UNet Nuke Solution (#68 LB)", "url": "/competitions/data-science-bowl-2018/discussion/54742", "author_name": "CPMP", "post_date": "2018-04-17T08:52:09.443000", "votes": 51, "comment_count": 20, "views": 0, "content": "

Some people asked for our solution and code, which surprises me a bit given there are so many people in front of us, some with way better scores than us. Watch for their write up as you'll learn more than from this one.

\n\n

First of all, let me thank my team mates, al.exe, Sven Hafeneger, Nicole Finnie, and Liam Finnie. I enjoyed the last 3 or 4 days of the competition when we tried to combine our approaches like crazy. I also thank those who shared so much, Heng Cher Keng, Allen Goodman, Anne Carpenter, Kjetil Åmdal-Sævik who kickstarted us with his kernel, and many others. Sorry I can't name you all.

\n\n

Last but not least, I thank Kaggle, Booz Allen Hamilton, and the organizers for setting this competition. Sure, there was some grumbling about stage 2 rules, but all in all the competition went pretty well.

\n\n

I started this competition to learn about deep learning and images. After reading few papers it was clear that there were 2 avenues for this: UNet or Mask-rcnn. I selected the former because it was way simpler, and also because Kjetil shared an amazing starter kernel. My final submission is a modification of his kernel.

\n\n

After 3 weeks or so I was in top 50 on the public LB with a 0.457 score, to my surprise. I interrupted my work on this to enter another competition (Toxic comment) and had a bad experience there as I got a gold medal then was removed because a team mate cheated. This is probably why I bugged many here with my post on 'cheating'. Net result however is that I lost motivation and did not resume work on DSB till I met some colleagues 4 days before competition end. They also had entered DSB and had worked on train time augmentation quite a bit. We decided to team, and spent 3 days cross pollinating each other.

\n\n

Let me now describe what I did, Nicole Finnie, one of my team mates, described their side of the house in the comments below .

\n\n

Preprocessing.

\n\n

Intensity is rescaled (put min to 0 and max to 255) and transform all images into gray images. Then I used Otsu thresholding, and inverted the intensity of images where the majority of pixels were above the threshold. The goal was to get images as similar as possible. I also did not use any clustering, thinking that training a single model would lead to better generalization. I now think that clustering with a small number of clusters, as my colleagues were doing, is better.

\n\n

Next step in preprocessing is to feed the model. I basically resized images to fit the CNN input size (256x256). This works quite fine except for images with a large aspect ratio, as it distorts the shape of nuclei. I decided to split images in 2 overlapping pieces that are as close as possible to squares. A square image yields 2 copies of it, and a very thin rectangle leads to two, less thin, rectangles. This led to less distortion.

\n\n

Issue with splitting images was that the boundary of predictions for the overlapping part was a bit wrong. I think it is because I used padding='same' instead of something more in line with the original UNet paper.

\n\n

I briefly tried using 256x256 crops of the images as an overlapping tiling , but it did not work as well. I must have been doing something wrong here because it worked fine for my team mates.

\n\n

Model

\n\n

It is very close to Kjetil model. I only modified the output layer and the loss function, and I added a Speckle noise layer on the input in some runs, thanks to a post form Heng. Here is the code for a Speckle noise layer. it is a multiplicative Gaussian noise. I merely modified the code of the built in Keras additive Gaussian noise.

\n\n
from keras.engine import Layer\nfrom keras import backend as K\n\nclass SpeckleNoise(Layer):\n\n    def __init__(self, stddev, **kwargs):\n        super(SpeckleNoise, self).__init__(**kwargs)\n        self.supports_masking = True\n        self.stddev = stddev\n\n    def call(self, inputs, training=None):\n        def noised():\n            return K.clip(inputs * K.random_normal(shape=K.shape(inputs),\n                                            mean=1.,\n                                            stddev=self.stddev), 0.0, 1.0)\n        return K.in_train_phase(noised, inputs, training=training)\n\n    def get_config(self):\n        config = {'stddev': self.stddev}\n        base_config = super(SpeckleNoise, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n    def compute_output_shape(self, input_shape):\n        return input_shape\n
\n\n

I modified the output layer to predict 3 channels with sigmoid activation. First channel predicts common boundary between adjacent nuclei, second channel predicts the union of nuclei, and third channel predicts background. The second channel is the same as in the original UNet model. In order to construct the ground truth for training, all masks are expanded by one pixel, and the union of these intersection is my first channel target. Second channel is the union of all masks, and last one is the complement of the other two. It is important to notice that the first two channels are not mutually exclusive, a pixel can be both on common boundary and in a mask.

\n\n

The loss function is a pixelwise weighted loss function. I used weights of 30, 3 and 1 for each channel. Weights were selected via cross validation.

\n\n
import tensorflow as tf\n\ndef pixelwise_crossentropy(target, output):\n    _epsilon = 10e-8\n    output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)\n    weight = 30 * target[:,:,:,0:1] + 3 * target[:,:,:,1:2] + 1 * target[:,:,:,2:3]\n    return - tf.reduce_sum(target * weight *  tf.log(output) +\n                           (1 - target)  *  tf.log(1 - output),\n                           len(output.get_shape()) - 1)\n
\n\n

Given the large weights I sometimes got exploding gradients (loss function grows to unreasonable levels). The way I fixed it was to clip gradient norm in the Adam optimizer. Maybe a better way would have to use SGD with proper settings, but I did not have time to try it. Adam is just working out of the box reasonable well.

\n\n

Training

\n\n

I used 6 fold cross validation, with a limit of 20 epochs, and early stopping with a patience of 5. Each of the 6 fold model is used to make predictions on the test data. These predictions are averaged before post processing test images. I shuffled images but made sure that all images (or crops) coming from one original image were in the same fold in order to avoid overfitting. Training time augmentation was limited to flipping and 90° rotations.

\n\n

Postprocessing

\n\n

First, images or crops coming from the same original images are merged, and rescaled to the original image size. Then the problem is how to segment the output as usual with UNet.

\n\n

I started with a watershed, like many, and explored different ways to seed it with good markers. Best results were obtained by defining markers as areas with a high probability to be in a mask (Second channel probability over 0.75), and far from being a common boundary (probability of first channel below 0.6). That use of the common boundary prediction was key in separating adjacent nuclei. Then watershed was expanded into the region where background probability was low (third channel probability below 0.4).

\n\n

All thresholds were selected by cross validation.

\n\n

Looking at results, I noticed that in some cases watershed was producing disconnected masks. I therefore tried another way to expand markers: random walker in scikit-image. And this was really better than watershed. When my team mates switched from watershed to random walker they also noticed a significant improvement. In some cases random walker was triggering an error. In such case I used watershed.

\n\n

Finally, I tried some ensembling. I averaged the NN output over several variants (with or without noise, with various gradient clipping) then post processed it as before. This yields a CV improvement but didn't improve the Private LB score to my surprise.

\n\n

Things I didn't have time to try

\n\n

Pretrained weights. I wanted to reuse pretrained weights as the NN I use basically contains a VGG-11 model. Reusing pretrained weights and make these layers not learnable would help fight overfitting. I wanted to reuse TernausNet, but it was in Pytorch, and I did not had time to convert it to TF or Keras.

\n\n

Mosaic. Many images were obtained by splitting larger images in 4. Working with the original images would help predict the nuclei that are split among two or more split images. Fortunately, my team mates implemented it after we merged, and this proved to be very useful

\n\n

More training time augmentation. Color shift, blur, were on my plate. Fortunately, my team mates did a lot in that area.

\n\n

Combine with Mask Rcnn. Thanks to Heng I tried using his output as markers for my postprocessing. It improved LB score by nearly 0.001. I wanted to include a mask rcnn output as input to my modified UNet, but did not had time.

\n\n

Things I did not think of

\n\n

Shrinking masks so that there are no adjacent asks, then expand predicted nuclei. This is a brilliant idea shared by Ildoo Kim. It makes UNet postprocessing way easier.

\n\n

Test time augmentation (ensembling). I did not find a way to merge my team mates predictions with mine, and I am eager to know what others did in that area.

\n\n

I'm sure Ill get many more from top performers write ups!

\n\n

Take away

\n\n

I learned a lot, and I now see there is a lot I have to learn still. One thing I had a hunch for, and that was confirmed here, is the importance of pre and post processing. Many proponents of deep learning say that there is no need to preprocess data. Well, in my case, stage1 score went from 0.263 to 0.457 via pre and post processing alone.

\n\n

Hope the above is of interest to some.

\n\n

I shared the code that led to my best stage 1 submission on github. My team mate shared the final submission code on github too.

", "messages": [ { "id": 315538, "postDate": "2018-04-17T08:52:09.443Z", "content": "

Some people asked for our solution and code, which surprises me a bit given there are so many people in front of us, some with way better scores than us. Watch for their write up as you'll learn more than from this one.

\n\n

First of all, let me thank my team mates, al.exe, Sven Hafeneger, Nicole Finnie, and Liam Finnie. I enjoyed the last 3 or 4 days of the competition when we tried to combine our approaches like crazy. I also thank those who shared so much, Heng Cher Keng, Allen Goodman, Anne Carpenter, Kjetil Åmdal-Sævik who kickstarted us with his kernel, and many others. Sorry I can't name you all.

\n\n

Last but not least, I thank Kaggle, Booz Allen Hamilton, and the organizers for setting this competition. Sure, there was some grumbling about stage 2 rules, but all in all the competition went pretty well.

\n\n

I started this competition to learn about deep learning and images. After reading few papers it was clear that there were 2 avenues for this: UNet or Mask-rcnn. I selected the former because it was way simpler, and also because Kjetil shared an amazing starter kernel. My final submission is a modification of his kernel.

\n\n

After 3 weeks or so I was in top 50 on the public LB with a 0.457 score, to my surprise. I interrupted my work on this to enter another competition (Toxic comment) and had a bad experience there as I got a gold medal then was removed because a team mate cheated. This is probably why I bugged many here with my post on 'cheating'. Net result however is that I lost motivation and did not resume work on DSB till I met some colleagues 4 days before competition end. They also had entered DSB and had worked on train time augmentation quite a bit. We decided to team, and spent 3 days cross pollinating each other.

\n\n

Let me now describe what I did, Nicole Finnie, one of my team mates, described their side of the house in the comments below .

\n\n

Preprocessing.

\n\n

Intensity is rescaled (put min to 0 and max to 255) and transform all images into gray images. Then I used Otsu thresholding, and inverted the intensity of images where the majority of pixels were above the threshold. The goal was to get images as similar as possible. I also did not use any clustering, thinking that training a single model would lead to better generalization. I now think that clustering with a small number of clusters, as my colleagues were doing, is better.

\n\n

Next step in preprocessing is to feed the model. I basically resized images to fit the CNN input size (256x256). This works quite fine except for images with a large aspect ratio, as it distorts the shape of nuclei. I decided to split images in 2 overlapping pieces that are as close as possible to squares. A square image yields 2 copies of it, and a very thin rectangle leads to two, less thin, rectangles. This led to less distortion.

\n\n

Issue with splitting images was that the boundary of predictions for the overlapping part was a bit wrong. I think it is because I used padding='same' instead of something more in line with the original UNet paper.

\n\n

I briefly tried using 256x256 crops of the images as an overlapping tiling , but it did not work as well. I must have been doing something wrong here because it worked fine for my team mates.

\n\n

Model

\n\n

It is very close to Kjetil model. I only modified the output layer and the loss function, and I added a Speckle noise layer on the input in some runs, thanks to a post form Heng. Here is the code for a Speckle noise layer. it is a multiplicative Gaussian noise. I merely modified the code of the built in Keras additive Gaussian noise.

\n\n
from keras.engine import Layer\nfrom keras import backend as K\n\nclass SpeckleNoise(Layer):\n\n    def __init__(self, stddev, **kwargs):\n        super(SpeckleNoise, self).__init__(**kwargs)\n        self.supports_masking = True\n        self.stddev = stddev\n\n    def call(self, inputs, training=None):\n        def noised():\n            return K.clip(inputs * K.random_normal(shape=K.shape(inputs),\n                                            mean=1.,\n                                            stddev=self.stddev), 0.0, 1.0)\n        return K.in_train_phase(noised, inputs, training=training)\n\n    def get_config(self):\n        config = {'stddev': self.stddev}\n        base_config = super(SpeckleNoise, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n    def compute_output_shape(self, input_shape):\n        return input_shape\n
\n\n

I modified the output layer to predict 3 channels with sigmoid activation. First channel predicts common boundary between adjacent nuclei, second channel predicts the union of nuclei, and third channel predicts background. The second channel is the same as in the original UNet model. In order to construct the ground truth for training, all masks are expanded by one pixel, and the union of these intersection is my first channel target. Second channel is the union of all masks, and last one is the complement of the other two. It is important to notice that the first two channels are not mutually exclusive, a pixel can be both on common boundary and in a mask.

\n\n

The loss function is a pixelwise weighted loss function. I used weights of 30, 3 and 1 for each channel. Weights were selected via cross validation.

\n\n
import tensorflow as tf\n\ndef pixelwise_crossentropy(target, output):\n    _epsilon = 10e-8\n    output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)\n    weight = 30 * target[:,:,:,0:1] + 3 * target[:,:,:,1:2] + 1 * target[:,:,:,2:3]\n    return - tf.reduce_sum(target * weight *  tf.log(output) +\n                           (1 - target)  *  tf.log(1 - output),\n                           len(output.get_shape()) - 1)\n
\n\n

Given the large weights I sometimes got exploding gradients (loss function grows to unreasonable levels). The way I fixed it was to clip gradient norm in the Adam optimizer. Maybe a better way would have to use SGD with proper settings, but I did not have time to try it. Adam is just working out of the box reasonable well.

\n\n

Training

\n\n

I used 6 fold cross validation, with a limit of 20 epochs, and early stopping with a patience of 5. Each of the 6 fold model is used to make predictions on the test data. These predictions are averaged before post processing test images. I shuffled images but made sure that all images (or crops) coming from one original image were in the same fold in order to avoid overfitting. Training time augmentation was limited to flipping and 90° rotations.

\n\n

Postprocessing

\n\n

First, images or crops coming from the same original images are merged, and rescaled to the original image size. Then the problem is how to segment the output as usual with UNet.

\n\n

I started with a watershed, like many, and explored different ways to seed it with good markers. Best results were obtained by defining markers as areas with a high probability to be in a mask (Second channel probability over 0.75), and far from being a common boundary (probability of first channel below 0.6). That use of the common boundary prediction was key in separating adjacent nuclei. Then watershed was expanded into the region where background probability was low (third channel probability below 0.4).

\n\n

All thresholds were selected by cross validation.

\n\n

Looking at results, I noticed that in some cases watershed was producing disconnected masks. I therefore tried another way to expand markers: random walker in scikit-image. And this was really better than watershed. When my team mates switched from watershed to random walker they also noticed a significant improvement. In some cases random walker was triggering an error. In such case I used watershed.

\n\n

Finally, I tried some ensembling. I averaged the NN output over several variants (with or without noise, with various gradient clipping) then post processed it as before. This yields a CV improvement but didn't improve the Private LB score to my surprise.

\n\n

Things I didn't have time to try

\n\n

Pretrained weights. I wanted to reuse pretrained weights as the NN I use basically contains a VGG-11 model. Reusing pretrained weights and make these layers not learnable would help fight overfitting. I wanted to reuse TernausNet, but it was in Pytorch, and I did not had time to convert it to TF or Keras.

\n\n

Mosaic. Many images were obtained by splitting larger images in 4. Working with the original images would help predict the nuclei that are split among two or more split images. Fortunately, my team mates implemented it after we merged, and this proved to be very useful

\n\n

More training time augmentation. Color shift, blur, were on my plate. Fortunately, my team mates did a lot in that area.

\n\n

Combine with Mask Rcnn. Thanks to Heng I tried using his output as markers for my postprocessing. It improved LB score by nearly 0.001. I wanted to include a mask rcnn output as input to my modified UNet, but did not had time.

\n\n

Things I did not think of

\n\n

Shrinking masks so that there are no adjacent asks, then expand predicted nuclei. This is a brilliant idea shared by Ildoo Kim. It makes UNet postprocessing way easier.

\n\n

Test time augmentation (ensembling). I did not find a way to merge my team mates predictions with mine, and I am eager to know what others did in that area.

\n\n

I'm sure Ill get many more from top performers write ups!

\n\n

Take away

\n\n

I learned a lot, and I now see there is a lot I have to learn still. One thing I had a hunch for, and that was confirmed here, is the importance of pre and post processing. Many proponents of deep learning say that there is no need to preprocess data. Well, in my case, stage1 score went from 0.263 to 0.457 via pre and post processing alone.

\n\n

Hope the above is of interest to some.

\n\n

I shared the code that led to my best stage 1 submission on github. My team mate shared the final submission code on github too.

", "rawMarkdown": "Some people asked for our solution and code, which surprises me a bit given there are so many people in front of us, some with way better scores than us. Watch for their write up as you'll learn more than from this one.\n\nFirst of all, let me thank my team mates, al.exe, Sven Hafeneger, Nicole Finnie, and Liam Finnie. I enjoyed the last 3 or 4 days of the competition when we tried to combine our approaches like crazy. I also thank those who shared so much, Heng Cher Keng, Allen Goodman, Anne Carpenter, Kjetil Åmdal-Sævik who kickstarted us with his kernel, and many others. Sorry I can't name you all. \n\nLast but not least, I thank Kaggle, Booz Allen Hamilton, and the organizers for setting this competition. Sure, there was some grumbling about stage 2 rules, but all in all the competition went pretty well. \n\nI started this competition to learn about deep learning and images. After reading few papers it was clear that there were 2 avenues for this: UNet or Mask-rcnn. I selected the former because it was way simpler, and also because Kjetil shared an amazing starter kernel. My final submission is a modification of his kernel. \n\nAfter 3 weeks or so I was in top 50 on the public LB with a 0.457 score, to my surprise. I interrupted my work on this to enter another competition (Toxic comment) and had a bad experience there as I got a gold medal then was removed because a team mate cheated. This is probably why I bugged many here with my post on 'cheating'. Net result however is that I lost motivation and did not resume work on DSB till I met some colleagues 4 days before competition end. They also had entered DSB and had worked on train time augmentation quite a bit. We decided to team, and spent 3 days cross pollinating each other.\n\nLet me now describe what I did, Nicole Finnie, one of my team mates, described their side of the house [in the comments below][1] . \n\n**Preprocessing.**\n\nIntensity is rescaled (put min to 0 and max to 255) and transform all images into gray images. Then I used Otsu thresholding, and inverted the intensity of images where the majority of pixels were above the threshold. The goal was to get images as similar as possible. I also did not use any clustering, thinking that training a single model would lead to better generalization. I now think that clustering with a small number of clusters, as my colleagues were doing, is better.\n\nNext step in preprocessing is to feed the model. I basically resized images to fit the CNN input size (256x256). This works quite fine except for images with a large aspect ratio, as it distorts the shape of nuclei. I decided to split images in 2 overlapping pieces that are as close as possible to squares. A square image yields 2 copies of it, and a very thin rectangle leads to two, less thin, rectangles. This led to less distortion.\n\nIssue with splitting images was that the boundary of predictions for the overlapping part was a bit wrong. I think it is because I used padding='same' instead of something more in line with the original UNet paper.\n\nI briefly tried using 256x256 crops of the images as an overlapping tiling , but it did not work as well. I must have been doing something wrong here because it worked fine for my team mates.\n\n**Model**\n\nIt is very close to Kjetil model. I only modified the output layer and the loss function, and I added a Speckle noise layer on the input in some runs, thanks to a post form Heng. Here is the code for a Speckle noise layer. it is a multiplicative Gaussian noise. I merely modified the code of the built in Keras additive Gaussian noise.\n\n from keras.engine import Layer\n from keras import backend as K\n \n class SpeckleNoise(Layer):\n \n def __init__(self, stddev, **kwargs):\n super(SpeckleNoise, self).__init__(**kwargs)\n self.supports_masking = True\n self.stddev = stddev\n \n def call(self, inputs, training=None):\n def noised():\n return K.clip(inputs * K.random_normal(shape=K.shape(inputs),\n mean=1.,\n stddev=self.stddev), 0.0, 1.0)\n return K.in_train_phase(noised, inputs, training=training)\n \n def get_config(self):\n config = {'stddev': self.stddev}\n base_config = super(SpeckleNoise, self).get_config()\n return dict(list(base_config.items()) + list(config.items()))\n \n def compute_output_shape(self, input_shape):\n return input_shape\n\n\nI modified the output layer to predict 3 channels with sigmoid activation. First channel predicts common boundary between adjacent nuclei, second channel predicts the union of nuclei, and third channel predicts background. The second channel is the same as in the original UNet model. In order to construct the ground truth for training, all masks are expanded by one pixel, and the union of these intersection is my first channel target. Second channel is the union of all masks, and last one is the complement of the other two. It is important to notice that the first two channels are not mutually exclusive, a pixel can be both on common boundary and in a mask.\n\nThe loss function is a pixelwise weighted loss function. I used weights of 30, 3 and 1 for each channel. Weights were selected via cross validation.\n\n import tensorflow as tf\n \n def pixelwise_crossentropy(target, output):\n _epsilon = 10e-8\n output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)\n weight = 30 * target[:,:,:,0:1] + 3 * target[:,:,:,1:2] + 1 * target[:,:,:,2:3]\n return - tf.reduce_sum(target * weight * tf.log(output) +\n (1 - target) * tf.log(1 - output),\n len(output.get_shape()) - 1)\n\nGiven the large weights I sometimes got exploding gradients (loss function grows to unreasonable levels). The way I fixed it was to clip gradient norm in the Adam optimizer. Maybe a better way would have to use SGD with proper settings, but I did not have time to try it. Adam is just working out of the box reasonable well.\n\n**Training**\n\nI used 6 fold cross validation, with a limit of 20 epochs, and early stopping with a patience of 5. Each of the 6 fold model is used to make predictions on the test data. These predictions are averaged before post processing test images. I shuffled images but made sure that all images (or crops) coming from one original image were in the same fold in order to avoid overfitting. Training time augmentation was limited to flipping and 90° rotations.\n\n**Postprocessing**\n\nFirst, images or crops coming from the same original images are merged, and rescaled to the original image size. Then the problem is how to segment the output as usual with UNet.\n\nI started with a watershed, like many, and explored different ways to seed it with good markers. Best results were obtained by defining markers as areas with a high probability to be in a mask (Second channel probability over 0.75), and far from being a common boundary (probability of first channel below 0.6). That use of the common boundary prediction was key in separating adjacent nuclei. Then watershed was expanded into the region where background probability was low (third channel probability below 0.4).\n\nAll thresholds were selected by cross validation.\n\nLooking at results, I noticed that in some cases watershed was producing disconnected masks. I therefore tried another way to expand markers: random walker in scikit-image. And this was really better than watershed. When my team mates switched from watershed to random walker they also noticed a significant improvement. In some cases random walker was triggering an error. In such case I used watershed.\n\nFinally, I tried some ensembling. I averaged the NN output over several variants (with or without noise, with various gradient clipping) then post processed it as before. This yields a CV improvement but didn't improve the Private LB score to my surprise.\n\n**Things I didn't have time to try**\n\nPretrained weights. I wanted to reuse pretrained weights as the NN I use basically contains a VGG-11 model. Reusing pretrained weights and make these layers not learnable would help fight overfitting. I wanted to reuse [TernausNet][2], but it was in Pytorch, and I did not had time to convert it to TF or Keras.\n\nMosaic. Many images were obtained by splitting larger images in 4. Working with the original images would help predict the nuclei that are split among two or more split images. Fortunately, my team mates implemented it after we merged, and this proved to be very useful\n\nMore training time augmentation. Color shift, blur, were on my plate. Fortunately, my team mates did a lot in that area.\n\nCombine with Mask Rcnn. Thanks to Heng I tried using his output as markers for my postprocessing. It improved LB score by nearly 0.001. I wanted to include a mask rcnn output as input to my modified UNet, but did not had time.\n\n**Things I did not think of**\n\nShrinking masks so that there are no adjacent asks, then expand predicted nuclei. This is a brilliant idea shared by Ildoo Kim. It makes UNet postprocessing way easier.\n\nTest time augmentation (ensembling). I did not find a way to merge my team mates predictions with mine, and I am eager to know what others did in that area.\n\nI'm sure Ill get many more from top performers write ups!\n\n**Take away**\n\nI learned a lot, and I now see there is a lot I have to learn still. One thing I had a hunch for, and that was confirmed here, is the importance of pre and post processing. Many proponents of deep learning say that there is no need to preprocess data. Well, in my case, stage1 score went from 0.263 to 0.457 via pre and post processing alone.\n\nHope the above is of interest to some.\n\nI shared the code that led to my best stage 1 submission [on github][3]. My team mate shared the final submission code on [github][4] too.\n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54742#315594\n [2]: https://github.com/ternaus/TernausNet\n [3]: https://github.com/jfpuget/DSB_2018\n [4]: https://github.com/nicolefinnie/kaggle-dsb2018", "votes": 51 }, { "id": 315594, "postDate": "2018-04-17T10:29:21.180Z", "content": "

https://github.com/nicolefinnie/kaggle-dsb2018

\n\n

Newbies' solution - as first time Kagglers (LB = 0.545)

\n\n

I'm @CPMP's team mate + colleague. Since most of us were first time Kagglers, we implemented a very simple U-net, taken from the U-net starter of @Kjetil Åmdal-Sævik that any DL newbies like us can do.

\n\n

Model

\n\n\n\n

Pre-processing

\n\n\n\n

K-mean cluster based on colours

\n\n\n\n

\"Allen's

\n\n

Mosaic for training and predicting

\n\n\n\n

Windowing images for training and predictions

\n\n\n\n

Data augmentation

\n\n\n\n

Average windowing predictions

\n\n\n\n

Post-processing

\n\n\n\n

Model ensemble using weighted majority voting

\n\n\n\n

Acknowledgement

\n\n

Thanks to @CPMP. If he didn't join our team in the last 3 days, we wouldn't have known that we had to train and submit models by the deadline of the first stage and we wouldn't have submitted any models that we could have used at the 2nd stage. Sadly, we didn't get to train all the models with data augmentation we planned to, however, we're happy with the results as first time Kagglers. It's pretty easy to implement our approach.

", "rawMarkdown": "https://github.com/nicolefinnie/kaggle-dsb2018\n\n**Newbies' solution - as first time Kagglers (LB = 0.545)**\n\nI'm @CPMP's team mate + colleague. Since most of us were first time Kagglers, we implemented a very simple U-net, taken from the U-net starter of @Kjetil Åmdal-Sævik that any DL newbies like us can do.\n\n**Model**\n\n- Same as the U-net starter except two output channels: `masks` and `contours` with the `binary cross entropy loss`\n\n**Pre-processing**\n \n - CLAHE (Contrast Limited Adaptive Histogram Equalization) \n - L-channel only\n - Inversion only when the background is lighter than the foreground\n - Reference: https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence\n - The goal is to make it easy for the U-net to learn their common features.\n\n\n**K-mean cluster based on colours**\n\n - K-mean cluster using `R, G, R-G deviation` on train/test images. As @Heng suggested, if we train grey and colour images together, their common trait is the shape/boundary, so it'd be hard for the neural net to learn the features inside the nucleus. In a result, our U-net did fairly well on the colour images at the 2nd stage, okay, maybe not too well on @Allen's art :)\n\n![Allen's art](https://kaggle2.blob.core.windows.net/forum-message-attachments/315594/9206/AllensArt.png)\n\n**Mosaic for training and predicting**\n\n- As suggested by @CPMP, we applied @Emil's mosaic idea (KNN on 4 edges of the image to find the matching neighbours) and merged all images to mosaic to gain the information on the edges \nhttps://www.kaggle.com/bonlime/train-test-image-mosaic\n![A mosaic trained image composed of 4 images](https://kaggle2.blob.core.windows.net/forum-message-attachments/315594/9207/mosaic_train_colour1.png)\n\n- Around `37%` of the grey test images and about `77%` of the colour test images can be formed into mosaic, this ratio still holds true for the stage 2 test images. That means we gained more hidden traits than most teams.\n\n**Windowing images for training and predictions**\n\n- Thanks to mosaic, now we have a lot more information on the edges, so we crop images to `256x256` using overlapping windows with a stride of `128`, because `256x256` yields the best mean IoU.\n\n- Mirror the edges and the corners as suggested by the U-net paper. However, we have less edges than the people who don't use the `mosaic` approach\n\n- Rotate and flip all windowing images, this augments images by `8x`\n\n**Data augmentation**\n \n- Transform perspective using the `imgaug` library. (this approach is used by the top 1 team)\n\n- Additive Gaussian noise + speckle noise. (used by the top 1 team)\n\n- Gaussian Blur (no time to train)\n\n- Image pyramid, upsampling and downsampling based on the average nuclei size per image (no time to train )\n\n- Regional gamma correction (no time to train) \n\n- Greyscale (no time to train)\n\n- Inversion, hurts our LB since it makes the U-net difficult to learn, so we discarded the idea just like the top 1 team\n\n**Average windowing predictions**\n\n- Predict on overlapping windowing images in rotated/flipped/mirrored versions\n- Average all windowing predictions to minimize the artifacts, this approach was used by the winners in the satellite competition last year.\n- Stitch predictions together to the original size\n\n\n**Post-processing**\n\n- Otsu threshold on predicted masks and predicted contours\n- Find sure foreground: `(predicted masks - predicted contours)` \n- Find sure background: dilates the predicted masks\n- Find unknown area: `(sure background - sure foreground)`\n- Find labels on the sure foreground \n- Random walker or watershed on the masks + contours to avoid `contour erosion`, adding the thresholding contour gave us a big LB boost. \n\n- Reference: https://docs.opencv.org/3.3.1/d3/db4/tutorial_py_watershed.html\n\n**Model ensemble using weighted majority voting**\n\n- @CPMP chose weights based on our public LB scores\n- `4 * ( baseline model + noise ) + 2 * ( baseline model + transform ) + 1 * ( baseline model)`\n\n\n**Acknowledgement**\n\nThanks to @CPMP. If he didn't join our team in the last 3 days, we wouldn't have known that we had to train and submit models by the deadline of the first stage and we wouldn't have submitted any models that we could have used at the 2nd stage. Sadly, we didn't get to train all the models with data augmentation we planned to, however, we're happy with the results as first time Kagglers. It's pretty easy to implement our approach. ", "votes": 12, "replies": [ { "id": 315607, "postDate": "2018-04-17T10:42:53.113Z", "content": "

Thanks for sharing Nicole. Next time we'll team earlier for sure!

", "rawMarkdown": "Thanks for sharing Nicole. Next time we'll team earlier for sure!", "votes": 1 } ] }, { "id": 315774, "postDate": "2018-04-17T15:24:51.047Z", "content": "

Thanks for sharing! I liked your speckled noise. I'll try to replace the Gaussian layer I used by it instead. Does it get turned on/off for training and evaluating, respectively?

", "rawMarkdown": "Thanks for sharing! I liked your speckled noise. I'll try to replace the Gaussian layer I used by it instead. Does it get turned on/off for training and evaluating, respectively?", "votes": 1, "replies": [ { "id": 315776, "postDate": "2018-04-17T15:27:56.363Z", "content": "

It is only active during training.

", "rawMarkdown": "It is only active during training.", "votes": 1 } ] }, { "id": 315564, "postDate": "2018-04-17T09:24:38.520Z", "content": "

Thanks for sharing!

\n\n

Due to lack of time and experience, I myself used a very simple unet, not so different from what you describe here. And due to lack of time and only moderate computational resources it was also poorly optimized & trained...

\n\n

Good thing: There's at least clearly no lack of ideas that I could try out the next time :)

\n\n

With more time I would mostly have wanted to work more on the initial cropping, because that seemed to matter quite a lot (splitting the image into 256x256 crops gave me better results than resizing). But same as you I had some issues with masks where crops overlapped...\nI would guess that first briefly looking for potential candidates than cropping smaller areas would have been better. I might even have added a spherical decay function towards the edge of the crops to make sure the model focuses more on the center part.

\n\n

And in the end I also became more interested in using autoencoders to augment the data (and maybe also to pre-train the first few layers?). \nHas anyone played around with this?

", "rawMarkdown": "Thanks for sharing!\n\nDue to lack of time and experience, I myself used a very simple unet, not so different from what you describe here. And due to lack of time and only moderate computational resources it was also poorly optimized & trained...\n\nGood thing: There's at least clearly no lack of ideas that I could try out the next time :)\n\nWith more time I would mostly have wanted to work more on the initial cropping, because that seemed to matter quite a lot (splitting the image into 256x256 crops gave me better results than resizing). But same as you I had some issues with masks where crops overlapped...\nI would guess that first briefly looking for potential candidates than cropping smaller areas would have been better. I might even have added a spherical decay function towards the edge of the crops to make sure the model focuses more on the center part.\n\nAnd in the end I also became more interested in using autoencoders to augment the data (and maybe also to pre-train the first few layers?). \nHas anyone played around with this?", "votes": 1, "replies": [ { "id": 315574, "postDate": "2018-04-17T09:39:10.357Z", "content": "

Thanks. I used a decay near the borders of my crops indeed, it helps but did not cure the problem entirely.

", "rawMarkdown": "Thanks. I used a decay near the borders of my crops indeed, it helps but did not cure the problem entirely.", "votes": 1 }, { "id": 315647, "postDate": "2018-04-17T11:55:00.833Z", "content": "

We saw a boost from using crops/sliding windows. Initially, we did not have any overlap, and that caused artifacts in the predicted masks at the edges of the windows. In the end, we used completely overlapping windows (see @Nicole's post above) (256x256 windows, of which 128x128 is always completely overlapping an adjacent window, with symmetric mirroring on the edges). Averaging together all the 128x128 overlapping portions avoided almost all the artifacts, and ended up improving our LB by about 0.01 or so, compared to when we just resized images.

", "rawMarkdown": "We saw a boost from using crops/sliding windows. Initially, we did not have any overlap, and that caused artifacts in the predicted masks at the edges of the windows. In the end, we used completely overlapping windows (see @Nicole's post above) (256x256 windows, of which 128x128 is always completely overlapping an adjacent window, with symmetric mirroring on the edges). Averaging together all the 128x128 overlapping portions avoided almost all the artifacts, and ended up improving our LB by about 0.01 or so, compared to when we just resized images.", "votes": 4 }, { "id": 315879, "postDate": "2018-04-17T18:07:01.747Z", "content": "

@Liam, thanks! I saw a difference of 0.1 to 0.15 when cropping instead of resizing. And mirroring is probably the most straight-forward way to avoid weird edge artefacts (at least much simpler than the decay towards the borders I had in mind). I'm more and more tempted to continue trying out things...

", "rawMarkdown": "@Liam, thanks! I saw a difference of 0.1 to 0.15 when cropping instead of resizing. And mirroring is probably the most straight-forward way to avoid weird edge artefacts (at least much simpler than the decay towards the borders I had in mind). I'm more and more tempted to continue trying out things...", "votes": 2 } ] }, { "id": 321959, "postDate": "2018-05-02T07:58:07.293Z", "content": "

Thank you @CPMP for sharing! I was testing your code to check how the model works, all worked till I got into \"python schwaebische_nuclei.py predict --loadmodel output_from_all_models/ --weights 4 2 1\", it gave me an error \"Traceback (most recent call last):\n File \"schwaebische_nuclei.py\", line 765, in \n util.predict(cluster_split_train_df_list, cluster_val_df_list, cluster_test_df_list)\n File \"schwaebische_nuclei.py\", line 588, in predict\n grey_models = load_models(self.LOAD_MODELS_GREY_PATH, self.GPU)\n File \"C:\\src\\modules\\unet.py\", line 227, in load_models\n (_, _, model_ids) = next(os.walk(model_path))\"\nCan you please specify what exactly on the output_from_all_models folder?

\n\n

Thank you for your help!

", "rawMarkdown": "Thank you @CPMP for sharing! I was testing your code to check how the model works, all worked till I got into \"python schwaebische_nuclei.py predict --loadmodel output_from_all_models/ --weights 4 2 1\", it gave me an error \"Traceback (most recent call last):\n File \"schwaebische_nuclei.py\", line 765, in ", "votes": 2, "replies": [ { "id": 321963, "postDate": "2018-05-02T08:02:28.207Z", "content": "

Hi, this is the code form my team mates, I'll let them have a look.

", "rawMarkdown": "Hi, this is the code form my team mates, I'll let them have a look." }, { "id": 321999, "postDate": "2018-05-02T09:31:21.467Z", "content": "

Hello,

\n\n

When you first run schwaebische_nuclei.py with the 'train' option, it will create a directory named output_. This has all output from the script, including two sub-directories - models_colour (which should contain a single .h5 model file), and models_grey (which should also contain a single .h5 model file). To generate the ensemble predictions with weighted averages, you will have first run the schwaebische_nuclei.py script 3 times with the 'train' option - and each run will have a separate output_ directory. Copy the colour .h5 models from each run into a new directory, say 'output_from_all_models/models_colour', and the grey .h5 models from each run into 'output_from_all_models/models_grey'. So, your resulting directory structure should look something like:

\n\n
output_from_all_models/models_colour/single--2018-04-11-12-34-31.h5\noutput_from_all_models/models_colour/single--2018-04-11-13-34-31.h5\noutput_from_all_models/models_colour/single--2018-04-11-14-34-31.h5\noutput_from_all_models/models_grey/single--2018-04-11-12-54-31.h5\noutput_from_all_models/models_grey/single--2018-04-11-13-54-31.h5\noutput_from_all_models/models_grey/single--2018-04-11-14-54-31.h5\n
\n\n

Then, you can run your command:

\n\n
python schwaebische_nuclei.py predict --loadmodel output_from_all_models/ --weights 4 2 1\n
\n\n

The weights are assigned based on the alphabetical order of the files - so in your command, the 'oldest' model will have a weight of 4.

\n\n

Hope this helps!

", "rawMarkdown": "Hello,\n\nWhen you first run schwaebische_nuclei.py with the 'train' option, it will create a directory named output_", "votes": 2 }, { "id": 322397, "postDate": "2018-05-02T21:18:36.147Z", "content": "

Thank you very much it worked :) I have a question, the output Excel sheet has many duplicates Ids is it normal?

", "rawMarkdown": "Thank you very much it worked :) I have a question, the output Excel sheet has many duplicates Ids is it normal?" }, { "id": 322505, "postDate": "2018-05-03T05:24:19.500Z", "content": "

Yes, that's expected. Each nucleus in the image will have it's run-length-encoded information stored on a separate line.

", "rawMarkdown": "Yes, that's expected. Each nucleus in the image will have it's run-length-encoded information stored on a separate line." } ] }, { "id": 316136, "postDate": "2018-04-18T08:33:34.820Z", "content": "

We shared our code, links at the end of main post above.

", "rawMarkdown": "We shared our code, links at the end of main post above.", "votes": 2 }, { "id": 315548, "postDate": "2018-04-17T09:04:46.870Z", "content": "

Thanks a lot @CPMP. Insightful indeed.\nI have been following your comments on discussion forums, throughout the competition. I need to say, they have been helpful in bringing clarity to things.\nHoping to see you soon in another competition.

", "rawMarkdown": "Thanks a lot @CPMP. Insightful indeed.\nI have been following your comments on discussion forums, throughout the competition. I need to say, they have been helpful in bringing clarity to things.\nHoping to see you soon in another competition.", "votes": 2, "replies": [ { "id": 315575, "postDate": "2018-04-17T09:40:02.177Z", "content": "

Thanks. Sure, now that I've tasted CNNs I'll be back in another image competition soon!

", "rawMarkdown": "Thanks. Sure, now that I've tasted CNNs I'll be back in another image competition soon!" } ] }, { "id": 322573, "postDate": "2018-05-03T08:08:04.150Z", "content": "

Hi @CPMP，I'm reading your code:\nZ_train[4*n + 0, :, :, 0] = (mask > 1)

\n\n

why use \" > 1\" ?

", "rawMarkdown": "Hi @CPMP，I'm reading your code:\nZ_train[4*n + 0, :, :, 0] = (mask > 1)\n\nwhy use \" > 1\" ?", "replies": [ { "id": 322587, "postDate": "2018-05-03T08:40:20.357Z", "content": "

Because I want to find common boundaries, i.e. overlapping areas coming from at least two different masks.

", "rawMarkdown": "Because I want to find common boundaries, i.e. overlapping areas coming from at least two different masks." } ] }, { "id": 315659, "postDate": "2018-04-17T12:34:28.557Z", "content": "

lmdsg

", "rawMarkdown": "lmdsg" }, { "id": 315567, "postDate": "2018-04-17T09:25:13.053Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 315594, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-17T10:29:21.180000", "content": "

https://github.com/nicolefinnie/kaggle-dsb2018

\n\n

Newbies' solution - as first time Kagglers (LB = 0.545)

\n\n

I'm @CPMP's team mate + colleague. Since most of us were first time Kagglers, we implemented a very simple U-net, taken from the U-net starter of @Kjetil Åmdal-Sævik that any DL newbies like us can do.

\n\n

Model

\n\n\n\n

Pre-processing

\n\n\n\n

K-mean cluster based on colours

\n\n\n\n

\"Allen's

\n\n

Mosaic for training and predicting

\n\n\n\n

Windowing images for training and predictions

\n\n\n\n

Data augmentation

\n\n\n\n

Average windowing predictions

\n\n\n\n

Post-processing

\n\n\n\n

Model ensemble using weighted majority voting

\n\n\n\n

Acknowledgement

\n\n

Thanks to @CPMP. If he didn't join our team in the last 3 days, we wouldn't have known that we had to train and submit models by the deadline of the first stage and we wouldn't have submitted any models that we could have used at the 2nd stage. Sadly, we didn't get to train all the models with data augmentation we planned to, however, we're happy with the results as first time Kagglers. It's pretty easy to implement our approach.

", "votes": 12, "replies": [ { "id": 315607, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T10:42:53.113000", "content": "

Thanks for sharing Nicole. Next time we'll team earlier for sure!

", "votes": 1, "replies": [] } ] }, { "id": 315774, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-17T15:24:51.047000", "content": "

Thanks for sharing! I liked your speckled noise. I'll try to replace the Gaussian layer I used by it instead. Does it get turned on/off for training and evaluating, respectively?

", "votes": 1, "replies": [ { "id": 315776, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T15:27:56.363000", "content": "

It is only active during training.

", "votes": 1, "replies": [] } ] }, { "id": 315564, "author_name": "Florian Huber", "author_url": "", "post_date": "2018-04-17T09:24:38.520000", "content": "

Thanks for sharing!

\n\n

Due to lack of time and experience, I myself used a very simple unet, not so different from what you describe here. And due to lack of time and only moderate computational resources it was also poorly optimized & trained...

\n\n

Good thing: There's at least clearly no lack of ideas that I could try out the next time :)

\n\n

With more time I would mostly have wanted to work more on the initial cropping, because that seemed to matter quite a lot (splitting the image into 256x256 crops gave me better results than resizing). But same as you I had some issues with masks where crops overlapped...\nI would guess that first briefly looking for potential candidates than cropping smaller areas would have been better. I might even have added a spherical decay function towards the edge of the crops to make sure the model focuses more on the center part.

\n\n

And in the end I also became more interested in using autoencoders to augment the data (and maybe also to pre-train the first few layers?). \nHas anyone played around with this?

", "votes": 1, "replies": [ { "id": 315574, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T09:39:10.357000", "content": "

Thanks. I used a decay near the borders of my crops indeed, it helps but did not cure the problem entirely.

", "votes": 1, "replies": [] }, { "id": 315647, "author_name": "Liam Finnie", "author_url": "", "post_date": "2018-04-17T11:55:00.833000", "content": "

We saw a boost from using crops/sliding windows. Initially, we did not have any overlap, and that caused artifacts in the predicted masks at the edges of the windows. In the end, we used completely overlapping windows (see @Nicole's post above) (256x256 windows, of which 128x128 is always completely overlapping an adjacent window, with symmetric mirroring on the edges). Averaging together all the 128x128 overlapping portions avoided almost all the artifacts, and ended up improving our LB by about 0.01 or so, compared to when we just resized images.

", "votes": 4, "replies": [] }, { "id": 315879, "author_name": "Florian Huber", "author_url": "", "post_date": "2018-04-17T18:07:01.747000", "content": "

@Liam, thanks! I saw a difference of 0.1 to 0.15 when cropping instead of resizing. And mirroring is probably the most straight-forward way to avoid weird edge artefacts (at least much simpler than the decay towards the borders I had in mind). I'm more and more tempted to continue trying out things...

", "votes": 2, "replies": [] } ] }, { "id": 321959, "author_name": "Wedad Anbtawi", "author_url": "", "post_date": "2018-05-02T07:58:07.293000", "content": "

Thank you @CPMP for sharing! I was testing your code to check how the model works, all worked till I got into \"python schwaebische_nuclei.py predict --loadmodel output_from_all_models/ --weights 4 2 1\", it gave me an error \"Traceback (most recent call last):\n File \"schwaebische_nuclei.py\", line 765, in \n util.predict(cluster_split_train_df_list, cluster_val_df_list, cluster_test_df_list)\n File \"schwaebische_nuclei.py\", line 588, in predict\n grey_models = load_models(self.LOAD_MODELS_GREY_PATH, self.GPU)\n File \"C:\\src\\modules\\unet.py\", line 227, in load_models\n (_, _, model_ids) = next(os.walk(model_path))\"\nCan you please specify what exactly on the output_from_all_models folder?

\n\n

Thank you for your help!

", "votes": 2, "replies": [ { "id": 321963, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-02T08:02:28.207000", "content": "

Hi, this is the code form my team mates, I'll let them have a look.

", "votes": 0, "replies": [] }, { "id": 321999, "author_name": "Liam Finnie", "author_url": "", "post_date": "2018-05-02T09:31:21.467000", "content": "

Hello,

\n\n

When you first run schwaebische_nuclei.py with the 'train' option, it will create a directory named output_. This has all output from the script, including two sub-directories - models_colour (which should contain a single .h5 model file), and models_grey (which should also contain a single .h5 model file). To generate the ensemble predictions with weighted averages, you will have first run the schwaebische_nuclei.py script 3 times with the 'train' option - and each run will have a separate output_ directory. Copy the colour .h5 models from each run into a new directory, say 'output_from_all_models/models_colour', and the grey .h5 models from each run into 'output_from_all_models/models_grey'. So, your resulting directory structure should look something like:

\n\n
output_from_all_models/models_colour/single--2018-04-11-12-34-31.h5\noutput_from_all_models/models_colour/single--2018-04-11-13-34-31.h5\noutput_from_all_models/models_colour/single--2018-04-11-14-34-31.h5\noutput_from_all_models/models_grey/single--2018-04-11-12-54-31.h5\noutput_from_all_models/models_grey/single--2018-04-11-13-54-31.h5\noutput_from_all_models/models_grey/single--2018-04-11-14-54-31.h5\n
\n\n

Then, you can run your command:

\n\n
python schwaebische_nuclei.py predict --loadmodel output_from_all_models/ --weights 4 2 1\n
\n\n

The weights are assigned based on the alphabetical order of the files - so in your command, the 'oldest' model will have a weight of 4.

\n\n

Hope this helps!

", "votes": 2, "replies": [] }, { "id": 322397, "author_name": "Wedad Anbtawi", "author_url": "", "post_date": "2018-05-02T21:18:36.147000", "content": "

Thank you very much it worked :) I have a question, the output Excel sheet has many duplicates Ids is it normal?

", "votes": 0, "replies": [] }, { "id": 322505, "author_name": "Liam Finnie", "author_url": "", "post_date": "2018-05-03T05:24:19.500000", "content": "

Yes, that's expected. Each nucleus in the image will have it's run-length-encoded information stored on a separate line.

", "votes": 0, "replies": [] } ] }, { "id": 316136, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T08:33:34.820000", "content": "

We shared our code, links at the end of main post above.

", "votes": 2, "replies": [] }, { "id": 315548, "author_name": "Burhan ", "author_url": "", "post_date": "2018-04-17T09:04:46.870000", "content": "

Thanks a lot @CPMP. Insightful indeed.\nI have been following your comments on discussion forums, throughout the competition. I need to say, they have been helpful in bringing clarity to things.\nHoping to see you soon in another competition.

", "votes": 2, "replies": [ { "id": 315575, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T09:40:02.177000", "content": "

Thanks. Sure, now that I've tasted CNNs I'll be back in another image competition soon!

", "votes": 0, "replies": [] } ] }, { "id": 322573, "author_name": "Kevin Liu", "author_url": "", "post_date": "2018-05-03T08:08:04.150000", "content": "

Hi @CPMP，I'm reading your code:\nZ_train[4*n + 0, :, :, 0] = (mask > 1)

\n\n

why use \" > 1\" ?

", "votes": 0, "replies": [ { "id": 322587, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-03T08:40:20.357000", "content": "

Because I want to find common boundaries, i.e. overlapping areas coming from at least two different masks.

", "votes": 0, "replies": [] } ] }, { "id": 315659, "author_name": "Annik J", "author_url": "", "post_date": "2018-04-17T12:34:28.557000", "content": "

lmdsg

", "votes": 0, "replies": [] }, { "id": 315567, "author_name": "", "author_url": "", "post_date": "2018-04-17T09:25:13.053000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "315538": "Some people asked for our solution and code, which surprises me a bit given there are so many people in front of us, some with way better scores than us. Watch for their write up as you'll learn more than from this one.\n\nFirst of all, let me thank my team mates, al.exe, Sven Hafeneger, Nicole Finnie, and Liam Finnie. I enjoyed the last 3 or 4 days of the competition when we tried to combine our approaches like crazy. I also thank those who shared so much, Heng Cher Keng, Allen Goodman, Anne Carpenter, Kjetil Åmdal-Sævik who kickstarted us with his kernel, and many others. Sorry I can't name you all. \n\nLast but not least, I thank Kaggle, Booz Allen Hamilton, and the organizers for setting this competition. Sure, there was some grumbling about stage 2 rules, but all in all the competition went pretty well. \n\nI started this competition to learn about deep learning and images. After reading few papers it was clear that there were 2 avenues for this: UNet or Mask-rcnn. I selected the former because it was way simpler, and also because Kjetil shared an amazing starter kernel. My final submission is a modification of his kernel. \n\nAfter 3 weeks or so I was in top 50 on the public LB with a 0.457 score, to my surprise. I interrupted my work on this to enter another competition (Toxic comment) and had a bad experience there as I got a gold medal then was removed because a team mate cheated. This is probably why I bugged many here with my post on 'cheating'. Net result however is that I lost motivation and did not resume work on DSB till I met some colleagues 4 days before competition end. They also had entered DSB and had worked on train time augmentation quite a bit. We decided to team, and spent 3 days cross pollinating each other.\n\nLet me now describe what I did, Nicole Finnie, one of my team mates, described their side of the house [in the comments below][1] . \n\n**Preprocessing.**\n\nIntensity is rescaled (put min to 0 and max to 255) and transform all images into gray images. Then I used Otsu thresholding, and inverted the intensity of images where the majority of pixels were above the threshold. The goal was to get images as similar as possible. I also did not use any clustering, thinking that training a single model would lead to better generalization. I now think that clustering with a small number of clusters, as my colleagues were doing, is better.\n\nNext step in preprocessing is to feed the model. I basically resized images to fit the CNN input size (256x256). This works quite fine except for images with a large aspect ratio, as it distorts the shape of nuclei. I decided to split images in 2 overlapping pieces that are as close as possible to squares. A square image yields 2 copies of it, and a very thin rectangle leads to two, less thin, rectangles. This led to less distortion.\n\nIssue with splitting images was that the boundary of predictions for the overlapping part was a bit wrong. I think it is because I used padding='same' instead of something more in line with the original UNet paper.\n\nI briefly tried using 256x256 crops of the images as an overlapping tiling , but it did not work as well. I must have been doing something wrong here because it worked fine for my team mates.\n\n**Model**\n\nIt is very close to Kjetil model. I only modified the output layer and the loss function, and I added a Speckle noise layer on the input in some runs, thanks to a post form Heng. Here is the code for a Speckle noise layer. it is a multiplicative Gaussian noise. I merely modified the code of the built in Keras additive Gaussian noise.\n\n from keras.engine import Layer\n from keras import backend as K\n \n class SpeckleNoise(Layer):\n \n def __init__(self, stddev, **kwargs):\n super(SpeckleNoise, self).__init__(**kwargs)\n self.supports_masking = True\n self.stddev = stddev\n \n def call(self, inputs, training=None):\n def noised():\n return K.clip(inputs * K.random_normal(shape=K.shape(inputs),\n mean=1.,\n stddev=self.stddev), 0.0, 1.0)\n return K.in_train_phase(noised, inputs, training=training)\n \n def get_config(self):\n config = {'stddev': self.stddev}\n base_config = super(SpeckleNoise, self).get_config()\n return dict(list(base_config.items()) + list(config.items()))\n \n def compute_output_shape(self, input_shape):\n return input_shape\n\n\nI modified the output layer to predict 3 channels with sigmoid activation. First channel predicts common boundary between adjacent nuclei, second channel predicts the union of nuclei, and third channel predicts background. The second channel is the same as in the original UNet model. In order to construct the ground truth for training, all masks are expanded by one pixel, and the union of these intersection is my first channel target. Second channel is the union of all masks, and last one is the complement of the other two. It is important to notice that the first two channels are not mutually exclusive, a pixel can be both on common boundary and in a mask.\n\nThe loss function is a pixelwise weighted loss function. I used weights of 30, 3 and 1 for each channel. Weights were selected via cross validation.\n\n import tensorflow as tf\n \n def pixelwise_crossentropy(target, output):\n _epsilon = 10e-8\n output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)\n weight = 30 * target[:,:,:,0:1] + 3 * target[:,:,:,1:2] + 1 * target[:,:,:,2:3]\n return - tf.reduce_sum(target * weight * tf.log(output) +\n (1 - target) * tf.log(1 - output),\n len(output.get_shape()) - 1)\n\nGiven the large weights I sometimes got exploding gradients (loss function grows to unreasonable levels). The way I fixed it was to clip gradient norm in the Adam optimizer. Maybe a better way would have to use SGD with proper settings, but I did not have time to try it. Adam is just working out of the box reasonable well.\n\n**Training**\n\nI used 6 fold cross validation, with a limit of 20 epochs, and early stopping with a patience of 5. Each of the 6 fold model is used to make predictions on the test data. These predictions are averaged before post processing test images. I shuffled images but made sure that all images (or crops) coming from one original image were in the same fold in order to avoid overfitting. Training time augmentation was limited to flipping and 90° rotations.\n\n**Postprocessing**\n\nFirst, images or crops coming from the same original images are merged, and rescaled to the original image size. Then the problem is how to segment the output as usual with UNet.\n\nI started with a watershed, like many, and explored different ways to seed it with good markers. Best results were obtained by defining markers as areas with a high probability to be in a mask (Second channel probability over 0.75), and far from being a common boundary (probability of first channel below 0.6). That use of the common boundary prediction was key in separating adjacent nuclei. Then watershed was expanded into the region where background probability was low (third channel probability below 0.4).\n\nAll thresholds were selected by cross validation.\n\nLooking at results, I noticed that in some cases watershed was producing disconnected masks. I therefore tried another way to expand markers: random walker in scikit-image. And this was really better than watershed. When my team mates switched from watershed to random walker they also noticed a significant improvement. In some cases random walker was triggering an error. In such case I used watershed.\n\nFinally, I tried some ensembling. I averaged the NN output over several variants (with or without noise, with various gradient clipping) then post processed it as before. This yields a CV improvement but didn't improve the Private LB score to my surprise.\n\n**Things I didn't have time to try**\n\nPretrained weights. I wanted to reuse pretrained weights as the NN I use basically contains a VGG-11 model. Reusing pretrained weights and make these layers not learnable would help fight overfitting. I wanted to reuse [TernausNet][2], but it was in Pytorch, and I did not had time to convert it to TF or Keras.\n\nMosaic. Many images were obtained by splitting larger images in 4. Working with the original images would help predict the nuclei that are split among two or more split images. Fortunately, my team mates implemented it after we merged, and this proved to be very useful\n\nMore training time augmentation. Color shift, blur, were on my plate. Fortunately, my team mates did a lot in that area.\n\nCombine with Mask Rcnn. Thanks to Heng I tried using his output as markers for my postprocessing. It improved LB score by nearly 0.001. I wanted to include a mask rcnn output as input to my modified UNet, but did not had time.\n\n**Things I did not think of**\n\nShrinking masks so that there are no adjacent asks, then expand predicted nuclei. This is a brilliant idea shared by Ildoo Kim. It makes UNet postprocessing way easier.\n\nTest time augmentation (ensembling). I did not find a way to merge my team mates predictions with mine, and I am eager to know what others did in that area.\n\nI'm sure Ill get many more from top performers write ups!\n\n**Take away**\n\nI learned a lot, and I now see there is a lot I have to learn still. One thing I had a hunch for, and that was confirmed here, is the importance of pre and post processing. Many proponents of deep learning say that there is no need to preprocess data. Well, in my case, stage1 score went from 0.263 to 0.457 via pre and post processing alone.\n\nHope the above is of interest to some.\n\nI shared the code that led to my best stage 1 submission [on github][3]. My team mate shared the final submission code on [github][4] too.\n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54742#315594\n [2]: https://github.com/ternaus/TernausNet\n [3]: https://github.com/jfpuget/DSB_2018\n [4]: https://github.com/nicolefinnie/kaggle-dsb2018", "315594": "https://github.com/nicolefinnie/kaggle-dsb2018\n\n**Newbies' solution - as first time Kagglers (LB = 0.545)**\n\nI'm @CPMP's team mate + colleague. Since most of us were first time Kagglers, we implemented a very simple U-net, taken from the U-net starter of @Kjetil Åmdal-Sævik that any DL newbies like us can do.\n\n**Model**\n\n- Same as the U-net starter except two output channels: `masks` and `contours` with the `binary cross entropy loss`\n\n**Pre-processing**\n \n - CLAHE (Contrast Limited Adaptive Histogram Equalization) \n - L-channel only\n - Inversion only when the background is lighter than the foreground\n - Reference: https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence\n - The goal is to make it easy for the U-net to learn their common features.\n\n\n**K-mean cluster based on colours**\n\n - K-mean cluster using `R, G, R-G deviation` on train/test images. As @Heng suggested, if we train grey and colour images together, their common trait is the shape/boundary, so it'd be hard for the neural net to learn the features inside the nucleus. In a result, our U-net did fairly well on the colour images at the 2nd stage, okay, maybe not too well on @Allen's art :)\n\n![Allen's art](https://kaggle2.blob.core.windows.net/forum-message-attachments/315594/9206/AllensArt.png)\n\n**Mosaic for training and predicting**\n\n- As suggested by @CPMP, we applied @Emil's mosaic idea (KNN on 4 edges of the image to find the matching neighbours) and merged all images to mosaic to gain the information on the edges \nhttps://www.kaggle.com/bonlime/train-test-image-mosaic\n![A mosaic trained image composed of 4 images](https://kaggle2.blob.core.windows.net/forum-message-attachments/315594/9207/mosaic_train_colour1.png)\n\n- Around `37%` of the grey test images and about `77%` of the colour test images can be formed into mosaic, this ratio still holds true for the stage 2 test images. That means we gained more hidden traits than most teams.\n\n**Windowing images for training and predictions**\n\n- Thanks to mosaic, now we have a lot more information on the edges, so we crop images to `256x256` using overlapping windows with a stride of `128`, because `256x256` yields the best mean IoU.\n\n- Mirror the edges and the corners as suggested by the U-net paper. However, we have less edges than the people who don't use the `mosaic` approach\n\n- Rotate and flip all windowing images, this augments images by `8x`\n\n**Data augmentation**\n \n- Transform perspective using the `imgaug` library. (this approach is used by the top 1 team)\n\n- Additive Gaussian noise + speckle noise. (used by the top 1 team)\n\n- Gaussian Blur (no time to train)\n\n- Image pyramid, upsampling and downsampling based on the average nuclei size per image (no time to train )\n\n- Regional gamma correction (no time to train) \n\n- Greyscale (no time to train)\n\n- Inversion, hurts our LB since it makes the U-net difficult to learn, so we discarded the idea just like the top 1 team\n\n**Average windowing predictions**\n\n- Predict on overlapping windowing images in rotated/flipped/mirrored versions\n- Average all windowing predictions to minimize the artifacts, this approach was used by the winners in the satellite competition last year.\n- Stitch predictions together to the original size\n\n\n**Post-processing**\n\n- Otsu threshold on predicted masks and predicted contours\n- Find sure foreground: `(predicted masks - predicted contours)` \n- Find sure background: dilates the predicted masks\n- Find unknown area: `(sure background - sure foreground)`\n- Find labels on the sure foreground \n- Random walker or watershed on the masks + contours to avoid `contour erosion`, adding the thresholding contour gave us a big LB boost. \n\n- Reference: https://docs.opencv.org/3.3.1/d3/db4/tutorial_py_watershed.html\n\n**Model ensemble using weighted majority voting**\n\n- @CPMP chose weights based on our public LB scores\n- `4 * ( baseline model + noise ) + 2 * ( baseline model + transform ) + 1 * ( baseline model)`\n\n\n**Acknowledgement**\n\nThanks to @CPMP. If he didn't join our team in the last 3 days, we wouldn't have known that we had to train and submit models by the deadline of the first stage and we wouldn't have submitted any models that we could have used at the 2nd stage. Sadly, we didn't get to train all the models with data augmentation we planned to, however, we're happy with the results as first time Kagglers. It's pretty easy to implement our approach. ", "315774": "Thanks for sharing! I liked your speckled noise. I'll try to replace the Gaussian layer I used by it instead. Does it get turned on/off for training and evaluating, respectively?", "315564": "Thanks for sharing!\n\nDue to lack of time and experience, I myself used a very simple unet, not so different from what you describe here. And due to lack of time and only moderate computational resources it was also poorly optimized & trained...\n\nGood thing: There's at least clearly no lack of ideas that I could try out the next time :)\n\nWith more time I would mostly have wanted to work more on the initial cropping, because that seemed to matter quite a lot (splitting the image into 256x256 crops gave me better results than resizing). But same as you I had some issues with masks where crops overlapped...\nI would guess that first briefly looking for potential candidates than cropping smaller areas would have been better. I might even have added a spherical decay function towards the edge of the crops to make sure the model focuses more on the center part.\n\nAnd in the end I also became more interested in using autoencoders to augment the data (and maybe also to pre-train the first few layers?). \nHas anyone played around with this?", "321959": "Thank you @CPMP for sharing! I was testing your code to check how the model works, all worked till I got into \"python schwaebische_nuclei.py predict --loadmodel output_from_all_models/ --weights 4 2 1\", it gave me an error \"Traceback (most recent call last):\n File \"schwaebische_nuclei.py\", line 765, in ", "316136": "We shared our code, links at the end of main post above.", "315548": "Thanks a lot @CPMP. Insightful indeed.\nI have been following your comments on discussion forums, throughout the competition. I need to say, they have been helpful in bringing clarity to things.\nHoping to see you soon in another competition.", "322573": "Hi @CPMP，I'm reading your code:\nZ_train[4*n + 0, :, :, 0] = (mask > 1)\n\nwhy use \" > 1\" ?", "315659": "lmdsg", "315567": "" }, "complete": true, "source": "existing_discussion", "competition_id": 8089, "competition_slug": "data-science-bowl-2018", "team_id": 1417255, "solution_url": "/competitions/data-science-bowl-2018/writeups/unet-nuke-unet-nuke-solution-68-lb" }