![\"Model](\"https://storage.googleapis.com/kaggle-forum-message-attachments/317711/9258/Vector%20unet.png\")
In our team we have evaluated both UNet and Mask-RCNN based solution, but for us Unet worked significantly better so we used Unet based model for submission.
\n\nThe UNet based solution is inspired by the Deep Watershed Transform paper: https://arxiv.org/pdf/1611.08303.pdf
\n\nFor each pixels, we predicted the x,y components of vector pointing from the instance border like described in the DWT paper and predicted the mask, watershed levels and nuclei centers using the second connected UNet.
\n\nAttached the diagram with our model
\n\n
Overall Unet predict the mask pretty well but it was necessary to find a way to reliably segment nuclei.
\n\nThe first approach was to predict using the single UNet model:
\n\nFor touching nuclei the vector value to the centers changes sign, so it changes sharply and the loss is the biggest on the nuclei border which forces model to learn to separate instances. The postprocessing was quite straightforward:
\n\nThis approach worked and scored over 0.5 on the public leaderboard.
\n\nThe second approach was to try the deep watershed transform idea to predict watershed energy levels (mask eroded by different offsets) by predicting intermediate unit vector fields pointing from the nearest border pixel. Instead of training 3 independent models for segmentation, vector field and watershed energy predictions I used the single UNet to predict everything. With the seeds we used the continuous areas after applying the threshold over the sum of energy levels, in the similar way to the DWT paper. The result was slightly better.
\n\nUse the predicted centers as seeds for watershed transform instead of the energy level with threshold. The score improvement was more significant comparing to switching to DWT.
\n\nPredict only vector fields with the first UNet, concatenated predicted fields with the last layer used to predictions of all the other fields using another UNet. This has little to no impact on predicted masks and the energy levels but helped to significantly improve the quality of predicted nuclei centers.
\n\nSimple mean ensembling of 8 TTA flips/rotations had very small but consistent improvement.
\n\nBetter postprocessing, the idea - since for prediction the center values we are using the 3x3 path of area 9, we can expect the total area of predicted center to be close to 9 as well, even for cases of complex connected nuclei hard to predict. \nThis allowed to do following post processing improvements:
\n\nWe added extra annotated datasets listed in the forum thread and annotated some images, mostly color histology images. We also used synthetic data to generate more cases of touching and overlapped nuclei.
\n\nAttached an example of the early model predictions, including predicted vectors, centers and energy levels.
\n\n![\"Example](\"https://storage.googleapis.com/kaggle-forum-message-attachments/317711/9259/nuclei_descr.png\")
We tried to use the imagenet pretrained models as UNet encoders but the result was the same or worse comparing to UNet trained from scratch.
\n\nFor unet encoders we used \nConv2d - BN - Relu - Conv2d - Relu\nDecoders:\nUpsample/concatenate - Conv2d - Relu
\n\nWe used quite a large number of filters (64/128) even on the high resolution levels as model had to predict many outputs but increased number of filters less than twice as we did not have as much complex semantic informations as with models trained on imagenet.
\n\nUnet1 was 6 levels deep (decreased the resolution 2x 6 times) while Unet2 was 4 levels deep.
\n\nAs input we used B/W image or H channel of HED stain decomposition. H worked slightly better for histology images but slightly worse for some other stains. For the final submission we ensembled 4 models trained on B/W images and 2 on H channel.
\n\nWe trained on 256x256 patches with significant level of augmentations and predicted on 1024 pixels tiles with 128 pix overlap on each side. We used the “SAME” padding and relied on the large tiles padding and overlap to avoid corner effects.
\n\nAt the end of competition we tried to train a model predicting only data used for post processing without vector fields and the result was significantly worse. Significantly reducing the loss of vectors to the center of nuclei had a little impact to the results, so most likely vectors from corners are more important. Even while not directly used, it helped to better predict centers and watershed energy levels.
\n\n", "messages": [ { "id": "317711", "postDate": "04/22/2018 10:20:44", "content": "In our team we have evaluated both UNet and Mask-RCNN based solution, but for us Unet worked significantly better so we used Unet based model for submission.
\n\nThe UNet based solution is inspired by the Deep Watershed Transform paper: https://arxiv.org/pdf/1611.08303.pdf
\n\nFor each pixels, we predicted the x,y components of vector pointing from the instance border like described in the DWT paper and predicted the mask, watershed levels and nuclei centers using the second connected UNet.
\n\nAttached the diagram with our model
\n\n
Overall Unet predict the mask pretty well but it was necessary to find a way to reliably segment nuclei.
\n\nThe first approach was to predict using the single UNet model:
\n\nFor touching nuclei the vector value to the centers changes sign, so it changes sharply and the loss is the biggest on the nuclei border which forces model to learn to separate instances. The postprocessing was quite straightforward:
\n\nThis approach worked and scored over 0.5 on the public leaderboard.
\n\nThe second approach was to try the deep watershed transform idea to predict watershed energy levels (mask eroded by different offsets) by predicting intermediate unit vector fields pointing from the nearest border pixel. Instead of training 3 independent models for segmentation, vector field and watershed energy predictions I used the single UNet to predict everything. With the seeds we used the continuous areas after applying the threshold over the sum of energy levels, in the similar way to the DWT paper. The result was slightly better.
\n\nUse the predicted centers as seeds for watershed transform instead of the energy level with threshold. The score improvement was more significant comparing to switching to DWT.
\n\nPredict only vector fields with the first UNet, concatenated predicted fields with the last layer used to predictions of all the other fields using another UNet. This has little to no impact on predicted masks and the energy levels but helped to significantly improve the quality of predicted nuclei centers.
\n\nSimple mean ensembling of 8 TTA flips/rotations had very small but consistent improvement.
\n\nBetter postprocessing, the idea - since for prediction the center values we are using the 3x3 path of area 9, we can expect the total area of predicted center to be close to 9 as well, even for cases of complex connected nuclei hard to predict. \nThis allowed to do following post processing improvements:
\n\nWe added extra annotated datasets listed in the forum thread and annotated some images, mostly color histology images. We also used synthetic data to generate more cases of touching and overlapped nuclei.
\n\nAttached an example of the early model predictions, including predicted vectors, centers and energy levels.
\n\n
We tried to use the imagenet pretrained models as UNet encoders but the result was the same or worse comparing to UNet trained from scratch.
\n\nFor unet encoders we used \nConv2d - BN - Relu - Conv2d - Relu\nDecoders:\nUpsample/concatenate - Conv2d - Relu
\n\nWe used quite a large number of filters (64/128) even on the high resolution levels as model had to predict many outputs but increased number of filters less than twice as we did not have as much complex semantic informations as with models trained on imagenet.
\n\nUnet1 was 6 levels deep (decreased the resolution 2x 6 times) while Unet2 was 4 levels deep.
\n\nAs input we used B/W image or H channel of HED stain decomposition. H worked slightly better for histology images but slightly worse for some other stains. For the final submission we ensembled 4 models trained on B/W images and 2 on H channel.
\n\nWe trained on 256x256 patches with significant level of augmentations and predicted on 1024 pixels tiles with 128 pix overlap on each side. We used the “SAME” padding and relied on the large tiles padding and overlap to avoid corner effects.
\n\nAt the end of competition we tried to train a model predicting only data used for post processing without vector fields and the result was significantly worse. Significantly reducing the loss of vectors to the center of nuclei had a little impact to the results, so most likely vectors from corners are more important. Even while not directly used, it helped to better predict centers and watershed energy levels.
\n\n", "rawMarkdown": "In our team we have evaluated both UNet and Mask-RCNN based solution, but for us Unet worked significantly better so we used Unet based model for submission.\n\nThe UNet based solution is inspired by the Deep Watershed Transform paper: https://arxiv.org/pdf/1611.08303.pdf\n\n\nFor each pixels, we predicted the x,y components of vector pointing from the instance border like described in the DWT paper and predicted the mask, watershed levels and nuclei centers using the second connected UNet. \n\nAttached the diagram with our model\n\n![Model description][1]\n\nOverall Unet predict the mask pretty well but it was necessary to find a way to reliably segment nuclei.\n\nApproaches we tried\n================\n\nThe first approach was to predict using the single UNet model:\n\n * Mask, BCE + DICE loss\n * Nuclei centers with 3x3 patches around the center of mass as a training label, BCE loss\n * Area of nuclei used to normalize loss from vectors for large and small nuclei\n * X,Y of vector to the center of nuclei, MSE loss normalized by nuclei area\n\n\nFor touching nuclei the vector value to the centers changes sign, so it changes sharply and the loss is the biggest on the nuclei border which forces model to learn to separate instances. The postprocessing was quite straightforward:\n\n 1. Find the centers of nuclei using predicted centers output, expecting the area of each prediction to be approx 9.0 (matring area of 3x3 training patch)\n 2. For each pixel in predicted masc, assign it to the cluster nearest to position predicted vector to the center points to.\n\nThis approach worked and scored over 0.5 on the public leaderboard.\n\n\nImprovement:\n-----------------\n\nThe second approach was to try the deep watershed transform idea to predict watershed energy levels (mask eroded by different offsets) by predicting intermediate unit vector fields pointing from the nearest border pixel. Instead of training 3 independent models for segmentation, vector field and watershed energy predictions I used the single UNet to predict everything. With the seeds we used the continuous areas after applying the threshold over the sum of energy levels, in the similar way to the DWT paper. The result was slightly better.\n\nImprovement:\n-----------------\n\nUse the predicted centers as seeds for watershed transform instead of the energy level with threshold. The score improvement was more significant comparing to switching to DWT.\n\nImprovement:\n-----------------\n\nPredict only vector fields with the first UNet, concatenated predicted fields with the last layer used to predictions of all the other fields using another UNet. This has little to no impact on predicted masks and the energy levels but helped to significantly improve the quality of predicted nuclei centers. \n\nImprovement:\n-----------------\nSimple mean ensembling of 8 TTA flips/rotations had very small but consistent improvement.\n\nImprovement:\n-----------------\n\nBetter postprocessing, the idea - since for prediction the center values we are using the 3x3 path of area 9, we can expect the total area of predicted center to be close to 9 as well, even for cases of complex connected nuclei hard to predict. \nThis allowed to do following post processing improvements:\n\n * After watershed transform from detected centers, check for missing large masks with the total center prediction > ~5.0, and add them.\n * Instances with the total center value integral > 9.0 * 1.5 are most likely have two nuclei connected but with very hard to predict centers. Split centers to two clusters using KNN and re-run watershed.\n * We tried the similar approach to topcoder team of estimating IOU using information about the shape and other predicted instance properties and run optimiser for the score to decide if particular instance should be included to submission or not, but the quality of IOU prediction was not sufficient for this to work reliably. Instead we calculated IOU between masks of the ensemble and individual TTA variants. If the median IOU is low, it means different models predicted different shapes and it’s better to exclude such mask from submission to avoid penalty at high IOU thresholds.\n\n\nExtra data used\n-------------------\n\nWe added extra annotated datasets listed in the forum thread and annotated some images, mostly color histology images. We also used synthetic data to generate more cases of touching and overlapped nuclei.\n\nFinal model details\n==============\n\nAttached an example of the early model predictions, including predicted vectors, centers and energy levels.\n\n![Example prediction][2]\n\nWe tried to use the imagenet pretrained models as UNet encoders but the result was the same or worse comparing to UNet trained from scratch.\n\nFor unet encoders we used \nConv2d - BN - Relu - Conv2d - Relu\nDecoders:\nUpsample/concatenate - Conv2d - Relu\n\nWe used quite a large number of filters (64/128) even on the high resolution levels as model had to predict many outputs but increased number of filters less than twice as we did not have as much complex semantic informations as with models trained on imagenet.\n\nUnet1 was 6 levels deep (decreased the resolution 2x 6 times) while Unet2 was 4 levels deep. \n\nAs input we used B/W image or H channel of HED stain decomposition. H worked slightly better for histology images but slightly worse for some other stains. For the final submission we ensembled 4 models trained on B/W images and 2 on H channel.\n\nWe trained on 256x256 patches with significant level of augmentations and predicted on 1024 pixels tiles with 128 pix overlap on each side. We used the “SAME” padding and relied on the large tiles padding and overlap to avoid corner effects.\n\nAt the end of competition we tried to train a model predicting only data used for post processing without vector fields and the result was significantly worse. Significantly reducing the loss of vectors to the center of nuclei had a little impact to the results, so most likely vectors from corners are more important. Even while not directly used, it helped to better predict centers and watershed energy levels.\n\n[Solution source code][3]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/317711/9258/Vector%20unet.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/317711/9259/nuclei_descr.png\n [3]: https://github.com/pdima/kaggle_2018_data_science_bowl_solution", "votes": null }, { "id": "317733", "postDate": "04/22/2018 12:17:21", "content": "@Dmytro, thank you very much for sharing your approach, a fascinating read. I just read the DWT paper you mentioned (happened to be from the same University I went to), I wish I had known about this at the beginning of this competition! The idea of using different energy levels to help with object segmentation is pretty powerful, and is actually similar in concept to the method described in the original Unet paper:\nUnet research paper
\n\nThe original Unet paper gave the highest weight to the object border pixels (so, similar to energy level 1 from the DWT paper). However, DWT seems more powerful, since it has many more energy levels - and using different energy levels for the cut based on the object size sounds like it might be helpful for this competition too (did you try this approach, based on the average predicted nuclei size?).
\n\nOur U-Net ended up being pretty basic compared to your approach, but our team was still pretty happy with our results in this competition, since this was the first Kaggle competition for most of us (we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly).
\n\nThanks again for sharing!
", "rawMarkdown": "Dmytro, thank you very much for sharing your approach, a fascinating read. I just read the DWT paper you mentioned (happened to be from the same University I went to), I wish I had known about this at the beginning of this competition! The idea of using different energy levels to help with object segmentation is pretty powerful, and is actually similar in concept to the method described in the original Unet paper:\n[Unet research paper][1]\n\nThe original Unet paper gave the highest weight to the object border pixels (so, similar to energy level 1 from the DWT paper). However, DWT seems more powerful, since it has many more energy levels - and using different energy levels for the cut based on the object size sounds like it might be helpful for this competition too (did you try this approach, based on the average predicted nuclei size?).\n\nOur U-Net ended up being pretty basic compared to your approach, but our team was still pretty happy with our results in this competition, since this was the first Kaggle competition for most of us (we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly).\n\nThanks again for sharing!\n\n [1]: http://www.cs.cmu.edu/~jeanoh/16-785/papers/ronnenberger-miccai2015-u-net.pdf", "votes": null }, { "id": "317743", "postDate": "04/22/2018 12:45:13", "content": "Thanks for sharing, and congrats on the result!
\n\n\n\n\nAfter watershed transform from detected centers, check for missing large masks with the total center prediction > ~5.0, and add them.
\n
Can you explain this a bit more? How do you define 'missed masks' given you don't have the masks when predicting?
", "rawMarkdown": "Thanks for sharing, and congrats on the result!\n\n> After watershed transform from detected centers, check for missing large masks with the total center prediction > ~5.0, and add them.\n\nCan you explain this a bit more? How do you define 'missed masks' given you don't have the masks when predicting?", "votes": null }, { "id": "317751", "postDate": "04/22/2018 13:17:34", "content": "My algorithm to select nuclei centers as seeds for watershed may have missed some nuclei if it's of complex form or uncertain shape while the integral of predicted center over nuclei area is still close to expected value of 9.0
\n\nSuch cases were easy to detect and fix by looking for connected areas of the mask (I used one of predicted energy levels) not included into resulting set of instances masks after the watershed transform.
", "rawMarkdown": "My algorithm to select nuclei centers as seeds for watershed may have missed some nuclei if it's of complex form or uncertain shape while the integral of predicted center over nuclei area is still close to expected value of 9.0\n\nSuch cases were easy to detect and fix by looking for connected areas of the mask (I used one of predicted energy levels) not included into resulting set of instances masks after the watershed transform.", "votes": null }, { "id": "317760", "postDate": "04/22/2018 13:40:57", "content": "More detailed explanation of centers selection following post processing:
\n\nThis allowed to better detect nuclei separation, sometimes center prediction worked better and sometimes predicted watershed energy levels, this approach helped to combine both predictions.
\n\nListed steps helped but the improvement was not as huge, I'm talking about local CV or public LB score changes from 0.565 to 0.570 or so.
", "rawMarkdown": "More detailed explanation of centers selection following post processing:\n\n1. For each peak in centers prediction, I tried to find which radius circle has sufficient sum of predicted values (this threshold a a function from radius was an important hyperparameter to tune). I then added the \npeak to the list of center seeds and cleared the circle, so it does not affect calculation of next centers.\n2. Run watershed using the sum of centers predictions and watershed energy levels as an energy and the first energy level as a mask. Using the first energy level instead of actual mask here helped in a few cases to separate touching nuclei if center prediction failed for one of them.\n3. As described above, I have checked if any of large enough continuous masks in energy level 1 does not have seed but has sufficient total sum of centers prediction, added center of mass as a new seed.\n4. Split some nuclei to two seeds using KNN if total integral of predicted center > ~15.0\n5. Re run watershed using the actual predicted mask and all found center seeds\n\nThis allowed to better detect nuclei separation, sometimes center prediction worked better and sometimes predicted watershed energy levels, this approach helped to combine both predictions.\n\nListed steps helped but the improvement was not as huge, I'm talking about local CV or public LB score changes from 0.565 to 0.570 or so.", "votes": null }, { "id": "317761", "postDate": "04/22/2018 13:47:08", "content": "Hi Liam,
\n\nI tried to use different thresholds using different watershed energy levels but overall it worked better to use predicted centers.
\n\nWhat is interesting, asking model to predict the vectors from the border like in the DWT paper improved the centers predictions. Allowing model to process such vectors further with a greater receptive field of the second unet, helped to improve predicted centers slightly more.
", "rawMarkdown": "Hi Liam,\n\nI tried to use different thresholds using different watershed energy levels but overall it worked better to use predicted centers.\n\nWhat is interesting, asking model to predict the vectors from the border like in the DWT paper improved the centers predictions. Allowing model to process such vectors further with a greater receptive field of the second unet, helped to improve predicted centers slightly more.", "votes": null }, { "id": "317765", "postDate": "04/22/2018 13:56:36", "content": "Thanks a lot, it is crystal clear now.
", "rawMarkdown": "Thanks a lot, it is crystal clear now.", "votes": null }, { "id": "317795", "postDate": "04/22/2018 15:19:18", "content": "@Dmytro Poplavskiy if you're interested in late submissions, you can try @CPMP's top secret post processing approach, random walker instead of watershed, that gave us an LB boost between 0.02~0.026 every time because it minimized contour erosion. just the algorithm is very slow, so we had to make our prediction code multi-threaded. And we did random walker on contours as well which is similar to the concept of your first energy level. If that works on your case too, it should improve your LB from 0.610 to 0.630 I hope :) Thanks a lot for sharing, very informative.
> we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly
\n\nI don't want that people think our final score is just due to my ideas. My own solo submission scored way lower than our team submission.
", "rawMarkdown": "> we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly\n\nI don't want that people think our final score is just due to my ideas. My own solo submission scored way lower than our team submission.", "votes": null }, { "id": "317899", "postDate": "04/22/2018 19:42:32", "content": "Congratulations and thanks for sharing this clear presentation. What language/tool did you use to implement the x,y vector computing layer?
", "rawMarkdown": "Congratulations and thanks for sharing this clear presentation. What language/tool did you use to implement the x,y vector computing layer?", "votes": null }, { "id": "317934", "postDate": "04/22/2018 21:35:11", "content": "The model itself is written in keras, but to prepare vector labels I used scipy.ndimage.morphology
\n\nscipy.ndimage.morphology.distance_transform_edt() is very useful to find distance and direction to the border as it can return the index of the nearest pixel outside of mask:
\n\nedt, inds = scipy.ndimage.morphology.distance_transform_edt(crop_smooth, return_distances=True, return_indices=True)\nborder_vector = np.array([\n np.expand_dims(np.arange(0, rows), axis=1) - inds[0],\n np.expand_dims(np.arange(0, cols), axis=0) - inds[1]])\n\nborder_vector_norm = border_vector / (np.linalg.norm(border_vector, axis=0, keepdims=True) + 1e-5)\nres_crop[:, :, 0] = border_vector_norm[0]\nres_crop[:, :, 1] = border_vector_norm[1]\nVector to the center is easier to calculate, it's a difference of the pixel position and the mask center of mass:
\n\n center_of_mass = scipy.ndimage.measurements.center_of_mass(crop)\n current_offset_field = np.zeros((CROP_SIZE, CROP_SIZE, 2))\n current_offset_field[:, :, 0] = np.expand_dims(center_of_mass[0] - np.arange(0, CROP_SIZE), axis=1)\n current_offset_field[:, :, 1] = np.expand_dims(center_of_mass[1] - np.arange(0, CROP_SIZE), axis=0)\nThis is very interesting idea, and the very significant boost to result. Would be very interesting to read in more detail, do you plan to publish you approach (or maybe I have missed it)?
\n\nIt's also interesting if the random walker approach can be combined with the information about the estimated energy levels, it may work better than either one separately.
\n\n[edited] I missed this bit in your team result description, certainly very interesting approach.
", "rawMarkdown": "This is very interesting idea, and the very significant boost to result. Would be very interesting to read in more detail, do you plan to publish you approach (or maybe I have missed it)?\n\nIt's also interesting if the random walker approach can be combined with the information about the estimated energy levels, it may work better than either one separately.\n\n[edited] I missed this bit in your team result description, certainly very interesting approach.", "votes": null }, { "id": "317941", "postDate": "04/22/2018 22:00:38", "content": "@Dmytro, yes, @CPMP published our code last week and you can look at our post processing random walker code if you're interested - - we only replaced watershed with random walker and the rest of the code stayed exactly the same. That's why I was thinking it should be very easy for you to plug-in the approach on your 2nd step and 5th step. The significant improvement was in the mean IoU range between 0.8 to 0.95. With watershed, we got a very low score on the mean IoU above 0.85 due to the nature of watershed (contour erosion).
Thank you very much for the clarification. I saw the paper during the competition but didn't come up with a good way to implement it. It makes sense how your team did it.
", "rawMarkdown": "Thank you very much for the clarification. I saw the paper during the competition but didn't come up with a good way to implement it. It makes sense how your team did it.", "votes": null }, { "id": "318012", "postDate": "04/23/2018 03:19:30", "content": "Impressive work once again.
", "rawMarkdown": "Impressive work once again.", "votes": null }, { "id": "318660", "postDate": "04/24/2018 08:20:24", "content": "Congrats @Dmytro Poplavskiy awesome work.
", "rawMarkdown": "Congrats @Dmytro Poplavskiy awesome work.", "votes": null }, { "id": "319151", "postDate": "04/25/2018 11:19:38", "content": "Thanks for sharing, Dmytro! :D
", "rawMarkdown": "Thanks for sharing, Dmytro! :D", "votes": null }, { "id": "320782", "postDate": "04/29/2018 20:29:29", "content": "Great exploratory approach! I'm guessing also adding predictions for diagonal vectors, not just x and y, would add a slight additional boost.
\n\nI think, one of the things W-nets (double-U-Nets, or some other multiple of them, of course) should add vs just going deeper in a U-net is a better positional accuracy of features. I also tried a W-net, though it's a bit tricky, especially the way I sometimes tried it, putting it as a residual. Sometimes it just exploded. I'm guessing it might be a good idea to try later in a segmentation task to make a multibranching net with nodes having branches that go upscale, downscale and same scale. Sort of combining U-nets and sub-U-nets going a different depth from different depth and merging them at different depths.
", "rawMarkdown": "Great exploratory approach! I'm guessing also adding predictions for diagonal vectors, not just x and y, would add a slight additional boost.\n\nI think, one of the things W-nets (double-U-Nets, or some other multiple of them, of course) should add vs just going deeper in a U-net is a better positional accuracy of features. I also tried a W-net, though it's a bit tricky, especially the way I sometimes tried it, putting it as a residual. Sometimes it just exploded. I'm guessing it might be a good idea to try later in a segmentation task to make a multibranching net with nodes having branches that go upscale, downscale and same scale. Sort of combining U-nets and sub-U-nets going a different depth from different depth and merging them at different depths.", "votes": null }, { "id": "320824", "postDate": "04/30/2018 00:51:19", "content": "Hi, Nicole. Thank you for sharing. However, random walker code hyperlink doesn't work. Could you please show the link again?
", "rawMarkdown": "Hi, Nicole. Thank you for sharing. However, random walker code hyperlink doesn't work. Could you please show the link again?", "votes": null }, { "id": "320888", "postDate": "04/30/2018 06:01:51", "content": "@Zhen fixed the link
", "rawMarkdown": "Zhen fixed the [link][1]\n\n\n [1]: https://github.com/nicolefinnie/kaggle-dsb2018/blob/master/src/modules/image_processing.py#L178", "votes": null }, { "id": "321050", "postDate": "04/30/2018 13:30:52", "content": "I'm really proud of my teammate Dmytro for this very innovative and detailed approach. We are thinking \"W-Net\" is indeed a better name, especially since the English letter \"W\" is pronounced \"Double U\" and it appears compatible with https://arxiv.org/abs/1711.08506. With watershed in there too we might should even call it \"W2-Net\", but \"W2\" is also the name of the wages/tax form in the U.S. and we don't want to go anywhere near that!
", "rawMarkdown": "I'm really proud of my teammate Dmytro for this very innovative and detailed approach. We are thinking \"W-Net\" is indeed a better name, especially since the English letter \"W\" is pronounced \"Double U\" and it appears compatible with [https://arxiv.org/abs/1711.08506][1]. With watershed in there too we might should even call it \"W2-Net\", but \"W2\" is also the name of the wages/tax form in the U.S. and we don't want to go anywhere near that! \n\n\n [1]: https://arxiv.org/abs/1711.08506", "votes": null }, { "id": "321161", "postDate": "04/30/2018 18:34:17", "content": "Totally agree! I was really surprised that such a good result could be achieved with a model trained from scratch.
", "rawMarkdown": "Totally agree! I was really surprised that such a good result could be achieved with a model trained from scratch.", "votes": null }, { "id": "321179", "postDate": "04/30/2018 19:22:26", "content": "Congrats! The name is very catchy and clever.
", "rawMarkdown": "Congrats! The name is very catchy and clever.", "votes": null }, { "id": "321852", "postDate": "05/02/2018 02:55:10", "content": "Nice work...
", "rawMarkdown": "Nice work...", "votes": null }, { "id": "323980", "postDate": "05/06/2018 20:32:51", "content": "Hi Dmytro, great job and thanks for sharing your methods. You decribed your first approach as:\n\"The first approach was to predict using the single UNet model:\nMask, BCE + DICE loss\nNuclei centers with 3x3 patches around the center of mass as a training label, BCE loss\nArea of nuclei used to normalize loss from vectors for large and small nuclei\nX,Y of vector to the center of nuclei, MSE loss normalized by nuclei area\"
\n\nMy questions is how, did you predict the area of nuclei? What were your training labels? Were they masks that contained the numeral value or their respective areas?
\n\nThank you for your time
", "rawMarkdown": "Hi Dmytro, great job and thanks for sharing your methods. You decribed your first approach as:\n\"The first approach was to predict using the single UNet model:\nMask, BCE + DICE loss\nNuclei centers with 3x3 patches around the center of mass as a training label, BCE loss\n**Area of nuclei** used to normalize loss from vectors for large and small nuclei\nX,Y of vector to the center of nuclei, MSE loss normalized by nuclei area\"\n\nMy questions is how, did you predict the area of nuclei? What were your training labels? Were they masks that contained the numeral value or their respective areas?\n\nThank you for your time", "votes": null }, { "id": "324169", "postDate": "05/07/2018 10:06:47", "content": "Hi, Your post is good. Its so useful for me about Data Science.\nData Science Training in Bangalore
", "rawMarkdown": "Hi, Your post is good. Its so useful for me about Data Science.\n[Data Science Training in Bangalore][1]\n\n\n [1]: https://www.besanttechnologies.com/training-courses/data-science-training-in-bangalore", "votes": null }, { "id": "326243", "postDate": "05/09/2018 13:28:07", "content": "Hi Andy,
\n\nI predicted the per pixel area value as one of UNET output layers using mae as a loss, but the main reason for using area was to normalize the loss of vector predicted to nuclei center, otherwise the mistake of center vector prediction for large nuclei would be penalized much more comparing to small nuclei.
\n\nI have not seen significant improvements from making the model to predict the area alone.
", "rawMarkdown": "Hi Andy,\n\nI predicted the per pixel area value as one of UNET output layers using mae as a loss, but the main reason for using area was to normalize the loss of vector predicted to nuclei center, otherwise the mistake of center vector prediction for large nuclei would be penalized much more comparing to small nuclei.\n\nI have not seen significant improvements from making the model to predict the area alone.", "votes": null }, { "id": "371696", "postDate": "08/17/2018 13:01:33", "content": "Source code for our solution is available at github: https://github.com/pdima/kaggle_2018_data_science_bowl_solution
", "rawMarkdown": "Source code for our solution is available at github: https://github.com/pdima/kaggle_2018_data_science_bowl_solution", "votes": null }, { "id": "374822", "postDate": "08/23/2018 23:14:27", "content": "Congratulations @Dmytro Poplavskiy, thanks for sharing your solution and code.
", "rawMarkdown": "Congratulations @Dmytro Poplavskiy, thanks for sharing your solution and code.", "votes": null }, { "id": "416501", "postDate": "11/06/2018 18:49:03", "content": "Hi @dmytropoplavskiy, congratulations for the great work! Please, correct me if I'm wrong, in the end (final model) you did not need the area prediction, right? Because you were already predicting the normalized angle vectors like the DWT paper, correct?
\n\nSo you only need mask, eroded mask and centers for watershed (the angle vector helped indirectly from what I understood), is this correct?
\n\nThank you
", "rawMarkdown": "Hi @dmytropoplavskiy, congratulations for the great work! Please, correct me if I'm wrong, in the end (final model) you did not need the area prediction, right? Because you were already predicting the normalized angle vectors like the DWT paper, correct?\n\nSo you only need mask, eroded mask and centers for watershed (the angle vector helped indirectly from what I understood), is this correct?\n\nThank you", "votes": null }, { "id": "418190", "postDate": "11/09/2018 12:58:11", "content": "This is correct, I hoped the area would give another signal for the model to predict nuclei size, but the impact was negligible.
", "rawMarkdown": "This is correct, I hoped the area would give another signal for the model to predict nuclei size, but the impact was negligible.", "votes": null }, { "id": "527151", "postDate": "05/04/2019 18:14:45", "content": "HI, what extra data did you use?
", "rawMarkdown": "HI, what extra data did you use?", "votes": null }, { "id": "527734", "postDate": "05/06/2019 07:24:13", "content": "We annotated a number of diverse external images, usually just a few images per type to validate how well model predicts the new types of images and add them to the training dataset
", "rawMarkdown": "We annotated a number of diverse external images, usually just a few images per type to validate how well model predicts the new types of images and add them to the training dataset", "votes": null } ], "comments": [ { "id": 317733, "author_name": "jliamfinnie", "author_url": "", "post_date": "04/22/2018 12:17:21", "content": "@Dmytro, thank you very much for sharing your approach, a fascinating read. I just read the DWT paper you mentioned (happened to be from the same University I went to), I wish I had known about this at the beginning of this competition! The idea of using different energy levels to help with object segmentation is pretty powerful, and is actually similar in concept to the method described in the original Unet paper:\nUnet research paper
\n\nThe original Unet paper gave the highest weight to the object border pixels (so, similar to energy level 1 from the DWT paper). However, DWT seems more powerful, since it has many more energy levels - and using different energy levels for the cut based on the object size sounds like it might be helpful for this competition too (did you try this approach, based on the average predicted nuclei size?).
\n\nOur U-Net ended up being pretty basic compared to your approach, but our team was still pretty happy with our results in this competition, since this was the first Kaggle competition for most of us (we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly).
\n\nThanks again for sharing!
", "votes": null, "replies": [ { "id": 317761, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "04/22/2018 13:47:08", "content": "Hi Liam,
\n\nI tried to use different thresholds using different watershed energy levels but overall it worked better to use predicted centers.
\n\nWhat is interesting, asking model to predict the vectors from the border like in the DWT paper improved the centers predictions. Allowing model to process such vectors further with a greater receptive field of the second unet, helped to improve predicted centers slightly more.
", "votes": null, "replies": [] }, { "id": 317892, "author_name": "cpmpml", "author_url": "", "post_date": "04/22/2018 19:19:20", "content": "> we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly
\n\nI don't want that people think our final score is just due to my ideas. My own solo submission scored way lower than our team submission.
", "votes": null, "replies": [] } ] }, { "id": 317743, "author_name": "cpmpml", "author_url": "", "post_date": "04/22/2018 12:45:13", "content": "Thanks for sharing, and congrats on the result!
\n\n\n\n\nAfter watershed transform from detected centers, check for missing large masks with the total center prediction > ~5.0, and add them.
\n
Can you explain this a bit more? How do you define 'missed masks' given you don't have the masks when predicting?
", "votes": null, "replies": [ { "id": 317751, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "04/22/2018 13:17:34", "content": "My algorithm to select nuclei centers as seeds for watershed may have missed some nuclei if it's of complex form or uncertain shape while the integral of predicted center over nuclei area is still close to expected value of 9.0
\n\nSuch cases were easy to detect and fix by looking for connected areas of the mask (I used one of predicted energy levels) not included into resulting set of instances masks after the watershed transform.
", "votes": null, "replies": [] }, { "id": 317760, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "04/22/2018 13:40:57", "content": "More detailed explanation of centers selection following post processing:
\n\nThis allowed to better detect nuclei separation, sometimes center prediction worked better and sometimes predicted watershed energy levels, this approach helped to combine both predictions.
\n\nListed steps helped but the improvement was not as huge, I'm talking about local CV or public LB score changes from 0.565 to 0.570 or so.
", "votes": null, "replies": [] }, { "id": 317765, "author_name": "cpmpml", "author_url": "", "post_date": "04/22/2018 13:56:36", "content": "Thanks a lot, it is crystal clear now.
", "votes": null, "replies": [] }, { "id": 317795, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/22/2018 15:19:18", "content": "@Dmytro Poplavskiy if you're interested in late submissions, you can try @CPMP's top secret post processing approach, random walker instead of watershed, that gave us an LB boost between 0.02~0.026 every time because it minimized contour erosion. just the algorithm is very slow, so we had to make our prediction code multi-threaded. And we did random walker on contours as well which is similar to the concept of your first energy level. If that works on your case too, it should improve your LB from 0.610 to 0.630 I hope :) Thanks a lot for sharing, very informative.
This is very interesting idea, and the very significant boost to result. Would be very interesting to read in more detail, do you plan to publish you approach (or maybe I have missed it)?
\n\nIt's also interesting if the random walker approach can be combined with the information about the estimated energy levels, it may work better than either one separately.
\n\n[edited] I missed this bit in your team result description, certainly very interesting approach.
", "votes": null, "replies": [] }, { "id": 317941, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/22/2018 22:00:38", "content": "@Dmytro, yes, @CPMP published our code last week and you can look at our post processing random walker code if you're interested - - we only replaced watershed with random walker and the rest of the code stayed exactly the same. That's why I was thinking it should be very easy for you to plug-in the approach on your 2nd step and 5th step. The significant improvement was in the mean IoU range between 0.8 to 0.95. With watershed, we got a very low score on the mean IoU above 0.85 due to the nature of watershed (contour erosion).
Hi, Nicole. Thank you for sharing. However, random walker code hyperlink doesn't work. Could you please show the link again?
", "votes": null, "replies": [] }, { "id": 320888, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/30/2018 06:01:51", "content": "@Zhen fixed the link
", "votes": null, "replies": [] } ] }, { "id": 317899, "author_name": "pliptor", "author_url": "", "post_date": "04/22/2018 19:42:32", "content": "Congratulations and thanks for sharing this clear presentation. What language/tool did you use to implement the x,y vector computing layer?
", "votes": null, "replies": [ { "id": 317934, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "04/22/2018 21:35:11", "content": "The model itself is written in keras, but to prepare vector labels I used scipy.ndimage.morphology
\n\nscipy.ndimage.morphology.distance_transform_edt() is very useful to find distance and direction to the border as it can return the index of the nearest pixel outside of mask:
\n\nedt, inds = scipy.ndimage.morphology.distance_transform_edt(crop_smooth, return_distances=True, return_indices=True)\nborder_vector = np.array([\n np.expand_dims(np.arange(0, rows), axis=1) - inds[0],\n np.expand_dims(np.arange(0, cols), axis=0) - inds[1]])\n\nborder_vector_norm = border_vector / (np.linalg.norm(border_vector, axis=0, keepdims=True) + 1e-5)\nres_crop[:, :, 0] = border_vector_norm[0]\nres_crop[:, :, 1] = border_vector_norm[1]\nVector to the center is easier to calculate, it's a difference of the pixel position and the mask center of mass:
\n\n center_of_mass = scipy.ndimage.measurements.center_of_mass(crop)\n current_offset_field = np.zeros((CROP_SIZE, CROP_SIZE, 2))\n current_offset_field[:, :, 0] = np.expand_dims(center_of_mass[0] - np.arange(0, CROP_SIZE), axis=1)\n current_offset_field[:, :, 1] = np.expand_dims(center_of_mass[1] - np.arange(0, CROP_SIZE), axis=0)\nThank you very much for the clarification. I saw the paper during the competition but didn't come up with a good way to implement it. It makes sense how your team did it.
", "votes": null, "replies": [] } ] }, { "id": 318012, "author_name": "chabir", "author_url": "", "post_date": "04/23/2018 03:19:30", "content": "Impressive work once again.
", "votes": null, "replies": [] }, { "id": 318660, "author_name": "iitrsamrat", "author_url": "", "post_date": "04/24/2018 08:20:24", "content": "Congrats @Dmytro Poplavskiy awesome work.
", "votes": null, "replies": [] }, { "id": 319151, "author_name": "lgmoneda", "author_url": "", "post_date": "04/25/2018 11:19:38", "content": "Thanks for sharing, Dmytro! :D
", "votes": null, "replies": [] }, { "id": 320782, "author_name": "thenickname", "author_url": "", "post_date": "04/29/2018 20:29:29", "content": "Great exploratory approach! I'm guessing also adding predictions for diagonal vectors, not just x and y, would add a slight additional boost.
\n\nI think, one of the things W-nets (double-U-Nets, or some other multiple of them, of course) should add vs just going deeper in a U-net is a better positional accuracy of features. I also tried a W-net, though it's a bit tricky, especially the way I sometimes tried it, putting it as a residual. Sometimes it just exploded. I'm guessing it might be a good idea to try later in a segmentation task to make a multibranching net with nodes having branches that go upscale, downscale and same scale. Sort of combining U-nets and sub-U-nets going a different depth from different depth and merging them at different depths.
", "votes": null, "replies": [] }, { "id": 321050, "author_name": "sasrdw", "author_url": "", "post_date": "04/30/2018 13:30:52", "content": "I'm really proud of my teammate Dmytro for this very innovative and detailed approach. We are thinking \"W-Net\" is indeed a better name, especially since the English letter \"W\" is pronounced \"Double U\" and it appears compatible with https://arxiv.org/abs/1711.08506. With watershed in there too we might should even call it \"W2-Net\", but \"W2\" is also the name of the wages/tax form in the U.S. and we don't want to go anywhere near that!
", "votes": null, "replies": [ { "id": 321161, "author_name": "selimsef", "author_url": "", "post_date": "04/30/2018 18:34:17", "content": "Totally agree! I was really surprised that such a good result could be achieved with a model trained from scratch.
", "votes": null, "replies": [] }, { "id": 321179, "author_name": "dskswu", "author_url": "", "post_date": "04/30/2018 19:22:26", "content": "Congrats! The name is very catchy and clever.
", "votes": null, "replies": [] } ] }, { "id": 321852, "author_name": "bhavya5491", "author_url": "", "post_date": "05/02/2018 02:55:10", "content": "Nice work...
", "votes": null, "replies": [] }, { "id": 323980, "author_name": "andyzhang50", "author_url": "", "post_date": "05/06/2018 20:32:51", "content": "Hi Dmytro, great job and thanks for sharing your methods. You decribed your first approach as:\n\"The first approach was to predict using the single UNet model:\nMask, BCE + DICE loss\nNuclei centers with 3x3 patches around the center of mass as a training label, BCE loss\nArea of nuclei used to normalize loss from vectors for large and small nuclei\nX,Y of vector to the center of nuclei, MSE loss normalized by nuclei area\"
\n\nMy questions is how, did you predict the area of nuclei? What were your training labels? Were they masks that contained the numeral value or their respective areas?
\n\nThank you for your time
", "votes": null, "replies": [ { "id": 326243, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "05/09/2018 13:28:07", "content": "Hi Andy,
\n\nI predicted the per pixel area value as one of UNET output layers using mae as a loss, but the main reason for using area was to normalize the loss of vector predicted to nuclei center, otherwise the mistake of center vector prediction for large nuclei would be penalized much more comparing to small nuclei.
\n\nI have not seen significant improvements from making the model to predict the area alone.
", "votes": null, "replies": [] } ] }, { "id": 324169, "author_name": "vicky25890", "author_url": "", "post_date": "05/07/2018 10:06:47", "content": "Hi, Your post is good. Its so useful for me about Data Science.\nData Science Training in Bangalore
", "votes": null, "replies": [] }, { "id": 371696, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "08/17/2018 13:01:33", "content": "Source code for our solution is available at github: https://github.com/pdima/kaggle_2018_data_science_bowl_solution
", "votes": null, "replies": [] }, { "id": 374822, "author_name": "sheriytm", "author_url": "", "post_date": "08/23/2018 23:14:27", "content": "Congratulations @Dmytro Poplavskiy, thanks for sharing your solution and code.
", "votes": null, "replies": [] }, { "id": 416501, "author_name": "arc144", "author_url": "", "post_date": "11/06/2018 18:49:03", "content": "Hi @dmytropoplavskiy, congratulations for the great work! Please, correct me if I'm wrong, in the end (final model) you did not need the area prediction, right? Because you were already predicting the normalized angle vectors like the DWT paper, correct?
\n\nSo you only need mask, eroded mask and centers for watershed (the angle vector helped indirectly from what I understood), is this correct?
\n\nThank you
", "votes": null, "replies": [ { "id": 418190, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "11/09/2018 12:58:11", "content": "This is correct, I hoped the area would give another signal for the model to predict nuclei size, but the impact was negligible.
", "votes": null, "replies": [] } ] }, { "id": 527151, "author_name": "alexvaladao", "author_url": "", "post_date": "05/04/2019 18:14:45", "content": "HI, what extra data did you use?
", "votes": null, "replies": [ { "id": 527734, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "05/06/2019 07:24:13", "content": "We annotated a number of diverse external images, usually just a few images per type to validate how well model predicts the new types of images and add them to the training dataset
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "317711": "In our team we have evaluated both UNet and Mask-RCNN based solution, but for us Unet worked significantly better so we used Unet based model for submission.\n\nThe UNet based solution is inspired by the Deep Watershed Transform paper: https://arxiv.org/pdf/1611.08303.pdf\n\n\nFor each pixels, we predicted the x,y components of vector pointing from the instance border like described in the DWT paper and predicted the mask, watershed levels and nuclei centers using the second connected UNet. \n\nAttached the diagram with our model\n\n![Model description][1]\n\nOverall Unet predict the mask pretty well but it was necessary to find a way to reliably segment nuclei.\n\nApproaches we tried\n================\n\nThe first approach was to predict using the single UNet model:\n\n * Mask, BCE + DICE loss\n * Nuclei centers with 3x3 patches around the center of mass as a training label, BCE loss\n * Area of nuclei used to normalize loss from vectors for large and small nuclei\n * X,Y of vector to the center of nuclei, MSE loss normalized by nuclei area\n\n\nFor touching nuclei the vector value to the centers changes sign, so it changes sharply and the loss is the biggest on the nuclei border which forces model to learn to separate instances. The postprocessing was quite straightforward:\n\n 1. Find the centers of nuclei using predicted centers output, expecting the area of each prediction to be approx 9.0 (matring area of 3x3 training patch)\n 2. For each pixel in predicted masc, assign it to the cluster nearest to position predicted vector to the center points to.\n\nThis approach worked and scored over 0.5 on the public leaderboard.\n\n\nImprovement:\n-----------------\n\nThe second approach was to try the deep watershed transform idea to predict watershed energy levels (mask eroded by different offsets) by predicting intermediate unit vector fields pointing from the nearest border pixel. Instead of training 3 independent models for segmentation, vector field and watershed energy predictions I used the single UNet to predict everything. With the seeds we used the continuous areas after applying the threshold over the sum of energy levels, in the similar way to the DWT paper. The result was slightly better.\n\nImprovement:\n-----------------\n\nUse the predicted centers as seeds for watershed transform instead of the energy level with threshold. The score improvement was more significant comparing to switching to DWT.\n\nImprovement:\n-----------------\n\nPredict only vector fields with the first UNet, concatenated predicted fields with the last layer used to predictions of all the other fields using another UNet. This has little to no impact on predicted masks and the energy levels but helped to significantly improve the quality of predicted nuclei centers. \n\nImprovement:\n-----------------\nSimple mean ensembling of 8 TTA flips/rotations had very small but consistent improvement.\n\nImprovement:\n-----------------\n\nBetter postprocessing, the idea - since for prediction the center values we are using the 3x3 path of area 9, we can expect the total area of predicted center to be close to 9 as well, even for cases of complex connected nuclei hard to predict. \nThis allowed to do following post processing improvements:\n\n * After watershed transform from detected centers, check for missing large masks with the total center prediction > ~5.0, and add them.\n * Instances with the total center value integral > 9.0 * 1.5 are most likely have two nuclei connected but with very hard to predict centers. Split centers to two clusters using KNN and re-run watershed.\n * We tried the similar approach to topcoder team of estimating IOU using information about the shape and other predicted instance properties and run optimiser for the score to decide if particular instance should be included to submission or not, but the quality of IOU prediction was not sufficient for this to work reliably. Instead we calculated IOU between masks of the ensemble and individual TTA variants. If the median IOU is low, it means different models predicted different shapes and it’s better to exclude such mask from submission to avoid penalty at high IOU thresholds.\n\n\nExtra data used\n-------------------\n\nWe added extra annotated datasets listed in the forum thread and annotated some images, mostly color histology images. We also used synthetic data to generate more cases of touching and overlapped nuclei.\n\nFinal model details\n==============\n\nAttached an example of the early model predictions, including predicted vectors, centers and energy levels.\n\n![Example prediction][2]\n\nWe tried to use the imagenet pretrained models as UNet encoders but the result was the same or worse comparing to UNet trained from scratch.\n\nFor unet encoders we used \nConv2d - BN - Relu - Conv2d - Relu\nDecoders:\nUpsample/concatenate - Conv2d - Relu\n\nWe used quite a large number of filters (64/128) even on the high resolution levels as model had to predict many outputs but increased number of filters less than twice as we did not have as much complex semantic informations as with models trained on imagenet.\n\nUnet1 was 6 levels deep (decreased the resolution 2x 6 times) while Unet2 was 4 levels deep. \n\nAs input we used B/W image or H channel of HED stain decomposition. H worked slightly better for histology images but slightly worse for some other stains. For the final submission we ensembled 4 models trained on B/W images and 2 on H channel.\n\nWe trained on 256x256 patches with significant level of augmentations and predicted on 1024 pixels tiles with 128 pix overlap on each side. We used the “SAME” padding and relied on the large tiles padding and overlap to avoid corner effects.\n\nAt the end of competition we tried to train a model predicting only data used for post processing without vector fields and the result was significantly worse. Significantly reducing the loss of vectors to the center of nuclei had a little impact to the results, so most likely vectors from corners are more important. Even while not directly used, it helped to better predict centers and watershed energy levels.\n\n[Solution source code][3]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/317711/9258/Vector%20unet.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/317711/9259/nuclei_descr.png\n [3]: https://github.com/pdima/kaggle_2018_data_science_bowl_solution", "317733": "Dmytro, thank you very much for sharing your approach, a fascinating read. I just read the DWT paper you mentioned (happened to be from the same University I went to), I wish I had known about this at the beginning of this competition! The idea of using different energy levels to help with object segmentation is pretty powerful, and is actually similar in concept to the method described in the original Unet paper:\n[Unet research paper][1]\n\nThe original Unet paper gave the highest weight to the object border pixels (so, similar to energy level 1 from the DWT paper). However, DWT seems more powerful, since it has many more energy levels - and using different energy levels for the cut based on the object size sounds like it might be helpful for this competition too (did you try this approach, based on the average predicted nuclei size?).\n\nOur U-Net ended up being pretty basic compared to your approach, but our team was still pretty happy with our results in this competition, since this was the first Kaggle competition for most of us (we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly).\n\nThanks again for sharing!\n\n [1]: http://www.cs.cmu.edu/~jeanoh/16-785/papers/ronnenberger-miccai2015-u-net.pdf", "317743": "Thanks for sharing, and congrats on the result!\n\n> After watershed transform from detected centers, check for missing large masks with the total center prediction > ~5.0, and add them.\n\nCan you explain this a bit more? How do you define 'missed masks' given you don't have the masks when predicting?", "317751": "My algorithm to select nuclei centers as seeds for watershed may have missed some nuclei if it's of complex form or uncertain shape while the integral of predicted center over nuclei area is still close to expected value of 9.0\n\nSuch cases were easy to detect and fix by looking for connected areas of the mask (I used one of predicted energy levels) not included into resulting set of instances masks after the watershed transform.", "317760": "More detailed explanation of centers selection following post processing:\n\n1. For each peak in centers prediction, I tried to find which radius circle has sufficient sum of predicted values (this threshold a a function from radius was an important hyperparameter to tune). I then added the \npeak to the list of center seeds and cleared the circle, so it does not affect calculation of next centers.\n2. Run watershed using the sum of centers predictions and watershed energy levels as an energy and the first energy level as a mask. Using the first energy level instead of actual mask here helped in a few cases to separate touching nuclei if center prediction failed for one of them.\n3. As described above, I have checked if any of large enough continuous masks in energy level 1 does not have seed but has sufficient total sum of centers prediction, added center of mass as a new seed.\n4. Split some nuclei to two seeds using KNN if total integral of predicted center > ~15.0\n5. Re run watershed using the actual predicted mask and all found center seeds\n\nThis allowed to better detect nuclei separation, sometimes center prediction worked better and sometimes predicted watershed energy levels, this approach helped to combine both predictions.\n\nListed steps helped but the improvement was not as huge, I'm talking about local CV or public LB score changes from 0.565 to 0.570 or so.", "317761": "Hi Liam,\n\nI tried to use different thresholds using different watershed energy levels but overall it worked better to use predicted centers.\n\nWhat is interesting, asking model to predict the vectors from the border like in the DWT paper improved the centers predictions. Allowing model to process such vectors further with a greater receptive field of the second unet, helped to improve predicted centers slightly more.", "317765": "Thanks a lot, it is crystal clear now.", "317795": "Dmytro Poplavskiy if you're interested in late submissions, you can try @CPMP's top secret post processing approach, `random walker` instead of watershed, that gave us an LB boost between 0.02~0.026 every time because it minimized contour erosion. just the algorithm is very slow, so we had to make our prediction code multi-threaded. And we did random walker on contours as well which is similar to the concept of your first energy level. If that works on your case too, it should improve your LB from 0.610 to 0.630 I hope :) Thanks a lot for sharing, very informative.", "317892": "> we teamed with @CPMP in the final 3 days, who shared many great ideas for improvements that we ran out of time to implement, sadly\n\nI don't want that people think our final score is just due to my ideas. My own solo submission scored way lower than our team submission.", "317899": "Congratulations and thanks for sharing this clear presentation. What language/tool did you use to implement the x,y vector computing layer?", "317934": "The model itself is written in keras, but to prepare vector labels I used scipy.ndimage.morphology\n\nscipy.ndimage.morphology.distance_transform_edt() is very useful to find distance and direction to the border as it can return the index of the nearest pixel outside of mask:\n\n edt, inds = scipy.ndimage.morphology.distance_transform_edt(crop_smooth, return_distances=True, return_indices=True)\n border_vector = np.array([\n np.expand_dims(np.arange(0, rows), axis=1) - inds[0],\n np.expand_dims(np.arange(0, cols), axis=0) - inds[1]])\n\n border_vector_norm = border_vector / (np.linalg.norm(border_vector, axis=0, keepdims=True) + 1e-5)\n res_crop[:, :, 0] = border_vector_norm[0]\n res_crop[:, :, 1] = border_vector_norm[1]\n\n \nVector to the center is easier to calculate, it's a difference of the pixel position and the mask center of mass:\n\n\n center_of_mass = scipy.ndimage.measurements.center_of_mass(crop)\n current_offset_field = np.zeros((CROP_SIZE, CROP_SIZE, 2))\n current_offset_field[:, :, 0] = np.expand_dims(center_of_mass[0] - np.arange(0, CROP_SIZE), axis=1)\n current_offset_field[:, :, 1] = np.expand_dims(center_of_mass[1] - np.arange(0, CROP_SIZE), axis=0)", "317937": "This is very interesting idea, and the very significant boost to result. Would be very interesting to read in more detail, do you plan to publish you approach (or maybe I have missed it)?\n\nIt's also interesting if the random walker approach can be combined with the information about the estimated energy levels, it may work better than either one separately.\n\n[edited] I missed this bit in your team result description, certainly very interesting approach.", "317941": "Dmytro, yes, @CPMP published our code last week and you can look at our post processing [random walker code][1] if you're interested - - we only replaced `watershed` with `random walker` and the rest of the code stayed exactly the same. That's why I was thinking it should be very easy for you to plug-in the approach on your 2nd step and 5th step. The significant improvement was in the mean IoU range between `0.8` to `0.95`. With watershed, we got a very low score on the mean IoU above `0.85` due to the nature of watershed (contour erosion). \n\n [1]: https://github.com/nicolefinnie/kaggle-dsb2018/blob/master/src/modules/image_processing.py#L178", "317944": "Thank you very much for the clarification. I saw the paper during the competition but didn't come up with a good way to implement it. It makes sense how your team did it.", "318012": "Impressive work once again.", "318660": "Congrats @Dmytro Poplavskiy awesome work.", "319151": "Thanks for sharing, Dmytro! :D", "320782": "Great exploratory approach! I'm guessing also adding predictions for diagonal vectors, not just x and y, would add a slight additional boost.\n\nI think, one of the things W-nets (double-U-Nets, or some other multiple of them, of course) should add vs just going deeper in a U-net is a better positional accuracy of features. I also tried a W-net, though it's a bit tricky, especially the way I sometimes tried it, putting it as a residual. Sometimes it just exploded. I'm guessing it might be a good idea to try later in a segmentation task to make a multibranching net with nodes having branches that go upscale, downscale and same scale. Sort of combining U-nets and sub-U-nets going a different depth from different depth and merging them at different depths.", "320824": "Hi, Nicole. Thank you for sharing. However, random walker code hyperlink doesn't work. Could you please show the link again?", "320888": "Zhen fixed the [link][1]\n\n\n [1]: https://github.com/nicolefinnie/kaggle-dsb2018/blob/master/src/modules/image_processing.py#L178", "321050": "I'm really proud of my teammate Dmytro for this very innovative and detailed approach. We are thinking \"W-Net\" is indeed a better name, especially since the English letter \"W\" is pronounced \"Double U\" and it appears compatible with [https://arxiv.org/abs/1711.08506][1]. With watershed in there too we might should even call it \"W2-Net\", but \"W2\" is also the name of the wages/tax form in the U.S. and we don't want to go anywhere near that! \n\n\n [1]: https://arxiv.org/abs/1711.08506", "321161": "Totally agree! I was really surprised that such a good result could be achieved with a model trained from scratch.", "321179": "Congrats! The name is very catchy and clever.", "321852": "Nice work...", "323980": "Hi Dmytro, great job and thanks for sharing your methods. You decribed your first approach as:\n\"The first approach was to predict using the single UNet model:\nMask, BCE + DICE loss\nNuclei centers with 3x3 patches around the center of mass as a training label, BCE loss\n**Area of nuclei** used to normalize loss from vectors for large and small nuclei\nX,Y of vector to the center of nuclei, MSE loss normalized by nuclei area\"\n\nMy questions is how, did you predict the area of nuclei? What were your training labels? Were they masks that contained the numeral value or their respective areas?\n\nThank you for your time", "324169": "Hi, Your post is good. Its so useful for me about Data Science.\n[Data Science Training in Bangalore][1]\n\n\n [1]: https://www.besanttechnologies.com/training-courses/data-science-training-in-bangalore", "326243": "Hi Andy,\n\nI predicted the per pixel area value as one of UNET output layers using mae as a loss, but the main reason for using area was to normalize the loss of vector predicted to nuclei center, otherwise the mistake of center vector prediction for large nuclei would be penalized much more comparing to small nuclei.\n\nI have not seen significant improvements from making the model to predict the area alone.", "371696": "Source code for our solution is available at github: https://github.com/pdima/kaggle_2018_data_science_bowl_solution", "374822": "Congratulations @Dmytro Poplavskiy, thanks for sharing your solution and code.", "416501": "Hi @dmytropoplavskiy, congratulations for the great work! Please, correct me if I'm wrong, in the end (final model) you did not need the area prediction, right? Because you were already predicting the normalized angle vectors like the DWT paper, correct?\n\nSo you only need mask, eroded mask and centers for watershed (the angle vector helped indirectly from what I understood), is this correct?\n\nThank you", "418190": "This is correct, I hoped the area would give another signal for the model to predict nuclei size, but the impact was negligible.", "527151": "HI, what extra data did you use?", "527734": "We annotated a number of diverse external images, usually just a few images per type to validate how well model predicts the new types of images and add them to the training dataset" }, "source": "existing_discussion", "complete": true, "competition_id": 8089, "competition_slug": "data-science-bowl-2018", "team_id": 1270068, "solution_url": "/competitions/data-science-bowl-2018/writeups/nuclear-vision-our-solution-4th-place-on-the-priva" }