![\"Prediction\"](\"https://www.dropbox.com/s/4igam47pqg0i82q/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37_predict.png?dl=1\")
Combined these tricks allowed to achieve 1st place on public LB even with a single model.
\n\nWe had experience in segmentation challenges before (Carvana, Urban 3D, Spacenet, Konica Minolta) where we actually competed against each other and therefore the choice was clear - UNet on steroids.
\n\nFirst we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). \nThis approach gave us around ~500 on Public LB. Clearly this is not enough to win.
\n\nThen we added second channel with contours, where width of contours depended on the nucleus size. \nThese masks + simple watershed postprocessing gave us ~525, not a breakthrough but a hint to the right direction.
\n\nLooking at the mistakes it was clear that networks easily predict contours in non ambiguous places and fail miserably in places where we actually need contours to separate the nuclei. Hence we decided to predict only the borders between the cells. This easily gave us 550+ on the LB with a single network results + watershed postprocessing.
\n\nStill if we have a full mask in one channel and a border in another, sometimes the seeds for watershed are not good enough. A better approach is to change nuclei masks and make pixels empty on the borders. This also allows to use softmax as the target activation instead of sigmoid. This better separates nuclei but actually decreases MaP because of high thresholds for IoU. We solved this issue with additional networks trained on full masks and combined the results in the postprocessing step.
\n\nFinal approach for targets:
\n\n2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)
2 channels full masks i.e. (mask, border)
For c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37 it looks like this\n
Final result after post processing:\n![\"Final](\"https://www.dropbox.com/s/isbils3jtelj297/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37.png?dl=1\")
As there were just some hundreds of training images, we needed to come up with specific augmentations to prevent our models from overfitting and make them more or less generalizable.\nWe used a lot of heavy augmentations (maybe too heavy)
\n\nWe used UNet like encoder-decoder architectures with encoders pretrained on ImageNet.
\n\nSurprisingly, simple encoders like VGG16 did not work in this competition at all. They failed on the hard cases when the tissue looked like nucleus but it was not, especially on color images like 59b35151d4a7a5ffdd7ab7f171b142db8cfe40beeee67277fac6adca4d042c4
\n\nAfter these experiments we decided that we have to go deeper!!!\nAs a result - top performing encoders in this competition were : DPN-92, Resnet-152, InceptionResnetV2, Resnet101.
\n\nLightGBM models trained on predicted nucleus candidates. Each base candidate selected with lowest threshold for separation and tried to be separated with few higher thresholds and erosion. Used few basic morphological features about candidate like solidity, circularity, convexity, area, neighbors median area, count, etc.. Target for prediction - iou with ground truth (0 if iou < 0.5).
\n\nThen best separation threshold selected for each candidate according to predicted iou. Candidates with small predicted iou just removed (iou < 0.3 and iou < 0.2 for 2 submissions - it was hard to find this threshold using OOF prediction, because there is small overfit to image types even with such hard augmentation).
\n\nWe added some images from janowczyk, nucleisegmentationbenchmark, isbi2009, BBBC020, TNBC datasets. Some of them decreased our score on Public LB, making less confident predictions on color images. Unfortunately we did not have time to find the culprits and left it as is.
\n\nWe used a simple approach for ensembling where we just averaged our masks before postprocessing
\n\nRandom Crops: 256x256
Batch Size: 16
optimizer: Adam
learning rate: initial 1e-4 with decay (we had different LR policies, but mostly small LR no more than 1e-4)
preprocessing: same as on ImageNet depending on network
For networks with sigmoid activation and 2 channel masks we used combination of binary_crossentropy with soft_dice per channel. For networks with softmax activation and 3 channel masks we used combination of categorical_crossentropy with soft_dice per channel (soft dice was applied only to mask and border channels).
\n\nStandard Flips/Rotations(0,90,180,270).
\n\nhttps://github.com/selimsef/dsb2018_topcoders/
", "messages": [ { "id": 315536, "postDate": "2018-04-17T08:33:46.713Z", "content": "Combined these tricks allowed to achieve 1st place on public LB even with a single model.
\n\nWe had experience in segmentation challenges before (Carvana, Urban 3D, Spacenet, Konica Minolta) where we actually competed against each other and therefore the choice was clear - UNet on steroids.
\n\nFirst we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). \nThis approach gave us around ~500 on Public LB. Clearly this is not enough to win.
\n\nThen we added second channel with contours, where width of contours depended on the nucleus size. \nThese masks + simple watershed postprocessing gave us ~525, not a breakthrough but a hint to the right direction.
\n\nLooking at the mistakes it was clear that networks easily predict contours in non ambiguous places and fail miserably in places where we actually need contours to separate the nuclei. Hence we decided to predict only the borders between the cells. This easily gave us 550+ on the LB with a single network results + watershed postprocessing.
\n\nStill if we have a full mask in one channel and a border in another, sometimes the seeds for watershed are not good enough. A better approach is to change nuclei masks and make pixels empty on the borders. This also allows to use softmax as the target activation instead of sigmoid. This better separates nuclei but actually decreases MaP because of high thresholds for IoU. We solved this issue with additional networks trained on full masks and combined the results in the postprocessing step.
\n\nFinal approach for targets:
\n\n2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)
2 channels full masks i.e. (mask, border)
For c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37 it looks like this\n
Final result after post processing:\n
As there were just some hundreds of training images, we needed to come up with specific augmentations to prevent our models from overfitting and make them more or less generalizable.\nWe used a lot of heavy augmentations (maybe too heavy)
\n\nWe used UNet like encoder-decoder architectures with encoders pretrained on ImageNet.
\n\nSurprisingly, simple encoders like VGG16 did not work in this competition at all. They failed on the hard cases when the tissue looked like nucleus but it was not, especially on color images like 59b35151d4a7a5ffdd7ab7f171b142db8cfe40beeee67277fac6adca4d042c4
\n\nAfter these experiments we decided that we have to go deeper!!!\nAs a result - top performing encoders in this competition were : DPN-92, Resnet-152, InceptionResnetV2, Resnet101.
\n\nLightGBM models trained on predicted nucleus candidates. Each base candidate selected with lowest threshold for separation and tried to be separated with few higher thresholds and erosion. Used few basic morphological features about candidate like solidity, circularity, convexity, area, neighbors median area, count, etc.. Target for prediction - iou with ground truth (0 if iou < 0.5).
\n\nThen best separation threshold selected for each candidate according to predicted iou. Candidates with small predicted iou just removed (iou < 0.3 and iou < 0.2 for 2 submissions - it was hard to find this threshold using OOF prediction, because there is small overfit to image types even with such hard augmentation).
\n\nWe added some images from janowczyk, nucleisegmentationbenchmark, isbi2009, BBBC020, TNBC datasets. Some of them decreased our score on Public LB, making less confident predictions on color images. Unfortunately we did not have time to find the culprits and left it as is.
\n\nWe used a simple approach for ensembling where we just averaged our masks before postprocessing
\n\nRandom Crops: 256x256
Batch Size: 16
optimizer: Adam
learning rate: initial 1e-4 with decay (we had different LR policies, but mostly small LR no more than 1e-4)
preprocessing: same as on ImageNet depending on network
For networks with sigmoid activation and 2 channel masks we used combination of binary_crossentropy with soft_dice per channel. For networks with softmax activation and 3 channel masks we used combination of categorical_crossentropy with soft_dice per channel (soft dice was applied only to mask and border channels).
\n\nStandard Flips/Rotations(0,90,180,270).
\n\nhttps://github.com/selimsef/dsb2018_topcoders/
", "rawMarkdown": "**Main contributions**\n------------------\n\n - targets - we predict touching borders along with the masks to solve\n the problem as instance segmentation\n - loss function - that combines crossentropy and soft dice loss in such\n a way that pixel imbalance doesn't affect the results\n - very deep encoder-decoder architectures that also achieve\n state-of-the-art results in other binary segmentation problems\n (SpaceNet, Inria and others)\n - tricky postprocessing that combines watershed, morphological features\n and second-level model with Gradient Boosted Trees\n - task specific data augmentations\n\n \nCombined these tricks allowed to achieve 1st place on public LB even with a single model.\n\n\n**U-Net vs Mask-RCNN?**\n-----------------------\n\nWe had experience in segmentation challenges before (Carvana, Urban 3D, Spacenet, Konica Minolta) where we actually competed against each other and therefore the choice was clear - **UNet on steroids**. \n\n\n**Target masks**\n-----------------\n\nFirst we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). \nThis approach gave us around ~500 on Public LB. Clearly this is not enough to win. \n\nThen we added second channel with contours, where width of contours depended on the nucleus size. \nThese masks + simple watershed postprocessing gave us ~525, not a breakthrough but a hint to the right direction.\n\nLooking at the mistakes it was clear that networks easily predict contours in non ambiguous places and fail miserably in places where we actually need contours to separate the nuclei. Hence we decided to predict only the borders between the cells. This easily gave us 550+ on the LB with a single network results + watershed postprocessing. \n\nStill if we have a full mask in one channel and a border in another, sometimes the seeds for watershed are not good enough. A better approach is to change nuclei masks and make pixels empty on the borders. This also allows to use softmax as the target activation instead of sigmoid. This better separates nuclei but actually decreases MaP because of high thresholds for IoU. We solved this issue with additional networks trained on full masks and combined the results in the postprocessing step.\n\n**Final approach for targets:**\n\n - 2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)\n\n - 2 channels full masks i.e. (mask, border)\n\nFor c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37 it looks like this\n![Prediction][1]\n\nFinal result after post processing:\n![Final result][2]\n\n\n\n**Augmentations**\n-----------------\n\nAs there were just some hundreds of training images, we needed to come up with specific augmentations to prevent our models from overfitting and make them more or less generalizable.\nWe used a lot of heavy augmentations (maybe too heavy)\n\n - Clahe, Sharpen, Emboss\n - Gaussian Noise\n - Color to Gray\n - Inverting - we should not have used it, some images were not\n predicted correctly on stage2 because of this augmentation\n - Remapping grayscale images to random color images\n - Blur, Median Blur, Motion Blur\n - contrast and brightness\n - random scale, rotates and flips\n - Heavy geometric transformations: Elastic Transform, Perspective\n Transform, Piecewise Affine transforms, pincushion distortion\n - Random HSV\n - Channel shuffle - I guess this one was very important due to the\n nature of the data\n - Nucleus copying on images. That created a lot of overlapping nuclei.\n It seemed to help networks to learn better borders for overlapping\n nuclei.\n\n**Network architectures**\n-------------------------\n\nWe used UNet like encoder-decoder architectures with encoders pretrained on ImageNet.\n\nSurprisingly, simple encoders like VGG16 did not work in this competition at all. They failed on the hard cases when the tissue looked like nucleus but it was not, especially on color images like 59b35151d4a7a5ffdd7ab7f171b142db8cfe40beeee67277fac6adca4d042c4\n\nAfter these experiments we decided that ***we have to go deeper***!!!\nAs a result - top performing encoders in this competition were : DPN-92, Resnet-152, InceptionResnetV2, Resnet101. \n\n**2nd Level model / Postprocessing**\n------------------------------------\n\nLightGBM models trained on predicted nucleus candidates. Each base candidate selected with lowest threshold for separation and tried to be separated with few higher thresholds and erosion. Used few basic morphological features about candidate like solidity, circularity, convexity, area, neighbors median area, count, etc.. Target for prediction - iou with ground truth (0 if iou < 0.5). \n\nThen best separation threshold selected for each candidate according to predicted iou. Candidates with small predicted iou just removed (iou < 0.3 and iou < 0.2 for 2 submissions - it was hard to find this threshold using OOF prediction, because there is small overfit to image types even with such hard augmentation).\n\n**Additional data:**\n--------------------\n\nWe added some images from janowczyk, nucleisegmentationbenchmark, isbi2009, BBBC020, TNBC datasets. Some of them decreased our score on Public LB, making less confident predictions on color images. Unfortunately we did not have time to find the culprits and left it as is. \n\n**Ensembling**\n--------------\n\nWe used a simple approach for ensembling where we just averaged our masks before postprocessing\n\n**Training**\n------------\n\n - Random Crops: 256x256\n\n - Batch Size: 16\n\n - optimizer: Adam\n\n - learning rate: initial 1e-4 with decay (we had different LR policies, but mostly small LR no more than 1e-4)\n\n - preprocessing: same as on ImageNet depending on network\n\n**Loss function**\n-----------------\n\n For networks with sigmoid activation and 2 channel masks we used combination of binary_crossentropy with soft_dice per channel. For networks with softmax activation and 3 channel masks we used combination of categorical_crossentropy with soft_dice per channel (soft dice was applied only to mask and border channels).\n\n**Test Time Augmentations (TTA)**\n-----------------------------\n\nStandard Flips/Rotations(0,90,180,270). \n\n [1]: https://www.dropbox.com/s/4igam47pqg0i82q/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37_predict.png?dl=1\n [2]: https://www.dropbox.com/s/isbils3jtelj297/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37.png?dl=1\n\n**GitHub repo**\n-----------------------------\n https://github.com/selimsef/dsb2018_topcoders/", "votes": 368 }, { "id": 315540, "postDate": "2018-04-17T08:56:09.843Z", "content": "Thanks for a great report! Which losses did you use and which optimizer? Any tricks with learning rate schedule? Which crop size did you use? How important/heavy were the scale augmentations? Did you use all augmentations at test time as well?
", "rawMarkdown": "Thanks for a great report! Which losses did you use and which optimizer? Any tricks with learning rate schedule? Which crop size did you use? How important/heavy were the scale augmentations? Did you use all augmentations at test time as well?", "votes": 11, "replies": [ { "id": 315587, "postDate": "2018-04-17T10:13:51.547Z", "content": "Thanks! Good point! Added training description and TTA to the post.\nDuring training random scale was in [0.55, 1.45]. Whenever we reduced augmentations we got better validation results and worse score on public leaderboard.
", "rawMarkdown": "Thanks! Good point! Added training description and TTA to the post.\nDuring training random scale was in [0.55, 1.45]. Whenever we reduced augmentations we got better validation results and worse score on public leaderboard.", "votes": 6 }, { "id": 315589, "postDate": "2018-04-17T10:21:54.257Z", "content": "I've also used 3 scales for TTA (0.75, 1, 1.25) * 2 flips * 4 rotations = 24 total
", "rawMarkdown": "I've also used 3 scales for TTA (0.75, 1, 1.25) * 2 flips * 4 rotations = 24 total", "votes": 6 }, { "id": 315605, "postDate": "2018-04-17T10:41:46.550Z", "content": "Thanks! Congrats on your well deserved win!
", "rawMarkdown": "Thanks! Congrats on your well deserved win!", "votes": 5 } ] }, { "id": 319237, "postDate": "2018-04-25T15:07:17.893Z", "content": "Fantastic work. This was my first kaggle competition. Like many others, I too started with UNet but gave up as I couldn't improve the score after a while. Your post explains how kaggle experience, domain knowledge and collaborative thinking helps to get past mediocre scores with known networks. Thanks for the post.
", "rawMarkdown": "Fantastic work. This was my first kaggle competition. Like many others, I too started with UNet but gave up as I couldn't improve the score after a while. Your post explains how kaggle experience, domain knowledge and collaborative thinking helps to get past mediocre scores with known networks. Thanks for the post. ", "votes": 9 }, { "id": 315954, "postDate": "2018-04-17T21:42:18.813Z", "content": "Bravo, awesome work topcoders! Really great to see building on knowledge from previous competitions then smart adaptations to this one.
", "rawMarkdown": "Bravo, awesome work topcoders! Really great to see building on knowledge from previous competitions then smart adaptations to this one. ", "votes": 9, "replies": [ { "id": 315962, "postDate": "2018-04-17T22:11:23.640Z", "content": "Highly interested in what you guys did as well!
", "rawMarkdown": "Highly interested in what you guys did as well!", "votes": 3 } ] }, { "id": 315592, "postDate": "2018-04-17T10:25:47.407Z", "content": "\"2nd Level model / Postprocessing\"
\n\nI think this is important. Rejecting false positive can boost your score by quite a lot! Is there any comparison results with and without post processing?
\n\nGood work!
", "rawMarkdown": "\"2nd Level model / Postprocessing\"\n\nI think this is important. Rejecting false positive can boost your score by quite a lot! Is there any comparison results with and without post processing?\n\nGood work!", "votes": 5, "replies": [ { "id": 315611, "postDate": "2018-04-17T10:49:59.167Z", "content": "Hi @Heng, \nThanks!\nIt is hard to estimate the gain on stage2 data. On Public LB this 2nd level model with adaptive thresholds and FP rejection increased the score by ~15.
", "rawMarkdown": "Hi @Heng, \nThanks!\nIt is hard to estimate the gain on stage2 data. On Public LB this 2nd level model with adaptive thresholds and FP rejection increased the score by ~15.", "votes": 2 } ] }, { "id": 315541, "postDate": "2018-04-17T08:57:08.637Z", "content": "Wow, thanks for sharing. Congratulations on the solution and the prize win! Can you expand a bit more on your use of lgb? Also, how do you compute predicted IoU?
", "rawMarkdown": "Wow, thanks for sharing. Congratulations on the solution and the prize win! Can you expand a bit more on your use of lgb? Also, how do you compute predicted IoU?", "votes": 4, "replies": [ { "id": 315547, "postDate": "2018-04-17T09:04:37.497Z", "content": "lgb just trained as regressor to predict IoU. So, we have nucleus candidate after Neural Networks prediction. We extracting some features and predicting IoU. I've used this method in Urban3D competition, but predicted binary target: 0 for FP, 1 for TP candidate. Here predicting IoU was more flexible for current MaP metric.
", "rawMarkdown": "lgb just trained as regressor to predict IoU. So, we have nucleus candidate after Neural Networks prediction. We extracting some features and predicting IoU. I've used this method in Urban3D competition, but predicted binary target: 0 for FP, 1 for TP candidate. Here predicting IoU was more flexible for current MaP metric.", "votes": 7 }, { "id": 315552, "postDate": "2018-04-17T09:09:53.343Z", "content": "\n\n\nlgb just trained as regressor to predict IoU
\n
Brilliant!
", "rawMarkdown": "> lgb just trained as regressor to predict IoU\n\nBrilliant!", "votes": 3 }, { "id": 315557, "postDate": "2018-04-17T09:15:55.853Z", "content": "Thanks! That gave us ~ +15
", "rawMarkdown": "Thanks! That gave us ~ +15", "votes": 6 }, { "id": 315907, "postDate": "2018-04-17T19:19:02.863Z", "content": "Sorry, I didn't understand this part well. I'm quite new to this..\nYou used lgb as method to predict the IoU with the ground truth or to decide which candidates to use? I mean you have several networks that make individual predictions and then use the lgb to decide the final ensemble. Sorry, I'm lost on this
", "rawMarkdown": "Sorry, I didn't understand this part well. I'm quite new to this..\nYou used lgb as method to predict the IoU with the ground truth or to decide which candidates to use? I mean you have several networks that make individual predictions and then use the lgb to decide the final ensemble. Sorry, I'm lost on this ", "votes": 3 } ] }, { "id": 362777, "postDate": "2018-07-27T05:39:00.027Z", "content": "I am working on the TGS competition and have been looking over the winners of old segmentation competitions and yours has a special significance because of the detail you have provided and the recency. My question is how did you apply regularization, if you did at all. Thank you
", "rawMarkdown": "I am working on the TGS competition and have been looking over the winners of old segmentation competitions and yours has a special significance because of the detail you have provided and the recency. My question is how did you apply regularization, if you did at all. Thank you", "votes": 1, "replies": [ { "id": 363521, "postDate": "2018-07-29T11:49:01.583Z", "content": "Applying agressive augmentations was enough to prevent overfitting. Spatial dropout or L2 regularisation only slowed down training speed.
", "rawMarkdown": "Applying agressive augmentations was enough to prevent overfitting. Spatial dropout or L2 regularisation only slowed down training speed. ", "votes": 2 } ] }, { "id": 388562, "postDate": "2018-09-17T07:34:05.503Z", "content": "Thanks for a great report! I have some question to ask you.
\n\nif i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models?
if i only one model, how will the LB score change, will it decrease, and how much?
train time. How long does it take to train a model and environmental configuration ?
predict time. because you use TTA, How much time does each image take?
Sorry to ask you so many questions.
\n\nHope you can reply me, thank you!
", "rawMarkdown": "Thanks for a great report! I have some question to ask you. \n\n1. if i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models?\n\n2. if i only one model, how will the LB score change, will it decrease, and how much? \n\n3. train time. How long does it take to train a model and environmental configuration ? \n\n4. predict time. because you use TTA, How much time does each image take? \n\nSorry to ask you so many questions. \n\nHope you can reply me, thank you!", "votes": 2, "replies": [ { "id": 389051, "postDate": "2018-09-18T04:16:42.840Z", "content": "Thank you very much！！！
", "rawMarkdown": "Thank you very much！！！" } ] }, { "id": 315863, "postDate": "2018-04-17T17:31:03.293Z", "content": "Congrats for your amazing work! I have two questions.
\n\nYou mentioned you got ~500 LB with the simplest approach and added watershed line for binary masks. I also tried that however, it cannot break ~400 LB. Could you please give more details about the Unet and watershed?
We have tried border in Mask-RCNN, however not sure how to combine it properly. Could you please give more details about the borders and masks combination?
BTW, the image you posted cannot read.
\n\nCongrats again!
", "rawMarkdown": "Congrats for your amazing work! I have two questions.\n\n1. You mentioned you got ~500 LB with the simplest approach and added watershed line for binary masks. I also tried that however, it cannot break ~400 LB. Could you please give more details about the Unet and watershed?\n\n2. We have tried border in Mask-RCNN, however not sure how to combine it properly. Could you please give more details about the borders and masks combination?\n\nBTW, the image you posted cannot read.\n\nCongrats again!", "votes": 1, "replies": [ { "id": 315887, "postDate": "2018-04-17T18:26:14.120Z", "content": "Images are visible in Chrome/Firefox, though in mobile Safari I don't see them . You can open the image by the link https://www.dropbox.com/s/4igam47pqg0i82q/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37_predict.png?dl=1
", "rawMarkdown": "1. Our networks are not plain UNets. We use pretrained and very deep encoders. That makes a huge difference when you don't have enough data, which is clearly the case. Watershed in that case is quite simple, you just use two thresholds, high for seeds, low for masks (something like 0.6 and 0.3 for a binary mask). Updated the description as well to make it more specific.\n2. Cannot say anything about Mask-RCNN, have not tried it in this competition. But it is designed in the way, that you don't need borders to separate instances. Though you can read MaskLab paper https://arxiv.org/abs/1712.04837 where they added third branch to predict directions. \n\nImages are visible in Chrome/Firefox, though in mobile Safari I don't see them . You can open the image by the link https://www.dropbox.com/s/4igam47pqg0i82q/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37_predict.png?dl=1\n", "votes": 4 }, { "id": 315970, "postDate": "2018-04-17T22:44:57.800Z", "content": "Interesting, you think having the encoder networks being pretrained on ImageNet was crucial?
", "rawMarkdown": "Interesting, you think having the encoder networks being pretrained on ImageNet was crucial?", "votes": 1 }, { "id": 316330, "postDate": "2018-04-18T18:38:23.973Z", "content": "@theNickName I tried to train a plain UNet from scratch using the same pipeline. The results were much worse.
", "rawMarkdown": "@theNickName I tried to train a plain UNet from scratch using the same pipeline. The results were much worse. ", "votes": 4 }, { "id": 316594, "postDate": "2018-04-19T12:27:12.197Z", "content": "First of all, congratulations for sharing this nice work ! really topcoders !
\n\nI have not a lot of experience with pretrained models. Could you clarify how you use it:\n- encoder parameters are frozen during training ? \n- encoder parameters are finetuned ? \n- encoder parameters are retrained starting from an initial value coming from imagenet weights ?
", "rawMarkdown": "First of all, congratulations for sharing this nice work ! really topcoders !\n\nI have not a lot of experience with pretrained models. Could you clarify how you use it:\n- encoder parameters are frozen during training ? \n- encoder parameters are finetuned ? \n- encoder parameters are retrained starting from an initial value coming from imagenet weights ?", "votes": 3 }, { "id": 316595, "postDate": "2018-04-19T12:30:19.817Z", "content": "jeandebleau, I'm reading this https://arxiv.org/pdf/1801.05746.pdf at the moment
", "rawMarkdown": "jeandebleau, I'm reading this https://arxiv.org/pdf/1801.05746.pdf at the moment", "votes": 1 }, { "id": 316597, "postDate": "2018-04-19T12:36:10.793Z", "content": "Thank you for the link. In the paper they use the third option that I mentionned : encoders are initialized with pretrained weights. However it is not clear to me if this is the case here ; or even if it is not possible to simply freeze the encoder.
", "rawMarkdown": "Thank you for the link. In the paper they use the third option that I mentionned : encoders are initialized with pretrained weights. However it is not clear to me if this is the case here ; or even if it is not possible to simply freeze the encoder. ", "votes": 2 }, { "id": 316603, "postDate": "2018-04-19T12:56:27.933Z", "content": "That's right. Encoders were initialized with pretrained weights from ImageNet. Then we trained models end to end.\nFrom my experience with a frozen encoder it is usually not possible to achieve good segmentation results even on datasets that are more or less similar to ImageNet.
", "rawMarkdown": "That's right. Encoders were initialized with pretrained weights from ImageNet. Then we trained models end to end.\nFrom my experience with a frozen encoder it is usually not possible to achieve good segmentation results even on datasets that are more or less similar to ImageNet. ", "votes": 5 }, { "id": 316605, "postDate": "2018-04-19T13:03:40.023Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much! " }, { "id": 316635, "postDate": "2018-04-19T14:55:14.907Z", "content": "Yes, I was also wondering about that, thanks for the tip!
", "rawMarkdown": "Yes, I was also wondering about that, thanks for the tip!" } ] }, { "id": 315847, "postDate": "2018-04-17T17:02:10.870Z", "content": "Congrats!
", "rawMarkdown": "Congrats! ", "votes": 1 }, { "id": 315737, "postDate": "2018-04-17T14:33:36.333Z", "content": "Thanks for your sharing!! it is very helpful and makes me think a lot. And, Congrats !!
", "rawMarkdown": "Thanks for your sharing!! it is very helpful and makes me think a lot. And, Congrats !!", "votes": 1 }, { "id": 315615, "postDate": "2018-04-17T10:55:43.863Z", "content": "thanks for sharing, guess it would be too much to ask for the code :)
\n\nespecially interested in the LGB part
\n\nbtw. with this:
\n\n2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)
2 channels full masks i.e. (mask, border)
you mean that you have one model predicting 4 (resp. 5) channels, or two models?
", "rawMarkdown": "thanks for sharing, guess it would be too much to ask for the code :)\n\nespecially interested in the LGB part\n\nbtw. with this:\n\n- 2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)\n\n- 2 channels full masks i.e. (mask, border)\n\nyou mean that you have one model predicting 4 (resp. 5) channels, or two models?\n", "votes": 1, "replies": [ { "id": 315617, "postDate": "2018-04-17T10:59:54.053Z", "content": "Actually different models. That gave a good profit from ensembling.
", "rawMarkdown": "Actually different models. That gave a good profit from ensembling. ", "votes": 3 } ] }, { "id": 315602, "postDate": "2018-04-17T10:40:39.010Z", "content": "I had just started to learn the distance/contours directly from the network (after a tip via Heng) just after the models were frozen. Definitely the way to go it seems! I wonder if obtaining seeds from an object detection network would also improve the score for this model too. Congratz and great job guys!
", "rawMarkdown": "I had just started to learn the distance/contours directly from the network (after a tip via Heng) just after the models were frozen. Definitely the way to go it seems! I wonder if obtaining seeds from an object detection network would also improve the score for this model too. Congratz and great job guys!", "votes": 1, "replies": [ { "id": 315815, "postDate": "2018-04-17T16:21:35.493Z", "content": "Can you tell me how to add a channel with contours. I don't know what it means.
", "rawMarkdown": "Can you tell me how to add a channel with contours. I don't know what it means." }, { "id": 315826, "postDate": "2018-04-17T16:29:24.827Z", "content": "topcoders may have implemented it differently to me, but I just create ground truth contours from the individual binary masks, then combine them in a single image channel. You can then use these as ground truth along with binary masks to teach U-Net. To output a second channel just increase the number of kernels in the final U-Net layer from 1 to 2, and make sure your inputs are consistent with this format.
", "rawMarkdown": "topcoders may have implemented it differently to me, but I just create ground truth contours from the individual binary masks, then combine them in a single image channel. You can then use these as ground truth along with binary masks to teach U-Net. To output a second channel just increase the number of kernels in the final U-Net layer from 1 to 2, and make sure your inputs are consistent with this format.", "votes": 2 }, { "id": 315836, "postDate": "2018-04-17T16:44:38.240Z", "content": "Yes, targets == ground truth masks as @DStjhb already described.\nIn general, Fully Convolutional Networks can predict and learn as many output channels as you wish. You just need to change the number of filters in the last convolution appropriately.
", "rawMarkdown": "Yes, targets == ground truth masks as @DStjhb already described.\nIn general, Fully Convolutional Networks can predict and learn as many output channels as you wish. You just need to change the number of filters in the last convolution appropriately. ", "votes": 3 }, { "id": 315917, "postDate": "2018-04-17T19:45:36.623Z", "content": "I tried that at the start and it looked nasty. Of course, it might be that I made a bug somewhere at the beginning, when I tried that, I make those quite often, sadly, but, at least I made up a different explanation along the way. Maybe, at least, the losses have to be somehow balanced. You're chasing the avg gradient generated by 2, or however many losses (from different masks) in the end. And the base data that you're dealing with is not precise at all. So if you're training borders and you get a high loss, that one doesn't necessarily lead you in the right direction and may pull away the weight updates from other masks. It surprises me this approach worked magic for you guys, as I discarded it very close to the start, seeing it didn't work too well for me. In the end I settled on building maps of inner parts of nucleuses, focused on recognizing touch zones, inner part of nucleus and boundaries (1 for inner portion of nucleus and 0 for everything else on the image), in one net and then using it as a pretrain for a net that is supposed to differentiate between nuclei and background. Seemed like a pretty great pretrain from experiments.
\n\nOn the other hand, if I actually struggled only due to some bug, it might be a better choice, because you're teaching the net to better recognize and appreciate important hard-to-notice details of images. Though, it remains up to experimentation to see what approach is better under what circumstances - first training for the hard task and then using that for the easy one, or the conjoined option.
\n\nI gotta say, how one loss-weights different pixels makes a huge difference. I wonder how you guys weighted them.
", "rawMarkdown": "I tried that at the start and it looked nasty. Of course, it might be that I made a bug somewhere at the beginning, when I tried that, I make those quite often, sadly, but, at least I made up a different explanation along the way. Maybe, at least, the losses have to be somehow balanced. You're chasing the avg gradient generated by 2, or however many losses (from different masks) in the end. And the base data that you're dealing with is not precise at all. So if you're training borders and you get a high loss, that one doesn't necessarily lead you in the right direction and may pull away the weight updates from other masks. It surprises me this approach worked magic for you guys, as I discarded it very close to the start, seeing it didn't work too well for me. In the end I settled on building maps of inner parts of nucleuses, focused on recognizing touch zones, inner part of nucleus and boundaries (1 for inner portion of nucleus and 0 for everything else on the image), in one net and then using it as a pretrain for a net that is supposed to differentiate between nuclei and background. Seemed like a pretty great pretrain from experiments.\n\nOn the other hand, if I actually struggled only due to some bug, it might be a better choice, because you're teaching the net to better recognize and appreciate important hard-to-notice details of images. Though, it remains up to experimentation to see what approach is better under what circumstances - first training for the hard task and then using that for the easy one, or the conjoined option.\n\nI gotta say, how one loss-weights different pixels makes a huge difference. I wonder how you guys weighted them.", "votes": 1 }, { "id": 316169, "postDate": "2018-04-18T10:21:35.937Z", "content": "My model had difficulties learning the exact euclidean distance. The initial training resulted in strange artefacts, such as vertical lines in the output. It helps to simplify things a little, e.g. into distance 'bins' etc.
\n\nWhen learning a border region between cells, not a full distance gradient basin, like the solution here, my model doesn't have many issues. Maybe you just need more depth.
", "rawMarkdown": "My model had difficulties learning the exact euclidean distance. The initial training resulted in strange artefacts, such as vertical lines in the output. It helps to simplify things a little, e.g. into distance 'bins' etc.\n\nWhen learning a border region between cells, not a full distance gradient basin, like the solution here, my model doesn't have many issues. Maybe you just need more depth." }, { "id": 316587, "postDate": "2018-04-19T11:49:23.760Z", "content": "DStjhb, if you're about me, I had plenty of depth. My model had residual units resnext style both ways - encoder and decoder.
", "rawMarkdown": "DStjhb, if you're about me, I had plenty of depth. My model had residual units resnext style both ways - encoder and decoder." } ] }, { "id": 315562, "postDate": "2018-04-17T09:24:28.313Z", "content": "Congratulations! that is a very impressive score.
", "rawMarkdown": "Congratulations! that is a very impressive score.", "votes": 1 }, { "id": 327086, "postDate": "2018-05-10T19:43:45.837Z", "content": "One further comment: Could you please elaborate on how you combined BCE with Dice. I have typically only seen one used or the other.
\n\nThanks
", "rawMarkdown": "One further comment: Could you please elaborate on how you combined BCE with Dice. I have typically only seen one used or the other. \n\nThanks", "votes": 2, "replies": [ { "id": 327162, "postDate": "2018-05-10T23:47:45.037Z", "content": "The logic behind why to combine those interests me too. I'm not strong at that sort of math, but it's interesting to know how it works on a, lets call it \"logical\", \"informational\" level.
", "rawMarkdown": "The logic behind why to combine those interests me too. I'm not strong at that sort of math, but it's interesting to know how it works on a, lets call it \"logical\", \"informational\" level." }, { "id": 327347, "postDate": "2018-05-11T10:40:46.197Z", "content": "informally
\n\nTheir combination is a workaround to alleviate the problems existing in pure bce or dice losses.
\n\nTo combine them the simplest approach is
\n\nloss = bce + (1 - dice)\nDepending on the data you can play with the weights for the loss terms and see the outcome
\n\nloss = w1 * bce + w2 * (1 - dice)\nGreat- thanks so much. This is a nice idea. One further question that I have been trying to figure out:
\n\nYou mentioned that you used the Janowczyk dataset. However, this is non-exhaustively labeled. Can you please comment on how you dealt with this non-exhaustive dataset.
\n\nCheers
", "rawMarkdown": "Great- thanks so much. This is a nice idea. One further question that I have been trying to figure out:\n\nYou mentioned that you used the Janowczyk dataset. However, this is non-exhaustively labeled. Can you please comment on how you dealt with this non-exhaustive dataset.\n\nCheers\n\n" }, { "id": 396701, "postDate": "2018-10-01T08:12:09.297Z", "content": "This would be of particular interest. The Janowczyk dataset contains large H&E images with nuclei partially annotated. How do you dealt with nuclear areas not annotated?
", "rawMarkdown": "This would be of particular interest. The Janowczyk dataset contains large H&E images with nuclei partially annotated. How do you dealt with nuclear areas not annotated?" }, { "id": 396703, "postDate": "2018-10-01T08:21:30.323Z", "content": "We just used few small crops from it where annotated areas larger some %. Wrote simple algorithm to find these patches.
", "rawMarkdown": "We just used few small crops from it where annotated areas larger some %. Wrote simple algorithm to find these patches." } ] }, { "id": 318069, "postDate": "2018-04-23T05:56:40.713Z", "content": "@Selim_Sef
\n\nI am trying to implement your method and repeat the below results:
\n\n\"First we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). This approach gave us around ~500 on Public LB. Clearly this is not enough to win.\"
\n\nI run into issues and have the following questions:
\n\nThe 0.5 public LB results is for one network without test-time augmentation (TTA) , one network with TTA or ensemble with TTA?
Do you try resent50 for your unet. If yes, how is the results?
Applying watershed seems straight forward but it is the most crucial part to get good results. Do you have code snippets to show how to post process mask and border channel output from your unet?
If possible, would you mind showing more images for the mask and border channel output from your unet, and also watershed post processing results? (or you can show us the 0.5 public LB submission csv file)
Thank you very much!
", "rawMarkdown": "@Selim_Sef\n\nI am trying to implement your method and repeat the below results:\n\n\"First we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). This approach gave us around ~500 on Public LB. Clearly this is not enough to win.\"\n\n---\n\nI run into issues and have the following questions:\n\n 1. The 0.5 public LB results is for one network without test-time augmentation (TTA) , one network with TTA or ensemble with TTA?\n\n 2. Do you try resent50 for your unet. If yes, how is the results?\n\n 3. Applying watershed seems straight forward but it is the most crucial part to get good results. Do you have code snippets to show how to post process mask and border channel output from your unet?\n\n 4. If possible, would you mind showing more images for the mask and border channel output from your unet, and also watershed post processing results? (or you can show us the 0.5 public LB submission csv file)\n\nThank you very much!\n\n", "votes": 2, "replies": [ { "id": 318196, "postDate": "2018-04-23T11:25:47.957Z", "content": "Hi Heng,
\n\n\n\n\ndef label_mask(mask_img, border_img, seed_ths, threshold, seed_size=8, obj_size=10):\n img_copy = np.copy(mask_img)\n m = img_copy * (1 - border_img)\n img_copy[m <= seed_ths] = 0\n img_copy[m > seed_ths] = 1\n img_copy = img_copy.astype(np.bool)\n img_copy = remove_small_objects(img_copy, seed_size).astype(np.uint8)\n mask_img[mask_img <= threshold] = 0\n mask_img[mask_img > threshold] = 1\n mask_img = mask_img.astype(np.bool)\n mask_img = remove_small_objects(mask_img, obj_size).astype(np.uint8)\n markers = ndimage.label(img_copy, output=np.uint32)[0]\n labels = watershed(mask_img, markers, mask=mask_img, watershed_line=True)\n return labels\n
\n\n4 You can get the idea how it works looking at these predictions from stage 2 (single resnet152)\n![\"00e6798\"](\"https://www.dropbox.com/s/jsnnsgwdsrp5wg3/00e67980783eaae30d41f6479beb9c04eb91de4608537673898039094800ddb0.png?dl=1\")
\n![\"1edd67f41\"](\"https://www.dropbox.com/s/vpqy6iqbihzaihd/1edd67f41a030b1c04828e2f58581751b960b5941e6d39239b5cb57653f6b0a7.png?dl=1\")
\ndef label_mask(mask_img, border_img, seed_ths, threshold, seed_size=8, obj_size=10):\n img_copy = np.copy(mask_img)\n m = img_copy * (1 - border_img)\n img_copy[m <= seed_ths] = 0\n img_copy[m > seed_ths] = 1\n img_copy = img_copy.astype(np.bool)\n img_copy = remove_small_objects(img_copy, seed_size).astype(np.uint8)\n mask_img[mask_img <= threshold] = 0\n mask_img[mask_img > threshold] = 1\n mask_img = mask_img.astype(np.bool)\n mask_img = remove_small_objects(mask_img, obj_size).astype(np.uint8)\n markers = ndimage.label(img_copy, output=np.uint32)[0]\n labels = watershed(mask_img, markers, mask=mask_img, watershed_line=True)\n return labels\n
\n\n\n4 You can get the idea how it works looking at these predictions from stage 2 (single resnet152)\n![00e6798][1]\n![1edd67f41][2]\n\n [1]: https://www.dropbox.com/s/jsnnsgwdsrp5wg3/00e67980783eaae30d41f6479beb9c04eb91de4608537673898039094800ddb0.png?dl=1\n [2]: https://www.dropbox.com/s/vpqy6iqbihzaihd/1edd67f41a030b1c04828e2f58581751b960b5941e6d39239b5cb57653f6b0a7.png?dl=1\n",
"votes": 7
},
{
"id": 318211,
"postDate": "2018-04-23T12:09:02.583Z",
"content": "Absolutely gorgeous work!
", "rawMarkdown": "Absolutely gorgeous work! ", "votes": 6 }, { "id": 318779, "postDate": "2018-04-24T13:15:20.567Z", "content": "@Selim_Sef
\n\nThank you very much the details. I will implement it and feedback to you later. Thanks again!
", "rawMarkdown": "@Selim_Sef\n\nThank you very much the details. I will implement it and feedback to you later. Thanks again!", "votes": 1 }, { "id": 318799, "postDate": "2018-04-24T13:57:20.587Z", "content": "After implementing @topcoders solution, I have to say their work is extraordinary smart.
\n\nIn my initial experiments, i actually tried Unet but i gave it up because i have problem in the post processing of unet outputs. Watershed is very sensitive to input marker and energy function (distance transform). That is why I switch to mask-rcnn.
\n\n@topcoders solution use unet to learn the \"border of nuclei\", which is made from watershed line of the ground truth. In this way they can control the accuracy of the watershed post processing.
\n\nresnet50 seems to insufficient for the task. I am now trying resnet101. i will post some results later.
", "rawMarkdown": "After implementing @topcoders solution, I have to say their work is extraordinary smart.\n\nIn my initial experiments, i actually tried Unet but i gave it up because i have problem in the post processing of unet outputs. Watershed is very sensitive to input marker and energy function (distance transform). That is why I switch to mask-rcnn.\n\n @topcoders solution use unet to learn the \"border of nuclei\", which is made from watershed line of the ground truth. In this way they can control the accuracy of the watershed post processing.\n\nresnet50 seems to insufficient for the task. I am now trying resnet101. i will post some results later.\n", "votes": 15 }, { "id": 331822, "postDate": "2018-05-21T23:31:29.500Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 315823, "postDate": "2018-04-17T16:28:09.193Z", "content": "Respect, impressive 2nd stage approach! Can't imagine to see gbt here.
", "rawMarkdown": "Respect, impressive 2nd stage approach! Can't imagine to see gbt here.", "votes": 2 }, { "id": 315577, "postDate": "2018-04-17T09:42:33.717Z", "content": "Congratulations! Brilliant work!
\n\nHow much LB increase did the augmentations give?
\n\nAny chance that we can look at your code?
", "rawMarkdown": "Congratulations! Brilliant work! \n\nHow much LB increase did the augmentations give?\n\nAny chance that we can look at your code?", "votes": 2, "replies": [ { "id": 315588, "postDate": "2018-04-17T10:15:27.843Z", "content": "It is quite hard to say how much exactly we gained from augmentations. But reducing them constanly led to better local validation/CV scores and much worse public scores.
", "rawMarkdown": "It is quite hard to say how much exactly we gained from augmentations. But reducing them constanly led to better local validation/CV scores and much worse public scores.", "votes": 2 } ] }, { "id": 315555, "postDate": "2018-04-17T09:14:49.367Z", "content": "Hats off! Great solution.
\n\nWhat HW resources did you use?
", "rawMarkdown": "Hats off! Great solution.\n\nWhat HW resources did you use?", "votes": 2, "replies": [ { "id": 315561, "postDate": "2018-04-17T09:20:34.537Z", "content": "Not so much actually) Everyone of us have by 2 GPUs (4*1080ti and 2*Titan XPs)
", "rawMarkdown": "Not so much actually) Everyone of us have by 2 GPUs (4*1080ti and 2*Titan XPs)", "votes": 4 } ] }, { "id": 443729, "postDate": "2018-12-22T09:12:00.163Z", "content": "Did anyone deploy this model? I would like to apply this to a set of H&E stained images but it would take few days to train on a not-so-good graphics card.
", "rawMarkdown": "Did anyone deploy this model? I would like to apply this to a set of H&E stained images but it would take few days to train on a not-so-good graphics card.", "votes": -2 }, { "id": 997692, "postDate": "2020-09-04T07:17:52.017Z", "content": "Nice Tricks!
", "rawMarkdown": "Nice Tricks!" }, { "id": 414396, "postDate": "2018-11-02T17:15:10.013Z", "content": "Hi @Selim, from what I'm looking at you code it seems that during test you inferred on the original test image's size. Wouldn't that be a problem because since you trained on 256 crops, your network does not have stride enough to get context from bigger images? Example: most networks have 1/32 or 1/16 final stride, so for a 1024 pixels image I believe one would not get enough context even in the last layers. Am I wrong?
\n\nWould not be best crop the original image into 256 pixels pieces with some overlap, infer to all of them and finally join them back together?
\n\nThanks!
", "rawMarkdown": "Hi @Selim, from what I'm looking at you code it seems that during test you inferred on the original test image's size. Wouldn't that be a problem because since you trained on 256 crops, your network does not have stride enough to get context from bigger images? Example: most networks have 1/32 or 1/16 final stride, so for a 1024 pixels image I believe one would not get enough context even in the last layers. Am I wrong?\n\nWould not be best crop the original image into 256 pixels pieces with some overlap, infer to all of them and finally join them back together?\n\nThanks!", "replies": [ { "id": 416840, "postDate": "2018-11-07T10:49:29.537Z", "content": "For that models, no. Because fully convolutional networks almost do not depend on the input size, unless there are some blocks like SE/SCSE
", "rawMarkdown": "For that models, no. Because fully convolutional networks almost do not depend on the input size, unless there are some blocks like SE/SCSE", "votes": 2 }, { "id": 416843, "postDate": "2018-11-07T11:10:16.413Z", "content": "Thanks, you are always helping me out! But why SE block are problem? I know there are FC layers there but its only after a GlobalPooling layer, so we always get 1x1xC at the FC layer, no matter the input size. Right?
", "rawMarkdown": "Thanks, you are always helping me out! But why SE block are problem? I know there are FC layers there but its only after a GlobalPooling layer, so we always get 1x1xC at the FC layer, no matter the input size. Right?" }, { "id": 417159, "postDate": "2018-11-07T21:44:12.303Z", "content": "Global avergae pooling will produce different features. Even though it will not break anything that may degrade performance.
", "rawMarkdown": "Global avergae pooling will produce different features. Even though it will not break anything that may degrade performance.", "votes": 2 } ] }, { "id": 410548, "postDate": "2018-10-26T08:34:51.903Z", "content": "Good job @selim. There are lot of interesting new aspects in you solution which I have to head on to.
", "rawMarkdown": "Good job @selim. There are lot of interesting new aspects in you solution which I have to head on to." }, { "id": 388560, "postDate": "2018-09-17T07:33:07.977Z", "content": "Thanks for a great report! I have some question to ask you.
\n\nif i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models?
if i only one model, how will the LB score change, will it decrease, and how much?
\n\npredict time. because you use TTA, How much time does each image take?
Sorry to ask you so many questions.
\n\nHope you can reply me, thank you!
", "rawMarkdown": "Thanks for a great report! I have some question to ask you. \n\n1. if i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models? \n\n2. if i only one model, how will the LB score change, will it decrease, and how much?\n\n 3. train time. How long does it take to train a model and environmental configuration ? \n\n4. predict time. because you use TTA, How much time does each image take? \n\nSorry to ask you so many questions. \n\nHope you can reply me, thank you!" }, { "id": 355229, "postDate": "2018-07-11T08:44:30.017Z", "content": "Thanks for a great report. In the code you have mentioned about label image. But in training we have mask and train image. Could you share me the details about label image?
\n\n lbl = cv2.imread(path.join(labels_folder, '{0}.tif'.format(img_id)), cv2.IMREAD_UNCHANGED)\nLabels where generated from original data, i.e. for each image there was a single grayscale tif file with labeled instances. Afair there should be some code on kernels/forums which does that.
", "rawMarkdown": "Labels where generated from original data, i.e. for each image there was a single grayscale tif file with labeled instances. Afair there should be some code on kernels/forums which does that.", "votes": 2 }, { "id": 443661, "postDate": "2018-12-22T04:05:46.553Z", "content": "I couldn't find a kernel that would generate those labels. Could you share what you did please? I'd love to replicate and try to understand your approach.
", "rawMarkdown": "I couldn't find a kernel that would generate those labels. Could you share what you did please? I'd love to replicate and try to understand your approach.", "votes": 2 }, { "id": 477226, "postDate": "2019-02-24T06:52:12.807Z", "content": "Hello, have you solved the problem? I am also a little confused about the generation of the label images.
", "rawMarkdown": "Hello, have you solved the problem? I am also a little confused about the generation of the label images.", "votes": 1 } ] }, { "id": 345546, "postDate": "2018-06-20T02:54:14.823Z", "content": "Hi @Selim, I've been trying to implement a solution similar to yours however when I use random crops for training my val score decreases comparing to resizing, even though I'm using soft dice + bce as loss. I believe that this have to do with class unbalancing. Did you use any kind of smart implementation to avoid unbalancing when cropping?
", "rawMarkdown": "Hi @Selim, I've been trying to implement a solution similar to yours however when I use random crops for training my val score decreases comparing to resizing, even though I'm using soft dice + bce as loss. I believe that this have to do with class unbalancing. Did you use any kind of smart implementation to avoid unbalancing when cropping?", "replies": [ { "id": 345982, "postDate": "2018-06-20T21:21:05.153Z", "content": "if you use channelwise dice loss then you will not have border/body pixelwise loss imbalance. Resizing should be used only for train (or multiscale prediction during inference) time augmentations - i.e. random sized crop is resized to the same size e.g. 224 or 256.
\n\nTraining with random crops using proper size usually works fine on all segmentation problems. For this problem crop size 224 or 256 was optimal from my experience.
", "rawMarkdown": "if you use channelwise dice loss then you will not have border/body pixelwise loss imbalance. Resizing should be used only for train (or multiscale prediction during inference) time augmentations - i.e. random sized crop is resized to the same size e.g. 224 or 256. \n\nTraining with random crops using proper size usually works fine on all segmentation problems. For this problem crop size 224 or 256 was optimal from my experience. " }, { "id": 346032, "postDate": "2018-06-21T00:07:27.183Z", "content": "Thanks for the reply.\n I'm not even using the borders yet. I'm having the imbalance between nuclei and background. \nMy net predicts two channels (foreground, background - one hot format), then I apply softmax and use soft dice + cross entropy as loss. Please, take a look:
\n\nelif self.loss == 'ce+wdice':\n # Binary cross entropy\n if self.use_weights:\n cls_weights = self.calc_class_weights(onehot_labels)\n weights = tf.squeeze(self.w_tf) + cls_weights\n else:\n weights = 1.0\n loss1 = tf.losses.softmax_cross_entropy(\n onehot_labels=onehot_labels,\n logits=self.logits_tf,\n weights=weights)\n loss1 = tf.reduce_mean(loss1)\n\n # Soft dice loss\n axis = [1, 2]\n epilson = 1e-5\n total = self.output_shape[0] * self.output_shape[1]\n w = tf.reduce_sum(onehot_labels, axis=axis) / total\n I = tf.reduce_sum(onehot_labels * self.logits_tf, axis=axis)\n balanced_I = I * (1 - w + epilson)\n l2_pred = tf.reduce_sum(tf.square(self.logits_tf), axis=axis)\n l2_true = tf.reduce_sum(tf.square(onehot_labels), axis=axis)\n dice_coeff = (2. * balanced_I + epilson) / \\\n (l2_true + l2_pred + epilson)\n loss2 = tf.subtract(1., tf.reduce_mean(dice_coeff))\n\n loss = loss1 + loss2\nyou should not use dice for background channel, that's the problem.
", "rawMarkdown": "you should not use dice for background channel, that's the problem." } ] }, { "id": 339900, "postDate": "2018-06-07T22:16:19.797Z", "content": "Hi @Selim, congratulations for your work!\nI noticed that you used InceptionResNetV2 as one of your encoder but you did not mentioned about the decoder architecture. How did you handle with the encoder's unpadded convolution? I mean, to make the encoder-decoder dimensions match.\nThanks a lot!
", "rawMarkdown": "Hi @Selim, congratulations for your work!\nI noticed that you used InceptionResNetV2 as one of your encoder but you did not mentioned about the decoder architecture. How did you handle with the encoder's unpadded convolution? I mean, to make the encoder-decoder dimensions match.\nThanks a lot! ", "replies": [ { "id": 339902, "postDate": "2018-06-07T22:34:04.687Z", "content": "HI, I simply changed encoders and replaced valid padding with same padding.
\nDecoders where of two flavours:\n - U-Net like with standard upsampling-conv approach\n - a custom FPN like decoder\n - their performance was the same
Thanks a lot!
", "rawMarkdown": "Thanks a lot!" } ] }, { "id": 326791, "postDate": "2018-05-10T10:48:13.250Z", "content": "Congrats on the great result! Please can you enlighten us on whether you used Dropout within your network at all? I'm guessing probably not if you used ResNet as the encoder...
", "rawMarkdown": "Congrats on the great result! Please can you enlighten us on whether you used Dropout within your network at all? I'm guessing probably not if you used ResNet as the encoder... ", "replies": [ { "id": 326795, "postDate": "2018-05-10T10:58:20.443Z", "content": "Ordinary Dropout is usually harmful for convolutional layers. But SpatialDropout2D could be used for segmentation tasks and gives good results from my experience. \nWe did not use it here though because we already had excessive augmentations and adding dropout will increase training time even more.
", "rawMarkdown": "Ordinary Dropout is usually harmful for convolutional layers. But SpatialDropout2D could be used for segmentation tasks and gives good results from my experience. \nWe did not use it here though because we already had excessive augmentations and adding dropout will increase training time even more." }, { "id": 326804, "postDate": "2018-05-10T11:13:47.797Z", "content": "Interesting, I wonder where you put SpatialDropout2D so that it adds precision. Where I tried putting it it basically destroyed the network.
", "rawMarkdown": "Interesting, I wonder where you put SpatialDropout2D so that it adds precision. Where I tried putting it it basically destroyed the network." }, { "id": 326836, "postDate": "2018-05-10T12:03:53.760Z", "content": "I used SpatialDropout2D just before the classification layer
", "rawMarkdown": "I used SpatialDropout2D just before the classification layer", "votes": 1 }, { "id": 326992, "postDate": "2018-05-10T16:14:35.923Z", "content": "Thanks for your recommendation! I will have to experiment! :)
", "rawMarkdown": "Thanks for your recommendation! I will have to experiment! :)" } ] }, { "id": 320870, "postDate": "2018-04-30T04:12:25.590Z", "content": "Thank you for the explanation.\nI was curious about which of the Mask-RCNN vs UNET wins the first solution. And it was very interesting for me that UNET wins by adding a extra pre/postprocessing like adding border etc!
", "rawMarkdown": "Thank you for the explanation.\nI was curious about which of the Mask-RCNN vs UNET wins the first solution. And it was very interesting for me that UNET wins by adding a extra pre/postprocessing like adding border etc!" }, { "id": 320292, "postDate": "2018-04-28T06:54:18.167Z", "content": "@Selim_Sef How did you handle color images? Is it:
\n\none unet, 2-channel for all images, color + gray (1-class unet)
one unet, 2-channel for color images and 2-channel gray images (2-class unet)
one unet, 2-channel for color images. Another separate unet, 2-channel for gray images. (two 1-class unets)
Thanks!
", "rawMarkdown": "@Selim_Sef How did you handle color images? Is it:\n\n1. one unet, 2-channel for all images, color + gray (1-class unet)\n\n2. one unet, 2-channel for color images and 2-channel gray images (2-class unet)\n\n3. one unet, 2-channel for color images. Another separate unet, 2-channel for gray images. (two 1-class unets)\n\nThanks!", "replies": [ { "id": 320524, "postDate": "2018-04-28T23:56:51.907Z", "content": "our solution is quite simple, we just use RGB everywhere
", "rawMarkdown": "our solution is quite simple, we just use RGB everywhere", "votes": 1 } ] }, { "id": 319828, "postDate": "2018-04-26T23:01:11.943Z", "content": "One more question: what modification of original training data did you use?
", "rawMarkdown": "One more question: what modification of original training data did you use?", "replies": [ { "id": 320028, "postDate": "2018-04-27T10:13:56.370Z", "content": "We did not change original training data. We just produced different masks from the original labels. And added a few external datasets as described in the post.
", "rawMarkdown": "We did not change original training data. We just produced different masks from the original labels. And added a few external datasets as described in the post. ", "votes": 2 }, { "id": 320130, "postDate": "2018-04-27T15:30:36.347Z", "content": "Yes, I meant what mods to labels did you generally do and if, maybe, you used some openly shared dataset remake like Lopuhin's.
", "rawMarkdown": "Yes, I meant what mods to labels did you generally do and if, maybe, you used some openly shared dataset remake like Lopuhin's." } ] }, { "id": 318787, "postDate": "2018-04-24T13:31:50.607Z", "content": "What would happen if you trained a mask-RCNN on your UNET channels + predictions + images. Could you get the best of both worlds?
", "rawMarkdown": "What would happen if you trained a mask-RCNN on your UNET channels + predictions + images. Could you get the best of both worlds?", "replies": [ { "id": 318853, "postDate": "2018-04-24T15:46:03.273Z", "content": "We actually thought about Mask-RCNN-like 2nd lvl model but did not have time to implement this. Most likely the results would be much better. The current approach with predicted masks' morphological features is very limited as it doesn't take image features into account.
", "rawMarkdown": "We actually thought about Mask-RCNN-like 2nd lvl model but did not have time to implement this. Most likely the results would be much better. The current approach with predicted masks' morphological features is very limited as it doesn't take image features into account.", "votes": 3 } ] }, { "id": 318045, "postDate": "2018-04-23T04:55:08.520Z", "content": "Congrats!!
\n\nAnd how to predict only the borders between the cells using unet ??\nI don't know how to judge if the borders between the cells or not...
", "rawMarkdown": "Congrats!!\n\nAnd how to predict only the borders between the cells using unet ??\nI don't know how to judge if the borders between the cells or not...", "replies": [ { "id": 318179, "postDate": "2018-04-23T10:30:24.383Z", "content": "It is quite easy to do with non vectorized code with loops but it is quite slow.\nA better approach is to use labels, dilation, watershed with watershed_line=True etc. Watershed line will be a border between the nuclei.
", "rawMarkdown": "It is quite easy to do with non vectorized code with loops but it is quite slow.\nA better approach is to use labels, dilation, watershed with watershed_line=True etc. Watershed line will be a border between the nuclei.", "votes": 1 }, { "id": 320242, "postDate": "2018-04-28T01:34:33.880Z", "content": "thanks
", "rawMarkdown": "thanks" } ] }, { "id": 316785, "postDate": "2018-04-19T22:21:33.520Z", "content": "Congrats, very impressive solution!
", "rawMarkdown": "Congrats, very impressive solution!" }, { "id": 316148, "postDate": "2018-04-18T09:04:03.683Z", "content": "Congratulations!! Thank you so much for sharing your strategy which is really helpful for us to understand and learn it. Hope you guys keep the great works! Now I realize how important is to have deep pre-trained networks for such a good competition.
", "rawMarkdown": "Congratulations!! Thank you so much for sharing your strategy which is really helpful for us to understand and learn it. Hope you guys keep the great works! Now I realize how important is to have deep pre-trained networks for such a good competition." }, { "id": 316025, "postDate": "2018-04-18T03:17:14.600Z", "content": "Congratulations .Phenomenal work .Love to learn this area in detail.Greatly appreciate if you could share some learning sites on the same for our initial understanding
", "rawMarkdown": "Congratulations .Phenomenal work .Love to learn this area in detail.Greatly appreciate if you could share some learning sites on the same for our initial understanding " }, { "id": 315958, "postDate": "2018-04-17T21:58:07.880Z", "content": "Hi Selim
\n\nI am quite new to all the techniques you are describing here and I've learned a lot.
\n\nI just want to clarify this to ensure my understanding is correct:\n\"Then we added second channel with contours, where width of contours depended on the nucleus size. \"
\n\nwhen you say adding contour I assume you mean， instead of binary mask， you have some sort of value that is decreasing from the centre of the nuclei towards the edge of the nuclei?
", "rawMarkdown": "Hi Selim\n\nI am quite new to all the techniques you are describing here and I've learned a lot.\n\nI just want to clarify this to ensure my understanding is correct:\n\"Then we added second channel with contours, where width of contours depended on the nucleus size. \"\n\nwhen you say adding contour I assume you mean， instead of binary mask， you have some sort of value that is decreasing from the centre of the nuclei towards the edge of the nuclei?", "replies": [ { "id": 315960, "postDate": "2018-04-17T22:10:01.983Z", "content": "sorry nvm just realise after reading your post carefully that contour you mean the seperation between nuclei/masks, i.e. the bright blue lines in your example image
", "rawMarkdown": "sorry nvm just realise after reading your post carefully that contour you mean the seperation between nuclei/masks, i.e. the bright blue lines in your example image" } ] }, { "id": 315894, "postDate": "2018-04-17T18:46:05.110Z", "content": "Congrats! \nHow much was this a contest over \"generalizability\" to you? And was the use of external data overall a helpful, or hurtful thing in your experience? I see you're the only other person in discussions who said external data and inverting was at least somewhat hurtful when everybody else was screaming how they were surprised at the fact that stage 2 data is somewhat different than stage 1 and that they would've used more external data and heavier augmentations if they knew what's coming.
\n\nMy guess is BBBC020 was definitely crucial to your victory, though. All BBBC datasets work very well with this challenge. With stage 2. Pretty much unlike any other datasets, most of which would only hurt.
", "rawMarkdown": "Congrats! \nHow much was this a contest over \"generalizability\" to you? And was the use of external data overall a helpful, or hurtful thing in your experience? I see you're the only other person in discussions who said external data and inverting was at least somewhat hurtful when everybody else was screaming how they were surprised at the fact that stage 2 data is somewhat different than stage 1 and that they would've used more external data and heavier augmentations if they knew what's coming.\n\nMy guess is BBBC020 was definitely crucial to your victory, though. All BBBC datasets work very well with this challenge. With stage 2. Pretty much unlike any other datasets, most of which would only hurt.", "replies": [ { "id": 315908, "postDate": "2018-04-17T19:22:19.203Z", "content": "Seems that TNBC dataset negatively affected performance.
\n\nInverting - predictions for neuron tissue were really bad, looks like the models predicted inverted images.
", "rawMarkdown": "Seems that TNBC dataset negatively affected performance. \n\nInverting - predictions for neuron tissue were really bad, looks like the models predicted inverted images. ", "votes": 2 }, { "id": 315919, "postDate": "2018-04-17T19:53:46.357Z", "content": "It's a really big surprise to me that someone who was handicapped by inversion and such external data ended up as a winner. I expected I'd see folks lucky enough to only train on the right data win this one. But your solution seems so well executed it survived that. I guess yours is the solution that organizers were looking for --- supergeneral and yet superperformant.
", "rawMarkdown": "It's a really big surprise to me that someone who was handicapped by inversion and such external data ended up as a winner. I expected I'd see folks lucky enough to only train on the right data win this one. But your solution seems so well executed it survived that. I guess yours is the solution that organizers were looking for --- supergeneral and yet superperformant.", "votes": 2 } ] }, { "id": 315563, "postDate": "2018-04-17T09:24:28.353Z", "content": "Congrats, Great work.
", "rawMarkdown": "Congrats, Great work. " }, { "id": 388559, "postDate": "2018-09-17T07:30:52.167Z", "rawMarkdown": "", "isDeleted": true }, { "id": 319755, "postDate": "2018-04-26T19:08:28.260Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 319830, "postDate": "2018-04-26T23:11:31.320Z", "content": "It seems Heng is adapting his method. You might want to follow recent Heng's posts.
", "rawMarkdown": "It seems Heng is adapting his method. You might want to follow recent Heng's posts.", "votes": 2 }, { "id": 320002, "postDate": "2018-04-27T09:08:16.980Z", "rawMarkdown": "", "votes": -24, "isDeleted": true }, { "id": 320021, "postDate": "2018-04-27T09:48:29.507Z", "content": "First, it is up to the organisers to verify that rules have been adhered to, not competitors. Athletes don't drug-test their fellow athletes.
\n\nSecond, the rules clearly state that although public code sharing is permitted, and an open source license is required, the competition sponsors are granted the license to the code, not the public:
\n\n\n\n\n\n
\n \n- WINNERS OBLIGATIONS.
\n(b) grant to Competition Sponsor the license to the winning Submission\n as set forth in the Competition Specific Rules above, and represent\n that you have the unrestricted right to grant that license;
\n
I agree that it would be great to see the code for the whole model, but I don’t believe they have an obligation to share it. It would be much better to allow topcoders to make it well-organised, well-documented, and readable first. They obviously don’t mind sharing parts of the code in good faith (Selim_Sef has posted code on this very thread), but I totally understand why they wouldn’t share the whole model in its current state.
", "rawMarkdown": "First, it is up to the organisers to verify that rules have been adhered to, not competitors. Athletes don't drug-test their fellow athletes.\n\nSecond, the rules clearly state that although public code sharing is *permitted*, and an open source license is required, the competition *sponsors are granted the license to the code*, not the public:\n\n> 10. WINNERS OBLIGATIONS.\n> \n> (b) grant to Competition Sponsor the license to the winning Submission\n> as set forth in the Competition Specific Rules above, and represent\n> that you have the unrestricted right to grant that license;\n\nI agree that it would be great to see the code for the whole model, but I don’t believe they have an obligation to share it. It would be much better to allow topcoders to make it well-organised, well-documented, and readable first. They obviously don’t mind sharing parts of the code in good faith (Selim_Sef has posted code on this very thread), but I totally understand why they wouldn’t share the whole model in its current state.", "votes": 9 }, { "id": 320107, "postDate": "2018-04-27T14:26:15.983Z", "content": "@DStjhb The quote you mention is part of the general rules. This competition has also specific rules:
\n\n\n\n\nA. SPECIFIC COMPETITION RULES\n\nIn addition to the provisions of the General Competition Rules below, you understand and agree to these Specific Competition Rules required by the Competition Sponsor:\n\nWinner License: Open Source\n\nOpen Source: You will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org>) that in no event limits commercial use of such code or model containing or depending on such code.\n
I think the winners will need to disclose the source to the public with an OSI license if they wish to claim the prize. But I agree that the goal is not for peer verification. It is meant to let anyone to benefit from the code after the prizes have been finalized by the sponsors.
", "rawMarkdown": "@DStjhb The quote you mention is part of the general rules. This competition has also specific rules:\n\n\n\nI think the winners will need to disclose the source to the public with an OSI license if they wish to claim the prize. But I agree that the goal is not for peer verification. It is meant to let anyone to benefit from the code after the prizes have been finalized by the sponsors.", "votes": 2 }, { "id": 320272, "postDate": "2018-04-28T04:51:40.967Z", "content": "\nA. SPECIFIC COMPETITION RULES\n\nIn addition to the provisions of the General Competition Rules below, you understand and agree to these Specific Competition Rules required by the Competition Sponsor:\n\nWinner License: Open Source\n\nOpen Source: You will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org>) that in no event limits commercial use of such code or model containing or depending on such code.\n
Well, strictly speaking the open source license does not require sharing the source code with everyone, it prohibits/limits restrictions of sharing. So kaggle/sponsors will receive the source code with appropriate license and it's up to either kaggle, sponsors or topcoders to decide to share the source code or not.
\n\nHowever I'd expect each side to benefit from the source code being published for a number of reasons, but the shared version does not have to be the same as used to reproduce the submission, it's much more useful to publish the cleaned up than exact solution.
", "rawMarkdown": "Well, strictly speaking the open source license does not require sharing the source code with everyone, it prohibits/limits restrictions of sharing. So kaggle/sponsors will receive the source code with appropriate license and it's up to either kaggle, sponsors or topcoders to decide to share the source code or not.\n\nHowever I'd expect each side to benefit from the source code being published for a number of reasons, but the shared version does not have to be the same as used to reproduce the submission, it's much more useful to publish the cleaned up than exact solution.", "votes": 16 }, { "id": 320276, "postDate": "2018-04-28T05:30:27.700Z", "content": "Good point. I thought it would become something public like the Linux kernel but I guess there is not a requirement for public disclosure in the license. I'm guilty for entering and agreeing with the rules without understanding the full implications of the rules ;)
", "rawMarkdown": "Good point. I thought it would become something public like the Linux kernel but I guess there is not a requirement for public disclosure in the license. I'm guilty for entering and agreeing with the rules without understanding the full implications of the rules ;) " }, { "id": 320302, "postDate": "2018-04-28T07:51:40.350Z", "content": "I concur with DStjhb and Dmytro, the obligation is to allow the sponsor to reuse part or all the prize winners solution without limitation. Afterall they pay for the prizes, they must get what they pay for.
\n\nSometimes prize winners make the extra effort to publish their solution (cleaned or not) source code for the benefit of the Kaggle community. But this is not always the case, either because it can be lots of work, and second because sometimes the winner cannot share widely to protect some intellectual property.
\n\nSo, let's encourage winners to share, but there is no point cornering them.
", "rawMarkdown": "I concur with DStjhb and Dmytro, the obligation is to allow the sponsor to reuse part or all the prize winners solution without limitation. Afterall they pay for the prizes, they must get what they pay for. \n\nSometimes prize winners make the extra effort to publish their solution (cleaned or not) source code for the benefit of the Kaggle community. But this is not always the case, either because it can be lots of work, and second because sometimes the winner cannot share widely to protect some intellectual property. \n\nSo, let's encourage winners to share, but there is no point cornering them.", "votes": 7 }, { "id": 320315, "postDate": "2018-04-28T09:02:59.510Z", "content": "I think this discussion is useful because we may not all know the legalities involved. I'd like to learn more. It seems to me that an OSI license won't work for a prize winner that intends to keep his intellectual property undisclosed because either party could in principle place the code in public domain.
", "rawMarkdown": "I think this discussion is useful because we may not all know the legalities involved. I'd like to learn more. It seems to me that an OSI license won't work for a prize winner that intends to keep his intellectual property undisclosed because either party could in principle place the code in public domain. " } ] }, { "id": 318978, "postDate": "2018-04-25T00:19:50.887Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 319142, "postDate": "2018-04-25T10:22:10.120Z", "content": "No, this blogpost and codes are not from our team. As described in his github repo - this is solution from 400th place on public and 100+ place on private.
", "rawMarkdown": "No, this blogpost and codes are not from our team. As described in his github repo - this is solution from 400th place on public and 100+ place on private. ", "votes": 3 } ] }, { "id": 317264, "postDate": "2018-04-21T04:42:31.053Z", "rawMarkdown": "", "isDeleted": true }, { "id": 315809, "postDate": "2018-04-17T16:17:12.850Z", "content": "Great explanation, thanks for sharing!
", "rawMarkdown": "Great explanation, thanks for sharing!", "votes": 1 }, { "id": 420032, "postDate": "2018-11-13T00:33:03.633Z", "content": "Thank you for the write up and Github!
", "rawMarkdown": "Thank you for the write up and Github!" }, { "id": 315539, "postDate": "2018-04-17T08:55:42.357Z", "content": "thanks
", "rawMarkdown": "thanks" } ], "comments": [ { "id": 315540, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-04-17T08:56:09.843000", "content": "Thanks for a great report! Which losses did you use and which optimizer? Any tricks with learning rate schedule? Which crop size did you use? How important/heavy were the scale augmentations? Did you use all augmentations at test time as well?
", "votes": 11, "replies": [ { "id": 315587, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-17T10:13:51.547000", "content": "Thanks! Good point! Added training description and TTA to the post.\nDuring training random scale was in [0.55, 1.45]. Whenever we reduced augmentations we got better validation results and worse score on public leaderboard.
", "votes": 6, "replies": [] }, { "id": 315589, "author_name": "Victor Durnov", "author_url": "", "post_date": "2018-04-17T10:21:54.257000", "content": "I've also used 3 scales for TTA (0.75, 1, 1.25) * 2 flips * 4 rotations = 24 total
", "votes": 6, "replies": [] }, { "id": 315605, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-04-17T10:41:46.550000", "content": "Thanks! Congrats on your well deserved win!
", "votes": 5, "replies": [] } ] }, { "id": 319237, "author_name": "Mathan", "author_url": "", "post_date": "2018-04-25T15:07:17.893000", "content": "Fantastic work. This was my first kaggle competition. Like many others, I too started with UNet but gave up as I couldn't improve the score after a while. Your post explains how kaggle experience, domain knowledge and collaborative thinking helps to get past mediocre scores with known networks. Thanks for the post.
", "votes": 9, "replies": [] }, { "id": 315954, "author_name": "Russ Wolfinger", "author_url": "", "post_date": "2018-04-17T21:42:18.813000", "content": "Bravo, awesome work topcoders! Really great to see building on knowledge from previous competitions then smart adaptations to this one.
", "votes": 9, "replies": [ { "id": 315962, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-17T22:11:23.640000", "content": "Highly interested in what you guys did as well!
", "votes": 3, "replies": [] } ] }, { "id": 315592, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-17T10:25:47.407000", "content": "\"2nd Level model / Postprocessing\"
\n\nI think this is important. Rejecting false positive can boost your score by quite a lot! Is there any comparison results with and without post processing?
\n\nGood work!
", "votes": 5, "replies": [ { "id": 315611, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-17T10:49:59.167000", "content": "Hi @Heng, \nThanks!\nIt is hard to estimate the gain on stage2 data. On Public LB this 2nd level model with adaptive thresholds and FP rejection increased the score by ~15.
", "votes": 2, "replies": [] } ] }, { "id": 315541, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T08:57:08.637000", "content": "Wow, thanks for sharing. Congratulations on the solution and the prize win! Can you expand a bit more on your use of lgb? Also, how do you compute predicted IoU?
", "votes": 4, "replies": [ { "id": 315547, "author_name": "Victor Durnov", "author_url": "", "post_date": "2018-04-17T09:04:37.497000", "content": "lgb just trained as regressor to predict IoU. So, we have nucleus candidate after Neural Networks prediction. We extracting some features and predicting IoU. I've used this method in Urban3D competition, but predicted binary target: 0 for FP, 1 for TP candidate. Here predicting IoU was more flexible for current MaP metric.
", "votes": 7, "replies": [] }, { "id": 315552, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T09:09:53.343000", "content": "\n\n\nlgb just trained as regressor to predict IoU
\n
Brilliant!
", "votes": 3, "replies": [] }, { "id": 315557, "author_name": "Victor Durnov", "author_url": "", "post_date": "2018-04-17T09:15:55.853000", "content": "Thanks! That gave us ~ +15
", "votes": 6, "replies": [] }, { "id": 315907, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-04-17T19:19:02.863000", "content": "Sorry, I didn't understand this part well. I'm quite new to this..\nYou used lgb as method to predict the IoU with the ground truth or to decide which candidates to use? I mean you have several networks that make individual predictions and then use the lgb to decide the final ensemble. Sorry, I'm lost on this
", "votes": 3, "replies": [] } ] }, { "id": 362777, "author_name": "Arpan Dhatt", "author_url": "", "post_date": "2018-07-27T05:39:00.027000", "content": "I am working on the TGS competition and have been looking over the winners of old segmentation competitions and yours has a special significance because of the detail you have provided and the recency. My question is how did you apply regularization, if you did at all. Thank you
", "votes": 1, "replies": [ { "id": 363521, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-07-29T11:49:01.583000", "content": "Applying agressive augmentations was enough to prevent overfitting. Spatial dropout or L2 regularisation only slowed down training speed.
", "votes": 2, "replies": [] } ] }, { "id": 388562, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-09-17T07:34:05.503000", "content": "Thanks for a great report! I have some question to ask you.
\n\nif i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models?
if i only one model, how will the LB score change, will it decrease, and how much?
train time. How long does it take to train a model and environmental configuration ?
predict time. because you use TTA, How much time does each image take?
Sorry to ask you so many questions.
\n\nHope you can reply me, thank you!
", "votes": 2, "replies": [ { "id": 389051, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-09-18T04:16:42.840000", "content": "Thank you very much！！！
", "votes": 0, "replies": [] } ] }, { "id": 315863, "author_name": "Zhen Chen", "author_url": "", "post_date": "2018-04-17T17:31:03.293000", "content": "Congrats for your amazing work! I have two questions.
\n\nYou mentioned you got ~500 LB with the simplest approach and added watershed line for binary masks. I also tried that however, it cannot break ~400 LB. Could you please give more details about the Unet and watershed?
We have tried border in Mask-RCNN, however not sure how to combine it properly. Could you please give more details about the borders and masks combination?
BTW, the image you posted cannot read.
\n\nCongrats again!
", "votes": 1, "replies": [ { "id": 315887, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-17T18:26:14.120000", "content": "Images are visible in Chrome/Firefox, though in mobile Safari I don't see them . You can open the image by the link https://www.dropbox.com/s/4igam47pqg0i82q/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37_predict.png?dl=1
", "votes": 4, "replies": [] }, { "id": 315970, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-17T22:44:57.800000", "content": "Interesting, you think having the encoder networks being pretrained on ImageNet was crucial?
", "votes": 1, "replies": [] }, { "id": 316330, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-18T18:38:23.973000", "content": "@theNickName I tried to train a plain UNet from scratch using the same pipeline. The results were much worse.
", "votes": 4, "replies": [] }, { "id": 316594, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-19T12:27:12.197000", "content": "First of all, congratulations for sharing this nice work ! really topcoders !
\n\nI have not a lot of experience with pretrained models. Could you clarify how you use it:\n- encoder parameters are frozen during training ? \n- encoder parameters are finetuned ? \n- encoder parameters are retrained starting from an initial value coming from imagenet weights ?
", "votes": 3, "replies": [] }, { "id": 316595, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-19T12:30:19.817000", "content": "jeandebleau, I'm reading this https://arxiv.org/pdf/1801.05746.pdf at the moment
", "votes": 1, "replies": [] }, { "id": 316597, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-19T12:36:10.793000", "content": "Thank you for the link. In the paper they use the third option that I mentionned : encoders are initialized with pretrained weights. However it is not clear to me if this is the case here ; or even if it is not possible to simply freeze the encoder.
", "votes": 2, "replies": [] }, { "id": 316603, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-19T12:56:27.933000", "content": "That's right. Encoders were initialized with pretrained weights from ImageNet. Then we trained models end to end.\nFrom my experience with a frozen encoder it is usually not possible to achieve good segmentation results even on datasets that are more or less similar to ImageNet.
", "votes": 5, "replies": [] }, { "id": 316605, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-19T13:03:40.023000", "content": "Thank you very much!
", "votes": 0, "replies": [] }, { "id": 316635, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-19T14:55:14.907000", "content": "Yes, I was also wondering about that, thanks for the tip!
", "votes": 0, "replies": [] } ] }, { "id": 315847, "author_name": "Vadim Borisov", "author_url": "", "post_date": "2018-04-17T17:02:10.870000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 315737, "author_name": "ILJI CHOI", "author_url": "", "post_date": "2018-04-17T14:33:36.333000", "content": "Thanks for your sharing!! it is very helpful and makes me think a lot. And, Congrats !!
", "votes": 1, "replies": [] }, { "id": 315615, "author_name": "steelrose", "author_url": "", "post_date": "2018-04-17T10:55:43.863000", "content": "thanks for sharing, guess it would be too much to ask for the code :)
\n\nespecially interested in the LGB part
\n\nbtw. with this:
\n\n2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)
2 channels full masks i.e. (mask, border)
you mean that you have one model predicting 4 (resp. 5) channels, or two models?
", "votes": 1, "replies": [ { "id": 315617, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-17T10:59:54.053000", "content": "Actually different models. That gave a good profit from ensembling.
", "votes": 3, "replies": [] } ] }, { "id": 315602, "author_name": "DStjhb", "author_url": "", "post_date": "2018-04-17T10:40:39.010000", "content": "I had just started to learn the distance/contours directly from the network (after a tip via Heng) just after the models were frozen. Definitely the way to go it seems! I wonder if obtaining seeds from an object detection network would also improve the score for this model too. Congratz and great job guys!
", "votes": 1, "replies": [ { "id": 315815, "author_name": "AncientMonk", "author_url": "", "post_date": "2018-04-17T16:21:35.493000", "content": "Can you tell me how to add a channel with contours. I don't know what it means.
", "votes": 0, "replies": [] }, { "id": 315826, "author_name": "DStjhb", "author_url": "", "post_date": "2018-04-17T16:29:24.827000", "content": "topcoders may have implemented it differently to me, but I just create ground truth contours from the individual binary masks, then combine them in a single image channel. You can then use these as ground truth along with binary masks to teach U-Net. To output a second channel just increase the number of kernels in the final U-Net layer from 1 to 2, and make sure your inputs are consistent with this format.
", "votes": 2, "replies": [] }, { "id": 315836, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-17T16:44:38.240000", "content": "Yes, targets == ground truth masks as @DStjhb already described.\nIn general, Fully Convolutional Networks can predict and learn as many output channels as you wish. You just need to change the number of filters in the last convolution appropriately.
", "votes": 3, "replies": [] }, { "id": 315917, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-17T19:45:36.623000", "content": "I tried that at the start and it looked nasty. Of course, it might be that I made a bug somewhere at the beginning, when I tried that, I make those quite often, sadly, but, at least I made up a different explanation along the way. Maybe, at least, the losses have to be somehow balanced. You're chasing the avg gradient generated by 2, or however many losses (from different masks) in the end. And the base data that you're dealing with is not precise at all. So if you're training borders and you get a high loss, that one doesn't necessarily lead you in the right direction and may pull away the weight updates from other masks. It surprises me this approach worked magic for you guys, as I discarded it very close to the start, seeing it didn't work too well for me. In the end I settled on building maps of inner parts of nucleuses, focused on recognizing touch zones, inner part of nucleus and boundaries (1 for inner portion of nucleus and 0 for everything else on the image), in one net and then using it as a pretrain for a net that is supposed to differentiate between nuclei and background. Seemed like a pretty great pretrain from experiments.
\n\nOn the other hand, if I actually struggled only due to some bug, it might be a better choice, because you're teaching the net to better recognize and appreciate important hard-to-notice details of images. Though, it remains up to experimentation to see what approach is better under what circumstances - first training for the hard task and then using that for the easy one, or the conjoined option.
\n\nI gotta say, how one loss-weights different pixels makes a huge difference. I wonder how you guys weighted them.
", "votes": 1, "replies": [] }, { "id": 316169, "author_name": "DStjhb", "author_url": "", "post_date": "2018-04-18T10:21:35.937000", "content": "My model had difficulties learning the exact euclidean distance. The initial training resulted in strange artefacts, such as vertical lines in the output. It helps to simplify things a little, e.g. into distance 'bins' etc.
\n\nWhen learning a border region between cells, not a full distance gradient basin, like the solution here, my model doesn't have many issues. Maybe you just need more depth.
", "votes": 0, "replies": [] }, { "id": 316587, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-19T11:49:23.760000", "content": "DStjhb, if you're about me, I had plenty of depth. My model had residual units resnext style both ways - encoder and decoder.
", "votes": 0, "replies": [] } ] }, { "id": 315562, "author_name": "Lewis Crawford", "author_url": "", "post_date": "2018-04-17T09:24:28.313000", "content": "Congratulations! that is a very impressive score.
", "votes": 1, "replies": [] }, { "id": 327086, "author_name": "SimonGraham", "author_url": "", "post_date": "2018-05-10T19:43:45.837000", "content": "One further comment: Could you please elaborate on how you combined BCE with Dice. I have typically only seen one used or the other.
\n\nThanks
", "votes": 2, "replies": [ { "id": 327162, "author_name": "theNickName", "author_url": "", "post_date": "2018-05-10T23:47:45.037000", "content": "The logic behind why to combine those interests me too. I'm not strong at that sort of math, but it's interesting to know how it works on a, lets call it \"logical\", \"informational\" level.
", "votes": 0, "replies": [] }, { "id": 327347, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-05-11T10:40:46.197000", "content": "informally
\n\nTheir combination is a workaround to alleviate the problems existing in pure bce or dice losses.
\n\nTo combine them the simplest approach is
\n\nloss = bce + (1 - dice)\nDepending on the data you can play with the weights for the loss terms and see the outcome
\n\nloss = w1 * bce + w2 * (1 - dice)\nGreat- thanks so much. This is a nice idea. One further question that I have been trying to figure out:
\n\nYou mentioned that you used the Janowczyk dataset. However, this is non-exhaustively labeled. Can you please comment on how you dealt with this non-exhaustive dataset.
\n\nCheers
", "votes": 0, "replies": [] }, { "id": 396701, "author_name": "kwiechen", "author_url": "", "post_date": "2018-10-01T08:12:09.297000", "content": "This would be of particular interest. The Janowczyk dataset contains large H&E images with nuclei partially annotated. How do you dealt with nuclear areas not annotated?
", "votes": 0, "replies": [] }, { "id": 396703, "author_name": "Victor Durnov", "author_url": "", "post_date": "2018-10-01T08:21:30.323000", "content": "We just used few small crops from it where annotated areas larger some %. Wrote simple algorithm to find these patches.
", "votes": 0, "replies": [] } ] }, { "id": 318069, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-23T05:56:40.713000", "content": "@Selim_Sef
\n\nI am trying to implement your method and repeat the below results:
\n\n\"First we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). This approach gave us around ~500 on Public LB. Clearly this is not enough to win.\"
\n\nI run into issues and have the following questions:
\n\nThe 0.5 public LB results is for one network without test-time augmentation (TTA) , one network with TTA or ensemble with TTA?
Do you try resent50 for your unet. If yes, how is the results?
Applying watershed seems straight forward but it is the most crucial part to get good results. Do you have code snippets to show how to post process mask and border channel output from your unet?
If possible, would you mind showing more images for the mask and border channel output from your unet, and also watershed post processing results? (or you can show us the 0.5 public LB submission csv file)
Thank you very much!
", "votes": 2, "replies": [ { "id": 318196, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-23T11:25:47.957000", "content": "Hi Heng,
\n\n\n\n\ndef label_mask(mask_img, border_img, seed_ths, threshold, seed_size=8, obj_size=10):\n img_copy = np.copy(mask_img)\n m = img_copy * (1 - border_img)\n img_copy[m <= seed_ths] = 0\n img_copy[m > seed_ths] = 1\n img_copy = img_copy.astype(np.bool)\n img_copy = remove_small_objects(img_copy, seed_size).astype(np.uint8)\n mask_img[mask_img <= threshold] = 0\n mask_img[mask_img > threshold] = 1\n mask_img = mask_img.astype(np.bool)\n mask_img = remove_small_objects(mask_img, obj_size).astype(np.uint8)\n markers = ndimage.label(img_copy, output=np.uint32)[0]\n labels = watershed(mask_img, markers, mask=mask_img, watershed_line=True)\n return labels\n
\n\n4 You can get the idea how it works looking at these predictions from stage 2 (single resnet152)\n\n
Absolutely gorgeous work!
", "votes": 6, "replies": [] }, { "id": 318779, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-24T13:15:20.567000", "content": "@Selim_Sef
\n\nThank you very much the details. I will implement it and feedback to you later. Thanks again!
", "votes": 1, "replies": [] }, { "id": 318799, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-24T13:57:20.587000", "content": "After implementing @topcoders solution, I have to say their work is extraordinary smart.
\n\nIn my initial experiments, i actually tried Unet but i gave it up because i have problem in the post processing of unet outputs. Watershed is very sensitive to input marker and energy function (distance transform). That is why I switch to mask-rcnn.
\n\n@topcoders solution use unet to learn the \"border of nuclei\", which is made from watershed line of the ground truth. In this way they can control the accuracy of the watershed post processing.
\n\nresnet50 seems to insufficient for the task. I am now trying resnet101. i will post some results later.
", "votes": 15, "replies": [] }, { "id": 331822, "author_name": "", "author_url": "", "post_date": "2018-05-21T23:31:29.500000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 315823, "author_name": "harshml", "author_url": "", "post_date": "2018-04-17T16:28:09.193000", "content": "Respect, impressive 2nd stage approach! Can't imagine to see gbt here.
", "votes": 2, "replies": [] }, { "id": 315577, "author_name": "Ashish Lal", "author_url": "", "post_date": "2018-04-17T09:42:33.717000", "content": "Congratulations! Brilliant work!
\n\nHow much LB increase did the augmentations give?
\n\nAny chance that we can look at your code?
", "votes": 2, "replies": [ { "id": 315588, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-17T10:15:27.843000", "content": "It is quite hard to say how much exactly we gained from augmentations. But reducing them constanly led to better local validation/CV scores and much worse public scores.
", "votes": 2, "replies": [] } ] }, { "id": 315555, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2018-04-17T09:14:49.367000", "content": "Hats off! Great solution.
\n\nWhat HW resources did you use?
", "votes": 2, "replies": [ { "id": 315561, "author_name": "Victor Durnov", "author_url": "", "post_date": "2018-04-17T09:20:34.537000", "content": "Not so much actually) Everyone of us have by 2 GPUs (4*1080ti and 2*Titan XPs)
", "votes": 4, "replies": [] } ] }, { "id": 443729, "author_name": "luistelmocosta", "author_url": "", "post_date": "2018-12-22T09:12:00.163000", "content": "Did anyone deploy this model? I would like to apply this to a set of H&E stained images but it would take few days to train on a not-so-good graphics card.
", "votes": -2, "replies": [] }, { "id": 997692, "author_name": "Lance Chu", "author_url": "", "post_date": "2020-09-04T07:17:52.017000", "content": "Nice Tricks!
", "votes": 0, "replies": [] }, { "id": 414396, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-11-02T17:15:10.013000", "content": "Hi @Selim, from what I'm looking at you code it seems that during test you inferred on the original test image's size. Wouldn't that be a problem because since you trained on 256 crops, your network does not have stride enough to get context from bigger images? Example: most networks have 1/32 or 1/16 final stride, so for a 1024 pixels image I believe one would not get enough context even in the last layers. Am I wrong?
\n\nWould not be best crop the original image into 256 pixels pieces with some overlap, infer to all of them and finally join them back together?
\n\nThanks!
", "votes": 0, "replies": [ { "id": 416840, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-11-07T10:49:29.537000", "content": "For that models, no. Because fully convolutional networks almost do not depend on the input size, unless there are some blocks like SE/SCSE
", "votes": 2, "replies": [] }, { "id": 416843, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-11-07T11:10:16.413000", "content": "Thanks, you are always helping me out! But why SE block are problem? I know there are FC layers there but its only after a GlobalPooling layer, so we always get 1x1xC at the FC layer, no matter the input size. Right?
", "votes": 0, "replies": [] }, { "id": 417159, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-11-07T21:44:12.303000", "content": "Global avergae pooling will produce different features. Even though it will not break anything that may degrade performance.
", "votes": 2, "replies": [] } ] }, { "id": 410548, "author_name": "Karthik Chowdary Tsaliki", "author_url": "", "post_date": "2018-10-26T08:34:51.903000", "content": "Good job @selim. There are lot of interesting new aspects in you solution which I have to head on to.
", "votes": 0, "replies": [] }, { "id": 388560, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-09-17T07:33:07.977000", "content": "Thanks for a great report! I have some question to ask you.
\n\nif i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models?
if i only one model, how will the LB score change, will it decrease, and how much?
\n\npredict time. because you use TTA, How much time does each image take?
Sorry to ask you so many questions.
\n\nHope you can reply me, thank you!
", "votes": 0, "replies": [] }, { "id": 355229, "author_name": "sabari nathan", "author_url": "", "post_date": "2018-07-11T08:44:30.017000", "content": "Thanks for a great report. In the code you have mentioned about label image. But in training we have mask and train image. Could you share me the details about label image?
\n\n lbl = cv2.imread(path.join(labels_folder, '{0}.tif'.format(img_id)), cv2.IMREAD_UNCHANGED)\nLabels where generated from original data, i.e. for each image there was a single grayscale tif file with labeled instances. Afair there should be some code on kernels/forums which does that.
", "votes": 2, "replies": [] }, { "id": 443661, "author_name": "luistelmocosta", "author_url": "", "post_date": "2018-12-22T04:05:46.553000", "content": "I couldn't find a kernel that would generate those labels. Could you share what you did please? I'd love to replicate and try to understand your approach.
", "votes": 2, "replies": [] }, { "id": 477226, "author_name": "Yujie Chi", "author_url": "", "post_date": "2019-02-24T06:52:12.807000", "content": "Hello, have you solved the problem? I am also a little confused about the generation of the label images.
", "votes": 1, "replies": [] } ] }, { "id": 345546, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-06-20T02:54:14.823000", "content": "Hi @Selim, I've been trying to implement a solution similar to yours however when I use random crops for training my val score decreases comparing to resizing, even though I'm using soft dice + bce as loss. I believe that this have to do with class unbalancing. Did you use any kind of smart implementation to avoid unbalancing when cropping?
", "votes": 0, "replies": [ { "id": 345982, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-06-20T21:21:05.153000", "content": "if you use channelwise dice loss then you will not have border/body pixelwise loss imbalance. Resizing should be used only for train (or multiscale prediction during inference) time augmentations - i.e. random sized crop is resized to the same size e.g. 224 or 256.
\n\nTraining with random crops using proper size usually works fine on all segmentation problems. For this problem crop size 224 or 256 was optimal from my experience.
", "votes": 0, "replies": [] }, { "id": 346032, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-06-21T00:07:27.183000", "content": "Thanks for the reply.\n I'm not even using the borders yet. I'm having the imbalance between nuclei and background. \nMy net predicts two channels (foreground, background - one hot format), then I apply softmax and use soft dice + cross entropy as loss. Please, take a look:
\n\nelif self.loss == 'ce+wdice':\n # Binary cross entropy\n if self.use_weights:\n cls_weights = self.calc_class_weights(onehot_labels)\n weights = tf.squeeze(self.w_tf) + cls_weights\n else:\n weights = 1.0\n loss1 = tf.losses.softmax_cross_entropy(\n onehot_labels=onehot_labels,\n logits=self.logits_tf,\n weights=weights)\n loss1 = tf.reduce_mean(loss1)\n\n # Soft dice loss\n axis = [1, 2]\n epilson = 1e-5\n total = self.output_shape[0] * self.output_shape[1]\n w = tf.reduce_sum(onehot_labels, axis=axis) / total\n I = tf.reduce_sum(onehot_labels * self.logits_tf, axis=axis)\n balanced_I = I * (1 - w + epilson)\n l2_pred = tf.reduce_sum(tf.square(self.logits_tf), axis=axis)\n l2_true = tf.reduce_sum(tf.square(onehot_labels), axis=axis)\n dice_coeff = (2. * balanced_I + epilson) / \\\n (l2_true + l2_pred + epilson)\n loss2 = tf.subtract(1., tf.reduce_mean(dice_coeff))\n\n loss = loss1 + loss2\nyou should not use dice for background channel, that's the problem.
", "votes": 0, "replies": [] } ] }, { "id": 339900, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-06-07T22:16:19.797000", "content": "Hi @Selim, congratulations for your work!\nI noticed that you used InceptionResNetV2 as one of your encoder but you did not mentioned about the decoder architecture. How did you handle with the encoder's unpadded convolution? I mean, to make the encoder-decoder dimensions match.\nThanks a lot!
", "votes": 0, "replies": [ { "id": 339902, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-06-07T22:34:04.687000", "content": "HI, I simply changed encoders and replaced valid padding with same padding.
\nDecoders where of two flavours:\n - U-Net like with standard upsampling-conv approach\n - a custom FPN like decoder\n - their performance was the same
Thanks a lot!
", "votes": 0, "replies": [] } ] }, { "id": 326791, "author_name": "SimonGraham", "author_url": "", "post_date": "2018-05-10T10:48:13.250000", "content": "Congrats on the great result! Please can you enlighten us on whether you used Dropout within your network at all? I'm guessing probably not if you used ResNet as the encoder...
", "votes": 0, "replies": [ { "id": 326795, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-05-10T10:58:20.443000", "content": "Ordinary Dropout is usually harmful for convolutional layers. But SpatialDropout2D could be used for segmentation tasks and gives good results from my experience. \nWe did not use it here though because we already had excessive augmentations and adding dropout will increase training time even more.
", "votes": 0, "replies": [] }, { "id": 326804, "author_name": "theNickName", "author_url": "", "post_date": "2018-05-10T11:13:47.797000", "content": "Interesting, I wonder where you put SpatialDropout2D so that it adds precision. Where I tried putting it it basically destroyed the network.
", "votes": 0, "replies": [] }, { "id": 326836, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-05-10T12:03:53.760000", "content": "I used SpatialDropout2D just before the classification layer
", "votes": 1, "replies": [] }, { "id": 326992, "author_name": "SimonGraham", "author_url": "", "post_date": "2018-05-10T16:14:35.923000", "content": "Thanks for your recommendation! I will have to experiment! :)
", "votes": 0, "replies": [] } ] }, { "id": 320870, "author_name": "corochann", "author_url": "", "post_date": "2018-04-30T04:12:25.590000", "content": "Thank you for the explanation.\nI was curious about which of the Mask-RCNN vs UNET wins the first solution. And it was very interesting for me that UNET wins by adding a extra pre/postprocessing like adding border etc!
", "votes": 0, "replies": [] }, { "id": 320292, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-28T06:54:18.167000", "content": "@Selim_Sef How did you handle color images? Is it:
\n\none unet, 2-channel for all images, color + gray (1-class unet)
one unet, 2-channel for color images and 2-channel gray images (2-class unet)
one unet, 2-channel for color images. Another separate unet, 2-channel for gray images. (two 1-class unets)
Thanks!
", "votes": 0, "replies": [ { "id": 320524, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-28T23:56:51.907000", "content": "our solution is quite simple, we just use RGB everywhere
", "votes": 1, "replies": [] } ] }, { "id": 319828, "author_name": "", "author_url": "", "post_date": "2018-04-26T23:01:11.943000", "content": "", "votes": 0, "replies": [ { "id": 320028, "author_name": "", "author_url": "", "post_date": "2018-04-27T10:13:56.370000", "content": "", "votes": 2, "replies": [] }, { "id": 320130, "author_name": "", "author_url": "", "post_date": "2018-04-27T15:30:36.347000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 318787, "author_name": "", "author_url": "", "post_date": "2018-04-24T13:31:50.607000", "content": "", "votes": 0, "replies": [ { "id": 318853, "author_name": "", "author_url": "", "post_date": "2018-04-24T15:46:03.273000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 318045, "author_name": "", "author_url": "", "post_date": "2018-04-23T04:55:08.520000", "content": "", "votes": 0, "replies": [ { "id": 318179, "author_name": "", "author_url": "", "post_date": "2018-04-23T10:30:24.383000", "content": "", "votes": 1, "replies": [] }, { "id": 320242, "author_name": "", "author_url": "", "post_date": "2018-04-28T01:34:33.880000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 316785, "author_name": "", "author_url": "", "post_date": "2018-04-19T22:21:33.520000", "content": "", "votes": 0, "replies": [] }, { "id": 316148, "author_name": "", "author_url": "", "post_date": "2018-04-18T09:04:03.683000", "content": "", "votes": 0, "replies": [] }, { "id": 316025, "author_name": "", "author_url": "", "post_date": "2018-04-18T03:17:14.600000", "content": "", "votes": 0, "replies": [] }, { "id": 315958, "author_name": "", "author_url": "", "post_date": "2018-04-17T21:58:07.880000", "content": "", "votes": 0, "replies": [ { "id": 315960, "author_name": "", "author_url": "", "post_date": "2018-04-17T22:10:01.983000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 315894, "author_name": "", "author_url": "", "post_date": "2018-04-17T18:46:05.110000", "content": "", "votes": 0, "replies": [ { "id": 315908, "author_name": "", "author_url": "", "post_date": "2018-04-17T19:22:19.203000", "content": "", "votes": 2, "replies": [] }, { "id": 315919, "author_name": "", "author_url": "", "post_date": "2018-04-17T19:53:46.357000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 315563, "author_name": "", "author_url": "", "post_date": "2018-04-17T09:24:28.353000", "content": "", "votes": 0, "replies": [] }, { "id": 388559, "author_name": "", "author_url": "", "post_date": "2018-09-17T07:30:52.167000", "content": "", "votes": 0, "replies": [] }, { "id": 319755, "author_name": "", "author_url": "", "post_date": "2018-04-26T19:08:28.260000", "content": "", "votes": 2, "replies": [ { "id": 319830, "author_name": "", "author_url": "", "post_date": "2018-04-26T23:11:31.320000", "content": "", "votes": 2, "replies": [] }, { "id": 320002, "author_name": "", "author_url": "", "post_date": "2018-04-27T09:08:16.980000", "content": "", "votes": -24, "replies": [] }, { "id": 320021, "author_name": "", "author_url": "", "post_date": "2018-04-27T09:48:29.507000", "content": "", "votes": 9, "replies": [] }, { "id": 320107, "author_name": "", "author_url": "", "post_date": "2018-04-27T14:26:15.983000", "content": "", "votes": 2, "replies": [] }, { "id": 320272, "author_name": "", "author_url": "", "post_date": "2018-04-28T04:51:40.967000", "content": "", "votes": 16, "replies": [] }, { "id": 320276, "author_name": "", "author_url": "", "post_date": "2018-04-28T05:30:27.700000", "content": "", "votes": 0, "replies": [] }, { "id": 320302, "author_name": "", "author_url": "", "post_date": "2018-04-28T07:51:40.350000", "content": "", "votes": 7, "replies": [] }, { "id": 320315, "author_name": "", "author_url": "", "post_date": "2018-04-28T09:02:59.510000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 318978, "author_name": "", "author_url": "", "post_date": "2018-04-25T00:19:50.887000", "content": "", "votes": 0, "replies": [ { "id": 319142, "author_name": "", "author_url": "", "post_date": "2018-04-25T10:22:10.120000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 317264, "author_name": "", "author_url": "", "post_date": "2018-04-21T04:42:31.053000", "content": "", "votes": 0, "replies": [] }, { "id": 315809, "author_name": "", "author_url": "", "post_date": "2018-04-17T16:17:12.850000", "content": "", "votes": 1, "replies": [] }, { "id": 420032, "author_name": "", "author_url": "", "post_date": "2018-11-13T00:33:03.633000", "content": "", "votes": 0, "replies": [] }, { "id": 315539, "author_name": "", "author_url": "", "post_date": "2018-04-17T08:55:42.357000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "315536": "**Main contributions**\n------------------\n\n - targets - we predict touching borders along with the masks to solve\n the problem as instance segmentation\n - loss function - that combines crossentropy and soft dice loss in such\n a way that pixel imbalance doesn't affect the results\n - very deep encoder-decoder architectures that also achieve\n state-of-the-art results in other binary segmentation problems\n (SpaceNet, Inria and others)\n - tricky postprocessing that combines watershed, morphological features\n and second-level model with Gradient Boosted Trees\n - task specific data augmentations\n\n \nCombined these tricks allowed to achieve 1st place on public LB even with a single model.\n\n\n**U-Net vs Mask-RCNN?**\n-----------------------\n\nWe had experience in segmentation challenges before (Carvana, Urban 3D, Spacenet, Konica Minolta) where we actually competed against each other and therefore the choice was clear - **UNet on steroids**. \n\n\n**Target masks**\n-----------------\n\nFirst we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). \nThis approach gave us around ~500 on Public LB. Clearly this is not enough to win. \n\nThen we added second channel with contours, where width of contours depended on the nucleus size. \nThese masks + simple watershed postprocessing gave us ~525, not a breakthrough but a hint to the right direction.\n\nLooking at the mistakes it was clear that networks easily predict contours in non ambiguous places and fail miserably in places where we actually need contours to separate the nuclei. Hence we decided to predict only the borders between the cells. This easily gave us 550+ on the LB with a single network results + watershed postprocessing. \n\nStill if we have a full mask in one channel and a border in another, sometimes the seeds for watershed are not good enough. A better approach is to change nuclei masks and make pixels empty on the borders. This also allows to use softmax as the target activation instead of sigmoid. This better separates nuclei but actually decreases MaP because of high thresholds for IoU. We solved this issue with additional networks trained on full masks and combined the results in the postprocessing step.\n\n**Final approach for targets:**\n\n - 2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)\n\n - 2 channels full masks i.e. (mask, border)\n\nFor c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37 it looks like this\n![Prediction][1]\n\nFinal result after post processing:\n![Final result][2]\n\n\n\n**Augmentations**\n-----------------\n\nAs there were just some hundreds of training images, we needed to come up with specific augmentations to prevent our models from overfitting and make them more or less generalizable.\nWe used a lot of heavy augmentations (maybe too heavy)\n\n - Clahe, Sharpen, Emboss\n - Gaussian Noise\n - Color to Gray\n - Inverting - we should not have used it, some images were not\n predicted correctly on stage2 because of this augmentation\n - Remapping grayscale images to random color images\n - Blur, Median Blur, Motion Blur\n - contrast and brightness\n - random scale, rotates and flips\n - Heavy geometric transformations: Elastic Transform, Perspective\n Transform, Piecewise Affine transforms, pincushion distortion\n - Random HSV\n - Channel shuffle - I guess this one was very important due to the\n nature of the data\n - Nucleus copying on images. That created a lot of overlapping nuclei.\n It seemed to help networks to learn better borders for overlapping\n nuclei.\n\n**Network architectures**\n-------------------------\n\nWe used UNet like encoder-decoder architectures with encoders pretrained on ImageNet.\n\nSurprisingly, simple encoders like VGG16 did not work in this competition at all. They failed on the hard cases when the tissue looked like nucleus but it was not, especially on color images like 59b35151d4a7a5ffdd7ab7f171b142db8cfe40beeee67277fac6adca4d042c4\n\nAfter these experiments we decided that ***we have to go deeper***!!!\nAs a result - top performing encoders in this competition were : DPN-92, Resnet-152, InceptionResnetV2, Resnet101. \n\n**2nd Level model / Postprocessing**\n------------------------------------\n\nLightGBM models trained on predicted nucleus candidates. Each base candidate selected with lowest threshold for separation and tried to be separated with few higher thresholds and erosion. Used few basic morphological features about candidate like solidity, circularity, convexity, area, neighbors median area, count, etc.. Target for prediction - iou with ground truth (0 if iou < 0.5). \n\nThen best separation threshold selected for each candidate according to predicted iou. Candidates with small predicted iou just removed (iou < 0.3 and iou < 0.2 for 2 submissions - it was hard to find this threshold using OOF prediction, because there is small overfit to image types even with such hard augmentation).\n\n**Additional data:**\n--------------------\n\nWe added some images from janowczyk, nucleisegmentationbenchmark, isbi2009, BBBC020, TNBC datasets. Some of them decreased our score on Public LB, making less confident predictions on color images. Unfortunately we did not have time to find the culprits and left it as is. \n\n**Ensembling**\n--------------\n\nWe used a simple approach for ensembling where we just averaged our masks before postprocessing\n\n**Training**\n------------\n\n - Random Crops: 256x256\n\n - Batch Size: 16\n\n - optimizer: Adam\n\n - learning rate: initial 1e-4 with decay (we had different LR policies, but mostly small LR no more than 1e-4)\n\n - preprocessing: same as on ImageNet depending on network\n\n**Loss function**\n-----------------\n\n For networks with sigmoid activation and 2 channel masks we used combination of binary_crossentropy with soft_dice per channel. For networks with softmax activation and 3 channel masks we used combination of categorical_crossentropy with soft_dice per channel (soft dice was applied only to mask and border channels).\n\n**Test Time Augmentations (TTA)**\n-----------------------------\n\nStandard Flips/Rotations(0,90,180,270). \n\n [1]: https://www.dropbox.com/s/4igam47pqg0i82q/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37_predict.png?dl=1\n [2]: https://www.dropbox.com/s/isbils3jtelj297/c43e356beedae15fec60ae3f8b06ea8e9036081951deb7e44f481b15b3acfc37.png?dl=1\n\n**GitHub repo**\n-----------------------------\n https://github.com/selimsef/dsb2018_topcoders/", "315540": "Thanks for a great report! Which losses did you use and which optimizer? Any tricks with learning rate schedule? Which crop size did you use? How important/heavy were the scale augmentations? Did you use all augmentations at test time as well?", "319237": "Fantastic work. This was my first kaggle competition. Like many others, I too started with UNet but gave up as I couldn't improve the score after a while. Your post explains how kaggle experience, domain knowledge and collaborative thinking helps to get past mediocre scores with known networks. Thanks for the post. ", "315954": "Bravo, awesome work topcoders! Really great to see building on knowledge from previous competitions then smart adaptations to this one. ", "315592": "\"2nd Level model / Postprocessing\"\n\nI think this is important. Rejecting false positive can boost your score by quite a lot! Is there any comparison results with and without post processing?\n\nGood work!", "315541": "Wow, thanks for sharing. Congratulations on the solution and the prize win! Can you expand a bit more on your use of lgb? Also, how do you compute predicted IoU?", "362777": "I am working on the TGS competition and have been looking over the winners of old segmentation competitions and yours has a special significance because of the detail you have provided and the recency. My question is how did you apply regularization, if you did at all. Thank you", "388562": "Thanks for a great report! I have some question to ask you. \n\n1. if i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models?\n\n2. if i only one model, how will the LB score change, will it decrease, and how much? \n\n3. train time. How long does it take to train a model and environmental configuration ? \n\n4. predict time. because you use TTA, How much time does each image take? \n\nSorry to ask you so many questions. \n\nHope you can reply me, thank you!", "315863": "Congrats for your amazing work! I have two questions.\n\n1. You mentioned you got ~500 LB with the simplest approach and added watershed line for binary masks. I also tried that however, it cannot break ~400 LB. Could you please give more details about the Unet and watershed?\n\n2. We have tried border in Mask-RCNN, however not sure how to combine it properly. Could you please give more details about the borders and masks combination?\n\nBTW, the image you posted cannot read.\n\nCongrats again!", "315847": "Congrats! ", "315737": "Thanks for your sharing!! it is very helpful and makes me think a lot. And, Congrats !!", "315615": "thanks for sharing, guess it would be too much to ask for the code :)\n\nespecially interested in the LGB part\n\nbtw. with this:\n\n- 2 channels masks for networks with sigmoid activation i.e. (mask - border, border) or 3 channels masks for networks with softmax activation i.e. (mask - border, border , 1 - mask - border)\n\n- 2 channels full masks i.e. (mask, border)\n\nyou mean that you have one model predicting 4 (resp. 5) channels, or two models?\n", "315602": "I had just started to learn the distance/contours directly from the network (after a tip via Heng) just after the models were frozen. Definitely the way to go it seems! I wonder if obtaining seeds from an object detection network would also improve the score for this model too. Congratz and great job guys!", "315562": "Congratulations! that is a very impressive score.", "327086": "One further comment: Could you please elaborate on how you combined BCE with Dice. I have typically only seen one used or the other. \n\nThanks", "318069": "@Selim_Sef\n\nI am trying to implement your method and repeat the below results:\n\n\"First we tried the simplest approach and added watershed line for binary masks(i.e. modified GT masks to always have gaps between nuclei). This approach gave us around ~500 on Public LB. Clearly this is not enough to win.\"\n\n---\n\nI run into issues and have the following questions:\n\n 1. The 0.5 public LB results is for one network without test-time augmentation (TTA) , one network with TTA or ensemble with TTA?\n\n 2. Do you try resent50 for your unet. If yes, how is the results?\n\n 3. Applying watershed seems straight forward but it is the most crucial part to get good results. Do you have code snippets to show how to post process mask and border channel output from your unet?\n\n 4. If possible, would you mind showing more images for the mask and border channel output from your unet, and also watershed post processing results? (or you can show us the 0.5 public LB submission csv file)\n\nThank you very much!\n\n", "315823": "Respect, impressive 2nd stage approach! Can't imagine to see gbt here.", "315577": "Congratulations! Brilliant work! \n\nHow much LB increase did the augmentations give?\n\nAny chance that we can look at your code?", "315555": "Hats off! Great solution.\n\nWhat HW resources did you use?", "443729": "Did anyone deploy this model? I would like to apply this to a set of H&E stained images but it would take few days to train on a not-so-good graphics card.", "997692": "Nice Tricks!", "414396": "Hi @Selim, from what I'm looking at you code it seems that during test you inferred on the original test image's size. Wouldn't that be a problem because since you trained on 256 crops, your network does not have stride enough to get context from bigger images? Example: most networks have 1/32 or 1/16 final stride, so for a 1024 pixels image I believe one would not get enough context even in the last layers. Am I wrong?\n\nWould not be best crop the original image into 256 pixels pieces with some overlap, infer to all of them and finally join them back together?\n\nThanks!", "410548": "Good job @selim. There are lot of interesting new aspects in you solution which I have to head on to.", "388560": "Thanks for a great report! I have some question to ask you. \n\n1. if i want to get your LB score 0.631！(Omg! So much higher than other), Do i need to trained several models? \n\n2. if i only one model, how will the LB score change, will it decrease, and how much?\n\n 3. train time. How long does it take to train a model and environmental configuration ? \n\n4. predict time. because you use TTA, How much time does each image take? \n\nSorry to ask you so many questions. \n\nHope you can reply me, thank you!", "355229": "Thanks for a great report. In the code you have mentioned about label image. But in training we have mask and train image. Could you share me the details about label image?\n\n lbl = cv2.imread(path.join(labels_folder, '{0}.tif'.format(img_id)), cv2.IMREAD_UNCHANGED)\n", "345546": "Hi @Selim, I've been trying to implement a solution similar to yours however when I use random crops for training my val score decreases comparing to resizing, even though I'm using soft dice + bce as loss. I believe that this have to do with class unbalancing. Did you use any kind of smart implementation to avoid unbalancing when cropping?", "339900": "Hi @Selim, congratulations for your work!\nI noticed that you used InceptionResNetV2 as one of your encoder but you did not mentioned about the decoder architecture. How did you handle with the encoder's unpadded convolution? I mean, to make the encoder-decoder dimensions match.\nThanks a lot! ", "326791": "Congrats on the great result! Please can you enlighten us on whether you used Dropout within your network at all? I'm guessing probably not if you used ResNet as the encoder... ", "320870": "Thank you for the explanation.\nI was curious about which of the Mask-RCNN vs UNET wins the first solution. And it was very interesting for me that UNET wins by adding a extra pre/postprocessing like adding border etc!", "320292": "@Selim_Sef How did you handle color images? Is it:\n\n1. one unet, 2-channel for all images, color + gray (1-class unet)\n\n2. one unet, 2-channel for color images and 2-channel gray images (2-class unet)\n\n3. one unet, 2-channel for color images. Another separate unet, 2-channel for gray images. (two 1-class unets)\n\nThanks!", "319828": "One more question: what modification of original training data did you use?", "318787": "What would happen if you trained a mask-RCNN on your UNET channels + predictions + images. Could you get the best of both worlds?", "318045": "Congrats!!\n\nAnd how to predict only the borders between the cells using unet ??\nI don't know how to judge if the borders between the cells or not...", "316785": "Congrats, very impressive solution!", "316148": "Congratulations!! Thank you so much for sharing your strategy which is really helpful for us to understand and learn it. Hope you guys keep the great works! Now I realize how important is to have deep pre-trained networks for such a good competition.", "316025": "Congratulations .Phenomenal work .Love to learn this area in detail.Greatly appreciate if you could share some learning sites on the same for our initial understanding ", "315958": "Hi Selim\n\nI am quite new to all the techniques you are describing here and I've learned a lot.\n\nI just want to clarify this to ensure my understanding is correct:\n\"Then we added second channel with contours, where width of contours depended on the nucleus size. \"\n\nwhen you say adding contour I assume you mean， instead of binary mask， you have some sort of value that is decreasing from the centre of the nuclei towards the edge of the nuclei?", "315894": "Congrats! \nHow much was this a contest over \"generalizability\" to you? And was the use of external data overall a helpful, or hurtful thing in your experience? I see you're the only other person in discussions who said external data and inverting was at least somewhat hurtful when everybody else was screaming how they were surprised at the fact that stage 2 data is somewhat different than stage 1 and that they would've used more external data and heavier augmentations if they knew what's coming.\n\nMy guess is BBBC020 was definitely crucial to your victory, though. All BBBC datasets work very well with this challenge. With stage 2. Pretty much unlike any other datasets, most of which would only hurt.", "315563": "Congrats, Great work. ", "388559": "", "319755": "", "318978": "", "317264": "", "315809": "Great explanation, thanks for sharing!", "420032": "Thank you for the write up and Github!", "315539": "thanks" }, "complete": true, "source": "existing_discussion", "competition_id": 8089, "competition_slug": "data-science-bowl-2018", "team_id": 1448280, "solution_url": "/competitions/data-science-bowl-2018/writeups/ods-ai-topcoders-ods-ai-topcoders-1st-place-soluti" }