![\"enter](\"http://openresearch.ai/uploads/default/optimized/1X/ec0ac2e2d2df8f213b916453375ccee95a254ac3_1_616x500.png\")
I know that many participants have tried with Mask-RCNN or other 2 staged models, but I only used UNet-based model. Here are few things that were important to improve LB score.
\n\n(1) Using VALID Padding
\n\nIn kernels and discussion, many of participants implemented UNet but they are different from the original one.
\n\n
As you can see here, they used VALID padding not SAME padding. Also, they matched lower and upper features after cropping lower feature. Because, 3x3 Convolutional operations and pooling operations remove precise location information but add more hierachical information, so they intentionally matched two feature losing LESS precise location information.
\n\n(2) Weighted Cross Entropy Loss Function
\n\nMany unet implementation in this competition are using cross entropy between ground truth and output map from the network. This is okay but they need to be post-processed since outputs are just semantic information not instance-separated information. I saw Scikit's label function and other morphological operations were used.
\n\nBut in the original work, they used weighted cross entropy loss to separate instances.
\n\n![\"enter](\"http://openresearch.ai/uploads/default/original/1X/c8ce2c1cb851d5b913ffac5d997eceb6581d3325.png\")
As you can see here, we can add more weights on small gaps between cells and cells. Because if we guarantee that the output from the model will separate all cells, then we don't need to use complicated methods to process the output.
\n\nI use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.
\n\n(3) Some augmentations
\n\nDue to its one-shot and end-to-end architecture, it has some drawbacks. Mainly, it needs to learn cells of variety sizes. So I added scale-augmentation to make the network predicts small and large cells. And I also used other augmentations but they are not special, probably most of participants used them.
\n\n(4) Tuning it
\n\nAfter Tuning network parameters and add small extra datas, a single model without any ensemble reached LB 0.504. Using test-augmentation ensemble model(with 6 augmentation) was beyond LB 0.53.
", "messages": [ { "id": "313293", "postDate": "04/13/2018 04:18:35", "content": "I know that many participants have tried with Mask-RCNN or other 2 staged models, but I only used UNet-based model. Here are few things that were important to improve LB score.
\n\n(1) Using VALID Padding
\n\nIn kernels and discussion, many of participants implemented UNet but they are different from the original one.
\n\n
As you can see here, they used VALID padding not SAME padding. Also, they matched lower and upper features after cropping lower feature. Because, 3x3 Convolutional operations and pooling operations remove precise location information but add more hierachical information, so they intentionally matched two feature losing LESS precise location information.
\n\n(2) Weighted Cross Entropy Loss Function
\n\nMany unet implementation in this competition are using cross entropy between ground truth and output map from the network. This is okay but they need to be post-processed since outputs are just semantic information not instance-separated information. I saw Scikit's label function and other morphological operations were used.
\n\nBut in the original work, they used weighted cross entropy loss to separate instances.
\n\n
As you can see here, we can add more weights on small gaps between cells and cells. Because if we guarantee that the output from the model will separate all cells, then we don't need to use complicated methods to process the output.
\n\nI use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.
\n\n(3) Some augmentations
\n\nDue to its one-shot and end-to-end architecture, it has some drawbacks. Mainly, it needs to learn cells of variety sizes. So I added scale-augmentation to make the network predicts small and large cells. And I also used other augmentations but they are not special, probably most of participants used them.
\n\n(4) Tuning it
\n\nAfter Tuning network parameters and add small extra datas, a single model without any ensemble reached LB 0.504. Using test-augmentation ensemble model(with 6 augmentation) was beyond LB 0.53.
", "rawMarkdown": "I know that many participants have tried with Mask-RCNN or other 2 staged models, but I only used UNet-based model. Here are few things that were important to improve LB score.\n\n(1) Using VALID Padding\n\nIn kernels and discussion, many of participants implemented UNet but they are different from the original one. \n\n![enter image description here][1]\n\nAs you can see here, they used VALID padding not SAME padding. Also, they matched lower and upper features after cropping lower feature. Because, 3x3 Convolutional operations and pooling operations remove precise location information but add more hierachical information, so they intentionally matched two feature losing LESS precise location information. \n\n(2) Weighted Cross Entropy Loss Function\n\nMany unet implementation in this competition are using cross entropy between ground truth and output map from the network. This is okay but they need to be post-processed since outputs are just semantic information not instance-separated information. I saw Scikit's label function and other morphological operations were used.\n\nBut in the original work, they used weighted cross entropy loss to separate instances.\n\n![enter image description here][2]\n\nAs you can see here, we can add more weights on small gaps between cells and cells. Because if we guarantee that the output from the model will separate all cells, then we don't need to use complicated methods to process the output.\n\nI use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.\n\n(3) Some augmentations\n\nDue to its one-shot and end-to-end architecture, it has some drawbacks. Mainly, it needs to learn cells of variety sizes. So I added scale-augmentation to make the network predicts small and large cells. And I also used other augmentations but they are not special, probably most of participants used them. \n\n(4) Tuning it\n\nAfter Tuning network parameters and add small extra datas, a single model without any ensemble reached LB 0.504. Using test-augmentation ensemble model(with 6 augmentation) was beyond LB 0.53.\n\n\n [1]: http://openresearch.ai/uploads/default/optimized/1X/ec0ac2e2d2df8f213b916453375ccee95a254ac3_1_616x500.png\n [2]: http://openresearch.ai/uploads/default/original/1X/c8ce2c1cb851d5b913ffac5d997eceb6581d3325.png", "votes": null }, { "id": "313296", "postDate": "04/13/2018 04:22:48", "content": "Good work!
", "rawMarkdown": "Good work!", "votes": null }, { "id": "313301", "postDate": "04/13/2018 04:36:56", "content": "Did you do cross validation? Was it much different from the LB score?
", "rawMarkdown": "Did you do cross validation? Was it much different from the LB score?", "votes": null }, { "id": "313304", "postDate": "04/13/2018 04:43:27", "content": "I did. I implement the same metric as the competition. After I add more data, it's validation score was very near LB score(like 0.5~1.5% difference). That made me confident when I submit a new result with better validation score.
", "rawMarkdown": "I did. I implement the same metric as the competition. After I add more data, it's validation score was very near LB score(like 0.5~1.5% difference). That made me confident when I submit a new result with better validation score.", "votes": null }, { "id": "313311", "postDate": "04/13/2018 05:09:43", "content": "Very nice.
", "rawMarkdown": "Very nice.", "votes": null }, { "id": "313315", "postDate": "04/13/2018 05:13:06", "content": "Thanks for sharing. We used a similar approach and reached the same LB, but this is way better than what we did:
\n\n> I use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.
\n\nSimple and efficient!
", "rawMarkdown": "Thanks for sharing. We used a similar approach and reached the same LB, but this is way better than what we did:\n\n> I use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.\n\nSimple and efficient!", "votes": null }, { "id": "313328", "postDate": "04/13/2018 05:45:13", "content": "I saw many differences between unet implementations and original one\nand wondered which one would be better,\nabove explanation gave me so much insights. \nThank you.
", "rawMarkdown": "I saw many differences between unet implementations and original one\nand wondered which one would be better,\nabove explanation gave me so much insights. \nThank you.", "votes": null }, { "id": "313343", "postDate": "04/13/2018 06:04:51", "content": "Our fresh team had almost the same approaches and I wish we had time to combine both of them to reach your LB. We only used very simple data augmentation to reach LB = 0.504, didn't have time to fix the rescaling code based on the nuclei size as suggested by @Allen Goodman. I believe that would help a lot. And if we merged the other approach of our new team mate into ours (weighted cross entropy like yours) as implemented by the original U-net paper, it would've made our models a lot more robust. Lesson learned -> team early :) I think your models have a very good chance at the 2nd stage.
", "rawMarkdown": "Our fresh team had almost the same approaches and I wish we had time to combine both of them to reach your LB. We only used very simple data augmentation to reach LB = 0.504, didn't have time to fix the rescaling code based on the nuclei size as suggested by @Allen Goodman. I believe that would help a lot. And if we merged the other approach of our new team mate into ours (weighted cross entropy like yours) as implemented by the original U-net paper, it would've made our models a lot more robust. Lesson learned -> team early :) I think your models have a very good chance at the 2nd stage.", "votes": null }, { "id": "313403", "postDate": "04/13/2018 08:03:00", "content": "hi do you mind talking about more details in Weighted Cross Entropy Loss Function? Or where to find example codes. Thanks~
", "rawMarkdown": "hi do you mind talking about more details in Weighted Cross Entropy Loss Function? Or where to find example codes. Thanks~", "votes": null }, { "id": "313426", "postDate": "04/13/2018 08:45:43", "content": "This is really interesting. Will you publish your code once the competition is over?
", "rawMarkdown": "This is really interesting. Will you publish your code once the competition is over?", "votes": null }, { "id": "313430", "postDate": "04/13/2018 08:46:52", "content": "Thanks for sharing your great work and idea! Wish you best of luck!
", "rawMarkdown": "Thanks for sharing your great work and idea! Wish you best of luck!", "votes": null }, { "id": "313431", "postDate": "04/13/2018 08:47:38", "content": "Hi @Ildoo Kim, could you explain your contribution in (1) in more detail? For example, what do you mean by
\n\n\n\n\nthey intentionally matched two feature losing LESS precise location information
\n
Thanks!
", "rawMarkdown": "Hi @Ildoo Kim, could you explain your contribution in (1) in more detail? For example, what do you mean by \n\n> they intentionally matched two feature losing LESS precise location information\n\nThanks!", "votes": null }, { "id": "313443", "postDate": "04/13/2018 09:08:57", "content": "One more point I would like to add for better results from U-Net with weighted cross entropy function is that undoing(applying) the sigmoid on the output logits.
", "rawMarkdown": "One more point I would like to add for better results from U-Net with weighted cross entropy function is that undoing(applying) the sigmoid on the output logits.", "votes": null }, { "id": "313491", "postDate": "04/13/2018 10:39:11", "content": "More details are in the original paper. Briefly speaking, adding more weights on gaps between cells make much separable cells in the output.
", "rawMarkdown": "More details are in the original paper. Briefly speaking, adding more weights on gaps between cells make much separable cells in the output.", "votes": null }, { "id": "313492", "postDate": "04/13/2018 10:39:54", "content": "I will. (Due to lack of time and resources, codes are messed up but it will help some people.. so.. ^^)
", "rawMarkdown": "I will. (Due to lack of time and resources, codes are messed up but it will help some people.. so.. ^^)", "votes": null }, { "id": "313493", "postDate": "04/13/2018 10:40:54", "content": "Really? without sigmoid, what loss function did you use?
", "rawMarkdown": "Really? without sigmoid, what loss function did you use?", "votes": null }, { "id": "313499", "postDate": "04/13/2018 10:54:48", "content": "I am saying it for with sigmoid, because a function like sigmoid_cross_entropy applies sigmoid to logits only but it's my intuition. I may be wrong here but it has always improved my score.
", "rawMarkdown": "I am saying it for with sigmoid, because a function like sigmoid_cross_entropy applies sigmoid to logits only but it's my intuition. I may be wrong here but it has always improved my score.", "votes": null }, { "id": "313507", "postDate": "04/13/2018 11:30:11", "content": "Excellent !
\n\nI implementd more or less the same weighting scheme, but I also added a little more weight for the cells core.
\n\nI also have a remark about the padding. VALID padding is probably the safest way to go. However if you do not want to loose spatial extension, you can also safely use periodic or symetric padding wich is nice because it does not introduce new frequencies.
", "rawMarkdown": "Excellent ! \n\nI implementd more or less the same weighting scheme, but I also added a little more weight for the cells core.\n\nI also have a remark about the padding. VALID padding is probably the safest way to go. However if you do not want to loose spatial extension, you can also safely use periodic or symetric padding wich is nice because it does not introduce new frequencies.", "votes": null }, { "id": "313555", "postDate": "04/13/2018 13:25:25", "content": "I will be very happy yo see your code
", "rawMarkdown": "I will be very happy yo see your code", "votes": null }, { "id": "313585", "postDate": "04/13/2018 14:12:37", "content": "just for your information.
\n\nassume that you unet has: feature_net --> prob_layer
\n\na boundary can be implemented as: max_pool(prob_layer, 3x3 kernel)- min_pool(prob_layer, 3x3 kernel)
\n\nmax pool is like erosion, while min pool is like dilation. This is a natural way to add morphological layer in conv net?
", "rawMarkdown": "just for your information. \n\nassume that you unet has: feature_net --> prob_layer\n\na boundary can be implemented as: max_pool(prob_layer, 3x3 kernel)- min_pool(prob_layer, 3x3 kernel)\n\nmax pool is like erosion, while min pool is like dilation. This is a natural way to add morphological layer in conv net?", "votes": null }, { "id": "313619", "postDate": "04/13/2018 15:25:59", "content": "Interesting idea. I always wondered why max pooling is so dominant in CNN for mage processing as I would expect min pooling to also be useful. you just gave a great example of that.
\n\nI guess max pooling proved useful for minist where numbers are white over black. Your point makes me understand that eroding these numbers make the crisper and easier to recognize.
", "rawMarkdown": "Interesting idea. I always wondered why max pooling is so dominant in CNN for mage processing as I would expect min pooling to also be useful. you just gave a great example of that.\n\nI guess max pooling proved useful for minist where numbers are white over black. Your point makes me understand that eroding these numbers make the crisper and easier to recognize.", "votes": null }, { "id": "313623", "postDate": "04/13/2018 15:28:22", "content": "yes you are right. I used maxpool-original in my loss function. However it is difficult to fine the right weighting parameter. But this is also true for other segmentation methods mixing a region and a boundary term.
", "rawMarkdown": "yes you are right. I used maxpool-original in my loss function. However it is difficult to fine the right weighting parameter. But this is also true for other segmentation methods mixing a region and a boundary term.", "votes": null }, { "id": "313626", "postDate": "04/13/2018 15:36:42", "content": "I'm also not quite familiar with CNN yet but I think max pooling is dominant simply because in most cases min pooling should technically give the same result if applied before the loss function. The reason is that the weights prior to the pooling just needs to be multiplied by -1 to flip between them. The optimizing algorithm would learn either. Is this reasoning correct?
", "rawMarkdown": "I'm also not quite familiar with CNN yet but I think max pooling is dominant simply because in most cases min pooling should technically give the same result if applied before the loss function. The reason is that the weights prior to the pooling just needs to be multiplied by -1 to flip between them. The optimizing algorithm would learn either. Is this reasoning correct?", "votes": null }, { "id": "313629", "postDate": "04/13/2018 15:43:01", "content": "It looks correct to me, except for pooling before any learned layer.
", "rawMarkdown": "It looks correct to me, except for pooling before any learned layer.", "votes": null }, { "id": "313633", "postDate": "04/13/2018 15:51:48", "content": "Yes, and the input layer too. BTW, I started with Mask-RCNN but switched to U-Net because it was lighter and motivated by your good results. Too bad this time XGBoost is not an option!
", "rawMarkdown": "Yes, and the input layer too. BTW, I started with Mask-RCNN but switched to U-Net because it was lighter and motivated by your good results. Too bad this time XGBoost is not an option!", "votes": null }, { "id": "313645", "postDate": "04/13/2018 16:12:56", "content": "Hi @Ildoo Kim,
\n\n\n\n\nAs you can see here, they used VALID padding not SAME padding.
\n
I actually think VALID(no padding) is worse than SAME(zero) padding.
\n\nAt the time when original U-Net was introduced it was a breaktrough not because of \"no padding and cropping\" but due to the skip connections.\nUnfortunately, the reason for that choice is not described. I guess it was just a personal preference not a result of a thorough study.
\n\nUsing zero padding have several benefits:
\n\nit can be used with really deep networks (Resnet50 for example)
it actually improves performance on borders
The benefits of padding are also described in Karpathy's lecture notes\nhttp://cs231n.github.io/convolutional-networks/
\n\n\n", "rawMarkdown": "Hi @Ildoo Kim, \n\n> As you can see here, they used VALID padding not SAME padding. \n\nI actually think VALID(no padding) is worse than SAME(zero) padding.\n\nAt the time when original U-Net was introduced it was a breaktrough not because of \"no padding and cropping\" but due to the skip connections.\nUnfortunately, the reason for that choice is not described. I guess it was just a personal preference not a result of a thorough study. \n\nUsing zero padding have several benefits:\n\n - it preserves spatial dimensions\n - it is much easier to use with encoder-decoder architecture\n - it can be used with really deep networks (Resnet50 for example)\n\n - it actually improves performance on borders\n\nThe benefits of padding are also described in Karpathy's lecture notes\nhttp://cs231n.github.io/convolutional-networks/\n\n> Why use padding? In addition to the aforementioned benefit of keeping the spatial sizes constant after CONV, doing this actually improves performance. If the CONV layers were to not zero-pad the inputs and only perform valid convolutions, then the size of the volumes would reduce by a small amount after each CONV, and the information at the borders would be “washed away” too quickly.", "votes": null }, { "id": "313651", "postDate": "04/13/2018 16:25:21", "content": "Why use padding? In addition to the aforementioned benefit of keeping the spatial sizes constant after CONV, doing this actually improves performance. If the CONV layers were to not zero-pad the inputs and only perform valid convolutions, then the size of the volumes would reduce by a small amount after each CONV, and the information at the borders would be “washed away” too quickly.
\n
Karphathy also says that:\nhttps://twitter.com/karpathy/status/720622989289644033?lang=fr
\n\n\"Zero padding in ConvNets is highly suspicious/wrong. Input distribution stats are off on each border differently yet params are all shared\"
\n\nReflection or periodic or symetric padding has the benefit to not introduce new freqencies in the image signal.
", "rawMarkdown": "Karphathy also says that:\nhttps://twitter.com/karpathy/status/720622989289644033?lang=fr\n\n\"Zero padding in ConvNets is highly suspicious/wrong. Input distribution stats are off on each border differently yet params are all shared\"\n\nReflection or periodic or symetric padding has the benefit to not introduce new freqencies in the image signal.", "votes": null }, { "id": "313665", "postDate": "04/13/2018 16:58:31", "content": "\n\n\nI guess it was just a personal preference not a result of a thorough study.
\n
@Selim_Sef I think you are right about this. Deeplab only uses same padding as well. There are even more (hidden) design choices for FCNs:
\n\n\n\n\n\"DeepLab's Four Alignment Rules\":
\n \n(1) Use of odd-sized kernels in all convolution and pooling ops.
\n \n(2) Use of SAME boundary conditions in all convolution and pooling ops.
\n \n(3) Use align_corners=True when upsampling feature maps with bilinear interpolation.
\n \n(4) Use of inputs with height/width equal to a multiple of the output_stride, plus one (for example, when the CNN output stride is 8, use height or width equal to 8 * n + 1, for some n, e.g., image HxW set to 321x513).
\n
I still don't know what (4) is good for. Any ideas?
", "rawMarkdown": "> I guess it was just a personal preference not a result of a thorough study.\n\n@Selim_Sef I think you are right about this. Deeplab only uses same padding as well. There are even more (hidden) [design choices for FCNs](https://github.com/tensorflow/tensorflow/issues/6720#issuecomment-298190596):\n> \"DeepLab's Four Alignment Rules\":\n\n> (1) Use of odd-sized kernels in all convolution and pooling ops.\n\n> (2) Use of SAME boundary conditions in all convolution and pooling ops.\n\n> (3) Use align_corners=True when upsampling feature maps with bilinear interpolation.\n\n> (4) Use of inputs with height/width equal to a multiple of the output_stride, plus one (for example, when the CNN output stride is 8, use height or width equal to 8 * n + 1, for some n, e.g., image HxW set to 321x513).\n\nI still don't know what (4) is good for. Any ideas?", "votes": null }, { "id": "313668", "postDate": "04/13/2018 17:15:37", "content": "@Selim_Sef Thanks for your kind explanation. Your reply gave me lots of insights. I will definately look into your thoughts.
\n\nBut UNet uses VALID padding with 'mirror(reflection)' preprocessing of the input image like below.
\n\n![\"enter](\"http://openresearch.ai/uploads/default/optimized/1X/9db500ba287c18df96388b6250d9e6a571c0759b_1_690x375.jpg\")
This is different with the normal valid padding. I think this method helps for not only coarse-fined segmentation but also precise segmentation at the borders.
\n\nAlso I agree with the reply from @jeandebleau.
", "rawMarkdown": "Selim_Sef Thanks for your kind explanation. Your reply gave me lots of insights. I will definately look into your thoughts.\n\nBut UNet uses VALID padding with 'mirror(reflection)' preprocessing of the input image like below. \n\n![enter image description here][1]\n\nThis is different with the normal valid padding. I think this method helps for not only coarse-fined segmentation but also precise segmentation at the borders. \n\nAlso I agree with the reply from @jeandebleau.\n\n [1]: http://openresearch.ai/uploads/default/optimized/1X/9db500ba287c18df96388b6250d9e6a571c0759b_1_690x375.jpg", "votes": null }, { "id": "313684", "postDate": "04/13/2018 17:35:22", "content": "@Ildoo Kim, then it makes sense for me as well, they \"simulated\" reflection padding with the tools they had at the time.
\n\nVALID = NO PADDING, without the reflection preprocessing it would degrade performance significantly.
\n\n\n\n\nI still don't know what (4) is good for. Any ideas?
\n
Step 4 is to ensure that 3 is a divisor of H and W, I guess it slightly improves segmentation quality as well.
", "rawMarkdown": "Ildoo Kim, then it makes sense for me as well, they \"simulated\" reflection padding with the tools they had at the time. \n\nVALID = NO PADDING, without the reflection preprocessing it would degrade performance significantly.\n\n>I still don't know what (4) is good for. Any ideas?\n\nStep 4 is to ensure that 3 is a divisor of H and W, I guess it slightly improves segmentation quality as well.", "votes": null }, { "id": "313829", "postDate": "04/13/2018 22:20:27", "content": "Ildoo Kim, it's good that you posted that image with the reflection padding. It clears things up. When I read your initial post I was wondering about the loss of information on the right and bottom of the image, if you were to simply use valid padding.\nI'm curious to know if you used reflective padding as well in your code for the competition or was it true valid padding (without any reflections).
", "rawMarkdown": "Ildoo Kim, it's good that you posted that image with the reflection padding. It clears things up. When I read your initial post I was wondering about the loss of information on the right and bottom of the image, if you were to simply use valid padding.\nI'm curious to know if you used reflective padding as well in your code for the competition or was it true valid padding (without any reflections).", "votes": null }, { "id": "313876", "postDate": "04/14/2018 01:38:27", "content": "At the beggining of the competition, I used normal valid padding and then I changed it to reflection valid padding. Only changing it, LB score was improved by a some margin.
", "rawMarkdown": "At the beggining of the competition, I used normal valid padding and then I changed it to reflection valid padding. Only changing it, LB score was improved by a some margin.", "votes": null }, { "id": "313882", "postDate": "04/14/2018 01:57:26", "content": "Great work indeed! Are you using image patches or resizing?
", "rawMarkdown": "Great work indeed! Are you using image patches or resizing?", "votes": null }, { "id": "313940", "postDate": "04/14/2018 05:32:37", "content": "I didn't use image patches. Just random resizing for augmentation and reflection paddings are used. If the resized image is smaller than the input size then I resize again. Also, when I resize images, aspect ratio of them are not changed.
", "rawMarkdown": "I didn't use image patches. Just random resizing for augmentation and reflection paddings are used. If the resized image is smaller than the input size then I resize again. Also, when I resize images, aspect ratio of them are not changed.", "votes": null }, { "id": "313991", "postDate": "04/14/2018 10:03:03", "content": "Thanks for sharing. I want to know if you split the test data into color and gray and predict it respectively?
", "rawMarkdown": "Thanks for sharing. I want to know if you split the test data into color and gray and predict it respectively?", "votes": null }, { "id": "314001", "postDate": "04/14/2018 10:23:05", "content": "No i only trained a single model with color image input. Also i predict all images using that model.
", "rawMarkdown": "No i only trained a single model with color image input. Also i predict all images using that model.", "votes": null }, { "id": "314047", "postDate": "04/14/2018 13:51:57", "content": "Sorry, Im not sure if I understood you correctly.. That means that you dont have a fixed input dimension to your model? Like 256x256, for exemple ?
", "rawMarkdown": "Sorry, Im not sure if I understood you correctly.. That means that you dont have a fixed input dimension to your model? Like 256x256, for exemple ?", "votes": null }, { "id": "314051", "postDate": "04/14/2018 14:15:01", "content": "Well, I used random crop for training data. Input size was 228x228 before reflection padding added.
", "rawMarkdown": "Well, I used random crop for training data. Input size was 228x228 before reflection padding added.", "votes": null }, { "id": "314057", "postDate": "04/14/2018 14:27:15", "content": "I see, thanks! But when doing the predictions you resize your test image to 228x228 predict it and then resize it back again?
", "rawMarkdown": "I see, thanks! But when doing the predictions you resize your test image to 228x228 predict it and then resize it back again?", "votes": null }, { "id": "314078", "postDate": "04/14/2018 14:50:48", "content": "Well, at the training phase, there is no need to resize the output.
\n\nBut at the inference time, Using sliding window, many 228x228 crops are fed into the network and then outputs are merged. For submission, I do resize the final output to fit the original image.
", "rawMarkdown": "Well, at the training phase, there is no need to resize the output. \n\nBut at the inference time, Using sliding window, many 228x228 crops are fed into the network and then outputs are merged. For submission, I do resize the final output to fit the original image.", "votes": null }, { "id": "314357", "postDate": "04/15/2018 10:39:21", "content": "With the mirror padding any potentially negative impact of the \"SAME\" padding is going to be discarded anyway, but there are many benefits listed by @Selim_Sef
\n\nAnother interesting link, first the MSRA team modified the xception model and alongside other changed switched to the \"SAME\" padding (added pad 1) to improve the spatial alignment for Deformable Cov Nets based model:
\n\nhttp://presentations.cocodataset.org/COCO17-Detect-MSRA.pdf
\n\nthe same idea/model has been adopted by Google for their DeepLab-v3+
\n\nhttps://arxiv.org/pdf/1802.02611.pdf
\n\nOne advantage of using the \"VALID\" padding for unet - less computations as borders are discarded earlier in encoder/decoder, comparing to the final result when \"SAME\" used.
", "rawMarkdown": "With the mirror padding any potentially negative impact of the \"SAME\" padding is going to be discarded anyway, but there are many benefits listed by @Selim_Sef\n\nAnother interesting link, first the MSRA team modified the xception model and alongside other changed switched to the \"SAME\" padding (added pad 1) to improve the spatial alignment for Deformable Cov Nets based model:\n\nhttp://presentations.cocodataset.org/COCO17-Detect-MSRA.pdf\n\nthe same idea/model has been adopted by Google for their DeepLab-v3+\n\nhttps://arxiv.org/pdf/1802.02611.pdf\n\n\nOne advantage of using the \"VALID\" padding for unet - less computations as borders are discarded earlier in encoder/decoder, comparing to the final result when \"SAME\" used.", "votes": null }, { "id": "314790", "postDate": "04/16/2018 09:20:02", "content": "Thanks for sharing. I'm curious about your ensemble method! Did you simple take average with 6 augmentation? Did you need to remove small artifacts after tta?
", "rawMarkdown": "Thanks for sharing. I'm curious about your ensemble method! Did you simple take average with 6 augmentation? Did you need to remove small artifacts after tta?", "votes": null }, { "id": "314873", "postDate": "04/16/2018 12:09:18", "content": "Voting and NMS were used. I will post a wiki with codes in github. :)
", "rawMarkdown": "Voting and NMS were used. I will post a wiki with codes in github. :)", "votes": null }, { "id": "314875", "postDate": "04/16/2018 12:10:54", "content": "Hi,\nThanks for the explanation. Do you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?
", "rawMarkdown": "Hi,\nThanks for the explanation. Do you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?", "votes": null }, { "id": "314876", "postDate": "04/16/2018 12:16:34", "content": "\n\n\nDo you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?
\n
You'll find one in my code when I share it after competition end.
", "rawMarkdown": "> Do you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?\n\nYou'll find one in my code when I share it after competition end.", "votes": null }, { "id": "314915", "postDate": "04/16/2018 13:49:47", "content": "Thanks. I am looking forward to see that
", "rawMarkdown": "Thanks. I am looking forward to see that", "votes": null }, { "id": "314923", "postDate": "04/16/2018 14:07:13", "content": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..
\n\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy
", "rawMarkdown": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..\n\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy", "votes": null }, { "id": "314944", "postDate": "04/16/2018 14:36:01", "content": "Right, I could not use that because my loss is weighted log loss, not cross entropy.
", "rawMarkdown": "Right, I could not use that because my loss is weighted log loss, not cross entropy.", "votes": null }, { "id": "314946", "postDate": "04/16/2018 14:37:36", "content": "I tried to do something the last day, but was stuck with this: what to do if one mask with one augmentation maps to 2 masks with another augmentation?
\n\nEdit: your approach may just avoid the problem altogether as you trained on separated masks. Your idea was really the right one for Unet!
", "rawMarkdown": "I tried to do something the last day, but was stuck with this: what to do if one mask with one augmentation maps to 2 masks with another augmentation?\n\nEdit: your approach may just avoid the problem altogether as you trained on separated masks. Your idea was really the right one for Unet!", "votes": null }, { "id": "314981", "postDate": "04/16/2018 15:41:38", "content": "Do you have any reason to use log loss? Log loss wasn't an option for me.
", "rawMarkdown": "Do you have any reason to use log loss? Log loss wasn't an option for me.", "votes": null }, { "id": "314984", "postDate": "04/16/2018 15:48:30", "content": "Yes, I predict 3 outcomes: common boundary, union of masks, and background. The first two are not exclusive of each other.
", "rawMarkdown": "Yes, I predict 3 outcomes: common boundary, union of masks, and background. The first two are not exclusive of each other.", "votes": null }, { "id": "315112", "postDate": "04/16/2018 19:02:57", "content": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..\n\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy\nJust a reminder.. These functions are used to add weights to different samples in the batch as you can see in the docs. To use them to add weights to different pixels you need to feed the flattened tensors to the function. It is just a workaround but I guess it is worth mentioning..
", "rawMarkdown": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..\n \n https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy\n\nJust a reminder.. These functions are used to add weights to different samples in the batch as you can see in the docs. To use them to add weights to different pixels you need to feed the flattened tensors to the function. It is just a workaround but I guess it is worth mentioning..", "votes": null }, { "id": "315217", "postDate": "04/16/2018 22:11:23", "content": "@Edu :)
\n\nNo, If you provide a parameter of weights as the same size of your label(or output), then it will add weights to different pixels. See the document,
\n\n\n\n\nweights: Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding losses dimension).
\n
So, weights will be broadcasted into the label values...
", "rawMarkdown": "Edu :) \n\nNo, If you provide a parameter of weights as the same size of your label(or output), then it will add weights to different pixels. See the document,\n\n> weights: Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding losses dimension).\n\nSo, weights will be broadcasted into the label values...", "votes": null }, { "id": "315346", "postDate": "04/17/2018 01:58:54", "content": "https://github.com/ildoonet/data-science-bowl-2018
\n\nI uploaded my model. I got 41th position in private lb, but at the last time, I switched my selection of models to submit, If I didn't changed it, I would get lb 0.580 which is slightly better than my submission.
\n\nI will update wiki too. (currently it is in Korean)
", "rawMarkdown": "https://github.com/ildoonet/data-science-bowl-2018\n\nI uploaded my model. I got 41th position in private lb, but at the last time, I switched my selection of models to submit, If I didn't changed it, I would get lb 0.580 which is slightly better than my submission.\n\nI will update wiki too. (currently it is in Korean)", "votes": null }, { "id": "315350", "postDate": "04/17/2018 02:11:14", "content": "Cool thanks @Ildoo Kim! Congrats to your team! I was certain you'd get a gold medal though, since you combined the approaches of both of our models and we didn't get a chance to merge them. However, 0.58 wouldn't have been far from the very top either. I guess the weakness of the U-net is still that it puts a big emphasis on the differences between the boundaries and the cells. And at the stage 2, we saw tons of cells in \"water beads shape\" and the nucleus is actually in the middle of the cell. However, our U-net detects the whole cell instead of the nucleus only. I can't think of a better way to improve that unless we can train our U-net with this type of data.
\n\nCan you do me a favour and point me to your weight loss function? Since we couldn't use the tensorflow weighted cross entropy function, @CPMP wrote the weight loss function himself. I'm interested to find out how you used the tensorflow API. 고맙습니다 !
", "rawMarkdown": "Cool thanks @Ildoo Kim! Congrats to your team! I was certain you'd get a gold medal though, since you combined the approaches of both of our models and we didn't get a chance to merge them. However, 0.58 wouldn't have been far from the very top either. I guess the weakness of the U-net is still that it puts a big emphasis on the differences between the boundaries and the cells. And at the stage 2, we saw tons of cells in \"water beads shape\" and the nucleus is actually in the middle of the cell. However, our U-net detects the whole cell instead of the nucleus only. I can't think of a better way to improve that unless we can train our U-net with this type of data. \n\nCan you do me a favour and point me to your weight loss function? Since we couldn't use the tensorflow weighted cross entropy function, @CPMP wrote the weight loss function himself. I'm interested to find out how you used the tensorflow API. 고맙습니다 !", "votes": null }, { "id": "315353", "postDate": "04/17/2018 02:14:14", "content": "I guess that our team needed to improve the model further. This is promising way to tackle this kind of problem.
\n\nHere is the source code that I wrote for weighted cross entropy :
\n\n", "rawMarkdown": "I guess that our team needed to improve the model further. This is promising way to tackle this kind of problem.\n\nHere is the source code that I wrote for weighted cross entropy :\n\nhttps://github.com/ildoonet/data-science-bowl-2018/blob/curtis/basic-improve/network_unet_valid.py#L129", "votes": null }, { "id": "315365", "postDate": "04/17/2018 02:34:36", "content": "@Ildoo Kim,thanks for sharing your solution. I will definitely take a look at it next weekend.
", "rawMarkdown": "Ildoo Kim,thanks for sharing your solution. I will definitely take a look at it next weekend.", "votes": null }, { "id": "315420", "postDate": "04/17/2018 04:46:37", "content": "Thank you for sharing your source code. It is helpful me. 감사합니다 :)
", "rawMarkdown": "Thank you for sharing your source code. It is helpful me. 감사합니다 :)", "votes": null }, { "id": "315838", "postDate": "04/17/2018 16:45:19", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "315854", "postDate": "04/17/2018 17:14:30", "content": "Thanks for sharing your code Kim and congratulations on the reuslt! Just one question, I've noticed that you implemented the reflect padding you used by hand using numpy. Is there any particular reason to for not using the already implemented tf.pad ?
", "rawMarkdown": "Thanks for sharing your code Kim and congratulations on the reuslt! Just one question, I've noticed that you implemented the reflect padding you used by hand using numpy. Is there any particular reason to for not using the already implemented tf.pad ?", "votes": null }, { "id": "316006", "postDate": "04/18/2018 01:37:34", "content": "I used relfection paddings at preprocessing stage. You can also use tensorflow's one but it will be included in the graph, which is on gpu. Also you need to crop image with reflection paddings, so I decided to implement it by myself.
\n\nBut you can do exactly same thing with tensorflow's one.
", "rawMarkdown": "I used relfection paddings at preprocessing stage. You can also use tensorflow's one but it will be included in the graph, which is on gpu. Also you need to crop image with reflection paddings, so I decided to implement it by myself. \n\nBut you can do exactly same thing with tensorflow's one.", "votes": null }, { "id": "316440", "postDate": "04/19/2018 03:27:51", "content": "Hi Ildoo Kim,
\nI was trying to implement the valid padding version of U-Net in Keras but the weighted loss never goes below 0.3 and mIOU was stuck at 0.4. Is there any implementation detail you know that can avoid this?
\nThank you.
@IIdoo Kim
\n\nThis is different in API v1.8, which states:
\n\nweights: Optional Tensor whose rank is either 0, or rank 1 and is broadcastable to the loss which is a Tensor of shape [batch_size].\nhttps://www.tensorflow.org/api_docs/python/tf/losses/softmax_cross_entropy
\n\nBut I'm trying here the way you suggested and it seems to be working fine :)\nThanks
", "rawMarkdown": "IIdoo Kim\n\nThis is different in API v1.8, which states:\n\n weights: Optional Tensor whose rank is either 0, or rank 1 and is broadcastable to the loss which is a Tensor of shape [batch_size].\nhttps://www.tensorflow.org/api_docs/python/tf/losses/softmax_cross_entropy\n\nBut I'm trying here the way you suggested and it seems to be working fine :)\nThanks", "votes": null }, { "id": "322368", "postDate": "05/02/2018 20:29:25", "content": "Although it makes sense that they added reflection padding to the input and then used 'VALID' convs, if you look at the image of the whole net (first IIdoo post, u-net architecture) you will see that after each conv the size is reduced by 2, which implies that they used regular 'VALID' paddings. This is quite weird, right?
", "rawMarkdown": "Although it makes sense that they added reflection padding to the input and then used 'VALID' convs, if you look at the image of the whole net (first IIdoo post, u-net architecture) you will see that after each conv the size is reduced by 2, which implies that they used regular 'VALID' paddings. This is quite weird, right?", "votes": null }, { "id": "482147", "postDate": "03/02/2019 12:08:49", "content": "Is there any one here who can help me regarding the mentioned github repository
", "rawMarkdown": "Is there any one here who can help me regarding the mentioned github repository", "votes": null } ], "comments": [ { "id": 313296, "author_name": "hengck23", "author_url": "", "post_date": "04/13/2018 04:22:48", "content": "Good work!
", "votes": null, "replies": [] }, { "id": 313301, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 04:36:56", "content": "Did you do cross validation? Was it much different from the LB score?
", "votes": null, "replies": [ { "id": 313304, "author_name": "ildoonet", "author_url": "", "post_date": "04/13/2018 04:43:27", "content": "I did. I implement the same metric as the competition. After I add more data, it's validation score was very near LB score(like 0.5~1.5% difference). That made me confident when I submit a new result with better validation score.
", "votes": null, "replies": [] }, { "id": 313311, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 05:09:43", "content": "Very nice.
", "votes": null, "replies": [] } ] }, { "id": 313315, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 05:13:06", "content": "Thanks for sharing. We used a similar approach and reached the same LB, but this is way better than what we did:
\n\n> I use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.
\n\nSimple and efficient!
", "votes": null, "replies": [ { "id": 313585, "author_name": "hengck23", "author_url": "", "post_date": "04/13/2018 14:12:37", "content": "just for your information.
\n\nassume that you unet has: feature_net --> prob_layer
\n\na boundary can be implemented as: max_pool(prob_layer, 3x3 kernel)- min_pool(prob_layer, 3x3 kernel)
\n\nmax pool is like erosion, while min pool is like dilation. This is a natural way to add morphological layer in conv net?
", "votes": null, "replies": [] }, { "id": 313619, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 15:25:59", "content": "Interesting idea. I always wondered why max pooling is so dominant in CNN for mage processing as I would expect min pooling to also be useful. you just gave a great example of that.
\n\nI guess max pooling proved useful for minist where numbers are white over black. Your point makes me understand that eroding these numbers make the crisper and easier to recognize.
", "votes": null, "replies": [] }, { "id": 313623, "author_name": "jeandebleau", "author_url": "", "post_date": "04/13/2018 15:28:22", "content": "yes you are right. I used maxpool-original in my loss function. However it is difficult to fine the right weighting parameter. But this is also true for other segmentation methods mixing a region and a boundary term.
", "votes": null, "replies": [] }, { "id": 313626, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 15:36:42", "content": "I'm also not quite familiar with CNN yet but I think max pooling is dominant simply because in most cases min pooling should technically give the same result if applied before the loss function. The reason is that the weights prior to the pooling just needs to be multiplied by -1 to flip between them. The optimizing algorithm would learn either. Is this reasoning correct?
", "votes": null, "replies": [] }, { "id": 313629, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 15:43:01", "content": "It looks correct to me, except for pooling before any learned layer.
", "votes": null, "replies": [] }, { "id": 313633, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 15:51:48", "content": "Yes, and the input layer too. BTW, I started with Mask-RCNN but switched to U-Net because it was lighter and motivated by your good results. Too bad this time XGBoost is not an option!
", "votes": null, "replies": [] } ] }, { "id": 313328, "author_name": "soonhwankwon", "author_url": "", "post_date": "04/13/2018 05:45:13", "content": "I saw many differences between unet implementations and original one\nand wondered which one would be better,\nabove explanation gave me so much insights. \nThank you.
", "votes": null, "replies": [] }, { "id": 313343, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/13/2018 06:04:51", "content": "Our fresh team had almost the same approaches and I wish we had time to combine both of them to reach your LB. We only used very simple data augmentation to reach LB = 0.504, didn't have time to fix the rescaling code based on the nuclei size as suggested by @Allen Goodman. I believe that would help a lot. And if we merged the other approach of our new team mate into ours (weighted cross entropy like yours) as implemented by the original U-net paper, it would've made our models a lot more robust. Lesson learned -> team early :) I think your models have a very good chance at the 2nd stage.
", "votes": null, "replies": [] }, { "id": 313403, "author_name": "chouchris", "author_url": "", "post_date": "04/13/2018 08:03:00", "content": "hi do you mind talking about more details in Weighted Cross Entropy Loss Function? Or where to find example codes. Thanks~
", "votes": null, "replies": [ { "id": 313491, "author_name": "ildoonet", "author_url": "", "post_date": "04/13/2018 10:39:11", "content": "More details are in the original paper. Briefly speaking, adding more weights on gaps between cells make much separable cells in the output.
", "votes": null, "replies": [] }, { "id": 313555, "author_name": "aviwolfson", "author_url": "", "post_date": "04/13/2018 13:25:25", "content": "I will be very happy yo see your code
", "votes": null, "replies": [] } ] }, { "id": 313426, "author_name": "fotomaterjal", "author_url": "", "post_date": "04/13/2018 08:45:43", "content": "This is really interesting. Will you publish your code once the competition is over?
", "votes": null, "replies": [ { "id": 313492, "author_name": "ildoonet", "author_url": "", "post_date": "04/13/2018 10:39:54", "content": "I will. (Due to lack of time and resources, codes are messed up but it will help some people.. so.. ^^)
", "votes": null, "replies": [] } ] }, { "id": 313430, "author_name": "muhammedazamkhan", "author_url": "", "post_date": "04/13/2018 08:46:52", "content": "Thanks for sharing your great work and idea! Wish you best of luck!
", "votes": null, "replies": [] }, { "id": 313431, "author_name": "kaliev", "author_url": "", "post_date": "04/13/2018 08:47:38", "content": "Hi @Ildoo Kim, could you explain your contribution in (1) in more detail? For example, what do you mean by
\n\n\n\n\nthey intentionally matched two feature losing LESS precise location information
\n
Thanks!
", "votes": null, "replies": [] }, { "id": 313443, "author_name": "vijaybj", "author_url": "", "post_date": "04/13/2018 09:08:57", "content": "One more point I would like to add for better results from U-Net with weighted cross entropy function is that undoing(applying) the sigmoid on the output logits.
", "votes": null, "replies": [ { "id": 313493, "author_name": "ildoonet", "author_url": "", "post_date": "04/13/2018 10:40:54", "content": "Really? without sigmoid, what loss function did you use?
", "votes": null, "replies": [] }, { "id": 313499, "author_name": "vijaybj", "author_url": "", "post_date": "04/13/2018 10:54:48", "content": "I am saying it for with sigmoid, because a function like sigmoid_cross_entropy applies sigmoid to logits only but it's my intuition. I may be wrong here but it has always improved my score.
", "votes": null, "replies": [] } ] }, { "id": 313507, "author_name": "jeandebleau", "author_url": "", "post_date": "04/13/2018 11:30:11", "content": "Excellent !
\n\nI implementd more or less the same weighting scheme, but I also added a little more weight for the cells core.
\n\nI also have a remark about the padding. VALID padding is probably the safest way to go. However if you do not want to loose spatial extension, you can also safely use periodic or symetric padding wich is nice because it does not introduce new frequencies.
", "votes": null, "replies": [] }, { "id": 313645, "author_name": "selimsef", "author_url": "", "post_date": "04/13/2018 16:12:56", "content": "Hi @Ildoo Kim,
\n\n\n\n\nAs you can see here, they used VALID padding not SAME padding.
\n
I actually think VALID(no padding) is worse than SAME(zero) padding.
\n\nAt the time when original U-Net was introduced it was a breaktrough not because of \"no padding and cropping\" but due to the skip connections.\nUnfortunately, the reason for that choice is not described. I guess it was just a personal preference not a result of a thorough study.
\n\nUsing zero padding have several benefits:
\n\nit can be used with really deep networks (Resnet50 for example)
it actually improves performance on borders
The benefits of padding are also described in Karpathy's lecture notes\nhttp://cs231n.github.io/convolutional-networks/
\n\n\n", "votes": null, "replies": [ { "id": 313651, "author_name": "jeandebleau", "author_url": "", "post_date": "04/13/2018 16:25:21", "content": "Why use padding? In addition to the aforementioned benefit of keeping the spatial sizes constant after CONV, doing this actually improves performance. If the CONV layers were to not zero-pad the inputs and only perform valid convolutions, then the size of the volumes would reduce by a small amount after each CONV, and the information at the borders would be “washed away” too quickly.
\n
Karphathy also says that:\nhttps://twitter.com/karpathy/status/720622989289644033?lang=fr
\n\n\"Zero padding in ConvNets is highly suspicious/wrong. Input distribution stats are off on each border differently yet params are all shared\"
\n\nReflection or periodic or symetric padding has the benefit to not introduce new freqencies in the image signal.
", "votes": null, "replies": [] }, { "id": 313665, "author_name": "seesee", "author_url": "", "post_date": "04/13/2018 16:58:31", "content": "\n\n\nI guess it was just a personal preference not a result of a thorough study.
\n
@Selim_Sef I think you are right about this. Deeplab only uses same padding as well. There are even more (hidden) design choices for FCNs:
\n\n\n\n\n\"DeepLab's Four Alignment Rules\":
\n \n(1) Use of odd-sized kernels in all convolution and pooling ops.
\n \n(2) Use of SAME boundary conditions in all convolution and pooling ops.
\n \n(3) Use align_corners=True when upsampling feature maps with bilinear interpolation.
\n \n(4) Use of inputs with height/width equal to a multiple of the output_stride, plus one (for example, when the CNN output stride is 8, use height or width equal to 8 * n + 1, for some n, e.g., image HxW set to 321x513).
\n
I still don't know what (4) is good for. Any ideas?
", "votes": null, "replies": [] }, { "id": 313668, "author_name": "ildoonet", "author_url": "", "post_date": "04/13/2018 17:15:37", "content": "@Selim_Sef Thanks for your kind explanation. Your reply gave me lots of insights. I will definately look into your thoughts.
\n\nBut UNet uses VALID padding with 'mirror(reflection)' preprocessing of the input image like below.
\n\n
This is different with the normal valid padding. I think this method helps for not only coarse-fined segmentation but also precise segmentation at the borders.
\n\nAlso I agree with the reply from @jeandebleau.
", "votes": null, "replies": [] }, { "id": 313684, "author_name": "selimsef", "author_url": "", "post_date": "04/13/2018 17:35:22", "content": "@Ildoo Kim, then it makes sense for me as well, they \"simulated\" reflection padding with the tools they had at the time.
\n\nVALID = NO PADDING, without the reflection preprocessing it would degrade performance significantly.
\n\n\n\n\nI still don't know what (4) is good for. Any ideas?
\n
Step 4 is to ensure that 3 is a divisor of H and W, I guess it slightly improves segmentation quality as well.
", "votes": null, "replies": [] }, { "id": 314357, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "04/15/2018 10:39:21", "content": "With the mirror padding any potentially negative impact of the \"SAME\" padding is going to be discarded anyway, but there are many benefits listed by @Selim_Sef
\n\nAnother interesting link, first the MSRA team modified the xception model and alongside other changed switched to the \"SAME\" padding (added pad 1) to improve the spatial alignment for Deformable Cov Nets based model:
\n\nhttp://presentations.cocodataset.org/COCO17-Detect-MSRA.pdf
\n\nthe same idea/model has been adopted by Google for their DeepLab-v3+
\n\nhttps://arxiv.org/pdf/1802.02611.pdf
\n\nOne advantage of using the \"VALID\" padding for unet - less computations as borders are discarded earlier in encoder/decoder, comparing to the final result when \"SAME\" used.
", "votes": null, "replies": [] }, { "id": 322368, "author_name": "arc144", "author_url": "", "post_date": "05/02/2018 20:29:25", "content": "Although it makes sense that they added reflection padding to the input and then used 'VALID' convs, if you look at the image of the whole net (first IIdoo post, u-net architecture) you will see that after each conv the size is reduced by 2, which implies that they used regular 'VALID' paddings. This is quite weird, right?
", "votes": null, "replies": [] } ] }, { "id": 313829, "author_name": "robertkag", "author_url": "", "post_date": "04/13/2018 22:20:27", "content": "Ildoo Kim, it's good that you posted that image with the reflection padding. It clears things up. When I read your initial post I was wondering about the loss of information on the right and bottom of the image, if you were to simply use valid padding.\nI'm curious to know if you used reflective padding as well in your code for the competition or was it true valid padding (without any reflections).
", "votes": null, "replies": [ { "id": 313876, "author_name": "ildoonet", "author_url": "", "post_date": "04/14/2018 01:38:27", "content": "At the beggining of the competition, I used normal valid padding and then I changed it to reflection valid padding. Only changing it, LB score was improved by a some margin.
", "votes": null, "replies": [] } ] }, { "id": 313882, "author_name": "arc144", "author_url": "", "post_date": "04/14/2018 01:57:26", "content": "Great work indeed! Are you using image patches or resizing?
", "votes": null, "replies": [ { "id": 313940, "author_name": "ildoonet", "author_url": "", "post_date": "04/14/2018 05:32:37", "content": "I didn't use image patches. Just random resizing for augmentation and reflection paddings are used. If the resized image is smaller than the input size then I resize again. Also, when I resize images, aspect ratio of them are not changed.
", "votes": null, "replies": [] }, { "id": 314047, "author_name": "arc144", "author_url": "", "post_date": "04/14/2018 13:51:57", "content": "Sorry, Im not sure if I understood you correctly.. That means that you dont have a fixed input dimension to your model? Like 256x256, for exemple ?
", "votes": null, "replies": [] }, { "id": 314051, "author_name": "ildoonet", "author_url": "", "post_date": "04/14/2018 14:15:01", "content": "Well, I used random crop for training data. Input size was 228x228 before reflection padding added.
", "votes": null, "replies": [] }, { "id": 314057, "author_name": "arc144", "author_url": "", "post_date": "04/14/2018 14:27:15", "content": "I see, thanks! But when doing the predictions you resize your test image to 228x228 predict it and then resize it back again?
", "votes": null, "replies": [] }, { "id": 314078, "author_name": "ildoonet", "author_url": "", "post_date": "04/14/2018 14:50:48", "content": "Well, at the training phase, there is no need to resize the output.
\n\nBut at the inference time, Using sliding window, many 228x228 crops are fed into the network and then outputs are merged. For submission, I do resize the final output to fit the original image.
", "votes": null, "replies": [] } ] }, { "id": 313991, "author_name": "zjj421", "author_url": "", "post_date": "04/14/2018 10:03:03", "content": "Thanks for sharing. I want to know if you split the test data into color and gray and predict it respectively?
", "votes": null, "replies": [ { "id": 314001, "author_name": "ildoonet", "author_url": "", "post_date": "04/14/2018 10:23:05", "content": "No i only trained a single model with color image input. Also i predict all images using that model.
", "votes": null, "replies": [] } ] }, { "id": 314790, "author_name": "naotous", "author_url": "", "post_date": "04/16/2018 09:20:02", "content": "Thanks for sharing. I'm curious about your ensemble method! Did you simple take average with 6 augmentation? Did you need to remove small artifacts after tta?
", "votes": null, "replies": [ { "id": 314873, "author_name": "ildoonet", "author_url": "", "post_date": "04/16/2018 12:09:18", "content": "Voting and NMS were used. I will post a wiki with codes in github. :)
", "votes": null, "replies": [] }, { "id": 314946, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 14:37:36", "content": "I tried to do something the last day, but was stuck with this: what to do if one mask with one augmentation maps to 2 masks with another augmentation?
\n\nEdit: your approach may just avoid the problem altogether as you trained on separated masks. Your idea was really the right one for Unet!
", "votes": null, "replies": [] } ] }, { "id": 314875, "author_name": "masih4", "author_url": "", "post_date": "04/16/2018 12:10:54", "content": "Hi,\nThanks for the explanation. Do you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?
", "votes": null, "replies": [ { "id": 314876, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 12:16:34", "content": "\n\n\nDo you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?
\n
You'll find one in my code when I share it after competition end.
", "votes": null, "replies": [] }, { "id": 314915, "author_name": "masih4", "author_url": "", "post_date": "04/16/2018 13:49:47", "content": "Thanks. I am looking forward to see that
", "votes": null, "replies": [] }, { "id": 314923, "author_name": "ildoonet", "author_url": "", "post_date": "04/16/2018 14:07:13", "content": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..
\n\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy
", "votes": null, "replies": [] }, { "id": 314944, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 14:36:01", "content": "Right, I could not use that because my loss is weighted log loss, not cross entropy.
", "votes": null, "replies": [] }, { "id": 314981, "author_name": "ildoonet", "author_url": "", "post_date": "04/16/2018 15:41:38", "content": "Do you have any reason to use log loss? Log loss wasn't an option for me.
", "votes": null, "replies": [] }, { "id": 314984, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 15:48:30", "content": "Yes, I predict 3 outcomes: common boundary, union of masks, and background. The first two are not exclusive of each other.
", "votes": null, "replies": [] }, { "id": 315112, "author_name": "arc144", "author_url": "", "post_date": "04/16/2018 19:02:57", "content": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..\n\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy\nJust a reminder.. These functions are used to add weights to different samples in the batch as you can see in the docs. To use them to add weights to different pixels you need to feed the flattened tensors to the function. It is just a workaround but I guess it is worth mentioning..
", "votes": null, "replies": [] }, { "id": 315217, "author_name": "ildoonet", "author_url": "", "post_date": "04/16/2018 22:11:23", "content": "@Edu :)
\n\nNo, If you provide a parameter of weights as the same size of your label(or output), then it will add weights to different pixels. See the document,
\n\n\n\n\nweights: Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding losses dimension).
\n
So, weights will be broadcasted into the label values...
", "votes": null, "replies": [] }, { "id": 321508, "author_name": "arc144", "author_url": "", "post_date": "05/01/2018 13:05:42", "content": "@IIdoo Kim
\n\nThis is different in API v1.8, which states:
\n\nweights: Optional Tensor whose rank is either 0, or rank 1 and is broadcastable to the loss which is a Tensor of shape [batch_size].\nhttps://www.tensorflow.org/api_docs/python/tf/losses/softmax_cross_entropy
\n\nBut I'm trying here the way you suggested and it seems to be working fine :)\nThanks
", "votes": null, "replies": [] } ] }, { "id": 315346, "author_name": "ildoonet", "author_url": "", "post_date": "04/17/2018 01:58:54", "content": "https://github.com/ildoonet/data-science-bowl-2018
\n\nI uploaded my model. I got 41th position in private lb, but at the last time, I switched my selection of models to submit, If I didn't changed it, I would get lb 0.580 which is slightly better than my submission.
\n\nI will update wiki too. (currently it is in Korean)
", "votes": null, "replies": [ { "id": 315350, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/17/2018 02:11:14", "content": "Cool thanks @Ildoo Kim! Congrats to your team! I was certain you'd get a gold medal though, since you combined the approaches of both of our models and we didn't get a chance to merge them. However, 0.58 wouldn't have been far from the very top either. I guess the weakness of the U-net is still that it puts a big emphasis on the differences between the boundaries and the cells. And at the stage 2, we saw tons of cells in \"water beads shape\" and the nucleus is actually in the middle of the cell. However, our U-net detects the whole cell instead of the nucleus only. I can't think of a better way to improve that unless we can train our U-net with this type of data.
\n\nCan you do me a favour and point me to your weight loss function? Since we couldn't use the tensorflow weighted cross entropy function, @CPMP wrote the weight loss function himself. I'm interested to find out how you used the tensorflow API. 고맙습니다 !
", "votes": null, "replies": [] }, { "id": 315353, "author_name": "ildoonet", "author_url": "", "post_date": "04/17/2018 02:14:14", "content": "I guess that our team needed to improve the model further. This is promising way to tackle this kind of problem.
\n\nHere is the source code that I wrote for weighted cross entropy :
\n\n", "votes": null, "replies": [] }, { "id": 315420, "author_name": "joongjum", "author_url": "", "post_date": "04/17/2018 04:46:37", "content": "Thank you for sharing your source code. It is helpful me. 감사합니다 :)
", "votes": null, "replies": [] } ] }, { "id": 315365, "author_name": "sheriytm", "author_url": "", "post_date": "04/17/2018 02:34:36", "content": "@Ildoo Kim,thanks for sharing your solution. I will definitely take a look at it next weekend.
", "votes": null, "replies": [] }, { "id": 315838, "author_name": "harshml", "author_url": "", "post_date": "04/17/2018 16:45:19", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 315854, "author_name": "arc144", "author_url": "", "post_date": "04/17/2018 17:14:30", "content": "Thanks for sharing your code Kim and congratulations on the reuslt! Just one question, I've noticed that you implemented the reflect padding you used by hand using numpy. Is there any particular reason to for not using the already implemented tf.pad ?
", "votes": null, "replies": [ { "id": 316006, "author_name": "ildoonet", "author_url": "", "post_date": "04/18/2018 01:37:34", "content": "I used relfection paddings at preprocessing stage. You can also use tensorflow's one but it will be included in the graph, which is on gpu. Also you need to crop image with reflection paddings, so I decided to implement it by myself.
\n\nBut you can do exactly same thing with tensorflow's one.
", "votes": null, "replies": [] } ] }, { "id": 316440, "author_name": "ishootlaser", "author_url": "", "post_date": "04/19/2018 03:27:51", "content": "Hi Ildoo Kim,
\nI was trying to implement the valid padding version of U-Net in Keras but the weighted loss never goes below 0.3 and mIOU was stuck at 0.4. Is there any implementation detail you know that can avoid this?
\nThank you.
Is there any one here who can help me regarding the mentioned github repository
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "313293": "I know that many participants have tried with Mask-RCNN or other 2 staged models, but I only used UNet-based model. Here are few things that were important to improve LB score.\n\n(1) Using VALID Padding\n\nIn kernels and discussion, many of participants implemented UNet but they are different from the original one. \n\n![enter image description here][1]\n\nAs you can see here, they used VALID padding not SAME padding. Also, they matched lower and upper features after cropping lower feature. Because, 3x3 Convolutional operations and pooling operations remove precise location information but add more hierachical information, so they intentionally matched two feature losing LESS precise location information. \n\n(2) Weighted Cross Entropy Loss Function\n\nMany unet implementation in this competition are using cross entropy between ground truth and output map from the network. This is okay but they need to be post-processed since outputs are just semantic information not instance-separated information. I saw Scikit's label function and other morphological operations were used.\n\nBut in the original work, they used weighted cross entropy loss to separate instances.\n\n![enter image description here][2]\n\nAs you can see here, we can add more weights on small gaps between cells and cells. Because if we guarantee that the output from the model will separate all cells, then we don't need to use complicated methods to process the output.\n\nI use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.\n\n(3) Some augmentations\n\nDue to its one-shot and end-to-end architecture, it has some drawbacks. Mainly, it needs to learn cells of variety sizes. So I added scale-augmentation to make the network predicts small and large cells. And I also used other augmentations but they are not special, probably most of participants used them. \n\n(4) Tuning it\n\nAfter Tuning network parameters and add small extra datas, a single model without any ensemble reached LB 0.504. Using test-augmentation ensemble model(with 6 augmentation) was beyond LB 0.53.\n\n\n [1]: http://openresearch.ai/uploads/default/optimized/1X/ec0ac2e2d2df8f213b916453375ccee95a254ac3_1_616x500.png\n [2]: http://openresearch.ai/uploads/default/original/1X/c8ce2c1cb851d5b913ffac5d997eceb6581d3325.png", "313296": "Good work!", "313301": "Did you do cross validation? Was it much different from the LB score?", "313304": "I did. I implement the same metric as the competition. After I add more data, it's validation score was very near LB score(like 0.5~1.5% difference). That made me confident when I submit a new result with better validation score.", "313311": "Very nice.", "313315": "Thanks for sharing. We used a similar approach and reached the same LB, but this is way better than what we did:\n\n> I use erosion operation at the preprocessing to make sure that every cells are separated, and then use dilation operation to the final instances to adjust their sizes.\n\nSimple and efficient!", "313328": "I saw many differences between unet implementations and original one\nand wondered which one would be better,\nabove explanation gave me so much insights. \nThank you.", "313343": "Our fresh team had almost the same approaches and I wish we had time to combine both of them to reach your LB. We only used very simple data augmentation to reach LB = 0.504, didn't have time to fix the rescaling code based on the nuclei size as suggested by @Allen Goodman. I believe that would help a lot. And if we merged the other approach of our new team mate into ours (weighted cross entropy like yours) as implemented by the original U-net paper, it would've made our models a lot more robust. Lesson learned -> team early :) I think your models have a very good chance at the 2nd stage.", "313403": "hi do you mind talking about more details in Weighted Cross Entropy Loss Function? Or where to find example codes. Thanks~", "313426": "This is really interesting. Will you publish your code once the competition is over?", "313430": "Thanks for sharing your great work and idea! Wish you best of luck!", "313431": "Hi @Ildoo Kim, could you explain your contribution in (1) in more detail? For example, what do you mean by \n\n> they intentionally matched two feature losing LESS precise location information\n\nThanks!", "313443": "One more point I would like to add for better results from U-Net with weighted cross entropy function is that undoing(applying) the sigmoid on the output logits.", "313491": "More details are in the original paper. Briefly speaking, adding more weights on gaps between cells make much separable cells in the output.", "313492": "I will. (Due to lack of time and resources, codes are messed up but it will help some people.. so.. ^^)", "313493": "Really? without sigmoid, what loss function did you use?", "313499": "I am saying it for with sigmoid, because a function like sigmoid_cross_entropy applies sigmoid to logits only but it's my intuition. I may be wrong here but it has always improved my score.", "313507": "Excellent ! \n\nI implementd more or less the same weighting scheme, but I also added a little more weight for the cells core.\n\nI also have a remark about the padding. VALID padding is probably the safest way to go. However if you do not want to loose spatial extension, you can also safely use periodic or symetric padding wich is nice because it does not introduce new frequencies.", "313555": "I will be very happy yo see your code", "313585": "just for your information. \n\nassume that you unet has: feature_net --> prob_layer\n\na boundary can be implemented as: max_pool(prob_layer, 3x3 kernel)- min_pool(prob_layer, 3x3 kernel)\n\nmax pool is like erosion, while min pool is like dilation. This is a natural way to add morphological layer in conv net?", "313619": "Interesting idea. I always wondered why max pooling is so dominant in CNN for mage processing as I would expect min pooling to also be useful. you just gave a great example of that.\n\nI guess max pooling proved useful for minist where numbers are white over black. Your point makes me understand that eroding these numbers make the crisper and easier to recognize.", "313623": "yes you are right. I used maxpool-original in my loss function. However it is difficult to fine the right weighting parameter. But this is also true for other segmentation methods mixing a region and a boundary term.", "313626": "I'm also not quite familiar with CNN yet but I think max pooling is dominant simply because in most cases min pooling should technically give the same result if applied before the loss function. The reason is that the weights prior to the pooling just needs to be multiplied by -1 to flip between them. The optimizing algorithm would learn either. Is this reasoning correct?", "313629": "It looks correct to me, except for pooling before any learned layer.", "313633": "Yes, and the input layer too. BTW, I started with Mask-RCNN but switched to U-Net because it was lighter and motivated by your good results. Too bad this time XGBoost is not an option!", "313645": "Hi @Ildoo Kim, \n\n> As you can see here, they used VALID padding not SAME padding. \n\nI actually think VALID(no padding) is worse than SAME(zero) padding.\n\nAt the time when original U-Net was introduced it was a breaktrough not because of \"no padding and cropping\" but due to the skip connections.\nUnfortunately, the reason for that choice is not described. I guess it was just a personal preference not a result of a thorough study. \n\nUsing zero padding have several benefits:\n\n - it preserves spatial dimensions\n - it is much easier to use with encoder-decoder architecture\n - it can be used with really deep networks (Resnet50 for example)\n\n - it actually improves performance on borders\n\nThe benefits of padding are also described in Karpathy's lecture notes\nhttp://cs231n.github.io/convolutional-networks/\n\n> Why use padding? In addition to the aforementioned benefit of keeping the spatial sizes constant after CONV, doing this actually improves performance. If the CONV layers were to not zero-pad the inputs and only perform valid convolutions, then the size of the volumes would reduce by a small amount after each CONV, and the information at the borders would be “washed away” too quickly.", "313651": "Karphathy also says that:\nhttps://twitter.com/karpathy/status/720622989289644033?lang=fr\n\n\"Zero padding in ConvNets is highly suspicious/wrong. Input distribution stats are off on each border differently yet params are all shared\"\n\nReflection or periodic or symetric padding has the benefit to not introduce new freqencies in the image signal.", "313665": "> I guess it was just a personal preference not a result of a thorough study.\n\n@Selim_Sef I think you are right about this. Deeplab only uses same padding as well. There are even more (hidden) [design choices for FCNs](https://github.com/tensorflow/tensorflow/issues/6720#issuecomment-298190596):\n> \"DeepLab's Four Alignment Rules\":\n\n> (1) Use of odd-sized kernels in all convolution and pooling ops.\n\n> (2) Use of SAME boundary conditions in all convolution and pooling ops.\n\n> (3) Use align_corners=True when upsampling feature maps with bilinear interpolation.\n\n> (4) Use of inputs with height/width equal to a multiple of the output_stride, plus one (for example, when the CNN output stride is 8, use height or width equal to 8 * n + 1, for some n, e.g., image HxW set to 321x513).\n\nI still don't know what (4) is good for. Any ideas?", "313668": "Selim_Sef Thanks for your kind explanation. Your reply gave me lots of insights. I will definately look into your thoughts.\n\nBut UNet uses VALID padding with 'mirror(reflection)' preprocessing of the input image like below. \n\n![enter image description here][1]\n\nThis is different with the normal valid padding. I think this method helps for not only coarse-fined segmentation but also precise segmentation at the borders. \n\nAlso I agree with the reply from @jeandebleau.\n\n [1]: http://openresearch.ai/uploads/default/optimized/1X/9db500ba287c18df96388b6250d9e6a571c0759b_1_690x375.jpg", "313684": "Ildoo Kim, then it makes sense for me as well, they \"simulated\" reflection padding with the tools they had at the time. \n\nVALID = NO PADDING, without the reflection preprocessing it would degrade performance significantly.\n\n>I still don't know what (4) is good for. Any ideas?\n\nStep 4 is to ensure that 3 is a divisor of H and W, I guess it slightly improves segmentation quality as well.", "313829": "Ildoo Kim, it's good that you posted that image with the reflection padding. It clears things up. When I read your initial post I was wondering about the loss of information on the right and bottom of the image, if you were to simply use valid padding.\nI'm curious to know if you used reflective padding as well in your code for the competition or was it true valid padding (without any reflections).", "313876": "At the beggining of the competition, I used normal valid padding and then I changed it to reflection valid padding. Only changing it, LB score was improved by a some margin.", "313882": "Great work indeed! Are you using image patches or resizing?", "313940": "I didn't use image patches. Just random resizing for augmentation and reflection paddings are used. If the resized image is smaller than the input size then I resize again. Also, when I resize images, aspect ratio of them are not changed.", "313991": "Thanks for sharing. I want to know if you split the test data into color and gray and predict it respectively?", "314001": "No i only trained a single model with color image input. Also i predict all images using that model.", "314047": "Sorry, Im not sure if I understood you correctly.. That means that you dont have a fixed input dimension to your model? Like 256x256, for exemple ?", "314051": "Well, I used random crop for training data. Input size was 228x228 before reflection padding added.", "314057": "I see, thanks! But when doing the predictions you resize your test image to 228x228 predict it and then resize it back again?", "314078": "Well, at the training phase, there is no need to resize the output. \n\nBut at the inference time, Using sliding window, many 228x228 crops are fed into the network and then outputs are merged. For submission, I do resize the final output to fit the original image.", "314357": "With the mirror padding any potentially negative impact of the \"SAME\" padding is going to be discarded anyway, but there are many benefits listed by @Selim_Sef\n\nAnother interesting link, first the MSRA team modified the xception model and alongside other changed switched to the \"SAME\" padding (added pad 1) to improve the spatial alignment for Deformable Cov Nets based model:\n\nhttp://presentations.cocodataset.org/COCO17-Detect-MSRA.pdf\n\nthe same idea/model has been adopted by Google for their DeepLab-v3+\n\nhttps://arxiv.org/pdf/1802.02611.pdf\n\n\nOne advantage of using the \"VALID\" padding for unet - less computations as borders are discarded earlier in encoder/decoder, comparing to the final result when \"SAME\" used.", "314790": "Thanks for sharing. I'm curious about your ensemble method! Did you simple take average with 6 augmentation? Did you need to remove small artifacts after tta?", "314873": "Voting and NMS were used. I will post a wiki with codes in github. :)", "314875": "Hi,\nThanks for the explanation. Do you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?", "314876": "> Do you know any Keras/Tensorflow implementation code for the Weighted Cross Entropy Loss Function?\n\nYou'll find one in my code when I share it after competition end.", "314915": "Thanks. I am looking forward to see that", "314923": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..\n\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy\nhttps://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy", "314944": "Right, I could not use that because my loss is weighted log loss, not cross entropy.", "314946": "I tried to do something the last day, but was stuck with this: what to do if one mask with one augmentation maps to 2 masks with another augmentation?\n\nEdit: your approach may just avoid the problem altogether as you trained on separated masks. Your idea was really the right one for Unet!", "314981": "Do you have any reason to use log loss? Log loss wasn't an option for me.", "314984": "Yes, I predict 3 outcomes: common boundary, union of masks, and background. The first two are not exclusive of each other.", "315112": "If you use tensorflow, you can just use default function. Tensorflow already add a weight argument in cross entropy loss..\n \n https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/softmax_cross_entropy https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/losses/sigmoid_cross_entropy\n\nJust a reminder.. These functions are used to add weights to different samples in the batch as you can see in the docs. To use them to add weights to different pixels you need to feed the flattened tensors to the function. It is just a workaround but I guess it is worth mentioning..", "315217": "Edu :) \n\nNo, If you provide a parameter of weights as the same size of your label(or output), then it will add weights to different pixels. See the document,\n\n> weights: Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding losses dimension).\n\nSo, weights will be broadcasted into the label values...", "315346": "https://github.com/ildoonet/data-science-bowl-2018\n\nI uploaded my model. I got 41th position in private lb, but at the last time, I switched my selection of models to submit, If I didn't changed it, I would get lb 0.580 which is slightly better than my submission.\n\nI will update wiki too. (currently it is in Korean)", "315350": "Cool thanks @Ildoo Kim! Congrats to your team! I was certain you'd get a gold medal though, since you combined the approaches of both of our models and we didn't get a chance to merge them. However, 0.58 wouldn't have been far from the very top either. I guess the weakness of the U-net is still that it puts a big emphasis on the differences between the boundaries and the cells. And at the stage 2, we saw tons of cells in \"water beads shape\" and the nucleus is actually in the middle of the cell. However, our U-net detects the whole cell instead of the nucleus only. I can't think of a better way to improve that unless we can train our U-net with this type of data. \n\nCan you do me a favour and point me to your weight loss function? Since we couldn't use the tensorflow weighted cross entropy function, @CPMP wrote the weight loss function himself. I'm interested to find out how you used the tensorflow API. 고맙습니다 !", "315353": "I guess that our team needed to improve the model further. This is promising way to tackle this kind of problem.\n\nHere is the source code that I wrote for weighted cross entropy :\n\nhttps://github.com/ildoonet/data-science-bowl-2018/blob/curtis/basic-improve/network_unet_valid.py#L129", "315365": "Ildoo Kim,thanks for sharing your solution. I will definitely take a look at it next weekend.", "315420": "Thank you for sharing your source code. It is helpful me. 감사합니다 :)", "315838": "Thanks for sharing!", "315854": "Thanks for sharing your code Kim and congratulations on the reuslt! Just one question, I've noticed that you implemented the reflect padding you used by hand using numpy. Is there any particular reason to for not using the already implemented tf.pad ?", "316006": "I used relfection paddings at preprocessing stage. You can also use tensorflow's one but it will be included in the graph, which is on gpu. Also you need to crop image with reflection paddings, so I decided to implement it by myself. \n\nBut you can do exactly same thing with tensorflow's one.", "316440": "Hi Ildoo Kim,