IoU on the semantic segmentation (union of masks vs background) can be implemented as a loss, in a way similar to to the dice loss. Trick is to use a smoothing constant, see for instance this dice loss implementation
\n\nI'm typing here from memory, hence there may be typos, but a IoU code could look like the following. It assumes that masks have value 1 and background value 0.
\n\ndef iou_coef(y_true, y_pred, smooth=1):\n \"\"\"\n IoU = (|X & Y|)/ (|X or Y|)\n \"\"\"\n intersection = K.sum(K.abs(y_true * y_pred), axis=-1)\n union = K.sum((y_true,-1) + K.sum(y_pred,-1) - intersection\n return (intersection + smooth) / ( union + smooth)\n\ndef iou_coef_loss(y_true, y_pred):\n return -iou_coef(y_true, y_pred)\nThis loss can be used for semantic segmentation, for instance with Unet. In my case it gave worse results than with binary cross entropy, but I'm sharing in case others get better results with it.
\n\nPlease let us know if you get better results with the above loss vs binary crossentropy.
\n\nNote that this is NOT the metrics of the competition, as the competition metric is about IoU object per object (instance segmentation). I don't think we can implement the competition metric as a differentiable loss, but any useful proxy would be more than welcome.
", "messages": [ { "id": "293610", "postDate": "03/10/2018 08:33:02", "content": "IoU on the semantic segmentation (union of masks vs background) can be implemented as a loss, in a way similar to to the dice loss. Trick is to use a smoothing constant, see for instance this dice loss implementation
\n\nI'm typing here from memory, hence there may be typos, but a IoU code could look like the following. It assumes that masks have value 1 and background value 0.
\n\ndef iou_coef(y_true, y_pred, smooth=1):\n \"\"\"\n IoU = (|X & Y|)/ (|X or Y|)\n \"\"\"\n intersection = K.sum(K.abs(y_true * y_pred), axis=-1)\n union = K.sum((y_true,-1) + K.sum(y_pred,-1) - intersection\n return (intersection + smooth) / ( union + smooth)\n\ndef iou_coef_loss(y_true, y_pred):\n return -iou_coef(y_true, y_pred)\nThis loss can be used for semantic segmentation, for instance with Unet. In my case it gave worse results than with binary cross entropy, but I'm sharing in case others get better results with it.
\n\nPlease let us know if you get better results with the above loss vs binary crossentropy.
\n\nNote that this is NOT the metrics of the competition, as the competition metric is about IoU object per object (instance segmentation). I don't think we can implement the competition metric as a differentiable loss, but any useful proxy would be more than welcome.
", "rawMarkdown": "IoU on the semantic segmentation (union of masks vs background) can be implemented as a loss, in a way similar to to the dice loss. Trick is to use a smoothing constant, see for instance this [dice loss implementation][1]\n\nI'm typing here from memory, hence there may be typos, but a IoU code could look like the following. It assumes that masks have value 1 and background value 0.\n\n def iou_coef(y_true, y_pred, smooth=1):\n \"\"\"\n IoU = (|X & Y|)/ (|X or Y|)\n \"\"\"\n intersection = K.sum(K.abs(y_true * y_pred), axis=-1)\n union = K.sum((y_true,-1) + K.sum(y_pred,-1) - intersection\n return (intersection + smooth) / ( union + smooth)\n\n def iou_coef_loss(y_true, y_pred):\n return -iou_coef(y_true, y_pred)\n\nThis loss can be used for semantic segmentation, for instance with Unet. In my case it gave worse results than with binary cross entropy, but I'm sharing in case others get better results with it. \n\nPlease let us know if you get better results with the above loss vs binary crossentropy.\n\nNote that this is NOT the metrics of the competition, as the competition metric is about IoU object per object (instance segmentation). I don't think we can implement the competition metric as a differentiable loss, but any useful proxy would be more than welcome.\n\n [1]: https://gist.github.com/wassname/7793e2058c5c9dacb5212c0ac0b18a8a", "votes": null }, { "id": "293625", "postDate": "03/10/2018 08:59:55", "content": "@CPMP wow, cool, thank you! We modified William Cukierski's mean IoU code for evaluation only. It's always nice to try it and I'll check the local loss, not sure if I want to submit it to LB :) Yeah, work caught up with us, didn't have much time for the competition in the past couple days. Have fun with the toxic comment competition.
", "rawMarkdown": "CPMP wow, cool, thank you! We modified William Cukierski's mean IoU code for evaluation only. It's always nice to try it and I'll check the local loss, not sure if I want to submit it to LB :) Yeah, work caught up with us, didn't have much time for the competition in the past couple days. Have fun with the toxic comment competition.", "votes": null }, { "id": "293627", "postDate": "03/10/2018 09:10:04", "content": "Yes, I don't think I'll be active here until Toxic competition ends. Please don't make too much progress while I'm away. I see I already lost 10 ranks or so ;)
", "rawMarkdown": "Yes, I don't think I'll be active here until Toxic competition ends. Please don't make too much progress while I'm away. I see I already lost 10 ranks or so ;)", "votes": null }, { "id": "385135", "postDate": "09/10/2018 11:45:28", "content": "Thanks for this. I'm checking if it's better than binary crossentropy for me on the TGS salt challenge now. I changed the final output function to return the negative log, instead of just the negative. Was that also what you meant?
", "rawMarkdown": "Thanks for this. I'm checking if it's better than binary crossentropy for me on the TGS salt challenge now. I changed the final output function to return the negative log, instead of just the negative. Was that also what you meant?", "votes": null }, { "id": "385138", "postDate": "09/10/2018 11:50:27", "content": "What output function do you speak about? The output of NN model? It has to be the probabilities, i.e. the output of a sigmoid activation. That's what goes into y_pred here.
\n\nWhy would you take the log of iou? That's not what the TGS metric is about.
\n\nThe main issue with the above code is that it computes IoU per batch, and not per image. That may be why it was not that effective in DSB.
", "rawMarkdown": "What output function do you speak about? The output of NN model? It has to be the probabilities, i.e. the output of a sigmoid activation. That's what goes into y_pred here.\n\nWhy would you take the log of iou? That's not what the TGS metric is about.\n\nThe main issue with the above code is that it computes IoU per batch, and not per image. That may be why it was not that effective in DSB.", "votes": null }, { "id": "385151", "postDate": "09/10/2018 12:41:02", "content": "I mean the output of function iou_coef_loss. If you just take the negative it will return a negative fraction, i.e. 0 > x > -1. I don't know if there's mathematical justification (probably not), but taking the negative log instead of just the negative will output positive values. And in my case it also results in a training graph (epochs vs loss) that doesn't start off with a plateau, which makes it easier to see that the network is not stuck in some local minimum when you start training it.
", "rawMarkdown": "I mean the output of function iou_coef_loss. If you just take the negative it will return a negative fraction, i.e. 0 > x > -1. I don't know if there's mathematical justification (probably not), but taking the negative log instead of just the negative will output positive values. And in my case it also results in a training graph (epochs vs loss) that doesn't start off with a plateau, which makes it easier to see that the network is not stuck in some local minimum when you start training it.", "votes": null }, { "id": "385153", "postDate": "09/10/2018 12:44:21", "content": "I'll dig into my archive, as I ended not using this IoU loss, hence it is not in my github repo. I don't remember my code being stuck.
", "rawMarkdown": "I'll dig into my archive, as I ended not using this IoU loss, hence it is not in my github repo. I don't remember my code being stuck.", "votes": null }, { "id": "403276", "postDate": "10/13/2018 08:19:08", "content": "I've written an IOU metric for multi label image classification in keras, check this out:\nhttps://github.com/JeniaNovellusDx/KerasExtras/blob/master/keras_functions.py
", "rawMarkdown": "I've written an IOU metric for multi label image classification in keras, check this out:\nhttps://github.com/JeniaNovellusDx/KerasExtras/blob/master/keras_functions.py", "votes": null } ], "comments": [ { "id": 293625, "author_name": "nicolefinnie", "author_url": "", "post_date": "03/10/2018 08:59:55", "content": "@CPMP wow, cool, thank you! We modified William Cukierski's mean IoU code for evaluation only. It's always nice to try it and I'll check the local loss, not sure if I want to submit it to LB :) Yeah, work caught up with us, didn't have much time for the competition in the past couple days. Have fun with the toxic comment competition.
", "votes": null, "replies": [ { "id": 293627, "author_name": "cpmpml", "author_url": "", "post_date": "03/10/2018 09:10:04", "content": "Yes, I don't think I'll be active here until Toxic competition ends. Please don't make too much progress while I'm away. I see I already lost 10 ranks or so ;)
", "votes": null, "replies": [] } ] }, { "id": 385135, "author_name": "chrisdinant", "author_url": "", "post_date": "09/10/2018 11:45:28", "content": "Thanks for this. I'm checking if it's better than binary crossentropy for me on the TGS salt challenge now. I changed the final output function to return the negative log, instead of just the negative. Was that also what you meant?
", "votes": null, "replies": [ { "id": 385138, "author_name": "cpmpml", "author_url": "", "post_date": "09/10/2018 11:50:27", "content": "What output function do you speak about? The output of NN model? It has to be the probabilities, i.e. the output of a sigmoid activation. That's what goes into y_pred here.
\n\nWhy would you take the log of iou? That's not what the TGS metric is about.
\n\nThe main issue with the above code is that it computes IoU per batch, and not per image. That may be why it was not that effective in DSB.
", "votes": null, "replies": [] }, { "id": 385151, "author_name": "chrisdinant", "author_url": "", "post_date": "09/10/2018 12:41:02", "content": "I mean the output of function iou_coef_loss. If you just take the negative it will return a negative fraction, i.e. 0 > x > -1. I don't know if there's mathematical justification (probably not), but taking the negative log instead of just the negative will output positive values. And in my case it also results in a training graph (epochs vs loss) that doesn't start off with a plateau, which makes it easier to see that the network is not stuck in some local minimum when you start training it.
", "votes": null, "replies": [] }, { "id": 385153, "author_name": "cpmpml", "author_url": "", "post_date": "09/10/2018 12:44:21", "content": "I'll dig into my archive, as I ended not using this IoU loss, hence it is not in my github repo. I don't remember my code being stuck.
", "votes": null, "replies": [] } ] }, { "id": 403276, "author_name": "jenianovellusdx", "author_url": "", "post_date": "10/13/2018 08:19:08", "content": "I've written an IOU metric for multi label image classification in keras, check this out:\nhttps://github.com/JeniaNovellusDx/KerasExtras/blob/master/keras_functions.py
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "293610": "IoU on the semantic segmentation (union of masks vs background) can be implemented as a loss, in a way similar to to the dice loss. Trick is to use a smoothing constant, see for instance this [dice loss implementation][1]\n\nI'm typing here from memory, hence there may be typos, but a IoU code could look like the following. It assumes that masks have value 1 and background value 0.\n\n def iou_coef(y_true, y_pred, smooth=1):\n \"\"\"\n IoU = (|X & Y|)/ (|X or Y|)\n \"\"\"\n intersection = K.sum(K.abs(y_true * y_pred), axis=-1)\n union = K.sum((y_true,-1) + K.sum(y_pred,-1) - intersection\n return (intersection + smooth) / ( union + smooth)\n\n def iou_coef_loss(y_true, y_pred):\n return -iou_coef(y_true, y_pred)\n\nThis loss can be used for semantic segmentation, for instance with Unet. In my case it gave worse results than with binary cross entropy, but I'm sharing in case others get better results with it. \n\nPlease let us know if you get better results with the above loss vs binary crossentropy.\n\nNote that this is NOT the metrics of the competition, as the competition metric is about IoU object per object (instance segmentation). I don't think we can implement the competition metric as a differentiable loss, but any useful proxy would be more than welcome.\n\n [1]: https://gist.github.com/wassname/7793e2058c5c9dacb5212c0ac0b18a8a", "293625": "CPMP wow, cool, thank you! We modified William Cukierski's mean IoU code for evaluation only. It's always nice to try it and I'll check the local loss, not sure if I want to submit it to LB :) Yeah, work caught up with us, didn't have much time for the competition in the past couple days. Have fun with the toxic comment competition.", "293627": "Yes, I don't think I'll be active here until Toxic competition ends. Please don't make too much progress while I'm away. I see I already lost 10 ranks or so ;)", "385135": "Thanks for this. I'm checking if it's better than binary crossentropy for me on the TGS salt challenge now. I changed the final output function to return the negative log, instead of just the negative. Was that also what you meant?", "385138": "What output function do you speak about? The output of NN model? It has to be the probabilities, i.e. the output of a sigmoid activation. That's what goes into y_pred here.\n\nWhy would you take the log of iou? That's not what the TGS metric is about.\n\nThe main issue with the above code is that it computes IoU per batch, and not per image. That may be why it was not that effective in DSB.", "385151": "I mean the output of function iou_coef_loss. If you just take the negative it will return a negative fraction, i.e. 0 > x > -1. I don't know if there's mathematical justification (probably not), but taking the negative log instead of just the negative will output positive values. And in my case it also results in a training graph (epochs vs loss) that doesn't start off with a plateau, which makes it easier to see that the network is not stuck in some local minimum when you start training it.", "385153": "I'll dig into my archive, as I ended not using this IoU loss, hence it is not in my github repo. I don't remember my code being stuck.", "403276": "I've written an IOU metric for multi label image classification in keras, check this out:\nhttps://github.com/JeniaNovellusDx/KerasExtras/blob/master/keras_functions.py" }, "source": "meta" }