I haven't been able to find a correctly implemented iou metric for Keras. I have tried several implementations from some example kernels and haven't found any that work. For example, the author of this one says that his iou metric is definitely not correct. Does anyone know of a correct implementation of this ?
", "messages": [ { "id": "293483", "postDate": "03/09/2018 23:54:51", "content": "I haven't been able to find a correctly implemented iou metric for Keras. I have tried several implementations from some example kernels and haven't found any that work. For example, the author of this one says that his iou metric is definitely not correct. Does anyone know of a correct implementation of this ?
", "rawMarkdown": "I haven't been able to find a correctly implemented iou metric for Keras. I have tried several implementations from some example kernels and haven't found any that work. For example, the author of [this one][1] says that his iou metric is definitely not correct. Does anyone know of a correct implementation of this ?\n\n [1]: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277", "votes": null }, { "id": "293514", "postDate": "03/10/2018 02:53:31", "content": "The IOU metric as used on the LB is a little more tricky than the example iou used in that kernel, because we need each object in both the true and predicted mask to calculate it. While you could do it in theory, since the metric doesn't actually inform the decision of the algorithm I think it probably wouldn't be worth the cost. Calculating it would be a little slow and trickier (because Keras will not pass through your object by object mask, just the training data which is the whole mask of the images, you'll have to maintain them externally or as metadata then look up the objects, as well as label each predicted mask each time). That being said you can more easily calculate the LB style IOU outside of a metrics function after you've created your model.
\n\nhttps://gist.github.com/ameerkat/fc2489ef7e8f2214f5a4e8a6cc2208f3\nHere's an example, although you'll probably have to modify it a little for how your workflow is setup. You'll have to remove the custom objects on the Keras load_model for sure.
\n\nIf you really want to use it as a metrics function in Keras, you'll have to perhaps load and cache the true masks externally. You might also be able to pass meta data along with the numpy array by subclassing it as well. You can hash a numpy array in python 3 with hash(<array>.data.tobytes()). This does make a copy of it though.
Thanks for the reply,
\n\nWould the example metric implementation here be able to give a correct result if run after training ?
", "rawMarkdown": "Thanks for the reply,\n\nWould the example metric implementation [here][1] be able to give a correct result if run after training ?\n\n [1]: https://www.kaggle.com/wcukierski/example-metric-implementation", "votes": null }, { "id": "293612", "postDate": "03/10/2018 08:35:28", "content": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553
", "rawMarkdown": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553", "votes": null } ], "comments": [ { "id": 293514, "author_name": "ameerkat", "author_url": "", "post_date": "03/10/2018 02:53:31", "content": "The IOU metric as used on the LB is a little more tricky than the example iou used in that kernel, because we need each object in both the true and predicted mask to calculate it. While you could do it in theory, since the metric doesn't actually inform the decision of the algorithm I think it probably wouldn't be worth the cost. Calculating it would be a little slow and trickier (because Keras will not pass through your object by object mask, just the training data which is the whole mask of the images, you'll have to maintain them externally or as metadata then look up the objects, as well as label each predicted mask each time). That being said you can more easily calculate the LB style IOU outside of a metrics function after you've created your model.
\n\nhttps://gist.github.com/ameerkat/fc2489ef7e8f2214f5a4e8a6cc2208f3\nHere's an example, although you'll probably have to modify it a little for how your workflow is setup. You'll have to remove the custom objects on the Keras load_model for sure.
\n\nIf you really want to use it as a metrics function in Keras, you'll have to perhaps load and cache the true masks externally. You might also be able to pass meta data along with the numpy array by subclassing it as well. You can hash a numpy array in python 3 with hash(<array>.data.tobytes()). This does make a copy of it though.
Thanks for the reply,
\n\nWould the example metric implementation here be able to give a correct result if run after training ?
", "votes": null, "replies": [] } ] }, { "id": 293612, "author_name": "cpmpml", "author_url": "", "post_date": "03/10/2018 08:35:28", "content": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "293483": "I haven't been able to find a correctly implemented iou metric for Keras. I have tried several implementations from some example kernels and haven't found any that work. For example, the author of [this one][1] says that his iou metric is definitely not correct. Does anyone know of a correct implementation of this ?\n\n [1]: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277", "293514": "The IOU metric as used on the LB is a little more tricky than the example iou used in that kernel, because we need each object in both the true and predicted mask to calculate it. While you *could* do it in theory, since the metric doesn't actually inform the decision of the algorithm I think it probably wouldn't be worth the cost. Calculating it would be a little slow and trickier (because Keras will not pass through your object by object mask, just the training data which is the whole mask of the images, you'll have to maintain them externally or as metadata then look up the objects, as well as label each predicted mask each time). That being said you can more easily calculate the LB style IOU outside of a metrics function after you've created your model.\n\nhttps://gist.github.com/ameerkat/fc2489ef7e8f2214f5a4e8a6cc2208f3\nHere's an example, although you'll probably have to modify it a little for how your workflow is setup. You'll have to remove the custom objects on the Keras load_model for sure.\n\nIf you really want to use it as a metrics function in Keras, you'll have to perhaps load and cache the true masks externally. You might also be able to pass meta data along with the numpy array by subclassing it as well. You can hash a numpy array in python 3 with `hash(", "293528": "Thanks for the reply,\n\nWould the example metric implementation [here][1] be able to give a correct result if run after training ?\n\n [1]: https://www.kaggle.com/wcukierski/example-metric-implementation", "293612": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553" }, "source": "meta" }