I am late to the competition but I am planning to work on a solution nonetheless and sharing the code.
\n\nI was checking the metric evaluation definition and was confused by the use of the term \"Precision\" to define this quantity:
\n\nPrecision(t)=TP(t) / (TP(t)+FP(t)+FN(t))
\n\nIf I understand correctly, precision (at threshold t) in the context of classification is defined as:
\n\nPrecision(t)=TP(t) / (TP(t)+FP(t)) = TP(t) / P(t), where P(t) is all the predicted positive classes (i.e. predict there is a nuclei).
\n\nHere is the Wikipedia page for more details: https://en.wikipedia.org/wiki/Precision_and_recall
\n\nThe competition can of course choose this definition of precision but it seemed confusing to me.
\n\nAny thoughts on that?
", "messages": [ { "id": "310777", "postDate": "04/08/2018 15:09:20", "content": "I am late to the competition but I am planning to work on a solution nonetheless and sharing the code.
\n\nI was checking the metric evaluation definition and was confused by the use of the term \"Precision\" to define this quantity:
\n\nPrecision(t)=TP(t) / (TP(t)+FP(t)+FN(t))
\n\nIf I understand correctly, precision (at threshold t) in the context of classification is defined as:
\n\nPrecision(t)=TP(t) / (TP(t)+FP(t)) = TP(t) / P(t), where P(t) is all the predicted positive classes (i.e. predict there is a nuclei).
\n\nHere is the Wikipedia page for more details: https://en.wikipedia.org/wiki/Precision_and_recall
\n\nThe competition can of course choose this definition of precision but it seemed confusing to me.
\n\nAny thoughts on that?
", "rawMarkdown": "I am late to the competition but I am planning to work on a solution nonetheless and sharing the code. \n\nI was checking the metric evaluation definition and was confused by the use of the term \"Precision\" to define this quantity: \n\nPrecision(t)=TP(t) / (TP(t)+FP(t)+FN(t)) \n\nIf I understand correctly, precision (at threshold t) in the context of classification is defined as: \n\nPrecision(t)=TP(t) / (TP(t)+FP(t)) = TP(t) / P(t), where P(t) is all the predicted positive classes (i.e. predict there is a nuclei). \n\nHere is the Wikipedia page for more details: https://en.wikipedia.org/wiki/Precision_and_recall \n\nThe competition can of course choose this definition of precision but it seemed confusing to me. \n\nAny thoughts on that?", "votes": null }, { "id": "310792", "postDate": "04/08/2018 15:56:54", "content": "It may have been answered elsewhere. If you look at the same wikipedia article, there is a definition for accuracy $$\\frac{TP+TN}{TP+TN+FP+FN}$$
\n\nNote a true negative (TN) is the background in the image. If you remove TN, it becomes the competition metric. Therefore the definition used in the competition can be interpreted as you are neither rewarded nor penalized by finding a true background part of the image.
", "rawMarkdown": "It may have been answered elsewhere. If you look at the same wikipedia article, there is a definition for accuracy $$\\frac{TP+TN}{TP+TN+FP+FN}$$\n\nNote a true negative (TN) is the background in the image. If you remove TN, it becomes the competition metric. Therefore the definition used in the competition can be interpreted as you are neither rewarded nor penalized by finding a true background part of the image.", "votes": null }, { "id": "310800", "postDate": "04/08/2018 16:20:15", "content": "Thanks for the accurate comment (pun intended :)). \nThus, it isn't precision but rather (an adapted) accuracy then. \nI guess this is, to some extent, a common practice in image recognition/segmentation? (I have very basic skills on this subject so far).
", "rawMarkdown": "Thanks for the accurate comment (pun intended :)). \nThus, it isn't precision but rather (an adapted) accuracy then. \nI guess this is, to some extent, a common practice in image recognition/segmentation? (I have very basic skills on this subject so far).", "votes": null }, { "id": "310809", "postDate": "04/08/2018 17:09:10", "content": "One thing to note though, despite the metric itself doesn't reward/penalize for finding background, the more background in the image the more opportunities for FP would arise in your algorithm. So it doesn't mean we shouldn't try to find background.
\n\nThis article might be a good read. Note the definition of Jaccard index (JAC) in equation (7) matches precisely the competition metric.
", "rawMarkdown": "One thing to note though, despite the metric itself doesn't reward/penalize for finding background, the more background in the image the more opportunities for FP would arise in your algorithm. So it doesn't mean we shouldn't try to find background. \n\nThis [**article**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4533825/) might be a good read. Note the definition of Jaccard index (JAC) in equation (7) matches precisely the competition metric.", "votes": null }, { "id": "310818", "postDate": "04/08/2018 17:55:58", "content": "Thanks a lot. That's the missing piece I was looking for.
\n\nSo, IoU (also called Jacard index) is used to create labels then Jacard similarity index (called precision in the competition metric definition) is used to compute the metric based on these labels and varying thresholds (averaging over the different thresholds then taking the average over all the images).
", "rawMarkdown": "Thanks a lot. That's the missing piece I was looking for. \n\nSo, IoU (also called Jacard index) is used to create labels then Jacard similarity index (called precision in the competition metric definition) is used to compute the metric based on these labels and varying thresholds (averaging over the different thresholds then taking the average over all the images).", "votes": null } ], "comments": [ { "id": 310792, "author_name": "pliptor", "author_url": "", "post_date": "04/08/2018 15:56:54", "content": "It may have been answered elsewhere. If you look at the same wikipedia article, there is a definition for accuracy $$\\frac{TP+TN}{TP+TN+FP+FN}$$
\n\nNote a true negative (TN) is the background in the image. If you remove TN, it becomes the competition metric. Therefore the definition used in the competition can be interpreted as you are neither rewarded nor penalized by finding a true background part of the image.
", "votes": null, "replies": [ { "id": 310800, "author_name": "yassinealouini", "author_url": "", "post_date": "04/08/2018 16:20:15", "content": "Thanks for the accurate comment (pun intended :)). \nThus, it isn't precision but rather (an adapted) accuracy then. \nI guess this is, to some extent, a common practice in image recognition/segmentation? (I have very basic skills on this subject so far).
", "votes": null, "replies": [] }, { "id": 310809, "author_name": "pliptor", "author_url": "", "post_date": "04/08/2018 17:09:10", "content": "One thing to note though, despite the metric itself doesn't reward/penalize for finding background, the more background in the image the more opportunities for FP would arise in your algorithm. So it doesn't mean we shouldn't try to find background.
\n\nThis article might be a good read. Note the definition of Jaccard index (JAC) in equation (7) matches precisely the competition metric.
", "votes": null, "replies": [] }, { "id": 310818, "author_name": "yassinealouini", "author_url": "", "post_date": "04/08/2018 17:55:58", "content": "Thanks a lot. That's the missing piece I was looking for.
\n\nSo, IoU (also called Jacard index) is used to create labels then Jacard similarity index (called precision in the competition metric definition) is used to compute the metric based on these labels and varying thresholds (averaging over the different thresholds then taking the average over all the images).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "310777": "I am late to the competition but I am planning to work on a solution nonetheless and sharing the code. \n\nI was checking the metric evaluation definition and was confused by the use of the term \"Precision\" to define this quantity: \n\nPrecision(t)=TP(t) / (TP(t)+FP(t)+FN(t)) \n\nIf I understand correctly, precision (at threshold t) in the context of classification is defined as: \n\nPrecision(t)=TP(t) / (TP(t)+FP(t)) = TP(t) / P(t), where P(t) is all the predicted positive classes (i.e. predict there is a nuclei). \n\nHere is the Wikipedia page for more details: https://en.wikipedia.org/wiki/Precision_and_recall \n\nThe competition can of course choose this definition of precision but it seemed confusing to me. \n\nAny thoughts on that?", "310792": "It may have been answered elsewhere. If you look at the same wikipedia article, there is a definition for accuracy $$\\frac{TP+TN}{TP+TN+FP+FN}$$\n\nNote a true negative (TN) is the background in the image. If you remove TN, it becomes the competition metric. Therefore the definition used in the competition can be interpreted as you are neither rewarded nor penalized by finding a true background part of the image.", "310800": "Thanks for the accurate comment (pun intended :)). \nThus, it isn't precision but rather (an adapted) accuracy then. \nI guess this is, to some extent, a common practice in image recognition/segmentation? (I have very basic skills on this subject so far).", "310809": "One thing to note though, despite the metric itself doesn't reward/penalize for finding background, the more background in the image the more opportunities for FP would arise in your algorithm. So it doesn't mean we shouldn't try to find background. \n\nThis [**article**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4533825/) might be a good read. Note the definition of Jaccard index (JAC) in equation (7) matches precisely the competition metric.", "310818": "Thanks a lot. That's the missing piece I was looking for. \n\nSo, IoU (also called Jacard index) is used to create labels then Jacard similarity index (called precision in the competition metric definition) is used to compute the metric based on these labels and varying thresholds (averaging over the different thresholds then taking the average over all the images)." }, "source": "meta" }