Hi,\n I have a confusion about evaluation metric.
\n\nI understand that LB score will be the mean precision across all images. To calculate precision for an image, TP, FP and FN need to be computed (at various threshold values). To compute TP, FP, and FN, I need to compute IoU between true masks and predicted masks (at some given threshold)
\n\nSuppose, for a given image, in truth there are 4 masks, while my model outputs 6 masks. Now, between which true masks and predicted masks, will I find IoU? Will I take all 4x6=24 IoUs and find TP, FP and FN from there? If yes, how?
", "messages": [ { "id": "270535", "postDate": "01/18/2018 13:01:10", "content": "Hi,\n I have a confusion about evaluation metric.
\n\nI understand that LB score will be the mean precision across all images. To calculate precision for an image, TP, FP and FN need to be computed (at various threshold values). To compute TP, FP, and FN, I need to compute IoU between true masks and predicted masks (at some given threshold)
\n\nSuppose, for a given image, in truth there are 4 masks, while my model outputs 6 masks. Now, between which true masks and predicted masks, will I find IoU? Will I take all 4x6=24 IoUs and find TP, FP and FN from there? If yes, how?
", "rawMarkdown": "Hi,\n I have a confusion about evaluation metric. \n\n\n I understand that LB score will be the mean precision across all images. To calculate precision for an image, TP, FP and FN need to be computed (at various threshold values). To compute TP, FP, and FN, I need to compute IoU between true masks and predicted masks (at some given threshold)\n \n Suppose, for a given image, in truth there are 4 masks, while my model outputs 6 masks. Now, between which true masks and predicted masks, will I find IoU? Will I take all 4x6=24 IoUs and find TP, FP and FN from there? If yes, how?", "votes": null }, { "id": "270538", "postDate": "01/18/2018 13:14:54", "content": "I have the same question)
", "rawMarkdown": "I have the same question)", "votes": null }, { "id": "270559", "postDate": "01/18/2018 14:15:28", "content": "We compute the intersection-over-union for each prediction (i.e. . we match the prediction object to the ground truth object by using the greatest IOU score). We’ll penalize the remaining two by scoring them as false positives.
", "rawMarkdown": "We compute the intersection-over-union for each prediction (i.e. . we match the prediction object to the ground truth object by using the greatest IOU score). We’ll penalize the remaining two by scoring them as false positives.", "votes": null }, { "id": "270631", "postDate": "01/18/2018 16:32:23", "content": "Thanks for your reply. I am writing down an example here.
\n\nSuppose, true masks are m1, m2, m3,m4 and I predict masks p1, p2, p3, p4, p5. Let's say I computing precision with threshold value of 0.7.
\n\nSay p1 is having highest iou with m1, p2 is having highest iou with m2, and these two iou's are > 0.7.\np3 is having highest iou with m3, but it is < 0.7. p4 is having highest iou with m1 (but it is < 0.7). p5 has iou = 0 with all the true masks.
\n\nThen TP = 2 (p1/m1 and p2/m2). TN = 2 (m3, m4), FN = 3 (p3, p4, p5).
", "rawMarkdown": "Thanks for your reply. I am writing down an example here.\n\nSuppose, true masks are m1, m2, m3,m4 and I predict masks p1, p2, p3, p4, p5. Let's say I computing precision with threshold value of 0.7.\n\nSay p1 is having highest iou with m1, p2 is having highest iou with m2, and these two iou's are > 0.7.\np3 is having highest iou with m3, but it is < 0.7. p4 is having highest iou with m1 (but it is < 0.7). p5 has iou = 0 with all the true masks.\n\nThen TP = 2 (p1/m1 and p2/m2). TN = 2 (m3, m4), FN = 3 (p3, p4, p5).", "votes": null }, { "id": "270655", "postDate": "01/18/2018 17:32:46", "content": "A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object.
\n\nI think in your case, TP(p1,p1) = 2, FP = 3(p3, p4, p5) and FN = 2(m3,m4).
", "rawMarkdown": "A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object.\n\nI think in your case, TP(p1,p1) = 2, FP = 3(p3, p4, p5) and FN = 2(m3,m4).", "votes": null }, { "id": "270890", "postDate": "01/19/2018 06:49:11", "content": "Almost clear now) But what happens when there are two different predicted objects, which have the same IoU with one of the ground truth objects? (That's highly improbable, but anyway)\nFor example, ground truth with 2 masks is like that (let them be m1 and m2):
\n\n1 1 0 0 0 0 0 0\n1 1 0 0 0 0 0 0\n0 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 1And I predict 3 objects (p1, p2, p3):
\n\n1 1 1 0 1 1 0 0 0 0 0 0\n1 1 0 0 1 1 0 0 0 0 0 0\n0 0 0 0 1 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 0 0 0 1 1\nThreshold = 0.7
\n\nHere both p1 and p2 have IoU = 0.8 > 0.7 with m1. What happens then? It's incorrect to consider both as true positive, and any of them as false positive (because \"A false positive indicates a predicted object had no associated ground truth object\", but they both have).
", "rawMarkdown": "Almost clear now) But what happens when there are two different predicted objects, which have the same IoU with one of the ground truth objects? (That's highly improbable, but anyway)\nFor example, ground truth with 2 masks is like that (let them be m1 and m2):\n1 1 0 0 0 0 0 0\n1 1 0 0 0 0 0 0\n0 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 11 1 1 0 1 1 0 0 0 0 0 0\n1 1 0 0 1 1 0 0 0 0 0 0\n0 0 0 0 1 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 0 0 0 1 1\nPredicted masks are not allowed to overlap.
", "rawMarkdown": "Predicted masks are not allowed to overlap.", "votes": null }, { "id": "271069", "postDate": "01/19/2018 16:16:38", "content": "I am a little bit confused. Some ground-truth mask do overlap, right?\nWould the perfect submission just be the merged masks of all the single nuclei masks?
", "rawMarkdown": "I am a little bit confused. Some ground-truth mask do overlap, right?\nWould the perfect submission just be the merged masks of all the single nuclei masks?", "votes": null }, { "id": "271089", "postDate": "01/19/2018 16:57:44", "content": "No pixel should belong to more than one mask in the data. If you find any that do, let us know. If you try to predict masks that overlap, you should get an error.
\n\nYou are predicting individual masks because nuclei are allowed to touch. A merged mask would not be able to differentiate where the border should be in the case of touching regions.
", "rawMarkdown": "No pixel should belong to more than one mask in the data. If you find any that do, let us know. If you try to predict masks that overlap, you should get an error.\n\nYou are predicting individual masks because nuclei are allowed to touch. A merged mask would not be able to differentiate where the border should be in the case of touching regions.", "votes": null }, { "id": "271376", "postDate": "01/20/2018 10:06:22", "content": "Alright, thank you! Now I understand.
", "rawMarkdown": "Alright, thank you! Now I understand.", "votes": null }, { "id": "272831", "postDate": "01/23/2018 20:03:21", "content": "Please can you confirm if the output is a row for each nucleus as per your comment; or for each image as per the sample output?
", "rawMarkdown": "Please can you confirm if the output is a row for each nucleus as per your comment; or for each image as per the sample output?", "votes": null }, { "id": "272855", "postDate": "01/23/2018 21:03:13", "content": "Each row represents an object. You’ll have multiple ImageIds (if the image has multiple objects).
", "rawMarkdown": "Each row represents an object. You’ll have multiple ImageIds (if the image has multiple objects).", "votes": null }, { "id": "279797", "postDate": "02/08/2018 18:07:46", "content": "I have a question. Evaluation is after decode RLE right? We upload csv is after encode then one mask may split into two parts, say\n1 1 0 0
\n1 1 0 0
\n0 0 0 0
\n0 0 0 0\nwill be 1, 2, 5, 2.\nWhen evaluating, it will first decode right?
Yes
", "rawMarkdown": "Yes", "votes": null } ], "comments": [ { "id": 270538, "author_name": "inscaven", "author_url": "", "post_date": "01/18/2018 13:14:54", "content": "I have the same question)
", "votes": null, "replies": [] }, { "id": 270559, "author_name": "agoodman", "author_url": "", "post_date": "01/18/2018 14:15:28", "content": "We compute the intersection-over-union for each prediction (i.e. . we match the prediction object to the ground truth object by using the greatest IOU score). We’ll penalize the remaining two by scoring them as false positives.
", "votes": null, "replies": [ { "id": 270631, "author_name": "thevivekpandey", "author_url": "", "post_date": "01/18/2018 16:32:23", "content": "Thanks for your reply. I am writing down an example here.
\n\nSuppose, true masks are m1, m2, m3,m4 and I predict masks p1, p2, p3, p4, p5. Let's say I computing precision with threshold value of 0.7.
\n\nSay p1 is having highest iou with m1, p2 is having highest iou with m2, and these two iou's are > 0.7.\np3 is having highest iou with m3, but it is < 0.7. p4 is having highest iou with m1 (but it is < 0.7). p5 has iou = 0 with all the true masks.
\n\nThen TP = 2 (p1/m1 and p2/m2). TN = 2 (m3, m4), FN = 3 (p3, p4, p5).
", "votes": null, "replies": [] }, { "id": 270655, "author_name": "hjzzhang", "author_url": "", "post_date": "01/18/2018 17:32:46", "content": "A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object.
\n\nI think in your case, TP(p1,p1) = 2, FP = 3(p3, p4, p5) and FN = 2(m3,m4).
", "votes": null, "replies": [] }, { "id": 270890, "author_name": "inscaven", "author_url": "", "post_date": "01/19/2018 06:49:11", "content": "Almost clear now) But what happens when there are two different predicted objects, which have the same IoU with one of the ground truth objects? (That's highly improbable, but anyway)\nFor example, ground truth with 2 masks is like that (let them be m1 and m2):
\n\n1 1 0 0 0 0 0 0\n1 1 0 0 0 0 0 0\n0 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 1And I predict 3 objects (p1, p2, p3):
\n\n1 1 1 0 1 1 0 0 0 0 0 0\n1 1 0 0 1 1 0 0 0 0 0 0\n0 0 0 0 1 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 0 0 0 1 1\nThreshold = 0.7
\n\nHere both p1 and p2 have IoU = 0.8 > 0.7 with m1. What happens then? It's incorrect to consider both as true positive, and any of them as false positive (because \"A false positive indicates a predicted object had no associated ground truth object\", but they both have).
", "votes": null, "replies": [] }, { "id": 271033, "author_name": "wcukierski", "author_url": "", "post_date": "01/19/2018 14:50:11", "content": "Predicted masks are not allowed to overlap.
", "votes": null, "replies": [] }, { "id": 271069, "author_name": "timjoseph", "author_url": "", "post_date": "01/19/2018 16:16:38", "content": "I am a little bit confused. Some ground-truth mask do overlap, right?\nWould the perfect submission just be the merged masks of all the single nuclei masks?
", "votes": null, "replies": [] }, { "id": 271089, "author_name": "wcukierski", "author_url": "", "post_date": "01/19/2018 16:57:44", "content": "No pixel should belong to more than one mask in the data. If you find any that do, let us know. If you try to predict masks that overlap, you should get an error.
\n\nYou are predicting individual masks because nuclei are allowed to touch. A merged mask would not be able to differentiate where the border should be in the case of touching regions.
", "votes": null, "replies": [] }, { "id": 271376, "author_name": "timjoseph", "author_url": "", "post_date": "01/20/2018 10:06:22", "content": "Alright, thank you! Now I understand.
", "votes": null, "replies": [] }, { "id": 272831, "author_name": "simonm3", "author_url": "", "post_date": "01/23/2018 20:03:21", "content": "Please can you confirm if the output is a row for each nucleus as per your comment; or for each image as per the sample output?
", "votes": null, "replies": [] }, { "id": 272855, "author_name": "agoodman", "author_url": "", "post_date": "01/23/2018 21:03:13", "content": "Each row represents an object. You’ll have multiple ImageIds (if the image has multiple objects).
", "votes": null, "replies": [] } ] }, { "id": 279797, "author_name": "zhen27", "author_url": "", "post_date": "02/08/2018 18:07:46", "content": "I have a question. Evaluation is after decode RLE right? We upload csv is after encode then one mask may split into two parts, say\n1 1 0 0
\n1 1 0 0
\n0 0 0 0
\n0 0 0 0\nwill be 1, 2, 5, 2.\nWhen evaluating, it will first decode right?
Yes
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "270535": "Hi,\n I have a confusion about evaluation metric. \n\n\n I understand that LB score will be the mean precision across all images. To calculate precision for an image, TP, FP and FN need to be computed (at various threshold values). To compute TP, FP, and FN, I need to compute IoU between true masks and predicted masks (at some given threshold)\n \n Suppose, for a given image, in truth there are 4 masks, while my model outputs 6 masks. Now, between which true masks and predicted masks, will I find IoU? Will I take all 4x6=24 IoUs and find TP, FP and FN from there? If yes, how?", "270538": "I have the same question)", "270559": "We compute the intersection-over-union for each prediction (i.e. . we match the prediction object to the ground truth object by using the greatest IOU score). We’ll penalize the remaining two by scoring them as false positives.", "270631": "Thanks for your reply. I am writing down an example here.\n\nSuppose, true masks are m1, m2, m3,m4 and I predict masks p1, p2, p3, p4, p5. Let's say I computing precision with threshold value of 0.7.\n\nSay p1 is having highest iou with m1, p2 is having highest iou with m2, and these two iou's are > 0.7.\np3 is having highest iou with m3, but it is < 0.7. p4 is having highest iou with m1 (but it is < 0.7). p5 has iou = 0 with all the true masks.\n\nThen TP = 2 (p1/m1 and p2/m2). TN = 2 (m3, m4), FN = 3 (p3, p4, p5).", "270655": "A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object.\n\nI think in your case, TP(p1,p1) = 2, FP = 3(p3, p4, p5) and FN = 2(m3,m4).", "270890": "Almost clear now) But what happens when there are two different predicted objects, which have the same IoU with one of the ground truth objects? (That's highly improbable, but anyway)\nFor example, ground truth with 2 masks is like that (let them be m1 and m2):\n1 1 0 0 0 0 0 0\n1 1 0 0 0 0 0 0\n0 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 11 1 1 0 1 1 0 0 0 0 0 0\n1 1 0 0 1 1 0 0 0 0 0 0\n0 0 0 0 1 0 0 0 0 0 1 1\n0 0 0 0 0 0 0 0 0 0 1 1\n