{ "id": 50060, "title": "Could you explain exactly what evaluation metric is?", "url": "/competitions/data-science-bowl-2018/discussion/50060", "author_name": "", "post_date": "2018-02-19T09:08:54.694781Z", "votes": 1, "comment_count": 6, "views": 0, "content": "

I am really being confused because I do not understand evaluation metric.

\n\n
    \n

  1. Is it okay that I just get all mask at once from a image or Do I have to get each mask for each nucleus?

    2. \n

  2. I can't figure out difference between FP and FN. could you explain for me about this?

    3. \n

  3. Do I have to give a label to each mask corresponding to each nucleus? (it might be the similar question to 1.)

    4. \n
\n\n

please help me...thank you.

", "messages": [ { "id": "285085", "postDate": "02/19/2018 09:08:54", "content": "

I am really being confused because I do not understand evaluation metric.

\n\n
    \n

  1. Is it okay that I just get all mask at once from a image or Do I have to get each mask for each nucleus?

    2. \n

  2. I can't figure out difference between FP and FN. could you explain for me about this?

    3. \n

  3. Do I have to give a label to each mask corresponding to each nucleus? (it might be the similar question to 1.)

    4. \n
\n\n

please help me...thank you.

", "rawMarkdown": "I am really being confused because I do not understand evaluation metric.\n\n1. Is it okay that I just get all mask at once from a image or Do I have to get each mask for each nucleus?\n\n2. I can't figure out difference between FP and FN. could you explain for me about this?\n\n3. Do I have to give a label to each mask corresponding to each nucleus? (it might be the similar question to 1.)\n\nplease help me...thank you.", "votes": null }, { "id": "286169", "postDate": "02/21/2018 20:11:37", "content": "

Yes, each nucleus should gets its own mask. You add a new row in the submission CSV file for every nucleus you find, so if you find 50 nuclei in one image you will add 50 rows in the CSV file.

\n\n

FP is a false positive. You have predicted that there is a nucleus at this position but there isn't (or at least, the IOU metric is not higher than the threshold, so for example you may have predicted a mask that only overlaps for 40% with the real nucleus but the threshold is 50% or higher and so this prediction does not count).

\n\n

FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)

", "rawMarkdown": "Yes, each nucleus should gets its own mask. You add a new row in the submission CSV file for every nucleus you find, so if you find 50 nuclei in one image you will add 50 rows in the CSV file.\n\nFP is a false positive. You have predicted that there is a nucleus at this position but there isn't (or at least, the IOU metric is not higher than the threshold, so for example you may have predicted a mask that only overlaps for 40% with the real nucleus but the threshold is 50% or higher and so this prediction does not count).\n\nFN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)", "votes": null }, { "id": "303229", "postDate": "03/25/2018 20:04:21", "content": "
\n

Human Analog wrote

\n \n

FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)

\n
\n\n

Hi Human Analog,

\n\n

Could you please clarify what do you mean by \"according to the IoU metric and the threshold\" for FN?\nI thought the calculation of false negatives would not use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).

\n\n

In other words for a threshold t

\n\n
\n

t=0.5

\n \n

TP = 0.5 <= IoU <= 1.0

\n \n

FP = 0 < IoU < 0.5

\n \n

FN = IoU = 0

\n
", "rawMarkdown": "> **Human Analog wrote**\n> \n> FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (**Again, according to the IOU metric and the threshold.**)\n\nHi Human Analog,\n\nCould you please clarify what do you mean by \"*according to the IoU metric and the threshold*\" for FN?\nI thought the calculation of false negatives would **not** use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).\n\nIn other words for a threshold t\n\n> t=0.5 \n\n> TP = 0.5 <= IoU <= 1.0 \n\n> FP = 0 < IoU < 0.5 \n\n> FN = IoU = 0", "votes": null }, { "id": "303474", "postDate": "03/26/2018 08:56:57", "content": "

If you make a prediction and its IOU with the ground-truth is less than the threshold it counts as a FP, right? And if there are no predictions for a ground-truth then it counts as FN.

\n\n

However, if all predictions for that ground-truth are FPs (i.e. if all of them are below the IOU threshold), then it also counts as a FN. After all, this is the same thing as saying there are no predictions for that ground-truth (because none of the existing predictions count).

", "rawMarkdown": "If you make a prediction and its IOU with the ground-truth is less than the threshold it counts as a FP, right? And if there are no predictions for a ground-truth then it counts as FN.\n\nHowever, if _all_ predictions for that ground-truth are FPs (i.e. if all of them are below the IOU threshold), then it also counts as a FN. After all, this is the same thing as saying there are no predictions for that ground-truth (because none of the existing predictions count).", "votes": null }, { "id": "304197", "postDate": "03/27/2018 06:46:46", "content": "

Could you clarify what you said more?

\n\n

Why if all predictions are FPs, FN is counted? I completely did not understand what you said.

", "rawMarkdown": "Could you clarify what you said more?\n\nWhy if all predictions are FPs, FN is counted? I completely did not understand what you said.", "votes": null }, { "id": "304198", "postDate": "03/27/2018 06:47:03", "content": "

d

", "rawMarkdown": "d", "votes": null }, { "id": "304201", "postDate": "03/27/2018 06:50:35", "content": "
\n

Peter Metz wrote

\n \n
\n

> Human Analog wrote\n > \n > FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)

\n
\n \n

Hi Human Analog,

\n \n

Could you please clarify what do you mean by \"according to the IoU metric and the threshold\" for FN?\n I thought the calculation of false negatives would not use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).

\n \n

In other words for a threshold t

\n \n

> t=0.5

\n \n

> TP = 0.5 <= IoU <= 1.0

\n \n

> FP = 0 < IoU < 0.5

\n \n

> FN = IoU = 0

\n
\n\n

What you said makes sense the most for me.\nBasically, we do not say which prediction indicate which ground-truth because we don't know what ground-truths there are. That's why we cannot say which ground-truth is missing.\nI will proceed my progress based on your comment

", "rawMarkdown": "> **Peter Metz wrote**\n> \n> > \n> > **Human Analog wrote**\n> > \n> > FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (**Again, according to the IOU metric and the threshold.**)\n> \n> Hi Human Analog,\n> \n> Could you please clarify what do you mean by \"*according to the IoU metric and the threshold*\" for FN?\n> I thought the calculation of false negatives would **not** use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).\n> \n> In other words for a threshold t\n> \n> > t=0.5 \n> \n> > TP = 0.5 <= IoU <= 1.0 \n> \n> > FP = 0 < IoU < 0.5 \n> \n> > FN = IoU = 0\n> \n\nWhat you said makes sense the most for me.\nBasically, we do not say which prediction indicate which ground-truth because we don't know what ground-truths there are. That's why we cannot say which ground-truth is missing.\nI will proceed my progress based on your comment", "votes": null } ], "comments": [ { "id": 286169, "author_name": "humananalog", "author_url": "", "post_date": "02/21/2018 20:11:37", "content": "

Yes, each nucleus should gets its own mask. You add a new row in the submission CSV file for every nucleus you find, so if you find 50 nuclei in one image you will add 50 rows in the CSV file.

\n\n

FP is a false positive. You have predicted that there is a nucleus at this position but there isn't (or at least, the IOU metric is not higher than the threshold, so for example you may have predicted a mask that only overlaps for 40% with the real nucleus but the threshold is 50% or higher and so this prediction does not count).

\n\n

FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)

", "votes": null, "replies": [ { "id": 303229, "author_name": "ksnoxygfwtvirymmg", "author_url": "", "post_date": "03/25/2018 20:04:21", "content": "
\n

Human Analog wrote

\n \n

FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)

\n
\n\n

Hi Human Analog,

\n\n

Could you please clarify what do you mean by \"according to the IoU metric and the threshold\" for FN?\nI thought the calculation of false negatives would not use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).

\n\n

In other words for a threshold t

\n\n
\n

t=0.5

\n \n

TP = 0.5 <= IoU <= 1.0

\n \n

FP = 0 < IoU < 0.5

\n \n

FN = IoU = 0

\n
", "votes": null, "replies": [] }, { "id": 303474, "author_name": "humananalog", "author_url": "", "post_date": "03/26/2018 08:56:57", "content": "

If you make a prediction and its IOU with the ground-truth is less than the threshold it counts as a FP, right? And if there are no predictions for a ground-truth then it counts as FN.

\n\n

However, if all predictions for that ground-truth are FPs (i.e. if all of them are below the IOU threshold), then it also counts as a FN. After all, this is the same thing as saying there are no predictions for that ground-truth (because none of the existing predictions count).

", "votes": null, "replies": [] }, { "id": 304197, "author_name": "tbop02", "author_url": "", "post_date": "03/27/2018 06:46:46", "content": "

Could you clarify what you said more?

\n\n

Why if all predictions are FPs, FN is counted? I completely did not understand what you said.

", "votes": null, "replies": [] }, { "id": 304198, "author_name": "tbop02", "author_url": "", "post_date": "03/27/2018 06:47:03", "content": "

d

", "votes": null, "replies": [] }, { "id": 304201, "author_name": "tbop02", "author_url": "", "post_date": "03/27/2018 06:50:35", "content": "
\n

Peter Metz wrote

\n \n
\n

> Human Analog wrote\n > \n > FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)

\n
\n \n

Hi Human Analog,

\n \n

Could you please clarify what do you mean by \"according to the IoU metric and the threshold\" for FN?\n I thought the calculation of false negatives would not use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).

\n \n

In other words for a threshold t

\n \n

> t=0.5

\n \n

> TP = 0.5 <= IoU <= 1.0

\n \n

> FP = 0 < IoU < 0.5

\n \n

> FN = IoU = 0

\n
\n\n

What you said makes sense the most for me.\nBasically, we do not say which prediction indicate which ground-truth because we don't know what ground-truths there are. That's why we cannot say which ground-truth is missing.\nI will proceed my progress based on your comment

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "285085": "I am really being confused because I do not understand evaluation metric.\n\n1. Is it okay that I just get all mask at once from a image or Do I have to get each mask for each nucleus?\n\n2. I can't figure out difference between FP and FN. could you explain for me about this?\n\n3. Do I have to give a label to each mask corresponding to each nucleus? (it might be the similar question to 1.)\n\nplease help me...thank you.", "286169": "Yes, each nucleus should gets its own mask. You add a new row in the submission CSV file for every nucleus you find, so if you find 50 nuclei in one image you will add 50 rows in the CSV file.\n\nFP is a false positive. You have predicted that there is a nucleus at this position but there isn't (or at least, the IOU metric is not higher than the threshold, so for example you may have predicted a mask that only overlaps for 40% with the real nucleus but the threshold is 50% or higher and so this prediction does not count).\n\nFN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (Again, according to the IOU metric and the threshold.)", "303229": "> **Human Analog wrote**\n> \n> FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (**Again, according to the IOU metric and the threshold.**)\n\nHi Human Analog,\n\nCould you please clarify what do you mean by \"*according to the IoU metric and the threshold*\" for FN?\nI thought the calculation of false negatives would **not** use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).\n\nIn other words for a threshold t\n\n> t=0.5 \n\n> TP = 0.5 <= IoU <= 1.0 \n\n> FP = 0 < IoU < 0.5 \n\n> FN = IoU = 0", "303474": "If you make a prediction and its IOU with the ground-truth is less than the threshold it counts as a FP, right? And if there are no predictions for a ground-truth then it counts as FN.\n\nHowever, if _all_ predictions for that ground-truth are FPs (i.e. if all of them are below the IOU threshold), then it also counts as a FN. After all, this is the same thing as saying there are no predictions for that ground-truth (because none of the existing predictions count).", "304197": "Could you clarify what you said more?\n\nWhy if all predictions are FPs, FN is counted? I completely did not understand what you said.", "304198": "d", "304201": "> **Peter Metz wrote**\n> \n> > \n> > **Human Analog wrote**\n> > \n> > FN is a false negative. These are the nuclei that you did not provide predictions for, i.e. the predictions that you missed. (**Again, according to the IOU metric and the threshold.**)\n> \n> Hi Human Analog,\n> \n> Could you please clarify what do you mean by \"*according to the IoU metric and the threshold*\" for FN?\n> I thought the calculation of false negatives would **not** use the thresholds, because it's a false negative when I completely missed a nuclei instance, e.g. I didn't just predict it with an accuracy below threshold (which is FP) but completely overlooked it's existence, meaning that there is no mask for that instance to compute IoU for at all! (which could be seen as an IoU of 0 I guess).\n> \n> In other words for a threshold t\n> \n> > t=0.5 \n> \n> > TP = 0.5 <= IoU <= 1.0 \n> \n> > FP = 0 < IoU < 0.5 \n> \n> > FN = IoU = 0\n> \n\nWhat you said makes sense the most for me.\nBasically, we do not say which prediction indicate which ground-truth because we don't know what ground-truths there are. That's why we cannot say which ground-truth is missing.\nI will proceed my progress based on your comment" }, "source": "meta" }