![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1739631%2F043d554a703ca8c792209d86ddbaa955%2Ff1.PNG?generation=1591612938712698&alt=media\")
Hi Everyone, I am new to object detection and I have a question on Kaggle mAP metric formulation.
\n\nI have carefully read the competition evaluation page as well as Peter's @pestipeti excellent script on metric calculation https://www.kaggle.com/pestipeti/competition-metric-details-script , as well as articles on standard mAP , e.g. https://manalelaidouni.github.io/manalelaidouni.github.io/Evaluating-Object-Detection-Models-Guide-to-Performance-Metrics.html
\n\nAll articles I found calculate area under PR curve for each IoU threshold, but Kaggle instead uses Jaccard formula (see https://clusteval.sdu.dk/1/clustering_quality_measures/7 )
\n\n> $$Precision(t)=\\frac{TP(t)}{TP(t) + FP(t) + FN(t)}$$
\n\nUnlike PR-AUC, the above Jaccard formula is actually not the common Precision definition where FN is not used : \n> https://en.wikipedia.org/wiki/Precision_and_recall\n> $$Precision =\\frac{TP}{TP + FP}$$
\n\nI am aware that this formulation also uses in the past llike pneumonia detection. But is there any explanation why the formulas are different? Or why we have to use the same 'precision' name?
\n\nFor example, if we have two bboxes predict the same GT with different IoU wrt. threshold of, say 0.5 (we can change similar IoU for other thresholds):
\n\nbox1 : IoU = 0.8 \nbox2 : IoU = 0.45
\n\nFollowing Peter's notebook to calculate TP, suppose we are more confident on box1, then box1 is counted as one TP, while box2 will be counted as one FP
\n\nHowever, even if we are more confident on box2, box2 will be counted as one FP first, and box1 will be one TP, so the results are actually the same, no matter of confidence scores? (even though in the evaluation page, Kaggle talks about confidence score sorting)
\n\n(confidence scores indeed make difference in this case if we calculate mAP using standard PR-AUC)
\n\n", "messages": [ { "id": "878124", "postDate": "06/08/2020 09:15:03", "content": "Hi Everyone, I am new to object detection and I have a question on Kaggle mAP metric formulation.
\n\nI have carefully read the competition evaluation page as well as Peter's @pestipeti excellent script on metric calculation https://www.kaggle.com/pestipeti/competition-metric-details-script , as well as articles on standard mAP , e.g. https://manalelaidouni.github.io/manalelaidouni.github.io/Evaluating-Object-Detection-Models-Guide-to-Performance-Metrics.html
\n\nAll articles I found calculate area under PR curve for each IoU threshold, but Kaggle instead uses Jaccard formula (see https://clusteval.sdu.dk/1/clustering_quality_measures/7 )
\n\n> $$Precision(t)=\\frac{TP(t)}{TP(t) + FP(t) + FN(t)}$$
\n\nUnlike PR-AUC, the above Jaccard formula is actually not the common Precision definition where FN is not used : \n> https://en.wikipedia.org/wiki/Precision_and_recall\n> $$Precision =\\frac{TP}{TP + FP}$$
\n\nI am aware that this formulation also uses in the past llike pneumonia detection. But is there any explanation why the formulas are different? Or why we have to use the same 'precision' name?
\n\nFor example, if we have two bboxes predict the same GT with different IoU wrt. threshold of, say 0.5 (we can change similar IoU for other thresholds):
\n\nbox1 : IoU = 0.8 \nbox2 : IoU = 0.45
\n\nFollowing Peter's notebook to calculate TP, suppose we are more confident on box1, then box1 is counted as one TP, while box2 will be counted as one FP
\n\nHowever, even if we are more confident on box2, box2 will be counted as one FP first, and box1 will be one TP, so the results are actually the same, no matter of confidence scores? (even though in the evaluation page, Kaggle talks about confidence score sorting)
\n\n(confidence scores indeed make difference in this case if we calculate mAP using standard PR-AUC)
\n\n", "rawMarkdown": "Hi Everyone, I am new to object detection and I have a question on Kaggle mAP metric formulation. \n\nI have carefully read the competition evaluation page as well as Peter's @pestipeti excellent script on metric calculation https://www.kaggle.com/pestipeti/competition-metric-details-script , as well as articles on standard mAP , e.g. https://manalelaidouni.github.io/manalelaidouni.github.io/Evaluating-Object-Detection-Models-Guide-to-Performance-Metrics.html \n\n### 1) Why does Kaggle mAP formula different from the standard mAP formula of object detection? \nAll articles I found calculate ***area under PR curve*** for each IoU threshold, but Kaggle instead uses ***Jaccard formula*** (see https://clusteval.sdu.dk/1/clustering_quality_measures/7 )\n\n> $$Precision(t)=\\frac{TP(t)}{TP(t) + FP(t) + FN(t)}$$\n\nUnlike PR-AUC, the above Jaccard formula is actually not the common Precision definition where FN is not used : \n> https://en.wikipedia.org/wiki/Precision_and_recall\n> $$Precision =\\frac{TP}{TP + FP}$$\n\nI am aware that this formulation also uses in the past llike pneumonia detection. But is there any explanation why the formulas are different? Or why we have to use the same 'precision' name?\n\n---------\n### 2) In Kaggle Precision formula, it seems like \"box's confidence\" scores are not really used, even though the evaluation page encourage us to sort our bboxes by scores.\n\nFor example, if we have two bboxes predict the same GT with different IoU wrt. threshold of, say 0.5 (we can change similar IoU for other thresholds): \n\nbox1 : IoU = 0.8 \nbox2 : IoU = 0.45\n\nFollowing Peter's notebook to calculate TP, suppose we are more confident on box1, then box1 is counted as one TP, while box2 will be counted as one FP\n\nHowever, even if we are more confident on box2, box2 will be counted as one FP first, and box1 will be one TP, so the results are actually the same, no matter of confidence scores? (even though in the evaluation page, Kaggle talks about confidence score sorting)\n\n(confidence scores indeed make difference in this case if we calculate mAP using standard PR-AUC)\n\n@maggiemd @stavness @bendvd", "votes": null }, { "id": "878187", "postDate": "06/08/2020 10:41:54", "content": "For the first question:
\n\nAdding FN(t) in the rate is very important as it makes sure that we're not missing some true boxes. As it combines between two classical metrics which are positive predictive value (PPV) 'or precision' and true positive rate (TPR) 'or recall' which make it mathematically closer to the F1 score but a bit different.
\n\n
Thanks Fares @ffares for your kind answer!
\n\nI am actually aware of that point. The questions I had in my mind regarding your explanations when I posted were :
\n\nA) standard mAP formula already handles FN automatically (making area under PR curve to zero on those FN examples) , so why not use the standard formula?
\n\nB) I am OK with the new formula, but just curious why it is still called by the same name ? (why don't change the name because it causes some confusion [or is it just me that confused :D ] )
", "rawMarkdown": "Thanks Fares @ffares for your kind answer! \n\nI am actually aware of that point. The questions I had in my mind regarding your explanations when I posted were : \n\nA) standard mAP formula already handles FN automatically (making area under PR curve to zero on those FN examples) , so why not use the standard formula?\n\nB) I am OK with the new formula, but just curious why it is still called by the same name ? (why don't change the name because it causes some confusion [or is it just me that confused :D ] )", "votes": null }, { "id": "878238", "postDate": "06/08/2020 11:31:26", "content": "I got you. Me too I am curious why it is still called 'Precision' so yeah, you're not the only one :D.
", "rawMarkdown": "I got you. Me too I am curious why it is still called 'Precision' so yeah, you're not the only one :D.", "votes": null } ], "comments": [ { "id": 878187, "author_name": "ffares", "author_url": "", "post_date": "06/08/2020 10:41:54", "content": "For the first question:
\n\nAdding FN(t) in the rate is very important as it makes sure that we're not missing some true boxes. As it combines between two classical metrics which are positive predictive value (PPV) 'or precision' and true positive rate (TPR) 'or recall' which make it mathematically closer to the F1 score but a bit different.
\n\n
Thanks Fares @ffares for your kind answer!
\n\nI am actually aware of that point. The questions I had in my mind regarding your explanations when I posted were :
\n\nA) standard mAP formula already handles FN automatically (making area under PR curve to zero on those FN examples) , so why not use the standard formula?
\n\nB) I am OK with the new formula, but just curious why it is still called by the same name ? (why don't change the name because it causes some confusion [or is it just me that confused :D ] )
", "votes": null, "replies": [] }, { "id": 878238, "author_name": "ffares", "author_url": "", "post_date": "06/08/2020 11:31:26", "content": "I got you. Me too I am curious why it is still called 'Precision' so yeah, you're not the only one :D.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "878124": "Hi Everyone, I am new to object detection and I have a question on Kaggle mAP metric formulation. \n\nI have carefully read the competition evaluation page as well as Peter's @pestipeti excellent script on metric calculation https://www.kaggle.com/pestipeti/competition-metric-details-script , as well as articles on standard mAP , e.g. https://manalelaidouni.github.io/manalelaidouni.github.io/Evaluating-Object-Detection-Models-Guide-to-Performance-Metrics.html \n\n### 1) Why does Kaggle mAP formula different from the standard mAP formula of object detection? \nAll articles I found calculate ***area under PR curve*** for each IoU threshold, but Kaggle instead uses ***Jaccard formula*** (see https://clusteval.sdu.dk/1/clustering_quality_measures/7 )\n\n> $$Precision(t)=\\frac{TP(t)}{TP(t) + FP(t) + FN(t)}$$\n\nUnlike PR-AUC, the above Jaccard formula is actually not the common Precision definition where FN is not used : \n> https://en.wikipedia.org/wiki/Precision_and_recall\n> $$Precision =\\frac{TP}{TP + FP}$$\n\nI am aware that this formulation also uses in the past llike pneumonia detection. But is there any explanation why the formulas are different? Or why we have to use the same 'precision' name?\n\n---------\n### 2) In Kaggle Precision formula, it seems like \"box's confidence\" scores are not really used, even though the evaluation page encourage us to sort our bboxes by scores.\n\nFor example, if we have two bboxes predict the same GT with different IoU wrt. threshold of, say 0.5 (we can change similar IoU for other thresholds): \n\nbox1 : IoU = 0.8 \nbox2 : IoU = 0.45\n\nFollowing Peter's notebook to calculate TP, suppose we are more confident on box1, then box1 is counted as one TP, while box2 will be counted as one FP\n\nHowever, even if we are more confident on box2, box2 will be counted as one FP first, and box1 will be one TP, so the results are actually the same, no matter of confidence scores? (even though in the evaluation page, Kaggle talks about confidence score sorting)\n\n(confidence scores indeed make difference in this case if we calculate mAP using standard PR-AUC)\n\n@maggiemd @stavness @bendvd", "878187": "For the first question:\n\nAdding FN(t) in the rate is very important as it makes sure that we're not missing some true boxes. As it combines between two classical metrics which are positive predictive value (PPV) 'or precision' and true positive rate (TPR) 'or recall' which make it mathematically closer to the F1 score but a bit different. \n\n", "878216": "Thanks Fares @ffares for your kind answer! \n\nI am actually aware of that point. The questions I had in my mind regarding your explanations when I posted were : \n\nA) standard mAP formula already handles FN automatically (making area under PR curve to zero on those FN examples) , so why not use the standard formula?\n\nB) I am OK with the new formula, but just curious why it is still called by the same name ? (why don't change the name because it causes some confusion [or is it just me that confused :D ] )", "878238": "I got you. Me too I am curious why it is still called 'Precision' so yeah, you're not the only one :D." }, "source": "meta" }