I am trying to understand how Kaggle calculates true-positives and false-positives when predictions match multiple ground-truths.
\nE.g. Let's consider a hypothetical sample and its predictions:
\ny_true_str = 'l x n o p|n o p q'
\ny_pred_str = 'x n p b|n o p a'
\nHere each character (l, x, n, etc.) could represent some word.
The Jaccard similarity matrix in this case would be:
\n| \n | n o p a | \nx n p b | \n
|---|---|---|
| l x n o p | \n0.5 | \n0.5 | \n
| n o p q | \n0.6 | \n0.33 | \n
Here, since there is a tie in the scores of the two predictions for the first ground-truth, the first prediction would be chosen, and the resultant true-prediction matrix would be:
\n| \n | n o p a | \nx n p b | \n
|---|---|---|
| l x n o p | \n1 | \n0 | \n
| n o p q | \n1 | \n0 | \n
Now, the first prediction matches two ground truth labels, and second does not match any.
\nBased on the description on the evaluation page, my understanding is that Kaggle counts False-positive as 1 since the second prediction does not 'match' any ground truth. Is this understanding correct?
\nAnd would the true-positive count be:
\nI am trying to understand how Kaggle calculates true-positives and false-positives when predictions match multiple ground-truths.
\nE.g. Let's consider a hypothetical sample and its predictions:
\ny_true_str = 'l x n o p|n o p q'
\ny_pred_str = 'x n p b|n o p a'
\nHere each character (l, x, n, etc.) could represent some word.
The Jaccard similarity matrix in this case would be:
\n| \n | n o p a | \nx n p b | \n
|---|---|---|
| l x n o p | \n0.5 | \n0.5 | \n
| n o p q | \n0.6 | \n0.33 | \n
Here, since there is a tie in the scores of the two predictions for the first ground-truth, the first prediction would be chosen, and the resultant true-prediction matrix would be:
\n| \n | n o p a | \nx n p b | \n
|---|---|---|
| l x n o p | \n1 | \n0 | \n
| n o p q | \n1 | \n0 | \n
Now, the first prediction matches two ground truth labels, and second does not match any.
\nBased on the description on the evaluation page, my understanding is that Kaggle counts False-positive as 1 since the second prediction does not 'match' any ground truth. Is this understanding correct?
\nAnd would the true-positive count be:
\nHey,
\nFirst of all, for the ground truth nopq , the matched label would be nopa and not mnpb, because of the higher Jaccard score. In this case, there would be 2 True positives because they have a score above or equal to 0.5 . (I am not sure whether the threshold for TP includes 0.5 or not).
Thanks, Ibtesam. Updated the post.
\nRegarding inclusion of 0.5, the evaluation page says 'Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP)', and therefore, 0.5 jaccard score predictions could be considered as TP's.
", "rawMarkdown": "Thanks, Ibtesam. Updated the post.\n\nRegarding inclusion of 0.5, the evaluation page says 'Any matched predictions where the Jaccard score *meets* or exceeds the threshold of 0.5 are counted as true positives (TP)', and therefore, 0.5 jaccard score predictions could be considered as TP's.", "votes": null }, { "id": "1270159", "postDate": "04/11/2021 11:18:25", "content": "Seems a relevant question to me
\nI would (also) say that the second prediction counts as a FP and the TP count should be 2.
\nBut I agree that the evaluation page is not (yet) clear about this and I couldn't find the answer in other posts but maybe I missed it.
@philculliton could you please shed some light on this?
", "rawMarkdown": "Seems a relevant question to me\nI would (also) say that the second prediction counts as a FP and the TP count should be 2.\nBut I agree that the evaluation page is not (yet) clear about this and I couldn't find the answer in other posts but maybe I missed it.\n\n@philculliton could you please shed some light on this?", "votes": null } ], "comments": [ { "id": 1262363, "author_name": "ibtesama", "author_url": "", "post_date": "04/04/2021 06:48:12", "content": "Hey,
\nFirst of all, for the ground truth nopq , the matched label would be nopa and not mnpb, because of the higher Jaccard score. In this case, there would be 2 True positives because they have a score above or equal to 0.5 . (I am not sure whether the threshold for TP includes 0.5 or not).
Thanks, Ibtesam. Updated the post.
\nRegarding inclusion of 0.5, the evaluation page says 'Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP)', and therefore, 0.5 jaccard score predictions could be considered as TP's.
", "votes": null, "replies": [] } ] }, { "id": 1270159, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "04/11/2021 11:18:25", "content": "Seems a relevant question to me
\nI would (also) say that the second prediction counts as a FP and the TP count should be 2.
\nBut I agree that the evaluation page is not (yet) clear about this and I couldn't find the answer in other posts but maybe I missed it.
@philculliton could you please shed some light on this?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1262123": "I am trying to understand how Kaggle calculates true-positives and false-positives when predictions match multiple ground-truths.\nE.g. Let's consider a hypothetical sample and its predictions:\ny_true_str = 'l x n o p|n o p q'\ny_pred_str = 'x n p b|n o p a'\nHere each character (l, x, n, etc.) could represent some word. \n\nThe Jaccard similarity matrix in this case would be:\n| | n o p a | x n p b |\n|---|---|---|\n|l x n o p | 0.5 |0.5|\n|n o p q |0.6|0.33|\n\nHere, since there is a tie in the scores of the two predictions for the first ground-truth, the first prediction would be chosen, and the resultant true-prediction matrix would be:\n| | n o p a | x n p b |\n|---|---|---|\n| l x n o p |1|0|\n|n o p q |1|0|\n\nNow, the first prediction matches two ground truth labels, and second does not match any.\n\nBased on the description on the evaluation page, my understanding is that Kaggle counts False-positive as 1 since the second prediction does not 'match' any ground truth. Is this understanding correct?\n\nAnd would the true-positive count be:\n- 2 (since two ground-truths are predicted correctly), or\n- 1 (since one predicted label has matched any of the ground truths)\nMy guess is that it should be the first option. But I would like to confirm this assumption.", "1262363": "Hey,\nFirst of all, for the ground truth `nopq` , the matched label would be `nopa` and not `mnpb`, because of the higher Jaccard score. In this case, there would be 2 True positives because they have a score above or equal to 0.5 . (I am not sure whether the threshold for TP includes 0.5 or not).", "1262370": "Thanks, Ibtesam. Updated the post.\n\nRegarding inclusion of 0.5, the evaluation page says 'Any matched predictions where the Jaccard score *meets* or exceeds the threshold of 0.5 are counted as true positives (TP)', and therefore, 0.5 jaccard score predictions could be considered as TP's.", "1270159": "Seems a relevant question to me\nI would (also) say that the second prediction counts as a FP and the TP count should be 2.\nBut I agree that the evaluation page is not (yet) clear about this and I couldn't find the answer in other posts but maybe I missed it.\n\n@philculliton could you please shed some light on this?" }, "source": "meta" }