{
  "id": 157393,
  "title": "Metric calculation",
  "url": "/competitions/global-wheat-detection/discussion/157393",
  "author_name": "cosmosaa",
  "post_date": "2020-06-10T14:28:28.064000",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I am confused on how to calculate the average precision for a single image at a single threshold for this competition.</p>\n\n<p>One part of the calculation that I am still confused about is the confidence. It doesn't seem to be used at all in the calculation. Let's say there is a ground truth box, and 3 predicted boxes that overlap with iou &gt; threshold. Only 1 is considered a true positive, and the other 2 are considered false positive. The confidence is irrelevant to this. How is the confidence used in the calculation?</p>",
  "messages": [
    {
      "id": 880776,
      "postDate": "2020-06-10T15:04:14.280Z",
      "content": "<p>It matters if you have more boxes, take a look at this example:</p>\n\n<p>GT1, GT2 - Ground truth boxes\nP1 - Predicted #1\nS1 - Score #1\nP2 - Predicted #2\nS2 - Score #2</p>\n\n<p>IoU-P1-GT1 = 0.7\nIoU-P1-GT2 = 0.6\nIoU-P2-GT1 = 0.5\nIoU-P2-GT2 = 0.4</p>\n\n<p>If S1 = 0.9 and S2 = 0.5 and\nImage precision (@0.5) = (0.7 + 0.0) / 2</p>\n\n<p>If S1 = 0.5 and S2 = 0.9\nImage precision (@0.5) = (0.5 + 0.6) / 2</p>",
      "rawMarkdown": "It matters if you have more boxes, take a look at this example:\n\nGT1, GT2 - Ground truth boxes\nP1 - Predicted #1\nS1 - Score #1\nP2 - Predicted #2\nS2 - Score #2\n\nIoU-P1-GT1 = 0.7\nIoU-P1-GT2 = 0.6\nIoU-P2-GT1 = 0.5\nIoU-P2-GT2 = 0.4\n\nIf S1 = 0.9 and S2 = 0.5 and\nImage precision (@0.5) = (0.7 + 0.0) / 2\n\nIf S1 = 0.5 and S2 = 0.9\nImage precision (@0.5) = (0.5 + 0.6) / 2",
      "votes": 1,
      "replies": [
        {
          "id": 880846,
          "postDate": "2020-06-10T15:45:16.810Z",
          "content": "<p>I see the difference now. But I calculated the precision differently.</p>\n\n<p>P1 and P2 match GT1\nP1 matches GT2</p>\n\n<p>For S1 = 0.9, S2 = 0.5\nP1 matches GT1, since it has a higher confidence than P2, and since its iou is greater with GT1 than GT2.\nP2 matches nothing since it only matches GT1 and P1 already matches with GT1\nTP: GT1-P1\nFP: GT1-P2\nFN: GT2\nPrecision: TP / (TP + FP + FN) = 1 / (1 + 1 + 1) = 1 / 3</p>\n\n<p>For S1 = 0.5, S2 = 0.9\nP2 matches GT1, since it has a higher confidence than P1\nP1 matches GT2\nTP: GT1-P2, GT2-P1\nFP: GT1-P1\nPrecision: TP / (TP + FP + FN) = 2 / (2 + 1 + 0) = 2 / 3</p>\n\n<p>Please let me know if you see a mistake in this.</p>",
          "rawMarkdown": "I see the difference now. But I calculated the precision differently.\n\nP1 and P2 match GT1\nP1 matches GT2\n\nFor S1 = 0.9, S2 = 0.5\nP1 matches GT1, since it has a higher confidence than P2, and since its iou is greater with GT1 than GT2.\nP2 matches nothing since it only matches GT1 and P1 already matches with GT1\nTP: GT1-P1\nFP: GT1-P2\nFN: GT2\nPrecision: TP / (TP + FP + FN) = 1 / (1 + 1 + 1) = 1 / 3\n\nFor S1 = 0.5, S2 = 0.9\nP2 matches GT1, since it has a higher confidence than P1\nP1 matches GT2\nTP: GT1-P2, GT2-P1\nFP: GT1-P1\nPrecision: TP / (TP + FP + FN) = 2 / (2 + 1 + 0) = 2 / 3\n\nPlease let me know if you see a mistake in this.",
          "votes": 1
        },
        {
          "id": 880883,
          "postDate": "2020-06-10T16:06:57.730Z",
          "content": "<p>Sorry, you are right with the precision calculation formula.</p>\n\n<p>Only the TP cases are in pairs (GTx-Py)\nFP: Predicted (Py) without GT match\nFN: Ground-truth (GTx) without prediction match</p>\n\n<p>In the first case:\nTP is correct\nFP is P2 because only P2-GT2 is available, but it is below the threshold\nFN correct\nPrecision: 1/3</p>\n\n<p>Second case:\nTP: correct\nFP: there are no false positives. All of the predicted objects have associated GT\nFN: correct. no false negatives.\nPrecision: 2 / (2 + 0 + 0) = 1.0</p>",
          "rawMarkdown": "Sorry, you are right with the precision calculation formula.\n\nOnly the TP cases are in pairs (GTx-Py)\nFP: Predicted (Py) without GT match\nFN: Ground-truth (GTx) without prediction match\n\nIn the first case:\nTP is correct\nFP is P2 because only P2-GT2 is available, but it is below the threshold\nFN correct\nPrecision: 1/3\n\nSecond case:\nTP: correct\nFP: there are no false positives. All of the predicted objects have associated GT\nFN: correct. no false negatives.\nPrecision: 2 / (2 + 0 + 0) = 1.0\n",
          "votes": 2
        }
      ]
    },
    {
      "id": 880733,
      "postDate": "2020-06-10T14:28:28.063Z",
      "content": "<p>I am confused on how to calculate the average precision for a single image at a single threshold for this competition.</p>\n\n<p>One part of the calculation that I am still confused about is the confidence. It doesn't seem to be used at all in the calculation. Let's say there is a ground truth box, and 3 predicted boxes that overlap with iou &gt; threshold. Only 1 is considered a true positive, and the other 2 are considered false positive. The confidence is irrelevant to this. How is the confidence used in the calculation?</p>",
      "rawMarkdown": "I am confused on how to calculate the average precision for a single image at a single threshold for this competition.\n\nOne part of the calculation that I am still confused about is the confidence. It doesn't seem to be used at all in the calculation. Let's say there is a ground truth box, and 3 predicted boxes that overlap with iou &gt; threshold. Only 1 is considered a true positive, and the other 2 are considered false positive. The confidence is irrelevant to this. How is the confidence used in the calculation?",
      "votes": 1
    },
    {
      "id": 880738,
      "postDate": "2020-06-10T14:32:01.110Z",
      "content": "<p>I believe boxes with the highest confidence are selected first. For a more complete look, maybe checkout Peter's <a href=\"https://www.kaggle.com/pestipeti/competition-metric-details-script\">notebook</a></p>",
      "rawMarkdown": "I believe boxes with the highest confidence are selected first. For a more complete look, maybe checkout Peter's [notebook](https://www.kaggle.com/pestipeti/competition-metric-details-script)",
      "replies": [
        {
          "id": 880743,
          "postDate": "2020-06-10T14:36:11.537Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 880776,
      "author_name": "Peter",
      "author_url": "",
      "post_date": "2020-06-10T15:04:14.280000",
      "content": "<p>It matters if you have more boxes, take a look at this example:</p>\n\n<p>GT1, GT2 - Ground truth boxes\nP1 - Predicted #1\nS1 - Score #1\nP2 - Predicted #2\nS2 - Score #2</p>\n\n<p>IoU-P1-GT1 = 0.7\nIoU-P1-GT2 = 0.6\nIoU-P2-GT1 = 0.5\nIoU-P2-GT2 = 0.4</p>\n\n<p>If S1 = 0.9 and S2 = 0.5 and\nImage precision (@0.5) = (0.7 + 0.0) / 2</p>\n\n<p>If S1 = 0.5 and S2 = 0.9\nImage precision (@0.5) = (0.5 + 0.6) / 2</p>",
      "votes": 1,
      "replies": [
        {
          "id": 880846,
          "author_name": "cosmosaa",
          "author_url": "",
          "post_date": "2020-06-10T15:45:16.810000",
          "content": "<p>I see the difference now. But I calculated the precision differently.</p>\n\n<p>P1 and P2 match GT1\nP1 matches GT2</p>\n\n<p>For S1 = 0.9, S2 = 0.5\nP1 matches GT1, since it has a higher confidence than P2, and since its iou is greater with GT1 than GT2.\nP2 matches nothing since it only matches GT1 and P1 already matches with GT1\nTP: GT1-P1\nFP: GT1-P2\nFN: GT2\nPrecision: TP / (TP + FP + FN) = 1 / (1 + 1 + 1) = 1 / 3</p>\n\n<p>For S1 = 0.5, S2 = 0.9\nP2 matches GT1, since it has a higher confidence than P1\nP1 matches GT2\nTP: GT1-P2, GT2-P1\nFP: GT1-P1\nPrecision: TP / (TP + FP + FN) = 2 / (2 + 1 + 0) = 2 / 3</p>\n\n<p>Please let me know if you see a mistake in this.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 880883,
          "author_name": "Peter",
          "author_url": "",
          "post_date": "2020-06-10T16:06:57.730000",
          "content": "<p>Sorry, you are right with the precision calculation formula.</p>\n\n<p>Only the TP cases are in pairs (GTx-Py)\nFP: Predicted (Py) without GT match\nFN: Ground-truth (GTx) without prediction match</p>\n\n<p>In the first case:\nTP is correct\nFP is P2 because only P2-GT2 is available, but it is below the threshold\nFN correct\nPrecision: 1/3</p>\n\n<p>Second case:\nTP: correct\nFP: there are no false positives. All of the predicted objects have associated GT\nFN: correct. no false negatives.\nPrecision: 2 / (2 + 0 + 0) = 1.0</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 880738,
      "author_name": "Matt",
      "author_url": "",
      "post_date": "2020-06-10T14:32:01.110000",
      "content": "<p>I believe boxes with the highest confidence are selected first. For a more complete look, maybe checkout Peter's <a href=\"https://www.kaggle.com/pestipeti/competition-metric-details-script\">notebook</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 880743,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-06-10T14:36:11.537000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "880776": "It matters if you have more boxes, take a look at this example:\n\nGT1, GT2 - Ground truth boxes\nP1 - Predicted #1\nS1 - Score #1\nP2 - Predicted #2\nS2 - Score #2\n\nIoU-P1-GT1 = 0.7\nIoU-P1-GT2 = 0.6\nIoU-P2-GT1 = 0.5\nIoU-P2-GT2 = 0.4\n\nIf S1 = 0.9 and S2 = 0.5 and\nImage precision (@0.5) = (0.7 + 0.0) / 2\n\nIf S1 = 0.5 and S2 = 0.9\nImage precision (@0.5) = (0.5 + 0.6) / 2",
    "880733": "I am confused on how to calculate the average precision for a single image at a single threshold for this competition.\n\nOne part of the calculation that I am still confused about is the confidence. It doesn't seem to be used at all in the calculation. Let's say there is a ground truth box, and 3 predicted boxes that overlap with iou &gt; threshold. Only 1 is considered a true positive, and the other 2 are considered false positive. The confidence is irrelevant to this. How is the confidence used in the calculation?",
    "880738": "I believe boxes with the highest confidence are selected first. For a more complete look, maybe checkout Peter's [notebook](https://www.kaggle.com/pestipeti/competition-metric-details-script)"
  }
}