{"cells":[{"metadata":{},"cell_type":"markdown","source":"I made this notebook because I was confused about the competition metric calculation, so I wanted to implement it myself. \n\nI noticed the following about the metric:\n* It is **not** \"mean average precision\" but rather is just \"mean precision\". Average precision uses a precision recall curve to calculate the average precision. This metric does not. Rather, this metric just calculates precision and takes the mean at all iou thresholds.\n* Precision the way it is defined is **not** the usual definition of precision. Rather precision is usually defined as TP / (TP + FP), but here it is defined as TP / (TP + FP + FN). As noted [here](https://www.kaggle.com/c/global-wheat-detection/discussion/156874) it is more like F1.\n* Confidences are not used in the precision-recall curve as they are normally used to compute average precision, but rather are used to break ties when multiple predicted boxes match a single ground truth box.\n\nThanks to https://www.kaggle.com/pestipeti/competition-metric-details-script. Note that I have used `box_iou` to calculate iou which does not add 1 when calculating area, so results are a bit different.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import defaultdict\nimport numpy as np\nimport torch\nfrom torchvision.ops.boxes import box_iou","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def calculate_precision(boxes_true: torch.tensor, boxes_pred: torch.tensor, confidences: list, threshold=0.5) -> float:\n    \"\"\"Calculates precision for GT - prediction pairs at one threshold.\"\"\"\n    \n    confidences = np.array(confidences)\n    \n    # edge case for no ground truth boxes\n    if boxes_true.size(1) == 0:\n        return 0.\n\n    iou = box_iou(boxes1=boxes_pred, boxes2=boxes_true)\n\n    pr_matches = set()\n    gt_matches = set()\n\n    # for each ground truth box, get list of pred boxes it matches with\n    match_candidates = (iou >= threshold).nonzero()\n    GT_PR_matches = defaultdict(list)\n    for PR, GT in match_candidates:\n        GT_PR_matches[GT.item()].append(PR.item())\n\n    # Find which pred matches a GT box\n    for GT, PRs in GT_PR_matches.items():\n        # if multiple preds match a single ground truth box,\n        # select the pred with the highest confidence\n        if len(PRs) > 1:\n            pr_match = PRs[confidences[PRs].argsort()[-1]]\n        # else only a single pred matches this GT box\n        else:\n            pr_match = PRs[0]\n\n        # only if we haven't seen a pred before can we mark a PR-GT pair as TP\n        # otherwise the pred matches a different GT box and this GT might instead be a FN\n        if pr_match not in pr_matches:\n            gt_matches.add(GT)\n\n        pr_matches.add(pr_match)\n\n    TP = len(pr_matches)\n\n    pr_idx = range(iou.size(0))\n    gt_idx = range(iou.size(1))\n\n    FP = len(set(pr_idx).difference(pr_matches))\n    FN = len(set(gt_idx).difference(gt_matches))\n\n    return TP / (TP + FP + FN)\n\ndef calculate_mean_precision(boxes_true: torch.tensor, boxes_pred: torch.tensor, confidences: np.array, thresholds=(0.5,)):\n    \"\"\"Calculates average precision over a set of thresholds\"\"\"\n    \n    precision = np.zeros(len(thresholds))\n\n    for i, threshold in enumerate(thresholds):\n        precision[i] = calculate_precision(boxes_true=boxes_true, boxes_pred=boxes_pred, confidences=confidences,\n                                                     threshold=threshold)\n    return precision.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def test_calc_precision():\n    boxes_true = torch.tensor([\n        [0., 0., 10., 10.],     # GT1\n        [0., 0., 12., 10.]      # GT2\n    ])\n    boxes_pred = torch.tensor([\n        [0., 0., 10., 6.],      # P1\n        [0., 0., 10., 5.]       # P2\n    ])\n    confidences = [.5, .9]\n    score = calculate_precision(boxes_true=boxes_true, boxes_pred=boxes_pred, confidences=confidences, threshold=.5)\n    assert score == 1.\n\n    confidences = [.9, .5]\n    score = calculate_precision(boxes_true=boxes_true, boxes_pred=boxes_pred, confidences=confidences, threshold=.5)\n    assert score == 1/3\n\n    score = calculate_precision(boxes_true=torch.tensor([[]]), boxes_pred=boxes_pred,\n                                confidences=confidences, threshold=.5)\n    assert score == 0\ntest_calc_precision()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}