{
  "id": 54852,
  "title": "matterport's \"compute_ap\" question",
  "url": "/competitions/data-science-bowl-2018/discussion/54852",
  "author_name": "",
  "post_date": "2018-04-18T18:22:18.337624800Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>hi,</p>\n\n<p>did anyone using matterport's MaskRCNN use the compute_ap to calculate their IOU scores? </p>\n\n<p>If yes, was it \"close\" to the public score (at least on stage 1)?</p>\n\n<p>For me, when calculated across the full training set, I came to around 0.61, but the stage1 score was 0.44 ... and that looks like a huge difference, so I'm thinking either I messed up my predictions or the function itself is not \"correct\" (I looked at the code, and it seems it should be fine)</p>",
  "messages": [
    {
      "id": "316324",
      "postDate": "04/18/2018 18:22:18",
      "content": "<p>hi,</p>\n\n<p>did anyone using matterport's MaskRCNN use the compute_ap to calculate their IOU scores? </p>\n\n<p>If yes, was it \"close\" to the public score (at least on stage 1)?</p>\n\n<p>For me, when calculated across the full training set, I came to around 0.61, but the stage1 score was 0.44 ... and that looks like a huge difference, so I'm thinking either I messed up my predictions or the function itself is not \"correct\" (I looked at the code, and it seems it should be fine)</p>",
      "rawMarkdown": "hi,\n\ndid anyone using matterport's MaskRCNN use the compute_ap to calculate their IOU scores? \n\nIf yes, was it \"close\" to the public score (at least on stage 1)?\n\nFor me, when calculated across the full training set, I came to around 0.61, but the stage1 score was 0.44 ... and that looks like a huge difference, so I'm thinking either I messed up my predictions or the function itself is not \"correct\" (I looked at the code, and it seems it should be fine)",
      "votes": null
    },
    {
      "id": "316498",
      "postDate": "04/19/2018 07:56:59",
      "content": "<p>If I remember correctly, the compute_ap() function in matterport's repo calculates the IOU of bounding boxes rather than masks. This is the biggest difference. </p>\n\n<p>Another difference is the definition of the precision function. The compute_ap() from matterport follows the canonical definition of precision (see wikipedia): TP/(TP+FP). In the evaluation page, you will see that we use TP/(TP+FP+FN). </p>\n\n<p>Check out this kernel for details: <a href=\"https://www.kaggle.com/stkbailey/step-by-step-explanation-of-scoring-metric\">https://www.kaggle.com/stkbailey/step-by-step-explanation-of-scoring-metric</a></p>\n\n<p>Computing the ious between all predicted and all ground-truth masks can be computationally intensive. I use the the cocoapi to do the calculation: </p>\n\n<pre><code>from pycocotools import mask as maskUtils\ndef mask_overlaps_rle(pred_masks, gt_masks):\n    pred_rle = [maskUtils.encode(np.asfortranarray(pred_masks[:,:,i])) \\\n                     for i in range(pred_masks.shape[2])]\n    gt_rle = [maskUtils.encode(np.asfortranarray(gt_masks[:,:,i])) \\\n                     for i in range(gt_masks.shape[2])]\n    iscrowd = [0 for i in gt_rle]\n    ious = maskUtils.iou(pred_rle, gt_rle, iscrowd)\n    return ious\n</code></pre>\n\n<p>`\n```</p>",
      "rawMarkdown": "If I remember correctly, the compute_ap() function in matterport's repo calculates the IOU of bounding boxes rather than masks. This is the biggest difference. \n\nAnother difference is the definition of the precision function. The compute_ap() from matterport follows the canonical definition of precision (see wikipedia): TP/(TP+FP). In the evaluation page, you will see that we use TP/(TP+FP+FN). \n\nCheck out this kernel for details: https://www.kaggle.com/stkbailey/step-by-step-explanation-of-scoring-metric\n\nComputing the ious between all predicted and all ground-truth masks can be computationally intensive. I use the the cocoapi to do the calculation: \n\n    from pycocotools import mask as maskUtils\n    def mask_overlaps_rle(pred_masks, gt_masks):\n        pred_rle = [maskUtils.encode(np.asfortranarray(pred_masks[:,:,i])) \\\n                         for i in range(pred_masks.shape[2])]\n        gt_rle = [maskUtils.encode(np.asfortranarray(gt_masks[:,:,i])) \\\n                         for i in range(gt_masks.shape[2])]\n        iscrowd = [0 for i in gt_rle]\n        ious = maskUtils.iou(pred_rle, gt_rle, iscrowd)\n        return ious\n\n`\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 316498,
      "author_name": "yueqiw",
      "author_url": "",
      "post_date": "04/19/2018 07:56:59",
      "content": "<p>If I remember correctly, the compute_ap() function in matterport's repo calculates the IOU of bounding boxes rather than masks. This is the biggest difference. </p>\n\n<p>Another difference is the definition of the precision function. The compute_ap() from matterport follows the canonical definition of precision (see wikipedia): TP/(TP+FP). In the evaluation page, you will see that we use TP/(TP+FP+FN). </p>\n\n<p>Check out this kernel for details: <a href=\"https://www.kaggle.com/stkbailey/step-by-step-explanation-of-scoring-metric\">https://www.kaggle.com/stkbailey/step-by-step-explanation-of-scoring-metric</a></p>\n\n<p>Computing the ious between all predicted and all ground-truth masks can be computationally intensive. I use the the cocoapi to do the calculation: </p>\n\n<pre><code>from pycocotools import mask as maskUtils\ndef mask_overlaps_rle(pred_masks, gt_masks):\n    pred_rle = [maskUtils.encode(np.asfortranarray(pred_masks[:,:,i])) \\\n                     for i in range(pred_masks.shape[2])]\n    gt_rle = [maskUtils.encode(np.asfortranarray(gt_masks[:,:,i])) \\\n                     for i in range(gt_masks.shape[2])]\n    iscrowd = [0 for i in gt_rle]\n    ious = maskUtils.iou(pred_rle, gt_rle, iscrowd)\n    return ious\n</code></pre>\n\n<p>`\n```</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "316324": "hi,\n\ndid anyone using matterport's MaskRCNN use the compute_ap to calculate their IOU scores? \n\nIf yes, was it \"close\" to the public score (at least on stage 1)?\n\nFor me, when calculated across the full training set, I came to around 0.61, but the stage1 score was 0.44 ... and that looks like a huge difference, so I'm thinking either I messed up my predictions or the function itself is not \"correct\" (I looked at the code, and it seems it should be fine)",
    "316498": "If I remember correctly, the compute_ap() function in matterport's repo calculates the IOU of bounding boxes rather than masks. This is the biggest difference. \n\nAnother difference is the definition of the precision function. The compute_ap() from matterport follows the canonical definition of precision (see wikipedia): TP/(TP+FP). In the evaluation page, you will see that we use TP/(TP+FP+FN). \n\nCheck out this kernel for details: https://www.kaggle.com/stkbailey/step-by-step-explanation-of-scoring-metric\n\nComputing the ious between all predicted and all ground-truth masks can be computationally intensive. I use the the cocoapi to do the calculation: \n\n    from pycocotools import mask as maskUtils\n    def mask_overlaps_rle(pred_masks, gt_masks):\n        pred_rle = [maskUtils.encode(np.asfortranarray(pred_masks[:,:,i])) \\\n                         for i in range(pred_masks.shape[2])]\n        gt_rle = [maskUtils.encode(np.asfortranarray(gt_masks[:,:,i])) \\\n                         for i in range(gt_masks.shape[2])]\n        iscrowd = [0 for i in gt_rle]\n        ious = maskUtils.iou(pred_rle, gt_rle, iscrowd)\n        return ious\n\n`\n```"
  },
  "source": "meta"
}