{"cells":[{"metadata":{"_cell_guid":"9364d9aa-e79d-4bd3-a4db-192a42eedc33","_uuid":"5dad81af38acece12b2f5dd8d41e3f96b879c99c"},"cell_type":"markdown","source":"### Rationale\nI found the explanation for the scoring metric on this competition a little confusing, and I wanted to create a  guide for those who are just entering or haven't made it too far yet. The metric used for this competition is defined as **the mean average precision at different intersection over union (IoU) thresholds**.  \n\nThis tells us there are a few different steps to getting the score reported on the leaderboard. For each image...\n1. For each submitted nuclei \"prediction\", calculate the Intersection of Union metric with each \"ground truth\" mask in the image.\n2. Calculate whether this mask fits at a range of IoU thresholds.\n3. At each threshold, calculate the precision across all your submitted masks. \n4. Average the precision across thresholds.\n\nAcross the dataset...\n1. Calculate the mean of the average precision for each image."},{"metadata":{"_cell_guid":"7028485a-a5ad-4828-98ae-0abf01db10bc","_uuid":"e8ee8282f9002f02784082388173429f5f3edbf4"},"cell_type":"markdown","source":"### Picking a test image\nI'm going to pick a sample image from the training dataset, load the masks, then create a \"mock predict\" set of masks from it by moving and dilating each individual nucleus mask. Here's the dataset:"},{"metadata":{"_kg_hide-output":false,"_uuid":"4b8449d0a3077ae0fc9fa8b6a3d964df84513119","_cell_guid":"9f4f974a-0c91-415b-9a6c-8d5e1220b415","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nfrom scipy import ndimage\nimport pandas as pd\nimport imageio\nfrom pathlib import Path\n\n# Get image\nim_id = '01d44a26f6680c42ba94c9bc6339228579a95d0e2695b149b7cc0c9592b21baf'\nim_dir = Path('../input/stage1_train/{}'.format(im_id))\nim_path = im_dir / 'images' / '{}.png'.format(im_id)\nim = imageio.imread(im_path.as_posix())\n\n# Get masks\ntarg_masks = []\nfor mask_path in im_dir.glob('masks/*.png'):\n    targ = imageio.imread(mask_path.as_posix())\n    targ_masks.append(targ)\ntarg_masks = np.stack(targ_masks)\n\n# Make messed up masks\npred_masks = np.zeros(targ_masks.shape)\nfor ind, orig_mask in enumerate(targ_masks):\n    aug_mask = ndimage.rotate(orig_mask, ind*1.5, \n                              mode='constant', reshape=False, order=0)\n    pred_masks[ind] = ndimage.binary_dilation(aug_mask, iterations=1)\n\n# Plot the objects\nfig, axes = plt.subplots(1,3, figsize=(16,9))\naxes[0].imshow(im)\naxes[1].imshow(targ_masks.sum(axis=0),cmap='hot')\naxes[2].imshow(pred_masks.sum(axis=0), cmap='hot')\n\nlabels = ['Original', '\"GroundTruth\" Masks', '\"Predicted\" Masks']\nfor ind, ax in enumerate(axes):\n    ax.set_title(labels[ind], fontsize=18)\n    ax.axis('off')","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"6c2537d3-948d-4a3a-a1f5-4d8d386ff011","_uuid":"51548ebf117cc5c17ee2eb0b1f7bdb2040f5c31f"},"cell_type":"markdown","source":"### Intersection Over Union (for a single Prediction-GroundTruth comparison)\n\n> The IoU of a proposed set of object pixels and a set of true object pixels is calculated as:\n$$\nIoU(A,B)=\\frac{A∩B}{A∪B}\n$$\n\nLet's take one of the nuclei masks from our GroundTruth and Predicted volumes. Their intersections and unions look like this:"},{"metadata":{"_uuid":"fa63320943a20203e923ca7a8f76e03adf456667","_cell_guid":"e6b9a27c-5118-4351-94d8-23bcd6663d43","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"A = targ_masks[3]\nB = pred_masks[3]\nintersection = np.logical_and(A, B)\nunion = np.logical_or(A, B)\n\nfig, axes = plt.subplots(1,4, figsize=(16,9))\naxes[0].imshow(A, cmap='hot')\naxes[0].annotate('npixels = {}'.format(np.sum(A>0)), \n                 xy=(114, 245), color='white', fontsize=16)\naxes[1].imshow(B, cmap='hot')\naxes[1].annotate('npixels = {}'.format(np.sum(B>0)), \n                 xy=(114, 245), color='white', fontsize=16)\n\naxes[2].imshow(intersection, cmap='hot')\naxes[2].annotate('npixels = {}'.format(np.sum(intersection>0)), \n                 xy=(114, 245), color='white', fontsize=16)\n\naxes[3].imshow(union, cmap='hot')\naxes[3].annotate('npixels = {}'.format(np.sum(union>0)), \n                 xy=(114, 245), color='white', fontsize=16)\n\nlabels = ['GroundTruth', 'Predicted', 'Intersection', 'Union']\nfor ind, ax in enumerate(axes):\n    ax.set_title(labels[ind], fontsize=18)\n    ax.axis('off')","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"94be852c-e848-47b5-ab16-c4b469eb5dd7","_uuid":"88feaff0515c487ba6116249bab0b77eed62950c"},"cell_type":"markdown","source":"Notice how the intersection will always be less than or equal to the size of the GroundTruth object, and the Union will always be greater than or equal to that size.\n\nSo, for this set of masks, the IoU metric is calculated as:\n$$\nIoU(A,B)=\\frac{A∩B}{A∪B} = \\frac{564}{849} = 0.664\n$$\n"},{"metadata":{"_cell_guid":"27da1107-a9e3-4e5c-8914-bd878f873757","_uuid":"2bd9575e375ebc9c188477fb6c0a9cc3790bc835"},"cell_type":"markdown","source":"### Thresholding the IoU value (for a single GroundTruth-Prediction comparison)\nNext, we sweep over a range of IoU thresholds to get a vector for each mask comparison.  The threshold values range from 0.5 to 0.95 with a step size of 0.05: `(0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95)`. \n\nIn other words, at a threshold of 0.5, a predicted object is considered a \"hit\" if its intersection over union with a ground truth object is greater than 0.5."},{"metadata":{"_uuid":"5fcee2a4355fe50d4482180cf71bb790021273c2","_cell_guid":"1b40c467-86cf-4fcf-a1a0-8ee29e46cd5a","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_iou_vector(A, B, n):\n    intersection = np.logical_and(A, B)\n    union = np.logical_or(A, B)\n    iou = np.sum(intersection > 0) / np.sum(union > 0)\n    s = pd.Series(name=n)\n    for thresh in np.arange(0.5,1,0.05):\n        s[thresh] = iou > thresh\n    return s\n\nprint('Does this IoU hit at each threshold?')\nprint(get_iou_vector(A, B, 'GT-P'))","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"98042056-b56f-4a1b-b320-7cb1a4a69075","_uuid":"f390bdce6f103d8843325871e8ea910eb25f4b08"},"cell_type":"markdown","source":"Now, for each prediction mask (P), we'll get a comparison with every ground truth mask (GT). In most cases, this will be zero since nuclei shouldn't overlap, but this also allows flexibility in matching up each mask to each potential nucleus."},{"metadata":{"_uuid":"94125816d69400de9d290549e0e7f0a033e9c7cb","_cell_guid":"c3e3b230-dde9-4c3c-87bd-01405e164d61","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = pd.DataFrame()\nfor ind, gt_mask in enumerate(targ_masks):\n    s = get_iou_vector(pred_masks[3], gt_mask, 'P3-GT{}'.format(ind))\n    df = df.append(s)\nprint('Performance of Predicted Mask 3 vs. each Ground Truth mask across IoU thresholds')\nprint(df)","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"503d8ac4-f506-4f63-a9fc-4743f5624200","_uuid":"6bc7a60ecb771b909d55a0329debb017b45ed199"},"cell_type":"markdown","source":"### Single-threshold precision for a single image\n\nNow, in our example, we've created 7 prediction masks ($P_i$) to compare with 7 ground truth masks ($GT_j$). At each threshold, we will have a $7*7$ matrix showing whether there was a hit with that object. The precision value is based on the number of true positives (TP), false negatives (FN), and false positives (FP) in this \"hit matrix\".\n\n$$\nPrecision(t) = \\frac{TP(t)}{TP(t)+FP(t)+FN(t)}\n$$\n\n* TP: a single predicted object matches a ground truth object with an IoU above the threshold\n* FP: a predicted object had no associated ground truth object. \n* FN: a ground truth object had no associated predicted object. "},{"metadata":{"_kg_hide-output":false,"_uuid":"1be9e056d72271b6e40b15d5b05c2e089bc29c05","_cell_guid":"741a6cad-1f26-49f8-9f2d-8534d1214c47","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"iou_vol = np.zeros([10, 7, 7])\nfor ii, pm in enumerate(pred_masks):\n    for jj, gt in enumerate(targ_masks):\n        s = get_iou_vector(pm, gt, 'P{}-GT{}'.format(ii,jj))\n        iou_vol[:,ii,jj] = s.values\n\nmask_labels = ['P{}'.format(x) for x in range(7)]\ntruth_labels = ['GT{}'.format(x) for x in range(7)]\n\nhits50 = iou_vol[0]\nhits75 = iou_vol[4]\n\nfig, axes = plt.subplots(1,2, figsize=(10,9))\n\naxes[0].imshow(hits50, cmap='hot')\naxes[0].set_xticks(range(7))\naxes[0].set_xticklabels(truth_labels, rotation=45, ha='right', fontsize=16)\naxes[0].tick_params(left=False, bottom=False)\naxes[0].set_yticks(range(7))\naxes[0].set_yticklabels(mask_labels, fontsize=16)\naxes[0].tick_params(left=False, bottom=False)\naxes[0].set_title('Hit Matrix at $thresh=0.50$', fontsize=18)\n\naxes[1].imshow(hits75, cmap='hot')\naxes[1].set_xticks(range(7))\naxes[1].set_xticklabels(truth_labels, rotation=45, ha='right', fontsize=16)\naxes[1].tick_params(left=False, bottom=False)\naxes[1].tick_params(left=False, bottom=False, labelleft=False)\naxes[1].set_title('Hit Matrix at $thresh=0.75$', fontsize=18)\n\nfor ax in axes:\n    # Minor ticks and turn grid on\n    ax.set_xticks(np.arange(-.5, 7, 1), minor=True);\n    ax.set_yticks(np.arange(-.5, 7, 1), minor=True);\n    ax.grid(which='minor', color='w', linestyle='-', linewidth=1)\n\nplt.tight_layout()\nplt.show()","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"d606fc43-282e-4d42-a1ca-60b8a6910350","_uuid":"1ce09a96d0f7aa8660d0163b82c48dfe716423cc"},"cell_type":"markdown","source":"In the above matrix...\n* The number of **true positives** is equal to the number of predictions with a \"hit\" on a true object.\n* The number of **false positives** is equal to the number of predictions that don't hit anything.\n* The number of **false negatives** is equal to the number of \"ground truth\" objects that aren't hit."},{"metadata":{"_uuid":"5792d3a0cb910f9544b7e4c3672d5a60c0039342","_cell_guid":"cf7c0f26-6622-4ede-80f6-06b589f5a33c","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def iou_thresh_precision(iou_mat):\n    tp = np.sum( iou_mat.sum(axis=1) > 0  )\n    fp = np.sum( iou_mat.sum(axis=1) == 0 )\n    fn = np.sum( iou_mat.sum(axis=0) == 0 )\n    p = tp / (tp + fp + fn)\n    return (tp, fp, fn, p)\n\nfor thresh, hits in [[0.5, hits50], [0.75, hits75]]:\n    tp, fp, fn, p = iou_thresh_precision(hits)\n    print('At a threshold of {:0.2f}...\\n\\tTP = {}\\n\\tFP = {}\\n\\tFN = {}\\n\\tp = {:0.3f}'.format(\n                thresh, tp, fp, fn, p))","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"a7253e00-c7cd-4a1a-8a80-9e67337e13e5","_uuid":"27e36f9ee0fff43604ca5a07784eeb77f970344b"},"cell_type":"markdown","source":"### Multi-threshold precision for a single image\n\n> The average precision of a single image is then calculated as the mean of the above precision values at each IoU threshold:\n$$\nAvg.\\ Precision = \\frac{1}{n_{thresh}}  \\sum_{t=1}^nprecision(t)\n$$\n\nHere, we simply take the average of the precision values at each threshold to get our mean precision for the image."},{"metadata":{"_uuid":"765f2cecc12fe5ce1810e1b0e595202c0035ff9f","_cell_guid":"c33fde6c-4151-4617-910a-04c1547448e3","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Precision values at each threshold:')\nps = []\nfor thresh, iou_mat in zip(np.arange(0.5, 1, 0.05), iou_vol):\n    _,_,_,p = iou_thresh_precision(iou_mat)\n    print('\\tt({:0.2f}) = {:0.3f}'.format(thresh, p))\n    ps.append(p)\nprint('Mean precision for image is: {:0.3f}'.format(np.mean(ps)))","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"263ab0f5-6e5c-455d-9526-f502e6132186","_uuid":"344fcf79419ee73b502bacce7f7a2021c6a68492"},"cell_type":"markdown","source":"\n### Mean average precision for the dataset\n\n>Lastly, the score returned by the competition metric is the mean taken over the individual average precisions of each image in the test dataset.\n\nTherefore, the leaderboard metric will simply be the mean of the precisions across all the images.\n\nHope you found this helpful -- I know it helped me to work through it!"}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}