{"cells":[{"metadata":{"_uuid":"2903ba531fb15d3943b80f28774e0d24d0377741","_cell_guid":"5cb12f78-e5a9-47d9-bcfd-ca322744c2ce"},"cell_type":"markdown","source":"This notebook is a reworked version of William Cukierski's kernel [here](https://www.kaggle.com/wcukierski/example-metric-implementation). It gives the same results.\n\nI wanted to use matrix multiplication for intersection and matrix maximum for union as this is easier to get my head around that the histogram approach. It will also help if I later code in C++ with binary maps. I also managed to simplify some of the formulae.\n\nIn the process, I noted that the evaluation method stipulates that the IOU score is based on the prediction masks, which is relevant where there is a mismatch between prediction and ground truth. Specifically if two true nuclei are fused together as one in the prediction or vice versa. So I think this code should work ok in the general case.\n\nHopefully this will help some of you. All comments welcome","outputs":[],"execution_count":null},{"metadata":{"_uuid":"60f807c1f159ae97c846698488dd5010b7215395","_cell_guid":"e0be591d-b0be-41ce-97a2-377e9612cb66"},"cell_type":"markdown","source":"This is an example notebook to demonstrate how the IoU metric works for a single image. Please note: this is not the official scoring implementation, but should work in the same manner.","outputs":[],"execution_count":null},{"metadata":{"_uuid":"3fd84142326a40f5f7ba094ce49c125d5a95837b","_cell_guid":"1b889269-1027-4261-a3e9-54f75223571b","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport numpy as np\nimport skimage.io\nimport matplotlib.pyplot as plt\nimport skimage.segmentation\n\n# Load a single image and its associated masks\nid = '0a7d30b252359a10fd298b638b90cb9ada3acced4e0c0e5a3692013f432ee4e9'\nfile = \"../input/stage1_train/{}/images/{}.png\".format(id,id)\nmfile = \"../input/stage1_train/{}/masks/*.png\".format(id)\nimage = skimage.io.imread(file)\nmasks = skimage.io.imread_collection(mfile).concatenate()\nheight, width, _ = image.shape\nnum_masks = masks.shape[0]\n\n# Make a ground truth array and summary label image\ny_true = np.zeros((num_masks, height, width), np.uint16)\ny_true[:,:,:] = masks[:,:,:] // 255  # Change ground truth mask to zeros and ones\n\nlabels = np.zeros((height, width), np.uint16)\nlabels[:,:] = np.sum(y_true, axis=0)  # Add up to plot all masks\n    \n# Show label images\nfig = plt.figure()\nplt.imshow(image)\nplt.title(\"Original image\")\nfig = plt.figure()\nplt.imshow(y_true[3])\nplt.title(\"One example ground truth mask\")\nfig = plt.figure()\nplt.imshow(labels)\nplt.title(\"All ground truth masks\")\n\n# Simulate an imperfect submission\noffset = 2 # offset pixels\ny_pr1 = y_true[:19, offset:, offset:]  # To remove 'item 20' as per other kernel\ny_pr2 = y_true[20:, offset:, offset:]\ny_pred = np.concatenate((y_pr1, y_pr2), axis=0)\ny_pred = np.pad(y_pred, ((0,0), (0, offset), (0, offset)), mode=\"constant\")\n#y_pred[y_pred == 20] = 0 # Remove one object\n#y_pred, _, _ = skimage.segmentation.relabel_sequential(y_pred) # Relabel objects\nyptot = np.sum(y_pred, axis=0)  # Sum individual predictions for plotting\n\n# Show simulated predictions\nfig = plt.figure()\nplt.imshow(y_pred[3])\nplt.title(\"An example simulated imperfect submission\")\nfig = plt.figure()\nplt.imshow(yptot)\nplt.title(\"All simulated imperfect submissions\")\nplt.show()\n\n# Compute number of objects\nnum_true = len(y_true)\nnum_pred = len(y_pred)\nprint(\"Number of true objects:\", num_true)\nprint(\"Number of predicted objects:\", num_pred)\n\n# Compute iou score for each prediction\niou = []\nfor pr in range(num_pred):\n    bol = 0  # best overlap\n    bun = 1e-9  # corresponding best union\n    for tr in range(num_true):\n        olap = y_pred[pr] * y_true[tr]  # Intersection points\n        osz = np.sum(olap)  # Add the intersection points to see size of overlap\n        if osz > bol:  # Choose the match with the biggest overlap\n            bol = osz\n            bun = np.sum(np.maximum(y_pred[pr], y_true[tr]))  # Union formed with sum of maxima\n    iou.append(bol / bun)\n\n# Loop over IoU thresholds\np = 0\nprint(\"Thresh\\tTP\\tFP\\tFN\\tPrec.\")\nfor t in np.arange(0.5, 1.0, 0.05):\n    matches = iou > t\n    tp = np.count_nonzero(matches)  # True positives\n    fp = num_pred - tp  # False positives\n    fn = num_true - tp  # False negatives\n    p += tp / (tp + fp + fn)\n    print(\"{:1.3f}\\t{}\\t{}\\t{}\\t{:1.3f}\".format(t, tp, fp, fn, tp / (tp + fp + fn)))\n\nprint(\"AP\\t-\\t-\\t-\\t{:1.3f}\".format(p / 10))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"542d2b226d90fc13cc852979f529962387873420","_cell_guid":"bc357ad2-d10c-4f5b-9f6a-55d47336779f","collapsed":true},"cell_type":"markdown","source":"This matches the table in Willam Cukierski's kernel.","outputs":[],"execution_count":null}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}