{"cells":[{"metadata":{"_uuid":"692d7077a3c68b2772b0e2f6f873aeaa15edb8c9"},"cell_type":"markdown","source":"Inspired by William Cukierski's [kernel](https://www.kaggle.com/wcukierski/example-metric-implementation), this notebook refactors the metric to be slightly faster. First, we import the packages needed:","outputs":[],"execution_count":null},{"metadata":{"_cell_guid":"59e67bfe-75ef-455f-a1f9-d4121b6303af","_uuid":"8880018fd9393c5a0f569f4a192ebcf38e68447a","collapsed":true,"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport skimage.io\nimport skimage.morphology\n\nsns.set(font_scale=1.3, palette=\"colorblind\", rc={\"figure.figsize\": (14,9) })","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f66b7e1da2ab9bc68ede652266219e36f67549d"},"cell_type":"markdown","source":"Next, we will load a image and construct a simulated prediction from the actual masks:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"51362a5a4dd17d6307206e6d5905fdae6a49ecf2"},"cell_type":"code","source":"img_id = '0a7d30b252359a10fd298b638b90cb9ada3acced4e0c0e5a3692013f432ee4e9'\nfile = f\"../input/stage1_train/{img_id}/images/{img_id}.png\"\nmfile = f\"../input/stage1_train/{img_id}/masks/*.png\"\nimage = skimage.io.imread(file)\nmasks = skimage.io.imread_collection(mfile).concatenate()\nheight, width, _ = image.shape\n\ny_true = (masks == 255).astype(np.uint8)\n\n# Remove item 20\ny_preds = np.concatenate((y_true[:19], y_true[20:]), axis=0)\ny_pred = np.empty_like(y_preds)\n\n# Reduce areas\nfor i, y_p in enumerate(y_preds):\n    skimage.morphology.binary_erosion(y_p, out=y_pred[i])\n\n# Label areas\nnum_true = y_true.shape[0]\nnum_pred = y_pred.shape[0]\nnp.multiply(y_true, \n            np.arange(1, num_true+1, dtype=np.uint8)[:, np.newaxis, np.newaxis], \n            out=y_true)\n\nnp.multiply(y_pred, \n            np.arange(1, num_pred+1, dtype=np.uint8)[:, np.newaxis, np.newaxis], \n            out=y_pred)\n\n# Combine labels into one image\ny_true = np.max(y_true, axis=0)\ny_pred = np.max(y_pred, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3cd066ffc32fad28b1cd8d01562618040a24caca"},"cell_type":"markdown","source":"We plot the true labels and the simulated predictions:","outputs":[],"execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0192dba4454b99aef5c3f781fa72a7c10e4b8fcd"},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2)\nax1.imshow(y_true)\nax1.set_title(\"True Labels\")\nax1.grid(False)\nax2.imshow(y_pred)\nax2.set_title(\"Simulated Predictions\")\nax2.grid(False);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"793d5c50164490f4b06e542e1dd38038821e6a38"},"cell_type":"markdown","source":"`y_pred` and `y_true` are two dimensional arrays labeled from 1 to `num_pred` and 1 to `num_true` respectively. The following defines the AP metric:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"04d9b868501688347aca6ec134e5517d12a5d6a0"},"cell_type":"code","source":"def ap(y_true, y_pred):\n    # remove one for background\n    num_true = len(np.unique(y_true)) - 1\n    num_pred = len(np.unique(y_pred)) - 1\n    \n    if num_true == 0 and num_pred == 0:\n        return 1\n    elif num_true == 0 or num_pred == 0:\n        return 0\n    \n    # bin size + 1 for background\n    intersect = np.histogram2d(\n        y_true.flatten(), y_pred.flatten(), bins=(num_true+1, num_pred+1))[0]\n    \n    area_t = np.histogram(y_true, bins=(num_true+1))[0][:, np.newaxis]\n    area_p = np.histogram(y_pred, bins=(num_pred+1))[0][np.newaxis, :]\n    \n    # get rid of background\n    union = area_t + area_p - intersect\n    intersect = intersect[1:, 1:]\n    union = union[1:, 1:]\n    iou = intersect / union \n    \n    threshold = np.arange(0.5, 1.0, 0.05)[np.newaxis, np.newaxis, :]\n    matches = iou[:,:, np.newaxis] > threshold\n    \n    tp = np.sum(matches, axis=(0,1))\n    fp = num_true - tp\n    fn = num_pred - tp\n    \n    return np.mean(tp/(tp+fp+fn))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"785883f05b06b20037f9c5272c22b2e69eac9498"},"cell_type":"markdown","source":"The AP Metric should be one when passed the correct labeling:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"c98358866bfa414465b6600708f90a680bdbc7f8"},"cell_type":"code","source":"ap(y_true, y_true)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a6879688ea4add3a69ca5dfa75433f05b32b5d6"},"cell_type":"markdown","source":"For our simulated prediction:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"35326fa940e35ab7323659dae0355dae32bbb861"},"cell_type":"code","source":"ap(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e1c95640745d1a18535cf340794971ad0ff753d"},"cell_type":"markdown","source":"","outputs":[],"execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"collapsed":true,"_uuid":"82dc6b80dfb5799be7440a521582c0b3b3959347"},"cell_type":"code","source":"def ap_wc(y_true_in, y_pred_in):\n    labels = y_true_in\n    y_pred = y_pred_in\n    \n    true_objects = len(np.unique(labels))\n    pred_objects = len(np.unique(y_pred))\n\n    intersection = np.histogram2d(labels.flatten(), y_pred.flatten(), bins=(true_objects, pred_objects))[0]\n\n    # Compute areas (needed for finding the union between all objects)\n    area_true = np.histogram(labels, bins = true_objects)[0]\n    area_pred = np.histogram(y_pred, bins = pred_objects)[0]\n    area_true = np.expand_dims(area_true, -1)\n    area_pred = np.expand_dims(area_pred, 0)\n\n    # Compute union\n    union = area_true + area_pred - intersection\n\n    # Exclude background from the analysis\n    intersection = intersection[1:,1:]\n    union = union[1:,1:]\n    union[union == 0] = 1e-9\n\n    # Compute the intersection over union\n    iou = intersection / union\n\n    # Precision helper function\n    def precision_at(threshold, iou):\n        matches = iou > threshold\n        true_positives = np.sum(matches, axis=1) == 1   # Correct objects\n        false_positives = np.sum(matches, axis=0) == 0  # Missed objects\n        false_negatives = np.sum(matches, axis=1) == 0  # Extra objects\n        tp, fp, fn = np.sum(true_positives), np.sum(false_positives), np.sum(false_negatives)\n        return tp, fp, fn\n\n    # Loop over IoU thresholds\n    prec = []\n    for t in np.arange(0.5, 1.0, 0.05):\n        tp, fp, fn = precision_at(t, iou)\n        if (tp + fp + fn) > 0:\n            p = tp / (tp + fp + fn)\n        else:\n            p = 0\n        prec.append(p)\n\n    return np.mean(prec)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9a2040b5015aa8ad61c96a86f4593b1168a89b1"},"cell_type":"markdown","source":"The metric from William Cukierski's [kernel](https://www.kaggle.com/wcukierski/example-metric-implementation) outputs the same result:","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"dbc4608323a2a050b5ea8ddef34a65b3ded9b1b3"},"cell_type":"code","source":"ap_wc(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2d24a54cef1b6875247bb7cda4548da0d2db498"},"cell_type":"markdown","source":"## Comparing Timings","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"244762d19a644f35ac0447fa1f24540a7f1937e0"},"cell_type":"code","source":"%%timeit\nap_wc(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"267aeb83d962e60eb64914d6e820b64373286278"},"cell_type":"code","source":"%%timeit\nap(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2ca1d3747c48a7afa5874aa4f9f12f5a2f93b62"},"cell_type":"markdown","source":"The refactored version is about 5% faster.  Success! :D","outputs":[],"execution_count":null}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}