{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Error analysis\n\nIn the previous competitions, I did not spend much time analyzing my model's performance. I did what almost all of the beginners:, tried different models, optimizers, losses, augmentations; I tried to implement more advanced techniques like mixup.\n\nI more or less entirely ignored the error analysis. That was a mistake!\n\nFortunately, now I had a bit of free time, so I published my first Error analysis notebook, and I have to say: After I finished it, I instantly had lots of ideas on how and where I should improve my models.\n\n--------------\n\nI think we have to improve our models (architecture, augmentation, pre-, post-process, etc) in two areas.\n\n## 1, Find the boxes\n\nFirst, I want to minimize the number of false-negative and false-positive predictions at a low threshold. For this, I used a 0.5 IoU threshold, but maybe we can go even lower. The goal is to make sure the model (or models) detects all of the wheat-heads and doesn't generate false positives. At this stage, I don't care about the overall precision (averaged over multiple thresholds).\n\n\n**Possible problems**:\n- The wheat-head is too small/large\n- The image (or part of it) is too dark/bright\n- The contrast is low\n- Multiple heads are in a close group\n- Head covered by a leaf or something else\n- Heads at the edge of the images\n- Label noise\n\n\n## 2, Improve precision\nAfter the model can detect most of the GT, we can start thinking about improving its precision. The question is, how can we detect the same number of (TP) boxes at higher `[0.5, 0.55, 0.6, 0.65, 0.7, 0.75]` thresholds.\n\n**Possible problems**\n- Anchor size/ratio is not good\n- Multiple heads are in a close group\n- Noisy targets\n- Weak loss function/s\n\n\n**In this notebook, I'll focus on the first area.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**NOTE**: *This is a starter notebook. There is plenty of room for improvement. I'll update as soon as I have time. (Vote if you are interested!)*","execution_count":null},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport cv2\nimport os\nimport random\nimport ast\nimport matplotlib.pyplot as plt\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nfrom tqdm.notebook import tqdm\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configure\nYou can analyze your data; you only need to set your model's OOF predictions to analyze your results.\n\n- Create a private dataset and upload your results (OOF predictions; same format as the submission.csv; see below)\n- Fork this notebook\n- Add your dataset\n- Modify the `DIR_RESULTS` and the `VALID_RESULTS` config params below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# =========================================\n# Replace these with your data.\n# =========================================\n#  - I have 1 .csv file per fold\n#  - All of these have the OOF predictions\n#  - You can use one fold too.\n#  - The format should be the same as the\n#    submission.csv\n\n# Your private dataset\nDIR_RESULTS = '/kaggle/input/global-wheat-detection-public'\n\n# Your OOF predictions\nVALID_RESULTS = [\n    f\"{DIR_RESULTS}/validation_results_fold0_best.csv\",\n    f\"{DIR_RESULTS}/validation_results_fold1_best.csv\",\n    f\"{DIR_RESULTS}/validation_results_fold2_best.csv\",\n    f\"{DIR_RESULTS}/validation_results_fold3_best.csv\",\n    f\"{DIR_RESULTS}/validation_results_fold4_best.csv\",\n]\n\n# =========================================\n\n# Below this area the size category of the box is 'small'\nAREA_SMALL = 56 * 56\n\n# Below this (and above small) is medium;\n# Above this is large.\nAREA_MEDIUM = 96 * 96\n\n# If the box is at most this far from either of the borders\n# we mark the box as 'is_border = True'\nBORDER_SIZE = 2\n\n# In these experiments I used 800px inputs.\n# For analysis, we have to scale back to 1024px\n# because the GT boxes are in that size.\nSCALE = 1024/800\n\n# Analizing at this threshold\nTHRESHOLD = 0.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DIR_INPUT = '/kaggle/input/global-wheat-detection'\nDIR_TRAIN = f'{DIR_INPUT}/train'\nDIR_TEST = f'{DIR_INPUT}/test'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def decode_prediction_string(pred_str):\n    data = list(map(float, pred_str.split(\" \")))\n    data = np.array(data)\n    \n    return data.reshape(-1, 5)[:, 1:]\n\ndef calculate_iou(gt, pr, form='pascal_voc') -> float:\n\n    if form == 'coco':\n        gt = gt.copy()\n        pr = pr.copy()\n\n        gt[2] = gt[0] + gt[2]\n        gt[3] = gt[1] + gt[3]\n        pr[2] = pr[0] + pr[2]\n        pr[3] = pr[1] + pr[3]\n\n    # Calculate overlap area\n    dx = min(gt[2], pr[2]) - max(gt[0], pr[0])\n\n    if dx < 0:\n        return 0.0\n\n    dy = min(gt[3], pr[3]) - max(gt[1], pr[1])\n\n    if dy < 0:\n        return 0.0\n\n    overlap_area = dx * dy\n\n    # Calculate union area\n    union_area = (\n            (gt[2] - gt[0]) * (gt[3] - gt[1]) +\n            (pr[2] - pr[0]) * (pr[3] - pr[1]) -\n            overlap_area\n    )\n\n    return overlap_area / union_area\n\n\ndef find_best_match(gts, pred, pred_idx, threshold=0.5, form='pascal_voc', ious=None) -> int:\n    best_match_iou = -np.inf\n    best_match_idx = -1\n\n    for gt_idx in range(len(gts)):\n\n        if gts[gt_idx][0] < 0:\n            # Already matched GT-box\n            continue\n\n        iou = -1 if ious is None else ious[gt_idx][pred_idx]\n\n        if iou < 0:\n            iou = calculate_iou(gts[gt_idx], pred, form=form)\n\n            if ious is not None:\n                ious[gt_idx][pred_idx] = iou\n\n        if iou < threshold:\n            continue\n\n        if iou > best_match_iou:\n            best_match_iou = iou\n            best_match_idx = gt_idx\n\n    return best_match_idx\n\ndef gen_images(data, filters, output_folder='./output', prefix='', limit=100):\n    \n    res = 'fp'\n    resdata = data.copy()\n\n    for filt in filters:\n        resdata = resdata[resdata[filt[0]] == filt[1]]\n        \n        prefix = f\"{prefix}_{filt[1]}\"\n        \n        if filt[0] == 'result':\n            res = filt[1]\n        \n        \n    if limit > 0:\n        resdata = resdata.sample(n=limit)\n        \n    image_ids = resdata['image_id'].unique()\n    res_images = []\n    \n    for image_id in image_ids:\n        img = cv2.imread(DIR_TRAIN + '/{}.jpg'.format(image_id))\n        \n        if res == 'fn':\n            boxes = resdata[resdata['image_id'] == image_id][['gt_x1', 'gt_y1', 'gt_x2', 'gt_y2']].values\n        elif res == 'fp':\n            boxes = resdata[resdata['image_id'] == image_id][['pred_x1', 'pred_y1', 'pred_x2', 'pred_y2']].values\n        \n        for box in boxes:\n            # tp\n            color = (0, 220, 0)\n\n            if res == 'fp':\n                # Showing GT boxes nearby\n                tpfilt = (\n                    (data['image_id'] == image_id) &\n                    (data['gt_x1'] < box[2] + 16) &\n                    (data['gt_x2'] > box[0] - 16) &\n                    \n                    (data['gt_y1'] < box[3] + 16) &\n                    (data['gt_y2'] > box[1] - 16)\n                )\n            \n                tps = data[tpfilt][['gt_x1', 'gt_y1', 'gt_x2', 'gt_y2']].values\n                for tpbox in tps:\n                    cv2.rectangle(img,\n                                  (int(tpbox[0]), int(tpbox[1])),\n                                  (int(tpbox[2]), int(tpbox[3])),\n                                  color, 3)\n            \n            if res == 'fn':\n                color = (40, 40, 198)\n            elif res == 'fp':\n                color = (198, 40, 40)\n\n            cv2.rectangle(img,\n                          (int(box[0]), int(box[1])),\n                          (int(box[2]), int(box[3])),\n                          color, 3)\n                \n            \n        res_images.append((img, f\"{output_folder}/{prefix}_{image_id}.jpg\"))\n        \n    return res_images\n    \ndef save_images(data, filters, output_folder='./output', prefix='', limit=100):\n    images = gen_images(data=data, filters=filters, limit=limit)\n    \n    for image, path in images:\n        cv2.imwrite(path, image)\n        \ndef show_images(data, filters, rows=2, cols=2):\n    \n    images = gen_images(data=data, filters=filters, output_folder='', limit=rows*cols)\n    \n    fig, ax = plt.subplots(rows, cols, figsize=(16,16))\n    ax = ax.flatten()\n    \n    for i, (image, path) in enumerate(images):\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        \n        ax[i].set_axis_off()\n        ax[i].imshow(image)\n        ax[i].set_title(path)\n        \n        \ndef show_image_boxes(train_df, data):\n    data = data.to_dict('records')\n\n    fig, ax = plt.subplots(1, 2, figsize=(16, 10))\n    ax = ax.flatten()\n    \n    image = cv2.imread(DIR_TRAIN + '/{}.jpg'.format(data[0]['image_id']))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n    src_img = image.copy()\n    \n    boxes = train_df[train_df['image_id'] == data[0]['image_id']][['x', 'y', 'x2', 'y2']].values\n    \n    for box in boxes:\n        cv2.rectangle(src_img,\n                      (int(box[0]), int(box[1])),\n                      (int(box[2]), int(box[3])),\n                      (0, 220, 0), 2)\n\n    ax[0].set_axis_off()\n    ax[0].imshow(src_img)\n    ax[0].set_title(\"Image + GT boxes\")\n        \n    # noisy targets\n    for box_data in data:\n        # fn\n        color = (40, 40, 198)\n        box = [0, 0, 0, 0]\n\n        if box_data['result'] == 'fn':\n            box[0], box[1], box[2], box[3] = box_data['gt_x1'],\\\n                                             box_data['gt_y1'],\\\n                                             box_data['gt_x2'],\\\n                                             box_data['gt_y2']\n\n\n        elif box_data['result'] == 'fp':\n            \n            box[0], box[1], box[2], box[3] = box_data['pred_x1'],\\\n                                             box_data['pred_y1'],\\\n                                             box_data['pred_x2'],\\\n                                             box_data['pred_y2']\n\n            color = (198, 40, 40)\n\n        cv2.rectangle(image,\n                      (int(box[0]), int(box[1])),\n                      (int(box[2]), int(box[3])),\n                      color, 2)\n\n    ax[1].set_axis_off()\n    ax[1].imshow(image)\n    ax[1].set_title(\"Blue: FP (predicted, no GT) | Red: FN (GT, no prediction)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load and prepare data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Training data","execution_count":null},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(f\"{DIR_INPUT}/train.csv\")\n\n# From Andrew's kernel\ntrain_df[['x', 'y', 'w', 'h']] = pd.DataFrame(\n    np.stack(train_df['bbox'].apply(lambda x: ast.literal_eval(x)))).astype(np.float32)\ntrain_df.drop(columns=['bbox'], inplace=True)\n\ntrain_df['x2'] = train_df['x'] + train_df['w']\ntrain_df['y2'] = train_df['y'] + train_df['h']\n\n# Calculate the area of the boxes.\ntrain_df['area'] = train_df['w'] * train_df['h']\n\n# Is the box at the edge of the image\ntrain_df['is_border'] = False\n\nborder_filt = ((train_df['x'] < BORDER_SIZE) | (train_df['y'] < BORDER_SIZE) |\n             (train_df['x2'] > 1024 - BORDER_SIZE) | (train_df['y2'] > 1024 - BORDER_SIZE))\ntrain_df.loc[border_filt, 'is_border'] = True\n\ntrain_df['size'] = 'large'\ntrain_df.loc[train_df['area'] < AREA_MEDIUM, 'size'] = 'medium'\ntrain_df.loc[train_df['area'] < AREA_SMALL, 'size'] = 'small'\n\n# These are the ground-truth boxes\ntrain_df['is_gt'] = True\n\ntrain_df['brightness'] = 0.0\ntrain_df['contrast'] = 0.0\ntrain_df['overlap_iou'] = 0.0\n\ntrain_df.sort_values(by='image_id', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Calculate box infos\n# - Brightness\n# - Contrast\n# - Hightest overlap with other GT box\n\nlast_src_id = None\nsrc = None\n\nfor i, row in tqdm(train_df.iterrows(), total=train_df.shape[0]):\n    \n    if last_src_id != row['image_id']:\n        src = cv2.imread(DIR_TRAIN + '/{}.jpg'.format(row['image_id']))\n        last_src_id = row['image_id']\n\n    \n    y1 = int(row['y'])\n    y2 = int(row['y2'])\n    x1 = int(row['x'])\n    x2 = int(row['x2'])\n\n    image = src[y1:y2, x1:x2].copy()\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n    \n    train_df.loc[i, 'brightness'] = image[:, :, 2].mean()\n    \n    image = cv2.cvtColor(image, cv2.COLOR_HSV2RGB)\n    image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n    \n    train_df.loc[i, 'contrast'] = image.std()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validation data\nI trained 5-folds; these validation results are the out-of-fold results. By combining these, we can analyze the entire training set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Format of the validation dataframes\npd.read_csv(VALID_RESULTS[0], usecols=['image_id', 'PredictionString']).head(3)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"valid_df = []\n\n# helper.\nimage_df = train_df.groupby(by=['image_id', 'source'])[['is_gt']].nunique().reset_index()[['image_id', 'source']]\n\nfor src in VALID_RESULTS:\n    valid = pd.read_csv(src, usecols=['image_id', 'PredictionString'])\n    valid = valid.merge(image_df[['image_id', 'source']], how='left', on='image_id')\n    valid.reset_index(drop=True, inplace=True)\n\n    res = []\n\n    for i, row in valid.iterrows():\n\n        boxes = decode_prediction_string(row['PredictionString'])\n\n        for box in boxes:\n\n            valid_df.append({\n                'image_id': row['image_id'],\n                'width': 1024,\n                'height': 1024,\n                'bbox': '',\n                'source': row['source'],\n                'x': box[0] * SCALE,\n                'y': box[1] * SCALE,\n                'x2': (box[0] + box[2]) * SCALE,\n                'y2': (box[1] + box[3]) * SCALE,\n                'w': box[2] * SCALE,\n                'h': box[3] * SCALE,\n                'area': (box[2] * box[3]) * SCALE,\n                'size': 'large',\n                'is_border': False,\n                'is_gt': False,\n                'brightness': 0.0,\n                'contrast': 0.0\n\n            })\n\n\n# Convert the list to a pd.DataFrame\nvalid_df = pd.DataFrame(valid_df)\n\nborder_filt = ((valid_df['x'] < BORDER_SIZE) | (valid_df['y'] < BORDER_SIZE) |\n             (valid_df['x2'] > 1024 - BORDER_SIZE) | (valid_df['y2'] > 1024 - BORDER_SIZE))\nvalid_df.loc[border_filt, 'is_border'] = True\n\nvalid_df.loc[valid_df['area'] < AREA_MEDIUM, 'size'] = 'medium'\nvalid_df.loc[valid_df['area'] < AREA_SMALL, 'size'] = 'small'\n\nvalid_df.sort_values(by='image_id', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Calculate box infos\n# - Brightness\n# - Contrast\n# - Hightest overlap with other GT box\n\nlast_src_id = None\nsrc = None\n\nfor i, row in tqdm(valid_df.iterrows(), total=valid_df.shape[0]):\n    \n    if last_src_id != row['image_id']:\n        src = cv2.imread(DIR_TRAIN + '/{}.jpg'.format(row['image_id']))\n        last_src_id = row['image_id']\n\n    \n    y1 = int(row['y'])\n    y2 = int(row['y2'])\n    x1 = int(row['x'])\n    x2 = int(row['x2'])\n\n    image = src[y1:y2, x1:x2].copy()\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n    \n    valid_df.loc[i, 'brightness'] = image[:, :, 2].mean()\n    \n    image = cv2.cvtColor(image, cv2.COLOR_HSV2RGB)\n    image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n    \n    valid_df.loc[i, 'contrast'] = image.std()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the final dataframe","execution_count":null},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def calc(gts, preds, threshold=0.5, form='pascal-voc'):\n    \n    def _get_data(image_id, res, gt, pr):\n        return {\n                'image_id': image_id,\n                'gt_x1': gt[1] if gt is not None else np.nan,\n                'gt_y1': gt[2] if gt is not None else np.nan,\n                'gt_x2': gt[3] if gt is not None else np.nan,\n                'gt_y2': gt[4] if gt is not None else np.nan,\n                'gt_w': gt[5] if gt is not None else np.nan,\n                'gt_h': gt[6] if gt is not None else np.nan,\n                'gt_area': gt[7] if gt is not None else np.nan,\n                'gt_is_border': gt[8] if gt is not None else False,\n                'gt_brightness': gt[12] if gt is not None else np.nan,\n                'gt_contrast': gt[13] if gt is not None else np.nan,\n                \n                'pred_x1': pr[1] if pr is not None else np.nan,\n                'pred_y1': pr[2] if pr is not None else np.nan,\n                'pred_x2': pr[3] if pr is not None else np.nan,\n                'pred_y2': pr[4] if pr is not None else np.nan,\n                'pred_w': pr[5] if pr is not None else np.nan,\n                'pred_h': pr[6] if pr is not None else np.nan,\n                'pred_area': pr[7] if pr is not None else np.nan,\n                'pred_is_border': pr[8] if pr is not None else False,\n                'pred_brightness': pr[12] if pr is not None else np.nan,\n                'pred_contrast': pr[13] if pr is not None else np.nan,\n                \n                'size': gt[10] if gt is not None else pr[10],\n                'source': gt[11] if gt is not None else pr[11],\n            \n                'result': res\n            }\n    \n    results = []\n    \n    # Number of predictions\n    n = len(preds)\n    \n    for pred_idx in range(n):\n        pr = preds[pred_idx]\n        \n        best_match_gt_idx = find_best_match(gts[:, 1:5], pr[1:5], pred_idx, threshold=threshold, form=form)\n        \n        if best_match_gt_idx >= 0:\n            # True positive: The predicted box matches a gt box with an IoU above the threshold.\n            gt = gts[best_match_gt_idx]\n            results.append(_get_data(gt[0], 'tp', gt, pr))\n            gts[best_match_gt_idx] = -1\n\n        else:\n            # No match\n            # False positive: indicates a predicted box had no associated gt box.\n            results.append(_get_data(pr[0], 'fp', None, pr))\n\n    for gt in gts:\n        if gt[1] < 0:\n            continue\n            \n        results.append(_get_data(gt[0], 'fn', gt, None))\n    \n    return results","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"cols = ['image_id', 'x', 'y', 'x2', 'y2', 'w', 'h', 'area', 'is_border',\n        'is_gt', 'size', 'source', 'brightness', 'contrast']\n\nvalid_img_ids = valid_df['image_id'].unique()\n\nresults = []\n\nfor img_id in tqdm(valid_img_ids, total=len(valid_img_ids)):\n    gt_boxes = train_df[train_df['image_id'] == img_id][cols].values\n    pred_boxes = valid_df[valid_df['image_id'] == img_id][cols].values\n    \n    results += calc(gt_boxes, pred_boxes, threshold=THRESHOLD, form='pascal-voc')\n    \nresults = pd.DataFrame(results)\n\nresults['is_border'] = False\nresults.loc[(results['gt_is_border'] == True) | (results['pred_is_border'] == True), 'is_border'] = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"------------------","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Overall results","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = ['True Positive', 'False Positive', 'False Negative']\nvalues = results['result'].value_counts().sort_index(ascending=False).values\n\nfig = make_subplots(rows=1, cols=2, specs=[[{\"type\": \"bar\"}, {\"type\": \"pie\"}]])\nfig.add_trace(go.Bar(x=labels, y=values, showlegend=False), row=1, col=1)\nfig.add_trace(go.Pie(labels=labels, values=values), row=1, col=2)\nfig.update_layout(\n    title={\n        'text': f'Number and ratio of TP/FP/FN at threshold {THRESHOLD}'\n    }\n)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sources = results['source'].value_counts().sort_index()\n\nfig = go.Figure([go.Bar(x=sources.index, y=sources.values)])\nfig.update_layout(\n    title = {\n        'text': f'Number of boxes: GT (TP or FN) + FP - at threshold {THRESHOLD}'\n    }\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sources = results['size'].value_counts().sort_index()\n\nfig = go.Figure([go.Bar(x=sources.index, y=sources.values)])\nfig.update_layout(\n    title={\n        'text': f'Number of boxes by size: GT (TP or FN) + FP - at threshold {THRESHOLD}'\n    }\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sources = results['is_border'].value_counts().sort_index().sort_index()\n\nfig = go.Figure([go.Bar(x=['Normal', 'Border'], y=sources.values)])\nfig.update_layout(\n    title={\n        'text': f'Number of boxes by type: GT (TP or FN) + FP - at threshold {THRESHOLD}'\n    }\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Histogram(x=results['gt_brightness']))\nfig.update_layout(title={\n    'text': \"Brightness of GT boxes\"\n})\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Histogram(x = results['gt_contrast']))\nfig.update_layout(title={\n    'text': 'Contrast value of GT boxes'\n})\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Results by source","execution_count":null},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def show_by_group(data, filt, group, idx, cols, title='', names=None, colors=None, order=None):\n    \n    if filt is not None:\n        data = data[filt] \n    \n    res = data.groupby(by=group).count()[['image_id']].reset_index().sort_index()\n    res = res.pivot(index=idx, columns=cols, values='image_id')\n    \n    fig = go.Figure()\n    \n    if order is None:\n        order = range(res.shape[0])\n\n    for row_idx in order:\n        fig.add_trace(go.Bar(\n            x=res.columns,\n            y=res.iloc[row_idx].values,\n            name=names[row_idx] if names is not None else res.index[row_idx],\n            marker_color=colors[row_idx] if colors is not None else None\n        ))\n        \n    fig.update_layout(\n        barmode='stack',\n        barnorm = 'percent',\n        title = {\n            'text': title\n        }\n    )\n    \n    return res, fig","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=None, group=['source', 'result'],\n                         idx='result',\n                         cols='source',\n                         names=['False Negative', 'False Positive', 'True Positive'],\n                         colors=['#c62828', '#3f51b5', '#4caf50'],\n                         title='Results (TP|FP|FN) by sources'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## False negatives","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sources = results[results['result'] == 'fn']['source'].value_counts().sort_index().sort_index()\n\nfig = go.Figure([go.Pie(labels=sources.index, values=sources.values)])\nfig.update_layout(\n    title = {\n        'text': f'False negatives - at threshold {THRESHOLD}'\n    }\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### False negatives by size","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=results['result'] == 'fn',\n                         group=['source', 'size'],\n                         idx='size',\n                         cols='source',\n                         names=['Large', 'Medium', 'Small'],\n                         colors=['#c62828', '#e57373','#ffcdd2'],\n                         order=[2, 1, 0],\n                         title='False negatives by size'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### False negatives at the borders","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=results['result'] == 'fn',\n                         group=['source', 'is_border'],\n                         idx='is_border',\n                         cols='source',\n                         names=['Normal', 'Border'],\n                         colors=['#ffcdd2', '#c62828'],\n                         order=[1,0],\n                         title='False negatives normal/border'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Examples: FN, rres_1, border","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filters = [\n    ('result', 'fn'),\n    ('is_border', True),\n    ('source', 'rres_1')\n]\n\nshow_images(results.copy(), filters, rows=2, cols=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Brightness differences","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Histogram(x=results[results['result'] == 'fn']['gt_brightness'],\n                           histnorm='probability', name='False negatives', marker={'color': '#c62828'}))\nfig.add_trace(go.Histogram(x=results[results['result'] == 'tp']['gt_brightness'],\n                           histnorm='probability', name='True positives', marker={'color': '#4caf50'}))\n\nfig.update_layout(barmode='overlay', title={\n    'text': 'Brightness'\n})\nfig.update_traces(opacity=0.75)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Contrast differences","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Histogram(x=results[results['result'] == 'fn']['gt_contrast'],\n                           histnorm='probability', name='False negatives', marker={'color': '#c62828'}))\nfig.add_trace(go.Histogram(x=results[results['result'] == 'tp']['gt_contrast'],\n                           histnorm='probability', name='True positives', marker={'color': '#4caf50'}))\n\nfig.update_layout(barmode='overlay', title={\n    'text': 'Contrast'\n})\nfig.update_traces(opacity=0.75)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## False positives","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sources = results[results['result'] == 'fp']['source'].value_counts().sort_index().sort_index()\n\nfig = go.Figure([go.Pie(labels=sources.index, values=sources.values)])\nfig.update_layout(\n    title = {\n        'text': f'False positives - at threshold {THRESHOLD}'\n    }\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### False positives by size","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=results['result'] == 'fp',\n                         group=['source', 'size'],\n                         idx='size',\n                         cols='source',\n                         names=['Large', 'Medium', 'Small'],\n                         colors=['#283593', '#3f51b5','#7986cb'],\n                         order=[2, 1, 0],\n                         title='False positives by size'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Examples: FP, inrae_1, large","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filters = [\n    ('result', 'fp'),\n    ('size', 'large'),\n    ('source', 'inrae_1')\n]\n\nshow_images(results.copy(), filters, rows=2, cols=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### False positives at the borders","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=results['result'] == 'fp',\n                         group=['source', 'is_border'],\n                         idx='is_border',\n                         cols='source',\n                         names=['Normal', 'Border'],\n                         colors=['#7986cb', '#283593'],\n                         order=[1, 0],\n                         title='False positives normal/border'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Examples: FP, inrae_1, border","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filters = [\n    ('result', 'fp'),\n    ('is_border', True),\n    ('source', 'inrae_1'),\n]\n\nshow_images(results.copy(), filters, rows=2, cols=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Brightness differences","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Histogram(x=results[results['result'] == 'fp']['pred_brightness'],\n                           histnorm='probability', name='False positives', marker={'color':'#3f51b5'}))\nfig.add_trace(go.Histogram(x=results[results['result'] == 'tp']['gt_brightness'],\n                           histnorm='probability', name='True positives', marker={'color': '#4caf50'}))\n\nfig.update_layout(barmode='overlay', title={\n    'text': 'Brightness'\n})\nfig.update_traces(opacity=0.75)\n\nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Contrast differences","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Histogram(x=results[results['result'] == 'fp']['pred_contrast'],\n                           histnorm='probability', name='False positives', marker={'color':'#3f51b5'}))\nfig.add_trace(go.Histogram(x=results[results['result'] == 'tp']['gt_contrast'],\n                           histnorm='probability', name='True positives', marker={'color': '#4caf50'}))\n\nfig.update_layout(barmode='overlay', title={\n    'text': 'Contrast'\n})\nfig.update_traces(opacity=0.75)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Results by size","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=None, group=['size', 'result'],\n                         idx='result',\n                         cols='size',\n                         names=['False Negative', 'False Positive', 'True Positive'],\n                         colors=['#c62828', '#3f51b5', '#4caf50'],\n                         title='Results by size'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Results of normal and \"border\" boxes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=None, group=['is_border', 'result'],\n                         idx='result',\n                         cols='is_border',\n                         names=['False Negative', 'False Positive', 'True Positive'],\n                         colors=['#c62828', '#3f51b5', '#4caf50'],\n                         title='Results normal/border'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results of \"border\" boxes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=results['is_border'] == True,\n                         group=['size', 'result'],\n                         idx='result',\n                         cols='size',\n                         names=['False Negative', 'False Positive', 'True Positive'],\n                         colors=['#c62828', '#3f51b5', '#4caf50'],\n                         title='Results at the borders by size'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Examples: FP, small, border","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filters = [\n    ('result', 'fp'),\n    ('size', 'small'),\n    ('is_border', True)\n]\n\nshow_images(results.copy(), filters, rows=2, cols=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Results of \"normal\" boxes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"res, fig = show_by_group(data=results, filt=results['is_border'] == False,\n                         group=['size', 'result'],\n                         idx='result',\n                         cols='size',\n                         names=['False Negative', 'False Positive', 'True Positive'],\n                         colors=['#c62828', '#3f51b5', '#4caf50'],\n                         title='Normal results by size'\n                        )\n\nres","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Examples: FP, small, normal","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filters = [\n    ('result', 'fp'),\n    ('size', 'small'),\n    ('is_border', False)\n]\n\nshow_images(results.copy(), filters, rows=2, cols=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Noisy targets","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image_ids = results['image_id'].unique()\n\nresults_noisy = []\n\nfor image_id in tqdm(image_ids, total=image_ids.shape[0]):\n    fps = results[(results['image_id'] == image_id) & (results['result'] == 'fp')]\n    fps.reset_index(drop=True, inplace=True)\n\n    fns = results[(results['image_id'] == image_id) & (results['result'] == 'fn')]\n    fns.reset_index(drop=True, inplace=True)\n\n    for fpi, fp in fps.iterrows():\n        \n        for fni, fn in fns.iterrows():\n            \n            if ((fp['pred_x1'] <= fn['gt_x1']) and\n                (fp['pred_y1'] <= fn['gt_y1']) and\n                (fp['pred_x2'] >= fn['gt_x2']) and   \n                (fp['pred_y2'] >= fn['gt_y2'])):\n                \n                # GT inside predicted\n                results_noisy.append(fp.to_dict())\n                results_noisy.append(fn.to_dict())\n            \n            elif ((fp['pred_x1'] >= fn['gt_x1']) and\n                  (fp['pred_y1'] >= fn['gt_y1']) and\n                  (fp['pred_x2'] <= fn['gt_x2']) and   \n                  (fp['pred_y2'] <= fn['gt_y2'])):\n                \n                # PREDICTED inside GT\n                results_noisy.append(fp.to_dict())\n                results_noisy.append(fn.to_dict())\n\n\nresults_noisy = pd.DataFrame(results_noisy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results_noisy.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"noisy_sources = pd.DataFrame(train_df['source'].value_counts().sort_index())\nnoisy_sources['noisy'] = (results_noisy['source'].value_counts() // 2).sort_index().values\nnoisy_sources['p'] = noisy_sources['noisy'] / noisy_sources['source'] * 100\n\nnoisy_sources.sort_values(by='p', ascending=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image_boxes(train_df, results_noisy[results_noisy['image_id'] == '4021d47d4'].copy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image_boxes(train_df, results_noisy[results_noisy['image_id'] == '01397a84c'].copy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image_boxes(train_df, results_noisy[results_noisy['image_id'] == '7b72ea0fb'].copy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}