{
  "id": 170175,
  "title": "Evaluation for EfficientDet?",
  "url": "/competitions/global-wheat-detection/discussion/170175",
  "author_name": "",
  "post_date": "2020-07-26T17:27:41.548466Z",
  "votes": 8,
  "comment_count": 19,
  "views": 0,
  "content": "<p>I am using this great <a href=\"https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning\"><strong>notebook</strong></a> to learn some object detection things generally and <a href=\"https://github.com/rwightman/efficientdet-pytorch\">EfficientDet</a> especially.</p>\n\n<p>I have everything working fine (I hope at least) but I am having trouble making the evaluation for the validation dataset work, i.e. compute mAP at the end of each epoch.</p>\n\n<p>I have some code so I am still trying things and hopefully will make it work. That being said, if you have some working snippets, I would be more than happy to check them. Thanks in advance. :)  </p>",
  "messages": [
    {
      "id": "946629",
      "postDate": "07/26/2020 17:27:41",
      "content": "<p>I am using this great <a href=\"https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning\"><strong>notebook</strong></a> to learn some object detection things generally and <a href=\"https://github.com/rwightman/efficientdet-pytorch\">EfficientDet</a> especially.</p>\n\n<p>I have everything working fine (I hope at least) but I am having trouble making the evaluation for the validation dataset work, i.e. compute mAP at the end of each epoch.</p>\n\n<p>I have some code so I am still trying things and hopefully will make it work. That being said, if you have some working snippets, I would be more than happy to check them. Thanks in advance. :)  </p>",
      "rawMarkdown": "I am using this great [**notebook**](https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning) to learn some object detection things generally and [EfficientDet](https://github.com/rwightman/efficientdet-pytorch) especially.\n\nI have everything working fine (I hope at least) but I am having trouble making the evaluation for the validation dataset work, i.e. compute mAP at the end of each epoch.\n\nI have some code so I am still trying things and hopefully will make it work. That being said, if you have some working snippets, I would be more than happy to check them. Thanks in advance. :)",
      "votes": null
    },
    {
      "id": "946754",
      "postDate": "07/26/2020 19:40:53",
      "content": "<p>Tell exactly what your problem is (or show code examples) because you need different input data to calculate the metric from Peter and the COCO metric. I'm using PytorchLightning and I've been able to befriend the EfficientDet with it.</p>",
      "rawMarkdown": "Tell exactly what your problem is (or show code examples) because you need different input data to calculate the metric from Peter and the COCO metric. I'm using PytorchLightning and I've been able to befriend the EfficientDet with it.",
      "votes": null
    },
    {
      "id": "947434",
      "postDate": "07/27/2020 09:10:38",
      "content": "<p>Yes, I will make a code snippet (or at least some pseudo-code close enough) and share it. Will let you know once it is ready. Thanks in advance for your help!</p>",
      "rawMarkdown": "Yes, I will make a code snippet (or at least some pseudo-code close enough) and share it. Will let you know once it is ready. Thanks in advance for your help!",
      "votes": null
    },
    {
      "id": "947572",
      "postDate": "07/27/2020 10:58:09",
      "content": "<p>How is your EfficientDet performance??</p>",
      "rawMarkdown": "How is your EfficientDet performance??",
      "votes": null
    },
    {
      "id": "948275",
      "postDate": "07/27/2020 19:13:54",
      "content": "<p>Well so far, I have only the loss on the train dataset so worth nothing much :D </p>",
      "rawMarkdown": "Well so far, I have only the loss on the train dataset so worth nothing much :D",
      "votes": null
    },
    {
      "id": "949667",
      "postDate": "07/28/2020 19:34:30",
      "content": "<p>Alright, here are some parts of the code (I am using <a href=\"https://github.com/PyTorchLightning/pytorch-lightning\"><strong>Pytroch Lightning</strong></a>) =&gt;</p>\n\n<p>```</p>\n\n<p>def val_dataloader(self):\n        valid_transforms = get_train_transforms()\n        valid_dataset = WheatDataset(\n            self.valid_df, self.image_dir, transforms=valid_transforms\n        )\n        valid_dataloader = DataLoader(\n            valid_dataset,\n            batch_size=self.batch_size,\n            sampler=SequentialSampler(valid_dataset),\n            pin_memory=False,\n            shuffle=False,\n            collate_fn=collate_fn,\n            num_workers=self.num_workers,\n        )\n        # TODO: Doesn't yet work, investigate...\n        iou_types = [\"bbox\"]\n        coco = convert_to_coco_api(valid_dataset)\n        self.coco_evaluator = CocoEvaluator(coco, iou_types)\n        return valid_dataloader</p>\n\n<p>def validation_step(self, batch, batch_idx):\n        images, targets = batch\n        #&nbsp;targets = [{k: v for k, v in t.items()} for t in targets]\n        print(len(images))\n        # separate losses\n        images = torch.stack(images).float()\n        targets2 = {}\n        targets2[\"bbox\"] = [\n            target[\"boxes\"].float() for target in targets\n        ]  # variable number of instances, so the entire structure can be forced to tensor\n        targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n        targets2[\"image_id\"] = torch.tensor(\n            [target[\"image_id\"] for target in targets]\n        ).float()\n        targets2[\"img_scale\"] = torch.tensor(\n            [target[\"img_scale\"] for target in targets]\n        ).float()\n        targets2[\"img_size\"] = torch.tensor(\n            [(IMG_SIZE, IMG_SIZE) for target in targets]\n        ).float()\n        losses_dict = self.model(images, targets2)\n        import pdb; pdb.set_trace()\n        res = {\n            target[\"image_id\"].item(): output\n            for target, output in zip(targets, losses_dict[\"detections\"])\n        } <br>\n        print(res)\n        # &nbsp;TODO: Doesn't yet work. Check why?\n        self.coco_evaluator.update(res)</p>\n\n<p>```</p>\n\n<p>Notice that <code>CocoEvaluator</code> and <code>convert_to_coco_api</code> come directly from the <a href=\"https://github.com/pytorch/vision\"><strong>torchvision</strong></a> <a href=\"https://github.com/pytorch/vision/tree/master/references/detection\"><strong>object detection</strong></a> folder. I have copy-pasted these since I haven't found any easy way to install them.</p>\n\n<p>Let me know if this is enough to let you help me. If not, I will add more details, i.e. make the code snippet more complete with all the methods and imports. Thanks in advance! \nI am still experimenting so if I ever happen to find the solution, I will share it. </p>",
      "rawMarkdown": "Alright, here are some parts of the code (I am using [**Pytroch Lightning**](https://github.com/PyTorchLightning/pytorch-lightning)) =&gt;\n\n```\n\ndef val_dataloader(self):\n        valid_transforms = get_train_transforms()\n        valid_dataset = WheatDataset(\n            self.valid_df, self.image_dir, transforms=valid_transforms\n        )\n        valid_dataloader = DataLoader(\n            valid_dataset,\n            batch_size=self.batch_size,\n            sampler=SequentialSampler(valid_dataset),\n            pin_memory=False,\n            shuffle=False,\n            collate_fn=collate_fn,\n            num_workers=self.num_workers,\n        )\n        # TODO: Doesn't yet work, investigate...\n        iou_types = [\"bbox\"]\n        coco = convert_to_coco_api(valid_dataset)\n        self.coco_evaluator = CocoEvaluator(coco, iou_types)\n        return valid_dataloader\n\ndef validation_step(self, batch, batch_idx):\n        images, targets = batch\n        #&nbsp;targets = [{k: v for k, v in t.items()} for t in targets]\n        print(len(images))\n        # separate losses\n        images = torch.stack(images).float()\n        targets2 = {}\n        targets2[\"bbox\"] = [\n            target[\"boxes\"].float() for target in targets\n        ]  # variable number of instances, so the entire structure can be forced to tensor\n        targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n        targets2[\"image_id\"] = torch.tensor(\n            [target[\"image_id\"] for target in targets]\n        ).float()\n        targets2[\"img_scale\"] = torch.tensor(\n            [target[\"img_scale\"] for target in targets]\n        ).float()\n        targets2[\"img_size\"] = torch.tensor(\n            [(IMG_SIZE, IMG_SIZE) for target in targets]\n        ).float()\n        losses_dict = self.model(images, targets2)\n        import pdb; pdb.set_trace()\n        res = {\n            target[\"image_id\"].item(): output\n            for target, output in zip(targets, losses_dict[\"detections\"])\n        }  \n        print(res)\n        # &nbsp;TODO: Doesn't yet work. Check why?\n        self.coco_evaluator.update(res)\n\n\n```\n\n\nNotice that `CocoEvaluator` and `convert_to_coco_api` come directly from the [**torchvision**](https://github.com/pytorch/vision) [**object detection**](https://github.com/pytorch/vision/tree/master/references/detection) folder. I have copy-pasted these since I haven't found any easy way to install them.\n\nLet me know if this is enough to let you help me. If not, I will add more details, i.e. make the code snippet more complete with all the methods and imports. Thanks in advance! \nI am still experimenting so if I ever happen to find the solution, I will share it.",
      "votes": null
    },
    {
      "id": "949698",
      "postDate": "07/28/2020 20:13:39",
      "content": "<p>Alright, some progress =&gt; I was able to make the code run for a little bit longer by changing the following in the <code>CocoEvaluator</code> class:  </p>\n\n<p>```\n    def prepare_for_coco_detection(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue</p>\n\n<pre><code>        #&nbsp;Changed the key selection with slicing =&gt; from \"bboxes\" to \":4\"\n        #&nbsp;and so on for the other variables. \n        boxes = prediction[:, :4]\n        #&nbsp;Change x,y, x2,y2 to xywh format.\n        boxes = convert_to_xywh(boxes).tolist()\n        scores = prediction[:, 4].tolist()\n        labels = prediction[:, 5].tolist()\n\n        coco_results.extend(\n            [\n                {\n                    \"image_id\": original_id,\n                    \"category_id\": labels[k],\n                    \"bbox\": box,\n                    \"score\": scores[k],\n                }\n                for k, box in enumerate(boxes)\n            ]\n        )\n    return coco_results\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "Alright, some progress =&gt; I was able to make the code run for a little bit longer by changing the following in the `CocoEvaluator` class:  \n\n```\n    def prepare_for_coco_detection(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue\n\n            #&nbsp;Changed the key selection with slicing =&gt; from \"bboxes\" to \":4\"\n            #&nbsp;and so on for the other variables. \n            boxes = prediction[:, :4]\n            #&nbsp;Change x,y, x2,y2 to xywh format.\n            boxes = convert_to_xywh(boxes).tolist()\n            scores = prediction[:, 4].tolist()\n            labels = prediction[:, 5].tolist()\n\n            coco_results.extend(\n                [\n                    {\n                        \"image_id\": original_id,\n                        \"category_id\": labels[k],\n                        \"bbox\": box,\n                        \"score\": scores[k],\n                    }\n                    for k, box in enumerate(boxes)\n                ]\n            )\n        return coco_results\n```",
      "votes": null
    },
    {
      "id": "950058",
      "postDate": "07/29/2020 06:41:05",
      "content": "<p>That's how I work:\n1. in the <code>__getitem__</code> function of the <code>Dataset</code>class, I use this code to correctly prepare the boxes for train and validation:\n<code>target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]] # yxyx: be warning (EFFDET)</code>\n2. in the <code>training_step</code>, I don't change anything.\n3. my <code>validation_step</code> looks like this:</p>\n\n<p>```\ndef validation_step(self, batch, batch_idx):\n        images, targets, image_ids = batch\n        targets = [{k: v for k, v in t.items()} for t in targets]</p>\n\n<pre><code>    # EFFDET\n    if self.model_type == DetBenchTrain:\n        images = torch.stack(images)\n        image_size = images.shape[-1]\n\n        target = {\n            'bbox': [target['boxes'].float() for target in targets],\n            'cls': [target['labels'].float() for target in targets],\n            'img_scale': torch.tensor([[1] for target in targets],\n                                      dtype=torch.float,\n                                      device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n            'img_size': torch.tensor([(image_size, image_size) for target in targets],\n                                      dtype=torch.float,\n                                      device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n        }\n        outputs = self.model(images, target)\n        # total loss\n        loss_val = outputs['loss']\n\n        outputs = outputs['detections']\n\n        # reconverting boxes\n        outputs[:, :, [1,0,3,2]] = outputs[:, :, [0,1,2,3]]\n\n        # xywh to xyxy\n        outputs[:, :, 2] += outputs[:, :, 0]\n        outputs[:, :, 3] += outputs[:, :, 1]\n\n        res = {target[\"image_id\"].item(): {\n                    'boxes': output[:, 0:4],\n                    'scores': output[:, 4],\n                    'labels': output[:, 5]}\n               for target, output in zip(targets, outputs)}\n\n        iou = self._calculate_iou(target, res, image_size)\n        iou = torch.as_tensor(iou)\n\n        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning\n        if self.distributed_backend:\n            loss_val = loss_val.unsqueeze(0)\n            iou = iou.unsqueeze(0)\n\n    else:\n        ...\n\n    self.coco_evaluator.update(res)\n\n    return {'val_loss': loss_val,\n            'main_score': iou}\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "That's how I work:\n1. in the `__getitem__` function of the `Dataset `class, I use this code to correctly prepare the boxes for train and validation:\n`target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]] # yxyx: be warning (EFFDET)`\n2. in the `training_step`, I don't change anything.\n3. my `validation_step` looks like this:\n\n```\ndef validation_step(self, batch, batch_idx):\n        images, targets, image_ids = batch\n        targets = [{k: v for k, v in t.items()} for t in targets]\n        \n        # EFFDET\n        if self.model_type == DetBenchTrain:\n            images = torch.stack(images)\n            image_size = images.shape[-1]\n            \n            target = {\n                'bbox': [target['boxes'].float() for target in targets],\n                'cls': [target['labels'].float() for target in targets],\n                'img_scale': torch.tensor([[1] for target in targets],\n                                          dtype=torch.float,\n                                          device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n                'img_size': torch.tensor([(image_size, image_size) for target in targets],\n                                          dtype=torch.float,\n                                          device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n            }\n            outputs = self.model(images, target)\n            # total loss\n            loss_val = outputs['loss']\n            \n            outputs = outputs['detections']\n            \n            # reconverting boxes\n            outputs[:, :, [1,0,3,2]] = outputs[:, :, [0,1,2,3]]\n            \n            # xywh to xyxy\n            outputs[:, :, 2] += outputs[:, :, 0]\n            outputs[:, :, 3] += outputs[:, :, 1]\n            \n            res = {target[\"image_id\"].item(): {\n                        'boxes': output[:, 0:4],\n                        'scores': output[:, 4],\n                        'labels': output[:, 5]}\n                   for target, output in zip(targets, outputs)}\n            \n            iou = self._calculate_iou(target, res, image_size)\n            iou = torch.as_tensor(iou)\n            \n            # in DP mode (default) make sure if result is scalar, there's another dim in the beginning\n            if self.distributed_backend:\n                loss_val = loss_val.unsqueeze(0)\n                iou = iou.unsqueeze(0)\n            \n        else:\n            ...\n        \n        self.coco_evaluator.update(res)\n        \n        return {'val_loss': loss_val,\n                'main_score': iou}\n```",
      "votes": null
    },
    {
      "id": "950059",
      "postDate": "07/29/2020 06:42:26",
      "content": "<p>I don't change COCO code and use <a href=\"/artgor\">@artgor</a> approach from this <a href=\"https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning\">notebook</a></p>",
      "rawMarkdown": "I don't change COCO code and use @artgor approach from this [notebook](https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning)",
      "votes": null
    },
    {
      "id": "950675",
      "postDate": "07/29/2020 14:32:13",
      "content": "<p>Awesome! Will give a try to your code! I have figured out most of what you did but not this part yet:\n```</p>\n\n<pre><code>        if self.distributed_backend:\n            loss_val = loss_val.unsqueeze(0)\n            iou = iou.unsqueeze(0)\n</code></pre>\n\n<p>```</p>\n\n<p>Thanks for sharing!</p>",
      "rawMarkdown": "Awesome! Will give a try to your code! I have figured out most of what you did but not this part yet:\n```\n\n            if self.distributed_backend:\n                loss_val = loss_val.unsqueeze(0)\n                iou = iou.unsqueeze(0)\n```\n\nThanks for sharing!",
      "votes": null
    },
    {
      "id": "950688",
      "postDate": "07/29/2020 14:48:26",
      "content": "<p>If you only use one gpu for training, that part won't work. If you use more than one gpu, it is necessary for the correct calculation of loss and metrics. You can see an example <a href=\"https://github.com/PyTorchLightning/pytorch-lightning/blob/0.7.0/pl_examples/full_examples/imagenet/imagenet_example.py\">here</a></p>",
      "rawMarkdown": "If you only use one gpu for training, that part won't work. If you use more than one gpu, it is necessary for the correct calculation of loss and metrics. You can see an example [here](https://github.com/PyTorchLightning/pytorch-lightning/blob/0.7.0/pl_examples/full_examples/imagenet/imagenet_example.py)",
      "votes": null
    },
    {
      "id": "952112",
      "postDate": "07/30/2020 16:21:41",
      "content": "<p>Indeed! And thanks for the ImageNet example. Next step: buy more GPUs so that I can run it. 😄 </p>",
      "rawMarkdown": "Indeed! And thanks for the ImageNet example. Next step: buy more GPUs so that I can run it. 😄",
      "votes": null
    },
    {
      "id": "952358",
      "postDate": "07/30/2020 20:59:40",
      "content": "<p>Also, I think this should be (with an s):</p>\n\n<p><code>iou = self._calculate_iou(targets, res, image_size)</code></p>\n\n<p>One last question, were do you define the <code>self._calculate_iou</code> method?</p>",
      "rawMarkdown": "Also, I think this should be (with an s):\n\n`iou = self._calculate_iou(targets, res, image_size)`\n\nOne last question, were do you define the `self._calculate_iou` method?",
      "votes": null
    },
    {
      "id": "953338",
      "postDate": "07/31/2020 17:47:32",
      "content": "<p>Sorry for the late answer</p>\n\n<p>```\ndef _calculate_iou(self, targets, predictions, image_size, th=0.50):\n        all_predictions = []\n        for gt_boxes, (image_id, preds) in zip(targets['bbox'], predictions.items()):\n            max_clip_value = self.data[\"original_image_size\"] - 1\n            img_size_coeff = self.data[\"original_image_size\"] / image_size</p>\n\n<pre><code>        all_predictions.append({\n            'image_id': image_id,\n            'gt_boxes': (gt_boxes.cpu().numpy()*img_size_coeff) \\\n                            .clip(min=0, max=max_clip_value).astype(int),\n            'pred_boxes': (preds['boxes'].detach().cpu().numpy()*img_size_coeff) \\\n                            .clip(min=0, max=max_clip_value).astype(int),\n            'scores': preds['scores'].detach().cpu().numpy(),\n        })\n\n    return calculate_mean_iou(all_predictions, th)\n</code></pre>\n\n<p>```</p>\n\n<p>```\ndef calculate_mean_iou(all_predictions, score_threshold):\n    final_scores = []\n    for i in range(len(all_predictions)):\n        image_id = all_predictions[i]['image_id']\n        gt_boxes = all_predictions[i]['gt_boxes'].copy()\n        pred_boxes = all_predictions[i]['pred_boxes'].copy()\n        scores = all_predictions[i]['scores'].copy()</p>\n\n<pre><code>    indexes = np.where(scores&gt;score_threshold)\n    pred_boxes = pred_boxes[indexes]\n    scores = scores[indexes]\n\n    image_precision = calculate_image_precision(gt_boxes, pred_boxes,\n                                                thresholds=IOU_THRESHOLDS,\n                                                form='pascal_voc')\n    final_scores.append(image_precision)\n\nreturn np.mean(final_scores)\n</code></pre>\n\n<p>```</p>\n\n<p>P.s.: other functions from Peter notebook</p>",
      "rawMarkdown": "Sorry for the late answer\n\n```\ndef _calculate_iou(self, targets, predictions, image_size, th=0.50):\n        all_predictions = []\n        for gt_boxes, (image_id, preds) in zip(targets['bbox'], predictions.items()):\n            max_clip_value = self.data[\"original_image_size\"] - 1\n            img_size_coeff = self.data[\"original_image_size\"] / image_size\n            \n            all_predictions.append({\n                'image_id': image_id,\n                'gt_boxes': (gt_boxes.cpu().numpy()*img_size_coeff) \\\n                                .clip(min=0, max=max_clip_value).astype(int),\n                'pred_boxes': (preds['boxes'].detach().cpu().numpy()*img_size_coeff) \\\n                                .clip(min=0, max=max_clip_value).astype(int),\n                'scores': preds['scores'].detach().cpu().numpy(),\n            })\n            \n        return calculate_mean_iou(all_predictions, th)\n```\n\n```\ndef calculate_mean_iou(all_predictions, score_threshold):\n    final_scores = []\n    for i in range(len(all_predictions)):\n        image_id = all_predictions[i]['image_id']\n        gt_boxes = all_predictions[i]['gt_boxes'].copy()\n        pred_boxes = all_predictions[i]['pred_boxes'].copy()\n        scores = all_predictions[i]['scores'].copy()\n\n        indexes = np.where(scores&gt;score_threshold)\n        pred_boxes = pred_boxes[indexes]\n        scores = scores[indexes]\n\n        image_precision = calculate_image_precision(gt_boxes, pred_boxes,\n                                                    thresholds=IOU_THRESHOLDS,\n                                                    form='pascal_voc')\n        final_scores.append(image_precision)\n\n    return np.mean(final_scores)\n```\n\nP.s.: other functions from Peter notebook",
      "votes": null
    },
    {
      "id": "953400",
      "postDate": "07/31/2020 18:53:46",
      "content": "<p>Thanks a lot for these, much appreciated. 👍 </p>",
      "rawMarkdown": "Thanks a lot for these, much appreciated. 👍",
      "votes": null
    },
    {
      "id": "953764",
      "postDate": "08/01/2020 04:38:24",
      "content": "<p>Good luck! 😊 </p>",
      "rawMarkdown": "Good luck! 😊",
      "votes": null
    },
    {
      "id": "954485",
      "postDate": "08/01/2020 19:13:30",
      "content": "<p>Thanks! I am working on a tutorial notebook, so you will have a special thumbsup. Stay tuned! Hopefully, I will release the notebook in two to three weeks max. :)</p>",
      "rawMarkdown": "Thanks! I am working on a tutorial notebook, so you will have a special thumbsup. Stay tuned! Hopefully, I will release the notebook in two to three weeks max. :)",
      "votes": null
    },
    {
      "id": "954496",
      "postDate": "08/01/2020 19:21:54",
      "content": "<p>Thank you! Tag me when notebook will be released :)</p>",
      "rawMarkdown": "Thank you! Tag me when notebook will be released :)",
      "votes": null
    },
    {
      "id": "1085244",
      "postDate": "11/20/2020 19:22:25",
      "content": "<p><a href=\"https://www.kaggle.com/mnikita\" target=\"_blank\">@mnikita</a> Your calculate_iou and mean_iou are really so helpful. Could you tell anyway if the mean_iou could be integrated to box_targets and box_outputs in the training process, like while the training gt boxes and training targets are input to the train loader?</p>\n<p>It would be of huge help.<br>\nRight now, I on't understand if how can I take bbox targets from all the levels of the BiFPN.<br>\nI hope I was deliverable?</p>\n<p>Thanks much</p>",
      "rawMarkdown": "mnikita Your calculate_iou and mean_iou are really so helpful. Could you tell anyway if the mean_iou could be integrated to box_targets and box_outputs in the training process, like while the training gt boxes and training targets are input to the train loader?\n\nIt would be of huge help.\nRight now, I on't understand if how can I take bbox targets from all the levels of the BiFPN.\nI hope I was deliverable?\n\nThanks much",
      "votes": null
    },
    {
      "id": "1085249",
      "postDate": "11/20/2020 19:29:31",
      "content": "<p><a href=\"https://www.kaggle.com/mnikita\" target=\"_blank\">@mnikita</a> Also, if you could tell where did you refer the function calculate_image_precision in the calculate_mean_iou function?</p>\n<p>Thanks again!</p>",
      "rawMarkdown": "mnikita Also, if you could tell where did you refer the function calculate_image_precision in the calculate_mean_iou function?\n\nThanks again!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 946754,
      "author_name": "mnikita",
      "author_url": "",
      "post_date": "07/26/2020 19:40:53",
      "content": "<p>Tell exactly what your problem is (or show code examples) because you need different input data to calculate the metric from Peter and the COCO metric. I'm using PytorchLightning and I've been able to befriend the EfficientDet with it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 947434,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/27/2020 09:10:38",
          "content": "<p>Yes, I will make a code snippet (or at least some pseudo-code close enough) and share it. Will let you know once it is ready. Thanks in advance for your help!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 949667,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/28/2020 19:34:30",
          "content": "<p>Alright, here are some parts of the code (I am using <a href=\"https://github.com/PyTorchLightning/pytorch-lightning\"><strong>Pytroch Lightning</strong></a>) =&gt;</p>\n\n<p>```</p>\n\n<p>def val_dataloader(self):\n        valid_transforms = get_train_transforms()\n        valid_dataset = WheatDataset(\n            self.valid_df, self.image_dir, transforms=valid_transforms\n        )\n        valid_dataloader = DataLoader(\n            valid_dataset,\n            batch_size=self.batch_size,\n            sampler=SequentialSampler(valid_dataset),\n            pin_memory=False,\n            shuffle=False,\n            collate_fn=collate_fn,\n            num_workers=self.num_workers,\n        )\n        # TODO: Doesn't yet work, investigate...\n        iou_types = [\"bbox\"]\n        coco = convert_to_coco_api(valid_dataset)\n        self.coco_evaluator = CocoEvaluator(coco, iou_types)\n        return valid_dataloader</p>\n\n<p>def validation_step(self, batch, batch_idx):\n        images, targets = batch\n        #&nbsp;targets = [{k: v for k, v in t.items()} for t in targets]\n        print(len(images))\n        # separate losses\n        images = torch.stack(images).float()\n        targets2 = {}\n        targets2[\"bbox\"] = [\n            target[\"boxes\"].float() for target in targets\n        ]  # variable number of instances, so the entire structure can be forced to tensor\n        targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n        targets2[\"image_id\"] = torch.tensor(\n            [target[\"image_id\"] for target in targets]\n        ).float()\n        targets2[\"img_scale\"] = torch.tensor(\n            [target[\"img_scale\"] for target in targets]\n        ).float()\n        targets2[\"img_size\"] = torch.tensor(\n            [(IMG_SIZE, IMG_SIZE) for target in targets]\n        ).float()\n        losses_dict = self.model(images, targets2)\n        import pdb; pdb.set_trace()\n        res = {\n            target[\"image_id\"].item(): output\n            for target, output in zip(targets, losses_dict[\"detections\"])\n        } <br>\n        print(res)\n        # &nbsp;TODO: Doesn't yet work. Check why?\n        self.coco_evaluator.update(res)</p>\n\n<p>```</p>\n\n<p>Notice that <code>CocoEvaluator</code> and <code>convert_to_coco_api</code> come directly from the <a href=\"https://github.com/pytorch/vision\"><strong>torchvision</strong></a> <a href=\"https://github.com/pytorch/vision/tree/master/references/detection\"><strong>object detection</strong></a> folder. I have copy-pasted these since I haven't found any easy way to install them.</p>\n\n<p>Let me know if this is enough to let you help me. If not, I will add more details, i.e. make the code snippet more complete with all the methods and imports. Thanks in advance! \nI am still experimenting so if I ever happen to find the solution, I will share it. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 949698,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/28/2020 20:13:39",
          "content": "<p>Alright, some progress =&gt; I was able to make the code run for a little bit longer by changing the following in the <code>CocoEvaluator</code> class:  </p>\n\n<p>```\n    def prepare_for_coco_detection(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue</p>\n\n<pre><code>        #&nbsp;Changed the key selection with slicing =&gt; from \"bboxes\" to \":4\"\n        #&nbsp;and so on for the other variables. \n        boxes = prediction[:, :4]\n        #&nbsp;Change x,y, x2,y2 to xywh format.\n        boxes = convert_to_xywh(boxes).tolist()\n        scores = prediction[:, 4].tolist()\n        labels = prediction[:, 5].tolist()\n\n        coco_results.extend(\n            [\n                {\n                    \"image_id\": original_id,\n                    \"category_id\": labels[k],\n                    \"bbox\": box,\n                    \"score\": scores[k],\n                }\n                for k, box in enumerate(boxes)\n            ]\n        )\n    return coco_results\n</code></pre>\n\n<p>```</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 950058,
          "author_name": "mnikita",
          "author_url": "",
          "post_date": "07/29/2020 06:41:05",
          "content": "<p>That's how I work:\n1. in the <code>__getitem__</code> function of the <code>Dataset</code>class, I use this code to correctly prepare the boxes for train and validation:\n<code>target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]] # yxyx: be warning (EFFDET)</code>\n2. in the <code>training_step</code>, I don't change anything.\n3. my <code>validation_step</code> looks like this:</p>\n\n<p>```\ndef validation_step(self, batch, batch_idx):\n        images, targets, image_ids = batch\n        targets = [{k: v for k, v in t.items()} for t in targets]</p>\n\n<pre><code>    # EFFDET\n    if self.model_type == DetBenchTrain:\n        images = torch.stack(images)\n        image_size = images.shape[-1]\n\n        target = {\n            'bbox': [target['boxes'].float() for target in targets],\n            'cls': [target['labels'].float() for target in targets],\n            'img_scale': torch.tensor([[1] for target in targets],\n                                      dtype=torch.float,\n                                      device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n            'img_size': torch.tensor([(image_size, image_size) for target in targets],\n                                      dtype=torch.float,\n                                      device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n        }\n        outputs = self.model(images, target)\n        # total loss\n        loss_val = outputs['loss']\n\n        outputs = outputs['detections']\n\n        # reconverting boxes\n        outputs[:, :, [1,0,3,2]] = outputs[:, :, [0,1,2,3]]\n\n        # xywh to xyxy\n        outputs[:, :, 2] += outputs[:, :, 0]\n        outputs[:, :, 3] += outputs[:, :, 1]\n\n        res = {target[\"image_id\"].item(): {\n                    'boxes': output[:, 0:4],\n                    'scores': output[:, 4],\n                    'labels': output[:, 5]}\n               for target, output in zip(targets, outputs)}\n\n        iou = self._calculate_iou(target, res, image_size)\n        iou = torch.as_tensor(iou)\n\n        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning\n        if self.distributed_backend:\n            loss_val = loss_val.unsqueeze(0)\n            iou = iou.unsqueeze(0)\n\n    else:\n        ...\n\n    self.coco_evaluator.update(res)\n\n    return {'val_loss': loss_val,\n            'main_score': iou}\n</code></pre>\n\n<p>```</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 950059,
          "author_name": "mnikita",
          "author_url": "",
          "post_date": "07/29/2020 06:42:26",
          "content": "<p>I don't change COCO code and use <a href=\"/artgor\">@artgor</a> approach from this <a href=\"https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning\">notebook</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 950675,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/29/2020 14:32:13",
          "content": "<p>Awesome! Will give a try to your code! I have figured out most of what you did but not this part yet:\n```</p>\n\n<pre><code>        if self.distributed_backend:\n            loss_val = loss_val.unsqueeze(0)\n            iou = iou.unsqueeze(0)\n</code></pre>\n\n<p>```</p>\n\n<p>Thanks for sharing!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 950688,
          "author_name": "mnikita",
          "author_url": "",
          "post_date": "07/29/2020 14:48:26",
          "content": "<p>If you only use one gpu for training, that part won't work. If you use more than one gpu, it is necessary for the correct calculation of loss and metrics. You can see an example <a href=\"https://github.com/PyTorchLightning/pytorch-lightning/blob/0.7.0/pl_examples/full_examples/imagenet/imagenet_example.py\">here</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 952112,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/30/2020 16:21:41",
          "content": "<p>Indeed! And thanks for the ImageNet example. Next step: buy more GPUs so that I can run it. 😄 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 952358,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/30/2020 20:59:40",
          "content": "<p>Also, I think this should be (with an s):</p>\n\n<p><code>iou = self._calculate_iou(targets, res, image_size)</code></p>\n\n<p>One last question, were do you define the <code>self._calculate_iou</code> method?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 953338,
          "author_name": "mnikita",
          "author_url": "",
          "post_date": "07/31/2020 17:47:32",
          "content": "<p>Sorry for the late answer</p>\n\n<p>```\ndef _calculate_iou(self, targets, predictions, image_size, th=0.50):\n        all_predictions = []\n        for gt_boxes, (image_id, preds) in zip(targets['bbox'], predictions.items()):\n            max_clip_value = self.data[\"original_image_size\"] - 1\n            img_size_coeff = self.data[\"original_image_size\"] / image_size</p>\n\n<pre><code>        all_predictions.append({\n            'image_id': image_id,\n            'gt_boxes': (gt_boxes.cpu().numpy()*img_size_coeff) \\\n                            .clip(min=0, max=max_clip_value).astype(int),\n            'pred_boxes': (preds['boxes'].detach().cpu().numpy()*img_size_coeff) \\\n                            .clip(min=0, max=max_clip_value).astype(int),\n            'scores': preds['scores'].detach().cpu().numpy(),\n        })\n\n    return calculate_mean_iou(all_predictions, th)\n</code></pre>\n\n<p>```</p>\n\n<p>```\ndef calculate_mean_iou(all_predictions, score_threshold):\n    final_scores = []\n    for i in range(len(all_predictions)):\n        image_id = all_predictions[i]['image_id']\n        gt_boxes = all_predictions[i]['gt_boxes'].copy()\n        pred_boxes = all_predictions[i]['pred_boxes'].copy()\n        scores = all_predictions[i]['scores'].copy()</p>\n\n<pre><code>    indexes = np.where(scores&gt;score_threshold)\n    pred_boxes = pred_boxes[indexes]\n    scores = scores[indexes]\n\n    image_precision = calculate_image_precision(gt_boxes, pred_boxes,\n                                                thresholds=IOU_THRESHOLDS,\n                                                form='pascal_voc')\n    final_scores.append(image_precision)\n\nreturn np.mean(final_scores)\n</code></pre>\n\n<p>```</p>\n\n<p>P.s.: other functions from Peter notebook</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 953400,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/31/2020 18:53:46",
          "content": "<p>Thanks a lot for these, much appreciated. 👍 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 953764,
          "author_name": "mnikita",
          "author_url": "",
          "post_date": "08/01/2020 04:38:24",
          "content": "<p>Good luck! 😊 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 954485,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "08/01/2020 19:13:30",
          "content": "<p>Thanks! I am working on a tutorial notebook, so you will have a special thumbsup. Stay tuned! Hopefully, I will release the notebook in two to three weeks max. :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 954496,
          "author_name": "mnikita",
          "author_url": "",
          "post_date": "08/01/2020 19:21:54",
          "content": "<p>Thank you! Tag me when notebook will be released :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1085244,
          "author_name": "ebhojwan",
          "author_url": "",
          "post_date": "11/20/2020 19:22:25",
          "content": "<p><a href=\"https://www.kaggle.com/mnikita\" target=\"_blank\">@mnikita</a> Your calculate_iou and mean_iou are really so helpful. Could you tell anyway if the mean_iou could be integrated to box_targets and box_outputs in the training process, like while the training gt boxes and training targets are input to the train loader?</p>\n<p>It would be of huge help.<br>\nRight now, I on't understand if how can I take bbox targets from all the levels of the BiFPN.<br>\nI hope I was deliverable?</p>\n<p>Thanks much</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1085249,
          "author_name": "ebhojwan",
          "author_url": "",
          "post_date": "11/20/2020 19:29:31",
          "content": "<p><a href=\"https://www.kaggle.com/mnikita\" target=\"_blank\">@mnikita</a> Also, if you could tell where did you refer the function calculate_image_precision in the calculate_mean_iou function?</p>\n<p>Thanks again!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 947572,
      "author_name": "xiyuanliu",
      "author_url": "",
      "post_date": "07/27/2020 10:58:09",
      "content": "<p>How is your EfficientDet performance??</p>",
      "votes": null,
      "replies": [
        {
          "id": 948275,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "07/27/2020 19:13:54",
          "content": "<p>Well so far, I have only the loss on the train dataset so worth nothing much :D </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "946629": "I am using this great [**notebook**](https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning) to learn some object detection things generally and [EfficientDet](https://github.com/rwightman/efficientdet-pytorch) especially.\n\nI have everything working fine (I hope at least) but I am having trouble making the evaluation for the validation dataset work, i.e. compute mAP at the end of each epoch.\n\nI have some code so I am still trying things and hopefully will make it work. That being said, if you have some working snippets, I would be more than happy to check them. Thanks in advance. :)",
    "946754": "Tell exactly what your problem is (or show code examples) because you need different input data to calculate the metric from Peter and the COCO metric. I'm using PytorchLightning and I've been able to befriend the EfficientDet with it.",
    "947434": "Yes, I will make a code snippet (or at least some pseudo-code close enough) and share it. Will let you know once it is ready. Thanks in advance for your help!",
    "947572": "How is your EfficientDet performance??",
    "948275": "Well so far, I have only the loss on the train dataset so worth nothing much :D",
    "949667": "Alright, here are some parts of the code (I am using [**Pytroch Lightning**](https://github.com/PyTorchLightning/pytorch-lightning)) =&gt;\n\n```\n\ndef val_dataloader(self):\n        valid_transforms = get_train_transforms()\n        valid_dataset = WheatDataset(\n            self.valid_df, self.image_dir, transforms=valid_transforms\n        )\n        valid_dataloader = DataLoader(\n            valid_dataset,\n            batch_size=self.batch_size,\n            sampler=SequentialSampler(valid_dataset),\n            pin_memory=False,\n            shuffle=False,\n            collate_fn=collate_fn,\n            num_workers=self.num_workers,\n        )\n        # TODO: Doesn't yet work, investigate...\n        iou_types = [\"bbox\"]\n        coco = convert_to_coco_api(valid_dataset)\n        self.coco_evaluator = CocoEvaluator(coco, iou_types)\n        return valid_dataloader\n\ndef validation_step(self, batch, batch_idx):\n        images, targets = batch\n        #&nbsp;targets = [{k: v for k, v in t.items()} for t in targets]\n        print(len(images))\n        # separate losses\n        images = torch.stack(images).float()\n        targets2 = {}\n        targets2[\"bbox\"] = [\n            target[\"boxes\"].float() for target in targets\n        ]  # variable number of instances, so the entire structure can be forced to tensor\n        targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n        targets2[\"image_id\"] = torch.tensor(\n            [target[\"image_id\"] for target in targets]\n        ).float()\n        targets2[\"img_scale\"] = torch.tensor(\n            [target[\"img_scale\"] for target in targets]\n        ).float()\n        targets2[\"img_size\"] = torch.tensor(\n            [(IMG_SIZE, IMG_SIZE) for target in targets]\n        ).float()\n        losses_dict = self.model(images, targets2)\n        import pdb; pdb.set_trace()\n        res = {\n            target[\"image_id\"].item(): output\n            for target, output in zip(targets, losses_dict[\"detections\"])\n        }  \n        print(res)\n        # &nbsp;TODO: Doesn't yet work. Check why?\n        self.coco_evaluator.update(res)\n\n\n```\n\n\nNotice that `CocoEvaluator` and `convert_to_coco_api` come directly from the [**torchvision**](https://github.com/pytorch/vision) [**object detection**](https://github.com/pytorch/vision/tree/master/references/detection) folder. I have copy-pasted these since I haven't found any easy way to install them.\n\nLet me know if this is enough to let you help me. If not, I will add more details, i.e. make the code snippet more complete with all the methods and imports. Thanks in advance! \nI am still experimenting so if I ever happen to find the solution, I will share it.",
    "949698": "Alright, some progress =&gt; I was able to make the code run for a little bit longer by changing the following in the `CocoEvaluator` class:  \n\n```\n    def prepare_for_coco_detection(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue\n\n            #&nbsp;Changed the key selection with slicing =&gt; from \"bboxes\" to \":4\"\n            #&nbsp;and so on for the other variables. \n            boxes = prediction[:, :4]\n            #&nbsp;Change x,y, x2,y2 to xywh format.\n            boxes = convert_to_xywh(boxes).tolist()\n            scores = prediction[:, 4].tolist()\n            labels = prediction[:, 5].tolist()\n\n            coco_results.extend(\n                [\n                    {\n                        \"image_id\": original_id,\n                        \"category_id\": labels[k],\n                        \"bbox\": box,\n                        \"score\": scores[k],\n                    }\n                    for k, box in enumerate(boxes)\n                ]\n            )\n        return coco_results\n```",
    "950058": "That's how I work:\n1. in the `__getitem__` function of the `Dataset `class, I use this code to correctly prepare the boxes for train and validation:\n`target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]] # yxyx: be warning (EFFDET)`\n2. in the `training_step`, I don't change anything.\n3. my `validation_step` looks like this:\n\n```\ndef validation_step(self, batch, batch_idx):\n        images, targets, image_ids = batch\n        targets = [{k: v for k, v in t.items()} for t in targets]\n        \n        # EFFDET\n        if self.model_type == DetBenchTrain:\n            images = torch.stack(images)\n            image_size = images.shape[-1]\n            \n            target = {\n                'bbox': [target['boxes'].float() for target in targets],\n                'cls': [target['labels'].float() for target in targets],\n                'img_scale': torch.tensor([[1] for target in targets],\n                                          dtype=torch.float,\n                                          device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n                'img_size': torch.tensor([(image_size, image_size) for target in targets],\n                                          dtype=torch.float,\n                                          device=f\"cuda:{self.hparams['trainer']['gpus']}\"),\n            }\n            outputs = self.model(images, target)\n            # total loss\n            loss_val = outputs['loss']\n            \n            outputs = outputs['detections']\n            \n            # reconverting boxes\n            outputs[:, :, [1,0,3,2]] = outputs[:, :, [0,1,2,3]]\n            \n            # xywh to xyxy\n            outputs[:, :, 2] += outputs[:, :, 0]\n            outputs[:, :, 3] += outputs[:, :, 1]\n            \n            res = {target[\"image_id\"].item(): {\n                        'boxes': output[:, 0:4],\n                        'scores': output[:, 4],\n                        'labels': output[:, 5]}\n                   for target, output in zip(targets, outputs)}\n            \n            iou = self._calculate_iou(target, res, image_size)\n            iou = torch.as_tensor(iou)\n            \n            # in DP mode (default) make sure if result is scalar, there's another dim in the beginning\n            if self.distributed_backend:\n                loss_val = loss_val.unsqueeze(0)\n                iou = iou.unsqueeze(0)\n            \n        else:\n            ...\n        \n        self.coco_evaluator.update(res)\n        \n        return {'val_loss': loss_val,\n                'main_score': iou}\n```",
    "950059": "I don't change COCO code and use @artgor approach from this [notebook](https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning)",
    "950675": "Awesome! Will give a try to your code! I have figured out most of what you did but not this part yet:\n```\n\n            if self.distributed_backend:\n                loss_val = loss_val.unsqueeze(0)\n                iou = iou.unsqueeze(0)\n```\n\nThanks for sharing!",
    "950688": "If you only use one gpu for training, that part won't work. If you use more than one gpu, it is necessary for the correct calculation of loss and metrics. You can see an example [here](https://github.com/PyTorchLightning/pytorch-lightning/blob/0.7.0/pl_examples/full_examples/imagenet/imagenet_example.py)",
    "952112": "Indeed! And thanks for the ImageNet example. Next step: buy more GPUs so that I can run it. 😄",
    "952358": "Also, I think this should be (with an s):\n\n`iou = self._calculate_iou(targets, res, image_size)`\n\nOne last question, were do you define the `self._calculate_iou` method?",
    "953338": "Sorry for the late answer\n\n```\ndef _calculate_iou(self, targets, predictions, image_size, th=0.50):\n        all_predictions = []\n        for gt_boxes, (image_id, preds) in zip(targets['bbox'], predictions.items()):\n            max_clip_value = self.data[\"original_image_size\"] - 1\n            img_size_coeff = self.data[\"original_image_size\"] / image_size\n            \n            all_predictions.append({\n                'image_id': image_id,\n                'gt_boxes': (gt_boxes.cpu().numpy()*img_size_coeff) \\\n                                .clip(min=0, max=max_clip_value).astype(int),\n                'pred_boxes': (preds['boxes'].detach().cpu().numpy()*img_size_coeff) \\\n                                .clip(min=0, max=max_clip_value).astype(int),\n                'scores': preds['scores'].detach().cpu().numpy(),\n            })\n            \n        return calculate_mean_iou(all_predictions, th)\n```\n\n```\ndef calculate_mean_iou(all_predictions, score_threshold):\n    final_scores = []\n    for i in range(len(all_predictions)):\n        image_id = all_predictions[i]['image_id']\n        gt_boxes = all_predictions[i]['gt_boxes'].copy()\n        pred_boxes = all_predictions[i]['pred_boxes'].copy()\n        scores = all_predictions[i]['scores'].copy()\n\n        indexes = np.where(scores&gt;score_threshold)\n        pred_boxes = pred_boxes[indexes]\n        scores = scores[indexes]\n\n        image_precision = calculate_image_precision(gt_boxes, pred_boxes,\n                                                    thresholds=IOU_THRESHOLDS,\n                                                    form='pascal_voc')\n        final_scores.append(image_precision)\n\n    return np.mean(final_scores)\n```\n\nP.s.: other functions from Peter notebook",
    "953400": "Thanks a lot for these, much appreciated. 👍",
    "953764": "Good luck! 😊",
    "954485": "Thanks! I am working on a tutorial notebook, so you will have a special thumbsup. Stay tuned! Hopefully, I will release the notebook in two to three weeks max. :)",
    "954496": "Thank you! Tag me when notebook will be released :)",
    "1085244": "mnikita Your calculate_iou and mean_iou are really so helpful. Could you tell anyway if the mean_iou could be integrated to box_targets and box_outputs in the training process, like while the training gt boxes and training targets are input to the train loader?\n\nIt would be of huge help.\nRight now, I on't understand if how can I take bbox targets from all the levels of the BiFPN.\nI hope I was deliverable?\n\nThanks much",
    "1085249": "mnikita Also, if you could tell where did you refer the function calculate_image_precision in the calculate_mean_iou function?\n\nThanks again!"
  },
  "source": "meta"
}