{"cells":[{"metadata":{},"cell_type":"markdown","source":"## How to make stable ensemble in object detection\n\nHi everyone!\n\nMy name is Alex Shonenkov, I am DL/NLP/CV/TS research engineer. Especially I am in Love with NLP & DL.\n\nToday I would like to share with you, my friends, super easy technique for bbox postprocessing and making ensemble in object detection tasks.\n\nI spied this idea from really good russian competitions grandmaster Roman Solovyev [ZFTurbo](https://www.kaggle.com/zfturbo) in [this repository](https://github.com/ZFTurbo/Weighted-Boxes-Fusion) that I used in this kernel as [dataset](https://www.kaggle.com/shonenkov/weightedboxesfusion).\n\noriginal paper: [Weighted Boxes Fusion: ensembling boxes for object detection models](https://arxiv.org/pdf/1910.13302.pdf)\n\n\n![](https://i.ibb.co/681bjVr/2020-05-12-20-43-52.png)\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## MAIN IDEA\n\nWe know about NMS (Non-maximum Suppression) method and\nits [Soft-NMS extension](https://arxiv.org/pdf/1704.04503.pdf). But here we will use WBF (Weighted Boxes Fusion), that gives really good boost in [Open Images Dataset](https://storage.googleapis.com/openimages/web/index.html) according to [paper]((https://arxiv.org/pdf/1910.13302.pdf).\n\n    \n| Method | mAP(0.5) Result | Best params | Elapsed time (sec) | \n| ------ | --------------- | ----------- | ------------ |\n| NMS | 0.5642 | IOU Thr: 0.5 | 47 |\n| Soft-NMS | 0.5616 | Sigma: 0.1, Confidence Thr: 0.001 | 88 |\n| NMW | 0.5667 | IOU Thr: 0.5 | 171 |\n| **WBF** | **0.5982** | IOU Thr: 0.6 | 249 |\n    \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Why WBF can be better than NMS or SoftNMS\n\nBoth NMS and Soft-NMS exclude some boxes, but WBF uses information from all boxes. It can fix some cases where\nall boxes are predicted inaccurate by all models. NMS will leave only one inaccurate box, while WBF will fix it using information from all 3 boxes (see the example in Fig. 1, red predictions, blue ground truth).\n\n<img src='https://i.ibb.co/d2P2pPL/2020-05-12-21-02-34.png' height=300 align=\"left\"> ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install --no-deps '../input/weightedboxesfusion/' > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.utils.data import Dataset,DataLoader\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom glob import glob\nimport torch\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nmarking = pd.read_csv('../input/global-wheat-detection/train.csv')\n\nbboxs = np.stack(marking['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=',')))\nfor i, column in enumerate(['x', 'y', 'w', 'h']):\n    marking[column] = bboxs[:,i]\nmarking.drop(columns=['bbox'], inplace=True)\n\nmarking.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"marking['source'].hist(bins=15);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stratify KFold\n\nI think that main important factor for getting stable ensemble solution is using cross-validation with correct data splitting. It will help us to avoid overfitting solution. \n\nBelow I would like to share with you my data split. I can't say that it is really correct way or not, but I believe it :)\n\n- Stratify by source\n- Stratify by count of bounding boxes\n\nI would like to see stratified distribution on 5 folds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nskf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\ndf_folds = marking[['image_id']].copy()\ndf_folds.loc[:, 'bbox_count'] = 1\ndf_folds = df_folds.groupby('image_id').count()\ndf_folds.loc[:, 'source'] = marking[['image_id', 'source']].groupby('image_id').min()['source']\ndf_folds.loc[:, 'stratify_group'] = np.char.add(\n    df_folds['source'].values.astype(str),\n    df_folds['bbox_count'].apply(lambda x: f'_{x // 15}').values.astype(str)\n)\ndf_folds.loc[:, 'fold'] = 0\n\nfor fold_number, (train_index, val_index) in enumerate(skf.split(X=df_folds.index, y=df_folds['stratify_group'])):\n    df_folds.loc[df_folds.iloc[val_index].index, 'fold'] = fold_number","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_folds.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare Dataset\n\nFor training folds for this kernel I have used [Pytorch Starter - FasterRCNN Train](https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train) with 512x512 image size.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_valid_transforms():\n    return A.Compose([\n            A.Resize(height=512, width=512, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_ROOT_PATH = '../input/global-wheat-detection/test'\n\nclass DatasetRetriever(Dataset):\n\n    def __init__(self, image_ids, transforms=None):\n        super().__init__()\n        self.image_ids = image_ids\n        self.transforms = transforms\n\n    def __getitem__(self, index: int):\n        image_id = self.image_ids[index]\n        image = cv2.imread(f'{DATA_ROOT_PATH}/{image_id}.jpg', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        if self.transforms:\n            sample = {'image': image}\n            sample = self.transforms(**sample)\n            image = sample['image']\n        return image, image_id\n\n    def __len__(self) -> int:\n        return self.image_ids.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = DatasetRetriever(\n    image_ids=np.array([path.split('/')[-1][:-4] for path in glob(f'{DATA_ROOT_PATH}/*.jpg')]),\n    transforms=get_valid_transforms()\n)\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\ndata_loader = DataLoader(\n    dataset,\n    batch_size=2,\n    shuffle=False,\n    num_workers=4,\n    drop_last=False,\n    collate_fn=collate_fn\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Pretrained Models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nimport torchvision\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection import FasterRCNN\nfrom torchvision.models.detection.rpn import AnchorGenerator\n\n\ndef load_net(checkpoint_path):\n    net = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)\n    num_classes = 2  # 1 class (wheat) + background\n    # get number of input features for the classifier\n    in_features = net.roi_heads.box_predictor.cls_score.in_features\n    # replace the pre-trained head with a new one\n    net.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\n\n    checkpoint = torch.load(checkpoint_path)\n    net.load_state_dict(checkpoint)\n    net = net.cuda()\n    net.eval()\n\n    del checkpoint\n    gc.collect()\n    return net\n\nmodels = [\n    load_net('../input/wheat-fasterrcnn-folds/fold0-best1.bin'),\n    load_net('../input/wheat-fasterrcnn-folds/fold1-best1.bin'),\n    load_net('../input/wheat-fasterrcnn-folds/fold2-best1.bin'),\n    load_net('../input/wheat-fasterrcnn-folds/fold3-best1.bin'),\n    load_net('../input/wheat-fasterrcnn-folds/fold4-best1.bin'),\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## WBF & Ensemble methods ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from ensemble_boxes import *\n\ndevice = torch.device('cuda:0')\n\ndef make_ensemble_predictions(images):\n    images = list(image.to(device) for image in images)    \n    result = []\n    for net in models:\n        outputs = net(images)\n        result.append(outputs)\n    return result\n\ndef run_wbf(predictions, image_index, image_size=512, iou_thr=0.55, skip_box_thr=0.7, weights=None):\n    boxes = [prediction[image_index]['boxes'].data.cpu().numpy()/(image_size-1) for prediction in predictions]\n    scores = [prediction[image_index]['scores'].data.cpu().numpy() for prediction in predictions]\n    labels = [np.ones(prediction[image_index]['scores'].shape[0]) for prediction in predictions]\n    boxes, scores, labels = weighted_boxes_fusion(boxes, scores, labels, weights=None, iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n    boxes = boxes*(image_size-1)\n    return boxes, scores, labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nfor j, (images, image_ids) in enumerate(data_loader):\n    if j > 0:\n        break\npredictions = make_ensemble_predictions(images)\n\ni = 1\nsample = images[i].permute(1,2,0).cpu().numpy()\nboxes, scores, labels = run_wbf(predictions, image_index=i)\nboxes = boxes.astype(np.int32).clip(min=0, max=511)\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nfor box in boxes:\n    cv2.rectangle(sample,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (220, 0, 0), 2)\n    \nax.set_axis_off()\nax.imshow(sample);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def format_prediction_string(boxes, scores):\n    pred_strings = []\n    for j in zip(scores, boxes):\n        pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], j[1][2], j[1][3]))\n    return \" \".join(pred_strings)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = []\n\nfor images, image_ids in data_loader:\n    predictions = make_ensemble_predictions(images)\n    for i, image in enumerate(images):\n        boxes, scores, labels = run_wbf(predictions, image_index=i)\n        boxes = (boxes*2).astype(np.int32).clip(min=0, max=1023)\n        image_id = image_ids[i]\n\n        boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n        boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n        \n        result = {\n            'image_id': image_id,\n            'PredictionString': format_prediction_string(boxes, scores)\n        }\n        results.append(result)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame(results, columns=['image_id', 'PredictionString'])\ntest_df.to_csv('submission.csv', index=False)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Thank you for reading my kernel!\n\nSo, I have demonstrated good technique for you, my friends! I hope you will make stable ensemble in this competition! :)\n\n\n\nJust recently I have started publishing my works, if you like this format of notebooks I would like continue to make kernels.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}