{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\nThis notebook aims to provide a working training script for the updated version of Effdet created by [@rwrightman](https://www.kaggle.com/rwightman). I realised the training code released by [@shonenkov](https://www.kaggle.com/shonenkov) does not work for the updated Effdet, hence, i made some changes accordingly.\n\n#### Main Changes:\n* **DetBenchTrain** forward() now takes in a dictionary object with key:value pair {bbox:, cls:, img_size:, img_scale:} as argument\n* **DetBenchPredict** forward() takes in (image, img_size, img_scale) as argumemnt\n\n(Im not sure what is the significance of img_scale and img_size, would appreciate if anyone can advise on this)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Credits\nFirst of all, i would like to thank and give credits to these great individuals who made it possible for beginners like me to implement state-of-the-art vision models for this competition.\n1. [@rwrightman](https://www.kaggle.com/rwightman) for his wonderful work in providing an a high level interface for us to easily implement pre-trained EfficientDet in Pytorch. Please Like his work [here](https://github.com/rwightman/efficientdet-pytorch) if you have also benefited from his amazing work :)\n2. [@shonenkov](https://www.kaggle.com/shonenkov). Most of the code below are from this amazing researcher which you can find [here](https://www.kaggle.com/shonenkov/training-efficientdet). The way he abstracted the different functions, with clean codes, really provides an easy way to understand the whole training pipeline. What i did from there is merely adjusting to the updated EffDet version and also minor changes. I have included some comments to better explain what some parts of the code is trying to do. Really learnt alot from his notebook!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Why EfficientDet?\nIt will be quite sad if, amidst the pursue of higher LB, we blindly pluck in any state-of-the-art model without making any effort to understand what makes the model stand out from the rest (i'm guilty of that). I guess the basic form of recognition we can give to the researchers who came up with these ideas is not only to implement it, but also appreciate how their novel techniques and ideas result in state-of-the-art performance.  \nIn this section, i would like to briefly go through the workings of [**EfficientDet**](https://arxiv.org/abs/1911.09070), starting from the backbone CNN, **EfficientNet**.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### EfficientNet\nWe have seen that over the past few years, alot(not all) of the models that hit the state-of-the-art performance are those with lots of parameters, reflecting the trade-off between model accuracy and model size. In simple terms, bigger model = better performance. However, there are still difficulties in coming up with better network design; How to increase the model size to provide better accuracy? There are a few ways by which we can scale up the networks:\n1. Increasing model depth\n2. Increasing model width\n3. Increasing resolution (image size)\n\nIn order to have higher accuracy and also efficiency, there is a need in finding ways to balance these options during the network scaling. This is where EfficientNet comes in, aiming to solve this difficult task of designing networks.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The authors propose a method called **Compound Scaling Method** that uses a compound coefficient $ \\phi $ to scale the width, depth and resolution of a network in a principed manner. \n<img src='https://miro.medium.com/max/700/1*lGcFaOK6lW5hHskGyINPgw.png'>\n\nGiven by the formula:\n<img src='https://www.learnopencv.com/wp-content/uploads/2019/06/compound-scaling-efficient-net.png' width=300, height=250>\nwhere $ \\phi $ is a user-defined coefficient that controls the resources (FLOPS) available for scaling\n$\\alpha, \\beta, \\gamma$ distribute resources to depth, width and resolution respectively.  \n\nUsing EfficientNet-B0 as the base, they come up with the following strategy:\n1. Fix $ \\phi$ = 1\n2. Grid search over $\\alpha, \\beta, \\gamma $ \n3. The authors found $ \\alpha = 1.2, \\beta = 1.1, \\gamma = 1.15$ as optimal\n4. Fixing the values of $\\alpha, \\beta, \\gamma $, scale up EfficientNet-B0 with different $\\phi$ to obtain EfficientNet-B1 to B7.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### EfficientDet\nAfter gaining a rough understanding of how EfficientNet works, lets observe the EfficientDet architecture.  \n<img src='https://1.bp.blogspot.com/-MQO5qKuTT8c/XpdE8_IwpsI/AAAAAAAAFtg/mSjhF2ws5FYxwcHN6h9_l5DqYzQlNYJwwCLcBGAsYHQ/s1600/image1.png' width=600, height=500>\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Key Optimizations\n\n* **Use of EfficientNet**  \nThe use of EfficientNet as the backbone helps to increase accuracy.\n* **BiFPN (BiDirectional Feature Pyramid Network) with fast normalized fusion technique**   \nFPN are networks that takes in features from the backbone and output fused features that has salient characteristics of the image. BiFPN allows information to flow from top-down and bottom-up direction\n* **Incorporate compound scaling factor into backbone, FPN, box/class predictor**  \nAllows us to determine easily how to scale the model for a given target resource constraint","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I have just given a very brief explanation on how EfficientNet works, and the various key optimization in EfficientDet that results in state-of-the-art performance. For more in-depth understanding, i would suggest reading the actual paper itself. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### References:\nhttps://www.learnopencv.com/efficientnet-theory-code/  \nhttps://ai.googleblog.com/2020/04/efficientdet-towards-scalable-and.html","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Download and Import Dependencies","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install albumentations\n!pip install --no-deps '../input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl' > /dev/null\n\nimport sys\nsys.path.insert(0,'../input/effdet-pytorch') #add packages to system path to allow import\nsys.path.insert(0,'../input/torch-img-model')\nsys.path.insert(0,'../input/omegaconf')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport torch\nimport os\nfrom glob import glob\nimport random\nfrom tqdm.notebook import tqdm\nimport cv2\nimport albumentations as A\nfrom torch.utils.data import Dataset,DataLoader\nfrom sklearn.model_selection import StratifiedKFold\nfrom albumentations.pytorch.transforms import ToTensorV2\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Preprocessing","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"TRAIN_PATH = \"../input/global-wheat-detection/train/\"\nIMG_SIZE = 512\n\ndef set_seed(seed):\n    os.environ['PYTHONHASHSEED'] = str(seed)  # set PYTHONHASHSEED env var at fixed value\n    random.seed(seed)  #set fixed value for python built-in pseudo-random generator\n    np.random.seed(seed) # for numpy pseudo-random generator\n    torch.manual_seed(seed) # pytorch (both CPU and CUDA)\n    \nset_seed(2020)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv('../input/global-wheat-detection/train.csv')\n\nprint(\"Shape of train csv: \", train_csv.shape)\nprint(\"Number of distinct img in data: \", train_csv['image_id'].nunique())\ntrain_csv.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# split bbox \nbbox = np.stack(train_csv['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=',')))\nfor i, col in enumerate(['x','y','w','h']):\n    train_csv[col] = bbox[:,i]\ntrain_csv.drop(columns=['bbox'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True)\n\ndf_folds = train_csv[['image_id']].copy()\ndf_folds['bbox_count'] = 1\ndf_folds = df_folds.groupby('image_id').count() #num bbox for each img_id\ndf_folds['source'] = train_csv[['image_id', 'source']].groupby('image_id').min()['source'] #get source from each img_id\ndf_folds.loc[:, 'stratify_group'] = np.char.add(\n    df_folds['source'].values.astype(str),\n    df_folds['bbox_count'].apply(lambda x: f'_{x // 15}').values.astype(str)\n)\ndf_folds.loc[:, 'fold'] = 0\n\nfor fold_num, (train_idx, val_idx) in enumerate(skf.split(X=df_folds.index, y=df_folds['stratify_group'])):\n    df_folds.loc[df_folds.iloc[val_idx].index,'fold'] = fold_num","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Configuration Settings","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class GlobalConfig:\n    num_workers = 2\n    batch_size = 4\n    n_epochs = 2\n    lr = 1e-4\n    \n    verbose = 1\n    verbose_step = 1\n    \n    folder = 'effdet_train'\n    \n    step_scheduler = False  # do scheduler.step after optimizer.step\n    validation_scheduler = True  # do scheduler.step after validation stage loss\n    \n    \n    SchedulerClass = torch.optim.lr_scheduler.ReduceLROnPlateau\n    scheduler_params = dict(\n        mode='min', #lr reduced when monitored quantity stopped decreasing\n        factor=0.5,\n        patience=1,\n        threshold_mode='abs',\n        min_lr=1e-8\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class WheatData(Dataset):\n    def __init__(self, df, img_ids, transform=None, test=False):\n        super().__init__()\n        self.df = df\n        self.img_ids = img_ids\n        self.transform = transform\n        self.test = test\n        \n    def __getitem__(self, index:int):\n        img_id = self.img_ids[index]\n    \n        if self.test or random.random() > 0.5:\n            image, boxes = self.load_image_and_boxes(index)\n        else:\n            image, boxes = self.load_cutmix_image_and_boxes(index)\n            \n        labels = torch.ones((boxes.shape[0],), dtype=torch.int64)\n        \n        target = {}\n        target['boxes'] = boxes\n        target['labels'] = labels\n        target['img_scale'] = torch.tensor([1.])\n        target['image_id'] = torch.tensor([index])\n        target['img_size'] = torch.tensor([(IMG_SIZE, IMG_SIZE)])\n        \n        if self.transform:\n            for i in range(10):\n                sample = {\n                    'image': image,\n                    'bboxes': target['boxes'],\n                    'labels': labels\n                }\n                sample = self.transform(**sample)\n                \n                if len(sample['bboxes']) > 0:\n                    image = sample['image']\n                    #print(sample['bboxes'])\n                    target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n                    #print(target['boxes'])\n                    target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]]  #yxyx format that is compatible with the model requirements\n                    break\n                    \n        return image, target, img_id\n    \n    def __len__(self):\n        return self.img_ids.shape[0]\n    \n    \n    def load_image_and_boxes(self, index):\n        image_id = self.img_ids[index]\n        image = cv2.imread(f'{TRAIN_PATH}/{image_id}.jpg', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        records = self.df[self.df['image_id'] == image_id]\n        boxes = records[['x', 'y', 'w', 'h']].values\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        return image, boxes\n    \n    \n    def load_cutmix_image_and_boxes(self, index, imsize=1024):\n        \"\"\" \n        This implementation of cutmix author:  https://www.kaggle.com/nvnnghia \n        Refactoring and adaptation: https://www.kaggle.com/shonenkov\n        \"\"\"\n        w, h = imsize, imsize\n        s = imsize // 2\n    \n        xc, yc = [int(random.uniform(imsize * 0.25, imsize * 0.75)) for _ in range(2)]  # center x, y\n        indexes = [index] + [random.randint(0, self.img_ids.shape[0] - 1) for _ in range(3)]\n\n        result_image = np.full((imsize, imsize, 3), 1, dtype=np.float32)\n        result_boxes = []\n\n        for i, index in enumerate(indexes):\n            image, boxes = self.load_image_and_boxes(index)\n            if i == 0:\n                x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc  # xmin, ymin, xmax, ymax (large image)\n                x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h  # xmin, ymin, xmax, ymax (small image)\n            elif i == 1:  # top right\n                x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc\n                x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h\n            elif i == 2:  # bottom left\n                x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)\n                x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, max(xc, w), min(y2a - y1a, h)\n            elif i == 3:  # bottom right\n                x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)\n                x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)\n            result_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]\n            padw = x1a - x1b\n            padh = y1a - y1b\n\n            boxes[:, 0] += padw\n            boxes[:, 1] += padh\n            boxes[:, 2] += padw\n            boxes[:, 3] += padh\n\n            result_boxes.append(boxes)\n\n        result_boxes = np.concatenate(result_boxes, 0)\n        np.clip(result_boxes[:, 0:], 0, 2 * s, out=result_boxes[:, 0:])\n        result_boxes = result_boxes.astype(np.int32)\n        result_boxes = result_boxes[np.where((result_boxes[:,2]-result_boxes[:,0])*(result_boxes[:,3]-result_boxes[:,1]) > 0)]\n        return result_image, result_boxes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Transformation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_transforms():\n    return A.Compose([\n            A.RandomSizedCrop(min_max_height=(800, 800), height=1024, width=1024, p=0.5),\n            A.OneOf([\n                A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n                                     val_shift_limit=0.2, p=0.9),\n                A.RandomBrightnessContrast(brightness_limit=0.2, \n                                           contrast_limit=0.2, p=0.9),\n                    ],p=0.9),\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.Resize(height=512, width=512, p=1),\n            A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n            ToTensorV2(p=1.0)], p=1.0, \n            bbox_params=A.BboxParams(\n                format='pascal_voc',\n                label_fields=['labels']\n        )\n    )\n\ndef valid_transforms():\n    return A.Compose(\n        [\n            A.Resize(height=512, width=512, p=1.0),\n            ToTensorV2(p=1.0),\n        ], \n        p=1.0, \n        bbox_params=A.BboxParams(\n            format='pascal_voc',\n            label_fields=['labels']\n        )\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_number = 0\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n# Instantiate dataset class\ntrain_dataset = WheatData(\n    img_ids=df_folds[df_folds['fold'] != fold_number].index.values,\n    df=train_csv,\n    transform=train_transforms(),\n    test=False)\n\nvalid_dataset = WheatData(\n    img_ids=df_folds[df_folds['fold'] == fold_number].index.values,\n    df=train_csv,\n    transform=valid_transforms(),\n    test=True)\n\n\n# Create dataloader\ntrain_loader = DataLoader(train_dataset,\n                         batch_size = GlobalConfig.batch_size,\n                         collate_fn = collate_fn,\n                         shuffle=True)\n\nvalid_loader = DataLoader(valid_dataset,\n                         batch_size = GlobalConfig.batch_size,\n                         collate_fn = collate_fn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image, target, image_id = train_dataset[0]\nboxes = target['boxes'].cpu().numpy().astype(np.int64)\n\nnumpy_image = image.permute(1,2,0).cpu().numpy()\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nfor box in boxes:\n    cv2.rectangle(numpy_image, (box[1], box[0]), (box[3],  box[2]), (0, 1, 0), 2)\n    \nax.set_axis_off()\nax.imshow(numpy_image);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Meter","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class AverageMeter():\n    def __init__(self):\n        self.reset()\n    \n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n        \n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Instantiation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import effdet\nfrom effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\n\ndef get_net():\n    config = get_efficientdet_config('tf_efficientdet_d5')\n    net = EfficientDet(config, pretrained_backbone=False)\n    checkpoint = torch.load('../input/efficientdet-model/eff_det_models/tf_efficientdet_d5-ef44aea8.pth') #d3-d7 ('efficientdet_model' folder) \n    net.load_state_dict(checkpoint)\n    config.num_classes = 1\n    config.image_size = 512\n    net.class_net = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=.001, momentum=.01))\n    return DetBenchTrain(net, config)\n\nnet = get_net()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Fitter:\n    def __init__(self, model, device, config):\n        self.config = config\n        self.model = model\n        self.device = device\n        \n        self.base_dir = f'./{config.folder}'\n        if not os.path.exists(self.base_dir):\n            os.makedirs(self.base_dir)\n\n        self.epoch = 0\n        self.best_summary_loss = 10**5\n        self.log_path = f'{self.base_dir}/log.txt'\n\n        param_optimizer = list(self.model.named_parameters())\n\n        self.optimizer = torch.optim.AdamW(self.model.parameters(), lr = config.lr)\n        self.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\n        \n        self.log(\"Begin training with {}\".format(self.device))\n    \n    \n    def fit(self, train_loader, valid_loader):\n        for i in range(self.config.n_epochs):\n            summary_loss = self.train_epoch(train_loader)\n            self.log(f'[TRAINING] Epoch {self.epoch}, Loss : {summary_loss.avg}')\n            self.save(f'{self.base_dir}/last-checkpoint.bin')\n\n            summary_loss = self.validation(valid_loader)\n            self.log(f'[VALIDATION] Epoch {self.epoch}, Loss : {summary_loss.avg}')\n\n            if self.best_summary_loss > summary_loss.avg:\n                self.best_summary_loss = summary_loss.avg\n                self.model.eval()\n                self.save(f'{self.base_dir}/best-checkpoint-{str(self.epoch).zfill(2)}epoch.bin')\n                for path in sorted(glob(f'{self.base_dir}/best-checkpoint-*epoch.bin'))[:-3]:\n                    os.remove(path)\n\n            if self.config.validation_scheduler:\n                self.scheduler.step(metrics=summary_loss.avg)\n\n            self.epoch += 1\n            \n    \n    def validation(self, valid_loader):\n        self.model.eval()\n        summary_loss = AverageMeter()\n        for steps, (images, targets, image_ids) in enumerate(valid_loader):\n            with torch.no_grad():\n                pred_res = {}\n                images = torch.stack(images)\n                batch_size = images.shape[0]\n                images = images.to(self.device).float()\n                \n                boxes = [target['boxes'].to(self.device).float() for target in targets]\n                labels = [target['labels'].to(self.device).float() for target in targets]\n                img_scale = torch.tensor([target['img_scale'].to(self.device) for target in targets])\n                img_size = torch.tensor([(IMG_SIZE, IMG_SIZE) for target in targets]).to(self.device).float()\n                \n                pred_res['bbox'] = boxes\n                pred_res['cls'] = labels\n                pred_res['img_scale'] = img_scale\n                pred_res['img_size'] = img_size\n\n                outputs = self.model(images, pred_res)\n                loss = outputs['loss']\n                summary_loss.update(loss.detach().item(), batch_size)\n\n        return summary_loss\n    \n    \n    def train_epoch(self, train_loader):\n        self.model.train()\n        summary_loss = AverageMeter()\n        for images, targets, image_ids in tqdm(train_loader):\n            target_res = {}\n            images = torch.stack(images)\n            images = images.to(self.device).float()\n            batch_size = images.shape[0]\n            \n            boxes = [target['boxes'].to(self.device).float() for target in targets]\n            labels = [target['labels'].to(self.device).float() for target in targets]\n            img_scale = torch.tensor([target['img_scale'] for target in targets]).to(self.device).float()\n            img_size = torch.tensor([(IMG_SIZE, IMG_SIZE) for target in targets]).to(self.device).float()\n            \n            target_res['bbox'] = boxes\n            target_res['cls'] = labels\n            target_res['img_scale'] = img_scale\n            target_res['img_size'] = img_size\n\n            self.optimizer.zero_grad()\n            \n            outputs = self.model(images, target_res)\n            loss = outputs['loss']\n            \n            loss.backward()\n\n            summary_loss.update(loss.detach().item(), batch_size)\n\n            self.optimizer.step()\n\n        return summary_loss\n    \n    \n    def save(self, path):\n        self.model.eval()\n        torch.save({\n            'model_state_dict': self.model.model.state_dict(),\n            'optimizer_state_dict': self.optimizer.state_dict(),\n            'scheduler_state_dict': self.scheduler.state_dict(),\n            'best_summary_loss': self.best_summary_loss,\n            'epoch': self.epoch,\n        }, path)\n        \n        \n#     def load(self, path):\n#         checkpoint = torch.load(path)\n#         self.model.model.load_state_dict(checkpoint['model_state_dict'])\n#         self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n#         self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n#         self.best_summary_loss = checkpoint['best_summary_loss']\n#         self.epoch = checkpoint['epoch'] + 1\n        \n    \n    def log(self, message):\n        if self.config.verbose:\n            print(message)\n        with open(self.log_path, 'a+') as logger:\n            logger.write(f'{message}\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\ndef training():\n    net.to(device)\n    \n    fitter = Fitter(model=net, device=device, config=GlobalConfig)\n    fitter.fit(train_loader, valid_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference\nFor the inference, i have referred to [@shonenkov](https://www.kaggle.com/shonenkov) insightful use of Weighted Box Fusion(WBF) that uses information from all boxes to fix the overlapping bounding boxes issues as well as the Test Time Augmentation(TTA) template that he has kindly provided [here](https://www.kaggle.com/shonenkov/wbf-over-tta-single-model-efficientdet). Do head over and read about it!","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sys.path.insert(0, \"../input/weightedboxfusion\")\n\nimport gc\nfrom effdet import DetBenchPredict\nfrom ensemble_boxes import *\n\nTEST_PATH = \"../input/global-wheat-detection/test/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class WheatData(Dataset):\n    def __init__(self, img_ids, transform=None):\n        self.img_ids = img_ids\n        self.transform = transform\n        \n    def __getitem__(self, index):\n        img_id = self.img_ids[index]\n        image = cv2.imread(f'{TEST_PATH}/{img_id}.jpg', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image = image /255.0\n        \n        if self.transform:\n            sample = {'image' : image}\n            sample = self.transform(**sample)\n            image = sample['image']\n        \n        target = {}\n        target['img_scale'] = torch.tensor([1.])\n            \n        return image, img_id, target\n        \n    def __len__(self) -> int: #annotate parameters with their expected type\n        return self.img_ids.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def valid_transform():\n    return A.Compose([\n            A.Resize(height=512, width=512, p=1.0),\n            ToTensorV2(p=1.0)], \n            p=1.0)\n\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n\ntest_dataset = WheatData(\n    img_ids=np.array([path.split('/')[-1][:-4] for path in glob(f'{TEST_PATH}/*.jpg')]),\n    transform=valid_transform())\n\ntest_loader = DataLoader(test_dataset,\n                         batch_size = 4,\n                         shuffle = False,\n                         drop_last = False,\n                         collate_fn = collate_fn) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load saved model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_net(checkpoint_path):\n    config = get_efficientdet_config('tf_efficientdet_d5')\n    net = EfficientDet(config, pretrained_backbone=False)\n    \n    config.num_classes = 1\n    config.image_size = 512\n    net.class_net = HeadNet(config, num_outputs=config.num_classes, \n                            norm_kwargs=dict(eps=.001, momentum=.01))\n    \n    checkpoint = torch.load(checkpoint_path)\n    net.load_state_dict(checkpoint['model_state_dict'])\n    \n    del checkpoint\n    gc.collect()\n    \n    net = DetBenchPredict(net, config)\n    net.eval()\n    \n    return net.cuda()\n\n# load\nnet = load_net('./effdet_train/best-checkpoint-01epoch.bin')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### TTA","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class BaseWheatTTA:\n    \"\"\" author: @shonenkov \"\"\"\n    image_size = 512\n\n    def augment(self, image):\n        raise NotImplementedError\n    \n    def batch_augment(self, images):\n        raise NotImplementedError\n    \n    def deaugment_boxes(self, boxes):\n        raise NotImplementedError\n\nclass TTAHorizontalFlip(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n\n    def augment(self, image):\n        return image.flip(1)\n    \n    def batch_augment(self, images):\n        return images.flip(2)\n    \n    def deaugment_boxes(self, boxes):\n        boxes[:, [1,3]] = self.image_size - boxes[:, [3,1]]\n        return boxes\n\nclass TTAVerticalFlip(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n    \n    def augment(self, image):\n        return image.flip(2)\n    \n    def batch_augment(self, images):\n        return images.flip(3)\n    \n    def deaugment_boxes(self, boxes):\n        boxes[:, [0,2]] = self.image_size - boxes[:, [2,0]]\n        return boxes\n    \nclass TTARotate90(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n    \n    def augment(self, image):\n        return torch.rot90(image, 1, (1, 2))\n\n    def batch_augment(self, images):\n        return torch.rot90(images, 1, (2, 3))\n    \n    def deaugment_boxes(self, boxes):\n        res_boxes = boxes.copy()\n        res_boxes[:, [0,2]] = self.image_size - boxes[:, [1,3]]\n        res_boxes[:, [1,3]] = boxes[:, [2,0]]\n        return res_boxes\n\nclass TTACompose(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n    def __init__(self, transforms):\n        self.transforms = transforms\n        \n    def augment(self, image):\n        for transform in self.transforms:\n            image = transform.augment(image)\n        return image\n    \n    def batch_augment(self, images):\n        for transform in self.transforms:\n            images = transform.batch_augment(images)\n        return images\n    \n    def prepare_boxes(self, boxes):\n        result_boxes = boxes.copy()\n        result_boxes[:,0] = np.min(boxes[:, [0,2]], axis=1)\n        result_boxes[:,2] = np.max(boxes[:, [0,2]], axis=1)\n        result_boxes[:,1] = np.min(boxes[:, [1,3]], axis=1)\n        result_boxes[:,3] = np.max(boxes[:, [1,3]], axis=1)\n        return result_boxes\n    \n    def deaugment_boxes(self, boxes):\n        for transform in self.transforms[::-1]:\n            boxes = transform.deaugment_boxes(boxes)\n        return self.prepare_boxes(boxes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Weighted Box Fusion","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def run_wbf(predictions, image_index, image_size=512, iou_thr=0.44, \n            skip_box_thr=0.43, weights=None):\n    boxes = [(prediction[image_index]['boxes']/(image_size-1)).tolist() for prediction in predictions]\n    scores = [prediction[image_index]['scores'].tolist() for prediction in predictions]\n    labels = [np.ones(prediction[image_index]['scores'].shape[0]).astype(int).tolist() for prediction in predictions]\n    boxes, scores, labels = ensemble_boxes_wbf.weighted_boxes_fusion(boxes, scores, labels, weights=None, iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n    boxes = boxes*(image_size-1)\n    return boxes, scores, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Running inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def format_prediction_string(boxes, scores):\n    pred_strings = []\n    for j in zip(scores, boxes):\n        pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], \n                                                             j[1][2], j[1][3]))\n    return \" \".join(pred_strings)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from itertools import product\n\ntta_transforms = []\nfor tta_combination in product([TTAHorizontalFlip(), None], \n                               [TTAVerticalFlip(), None],\n                               [TTARotate90(), None]):\n    tta_transforms.append(TTACompose([tta_transform for tta_transform in tta_combination if tta_transform]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# WBF over TTA\n\ndef predict(images, target, score_thres=0.25):\n    with torch.no_grad():\n        prediction = []\n        images = torch.stack(images).to(device).float()\n        img_scale = torch.tensor([target['img_scale'].to(device) for target in targets])\n        img_size = torch.tensor([(IMG_SIZE, IMG_SIZE) for target in targets]).to(device)\n\n        '''\n\n        Within the forward function of the DetBenchPredict class, it takes in 3 arguments (image, image_scale, image_size)\n        The return object is as follows: \n        detections = torch.cat([boxes, scores, classes.float()], dim=1) \n        where the first 4 col will be the bboxes, 5th col the scores\n        Find out more at https://github.com/rwightman/efficientdet-pytorch/blob/master/effdet/bench.py\n\n        '''\n\n        for tta_transform in tta_transforms:\n            result = []\n            det = net(tta_transform.batch_augment(images.clone()),\n                      img_scales = img_scale,\n                      img_size = img_size)\n\n            for i in range(images.shape[0]):\n                boxes = det[i].detach().cpu().numpy()[:,:4]    \n                scores = det[i].detach().cpu().numpy()[:,4]\n                indexes = np.where(scores > score_thres)[0]\n                boxes = boxes[indexes]\n                boxes[:, 2] = boxes[:, 2] + boxes[:, 0]\n                boxes[:, 3] = boxes[:, 3] + boxes[:, 1]\n                boxes = tta_transform.deaugment_boxes(boxes.copy())\n                result.append({\n                    'boxes': boxes,\n                    'scores': scores[indexes],\n                })\n\n            prediction.append(result)\n\n    return prediction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = []\nfor images, image_ids, targets in test_loader:\n    predictions = predict(images, targets)\n    for i, image in enumerate(images):\n        boxes, scores, labels = run_wbf(predictions, image_index=i)\n        boxes = (boxes*2).round().astype(np.int32).clip(min=0, max=1023)\n        image_id = image_ids[i]\n        \n        boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n        boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n\n        result = {\n            'image_id': image_id,\n            'PredictionString': format_prediction_string(boxes, scores)\n        }\n        results.append(result)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame(results, columns=['image_id', 'PredictionString'])\ntest_df.to_csv('submission.csv', index=False)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"I hope that with this notebook, you are able to implement the updated version of EfficientDet and try out the various interesting ideas you may have going forward! :)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}