{"cells":[{"metadata":{},"cell_type":"markdown","source":"## General information\n\n```\nTo get large and accurate data about wheat fields worldwide, plant scientists use image detection of \"wheat heads\"—spikes atop the plant containing grain. These images are used to estimate the density and size of wheat heads in different varieties. Farmers can use the data to assess health and maturity when making management decisions in their fields.\nHowever, accurate wheat head detection in outdoor field images can be visually challenging. There is often overlap of dense wheat plants, and the wind can blur the photographs. Both make it difficult to identify single heads. Additionally, appearances vary due to maturity, color, genotype, and head orientation. Finally, because wheat is grown worldwide, different varieties, planting densities, patterns, and field conditions must be considered.\n```\n\nIn this competition we detect wheat heads on images and return coordinates of bboxes and their confidence.\n\nYou can see a repository with code (can be run as a script) here: https://github.com/Erlemar/wheat\n\nThe metric is IoU:\n![](https://pyimagesearch.com/wp-content/uploads/2016/09/iou_equation.png)\n\nIt is calculated at at different thresholds from 0.5 to 0.75 with a step size of 0.05.\n\nMy kernel is based on official tutorial: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html and could intersect with this cool kernel by Peter: https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train\n\nBut this kernel I want to show you how to use Pytorch-Lightning framework for deep learning. I like the concept of this library and hope you will also find it useful.\n\n**UPD**: code updated for pytorch-lightning 0.7.6\n\n**Version 8**: update pytorch-lightning to 0.8.1 and add various small fixes.\n\n![](https://media.publika.md/en/image/201807/full/wheat_21872400_87580900_75211700.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Import libraries","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip uninstall pycocotools -y\n!pip install -q git+https://github.com/waleedka/coco.git#subdirectory=PythonAPI","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install hydra-core\n!pip install pytorch-lightning==0.8.1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### import libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from __future__ import print_function\nfrom PIL import Image\nfrom albumentations.core.composition import Compose\nfrom collections import defaultdict, deque\nfrom itertools import product\nfrom omegaconf import DictConfig, OmegaConf\nfrom pycocotools import mask as coco_mask\nfrom pycocotools.coco import COCO\nfrom pycocotools.cocoeval import COCOeval\nfrom pytorch_lightning.loggers import TensorBoardLogger\nfrom sklearn.model_selection import train_test_split\nfrom torch.utils.data import Dataset\nfrom torchvision.transforms import functional as F\nfrom typing import Any, Dict, List\nimport albumentations as A\nimport ast\nimport copy\nimport cv2\nimport datetime\nimport errno\nimport hydra\nimport importlib\nimport json\nimport numpy as np\nimport os\nimport pandas as pd\nimport pickle\nimport pycocotools.mask as mask_util\nimport pytorch_lightning as pl\nimport random\nimport shutil\nimport tempfile\nimport time\nimport torch\nimport torch._six\nimport torch.distributed as dist\nimport torch.utils.data\nimport torchvision\nimport matplotlib.pyplot as plt\nimport collections\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper functions\n\nmany functions are taken from https://github.com/pytorch/vision/tree/v0.3.0/references/detection\nsome of them were modified for this competition","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# different small utils\n\nclass SmoothedValue(object):\n    \"\"\"Track a series of values and provide access to smoothed values over a\n    window or the global series average.\n    \"\"\"\n\n    def __init__(self, window_size=20, fmt=None):\n        if fmt is None:\n            fmt = \"{median:.4f} ({global_avg:.4f})\"\n        self.deque = deque(maxlen=window_size)\n        self.total = 0.0\n        self.count = 0\n        self.fmt = fmt\n\n    def update(self, value, n=1):\n        self.deque.append(value)\n        self.count += n\n        self.total += value * n\n\n    def synchronize_between_processes(self):\n        \"\"\"\n        Warning: does not synchronize the deque!\n        \"\"\"\n        if not is_dist_avail_and_initialized():\n            return\n        t = torch.tensor([self.count, self.total], dtype=torch.float64, device='cuda')\n        dist.barrier()\n        dist.all_reduce(t)\n        t = t.tolist()\n        self.count = int(t[0])\n        self.total = t[1]\n\n    @property\n    def median(self):\n        d = torch.tensor(list(self.deque))\n        return d.median().item()\n\n    @property\n    def avg(self):\n        d = torch.tensor(list(self.deque), dtype=torch.float32)\n        return d.mean().item()\n\n    @property\n    def global_avg(self):\n        return self.total / self.count\n\n    @property\n    def max(self):\n        return max(self.deque)\n\n    @property\n    def value(self):\n        return self.deque[-1]\n\n    def __str__(self):\n        return self.fmt.format(\n            median=self.median,\n            avg=self.avg,\n            global_avg=self.global_avg,\n            max=self.max,\n            value=self.value)\n\n\ndef all_gather(data):\n    \"\"\"\n    Run all_gather on arbitrary picklable data (not necessarily tensors)\n    Args:\n        data: any picklable object\n    Returns:\n        list[data]: list of data gathered from each rank\n    \"\"\"\n    world_size = get_world_size()\n    if world_size == 1:\n        return [data]\n\n    # serialized to a Tensor\n    buffer = pickle.dumps(data)\n    storage = torch.ByteStorage.from_buffer(buffer)\n    tensor = torch.ByteTensor(storage).to(\"cuda\")\n\n    # obtain Tensor size of each rank\n    local_size = torch.tensor([tensor.numel()], device=\"cuda\")\n    size_list = [torch.tensor([0], device=\"cuda\") for _ in range(world_size)]\n    dist.all_gather(size_list, local_size)\n    size_list = [int(size.item()) for size in size_list]\n    max_size = max(size_list)\n\n    # receiving Tensor from all ranks\n    # we pad the tensor because torch all_gather does not support\n    # gathering tensors of different shapes\n    tensor_list = []\n    for _ in size_list:\n        tensor_list.append(torch.empty((max_size,), dtype=torch.uint8, device=\"cuda\"))\n    if local_size != max_size:\n        padding = torch.empty(size=(max_size - local_size,), dtype=torch.uint8, device=\"cuda\")\n        tensor = torch.cat((tensor, padding), dim=0)\n    dist.all_gather(tensor_list, tensor)\n\n    data_list = []\n    for size, tensor in zip(size_list, tensor_list):\n        buffer = tensor.cpu().numpy().tobytes()[:size]\n        data_list.append(pickle.loads(buffer))\n\n    return data_list\n\n\ndef reduce_dict(input_dict, average=True):\n    \"\"\"\n    Args:\n        input_dict (dict): all the values will be reduced\n        average (bool): whether to do average or sum\n    Reduce the values in the dictionary from all processes so that all processes\n    have the averaged results. Returns a dict with the same fields as\n    input_dict, after reduction.\n    \"\"\"\n    world_size = get_world_size()\n    if world_size < 2:\n        return input_dict\n    with torch.no_grad():\n        names = []\n        values = []\n        # sort the keys so that they are consistent across processes\n        for k in sorted(input_dict.keys()):\n            names.append(k)\n            values.append(input_dict[k])\n        values = torch.stack(values, dim=0)\n        dist.all_reduce(values)\n        if average:\n            values /= world_size\n        reduced_dict = {k: v for k, v in zip(names, values)}\n    return reduced_dict\n\n\nclass MetricLogger(object):\n    def __init__(self, delimiter=\"\\t\"):\n        self.meters = defaultdict(SmoothedValue)\n        self.delimiter = delimiter\n\n    def update(self, **kwargs):\n        for k, v in kwargs.items():\n            if isinstance(v, torch.Tensor):\n                v = v.item()\n            assert isinstance(v, (float, int))\n            self.meters[k].update(v)\n\n    def __getattr__(self, attr):\n        if attr in self.meters:\n            return self.meters[attr]\n        if attr in self.__dict__:\n            return self.__dict__[attr]\n        raise AttributeError(\"'{}' object has no attribute '{}'\".format(\n            type(self).__name__, attr))\n\n    def __str__(self):\n        loss_str = []\n        for name, meter in self.meters.items():\n            loss_str.append(\n                \"{}: {}\".format(name, str(meter))\n            )\n        return self.delimiter.join(loss_str)\n\n    def synchronize_between_processes(self):\n        for meter in self.meters.values():\n            meter.synchronize_between_processes()\n\n    def add_meter(self, name, meter):\n        self.meters[name] = meter\n\n    def log_every(self, iterable, print_freq, header=None):\n        i = 0\n        if not header:\n            header = ''\n        start_time = time.time()\n        end = time.time()\n        iter_time = SmoothedValue(fmt='{avg:.4f}')\n        data_time = SmoothedValue(fmt='{avg:.4f}')\n        space_fmt = ':' + str(len(str(len(iterable)))) + 'd'\n        log_msg = self.delimiter.join([\n            header,\n            '[{0' + space_fmt + '}/{1}]',\n            'eta: {eta}',\n            '{meters}',\n            'time: {time}',\n            'data: {data}',\n            'max mem: {memory:.0f}'\n        ])\n        MB = 1024.0 * 1024.0\n        for obj in iterable:\n            data_time.update(time.time() - end)\n            yield obj\n            iter_time.update(time.time() - end)\n            if i % print_freq == 0 or i == len(iterable) - 1:\n                eta_seconds = iter_time.global_avg * (len(iterable) - i)\n                eta_string = str(datetime.timedelta(seconds=int(eta_seconds)))\n                print(log_msg.format(\n                    i, len(iterable), eta=eta_string,\n                    meters=str(self),\n                    time=str(iter_time), data=str(data_time),\n                    memory=torch.cuda.max_memory_allocated() / MB))\n            i += 1\n            end = time.time()\n        total_time = time.time() - start_time\n        total_time_str = str(datetime.timedelta(seconds=int(total_time)))\n        print('{} Total time: {} ({:.4f} s / it)'.format(\n            header, total_time_str, total_time / len(iterable)))\n\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n\ndef warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor):\n\n    def f(x):\n        if x >= warmup_iters:\n            return 1\n        alpha = float(x) / warmup_iters\n        return warmup_factor * (1 - alpha) + alpha\n\n    return torch.optim.lr_scheduler.LambdaLR(optimizer, f)\n\n\ndef mkdir(path):\n    try:\n        os.makedirs(path)\n    except OSError as e:\n        if e.errno != errno.EEXIST:\n            raise\n\n\ndef setup_for_distributed(is_master):\n    \"\"\"\n    This function disables printing when not in master process\n    \"\"\"\n    import builtins as __builtin__\n    builtin_print = __builtin__.print\n\n    def print(*args, **kwargs):\n        force = kwargs.pop('force', False)\n        if is_master or force:\n            builtin_print(*args, **kwargs)\n\n    __builtin__.print = print\n\n\ndef is_dist_avail_and_initialized():\n    if not dist.is_available():\n        return False\n    if not dist.is_initialized():\n        return False\n    return True\n\n\ndef get_world_size():\n    if not is_dist_avail_and_initialized():\n        return 1\n    return dist.get_world_size()\n\n\ndef get_rank():\n    if not is_dist_avail_and_initialized():\n        return 0\n    return dist.get_rank()\n\n\ndef is_main_process():\n    return get_rank() == 0\n\n\ndef save_on_master(*args, **kwargs):\n    if is_main_process():\n        torch.save(*args, **kwargs)\n\n\ndef init_distributed_mode(args):\n    if 'RANK' in os.environ and 'WORLD_SIZE' in os.environ:\n        args.rank = int(os.environ[\"RANK\"])\n        args.world_size = int(os.environ['WORLD_SIZE'])\n        args.gpu = int(os.environ['LOCAL_RANK'])\n    elif 'SLURM_PROCID' in os.environ:\n        args.rank = int(os.environ['SLURM_PROCID'])\n        args.gpu = args.rank % torch.cuda.device_count()\n    else:\n        print('Not using distributed mode')\n        args.distributed = False\n        return\n\n    args.distributed = True\n\n    torch.cuda.set_device(args.gpu)\n    args.dist_backend = 'nccl'\n    print('| distributed init (rank {}): {}'.format(\n        args.rank, args.dist_url), flush=True)\n    torch.distributed.init_process_group(backend=args.dist_backend, init_method=args.dist_url,\n                                         world_size=args.world_size, rank=args.rank)\n    torch.distributed.barrier()\n    setup_for_distributed(args.rank == 0)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### coco evaluator","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class CocoEvaluator(object):\n    def __init__(self, coco_gt, iou_types):\n        assert isinstance(iou_types, (list, tuple))\n        coco_gt = copy.deepcopy(coco_gt)\n        self.coco_gt = coco_gt\n\n        self.iou_types = iou_types\n        self.coco_eval = {}\n        for iou_type in iou_types:\n            self.coco_eval[iou_type] = COCOeval(coco_gt, iouType=iou_type)\n\n        self.img_ids = []\n        self.eval_imgs = {k: [] for k in iou_types}\n\n    def update(self, predictions):\n        img_ids = list(np.unique(list(predictions.keys())))\n        self.img_ids.extend(img_ids)\n\n        for iou_type in self.iou_types:\n            results = self.prepare(predictions, iou_type)\n            coco_dt = loadRes(self.coco_gt, results) if results else COCO()\n            coco_eval = self.coco_eval[iou_type]\n\n            coco_eval.cocoDt = coco_dt\n            coco_eval.params.imgIds = list(img_ids)\n            img_ids, eval_imgs = evaluate(coco_eval)\n\n            self.eval_imgs[iou_type].append(eval_imgs)\n\n    def synchronize_between_processes(self):\n        for iou_type in self.iou_types:\n            self.eval_imgs[iou_type] = np.concatenate(self.eval_imgs[iou_type], 2)\n            create_common_coco_eval(self.coco_eval[iou_type], self.img_ids, self.eval_imgs[iou_type])\n\n    def accumulate(self):\n        for coco_eval in self.coco_eval.values():\n            coco_eval.evaluate()\n            coco_eval.accumulate()\n\n    def summarize(self):\n        for iou_type, coco_eval in self.coco_eval.items():\n            print(\"IoU metric: {}\".format(iou_type))\n            coco_eval.summarize()\n\n    def prepare(self, predictions, iou_type):\n        if iou_type == \"bbox\":\n            return self.prepare_for_coco_detection(predictions)\n        elif iou_type == \"segm\":\n            return self.prepare_for_coco_segmentation(predictions)\n        elif iou_type == \"keypoints\":\n            return self.prepare_for_coco_keypoint(predictions)\n        else:\n            raise ValueError(\"Unknown iou type {}\".format(iou_type))\n\n    def prepare_for_coco_detection(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue\n\n            boxes = prediction[\"boxes\"]\n            boxes = convert_to_xywh(boxes).tolist()\n            scores = prediction[\"scores\"].tolist()\n            labels = prediction[\"labels\"].tolist()\n\n            coco_results.extend(\n                [\n                    {\n                        \"image_id\": original_id,\n                        \"category_id\": labels[k],\n                        \"bbox\": box,\n                        \"score\": scores[k],\n                    }\n                    for k, box in enumerate(boxes)\n                ]\n            )\n        return coco_results\n\n    def prepare_for_coco_segmentation(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue\n\n            scores = prediction[\"scores\"]\n            labels = prediction[\"labels\"]\n            masks = prediction[\"masks\"]\n\n            masks = masks > 0.5\n\n            scores = prediction[\"scores\"].tolist()\n            labels = prediction[\"labels\"].tolist()\n\n            rles = [\n                mask_util.encode(np.array(mask[0, :, :, np.newaxis], dtype=np.uint8, order=\"F\"))[0]\n                for mask in masks\n            ]\n            for rle in rles:\n                rle[\"counts\"] = rle[\"counts\"].decode(\"utf-8\")\n\n            coco_results.extend(\n                [\n                    {\n                        \"image_id\": original_id,\n                        \"category_id\": labels[k],\n                        \"segmentation\": rle,\n                        \"score\": scores[k],\n                    }\n                    for k, rle in enumerate(rles)\n                ]\n            )\n        return coco_results\n\n    def prepare_for_coco_keypoint(self, predictions):\n        coco_results = []\n        for original_id, prediction in predictions.items():\n            if len(prediction) == 0:\n                continue\n\n            boxes = prediction[\"boxes\"]\n            boxes = convert_to_xywh(boxes).tolist()\n            scores = prediction[\"scores\"].tolist()\n            labels = prediction[\"labels\"].tolist()\n            keypoints = prediction[\"keypoints\"]\n            keypoints = keypoints.flatten(start_dim=1).tolist()\n\n            coco_results.extend(\n                [\n                    {\n                        \"image_id\": original_id,\n                        \"category_id\": labels[k],\n                        'keypoints': keypoint,\n                        \"score\": scores[k],\n                    }\n                    for k, keypoint in enumerate(keypoints)\n                ]\n            )\n        return coco_results\n\n\ndef convert_to_xywh(boxes):\n    xmin, ymin, xmax, ymax = boxes.unbind(1)\n    return torch.stack((xmin, ymin, xmax - xmin, ymax - ymin), dim=1)\n\n\ndef merge(img_ids, eval_imgs):\n    all_img_ids = all_gather(img_ids)\n    all_eval_imgs = all_gather(eval_imgs)\n\n    merged_img_ids = []\n    for p in all_img_ids:\n        merged_img_ids.extend(p)\n\n    merged_eval_imgs = []\n    for p in all_eval_imgs:\n        merged_eval_imgs.append(p)\n\n    merged_img_ids = np.array(merged_img_ids)\n    merged_eval_imgs = np.concatenate(merged_eval_imgs, 2)\n\n    # keep only unique (and in sorted order) images\n    merged_img_ids, idx = np.unique(merged_img_ids, return_index=True)\n    merged_eval_imgs = merged_eval_imgs[..., idx]\n\n    return merged_img_ids, merged_eval_imgs\n\n\ndef create_common_coco_eval(coco_eval, img_ids, eval_imgs):\n    img_ids, eval_imgs = merge(img_ids, eval_imgs)\n    img_ids = list(img_ids)\n    eval_imgs = list(eval_imgs.flatten())\n\n    coco_eval.evalImgs = eval_imgs\n    coco_eval.params.imgIds = img_ids\n    coco_eval._paramsEval = copy.deepcopy(coco_eval.params)\n\n\n#################################################################\n# From pycocotools, just removed the prints and fixed\n# a Python3 bug about unicode not defined\n#################################################################\n\n# Ideally, pycocotools wouldn't have hard-coded prints\n# so that we could avoid copy-pasting those two functions\n\ndef createIndex(self):\n    # create index\n    # print('creating index...')\n    anns, cats, imgs = {}, {}, {}\n    imgToAnns, catToImgs = defaultdict(list), defaultdict(list)\n    if 'annotations' in self.dataset:\n        for ann in self.dataset['annotations']:\n            imgToAnns[ann['image_id']].append(ann)\n            anns[ann['id']] = ann\n\n    if 'images' in self.dataset:\n        for img in self.dataset['images']:\n            imgs[img['id']] = img\n\n    if 'categories' in self.dataset:\n        for cat in self.dataset['categories']:\n            cats[cat['id']] = cat\n\n    if 'annotations' in self.dataset and 'categories' in self.dataset:\n        for ann in self.dataset['annotations']:\n            catToImgs[ann['category_id']].append(ann['image_id'])\n\n    # print('index created!')\n\n    # create class members\n    self.anns = anns\n    self.imgToAnns = imgToAnns\n    self.catToImgs = catToImgs\n    self.imgs = imgs\n    self.cats = cats\n\n\nmaskUtils = mask_util\n\n\ndef loadRes(self, resFile):\n    \"\"\"\n    Load result file and return a result api object.\n    :param   resFile (str)     : file name of result file\n    :return: res (obj)         : result api object\n    \"\"\"\n    res = COCO()\n    res.dataset['images'] = [img for img in self.dataset['images']]\n\n    # print('Loading and preparing results...')\n    # tic = time.time()\n    if isinstance(resFile, torch._six.string_classes):\n        anns = json.load(open(resFile))\n    elif type(resFile) == np.ndarray:\n        anns = self.loadNumpyAnnotations(resFile)\n    else:\n        anns = resFile\n    assert type(anns) == list, 'results in not an array of objects'\n    annsImgIds = [ann['image_id'] for ann in anns]\n    assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \\\n        'Results do not correspond to current coco set'\n    if 'caption' in anns[0]:\n        imgIds = set([img['id'] for img in res.dataset['images']]) & set([ann['image_id'] for ann in anns])\n        res.dataset['images'] = [img for img in res.dataset['images'] if img['id'] in imgIds]\n        for id, ann in enumerate(anns):\n            ann['id'] = id + 1\n    elif 'bbox' in anns[0] and not anns[0]['bbox'] == []:\n        res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n        for id, ann in enumerate(anns):\n            bb = ann['bbox']\n            x1, x2, y1, y2 = [bb[0], bb[0] + bb[2], bb[1], bb[1] + bb[3]]\n            if 'segmentation' not in ann:\n                ann['segmentation'] = [[x1, y1, x1, y2, x2, y2, x2, y1]]\n            ann['area'] = bb[2] * bb[3]\n            ann['id'] = id + 1\n            ann['iscrowd'] = 0\n    elif 'segmentation' in anns[0]:\n        res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n        for id, ann in enumerate(anns):\n            # now only support compressed RLE format as segmentation results\n            ann['area'] = maskUtils.area(ann['segmentation'])\n            if 'bbox' not in ann:\n                ann['bbox'] = maskUtils.toBbox(ann['segmentation'])\n            ann['id'] = id + 1\n            ann['iscrowd'] = 0\n    elif 'keypoints' in anns[0]:\n        res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n        for id, ann in enumerate(anns):\n            s = ann['keypoints']\n            x = s[0::3]\n            y = s[1::3]\n            x1, x2, y1, y2 = np.min(x), np.max(x), np.min(y), np.max(y)\n            ann['area'] = (x2 - x1) * (y2 - y1)\n            ann['id'] = id + 1\n            ann['bbox'] = [x1, y1, x2 - x1, y2 - y1]\n    # print('DONE (t={:0.2f}s)'.format(time.time()- tic))\n\n    res.dataset['annotations'] = anns\n    createIndex(res)\n    return res\n\n\ndef evaluate(self):\n    '''\n    Run per image evaluation on given images and store results (a list of dict) in self.evalImgs\n    :return: None\n    '''\n    # tic = time.time()\n    # print('Running per image evaluation...')\n    p = self.params\n    # add backward compatibility if useSegm is specified in params\n    if p.useSegm is not None:\n        p.iouType = 'segm' if p.useSegm == 1 else 'bbox'\n        print('useSegm (deprecated) is not None. Running {} evaluation'.format(p.iouType))\n    # print('Evaluate annotation type *{}*'.format(p.iouType))\n    p.imgIds = list(np.unique(p.imgIds))\n    if p.useCats:\n        p.catIds = list(np.unique(p.catIds))\n    p.maxDets = sorted(p.maxDets)\n    self.params = p\n\n    self._prepare()\n    # loop through images, area range, max detection number\n    catIds = p.catIds if p.useCats else [-1]\n\n    if p.iouType == 'segm' or p.iouType == 'bbox':\n        computeIoU = self.computeIoU\n    elif p.iouType == 'keypoints':\n        computeIoU = self.computeOks\n    self.ious = {\n        (imgId, catId): computeIoU(imgId, catId)\n        for imgId in p.imgIds\n        for catId in catIds}\n\n    evaluateImg = self.evaluateImg\n    maxDet = p.maxDets[-1]\n    evalImgs = [\n        evaluateImg(imgId, catId, areaRng, maxDet)\n        for catId in catIds\n        for areaRng in p.areaRng\n        for imgId in p.imgIds\n    ]\n    # this is NOT in the pycocotools code, but could be done outside\n    evalImgs = np.asarray(evalImgs).reshape(len(catIds), len(p.areaRng), len(p.imgIds))\n    self._paramsEval = copy.deepcopy(self.params)\n    # toc = time.time()\n    # print('DONE (t={:0.2f}s).'.format(toc-tic))\n    return p.imgIds, evalImgs\n\n#################################################################\n# end of straight copy from pycocotools, just removing the prints\n#################################################################\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### coco transforms","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def _flip_coco_person_keypoints(kps, width):\n    flip_inds = [0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 16, 15]\n    flipped_data = kps[:, flip_inds]\n    flipped_data[..., 0] = width - flipped_data[..., 0]\n    # Maintain COCO convention that if visibility == 0, then x, y = 0\n    inds = flipped_data[..., 2] == 0\n    flipped_data[inds] = 0\n    return flipped_data\n\n\nclass Compose(object):\n    def __init__(self, transforms):\n        self.transforms = transforms\n\n    def __call__(self, image, target):\n        for t in self.transforms:\n            image, target = t(image, target)\n        return image, target\n\n\nclass RandomHorizontalFlip(object):\n    def __init__(self, prob):\n        self.prob = prob\n\n    def __call__(self, image, target):\n        if random.random() < self.prob:\n            height, width = image.shape[-2:]\n            image = image.flip(-1)\n            bbox = target[\"boxes\"]\n            bbox[:, [0, 2]] = width - bbox[:, [2, 0]]\n            target[\"boxes\"] = bbox\n            if \"masks\" in target:\n                target[\"masks\"] = target[\"masks\"].flip(-1)\n            if \"keypoints\" in target:\n                keypoints = target[\"keypoints\"]\n                keypoints = _flip_coco_person_keypoints(keypoints, width)\n                target[\"keypoints\"] = keypoints\n        return image, target\n\n\nclass ToTensor(object):\n    def __call__(self, image, target):\n        image = F.to_tensor(image)\n        return image, target\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### coco utils","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class FilterAndRemapCocoCategories(object):\n    def __init__(self, categories, remap=True):\n        self.categories = categories\n        self.remap = remap\n\n    def __call__(self, image, target):\n        anno = target[\"annotations\"]\n        anno = [obj for obj in anno if obj[\"category_id\"] in self.categories]\n        if not self.remap:\n            target[\"annotations\"] = anno\n            return image, target\n        anno = copy.deepcopy(anno)\n        for obj in anno:\n            obj[\"category_id\"] = self.categories.index(obj[\"category_id\"])\n        target[\"annotations\"] = anno\n        return image, target\n\n\ndef convert_coco_poly_to_mask(segmentations, height, width):\n    masks = []\n    for polygons in segmentations:\n        rles = coco_mask.frPyObjects(polygons, height, width)\n        mask = coco_mask.decode(rles)\n        if len(mask.shape) < 3:\n            mask = mask[..., None]\n        mask = torch.as_tensor(mask, dtype=torch.uint8)\n        mask = mask.any(dim=2)\n        masks.append(mask)\n    if masks:\n        masks = torch.stack(masks, dim=0)\n    else:\n        masks = torch.zeros((0, height, width), dtype=torch.uint8)\n    return masks\n\n\nclass ConvertCocoPolysToMask(object):\n    def __call__(self, image, target):\n        w, h = image.size\n\n        image_id = target[\"image_id\"]\n        image_id = torch.tensor([image_id])\n\n        anno = target[\"annotations\"]\n\n        anno = [obj for obj in anno if obj['iscrowd'] == 0]\n\n        boxes = [obj[\"bbox\"] for obj in anno]\n        # guard against no boxes via resizing\n        boxes = torch.as_tensor(boxes, dtype=torch.float32).reshape(-1, 4)\n        boxes[:, 2:] += boxes[:, :2]\n        boxes[:, 0::2].clamp_(min=0, max=w)\n        boxes[:, 1::2].clamp_(min=0, max=h)\n\n        classes = [obj[\"category_id\"] for obj in anno]\n        classes = torch.tensor(classes, dtype=torch.int64)\n\n        segmentations = [obj[\"segmentation\"] for obj in anno]\n        masks = convert_coco_poly_to_mask(segmentations, h, w)\n\n        keypoints = None\n        if anno and \"keypoints\" in anno[0]:\n            keypoints = [obj[\"keypoints\"] for obj in anno]\n            keypoints = torch.as_tensor(keypoints, dtype=torch.float32)\n            num_keypoints = keypoints.shape[0]\n            if num_keypoints:\n                keypoints = keypoints.view(num_keypoints, -1, 3)\n\n        keep = (boxes[:, 3] > boxes[:, 1]) & (boxes[:, 2] > boxes[:, 0])\n        boxes = boxes[keep]\n        classes = classes[keep]\n        masks = masks[keep]\n        if keypoints is not None:\n            keypoints = keypoints[keep]\n\n        target = {}\n        target[\"boxes\"] = boxes\n        target[\"labels\"] = classes\n        target[\"masks\"] = masks\n        target[\"image_id\"] = image_id\n        if keypoints is not None:\n            target[\"keypoints\"] = keypoints\n\n        # for conversion to coco api\n        area = torch.tensor([obj[\"area\"] for obj in anno])\n        iscrowd = torch.tensor([obj[\"iscrowd\"] for obj in anno])\n        target[\"area\"] = area\n        target[\"iscrowd\"] = iscrowd\n\n        return image, target\n\n\ndef _coco_remove_images_without_annotations(dataset, cat_list=None):\n    def _has_only_empty_bbox(anno):\n        return all(any(o <= 1 for o in obj[\"bbox\"][2:]) for obj in anno)\n\n    def _count_visible_keypoints(anno):\n        return sum(sum(1 for v in ann[\"keypoints\"][2::3] if v > 0) for ann in anno)\n\n    min_keypoints_per_image = 10\n\n    def _has_valid_annotation(anno):\n        # if it's empty, there is no annotation\n        if len(anno) == 0:\n            return False\n        # if all boxes have close to zero area, there is no annotation\n        if _has_only_empty_bbox(anno):\n            return False\n        # keypoints task have a slight different critera for considering\n        # if an annotation is valid\n        if \"keypoints\" not in anno[0]:\n            return True\n        # for keypoint detection tasks, only consider valid images those\n        # containing at least min_keypoints_per_image\n        if _count_visible_keypoints(anno) >= min_keypoints_per_image:\n            return True\n        return False\n\n    assert isinstance(dataset, torchvision.datasets.CocoDetection)\n    ids = []\n    for ds_idx, img_id in enumerate(dataset.ids):\n        ann_ids = dataset.coco.getAnnIds(imgIds=img_id, iscrowd=None)\n        anno = dataset.coco.loadAnns(ann_ids)\n        if cat_list:\n            anno = [obj for obj in anno if obj[\"category_id\"] in cat_list]\n        if _has_valid_annotation(anno):\n            ids.append(ds_idx)\n\n    dataset = torch.utils.data.Subset(dataset, ids)\n    return dataset\n\n\ndef convert_to_coco_api(ds):\n    coco_ds = COCO()\n    ann_id = 0\n    dataset = {'images': [], 'categories': [], 'annotations': []}\n    categories = set()\n    for img_idx in range(len(ds)):\n        # find better way to get target\n        # targets = ds.get_annotations(img_idx)\n        img, targets, _ = ds[img_idx]\n        image_id = targets[\"image_id\"].item()\n        img_dict = {}\n        img_dict['id'] = image_id\n        img_dict['height'] = img.shape[-2]\n        img_dict['width'] = img.shape[-1]\n        dataset['images'].append(img_dict)\n        bboxes = targets[\"boxes\"]\n        bboxes[:, 2:] -= bboxes[:, :2]\n        bboxes = bboxes.tolist()\n        labels = targets['labels'].tolist()\n        areas = targets['area'].tolist()\n        iscrowd = targets['iscrowd'].tolist()\n        if 'masks' in targets:\n            masks = targets['masks']\n            # make masks Fortran contiguous for coco_mask\n            masks = masks.permute(0, 2, 1).contiguous().permute(0, 2, 1)\n        if 'keypoints' in targets:\n            keypoints = targets['keypoints']\n            keypoints = keypoints.reshape(keypoints.shape[0], -1).tolist()\n        num_objs = len(bboxes)\n        for i in range(num_objs):\n            ann = {}\n            ann['image_id'] = image_id\n            ann['bbox'] = bboxes[i]\n            ann['category_id'] = labels[i]\n            categories.add(labels[i])\n            ann['area'] = areas[i]\n            ann['iscrowd'] = iscrowd[i]\n            ann['id'] = ann_id\n            if 'masks' in targets:\n                ann[\"segmentation\"] = coco_mask.encode(masks[i].numpy())\n            if 'keypoints' in targets:\n                ann['keypoints'] = keypoints[i]\n                ann['num_keypoints'] = sum(k != 0 for k in keypoints[i][2::3])\n            dataset['annotations'].append(ann)\n            ann_id += 1\n    dataset['categories'] = [{'id': i} for i in sorted(categories)]\n    coco_ds.dataset = dataset\n    coco_ds.createIndex()\n    return coco_ds\n\n\ndef get_coco_api_from_dataset(dataset):\n    for i in range(10):\n        if isinstance(dataset, torchvision.datasets.CocoDetection):\n            break\n        if isinstance(dataset, torch.utils.data.Subset):\n            dataset = dataset.dataset\n    if isinstance(dataset, torchvision.datasets.CocoDetection):\n        return dataset.coco\n    return convert_to_coco_api(dataset)\n\n\nclass CocoDetection(torchvision.datasets.CocoDetection):\n    def __init__(self, img_folder, ann_file, transforms):\n        super(CocoDetection, self).__init__(img_folder, ann_file)\n        self._transforms = transforms\n\n    def __getitem__(self, idx):\n        img, target = super(CocoDetection, self).__getitem__(idx)\n        image_id = self.ids[idx]\n        target = dict(image_id=image_id, annotations=target)\n        if self._transforms is not None:\n            img, target = self._transforms(img, target)\n        return img, target\n\n\ndef get_coco(root, image_set, transforms, mode='instances'):\n    anno_file_template = \"{}_{}2017.json\"\n    PATHS = {\n        \"train\": (\"train2017\", os.path.join(\"annotations\", anno_file_template.format(mode, \"train\"))),\n        \"val\": (\"val2017\", os.path.join(\"annotations\", anno_file_template.format(mode, \"val\"))),\n        # \"train\": (\"val2017\", os.path.join(\"annotations\", anno_file_template.format(mode, \"val\")))\n    }\n\n    t = [ConvertCocoPolysToMask()]\n\n    if transforms is not None:\n        t.append(transforms)\n    transforms = Compose(t)\n\n    img_folder, ann_file = PATHS[image_set]\n    img_folder = os.path.join(root, img_folder)\n    ann_file = os.path.join(root, ann_file)\n\n    dataset = CocoDetection(img_folder, ann_file, transforms=transforms)\n\n    if image_set == \"train\":\n        dataset = _coco_remove_images_without_annotations(dataset)\n\n    # dataset = torch.utils.data.Subset(dataset, [i for i in range(500)])\n\n    return dataset\n\n\ndef get_coco_kp(root, image_set, transforms):\n    return get_coco(root, image_set, transforms, mode=\"person_keypoints\")\n\n\ndef _get_iou_types(model):\n    model_without_ddp = model\n    if isinstance(model, torch.nn.parallel.DistributedDataParallel):\n        model_without_ddp = model.module\n    iou_types = [\"bbox\"]\n    if isinstance(model_without_ddp, torchvision.models.detection.MaskRCNN):\n        iou_types.append(\"segm\")\n    if isinstance(model_without_ddp, torchvision.models.detection.KeypointRCNN):\n        iou_types.append(\"keypoints\")\n    return iou_types\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### basic utils","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# https://github.com/quantumblacklabs/kedro/blob/9809bd7ca0556531fa4a2fc02d5b2dc26cf8fa97/kedro/utils.py\ndef load_obj(obj_path: str, default_obj_path: str = \"\") -> Any:\n    \"\"\"Extract an object from a given path.\n        Args:\n            obj_path: Path to an object to be extracted, including the object name.\n            default_obj_path: Default object path.\n        Returns:\n            Extracted object.\n        Raises:\n            AttributeError: When the object does not have the given named attribute.\n    \"\"\"\n    obj_path_list = obj_path.rsplit(\".\", 1)\n    obj_path = obj_path_list.pop(0) if len(obj_path_list) > 1 else default_obj_path\n    obj_name = obj_path_list[0]\n    module_obj = importlib.import_module(obj_path)\n    if not hasattr(module_obj, obj_name):\n        raise AttributeError(\n            f\"Object `{obj_name}` cannot be loaded from `{obj_path}`.\"\n        )\n    return getattr(module_obj, obj_name)\n\n\ndef set_seed(seed: int = 666):\n    np.random.seed(seed)\n    random.seed(seed)\n    torch.backends.cudnn.benchmark = False\n    torch.backends.cudnn.deterministic = True\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n\n\ndef save_useful_info():\n    shutil.copytree(os.path.join(hydra.utils.get_original_cwd(), 'src'),\n                    os.path.join(os.getcwd(), 'code/src'))\n    shutil.copy2(os.path.join(hydra.utils.get_original_cwd(), 'hydra_run.py'), os.path.join(os.getcwd(), 'code'))\n\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n\ndef product_dict(**kwargs) -> List[List]:\n    \"\"\"\n    Convert dict with lists in values into lists of all combinations\n\n    This is necessary to convert config with experiment values\n    into format usable by hydra\n    Args:\n        **kwargs:\n\n    Returns:\n        list of lists\n\n    ---\n    Example:\n        >>> list_dict = {'a': [1, 2], 'b': [2, 3]}\n        >>> list(product_dict(**list_dict))\n        >>> [['a=1', 'b=2'], ['a=1', 'b=3'], ['a=2', 'b=2'], ['a=2', 'b=3']]\n\n    \"\"\"\n    keys = kwargs.keys()\n    vals = kwargs.values()\n    for instance in product(*vals):\n        zip_list = list(zip(keys, instance))\n        yield [f'{i}={j}' for i, j in zip_list]\n\n\ndef config_to_hydra_dict(cfg: DictConfig) -> Dict:\n    \"\"\"\n    Convert config into dict with lists of values, where key is full name of parameter\n\n    This fuction is used to get key names which can be used in hydra.\n\n    Args:\n        cfg:\n\n    Returns:\n\n    \"\"\"\n    experiment_dict = {}\n    for k, v in cfg.items():\n        for k1, v1 in v.items():\n            experiment_dict[f'{k}.{k1}'] = v1\n\n    return experiment_dict\n\n\ndef flatten_omegaconf(d, sep=\"_\"):\n\n    d = OmegaConf.to_container(d)\n\n    obj = collections.OrderedDict()\n\n    def recurse(t, parent_key=\"\"):\n\n        if isinstance(t, list):\n            for i in range(len(t)):\n                recurse(t[i], parent_key + sep + str(i) if parent_key else str(i))\n        elif isinstance(t, dict):\n            for k, v in t.items():\n                recurse(v, parent_key + sep + k if parent_key else k)\n        else:\n            obj[parent_key] = t\n\n    recurse(d)\n    obj = {k: v for k, v in obj.items() if type(v) in [int, float]}\n    # obj = {k: v for k, v in obj.items()}\n\n    return obj\n\n\n\ndef get_training_datasets(cfg: DictConfig) -> dict:\n    \"\"\"\n    Get datases for modelling\n\n    Args:\n        cfg: config\n\n    Returns:\n\n    \"\"\"\n\n    train = pd.read_csv(f'{cfg.data.folder_path}/train.csv')\n\n    train[['x', 'y', 'w', 'h']] = pd.DataFrame(\n        np.stack(train['bbox'].apply(lambda x: ast.literal_eval(x)))).astype(np.float32)\n\n    # precalculate some values\n    train['x1'] = train['x'] + train['w']\n    train['y1'] = train['y'] + train['h']\n    train['area'] = train['w'] * train['h']\n    train_ids, valid_ids = train_test_split(train['image_id'].unique(), test_size=0.1, random_state=cfg.training.seed)\n\n    # for fast training\n    if cfg.training.debug:\n        train_ids = train_ids[:10]\n        valid_ids = valid_ids[:10]\n\n    train_df = train.loc[train['image_id'].isin(train_ids)]\n    valid_df = train.loc[train['image_id'].isin(valid_ids)]\n\n    train_img_dir = f'{cfg.data.folder_path}/train'\n\n    # initialize augmentations\n    train_augs_list = [load_obj(i['class_name'])(**i['params']) for i in cfg['augmentation']['train']['augs']]\n    train_bbox_params = OmegaConf.to_container((cfg['augmentation']['train']['bbox_params']))\n    train_augs = A.Compose(train_augs_list, bbox_params=train_bbox_params)\n\n    valid_augs_list = [load_obj(i['class_name'])(**i['params']) for i in cfg['augmentation']['valid']['augs']]\n    valid_bbox_params = OmegaConf.to_container((cfg['augmentation']['valid']['bbox_params']))\n    valid_augs = A.Compose(valid_augs_list, bbox_params=valid_bbox_params)\n\n    train_dataset = WheatDataset(train_df,\n                                  'train',\n                                  train_img_dir,\n                                  cfg,\n                                  train_augs)\n\n    valid_dataset = WheatDataset(valid_df,\n                                  'valid',\n                                  train_img_dir,\n                                  cfg,\n                                  valid_augs)\n\n    return {'train': train_dataset, 'valid': valid_dataset}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Hydra config\n\n[Hydra](https://hydra.cc/) is a cool framework for working with configs. The basic idea is that you define many small configs separately and can combine them. This way you can easily change some parts of configuration.\n\nIt isn't very convenient to work with it on Kaggle, so I'll define a dictionary and convert in into Hydra format.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg = {'general': {'save_dir': 'logs/', 'workspace': 'erlemar', 'project_name': 'wheat'},\n       'dataset': {'class_name': 'WheatDataset'},\n       'trainer': {'gpus': 1, 'distributed_backend': 'dp', 'accumulate_grad_batches': 1, 'profiler': False, 'max_epochs': 13, 'gradient_clip_val': 0.5,\n                   'num_sanity_val_steps': 0, 'weights_summary': None},\n       'training': {'lr': 0.0001, 'metric': 'main_score', 'seed': 666, 'debug': False, 'mode': 'max'},\n       'logging': {'log': True},\n       'optimizer': {'class_name': 'torch.optim.AdamW', 'params': {'lr': '${training.lr}', 'weight_decay': 0.001}},\n       'scheduler': {'class_name': 'torch.optim.lr_scheduler.ReduceLROnPlateau', 'step': 'epoch', 'monitor': '${training.metric}', 'params': {'mode': '${training.mode}',\n                                                                                                                                              'factor': 0.1, 'patience': 5}},\n       'model': {'backbone': {'class_name': 'torchvision.models.detection.fasterrcnn_resnet50_fpn', 'params': {'pretrained': True}},\n                 'head': {'class_name': 'torchvision.models.detection.faster_rcnn.FastRCNNPredictor', 'params': {'num_classes': 2}}},\n       'callbacks': {'early_stopping': {'class_name': 'pl.callbacks.EarlyStopping', 'params': {'monitor': '${training.metric}', 'patience': 10, 'mode': '${training.mode}'}},\n                     'model_checkpoint': {'class_name': 'pl.callbacks.ModelCheckpoint', 'params': {'monitor': '${training.metric}', 'save_top_k': 3,\n                                                                                                   'filepath': 'saved_models/', 'mode': '${training.mode}'}}},\n       'private': {'comet_api': 'fOmVZaafsPuJ6OP3myaJUd4fC'},\n       'data': {'folder_path': '/kaggle/input/global-wheat-detection', 'num_workers': 0, 'batch_size': 12},\n       'augmentation': {'train': {'augs': [{'class_name': 'albumentations.Flip', 'params': {'p': 0.6}},\n                                           {'class_name': 'albumentations.RandomBrightnessContrast', 'params': {'p': 0.6}},\n                                           {'class_name': 'albumentations.pytorch.transforms.ToTensorV2', 'params': {'p': 1.0}}],\n                                  'bbox_params': {'format': 'pascal_voc', 'label_fields': ['labels']}},\n                        'valid': {'augs': [{'class_name': 'albumentations.pytorch.transforms.ToTensorV2', 'params': {'p': 1.0}}],\n                                  'bbox_params': {'format': 'pascal_voc', 'label_fields': ['labels']}}}}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg = OmegaConf.create(cfg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at our config","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(cfg.pretty())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This way you can see all the settings of our project.\n\nBy the way, you can see that there are keys `class_name` which have the full path to some class, for example `torchvision.models.detection.fasterrcnn_resnet50_fpn`. If you define the classes like this, you can load them later with function `load_obj`, let's see an example:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Preparing the model\n\nWe need to define the backbone net and to change the head. Let's see how this can be done using my config `cfg` and function `load_obj`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = load_obj(cfg.model.backbone.class_name)\nmodel = model(**cfg.model.backbone.params)\n\n# get number of input features for the classifier\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n\nhead = load_obj(cfg.model.head.class_name)\n\n# replace the pre-trained head with a new one\nmodel.roi_heads.box_predictor = head(in_features, cfg.model.head.params.num_classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset\n\nIf you have never worked with object detection, then writing the dataset class could be challenging.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class WheatDataset(Dataset):\n\n    def __init__(self,\n                 dataframe: pd.DataFrame = None,\n                 mode: str = 'train',\n                 image_dir: str = '',\n                 cfg: DictConfig = None,\n                 transforms: Compose = None):\n        \"\"\"\n        Prepare data for wheat competition.\n\n        Args:\n            dataframe: dataframe with image id and bboxes\n            mode: train/val/test\n            cfg: config with parameters\n            image_dir: path to images\n            transforms: albumentations\n        \"\"\"\n        self.image_dir = image_dir\n        self.df = dataframe\n        self.mode = mode\n        self.cfg = cfg\n        self.image_ids = os.listdir(self.image_dir) if self.df is None else self.df['image_id'].unique()\n        self.transforms = transforms\n\n    def __getitem__(self, idx: int):\n        image_id = self.image_ids[idx].split('.')[0]\n        # print(image_id)\n        image = cv2.imread(f'{self.image_dir}/{image_id}.jpg', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n\n        # normalization.\n        # TO DO: refactor preprocessing\n        image /= 255.0\n\n        # test dataset must have some values so that transforms work.\n        target = {'labels': torch.as_tensor([[0]], dtype=torch.float32),\n                  'boxes': torch.as_tensor([[0, 0, 0, 0]], dtype=torch.float32)}\n\n        # for train and valid test create target dict.\n        if self.mode != 'test':\n            image_data = self.df.loc[self.df['image_id'] == image_id]\n            boxes = image_data[['x', 'y', 'x1', 'y1']].values\n\n            areas = image_data['area'].values\n            areas = torch.as_tensor(areas, dtype=torch.float32)\n\n            # there is only one class\n            labels = torch.ones((image_data.shape[0],), dtype=torch.int64)\n            iscrowd = torch.zeros((image_data.shape[0],), dtype=torch.int64)\n\n            target['boxes'] = boxes\n            target['labels'] = labels\n            target['image_id'] = torch.tensor([idx])\n            target['area'] = areas\n            target['iscrowd'] = iscrowd\n\n            if self.transforms:\n                image_dict = {\n                    'image': image,\n                    'bboxes': target['boxes'],\n                    'labels': labels\n                }\n                image_dict = self.transforms(**image_dict)\n                image = image_dict['image']\n                target['boxes'] = torch.as_tensor(image_dict['bboxes'], dtype=torch.float32)\n\n        else:\n            image_dict = {\n                'image': image,\n                'bboxes': target['boxes'],\n                'labels': target['labels']\n            }\n            image = self.transforms(**image_dict)['image']\n\n        return image, target, image_id\n\n    def __len__(self) -> int:\n        return len(self.image_ids)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(f'{cfg.data.folder_path}/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In each row we have image name, width and height of the image, bbox coordinates and the source (I suppose annotator).\n\nLet's work with the data:\n* extract values of bbox into separate columns;\n* calculate top right coordinates of the bboxes;\n* calculate area;","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[['x', 'y', 'w', 'h']] = pd.DataFrame(\n    np.stack(train['bbox'].apply(lambda x: ast.literal_eval(x)))).astype(np.float32)\n\n# precalculate some values\ntrain['x1'] = train['x'] + train['w']\ntrain['y1'] = train['y'] + train['h']\ntrain['area'] = train['w'] * train['h']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"There are {train['image_id'].nunique()} unique images in train data\")\nprint(f\"There are {train.shape[0]} bboxes in train data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(train['area'], bins=20)\nplt.title('Area value distribution');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(50, 100, 10):\n    perc = np.percentile(train['area'], i)\n    print(f\"{i} percentile of area is {perc}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that most bboxes are quite small, but there are some really huge bboxes. It would be interesting to look at them","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_image_bboxes(img_id: str = '',\n                      df: pd.DataFrame = None):\n    # based on Peter's kernel\n    image = cv2.imread(f'/kaggle/input/global-wheat-detection/train/{img_id}.jpg', cv2.IMREAD_COLOR)\n    # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n    bboxes = df.loc[df['image_id'] == img_id, ['x', 'y1', 'x1', 'y']].values\n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n    for box in bboxes:\n        cv2.rectangle(image,\n                      (box[0], box[1]),\n                      (box[2], box[3]),\n                      (220, 0, 0), 3)\n    ax.set_axis_off()\n    ax.imshow(image / 255)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image_bboxes('b6ab77fd7', train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image_bboxes('42e6efaaa', train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow! There is a huge bbox on this image. I wonder whether this is normal or not.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_augs_list = [load_obj(i['class_name'])(**i['params']) for i in cfg['augmentation']['valid']['augs']]\nvalid_bbox_params = OmegaConf.to_container((cfg['augmentation']['valid']['bbox_params']))\nvalid_augs = A.Compose(valid_augs_list, bbox_params=valid_bbox_params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_img_dir = f'{cfg.data.folder_path}/test'\n\ntest_dataset = WheatDataset(None,\n                             'test',\n                             test_img_dir,\n                             cfg,\n                             valid_augs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Lightning class\n\nTo use pytorch-lightning, we need to define a main class, which has the following parts:\n* self.hparams. This is optional parameter, but it better to use it - it is a dictionary with hyperparameters;\n* forward method - making predictions with the model. The model itself can be defined outside this class;\n* prepare data - preparing datasets;\n* train_dataloader, val_dataloader - these methods should return the relevant dataloaders;\n* configure configure_optimizers - should return lists of optimizers and schedulers;\n* training_step - define what happend inside the train loop;\n* validation_step - define what happend inside the validation loop;\n* validation_epoch_end - define what happend at epoch end;","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Please, note that some lines with `coco_evaluator` are commented out. This is because it doesn't work with numpy 1.18 - only with 1.17. And I can't downgrade the version.\nI'll try to think what to do with this later.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class LitWheat(pl.LightningModule):\n\n    def __init__(self, hparams: DictConfig = None, cfg: DictConfig = None, model = None):\n        super(LitWheat, self).__init__()\n        self.cfg = cfg\n        self.hparams = hparams\n        self.model = model\n\n    def forward(self, x, *args, **kwargs):\n        return self.model(x)\n\n    def prepare_data(self):\n        datasets = get_training_datasets(self.cfg)\n        self.train_dataset = datasets['train']\n        self.valid_dataset = datasets['valid']\n\n    def train_dataloader(self):\n        train_loader = torch.utils.data.DataLoader(self.train_dataset,\n                                                   batch_size=self.cfg.data.batch_size,\n                                                   num_workers=self.cfg.data.num_workers,\n                                                   shuffle=True,\n                                                   collate_fn=collate_fn)\n        return train_loader\n\n    def val_dataloader(self):\n        valid_loader = torch.utils.data.DataLoader(self.valid_dataset,\n                                                   batch_size=self.cfg.data.batch_size,\n                                                   num_workers=self.cfg.data.num_workers,\n                                                   shuffle=False,\n                                                   collate_fn=collate_fn)\n\n        # prepare coco evaluator\n#         coco = get_coco_api_from_dataset(valid_loader.dataset)\n#         iou_types = _get_iou_types(self.model)\n#         self.coco_evaluator = CocoEvaluator(coco, iou_types)\n\n        return valid_loader\n\n    def configure_optimizers(self):\n        if 'decoder_lr' in self.cfg.optimizer.params.keys():\n            params = [\n                {'params': self.model.decoder.parameters(), 'lr': self.cfg.optimizer.params.lr},\n                {'params': self.model.encoder.parameters(), 'lr': self.cfg.optimizer.params.decoder_lr},\n            ]\n            optimizer = load_obj(self.cfg.optimizer.class_name)(params)\n\n        else:\n            optimizer = load_obj(self.cfg.optimizer.class_name)(self.model.parameters(), **self.cfg.optimizer.params)\n        scheduler = load_obj(self.cfg.scheduler.class_name)(optimizer, **self.cfg.scheduler.params)\n\n        return [optimizer], [{\"scheduler\": scheduler,\n                              \"interval\": self.cfg.scheduler.step,\n                              'monitor': self.cfg.scheduler.monitor}]\n\n    def training_step(self, batch, batch_idx):\n        images, targets, image_ids = batch\n        targets = [{k: v for k, v in t.items()} for t in targets]\n        # separate losses\n        loss_dict = self.model(images, targets)\n        # total loss\n        losses = sum(loss for loss in loss_dict.values())\n\n        return {'loss': losses, 'log': loss_dict, 'progress_bar': loss_dict}\n\n    def validation_step(self, batch, batch_idx):\n        images, targets, image_ids = batch\n        targets = [{k: v for k, v in t.items()} for t in targets]\n        outputs = self.model(images, targets)\n        res = {target[\"image_id\"].item(): output for target, output in zip(targets, outputs)}\n#         self.coco_evaluator.update(res)\n\n        return {}\n\n    def validation_epoch_end(self, outputs):\n#         self.coco_evaluator.accumulate()\n#         self.coco_evaluator.summarize()\n#         # coco main metric\n#         metric = self.coco_evaluator.coco_eval['bbox'].stats[0]\n        metric = 0\n        tensorboard_logs = {'main_score': metric}\n        return {'val_loss': metric, 'log': tensorboard_logs, 'progress_bar': tensorboard_logs}\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing for training\n\nThere are several steps left before we can start training:\n* setting random seed\n* initializing the lightning class\n* defining callbacks and loggers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(cfg.training.seed)\nhparams = flatten_omegaconf(cfg)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"lit_model = LitWheat(hparams=hparams, cfg=cfg, model=model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"early_stopping = pl.callbacks.EarlyStopping(**cfg.callbacks.early_stopping.params)\nmodel_checkpoint = pl.callbacks.ModelCheckpoint(**cfg.callbacks.model_checkpoint.params)\n\ntb_logger = TensorBoardLogger(save_dir=cfg.general.save_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## training\n\nTraining is done using special `Trainer` class","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer = pl.Trainer(logger=[tb_logger],\n                     early_stop_callback=early_stopping,\n                     checkpoint_callback=model_checkpoint,\n                     **cfg.trainer)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer.fit(lit_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference\n\nInference will be done in another kernel. Still, below you can see how could it be done in this kernel.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_loader = torch.utils.data.DataLoader(test_dataset,\n                                          batch_size=cfg.data.batch_size,\n                                          num_workers=cfg.data.num_workers,\n                                          shuffle=False,\n                                          collate_fn=collate_fn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def format_prediction_string(boxes, scores):\n    pred_strings = []\n    for s, b in zip(scores, boxes.astype(int)):\n        pred_strings.append(f'{s:.4f} {b[0]} {b[1]} {b[2] - b[0]} {b[3] - b[1]}')\n\n    return \" \".join(pred_strings)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"detection_threshold = 0.5\nresults = []\ndevice = 'cuda'\nmodel.eval()\nfor images, _, image_ids in test_loader:\n\n    images = list(image.to(device) for image in images)\n    outputs = lit_model(images)\n\n    for i, image in enumerate(images):\n\n        boxes = outputs[i]['boxes'].data.cpu().numpy()\n        scores = outputs[i]['scores'].data.cpu().numpy()\n        \n        boxes = boxes[scores >= detection_threshold].astype(np.int32)\n        scores = scores[scores >= detection_threshold]\n        image_id = image_ids[i]\n        \n        result = {\n            'image_id': image_id,\n            'PredictionString': format_prediction_string(boxes, scores)\n        }\n\n        \n        results.append(result)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame(results, columns=['image_id', 'PredictionString'])\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(lit_model.model.state_dict(), 'fasterrcnn_resnet50_fpn.pth')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}