{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":19989,"databundleVersionId":1160143,"sourceType":"competition"},{"sourceId":1084583,"sourceType":"datasetVersion","datasetId":604593}],"dockerImageVersionId":29963,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# Table of Content","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"![efficientdet_cover](attachment:object_detection_101_new.png)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Welcome to this beginner friendly guide to **object detection** using **EfficientDet**. Similarly to what I have done in the NLP guide (check it [here](https://www.kaggle.com/yassinealouini/roberta-meets-tpus) if you haven't yet already), there \nwill be a mix of **theory**, **practice**, and an **application** to the [global wheat competition dataset](https://www.kaggle.com/c/global-wheat-detection). \n\n\n\nThis will be a very long notebook, so use the following table of content if necessary. Grab something to drink and enjoy!\n\n- [Introduction](#intro)\n- [Modern Computer Vision](#modern-cv) \n * [Different Detection Tasks](#different-tasks)\n * [Object Detection 101](#od-101)\n * [Object Detection Models Zoology](#od-zoology)\n- [EfficientDet Explained](#efficientdet)\n * [EfficientNet as a Backbone](#\"effnet-backbone\")\n * [BiFPN as a Multi-Scale Fusion](#bifpn)\n * [Two Heads](#heads)\n * [Compound Scalings](#scalings)\n- [Application: Global Wheat Detection](#application)\n * [Understanding the Task](#understanding)\n * [Evaluation Metric](#evaluation-metric)\n * [Data processing](#processing)\n * [The Model](#model)\n * [The Dataset](#dataset)\n * [Pytorch Complete Training Pipeline](#pipeline)\n * [Evaluation](#evaluation)\n * [Inference](#inference)\n \n- [Advanced Concepts](#advanced)\n * [Mosaic Augmentation](#augmentation)\n * [TTA](#tta)\n * [WBF](#wbf)\n \n- [To Wrap up](#wrap-up)\n- [To Go Beyond](#going-beyond)\n\n\nOne last thing before I start, I have benefited immensely from the following two notebooks: \n\n- A Pytorch Lightning end-to-end training [pipeline](https://www.kaggle.com/artgor/object-detection-with-pytorch-lightning) by the great [Andrew Lukyanenko](https://www.kaggle.com/artgor). There is a Github [repo](https://github.com/Erlemar/wheat) as well if you want better organised code. If youy don't know who Andrew \"artgor\" is, now is the time to discover lots of cool notebooks. ;)\n\n- A Pytorch (no Lightning this time) end-to-end training [pipeline](https://www.kaggle.com/shonenkov/training-efficientdet) by the great [Alex Shonenkov](https://www.kaggle.com/shonenkov). Check his work, he has many other great CV notebooks.\n\nWith these shoutouts out of the way, let's start.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Introduction ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Computer vision went through a fast cycle of innovations and improvements starting in **2012** with \nthe [AlexNet](https://en.wikipedia.org/wiki/AlexNet) network revolution.\n\nIndeed, this was according to many one of the moments that launched again the field of deep learning from the last [\"AI winter\"](https://en.wikipedia.org/wiki/AI_winter): for the first time, it was possible to train neural networks unsing **large dataset**s and **lots of compute** (well according to 2012's standards anyway :p).\n\nThis is what made the model achive state of the art performance on [ImageNet](http://www.image-net.org/). Indeed, it has achieved in 2012 a **top-5 error of 15.3%** which was 10% better than pervious year's performance (the lower, the better). \n\nSo, what does this network contain?\n\nThe **AlexNet** network is quite simple: different **[CNN](https://cs231n.github.io/convolutional-networks/)** layers followed by max-pooling then a fully-connected layer. \n\nFast forward to 2020, a lot of things happened and the performances kept improving year after year: ImageNet is now considered a \"solved\" (kind of in most cases at least) dataset. In fact, it has been solved since at least **2015** with the introduction of the [**ResNet**](https://en.wikipedia.org/wiki/Residual_neural_network) model, i.e. getting performance better than human-level for the first time (see the graph below). \n\n\n\n\n\n\nSource: https://semiengineering.com/new-vision-technologies-for-real-world-applications/\n\nTo read more about what happened over the last years in the field of deep learning, check the following [blog post](https://adeshpande3.github.io/adeshpande3.github.io/The-Last-5-Years-in-Deep-Learning)\n\nMost of these systems share similar architectures and modern \"tips\" to make them work best.\n\nIn what follows, I will detail some of what makes the modern computer vision ecosystem.\n\nHere we go!","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Modern Computer Vision ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Before focusing on **object detection**, let's move one step back and explore the modern computer vision landscape. \n\nThe first step was making CNNs work: that's roughly what happened around 2012 and has been improved ever since as discussed in the previous section. \n\nThen, the idea was to make the networks deeper to get better performances. Two main problems appeared then: \n\n1. networks got too many parameters\n2. convergence wasn't fast or happening at all\n\nTo solve these issues, few interesting ideas were introduced. As stated by Karpathy in one talk (haven't found the link yet, if you know it, please share in the comments): \"more zeros, better performance\". What does that mean? In short, more 0s everywhere in the network: \n\n- Max-pooling instead of average pooling\n- ReLU layer\n- Dropout \n\nThus less parameters.\n\nSomething else happened: **code discovery**, **sharing**, and **reproducibility** got better. I don't have \nany data to back this claim but that's what I have observed through experience. For instance, [papers with code](https://paperswithcode.com/) is probably the best place to look for a code implementation.\n\nAlso, the **code ecosystem** got much better. Among the recent code libraries and tools: \n\n- [Pytorch](https://pytorch.org/) and the [torchvision](https://pytorch.org/docs/stable/torchvision/index.html) library particularly for computer vision tasks\n- [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) for easier training\n- [Fastai](https://www.fast.ai/) and its CV module for an intrigated experience with latest CV best practices.\n\nFinally, some of the recent research trends:\n\n- more efficient architectures, i.e. even less parameters. More on this in the following sections.\n- automated neural architecture search\n- transfer learning\n- application of attention ideas from the recent NLP progress. Check this blog [post](https://towardsdatascience.com/self-attention-in-computer-vision-2782727021f6) for some details.\n\nAll these things make the modern-days computer vision ecosystem. Let's now move to the subfield of **object detection**.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Different Detection Tasks","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"We will mainly focus on **three** similar **detection tasks** (from simpler to more complex):\n \n- object detection\n- semantic segmentation\n- instance segmentation\n\nObject detection is the easiest one. It consists in placing bounding boxes around detected objects. The two remaining tasks are best described in the image below:\n\n\n\nSource: https://datascience.stackexchange.com/questions/52015/what-is-the-difference-between-semantic-segmentation-object-detection-and-insta\n\n\nAll three tasks share the following thing: given an **object** the aim is to **locate** some pixels that identify it: \n\n- for object detection, 4 numbers are predicted (either two corners or center and width and height, check the figure below for more details) and a label's probability. \n- for semantic segmentation, the countours of each label are predicted (more numbers), i.e. for each pixel predict to which class it belongs.\n- for instance segmentation, the instances of each label are predicted (much more numnbers), i.e. for each pixel predict to which class it belongs + its id.\n\n\n![object_detection_bbox.png](attachment:object_detection_bbox.png)\n\n\nThe two most popular ways to annotate a bounding box. Left for the reader: how to move from one to the other?\n\nNow that we are more familiar with the general detection tasks, we can move to **object detection**. ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Object Detection 101","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"What is object detection?\n\nFor humans, this is a straightforward question: given an **image** and a **label**, draw a **bounding box** around the detected objects. In other words, it is localization plus classification of objects.\n\nHowever, translating this task into an algorithm is on another level of hard. \nIndeed, where do you even start?\n\n\n![object_detection_explained.png](attachment:object_detection_explained.png)\n\n\nLegend: on top, human perspective. Bottom, computer perspective.\n\n\nI also like this definition from [paperswithcode](https://paperswithcode.com/task/object-detection/latest): \n\n> Object detection is the task of detecting instances of objects of a certain class within an image. The state-of-the-art methods can be categorized into two main types: one-stage methods and two stage-methods. One-stage methods prioritize inference speed, and example models include YOLO, SSD and RetinaNet. Two-stage methods prioritize detection accuracy, and example models include Faster R-CNN, Mask R-CNN and Cascade R-CNN.\nThe most popular benchmark is the MSCOCO dataset. Models are typically evaluated according to a Mean Average Precision metric.\n\n\n\nFortunately, we don't need to really understand what goes inside our brains. It appears that \ndeep learning models are very good at that. Indeed, given the latest [ImageNet](http://www.image-net.org/) and [COCO](https://cocodataset.org/#home) benchmarks, deep learning models are beating human-level performance: that was the case on the ImageNet dataset since 2015 and that's the case for the COCO dataset since . \n\nSo, the next question is: how to design a good neural network for object detection?\n\n\nLet's have a look at the most common techniques to have a feel for what works. From Wikipedia's [object detection](https://en.wikipedia.org/wiki/Object_detection) page: \n \n* Region Proposals (R-CNN,Fast R-CNN, Faster R-CNN, cascade R-CNN, and so on)\n* Single Shot MultiBox Detector (SSD) \n* You Only Look Once (YOLO) \n* Single-Shot Refinement Neural Network for Object Detection (RefineDet)\n* Retina-Net \n* Deformable convolutional networks\n\nWe can also add **EfficientDet** ;). Well this list isn't very informative, is it? \nBased on my own experience (basically reading a lot of blog posts and papers), there are **two** major families: \n\n- Two steps networks\n- One step networks\n\nR-CNN and its \"cousins\" fall into the first family of models. \nIn contrast to the R-CNN family, SSD, YOLO, RetinaNet, CeneterNet and EfficientDet fall into the faster, one step family. \n\n\nFinally, to learn more about these two families of models, check this [blog post](https://cv-tricks.com/object-detection/faster-r-cnn-yolo-ssd/) and this [slides deck](https://fr.slideshare.net/pfi/a-brief-history-of-object-detection-tommi-kerola) telling a brief history of object detection. \n\nLet's explore further this rich ecosystem of models.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Object Detection Models Zoology","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"As stated above, it seems that deep learning models, particularly convolutional ones are very good at image processing tasks, and in particular object detection. We won't explore why this is the case so feel free to explore this subject on your own. One possible explanation is that deep neural networks are good at distilling features from noisy inputs and this can be explained by the \n[information bottleneck theory](https://en.wikipedia.org/wiki/Information_bottleneck_method). \n\nInstead, let's explore some SOTA (state-of-the-art) models and see what they have in common: \n\n\n* **SSD**: [paper](https://arxiv.org/pdf/1512.02325.pdf) and pytorch [implementation]. (https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/). Notice that SSD stands for \"Single Shot MultiBox Detector\".\n\n* **FPN**: [paper](https://openaccess.thecvf.com/content_cvpr_2017/papers/Lin_Feature_Pyramid_Networks_CVPR_2017_paper.pdf) and pytorch [implementation](https://github.com/jwyang/fpn.pytorch). Notice that FPN stands for \"Feature Pyramid Networks\". \n\n* **RetinaNET**: [paper](https://arxiv.org/pdf/1708.02002.pdf) and pytorch [implementation](https://github.com/yhenon/pytorch-retinanet). \n\n* **Mask RCNN**: [paper](https://arxiv.org/pdf/1703.06870.pdf) and pytorch [tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) on how to fine-tune it. Notice that this model is a generalization of [Faster RCNN](https://arxiv.org/abs/1506.01497) that adds instance segmentation on top of object detection. \n\n* **CenterNet**: [paper](https://arxiv.org/pdf/1904.08189.pdf) and pytorch [implementation](https://github.com/xingyizhou/CenterNet).\n\n* **YOLO**: [website](https://pjreddie.com/darknet/yolo/) and v3 [paper](https://pjreddie.com/media/files/papers/YOLOv3.pdf). As the website claims, it is 100 times faster than Mask RCNN. There is even a v5 [now](https://github.com/ultralytics/yolov5). Notice that YOLO stands for \"You Only Look Once\". \n\n* **EfficientDet**: [paper](https://arxiv.org/pdf/1911.09070.pdf) and pytorch [implementation](https://github.com/rwightman/efficientdet-pytorch). \n\n* **DETR**: model detection using the transformer architecture. [paper](https://arxiv.org/abs/2005.12872) and pytorch [implementation](https://github.com/facebookresearch/detr). \n\n\n\nTo dig deeper, here are some more details:\n\n\n* **SSD**\n\n * SSD is short for single shot detector\n * As it is clear from its name, it belonges to the one step family\n * Paper accepted around 2016 (end of)\n \n \n \n \n* **FPN**\n\n * FPN is short for feature pyarmic network\n * This isn't an object detection network per say but rather a part\n * It is used in many architecture as the features extraction part\n * Paper accepted in 2017\n\n* **Mask R-CNN**\n\n * From the region proposals series of models, this is the latest one\n * It is an extension of faster r-cnn for instance segmentation\n * It seems that it gives better results even for object detection\n * To learn more, read [this](https://medium.com/@jonathan_hui/what-do-we-learn-from-region-based-object-detectors-faster-r-cnn-r-fcn-fpn-7e354377a7c9) blog post and [this](https://paperswithcode.com/method/mask-r-cnn) paperswithcode page.\n\n\n* **RetinaNet**\n\n * Uses focal loss\n \n \n \n * Uses FPN as its features extraction layer \n \n \n\n* **CenterNet**\n\n * Paper accepted in 2019\n * Estimates the center of the bounding box in addition to the corners\n * Belongs to the one stage family of models \n * Claims to have the best performance for the one stage family and not very far from the best in two stages as well\n * Has two networks: one for finding the center and one for finding the corners\n * As usual, to learn more check the paperswithcode [page](https://paperswithcode.com/method/centernet)\n \n \n\n* **YOLO**\n\n * Uses a different format for the outputted bounding box: the center + width and height instead of top-left and bottom-right corners. \n * Among the one stage family of models\n * Fast predictions with great mAP\n * Latest versions use FPN as a features extracting layer\n * Has been through many iterations (latest being v4). If you want a very complete blog post, check this [one](https://medium.com/@jonathan_hui/yolov4-c9901eaa8e61) and the [paperswithcode](https://paperswithcode.com/method/yolov4) page. \n * V4 paper accepted in 2020\n \n \n\n\n\n* **EfficientDet**\n\n * Originally developed by the google brain team using architecture search. \n * The original accompanying code could be found [here](https://github.com/google/automl/tree/master/efficientdet) \n * A great [notebook](https://www.kaggle.com/shonenkov/training-efficientdet/notebook) to get started training an EfficientDet model on this competition dataset. Thanks to [Alex Shonenkov](https://www.kaggle.com/shonenkov). \n * To get the backbon weights, go to the [README](https://github.com/rwightman/efficientdet-pytorch/blob/master/README.md#models) page, models section, where you can find this table and click on the weights you want\n\n \n \n * COCO AP (average precision) vs FLOPS (number of operations per second) graph:\n\n \n\n\n * SOTA according to [paperwithcode](https://paperswithcode.com/paper/efficientdet-scalable-and-efficient-object). Here is graph that shows that EfficientDet-D7x (largest model) achieves it on the COCO object detection benchmark:\n \n ![coco_object_detection_score.png](attachment:coco_object_detection_score.png)\n \n\n* **DETR** \n\n * DETR is short for detection with transformer\n * Paper accepted in 2020\n * Uses a transformer layer after the feature extraction one\n * Developed by FAIR\n * To learn more, check the paperwithcodes [page](https://paperswithcode.com/method/detr)\n \n \n\n\nThat was a lot of new concepts to groak, so as a reward, here is an approximate timeline: in red the one step models, and green the two steps ones. Notice that FPN doesn't have a category since it is used in both:\n\n\n\n\n\nThat's enough general architecture for now!\nIn the remaining parts, we will be focusing on the **EfficienDet** family of models, more specifically on the following Pytorch [implementation](https://github.com/google/automl/tree/master/efficientdet).","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n# EfficientDet Explained\n\nWhat is **EfficientDet** and how does it work?\n\n\nIn short, it is a recent (first submitted at the end of 2019, accepted in CVPR in 2020) very efficient (surprise surprise) object detection model.\n\nIn more details, it is a family of models designed by researchers from **Google brain**. The interesting thing is that some parts of it were designed using automatic **architecture search**, i.e. there was a meta-model that was trained to find the best hyper-parameters of the trained model automatically. Let's dive in more details.\n\n\n\nFirst, let's start with the model's **architecture**: \n\n\n\n\nAs you can see, there are **3 main** building blocks:\n\n1. a **classification** backbone block\n2. a **multi-scale features fusion** block\n3. a **class** and **bounding box** networks block\n\nIn more details, we have: \n\n* an **EfficientNet** (**Net** and not **Det**) backbone. This backbone was found using NAS (neural-architecture search) through the use of the MnasNet framework. More on this in the next section\n* **BiFPN** as a multi-scale fusion layer: once we have a classification network we need to efficiently extract features for the detection step. One extra challenge is to do so for multiple scales (learn more about this [here](https://d2l.ai/chapter_computer-vision/multiscale-object-detection.html)). These two challenges are solved efficiently using the BiFPN layer. Again, more on this in the next section.\n* **Two heads**: finally, everything is connected to two output networks. One for predicting the most likely label and one for predicting the detected bounding box (if any).\n\n\nTo experiment with the model and learn more, check this [colab notebook](https://github.com/google/automl/blob/master/efficientdet/tutorial.ipynb), the original [paper](https://arxiv.org/pdf/1911.09070.pdf), and the original [implementation](https://github.com/google/automl/tree/master/efficientdet). \n\n\nIn the following sections, we will explore each part in more details. Let's go!\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n## EfficientNet as a Backbone\n\nThis is the first part of the archtiecture. It uses the **EfficientNet** architecture and includes pre-trained weights.\n\nHere is a look at the overall archtitecture:\n\n\n\n\n\nAs you can see, the interesting thing is that many elements of the model have been optimized using neural automatic search: **width**, **depth**, **number of channels**, and **resolution**.\n\nTo learn more about this **EfficientNet** backbone, read the original paper [here](https://arxiv.org/pdf/1905.11946.pdf), the blog [post](https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html), and the implementation [here](https://github.com/google/automl/tree/master/efficientdet/backbone) (from the **efficientdet** repo).","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n## BiFPN as a Multi-Scale Fusion Layer\n\nOnce we have a classification network, one important question to ask is: how to move from the **classification** model to an **object detection** one? This is cleverly solved by using features maps (i.e. intermediate learned representations). Another question is how to detect multiple objects having different sizes (for example a small cat and a car on the same image)? This is solved by taking features maps having different sizes. \n\nIn most modern object detection architectures, both problems are solved at once using an FPN layer. BiFPN, as its name indicates, builds upon the ideas of the FPN layer and adds one simple idea: instead of simply aggregating the different representations in a top-down fashion, it takes the PANet approach (i.e. top-down and bottom-up connections) and optimizes cross-scale connections.\n\nOne last trick is to use a weighted feature fusion instead of the unweighted FPN approach (the weights are learned as well) and normalize these weights using a fast normalization procedure (i.e. normalize by the sum instead of using softmax). Here is one example of a BiFPN computation from the paper:\n\n\n\n\n\nAll these tricks make the BiFPN layer both efficient and accurate. We will see one additional trick applied in the next and last section. Check the following graph to learn more about other FPN variations (including PANet and BiFPN):\n\n\n\n\n\n\nFinally, to learn more about FPN and BiFPN, check the papeswithcode [page](https://paperswithcode.com/method/bifpn), this [blog post](https://towardsdatascience.com/review-fpn-feature-pyramid-network-object-detection-262fc7482610), and this [one](https://medium.com/@jonathan_hui/understanding-feature-pyramid-networks-for-object-detection-fpn-45b227b9106c) as well. \n\nLet's move to the next step: the prediction networks.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Two Heads\n\nThe third ingredient to the mix is the **two heads network**:\n\n- one for predicting the coordinates of the **bounding box**\n- one for predicting the **class** (the probability of each class to be more precise)\n\nEach of the two networks take as input all the ouptuts of the previous **BiFPN layer**. Then we have two fully-connected layers before getting the final values. As simple as that!\n\nWe close this section with the last trick: compound scaling.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Compound Scaling\n\n\nOne last ingredient to the mix is how the network is scaled. Indeed, this is a trick often used to improve the performance \nof networks. The most common way to do it is to scale the backbone's width and depth. \n\nIn the EfficientDet model, the EffincientNet backbone is scaled reusing the same values in the original network.\nThe interesting additional thing is the scaling of the BiFPN and prediction networks.\nIn fact, instead of using only one BiFPN and one prediction networks, these components are repeated. Finally, width ($W$), depth ($D$), and input resolution ($R$) in the different sections are scaled usign one single parammeter $\\phi$ in this fashion:\n\n\n\n- 1. $W_{bifpn} = 64 . (1.35^{\\phi})$ and $D_{bifpn} = 3 + \\phi$ for the **BiFPN** part\n- 2. $W_{pred} = W_{bifpn}$ and $D_{bifpn} = 3 + \\phi$ for the prediction part\n- 3. $R_{input} = 512 + \\phi . 128$ for the input image resolution. Notice that it should be divisible by 128 since we take features from levels 3 to 7 ($128 = 2 ^{7}$).\n\n\nHere is the table for the different **EfficientDet** variants: the bigger $\\phi$ is, the bigger the model:\n \n\n\n\nTo finish this section, notice that the $1.35$ value in the first equation has been optimized using grid search (i.e. different values are tested and the one giving the best score is selected) over ${1.2, 1.25, 1.3, 1.35, 1.4, 1.45}$ values.\n\n\nThat's it for the model's architecture. Time to move to the application!","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Application to Global Wheat Challenge","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n\nNow that the theory makes (hopefully) more sense, let's move to the application side of things.\nIn the above figure, you can see a patch of wheat heads.\n\nThat's the competition's banner and that's what we are predicting: finding the bounding boxes of wheat heads.\n\nAs stated in the introduction, we will be using the **EfficientDet** model introduced above. \n\nHere is a rough sketch of the plan:\n\n1. Understand the dataset, the task, and the evaluation metric\n2. Process the data so that it can be used with an efficientdet model\n3. Train the model \n4. Predict on test images","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Understanding the Task\n\nAs you have guessed it, the task of this competition is to predict the bounding boxes of wheat heads in different images.\nThe images have a varying number of wheat heads, colors, orientations, and so on making the task more challenging.\n\nTo further understand the task, let's explore the wheat images and the associated bounding boxes: \n\n* Train images: \n\n * There is a toal of **3422** unique train images. The code to get this number:\n \n ``` python \n from pathlib import Path\n train_paths = Path(\"/path/to/train\")\n len(list(train_paths.glob(\"*\")))\n ``` \n \n * There is a total of **3373** unique images with masks\n * Thus some images don't have any masks\n * There is a toal of **147793** masks\n * Thus, on average, there are **43.8** masks per image\n * The image with the most masks contains **116**. It is the image with id `35b935b6c`.\n * All the train images have the same size: 1024 x 1024. \n * There are 3 channels: R, G, B.\n * The train images come from **7** different sources. Here is the distribution: \n\n ``` python\n {'arvalis_1': 1055,\n 'arvalis_2': 204,\n 'arvalis_3': 559,\n 'ethz_1': 747,\n 'inrae_1': 176,\n 'rres_1': 432,\n 'usask_1': 200}\n ``` \n \n * **3** of these sources are probably similar: *arvalis_1*, *arvalis_2*, and *arvalis_3*. Most likely, the breakdown is by the competition sponsors/organisers?\n * Bounding boxes where added using this [tool](https://github.com/jsbroks/coco-annotator).\n * You can find resized datasets [here](https://www.kaggle.com/c/global-wheat-detection/discussion/160408).\n * There are some data quality issues: masks [too big](https://www.kaggle.com/c/global-wheat-detection/discussion/159578), or [missing masks](https://www.kaggle.com/c/global-wheat-detection/discussion/163263). The organizers are working on a fix for the private test dataset (as far as I know).\n * The average bounding box area is **6843**, the smallest is only **2**, and the largest is **529788**! Here attached the two images with the largest (upper right including many smaller) and the smallest (it is the in the lower right corner, at the edge) bounding boxes: \n \n\n \n\n * The bbox area histogram with the top 1000 values truncated and the bottom 100 as well:\n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F172860%2F6397d02cd4ada74403ae0eaa955394ce%2Fhist_truncated.png?generation=1594242665230885&alt=media)\n\n * For more details about the dataset creation, check this [paper](https://arxiv.org/pdf/2005.02162.pdf). It contains a lot of statistics and data acquisition methodolgy descritption, so make sure to check it. Here are some graphs from the paper: \n \n \n \n\n\n\n* Test images:\n\n * Only **10** images have been made public. The code to get this: \n \n ``` python \n from pathlib import Path\n test_paths = Path(\"/path/to/test\")\n len(list(test_paths.glob(\"*\")))\n ```\n \n * Don't worry though, there is a \"real\" test dataset that runs when you submit your inference kernel.\n * The test dataset has been collected from different sources than the train. This is an important aspect of the competition for sure.\n * There are few images in the test dataset that aren't 1024 x 1024. These will be fixed in the private part as far as I know.\n * More insights about the test dataset could be found in [this](https://www.kaggle.com/c/global-wheat-detection/discussion/149770) discussion. \n\n* Short code snippets: \n\n\n * Load an image and overlay the corresponding mask(s): \n \n ``` python \n from pathlib import Path\n\n img_id = \"ID_OF_IMAGE\"\n img_path = Path(f\"/path/to/train/{img_id}.jpg\")\n img = Image.open(img_path)\n draw = ImageDraw.Draw(img)\n bboxes = train_labels_df.loc[lambda df: df[\"image_id\"] == img_id, \"bbox\"].tolist()\n # The box contains the upper left corner (x, y) coordinates then width and height.\n # So we need to change these to (x1, y1) and (x2, y2) where they are the upper \n # left and lower right corners\n for bbox in bboxes:\n x, y, w, h = eval(bbox)\n transformed_bbox = [x, y, x + w, y + h]\n draw.rectangle(transformed_bbox, outline=\"black\", width=3)\n img.save(\"/path/to/output/{img_id}.png\")\n img\n ```\n \n \n\n * Extracting bounding boxes coordinates, width, height, and area: \n \n ``` python \n import pandas as pd\n train_labels_df = pd.read_csv(\"path/to/train/labels.csv\")\n\n x = []\n y = []\n w = []\n h = []\n for bbox in train_labels_df['bbox']:\n bbox = eval(bbox)\n x.append(bbox[0])\n y.append(bbox[1])\n w.append(bbox[2])\n h.append(bbox[3])\n\n processed_train_labels_df = train_labels_df.copy()\n processed_train_labels_df[\"x\"] = x\n processed_train_labels_df[\"y\"] = y\n processed_train_labels_df[\"w\"] = w\n processed_train_labels_df[\"h\"] = h\n\n processed_train_labels_df[\"area\"] = processed_train_labels_df[\"w\"] * processed_train_labels_df[\"h\"]\n processed_train_labels_df[\"x2\"] = processed_train_labels_df[\"x\"] + processed_train_labels_df[\"w\"]\n processed_train_labels_df[\"y2\"] = processed_train_labels_df[\"y\"] + processed_train_labels_df[\"h\"]\n ```\n\n\n* Metric\n\n * There are many components to this metric: first, we compute Jaccard/[IoU](https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/) scores, then for some threshold `t`, the wheat head is a TP if the IoU is above it, finally we compute the [precision](https://en.wikipedia.org/wiki/Precision_and_recall) for this threshold. Then, average over the thresholds (given the AP) and over images (given mAP, mean average prediction).\n * Thus, there are two levels of averaging: over the thresholds and then over the images. \n * Notice that there are some edge-cases where the precision will be 0: no mask is predicted and there is at least one, a mask is predicted but there aren't any.\n\n* Some backbone models that work well for object detection: \n\n * [YOLOv5](https://github.com/ultralytics/yolov5)\n * [Detectron2](https://github.com/facebookresearch/detectron2): this isn't a single model but rather a collection of object detection models. It contains Faster R-CNN, RetinaNet and many more. \n\n* Previous similar competitions:\n\n * [TGS Salt Identification](https://www.kaggle.com/c/tgs-salt-identification-challenge/overview). This is an object detection competition with exactly the same metric. Here is a link to the [first place](https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/69291) solution, the code [repo](https://github.com/ybabakhin/kaggle_salt_bes_phalanx), and a [paper](https://arxiv.org/pdf/1904.04445.pdf) published.\n * [Severstal: Steel Defect Detection](https://www.kaggle.com/c/severstal-steel-defect-detection/). This isn't exactly an object detection competition but rather an **semantic segmentation** one. Here is a link the [first place](https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/114254) solution. \n * [Airbus Ship Detection](https://www.kaggle.com/c/airbus-ship-detection). Another instance segmentation competition. Here is the 4th place solution. The metric here is close to the current competition but instead of AP, it uses [F2](https://www.quora.com/What-is-the-F2-score-in-machine-learning) score. \n * [Data Science Bowl 2018](https://www.kaggle.com/c/data-science-bowl-2018/overview/evaluation). Yet another semantic segmentation competition. The evaluation metric is exactly the same. Here is the [top solution](https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741). \n \n* Some interesting ideas:\n\n * Multi-rounds pseudo-labelling (graph below from [here](https://arxiv.org/pdf/1904.04445.pdf))\n * Train a classifier first to find images without wheat heads\n * Randomly cropping the train images\n * Using augmentation. [Albumentations](https://github.com/albumentations-team/albumentations) is a great library for that.\n * Use BCE (binary cross entropy) + Jaccard as a loss (given some weight to each) or [Lovasz](https://arxiv.org/pdf/1705.08790.pdf)\n * Train using one loss for some epochs then use another one\n * Average over the last n snapshots for more generalization\n \n \n \n ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Evaluation Metric","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"The evaluation metric for object detection is quite tricky (when compared to object classification).\nHere is how it is computed:\n\n1. Predict the bounding box for each wheat head and give a confidence score\n2. Compute the intersection over union of each bounding box \n3. Re-order the predictions from highest score to lowest\n4. Now we can compute the recall and precision for the ranked bounding boxes\n5. From there, we can compute the average precision: this is the sum over 11 break points of the precision vs recall curve.\n6. Since we have only one class, this is also the mAP.\n\n\nIf this explanation isn't clear enough, check the following [paper](https://raw.githubusercontent.com/rafaelpadilla/Object-Detection-Metrics/master/paper_survey_on_performance_metrics_for_object_detection_algorithms.pdf) detailing object detection metrics and how they are computed.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## The Model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Enough with theory, let's move to the implementation!\n\nSimilarly the the previous NLP notebook, I will be using [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) to implement the model. \nFor that, we will start by defining the model part then move to the data processing, evaluation, and prediction.\n\nLet's get started!","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"To make things easier (DRY principle), I will be using the following **EfficientDet** [model](https://github.com/rwightman/efficientdet-pytorch).\n\n\nTo create the model, the code snippet is quite simple (D5 **EfficientDet** here but you can use the one that suits your needs): \n\n\n``` python\n\nfrom effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\n\n\n\ndef get_train_efficientdet():\n # Get the model's config\n config = get_efficientdet_config(\"tf_efficientdet_d5\")\n # Create the model\n net = EfficientDet(config, pretrained_backbone=False)\n # Load pretrained EfficientDet weights\n checkpoint = torch.load(\"path/to/weights\")\n net.load_state_dict(checkpoint)\n # Change the number of classes to 1 (only predicting wheat)\n config.num_classes = 1\n config.image_size = 256\n # Add the head with the updated parameters\n head = HeadNet(\n config,\n num_outputs=config.num_classes,\n norm_kwargs=dict(eps=0.001, momentum=0.01),\n )\n # Attach the head\n net.class_net = head\n return DetBenchTrain(net, config)\n\n```","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Notice that for the most recent version of the `effdet` library, you can use the create_model and/or the create_mode_from_config functions for a cleaner code.\n\n\nHere is a sample code: \n\n\n```\nfrom effdet import create_model\nnet = create_model('tf_efficientdet_d5', bench_task='train', num_classes=1, bench_labeler=True)\n```","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"In what follows, we will use this code snippet to create the model. Next, let's see how we can \ncreate a `WheatDataset`: if you aren't familiar with Pytorch's `Dataset` concept, check it [here](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset). ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n## The Dataset","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Before I start, notice that some of the code (mostly?) is inspired from this [notebook]( https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train), so go check it out and thank [Peter](https://www.kaggle.com/pestipeti) for making it!","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import DataLoader, Dataset\nimport torch\nfrom PIL import Image\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path\n\n# Using a small image size so it trains faster but do try bigger images \n# for better performance.\nIMG_SIZE = 256\n\ndef get_train_transforms():\n return A.Compose(\n [\n A.RandomSizedCrop(min_max_height=(800, 800), height=IMG_SIZE, width=IMG_SIZE, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.ToGray(p=0.01),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.Resize(height=256, width=256, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\n\ndef get_valid_transforms():\n return A.Compose(\n [\n A.Resize(height=IMG_SIZE, width=IMG_SIZE, p=1.0),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\n\n\n\ndef get_test_transforms():\n return A.Compose([\n A.Resize(height=IMG_SIZE, width=IMG_SIZE, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.0)\n\n\n\n\nclass WheatDataset(Dataset):\n\n def __init__(self, df = None, mode = \"train\", image_dir = \"\", transforms=None):\n super().__init__()\n if df is not None:\n self.df = df.copy()\n self.image_ids = df['image_id'].unique()\n else:\n # Test case\n self.df = None\n self.image_ids = [p.stem for p in Path(image_dir).glob(\"*.jpg\")]\n self.image_dir = image_dir\n self.transforms = transforms\n self.mode = mode\n\n def __getitem__(self, index: int):\n\n image_id = self.image_ids[index]\n # Could be one or many rows.\n\n image = Image.open(f'{self.image_dir}/{image_id}.jpg').convert(\"RGB\")\n # Convert to Numpy array\n image = np.array(image)\n image = image / 255.0\n image = image.astype(np.float32)\n\n\n if self.mode != \"test\":\n \n records = self.df[self.df['image_id'] == image_id]\n\n area = records[\"area\"].values\n area = torch.as_tensor(area, dtype=torch.float32)\n\n boxes = records[[\"x\", \"y\", \"x2\", \"y2\"]].values\n\n # there is only one class, so always 1.\n labels = torch.ones((records.shape[0],), dtype=torch.int64)\n \n # suppose all instances are not crowd.\n iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)\n \n target = {}\n target['boxes'] = boxes\n target['labels'] = labels\n # target['masks'] = None\n target['image_id'] = torch.tensor([index])\n target['area'] = area\n target['iscrowd'] = iscrowd\n # These are needed as well by the efficientdet model.\n target['img_size'] = torch.tensor([(IMG_SIZE, IMG_SIZE)])\n target['img_scale'] = torch.tensor([1.])\n \n else:\n\n # test dataset must have some values so that transforms work.\n target = {'cls': torch.as_tensor([[0]], dtype=torch.float32),\n 'bbox': torch.as_tensor([[0, 0, 0, 0]], dtype=torch.float32),\n 'img_size': torch.tensor([(IMG_SIZE, IMG_SIZE)]),\n 'img_scale': torch.tensor([1.])}\n\n\n if self.mode != \"test\":\n if self.transforms:\n sample = {\n 'image': image,\n 'bboxes': target['boxes'],\n 'labels': labels\n }\n if len(sample['bboxes']) > 0:\n # Apply some augmentation on the fly. \n sample = self.transforms(**sample)\n image = sample['image']\n boxes = sample['bboxes']\n # Need yxyx format for EfficientDet.\n target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*boxes)))).permute(1, 0)\n\n else:\n sample = {\n 'image': image,\n 'bbox': target['bbox'],\n 'cls': target['cls']\n }\n image = self.transforms(**sample)['image']\n\n return image, target\n\n def __len__(self) -> int:\n return len(self.image_ids)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"So, what have we done above?\n\n1. Defined some transforms using the [albumentations](https://albumentations.ai/) library for the three phases: train, validation, and test.\n2. Setup the `WheatDataset`. Notice there are some data wrangling to get the correct format while using the `EfficientDet` implementation.\n\nLet's check that this implementation works by sampling from the `WheatDataset`. \nBefore that one small detour to process the labels' DataFrame and add some useful metadata such as the bounding box `area` and\nthe lower corner's coordiantes.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Some processing to add some metadata to the labels DataFrame\ntrain_labels_df = pd.read_csv(\"../input/global-wheat-detection/train.csv\")\n\ntrain_labels_df['bbox'] = train_labels_df['bbox'].apply(eval)\n\nx = []\ny = []\nw = []\nh = []\nfor bbox in train_labels_df['bbox']:\n x.append(bbox[0])\n y.append(bbox[1])\n w.append(bbox[2])\n h.append(bbox[3])\n \nprocessed_train_labels_df = train_labels_df.copy()\nprocessed_train_labels_df[\"x\"] = x\nprocessed_train_labels_df[\"y\"] = y\nprocessed_train_labels_df[\"w\"] = w\nprocessed_train_labels_df[\"h\"] = h\n\n\nprocessed_train_labels_df[\"area\"] = processed_train_labels_df[\"w\"] * processed_train_labels_df[\"h\"]\nprocessed_train_labels_df[\"x2\"] = processed_train_labels_df[\"x\"] + processed_train_labels_df[\"w\"]\nprocessed_train_labels_df[\"y2\"] = processed_train_labels_df[\"y\"] + processed_train_labels_df[\"h\"]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create stratified folds, here using the source.\n# This isn't the most optimal way to do it but I will leave it to you \n# to find a better one. ;)\nfrom sklearn.model_selection import StratifiedKFold\nskf = StratifiedKFold(n_splits=5)\nfor fold, (train_index, valid_index) in enumerate(skf.split(processed_train_labels_df, \n y=processed_train_labels_df[\"source\"])):\n processed_train_labels_df.loc[valid_index, \"fold\"] = fold","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_train_labels_df.sample(2).T","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We need to set a folder to get images\nTRAIN_IMG_FOLDER = \"../input/global-wheat-detection/train\"\n\n# This is the transforms for the training phase\ntrain_transforms = get_train_transforms()\n\n\n\n\ntrain_dataset = WheatDataset(\n \n processed_train_labels_df, mode=\"train\", image_dir=TRAIN_IMG_FOLDER, transforms=train_transforms\n)\n\nimage, target = train_dataset[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(image)\nprint(target)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pylab as plt\nfrom PIL import Image, ImageDraw\nimport pandas as pd\nfrom pathlib import Path\n\n\ndef plot_image_with_bboxes(img_id, df):\n img_path = Path(TRAIN_IMG_FOLDER + f\"/{img_id}.jpg\")\n img = Image.open(img_path)\n draw = ImageDraw.Draw(img)\n bboxes = df.loc[lambda df: df[\"image_id\"] == img_id, \"bbox\"]\n # The box contains the upper left corner (x, y) coordinates then width and height.\n # So we need to change these to (x1, y1) and (x2, y2) where they are the upper \n # left and lower right corners\n for bbox in bboxes:\n x, y, w, h = bbox\n transformed_bbox = [x, y, x + w, y + h]\n draw.rectangle(transformed_bbox, outline=\"red\", width=3)\n return img\n\n\n\nplot_image_with_bboxes(\"00333207f\", processed_train_labels_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Looks fine, awesome! Time to move to the training pipeline.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n## Pytorch Lightning Complete Pipeline","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Alright, time to build a model. For that, we will be using the belvoed **Pytorch Lightning**. \nAs usual, we will start with the **model building block** then add the **processing** steps. As a first step, we install the **Pytorch Lightning** library using pip: `pip install pytorch_lightning`. \n\nNext, we install the [efficientdet](https://github.com/rwightman/efficientdet-pytorch) library, again using pip: `pip install effdet`. Notice that the **efficientdet** library needs **timm** (PyTorch Image Models library). For that, we also need to run: \n`pip install timm`. There is finally **omegaconf** and **pycocotools** (same using pip).","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install --upgrade pip\n!pip install pytorch_lightning\n!pip install effdet --upgrade\n!pip install timm\n!pip install omegaconf\n!pip install pycocotools","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Everything is set, time to create the **EfficientDet** model using the code snippet presented above: the new `create_model` code snippet instead of the `get_train_efficientdet` function since we are using the latest `effdet` version. ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nfrom effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\n\n\n# No longer works. :) \ndef get_train_efficientdet():\n config = get_efficientdet_config('tf_efficientdet_d5')\n net = EfficientDet(config, pretrained_backbone=False)\n checkpoint = torch.load('../input/efficientdet/efficientdet_d5-ef44aea8.pth')\n net.load_state_dict(checkpoint)\n config.num_classes = 1\n config.image_size = IMG_SIZE\n net.class_net = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=.001, momentum=.01))\n return DetBenchTrain(net, config)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# New way that works!\n\nfrom effdet import create_model\n\n\ndef get_train_efficientdet():\n # checkpoint_path='../input/efficientdet/efficientdet_d5-ef44aea8.pth'\n return create_model('tf_efficientdet_d5', bench_task='train', num_classes=2, bench_labeler=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pytorch_lightning import LightningModule\n\nclass WheatModel(LightningModule):\n \n \n def __init__(self, df, fold):\n super().__init__()\n self.df = df\n self.train_df = self.df.loc[lambda df: df[\"fold\"] != fold]\n self.valid_df = self.df.loc[lambda df: df[\"fold\"] == fold]\n self.image_dir = TRAIN_IMG_FOLDER\n self.model = get_train_efficientdet()\n self.num_workers = 4\n self.batch_size = 8\n \n \n def forward(self, image, target):\n return self.model(image, target)\n \n\n# Create a model for one fold.\nmodel = WheatModel(processed_train_labels_df, fold=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n\ndef collate_fn(batch):\n return tuple(zip(*batch))\n\n# Let's add the train and validation data loaders.\n\ndef train_dataloader(self):\n train_transforms = get_train_transforms()\n train_dataset = WheatDataset(\n self.train_df, image_dir=self.image_dir, transforms=train_transforms\n )\n return DataLoader(\n train_dataset,\n batch_size=self.batch_size,\n sampler=RandomSampler(train_dataset),\n pin_memory=False,\n drop_last=True,\n collate_fn=collate_fn,\n num_workers=self.num_workers,\n )\n\ndef val_dataloader(self):\n valid_transforms = get_train_transforms()\n valid_dataset = WheatDataset(\n self.valid_df, image_dir=self.image_dir, transforms=valid_transforms\n )\n valid_dataloader = DataLoader(\n valid_dataset,\n batch_size=self.batch_size,\n sampler=SequentialSampler(valid_dataset),\n pin_memory=False,\n shuffle=False,\n collate_fn=collate_fn,\n num_workers=self.num_workers,\n )\n iou_types = [\"bbox\"]\n # Had to comment these since the evaluation doesn't work yet.\n # More on this in the next section.\n # coco = convert_to_coco_api(valid_dataset)\n # self.coco_evaluator = CocoEvaluator(coco, iou_types)\n return valid_dataloader\n\n\n\nWheatModel.train_dataloader = train_dataloader\nWheatModel.val_dataloader = val_dataloader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# TODO: What should be changed in order to make it work?\ndef training_step(self, batch, batch_idx):\n images, targets = batch\n targets = [{k: v for k, v in t.items()} for t in targets]\n # separate losses\n images = torch.stack(images).float()\n targets2 = {}\n targets2[\"bbox\"] = [\n target[\"boxes\"].float() for target in targets\n ] # variable number of instances, so the entire structure can be forced to tensor\n targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n \"\"\"\n targets2[\"image_id\"] = torch.tensor(\n [target[\"image_id\"] for target in targets]\n ).float()\n targets2[\"img_scale\"] = torch.tensor(\n [target[\"img_scale\"] for target in targets], device=\"cuda\"\n ).float()\n targets2[\"img_size\"] = torch.tensor(\n [(IMG_SIZE, IMG_SIZE) for target in targets], device=\"cuda\"\n ).float()\n \"\"\"\n losses_dict = self.model(images, targets2)\n\n return {\"loss\": losses_dict[\"loss\"], \"log\": losses_dict}\n\ndef validation_step(self, batch, batch_idx):\n images, targets = batch\n targets = [{k: v for k, v in t.items()} for t in targets]\n # separate losses\n images = torch.stack(images).float()\n targets2 = {}\n targets2[\"bbox\"] = [\n target[\"boxes\"].float() for target in targets\n ] # variable number of instances, so the entire structure can be forced to tensor\n targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n \"\"\"\n targets2[\"image_id\"] = torch.tensor(\n [target[\"image_id\"] for target in targets]\n ).float()\n targets2[\"img_scale\"] = torch.tensor(\n [target[\"img_scale\"] for target in targets], device=\"cuda\"\n ).float()\n targets2[\"img_size\"] = torch.tensor(\n [(IMG_SIZE, IMG_SIZE) for target in targets], device=\"cuda\"\n ).float()\n \"\"\"\n losses_dict = self.model(images, targets2)\n loss_val = losses_dict[\"loss\"]\n detections = losses_dict[\"detections\"]\n # Back to xyxy format.\n detections[:, :, [1,0,3,2]] = detections[:, :, [0,1,2,3]]\n # xywh to xyxy => not necessary.\n # detections[:, :, 2] += detections[:, :, 0]\n # detections[:, :, 3] += detections[:, :, 1]\n\n res = {target[\"image_id\"].item(): {\n 'boxes': output[:, 0:4],\n 'scores': output[:, 4],\n 'labels': output[:, 5]}\n for target, output in zip(targets, detections)}\n # iou = self._calculate_iou(targets, res, IMG_SIZE)\n # iou = torch.as_tensor(iou)\n # self.coco_evaluator.update(res)\n return {\"loss\": loss_val, \"log\": losses_dict}\n\ndef validation_epoch_end(self, outputs):\n # self.coco_evaluator.accumulate()\n # self.coco_evaluator.summarize()\n # coco main metric\n # metric = self.coco_evaluator.coco_eval[\"bbox\"].stats[0]\n # metric = torch.as_tensor(metric)\n # tensorboard_logs = {\"main_score\": metric}\n # return {\n # \"val_loss\": metric,\n # \"log\": tensorboard_logs,\n # \"progress_bar\": tensorboard_logs,\n # }\n pass\n\ndef configure_optimizers(self):\n return torch.optim.AdamW(self.model.parameters(), lr=1e-4)\n\n\n\nWheatModel.training_step = training_step\n# Had to comment these since the evaluation doesn't work yet. More on this \n# in the next section.\n# WheatModel.validation_step = validation_step\n# WheatModel.validation_epoch_end = validation_epoch_end\nWheatModel.configure_optimizers = configure_optimizers\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"That's it, our `WheatModel` is ready now. Let's train it. For that, we will create a `Trainer` and set it \nto `fast_dev_run=True` for a quicker demo. Also, since it is in this mode, the `Trainer`\ndoesn't automatically save the weights at the end (correct me if I am wrong of course) so we\nneed to add a `torch.save` call at the end.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pytorch_lightning import Trainer, seed_everything, loggers\n\nseed_everything(314)\n# Create a model for one fold.\n# As an exercise, try doing it for the other folds and chaning the Trainer. ;)\nmodel = WheatModel(processed_train_labels_df, fold=0)\nlogger = loggers.TensorBoardLogger(\"logs\", name=\"effdet-b5\", version=\"fold_0\") \ntrainer = Trainer(gpus=1, logger=logger, fast_dev_run=True)\ntrainer.fit(model)\ntorch.save(model.model.state_dict(), \"wheatdet.pth\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n# Evaluation ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Believe it or not, this is a part where I struggled the most. This isn't because of the tool but it was due to the \ncode that I was using. Indeed, the evaluation part is tied to the `torchvision` repo and I haven't found a simple way to import it.\nThus I needed to copy a lot of code as was done in artgor's notebook. I won't show this part since it is tedious and not very elegant. If you have found a better method, please share it with me in the comments section.\n\nAnother route I have tried is a similar one used [here](https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=DBIoe_tHTQgV) where they have cloned the torchvision part from the Pytorch's repo then copying the necessary parts using cp (instead of manually doing it). I am sure it works eventually but I won't do this here since the notebook is long enough already and I am being a little lazy. I will try to make this section work later on. ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Download TorchVision repo to use some files from\n# references/detection\n\"\"\"\n!git clone https://github.com/pytorch/vision.git\n!cd vision\n!git checkout v0.3.0\n\n!cp references/detection/utils.py ../\n!cp references/detection/transforms.py ../\n!cp references/detection/coco_eval.py ../\n!cp references/detection/engine.py ../\n!cp references/detection/coco_utils.py ../\n\"\"\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"To finish this section, let's make some predictions.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Inference","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Mostly inspired (i.e. copied ;)) from [here](https://www.kaggle.com/shonenkov/inference-efficientdet). Again, thanks \n**Alex Shonenkov**. I won't go over the formatting step to make a proper submission file. \nCheck the notebook if you want to learn about these final steps.\n\nAlso, in a better world, I would have made this part using Pytorch Lightning but I am little overwhelmed with \nthis very lengthy notebook so it is enough for now. ;)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from effdet import DetBenchPredict\n\n\ndef get_test_efficientdet(checkpoint_path):\n config = get_efficientdet_config('tf_efficientdet_d5')\n net = EfficientDet(config, pretrained_backbone=False)\n\n config.num_classes = 1\n config.image_size=IMG_SIZE\n net.class_net = HeadNet(config, num_outputs=config.num_classes, \n norm_kwargs=dict(eps=.001, momentum=.01))\n\n checkpoint = torch.load(checkpoint_path)\n net.load_state_dict(checkpoint, strict=False)\n net = DetBenchPredict(net, config)\n return net\n\n\n\ndef make_predictions(model, images, score_threshold=0.22):\n images = torch.stack(images).float()\n predictions = []\n targets = torch.tensor([1]*images.shape[0])\n img_scales = torch.tensor(\n [1 for target in targets]\n ).float()\n img_sizes = torch.tensor(\n [(IMG_SIZE, IMG_SIZE) for target in targets]\n ).float()\n with torch.no_grad():\n detections = model(images, img_scales, img_sizes)\n for i in range(images.shape[0]):\n pred = detections[i].detach().cpu().numpy()\n boxes = pred[:,:4] \n scores = pred[:,4]\n indexes = np.where(scores > score_threshold)[0]\n boxes = boxes[indexes]\n boxes[:, 2] = boxes[:, 2] + boxes[:, 0]\n boxes[:, 3] = boxes[:, 3] + boxes[:, 1]\n predictions.append({\n 'boxes': boxes[indexes],\n 'scores': scores[indexes],\n })\n return predictions","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TEST_IMG_FOLDER = \"../input/global-wheat-detection/test\"\n\n\ntest_model = get_test_efficientdet(\"wheatdet.pth\")\n\ntest_dataset = WheatDataset(\n \n None, mode=\"test\", image_dir=TEST_IMG_FOLDER, transforms=get_test_transforms()\n)\n\nimage, _ = test_dataset[0]\n\npredictions = make_predictions(test_model, [image])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Hurray, we have predictions!","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n# Advanced Concepts\n\n\nBefore finishing this (lengthy) notebook, I wanted to talk about few **advanced concepts** to get better results\nparticularly for object detection but they do work for many computer vision tasks. \nAs you might have seen, the results you get from training the model presented above are quite good but not \nenough to be competitive. Indeed, winning solutions used EfficientDet augmented with few more tricks. \n\nHere are some advanced concepts to get you better results: \n\n- Mosaic augmentation\n- TTA \n- WBF blending\n\nLet's start with **mosaic augmentation**.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## Mosaic Augmentation","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"This technique consists in stitching together different images to form a bigger one. \nThe more diverse the images, the better it is. \nIf you are curious, you can check an example of how to use it\n[here](https://www.kaggle.com/c/global-wheat-detection/discussion/172418) from the winning competition solution. The following example is extracted from the same post, thanks again to [DungNB](https://www.kaggle.com/nguyenbadung) for the write-up!\n\n![custom_mosaic_augmentation.png](attachment:custom_mosaic_augmentation.png)\n\n\nHere is a simple explanation using cat images: \n\n\n\n![mosaic_augmentation_resized.png](attachment:mosaic_augmentation_resized.png)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## TTA\n\n\nTTA, short for **test-time augmentation**, is a popular technique for getting better performance for computer vision tasks and you will often encounter it when doing Kaggle computer vision competitions.\n\nHow does it work?\n\nThe basic idea is quite simple: \n\n1. Create additional test images from the provided ones using **augmentation** techniques.\n2. Predict using the trained model: you will have the prediction for the original image plus the augmented ones.\n3. Blend the predictions: more on this in the next section \n\nFor the augmentation step, horizantal and vertical flips are quite common strategies for TTA. Notice that since this step happens during inference time and since code competitions have time limits, you can't use a lot of augmentation. \n2 or 3 additional images per original test one is more than enough.\nFinally, for object detection, since there is a bounding box to predict, the augmentation step should preserve the shape of the bounding box. Thus some augmentations aren't allowed (for instance 90 degrees rotations). \n\nTo finish this section, here is an illustration (again, using cats :p)\n\n\n![tta_resized.png](attachment:tta_resized.png)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n## WBF Blending\n\n\nIf you are accustomed to **object detection** tasks and challenges, you may have heard about **NMS** (short for non-maximum suppresion). Here is a short description of its pseudo-code:\n\n\n\n\n\n\n\nCheck this [post](https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c) for more details (the figure above is from it).\n\n\nThis technique is used to merge many boxes (called proposals) from one model and can be used to merge the results of many models as well. \n\nHere, we will go over another technique called **weighted-boxes fusion** or (**WBF** in short). \n\nTo get started, you can check the Github implementation [here](https://github.com/ZFTurbo/Weighted-Boxes-Fusion), the [paper](https://arxiv.org/pdf/1910.13302.pdf) or this great [notebook](https://www.kaggle.com/shonenkov/wbf-approach-for-ensemble) again from [Alex Shonenkov](https://www.kaggle.com/shonenkov), so thanks!\n\nSo how does it work and how is it different? Let's have a look at the following schema (from the **WBF** paper)\n\n\n![wbf_schema.png](attachment:wbf_schema.png)\n\n\nIf you may have noticed, the **WBF** algorithm \"made\" a bounding box that isn't any of the proposed ones whereas NMS came up with one from the proposals. Indeed, that's one of the big differences between the two: how different proposals are merged. For the WBF case, the main objective is to compute a new confidence score C from existing confidence scores and then use it to compute the blended bounding box coordinates and confidence as a weighted mean (or other functions).\n\n\nFinally, if you want even more blending techniques, you can explore these techniques:\n\n- Soft-NMS: similar to NMS (non-maximum suppression) but with a soft selection criterion\n\n\n\n\n- Non-maximum weighted (NMW): this is the same as the NMS but computing a weighted a bounding box instead\nof computing only one.\n\n\n\n\n\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n# To Wrap Up","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"That's it for today. If you made it this far, congratulations!\n\nBy now, you should have acquired basic knowledge of **object detection** and **computer vision** more generally, you can \nnow go and build models. Congratulations on gaining a new (super) power!\n\nStay tuned for the next notebook and in the meantime, happy kaggling!","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n## To Go Beyond","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Going further, check these resources to expand your understanding: \n \n- Pytorch's torchvision object detection tutorialn very: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html\n- Learn more about efficientdet here: https://blog.roboflow.ai/breaking-down-efficientdet/\n- If you want to learn more about computer vision using deep learning: A great playlist if you like videos: https://www.youtube.com/playlist?list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r\n- More specifically, this video is great for object detection: https://www.youtube.com/watch?v=TB-fdISzpHQ&list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r&index=16&t=0s\n\n- A good explanation of the differences between evaluation metrics: https://stats.stackexchange.com/questions/238684/what-are-the-difference-between-dice-jaccard-and-overlap-coefficients\n\n- A complete explanation of EfficientDet: https://towardsdatascience.com/googles-efficientdet-an-overview-8d010fa15860\n- Another EfficientDet Medium post: https://medium.com/@nainaakash012/efficientdet-scalable-and-efficient-object-detection-ea05ccd28427\n- A paper comparing human-level performance vs some deep learning algorithms on vision tasks: https://arxiv.org/pdf/1706.06969.pdf\n- Some augmentations notebook: https://www.kaggle.com/nvnnghia/awesome-augmentation\n- Another EfficientDet Pytorch implementation: https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch\n- A very thorough series of object detection medium posts: https://medium.com/@jonathan_hui/object-detection-series-24d03a12f904","metadata":{},"execution_count":null,"outputs":[]}]}