{"cells":[{"metadata":{"_uuid":"10527e38-4e4e-437a-b301-7b6247f155e1","_cell_guid":"fdf0f796-96dc-4c6b-a82e-f31f7917f35c","trusted":true},"cell_type":"markdown","source":"# Frameworks for Object Detection\n\nImplementing models for object detection from scratch is a bit complicated: we need to start with building a [Feature Pyramid Networks](https://arxiv.org/abs/1612.03144) combined with [Region Proposal Networks](https://medium.com/@nabil.madali/demystifying-region-proposal-network-rpn-faa5a8fb8fce) if we want to use region proposal algorithms such as Faster R-CNN. To understand these models it is important to build it at least once from scratch. Nevertheless, object detection frameworks come to the rescue if we want to train a model with a custom dataset without dwelling too much on the specific implementation.\n\nHere you can find an incomplete list of popular object detection frameworks:\n\n- Detectron2 -> [repository](https://github.com/facebookresearch/detectron2)\n- MMDetection -> [paper](https://arxiv.org/pdf/1906.07155.pdf) -> [repository](https://github.com/open-mmlab/mmdetection) -> [tutorial](https://medium.com/@ravisingh93362/object-detection-using-mmdetection-c7f0eb26a2c9)\n- SimpleDet -> [paper](https://arxiv.org/pdf/1903.05831.pdf) -> [repository](https://github.com/TuSimple/simpledet)\n- TorchVision -> [repository](https://github.com/pytorch/vision)\n- Tensorflow Object Detection -> [repository](https://github.com/tensorflow/models/tree/master/research/object_detection)\n\nSuch frameworks usually lag behind the state of the art, as developers need to integrate the most recent algorithms into the framework. If you want to use a very recent implementation you should consider repositories that usually being published together with the papers. For the most recent state of the art models and their implementations consider visiting [Papers with Code](https://paperswithcode.com/task/object-detection).\n- YOLOv5 -> [repository](https://github.com/ultralytics/yolov5)\n- EfficientDet [paper](https://arxiv.org/pdf/1911.09070.pdf) [repository](https://github.com/rwightman/efficientdet-pytorch)\n\n# Detectron2 for Object Detection\n\n\nIn this notebook, we are going to use [Detectron2](https://github.com/facebookresearch/detectron2) framework from Facebook Artificial Intelligence Research (FAIR) to train a deep neural network model for object detection. Of course, you can use any implementation of object detection algorithms out there. However, there are several benefits associated with using such a framework:\n - A variety of pre-trained models available in [the model zoo](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md).\n - Unified configuration file for specifying models \n - Standardized way of handling labels and annotations\n - Efficient implementation based on PyTorch deep learning library\n - Extensible framework, that allows you to build new models without rewriting tons of boilerplate code, such as handling datasets, statistics collection, etc."},{"metadata":{"_uuid":"be489c7b-ab10-42b7-ae0e-6ab90306ade1","_cell_guid":"dfb561a1-427a-4d9d-b93b-972e06564de0","trusted":true},"cell_type":"markdown","source":"## High Level Structure of Detectron2\n\nBellow, you can find a short overview of the directory structure of the Detectron2:\n\n\n\n\n\n\nThe framework was developed to facilitate complete object detection pipeline. There are multiple ways to use Detectron2, we list three of them with increasing complexity:\n\n1. Run inference on existing pre-trained models. Check [Detectron2 Begnner's Tutorial](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5) on colab.\n2. Configure and train existing models. This step is the focus of this notebook. We are going to answer the following questions: \n - How to change configuration file?\n - How to load pre-trained weights?\n - How to adjust cusotm data and annotations to fit Detectron2 format?\n - How to save the model?\n - How to run inference?\n - How to visualize results?\n3. Use Detectron2 framework for building your own models and architectures. Check [Projects](https://github.com/facebookresearch/detectron2/tree/master/projects) folder."},{"metadata":{"_uuid":"4963d3e1-e1d7-44ae-9c91-3b644f0bc926","_cell_guid":"a8c6e33e-f879-4fa4-b8cd-f2f31c65870a","trusted":true},"cell_type":"markdown","source":"## Install and load neccessary packages."},{"metadata":{"_uuid":"6bbdddc9-c8ab-443c-ad1c-aa2399d5c038","_cell_guid":"7e6842b5-346f-4e6e-9b84-795b78030ceb","trusted":true},"cell_type":"markdown","source":"**Important Note**: This particular competition does not allow the use of the internet in the notebook for the submission to be valid! Therefore **do not** activate internet in the notebook settings. Bellow we present an installation process that is specific for this competition: we have added all important packages and model weights as a datasets (top right corner `Add data` button). In case you want to install Detectron2 locally, you can follow the official [installation guide](https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md)."},{"metadata":{"_uuid":"d07b8393-15fc-461b-b8cb-2d74b6df80d5","_cell_guid":"ed128593-5463-4cc0-bc65-ccc8993c0fde","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install '/kaggle/input/torch-15/torch-1.5.0cu101-cp37-cp37m-linux_x86_64.whl'\n!pip install '/kaggle/input/torch-15/torchvision-0.6.0cu101-cp37-cp37m-linux_x86_64.whl'\n!pip install '/kaggle/input/torch-15/yacs-0.1.7-py3-none-any.whl'\n!pip install '/kaggle/input/torch-15/fvcore-0.1.1.post200513-py3-none-any.whl'\n!pip install '/kaggle/input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl'\n!pip install '/kaggle/input/detectron2/detectron2-0.1.3cu101-cp37-cp37m-linux_x86_64.whl'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"05e69810-08bc-40b0-bafd-ad11ac7703a7","_cell_guid":"cac59c9a-9e84-4431-9cd9-843c32027f36","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom tqdm.notebook import tqdm\n\nimport gc\nimport os\nimport copy\nfrom glob import glob\n\nimport cv2\nfrom PIL import Image\n\nimport random\n\nfrom collections import deque, defaultdict\nfrom multiprocessing import Pool, Process\nfrom functools import partial\n\nimport torch\nimport torch.nn as nn\n\nimport pycocotools\nimport detectron2\nfrom detectron2.config import get_cfg\nfrom detectron2 import model_zoo\nfrom detectron2.engine import DefaultPredictor, DefaultTrainer\nfrom detectron2.utils.visualizer import Visualizer, ColorMode\nfrom detectron2.structures import BoxMode\nfrom detectron2.data import datasets, DatasetCatalog, MetadataCatalog, build_detection_train_loader, build_detection_test_loader\nfrom detectron2.data import transforms as T\nfrom detectron2.data import detection_utils as utils\nfrom detectron2.evaluation import COCOEvaluator, verify_results\nfrom detectron2.modeling import GeneralizedRCNNWithTTA\nfrom detectron2.data.transforms import TransformGen\nfrom detectron2.utils.logger import setup_logger\nsetup_logger()\n\nfrom fvcore.transforms.transform import TransformList, Transform, NoOpTransform\nfrom contextlib import contextmanager","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f680ce25-c7d4-49f8-8dc3-65bed5cd103d","_cell_guid":"2be1f007-51b1-4c26-a231-8f47dd108426","trusted":true},"cell_type":"markdown","source":"# Main Configuration"},{"metadata":{"_uuid":"a9bd6817-8f43-416f-a180-fc301786a364","_cell_guid":"54111ce9-087e-47be-9f33-d468efbab66f","trusted":true},"cell_type":"markdown","source":"To use different your models, you would need to go to the [model zoo](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md) select your configuration file and download the model weights. You can upload model weights by using `+Add data` in the top right corner. Afterward, you only need to specify a path to configuration and the weights by adding a new entry to the dictionary below. We have added four models for you to try."},{"metadata":{"_uuid":"dea4760e-de5c-4930-a78e-6c076d097cf8","_cell_guid":"ccd3d344-32c5-43a8-9022-6033fe5f61ba","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"MAIN_PATH = '/kaggle/input/global-wheat-detection'\nTRAIN_IMAGE_PATH = os.path.join(MAIN_PATH, 'train/')\nTEST_IMAGE_PATH = os.path.join(MAIN_PATH, 'test/')\nTRAIN_PATH = os.path.join(MAIN_PATH, 'train.csv')\nSUB_PATH = os.path.join(MAIN_PATH, 'sample_submission.csv')\n\nmodels = {\n 'rpn_r50_fpn': {\n 'model_path': 'COCO-Detection/rpn_R_50_FPN_1x.yaml',\n 'weights_path': '/kaggle/input/rpn-r-50-fpn-1x/model_final_02ce48.pkl'\n },\n 'faster_rcnn_50': {\n 'model_path': 'COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml',\n 'weights_path': '/kaggle/input/faster-r-cnn-r50-fpn-x3/model_final_280758.pkl'\n },\n 'faster_rcnn_101': {\n 'model_path': 'COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml',\n 'weights_path': '/kaggle/input/detectron2-faster-rcnn-101/model_final_f6e8b1.pkl'\n },\n 'retinanet': {\n 'model_path': 'COCO-Detection/retinanet_R_101_FPN_3x.yaml',\n 'weights_path': '/kaggle/input/detectron2-faster-rcnn-101/model_final_971ab9.pkl'\n }\n}\n\n# Uncomment the model you would like to use\n# MODEL_USE = 'rpn_r50_fpn'\nMODEL_USE = 'faster_rcnn_50'\n# MODEL_USE = 'faster_rcnn_101'\n# MODEL_USE = 'retinanet'\n\n# Load the model config and its pre-trained weights\nMODEL_PATH = models[MODEL_USE]['model_path']\nWEIGHT_PATH = models[MODEL_USE]['weights_path']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The models and datasets can be fully configured with a configuration file. [Here](https://github.com/facebookresearch/detectron2/tree/master/configs) you can find the list of all base configuration files. The folders in this directory corresponds to specific models pre-trained on a particular dataset. If we have a look at one of them more closely we will see that it still inherits from the `_BASE_: \"../Base-RCNN-FPN.yaml\"` configuration.\n\n\n\n\nEverytime you are working with a detectron2 you are going to override an already existing configuration. This is what makes detectron2 easy to use: you never start from scratch and most of the configurations are fully working out of the box.\n\n#### Some context for the configuration files\nYou can read more about different configurations on the [model zoo](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md#common-settings-for-coco-models) page.\n\nFor Faster/Mask R-CNN, Detectron provides baselines based on 3 different backbone combinations:\n- FPN: Use a ResNet+FPN backbone with standard conv and FC heads for mask and box prediction, respectively. It obtains the best speed/accuracy tradeoff, but the other two are still useful for research.\n- C4: Use a ResNet conv4 backbone with conv5 head. The original baseline in the [Faster R-CNN paper](https://arxiv.org/abs/1506.01497).\n- DC5 (Dilated-C5): Use a ResNet conv5 backbone with dilations in conv5, and standard conv and FC heads for mask and box prediction, respectively. This is used by the [Deformable ConvNet paper](https://arxiv.org/abs/1703.06211).\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Configuration file.\nConfiguration file consists of several main components. Lets load a configuration file and look at the most important ones. You can find a more detailed explanation [here](https://detectron2.readthedocs.io/modules/config.html).\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loads the default configuration setup\ncfg = get_cfg()\ncfg.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- 'VERSION' - indicates the version of the Detectron\n- 'OUTPUT_DIR' - contains folder where output of the training is going to be saved\n- 'SEED' - random seed, set to negative to fully randomize everything\n- 'CUDNN_BENCHMARK' - if true cudnn algorithms are going to be benchmarked\n- 'VIS_PERIOD' - the period (in terms of steps) for minibatch visualization at train time. Set to 0 to disable."},{"metadata":{},"cell_type":"markdown","source":"### 'MODEL'\nThis part of the configuration file is responsible for setting the model parameters. Each key in this dictionary corresponds to a different components of the model. Not all of the components are used at the same time, however they all need to be present in the configuration for backward compatibility."},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg['MODEL'].keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 'RESNETS' - Backbone ResNet settings"},{"metadata":{"trusted":true},"cell_type":"code","source":"from pprint import pprint\npprint(cfg['MODEL']['RESNETS'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- 'NUM_GROUPS' - Number of groups to use; 1 ==> [ResNet](https://arxiv.org/abs/1512.03385); > 1 ==> [ResNeXt](https://arxiv.org/abs/1611.05431)\n- 'NORM' - Options: FrozenBN, GN, \"SyncBN\", \"BN\"\n- 'WIDTH_PER_GROUP' - Scaling this parameters will scale the width of all bottleneck layers.\n- 'STRIDE_IN_1X1' - Place the stride 2 conv on the 1x1 filter. Use True only for the original MSRA ResNet; use False for C2 and Torch models\n- 'RES2_OUT_CHANNELS' - Output width of res2. Scaling this parameters will scale the width of all 1x1 convs in ResNet"},{"metadata":{},"cell_type":"markdown","source":"#### 'RPN'- Region Proposal Nework settings"},{"metadata":{"trusted":true},"cell_type":"code","source":"pprint(cfg['MODEL']['RPN'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 'ROI_HEADS' - Region of Interest Heads"},{"metadata":{"trusted":true},"cell_type":"code","source":"pprint(cfg['MODEL']['ROI_HEADS'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 'RETINANET' - [RetinaNet](https://arxiv.org/abs/1708.02002)"},{"metadata":{"trusted":true},"cell_type":"code","source":"pprint(cfg['MODEL']['RETINANET'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 'DATASETS'\nIn this part of the configuration we are going to indicate what dataset to use for training and testing."},{"metadata":{"trusted":true},"cell_type":"code","source":"pprint(cfg['DATASETS'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 'SOLVER'\nSolver constains configurations for the learning rate, number of iterations and other Gradient Descent settings."},{"metadata":{"trusted":true},"cell_type":"code","source":"pprint(cfg['SOLVER'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- 'BASE_LR' - base learning rate\n- 'IMS_PER_BATCH' - Number of images per batch across all machines. If we have 16 GPUs and IMS_PER_BATCH = 32, each GPU will see 2 images per batch."},{"metadata":{"_uuid":"b9fc9b78-d112-452f-bee7-10d68fced3bd","_cell_guid":"5c382831-d1d7-404f-8469-16261d49257a","trusted":true},"cell_type":"markdown","source":"# Data"},{"metadata":{"_uuid":"bfaff404-1c79-45b4-9dd6-588b39b07b9b","_cell_guid":"133374f6-965e-42f5-a48a-0dbb83eb554a","trusted":true},"cell_type":"code","source":"train_img = glob(f'{TRAIN_IMAGE_PATH}/*.jpg')\ntest_img = glob(f'{TEST_IMAGE_PATH}/*.jpg')\n\nprint('Number of train images: {}, test images: {}'.format(len(train_img), len(test_img)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f36fd80b-c3da-4401-bd39-cc3a03a7e509","_cell_guid":"24d0270b-9c64-49ef-aab3-3bbfb1bae583","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_PATH)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b7ab09a-9efe-4bb1-ad34-08e0ee5f8381","_cell_guid":"e4d94972-0a73-4874-bf01-1502043299e6","trusted":true},"cell_type":"markdown","source":"## Display Images with Bounding Boxes"},{"metadata":{"_uuid":"7a5b33ff-4e6d-49c8-a035-993cd62e50e6","_cell_guid":"2317b180-6d6b-45ad-b09c-3e4555f80474","trusted":true},"cell_type":"code","source":"def display_images(df, folder, num_img=1, bb_color=(0, 50, 255)):\n \n fig, axs = plt.subplots(nrows=1, ncols=num_img,figsize=(15, 15), squeeze=False)\n axs = axs.flatten()\n for i, ax in enumerate(axs):\n # randomly pick an image\n img_random = random.choice(df['image_id'].unique())\n assert (img_random + '.jpg') in os.listdir(folder)\n \n img_df = df[df['image_id']==img_random]\n img_df.reset_index(drop=True, inplace=True)\n \n img = cv2.imread(os.path.join(folder, img_random + '.jpg'))\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n \n for row in range(len(img_df)):\n source = img_df.loc[row, 'source']\n box = img_df.loc[row, 'bbox'][1:-1]\n box = list(map(float, box.split(', ')))\n x, y, w, h = list(map(int, box))\n cv2.rectangle(img, (x, y), (x+w, y+h), bb_color, 2)\n \n ax.set_title(f'{img_random} have {len(img_df)} bounding boxes')\n ax.imshow(img) \n \n plt.show() \n plt.tight_layout()\n \ndisplay_images(train_df, TRAIN_IMAGE_PATH, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78724aec-ea87-4531-8794-29cd425115db","_cell_guid":"40333fe3-263f-4daf-9382-b62abe84d607","trusted":true},"cell_type":"markdown","source":"## Image sources\nIn the dataset we see that there is a number of sources.\nLest visualize the distribution of the sources."},{"metadata":{"_uuid":"22863fc3-6361-4b99-bc32-0c40f7f73717","_cell_guid":"9c0db1ff-1720-426f-8d47-b4f4c5ff892b","trusted":true},"cell_type":"code","source":"def display_feature(df, feature):\n \n plt.figure(figsize=(15,8))\n ax = sns.countplot(y=feature, data=df, order=df[feature].value_counts().index)\n\n for p in ax.patches:\n ax.annotate('{:.2f}%'.format(100*p.get_width()/df.shape[0]), (p.get_x() + p.get_width() + 0.02, p.get_y() + p.get_height()/2))\n\n plt.title(f'Distribution of {feature}', size=25, color='b') \n plt.show()\n\nlist_source = train_df['source'].unique().tolist()\nprint(list_source)\ndisplay_feature(train_df, 'source')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d652587-dc6b-44c8-b66f-23d12850d15d","_cell_guid":"c1d74238-9329-4506-b618-c1a38997593f","trusted":true},"cell_type":"markdown","source":"### Number of bounding boxes per image"},{"metadata":{"_uuid":"8503b1d6-1231-4118-89ad-f77fefd77475","_cell_guid":"0198608a-9788-4221-a2dd-9157a851d183","trusted":true},"cell_type":"code","source":"num_box = train_df.groupby('image_id')['bbox'].count().reset_index().add_prefix('Number_').sort_values('Number_bbox', ascending=False)\nnum_box.head()\nprint('Average number of bounding boxes per image: {:.2f}'.format(num_box.mean().values[0]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69332612-805f-4b79-9d1b-dab341dc5427","_cell_guid":"45574f10-f0a3-49eb-90b4-66e03dd9928e","trusted":true},"cell_type":"markdown","source":"## Check if the number of images in the training file is the same as in the training folder."},{"metadata":{"_uuid":"31855776-bd06-4aaa-a0cb-eea601ee6141","_cell_guid":"2f9661f0-ac8b-4e52-b9c0-48edfe46f6dc","trusted":true},"cell_type":"code","source":"image_unique = train_df['image_id'].unique()\nimage_unique_in_train_path = [i for i in image_unique if i + '.jpg' in os.listdir(TRAIN_IMAGE_PATH)]\n\nprint(f'Number unique images: {len(image_unique)}, in train path: {len(image_unique_in_train_path)}')\n\ndel image_unique, image_unique_in_train_path\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e3d762e-1d25-46b3-a1fe-12f5e635625b","_cell_guid":"f7f894fb-1498-40b2-906c-97e96b44f0bf","trusted":true},"cell_type":"markdown","source":"# Example Submission\nWe will use an example submission to make our submission by overriding the `PredicitionString` column.\nMoreover we are going to use it as input in our testing dataset."},{"metadata":{"_uuid":"bcd8d36f-4c1a-45eb-b88f-8b68b1a6e41d","_cell_guid":"92bb0f29-b8fd-4fa3-a4b6-f3094e4f8310","trusted":true},"cell_type":"code","source":"sub_df = pd.read_csv(SUB_PATH)\nsub_df.tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b4296e0-29c2-4e73-a766-8a3b61fc19d1","_cell_guid":"43e88a3a-9d64-4f28-b48c-a7bd4f99aa7c","trusted":true},"cell_type":"markdown","source":"# Building a Dataset"},{"metadata":{"_uuid":"a1674d4a-0016-4a08-b33c-ea724e4b542c","_cell_guid":"2a47b2cd-1ddb-4b30-93e4-288e60672ad6","trusted":true},"cell_type":"markdown","source":"In order to use our dataset in a Detectron2 framwork we would need to preprocess the images and its annotations.\n\nCurrently, each line in our `train_df` corresponds to a bounding box. This bounding box is associated with an image. First thing that we need to do is to represent our data as a list of dictionaries. Every element in that list will correspond to an image that has multiple attributes. Lets have a closer look."},{"metadata":{"_uuid":"1d721b2e-c098-4e9d-9cd6-ac982832ac7d","_cell_guid":"3c6e03cc-01e6-42af-b1fb-e758451688c0","trusted":true},"cell_type":"markdown","source":"## Dataset Dictionary"},{"metadata":{"_uuid":"f2dae545-db05-40b0-87d8-196002251659","_cell_guid":"4c84af04-f818-45c0-8877-25129977ec4a","trusted":true},"cell_type":"markdown","source":"Kaggle provides data in this format:"},{"metadata":{"_uuid":"628eec1e-f0ac-4107-b734-926dd05cf112","_cell_guid":"98cdd9f7-9586-4e04-baac-8cbbf9c71ef2","trusted":true},"cell_type":"code","source":"train_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5764f542-fa4f-4ef4-a435-3ebcaa415aff","_cell_guid":"63cd8893-fa42-4a2a-a131-5a3fd3a3ab09","trusted":true},"cell_type":"markdown","source":"We would need to preprocess these data so that for every image we have the following entry:\n```python\n{'file_name': '/kaggle/input/global-wheat-detection/train/b6ab77fd7.jpg',\n 'image_id': 0,\n 'height': 1024,\n 'width': 1024,\n 'annotations': [\n {'bbox': (834, 222, 890, 258),\n 'bbox_mode': ,\n 'category_id': 0},\n {'bbox': (226, 549, 356, 606),\n 'bbox_mode': ,\n 'category_id': 0},\n ...\n ]\n ```\nEach entry in our dataset should be a unique image with properties `file_name`, `image_id`, `height`, `width`. The `annotations` property should contain a list of all bounding box annotations.\nEach annotations is characterised by a bounding box and a category. Next we are going to write couple of functions that will allow us to transform our annotations to a suitable format."},{"metadata":{"_uuid":"5380b943-e690-4975-a26c-a28c49f014d1","_cell_guid":"fd543b9d-fcce-4516-a587-4f51753954a2","trusted":true},"cell_type":"code","source":"def wheat_dataset(df, folder, is_train):\n unique_img_names = df[\"image_id\"].unique().tolist() # Take unique image names\n df_group = df.groupby(\"image_id\") # Group the training by the image ids\n dataset_dicts = []\n\n for img_id, img_name in enumerate(tqdm(unique_img_names)):\n img_group = df_group.get_group(img_name) # Take all annotations for an image\n img_path = os.path.join(folder, img_name + \".jpg\") # Create path for an image\n if (\n is_train\n ): # Using training set, where we have multiple bounding boxes per image\n\n record = dict() # Create a record dictionary\n\n # Add image properties to the record\n record[\"file_name\"] = img_path\n record[\"image_id\"] = img_id\n record[\"height\"] = int(img_group[\"height\"].values[0])\n record[\"width\"] = int(img_group[\"width\"].values[0])\n\n annots = [] # Create annotation list\n for _, ant in img_group.iterrows():\n # bounding box is a string, so remove square brackets\n box = ant.bbox[1:-1]\n # Split the bbox values and convert it to float\n box = list(map(float, box.split(\", \")))\n # Convert to int\n x, y, w, h = list(map(int, box))\n\n # Create annotation dictionary\n annot = {\n \"bbox\": (\n x,\n y,\n x + w,\n y + h,\n ), # change to XYXY format. Original was in XYWH\n \"bbox_mode\": BoxMode.XYXY_ABS,\n \"category_id\": 0, # only one category is present in this dataset\n }\n\n # Append each annotation to the list of annotation\n annots.append(dict(annot))\n\n record[\"annotations\"] = list(annots)\n\n else: # Using submission file, where each line is an image\n\n img = cv2.imread(img_path)\n h, w = img.shape[:2]\n\n record = dict()\n record[\"file_name\"] = img_path\n record[\"image_id\"] = img_id\n record[\"height\"] = int(h)\n record[\"width\"] = int(w)\n\n dataset_dicts.append(record)\n\n return dataset_dicts\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c77b608f-e0bc-4b02-bb22-ac17f671d9ea","_cell_guid":"9cc799f9-1c14-49fe-bfdf-b78d5b3ab308","trusted":true},"cell_type":"code","source":"%%time\nimg_uniques = list(zip(range(train_df['image_id'].nunique()), train_df['image_id'].unique()))\ndataset_dicts = wheat_dataset(train_df, TRAIN_IMAGE_PATH, True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"11f805c9-1ca5-4899-91c5-ffe1c54cffb8","_cell_guid":"2ffcb817-c071-4422-b585-b2cb6c7522b4","trusted":true},"cell_type":"code","source":"from pprint import pprint\n# display the first entry to check for errors\nprint('Dataset:')\npprint(dataset_dicts[0], depth=1)\nprint('Annotation:')\npprint(dataset_dicts[0]['annotations'][0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### [CutMix](https://arxiv.org/pdf/1905.04899.pdf) Data Augmentation"},{"metadata":{},"cell_type":"markdown","source":"Here we define a CutMix data augmentation class. You can use this as a reference for implementing custom data augmentations."},{"metadata":{"_uuid":"2e473a27-35f4-4993-ac1a-8e5b0d4399b9","_cell_guid":"d1eb4335-580d-416f-89f9-4c6c33faa22c","trusted":true},"cell_type":"code","source":"class CutMix(Transform):\n def __init__(self, box_size=50, prob_cutmix=0.5):\n super().__init__()\n\n self.box_size = box_size\n self.prob_cutmix = prob_cutmix\n\n def apply_image(self, img):\n\n if random.random() < self.prob_cutmix:\n\n h, w = img.shape[:2]\n num_rand = np.random.randint(10, 20)\n for num_cut in range(num_rand):\n x_rand = random.randint(0, w - self.box_size)\n y_rand = random.randint(0, h - self.box_size)\n img[x_rand: x_rand + self.box_size, y_rand: y_rand + self.box_size, :] = 0\n\n return np.asarray(img)\n\n def apply_coords(self, coords):\n return coords.astype(np.float32)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"993235b8-d7cb-49b4-a6c7-c2b5baf3b857","_cell_guid":"91e0012c-7d66-4c7a-9d60-029cdc4d86eb","trusted":true},"cell_type":"code","source":"# visualize cutmix augmentation\nimg = CutMix(prob_cutmix=1.).apply_image(cv2.imread(dataset_dicts[0]['file_name']))\nplt.figure(figsize=(10, 10))\nplt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca798850-f302-4ca7-87fa-7402dc034d26","_cell_guid":"4c2465e2-b13b-4fe7-baec-bbfea69c8d6e","trusted":true},"cell_type":"markdown","source":"## Dataset Mapper\nNow we would need to implement a class that will load out images and annotations in compatible with Detectron2 format. We have already did half of the work by transoforming our data in the dictionary format. An example of modified [DataSetMapper](https://detectron2.readthedocs.io/_modules/detectron2/data/dataset_mapper.html)."},{"metadata":{"_uuid":"0a8d0011-5ec0-40be-8564-a27b8870b979","_cell_guid":"49ec5def-47db-42c3-8572-780e8a26c17b","trusted":true},"cell_type":"code","source":"class DatasetMapper:\n \"\"\"\n A callable which takes a dataset dict in Detectron2 Dataset format,\n and map it into a format used by the model.\n\n This is the default callable to be used to map your dataset dict into training data.\n You may need to follow it to implement your own one for customized logic,\n such as a different way to read or transform images.\n See :doc:`/tutorials/data_loading` for details.\n\n The callable currently does the following:\n\n 1. Read the image from \"file_name\"\n 2. Applies cropping/geometric transforms to the image and annotations\n 3. Prepare data and annotations to Tensor and :class:`Instances`\n \"\"\"\n\n def __init__(self, cfg, is_train=True):\n if cfg.INPUT.CROP.ENABLED and is_train:\n self.crop_gen = T.RandomCrop(cfg.INPUT.CROP.TYPE, cfg.INPUT.CROP.SIZE)\n logging.getLogger(__name__).info(\n \"CropGen used in training: \" + str(self.crop_gen)\n )\n else:\n self.crop_gen = None\n\n # Data Augmentations\n self.tfm_gens = [\n T.RandomBrightness(0.1, 1.6),\n T.RandomContrast(0.1, 3),\n T.RandomSaturation(0.1, 2),\n T.RandomRotation(angle=[90, 90]),\n T.RandomFlip(prob=0.4, horizontal=False, vertical=True),\n T.RandomCrop(\"relative_range\", (0.4, 0.6)),\n CutMix(), # custom augmentation!\n ]\n\n self.img_format = cfg.INPUT.FORMAT\n self.mask_on = cfg.MODEL.MASK_ON\n self.mask_format = cfg.INPUT.MASK_FORMAT\n self.keypoint_on = cfg.MODEL.KEYPOINT_ON\n self.load_proposals = cfg.MODEL.LOAD_PROPOSALS\n\n if self.keypoint_on and is_train:\n # Flip only makes sense in training\n self.keypoint_hflip_indices = utils.create_keypoint_hflip_indices(\n cfg.DATASETS.TRAIN\n )\n else:\n self.keypoint_hflip_indices = None\n\n if self.load_proposals:\n self.min_box_side_len = cfg.MODEL.PROPOSAL_GENERATOR.MIN_SIZE\n self.proposal_topk = (\n cfg.DATASETS.PRECOMPUTED_PROPOSAL_TOPK_TRAIN\n if is_train\n else cfg.DATASETS.PRECOMPUTED_PROPOSAL_TOPK_TEST\n )\n self.is_train = is_train\n\n def __call__(self, dataset_dict):\n \"\"\"\n Args:\n dataset_dict (dict): Metadata of one image, in Detectron2 Dataset format.\n\n Returns:\n dict: a format that builtin models in detectron2 accept\n \"\"\"\n dataset_dict = copy.deepcopy(dataset_dict) # it will be modified by code below\n # USER: Write your own image loading if it's not from a file\n image = utils.read_image(dataset_dict[\"file_name\"], format=self.img_format)\n utils.check_image_size(dataset_dict, image)\n\n if \"annotations\" not in dataset_dict:\n image, transforms = T.apply_transform_gens(\n ([self.crop_gen] if self.crop_gen else []) + self.tfm_gens, image\n )\n else:\n # Crop around an instance if there are instances in the image.\n # USER: Remove if you don't use cropping\n if self.crop_gen:\n crop_tfm = utils.gen_crop_transform_with_instance(\n self.crop_gen.get_crop_size(image.shape[:2]),\n image.shape[:2],\n np.random.choice(dataset_dict[\"annotations\"]),\n )\n image = crop_tfm.apply_image(image)\n image, transforms = T.apply_transform_gens(self.tfm_gens, image)\n if self.crop_gen:\n transforms = crop_tfm + transforms\n\n image_shape = image.shape[:2] # h, w\n\n # Pytorch's dataloader is efficient on torch.Tensor due to shared-memory,\n # but not efficient on large generic data structures due to the use of pickle & mp.Queue.\n # Therefore it's important to use torch.Tensor.\n dataset_dict[\"image\"] = torch.as_tensor(\n np.ascontiguousarray(image.transpose(2, 0, 1))\n )\n\n if not self.is_train:\n # USER: Modify this if you want to keep them for some reason.\n dataset_dict.pop(\"annotations\", None)\n dataset_dict.pop(\"sem_seg_file_name\", None)\n return dataset_dict\n\n if \"annotations\" in dataset_dict:\n # USER: Modify this if you want to keep them for some reason.\n for anno in dataset_dict[\"annotations\"]:\n if not self.mask_on:\n anno.pop(\"segmentation\", None)\n if not self.keypoint_on:\n anno.pop(\"keypoints\", None)\n\n # USER: Implement additional transformations if you have other types of data\n annos = [\n utils.transform_instance_annotations(\n obj,\n transforms,\n image_shape,\n keypoint_hflip_indices=self.keypoint_hflip_indices,\n )\n for obj in dataset_dict.pop(\"annotations\")\n if obj.get(\"iscrowd\", 0) == 0\n ]\n instances = utils.annotations_to_instances(\n annos, image_shape, mask_format=self.mask_format\n )\n # Create a tight bounding box from masks, useful when image is cropped\n if self.crop_gen and instances.has(\"gt_masks\"):\n instances.gt_boxes = instances.gt_masks.get_bounding_boxes()\n dataset_dict[\"instances\"] = utils.filter_empty_instances(instances)\n\n return dataset_dict\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once we have defined our DatasetMapper we need to register it. Each time your register a dataset a unique name should be provided."},{"metadata":{"_uuid":"5c2507a1-9eff-4895-8ee6-2101c727a8a8","_cell_guid":"c1228cf1-7ac9-4a86-83b7-3ad4d2c5f361","trusted":true},"cell_type":"code","source":"data_set_prefix = 'wheat'\nfor d in [\"train\", \"test\"]:\n DatasetCatalog.register(\n f\"{data_set_prefix}_{d}\",\n lambda d=d: wheat_dataset(\n train_df if d == \"train\" else sub_df,\n TRAIN_IMAGE_PATH if d == \"train\" else TEST_IMAGE_PATH,\n True if d == \"train\" else False,\n ),\n )\n\nmicro_metadata = MetadataCatalog.get(\"wheat_train\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets visualize our dataset."},{"metadata":{"_uuid":"9de30dd1-f84e-4320-946e-4d8083467156","_cell_guid":"09d8f796-6329-4748-8efb-e7447a6a54f7","trusted":true},"cell_type":"code","source":"%%time\ndef visualize_training_set(dataset, n_sampler=1):\n for sample in random.sample(dataset, n_sampler):\n img = cv2.imread(sample['file_name'])\n v = Visualizer(img[:, :, ::-1], metadata=micro_metadata, scale=0.5)\n v = v.draw_dataset_dict(sample)\n plt.figure(figsize = (14, 10))\n plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))\n plt.show()\n \n# dataset_dicts = wheat_dataset(train_df, TRAIN_IMAGE_PATH, True) # we have already performed this action before \nvisualize_training_set(dataset_dicts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c443f142-c388-43c9-a84b-c611bbede9de","_cell_guid":"12b21f93-b2dc-4a13-8e81-4d502d8eeb69","trusted":true},"cell_type":"markdown","source":"# Modeling"},{"metadata":{"_uuid":"9fe1b69c-dd74-4202-961f-23b86624c22d","_cell_guid":"55cccc36-9863-4715-8a8a-3eacafa4e619","trusted":true},"cell_type":"markdown","source":"We have already reviewed the structure of the configuration file. Now you we will modify the configuration file.\n1. We start with default configuration `get_cfg()`\n2. We override some fields of a default config with a config file provided in the [Model Zoo](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md)\n3. We further modify the configuration (if needed) and specify train and test dataset."},{"metadata":{"_uuid":"3822eb34-56d4-48a3-9a9d-c7521c50a565","_cell_guid":"b04ef1c8-7a75-44fc-b358-35e7a820a7c3","trusted":true},"cell_type":"markdown","source":"Lets look at the pre-defined configuration we are using in this example."},{"metadata":{"_uuid":"f0f6accc-3565-4271-9769-a7f31881512f","_cell_guid":"64762059-7ae7-4e6c-9c72-bcc89b6ee530","trusted":true},"cell_type":"code","source":"from yaml import load, FullLoader\nfrom pprint import pprint\npprint(load(open(model_zoo.get_config_file(MODEL_PATH)), Loader=FullLoader))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c21d3363-adf7-41f9-9c05-3a63026389d4","_cell_guid":"c78f6c1c-a714-496a-b42e-700eff7ccc3b","trusted":true},"cell_type":"code","source":"def cfg_setup():\n # Load default config\n cfg = get_cfg()\n # merge with specific config chosen before\n cfg.merge_from_file(model_zoo.get_config_file(MODEL_PATH))\n # since internet is not available specify exact path to the weights\n cfg.MODEL.WEIGHTS = WEIGHT_PATH\n # specify number of classes\n cfg.MODEL.RETINANET.NUM_CLASSES = 1\n \n # Number of regions per image used to train RPN\n cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256\n \n # set trianing dataset\n cfg.DATASETS.TRAIN = ('wheat_train',)\n cfg.DATASETS.TEST = ()\n \n # Number of data loading threads\n cfg.DATALOADER.NUM_WORKERS = 4\n \n cfg.SOLVER.IMS_PER_BATCH = 2\n \n # Learning rate scheduler\n cfg.SOLVER.LR_SCHEDULER_NAME = 'WarmupCosineLR'\n cfg.SOLVER.BASE_LS = 0.0002\n cfg.SOLVER.MAX_ITER = 10000\n # create a folder for the output files\n os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)\n \n return cfg\n\n\n# And now we can create a trainer taking the configuration file and Dataset Mapper\nclass WheatTrainer(DefaultTrainer):\n \n @classmethod\n def build_train_loader(cls, cfg):\n return build_detection_train_loader(cfg, mapper=DatasetMapper(cfg))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"315cd2e6-2ac7-43ba-a49e-30d98fd9f730","_cell_guid":"0113b08b-459d-44bd-bdd7-f93762e8188d","trusted":true},"cell_type":"markdown","source":"### Create the Model Using Configuration File"},{"metadata":{"_uuid":"cab7c097-b710-4d28-9786-f74443904a58","_cell_guid":"60fcc71f-f700-4e86-87b5-af20d169be0e","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\ncfg = cfg_setup()\ntrainer = WheatTrainer(cfg)\ntrainer.resume_or_load(resume=False)\n\n# Garbage collect unused files to free up some memory\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27760b34-3085-4ea3-93a0-54b3b8bc49f8","_cell_guid":"7e994c39-865d-4f46-84f4-ce805f8f6195","trusted":true},"cell_type":"markdown","source":"### Train the model"},{"metadata":{"_uuid":"c75e9c9a-6bda-454a-bfd9-d1db09b494bb","_cell_guid":"74384567-0f87-42d0-8016-f46993916369","trusted":true},"cell_type":"code","source":"%%time\ntrainer.train()\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"feaf299b-b278-482b-94fc-d48a5db5b559","_cell_guid":"a6033b9f-c2df-4370-83c5-7b5584815069","trusted":true},"cell_type":"markdown","source":"### Saving/Loading model"},{"metadata":{"_uuid":"db83350e-e4db-4493-aa44-75d957eea3ae","_cell_guid":"0f0e7a2f-1bcd-498a-adf5-0d0388de904c","trusted":true},"cell_type":"code","source":"from detectron2.checkpoint import DetectionCheckpointer\n\nfrom pathlib import Path\nPath(\"saved_models\").mkdir(exist_ok=True)\n\ncheckpointer = DetectionCheckpointer(trainer.model, save_dir=\"saved_models\")\ncheckpointer.save(\"{}\".format(MODEL_USE))\n# DetectionCheckpointer(trainer.model).load('./saved_models/rpn_r50_fpn.pth') # load a file, usually from cfg.MODEL.WEIGHTS","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74f8dda0-1895-4cc0-9c2b-dc60bf3e2736","_cell_guid":"3692e0fa-6665-4f7f-a6eb-d190096874eb","trusted":true},"cell_type":"markdown","source":"## Configuring Model for Inference"},{"metadata":{"_uuid":"98a27272-ba25-4e95-96a5-5fac15263bfc","_cell_guid":"52a957fb-8eed-4c7b-8c2c-52be4cd2a228","trusted":true},"cell_type":"code","source":"def cfg_test():\n cfg = get_cfg()\n cfg.merge_from_file(model_zoo.get_config_file(MODEL_PATH))\n cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, 'model_final.pth')\n # Here you could specify a custom path to the saved model\n# cfg.MODEL.WEIGHTS = './saved_models/rpn_r50_fpn.pth'\n\n\n cfg.DATASETS.TEST = ('wheat_test',)\n cfg.MODEL.RETINANET.NUM_CLASSES = 1\n cfg.MODEL.RETINANET.SCORE_THRESH_TEST = 0.45\n \n return cfg\n\ncfg = cfg_test()\npredict = DefaultPredictor(cfg)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7dba463a-1576-4ef1-917c-48529fda83a0","_cell_guid":"27387a6b-6321-46ef-9fac-26a1e8a050f4","trusted":true},"cell_type":"markdown","source":"### Visualize predictions"},{"metadata":{"_uuid":"bc52e50c-9e82-41a8-ae4d-c7785aa0208e","_cell_guid":"360a95a3-1263-4642-aef0-60ebb5d96f5c","trusted":true},"cell_type":"code","source":"%%time\n\ndef visual_predict(dataset):\n for sample in dataset:\n img = cv2.imread(sample['file_name'])\n output = predict(img)\n \n v = Visualizer(img[:, :, ::-1], metadata=micro_metadata, scale=0.5)\n v = v.draw_instance_predictions(output['instances'].to('cpu'))\n plt.figure(figsize = (14, 10))\n plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))\n plt.show()\n\ntest_dataset = wheat_dataset(sub_df, TEST_IMAGE_PATH, False)\nvisual_predict(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30588792-226e-4603-8019-2609b90fe14d","_cell_guid":"4547b0b1-3208-4831-801e-9ccdce0da745","trusted":true},"cell_type":"markdown","source":"# Submit!"},{"metadata":{"_uuid":"f9f326f8-fc44-40ec-8b85-0344d281039b","_cell_guid":"8d0e4e8b-d69c-4eeb-8306-f94d0ef8fe71","trusted":true},"cell_type":"code","source":"def submit():\n for idx, row in tqdm(sub_df.iterrows(), total=len(sub_df)):\n img_path = os.path.join(TEST_IMAGE_PATH, row.image_id+'.jpg')\n img = cv2.imread(img_path)\n outputs = predict(img)['instances']\n boxes = [i.cpu().detach().numpy() for i in outputs.pred_boxes]\n scores = outputs.scores.cpu().detach().numpy()\n list_str = []\n for box, score in zip(boxes, scores):\n box[3] -= box[1]\n box[2] -= box[0]\n box = list(map(int, box))\n score = round(score, 4)\n list_str.append(score) \n list_str.extend(box)\n sub_df.loc[idx, 'PredictionString'] = ' '.join(map(str, list_str))\n \n return sub_df\n\nsub_df = submit() \nsub_df.to_csv('submission.csv', index=False)\nsub_df","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}