{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Version Update [3]**<br>\nAdded **ResNet-101**. It improves score `0.61+ -> 0.63+`.\n\n**Version Update [2]**<br>\nAdded **Weather** augmentation (**Rain/Snow Fall**) using `img_aug` library. It improves the scores. `0.59+ -> 0.61+`. However, the latest `img_aug` library MUST be installed, by default `(0.2.6)`, upgrade to `(0.4)`. It's done in the augmentation section.\n\n**Version Update [1]**<br>\nThere was a bug in data loader. The defined function returned all the images for `train` and `validation`. The function is re-write and OK so far.\n\n# Global Wheat Detection\n\n\nHi.<br>\nThis is a baseline [Matterport](https://github.com/matterport/Mask_RCNN) Keras implementation of **Mask-RCNN** for **Global Wheat Detection** task. \n\n---\n**Please Note**\n\nI will be using [Matterport](https://github.com/matterport/Mask_RCNN), Inc implementation. Initially I planned to use it in `TF 2.1` but ended up with `TF 1.x` because of compatible error issue. So previously when working on `TF 2.1`, I manually upgrade the necessary scripts of [**Mask-RCNN**](https://github.com/matterport/Mask_RCNN) using [tf_upgrade_v2](https://www.tensorflow.org/guide/upgrade). But though I am now using `TF 1.x` but still the converted scripts are usable. One can find the upgraded files from here [MaskRCNN Keras Source Code](https://www.kaggle.com/ipythonx/maskrcnn-keras-source-code). In this, we removed some unnecessary example notebooks, unwanted sample images and anything that are not necessary to keep work space neat and clean.\n\n\n---\n\n## Content\n* [EDA and Model Config](#1)\n    * [Simple EDA](#1)\n    * [Mask RCNN Model Configuration](#2)\n* [Preparing the Training Set](#3)  \n    * [Mask-RCNN Dataloader](#3)\n    * [Data Split](#4)\n* [Training Sample Visualization](#5)\n    * [Top Mask Position](#5)\n    * [All Mask | Sample with Masked BBox](#6)\n* [Augmentation](#7)\n* [Model Definition and Training || Inference](#8)\n* [Evaluation](#9)\n    * [Visual Evaluation](#9)\n    * [Numerical Evaluation (Comp. Metrics)](#10)\n* [Inference on Test Set](#11)\n    * [Visual Prediction](#11)\n    * [Submission](#12)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# copy to working directory\n!cp -r ../input/maskrcnn-keras-source-code/MaskRCNN/* ./","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Imports**","execution_count":null},{"metadata":{"id":"4kjcC6QqywWl","trusted":true,"_uuid":"40c67b3ff0fa04587dec508363308adaa3ceaf34","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport seaborn as sns\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport sys, os, random, glob, cv2, math\n\nfrom mrcnn import utils\nfrom mrcnn.model import log\nfrom mrcnn import visualize\nimport mrcnn.model as modellib\nfrom mrcnn.config import Config","execution_count":null,"outputs":[]},{"metadata":{"id":"yP0XLJx_x_6o","trusted":true,"_uuid":"6e5764759e6a0a9b698b44645658f66873edd807"},"cell_type":"code","source":"# for reproducibility\ndef seed_all(SEED):\n    random.seed(SEED)\n    np.random.seed(SEED)\n    os.environ['PYTHONHASHSEED'] = str(SEED)\n\nseed_all(42)\nsns.set(style=\"darkgrid\")\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simple EDA <a id=\"1\"></a>","execution_count":null},{"metadata":{"id":"KgllzLnDr7kF","outputId":"6c978df7-2013-437e-acd1-5011048dfb53","trusted":true,"_uuid":"b37d22551d332f0f7b722cc7204eb614524b6c21"},"cell_type":"code","source":"ORIG_SIZE     = 1024\nepoch         = 100\ndata_root     = '/kaggle/input'\npackages_root = '/kaggle/working'","execution_count":null,"outputs":[]},{"metadata":{"id":"-KZXyWwhzOVU","outputId":"2576cc17-7484-4311-ad72-3c5643dcb5bb","trusted":true,"_uuid":"3acbbbe055b6a409d3c50ae0f893acf51b5ae7ba"},"cell_type":"code","source":"# load annotation files\ndf = pd.read_csv(os.path.join(data_root , 'global-wheat-detection/train.csv'))\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"FghMmiMjzOX2","trusted":true,"_uuid":"50089cc61791871cdf6a5c0037dc4f28b7b7d7cc"},"cell_type":"code","source":"# information summary\ndf.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Check source distribution**","execution_count":null},{"metadata":{"trusted":true,"_uuid":"c3ee0cd0ee0b1defdec97b94bc736587c1f7631f"},"cell_type":"code","source":"plt.figure(figsize=(9,5))\nsns.countplot(df.source)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Organization informed that ` Not all images include wheat heads / bounding boxes.` We can justify that easily by following. There're about 49 image that doesn't have bbox.","execution_count":null},{"metadata":{"id":"ivqC4cnszOaM","trusted":true,"_uuid":"778cb19865d7cc63440491aef9202b71c61e8bb2"},"cell_type":"code","source":"# image directory\nimg_root = '../input/global-wheat-detection/train/'\nlen(os.listdir(img_root)) - len(df.image_id.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's modify the annotation file for feasible use. The `bbox` values are in one column, we will make them separate in different attributes.","execution_count":null},{"metadata":{"id":"_SfzTa-1zOck","outputId":"91ae8935-bccb-4b8e-9a7e-aa690f95fd9b","trusted":true,"_uuid":"dfcffc4eaa94a41497717851dee9f702d8a2a73b"},"cell_type":"code","source":"df['bbox'] = df['bbox'].apply(lambda x: x[1:-1].split(\",\"))\n\ndf['x'] = df['bbox'].apply(lambda x: x[0]).astype('float32')\ndf['y'] = df['bbox'].apply(lambda x: x[1]).astype('float32')\ndf['w'] = df['bbox'].apply(lambda x: x[2]).astype('float32')\ndf['h'] = df['bbox'].apply(lambda x: x[3]).astype('float32')\n\ndf = df[['image_id','x', 'y', 'w', 'h']]\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mask-RCNN Model Configuration <a id=\"2\"></a>","execution_count":null},{"metadata":{"id":"8EBVA1M60yAj","trusted":true,"_uuid":"52bd3ffbdde0173a363055482d675da51c2aba99"},"cell_type":"code","source":"class WheatDetectorConfig(Config):\n    # Give the configuration a recognizable name  \n    NAME = 'wheat'\n    \n    # set the number of GPUs to use along with the number of images\n    # per GPU\n    GPU_COUNT = 1\n    IMAGES_PER_GPU = 2\n    BACKBONE = 'resnet101'\n    \n    # number of classes (we would normally add +1 for the background)\n    # BG + Wheat\n    NUM_CLASSES = 2\n    \n    IMAGE_RESIZE_MODE = \"square\"\n    IMAGE_MIN_DIM = 1024\n    IMAGE_MAX_DIM = 1024\n    \n    # Number of training steps per epoch\n    STEPS_PER_EPOCH = 120\n    \n    # Use different size anchors because our target objects are multi-scale (wheats are some too big, some too small)\n    RPN_ANCHOR_SCALES = (16, 32, 64, 128)  # anchor side in pixels\n    \n    # Learning rate\n    LEARNING_RATE = 0.005\n    WEIGHT_DECAY  = 0.0005\n    \n    # Maximum number of ROI’s, the Region Proposal Network (RPN) will generate for the image\n    TRAIN_ROIS_PER_IMAGE = 350 \n    \n    # Skip detections with < 60% confidence\n    DETECTION_MIN_CONFIDENCE = 0.60\n    \n    # Increase with larger training\n    VALIDATION_STEPS = 60\n    \n    # Maximum number of instances that can be detected in one image.\n    MAX_GT_INSTANCES = 500 # 200 \n \n    # Loss weights for more precise optimization.\n    # Can be used for R-CNN training setup.\n    LOSS_WEIGHTS = {\n        \"rpn_class_loss\": 1.0,\n        \"rpn_bbox_loss\": 1.0,\n        \"mrcnn_class_loss\": 1.0,\n        \"mrcnn_bbox_loss\": 1.0,\n        \"mrcnn_mask_loss\": 1.0\n        }\n\nconfig = WheatDetectorConfig()\nconfig.display()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Preparing <a id=\"3\"></a>","execution_count":null},{"metadata":{"id":"EdhUEFDr0yDA","outputId":"1715a5df-a577-41fd-bf20-f1a27aadb28c","trusted":true,"_uuid":"793b1c6c6ba4e5f0d51e130080aa799f230b5ef6"},"cell_type":"code","source":"def get_jpg(img_dir, anns):\n    '''\n    input:\n        img_dir: image directory of the train sets\n        anns: specified image ids for train or validation\n    return:\n        img files with specified image ids\n    '''\n    id      = []\n    jpg_fps = []\n\n    for index, row in anns.iterrows():\n        id.append(row['image_id'])\n\n    for i in os.listdir(img_dir):\n        if os.path.splitext(i)[0] not in id:\n            continue\n        else:\n            jpg_fps.append(os.path.join(img_dir, i))\n\n    return list(set(jpg_fps))\n\ndef get_dataset(img_dir, anns): \n    image_fps = get_jpg(img_dir, anns)\n\n    image_annotations = {fp: [] for fp in image_fps}\n\n    for index, row in anns.iterrows(): \n        fp = os.path.join(img_dir, row['image_id'] + '.jpg')\n        image_annotations[fp].append(row)\n\n    return image_fps, image_annotations ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Generator for Mask-RCNN <a id=\"3\"></a>","execution_count":null},{"metadata":{"id":"Mxz-pNbt5txY","trusted":true,"_uuid":"7aebc88f910b232e3b8759421914a007c6ffed94"},"cell_type":"code","source":"class DetectorDataset(utils.Dataset):\n    def __init__(self, image_fps, image_annotations, orig_height, orig_width):\n        super().__init__(self)\n        \n        # Add classes\n        self.add_class('GlobalWheat', 1 , 'Wheat') # only one class, wheat\n        \n        # add images \n        for id, fp in enumerate(image_fps):\n            annotations = image_annotations[fp]\n            self.add_image('GlobalWheat', image_id=id, \n                           path=fp, annotations=annotations, \n                           orig_height=orig_height, orig_width=orig_width)\n\n    # load bbox, most important function so far        \n    def load_mask(self, image_id):\n        info = self.image_info[image_id]\n        annotations = info['annotations']\n        count = len(annotations)\n    \n        if count == 0:\n            mask = np.zeros((info['orig_height'], info['orig_width'], 1), \n                            dtype=np.uint8)\n            class_ids = np.zeros((1,), dtype=np.int32)\n        else:\n            mask = np.zeros((info['orig_height'], info['orig_width'], count),\n                            dtype=np.uint8)\n            class_ids = np.zeros((count,), dtype=np.int32)\n            for i, a in enumerate(annotations):\n                x = int(a['x'])\n                y = int(a['y'])\n                w = int(a['w'])\n                h = int(a['h'])\n                mask_instance = mask[:, :, i].copy()\n                cv2.rectangle(mask_instance, (x, y), (x+w, y+h), 255, -1)\n                mask[:, :, i] = mask_instance\n                class_ids[i] = 1\n        return mask.astype(np.bool), class_ids.astype(np.int32)\n    \n    # simple image loader \n    def load_image(self, image_id):\n        info = self.image_info[image_id]\n        fp = info['path']\n        image = cv2.imread(fp, cv2.IMREAD_COLOR)\n        # If grayscale. Convert to RGB for consistency.\n        if len(image.shape) != 3 or image.shape[2] != 3:\n            image = np.stack((image,) * 3, -1)\n        return image\n    \n    # simply return the image path\n    def image_reference(self, image_id):\n        info = self.image_info[image_id]\n        return info['path']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Splits Data Sets <a id=\"4\"></a>","execution_count":null},{"metadata":{"id":"YPqjEIXWRhSf","trusted":true,"_uuid":"6c386dcef041b972f6209dd19e247d547c3c349f"},"cell_type":"code","source":"image_ids = df['image_id'].unique()\n\nvalid_ids = image_ids[-700:]\ntrain_ids = image_ids[:-700]\n\nvalid_df = df[df['image_id'].isin(valid_ids)]\ntrain_df = df[df['image_id'].isin(train_ids)]\ntrain_df.shape, valid_df.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"81lovwF2Ro5R","outputId":"e2263fe2-1a32-432a-ec75-b9220a24e697","trusted":true,"_uuid":"0ef68a41cf1a5e842e86a219b6392e3695004720"},"cell_type":"code","source":"len(train_df.image_id.unique()), len(valid_df.image_id.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build Train Set","execution_count":null},{"metadata":{"id":"gYNSd1AhRqOV","trusted":true,"_uuid":"74277ae9af4a3b044e62b664d10d76b23848bb43"},"cell_type":"code","source":"# grab all image file path with concern annotation\ntrain_image_fps, train_image_annotations = get_dataset(img_root,\n                                                       anns=train_df)\n\n# make data generator with that\ndataset_train = DetectorDataset(train_image_fps, \n                                train_image_annotations,\n                                ORIG_SIZE, ORIG_SIZE)\ndataset_train.prepare()\n\nprint(\"Class Count: {}\".format(dataset_train.num_classes))\nfor i, info in enumerate(dataset_train.class_info):\n    print(\"{:3}. {:50}\".format(i, info['name']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build Validation Set","execution_count":null},{"metadata":{"id":"7jByVCZt-ZOC","outputId":"f1aa267d-7530-4620-ffc5-2f7aa39083bb","trusted":true,"_uuid":"6175c72e73639e3190e127f67783988eadced9ba"},"cell_type":"code","source":"# grab all image file path with concern annotation\nvalid_image_fps, valid_image_annotations = get_dataset(img_root, \n                                           anns=valid_df)\n\n# make data generator with that\ndataset_valid = DetectorDataset(valid_image_fps, valid_image_annotations,\n                                ORIG_SIZE, ORIG_SIZE)\ndataset_valid.prepare()\n\nprint(\"Class Count: {}\".format(dataset_valid.num_classes))\nfor i, info in enumerate(dataset_valid.class_info):\n    print(\"{:3}. {:50}\".format(i, info['name']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Samples <a id=\"5\"></a>\n\nUsing `dataset_train`, let's observe some sample data.","execution_count":null},{"metadata":{"id":"jwMkhotP0yFf","trusted":true,"_uuid":"86c3333d4dfb8b7d00ce1f401693d0df4e6254e1"},"cell_type":"code","source":"class_ids = [0]\n\nwhile class_ids[0] == 0:  ## look for a mask\n    image_id = random.choice(dataset_train.image_ids)\n    image_fp = dataset_train.image_reference(image_id)\n    image = dataset_train.load_image(image_id)\n    mask, class_ids = dataset_train.load_mask(image_id)\n\nprint(image.shape)\n\nplt.figure(figsize=(15, 15))\nplt.subplot(1, 2, 1)\nplt.imshow(image)\nplt.axis('off')\n\nplt.subplot(1, 2, 2)\nmasked = np.zeros(image.shape[:2])\nfor i in range(mask.shape[2]):\n    masked += image[:, :, 0] * mask[:, :, i]\nplt.imshow(masked, cmap='gray')\nplt.axis('off')\n\nprint(class_ids)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Top Mask Position <a id=\"5\"></a>\n\nLet's display some sample and corresponding mask (here which is bounding box indicator).","execution_count":null},{"metadata":{"id":"0xEc47Jz59x5","outputId":"129edfbc-cf9d-46c7-b569-d804a50cd12d","trusted":true,"_uuid":"93da5a58731ad483a4bd2b20543f2b1df4b8ad74"},"cell_type":"code","source":"# Load and display random samples\nimage_ids = np.random.choice(dataset_train.image_ids,5)\nfor image_id in image_ids:\n    image = dataset_train.load_image(image_id)\n    mask, class_ids = dataset_train.load_mask(image_id)\n    visualize.display_top_masks(image, mask, class_ids, \n                                dataset_train.class_names, limit=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# BBoxes with Masked Sample <a id=\"6\"></a>\n\nIn `Mask-RCNN`, the aspect ratio is preserved, though. If an image is not square, then zero padding is added at the `top/bottom` or `right/left`.","execution_count":null},{"metadata":{"id":"K1TkWuGP0yHl","trusted":true,"_uuid":"313347d838fa8321a714858c8073f98c50c5be26"},"cell_type":"code","source":"# Load random image and mask.\nimage_id = np.random.choice(dataset_train.image_ids, 1)[0]\nimage = dataset_train.load_image(image_id)\nmask, class_ids = dataset_train.load_mask(image_id)\noriginal_shape = image.shape\n\n# Resize\nimage, window, scale, padding, _ = utils.resize_image(image, \n                                                      min_dim=config.IMAGE_MIN_DIM, \n                                                      max_dim=config.IMAGE_MAX_DIM,\n                                                      mode=config.IMAGE_RESIZE_MODE)\nmask = utils.resize_mask(mask, scale, padding)\n\n# Compute Bounding box\nbbox = utils.extract_bboxes(mask)\n\n# Display image and additional stats\nprint(\"Original shape: \", original_shape)\nlog(\"image\", image)\nlog(\"mask\", mask)\nlog(\"class_ids\", class_ids)\nlog(\"bbox\", bbox)\n\n# Display image and instances\nvisualize.display_instances(image, bbox, mask, class_ids, \n                            dataset_train.class_names)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentation <a id=\"7\"></a>\n\nAugmentation is the key part to boost performance. Here are some `weather` looking augmentation which are implemented using `img_aug` library. However the current version of `img_aug` is `0.2.6` which needs to upgrade to `0.4` for such augmentation.\n\n```\n- Afine Transform\n- Flip\n- Cutout + CoarseDropout\n- Snowflakes\n- Rain\n```","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install ../input/img-aug-v04/imgaug-0.4.0-py2.py3-none-any.whl","execution_count":null,"outputs":[]},{"metadata":{"id":"4xwsrf9G1lHR","outputId":"a13386d3-a918-41fe-8824-13625c9d7b08","trusted":true,"_uuid":"491b78ec96d28fcdbbf8e2d7f9320a05d64c9249"},"cell_type":"code","source":"import warnings\nfrom imgaug import augmenters as iaa\nwarnings.filterwarnings(\"ignore\")\n\naugmentation = iaa.Sequential([\n        iaa.OneOf([ ## rotate\n            iaa.Affine(rotate=0),\n            iaa.Affine(rotate=90),\n            iaa.Affine(rotate=180),\n            iaa.Affine(rotate=270),\n        ]),\n\n        iaa.Fliplr(0.5),\n        iaa.Flipud(0.2),\n\n        iaa.OneOf([ # drop out augmentation\n            iaa.Cutout(fill_mode=\"constant\", cval=255),\n            iaa.CoarseDropout((0.0, 0.05), size_percent=(0.02, 0.25)),\n            ]),\n\n        iaa.OneOf([ ## weather augmentation\n            iaa.Snowflakes(flake_size=(0.2, 0.4), speed=(0.01, 0.07)),\n            iaa.Rain(speed=(0.3, 0.5)),\n        ]),  \n\n        iaa.OneOf([ ## brightness or contrast\n            iaa.Multiply((0.8, 1.0)),\n            iaa.contrast.LinearContrast((0.9, 1.1)),\n        ]),\n\n        iaa.OneOf([ ## blur or sharpen\n            iaa.GaussianBlur(sigma=(0.0, 0.1)),\n            iaa.Sharpen(alpha=(0.0, 0.1)),\n        ])\n    ],\n    # do all of the above augmentations in random order\n    random_order=True\n)","execution_count":null,"outputs":[]},{"metadata":{"id":"STZnQTE61lME","trusted":true,"_uuid":"4ab9d6086ce611a46f189c047956c43b29783e6d"},"cell_type":"code","source":"# from official repo\ndef get_ax(rows=1, cols=1, size=7):\n    \"\"\"Return a Matplotlib Axes array to be used in\n    all visualizations in the notebook. Provide a\n    central point to control graph sizes.\n    \n    Adjust the size attribute to control how big to render images\n    \"\"\"\n    _, ax = plt.subplots(rows, cols, figsize=(size*cols, size*rows))\n    return ax\n\n\n# Load the image multiple times to show augmentations\nlimit = 4\nax = get_ax(rows=2, cols=limit//2)\n\nfor i in range(limit):\n    image, image_meta, class_ids,\\\n    bbox, mask = modellib.load_image_gt(\n        dataset_train, config, image_id, use_mini_mask=False, \n        augment=False, augmentation=augmentation)\n    \n    visualize.display_instances(image, bbox, mask, class_ids,\n                                dataset_train.class_names, ax=ax[i//2, i % 2],\n                                show_mask=False, show_bbox=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Model <a id=\"8\"></a>\n\nTime to build the model. I will use [`mask_rcnn_coco.h5`](https://www.kaggle.com/ipythonx/cocowg) pre-trained model and train the model by initializing with it.","execution_count":null},{"metadata":{"trusted":true,"_uuid":"138d6197fc8dce9f1f8a7b5a6c27aa2069698e03"},"cell_type":"code","source":"def model_definition():\n    print(\"loading mask R-CNN model\")\n    model = modellib.MaskRCNN(mode='training', \n                              config=config, \n                              model_dir=packages_root)\n    \n    # load the weights for COCO\n    model.load_weights(data_root + '/cocowg/mask_rcnn_coco.h5',\n                       by_name=True, \n                       exclude=[\"mrcnn_class_logits\",\n                                \"mrcnn_bbox_fc\",  \n                                \"mrcnn_bbox\",\"mrcnn_mask\"])\n    return model   \n\nmodel = model_definition()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import (ModelCheckpoint, ReduceLROnPlateau, CSVLogger)\n\ndef callback():\n    cb = []\n    checkpoint = ModelCheckpoint(packages_root+'wheat_wg.h5',\n                                 save_best_only=True,\n                                 mode='min',\n                                 monitor='val_loss',\n                                 save_weights_only=True, verbose=1)\n    cb.append(checkpoint)\n    reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss',\n                                   factor=0.3, patience=5,\n                                   verbose=1, mode='auto',\n                                   epsilon=0.0001, cooldown=1, min_lr=0.00001)\n    log = CSVLogger(packages_root+'wheat_history.csv')\n    cb.append(log)\n    cb.append(reduceLROnPlat)\n    return cb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Inference Configuration**\n\nI've trained the model on-site. I set `epoch` 100 but the model converged within `50` but later slighty improved in next few more epoch. I didn't have the intention to train longer though. I started the training and went to sleep; next is history. :D","execution_count":null},{"metadata":{"trusted":true,"_uuid":"8004790d27f041793562e994bbe95edf67f8978b","_kg_hide-output":true},"cell_type":"code","source":"%%time\nCB = callback()\nTRAIN = False\n\nclass WheatInferenceConfig(WheatDetectorConfig):\n    GPU_COUNT = 1\n    IMAGES_PER_GPU = 1\n\nif TRAIN:\n    model.train(dataset_train, dataset_valid, \n                augmentation=augmentation, \n                learning_rate=config.LEARNING_RATE,\n                custom_callbacks = CB,\n                epochs=epoch, layers='all') \nelse:\n    inference_config = WheatInferenceConfig()\n    # Recreate the model in inference mode\n    model = modellib.MaskRCNN(mode='inference', \n                              config=inference_config,\n                              model_dir=packages_root)\n    \n    model.load_weights(data_root + '/096269-wheat-r101/wheat_096269_101_1024.h5', \n                       by_name = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Learning Curves**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = pd.read_csv(data_root + '/wheatweight/wheat_history.csv') \n\n# find the lowest validation loss score\nprint(history.loc[history['val_loss'].idxmin()])\nhistory.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(19,6))\n\nplt.subplot(131)\nplt.plot(history.epoch, history.loss, label=\"Train loss\")\nplt.plot(history.epoch, history.val_loss, label=\"Valid loss\")\nplt.legend()\n\nplt.subplot(132)\nplt.plot(history.epoch, history.mrcnn_class_loss, label=\"Train class ce\")\nplt.plot(history.epoch, history.val_mrcnn_class_loss, label=\"Valid class ce\")\nplt.legend()\n\nplt.subplot(133)\nplt.plot(history.epoch, history.mrcnn_bbox_loss, label=\"Train box loss\")\nplt.plot(history.epoch, history.val_mrcnn_bbox_loss, label=\"Valid box loss\")\nplt.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation <a id=\"9\"></a>  \n\nWe will evaluate the model performance in both ways: `visual interpretation` and `numerical` or mainly competition metrices (`mAP(0.5:0.75:0.05)`. But I know that most of the cases `visual interpretation` doesn't really matter (except in medical domain). ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_id = np.random.choice(dataset_valid.image_ids, 2)\n\nfor img_id in image_id:\n    original_image, image_meta, gt_class_id, gt_bbox, gt_mask =\\\n        modellib.load_image_gt(dataset_valid, inference_config,     \n                               img_id, use_mini_mask=False)\n\n    info = dataset_valid.image_info[img_id]\n    results = model.detect([original_image], verbose=1)\n    r = results[0]\n\n    visualize.display_instances(original_image, r['rois'], r['masks'], r['class_ids'], \n                                dataset_valid.class_names, r['scores'], ax=get_ax(), title=\"Predictions\")\n    \n    log(\"image_meta\", image_meta)\n    log(\"gt_class_id\", gt_class_id)\n    log(\"gt_bbox\", gt_bbox)\n    log(\"gt_mask\", gt_mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Competition Metrics <a id=\"10\"></a>\n\nThe following functons takes good amount of time to evaluate the average precision scores withing the given `IoU` threshold scores on the validation set. So, please consider if you want to use it.","execution_count":null},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"%%time\n\nthresh_score = [0.5 , 0.55, 0.6 , 0.65, 0.7 , 0.75]\n\ndef evaluate_threshold_range(test_set, image_ids, model, \n                             iou_thresholds, inference_config):\n    '''Calculate mAP based on iou_threshold range\n    inputs:\n        test_set        : test samples\n        image_ids       : image ids of the test samples\n        model           : trained model\n        inference_config: test configuration\n        iou_threshold   : by default [0.5:0.75:0.05]\n    return:\n        AP : mAP[@0.5:0.75] scores lists of the test samples\n    '''\n    # placeholder for all the ap of all classes for IoU socres 0.5 to 0.95 with step size 0.05\n    AP = []\n    np.seterr(divide='ignore', invalid='ignore') \n    \n    for image_id in image_ids:\n        # Load image and ground truth data\n        image, image_meta, gt_class_id, gt_bbox, gt_mask =\\\n            modellib.load_image_gt(test_set, inference_config,\n                                   image_id, use_mini_mask=False)\n\n        # Run object detection\n        results = model.detect([image], verbose=0)\n        r = results[0]\n        AP_range = utils.compute_ap_range(gt_bbox, gt_class_id, gt_mask, \n                                          r[\"rois\"], r[\"class_ids\"], r[\"scores\"], r['masks'],\n                                          iou_thresholds=iou_thresholds, verbose=0)\n        \n        if math.isnan(AP_range):\n            continue\n            \n        # append the scores of each samples\n        AP.append(AP_range)   \n        \n    return AP\n\nAP = evaluate_threshold_range(dataset_valid, dataset_valid.image_ids,\n                              model, thresh_score, inference_config)\n\nprint(\"AP[0.5:0.75]: \", np.mean(AP))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference on Test Set <a id=\"1\"></a>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_jpg(img_dir):\n    jpg_fps = glob.glob(img_dir + '*.jpg')\n    return list(set(jpg_fps))\n\n# Get filenames of test dataset jpg images\ntest_img_root  = data_root + '/global-wheat-detection/test/'\ntest_image_fps = get_jpg(test_img_root)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visual Prediction <a id=\"11\"></a>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# show a few test image detection example\nfor image_id in test_image_fps:\n    image = cv2.imread(image_id, cv2.IMREAD_COLOR)\n\n    # assume square image \n    resize_factor = ORIG_SIZE / config.IMAGE_SHAPE[0]\n\n    # If grayscale. Convert to RGB for consistency.\n    if len(image.shape) != 3 or image.shape[2] != 3:\n        image = np.stack((image,) * 3, -1) \n\n    resized_image, window, scale, padding, crop = utils.resize_image(\n        image,\n        min_dim=config.IMAGE_MIN_DIM,\n        min_scale=config.IMAGE_MIN_SCALE,\n        max_dim=config.IMAGE_MAX_DIM,\n        mode=config.IMAGE_RESIZE_MODE)\n\n    image_id = os.path.splitext(os.path.basename(image_id))[0]\n\n    results = model.detect([resized_image])\n    r = results[0]\n    for bbox in r['rois']: \n        x1 = int(bbox[1] * resize_factor)\n        y1 = int(bbox[0] * resize_factor)\n        x2 = int(bbox[3] * resize_factor)\n        y2 = int(bbox[2] * resize_factor)\n        cv2.rectangle(image, (x1,y1), (x2,y2), (77, 255, 9), 3, 1)\n        width  = x2 - x1 \n        height = y2 - y1 \n\n    plt.figure(figsize=(10,10)) \n    plt.grid(False)\n    plt.imshow(image, cmap=plt.cm.gist_gray)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission <a id=\"12\"></a>\n\nYes bro! Like you, I've also faced stupid `Submission Scoring Error` around `15` times. And when I solved, it felt as same as winning the competition. LoL :D","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make predictions on test images, write out sample submission\ndef predict(image_fps, filepath='submission.csv', min_conf=0.50):\n    # assume square image\n    resize_factor = ORIG_SIZE / config.IMAGE_SHAPE[0]\n\n    with open(filepath, 'w') as file:\n        file.write(\"image_id,PredictionString\\n\")\n\n        for image_id in tqdm(image_fps):\n            image = cv2.imread(image_id, cv2.IMREAD_COLOR)\n            # If grayscale. Convert to RGB for consistency.\n            if len(image.shape) != 3 or image.shape[2] != 3:\n                image = np.stack((image,) * 3, -1)\n                \n            image, window, scale, padding, crop = utils.resize_image(\n                image,\n                min_dim=config.IMAGE_MIN_DIM,\n                min_scale=config.IMAGE_MIN_SCALE,\n                max_dim=config.IMAGE_MAX_DIM,\n                mode=config.IMAGE_RESIZE_MODE)\n\n            image_id = os.path.splitext(os.path.basename(image_id))[0]\n\n            results = model.detect([image])\n            r = results[0]\n\n            out_str = \"\"\n            out_str += image_id\n            out_str += \",\"\n            \n            assert( len(r['rois']) == len(r['class_ids']) == len(r['scores']) )\n            \n            if len(r['rois']) == 0:\n                pass\n            else:\n                num_instances = len(r['rois'])\n                for i in range(num_instances):\n                    if r['scores'][i] > min_conf:\n                               \n                        out_str += ' '\n                        out_str += \"{0:.4f}\".format(r['scores'][i])\n                        out_str += ' '\n\n                        # x1, y1, width, height\n                        x1 = r['rois'][i][1]\n                        y1 = r['rois'][i][0]\n                        width = r['rois'][i][3] - x1\n                        height = r['rois'][i][2] - y1\n                        bboxes_str = \"{} {} {} {}\".format( x1*resize_factor, y1*resize_factor, \\\n                                                           width*resize_factor, height*resize_factor )\n                        out_str += bboxes_str\n\n            file.write(out_str+\"\\n\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = os.path.join(packages_root, 'submission.csv')\npredict(test_image_fps, filepath=submission)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submit = pd.read_csv(submission)\nsubmit.head(10)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}