{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Keras CenterNet Baseline Training\n\nThis is simple baseline kernal for training CenterNet with Hourglass backbone for Global Wheat Detection.\n\nThanks to [@see--](https://www.kaggle.com/seesee) for their CenterNet Keras port repo: https://github.com/see--/keras-centernet\n\nThanks to [@pestipeti](https://www.kaggle.com/pestipeti) for their competition metric kernal and visualization script: https://www.kaggle.com/pestipeti/competition-metric-details-script\n\nThanks to [@diegojohnson](https://www.kaggle.com/diegojohnson) for their heatmap script: https://www.kaggle.com/diegojohnson/centernet-objects-as-points"},{"metadata":{"id":"WzCI0t0w0HzN"},"cell_type":"markdown","source":"# Imports and Load data"},{"metadata":{"id":"GVS4Sxv4ifrY","outputId":"f6f50d68-601d-462a-ffa1-7e95aabdc586","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport cv2\nimport os\nfrom tqdm import tqdm,trange\nfrom sklearn.model_selection import train_test_split\nimport sklearn.metrics\n\nimport keras\nfrom keras.callbacks import Callback, ModelCheckpoint\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nclass config:\n    epochs = 1\n    batch_size = 1\n    num_classes = 3\n    IMAGE_PATH = '../input/global-wheat-detection/train/'\n    lr = 1e-4\n    seed = 42\n\nimport random\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n\nseed_everything(config.seed)","execution_count":null,"outputs":[]},{"metadata":{"id":"lW0yWoEjilg8","outputId":"7143e32a-5837-4fc9-e995-18438099e02b","trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/global-wheat-detection/train.csv')\ntrain_df = os.listdir(config.IMAGE_PATH)\nprint(len(train_df))\ntrain_df, val_df = train_test_split(train_df, random_state=config.seed, test_size=0.2)\nprint(len(train_df), len(val_df))\n\n# Remove empty images from training\nbad_imgs = []\nfor id in train_df:\n  if len(df[df['image_id']==id[:-4]]) == 0:\n    bad_imgs.append(id)\n\nfor im in bad_imgs:\n  train_df.remove(im)\nprint(len(train_df), len(val_df))","execution_count":null,"outputs":[]},{"metadata":{"id":"xlHJokLKYUc-"},"cell_type":"markdown","source":"# Utils"},{"metadata":{"id":"ykTGLjG605BO","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def normalize_image(image):\n  \"\"\"Normalize the image for the Hourglass network.\n  # Arguments\n    image: BGR uint8\n  # Returns\n    float32 image with the same shape as the input\n  \"\"\"\n  mean = [0.40789655, 0.44719303, 0.47026116]\n  std = [0.2886383, 0.27408165, 0.27809834]\n  return ((np.float32(image) / 255.) - mean) / std\n\nfrom math import floor\n\ndef get_boxes(bbox):\n  boxes = []\n  for box in bbox:\n    box = box[1:-1].split(',')\n    box = [float(b) for b in box]\n    box = [int(b) for b in box]\n    boxes.append(box)\n\n  boxes = np.array(boxes, dtype=np.int32)\n  return boxes\n\ndef heatmap(bbox):\n    def get_coords(bbox):\n      xs,ys,w,h=[],[],[],[]\n      for box in bbox:\n        box = box[1:-1].split(',')\n        box = [float(b) for b in box]\n        box = [int(b) for b in box]\n\n        x1, y1, width, height = box\n        xs.append(x1+int(width/2))\n        ys.append(y1+int(height/2))\n        w.append(width)\n        h.append(height)\n      \n      return xs, ys, w, h\n    \n    def get_heatmap(p_x, p_y):\n        # Ref: https://www.kaggle.com/diegojohnson/centernet-objects-as-points\n        X1 = np.linspace(1, 1024, 1024)\n        Y1 = np.linspace(1, 1024, 1024)\n        [X, Y] = np.meshgrid(X1, Y1)\n        X = X - floor(p_x)\n        Y = Y - floor(p_y)\n        D2 = X * X + Y * Y\n        sigma_ = 10\n        E2 = 2.0 * sigma_ ** 2\n        Exponent = D2 / E2\n        heatmap = np.exp(-Exponent)\n        heatmap = heatmap[:, :, np.newaxis]\n        return heatmap\n\n    coors = []\n    size = 20\n    y_ = size\n    while y_ > -size - 1:\n      x_ = -size\n      while x_ < size + 1:\n        coors.append([x_, y_])\n        x_ += 1\n      y_ -= 1\n\n    u, v, w, h = get_coords(bbox)\n    \n    if len(bbox) == 0:\n      u = np.array([512])\n      v = np.array([512])\n      w = np.array([10])\n      h = np.array([10])\n    \n    hm = np.zeros((1024,1024,1))\n    width = np.zeros((1024,1024,1))\n    height = np.zeros((1024,1024,1))\n    for i in range(len(u)):\n      for coor in coors:\n        try:\n          width[int(v[i])+coor[0], int(u[i])+coor[1]] = w[i] / 256\n          height[int(v[i])+coor[0], int(u[i])+coor[1]] = h[i] / 256\n        except:\n          pass\n      heatmap = get_heatmap(u[i], v[i])\n      hm[:,:] = np.maximum(hm[:,:],heatmap[:,:])\n      \n    hm = cv2.resize(hm, (256,256))[:,:,None]\n    width = cv2.resize(width, (256,256))[:,:,None]\n    height = cv2.resize(height, (256,256))[:,:,None]\n    return hm, width, height","execution_count":null,"outputs":[]},{"metadata":{"id":"4_kpj0kGrXmZ"},"cell_type":"markdown","source":"# Dataset"},{"metadata":{"id":"e9ZFXHO0pjXg","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, list_IDs, df, target_df=None, mode='fit',\n                 base_path=config.IMAGE_PATH, image_paths=None,\n                 batch_size=4, dim=(128, 128), n_channels=3,\n                 n_classes=3, random_state=config.seed, shuffle=True):\n        self.dim = dim\n        self.batch_size = batch_size\n        self.df = df\n        self.mode = mode\n        self.base_path = base_path\n        self.target_df = target_df\n        self.list_IDs = list_IDs\n        self.n_channels = n_channels\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.random_state = random_state\n        self.image_paths = image_paths\n        \n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_batch = [self.list_IDs[k] for k in indexes]\n        \n        X = self.__generate_X(list_IDs_batch)\n        \n        if self.mode == 'fit':\n            y = self.__generate_y(list_IDs_batch)\n            return X, y\n        \n        elif self.mode == 'predict':\n            return X\n\n        else:\n            raise AttributeError('The mode parameter should be set to \"fit\" or \"predict\".')\n        \n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indexes)\n    \n    def __generate_X(self, list_IDs_batch):\n        'Generates data containing batch_size samples'\n        X = []\n        \n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.image_paths[ID]\n            img_path = f\"{self.base_path}{im_name}\"\n            img = self.__load_rgb(img_path)\n            \n            X.append(img)\n\n        X = np.array(X)\n        return X\n    \n    def __generate_y(self, list_IDs_batch):\n        y1 = []\n        y2 = []\n        for i, ID in enumerate(list_IDs_batch):\n            image_id = self.image_paths[ID][:-4]\n            bbox = self.df[self.df['image_id']==image_id]['bbox']\n            mask, width, height = heatmap(bbox)\n            y1.append(np.concatenate([mask,width,height], axis=-1))\n            y2.append(mask)\n        \n        y1 = np.array(y1)\n        y2 = np.array(y2)\n        return [y1,y2]\n    \n    def __load_grayscale(self, img_path):\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        img = img.astype(np.float32) / 255.\n        img = np.expand_dims(img, axis=-1)\n\n        return img\n    \n    def __load_rgb(self, img_path):\n        img = cv2.cvtColor(cv2.imread(img_path), cv2.COLOR_BGR2RGB)\n\n        img = normalize_image(img)\n        return img","execution_count":null,"outputs":[]},{"metadata":{"id":"1VIZm4bYUCRA","outputId":"1d12b3af-b886-491f-f105-7ee745af8604","trusted":true},"cell_type":"code","source":"train_gen = DataGenerator(\n    list(range(len(train_df))), \n    df=df,\n    target_df=df,\n    batch_size=config.batch_size,\n    dim=(1024,1024),\n    n_classes=config.num_classes,\n    image_paths=train_df,\n    shuffle=True\n)\n\nimg, regr = train_gen.__getitem__(1)\nhm = regr[1][0][:,:,0]\nwidth = regr[0][0][:,:,1]\nheight = regr[0][0][:,:,2]\n\nimg = cv2.resize(img[0], (256,256))\n\nnrow, ncol = 3, 1\nfig, axes = plt.subplots(nrow, ncol, figsize=(20, 20))\naxes = axes.flatten()\nfor i, ax in enumerate(axes):\n  if i == 0:\n    ax.imshow(img)\n    ax.imshow(hm, alpha=0.5)\n  elif i == 1:\n    ax.imshow(img)\n    ax.imshow(width, alpha=0.5)\n  else:\n    ax.imshow(img)\n    ax.imshow(height, alpha=0.5)","execution_count":null,"outputs":[]},{"metadata":{"id":"d4d302jIOsX9"},"cell_type":"markdown","source":"# Centernet Model"},{"metadata":{"id":"O1j5pvAUsNqt","outputId":"da0ac8e6-51ac-4a47-cd08-ca7d08c46005","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import keras\nimport tensorflow as tf\nimport keras.backend as K\nfrom keras.layers import Dense, Activation, Input, Conv2D, BatchNormalization, Add, UpSampling2D, ZeroPadding2D, Lambda, Concatenate, Dropout, SpatialDropout2D\nfrom keras.models import Sequential, Model\nfrom keras.utils import get_file\n\nweights_path = get_file('centernet.hdf5',\n                          'https://github.com/see--/keras-centernet/releases/download/0.1.0/ctdet_coco_hg.hdf5',\n                          cache_subdir='/kaggle/working/', \n                          file_hash='ce01e92f75b533e3ff8e396c76d55d97ff3ec27e99b1bdac1d7b0d6dcf5d90eb')\n\ndef HourglassNetwork(heads, num_stacks, cnv_dim=256, inres=(512, 512), weights_path='/kaggle/working/centernet.hdf5',\n                     dims=[256, 384, 384, 384, 512]):\n    \"\"\"Instantiates the Hourglass architecture.\n    Optionally loads weights pre-trained on COCO.\n    Note that the data format convention used by the model is\n    the one specified in your Keras config at `~/.keras/keras.json`.\n    # Arguments\n      num_stacks: number of hourglass modules.\n      cnv_dim: number of filters after the resolution is decreased.\n      inres: network input shape, should be a multiple of 128.\n      weights: one of `None` (random initialization),\n            'ctdet_coco' (pre-training on COCO for 2D object detection),\n            'hpdet_coco' (pre-training on COCO for human pose detection),\n            or the path to the weights file to be loaded.\n      dims: numbers of channels in the hourglass blocks.\n    # Returns\n      A Keras model instance.\n    # Raises\n      ValueError: in case of invalid argument for `weights`,\n          or invalid input shape.\n    \"\"\"\n    \n    input_layer = Input(shape=(inres[0], inres[1], 3), name='HGInput')\n    inter = pre(input_layer, cnv_dim)\n    prev_inter = None\n    outputs = []\n    for i in range(num_stacks):\n        prev_inter = inter\n        _heads, inter = hourglass_module(heads, inter, cnv_dim, i, dims)\n        if i == 1:\n            if _heads is not None:\n                outputs.extend(_heads)\n        if i < num_stacks - 1:\n            inter_ = Conv2D(cnv_dim, 1, use_bias=False, name='inter_.%d.0' % i)(prev_inter)\n            inter_ = BatchNormalization(epsilon=1e-5, name='inter_.%d.1' % i)(inter_)\n\n            cnv_ = Conv2D(cnv_dim, 1, use_bias=False, name='cnv_.%d.0' % i)(inter)\n            cnv_ = BatchNormalization(epsilon=1e-5, name='cnv_.%d.1' % i)(cnv_)\n\n            inter = Add(name='inters.%d.inters.add' % i)([inter_, cnv_])\n            inter = Activation('relu', name='inters.%d.inters.relu' % i)(inter)\n            inter = residual(inter, cnv_dim, 'inters.%d' % i)\n\n    model = Model(inputs=input_layer, outputs=outputs)\n\n    # load weights\n    print('Loading weights...')\n    model.load_weights(weights_path, by_name=True)\n    print('Done!')\n\n    return model\n\ndef hourglass_module(heads, bottom, cnv_dim, hgid, dims):\n    # create left features , f1, f2, f4, f8, f16 and f32\n    lfs = left_features(bottom, hgid, dims)\n\n    # create right features, connect with left features\n    rf1 = right_features(lfs, hgid, dims)\n    rf1 = convolution(rf1, 3, cnv_dim, name='cnvs.%d' % hgid)\n\n    # add 1x1 conv with two heads, inter is sent to next stage\n    # head_parts is used for intermediate supervision\n    if heads is not None:\n        heads = create_heads(heads, rf1, hgid)\n    return heads, rf1\n\ndef convolution(_x, k, out_dim, name, stride=1):\n    padding = (k - 1) // 2\n    _x = ZeroPadding2D(padding=padding, name=name + '.pad')(_x)\n    _x = Conv2D(out_dim, k, strides=stride, use_bias=False, name=name + '.conv')(_x)\n    _x = BatchNormalization(epsilon=1e-5, name=name + '.bn')(_x)\n    _x = Activation('relu', name=name + '.relu')(_x)\n    return _x\n\ndef residual(_x, out_dim, name, stride=1):\n    shortcut = _x\n    num_channels = K.int_shape(shortcut)[-1]\n    _x = ZeroPadding2D(padding=1, name=name + '.pad1')(_x)\n    _x = Conv2D(out_dim, 3, strides=stride, use_bias=False, name=name + '.conv1')(_x)\n    _x = BatchNormalization(epsilon=1e-5, name=name + '.bn1')(_x)\n    _x = Activation('relu', name=name + '.relu1')(_x)\n\n    _x = Conv2D(out_dim, 3, padding='same', use_bias=False, name=name + '.conv2')(_x)\n    _x = BatchNormalization(epsilon=1e-5, name=name + '.bn2')(_x)\n\n    if num_channels != out_dim or stride != 1:\n        shortcut = Conv2D(out_dim, 1, strides=stride, use_bias=False, name=name + '.shortcut.0')(\n            shortcut)\n        shortcut = BatchNormalization(epsilon=1e-5, name=name + '.shortcut.1')(shortcut)\n\n    _x = Add(name=name + '.add')([_x, shortcut])\n    _x = Activation('relu', name=name + '.relu')(_x)\n    return _x\n\ndef pre(_x, num_channels):\n    # front module, input to 1/4 resolution\n    _x = convolution(_x, 7, 128, name='pre.0', stride=2)\n    _x = residual(_x, num_channels, name='pre.1', stride=2)\n    return _x\n\ndef left_features(bottom, hgid, dims):\n    # create left half blocks for hourglass module\n    # f1, f2, f4 , f8, f16, f32 : 1, 1/2, 1/4 1/8, 1/16, 1/32 resolution\n    # 5 times reduce/increase: (256, 384, 384, 384, 512)\n    features = [bottom]\n    for kk, nh in enumerate(dims):\n        pow_str = ''\n        for _ in range(kk):\n            pow_str += '.center'\n        _x = residual(features[-1], nh, name='kps.%d%s.down.0' % (hgid, pow_str), stride=2)\n        _x = residual(_x, nh, name='kps.%d%s.down.1' % (hgid, pow_str))\n        features.append(_x)\n    return features\n\ndef connect_left_right(left, right, num_channels, num_channels_next, name):\n    # left: 2 residual modules\n    left = residual(left, num_channels_next, name=name + 'skip.0')\n    left = residual(left, num_channels_next, name=name + 'skip.1')\n\n    # up: 2 times residual & nearest neighbour\n    out = residual(right, num_channels, name=name + 'out.0')\n    out = residual(out, num_channels_next, name=name + 'out.1')\n    out = UpSampling2D(name=name + 'out.upsampleNN')(out)\n    out = Add(name=name + 'out.add')([left, out])\n    return out\n\ndef bottleneck_layer(_x, num_channels, hgid):\n    # 4 residual blocks with 512 channels in the middle\n    pow_str = 'center.' * 5\n    _x = residual(_x, num_channels, name='kps.%d.%s0' % (hgid, pow_str))\n    _x = residual(_x, num_channels, name='kps.%d.%s1' % (hgid, pow_str))\n    _x = residual(_x, num_channels, name='kps.%d.%s2' % (hgid, pow_str))\n    _x = residual(_x, num_channels, name='kps.%d.%s3' % (hgid, pow_str))\n    return _x\n\ndef right_features(leftfeatures, hgid, dims):\n    rf = bottleneck_layer(leftfeatures[-1], dims[-1], hgid)\n    for kk in reversed(range(len(dims))):\n        pow_str = ''\n        for _ in range(kk):\n            pow_str += 'center.'\n        rf = connect_left_right(leftfeatures[kk], rf, dims[kk], dims[max(kk - 1, 0)], name='kps.%d.%s' % (hgid, pow_str))\n    return rf\n\ndef create_heads(heads, rf1, hgid):\n    _heads = []\n    for head in heads:\n        num_channels = heads[head]\n        _x = Conv2D(256, 3, use_bias=True, padding='same', name=head + '.%d.0.conv' % hgid)(rf1)\n        _x = Activation('relu', name=head + '.%d.0.relu' % hgid)(_x)\n        _x = Conv2D(num_channels, 1, use_bias=True, name=head + '.%d.1' % hgid)(_x)\n        _heads.append(_x)\n    return _heads\n\nkwargs = {\n        'num_stacks': 2,\n        'cnv_dim': 256,\n        'inres': (1024, 1024),\n        }\nheads = {\n        'regr': 2,\n        'confidence': 1\n        }\nmodel = HourglassNetwork(heads=heads, **kwargs)","execution_count":null,"outputs":[]},{"metadata":{"id":"b5dXgTmWlVg7"},"cell_type":"markdown","source":"# Compile Model"},{"metadata":{"id":"4Zj1RfQBPzS2","trusted":true},"cell_type":"code","source":"# Ref: https://lars76.github.io/neural-networks/object-detection/losses-for-segmentation/\nalpha = .25\ngamma = 2\ndef focal_loss_with_logits(logits, targets, alpha, gamma, y_pred):\n  weight_a = alpha * (1 - y_pred) ** gamma * targets\n  weight_b = (1 - alpha) * y_pred ** gamma * (1 - targets)\n    \n  return (tf.math.log1p(tf.exp(-tf.abs(logits))) + tf.nn.relu(-logits)) * (weight_a + weight_b) + logits * weight_b \n\ndef focal_loss(y_true, y_pred):\n  y_pred = tf.clip_by_value(y_pred, K.epsilon(), 1 - K.epsilon())\n  logits = tf.math.log(y_pred / (1 - y_pred))\n\n  loss = focal_loss_with_logits(logits=logits, targets=y_true, alpha=alpha, gamma=gamma, y_pred=y_pred)\n\n  # or reduce_sum and/or axis=-1\n  return tf.reduce_mean(loss)\n\nfrom keras.losses import mean_squared_error\n\ndef criterion(y_true, y_pred): # Regression Loss\n    mask = y_true[:,:,:, 0][:,:,:,np.newaxis]\n    regr = y_true[:,:,:, 1:]\n\n    regr_loss = mean_squared_error(regr, y_pred)\n    loss = regr_loss\n\n    return loss\n\ndef criterion2(y_true, y_pred): # Heatmap Loss\n    mask = y_true[:,:,:, 0][:,:,:,np.newaxis]\n    prediction = y_pred\n\n    # Binary mask loss\n    pred_mask = tf.sigmoid(prediction[:,:,:, 0])[:,:,:,np.newaxis]\n    mask_loss = focal_loss(mask, pred_mask)\n    mask_loss = tf.reduce_mean(mask_loss)\n\n    loss = mask_loss\n    return loss\n\nfrom keras.optimizers import Adam\nopt = Adam(lr=config.lr)\nmodel.compile(optimizer=opt, loss=[criterion, criterion2], loss_weights=[5, 1])","execution_count":null,"outputs":[]},{"metadata":{"id":"aCB95wmi2iHI"},"cell_type":"markdown","source":"# Decoder"},{"metadata":{"id":"9wV-LC_74k8G","trusted":true},"cell_type":"code","source":"def _nms(heat, kernel=3):\n  hmax = K.pool2d(heat, (kernel, kernel), padding='same', pool_mode='max')\n  keep = K.cast(K.equal(hmax, heat), K.floatx())\n  return heat * keep\n\ndef decode_ddd(regr_, hm_, k, output_stride):\n  hm = K.sigmoid(K.expand_dims(hm_[:,:,:,0]))\n  regr = regr_\n  hm = _nms(hm)\n  hm_shape = K.shape(hm)\n  regr_shape = K.shape(regr)\n  batch, width, cat = hm_shape[0], hm_shape[2], hm_shape[3]\n\n  hm_flat = K.reshape(hm, (batch, -1))\n  regr_flat = K.reshape(regr, (regr_shape[0], -1, regr_shape[-1]))\n\n  def _process_sample(args):\n    _hm, _regr = args\n    _scores, _inds = tf.math.top_k(_hm, k=k, sorted=True)\n    _classes = K.cast(_inds % cat, 'float32')\n    _inds = K.cast(_inds / cat, 'int32')\n    _xs = K.cast(_inds % width, 'float32')\n    _ys = K.cast(K.cast(_inds / width, 'int32'), 'float32')\n    _xs *= output_stride\n    _ys *= output_stride\n\n    _regr = K.gather(_regr, _inds)\n\n    _width = _regr[:,0] * 256\n    _height = _regr[:,1] * 256\n\n    _detection = K.stack([_xs, _ys, _scores, _classes, _width, _height], -1)\n    return _detection\n\n  detections = K.map_fn(_process_sample, [hm_flat, regr_flat], dtype=K.floatx())\n  return detections\n\ndef add_decoder(model, k=125, output_stride=4):\n  def _decode(args):\n    _regr, _hm = args\n    return decode_ddd(_regr, _hm, k=k, output_stride=output_stride)\n\n  output = Lambda(_decode)([*model.outputs])\n  model = Model(model.input, output)\n  return model","execution_count":null,"outputs":[]},{"metadata":{"id":"9YsAQlCbrkIw"},"cell_type":"markdown","source":"# IOU/Precision Utils"},{"metadata":{"id":"_UKt2wOHriPD","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Ref: https://www.kaggle.com/pestipeti/competition-metric-details-script\n\nfrom collections import namedtuple\nfrom typing import List, Union\n\nBox = namedtuple('Box', 'xmin ymin xmax ymax')\n\ndef calculate_iou(gt: List[Union[int, float]],\n                  pred: List[Union[int, float]],\n                  form: str = 'pascal_voc') -> float:\n    \"\"\"Calculates the IoU.\n    \n    Args:\n        gt: List[Union[int, float]] coordinates of the ground-truth box\n        pred: List[Union[int, float]] coordinates of the prdected box\n        form: str gt/pred coordinates format\n            - pascal_voc: [xmin, ymin, xmax, ymax]\n            - coco: [xmin, ymin, w, h]\n    Returns:\n        IoU: float Intersection over union (0.0 <= iou <= 1.0)\n    \"\"\"\n    if form == 'coco':\n        bgt = Box(gt[0], gt[1], gt[0] + gt[2], gt[1] + gt[3])\n        bpr = Box(pred[0], pred[1], pred[0] + pred[2], pred[1] + pred[3])\n    else:\n        bgt = Box(gt[0], gt[1], gt[2], gt[3])\n        bpr = Box(pred[0], pred[1], pred[2], pred[3])\n        \n\n    overlap_area = 0.0\n    union_area = 0.0\n\n    # Calculate overlap area\n    dx = min(bgt.xmax, bpr.xmax) - max(bgt.xmin, bpr.xmin)\n    dy = min(bgt.ymax, bpr.ymax) - max(bgt.ymin, bpr.ymin)\n\n    if (dx > 0) and (dy > 0):\n        overlap_area = dx * dy\n\n    # Calculate union area\n    union_area = (\n            (bgt.xmax - bgt.xmin) * (bgt.ymax - bgt.ymin) +\n            (bpr.xmax - bpr.xmin) * (bpr.ymax - bpr.ymin) -\n            overlap_area\n    )\n\n    return overlap_area / union_area\n\ndef find_best_match(gts, predd, threshold=0.5, form='pascal_voc'):\n    \"\"\"Returns the index of the 'best match' between the\n    ground-truth boxes and the prediction. The 'best match'\n    is the highest IoU. (0.0 IoUs are ignored).\n    \n    Args:\n        gts: Coordinates of the available ground-truth boxes\n        pred: Coordinates of the predicted box\n        threshold: Threshold\n        form: Format of the coordinates\n        \n    Return:\n        Index of the best match GT box (-1 if no match above threshold)\n    \"\"\"\n    best_match_iou = -np.inf\n    best_match_idx = -1\n    \n    for gt_idx, ggt in enumerate(gts):\n        iou = calculate_iou(ggt, predd, form=form)\n        \n        if iou < threshold:\n            continue\n        \n        if iou > best_match_iou:\n            best_match_iou = iou\n            best_match_idx = gt_idx\n\n    return best_match_idx\n\ndef calculate_precision(preds_sorted, gt_boxes, threshold=0.5, form='coco'):\n    \"\"\"Calculates precision per at one threshold.\n    \n    Args:\n        preds_sorted: \n    \"\"\"\n    tp = 0\n    fp = 0\n    fn = 0\n\n    fn_boxes = []\n\n    for pred_idx, pred in enumerate(preds_sorted):\n        best_match_gt_idx = find_best_match(gt_boxes, pred, threshold=threshold, form='coco')\n\n        if best_match_gt_idx >= 0:\n            # True positive: The predicted box matches a gt box with an IoU above the threshold.\n            tp += 1\n\n            # Remove the matched GT box\n            gt_boxes = np.delete(gt_boxes, best_match_gt_idx, axis=0)\n\n        else:\n            # No match\n            # False positive: indicates a predicted box had no associated gt box.\n            fn += 1\n            fn_boxes.append(pred)\n\n    # False negative: indicates a gt box had no associated predicted box.\n    fp = len(gt_boxes)\n    precision = tp / (tp + fp + fn)\n    return precision, fn_boxes, gt_boxes\n\ndef calculate_image_precision(preds_sorted, gt_boxes, thresholds=(0.5), form='coco', debug=False):\n    \n    n_threshold = len(thresholds)\n    image_precision = 0.0\n    \n    for threshold in thresholds:\n        precision_at_threshold, _, _ = calculate_precision(preds_sorted,\n                                                           gt_boxes,\n                                                           threshold=threshold,\n                                                           form=form\n                                                          )\n        if debug:\n            print(\"@{0:.2f} = {1:.4f}\".format(threshold, precision_at_threshold))\n\n        image_precision += precision_at_threshold / n_threshold\n    \n    return image_precision","execution_count":null,"outputs":[]},{"metadata":{"id":"7My3BSd5rnlJ"},"cell_type":"markdown","source":"# MAP"},{"metadata":{"id":"Wb-LeTch2hxR","trusted":true},"cell_type":"code","source":"def calcmAP(model, threshold=0.5):\n  model_ = add_decoder(model)\n  \n  iou_thresholds = [x for x in np.arange(0.5, 0.76, 0.05)]\n  \n  precision = []\n\n  for idx in trange(len(val_df)):\n    image_path = val_df[idx]\n    img = cv2.cvtColor(cv2.imread(f'{config.IMAGE_PATH}{image_path}'), cv2.COLOR_BGR2RGB)\n    img = normalize_image(img)\n    \n    image_id = image_path[:-4]\n    bbox = df[df['image_id']==image_id]['bbox']\n    boxes = get_boxes(bbox)\n\n    out = model_.predict(img[None])\n\n    pred_box,scores=[],[]\n\n    for detection in out[0]:\n      if detection[2] > threshold:\n        x, y, score, _, width, height = detection\n        pred_box.append([max(x-(width/2.), 0), max(y-(height/2.), 0), width, height])\n        scores.append(score)\n\n    pred_box = np.array(pred_box, dtype=np.int32)\n    scores = np.array(scores)\n\n    preds_sorted_idx = np.argsort(scores)[::-1]\n    preds_sorted = pred_box[preds_sorted_idx]\n\n    if len(boxes) > 0:\n      image_precision = calculate_image_precision(preds_sorted, boxes,\n                                                thresholds=iou_thresholds,\n                                                form='coco', debug=False)\n      precision.append(image_precision)\n    else:\n      if len(preds_sorted) > 0:\n        precision.append(0)\n  \n  precision = np.array(precision)\n  return np.mean(precision)","execution_count":null,"outputs":[]},{"metadata":{"id":"9qO3MRZ44cza","trusted":true},"cell_type":"code","source":"class SaveBestmAP(tf.keras.callbacks.Callback):\n  def __init__(self, path):\n    super(SaveBestmAP, self).__init__()\n    self.best_weights = None\n    self.path = path\n\n  def on_train_begin(self, logs=None):\n    self.best = 0\n\n  def on_epoch_end(self, epoch, logs=None):\n    current = calcmAP(self.model, 0.4)\n    if np.greater(current, self.best):\n      self.best = current\n      self.best_weights = self.model.get_weights()\n      print(f'Best mAP: {current}, saving...')\n      self.model.save_weights(self.path)\n    else:\n      print(f'Current mAP: {current}')\n  \n  def on_train_end(self, logs=None):\n    print(f'Loading best model...')\n    self.model.load_weights(self.path)","execution_count":null,"outputs":[]},{"metadata":{"id":"FzLzsCW8Qy5d"},"cell_type":"markdown","source":"# Training"},{"metadata":{"id":"hIhGlKuIP0lB","outputId":"d88790fd-4703-48ed-f66e-e386addee142","trusted":true},"cell_type":"code","source":"from keras.callbacks import ReduceLROnPlateau\n\ntrain_gen = DataGenerator(\n    list(range(len(train_df))), \n    df=df,\n    target_df=df,\n    batch_size=config.batch_size,\n    dim=(1024,1024),\n    n_classes=config.num_classes,\n    image_paths=train_df,\n    shuffle=True\n)\n\nval_gen = DataGenerator(\n    list(range(len(val_df))), \n    df=df,\n    target_df=df,\n    batch_size=config.batch_size,\n    dim=(1024,1024),\n    n_classes=config.num_classes,\n    image_paths=val_df,\n    shuffle=False\n)\n\ncheckpoint1 = ModelCheckpoint(\n    'hourglass1.h5',\n    monitor='loss', \n    verbose=0, \n    save_best_only=True,\n    save_weights_only=True,\n    mode='auto'\n)\n\ncheckpoint2 = ModelCheckpoint(\n    'hourglass1-2.h5',\n    monitor='loss', \n    verbose=0, \n    save_best_only=False,\n    save_weights_only=True,\n    mode='auto'\n)\n\nreducelr = ReduceLROnPlateau(\n    monitor='loss',\n    factor=0.25,\n    patience=2,\n    min_lr=1e-5,\n    verbose=1\n)\n\nsavemAP = SaveBestmAP('hourglass1-3.h5')\n\nhistory = model.fit_generator(\n    train_gen,\n    validation_data=val_gen,\n    epochs=config.epochs,\n    callbacks=[reducelr, checkpoint1, checkpoint2, savemAP],\n    use_multiprocessing=False,\n    workers=4\n)","execution_count":null,"outputs":[]},{"metadata":{"id":"4hotMXICK8DD"},"cell_type":"markdown","source":"# Testing"},{"metadata":{"id":"MSE35kPDSMoN","trusted":true},"cell_type":"code","source":"test_img = f'../input/global-wheat-detection/train/{val_df[0]}'\n\nbbox = df[df['image_id']==val_df[0][:-4]]['bbox']\nboxes = get_boxes(bbox)\n\nimg = cv2.cvtColor(cv2.imread(test_img), cv2.COLOR_BGR2RGB)\n\nplt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"id":"d-EQyoLziEXC","trusted":true},"cell_type":"code","source":"img = normalize_image(img)\nregr, hm = model.predict(img[None])","execution_count":null,"outputs":[]},{"metadata":{"id":"ezTYbnnYiGaz","trusted":true},"cell_type":"code","source":"img_ = cv2.resize(img, (256,256))\nplt.imshow(tf.sigmoid(hm[0][:,:,0]))","execution_count":null,"outputs":[]},{"metadata":{"id":"Oiw_ZJl8iQcH","trusted":true},"cell_type":"code","source":"plt.imshow(regr[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"id":"GMKggJQti1bi","trusted":true},"cell_type":"code","source":"plt.imshow(regr[0][:,:,1])","execution_count":null,"outputs":[]},{"metadata":{"id":"bXMgGBE2i3jH","trusted":true},"cell_type":"code","source":"model_ = add_decoder(model)","execution_count":null,"outputs":[]},{"metadata":{"id":"jt7_o6Ml6uC4","trusted":true},"cell_type":"code","source":"img = cv2.cvtColor(cv2.imread(test_img), cv2.COLOR_BGR2RGB)\nimg = normalize_image(img)\nout = model_.predict(img[None]) # bs x 125 x 6","execution_count":null,"outputs":[]},{"metadata":{"id":"G-1inlp4METZ","trusted":true},"cell_type":"code","source":"pred_box,scores=[],[]\n\nfor detection in out[0]:\n  if detection[2] > 0.25:\n    x, y, score, _, width, height = detection\n    pred_box.append([max(x-(width/2.), 0), max(y-(height/2.), 0), width, height])\n    scores.append(score)\n\npred_box = np.array(pred_box, dtype=np.int32)\nscores = np.array(scores)\n\npreds_sorted_idx = np.argsort(scores)[::-1]\npreds_sorted = pred_box[preds_sorted_idx]","execution_count":null,"outputs":[]},{"metadata":{"id":"0gLEuWmMCj5B","trusted":true},"cell_type":"code","source":"preds_sorted","execution_count":null,"outputs":[]},{"metadata":{"id":"cqm_iTRZMppi","trusted":true},"cell_type":"code","source":"def show_result(sample_id, preds, gt_boxes):\n    sample = cv2.cvtColor(cv2.imread(test_img), cv2.COLOR_BGR2RGB)\n\n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\n    for pred_box in preds:\n        cv2.rectangle(\n            sample,\n            (pred_box[0], pred_box[1]),\n            (pred_box[0] + pred_box[2], pred_box[1] + pred_box[3]),\n            (220, 0, 0), 2\n        )\n\n    if gt_boxes is not None:\n      for gt_box in gt_boxes:    \n          cv2.rectangle(\n              sample,\n              (gt_box[0], gt_box[1]),\n              (gt_box[0] + gt_box[2], gt_box[1] + gt_box[3]),\n              (0, 0, 220), 2\n          )\n\n    ax.set_axis_off()\n    ax.imshow(sample)\n    ax.set_title(\"RED: Predicted | BLUE - Ground-truth\")","execution_count":null,"outputs":[]},{"metadata":{"id":"PWFKczDpN4-M","trusted":true},"cell_type":"code","source":"iou_thresholds = [x for x in np.arange(0.5, 0.76, 0.05)]\nimage_precision = calculate_image_precision(preds_sorted, boxes,\n                                                thresholds=iou_thresholds,\n                                                form='coco', debug=False)\nprint(f'Score: {image_precision}')\n\nshow_result(1, preds_sorted, boxes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## And that is all! If you found this kernal helpful consider upvoting!\n\n## Things to experiment with:\n\n- Train for longer (100+ epochs)\n- Add augmentation\n- Change loss functions\n- Change heatmap and regression maps (change sizes)\n- Change backbone (Resnet)\n- Change model architecture (features, length)\n- Add Offset regression\n- Incorporate empty images into training\n\n## If you are interested in learning more about CenterNet I recommend these other great kernals\n\n'CenterNet -Keypoint Detector-' https://www.kaggle.com/kmat2019/centernet-keypoint-detector by [@kmat2019](https://www.kaggle.com/kmat2019) for Kuzushiji Recognition Competition\n\n'CenterNet Baseline' https://www.kaggle.com/hocop1/centernet-baseline by [@hocop1](https://www.kaggle.com/hocop1) for Peking University/Baidu - Autonomous Driving Competition\n\n#### as well as the offical Pytorch implementation and paper:\n> https://github.com/xingyizhou/CenterNet\n\n> [**Objects as Points**](http://arxiv.org/abs/1904.07850),            \n> Xingyi Zhou, Dequan Wang, Philipp Kr&auml;henb&uuml;hl,        \n> *arXiv technical report ([arXiv 1904.07850](http://arxiv.org/abs/1904.07850))*"}],"metadata":{"colab":{"name":"EfficientDet.ipynb","provenance":[],"collapsed_sections":["5KEMBfC00FYt","xlHJokLKYUc-","4_kpj0kGrXmZ","d4d302jIOsX9","b5dXgTmWlVg7","aCB95wmi2iHI","9YsAQlCbrkIw","7My3BSd5rnlJ"],"machine_shape":"hm"},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"nbformat":4,"nbformat_minor":4}