{
  "id": 52621,
  "title": "faster-rcnn + crop image + mask estimation",
  "url": "/competitions/data-science-bowl-2018/discussion/52621",
  "author_name": "",
  "post_date": "2018-03-21T15:34:42.567873Z",
  "votes": 12,
  "comment_count": 12,
  "views": 0,
  "content": "<p>I decided to try a new approach, split a end-to-end mask rcnn into two-stage approach: detect nuclei first (faster-rcnn) and then crop image and predict mask (fcn).</p>\n\n<p>From application point of view, this is not so efficient. But from competition point of view, this seems to be easier to try new ideas, especially with regards to ensembling, mask refinement and pesudo-labeling and deeper network.</p>\n\n<p>(Note: using method of data or knowledge distillation, we still can transfer knowledge to end-to-end system later for real application)</p>\n\n<p>Here i show results </p>\n\n<p>stage-1: i.e. faster-rcnn alone:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/300392/8836/432f367a4c5b5674de2e2977744d10289a064e5704b21af6607b4975be47c580.rcnn_proposal.png\" alt=\"enter image description here\"></p>\n\n<p>stage-2: i.e. crop image + mask prediction</p>\n\n<p>... to be updated ...</p>\n\n<p>there is no submission to LB yet</p>",
  "messages": [
    {
      "id": "300392",
      "postDate": "03/21/2018 15:34:42",
      "content": "<p>I decided to try a new approach, split a end-to-end mask rcnn into two-stage approach: detect nuclei first (faster-rcnn) and then crop image and predict mask (fcn).</p>\n\n<p>From application point of view, this is not so efficient. But from competition point of view, this seems to be easier to try new ideas, especially with regards to ensembling, mask refinement and pesudo-labeling and deeper network.</p>\n\n<p>(Note: using method of data or knowledge distillation, we still can transfer knowledge to end-to-end system later for real application)</p>\n\n<p>Here i show results </p>\n\n<p>stage-1: i.e. faster-rcnn alone:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/300392/8836/432f367a4c5b5674de2e2977744d10289a064e5704b21af6607b4975be47c580.rcnn_proposal.png\" alt=\"enter image description here\"></p>\n\n<p>stage-2: i.e. crop image + mask prediction</p>\n\n<p>... to be updated ...</p>\n\n<p>there is no submission to LB yet</p>",
      "rawMarkdown": "I decided to try a new approach, split a end-to-end mask rcnn into two-stage approach: detect nuclei first (faster-rcnn) and then crop image and predict mask (fcn).\n\nFrom application point of view, this is not so efficient. But from competition point of view, this seems to be easier to try new ideas, especially with regards to ensembling, mask refinement and pesudo-labeling and deeper network.\n\n(Note: using method of data or knowledge distillation, we still can transfer knowledge to end-to-end system later for real application)\n\nHere i show results \n\nstage-1: i.e. faster-rcnn alone:\n\n  ![enter image description here][1]\n\nstage-2: i.e. crop image + mask prediction\n\n... to be updated ...\n\nthere is no submission to LB yet\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/300392/8836/432f367a4c5b5674de2e2977744d10289a064e5704b21af6607b4975be47c580.rcnn_proposal.png",
      "votes": null
    },
    {
      "id": "300601",
      "postDate": "03/21/2018 18:47:16",
      "content": "<p>Congrats for your new LB. Hope this is a better idea.</p>",
      "rawMarkdown": "Congrats for your new LB. Hope this is a better idea.",
      "votes": null
    },
    {
      "id": "300842",
      "postDate": "03/22/2018 00:56:58",
      "content": "<p>As you said, we have no benefit for this way. But do you think fasterrcnn only can provide better detection result than maskrcnn? Because it has more simple loss  ( ignore mask loss) tends to easy to optimal. However, the main disadvantage of this way is that provides more than one cell in one box thus difficult to seperate them. And the spatial resolution of cell in box is very small ( 56x56, I guess) thus the deeper network may not good for segmentation</p>",
      "rawMarkdown": "As you said, we have no benefit for this way. But do you think fasterrcnn only can provide better detection result than maskrcnn? Because it has more simple loss  ( ignore mask loss) tends to easy to optimal. However, the main disadvantage of this way is that provides more than one cell in one box thus difficult to seperate them. And the spatial resolution of cell in box is very small ( 56x56, I guess) thus the deeper network may not good for segmentation",
      "votes": null
    },
    {
      "id": "302148",
      "postDate": "03/23/2018 18:29:13",
      "content": "<p>Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... </p>\n\n<ol>\n<li>RPN for proposals</li>\n<li>MaskHead for predicting masks for each proposal.</li>\n</ol>\n\n<p>That's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.</p>",
      "rawMarkdown": "Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... \n\n1. RPN for proposals\n2. MaskHead for predicting masks for each proposal.\n\nThat's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.",
      "votes": null
    },
    {
      "id": "302164",
      "postDate": "03/23/2018 18:56:07",
      "content": "<p>Unless I am missing something in my understanding as I am still new to Mask-RCNN, here is my understanding. </p>\n\n<p>Technically we do still have 2 classes = Background + Nuclei. I think it helps for the CNN to classify the features that would be associated with the object we want versus everything else. From there those features will get captured by the anchor boxes in the RPN and refined to become the bounding box. </p>\n\n<p>So without the CNN classification step, I'm not sure how the RPN proposals would actually work as those are scanned over the feature maps that were generated by the CNN. </p>\n\n<blockquote>\n  <p><strong>xuan wrote</strong></p>\n  \n  <blockquote>\n    <p>Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... </p>\n  </blockquote>\n  \n  <ol>\n  <li>RPN for proposals</li>\n  <li>MaskHead for predicting masks for each proposal.</li>\n  </ol>\n  \n  <p>That's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.</p>\n</blockquote>",
      "rawMarkdown": "Unless I am missing something in my understanding as I am still new to Mask-RCNN, here is my understanding. \n\nTechnically we do still have 2 classes = Background + Nuclei. I think it helps for the CNN to classify the features that would be associated with the object we want versus everything else. From there those features will get captured by the anchor boxes in the RPN and refined to become the bounding box. \n\nSo without the CNN classification step, I'm not sure how the RPN proposals would actually work as those are scanned over the feature maps that were generated by the CNN. \n\n&gt; **xuan wrote**\n&gt; \n&gt; &gt; Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... \n&gt; \n&gt; 1. RPN for proposals\n&gt; 2. MaskHead for predicting masks for each proposal.\n&gt; \n&gt; That's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.",
      "votes": null
    },
    {
      "id": "303357",
      "postDate": "03/26/2018 04:06:19",
      "content": "<p>i am surprised that this does not work well.</p>\n\n<p>Visually, the bounding box i used for training the instance mask segmentation is similar to the output from faster rcnn.\nHowever, there is a difference and it causes the error shown below:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/303357/8879/separate3.png\" alt=\"enter image description here\"></p>\n\n<p>a small shift in the bounding box causes boundary shift in mask prediction. So i guess it is better to train end-to-end.\nThe results also suggest to me how i should modify my current mask-rcnn end to end code. </p>",
      "rawMarkdown": "i am surprised that this does not work well.\n\nVisually, the bounding box i used for training the instance mask segmentation is similar to the output from faster rcnn.\nHowever, there is a difference and it causes the error shown below:\n\n  ![enter image description here][1]\n\n\n\na small shift in the bounding box causes boundary shift in mask prediction. So i guess it is better to train end-to-end.\nThe results also suggest to me how i should modify my current mask-rcnn end to end code. \n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/303357/8879/separate3.png",
      "votes": null
    },
    {
      "id": "303361",
      "postDate": "03/26/2018 04:10:55",
      "content": "<p>I saw your result is very similar my result that obtained by keras implementation. For this result, I think the post-processing may be useful. Let try the dilation for correction the results and I guess it may provide better performance.</p>",
      "rawMarkdown": "I saw your result is very similar my result that obtained by keras implementation. For this result, I think the post-processing may be useful. Let try the dilation for correction the results and I guess it may provide better performance.",
      "votes": null
    },
    {
      "id": "303497",
      "postDate": "03/26/2018 10:00:35",
      "content": "<p>if you train end-to-end, then problems don't occurs. Of course, post-processing is another solution too.</p>",
      "rawMarkdown": "if you train end-to-end, then problems don't occurs. Of course, post-processing is another solution too.",
      "votes": null
    },
    {
      "id": "303531",
      "postDate": "03/26/2018 11:11:42",
      "content": "<p>I also train end to end but different with your code is that batch norm is frozen ( batch size is 1) and the bboxes created from masks with 1 pixel different. I also think your problem relates to resize masks in seg. phase</p>",
      "rawMarkdown": "I also train end to end but different with your code is that batch norm is frozen ( batch size is 1) and the bboxes created from masks with 1 pixel different. I also think your problem relates to resize masks in seg. phase",
      "votes": null
    },
    {
      "id": "303988",
      "postDate": "03/26/2018 21:18:25",
      "content": "<p>Yes, RPN can do the job. You can refer to face detection papers. I guess a second stage RCNN will refine the results from RPN</p>",
      "rawMarkdown": "Yes, RPN can do the job. You can refer to face detection papers. I guess a second stage RCNN will refine the results from RPN",
      "votes": null
    },
    {
      "id": "304962",
      "postDate": "03/28/2018 09:38:59",
      "content": "<p>Hello Heng, Could you tell me how to train the faster-rcnn only using your publish source code (contains faster-rcnn+mask)? I have not enough memory so I just want to train the faster-rcnn only to use larger batch size.</p>",
      "rawMarkdown": "Hello Heng, Could you tell me how to train the faster-rcnn only using your publish source code (contains faster-rcnn+mask)? I have not enough memory so I just want to train the faster-rcnn only to use larger batch size.",
      "votes": null
    },
    {
      "id": "309298",
      "postDate": "04/05/2018 01:50:12",
      "content": "<p>Hi Heng,</p>\n\n<p>Recently I have read your Mask RCNN code, which is very great and helpful!\nAnd after reading this post I remember a question when I was reading your code.\nWhy you used different box to delta encoding and decoding method for rnp and rcnn?\nFor rpn you used</p>\n\n<pre><code>def rpn_encode(window, truth_box):\n    cx = (window[:, 0] + window[:, 2]) / 2\n    cy = (window[:, 1] + window[:, 3]) / 2\n    w = (window[:, 2] - window[:, 0] + 1)\n    h = (window[:, 3] - window[:, 1] + 1)\n\n    target = (truth_box - np.column_stack([cx, cy, cx, cy])) / np.column_stack([w, h, w, h])\n    target = target * np.array([-1, -1, 1, 1], np.float32)\n    return target\n\n\ndef rpn_decode(window, delta):\n    cx = (window[:, 0] + window[:, 2]) / 2\n    cy = (window[:, 1] + window[:, 3]) / 2\n    w = (window[:, 2] - window[:, 0] + 1)\n    h = (window[:, 3] - window[:, 1] + 1)\n\n    delta = delta * np.array([-1, -1, 1, 1], np.float32)  # TODO if delta = (0.5, 0.5, 0.5, 0.5), then window = box\n    box = delta * np.column_stack([w, h, w, h]) + np.column_stack([cx, cy, cx, cy])\n\n    return box\n</code></pre>\n\n<p>while for rcnn you used:</p>\n\n<pre><code>def box_transform(windows, targets):\n    # targets : ground truth\n\n    bw = windows[:, 2] - windows[:, 0] + 1.0\n    bh = windows[:, 3] - windows[:, 1] + 1.0\n    bx = windows[:, 0] + 0.5 * bw\n    by = windows[:, 1] + 0.5 * bh\n\n    tw = targets[:, 2] - targets[:, 0] + 1.0\n    th = targets[:, 3] - targets[:, 1] + 1.0\n    tx = targets[:, 0] + 0.5 * tw\n    ty = targets[:, 1] + 0.5 * th\n\n    dx = (tx - bx) / bw\n    dy = (ty - by) / bh\n    dw = np.log(tw / bw)\n    dh = np.log(th / bh)\n\n    deltas = np.vstack((dx, dy, dw, dh)).transpose()\n    return deltas\n\n\ndef box_transform_inv(windows, deltas):\n    num = len(windows)\n    predictions = np.zeros((num, 4), dtype=np.float32)\n    # if num == 0: return predictions  #not possible?\n\n    bw = windows[:, 2] - windows[:, 0] + 1.0\n    bh = windows[:, 3] - windows[:, 1] + 1.0\n    bx = windows[:, 0] + 0.5 * bw\n    by = windows[:, 1] + 0.5 * bh\n    bw = bw[:, np.newaxis]\n    bh = bh[:, np.newaxis]\n    bx = bx[:, np.newaxis]\n    by = by[:, np.newaxis]\n\n    dx = deltas[:, 0::4]\n    dy = deltas[:, 1::4]\n    dw = deltas[:, 2::4]\n    dh = deltas[:, 3::4]\n\n    x = dx * bw + bx\n    y = dy * bh + by\n    w = np.exp(dw) * bw\n    h = np.exp(dh) * bh\n\n    predictions[:, 0::4] = x - 0.5 * w  # x0,y0,x1,y1\n    predictions[:, 1::4] = y - 0.5 * h  \n    predictions[:, 2::4] = x + 0.5 * w  # should there be a -1 ?\n    predictions[:, 3::4] = y + 0.5 * h  # should there be a -1 ?\n\n    return predictions\n</code></pre>\n\n<p>And also for the rcnn box_transform_inv() one, should there be a -1 in the final as in the comment?\nAfter modified, the rcnn can align a little better with the ground truth, (forgive me if I am wrong :))</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "Hi Heng,\n\nRecently I have read your Mask RCNN code, which is very great and helpful!\nAnd after reading this post I remember a question when I was reading your code.\nWhy you used different box to delta encoding and decoding method for rnp and rcnn?\nFor rpn you used\n\n    def rpn_encode(window, truth_box):\n        cx = (window[:, 0] + window[:, 2]) / 2\n        cy = (window[:, 1] + window[:, 3]) / 2\n        w = (window[:, 2] - window[:, 0] + 1)\n        h = (window[:, 3] - window[:, 1] + 1)\n    \n        target = (truth_box - np.column_stack([cx, cy, cx, cy])) / np.column_stack([w, h, w, h])\n        target = target * np.array([-1, -1, 1, 1], np.float32)\n        return target\n    \n    \n    def rpn_decode(window, delta):\n        cx = (window[:, 0] + window[:, 2]) / 2\n        cy = (window[:, 1] + window[:, 3]) / 2\n        w = (window[:, 2] - window[:, 0] + 1)\n        h = (window[:, 3] - window[:, 1] + 1)\n    \n        delta = delta * np.array([-1, -1, 1, 1], np.float32)  # TODO if delta = (0.5, 0.5, 0.5, 0.5), then window = box\n        box = delta * np.column_stack([w, h, w, h]) + np.column_stack([cx, cy, cx, cy])\n    \n        return box\n\nwhile for rcnn you used:\n\n    def box_transform(windows, targets):\n        # targets : ground truth\n    \n        bw = windows[:, 2] - windows[:, 0] + 1.0\n        bh = windows[:, 3] - windows[:, 1] + 1.0\n        bx = windows[:, 0] + 0.5 * bw\n        by = windows[:, 1] + 0.5 * bh\n    \n        tw = targets[:, 2] - targets[:, 0] + 1.0\n        th = targets[:, 3] - targets[:, 1] + 1.0\n        tx = targets[:, 0] + 0.5 * tw\n        ty = targets[:, 1] + 0.5 * th\n    \n        dx = (tx - bx) / bw\n        dy = (ty - by) / bh\n        dw = np.log(tw / bw)\n        dh = np.log(th / bh)\n    \n        deltas = np.vstack((dx, dy, dw, dh)).transpose()\n        return deltas\n\n\n    def box_transform_inv(windows, deltas):\n        num = len(windows)\n        predictions = np.zeros((num, 4), dtype=np.float32)\n        # if num == 0: return predictions  #not possible?\n    \n        bw = windows[:, 2] - windows[:, 0] + 1.0\n        bh = windows[:, 3] - windows[:, 1] + 1.0\n        bx = windows[:, 0] + 0.5 * bw\n        by = windows[:, 1] + 0.5 * bh\n        bw = bw[:, np.newaxis]\n        bh = bh[:, np.newaxis]\n        bx = bx[:, np.newaxis]\n        by = by[:, np.newaxis]\n    \n        dx = deltas[:, 0::4]\n        dy = deltas[:, 1::4]\n        dw = deltas[:, 2::4]\n        dh = deltas[:, 3::4]\n    \n        x = dx * bw + bx\n        y = dy * bh + by\n        w = np.exp(dw) * bw\n        h = np.exp(dh) * bh\n    \n        predictions[:, 0::4] = x - 0.5 * w  # x0,y0,x1,y1\n        predictions[:, 1::4] = y - 0.5 * h  \n        predictions[:, 2::4] = x + 0.5 * w  # should there be a -1 ?\n        predictions[:, 3::4] = y + 0.5 * h  # should there be a -1 ?\n    \n        return predictions\n\nAnd also for the rcnn box_transform_inv() one, should there be a -1 in the final as in the comment?\nAfter modified, the rcnn can align a little better with the ground truth, (forgive me if I am wrong :))\n\nThanks!",
      "votes": null
    },
    {
      "id": "309494",
      "postDate": "04/05/2018 13:11:20",
      "content": "<p>Do you have a repo for your solution?  </p>",
      "rawMarkdown": "Do you have a repo for your solution?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 300601,
      "author_name": "zhen27",
      "author_url": "",
      "post_date": "03/21/2018 18:47:16",
      "content": "<p>Congrats for your new LB. Hope this is a better idea.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 300842,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "03/22/2018 00:56:58",
      "content": "<p>As you said, we have no benefit for this way. But do you think fasterrcnn only can provide better detection result than maskrcnn? Because it has more simple loss  ( ignore mask loss) tends to easy to optimal. However, the main disadvantage of this way is that provides more than one cell in one box thus difficult to seperate them. And the spatial resolution of cell in box is very small ( 56x56, I guess) thus the deeper network may not good for segmentation</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 302148,
      "author_name": "atm584",
      "author_url": "",
      "post_date": "03/23/2018 18:29:13",
      "content": "<p>Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... </p>\n\n<ol>\n<li>RPN for proposals</li>\n<li>MaskHead for predicting masks for each proposal.</li>\n</ol>\n\n<p>That's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 302164,
          "author_name": "jamesrequa",
          "author_url": "",
          "post_date": "03/23/2018 18:56:07",
          "content": "<p>Unless I am missing something in my understanding as I am still new to Mask-RCNN, here is my understanding. </p>\n\n<p>Technically we do still have 2 classes = Background + Nuclei. I think it helps for the CNN to classify the features that would be associated with the object we want versus everything else. From there those features will get captured by the anchor boxes in the RPN and refined to become the bounding box. </p>\n\n<p>So without the CNN classification step, I'm not sure how the RPN proposals would actually work as those are scanned over the feature maps that were generated by the CNN. </p>\n\n<blockquote>\n  <p><strong>xuan wrote</strong></p>\n  \n  <blockquote>\n    <p>Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... </p>\n  </blockquote>\n  \n  <ol>\n  <li>RPN for proposals</li>\n  <li>MaskHead for predicting masks for each proposal.</li>\n  </ol>\n  \n  <p>That's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.</p>\n</blockquote>",
          "votes": null,
          "replies": []
        },
        {
          "id": 303988,
          "author_name": "wayne999",
          "author_url": "",
          "post_date": "03/26/2018 21:18:25",
          "content": "<p>Yes, RPN can do the job. You can refer to face detection papers. I guess a second stage RCNN will refine the results from RPN</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 303357,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "03/26/2018 04:06:19",
      "content": "<p>i am surprised that this does not work well.</p>\n\n<p>Visually, the bounding box i used for training the instance mask segmentation is similar to the output from faster rcnn.\nHowever, there is a difference and it causes the error shown below:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/303357/8879/separate3.png\" alt=\"enter image description here\"></p>\n\n<p>a small shift in the bounding box causes boundary shift in mask prediction. So i guess it is better to train end-to-end.\nThe results also suggest to me how i should modify my current mask-rcnn end to end code. </p>",
      "votes": null,
      "replies": [
        {
          "id": 303361,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "03/26/2018 04:10:55",
          "content": "<p>I saw your result is very similar my result that obtained by keras implementation. For this result, I think the post-processing may be useful. Let try the dilation for correction the results and I guess it may provide better performance.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 303497,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "03/26/2018 10:00:35",
          "content": "<p>if you train end-to-end, then problems don't occurs. Of course, post-processing is another solution too.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 303531,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "03/26/2018 11:11:42",
          "content": "<p>I also train end to end but different with your code is that batch norm is frozen ( batch size is 1) and the bboxes created from masks with 1 pixel different. I also think your problem relates to resize masks in seg. phase</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 309298,
          "author_name": "liyang2034",
          "author_url": "",
          "post_date": "04/05/2018 01:50:12",
          "content": "<p>Hi Heng,</p>\n\n<p>Recently I have read your Mask RCNN code, which is very great and helpful!\nAnd after reading this post I remember a question when I was reading your code.\nWhy you used different box to delta encoding and decoding method for rnp and rcnn?\nFor rpn you used</p>\n\n<pre><code>def rpn_encode(window, truth_box):\n    cx = (window[:, 0] + window[:, 2]) / 2\n    cy = (window[:, 1] + window[:, 3]) / 2\n    w = (window[:, 2] - window[:, 0] + 1)\n    h = (window[:, 3] - window[:, 1] + 1)\n\n    target = (truth_box - np.column_stack([cx, cy, cx, cy])) / np.column_stack([w, h, w, h])\n    target = target * np.array([-1, -1, 1, 1], np.float32)\n    return target\n\n\ndef rpn_decode(window, delta):\n    cx = (window[:, 0] + window[:, 2]) / 2\n    cy = (window[:, 1] + window[:, 3]) / 2\n    w = (window[:, 2] - window[:, 0] + 1)\n    h = (window[:, 3] - window[:, 1] + 1)\n\n    delta = delta * np.array([-1, -1, 1, 1], np.float32)  # TODO if delta = (0.5, 0.5, 0.5, 0.5), then window = box\n    box = delta * np.column_stack([w, h, w, h]) + np.column_stack([cx, cy, cx, cy])\n\n    return box\n</code></pre>\n\n<p>while for rcnn you used:</p>\n\n<pre><code>def box_transform(windows, targets):\n    # targets : ground truth\n\n    bw = windows[:, 2] - windows[:, 0] + 1.0\n    bh = windows[:, 3] - windows[:, 1] + 1.0\n    bx = windows[:, 0] + 0.5 * bw\n    by = windows[:, 1] + 0.5 * bh\n\n    tw = targets[:, 2] - targets[:, 0] + 1.0\n    th = targets[:, 3] - targets[:, 1] + 1.0\n    tx = targets[:, 0] + 0.5 * tw\n    ty = targets[:, 1] + 0.5 * th\n\n    dx = (tx - bx) / bw\n    dy = (ty - by) / bh\n    dw = np.log(tw / bw)\n    dh = np.log(th / bh)\n\n    deltas = np.vstack((dx, dy, dw, dh)).transpose()\n    return deltas\n\n\ndef box_transform_inv(windows, deltas):\n    num = len(windows)\n    predictions = np.zeros((num, 4), dtype=np.float32)\n    # if num == 0: return predictions  #not possible?\n\n    bw = windows[:, 2] - windows[:, 0] + 1.0\n    bh = windows[:, 3] - windows[:, 1] + 1.0\n    bx = windows[:, 0] + 0.5 * bw\n    by = windows[:, 1] + 0.5 * bh\n    bw = bw[:, np.newaxis]\n    bh = bh[:, np.newaxis]\n    bx = bx[:, np.newaxis]\n    by = by[:, np.newaxis]\n\n    dx = deltas[:, 0::4]\n    dy = deltas[:, 1::4]\n    dw = deltas[:, 2::4]\n    dh = deltas[:, 3::4]\n\n    x = dx * bw + bx\n    y = dy * bh + by\n    w = np.exp(dw) * bw\n    h = np.exp(dh) * bh\n\n    predictions[:, 0::4] = x - 0.5 * w  # x0,y0,x1,y1\n    predictions[:, 1::4] = y - 0.5 * h  \n    predictions[:, 2::4] = x + 0.5 * w  # should there be a -1 ?\n    predictions[:, 3::4] = y + 0.5 * h  # should there be a -1 ?\n\n    return predictions\n</code></pre>\n\n<p>And also for the rcnn box_transform_inv() one, should there be a -1 in the final as in the comment?\nAfter modified, the rcnn can align a little better with the ground truth, (forgive me if I am wrong :))</p>\n\n<p>Thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 304962,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "03/28/2018 09:38:59",
      "content": "<p>Hello Heng, Could you tell me how to train the faster-rcnn only using your publish source code (contains faster-rcnn+mask)? I have not enough memory so I just want to train the faster-rcnn only to use larger batch size.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 309494,
      "author_name": "hireme",
      "author_url": "",
      "post_date": "04/05/2018 13:11:20",
      "content": "<p>Do you have a repo for your solution?  </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "300392": "I decided to try a new approach, split a end-to-end mask rcnn into two-stage approach: detect nuclei first (faster-rcnn) and then crop image and predict mask (fcn).\n\nFrom application point of view, this is not so efficient. But from competition point of view, this seems to be easier to try new ideas, especially with regards to ensembling, mask refinement and pesudo-labeling and deeper network.\n\n(Note: using method of data or knowledge distillation, we still can transfer knowledge to end-to-end system later for real application)\n\nHere i show results \n\nstage-1: i.e. faster-rcnn alone:\n\n  ![enter image description here][1]\n\nstage-2: i.e. crop image + mask prediction\n\n... to be updated ...\n\nthere is no submission to LB yet\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/300392/8836/432f367a4c5b5674de2e2977744d10289a064e5704b21af6607b4975be47c580.rcnn_proposal.png",
    "300601": "Congrats for your new LB. Hope this is a better idea.",
    "300842": "As you said, we have no benefit for this way. But do you think fasterrcnn only can provide better detection result than maskrcnn? Because it has more simple loss  ( ignore mask loss) tends to easy to optimal. However, the main disadvantage of this way is that provides more than one cell in one box thus difficult to seperate them. And the spatial resolution of cell in box is very small ( 56x56, I guess) thus the deeper network may not good for segmentation",
    "302148": "Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... \n\n1. RPN for proposals\n2. MaskHead for predicting masks for each proposal.\n\nThat's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.",
    "302164": "Unless I am missing something in my understanding as I am still new to Mask-RCNN, here is my understanding. \n\nTechnically we do still have 2 classes = Background + Nuclei. I think it helps for the CNN to classify the features that would be associated with the object we want versus everything else. From there those features will get captured by the anchor boxes in the RPN and refined to become the bounding box. \n\nSo without the CNN classification step, I'm not sure how the RPN proposals would actually work as those are scanned over the feature maps that were generated by the CNN. \n\n&gt; **xuan wrote**\n&gt; \n&gt; &gt; Since there's only one class we could predict (just nuclei, no different types of them without extra labelling). Why couldn't we just remove the rcnn-head? Actually I'm doing this right now... \n&gt; \n&gt; 1. RPN for proposals\n&gt; 2. MaskHead for predicting masks for each proposal.\n&gt; \n&gt; That's a bit similar to your idea here. Why do we need rcnn-head? I have no idea...would appreciate to hear the story behind it.",
    "303357": "i am surprised that this does not work well.\n\nVisually, the bounding box i used for training the instance mask segmentation is similar to the output from faster rcnn.\nHowever, there is a difference and it causes the error shown below:\n\n  ![enter image description here][1]\n\n\n\na small shift in the bounding box causes boundary shift in mask prediction. So i guess it is better to train end-to-end.\nThe results also suggest to me how i should modify my current mask-rcnn end to end code. \n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/303357/8879/separate3.png",
    "303361": "I saw your result is very similar my result that obtained by keras implementation. For this result, I think the post-processing may be useful. Let try the dilation for correction the results and I guess it may provide better performance.",
    "303497": "if you train end-to-end, then problems don't occurs. Of course, post-processing is another solution too.",
    "303531": "I also train end to end but different with your code is that batch norm is frozen ( batch size is 1) and the bboxes created from masks with 1 pixel different. I also think your problem relates to resize masks in seg. phase",
    "303988": "Yes, RPN can do the job. You can refer to face detection papers. I guess a second stage RCNN will refine the results from RPN",
    "304962": "Hello Heng, Could you tell me how to train the faster-rcnn only using your publish source code (contains faster-rcnn+mask)? I have not enough memory so I just want to train the faster-rcnn only to use larger batch size.",
    "309298": "Hi Heng,\n\nRecently I have read your Mask RCNN code, which is very great and helpful!\nAnd after reading this post I remember a question when I was reading your code.\nWhy you used different box to delta encoding and decoding method for rnp and rcnn?\nFor rpn you used\n\n    def rpn_encode(window, truth_box):\n        cx = (window[:, 0] + window[:, 2]) / 2\n        cy = (window[:, 1] + window[:, 3]) / 2\n        w = (window[:, 2] - window[:, 0] + 1)\n        h = (window[:, 3] - window[:, 1] + 1)\n    \n        target = (truth_box - np.column_stack([cx, cy, cx, cy])) / np.column_stack([w, h, w, h])\n        target = target * np.array([-1, -1, 1, 1], np.float32)\n        return target\n    \n    \n    def rpn_decode(window, delta):\n        cx = (window[:, 0] + window[:, 2]) / 2\n        cy = (window[:, 1] + window[:, 3]) / 2\n        w = (window[:, 2] - window[:, 0] + 1)\n        h = (window[:, 3] - window[:, 1] + 1)\n    \n        delta = delta * np.array([-1, -1, 1, 1], np.float32)  # TODO if delta = (0.5, 0.5, 0.5, 0.5), then window = box\n        box = delta * np.column_stack([w, h, w, h]) + np.column_stack([cx, cy, cx, cy])\n    \n        return box\n\nwhile for rcnn you used:\n\n    def box_transform(windows, targets):\n        # targets : ground truth\n    \n        bw = windows[:, 2] - windows[:, 0] + 1.0\n        bh = windows[:, 3] - windows[:, 1] + 1.0\n        bx = windows[:, 0] + 0.5 * bw\n        by = windows[:, 1] + 0.5 * bh\n    \n        tw = targets[:, 2] - targets[:, 0] + 1.0\n        th = targets[:, 3] - targets[:, 1] + 1.0\n        tx = targets[:, 0] + 0.5 * tw\n        ty = targets[:, 1] + 0.5 * th\n    \n        dx = (tx - bx) / bw\n        dy = (ty - by) / bh\n        dw = np.log(tw / bw)\n        dh = np.log(th / bh)\n    \n        deltas = np.vstack((dx, dy, dw, dh)).transpose()\n        return deltas\n\n\n    def box_transform_inv(windows, deltas):\n        num = len(windows)\n        predictions = np.zeros((num, 4), dtype=np.float32)\n        # if num == 0: return predictions  #not possible?\n    \n        bw = windows[:, 2] - windows[:, 0] + 1.0\n        bh = windows[:, 3] - windows[:, 1] + 1.0\n        bx = windows[:, 0] + 0.5 * bw\n        by = windows[:, 1] + 0.5 * bh\n        bw = bw[:, np.newaxis]\n        bh = bh[:, np.newaxis]\n        bx = bx[:, np.newaxis]\n        by = by[:, np.newaxis]\n    \n        dx = deltas[:, 0::4]\n        dy = deltas[:, 1::4]\n        dw = deltas[:, 2::4]\n        dh = deltas[:, 3::4]\n    \n        x = dx * bw + bx\n        y = dy * bh + by\n        w = np.exp(dw) * bw\n        h = np.exp(dh) * bh\n    \n        predictions[:, 0::4] = x - 0.5 * w  # x0,y0,x1,y1\n        predictions[:, 1::4] = y - 0.5 * h  \n        predictions[:, 2::4] = x + 0.5 * w  # should there be a -1 ?\n        predictions[:, 3::4] = y + 0.5 * h  # should there be a -1 ?\n    \n        return predictions\n\nAnd also for the rcnn box_transform_inv() one, should there be a -1 in the final as in the comment?\nAfter modified, the rcnn can align a little better with the ground truth, (forgive me if I am wrong :))\n\nThanks!",
    "309494": "Do you have a repo for your solution?"
  },
  "source": "meta"
}