{
  "id": 53720,
  "title": "my solution : moving from 0.500 onwards",
  "url": "/competitions/data-science-bowl-2018/discussion/53720",
  "author_name": "hengck23",
  "post_date": "2018-04-04T06:24:20.047000",
  "votes": 61,
  "comment_count": 94,
  "views": 0,
  "content": "<p>After getting LB score of 0.48 to 0.50, here are 2 things i did. Strategy is simple. Collect enough errors, correct them and relearn.</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308811/8929/Slide1.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308811/8930/Slide2.png\" alt=\"enter image description here\"></p>",
  "messages": [
    {
      "id": 308811,
      "postDate": "2018-04-04T06:24:20.047Z",
      "content": "<p>After getting LB score of 0.48 to 0.50, here are 2 things i did. Strategy is simple. Collect enough errors, correct them and relearn.</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308811/8929/Slide1.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308811/8930/Slide2.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "After getting LB score of 0.48 to 0.50, here are 2 things i did. Strategy is simple. Collect enough errors, correct them and relearn.\n\n  ![enter image description here][1]\n  \n  ![enter image description here][2]\n  \n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308811/8929/Slide1.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308811/8930/Slide2.png",
      "votes": 61
    },
    {
      "id": 309550,
      "postDate": "2018-04-05T14:59:21.150Z",
      "content": "<p>ensmeble code</p>",
      "rawMarkdown": "ensmeble code",
      "votes": 10,
      "replies": [
        {
          "id": 309676,
          "postDate": "2018-04-05T19:31:30.567Z",
          "content": "<p>Thank you Heng for sharing the ensemble code. Could you also share the code for the function image_to_mask in \"mask       = image_to_mask(mask_image)\"?  Thanks much.</p>",
          "rawMarkdown": "Thank you Heng for sharing the ensemble code. Could you also share the code for the function image_to_mask in \"mask       = image_to_mask(mask_image)\"?  Thanks much."
        },
        {
          "id": 309816,
          "postDate": "2018-04-06T02:45:42.610Z",
          "content": "<p>Very useful. Could you also provide script to test with scale image likes scale_1.2, scale_0.8. I can scale image to 1.2 and result in x1.2 times with original image. However, I do not know how scale back to input image size. I think it is in the <code>def revert(net, images):</code> function</p>",
          "rawMarkdown": "Very useful. Could you also provide script to test with scale image likes scale_1.2, scale_0.8. I can scale image to 1.2 and result in x1.2 times with original image. However, I do not know how scale back to input image size. I think it is in the `def revert(net, images):` function\n"
        },
        {
          "id": 309825,
          "postDate": "2018-04-06T03:02:21.827Z",
          "content": "<p>You have the raw mask probability at fixed size of e.g 28x28. Resize these to the original image detection box size</p>",
          "rawMarkdown": "You have the raw mask probability at fixed size of e.g 28x28. Resize these to the original image detection box size",
          "votes": 1
        },
        {
          "id": 309952,
          "postDate": "2018-04-06T09:13:05.533Z",
          "content": "<p>mask       = image_to_mask(mask_image)\nNameError: name 'image_to_mask' is not defined</p>\n\n<p>How to define this function?</p>",
          "rawMarkdown": "mask       = image_to_mask(mask_image)\nNameError: name 'image_to_mask' is not defined\n\nHow to define this function?"
        },
        {
          "id": 309959,
          "postDate": "2018-04-06T09:30:45.827Z",
          "content": "<p>You do not need to use it. Just using the function to load from npy file. See submit.py for that or uncomment two above lines in his code. FYI, my score drop from 0.455 to 0.45 using fliplr+flipud+original</p>",
          "rawMarkdown": "You do not need to use it. Just using the function to load from npy file. See submit.py for that or uncomment two above lines in his code. FYI, my score drop from 0.455 to 0.45 using fliplr+flipud+original",
          "votes": 1
        },
        {
          "id": 310287,
          "postDate": "2018-04-07T02:33:19.360Z",
          "content": "<p>Youvare using intersection, union or average in ensemble? Hiw about comparing the visual results between the orginal and ensemble submission. What \nis the difference? You can use the draw mask function, just treat one of them as ground truth.</p>",
          "rawMarkdown": "Youvare using intersection, union or average in ensemble? Hiw about comparing the visual results between the orginal and ensemble submission. What \nis the difference? You can use the draw mask function, just treat one of them as ground truth.",
          "votes": 2
        },
        {
          "id": 310288,
          "postDate": "2018-04-07T02:40:00.600Z",
          "content": "<p>@Heng: i am using original ensemble code (average) but different is that I only have three input: original, flipud and fliplr. I do not have scale 0.8 and 1.2 outputs. I spend 1 day to implement it based on your suggestion but it did not work. Could you please share the code to do scale prediction in the file submit.py?</p>",
          "rawMarkdown": "@Heng: i am using original ensemble code (average) but different is that I only have three input: original, flipud and fliplr. I do not have scale 0.8 and 1.2 outputs. I spend 1 day to implement it based on your suggestion but it did not work. Could you please share the code to do scale prediction in the file submit.py?"
        },
        {
          "id": 310293,
          "postDate": "2018-04-07T02:50:35.720Z",
          "content": "<p>Thanks, I know how to ensemble now.</p>",
          "rawMarkdown": "Thanks, I know how to ensemble now."
        },
        {
          "id": 310428,
          "postDate": "2018-04-07T13:42:55.797Z",
          "content": "<p>i will put my code later.</p>\n\n<p>you can also try this\n<a href=\"https://www.kaggle.com/c/data-science-bowl-2018/discussion/50795\">https://www.kaggle.com/c/data-science-bowl-2018/discussion/50795</a>, search for comments\n\"FYI: how to ensemble objector detector from resnet paper:\"</p>\n\n<p>say you have image x0, test-augments x1,x2,x3 ....</p>\n\n<p>let d0,d1 ,d2 .... be detection results of x0,x1,x2 respectively,</p>\n\n<p>compute: </p>\n\n<p>m0 = mask_rcnn_crop_and_mask_head_only( x0, (d0,d1 ,d2, ....))</p>\n\n<p>m1 = mask_rcnn_crop_and_mask_head_only( x1, (d0,d1 ,d2, ....))</p>\n\n<p>...</p>\n\n<p>then ensemble m0,m1,m2 ....</p>\n\n<p>We collect all detection boxes from all test augments. hence now we have more detection per augment image. Maybe mask prediction is more accurate in some augment image, so we average the results for each box</p>",
          "rawMarkdown": "i will put my code later.\n\nyou can also try this\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/50795, search for comments\n\"FYI: how to ensemble objector detector from resnet paper:\"\n\nsay you have image x0, test-augments x1,x2,x3 ....\n\nlet d0,d1 ,d2 .... be detection results of x0,x1,x2 respectively,\n\ncompute: \n\nm0 = mask_rcnn_crop_and_mask_head_only( x0, (d0,d1 ,d2, ....))\n\nm1 = mask_rcnn_crop_and_mask_head_only( x1, (d0,d1 ,d2, ....))\n\n...\n\nthen ensemble m0,m1,m2 ....\n\nWe collect all detection boxes from all test augments. hence now we have more detection per augment image. Maybe mask prediction is more accurate in some augment image, so we average the results for each box\n\n",
          "votes": 4
        }
      ]
    },
    {
      "id": 310906,
      "postDate": "2018-04-09T02:25:57.833Z",
      "content": "<p>compare ensemble and single  model results:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310906/9029/single.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310906/9030/ensemble.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "compare ensemble and single  model results:\n\n\n   ![enter image description here][1]\n\n   ![enter image description here][2]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310906/9029/single.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310906/9030/ensemble.png",
      "votes": 5,
      "replies": [
        {
          "id": 310919,
          "postDate": "2018-04-09T03:11:21.377Z",
          "content": "<p>The last (4th) image of each row is the ensembled results.</p>\n\n<p>Looking at it, you can actually train another model on the ensembled results. (i.e. 2 layer stacked framework)</p>",
          "rawMarkdown": "The last (4th) image of each row is the ensembled results.\n\nLooking at it, you can actually train another model on the ensembled results. (i.e. 2 layer stacked framework)",
          "votes": 2
        },
        {
          "id": 311106,
          "postDate": "2018-04-09T11:57:08.840Z",
          "content": "<p>Hi Heng, I am using GTX 1080 Ti. I have compiled all the '.so' modules using 'sm_61' for 1080 Ti but I am still getting OOM errors. Can you please include a README (to compile any other modules for 1080 Ti)?</p>",
          "rawMarkdown": "Hi Heng, I am using GTX 1080 Ti. I have compiled all the '.so' modules using 'sm_61' for 1080 Ti but I am still getting OOM errors. Can you please include a README (to compile any other modules for 1080 Ti)?"
        },
        {
          "id": 311110,
          "postDate": "2018-04-09T12:14:20.927Z",
          "content": "<p>Sorry. Pls igore. Found the answer below.</p>",
          "rawMarkdown": "Sorry. Pls igore. Found the answer below."
        },
        {
          "id": 311154,
          "postDate": "2018-04-09T13:57:16.417Z",
          "content": "<p>@Watts </p>\n\n<p>How did you get Heng's code to work on a sm_61? Any assistance would be greatly appreciated. </p>",
          "rawMarkdown": "@Watts \n\nHow did you get Heng's code to work on a sm_61? Any assistance would be greatly appreciated. "
        },
        {
          "id": 311756,
          "postDate": "2018-04-10T16:15:31.230Z",
          "content": "<p>@ WIlliam Green</p>\n\n<p>Please use the README file from google drive distribution dtd 20180306 (see 0.463 thread). Replace sm_52 by sm_61 in all the commands that are used to build '<em>.so' files (including setup.py, build.py etc files as required). Then copy '</em>.so' files in the right places. That's about it. After that it runs pretty much as is.</p>",
          "rawMarkdown": "@ WIlliam Green\n\nPlease use the README file from google drive distribution dtd 20180306 (see 0.463 thread). Replace sm_52 by sm_61 in all the commands that are used to build '*.so' files (including setup.py, build.py etc files as required). Then copy '*.so' files in the right places. That's about it. After that it runs pretty much as is."
        }
      ]
    },
    {
      "id": 310484,
      "postDate": "2018-04-07T17:16:16.910Z",
      "content": "<p>A much better way to ensemble:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8957/Slide1.png\" alt=\"enter image description here\">\n   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8958/Slide2.png\" alt=\"enter image description here\">\n   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8959/Slide3.png\" alt=\"enter image description here\">\n   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8960/Slide4.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "A much better way to ensemble:\n\n   ![enter image description here][1]\n   ![enter image description here][2]\n   ![enter image description here][3]\n   ![enter image description here][4]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8957/Slide1.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8958/Slide2.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8959/Slide3.png\n  [4]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8960/Slide4.png",
      "votes": 6,
      "replies": [
        {
          "id": 315236,
          "postDate": "2018-04-16T23:10:10.563Z",
          "content": "<p>Heng CherKeng,  Thank you so much for the code. It was a great learning experience. In box.py, i made some changes  in the box_transform function. I noticed that in most of the cases\n    dx = (tx - bx) / bw\n    dy = (ty - by) / bh\n    dw = np.log(tw / bw)\n    dh = np.log(th / bh)</p>\n\n<p>dx, dy, dw, dh would be zero. This would then affect/reduce  the number of rcnn targets. </p>\n\n<p>i added a loop </p>\n\n<pre><code>if (dx.all() == 0 or dy.all() == 0):\n    dx = tx\n    dy = ty\n    dw = tw\n    dh = th\n</code></pre>\n\n<p>I played around with dx = ( tx/1024) dy = ( ty/1024);  dx = ( tx/512) etc. This greatly increased the number of rcnn targets. </p>",
          "rawMarkdown": "Heng CherKeng,  Thank you so much for the code. It was a great learning experience. In box.py, i made some changes  in the box_transform function. I noticed that in most of the cases\n    dx = (tx - bx) / bw\n    dy = (ty - by) / bh\n    dw = np.log(tw / bw)\n    dh = np.log(th / bh)\n\ndx, dy, dw, dh would be zero. This would then affect/reduce  the number of rcnn targets. \n\ni added a loop \n\n    if (dx.all() == 0 or dy.all() == 0):\n        dx = tx\n        dy = ty\n        dw = tw\n        dh = th\n\nI played around with dx = ( tx/1024) dy = ( ty/1024);  dx = ( tx/512) etc. This greatly increased the number of rcnn targets. \n\n"
        },
        {
          "id": 315322,
          "postDate": "2018-04-17T01:14:51.050Z",
          "content": "<p>@avsanjay</p>\n\n<p>thanks!</p>",
          "rawMarkdown": "@avsanjay\n\nthanks!"
        }
      ]
    },
    {
      "id": 310430,
      "postDate": "2018-04-07T14:04:13.453Z",
      "content": "<p>as requested, test time augmentation code</p>",
      "rawMarkdown": "as requested, test time augmentation code",
      "votes": 6
    },
    {
      "id": 311615,
      "postDate": "2018-04-10T11:24:45.300Z",
      "content": "<p>btw, do we upload code by 11-Apr 23:59 hrs or 11-Apr 00:00 hrs. Now i am busy cleaning code!</p>\n\n<p>\"All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.\"</p>\n\n<p>So i think it is  \"11-Apr 23:59 hrs\". </p>",
      "rawMarkdown": "btw, do we upload code by 11-Apr 23:59 hrs or 11-Apr 00:00 hrs. Now i am busy cleaning code!\n\n\"All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.\"\n\nSo i think it is  \"11-Apr 23:59 hrs\". ",
      "votes": 3,
      "replies": [
        {
          "id": 311892,
          "postDate": "2018-04-10T21:11:17.667Z",
          "content": "<p>If you are in the UTC time zone, yes. <a href=\"https://www.wolframalpha.com/input/?i=april+11+2018+23:59+utc\">Conversion to detected local time zone</a></p>",
          "rawMarkdown": "If you are in the UTC time zone, yes. [Conversion to detected local time zone](https://www.wolframalpha.com/input/?i=april+11+2018+23:59+utc)",
          "votes": 1
        }
      ]
    },
    {
      "id": 311096,
      "postDate": "2018-04-09T11:23:14.237Z",
      "content": "<p>this movie gives you an idea how to post process:</p>\n\n<p>from left to right: </p>\n\n<ul>\n<li>box of average mask, </li>\n<li>average mask, </li>\n<li>each mask in ensemble</li>\n</ul>",
      "rawMarkdown": "this movie gives you an idea how to post process:\n\nfrom left to right: \n\n-  box of average mask, \n-  average mask, \n-  each mask in ensemble",
      "votes": 3
    },
    {
      "id": 311015,
      "postDate": "2018-04-09T07:44:51.533Z",
      "content": "<p>i put more ensemble results are the google drive:</p>\n\n<p><a href=\"https://drive.google.com/drive/folders/1Z9-eS5yr4jDPlCyyWqiV_3-m64lAPqRG\">https://drive.google.com/drive/folders/1Z9-eS5yr4jDPlCyyWqiV_3-m64lAPqRG</a></p>\n\n<p>I also put the trained model (00049000_model.pth) there.</p>\n\n<p>The results are visually very attractive.\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311015/9036/a984e7fb886aa02e29d112766d3ce26a4f78eac540ce7bbdbd42af2761928f6d.png\" alt=\"enter image description here\">\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311015/9034/43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af.png\" alt=\"enter image description here\">\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311015/9037/bdc789019cee8ddfae20d5f769299993b4b330b2d38d1218646cf89e77fbbd4d.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "i put more ensemble results are the google drive:\n\nhttps://drive.google.com/drive/folders/1Z9-eS5yr4jDPlCyyWqiV_3-m64lAPqRG\n\nI also put the trained model (00049000_model.pth) there.\n\nThe results are visually very attractive.\n  ![enter image description here][1]\n  ![enter image description here][2]\n  ![enter image description here][3]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311015/9036/a984e7fb886aa02e29d112766d3ce26a4f78eac540ce7bbdbd42af2761928f6d.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311015/9034/43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311015/9037/bdc789019cee8ddfae20d5f769299993b4b330b2d38d1218646cf89e77fbbd4d.png",
      "votes": 3
    },
    {
      "id": 310917,
      "postDate": "2018-04-09T03:07:41.090Z",
      "content": "<p>You do not need perfect labels to learn. But you do need many labels per sample if they are ambiguous. For this reason, this is why pseudo-labels from ensemble classifiers should work. This is distilling knowledge from ensemble.\nThe problem is to control the percentage of label noise.  </p>\n\n<p>This is a neat trick that you can use in many kaggle competition. You now have a way to \"fit test data\"</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310917/9032/self-learning.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "You do not need perfect labels to learn. But you do need many labels per sample if they are ambiguous. For this reason, this is why pseudo-labels from ensemble classifiers should work. This is distilling knowledge from ensemble.\nThe problem is to control the percentage of label noise.  \n\nThis is a neat trick that you can use in many kaggle competition. You now have a way to \"fit test data\"\n\n   ![enter image description here][1]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310917/9032/self-learning.png",
      "votes": 4,
      "replies": [
        {
          "id": 312464,
          "postDate": "2018-04-11T19:44:48.663Z",
          "content": "<p>Have you tried implementing \"bootstrapping\" on noisy labels: </p>\n\n<p><a href=\"https://arxiv.org/pdf/1412.6596.pdf\">https://arxiv.org/pdf/1412.6596.pdf</a></p>\n\n<p>They only test the strategy on object detection with Multibox, but I think the loss function in 3.2 can be used in place of rpn class loss, rcnn class loss, and maybe the mask loss itself. When I  tried it, it did detect more nuclei that had been missed otherwise, but the masks were not very good. A more accurate model trained on clean labels might be able to use it effectively though.</p>",
          "rawMarkdown": "Have you tried implementing \"bootstrapping\" on noisy labels: \n\n[https://arxiv.org/pdf/1412.6596.pdf][1]\n\nThey only test the strategy on object detection with Multibox, but I think the loss function in 3.2 can be used in place of rpn class loss, rcnn class loss, and maybe the mask loss itself. When I  tried it, it did detect more nuclei that had been missed otherwise, but the masks were not very good. A more accurate model trained on clean labels might be able to use it effectively though.\n\n\n  [1]: https://arxiv.org/pdf/1412.6596.pdf"
        }
      ]
    },
    {
      "id": 308815,
      "postDate": "2018-04-04T06:38:03.253Z",
      "content": "<p>some question you may ask:</p>\n\n<ol>\n<li><p>what if given an external image, there are some nuclei that i know how to label and some that i don't know?</p>\n\n<ul><li>draw a box to cover the nuclei that you don't know how to label and filled with black (or back ground color).</li></ul></li>\n<li><p>Does external data helps?</p>\n\n<ul><li>Yes, only if you annotate them correctly</li></ul></li>\n</ol>",
      "rawMarkdown": "some question you may ask:\n\n1. what if given an external image, there are some nuclei that i know how to label and some that i don't know?\n\n- draw a box to cover the nuclei that you don't know how to label and filled with black (or back ground color).\n\n\n2. Does external data helps?\n\n- Yes, only if you annotate them correctly\n\n",
      "votes": 4,
      "replies": [
        {
          "id": 309014,
          "postDate": "2018-04-04T13:44:49.180Z",
          "content": "<p>Hi @Heng CherKeng thanks for sharing your thoughts and the code. I just need to ask a question since I'm not able to make sense out of the training phase. I've run your code and multiple times and for various iterations up to 30k and there's nowhere near that it reaches the 0.419 LB that you had in your <code>20180306</code> solution results. Any ideas on why is that since it is the same solution unless something has changed? Beyond that I see that the results are fluctuating up and down but in the same range from iteration to iteration? Here's an example:</p>\n\n<p><code>\n4418 0.0010  10.3 k  144.2  0.1 m | 0.559   0.02 0.02   0.35 0.01   0.16 | 0.660   0.03 0.03   0.38 0.02   0.20 | 0.515   0.02 0.03   0.26 0.01   0.19 |  7 hr      29 min\n</code></p>\n\n<p><code>\n4419 0.0010  10.4 k  145.6  0.1 m | 0.492   0.02 0.02   0.28 0.01   0.16 | 0.648   0.03 0.03   0.38 0.02   0.19 | 0.691   0.02 0.03   0.41 0.02   0.21 |  7 hr      37 min\n</code></p>\n\n<p><code>\n4420 0.0010  10.5 k  147.0  0.1 m | 0.529   0.02 0.02   0.32 0.01   0.16 | 0.605   0.03 0.03   0.34 0.02   0.19 | 0.639   0.03 0.03   0.36 0.01   0.21 |  7 hr      44 min\n</code></p>\n\n<p><code>\n4421 0.0010  10.6 k  148.4  0.1 m | 0.497   0.02 0.02   0.29 0.01   0.16 | 0.607   0.03 0.03   0.34 0.02   0.18 | 0.484   0.04 0.03   0.26 0.01   0.14 |  7 hr      51 min\n</code></p>\n\n<p><code>\n4422 0.0010  10.7 k  149.8  0.1 m | 0.482   0.02 0.02   0.27 0.01   0.16 | 0.597   0.03 0.03   0.35 0.01   0.18 | 0.829   0.05 0.03   0.55 0.01   0.20 |  7 hr      59 min\n</code></p>\n\n<p><code>\n4423 0.0010  10.8 k  151.2  0.1 m | 0.462   0.02 0.02   0.26 0.01   0.15 | 0.611   0.03 0.03   0.33 0.02   0.20 | 0.657   0.03 0.03   0.40 0.02   0.18 |  8 hr      06 min\n</code></p>\n\n<p><code>\n4424 0.0010  10.9 k  152.6  0.1 m | 0.480   0.02 0.02   0.27 0.01   0.16 | 0.575   0.02 0.03   0.31 0.02   0.19 | 0.508   0.03 0.03   0.26 0.01   0.18 |  8 hr      13 min\n</code></p>\n\n<p><code>\n4425 0.0010  11.0 k  154.0  0.1 m | 0.527   0.02 0.02   0.32 0.01   0.15 | 0.631   0.04 0.03   0.35 0.02   0.19 | 0.455   0.04 0.03   0.24 0.01   0.14 |  8 hr      21 min\n</code></p>\n\n<p><code>\n4426 0.0010  11.1 k  155.4  0.1 m | 0.461   0.02 0.02   0.26 0.01   0.15 | 0.549   0.03 0.03   0.30 0.01   0.18 | 0.702   0.01 0.05   0.42 0.02   0.20 |  8 hr      28 min\n</code></p>\n\n<p><code>\n4427 0.0010  11.2 k  156.8  0.1 m | 0.533   0.02 0.02   0.32 0.01   0.16 | 0.592   0.02 0.03   0.34 0.02   0.18 | 0.517   0.03 0.03   0.28 0.02   0.16 |  8 hr      32 min\n</code></p>\n\n<p><code>\n4428 0.0010  11.3 k  158.2  0.1 m | 0.508   0.02 0.02   0.30 0.01   0.16 | 0.636   0.03 0.03   0.37 0.02   0.19 | 0.534   0.03 0.04   0.29 0.01   0.16 |  8 hr      37 min\n</code> </p>\n\n<p><code>\n4429 0.0010  11.4 k  159.6  0.1 m | 0.581   0.02 0.02   0.38 0.01   0.15 | 0.661   0.03 0.03   0.37 0.02   0.20 | 0.587   0.03 0.03   0.34 0.02   0.17 |  8 hr      42 min\n</code></p>\n\n<p><code>\n4430 0.0010  11.5 k  161.0  0.1 m | 0.445   0.02 0.02   0.24 0.01   0.16 | 0.618   0.03 0.03   0.35 0.02   0.20 | 0.367   0.02 0.02   0.15 0.01   0.17 |  8 hr      46 min\n</code></p>\n\n<p><code>\n4431 0.0010  11.6 k  162.4  0.1 m | 0.518   0.02 0.02   0.32 0.01   0.15 | 0.561   0.02 0.03   0.32 0.01   0.18 | 0.745   0.05 0.04   0.45 0.01   0.19 |  8 hr      50 min\n</code></p>\n\n<p><code>\n4432 0.0010  11.7 k  163.8  0.1 m | 0.541   0.02 0.02   0.34 0.01   0.15 | 0.602   0.03 0.03   0.34 0.02   0.19 | 0.729   0.03 0.04   0.40 0.02   0.24 |  8 hr\n</code></p>",
          "rawMarkdown": "Hi @Heng CherKeng thanks for sharing your thoughts and the code. I just need to ask a question since I'm not able to make sense out of the training phase. I've run your code and multiple times and for various iterations up to 30k and there's nowhere near that it reaches the 0.419 LB that you had in your `20180306` solution results. Any ideas on why is that since it is the same solution unless something has changed? Beyond that I see that the results are fluctuating up and down but in the same range from iteration to iteration? Here's an example:\n\n```\n4418 0.0010  10.3 k  144.2  0.1 m | 0.559   0.02 0.02   0.35 0.01   0.16 | 0.660   0.03 0.03   0.38 0.02   0.20 | 0.515   0.02 0.03   0.26 0.01   0.19 |  7 hr      29 min\n```\n\n```\n4419 0.0010  10.4 k  145.6  0.1 m | 0.492   0.02 0.02   0.28 0.01   0.16 | 0.648   0.03 0.03   0.38 0.02   0.19 | 0.691   0.02 0.03   0.41 0.02   0.21 |  7 hr      37 min\n```\n\n```\n4420 0.0010  10.5 k  147.0  0.1 m | 0.529   0.02 0.02   0.32 0.01   0.16 | 0.605   0.03 0.03   0.34 0.02   0.19 | 0.639   0.03 0.03   0.36 0.01   0.21 |  7 hr      44 min\n```\n\n```\n4421 0.0010  10.6 k  148.4  0.1 m | 0.497   0.02 0.02   0.29 0.01   0.16 | 0.607   0.03 0.03   0.34 0.02   0.18 | 0.484   0.04 0.03   0.26 0.01   0.14 |  7 hr      51 min\n```\n\n```\n4422 0.0010  10.7 k  149.8  0.1 m | 0.482   0.02 0.02   0.27 0.01   0.16 | 0.597   0.03 0.03   0.35 0.01   0.18 | 0.829   0.05 0.03   0.55 0.01   0.20 |  7 hr      59 min\n```\n\n```\n4423 0.0010  10.8 k  151.2  0.1 m | 0.462   0.02 0.02   0.26 0.01   0.15 | 0.611   0.03 0.03   0.33 0.02   0.20 | 0.657   0.03 0.03   0.40 0.02   0.18 |  8 hr      06 min\n```\n\n```\n4424 0.0010  10.9 k  152.6  0.1 m | 0.480   0.02 0.02   0.27 0.01   0.16 | 0.575   0.02 0.03   0.31 0.02   0.19 | 0.508   0.03 0.03   0.26 0.01   0.18 |  8 hr      13 min\n```\n\n```\n4425 0.0010  11.0 k  154.0  0.1 m | 0.527   0.02 0.02   0.32 0.01   0.15 | 0.631   0.04 0.03   0.35 0.02   0.19 | 0.455   0.04 0.03   0.24 0.01   0.14 |  8 hr      21 min\n```\n\n```\n4426 0.0010  11.1 k  155.4  0.1 m | 0.461   0.02 0.02   0.26 0.01   0.15 | 0.549   0.03 0.03   0.30 0.01   0.18 | 0.702   0.01 0.05   0.42 0.02   0.20 |  8 hr      28 min\n```\n\n```\n4427 0.0010  11.2 k  156.8  0.1 m | 0.533   0.02 0.02   0.32 0.01   0.16 | 0.592   0.02 0.03   0.34 0.02   0.18 | 0.517   0.03 0.03   0.28 0.02   0.16 |  8 hr      32 min\n```\n\n```\n4428 0.0010  11.3 k  158.2  0.1 m | 0.508   0.02 0.02   0.30 0.01   0.16 | 0.636   0.03 0.03   0.37 0.02   0.19 | 0.534   0.03 0.04   0.29 0.01   0.16 |  8 hr      37 min\n``` \n\n``` \n4429 0.0010  11.4 k  159.6  0.1 m | 0.581   0.02 0.02   0.38 0.01   0.15 | 0.661   0.03 0.03   0.37 0.02   0.20 | 0.587   0.03 0.03   0.34 0.02   0.17 |  8 hr      42 min\n```\n\n```\n4430 0.0010  11.5 k  161.0  0.1 m | 0.445   0.02 0.02   0.24 0.01   0.16 | 0.618   0.03 0.03   0.35 0.02   0.20 | 0.367   0.02 0.02   0.15 0.01   0.17 |  8 hr      46 min\n```\n\n```\n4431 0.0010  11.6 k  162.4  0.1 m | 0.518   0.02 0.02   0.32 0.01   0.15 | 0.561   0.02 0.03   0.32 0.01   0.18 | 0.745   0.05 0.04   0.45 0.01   0.19 |  8 hr      50 min\n```\n\n```\n4432 0.0010  11.7 k  163.8  0.1 m | 0.541   0.02 0.02   0.34 0.01   0.15 | 0.602   0.03 0.03   0.34 0.02   0.19 | 0.729   0.03 0.04   0.40 0.02   0.24 |  8 hr\n```"
        },
        {
          "id": 309052,
          "postDate": "2018-04-04T15:02:36.290Z",
          "content": "<p>To the second question: Since we are measuring mAP at different IoU thresholds (up to 0.95) I would think that if the annotations on the external data are performed only in a slightly different way (e.g. the contours are drawn wider etc.) to the original data, this might pose a problem for our final score. At these high thresholds, it seems that overfitting to the individual annotators is actually happening.</p>",
          "rawMarkdown": "To the second question: Since we are measuring mAP at different IoU thresholds (up to 0.95) I would think that if the annotations on the external data are performed only in a slightly different way (e.g. the contours are drawn wider etc.) to the original data, this might pose a problem for our final score. At these high thresholds, it seems that overfitting to the individual annotators is actually happening.",
          "votes": 2
        },
        {
          "id": 309078,
          "postDate": "2018-04-04T15:49:27.843Z",
          "content": "<p>you need to measure your annotation error. Then make some calculation. Is it better to miss the detection or detect it at some iou?</p>",
          "rawMarkdown": "you need to measure your annotation error. Then make some calculation. Is it better to miss the detection or detect it at some iou?",
          "votes": 1
        }
      ]
    },
    {
      "id": 315320,
      "postDate": "2018-04-17T01:12:47.010Z",
      "content": "<p>here is the ensemble code  uploaded (fixed for zero detection) for submission and making results for stage2</p>",
      "rawMarkdown": "here is the ensemble code  uploaded (fixed for zero detection) for submission and making results for stage2",
      "votes": 1
    },
    {
      "id": 311692,
      "postDate": "2018-04-10T14:14:04.893Z",
      "content": "<p>TTA trick. if it works in test, do it in train augmentation too.</p>\n\n<p>there is a bug in my scale_to_factor() code. The is the corrected one:</p>\n\n<pre><code>def scale_to_factor(image, scale_x, scale_y, factor=16):\nheight,width = image.shape[:2]\nh = math.ceil(scale_y*height/factor)*factor\nw = math.ceil(scale_x*width/factor)*factor\nimage = cv2.resize(image,(w,h))\nreturn image\n</code></pre>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311692/9071/test_time_trick.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "TTA trick. if it works in test, do it in train augmentation too.\n\nthere is a bug in my scale_to_factor() code. The is the corrected one:\n\n    def scale_to_factor(image, scale_x, scale_y, factor=16):\n    height,width = image.shape[:2]\n    h = math.ceil(scale_y*height/factor)*factor\n    w = math.ceil(scale_x*width/factor)*factor\n    image = cv2.resize(image,(w,h))\n    return image\n\n\n![enter image description here][1]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311692/9071/test_time_trick.png",
      "votes": 1
    },
    {
      "id": 311611,
      "postDate": "2018-04-10T11:14:45.847Z",
      "content": "<p>Overlapping boundary. This is where mask rcnn is better than unet. However, this is not required in the challenge. </p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311611/9066/overlapping_boundary.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311611/9067/overlapping_boundary1.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "Overlapping boundary. This is where mask rcnn is better than unet. However, this is not required in the challenge. \n\n![enter image description here][1]\n\n![enter image description here][2]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311611/9066/overlapping_boundary.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311611/9067/overlapping_boundary1.png",
      "votes": 1,
      "replies": [
        {
          "id": 311616,
          "postDate": "2018-04-10T11:25:50.313Z",
          "content": "<p>赞</p>",
          "rawMarkdown": "赞"
        }
      ]
    },
    {
      "id": 310636,
      "postDate": "2018-04-08T06:12:41.867Z",
      "content": "<p>@chenchen</p>\n\n<p>\" I did't know the 'aux' is ?\" as attached</p>",
      "rawMarkdown": "@chenchen\n\n\" I did't know the 'aux' is ?\" as attached",
      "votes": 1,
      "replies": [
        {
          "id": 310652,
          "postDate": "2018-04-08T07:10:33.817Z",
          "content": "<p>I mean the \"aux\" in your predict_arguments.py . But there are not \"aux\" in your rcnn_nms\ndef undo_test_augment_identity(net, image):</p>\n\n<pre><code>height,width = image.shape[:2]\n# h = math.ceil(height/AUG_FACTOR)*AUG_FACTOR\n# w = math.ceil(width /AUG_FACTOR)*AUG_FACTOR\n# dx = w-width\n# dy = h-height\n\nrcnn_proposal = net.rcnn_proposals.cpu().numpy()\ndetection = net.detections.data.cpu().numpy()\nmask =  net.masks[0]\n\n\n# rcnn_proposal[:,1]=np.clip(rcnn_proposal[:,1]-AUG_BORDER,0,width -1)\n# rcnn_proposal[:,2]=np.clip(rcnn_proposal[:,2]-AUG_BORDER,0,height-1)\n# rcnn_proposal[:,3]=np.clip(rcnn_proposal[:,3]-AUG_BORDER,0,width -1)\n# rcnn_proposal[:,4]=np.clip(rcnn_proposal[:,4]-AUG_BORDER,0,height-1)\nps = rcnn_proposal.copy()\nrcnn_proposal=[]\nfor p in ps:\n    i,x0,y0,x1,y1, score, label, aux = p\n    x0 = max(min(x0,width -1),0)\n    x1 = max(min(x1,width -1),0)\n    y0 = max(min(y0,height-1),0)\n    y1 = max(min(y1,height-1),0)\n    w = x1-x0 + 1\n    h = y1-y0 + 1\n\n    if w&gt;2 and h&gt;2:\n        rcnn_proposal.append([i,x0,y0,x1,y1, score, label, aux])\n</code></pre>",
          "rawMarkdown": "I mean the \"aux\" in your predict_arguments.py . But there are not \"aux\" in your rcnn_nms\ndef undo_test_augment_identity(net, image):\n\n    height,width = image.shape[:2]\n    # h = math.ceil(height/AUG_FACTOR)*AUG_FACTOR\n    # w = math.ceil(width /AUG_FACTOR)*AUG_FACTOR\n    # dx = w-width\n    # dy = h-height\n\n    rcnn_proposal = net.rcnn_proposals.cpu().numpy()\n    detection = net.detections.data.cpu().numpy()\n    mask =  net.masks[0]\n\n\n    # rcnn_proposal[:,1]=np.clip(rcnn_proposal[:,1]-AUG_BORDER,0,width -1)\n    # rcnn_proposal[:,2]=np.clip(rcnn_proposal[:,2]-AUG_BORDER,0,height-1)\n    # rcnn_proposal[:,3]=np.clip(rcnn_proposal[:,3]-AUG_BORDER,0,width -1)\n    # rcnn_proposal[:,4]=np.clip(rcnn_proposal[:,4]-AUG_BORDER,0,height-1)\n    ps = rcnn_proposal.copy()\n    rcnn_proposal=[]\n    for p in ps:\n        i,x0,y0,x1,y1, score, label, aux = p\n        x0 = max(min(x0,width -1),0)\n        x1 = max(min(x1,width -1),0)\n        y0 = max(min(y0,height-1),0)\n        y1 = max(min(y1,height-1),0)\n        w = x1-x0 + 1\n        h = y1-y0 + 1\n\n        if w&gt;2 and h&gt;2:\n            rcnn_proposal.append([i,x0,y0,x1,y1, score, label, aux])"
        },
        {
          "id": 310683,
          "postDate": "2018-04-08T09:16:30.457Z",
          "content": "<p>Could you show me your rcnn_nms.py ?thanks very much</p>",
          "rawMarkdown": "Could you show me your rcnn_nms.py ?thanks very much"
        },
        {
          "id": 310751,
          "postDate": "2018-04-08T13:34:39.667Z",
          "content": "<p>same error when ran new mask_nms.py</p>\n\n<p>/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/layer/mask_nms.py\", line 346, in mask_nms\n    b, x0, y0, x1, y1, score, label, _ = proposals[t]\nValueError: not enough values to unpack (expected 8, got 7)</p>\n\n<p>Note that, it can run if i delete one return value \"_\" (aux) in above line. However , I am not sure if it affect to program or not</p>",
          "rawMarkdown": "same error when ran new mask_nms.py\n\n/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/layer/mask_nms.py\", line 346, in mask_nms\n    b, x0, y0, x1, y1, score, label, _ = proposals[t]\nValueError: not enough values to unpack (expected 8, got 7)\n\nNote that, it can run if i delete one return value \"_\" (aux) in above line. However , I am not sure if it affect to program or not"
        },
        {
          "id": 310889,
          "postDate": "2018-04-09T01:36:53.420Z",
          "content": "<p>@chenchen: Have you successful to run his augmentation code?</p>\n\n<p>@Heng: I found that the current code cannot run your <code>predict_arguments.py</code> because the old code only provides 7 return values in  <code>mask_nms.py</code>. I have updated your new <code>mask_nms.py</code> but it also has some errors. Besides, could you show the <code>scale_to_factor</code> function in the <code>predict_arguments.py</code></p>",
          "rawMarkdown": "@chenchen: Have you successful to run his augmentation code?\n\n@Heng: I found that the current code cannot run your `predict_arguments.py` because the old code only provides 7 return values in  `mask_nms.py`. I have updated your new `mask_nms.py` but it also has some errors. Besides, could you show the `scale_to_factor` function in the `predict_arguments.py`"
        },
        {
          "id": 310907,
          "postDate": "2018-04-09T02:26:19.223Z",
          "content": "<p>use \"mask-rcnn-ver-12.tar.gz (5.84 MB)\" below</p>",
          "rawMarkdown": "use \"mask-rcnn-ver-12.tar.gz (5.84 MB)\" below",
          "votes": 2
        },
        {
          "id": 310926,
          "postDate": "2018-04-09T03:29:43.497Z",
          "content": "<p>Will I be able to use Cuda 8.0 or 9.0?</p>",
          "rawMarkdown": "Will I be able to use Cuda 8.0 or 9.0?",
          "votes": 1
        },
        {
          "id": 310931,
          "postDate": "2018-04-09T03:59:35.253Z",
          "content": "<p>Hello Heng, the code worked well. However, I got the cuda memory error after 3 mins. I have reduced the batch size to 3 (TitanX 12GB) but it still happens, do we have any configuration need to change?</p>\n\n<pre><code>     rate    iter   epoch  num   | valid_loss               | train_loss               | batch_loss               |  time          \n-------------------------------------------------------------------------------------------------------------------------------\n0.0000   0.0 k    0.0  0.0 m | 1.989   0.17 0.45   0.69 0.00   0.68 | 0.000   0.00 0.00   0.00 0.00   0.00 | 0.000   0.00 0.00   0.00 0.00   0.00 |  0 hr 00 min\n0.0100   0.1 k    0.7  0.0 m | 1.172   0.05 0.13   0.59 0.06   0.34 | 1.143   0.07 0.15   0.47 0.11   0.35 | 1.066   0.07 0.12   0.42 0.07   0.39 |  0 hr 04 min  99,99,\n0.0100   0.1 k    0.9  0.0 m | 1.172   0.05 0.13   0.59 0.06   0.34 | 1.048   0.07 0.13   0.46 0.07   0.31 | 1.167   0.10 0.13   0.48 0.11   0.35 |  0 hr 06 min  122,122\n\n,THCudaCheck FAIL file=/home/john/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory\nTraceback (most recent call last):\n  File \"train_mask_rcnn_net_3.py\", line 530, in &lt;module&gt;\n    run_train()\n  File \"train_mask_rcnn_net_3.py\", line 414, in run_train\n    net.forward( inputs, truth_boxes, truth_labels, truth_instances )\n  File \"/home/john/mask-rcnn-ver-12/net/se_resnext50_mask_rcnn_2crop/model.py\", line 459, in forward\n    self.mask_logits = data_parallel(self.mask_head, mask_crops)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py\", line 142, in data_parallel\n    return module(*inputs[0], **module_kwargs[0])\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n    result = self.forward(*input, **kwargs)\n  File \"/home/john/mask-rcnn-ver-12/net/se_resnext50_mask_rcnn_2crop/model.py\", line 388, in forward\n    x = self.up(x)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n    result = self.forward(*input, **kwargs)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py\", line 663, in forward\n    output_padding, self.groups, self.dilation)\nRuntimeError: cuda runtime error (2) : out of memory at /home/john/pytorch/aten/src/THC/generic/THCStorage.cu:58\n</code></pre>",
          "rawMarkdown": "Hello Heng, the code worked well. However, I got the cuda memory error after 3 mins. I have reduced the batch size to 3 (TitanX 12GB) but it still happens, do we have any configuration need to change?\n\n\n         rate    iter   epoch  num   | valid_loss               | train_loss               | batch_loss               |  time          \n    -------------------------------------------------------------------------------------------------------------------------------\n    0.0000   0.0 k    0.0  0.0 m | 1.989   0.17 0.45   0.69 0.00   0.68 | 0.000   0.00 0.00   0.00 0.00   0.00 | 0.000   0.00 0.00   0.00 0.00   0.00 |  0 hr 00 min\n    0.0100   0.1 k    0.7  0.0 m | 1.172   0.05 0.13   0.59 0.06   0.34 | 1.143   0.07 0.15   0.47 0.11   0.35 | 1.066   0.07 0.12   0.42 0.07   0.39 |  0 hr 04 min  99,99,\n    0.0100   0.1 k    0.9  0.0 m | 1.172   0.05 0.13   0.59 0.06   0.34 | 1.048   0.07 0.13   0.46 0.07   0.31 | 1.167   0.10 0.13   0.48 0.11   0.35 |  0 hr 06 min  122,122\n\n    ,THCudaCheck FAIL file=/home/john/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory\n    Traceback (most recent call last):\n      File \"train_mask_rcnn_net_3.py\", line 530, in "
        },
        {
          "id": 310938,
          "postDate": "2018-04-09T04:31:10.353Z",
          "content": "<p>you can reduce batch size for rpn,rcnn, mask at the configuration.py. I think you don't have to retrain. you can just put in your old trained model. should be compatible. the main objective is to test the ensemble part</p>",
          "rawMarkdown": "you can reduce batch size for rpn,rcnn, mask at the configuration.py. I think you don't have to retrain. you can just put in your old trained model. should be compatible. the main objective is to test the ensemble part\n",
          "votes": 1
        },
        {
          "id": 310941,
          "postDate": "2018-04-09T04:35:36.273Z",
          "content": "<p>I'm having trouble making it compatible to CUDA 9.0. What should I change in the config file? I tried changing it in the maker file but it didn't update. Thanks!</p>",
          "rawMarkdown": "I'm having trouble making it compatible to CUDA 9.0. What should I change in the config file? I tried changing it in the maker file but it didn't update. Thanks!"
        },
        {
          "id": 310944,
          "postDate": "2018-04-09T04:41:25.210Z",
          "content": "<p>For your information, I have changed many setting to solve it but it have not success. Only one way to solve it is that reduce <code>WIDTH, HEIGHT = 192,192</code> from <code>WIDTH, HEIGHT = 256,256</code></p>",
          "rawMarkdown": "For your information, I have changed many setting to solve it but it have not success. Only one way to solve it is that reduce `WIDTH, HEIGHT = 192,192` from `WIDTH, HEIGHT = 256,256`"
        },
        {
          "id": 310947,
          "postDate": "2018-04-09T04:43:59.453Z",
          "content": "<p>you can switch back to 256,256 later where the model become trained and there will be less rcnn detection per image. Maybe in the initial training, there are many random detection per image and memory requirement is larger.</p>",
          "rawMarkdown": "you can switch back to 256,256 later where the model become trained and there will be less rcnn detection per image. Maybe in the initial training, there are many random detection per image and memory requirement is larger.",
          "votes": 2
        },
        {
          "id": 310951,
          "postDate": "2018-04-09T04:59:24.963Z",
          "content": "<p>Great idea. I have run it and it worked well. I have not changed back to <code>256x256</code>. This is way that I ran Heng's code, hope it helps someone </p>\n\n<ol>\n<li>Rebuild his .so as readme.pdf in his thread [0.463]</li>\n<li>Train the model <code>train_mask_rcnn_net_3.py</code></li>\n<li>Run the <code>predict.py</code></li>\n<li>Run the <code>emsemble.py</code></li>\n</ol>\n\n<p>Let me know if I was missing any step. Nice code and good job @Heng</p>",
          "rawMarkdown": "Great idea. I have run it and it worked well. I have not changed back to `256x256`. This is way that I ran Heng's code, hope it helps someone \n\n1. Rebuild his .so as readme.pdf in his thread [0.463]\n2. Train the model `train_mask_rcnn_net_3.py`\n3. Run the `predict.py`\n4. Run the `emsemble.py`\n\nLet me know if I was missing any step. Nice code and good job @Heng",
          "votes": 1
        },
        {
          "id": 310957,
          "postDate": "2018-04-09T05:08:12.093Z",
          "content": "<p>@John1231983 </p>\n\n<p>I tried rebuilding the .so files and unsuccessful.  I tried with both cuda 9.0 (telsa v100 and 8.0 (K80) on floydhub. I also tried updating the makefile and setup.py file as well. </p>\n\n<p>I keep getting the following error: </p>\n\n<pre><code>---------------------------------------------------------------------------\n  ImportError                               Traceback (most recent call last)\n  &lt;ipython-input-1-0307244cc3a6&gt; in &lt;module&gt;()\n  6 from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\n  7 from net.lib.box.nms.torch_nms import torch_nms\n  ----&gt; 8 from net.lib.box.nms.gpu_nms import gpu_nms\n  9 from net.lib.box.nms.cython_nms import cython_nms\n 10 \n\n  ImportError: libcudart.so.9.1: cannot open shared object file: No such file or directory\n</code></pre>",
          "rawMarkdown": "@John1231983 \n\nI tried rebuilding the .so files and unsuccessful.  I tried with both cuda 9.0 (telsa v100 and 8.0 (K80) on floydhub. I also tried updating the makefile and setup.py file as well. \n\nI keep getting the following error: \n\n    ---------------------------------------------------------------------------\n      ImportError                               Traceback (most recent call last)\n      "
        },
        {
          "id": 310958,
          "postDate": "2018-04-09T05:10:30.007Z",
          "content": "<p>Sorry, I am using cuda 9.1. I have tried cuda 9.0 and got your error. After that,  I upgrade my gpu to 9.1, it takes short time than find the solution :)</p>",
          "rawMarkdown": "Sorry, I am using cuda 9.1. I have tried cuda 9.0 and got your error. After that,  I upgrade my gpu to 9.1, it takes short time than find the solution :)"
        },
        {
          "id": 310962,
          "postDate": "2018-04-09T05:26:47.027Z",
          "content": "<p>@John Thanks you </p>",
          "rawMarkdown": "@John Thanks you "
        },
        {
          "id": 311208,
          "postDate": "2018-04-09T16:05:48.910Z",
          "content": "<p>predict.py</p>\n\n<p>emsemble.py (run_ensemble_box)</p>\n\n<p>predict_mask_only</p>\n\n<p>emsemble.py (run_ensemble_mask)</p>",
          "rawMarkdown": "predict.py\n\nemsemble.py (run_ensemble_box)\n\npredict_mask_only\n\nemsemble.py (run_ensemble_mask)"
        },
        {
          "id": 311218,
          "postDate": "2018-04-09T16:20:14.137Z",
          "content": "<p>I run your code , but get some errors:\nbuild/mask-rcnn-ver-12/net/layer/roi_align_pool_tf/module.py\", line 40, in forward\n    ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1)\nValueError: too many values to unpack (expected 5)\nhow should I do?\nthanks very much!</p>\n\n<p>I use wrong .so files.... I have fixed it.</p>",
          "rawMarkdown": "I run your code , but get some errors:\nbuild/mask-rcnn-ver-12/net/layer/roi_align_pool_tf/module.py\", line 40, in forward\n    ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1)\nValueError: too many values to unpack (expected 5)\nhow should I do?\nthanks very much!\n\nI use wrong .so files.... I have fixed it."
        }
      ]
    },
    {
      "id": 308842,
      "postDate": "2018-04-04T07:23:29.477Z",
      "content": "<p>So.. Using these apporaches, how much did you improve your LB?</p>",
      "rawMarkdown": "So.. Using these apporaches, how much did you improve your LB?",
      "votes": 1,
      "replies": [
        {
          "id": 309439,
          "postDate": "2018-04-05T10:35:15.013Z",
          "content": "<p>for example:</p>\n\n<p>single se-resnext-50, 53 gray test images: LB=0.43</p>\n\n<p>single se-resnext-50 + ensemble on test augmentation, 53 gray test images: LB=0.46 to 0.47</p>",
          "rawMarkdown": "for example:\n\nsingle se-resnext-50, 53 gray test images: LB=0.43\n\nsingle se-resnext-50 + ensemble on test augmentation, 53 gray test images: LB=0.46 to 0.47",
          "votes": 4
        },
        {
          "id": 309444,
          "postDate": "2018-04-05T10:53:24.163Z",
          "content": "<p>Amazing. I have tested with flipud + fliplr+orginal but hurt performance (0.498 to 0.48). What kind of tech. in test agumentation do you use? Increase 3-4% only from test agu.?</p>",
          "rawMarkdown": "Amazing. I have tested with flipud + fliplr+orginal but hurt performance (0.498 to 0.48). What kind of tech. in test agumentation do you use? Increase 3-4% only from test agu.?"
        },
        {
          "id": 309710,
          "postDate": "2018-04-05T20:55:34.110Z",
          "content": "<p>I agree with Heng, if we have large overlaps with time augment results then prediction looks confident.\nHere is what I get:</p>\n\n<ul>\n<li>Original LB=0.482</li>\n<li>FlipLR LB=0.484 </li>\n<li>FlipUD LB=0.482</li>\n</ul>\n\n<p>Then by ensembling these results:</p>\n\n<ul>\n<li>With NMS ensembling, I get LB 0.487</li>\n<li>With IoU ensembling, I get LB 0.50</li>\n</ul>",
          "rawMarkdown": "I agree with Heng, if we have large overlaps with time augment results then prediction looks confident.\nHere is what I get:\n\n * Original LB=0.482\n * FlipLR LB=0.484 \n * FlipUD LB=0.482\n\nThen by ensembling these results:\n\n* With NMS ensembling, I get LB 0.487\n* With IoU ensembling, I get LB 0.50"
        },
        {
          "id": 309814,
          "postDate": "2018-04-06T02:41:00.063Z",
          "content": "<p>actually we can train a classifier to predict the best form of ensemble method. it can be different ensemble method for different characteristics of the input(e.g. overlap area, confident score, etc). This is also where external data is useful. And we do not need accurate labels on this external label, because we can set the learning problem as:</p>\n\n<p>y = esemble_of_xs = Model(x1,x2,x3 ... )</p>\n\n<p>learning objective:  distance(y,y_hat) is better than distance(y1,y_hat),  distance(y2,y_hat), .... </p>\n\n<p>learning objective is NOT minimize distance(y,y_hat) to 0, since y_hat is unknown.</p>",
          "rawMarkdown": "actually we can train a classifier to predict the best form of ensemble method. it can be different ensemble method for different characteristics of the input(e.g. overlap area, confident score, etc). This is also where external data is useful. And we do not need accurate labels on this external label, because we can set the learning problem as:\n\ny = esemble_of_xs = Model(x1,x2,x3 ... )\n\nlearning objective:  distance(y,y_hat) is better than distance(y1,y_hat),  distance(y2,y_hat), .... \n\nlearning objective is NOT minimize distance(y,y_hat) to 0, since y_hat is unknown.",
          "votes": 2
        },
        {
          "id": 309815,
          "postDate": "2018-04-06T02:43:53.020Z",
          "content": "<blockquote>\n  <blockquote>\n    <p>With IoU ensembling, I get LB 0.50</p>\n  </blockquote>\n</blockquote>\n\n<p>imagine if your retrain with pseudo labels that gives \"LB 0.50\", and repeat the ensemble process. Maybe results will improve.</p>\n\n<p>Since without ensemble, results is \"LB 0.48\". pseudo labels that gives \"LB 0.50\" can product new supervisory signals to improve each original model.</p>",
          "rawMarkdown": "&gt;&gt;With IoU ensembling, I get LB 0.50\n\nimagine if your retrain with pseudo labels that gives \"LB 0.50\", and repeat the ensemble process. Maybe results will improve.\n\nSince without ensemble, results is \"LB 0.48\". pseudo labels that gives \"LB 0.50\" can product new supervisory signals to improve each original model."
        },
        {
          "id": 310016,
          "postDate": "2018-04-06T12:03:25.317Z",
          "content": "<p>@MPWare Team : What is IOU Ensemble? </p>",
          "rawMarkdown": "@MPWare Team : What is IOU Ensemble? "
        }
      ]
    },
    {
      "id": 311065,
      "postDate": "2018-04-09T09:49:51.990Z",
      "content": "<p>added scale_filp_transpose argument(). you now can have up to 26 test augmentation per image</p>",
      "rawMarkdown": "added scale_filp_transpose argument(). you now can have up to 26 test augmentation per image\n\n\n",
      "votes": 2,
      "replies": [
        {
          "id": 311144,
          "postDate": "2018-04-09T13:43:03.253Z",
          "content": "<p>@Heng. After run the <code>emsemble.py</code>, we have</p>\n\n<p>overall = overall/num_ensemble</p>\n\n<p>How to convert it to mask (int value instead of prob.) to run rle encoding?</p>",
          "rawMarkdown": "@Heng. After run the `emsemble.py`, we have\n\noverall = overall/num_ensemble\n\nHow to convert it to mask (int value instead of prob.) to run rle encoding?\n"
        },
        {
          "id": 311192,
          "postDate": "2018-04-09T15:15:56.013Z",
          "content": "<p>This is the tricky part. To get best results, there are several post processing which i can only reveal after the competition.</p>\n\n<p>For a simple solution:</p>\n\n<ul>\n<li><p>there is instance = num_augments * num_masks_in_one_image * image_height * image_width.\naverage over all augments, then you have average mask per instance object. you can then threshold at 0.5.</p></li>\n<li><p>try to score this average instant mask. if the variance is large (different augments disagree), score is low</p></li>\n<li><p>apply mask mns on these average instance mask, using the score. This is to decide which mask will be on top of others.</p></li>\n<li><p>further post process to remove small fragments, fill holes, etc</p></li>\n</ul>\n\n<p>Or you can actually trained a mask-rcnn using \"overall+ original_image\" as input</p>",
          "rawMarkdown": "This is the tricky part. To get best results, there are several post processing which i can only reveal after the competition.\n\nFor a simple solution:\n\n- there is instance = num_augments * num_masks_in_one_image * image_height * image_width.\n   average over all augments, then you have average mask per instance object. you can then threshold at 0.5.\n\n- try to score this average instant mask. if the variance is large (different augments disagree), score is low\n\n- apply mask mns on these average instance mask, using the score. This is to decide which mask will be on top of others.\n\n- further post process to remove small fragments, fill holes, etc\n\n\nOr you can actually trained a mask-rcnn using \"overall+ original_image\" as input",
          "votes": 1
        },
        {
          "id": 311450,
          "postDate": "2018-04-10T04:55:43.173Z",
          "content": "<p>Thanks. I used your previous ensemble code and it hurt my performance. It reduce to 0.44 from 0.46 (without ensemble). I guess ensemble make many isolate regions and makes contour between nucleic closer</p>",
          "rawMarkdown": "Thanks. I used your previous ensemble code and it hurt my performance. It reduce to 0.44 from 0.46 (without ensemble). I guess ensemble make many isolate regions and makes contour between nucleic closer"
        }
      ]
    },
    {
      "id": 308940,
      "postDate": "2018-04-04T12:02:19.220Z",
      "content": "<p>this is how it looks when you do test-time-augmentation (e.g. flip test images).</p>\n\n<p>Intensity = number of time detected</p>\n\n<p>The trick to get good results is how to ensemble these results!</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308940/8935/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\" alt=\"enter image description here\">   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308940/8933/4be73d68f433869188fe5e7f09c7f681ed51003da6aa5d19ce368726d8e271ee.png\" alt=\"enter image description here\">  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308940/8932/7f4cbe0b36b5d09466476a7d4e01f4f976c67872d549f4ff47b3e1e3a2b403af.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "this is how it looks when you do test-time-augmentation (e.g. flip test images).\n\nIntensity = number of time detected\n\nThe trick to get good results is how to ensemble these results!\n\n\n  ![enter image description here][1]   ![enter image description here][2]  ![enter image description here][3]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308940/8935/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308940/8933/4be73d68f433869188fe5e7f09c7f681ed51003da6aa5d19ce368726d8e271ee.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308940/8932/7f4cbe0b36b5d09466476a7d4e01f4f976c67872d549f4ff47b3e1e3a2b403af.png",
      "votes": 2,
      "replies": [
        {
          "id": 309443,
          "postDate": "2018-04-05T10:52:01.740Z",
          "content": "<p>you note that if there are multiple overlap from different augmentations, the prediction is confident and mostly correct.</p>\n\n<p>we can also identify non-confident regions (few overlap).</p>\n\n<p>This give a clue of how to do automatic pseudo-labeling:</p>\n\n<ul>\n<li><p>confident region : label as mask</p></li>\n<li><p>non-confident region: just color with background color (don't care region during learning)</p></li>\n</ul>\n\n<p>human error is accuracy is about LB 0.55. So if your algorithm has accuracy greater then this, you can try pseudo-labeling</p>",
          "rawMarkdown": "you note that if there are multiple overlap from different augmentations, the prediction is confident and mostly correct.\n\nwe can also identify non-confident regions (few overlap).\n\nThis give a clue of how to do automatic pseudo-labeling:\n\n- confident region : label as mask\n\n- non-confident region: just color with background color (don't care region during learning)\n\nhuman error is accuracy is about LB 0.55. So if your algorithm has accuracy greater then this, you can try pseudo-labeling\n\n",
          "votes": 1
        }
      ]
    },
    {
      "id": 308819,
      "postDate": "2018-04-04T06:41:03.890Z",
      "content": "<p>Great thank you for your strategy. I am finding the way to do the step 1 and your sentence \"cluster subject based on IoU\" give me many motivation and save my life :). The second point I did not try because it likes semi supervised learning</p>\n\n<p>For more clear to other people, I will explain more about point 1 of Heng, let me know if I was misunderstand\nAssume we have two outputs from two models for an image. The first output results in 5 masks, while the second results in 1 mask (background and mask if using Unet, FCN...) or 4 masks if using other checkpoints of masrcnn.  The first task of the first points is clustering masks into same cluster by measuring IoU. </p>\n\n<pre><code> for m1 in mask1\n\n      for m2 in mask2\n\n           max_IoU=0\n\n          if (max_IoU &lt; compute_IoU(m1,m2))\n\n              max_IoU=compute_IoU(m1,m2)\n\n      select m1,m2 that have max_IoU in same group\n</code></pre>\n\n<p>After first step, we can group masks for each output together to computer average, majorvoting. ....Let correct if I was wrong. Thanks</p>\n\n<p>Edit: Question: Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. Thus, we have to do extra step to separate cells by using finding contour, wastershed....which kind of method did you use to separate cells?</p>",
      "rawMarkdown": "Great thank you for your strategy. I am finding the way to do the step 1 and your sentence \"cluster subject based on IoU\" give me many motivation and save my life :). The second point I did not try because it likes semi supervised learning\n\nFor more clear to other people, I will explain more about point 1 of Heng, let me know if I was misunderstand\nAssume we have two outputs from two models for an image. The first output results in 5 masks, while the second results in 1 mask (background and mask if using Unet, FCN...) or 4 masks if using other checkpoints of masrcnn.  The first task of the first points is clustering masks into same cluster by measuring IoU. \n\n     for m1 in mask1\n\n          for m2 in mask2\n\n               max_IoU=0\n\n              if (max_IoU &lt; compute_IoU(m1,m2))\n\n                  max_IoU=compute_IoU(m1,m2)\n\n          select m1,m2 that have max_IoU in same group\n\nAfter first step, we can group masks for each output together to computer average, majorvoting. ....Let correct if I was wrong. Thanks\n\nEdit: Question: Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. Thus, we have to do extra step to separate cells by using finding contour, wastershed....which kind of method did you use to separate cells?",
      "votes": 2,
      "replies": [
        {
          "id": 308821,
          "postDate": "2018-04-04T06:46:17.203Z",
          "content": "<p>it is correct. Just one thing to note. I read that there are about 3000 stage2 test images, so the clustering algorithm should be efficient. Example is to measure box iou first, before measure mask iou, e.g.</p>\n\n<pre><code>def mask_iou(m1,m2):\n    if iou of box(m1) box(m2) == zero, return 0\n    else ... further processing ....\n</code></pre>\n\n<p>for \"Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. \", we label using our predictor first, then correct by hand</p>",
          "rawMarkdown": "it is correct. Just one thing to note. I read that there are about 3000 stage2 test images, so the clustering algorithm should be efficient. Example is to measure box iou first, before measure mask iou, e.g.\n\n    def mask_iou(m1,m2):\n        if iou of box(m1) box(m2) == zero, return 0\n        else ... further processing ....\n\n\n\nfor \"Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. \", we label using our predictor first, then correct by hand",
          "votes": 3
        },
        {
          "id": 309360,
          "postDate": "2018-04-05T06:11:32.293Z",
          "content": "<p>Hello Heng, I did it and it looks that combine testing augmentation may not good for case of fliplr and flipud. I will test with rotation some angle. For the BBBC06 dataset, I found many raw images (likes BBBC006_v1_images_z_18.zip, BBBC006_v1_images_z_19.zip...), which should I selecte for raw image? </p>",
          "rawMarkdown": "Hello Heng, I did it and it looks that combine testing augmentation may not good for case of fliplr and flipud. I will test with rotation some angle. For the BBBC06 dataset, I found many raw images (likes BBBC006_v1_images_z_18.zip, BBBC006_v1_images_z_19.zip...), which should I selecte for raw image? "
        },
        {
          "id": 309549,
          "postDate": "2018-04-05T14:58:50.493Z",
          "content": "<p>BBBC006_v1_images_z_16.zip</p>",
          "rawMarkdown": "BBBC006_v1_images_z_16.zip",
          "votes": 3
        }
      ]
    },
    {
      "id": 313962,
      "postDate": "2018-04-14T07:12:43.863Z",
      "content": "<p>Great!</p>",
      "rawMarkdown": "Great!"
    },
    {
      "id": 313918,
      "postDate": "2018-04-14T04:24:46.657Z",
      "content": "<p>Hi, Heng, where is your function of emsemble.py (run_ensemble_mask)?</p>",
      "rawMarkdown": "Hi, Heng, where is your function of emsemble.py (run_ensemble_mask)?"
    },
    {
      "id": 313568,
      "postDate": "2018-04-13T13:53:40.353Z",
      "content": "<p>in stage2 I have some images that I don't predict anything and there are empty. how do I change the ensembling code to support that? </p>",
      "rawMarkdown": "in stage2 I have some images that I don't predict anything and there are empty. how do I change the ensembling code to support that? ",
      "replies": [
        {
          "id": 313582,
          "postDate": "2018-04-13T14:05:47.543Z",
          "content": "<p>three ways:\n1) add a check. if sum of instance ==0 continue</p>\n\n<p>2) try and catch. if there is error move to next image</p>\n\n<p>3) this is without code modification, but i think 1 and 2 does not violate the rules and are better solution. The third solution is to create an input image file list. if there is error remove it from the input list and put in a reject list. Then break the list into another new list, starting from the next image file. Kind of stupid solution, but this is one that stricting does not change code.</p>\n\n<p>Since we are data scientist and we know what is \"model change\" verus \"code change\". I think if you don't check your model, you can change your code. kaggle will check your code if they need to and you can explain to them.</p>\n\n<p>Without code modification, the code cannot run.</p>",
          "rawMarkdown": "three ways:\n1) add a check. if sum of instance ==0 continue\n\n2) try and catch. if there is error move to next image\n\n3) this is without code modification, but i think 1 and 2 does not violate the rules and are better solution. The third solution is to create an input image file list. if there is error remove it from the input list and put in a reject list. Then break the list into another new list, starting from the next image file. Kind of stupid solution, but this is one that stricting does not change code.\n\nSince we are data scientist and we know what is \"model change\" verus \"code change\". I think if you don't check your model, you can change your code. kaggle will check your code if they need to and you can explain to them.\n\nWithout code modification, the code cannot run.\n"
        }
      ]
    },
    {
      "id": 311678,
      "postDate": "2018-04-10T13:50:56.503Z",
      "content": "<p>AttributeError: 'MaskRcnnNet' object has no attribute 'mask_instances'</p>",
      "rawMarkdown": "AttributeError: 'MaskRcnnNet' object has no attribute 'mask_instances'",
      "replies": [
        {
          "id": 311680,
          "postDate": "2018-04-10T13:53:33.033Z",
          "content": "<p>this is becuase there is no detection.\n(see the forward function)\nmake a dummy mask_instances = zero arrays ( like mask = zero arrays).</p>\n\n<p>better still, your code should len(net.detections)&gt;0 before calling net.mask_instances</p>",
          "rawMarkdown": "this is becuase there is no detection.\n(see the forward function)\nmake a dummy mask_instances = zero arrays ( like mask = zero arrays).\n\nbetter still, your code should len(net.detections)&gt;0 before calling net.mask_instances",
          "votes": 1
        }
      ]
    },
    {
      "id": 311582,
      "postDate": "2018-04-10T09:39:05.430Z",
      "content": "<p>how much did u got before doing all those manipulations? </p>\n\n<p>our LB only from the maskrcnn is 0.532</p>",
      "rawMarkdown": "how much did u got before doing all those manipulations? \n\nour LB only from the maskrcnn is 0.532",
      "replies": [
        {
          "id": 311604,
          "postDate": "2018-04-10T11:00:58.127Z",
          "content": "<p>I get lb0.500 base on Heng CherKeng's maskrcnn code.\nHave not do ensemble yet. </p>",
          "rawMarkdown": "I get lb0.500 base on Heng CherKeng's maskrcnn code.\nHave not do ensemble yet. "
        },
        {
          "id": 311613,
          "postDate": "2018-04-10T11:19:50.420Z",
          "content": "<p>Hi Avi, how do you train color image? I use original color image and H&amp;E and mosaic but only got 0.061 LB ( sometime got 0.063)</p>",
          "rawMarkdown": "Hi Avi, how do you train color image? I use original color image and H&amp;E and mosaic but only got 0.061 LB ( sometime got 0.063)"
        },
        {
          "id": 311750,
          "postDate": "2018-04-10T15:53:52.300Z",
          "content": "<p>How many iterations did you need to reach 0.500 or 0.53? </p>",
          "rawMarkdown": "How many iterations did you need to reach 0.500 or 0.53? "
        },
        {
          "id": 311846,
          "postDate": "2018-04-10T19:05:25.210Z",
          "content": "<p>@MPWare: 0.53 is from matterport code. </p>",
          "rawMarkdown": "@MPWare: 0.53 is from matterport code. "
        },
        {
          "id": 311969,
          "postDate": "2018-04-11T02:07:32.787Z",
          "content": "<p>my single model without external data don't do well, about 0.45+0.06 (gray + color).</p>\n\n<p>With external data, ensemble and post processing, i can have a gain of +0.05 to 0.06</p>",
          "rawMarkdown": "my single model without external data don't do well, about 0.45+0.06 (gray + color).\n\nWith external data, ensemble and post processing, i can have a gain of +0.05 to 0.06"
        },
        {
          "id": 312001,
          "postDate": "2018-04-11T03:32:01.517Z",
          "content": "<p>I got 0.063-0.066 on color.  But I trained the color without preprocessing (using original color information)+ mosaic. I have two models: 1 trained on gray images, and 1 trained on color images +H&amp;E. Have you merge color and gray into one set when train the color? Someone informed that it has no benefit</p>",
          "rawMarkdown": "I got 0.063-0.066 on color.  But I trained the color without preprocessing (using original color information)+ mosaic. I have two models: 1 trained on gray images, and 1 trained on color images +H&amp;E. Have you merge color and gray into one set when train the color? Someone informed that it has no benefit",
          "votes": 1
        }
      ]
    },
    {
      "id": 311537,
      "postDate": "2018-04-10T08:06:02.497Z",
      "content": "<p>What is IOU Ensemble? \nHow can I make submissions from ensemble folders \".npy\" or overlays \"/%s.png\"?</p>",
      "rawMarkdown": "What is IOU Ensemble? \nHow can I make submissions from ensemble folders \".npy\" or overlays \"/%s.png\"?"
    },
    {
      "id": 311486,
      "postDate": "2018-04-10T06:19:29.323Z",
      "content": "<p>this is how to do boundary detection with mask rcnn</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311486/9060/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311486/9061/0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311486/9062/0ed3555a4bd48046d3b63d8baf03a5aa97e523aa483aaa07459e7afa39fb96c6.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "this is how to do boundary detection with mask rcnn\n\n\n  ![enter image description here][1]\n\n  ![enter image description here][2]\n\n  ![enter image description here][3]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311486/9060/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311486/9061/0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311486/9062/0ed3555a4bd48046d3b63d8baf03a5aa97e523aa483aaa07459e7afa39fb96c6.png"
    },
    {
      "id": 311246,
      "postDate": "2018-04-09T17:24:24.763Z",
      "content": "<p>I have an error  after run 20k iterations. have you see it too?</p>\n\n<pre><code>File \"train_mask_rcnn_net_3.py\", line 530, in &lt;module&gt;\n    run_train()\n  File \"train_mask_rcnn_net_3.py\", line 471, in run_train\n    all7 = draw_mask_metric(image, mask, truth_mask)\n  File \"/home/john/mask-rcnn-ver-12/draw.py\", line 315, in draw_mask_metric\n    overlap = np.max(iou,1)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py\", line 2320, in amax\n    out=out, **kwargs)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/numpy/core/_methods.py\", line 26, in _amax\n    return umr_maximum(a, axis, None, out, keepdims)\nValueError: zero-size array to reduction operation maximum which has no identity\n</code></pre>",
      "rawMarkdown": "I have an error  after run 20k iterations. have you see it too?\n    \n    File \"train_mask_rcnn_net_3.py\", line 530, in "
    },
    {
      "id": 310761,
      "postDate": "2018-04-08T14:25:18.893Z",
      "content": "<p>good job =)</p>",
      "rawMarkdown": "good job =)"
    },
    {
      "id": 310645,
      "postDate": "2018-04-08T06:42:24.400Z",
      "content": "<p>Hello Heng, may i ask you a question about LB in color set? Most of report here or other thread related to gray (based LB is 0.43+).  What is LB can we expect on 12 color set without using test augmentation? I remember that you said you achieved 0.05 LB on 12 color images, that is too low than me 0.063. My strategy is train all color images +H&amp;E aug in 1 network with initial checkpoint is from gray set. I did not apply any color normalization. Do you find any improvement when we using color normalization  like stain normalization?</p>",
      "rawMarkdown": "Hello Heng, may i ask you a question about LB in color set? Most of report here or other thread related to gray (based LB is 0.43+).  What is LB can we expect on 12 color set without using test augmentation? I remember that you said you achieved 0.05 LB on 12 color images, that is too low than me 0.063. My strategy is train all color images +H&amp;E aug in 1 network with initial checkpoint is from gray set. I did not apply any color normalization. Do you find any improvement when we using color normalization  like stain normalization?",
      "replies": [
        {
          "id": 311003,
          "postDate": "2018-04-09T07:17:19.407Z",
          "content": "<p>Hi, for color images, I train a model initial from gray set using 108 purple images. But I only got 0.044 for 12 color images, even worse than directly using model trained on 500 gray images. Can you give me some suggestion how to improve? For color images, I normalize them by reverting the color (i.e., 1-rgb2gray(img)). Thanks!</p>",
          "rawMarkdown": "Hi, for color images, I train a model initial from gray set using 108 purple images. But I only got 0.044 for 12 color images, even worse than directly using model trained on 500 gray images. Can you give me some suggestion how to improve? For color images, I normalize them by reverting the color (i.e., 1-rgb2gray(img)). Thanks!"
        },
        {
          "id": 311005,
          "postDate": "2018-04-09T07:19:09.270Z",
          "content": "<p>This model is overfitting a lot on these purple images.</p>",
          "rawMarkdown": "This model is overfitting a lot on these purple images."
        },
        {
          "id": 311008,
          "postDate": "2018-04-09T07:26:53.350Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 311010,
          "postDate": "2018-04-09T07:29:54.537Z",
          "content": "<p>I just used color information because converting to gray lost a lot of color information. I also use mosaic and H&amp;E external dataset. </p>\n\n<p>@Heng: What is LB on color image using your code?  </p>",
          "rawMarkdown": "I just used color information because converting to gray lost a lot of color information. I also use mosaic and H&amp;E external dataset. \n\n@Heng: What is LB on color image using your code?  "
        },
        {
          "id": 311045,
          "postDate": "2018-04-09T08:38:37.317Z",
          "content": "<p>Thanks! How much improvement are you get from mosaic and external dataset?</p>",
          "rawMarkdown": "Thanks! How much improvement are you get from mosaic and external dataset?"
        }
      ]
    },
    {
      "id": 310630,
      "postDate": "2018-04-08T06:05:05.743Z",
      "content": "<p>hi,thanks very much for your  'predict_arguments.py' code. I found in \"undo_test_augment_identity\" function ,it need 8 values--'i,x0,y0,x1,y1, score, label, aux' from rcnn_proposals ,but there are only 7 values in rcnn_nms function in your old codes. I did't know the 'aux' is ? Thank u for helping me!</p>",
      "rawMarkdown": "hi,thanks very much for your  'predict_arguments.py' code. I found in \"undo_test_augment_identity\" function ,it need 8 values--'i,x0,y0,x1,y1, score, label, aux' from rcnn_proposals ,but there are only 7 values in rcnn_nms function in your old codes. I did't know the 'aux' is ? Thank u for helping me!"
    },
    {
      "id": 310303,
      "postDate": "2018-04-07T03:50:13.823Z",
      "content": "<p>other ways to ensemble which i have not tried are:</p>\n\n<ul>\n<li><p>transform mask rcnn results into distance transform map, then average the maps and find local peaks.</p></li>\n<li><p>ensemble of contours, centers, etc</p></li>\n</ul>",
      "rawMarkdown": "other ways to ensemble which i have not tried are:\n\n-  transform mask rcnn results into distance transform map, then average the maps and find local peaks.\n\n- ensemble of contours, centers, etc"
    }
  ],
  "comments": [
    {
      "id": 309550,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-05T14:59:21.150000",
      "content": "<p>ensmeble code</p>",
      "votes": 10,
      "replies": [
        {
          "id": 309676,
          "author_name": "Yibing Wu",
          "author_url": "",
          "post_date": "2018-04-05T19:31:30.567000",
          "content": "<p>Thank you Heng for sharing the ensemble code. Could you also share the code for the function image_to_mask in \"mask       = image_to_mask(mask_image)\"?  Thanks much.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309816,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-06T02:45:42.610000",
          "content": "<p>Very useful. Could you also provide script to test with scale image likes scale_1.2, scale_0.8. I can scale image to 1.2 and result in x1.2 times with original image. However, I do not know how scale back to input image size. I think it is in the <code>def revert(net, images):</code> function</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309825,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-06T03:02:21.827000",
          "content": "<p>You have the raw mask probability at fixed size of e.g 28x28. Resize these to the original image detection box size</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 309952,
          "author_name": "yyll008",
          "author_url": "",
          "post_date": "2018-04-06T09:13:05.533000",
          "content": "<p>mask       = image_to_mask(mask_image)\nNameError: name 'image_to_mask' is not defined</p>\n\n<p>How to define this function?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309959,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-06T09:30:45.827000",
          "content": "<p>You do not need to use it. Just using the function to load from npy file. See submit.py for that or uncomment two above lines in his code. FYI, my score drop from 0.455 to 0.45 using fliplr+flipud+original</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 310287,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-07T02:33:19.360000",
          "content": "<p>Youvare using intersection, union or average in ensemble? Hiw about comparing the visual results between the orginal and ensemble submission. What \nis the difference? You can use the draw mask function, just treat one of them as ground truth.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 310288,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-07T02:40:00.600000",
          "content": "<p>@Heng: i am using original ensemble code (average) but different is that I only have three input: original, flipud and fliplr. I do not have scale 0.8 and 1.2 outputs. I spend 1 day to implement it based on your suggestion but it did not work. Could you please share the code to do scale prediction in the file submit.py?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310293,
          "author_name": "yyll008",
          "author_url": "",
          "post_date": "2018-04-07T02:50:35.720000",
          "content": "<p>Thanks, I know how to ensemble now.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310428,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-07T13:42:55.797000",
          "content": "<p>i will put my code later.</p>\n\n<p>you can also try this\n<a href=\"https://www.kaggle.com/c/data-science-bowl-2018/discussion/50795\">https://www.kaggle.com/c/data-science-bowl-2018/discussion/50795</a>, search for comments\n\"FYI: how to ensemble objector detector from resnet paper:\"</p>\n\n<p>say you have image x0, test-augments x1,x2,x3 ....</p>\n\n<p>let d0,d1 ,d2 .... be detection results of x0,x1,x2 respectively,</p>\n\n<p>compute: </p>\n\n<p>m0 = mask_rcnn_crop_and_mask_head_only( x0, (d0,d1 ,d2, ....))</p>\n\n<p>m1 = mask_rcnn_crop_and_mask_head_only( x1, (d0,d1 ,d2, ....))</p>\n\n<p>...</p>\n\n<p>then ensemble m0,m1,m2 ....</p>\n\n<p>We collect all detection boxes from all test augments. hence now we have more detection per augment image. Maybe mask prediction is more accurate in some augment image, so we average the results for each box</p>",
          "votes": 4,
          "replies": []
        }
      ]
    },
    {
      "id": 310906,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-09T02:25:57.833000",
      "content": "<p>compare ensemble and single  model results:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310906/9029/single.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310906/9030/ensemble.png\" alt=\"enter image description here\"></p>",
      "votes": 5,
      "replies": [
        {
          "id": 310919,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-09T03:11:21.377000",
          "content": "<p>The last (4th) image of each row is the ensembled results.</p>\n\n<p>Looking at it, you can actually train another model on the ensembled results. (i.e. 2 layer stacked framework)</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 311106,
          "author_name": "Ashish Lal",
          "author_url": "",
          "post_date": "2018-04-09T11:57:08.840000",
          "content": "<p>Hi Heng, I am using GTX 1080 Ti. I have compiled all the '.so' modules using 'sm_61' for 1080 Ti but I am still getting OOM errors. Can you please include a README (to compile any other modules for 1080 Ti)?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311110,
          "author_name": "Ashish Lal",
          "author_url": "",
          "post_date": "2018-04-09T12:14:20.927000",
          "content": "<p>Sorry. Pls igore. Found the answer below.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311154,
          "author_name": "William Green",
          "author_url": "",
          "post_date": "2018-04-09T13:57:16.417000",
          "content": "<p>@Watts </p>\n\n<p>How did you get Heng's code to work on a sm_61? Any assistance would be greatly appreciated. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311756,
          "author_name": "Ashish Lal",
          "author_url": "",
          "post_date": "2018-04-10T16:15:31.230000",
          "content": "<p>@ WIlliam Green</p>\n\n<p>Please use the README file from google drive distribution dtd 20180306 (see 0.463 thread). Replace sm_52 by sm_61 in all the commands that are used to build '<em>.so' files (including setup.py, build.py etc files as required). Then copy '</em>.so' files in the right places. That's about it. After that it runs pretty much as is.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 310484,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-07T17:16:16.910000",
      "content": "<p>A much better way to ensemble:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8957/Slide1.png\" alt=\"enter image description here\">\n   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8958/Slide2.png\" alt=\"enter image description here\">\n   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8959/Slide3.png\" alt=\"enter image description here\">\n   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310484/8960/Slide4.png\" alt=\"enter image description here\"></p>",
      "votes": 6,
      "replies": [
        {
          "id": 315236,
          "author_name": "avsanjay",
          "author_url": "",
          "post_date": "2018-04-16T23:10:10.563000",
          "content": "<p>Heng CherKeng,  Thank you so much for the code. It was a great learning experience. In box.py, i made some changes  in the box_transform function. I noticed that in most of the cases\n    dx = (tx - bx) / bw\n    dy = (ty - by) / bh\n    dw = np.log(tw / bw)\n    dh = np.log(th / bh)</p>\n\n<p>dx, dy, dw, dh would be zero. This would then affect/reduce  the number of rcnn targets. </p>\n\n<p>i added a loop </p>\n\n<pre><code>if (dx.all() == 0 or dy.all() == 0):\n    dx = tx\n    dy = ty\n    dw = tw\n    dh = th\n</code></pre>\n\n<p>I played around with dx = ( tx/1024) dy = ( ty/1024);  dx = ( tx/512) etc. This greatly increased the number of rcnn targets. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 315322,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-17T01:14:51.050000",
          "content": "<p>@avsanjay</p>\n\n<p>thanks!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 310430,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-07T14:04:13.453000",
      "content": "<p>as requested, test time augmentation code</p>",
      "votes": 6,
      "replies": []
    },
    {
      "id": 311615,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-10T11:24:45.300000",
      "content": "<p>btw, do we upload code by 11-Apr 23:59 hrs or 11-Apr 00:00 hrs. Now i am busy cleaning code!</p>\n\n<p>\"All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.\"</p>\n\n<p>So i think it is  \"11-Apr 23:59 hrs\". </p>",
      "votes": 3,
      "replies": [
        {
          "id": 311892,
          "author_name": "Ian Oliver",
          "author_url": "",
          "post_date": "2018-04-10T21:11:17.667000",
          "content": "<p>If you are in the UTC time zone, yes. <a href=\"https://www.wolframalpha.com/input/?i=april+11+2018+23:59+utc\">Conversion to detected local time zone</a></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 311096,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-09T11:23:14.237000",
      "content": "<p>this movie gives you an idea how to post process:</p>\n\n<p>from left to right: </p>\n\n<ul>\n<li>box of average mask, </li>\n<li>average mask, </li>\n<li>each mask in ensemble</li>\n</ul>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 311015,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-09T07:44:51.533000",
      "content": "<p>i put more ensemble results are the google drive:</p>\n\n<p><a href=\"https://drive.google.com/drive/folders/1Z9-eS5yr4jDPlCyyWqiV_3-m64lAPqRG\">https://drive.google.com/drive/folders/1Z9-eS5yr4jDPlCyyWqiV_3-m64lAPqRG</a></p>\n\n<p>I also put the trained model (00049000_model.pth) there.</p>\n\n<p>The results are visually very attractive.\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311015/9036/a984e7fb886aa02e29d112766d3ce26a4f78eac540ce7bbdbd42af2761928f6d.png\" alt=\"enter image description here\">\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311015/9034/43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af.png\" alt=\"enter image description here\">\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311015/9037/bdc789019cee8ddfae20d5f769299993b4b330b2d38d1218646cf89e77fbbd4d.png\" alt=\"enter image description here\"></p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 310917,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-09T03:07:41.090000",
      "content": "<p>You do not need perfect labels to learn. But you do need many labels per sample if they are ambiguous. For this reason, this is why pseudo-labels from ensemble classifiers should work. This is distilling knowledge from ensemble.\nThe problem is to control the percentage of label noise.  </p>\n\n<p>This is a neat trick that you can use in many kaggle competition. You now have a way to \"fit test data\"</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/310917/9032/self-learning.png\" alt=\"enter image description here\"></p>",
      "votes": 4,
      "replies": [
        {
          "id": 312464,
          "author_name": "conradr71",
          "author_url": "",
          "post_date": "2018-04-11T19:44:48.663000",
          "content": "<p>Have you tried implementing \"bootstrapping\" on noisy labels: </p>\n\n<p><a href=\"https://arxiv.org/pdf/1412.6596.pdf\">https://arxiv.org/pdf/1412.6596.pdf</a></p>\n\n<p>They only test the strategy on object detection with Multibox, but I think the loss function in 3.2 can be used in place of rpn class loss, rcnn class loss, and maybe the mask loss itself. When I  tried it, it did detect more nuclei that had been missed otherwise, but the masks were not very good. A more accurate model trained on clean labels might be able to use it effectively though.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 308815,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-04T06:38:03.253000",
      "content": "<p>some question you may ask:</p>\n\n<ol>\n<li><p>what if given an external image, there are some nuclei that i know how to label and some that i don't know?</p>\n\n<ul><li>draw a box to cover the nuclei that you don't know how to label and filled with black (or back ground color).</li></ul></li>\n<li><p>Does external data helps?</p>\n\n<ul><li>Yes, only if you annotate them correctly</li></ul></li>\n</ol>",
      "votes": 4,
      "replies": [
        {
          "id": 309014,
          "author_name": "kirk",
          "author_url": "",
          "post_date": "2018-04-04T13:44:49.180000",
          "content": "<p>Hi @Heng CherKeng thanks for sharing your thoughts and the code. I just need to ask a question since I'm not able to make sense out of the training phase. I've run your code and multiple times and for various iterations up to 30k and there's nowhere near that it reaches the 0.419 LB that you had in your <code>20180306</code> solution results. Any ideas on why is that since it is the same solution unless something has changed? Beyond that I see that the results are fluctuating up and down but in the same range from iteration to iteration? Here's an example:</p>\n\n<p><code>\n4418 0.0010  10.3 k  144.2  0.1 m | 0.559   0.02 0.02   0.35 0.01   0.16 | 0.660   0.03 0.03   0.38 0.02   0.20 | 0.515   0.02 0.03   0.26 0.01   0.19 |  7 hr      29 min\n</code></p>\n\n<p><code>\n4419 0.0010  10.4 k  145.6  0.1 m | 0.492   0.02 0.02   0.28 0.01   0.16 | 0.648   0.03 0.03   0.38 0.02   0.19 | 0.691   0.02 0.03   0.41 0.02   0.21 |  7 hr      37 min\n</code></p>\n\n<p><code>\n4420 0.0010  10.5 k  147.0  0.1 m | 0.529   0.02 0.02   0.32 0.01   0.16 | 0.605   0.03 0.03   0.34 0.02   0.19 | 0.639   0.03 0.03   0.36 0.01   0.21 |  7 hr      44 min\n</code></p>\n\n<p><code>\n4421 0.0010  10.6 k  148.4  0.1 m | 0.497   0.02 0.02   0.29 0.01   0.16 | 0.607   0.03 0.03   0.34 0.02   0.18 | 0.484   0.04 0.03   0.26 0.01   0.14 |  7 hr      51 min\n</code></p>\n\n<p><code>\n4422 0.0010  10.7 k  149.8  0.1 m | 0.482   0.02 0.02   0.27 0.01   0.16 | 0.597   0.03 0.03   0.35 0.01   0.18 | 0.829   0.05 0.03   0.55 0.01   0.20 |  7 hr      59 min\n</code></p>\n\n<p><code>\n4423 0.0010  10.8 k  151.2  0.1 m | 0.462   0.02 0.02   0.26 0.01   0.15 | 0.611   0.03 0.03   0.33 0.02   0.20 | 0.657   0.03 0.03   0.40 0.02   0.18 |  8 hr      06 min\n</code></p>\n\n<p><code>\n4424 0.0010  10.9 k  152.6  0.1 m | 0.480   0.02 0.02   0.27 0.01   0.16 | 0.575   0.02 0.03   0.31 0.02   0.19 | 0.508   0.03 0.03   0.26 0.01   0.18 |  8 hr      13 min\n</code></p>\n\n<p><code>\n4425 0.0010  11.0 k  154.0  0.1 m | 0.527   0.02 0.02   0.32 0.01   0.15 | 0.631   0.04 0.03   0.35 0.02   0.19 | 0.455   0.04 0.03   0.24 0.01   0.14 |  8 hr      21 min\n</code></p>\n\n<p><code>\n4426 0.0010  11.1 k  155.4  0.1 m | 0.461   0.02 0.02   0.26 0.01   0.15 | 0.549   0.03 0.03   0.30 0.01   0.18 | 0.702   0.01 0.05   0.42 0.02   0.20 |  8 hr      28 min\n</code></p>\n\n<p><code>\n4427 0.0010  11.2 k  156.8  0.1 m | 0.533   0.02 0.02   0.32 0.01   0.16 | 0.592   0.02 0.03   0.34 0.02   0.18 | 0.517   0.03 0.03   0.28 0.02   0.16 |  8 hr      32 min\n</code></p>\n\n<p><code>\n4428 0.0010  11.3 k  158.2  0.1 m | 0.508   0.02 0.02   0.30 0.01   0.16 | 0.636   0.03 0.03   0.37 0.02   0.19 | 0.534   0.03 0.04   0.29 0.01   0.16 |  8 hr      37 min\n</code> </p>\n\n<p><code>\n4429 0.0010  11.4 k  159.6  0.1 m | 0.581   0.02 0.02   0.38 0.01   0.15 | 0.661   0.03 0.03   0.37 0.02   0.20 | 0.587   0.03 0.03   0.34 0.02   0.17 |  8 hr      42 min\n</code></p>\n\n<p><code>\n4430 0.0010  11.5 k  161.0  0.1 m | 0.445   0.02 0.02   0.24 0.01   0.16 | 0.618   0.03 0.03   0.35 0.02   0.20 | 0.367   0.02 0.02   0.15 0.01   0.17 |  8 hr      46 min\n</code></p>\n\n<p><code>\n4431 0.0010  11.6 k  162.4  0.1 m | 0.518   0.02 0.02   0.32 0.01   0.15 | 0.561   0.02 0.03   0.32 0.01   0.18 | 0.745   0.05 0.04   0.45 0.01   0.19 |  8 hr      50 min\n</code></p>\n\n<p><code>\n4432 0.0010  11.7 k  163.8  0.1 m | 0.541   0.02 0.02   0.34 0.01   0.15 | 0.602   0.03 0.03   0.34 0.02   0.19 | 0.729   0.03 0.04   0.40 0.02   0.24 |  8 hr\n</code></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309052,
          "author_name": "Branislav Holländer",
          "author_url": "",
          "post_date": "2018-04-04T15:02:36.290000",
          "content": "<p>To the second question: Since we are measuring mAP at different IoU thresholds (up to 0.95) I would think that if the annotations on the external data are performed only in a slightly different way (e.g. the contours are drawn wider etc.) to the original data, this might pose a problem for our final score. At these high thresholds, it seems that overfitting to the individual annotators is actually happening.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 309078,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-04T15:49:27.843000",
          "content": "<p>you need to measure your annotation error. Then make some calculation. Is it better to miss the detection or detect it at some iou?</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 315320,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-17T01:12:47.010000",
      "content": "<p>here is the ensemble code  uploaded (fixed for zero detection) for submission and making results for stage2</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 311692,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-10T14:14:04.893000",
      "content": "<p>TTA trick. if it works in test, do it in train augmentation too.</p>\n\n<p>there is a bug in my scale_to_factor() code. The is the corrected one:</p>\n\n<pre><code>def scale_to_factor(image, scale_x, scale_y, factor=16):\nheight,width = image.shape[:2]\nh = math.ceil(scale_y*height/factor)*factor\nw = math.ceil(scale_x*width/factor)*factor\nimage = cv2.resize(image,(w,h))\nreturn image\n</code></pre>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311692/9071/test_time_trick.png\" alt=\"enter image description here\"></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 311611,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-10T11:14:45.847000",
      "content": "<p>Overlapping boundary. This is where mask rcnn is better than unet. However, this is not required in the challenge. </p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311611/9066/overlapping_boundary.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311611/9067/overlapping_boundary1.png\" alt=\"enter image description here\"></p>",
      "votes": 1,
      "replies": [
        {
          "id": 311616,
          "author_name": "chenchen",
          "author_url": "",
          "post_date": "2018-04-10T11:25:50.313000",
          "content": "<p>赞</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 310636,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-08T06:12:41.867000",
      "content": "<p>@chenchen</p>\n\n<p>\" I did't know the 'aux' is ?\" as attached</p>",
      "votes": 1,
      "replies": [
        {
          "id": 310652,
          "author_name": "chenchen",
          "author_url": "",
          "post_date": "2018-04-08T07:10:33.817000",
          "content": "<p>I mean the \"aux\" in your predict_arguments.py . But there are not \"aux\" in your rcnn_nms\ndef undo_test_augment_identity(net, image):</p>\n\n<pre><code>height,width = image.shape[:2]\n# h = math.ceil(height/AUG_FACTOR)*AUG_FACTOR\n# w = math.ceil(width /AUG_FACTOR)*AUG_FACTOR\n# dx = w-width\n# dy = h-height\n\nrcnn_proposal = net.rcnn_proposals.cpu().numpy()\ndetection = net.detections.data.cpu().numpy()\nmask =  net.masks[0]\n\n\n# rcnn_proposal[:,1]=np.clip(rcnn_proposal[:,1]-AUG_BORDER,0,width -1)\n# rcnn_proposal[:,2]=np.clip(rcnn_proposal[:,2]-AUG_BORDER,0,height-1)\n# rcnn_proposal[:,3]=np.clip(rcnn_proposal[:,3]-AUG_BORDER,0,width -1)\n# rcnn_proposal[:,4]=np.clip(rcnn_proposal[:,4]-AUG_BORDER,0,height-1)\nps = rcnn_proposal.copy()\nrcnn_proposal=[]\nfor p in ps:\n    i,x0,y0,x1,y1, score, label, aux = p\n    x0 = max(min(x0,width -1),0)\n    x1 = max(min(x1,width -1),0)\n    y0 = max(min(y0,height-1),0)\n    y1 = max(min(y1,height-1),0)\n    w = x1-x0 + 1\n    h = y1-y0 + 1\n\n    if w&gt;2 and h&gt;2:\n        rcnn_proposal.append([i,x0,y0,x1,y1, score, label, aux])\n</code></pre>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310683,
          "author_name": "chenchen",
          "author_url": "",
          "post_date": "2018-04-08T09:16:30.457000",
          "content": "<p>Could you show me your rcnn_nms.py ?thanks very much</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310751,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-08T13:34:39.667000",
          "content": "<p>same error when ran new mask_nms.py</p>\n\n<p>/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/layer/mask_nms.py\", line 346, in mask_nms\n    b, x0, y0, x1, y1, score, label, _ = proposals[t]\nValueError: not enough values to unpack (expected 8, got 7)</p>\n\n<p>Note that, it can run if i delete one return value \"_\" (aux) in above line. However , I am not sure if it affect to program or not</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310889,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T01:36:53.420000",
          "content": "<p>@chenchen: Have you successful to run his augmentation code?</p>\n\n<p>@Heng: I found that the current code cannot run your <code>predict_arguments.py</code> because the old code only provides 7 return values in  <code>mask_nms.py</code>. I have updated your new <code>mask_nms.py</code> but it also has some errors. Besides, could you show the <code>scale_to_factor</code> function in the <code>predict_arguments.py</code></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310907,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-09T02:26:19.223000",
          "content": "<p>use \"mask-rcnn-ver-12.tar.gz (5.84 MB)\" below</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 310926,
          "author_name": "William Green",
          "author_url": "",
          "post_date": "2018-04-09T03:29:43.497000",
          "content": "<p>Will I be able to use Cuda 8.0 or 9.0?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 310931,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T03:59:35.253000",
          "content": "<p>Hello Heng, the code worked well. However, I got the cuda memory error after 3 mins. I have reduced the batch size to 3 (TitanX 12GB) but it still happens, do we have any configuration need to change?</p>\n\n<pre><code>     rate    iter   epoch  num   | valid_loss               | train_loss               | batch_loss               |  time          \n-------------------------------------------------------------------------------------------------------------------------------\n0.0000   0.0 k    0.0  0.0 m | 1.989   0.17 0.45   0.69 0.00   0.68 | 0.000   0.00 0.00   0.00 0.00   0.00 | 0.000   0.00 0.00   0.00 0.00   0.00 |  0 hr 00 min\n0.0100   0.1 k    0.7  0.0 m | 1.172   0.05 0.13   0.59 0.06   0.34 | 1.143   0.07 0.15   0.47 0.11   0.35 | 1.066   0.07 0.12   0.42 0.07   0.39 |  0 hr 04 min  99,99,\n0.0100   0.1 k    0.9  0.0 m | 1.172   0.05 0.13   0.59 0.06   0.34 | 1.048   0.07 0.13   0.46 0.07   0.31 | 1.167   0.10 0.13   0.48 0.11   0.35 |  0 hr 06 min  122,122\n\n,THCudaCheck FAIL file=/home/john/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory\nTraceback (most recent call last):\n  File \"train_mask_rcnn_net_3.py\", line 530, in &lt;module&gt;\n    run_train()\n  File \"train_mask_rcnn_net_3.py\", line 414, in run_train\n    net.forward( inputs, truth_boxes, truth_labels, truth_instances )\n  File \"/home/john/mask-rcnn-ver-12/net/se_resnext50_mask_rcnn_2crop/model.py\", line 459, in forward\n    self.mask_logits = data_parallel(self.mask_head, mask_crops)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py\", line 142, in data_parallel\n    return module(*inputs[0], **module_kwargs[0])\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n    result = self.forward(*input, **kwargs)\n  File \"/home/john/mask-rcnn-ver-12/net/se_resnext50_mask_rcnn_2crop/model.py\", line 388, in forward\n    x = self.up(x)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n    result = self.forward(*input, **kwargs)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py\", line 663, in forward\n    output_padding, self.groups, self.dilation)\nRuntimeError: cuda runtime error (2) : out of memory at /home/john/pytorch/aten/src/THC/generic/THCStorage.cu:58\n</code></pre>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310938,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-09T04:31:10.353000",
          "content": "<p>you can reduce batch size for rpn,rcnn, mask at the configuration.py. I think you don't have to retrain. you can just put in your old trained model. should be compatible. the main objective is to test the ensemble part</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 310941,
          "author_name": "Tejpal Virdi",
          "author_url": "",
          "post_date": "2018-04-09T04:35:36.273000",
          "content": "<p>I'm having trouble making it compatible to CUDA 9.0. What should I change in the config file? I tried changing it in the maker file but it didn't update. Thanks!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310944,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T04:41:25.210000",
          "content": "<p>For your information, I have changed many setting to solve it but it have not success. Only one way to solve it is that reduce <code>WIDTH, HEIGHT = 192,192</code> from <code>WIDTH, HEIGHT = 256,256</code></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310947,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-09T04:43:59.453000",
          "content": "<p>you can switch back to 256,256 later where the model become trained and there will be less rcnn detection per image. Maybe in the initial training, there are many random detection per image and memory requirement is larger.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 310951,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T04:59:24.963000",
          "content": "<p>Great idea. I have run it and it worked well. I have not changed back to <code>256x256</code>. This is way that I ran Heng's code, hope it helps someone </p>\n\n<ol>\n<li>Rebuild his .so as readme.pdf in his thread [0.463]</li>\n<li>Train the model <code>train_mask_rcnn_net_3.py</code></li>\n<li>Run the <code>predict.py</code></li>\n<li>Run the <code>emsemble.py</code></li>\n</ol>\n\n<p>Let me know if I was missing any step. Nice code and good job @Heng</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 310957,
          "author_name": "William Green",
          "author_url": "",
          "post_date": "2018-04-09T05:08:12.093000",
          "content": "<p>@John1231983 </p>\n\n<p>I tried rebuilding the .so files and unsuccessful.  I tried with both cuda 9.0 (telsa v100 and 8.0 (K80) on floydhub. I also tried updating the makefile and setup.py file as well. </p>\n\n<p>I keep getting the following error: </p>\n\n<pre><code>---------------------------------------------------------------------------\n  ImportError                               Traceback (most recent call last)\n  &lt;ipython-input-1-0307244cc3a6&gt; in &lt;module&gt;()\n  6 from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\n  7 from net.lib.box.nms.torch_nms import torch_nms\n  ----&gt; 8 from net.lib.box.nms.gpu_nms import gpu_nms\n  9 from net.lib.box.nms.cython_nms import cython_nms\n 10 \n\n  ImportError: libcudart.so.9.1: cannot open shared object file: No such file or directory\n</code></pre>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310958,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T05:10:30.007000",
          "content": "<p>Sorry, I am using cuda 9.1. I have tried cuda 9.0 and got your error. After that,  I upgrade my gpu to 9.1, it takes short time than find the solution :)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310962,
          "author_name": "William Green",
          "author_url": "",
          "post_date": "2018-04-09T05:26:47.027000",
          "content": "<p>@John Thanks you </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311208,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-09T16:05:48.910000",
          "content": "<p>predict.py</p>\n\n<p>emsemble.py (run_ensemble_box)</p>\n\n<p>predict_mask_only</p>\n\n<p>emsemble.py (run_ensemble_mask)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311218,
          "author_name": "leodestiny",
          "author_url": "",
          "post_date": "2018-04-09T16:20:14.137000",
          "content": "<p>I run your code , but get some errors:\nbuild/mask-rcnn-ver-12/net/layer/roi_align_pool_tf/module.py\", line 40, in forward\n    ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1)\nValueError: too many values to unpack (expected 5)\nhow should I do?\nthanks very much!</p>\n\n<p>I use wrong .so files.... I have fixed it.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 308842,
      "author_name": "Ildoo Kim",
      "author_url": "",
      "post_date": "2018-04-04T07:23:29.477000",
      "content": "<p>So.. Using these apporaches, how much did you improve your LB?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 309439,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-05T10:35:15.013000",
          "content": "<p>for example:</p>\n\n<p>single se-resnext-50, 53 gray test images: LB=0.43</p>\n\n<p>single se-resnext-50 + ensemble on test augmentation, 53 gray test images: LB=0.46 to 0.47</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 309444,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-05T10:53:24.163000",
          "content": "<p>Amazing. I have tested with flipud + fliplr+orginal but hurt performance (0.498 to 0.48). What kind of tech. in test agumentation do you use? Increase 3-4% only from test agu.?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309710,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2018-04-05T20:55:34.110000",
          "content": "<p>I agree with Heng, if we have large overlaps with time augment results then prediction looks confident.\nHere is what I get:</p>\n\n<ul>\n<li>Original LB=0.482</li>\n<li>FlipLR LB=0.484 </li>\n<li>FlipUD LB=0.482</li>\n</ul>\n\n<p>Then by ensembling these results:</p>\n\n<ul>\n<li>With NMS ensembling, I get LB 0.487</li>\n<li>With IoU ensembling, I get LB 0.50</li>\n</ul>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309814,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-06T02:41:00.063000",
          "content": "<p>actually we can train a classifier to predict the best form of ensemble method. it can be different ensemble method for different characteristics of the input(e.g. overlap area, confident score, etc). This is also where external data is useful. And we do not need accurate labels on this external label, because we can set the learning problem as:</p>\n\n<p>y = esemble_of_xs = Model(x1,x2,x3 ... )</p>\n\n<p>learning objective:  distance(y,y_hat) is better than distance(y1,y_hat),  distance(y2,y_hat), .... </p>\n\n<p>learning objective is NOT minimize distance(y,y_hat) to 0, since y_hat is unknown.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 309815,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-06T02:43:53.020000",
          "content": "<blockquote>\n  <blockquote>\n    <p>With IoU ensembling, I get LB 0.50</p>\n  </blockquote>\n</blockquote>\n\n<p>imagine if your retrain with pseudo labels that gives \"LB 0.50\", and repeat the ensemble process. Maybe results will improve.</p>\n\n<p>Since without ensemble, results is \"LB 0.48\". pseudo labels that gives \"LB 0.50\" can product new supervisory signals to improve each original model.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310016,
          "author_name": "Ildoo Kim",
          "author_url": "",
          "post_date": "2018-04-06T12:03:25.317000",
          "content": "<p>@MPWare Team : What is IOU Ensemble? </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 311065,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-09T09:49:51.990000",
      "content": "<p>added scale_filp_transpose argument(). you now can have up to 26 test augmentation per image</p>",
      "votes": 2,
      "replies": [
        {
          "id": 311144,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T13:43:03.253000",
          "content": "<p>@Heng. After run the <code>emsemble.py</code>, we have</p>\n\n<p>overall = overall/num_ensemble</p>\n\n<p>How to convert it to mask (int value instead of prob.) to run rle encoding?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311192,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-09T15:15:56.013000",
          "content": "<p>This is the tricky part. To get best results, there are several post processing which i can only reveal after the competition.</p>\n\n<p>For a simple solution:</p>\n\n<ul>\n<li><p>there is instance = num_augments * num_masks_in_one_image * image_height * image_width.\naverage over all augments, then you have average mask per instance object. you can then threshold at 0.5.</p></li>\n<li><p>try to score this average instant mask. if the variance is large (different augments disagree), score is low</p></li>\n<li><p>apply mask mns on these average instance mask, using the score. This is to decide which mask will be on top of others.</p></li>\n<li><p>further post process to remove small fragments, fill holes, etc</p></li>\n</ul>\n\n<p>Or you can actually trained a mask-rcnn using \"overall+ original_image\" as input</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 311450,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-10T04:55:43.173000",
          "content": "<p>Thanks. I used your previous ensemble code and it hurt my performance. It reduce to 0.44 from 0.46 (without ensemble). I guess ensemble make many isolate regions and makes contour between nucleic closer</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 308940,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-04T12:02:19.220000",
      "content": "<p>this is how it looks when you do test-time-augmentation (e.g. flip test images).</p>\n\n<p>Intensity = number of time detected</p>\n\n<p>The trick to get good results is how to ensemble these results!</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308940/8935/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\" alt=\"enter image description here\">   <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308940/8933/4be73d68f433869188fe5e7f09c7f681ed51003da6aa5d19ce368726d8e271ee.png\" alt=\"enter image description here\">  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/308940/8932/7f4cbe0b36b5d09466476a7d4e01f4f976c67872d549f4ff47b3e1e3a2b403af.png\" alt=\"enter image description here\"></p>",
      "votes": 2,
      "replies": [
        {
          "id": 309443,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-05T10:52:01.740000",
          "content": "<p>you note that if there are multiple overlap from different augmentations, the prediction is confident and mostly correct.</p>\n\n<p>we can also identify non-confident regions (few overlap).</p>\n\n<p>This give a clue of how to do automatic pseudo-labeling:</p>\n\n<ul>\n<li><p>confident region : label as mask</p></li>\n<li><p>non-confident region: just color with background color (don't care region during learning)</p></li>\n</ul>\n\n<p>human error is accuracy is about LB 0.55. So if your algorithm has accuracy greater then this, you can try pseudo-labeling</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 308819,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "2018-04-04T06:41:03.890000",
      "content": "<p>Great thank you for your strategy. I am finding the way to do the step 1 and your sentence \"cluster subject based on IoU\" give me many motivation and save my life :). The second point I did not try because it likes semi supervised learning</p>\n\n<p>For more clear to other people, I will explain more about point 1 of Heng, let me know if I was misunderstand\nAssume we have two outputs from two models for an image. The first output results in 5 masks, while the second results in 1 mask (background and mask if using Unet, FCN...) or 4 masks if using other checkpoints of masrcnn.  The first task of the first points is clustering masks into same cluster by measuring IoU. </p>\n\n<pre><code> for m1 in mask1\n\n      for m2 in mask2\n\n           max_IoU=0\n\n          if (max_IoU &lt; compute_IoU(m1,m2))\n\n              max_IoU=compute_IoU(m1,m2)\n\n      select m1,m2 that have max_IoU in same group\n</code></pre>\n\n<p>After first step, we can group masks for each output together to computer average, majorvoting. ....Let correct if I was wrong. Thanks</p>\n\n<p>Edit: Question: Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. Thus, we have to do extra step to separate cells by using finding contour, wastershed....which kind of method did you use to separate cells?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 308821,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-04T06:46:17.203000",
          "content": "<p>it is correct. Just one thing to note. I read that there are about 3000 stage2 test images, so the clustering algorithm should be efficient. Example is to measure box iou first, before measure mask iou, e.g.</p>\n\n<pre><code>def mask_iou(m1,m2):\n    if iou of box(m1) box(m2) == zero, return 0\n    else ... further processing ....\n</code></pre>\n\n<p>for \"Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. \", we label using our predictor first, then correct by hand</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 309360,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-05T06:11:32.293000",
          "content": "<p>Hello Heng, I did it and it looks that combine testing augmentation may not good for case of fliplr and flipud. I will test with rotation some angle. For the BBBC06 dataset, I found many raw images (likes BBBC006_v1_images_z_18.zip, BBBC006_v1_images_z_19.zip...), which should I selecte for raw image? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 309549,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-05T14:58:50.493000",
          "content": "<p>BBBC006_v1_images_z_16.zip</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 313962,
      "author_name": "Daler Dzhabborov",
      "author_url": "",
      "post_date": "2018-04-14T07:12:43.863000",
      "content": "<p>Great!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 313918,
      "author_name": "NullPoint",
      "author_url": "",
      "post_date": "2018-04-14T04:24:46.657000",
      "content": "<p>Hi, Heng, where is your function of emsemble.py (run_ensemble_mask)?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 313568,
      "author_name": "AviWolfson",
      "author_url": "",
      "post_date": "2018-04-13T13:53:40.353000",
      "content": "<p>in stage2 I have some images that I don't predict anything and there are empty. how do I change the ensembling code to support that? </p>",
      "votes": 0,
      "replies": [
        {
          "id": 313582,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-13T14:05:47.543000",
          "content": "<p>three ways:\n1) add a check. if sum of instance ==0 continue</p>\n\n<p>2) try and catch. if there is error move to next image</p>\n\n<p>3) this is without code modification, but i think 1 and 2 does not violate the rules and are better solution. The third solution is to create an input image file list. if there is error remove it from the input list and put in a reject list. Then break the list into another new list, starting from the next image file. Kind of stupid solution, but this is one that stricting does not change code.</p>\n\n<p>Since we are data scientist and we know what is \"model change\" verus \"code change\". I think if you don't check your model, you can change your code. kaggle will check your code if they need to and you can explain to them.</p>\n\n<p>Without code modification, the code cannot run.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 311678,
      "author_name": "NullPoint",
      "author_url": "",
      "post_date": "2018-04-10T13:50:56.503000",
      "content": "<p>AttributeError: 'MaskRcnnNet' object has no attribute 'mask_instances'</p>",
      "votes": 0,
      "replies": [
        {
          "id": 311680,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-10T13:53:33.033000",
          "content": "<p>this is becuase there is no detection.\n(see the forward function)\nmake a dummy mask_instances = zero arrays ( like mask = zero arrays).</p>\n\n<p>better still, your code should len(net.detections)&gt;0 before calling net.mask_instances</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 311582,
      "author_name": "AviWolfson",
      "author_url": "",
      "post_date": "2018-04-10T09:39:05.430000",
      "content": "<p>how much did u got before doing all those manipulations? </p>\n\n<p>our LB only from the maskrcnn is 0.532</p>",
      "votes": 0,
      "replies": [
        {
          "id": 311604,
          "author_name": "yyll008",
          "author_url": "",
          "post_date": "2018-04-10T11:00:58.127000",
          "content": "<p>I get lb0.500 base on Heng CherKeng's maskrcnn code.\nHave not do ensemble yet. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311613,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-10T11:19:50.420000",
          "content": "<p>Hi Avi, how do you train color image? I use original color image and H&amp;E and mosaic but only got 0.061 LB ( sometime got 0.063)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311750,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2018-04-10T15:53:52.300000",
          "content": "<p>How many iterations did you need to reach 0.500 or 0.53? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311846,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-10T19:05:25.210000",
          "content": "<p>@MPWare: 0.53 is from matterport code. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311969,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2018-04-11T02:07:32.787000",
          "content": "<p>my single model without external data don't do well, about 0.45+0.06 (gray + color).</p>\n\n<p>With external data, ensemble and post processing, i can have a gain of +0.05 to 0.06</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 312001,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-11T03:32:01.517000",
          "content": "<p>I got 0.063-0.066 on color.  But I trained the color without preprocessing (using original color information)+ mosaic. I have two models: 1 trained on gray images, and 1 trained on color images +H&amp;E. Have you merge color and gray into one set when train the color? Someone informed that it has no benefit</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 311537,
      "author_name": "yyll008",
      "author_url": "",
      "post_date": "2018-04-10T08:06:02.497000",
      "content": "<p>What is IOU Ensemble? \nHow can I make submissions from ensemble folders \".npy\" or overlays \"/%s.png\"?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 311486,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-10T06:19:29.323000",
      "content": "<p>this is how to do boundary detection with mask rcnn</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311486/9060/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311486/9061/0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/311486/9062/0ed3555a4bd48046d3b63d8baf03a5aa97e523aa483aaa07459e7afa39fb96c6.png\" alt=\"enter image description here\"></p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 311246,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "2018-04-09T17:24:24.763000",
      "content": "<p>I have an error  after run 20k iterations. have you see it too?</p>\n\n<pre><code>File \"train_mask_rcnn_net_3.py\", line 530, in &lt;module&gt;\n    run_train()\n  File \"train_mask_rcnn_net_3.py\", line 471, in run_train\n    all7 = draw_mask_metric(image, mask, truth_mask)\n  File \"/home/john/mask-rcnn-ver-12/draw.py\", line 315, in draw_mask_metric\n    overlap = np.max(iou,1)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py\", line 2320, in amax\n    out=out, **kwargs)\n  File \"/home/john/anaconda3/lib/python3.6/site-packages/numpy/core/_methods.py\", line 26, in _amax\n    return umr_maximum(a, axis, None, out, keepdims)\nValueError: zero-size array to reduction operation maximum which has no identity\n</code></pre>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 310761,
      "author_name": "Wassim Ghaddab",
      "author_url": "",
      "post_date": "2018-04-08T14:25:18.893000",
      "content": "<p>good job =)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 310645,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "2018-04-08T06:42:24.400000",
      "content": "<p>Hello Heng, may i ask you a question about LB in color set? Most of report here or other thread related to gray (based LB is 0.43+).  What is LB can we expect on 12 color set without using test augmentation? I remember that you said you achieved 0.05 LB on 12 color images, that is too low than me 0.063. My strategy is train all color images +H&amp;E aug in 1 network with initial checkpoint is from gray set. I did not apply any color normalization. Do you find any improvement when we using color normalization  like stain normalization?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 311003,
          "author_name": "Brian Luo",
          "author_url": "",
          "post_date": "2018-04-09T07:17:19.407000",
          "content": "<p>Hi, for color images, I train a model initial from gray set using 108 purple images. But I only got 0.044 for 12 color images, even worse than directly using model trained on 500 gray images. Can you give me some suggestion how to improve? For color images, I normalize them by reverting the color (i.e., 1-rgb2gray(img)). Thanks!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311005,
          "author_name": "Brian Luo",
          "author_url": "",
          "post_date": "2018-04-09T07:19:09.270000",
          "content": "<p>This model is overfitting a lot on these purple images.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311008,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-04-09T07:26:53.350000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311010,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "2018-04-09T07:29:54.537000",
          "content": "<p>I just used color information because converting to gray lost a lot of color information. I also use mosaic and H&amp;E external dataset. </p>\n\n<p>@Heng: What is LB on color image using your code?  </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 311045,
          "author_name": "Brian Luo",
          "author_url": "",
          "post_date": "2018-04-09T08:38:37.317000",
          "content": "<p>Thanks! How much improvement are you get from mosaic and external dataset?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 310630,
      "author_name": "chenchen",
      "author_url": "",
      "post_date": "2018-04-08T06:05:05.743000",
      "content": "<p>hi,thanks very much for your  'predict_arguments.py' code. I found in \"undo_test_augment_identity\" function ,it need 8 values--'i,x0,y0,x1,y1, score, label, aux' from rcnn_proposals ,but there are only 7 values in rcnn_nms function in your old codes. I did't know the 'aux' is ? Thank u for helping me!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 310303,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-04-07T03:50:13.823000",
      "content": "<p>other ways to ensemble which i have not tried are:</p>\n\n<ul>\n<li><p>transform mask rcnn results into distance transform map, then average the maps and find local peaks.</p></li>\n<li><p>ensemble of contours, centers, etc</p></li>\n</ul>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "308811": "After getting LB score of 0.48 to 0.50, here are 2 things i did. Strategy is simple. Collect enough errors, correct them and relearn.\n\n  ![enter image description here][1]\n  \n  ![enter image description here][2]\n  \n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308811/8929/Slide1.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308811/8930/Slide2.png",
    "309550": "ensmeble code",
    "310906": "compare ensemble and single  model results:\n\n\n   ![enter image description here][1]\n\n   ![enter image description here][2]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310906/9029/single.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310906/9030/ensemble.png",
    "310484": "A much better way to ensemble:\n\n   ![enter image description here][1]\n   ![enter image description here][2]\n   ![enter image description here][3]\n   ![enter image description here][4]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8957/Slide1.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8958/Slide2.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8959/Slide3.png\n  [4]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310484/8960/Slide4.png",
    "310430": "as requested, test time augmentation code",
    "311615": "btw, do we upload code by 11-Apr 23:59 hrs or 11-Apr 00:00 hrs. Now i am busy cleaning code!\n\n\"All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.\"\n\nSo i think it is  \"11-Apr 23:59 hrs\". ",
    "311096": "this movie gives you an idea how to post process:\n\nfrom left to right: \n\n-  box of average mask, \n-  average mask, \n-  each mask in ensemble",
    "311015": "i put more ensemble results are the google drive:\n\nhttps://drive.google.com/drive/folders/1Z9-eS5yr4jDPlCyyWqiV_3-m64lAPqRG\n\nI also put the trained model (00049000_model.pth) there.\n\nThe results are visually very attractive.\n  ![enter image description here][1]\n  ![enter image description here][2]\n  ![enter image description here][3]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311015/9036/a984e7fb886aa02e29d112766d3ce26a4f78eac540ce7bbdbd42af2761928f6d.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311015/9034/43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311015/9037/bdc789019cee8ddfae20d5f769299993b4b330b2d38d1218646cf89e77fbbd4d.png",
    "310917": "You do not need perfect labels to learn. But you do need many labels per sample if they are ambiguous. For this reason, this is why pseudo-labels from ensemble classifiers should work. This is distilling knowledge from ensemble.\nThe problem is to control the percentage of label noise.  \n\nThis is a neat trick that you can use in many kaggle competition. You now have a way to \"fit test data\"\n\n   ![enter image description here][1]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/310917/9032/self-learning.png",
    "308815": "some question you may ask:\n\n1. what if given an external image, there are some nuclei that i know how to label and some that i don't know?\n\n- draw a box to cover the nuclei that you don't know how to label and filled with black (or back ground color).\n\n\n2. Does external data helps?\n\n- Yes, only if you annotate them correctly\n\n",
    "315320": "here is the ensemble code  uploaded (fixed for zero detection) for submission and making results for stage2",
    "311692": "TTA trick. if it works in test, do it in train augmentation too.\n\nthere is a bug in my scale_to_factor() code. The is the corrected one:\n\n    def scale_to_factor(image, scale_x, scale_y, factor=16):\n    height,width = image.shape[:2]\n    h = math.ceil(scale_y*height/factor)*factor\n    w = math.ceil(scale_x*width/factor)*factor\n    image = cv2.resize(image,(w,h))\n    return image\n\n\n![enter image description here][1]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311692/9071/test_time_trick.png",
    "311611": "Overlapping boundary. This is where mask rcnn is better than unet. However, this is not required in the challenge. \n\n![enter image description here][1]\n\n![enter image description here][2]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311611/9066/overlapping_boundary.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311611/9067/overlapping_boundary1.png",
    "310636": "@chenchen\n\n\" I did't know the 'aux' is ?\" as attached",
    "308842": "So.. Using these apporaches, how much did you improve your LB?",
    "311065": "added scale_filp_transpose argument(). you now can have up to 26 test augmentation per image\n\n\n",
    "308940": "this is how it looks when you do test-time-augmentation (e.g. flip test images).\n\nIntensity = number of time detected\n\nThe trick to get good results is how to ensemble these results!\n\n\n  ![enter image description here][1]   ![enter image description here][2]  ![enter image description here][3]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308940/8935/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308940/8933/4be73d68f433869188fe5e7f09c7f681ed51003da6aa5d19ce368726d8e271ee.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/308940/8932/7f4cbe0b36b5d09466476a7d4e01f4f976c67872d549f4ff47b3e1e3a2b403af.png",
    "308819": "Great thank you for your strategy. I am finding the way to do the step 1 and your sentence \"cluster subject based on IoU\" give me many motivation and save my life :). The second point I did not try because it likes semi supervised learning\n\nFor more clear to other people, I will explain more about point 1 of Heng, let me know if I was misunderstand\nAssume we have two outputs from two models for an image. The first output results in 5 masks, while the second results in 1 mask (background and mask if using Unet, FCN...) or 4 masks if using other checkpoints of masrcnn.  The first task of the first points is clustering masks into same cluster by measuring IoU. \n\n     for m1 in mask1\n\n          for m2 in mask2\n\n               max_IoU=0\n\n              if (max_IoU &lt; compute_IoU(m1,m2))\n\n                  max_IoU=compute_IoU(m1,m2)\n\n          select m1,m2 that have max_IoU in same group\n\nAfter first step, we can group masks for each output together to computer average, majorvoting. ....Let correct if I was wrong. Thanks\n\nEdit: Question: Almost avaiable dataset are using for semantic segmentation, not for instance segmentation. Thus, we have to do extra step to separate cells by using finding contour, wastershed....which kind of method did you use to separate cells?",
    "313962": "Great!",
    "313918": "Hi, Heng, where is your function of emsemble.py (run_ensemble_mask)?",
    "313568": "in stage2 I have some images that I don't predict anything and there are empty. how do I change the ensembling code to support that? ",
    "311678": "AttributeError: 'MaskRcnnNet' object has no attribute 'mask_instances'",
    "311582": "how much did u got before doing all those manipulations? \n\nour LB only from the maskrcnn is 0.532",
    "311537": "What is IOU Ensemble? \nHow can I make submissions from ensemble folders \".npy\" or overlays \"/%s.png\"?",
    "311486": "this is how to do boundary detection with mask rcnn\n\n\n  ![enter image description here][1]\n\n  ![enter image description here][2]\n\n  ![enter image description here][3]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311486/9060/1962d0c5faf3e85cda80e0578e0cb7aca50826d781620e5c1c4cc586bc69f81a.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311486/9061/0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac.png\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/311486/9062/0ed3555a4bd48046d3b63d8baf03a5aa97e523aa483aaa07459e7afa39fb96c6.png",
    "311246": "I have an error  after run 20k iterations. have you see it too?\n    \n    File \"train_mask_rcnn_net_3.py\", line 530, in ",
    "310761": "good job =)",
    "310645": "Hello Heng, may i ask you a question about LB in color set? Most of report here or other thread related to gray (based LB is 0.43+).  What is LB can we expect on 12 color set without using test augmentation? I remember that you said you achieved 0.05 LB on 12 color images, that is too low than me 0.063. My strategy is train all color images +H&amp;E aug in 1 network with initial checkpoint is from gray set. I did not apply any color normalization. Do you find any improvement when we using color normalization  like stain normalization?",
    "310630": "hi,thanks very much for your  'predict_arguments.py' code. I found in \"undo_test_augment_identity\" function ,it need 8 values--'i,x0,y0,x1,y1, score, label, aux' from rcnn_proposals ,but there are only 7 values in rcnn_nms function in your old codes. I did't know the 'aux' is ? Thank u for helping me!",
    "310303": "other ways to ensemble which i have not tried are:\n\n-  transform mask rcnn results into distance transform map, then average the maps and find local peaks.\n\n- ensemble of contours, centers, etc"
  }
}