contour of instance segmentation from above
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/282662/8563/example_results.png\")
This thread serve as a reference for those who are writing mask-rcnn from stratch.
\n\nIt is part of the work from : https://www.kaggle.com/c/data-science-bowl-2018/discussion/47686. (actually, it is [version.0b] Feb-20)
\n\nPlease feel free to contribute (e.g. bug report/fix, experimental results, documentation, parameter tunning, etc)
\n\nI will update with more documentation when i am free later. LB 0.463 as for now, without pre or post processing.
\n\nCurrently, this is the results you can get:
\n\nfrom left to right:
\n\ncontour of instance segmentation from above
\n\n
code download: \nhttps://drive.google.com/drive/folders/1EtBdlPK3BVahHBZtB103QQxNUdWaJ6di
\n\nlatest version:
\n\nfolder 20180306, see readme.pdf
submission code and sample image/csv results are provided.
sample model and training log (train parameters and loss at each iteration)
This thread serve as a reference for those who are writing mask-rcnn from stratch.
\n\nIt is part of the work from : https://www.kaggle.com/c/data-science-bowl-2018/discussion/47686. (actually, it is [version.0b] Feb-20)
\n\nPlease feel free to contribute (e.g. bug report/fix, experimental results, documentation, parameter tunning, etc)
\n\nI will update with more documentation when i am free later. LB 0.463 as for now, without pre or post processing.
\n\nCurrently, this is the results you can get:
\n\nfrom left to right:
\n\ncontour of instance segmentation from above
\n\n
code download: \nhttps://drive.google.com/drive/folders/1EtBdlPK3BVahHBZtB103QQxNUdWaJ6di
\n\nlatest version:
\n\nfolder 20180306, see readme.pdf
submission code and sample image/csv results are provided.
sample model and training log (train parameters and loss at each iteration)
Thanks again Heng for providing great baseline!\nbtw, you're famous now (spotted somewhere in the russian internet): \n ![\"Spotted](\"https://files.slack.com/files-pri/T040HKJE3-F9A4WTAD6/screenshot_20180217-222506.png?pub_secret=a331e9f542\")
For reference, this meme is not really known outside Russia, so here is my attempt at translating the original: https://i.imgur.com/gClRuVH.jpg
", "rawMarkdown": "For reference, this meme is not really known outside Russia, so here is my attempt at translating the original: https://i.imgur.com/gClRuVH.jpg", "votes": 28 }, { "id": 284655, "postDate": "2018-02-17T21:18:07.757Z", "content": "yay
", "rawMarkdown": "yay" } ] }, { "id": 284072, "postDate": "2018-02-16T15:12:08.790Z", "content": "Github version will be much better.
", "rawMarkdown": "Github version will be much better.", "votes": 16 }, { "id": 284217, "postDate": "2018-02-16T19:07:24.817Z", "content": "suggested way to train mask rcnn:
\n\nThere is no training, just let prediction = target ground truth for each of the rpn, rcnn and mask heads. Then measure the metric accuracy. We expect about 0.95 to 1.00 accuracy. There are several cropping operations and down-sizing/up-sizing of the mask etc. When we define a bounding box as x0,y0,x1,y1, are these coordinates inclusive or exclusive? should i predict instance mask at 14x14, 16x16 or 28x28? These may affects results. This step verify that such operations do not result in significant drop in accuracy.
Train with small set, say 32 images. valid with the same 32 train images and measure the metric accuracy. There is no generalization problem and we expect about 0.85 to 0.95 accuracy. Because i am using cropped patches, we do not see all samples at one iteration during training. Use this step to check the loss of different rpn, rcnn and mask heads. Are they balanced? What loss is required for a metric of 0.95? Can your network overfit you data?
Now this is the actual training! Use the loss values in step.2 above to decide if there is generalization problem now. if so, will dropout helps? how about data augmentation?
Task loss balancing. Training should proceed such that rpn loss decrease first, then followed by rcnn and then mask. This is because rcnn depends on rpn proposals and mask depends on rcnn proposals.
Class loss balancing. Especially for rcnn part, you may need to use OHEM or focal loss. for rpn part you may need focal loss.
in summary, ask this question: \"what is the upper bound accuracy of your pipeline?\"
", "rawMarkdown": "suggested way to train mask rcnn:\n\n1. There is no training, just let prediction = target ground truth for each of the rpn, rcnn and mask heads. Then measure the metric accuracy. We expect about 0.95 to 1.00 accuracy. There are several cropping operations and down-sizing/up-sizing of the mask etc. When we define a bounding box as x0,y0,x1,y1, are these coordinates inclusive or exclusive? should i predict instance mask at 14x14, 16x16 or 28x28? These may affects results. This step verify that such operations do not result in significant drop in accuracy.\n\n2. Train with small set, say 32 images. valid with the same 32 train images and measure the metric accuracy. There is no generalization problem and we expect about 0.85 to 0.95 accuracy. Because i am using cropped patches, we do not see all samples at one iteration during training. Use this step to check the loss of different rpn, rcnn and mask heads. Are they balanced? What loss is required for a metric of 0.95? Can your network overfit you data?\n\n3. Now this is the actual training! Use the loss values in step.2 above to decide if there is generalization problem now. if so, will dropout helps? how about data augmentation? \n\n4. Task loss balancing. Training should proceed such that rpn loss decrease first, then followed by rcnn and then mask. This is because rcnn depends on rpn proposals and mask depends on rcnn proposals. \n\n5. Class loss balancing. Especially for rcnn part, you may need to use OHEM or focal loss. for rpn part you may need focal loss.\n\nin summary, ask this question: \"what is the upper bound accuracy of your pipeline?\"", "votes": 12, "replies": [ { "id": 284226, "postDate": "2018-02-16T19:17:27.963Z", "content": "Hmm... So you are only training the heads not the backbone. And for the backbone you are using resnet50, right?
\n\nI like the idea of training on a smaller sample first. That way I don't have to wait for hours to get an idea of how the model is doing.
\n\nAs for the augmentation, I am only using random flips left to right and up to down.
", "rawMarkdown": "Hmm... So you are only training the heads not the backbone. And for the backbone you are using resnet50, right?\n\nI like the idea of training on a smaller sample first. That way I don't have to wait for hours to get an idea of how the model is doing.\n\nAs for the augmentation, I am only using random flips left to right and up to down. ", "votes": 1 }, { "id": 284227, "postDate": "2018-02-16T19:19:02.437Z", "content": "both the backbone and heads are trained.
", "rawMarkdown": "both the backbone and heads are trained.", "votes": 3 }, { "id": 287931, "postDate": "2018-02-26T08:56:49.493Z", "content": "Hi Heng, this is one of the most important tip i heard, the training strategy as i don't have much GPU resource, instead of waiting for a day, this is really speed up things. When train with small set (step 2), will network suffer from overfitting? The reason i am asking is that, i am using another Mask RCNN implementation (https://github.com/matterport/Mask_RCNN) and trying to train entire backbone network, even if using full training set, network starts to overfit after 40 epoch (only flip left/right argumentation), loss continuously decreasing while val_loss starts to increase. So i am afraid just taking 32 images will be the same pb, or did i miss anything?
", "rawMarkdown": "Hi Heng, this is one of the most important tip i heard, the training strategy as i don't have much GPU resource, instead of waiting for a day, this is really speed up things. When train with small set (step 2), will network suffer from overfitting? The reason i am asking is that, i am using another Mask RCNN implementation (https://github.com/matterport/Mask_RCNN) and trying to train entire backbone network, even if using full training set, network starts to overfit after 40 epoch (only flip left/right argumentation), loss continuously decreasing while val_loss starts to increase. So i am afraid just taking 32 images will be the same pb, or did i miss anything? " } ] }, { "id": 286555, "postDate": "2018-02-22T11:33:47.773Z", "content": "for cython_mns.so and gpu_mns.so, please refer to attachment here. (you can ignore the rest of unrelated files here. i will release dummy-14.7 later)
", "rawMarkdown": "for cython_mns.so and gpu_mns.so, please refer to attachment here. (you can ignore the rest of unrelated files here. i will release dummy-14.7 later)", "votes": 7, "replies": [ { "id": 286872, "postDate": "2018-02-23T00:07:53.587Z", "content": "thanks for the code. It has been a great learning experience. Where can i find resize_to_fractor or pad_to_factor? Also i run into this strange error when i am running the test data set about illegal memory access at THCTensorcopy.c. This seems to get trigerred at mask_rcnn_resnet_50_fpn_net.py/forward/index.append(ids[mask]. The origin seems to be \nrcnn_crops = self.crop(features, self.rpn.proposals). Any pointers would be much appreciated
", "rawMarkdown": "thanks for the code. It has been a great learning experience. Where can i find resize_to_fractor or pad_to_factor? Also i run into this strange error when i am running the test data set about illegal memory access at THCTensorcopy.c. This seems to get trigerred at mask_rcnn_resnet_50_fpn_net.py/forward/index.append(ids[mask]. The origin seems to be \nrcnn_crops = self.crop(features, self.rpn.proposals). Any pointers would be much appreciated", "votes": 1 }, { "id": 286951, "postDate": "2018-02-23T04:35:18.920Z", "content": "Thank you. I found the functions in transform.py
", "rawMarkdown": "Thank you. I found the functions in transform.py" }, { "id": 290089, "postDate": "2018-03-02T22:40:38.660Z", "content": "Hi Heng!
\n\nWhat hardware did you use to run the dummy-14.7 code with batch_size=2?
open source pytorch mask rcnn:
\n\nhttps://github.com/soeaver/Pytorch_Mask_RCNN
", "rawMarkdown": "open source pytorch mask rcnn:\n\nhttps://github.com/soeaver/Pytorch_Mask_RCNN", "votes": 8 }, { "id": 282671, "postDate": "2018-02-14T11:32:19.687Z", "content": "block diagrams
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/282671/8564/inference.png\")
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/282671/8565/back_prop.png\")
Great job! The original paper used parallel pipeline to predict detection class and mask in parallel. But you do not seem to follow it. Is it the result of your experiments?
", "rawMarkdown": "Great job! The original paper used parallel pipeline to predict detection class and mask in parallel. But you do not seem to follow it. Is it the result of your experiments?" }, { "id": 284908, "postDate": "2018-02-18T19:00:31.407Z", "content": "the paper mention:
\n\n\"Inference: At test time, the proposal number is 300 for the\nC4 backbone (as in [36]) and 1000 for FPN (as in [27]). We\nrun the box prediction branch on these proposals, followed\nby non-maximum suppression [14]. The mask branch is\nthen applied to the highest scoring 100 detection boxes. Although\nthis differs from the parallel computation used in\ntraining, it speeds up inference and improves accuracy (due\nto the use of fewer, more accurate RoIs)\"
", "rawMarkdown": "the paper mention:\n\n\"Inference: At test time, the proposal number is 300 for the\nC4 backbone (as in [36]) and 1000 for FPN (as in [27]). We\nrun the box prediction branch on these proposals, followed\nby non-maximum suppression [14]. The mask branch is\nthen applied to the highest scoring 100 detection boxes. Although\nthis differs from the parallel computation used in\ntraining, it speeds up inference and improves accuracy (due\nto the use of fewer, more accurate RoIs)\"", "votes": 2 }, { "id": 285023, "postDate": "2018-02-19T04:53:45.267Z", "content": "Oh sorry it's my mistake. If you don't mind, I still have another question. The resnet 50 use stride 2 conv and maxpooling on the first stage, and stride 1 on the second stage, which seems different from your implementation. Is that true?
", "rawMarkdown": "Oh sorry it's my mistake. If you don't mind, I still have another question. The resnet 50 use stride 2 conv and maxpooling on the first stage, and stride 1 on the second stage, which seems different from your implementation. Is that true?" }, { "id": 285025, "postDate": "2018-02-19T04:57:15.027Z", "content": "yes. this is to detect small nuclei of radius 4 pixel
", "rawMarkdown": "yes. this is to detect small nuclei of radius 4 pixel", "votes": 2 }, { "id": 302512, "postDate": "2018-03-24T08:04:29.940Z", "content": "Asian people always love diagrams! It's a hieroglyphics mind style.
", "rawMarkdown": "Asian people always love diagrams! It's a hieroglyphics mind style.", "votes": -3 } ] }, { "id": 306198, "postDate": "2018-03-30T02:49:16.570Z", "content": "Has anyone encountered ModuleNotFoundError: No module named 'net' ? I seemed to be stuck there. Any assistance would be greatly appreciated.
\n\n![https://i.imgur.com/gClRuVH.jpg](\"https://i.imgur.com/gClRuVH.jpg\"){kind=link}
---------------------------------------------------------------------------\n ModuleNotFoundError Traceback (most recent call last)\n <ipython-input-4-daf7d55d62fc> in <module>()\n 1 #from common import *\n ----> 2 from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\n 3 from net.lib.box.nms.torch_nms import torch_nms\n 4 from net.lib.box.nms.gpu_nms import gpu_nms\n 5 from net.lib.box.nms.cython_nms import cython_nms\n\n ModuleNotFoundError: No module named 'net'\nDid y get any error when execute \"from common import * \" ?
", "rawMarkdown": "Did y get any error when execute \"from common import * \" ? " }, { "id": 306211, "postDate": "2018-03-30T03:22:15.883Z", "content": "Yes, I get no module found.
", "rawMarkdown": "Yes, I get no module found. " }, { "id": 306218, "postDate": "2018-03-30T03:53:09.830Z", "content": "Add this two lines in the front of file:\nimport sys\nsys.path.append('../')
", "rawMarkdown": "Add this two lines in the front of file:\nimport sys\nsys.path.append('../')" }, { "id": 306224, "postDate": "2018-03-30T04:14:42.113Z", "content": "same error
", "rawMarkdown": "same error" }, { "id": 307166, "postDate": "2018-04-01T01:00:21.417Z", "content": "cd into the mask_rcnn_xxx.a folder\nthen execute\nexport PYTHONPATH=\"pwd\"
\n\nReplace \" with ` in pwd command.
", "rawMarkdown": "cd into the mask_rcnn_xxx.a folder\nthen execute\nexport PYTHONPATH=\"pwd\"\n\nReplace \" with ` in pwd command." }, { "id": 308892, "postDate": "2018-04-04T09:05:07.047Z", "content": "I got same error when install in the TitanXp. I changed PYTHONPATH and sys but it does not help. Any solution? Thanks
\n\n set random seed\n SEED=35202\n set cuda environment\n torch.__version__ = 0.4.0a0+b125033\n torch.version.cuda = 9.1.85\n torch.backends.cudnn.version() = 7005\n os['CUDA_VISIBLE_DEVICES'] = 0\n torch.cuda.device_count() = 1\n torch.cuda.current_device() = 0\n\nTraceback (most recent call last):\n File \"train_3.py\", line 6, in <module>\n from dataset.reader import *\n File \"/home/john/maskrcnn/build/mask-rcnn-resnet50-ver-01.a/dataset/reader.py\", line 11, in <module>\n from net.lib.box.process import *\n File \"/home/john/maskrcnn/build/mask-rcnn-resnet50-ver-01.a/net/lib/box/process.py\", line 7, in <module>\n from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\nModuleNotFoundError: No module named 'net.lib.box.overlap.cython_box_overlap'\nI haven't figured it out. How ever, 城璐 has a solution that allows you to add your cuda (gpu) version. \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/53588#308848
", "rawMarkdown": "I haven't figured it out. How ever, 城璐 has a solution that allows you to add your cuda (gpu) version. \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/53588#308848 " }, { "id": 309291, "postDate": "2018-04-05T01:21:15.173Z", "content": "I solved it. The reason is that after rebuild it, you have to copy *.so to root folder as the figure in the document of Heng. For example, cython_box_overlap.cpython-36m-x86_64-linux-gnu.so to overlap folder. That all
", "rawMarkdown": "I solved it. The reason is that after rebuild it, you have to copy *.so to root folder as the figure in the document of Heng. For example, cython_box_overlap.cpython-36m-x86_64-linux-gnu.so to overlap folder. That all" } ] }, { "id": 297598, "postDate": "2018-03-17T14:02:01.520Z", "content": "Hello Heng CherKeng, i have tried to run your code, but i got error when compiling. I use Pytorch 0.3.0 with Anaconda. After running python build.py, the error is below:\ncffi.error.VerificationError: CompileError: command 'gcc' failed with exit status 1
\n\nIs it because of the version of Pytorch? Is Pytorch 0.4.0 necessary?
", "rawMarkdown": "Hello Heng CherKeng, i have tried to run your code, but i got error when compiling. I use Pytorch 0.3.0 with Anaconda. After running python build.py, the error is below:\ncffi.error.VerificationError: CompileError: command 'gcc' failed with exit status 1\n\n Is it because of the version of Pytorch? Is Pytorch 0.4.0 necessary?", "votes": 3 }, { "id": 294171, "postDate": "2018-03-11T13:05:59.993Z", "content": "some tip:
\n\nchanging from resnet50 to se-resnext50 backbone improve the gray image score from 0.419 to 0.430.
\n\nnote:
\n\nif you want to get high LB score, you most improve the mask non-max suppression provided. Together with other improvement like class balancing, adjusting network structure, etc you can above 0.450 just with the gray images alone.
", "rawMarkdown": "some tip: \n\nchanging from resnet50 to se-resnext50 backbone improve the gray image score from 0.419 to 0.430.\n\n\nnote:\n\nif you want to get high LB score, you most improve the mask non-max suppression provided. Together with other improvement like class balancing, adjusting network structure, etc you can above 0.450 just with the gray images alone.", "votes": 3, "replies": [ { "id": 294381, "postDate": "2018-03-11T20:58:50.623Z", "content": "Thank you very much Heng! for all the ideas, suggestions and open-source code! \n I am running your latest code on a K80 on google cloud compute. I was able to compile everything and follow all the steps in your pdf :-). \n i am running the 1st experiment with grayscale images to make 00016500_model.pth.\nFor me 17k iterations took 16hrs with image size of 128-by-128:
\n\n17.0 k 544.0 0.3 m | 0.335 0.02 0.02 0.17 0.01 0.12 | 0.440 0.01 0.02 0.23 0.01 0.16 | 0.270 0.00 0.02 0.11 0.01 0.13 | 16 hr 07 min 16999,16999,
\n\nOne basic question: I saw in train_0.py that num_iters = 1000 *1000; does this mean 1000k iterations ? can i use any early stopping criteria or experiment with num_iters ?
\n\nThanks\nMehul
", "rawMarkdown": "Thank you very much Heng! for all the ideas, suggestions and open-source code! \n I am running your latest code on a K80 on google cloud compute. I was able to compile everything and follow all the steps in your pdf :-). \n i am running the 1st experiment with grayscale images to make 00016500_model.pth.\nFor me 17k iterations took 16hrs with image size of 128-by-128: \n\n\n 17.0 k 544.0 0.3 m | 0.335 0.02 0.02 0.17 0.01 0.12 | 0.440 0.01 0.02 0.23 0.01 0.16 | 0.270 0.00 0.02 0.11 0.01 0.13 | 16 hr 07 min 16999,16999,\n\nOne basic question: I saw in train_0.py that num_iters = 1000 *1000; does this mean 1000k iterations ? can i use any early stopping criteria or experiment with num_iters ? \n\nThanks\nMehul\n" }, { "id": 294391, "postDate": "2018-03-11T21:51:25.637Z", "content": "Hi Heng, I was trying to modify the network structure but getting some error when running submit.py, as I mentioned two days ago. I just changed some parameters in the model.py. It seems the net.detections has a size of 0. Do you have any comments on this issue?
", "rawMarkdown": "Hi Heng, I was trying to modify the network structure but getting some error when running submit.py, as I mentioned two days ago. I just changed some parameters in the model.py. It seems the net.detections has a size of 0. Do you have any comments on this issue?", "votes": 1 } ] }, { "id": 291540, "postDate": "2018-03-06T10:19:14.473Z", "content": "the latest version (mask-rcnn-resnet50-ver-01.a) is up! See folder 20180306 and readme.pdf
\n\nsubmission code and sample image/csv results are provided.
sample model and training log (train parameters and loss at each iteration) are provided
Hi. Could you tell me the gain do you achieve when you use focal loss? How do you compute the label weight for the loss. I currently use binary softmax loss. Thanks
", "rawMarkdown": "Hi. Could you tell me the gain do you achieve when you use focal loss? How do you compute the label weight for the loss. I currently use binary softmax loss. Thanks" }, { "id": 293455, "postDate": "2018-03-09T22:05:11.877Z", "content": "focal loss is used for single step (stage) object detection, one doesnt need to use focal loss if you you're using a two stage network like MaskRCNN which has the region proposal network (RPN) as 1st stage.
", "rawMarkdown": "focal loss is used for single step (stage) object detection, one doesnt need to use focal loss if you you're using a two stage network like MaskRCNN which has the region proposal network (RPN) as 1st stage." } ] }, { "id": 285163, "postDate": "2018-02-19T13:13:37.690Z", "content": "during training, you can ignore prediction boxes (for rpn, rcnn and mask) that are too small or those at the boundary of the image. you can try to modify my code (see make_rpn_target(), make_rcnn_target(), make_mask_target()). i.e. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\".
\n\nAttached is some modification i made for rpn. I will post the rest of complete solution for rcnn and mask later.
\n\nspend some time to write code to visualize results, e.g. mark the hit/miss true-positive/false-positive detections of rcnn. you can discover many improvement points. Draw the predicted and ground truth mask as well.
\n\nthis is the reason why i write my own mask rcnn code instead of using the open source. it is to have complete control and visualize results. I think after several observations of the results, you can think of ways to get LB in the range of 0.50 to 0.55 range.
", "rawMarkdown": "during training, you can ignore prediction boxes (for rpn, rcnn and mask) that are too small or those at the boundary of the image. you can try to modify my code (see make_rpn_target(), make_rcnn_target(), make_mask_target()). i.e. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\".\n\nAttached is some modification i made for rpn. I will post the rest of complete solution for rcnn and mask later.\n\nspend some time to write code to visualize results, e.g. mark the hit/miss true-positive/false-positive detections of rcnn. you can discover many improvement points. Draw the predicted and ground truth mask as well. \n\nthis is the reason why i write my own mask rcnn code instead of using the open source. it is to have complete control and visualize results. I think after several observations of the results, you can think of ways to get LB in the range of 0.50 to 0.55 range. ", "votes": 4, "replies": [ { "id": 285235, "postDate": "2018-02-19T16:59:12.333Z", "content": "Thanks for your wonderful work. Appreciate it! I really like your pytorch maskrcnn code. While I have some questions:\n1. I finished the training processing and find all the testing color images are failed, the results are poor, no bounding box got. So, I must training the model separate by color and gray? Is it enough for training samples.\n2. I submit the results, only using (masks = net.masks) results, while the performance only have 0.33. Anything wrong of my inference processing. I know four rgb image are failed, but this may lead score drop from 0.46 to 0.33?\n3. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\". So you mean I must delete tiny object in the training processing, it lead to great improvement? Interesting, why?
\n\nThanks for your code again. I really like the visualize results. And it has the same performance compared with caffe2 maskrcnn.
", "rawMarkdown": "Thanks for your wonderful work. Appreciate it! I really like your pytorch maskrcnn code. While I have some questions:\n1. I finished the training processing and find all the testing color images are failed, the results are poor, no bounding box got. So, I must training the model separate by color and gray? Is it enough for training samples.\n2. I submit the results, only using (masks = net.masks) results, while the performance only have 0.33. Anything wrong of my inference processing. I know four rgb image are failed, but this may lead score drop from 0.46 to 0.33?\n3. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\". So you mean I must delete tiny object in the training processing, it lead to great improvement? Interesting, why?\n\nThanks for your code again. I really like the visualize results. And it has the same performance compared with caffe2 maskrcnn.\n", "votes": 1 }, { "id": 285330, "postDate": "2018-02-20T00:29:44.770Z", "content": "i visualize the errors of rpn and rcnn. Many at are the boundary of the images and are ambiguous (i.e. if you were annotate a nucleus by hand, it is sometimes hard to decide if you want to mark it when say only 10% of it is visible). In fact in academia dataset like VOC or Caltech Pedestrian dataset, the ground truth box is given the attribute \"difficult\" or \"don't care\", which means that loss is not computed for these cases.
\n\nfrom the faster rcnn paper:
\n\n\"The anchor boxes that cross image boundaries need to be handled with care. During training, we\nignore all cross-boundary anchors so they do not contribute to the loss. For a typical 1000 × 600\nimage, there will be roughly 20k (≈ 60 × 40 × 9) anchors in total. With the cross-boundary anchors\nignored, there are about 6k anchors per image for training. If the boundary-crossing outliers are not\nignored in training, they introduce large, difficult to correct error terms in the objective, and training\ndoes not converge\"
", "rawMarkdown": "i visualize the errors of rpn and rcnn. Many at are the boundary of the images and are ambiguous (i.e. if you were annotate a nucleus by hand, it is sometimes hard to decide if you want to mark it when say only 10% of it is visible). In fact in academia dataset like VOC or Caltech Pedestrian dataset, the ground truth box is given the attribute \"difficult\" or \"don't care\", which means that loss is not computed for these cases.\n\n---\n\nfrom the faster rcnn paper:\n\n\"The anchor boxes that cross image boundaries need to be handled with care. During training, we\nignore all cross-boundary anchors so they do not contribute to the loss. For a typical 1000 × 600\nimage, there will be roughly 20k (≈ 60 × 40 × 9) anchors in total. With the cross-boundary anchors\nignored, there are about 6k anchors per image for training. If the boundary-crossing outliers are not\nignored in training, they introduce large, difficult to correct error terms in the objective, and training\ndoes not converge\"\n", "votes": 3 }, { "id": 293756, "postDate": "2018-03-10T14:21:36.440Z", "content": "Training with mixed gray and color images seems to be difficult, can you give me some suggestions on how to deal with different colors?
", "rawMarkdown": "Training with mixed gray and color images seems to be difficult, can you give me some suggestions on how to deal with different colors?" } ] }, { "id": 282783, "postDate": "2018-02-14T15:35:00.847Z", "content": "Cool. Mask R-CNN in PyTorch is still rare today. Placing the code on github will make it easier for people to follow and take part in.
", "rawMarkdown": "Cool. Mask R-CNN in PyTorch is still rare today. Placing the code on github will make it easier for people to follow and take part in.", "votes": 4 }, { "id": 309590, "postDate": "2018-04-05T16:29:34.247Z", "content": "You should try to understand and adjust the parameters in the configuration.py. E.g. score threshold, overlap threshold of rpn, rcnn and mask
\n\nYou should draw some visualization to see effect of the visualization. It is possible to achieve 0.45 for gray 53 images with se-resnext50 (and maybe also resnet50?)
", "rawMarkdown": "You should try to understand and adjust the parameters in the configuration.py. E.g. score threshold, overlap threshold of rpn, rcnn and mask\n\nYou should draw some visualization to see effect of the visualization. It is possible to achieve 0.45 for gray 53 images with se-resnext50 (and maybe also resnet50?)\n\n\n", "votes": 1 }, { "id": 309558, "postDate": "2018-04-05T15:05:26.763Z", "content": "Did anyone get this error msg when they tried to train their model
\n\nTHCudaCheck FAIL file=/home/uesu/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory\nTraceback (most recent call last):\n File \"train_0.py\", line 515, in <module>\n run_train()\n File \"train_0.py\", line 347, in run_train\n net( inputs, truth_boxes, truth_labels, truth_instances )\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n result = self.forward(*input, **kwargs)\n File \"/home/uesu/20180306/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/model.py\", line 400, in forward\n self.mask_logits = data_parallel(self.mask_head, mask_crops)\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py\", line 142, in data_parallel\n return module(*inputs[0], **module_kwargs[0])\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n result = self.forward(*input, **kwargs)\n File \"/home/uesu/20180306/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/model.py\", line 329, in forward\n x = F.relu(self.bn1(self.conv1(crops)),inplace=True)\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py\", line 621, in relu\n return torch.relu_(input)\nRuntimeError: cuda runtime error (2) : out of memory at /home/uesu/pytorch/aten/src/THC/generic/THCStorage.cu:58\nI am training this inside a nvidia enabled docker container on a Titian xp, based on what i have found this error is caused by the lack of shared memory. Does anybody know what is the shared memory used by the model and how to address this issue?
", "rawMarkdown": "Did anyone get this error msg when they tried to train their model \n\n THCudaCheck FAIL file=/home/uesu/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory\n Traceback (most recent call last):\n File \"train_0.py\", line 515, in ", "votes": 1, "replies": [ { "id": 309928, "postDate": "2018-04-06T07:58:38.410Z", "content": "Hi guys, managed to solve this issue this morning. I reduced the batch size to 8 and got the model to work. If anybody has any other suggestion you can post on this thread.
", "rawMarkdown": "Hi guys, managed to solve this issue this morning. I reduced the batch size to 8 and got the model to work. If anybody has any other suggestion you can post on this thread. ", "votes": 2 }, { "id": 361121, "postDate": "2018-07-23T21:51:31.297Z", "content": "I got the same problem in my code. And reducing batch size also solved the problem! \nDo you know why this happens?
", "rawMarkdown": "I got the same problem in my code. And reducing batch size also solved the problem! \nDo you know why this happens?" } ] }, { "id": 308711, "postDate": "2018-04-04T00:51:29.880Z", "content": "Hello Heng, I have a question with pretrain model in seresnetxt. I have successful to convert caffe model to pytorch model. It worked as expected in classification. Now, I apply it to your code, I found something:
\n\nThe original seresnetxt-50 uses max-pooling in layer_c0, but you did not use it. As my opinion, your crop size is 256, so if we use it, the resolution will divide by 4, thus we lost many information and some small mask will be come zeros. Hence, the original seresnetxt-50 may not good in this challenge. And you remove the pooling for the purpose. Am I right?
To use pretrain model, we have to insert the max-pooling in the layer_c0, I get error at the first epoch in evaluate. Have you success to use pretrain model?
\n\n\n valid_loss, valid_acc = evaluate(net, valid_loader)\n File \"train_2.py\", line 106, in evaluate\n net(inputs, truth_boxes, truth_labels, truth_instances )\nFile \"/home/john/heng_pytorch/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/layer/rpn_multi_target.py\", line 248, in make_one_rpn_target\n label_weight[bg_index] = num_fg/num_bg\n ZeroDivisionError: division by zero\n
i haven't make comparison, so i cannot say for sure. if you still one to use the original seresnetxt-50, you can always up scale your image for input
\n\ni think you have to change your anchor size, etc ... in the configuration file. currently it is assumed that the first feature layer (of the fpn) is 1/2 of input image, second feature is 1/4 , etc ... You have to change the configuration.py
", "rawMarkdown": "i haven't make comparison, so i cannot say for sure. if you still one to use the original seresnetxt-50, you can always up scale your image for input\n\ni think you have to change your anchor size, etc ... in the configuration file. currently it is assumed that the first feature layer (of the fpn) is 1/2 of input image, second feature is 1/4 , etc ... You have to change the configuration.py\n" } ] }, { "id": 302345, "postDate": "2018-03-23T23:44:04.630Z", "content": "While training with multi-gpu i get this error
\n\nloss.backward()\n File \"/home/ashwin/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py\", line 120, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph)\n File \"/home/ashwin/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py\", line 81, in backward\n variables, grad_variables, retain_graph, create_graph)\nRuntimeError: arguments are located on different GPUs at /home/ashwin/pytorch/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:231\nAny idea how to resolve this ?
", "rawMarkdown": "While training with multi-gpu i get this error\n\n loss.backward()\n File \"/home/ashwin/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py\", line 120, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph)\n File \"/home/ashwin/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py\", line 81, in backward\n variables, grad_variables, retain_graph, create_graph)\n RuntimeError: arguments are located on different GPUs at /home/ashwin/pytorch/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:231\n\nAny idea how to resolve this ?", "votes": 1, "replies": [ { "id": 302349, "postDate": "2018-03-24T00:03:33.443Z", "content": "try to comment out \" os.environ['CUDA_VISIBLE_DEVICES'] = 0 # '3,2,1,0' \".
", "rawMarkdown": "try to comment out \" os.environ['CUDA_VISIBLE_DEVICES'] = 0 # '3,2,1,0' \". " }, { "id": 302441, "postDate": "2018-03-24T03:48:23.960Z", "content": "I did comment this line but still getting the error
", "rawMarkdown": "I did comment this line but still getting the error" }, { "id": 302455, "postDate": "2018-03-24T04:43:42.730Z", "content": "I followed the instructions. I first saw the same error as you posted and it was gone after I commented out the line.
", "rawMarkdown": "I followed the instructions. I first saw the same error as you posted and it was gone after I commented out the line." }, { "id": 304884, "postDate": "2018-03-28T07:05:39.413Z", "content": "I've got the same problem. I found that we should use self.register_buffer('variable', variable), but still no idea where put it in the code.
Thanks, Heng, I am trying run your code. It is super fast than Keras based. I will try to convert my scheme to your code to run it. I have some questions:
\n\nHow do you separate purple and gray image into two subsets? Using K-means?
Does your code have any license? Because we have to submit the source code to Kaggle, thus, if I modified it and achieve top-5 performance (Hope!!!), Is it fine to use your code as the baseline and modify from it?
Why not using pre-train for backbone as resnet-50? In my Keras code, pre-train is much better. The training schedule likes: train on heads, then train all. You are train all in current code
Hi, I was trying to run Heng's code, too. But I encounter a problem when running 'train_0.py',
\n\nTraceback (most recent call last):\n File \"/data/yiwei/fun/Bowl/pytorch-maskrcnn/20180306/build/mask-rcnn-resnet50-ver-01.a/train_0.py\", line 515, in <module>\n run_train()\n File \"/data/yiwei/fun/Bowl/pytorch-maskrcnn/20180306/build/mask-rcnn-resnet50-ver-01.a/train_0.py\", line 351, in run_train\n loss.backward()\n File \"/usr/local/lib/python3.5/dist-packages/torch/autograd/variable.py\", line 120, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph)\n File \"/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py\", line 81, in backward\n variables, grad_variables, retain_graph, create_graph)\nRuntimeError: element 0 of variables does not require grad and does not have a grad_fn\nThe error points to the loss.backward(), do you have any ideas?\nThank you very much
Hi Heng! Thank you for your opensource contribution to Kaggle community. I have a little question - I started training using all data, not only grayscale but for color images I get very bad results after 12k iterations, for gray they are ok and close to yours. Can you pls help to undrstand what Iam missing? I use your last code.
", "rawMarkdown": "Hi Heng! Thank you for your opensource contribution to Kaggle community. I have a little question - I started training using all data, not only grayscale but for color images I get very bad results after 12k iterations, for gray they are ok and close to yours. Can you pls help to undrstand what Iam missing? I use your last code.", "votes": 1, "replies": [ { "id": 294170, "postDate": "2018-03-11T13:02:00.730Z", "content": "for color images, you need to treat it as a separate class.
\n\nif you treat all images as single class, then the common thing for all of them is only outline, which is very weak feature.
\n\nif you treat them as separate classes, you will have more common things within a class (e.g. the interior is also the same within the class)
\n\nI train two separate mask-rcnn, one on gray images and another on purple images (converted to gray scale). both perform reasonably well.
", "rawMarkdown": "for color images, you need to treat it as a separate class.\n\nif you treat all images as single class, then the common thing for all of them is only outline, which is very weak feature.\n\nif you treat them as separate classes, you will have more common things within a class (e.g. the interior is also the same within the class)\n\nI train two separate mask-rcnn, one on gray images and another on purple images (converted to gray scale). both perform reasonably well.\n\n", "votes": 4 }, { "id": 294440, "postDate": "2018-03-12T00:48:42.320Z", "content": "Hi Heng, how much improvement do you think when you add focal loss to the rpn class graph?
", "rawMarkdown": "Hi Heng, how much improvement do you think when you add focal loss to the rpn class graph?" } ] }, { "id": 290097, "postDate": "2018-03-02T22:52:20.517Z", "content": "Hi Heng, can you comment on how to reproduce LB 0.463 using dummy-14.2? We run your code, but feels it not easy to reproduce the LB score.
", "rawMarkdown": "Hi Heng, can you comment on how to reproduce LB 0.463 using dummy-14.2? We run your code, but feels it not easy to reproduce the LB score.", "votes": 1, "replies": [ { "id": 290129, "postDate": "2018-03-03T02:23:18.247Z", "content": "what score are you getting? should be in the 4.0 to 4.5 range?
", "rawMarkdown": "what score are you getting? should be in the 4.0 to 4.5 range?", "votes": 1 }, { "id": 290362, "postDate": "2018-03-03T21:47:46.797Z", "content": "So far we got 0.377
", "rawMarkdown": "So far we got 0.377" }, { "id": 291486, "postDate": "2018-03-06T08:16:06.573Z", "content": "I am also using dummy-14.2. I get a validation loss of 0.193 using train1_ids_gray_only1_500 and valid1_ids_gray_only1_43 after training for num_iters=1000*50. Hang is correct, removing small boxes improves validation loss isgnificantly. Is this good validation loss that can get you in the 0.4 to 0.45 range?
", "rawMarkdown": "I am also using dummy-14.2. I get a validation loss of 0.193 using train1_ids_gray_only1_500 and valid1_ids_gray_only1_43 after training for num_iters=1000*50. Hang is correct, removing small boxes improves validation loss isgnificantly. Is this good validation loss that can get you in the 0.4 to 0.45 range?" }, { "id": 291489, "postDate": "2018-03-06T08:21:27.117Z", "content": "Please wait for a while. I am releasing the final version mask-rcnn-resnet 50 end of today, together with train logfiles, trained models and submission CVS. Hopefully, you don't have to tune parameters and run this version as it is.
\n\nAll trained model and results are on gray images only. By submitting 53 gray images, you get LB score of 0.419 (i.e. 0.5139 = 0.419/65*53 per image on average)
", "rawMarkdown": "Please wait for a while. I am releasing the final version mask-rcnn-resnet 50 end of today, together with train logfiles, trained models and submission CVS. Hopefully, you don't have to tune parameters and run this version as it is.\n\nAll trained model and results are on gray images only. By submitting 53 gray images, you get LB score of 0.419 (i.e. 0.5139 = 0.419/65*53 per image on average)\n", "votes": 6 }, { "id": 291530, "postDate": "2018-03-06T09:45:45.883Z", "content": "Thanks, you do lots of great work!
", "rawMarkdown": "Thanks, you do lots of great work!" }, { "id": 291536, "postDate": "2018-03-06T10:06:16.413Z", "content": "Heng, \nThanks! I will figure it out. You have given enough information already. Thanks for the excellent work. Hope to see you reach the #1 position.
\n\nYou write all this code from scratch. Awesome!
", "rawMarkdown": "Heng, \nThanks! I will figure it out. You have given enough information already. Thanks for the excellent work. Hope to see you reach the #1 position. \n\nYou write all this code from scratch. Awesome! " } ] }, { "id": 285539, "postDate": "2018-02-20T14:54:10.170Z", "content": "How do deal with different color schemes? Do you convert to grayscale or use original images?
", "rawMarkdown": "How do deal with different color schemes? Do you convert to grayscale or use original images?", "votes": 1, "replies": [ { "id": 285541, "postDate": "2018-02-20T14:57:01.937Z", "content": "i haven't decide for mask rcnn yet. i intend to use both gray-scale and color. i.e. three class:
\n\nwhite on black
black on white
color
in my previous unet results, i just use color.
", "rawMarkdown": "i haven't decide for mask rcnn yet. i intend to use both gray-scale and color. i.e. three class:\n\n- white on black\n\n- black on white\n\n- color\n\n\nin my previous unet results, i just use color.", "votes": 1 } ] }, { "id": 285537, "postDate": "2018-02-20T14:46:15.007Z", "content": "error analysis on dummy images:
\n\ntop image:
\n\nmiddle image:
\n\nbottom image:
\n\nconclusion:
\n\nthese errors should be corrected
\n\nsome pixel shift from roi align
16x16 mask prediction is not sufficient. maybe i should try 24x24 or 28x28
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/285537/8605/Selection_061.png\")
there is a bug in crop_instance() for make_mask_target().
\n\n\"crop = instance[y0:y1,x0:x1]\" is wrong because the box coordinates are inclusive. I estimate this case a loss of about 0.02 loss in the LB score. Hope the the LB score is higher after this fix.
\n\nAlso i changed the implementation of roi_align to this: https://github.com/longcw/RoIAlign.pytorch
\n\nI will release all fix later.
\n\ncorrected code:
\n\ndef crop_instance(instance, box, size, threshold=0.5):\nH,W = instance.shape\nx0,y0,x1,y1 = np.rint(box).astype(np.int32)\n\n# x0 = max(0,x0) #input should already be clipped\n# y0 = max(0,y0)\n# x1 = min(W,x1)\n# y1 = min(H,y1)\n#print(x0,y0,x1,y1)\n\ncrop = instance[y0:y1+1,x0:x1+1]\ncrop = cv2.resize(crop,(size,size))\ncrop = (crop>threshold).astype(np.float32)\nreturn crop\nThanks for your wonderful work. Appreciate it! I really like your pytorch maskrcnn code. While I have some questions: \n1. I finished the training processing and find all the testing color images are failed, the results are poor, no bounding box got. So, I must training the model separate by color and gray? Is it enough for training samples. \n2. I submit the results, only using (masks = net.masks) results, while the performance only have 0.33. Anything wrong of my inference processing. I know four rgb image are failed, but this may lead score drop from 0.46 to 0.33?\n 3. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\". So you mean I must delete tiny object in the training processing, it lead to great improvement? Interesting, why?\nThanks for your code again. I really like the visualize results. And it has the same performance compared with caffe2 maskrcnn.
", "rawMarkdown": "Thanks for your wonderful work. Appreciate it! I really like your pytorch maskrcnn code. While I have some questions: \n1. I finished the training processing and find all the testing color images are failed, the results are poor, no bounding box got. So, I must training the model separate by color and gray? Is it enough for training samples. \n2. I submit the results, only using (masks = net.masks) results, while the performance only have 0.33. Anything wrong of my inference processing. I know four rgb image are failed, but this may lead score drop from 0.46 to 0.33?\n 3. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\". So you mean I must delete tiny object in the training processing, it lead to great improvement? Interesting, why?\nThanks for your code again. I really like the visualize results. And it has the same performance compared with caffe2 maskrcnn.", "votes": 1, "replies": [ { "id": 286968, "postDate": "2018-02-23T05:08:27.297Z", "content": "May I ask a question about the submit.py? In the submit file I found this line :
\n\ntest_dataset = ScienceDataset(\n #'test1_ids_gray_only_53', mode='test',\n #'train1_ids_gray_only1_500', mode='test',\n 'valid1_ids_gray_only1_43', mode='test',\n transform = submit_augment)
\n\nI think it means I did not predict the test data but predicted the verification data，What the correct code should be?
", "rawMarkdown": "May I ask a question about the submit.py? In the submit file I found this line :\n\n\ntest_dataset = ScienceDataset(\n #'test1_ids_gray_only_53', mode='test',\n #'train1_ids_gray_only1_500', mode='test',\n 'valid1_ids_gray_only1_43', mode='test',\n transform = submit_augment)\n\n\nI think it means I did not predict the test data but predicted the verification data，What the correct code should be?" } ] }, { "id": 283750, "postDate": "2018-02-15T23:18:17.827Z", "content": "Great work!
\n\nAs the trunk of the model you have chosen the resnet50, pretrained I suppose. I don't know in details this architecture but if I'm not mistaken, it's part of a classifier model. Is it so directly shareable with a segmentation based task? Recent works such as google's Multimodel show us how multitasking / weight sharing can find improvements in \"hopeless places\", so I'm not surprised that it works very well, but I'm curious to know the reasons for this choice.
\n\nEdit: looking at the papers https://arxiv.org/pdf/1512.03385.pdf and https://arxiv.org/pdf/1610.02915.pdf i've understood that atleast the architecture fits the problem, it preserves the input dimensions and low level informations, necessary for segmentation. There's also a U-net like version https://link.springer.com/chapter/10.1007/978-3-319-70096-0_38?no-access=true .
", "rawMarkdown": "Great work! \n\nAs the trunk of the model you have chosen the resnet50, pretrained I suppose. I don't know in details this architecture but if I'm not mistaken, it's part of a classifier model. Is it so directly shareable with a segmentation based task? Recent works such as google's Multimodel show us how multitasking / weight sharing can find improvements in \"hopeless places\", so I'm not surprised that it works very well, but I'm curious to know the reasons for this choice.\n\nEdit: looking at the papers https://arxiv.org/pdf/1512.03385.pdf and https://arxiv.org/pdf/1610.02915.pdf i've understood that atleast the architecture fits the problem, it preserves the input dimensions and low level informations, necessary for segmentation. There's also a U-net like version https://link.springer.com/chapter/10.1007/978-3-319-70096-0_38?no-access=true .", "votes": 1 }, { "id": 283286, "postDate": "2018-02-15T07:04:26.967Z", "content": "please see \"20180214/dummy-14.2\" and \"20180214/example_results/14.2\".
\n\ni added a trained model for your experiment, debugging, etc you can produce results like this on the whole image:
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/283286/8572/bb481eae02085bbae08742f702b9ab7d8b2ff9df2fbefeee9fac51f7c77dd01f.multi_mask.png\")
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/283286/8573/bb481eae02085bbae08742f702b9ab7d8b2ff9df2fbefeee9fac51f7c77dd01f.contour.png\")
Hi Heng, I managed to run your 20180314 version. However, I think I am not running it with full function because of your modifications in mask_nms, which is called here in the model.py:
self.masks, self.mask_proposals = mask_nms(cfg, mode, inputs, self.rcnn_proposals, self.mask_logits)\n\nself.detections = self.mask_proposals\nWhat is self.mask_proposals? In my current running, I just change the above two lines back to the 20180306 version.
Also I modified crop = self.crops[l](fs[l], proposals[index].view(-1,8)) to crop = self.crops[l](fs[l], proposals[index].view(-1,7)).
@Yan: Me too. I also got the error and thanks for your points to solve it. I am running it for training from scratch. What is your performance when ran the se-resnext version? Do you train with full images (color+gray)? For version of 20180306, I achieved 0.44 LB on full image set (too low than what I expected). I think train with pretrain model will be better. Do you try with pretrain model?
", "rawMarkdown": "@Yan: Me too. I also got the error and thanks for your points to solve it. I am running it for training from scratch. What is your performance when ran the se-resnext version? Do you train with full images (color+gray)? For version of 20180306, I achieved 0.44 LB on full image set (too low than what I expected). I think train with pretrain model will be better. Do you try with pretrain model?" }, { "id": 296620, "postDate": "2018-03-15T14:28:44.803Z", "content": "I am running se-resnext with only gray images now, not finished yet. Also I do not use pretrained weights. Using imagenet weight for resnet is possible from the official pytorch weights, I am not sure about resnext or se-resnext. You are using cuda 8 or cuda 9? I am using cuda 9, but want to know if anyone succeeds with cuda 8.
", "rawMarkdown": "I am running se-resnext with only gray images now, not finished yet. Also I do not use pretrained weights. Using imagenet weight for resnet is possible from the official pytorch weights, I am not sure about resnext or se-resnext. You are using cuda 8 or cuda 9? I am using cuda 9, but want to know if anyone succeeds with cuda 8. " }, { "id": 296642, "postDate": "2018-03-15T14:49:58.267Z", "content": "Good. I am also running it in cuda 9.0 and cudnn 7.0.5 using gray only. My step size is [0.01 in 10k, 0.001 in 20k, 0.0001 in 25k] I hope it will complete tomorrow. Actually, it takes more GPU resource than resnet50, so I can only run with batch size of 8. For resnet50, Do you achieve similar result as author reported (0.41LB on gray)? I only got 0.445 LB but on both gray and color image. I guess if I use gray image only, this score will lower than 0.41. How about you?
", "rawMarkdown": "Good. I am also running it in cuda 9.0 and cudnn 7.0.5 using gray only. My step size is [0.01 in 10k, 0.001 in 20k, 0.0001 in 25k] I hope it will complete tomorrow. Actually, it takes more GPU resource than resnet50, so I can only run with batch size of 8. For resnet50, Do you achieve similar result as author reported (0.41LB on gray)? I only got 0.445 LB but on both gray and color image. I guess if I use gray image only, this score will lower than 0.41. How about you?" }, { "id": 296661, "postDate": "2018-03-15T15:25:48.420Z", "content": "I also got ~0.445 on both gray and color image using the ResNeXt model, but I only ran for about 5k iters, seems the loss (especially rcnn cls loss) cannot get down for more iters. Is that right? I saw you've ran it for 25k iters.\nBtw, the model has poor performance on color image & slim nuclei, as well as some dark images. May be we neet to find other ways to handle the color ones using pre/post process
", "rawMarkdown": "I also got ~0.445 on both gray and color image using the ResNeXt model, but I only ran for about 5k iters, seems the loss (especially rcnn cls loss) cannot get down for more iters. Is that right? I saw you've ran it for 25k iters.\nBtw, the model has poor performance on color image & slim nuclei, as well as some dark images. May be we neet to find other ways to handle the color ones using pre/post process" }, { "id": 296666, "postDate": "2018-03-15T15:35:57.050Z", "content": "To run resneXt, Have you change the code? Because the code only has two options: Resnet-50 and SE-resneXt. The score 0.445LB which I achieved that used Resnet-50 and trained with Adam optimization on 20k iterations. I am run SE-ResnetXt now, I did not complete yet.
\n\nYou are right about color image. The author suggested that we can convert it to gray and purple image and trained it with two models: 1 for gray and 1 for purple. Then concatenate the result at the end. For post-processing, he did it with removing small cell and dilation. I think it is fine
", "rawMarkdown": "To run resneXt, Have you change the code? Because the code only has two options: Resnet-50 and SE-resneXt. The score 0.445LB which I achieved that used Resnet-50 and trained with Adam optimization on 20k iterations. I am run SE-ResnetXt now, I did not complete yet. \n\nYou are right about color image. The author suggested that we can convert it to gray and purple image and trained it with two models: 1 for gray and 1 for purple. Then concatenate the result at the end. For post-processing, he did it with removing small cell and dilation. I think it is fine" }, { "id": 296675, "postDate": "2018-03-15T15:51:53.207Z", "content": "I modified the code with similar process as Yan Wang posted up there.\nI had trained separate model using gray/color images, but the result is no better than current version. Model trained on gray images performs well, but color one is too poor and I'm afraid of overfitting with such few train/valid images.
\n\nAnyway, thanks for feedback! There's still time and I have some trials to struggle :D
", "rawMarkdown": "I modified the code with similar process as Yan Wang posted up there.\nI had trained separate model using gray/color images, but the result is no better than current version. Model trained on gray images performs well, but color one is too poor and I'm afraid of overfitting with such few train/valid images. \n\nAnyway, thanks for feedback! There's still time and I have some trials to struggle :D" }, { "id": 296678, "postDate": "2018-03-15T15:57:47.017Z", "content": "You are welcome. Actually, I got 0.47 LB on Keras version (https://github.com/matterport/Mask_RCNN). but it takes too too much time for training (1 epoch ~6 min, while Heng code ~2 min) . So I moved to pytorch code. The different one is Keras based uses pretrained model (resnet-50) and I was used it to achieved the score. The strategy for training: train on head and then train all. Because we want to keep the good features from pretrained. However, I did not find any way to use pretrain in this implementation. If you find it, please let me know.
", "rawMarkdown": "You are welcome. Actually, I got 0.47 LB on Keras version (https://github.com/matterport/Mask_RCNN). but it takes too too much time for training (1 epoch ~6 min, while Heng code ~2 min) . So I moved to pytorch code. The different one is Keras based uses pretrained model (resnet-50) and I was used it to achieved the score. The strategy for training: train on head and then train all. Because we want to keep the good features from pretrained. However, I did not find any way to use pretrain in this implementation. If you find it, please let me know. " }, { "id": 296685, "postDate": "2018-03-15T16:12:33.320Z", "content": "Since the model has been made a little complicated, I'm not sure if a pretrained model based on other code (i.e. models from the official torchvision repo) could be fitted into this model. Will let u know if I have some work-around. And thanks for sharing training strategy!
", "rawMarkdown": "Since the model has been made a little complicated, I'm not sure if a pretrained model based on other code (i.e. models from the official torchvision repo) could be fitted into this model. Will let u know if I have some work-around. And thanks for sharing training strategy!" }, { "id": 296759, "postDate": "2018-03-15T18:11:30.440Z", "content": "please compare your loss with the log files in the google drive. A good loss at the end should be:
\n\nrate = 0.001 (using SGD)
\n\ntotal = 0.393 (variable, becuase of rcnn,mask sampling of samples)
\n\nrpn = 0.01 0.01 (you should this number)
\n\nrcnn = 0.25 0.01 (rcnn reg loss should be 0.01, rcnn cls loss should be 0.0.25)
\n\nmask = 0.11 (mask loss is about 0.12 to 0.11)
\n\nWith this , you should get LB = 0.42 on gray images only for resnet50-fpn\nthere are some easy mask post processing you can do. try to draw the mask one by one, one at a time. you will see some single pixel noise, try to clean them out.
\n\ntry to train longer iterations. the mask prediction needs long iteration to get to pixel accuracy (i train 29K iterations (that is about 300 epoches!) which takes 24hr). Don't use epoch, use iterations to count.
\n\nthere are 12 color images.
\n\nfor the 8 (purple on yellow), convert to gray. these 8 images give LB score of bout 0.03.
\n\nthe remaining 4 images are H&E stain. train this with external labelled images (you can find this at the kernel thread). The LB score should be about 0.010 to 0.015.
\n\nremove scale augmentation at the last training iterations with small learning rate of 0.001
", "rawMarkdown": "please compare your loss with the log files in the google drive. A good loss at the end should be:\n\nrate = 0.001 (using SGD)\n\ntotal = 0.393 (variable, becuase of rcnn,mask sampling of samples)\n\nrpn = 0.01 0.01 (you should this number) \n\nrcnn = 0.25 0.01 (rcnn reg loss should be 0.01, rcnn cls loss should be 0.0.25)\n\nmask = 0.11 (mask loss is about 0.12 to 0.11)\n\nWith this , you should get LB = 0.42 on gray images only for resnet50-fpn\nthere are some easy mask post processing you can do. try to draw the mask one by one, one at a time. you will see some single pixel noise, try to clean them out.\n\ntry to train longer iterations. the mask prediction needs long iteration to get to pixel accuracy (i train 29K iterations (that is about 300 epoches!) which takes 24hr). Don't use epoch, use iterations to count.\n\n\nthere are 12 color images.\n\nfor the 8 (purple on yellow), convert to gray. these 8 images give LB score of bout 0.03.\n\nthe remaining 4 images are H&E stain. train this with external labelled images (you can find this at the kernel thread). The LB score should be about 0.010 to 0.015.\n\nremove scale augmentation at the last training iterations with small learning rate of 0.001", "votes": 2 }, { "id": 296788, "postDate": "2018-03-15T18:59:15.987Z", "content": "Thanks Heng. It is a very nice log. Could you tell me why did not you use pretrain model? I used keras with pretrain and it provides better than without pretrain. Using pretrain model we can have a good strategy for training likes paper mentioned. Any problem if we implement model.py with pretrain?
\n\nFYI, I completed training using 500 gray and evaluate in 53 testing gray, validate in 43 grays. It achieved 0.429 after 25k iteration with learning schedule as LR = StepLR([ (0, 0.01), (10000, 0.001), (20000, 0.0001), (25000, 0.00001)])\nThe loss is
\n 0.0001 25.0 k 400.0 0.2 m | 0.366 0.01 0.01 0.22 0.01 0.11 | 0.498 0.02 0.02 0.29 0.01 0.16 | 0.547 0.04 0.03 0.31 0.01 0.15 | 12 hr 07 min\n
Update: After training on other color image (12 images) without modifying the code, Then merge result with gray (have trained on gray image), then my current score is 0.47LB after 3.5K. I think the result will improve if I train on three networks, instead of two as I did. Three network means: 1 for gray, 1 for gray of color and 1 for purple convered from color image.\nThis is log of training on color
\n\n\n0.0001 3.5 k 247.8 0.0 m | 0.792 0.04 0.04 0.48 0.01 0.22 | 0.707 0.03 0.04 0.40 0.02 0.21 | 0.677 0.01 0.04 0.40 0.03 0.20 | 1 hr 51 min\n
@john1231983 ... 0.47LB after 3.5K
\n\nYes this should be around the baseline of the code. The rest are just augmentation (e.g. using merged mosiac images), sample balancing, learning rate, etc ... should be about to go up a little more to 0.48 to 0.49 i think.
\n\nThe rest are heavy mask post processing to break the 0.500 mark.
", "rawMarkdown": "@john1231983 ... 0.47LB after 3.5K\n\nYes this should be around the baseline of the code. The rest are just augmentation (e.g. using merged mosiac images), sample balancing, learning rate, etc ... should be about to go up a little more to 0.48 to 0.49 i think.\n\nThe rest are heavy mask post processing to break the 0.500 mark.\n ", "votes": 3 }, { "id": 297286, "postDate": "2018-03-16T17:06:08.023Z", "content": "the 20180306 code got me 0.427 after training in the gray500 - nice!
\n\nI downloaded the 20180314 model and made the changes you suggested but am getting an error...
\n\n File \"/home/james/Dropbox/LINUX/remote/HKC_c/net/resnet50_mask_rcnn/modelSE.py\", line 297, in forward\n crop = self.crops[l](fs[l], proposals[index].view(-1,7))\n File \"/home/james/anaconda3/envs/HKC/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 365, in __call__\n result = self.forward(*input, **kwargs)\n File \"/home/james/Dropbox/LINUX/remote/HKC_c/net/lib/roi_align_pool_tf/module.py\", line 40, in forward\n ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1)\nValueError: too many values to unpack (expected 5)\nAny idea what's causing that?
\n\nit's happening just after the line...
\n\ncrop = self.crops[l](fs[l], proposals[index].view(-1,7))\n@Jame: You should you the lines of the old version to fix it.
\n\n@Heng: We have a total of 664 images, then we separate them into 2 classes: gray and color. The gray will use as you did (543 images). The remaining 121 color images will be converted to gray and purple and train on two separate networks. But you said
\n\n\n\n\nor the 8 (purple on yellow), convert to gray. these 8 images give LB score of bout 0.03.\n the remaining 4 images are H&E stain. train this with external labelled images (you can find this at the kernel thread). The LB score should be about 0.010 to 0.015.
\n
It looks that we have to separate the color training set into two classes: purple on yellow and H&E, while H&E using augmentation data, and purple using above scheme (convert gray+purple). That means we have three networks for color images. Please correct me if I was wrong? Thanks
", "rawMarkdown": "@Jame: You should you the lines of the old version to fix it. \n\n@Heng: We have a total of 664 images, then we separate them into 2 classes: gray and color. The gray will use as you did (543 images). The remaining 121 color images will be converted to gray and purple and train on two separate networks. But you said\n\n> or the 8 (purple on yellow), convert to gray. these 8 images give LB score of bout 0.03.\nthe remaining 4 images are H&E stain. train this with external labelled images (you can find this at the kernel thread). The LB score should be about 0.010 to 0.015.\n\nIt looks that we have to separate the color training set into two classes: purple on yellow and H&E, while H&E using augmentation data, and purple using above scheme (convert gray+purple). That means we have three networks for color images. Please correct me if I was wrong? Thanks" }, { "id": 297370, "postDate": "2018-03-16T23:15:22.923Z", "content": "@James: You should use the lines of the old version to fix it.
\n\nThanks, @john
\n\nThe good news is that it's training. My entire CropRoi class is now the same as the in original model. \nSpecifically...
\n\ncrop = self.crops[l](fs[l], proposals[index].view(-1,7))\nis now back to
\n\ncrop = self.crops[l](fs[l], rois[index].view(-1,5))\nand I (re)inserted
\n\nrois = proposals.detach().data[:,0:5]\nrois = Variable(rois)\nAND I had to reduce my batch size from 16 to 8 because I was getting oom errors.
\n\nAnyone else had to reduce the batch size? I have two GPUs a 1080Ti and a 1080. So I guess they are running a batch size of 4 on each.
\n\nI am still curious to know how @Yan got this line working:
\n\ncrop = self.crops[l](fs[l], proposals[index].view(-1,7))\nI'll post my new score when I get it.
", "rawMarkdown": "@James: You should use the lines of the old version to fix it.\n\nThanks, @john\n\nThe good news is that it's training. My entire CropRoi class is now the same as the in original model. \nSpecifically...\n\n crop = self.crops[l](fs[l], proposals[index].view(-1,7))\n\nis now back to \n\n crop = self.crops[l](fs[l], rois[index].view(-1,5))\n\nand I (re)inserted\n\n rois = proposals.detach().data[:,0:5]\n rois = Variable(rois)\n\nAND I had to reduce my batch size from 16 to 8 because I was getting oom errors.\n\nAnyone else had to reduce the batch size? I have two GPUs a 1080Ti and a 1080. So I guess they are running a batch size of 4 on each.\n\nI am still curious to know how @Yan got this line working:\n\n crop = self.crops[l](fs[l], proposals[index].view(-1,7))\n\nI'll post my new score when I get it.\n" }, { "id": 297380, "postDate": "2018-03-17T00:15:43.613Z", "content": "@Jame You are welcome. For my score, I got 0.473 now without modify the code. 0.03 got from extra H&E dataset. I group all color images in same group and train with single network. I did not seperate them to two network as Henh mentioned. Do you did same thing?
", "rawMarkdown": "@Jame You are welcome. For my score, I got 0.473 now without modify the code. 0.03 got from extra H&E dataset. I group all color images in same group and train with single network. I did not seperate them to two network as Henh mentioned. Do you did same thing?" }, { "id": 297420, "postDate": "2018-03-17T03:09:53.337Z", "content": "@James: Yes, this line will cause some error, we can fix the error by adding two extra return values for this line: ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1) in module.py
@Yan what is your score now? I got 0.429 on 500 gray and 0.47 on all gray and color.
", "rawMarkdown": "@Yan what is your score now? I got 0.429 on 500 gray and 0.47 on all gray and color." }, { "id": 297442, "postDate": "2018-03-17T03:53:41.790Z", "content": "Hi john1231983! \nCould you tell me how many iterations it took to get to 0.429 and 0.47 \nI only have access to K80 GPU on google cloud compute so training takes a long time for me.
\n\nthanks\nMehul
", "rawMarkdown": "Hi john1231983! \nCould you tell me how many iterations it took to get to 0.429 and 0.47 \nI only have access to K80 GPU on google cloud compute so training takes a long time for me.\n\nthanks\nMehul" }, { "id": 297597, "postDate": "2018-03-17T13:53:45.510Z", "content": "@John, We are using a K80 for this code too, not finished yet. I guess your score is good enough for this code. We are trying to run the code on our own server now, which has more GPUs.
", "rawMarkdown": "@John, We are using a K80 for this code too, not finished yet. I guess your score is good enough for this code. We are trying to run the code on our own server now, which has more GPUs." }, { "id": 297601, "postDate": "2018-03-17T14:18:20.453Z", "content": "Hi mehul. I have posted it in this thread. Please read it in above\n@yan I see, FYI. I got 0.473 when using HE external dataset. I also finding the way to use pretrained model ( convert from caffe). I guess the performance increases if we use it. But I did not try
", "rawMarkdown": "Hi mehul. I have posted it in this thread. Please read it in above\n@yan I see, FYI. I got 0.473 when using HE external dataset. I also finding the way to use pretrained model ( convert from caffe). I guess the performance increases if we use it. But I did not try", "votes": 2 }, { "id": 297847, "postDate": "2018-03-18T05:25:17.630Z", "content": "@Heng CherKeng\n Hello Heng, how do you calculate accuracy?
", "rawMarkdown": "@Heng CherKeng\n Hello Heng, how do you calculate accuracy?" }, { "id": 297861, "postDate": "2018-03-18T06:38:54.270Z", "content": "Hi @John. What's the extra H&E dataset you used? Possible to give a link here? Thanks a lot!
", "rawMarkdown": "Hi @John. What's the extra H&E dataset you used? Possible to give a link here? Thanks a lot!" }, { "id": 297869, "postDate": "2018-03-18T07:17:23.293Z", "content": "This one https://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations Let me know if it improve your perfomance. I got 0.002 improve
", "rawMarkdown": "This one https://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations Let me know if it improve your perfomance. I got 0.002 improve" }, { "id": 297972, "postDate": "2018-03-18T12:53:59.693Z", "content": "Sure. I will let you know when I get the result. Thanks!
", "rawMarkdown": "Sure. I will let you know when I get the result. Thanks!" }, { "id": 298322, "postDate": "2018-03-19T11:02:22.923Z", "content": "Hi John, As you mentioned, After training on other color image (12 images). Are these 12 images are in training data set? I am thinking these 12 images in test data set, correct? If so , which training set and valid set do you use to create the model? I am trying Heng's code to create model to predict 12 purple images. I am training on 108(random split into 100 in train set, 8 in valid set) colorful images , the coding running with 2 1080ti GPU for 9 hours. but the loss still high. Any idea?\n0.0100 7.9 k 1258.4 0.1 m | 0.816 0.04 0.03 0.55 0.01 0.18 | 0.691 0.01 0.03 0.44 0.02 0.20 | 0.647 0.04 0.03 0.37 0.02 0.18 | 9 hr 12 min
", "rawMarkdown": "Hi John, As you mentioned, After training on other color image (12 images). Are these 12 images are in training data set? I am thinking these 12 images in test data set, correct? If so , which training set and valid set do you use to create the model? I am trying Heng's code to create model to predict 12 purple images. I am training on 108(random split into 100 in train set, 8 in valid set) colorful images , the coding running with 2 1080ti GPU for 9 hours. but the loss still high. Any idea?\n0.0100 7.9 k 1258.4 0.1 m | 0.816 0.04 0.03 0.55 0.01 0.18 | 0.691 0.01 0.03 0.44 0.02 0.20 | 0.647 0.04 0.03 0.37 0.02 0.18 | 9 hr 12 min" }, { "id": 298340, "postDate": "2018-03-19T11:54:50.260Z", "content": "Sorry. It was wrong. It means train with color image ime 113 image and 30 augment image ( link above) . You can get some loss as me ( as above) . After that, i test with 12 image and got 0.473. How about your current score now? For your problem , let reduce learning rate to 10 times after each 3k iters
", "rawMarkdown": "Sorry. It was wrong. It means train with color image ime 113 image and 30 augment image ( link above) . You can get some loss as me ( as above) . After that, i test with 12 image and got 0.473. How about your current score now? For your problem , let reduce learning rate to 10 times after each 3k iters" }, { "id": 298365, "postDate": "2018-03-19T13:02:28.473Z", "content": "thanks john, I random picked up 150 train images (gray and purple) and predict on the 8 purple test images, then merged with the gray result(0.417) . my score is 0.439 at that time. I am trying to train a colorful images model then predict on the 8 purple to see the result. will test the learning rate 0.001.
", "rawMarkdown": "thanks john, I random picked up 150 train images (gray and purple) and predict on the 8 purple test images, then merged with the gray result(0.417) . my score is 0.439 at that time. I am trying to train a colorful images model then predict on the 8 purple to see the result. will test the learning rate 0.001." }, { "id": 298375, "postDate": "2018-03-19T13:24:27.833Z", "content": "Maybe something wrong. I have trained on 500 gray images and testing on 53 gray images, it must be around 0.429. Then the 113 color image will combined with 30 extra image to make training color set. Then I train it and test on 12 color image. Combined RLE with 53 testing above image, I got 0.473 LB. What is your batch size? I am using 7 because my GPU resource.
", "rawMarkdown": "Maybe something wrong. I have trained on 500 gray images and testing on 53 gray images, it must be around 0.429. Then the 113 color image will combined with 30 extra image to make training color set. Then I train it and test on 12 color image. Combined RLE with 53 testing above image, I got 0.473 LB. What is your batch size? I am using 7 because my GPU resource. " }, { "id": 298437, "postDate": "2018-03-19T15:13:38.247Z", "content": "thanks @Yan. The extra two params nailed it.
\n\nBefore I fixed that, I got the following results on the SE-ResnetXt.
\n\nTraining on gray500 only and submitting on gray53. Straight out the box, in other words...
\n\n12000 iterations: 0.423
\n\n14500 iterations: 0.424
\n\n15000 iterations: 0.436*
\n\n16000 iterations: 0.419
\n\n18000 iterations: 0.414
\n\nWith the regular Res50 model I got 0.427 using 16500 iterations (didn't try and other iterations)
\n\nSeems odd to me that 15000 should spike up like that. What's going on?
", "rawMarkdown": "thanks @Yan. The extra two params nailed it.\n\nBefore I fixed that, I got the following results on the SE-ResnetXt.\n\nTraining on gray500 only and submitting on gray53. Straight out the box, in other words...\n\n12000 iterations: 0.423 \n\n14500 iterations: 0.424 \n\n15000 iterations: 0.436*\n\n16000 iterations: 0.419 \n\n18000 iterations: 0.414\n\nWith the regular Res50 model I got 0.427 using 16500 iterations (didn't try and other iterations)\n\nSeems odd to me that 15000 should spike up like that. What's going on?" }, { "id": 298441, "postDate": "2018-03-19T15:18:55.470Z", "content": "Impressive with 0.436 LB. Could you tell me your learning schedule? How do you set the learning rate schedule? I am using 0.01 from 0 to 10k, 0.001 from 10k to 15k and 0.0001 from 15k to 30k
", "rawMarkdown": "Impressive with 0.436 LB. Could you tell me your learning schedule? How do you set the learning rate schedule? I am using 0.01 from 0 to 10k, 0.001 from 10k to 15k and 0.0001 from 15k to 30k " }, { "id": 298445, "postDate": "2018-03-19T15:27:03.187Z", "content": "learning rate was 0.01 all the way (out of the box).\nI am running a version now with [(0, 0.01), (10000, 0.001), (20000, 0.0001), (25000, 0.00001)]
", "rawMarkdown": "learning rate was 0.01 all the way (out of the box).\nI am running a version now with [(0, 0.01), (10000, 0.001), (20000, 0.0001), (25000, 0.00001)]" }, { "id": 298454, "postDate": "2018-03-19T15:37:28.743Z", "content": "@Jam: What is return should I change ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1) in module.py? Does it like ids, x0, y0, x1, y1,_,_= torch.split(rois, 1, dim=1) I have corrected based on the previous version, but for safety, I would like to change as small as possible comparsion with current model.py
yes...
\n\nids, x0, y0, x1, y1,_,_ = torch.split(rois, 1, dim=1)
", "rawMarkdown": "yes...\n\nids, x0, y0, x1, y1,\\_,\\_ = torch.split(rois, 1, dim=1)\n" }, { "id": 302469, "postDate": "2018-03-24T05:09:00.123Z", "content": "Thanks for sharing.Can you tell me how to merge the submissions？
", "rawMarkdown": "Thanks for sharing.Can you tell me how to merge the submissions？" }, { "id": 305010, "postDate": "2018-03-28T10:59:54.333Z", "content": "Thanks !\n@Yan Wang
\n\nWhat is self.mask_proposals? How do you solve this :
\n\nself.masks, self.mask_proposals = mask_nms(cfg, mode, inputs, self.rcnn_proposals, self.mask_logits)\nself.detections = self.mask_proposals
", "rawMarkdown": "Thanks !\n@Yan Wang\n\nWhat is self.mask_proposals? How do you solve this :\n\nself.masks, self.mask_proposals = mask_nms(cfg, mode, inputs, self.rcnn_proposals, self.mask_logits)\nself.detections = self.mask_proposals" }, { "id": 307410, "postDate": "2018-04-01T14:52:24.213Z", "content": "Hi John. May I ask what configuration are you using with matterport's keras+tensorflow implementation? I using set the image shape to 512 x 512 and it takes me about an hour to train an epoch on a Titan X pascal. ......
", "rawMarkdown": "Hi John. May I ask what configuration are you using with matterport's keras+tensorflow implementation? I using set the image shape to 512 x 512 and it takes me about an hour to train an epoch on a Titan X pascal. ......" }, { "id": 307411, "postDate": "2018-04-01T14:57:11.620Z", "content": "John, also may i ask what score are you getting for rpn class loss and rpn bbox loss ... for me these two scores remain unchanged at around 0.7 and 2.1 respectively, which seems a bit odd to me...
", "rawMarkdown": "John, also may i ask what score are you getting for rpn class loss and rpn bbox loss ... for me these two scores remain unchanged at around 0.7 and 2.1 respectively, which seems a bit odd to me..." }, { "id": 310691, "postDate": "2018-04-08T10:08:33.970Z", "content": "John, I am getting a similar spike around 15K. Were you able to figure out the cause?
", "rawMarkdown": "John, I am getting a similar spike around 15K. Were you able to figure out the cause?" } ] }, { "id": 295550, "postDate": "2018-03-13T20:51:54.267Z", "content": "Hey @Heng CherKeng, a first time Kaggler here, just found this discussion and I have a quick question: it seems you only submit 53 gray images. If we also submit the rest of (color) images and don't do well on those, will we get 0 or get penalized by doing bad on those images? I didn't know we could submit just a subset of images.
", "rawMarkdown": "Hey @Heng CherKeng, a first time Kaggler here, just found this discussion and I have a quick question: it seems you only submit 53 gray images. If we also submit the rest of (color) images and don't do well on those, will we get 0 or get penalized by doing bad on those images? I didn't know we could submit just a subset of images. ", "votes": 2, "replies": [ { "id": 295634, "postDate": "2018-03-14T00:16:28.270Z", "content": "I believe we will get 0 for bad predictions, you may try it, since we have 5 submissions every day.
", "rawMarkdown": "I believe we will get 0 for bad predictions, you may try it, since we have 5 submissions every day.", "votes": 1 }, { "id": 295699, "postDate": "2018-03-14T03:06:48.877Z", "content": "The min score for an image is zero. It will not be negative. The purpose of submitting separately is for me to find out the weakness of my algo and the difficulty of the test images.
", "rawMarkdown": "The min score for an image is zero. It will not be negative. The purpose of submitting separately is for me to find out the weakness of my algo and the difficulty of the test images.", "votes": 3 }, { "id": 295935, "postDate": "2018-03-14T12:46:23.053Z", "content": "FYI, I have ran the code with gray only and test with gray only. I achieve 0.36LB after 35k iterations ~2hours.
", "rawMarkdown": "FYI, I have ran the code with gray only and test with gray only. I achieve 0.36LB after 35k iterations ~2hours. " } ] }, { "id": 293471, "postDate": "2018-03-09T22:54:07.453Z", "content": "Hi Heng, \nThanks again for sharing. I meet some problems when I'm trying to modify some parameters of the network in net/resnet../model.py. My case is, running model.py check is fine, then train the network is fine, but when I use the new .ptb to run submit.py, not a single image can pass. The errors are in\n\nrevert(net, images) #unpad, undo test-time augment etc ....\n\n, then in this \"revert\" function \n\nnet.detections[:,0]\n\ndimension error. Actually, the net.detections is of size 0 for every image.
How could I make modifications to the structure of the network? Am I miss something?
", "rawMarkdown": "Hi Heng, \nThanks again for sharing. I meet some problems when I'm trying to modify some parameters of the network in net/resnet../model.py. My case is, running model.py check is fine, then train the network is fine, but when I use the new .ptb to run submit.py, not a single image can pass. The errors are in\n```\nrevert(net, images) #unpad, undo test-time augment etc ....\n```\n, then in this \"revert\" function \n```\nnet.detections[:,0]\n```\ndimension error. Actually, the net.detections is of size 0 for every image.\n\nHow could I make modifications to the structure of the network? Am I miss something?", "votes": 2, "replies": [ { "id": 294012, "postDate": "2018-03-11T03:43:20.410Z", "content": "Hit the same error as yours when execute submit.py:
\n\n\n\n\nindex = (net.detections[:,0]==b).nonzero().view(-1)\n IndexError: too many indices for tensor of dimension 1
\n
Link here may help https://github.com/numpy/numpy/issues/10447. If true, still i did not figure out how to modify .nonzero(). to make it work.
", "rawMarkdown": "Hit the same error as yours when execute submit.py: \n> index = (net.detections[:,0]==b).nonzero().view(-1)\nIndexError: too many indices for tensor of dimension 1\n\nLink here may help https://github.com/numpy/numpy/issues/10447. If true, still i did not figure out how to modify *.nonzero().* to make it work." }, { "id": 294128, "postDate": "2018-03-11T10:50:07.420Z", "content": "If you print net.detections, you can see it is a variable of size 0, for all images. I think some other part code should also be modified together. Haven't figure it out yet.
", "rawMarkdown": "If you print net.detections, you can see it is a variable of size 0, for all images. I think some other part code should also be modified together. Haven't figure it out yet." }, { "id": 312855, "postDate": "2018-04-12T13:20:36.263Z", "content": "Hi, i got this error when using submit.py for stage2 dataset. Did anybody managed to find out what causes this error ?
", "rawMarkdown": "Hi, i got this error when using submit.py for stage2 dataset. Did anybody managed to find out what causes this error ? " }, { "id": 312858, "postDate": "2018-04-12T13:24:38.543Z", "content": "you can modify the code.
\n\n net.set_mode('test')\n with torch.no_grad():\n input = torch.from_numpy(augment_image.transpose((2,0,1))).float().div(255).unsqueeze(0)\n input = Variable(input).cuda()\n net.forward(input)\n\n if (len(net.detections)==0):\n ... save a zero array or just ignore .....\n continue\n else:\n\n rcnn_proposal, detection, mask, instance = undo_test_augment(net, image, **params)\nThanks @Heng CherKeng for your help, however because i am using your 03062018 version of your code i don't have the undo_test_augment function. Instead I found out that by modifying the submit.py code at line 221 like so
\n\n with torch.no_grad():\n inputs = Variable(inputs).cuda()\n net(inputs )\n if (len(net.detections)==0):\n revert(net, images)\nthe error resolved
", "rawMarkdown": "Thanks @Heng CherKeng for your help, however because i am using your 03062018 version of your code i don't have the undo_test_augment function. Instead I found out that by modifying the submit.py code at line 221 like so\n\n with torch.no_grad():\n inputs = Variable(inputs).cuda()\n net(inputs )\n if (len(net.detections)==0):\n revert(net, images)\n\nthe error resolved" } ] }, { "id": 289000, "postDate": "2018-02-28T18:03:21.750Z", "content": "I'm trying to run the mask-rcnn on a tesla k80 but I'm running into this error:
\n\nRuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1518244507981/work/torch/lib/THC/generic/THCTensorMath.cu:15\nIs the problem just my hardware? I can run the Unet code with the same setup.
\n\nEDIT:\nIf you're using cloud you need to use a p100/v100 gpu to run the mask-rcnns.
", "rawMarkdown": "I'm trying to run the mask-rcnn on a tesla k80 but I'm running into this error: \n\n RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1518244507981/work/torch/lib/THC/generic/THCTensorMath.cu:15\n\nIs the problem just my hardware? I can run the Unet code with the same setup.\n\nEDIT:\nIf you're using cloud you need to use a p100/v100 gpu to run the mask-rcnns.", "votes": 2, "replies": [ { "id": 290423, "postDate": "2018-03-04T02:33:26.980Z", "content": "Yes I have the exact same issue with you. Have you fixed this problem?
", "rawMarkdown": "Yes I have the exact same issue with you. Have you fixed this problem? " }, { "id": 290471, "postDate": "2018-03-04T07:05:19.040Z", "content": "Hi Steven, I think change the setting of arch sm_52 when build .so might make a difference. This is what I found from a readme file of a pytorch mask rcnn on github\nWhich CUDA capabilities do we want to pre-build for?\nhttps://developer.nvidia.com/cuda-gpus\nCompute/shader model Cards
\n\n```\n# Which CUDA capabilities do we want to pre-build for?\n# https://developer.nvidia.com/cuda-gpus\n# Compute/shader model Cards\n# 6.1 P4, P40, Titan Xp, GTX 1080 Ti, GTX 1080\n# 6.0 P100\n# 5.2 M40, Titan X, GTX 980\n# 3.7 K80\n# 3.5 K40, K20\n# 3.0 K10, Grid K520 (AWS G2)\n```\nP100 is way too expensive on cloud
", "rawMarkdown": "Hi Steven, I think change the setting of arch sm_52 when build .so might make a difference. This is what I found from a readme file of a pytorch mask rcnn on github\nWhich CUDA capabilities do we want to pre-build for?\nhttps://developer.nvidia.com/cuda-gpus\nCompute/shader model Cards\n\n ```\n # Which CUDA capabilities do we want to pre-build for?\n # https://developer.nvidia.com/cuda-gpus\n # Compute/shader model Cards\n # 6.1 P4, P40, Titan Xp, GTX 1080 Ti, GTX 1080\n # 6.0 P100\n # 5.2 M40, Titan X, GTX 980\n # 3.7 K80\n # 3.5 K40, K20\n # 3.0 K10, Grid K520 (AWS G2)\n ```\n\nP100 is way too expensive on cloud" }, { "id": 290639, "postDate": "2018-03-04T15:54:38.327Z", "content": "Hi Chen, I'm using preemptible vms on gcp for my p100s, which end up being about $1/hr. With checkpoints there really much of an issue.
", "rawMarkdown": "Hi Chen, I'm using [preemptible vms on gcp][1] for my p100s, which end up being about $1/hr. With checkpoints there really much of an issue. \n\n [1]: https://cloud.google.com/preemptible-vms/" }, { "id": 292344, "postDate": "2018-03-07T20:56:10.860Z", "content": "Use -arch=sm_30 for Tesla K80. Pay attention sm_52 is hardcoded in gpu_nms setup.py, so update it manually too. All arch mapping here:\nhttp://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
", "rawMarkdown": "Use -arch=sm_30 for Tesla K80. Pay attention sm_52 is hardcoded in gpu_nms setup.py, so update it manually too. All arch mapping here:\nhttp://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/", "votes": 3 } ] }, { "id": 286978, "postDate": "2018-02-23T05:57:14.347Z", "content": "the correct making of the anchor boxes in rpn_mns.py should be corrected as below:
\n\ndef make_bases(base_size, base_apsect_ratios):\nbases = []\nfor ratio in base_apsect_ratios:\n w = base_size/math.sqrt(ratio)\n h = w*ratio\n rw = round(w/2)\n rh = round(h/2)\n base =(-rw, -rh, rw, rh, )\n bases.append(base)\n\nbases = np.array(bases,np.float32)\nreturn bases\nfor those who need to rebuild gpu_mns:
\n\nyou need to cythonize \"'gpu_nms.pyx'\" from the attached.
in box, there is a test code \"def run_check_nms():\", try keep = gpu_nms(rois, 0.5), keep = py_nms(rois, 0.5), keep = torch_nms(rois, 0.5), you should get same results
Decreasing batch_size to 2 still puts my 2GB GPU out of memory, and When I set batch_size to 1, got this error in startup:\nTraceback (most recent call last):\n File \"train.py\", line 475, in \n run_train()\n File \"train.py\", line 370, in run_train\n valid_loss, valid_acc = evaluate(net, valid_loader)\n File \"train.py\", line 153, in evaluate\n net(inputs, boxes, labels, instances )\n File \"/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py\", line 325, in call\n result = self.forward(*input, **kwargs)\n File \".../source/mask-rcnn/model/mask_rcnn_resnet50_fpn_net.py\", line 370, in forward\n make_rcnn_target(cfg, self.rpn_proposals, truth_boxes, truth_labels)\n File \".../source/mask-rcnn/model/mask_rcnn_lib/rcnn_target.py\", line 152, in make_rcnn_target\n sampled_proposals = torch.cat(sampled_proposals,0)\nRuntimeError: seq can't be empty
\n\n@Heng CherKeng is there a way to run it on 2GB by any chance?
", "rawMarkdown": "Decreasing batch_size to 2 still puts my 2GB GPU out of memory, and When I set batch_size to 1, got this error in startup:\nTraceback (most recent call last):\n File \"train.py\", line 475, in " }, { "id": 285439, "postDate": "2018-02-20T08:41:04.367Z", "content": "@Heng CherKeng\nHow do I rebuild gpu_mns? Its still looking for cuda 8, even though I am on 9.1
", "rawMarkdown": "@Heng CherKeng\nHow do I rebuild gpu_mns? Its still looking for cuda 8, even though I am on 9.1" }, { "id": 285586, "postDate": "2018-02-20T16:46:19.377Z", "content": "from model.mask_rcnn_lib.nms.gpu_nms import gpu_nms\nImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
\n\nThe same issue?
", "rawMarkdown": "from model.mask_rcnn_lib.nms.gpu_nms import gpu_nms\nImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory\n\nThe same issue?" }, { "id": 285799, "postDate": "2018-02-21T02:16:16.467Z", "content": "\"RuntimeError: seq can't be empty\" issue can be solved by skipping cases if all boxes are empty in the forward phase.
\n\ntrain.py:
\n\ndef all_boxes_none(boxes=None):\n all_none = False\n for j in range(len(boxes)):\n try:\n if boxes[j] == None: all_none = True\n except:\n pass\n return all_none\n\ndef evaluate(net, test_loader):\n test_num = 0\n skipped = 0\n test_loss = np.zeros(6, np.float32)\n test_acc = 0\n for i, (inputs, boxes, labels, instances, indices) in enumerate(test_loader, 0):\n inputs = Variable(inputs, volatile=True).cuda()\n if all_boxes_none(boxes):\n skipped += len(boxes)\n continue\n net(inputs, boxes, labels, instances)\n ...\n assert ((test_num + skipped) == len(test_loader.sampler))\n ...\n\ndef run_train():\n ...\n if all_boxes_none(truth_boxes):\n j = j + 1\n continue\n net(inputs, truth_boxes, truth_labels, truth_instances)\n ...\nnvidia-smi shows 3037MiB / 6078MiB during training, there may be spikes though.
\n\nI set batch_size to 1 to avoid out of memory situation.
", "rawMarkdown": "\"RuntimeError: seq can't be empty\" issue can be solved by skipping cases if all boxes are empty in the forward phase.\n\ntrain.py:\n\n def all_boxes_none(boxes=None):\n all_none = False\n for j in range(len(boxes)):\n try:\n if boxes[j] == None: all_none = True\n except:\n pass\n return all_none\n\n def evaluate(net, test_loader):\n test_num = 0\n skipped = 0\n test_loss = np.zeros(6, np.float32)\n test_acc = 0\n for i, (inputs, boxes, labels, instances, indices) in enumerate(test_loader, 0):\n inputs = Variable(inputs, volatile=True).cuda()\n if all_boxes_none(boxes):\n skipped += len(boxes)\n continue\n net(inputs, boxes, labels, instances)\n ...\n assert ((test_num + skipped) == len(test_loader.sampler))\n ...\n\n def run_train():\n ...\n if all_boxes_none(truth_boxes):\n j = j + 1\n continue\n net(inputs, truth_boxes, truth_labels, truth_instances)\n ...\n\nnvidia-smi shows 3037MiB / 6078MiB during training, there may be spikes though. \n\nI set batch_size to 1 to avoid out of memory situation. ", "votes": 3 }, { "id": 285802, "postDate": "2018-02-21T02:19:20.020Z", "content": "thanks for the code. the training batch can have empty box if the batch size is small (in my experiment you need batch size >=8 to avoid this).
\n\ni will use your fix in the next release!
\n\nhere is a simpler version:
\n\n #skip if there is no truth boxes\n if all(b is None for b in truth_boxes): continue\n\n # one iteration update -------------\n inputs = Variable(inputs).cuda()\n net( inputs, truth_boxes, truth_labels, truth_instances )\nThanks @Alexander Popov and @Heng CherKeng (for leading the pack), I was afraid my 2GB GPU will not be enough for this so I need a new machine and unfortunately Costco is out of gtx 1080 ti desktops :)
", "rawMarkdown": "Thanks @Alexander Popov and @Heng CherKeng (for leading the pack), I was afraid my 2GB GPU will not be enough for this so I need a new machine and unfortunately Costco is out of gtx 1080 ti desktops :)" }, { "id": 286693, "postDate": "2018-02-22T16:56:54.277Z", "content": "Hi @Alexander Popov ,thx for your solution , the code is beautiful,but I encountered a problem : \" assert (test_num == len(test_loader.sampler)) AssertionError\", Could you please give me some suggestions?
", "rawMarkdown": "Hi @Alexander Popov ,thx for your solution , the code is beautiful,but I encountered a problem : \" assert (test_num == len(test_loader.sampler)) AssertionError\", Could you please give me some suggestions?" }, { "id": 287639, "postDate": "2018-02-25T14:54:22.827Z", "content": "See the code above assert statement should be corrected, too:
\n\nassert ((test_num + skipped) == len(test_loader.sampler))\nor you could simply comment the assert out
", "rawMarkdown": "See the code above assert statement should be corrected, too:\n\n assert ((test_num + skipped) == len(test_loader.sampler))\n\nor you could simply comment the assert out", "votes": 1 }, { "id": 292376, "postDate": "2018-03-07T21:33:23.220Z", "rawMarkdown": "" } ] }, { "id": 596383, "postDate": "2019-08-10T15:11:21.013Z", "content": "Can I train mask r-cnn, if I have annotation data with only segmentation values and no bounding boxes? If yes, how?
", "rawMarkdown": "Can I train mask r-cnn, if I have annotation data with only segmentation values and no bounding boxes? If yes, how? " }, { "id": 315307, "postDate": "2018-04-17T01:04:03.073Z", "content": "Fantastic work! Really helps a lot!
", "rawMarkdown": "Fantastic work! Really helps a lot!" }, { "id": 308496, "postDate": "2018-04-03T15:25:35.443Z", "content": "Has anyone encountered this error before ? Any insight into this error would be greatly appreciated :)
\n\nTraceback (most recent call last):\n File \"submit.py\", line 382, in <module>\n run_npy_to_sumbit_csv()\n File \"submit.py\", line 355, in run_npy_to_sumbit_csv\n all = np.hstack((image, contour_overlay, color1_overlay)).astype(np.uint8)\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/numpy/core/shape_base.py\", line 286, in hstack\n return _nx.concatenate(arrs, 0)\nAhh i found out what was wrong already, I forgot to change my data_dir in the code to my own data_dir. Hopefully this might help anyone encountering this problem as well.
", "rawMarkdown": "Ahh i found out what was wrong already, I forgot to change my data_dir in the code to my own data_dir. Hopefully this might help anyone encountering this problem as well. " } ] }, { "id": 308153, "postDate": "2018-04-03T03:28:36.527Z", "content": "@Heng CherKeng Hi, thanks for the nice work and sharing your code. I am just observing some things and I don't know to exactly interpret them. For instance during training I see a lot of fluctuation on the gpu. It goes to 0% volatility sometimes even for a minute and then back up to 80% - 90%. I presume that might be because you're doing a lot of augmentation and passing values between cpu and gpu but I just wanted to ask if anyone has noticed the same thing?
", "rawMarkdown": "@Heng CherKeng Hi, thanks for the nice work and sharing your code. I am just observing some things and I don't know to exactly interpret them. For instance during training I see a lot of fluctuation on the gpu. It goes to 0% volatility sometimes even for a minute and then back up to 80% - 90%. I presume that might be because you're doing a lot of augmentation and passing values between cpu and gpu but I just wanted to ask if anyone has noticed the same thing?" }, { "id": 307816, "postDate": "2018-04-02T14:26:02.973Z", "content": "Thanks for the code. When I ran evaluate.py or train_0.py, encountered TypeError as shown bellow, I seemed to be stuck there. Any assistance would be greatly appreciated. I have been rebuilt the *.so following the suggestions in the readme.pptx.
\n\n![\"enter](\"http://c/Users/eden/Desktop/error/error1.png\")
Traceback (most recent call last):\n File \"/home/paopao/Competition-Kaggle/Mask-RCNN-nucleus-v0.3.1/evaluate.py\", line 352, in \n run_evaluate()\n File \"/home/paopao/Competition-Kaggle/Mask-RCNN-nucleus-v0.3.1/evaluate.py\", line 161, in run_evaluate\n net(inputs, truth_boxes, truth_labels, truth_instances )\n File \"/home/paopao/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in call\n result = self.forward(*input, **kwargs)\n File \"/home/paopao/Competition-Kaggle/Mask-RCNN-nucleus-v0.3.1/net/resnet50_mask_rcnn/model.py\", line 371, in forward\n self.rpn_proposals = rpn_nms(cfg, mode, inputs, self.rpn_window, self.rpn_logits_flat, self.rpn_deltas_flat)\n File \"/home/paopao/Competition-Kaggle/Mask-RCNN-nucleus-v0.3.1/net/resnet50_mask_rcnn/layer/rpn_multi_nms.py\", line 138, in rpn_nms\n keep = gpu_nms(np.hstack((box, p)), nms_overlap_threshold)\nTypeError: 'module' object is not callable
\n\nI couldn’t load the module of cython_box_overlap and cython_nms also in process.py, which are marked with red underline.
", "rawMarkdown": "Thanks for the code. When I ran evaluate.py or train_0.py, encountered TypeError as shown bellow, I seemed to be stuck there. Any assistance would be greatly appreciated. I have been rebuilt the *.so following the suggestions in the readme.pptx.\n\n![enter image description here][1]\n\nTraceback (most recent call last):\n File \"/home/paopao/Competition-Kaggle/Mask-RCNN-nucleus-v0.3.1/evaluate.py\", line 352, in ", "replies": [ { "id": 309025, "postDate": "2018-04-04T14:06:48.403Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 307602, "postDate": "2018-04-02T05:23:28.963Z", "content": "Thank you for the code and documentation. The loss looks oscillating from the beginning. Although the LB improves from longer training (even though valid loss is similar). Should I decrease the learning rate to 0.001? If so, when?
", "rawMarkdown": "Thank you for the code and documentation. The loss looks oscillating from the beginning. Although the LB improves from longer training (even though valid loss is similar). Should I decrease the learning rate to 0.001? If so, when?", "replies": [ { "id": 307610, "postDate": "2018-04-02T05:40:54.030Z", "content": "you can only know after trial and error.
\n\nif you can get 0.443 (your current LB score) with 53 test images after long training, continue to do so until results does not improve. Do not need to change learning rate ,etc. the validation results sometimes do reflect the LB results due to difference in dataset.
\n\nIf the results does not improve, then you can try different learning rate etc.
", "rawMarkdown": "you can only know after trial and error.\n\nif you can get 0.443 (your current LB score) with 53 test images after long training, continue to do so until results does not improve. Do not need to change learning rate ,etc. the validation results sometimes do reflect the LB results due to difference in dataset.\n\nIf the results does not improve, then you can try different learning rate etc. \n\n", "votes": 2 }, { "id": 307618, "postDate": "2018-04-02T05:54:12.270Z", "content": "Which iter you select ? What' LB score the submission get ?
", "rawMarkdown": " Which iter you select ? What' LB score the submission get ?" }, { "id": 307904, "postDate": "2018-04-02T17:12:23.277Z", "content": "so, the learning rate strategies are we should not change the learning rate until the loss does not decay anymore? this might seems strange. and this network might ignore many nucleis, so how do you solve this problem. actully we solve this problem. however, the false positive rate becomes very high. how can we solve this???
", "rawMarkdown": "so, the learning rate strategies are we should not change the learning rate until the loss does not decay anymore? this might seems strange. and this network might ignore many nucleis, so how do you solve this problem. actully we solve this problem. however, the false positive rate becomes very high. how can we solve this???" }, { "id": 307942, "postDate": "2018-04-02T17:57:45.507Z", "content": "google for \"PVANET: Deep but Lightweight Neural Networks for Real-time Object detection\"
\n\n\"and learning rate scheduling based on plateau detection\"
", "rawMarkdown": "google for \"PVANET: Deep but Lightweight Neural Networks for Real-time Object detection\"\n\n\"and learning rate scheduling based on plateau detection\"" } ] }, { "id": 305889, "postDate": "2018-03-29T15:18:00.473Z", "content": "If I use pytorch0.3.1, can it work for train?
", "rawMarkdown": "If I use pytorch0.3.1, can it work for train?" }, { "id": 305785, "postDate": "2018-03-29T12:04:26.797Z", "rawMarkdown": "" }, { "id": 305497, "postDate": "2018-03-29T00:33:15.337Z", "content": "If using macos, what version of xcode does it need to be to install pytorch 0.4.0?
", "rawMarkdown": "If using macos, what version of xcode does it need to be to install pytorch 0.4.0? " }, { "id": 303951, "postDate": "2018-03-26T20:03:06.510Z", "content": "Do we have to use Cuda 9.1? Will a different version work?
", "rawMarkdown": "Do we have to use Cuda 9.1? Will a different version work? ", "replies": [ { "id": 305113, "postDate": "2018-03-28T13:55:55.893Z", "content": "Have you tried it? Seems that pytorch 0.4 is a must, not sure about the cuda version
", "rawMarkdown": "Have you tried it? Seems that pytorch 0.4 is a must, not sure about the cuda version" }, { "id": 305212, "postDate": "2018-03-28T15:49:52.687Z", "content": "I think it's the cuda version. Floydhub on goes up to .3 and cuda is 9.0. Is there place is the code to change what version of cuda? Maybe in the setup.py in the gpu_nms folder?
", "rawMarkdown": "I think it's the cuda version. Floydhub on goes up to .3 and cuda is 9.0. Is there place is the code to change what version of cuda? Maybe in the setup.py in the gpu_nms folder? " }, { "id": 305338, "postDate": "2018-03-28T18:28:05.457Z", "content": "@ Yan Wang
\n\nI think I figured out the solution. I am still testing to see if this worked or not. However, you have to change the gencode for gpu version which is located in the setup.py (*build/mask-rcnn-resnet50-ver-01.a/net/lib/box/nms/gpu_nms*).
\n\nThe following line of code needs to be change:
\n\nfrom:
\n\n'nvcc': ['-arch=sm_52', '--ptxas-options=-v', '-c', '--compiler-options', \"'-fPIC'\"],\nto:
\n\n'nvcc': ['-arch=sm_70', '--ptxas-options=-v', '-c', '--compiler-options', \"'-fPIC'\"],\nnote: In my case I am using floydhub GPU2 which is Telso V100. I am confirming with floydub if this is correct or not.
\n\nI am currently trying to figure out why I am getting \"ModuleNotFoundError:\" error.
\n\nI found this information on this site: Matching SM architectures (CUDA arch and CUDA gencode) for various NVIDIA cards
\n\nI hope that this works for those who are having a difficult time get Heng's code to work. Also if you are using torch .3, you have add \" \" around file in all the files. Good luck :)
", "rawMarkdown": "@ Yan Wang\n\nI think I figured out the solution. I am still testing to see if this worked or not. However, you have to change the gencode for gpu version which is located in the setup.py (***build/mask-rcnn-resnet50-ver-01.a/net/lib/box/nms/gpu_nms***). \n\nThe following line of code needs to be change:\n\n**from:**\n\n 'nvcc': ['-arch=sm_52', '--ptxas-options=-v', '-c', '--compiler-options', \"'-fPIC'\"],\n\n**to:** \n\n 'nvcc': ['-arch=sm_70', '--ptxas-options=-v', '-c', '--compiler-options', \"'-fPIC'\"],\n**note:** In my case I am using floydhub GPU2 which is Telso V100. I am confirming with floydub if this is correct or not. \n\nI am currently trying to figure out why I am getting \"ModuleNotFoundError:\" error. \n\nI found this information on this site: [Matching SM architectures (CUDA arch and CUDA gencode) for various NVIDIA cards][1]\n\n\n [1]: http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/\n\nI hope that this works for those who are having a difficult time get Heng's code to work. Also if you are using torch .3, you have add \" \" around __file__ in all the files. Good luck :)" } ] }, { "id": 303406, "postDate": "2018-03-26T05:56:50.853Z", "content": "Hello, Nice Codes! I had a question, when I run process.py there seem an error with torch_nms(gpu_nms and cython_nms work well), I attached the code below, can you show me some clues on solving this problem? Thx.
\n\nprocess.py: calling main function ...\ngpu_nms : [5, 55, 37, 27, 22, 38, 35, 33, 20, 54, 44, 2, 50, 30, 0, 34, 21, 23, 28, 39, 43, 51, 3, 10, 16, 56, 8, 32, 45, 13, 52, 19, 36, 6, 29, 12, 9, 31, 41, 40, 25, 17, 24, 4, 53, 11, 14, 42, 46, 1, 15, 49, 47, 7, 48, 26, 18]\ncython_nms : [5, 55, 37, 27, 35, 20, 0, 45, 52, 19, 29, 24, 4, 11]\nTHCudaCheck FAIL file=/data_1/pytorch/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu line=386 error=8 : invalid device function\nTraceback (most recent call last):\n File \"process.py\", line 341, in \n run_check_nms()\n File \"process.py\", line 330, in run_check_nms\n keep = torch_nms(rois, 0.5)\n File \"/data_1/2018_bowl/build/mask-rcnn-resnet50-ver-01.a/net/lib/box/nms/torch_nms/init.py\", line 34, in torch_nms\n areas = (x2 - x1 + 1) * (y2 - y1 + 1)\nRuntimeError: cuda runtime error (8) : invalid device function at /data_1/pytorch/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:386
", "rawMarkdown": "Hello, Nice Codes! I had a question, when I run process.py there seem an error with torch_nms(gpu_nms and cython_nms work well), I attached the code below, can you show me some clues on solving this problem? Thx.\n\n\nprocess.py: calling main function ...\ngpu_nms : [5, 55, 37, 27, 22, 38, 35, 33, 20, 54, 44, 2, 50, 30, 0, 34, 21, 23, 28, 39, 43, 51, 3, 10, 16, 56, 8, 32, 45, 13, 52, 19, 36, 6, 29, 12, 9, 31, 41, 40, 25, 17, 24, 4, 53, 11, 14, 42, 46, 1, 15, 49, 47, 7, 48, 26, 18]\ncython_nms : [5, 55, 37, 27, 35, 20, 0, 45, 52, 19, 29, 24, 4, 11]\nTHCudaCheck FAIL file=/data_1/pytorch/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu line=386 error=8 : invalid device function\nTraceback (most recent call last):\n File \"process.py\", line 341, in " }, { "id": 303306, "postDate": "2018-03-26T01:17:10.420Z", "content": "Thanks Heng for the wonderful open-source project. From Readme, I see that you have to build *.so files, which from my limited experience, I think can only work in linux systems. Is there a workaround for the code to work in Mac os? Thanks.
", "rawMarkdown": "Thanks Heng for the wonderful open-source project. From Readme, I see that you have to build *.so files, which from my limited experience, I think can only work in linux systems. Is there a workaround for the code to work in Mac os? Thanks. " }, { "id": 302647, "postDate": "2018-03-24T14:20:16.007Z", "content": "Has anyone run into the issue below:
\n\n\nan illegal memory access was encountered\nTHCudaCHeck FAIL \nRuntimeError: cuda runtime error (77) : an illegal memory access was encountered at pytorch/aten/src/THC/generic/THCTensorcopy.c:20\n
Somehow the first epochs work fine and this error pops up
", "rawMarkdown": "Has anyone run into the issue below: \n\n```\nan illegal memory access was encountered\nTHCudaCHeck FAIL \nRuntimeError: cuda runtime error (77) : an illegal memory access was encountered at pytorch/aten/src/THC/generic/THCTensorcopy.c:20\n```\n\nSomehow the first epochs work fine and this error pops up", "replies": [ { "id": 303051, "postDate": "2018-03-25T11:34:32.547Z", "content": "I think I solved the issue, if you get something similar to this then just decrease batch size to 12 and it should run perfectly fine (note: I'm using a P100 instance on Google Cloud, with single GPU, so 12 should fit within a single 12 GB gpu)
", "rawMarkdown": "I think I solved the issue, if you get something similar to this then just decrease batch size to 12 and it should run perfectly fine (note: I'm using a P100 instance on Google Cloud, with single GPU, so 12 should fit within a single 12 GB gpu)" } ] }, { "id": 301559, "postDate": "2018-03-22T23:26:10.873Z", "content": "Hi All, I'm trying to implement Heng's 20180306 code, but met the error :
\nFile \"~/20180306_O/build/net/lib/roi_align_pool_tf/function.py\", line 21, in forward\n crop_and_resize_gpu_forward(\nNameError: name 'crop_and_resize_gpu_forward' is not defined
I'm using GeForce GTX 1080 and tried sm_52 and sm_61. Thank you for your reply.
", "rawMarkdown": "Hi All, I'm trying to implement Heng's 20180306 code, but met the error : \nFile \"~/20180306_O/build/net/lib/roi_align_pool_tf/function.py\", line 21, in forward\n crop_and_resize_gpu_forward(\nNameError: name 'crop_and_resize_gpu_forward' is not defined\n\nI'm using GeForce GTX 1080 and tried sm_52 and sm_61. Thank you for your reply.", "replies": [ { "id": 302417, "postDate": "2018-03-24T02:48:22.050Z", "content": "did you follow the instructions given in his readme file ? building the c wrapper/cython code
", "rawMarkdown": "did you follow the instructions given in his readme file ? building the c wrapper/cython code" } ] }, { "id": 301493, "postDate": "2018-03-22T20:27:06.397Z", "content": "I am trying to implement Heng's code using Floydhub and encounter this error when running common.py. Has anyone try implement with cloud service platforms (i.e aws, google, colab, and etc)? \n ---------------------------------------------------------------------------\n AssertionError Traceback (most recent call last)\n <ipython-input-3-8a272e0e0f88> in <module>()\n102 \n103 print ('\\t\\ttorch.cuda.device_count() =', torch.cuda.device_count())\n --> 104 print ('\\t\\ttorch.cuda.current_device() =', torch.cuda.current_device())\n105 \n106 \n\n /usr/local/lib/python3.6/site-packages/torch/cuda/__init__.py in current_device()\n281 def current_device():\n282 \"\"\"Returns the index of a currently selected device.\"\"\"\n --> 283 _lazy_init()\n284 return torch._C._cuda_getDevice()\n285 \n\n/usr/local/lib/python3.6/site-packages/torch/cuda/__init__.py in _lazy_init()\n118 raise RuntimeError(\n119 \"Cannot re-initialize CUDA in forked subprocess. \" + msg)\n --> 120 _check_driver()\n121 torch._C._cuda_init()\n122 torch._C._cuda_sparse_init()\n\n /usr/local/lib/python3.6/site-packages/torch/cuda/__init__.py in _check_driver()\n 60 Found no NVIDIA driver on your system. Please check that you\n 61 have an NVIDIA GPU and installed a driver from\n ---> 62 http://www.nvidia.com/Download/index.aspx\"\"\")\n 63 else:\n 64 # TODO: directly link to the alternative bin that needs install\n\nAssertionError: \nFound no NVIDIA driver on your system. Please check that you\nhave an NVIDIA GPU and installed a driver from\nhttp://www.nvidia.com/Download/index.aspx\nI was able solve the assertion error. Now, I am trying to figure out how to build *.so file. A little bit new to pytorch.
", "rawMarkdown": "I was able solve the assertion error. Now, I am trying to figure out how to build *.so file. A little bit new to pytorch. " }, { "id": 301648, "postDate": "2018-03-23T03:20:29.483Z", "content": "@William Green, how did you setup Heng's code on Floydhub? I want to do the same and have not used pytorch yet. Any step by step will be appreciated.
", "rawMarkdown": "@William Green, how did you setup Heng's code on Floydhub? I want to do the same and have not used pytorch yet. Any step by step will be appreciated." }, { "id": 301654, "postDate": "2018-03-23T03:38:11.543Z", "content": "@YaGana,\nI was only able to get common.py to run by placing \" \" around file (ex. \"file\"). I'm stuck on creating the .so files. Also , I am using the 0.3.0 version. Common.py will have error message if you are not using gpu.
", "rawMarkdown": "@YaGana,\nI was only able to get common.py to run by placing \" \" around _file_ (ex. \"_file_\"). I'm stuck on creating the .so files. Also , I am using the 0.3.0 version. Common.py will have error message if you are not using gpu.", "votes": 1 }, { "id": 301660, "postDate": "2018-03-23T03:50:34.840Z", "content": "Thanks @William Green. Let me know when you are able to get Heng's code working on Floydhub. I am currently using the matterport RCNN. Will try Heng's when I am done with this one.
", "rawMarkdown": "Thanks @William Green. Let me know when you are able to get Heng's code working on Floydhub. I am currently using the matterport RCNN. Will try Heng's when I am done with this one." }, { "id": 301673, "postDate": "2018-03-23T04:34:59.623Z", "content": "@WilliamGreen and @YaGana, I was able to compile the Heng code on Google Cloud Compute (GCP). I followed the steps in the readme in the 03062018 package. For the .so i believe you need to compile it and readme.pdf has the steps. In the weekend i will try and write down all the steps I had to do to get it running. The reason I pick GCP is that they give $300 credit free to all Kaggle users (I had to make a deposit of $35 to get access to 1 K80 gpu). I will also post the results I got so far from the different matterport versions. i am still stuck in 0.415-0.418 range. The learning rate seems to make a big impact and also initialization. with same code and same number of iterations i got 0.4 while some others got 0.44.
", "rawMarkdown": "@WilliamGreen and @YaGana, I was able to compile the Heng code on Google Cloud Compute (GCP). I followed the steps in the readme in the 03062018 package. For the .so i believe you need to compile it and readme.pdf has the steps. In the weekend i will try and write down all the steps I had to do to get it running. The reason I pick GCP is that they give $300 credit free to all Kaggle users (I had to make a deposit of $35 to get access to 1 K80 gpu). I will also post the results I got so far from the different matterport versions. i am still stuck in 0.415-0.418 range. The learning rate seems to make a big impact and also initialization. with same code and same number of iterations i got 0.4 while some others got 0.44." }, { "id": 301751, "postDate": "2018-03-23T07:38:53.790Z", "content": "@Mehul, thank you.
", "rawMarkdown": "@Mehul, thank you." }, { "id": 303721, "postDate": "2018-03-26T15:31:29.803Z", "content": "@YaGana Sheriff-Hussaini\nI made progress. I was able to create the *.so files. I am currently trying to \"check mns, see \"box/process.py\" run_check_nms()\". Once I get it to work, I'll type up the instructions so it is a little easier to follow.
", "rawMarkdown": "@YaGana Sheriff-Hussaini\nI made progress. I was able to create the *.so files. I am currently trying to \"check mns, see \"box/process.py\" run_check_nms()\". Once I get it to work, I'll type up the instructions so it is a little easier to follow. \n", "votes": 1 }, { "id": 303738, "postDate": "2018-03-26T15:50:10.007Z", "content": "@William, i am curious about speed for the Heng code on your side. the processing show 1k, 2k, 3k .. etc for me on K80 it took 1hr per 1k; so this means you have to wait 20hr for 20k. curious to see if it is faster on floydhub
", "rawMarkdown": "@William, i am curious about speed for the Heng code on your side. the processing show 1k, 2k, 3k .. etc for me on K80 it took 1hr per 1k; so this means you have to wait 20hr for 20k. curious to see if it is faster on floydhub" }, { "id": 303755, "postDate": "2018-03-26T16:15:10.913Z", "content": "@Mehul Samput \nFloydhub has GPU and GPU2. GPU is K80 and GPU2 is V100. The V100 will be faster, but not much faster than K80. It may shave down 2-3 hrs. Just my guess. I am trying to get the code to work now in floydhub and will let you know how long it takes.
\n\nI tried Yang's code in Keras and GPU2 is 5 hrs.
", "rawMarkdown": "@Mehul Samput \nFloydhub has GPU and GPU2. GPU is K80 and GPU2 is V100. The V100 will be faster, but not much faster than K80. It may shave down 2-3 hrs. Just my guess. I am trying to get the code to work now in floydhub and will let you know how long it takes. \n\nI tried Yang's code in Keras and GPU2 is 5 hrs. " }, { "id": 303758, "postDate": "2018-03-26T16:16:38.243Z", "content": "@Mehul
\n\nDid you get a chance to write the instructions for running Heng's code in GCP?
", "rawMarkdown": "@Mehul\n\nDid you get a chance to write the instructions for running Heng's code in GCP? " }, { "id": 303766, "postDate": "2018-03-26T16:27:07.973Z", "content": "Thanks @William Green, I look forward to your step by step on Floydhub. I have a windows 10 machine, what is your environment? I usually set up my code locally, then upload it to floydhub, upload the data, and then run the code in the cloud.
", "rawMarkdown": "Thanks @William Green, I look forward to your step by step on Floydhub. I have a windows 10 machine, what is your environment? I usually set up my code locally, then upload it to floydhub, upload the data, and then run the code in the cloud." }, { "id": 303787, "postDate": "2018-03-26T16:57:38.810Z", "content": "@ YaGana Sheriff-Hussaini
\n\nWelcome! I'm on windows 10. I went forward and setup on Floydhub vice locally. I just tried to run \ntrain_0.py and received import error.
\n\n---------------------------------------------------------------------------\n ImportError Traceback (most recent call last)\n <ipython-input-6-ac33b71eee94> in <module>()\n 4 from common import *\n 5 from utility.file import *\n ----> 6 from dataset.reader import *\n 7 from net.rate import *\n 8 from net.metric import *\n\n /output/build/mask-rcnn-resnet50-ver-01.a/dataset/reader.py in <module>()\n 6 from utility.draw import *\n 7 \n ----> 8 from net.lib.box.process import *\n 9 \n 10 #data reader ----------------------------------------------------------------\n\n /output/build/mask-rcnn-resnet50-ver-01.a/net/lib/box/process.py in <module>()\n 2 from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\n 3 from net.lib.box.nms.torch_nms import torch_nms\n ----> 4 from net.lib.box.nms.gpu_nms import gpu_nms\n 5 from net.lib.box.nms.cython_nms import cython_nms\n 6 \n\n ImportError: libcudart.so.9.1: cannot open shared object file: No such file or directory\nI see all the *.so files in the correct folder and subfolder locations. I have to figure out why I'm getting this error.
", "rawMarkdown": "@ YaGana Sheriff-Hussaini\n\nWelcome! I'm on windows 10. I went forward and setup on Floydhub vice locally. I just tried to run \ntrain_0.py and received import error. \n\n ---------------------------------------------------------------------------\n ImportError Traceback (most recent call last)\n " }, { "id": 304007, "postDate": "2018-03-26T21:49:28.317Z", "content": "@YaGana
\n\nI am able to get the *.so file. However, I cannot get past the error that I posted. In order to build *.so files in floydhub you have to do the following:
\n\nYou have to point to torch_nms in the terminal and run >>python build. That will create the _extension.so file. Do that for each step. Point to the folder and run the command line.
\n\nLet me know if you are able to get the code to run. I can be reached at dskswu@gmail.com if you need additional assistance. \nI'm not sure if you it's required to have cuda 9.1 or not. Hopefully, someone can assist and respond.
", "rawMarkdown": "@YaGana \n\nI am able to get the *.so file. However, I cannot get past the error that I posted. In order to build *.so files in floydhub you have to do the following:\n\nYou have to point to torch_nms in the terminal and run >>python build. That will create the _extension.so file. Do that for each step. Point to the folder and run the command line. \n\nLet me know if you are able to get the code to run. I can be reached at dskswu@gmail.com if you need additional assistance. \nI'm not sure if you it's required to have cuda 9.1 or not. Hopefully, someone can assist and respond. " }, { "id": 304023, "postDate": "2018-03-26T22:25:00.177Z", "content": "Thanks @William Green, I am going to try out Hengs code next weekend so will get in touch if I need any help by then. I have not looked at his code at all at this point but my suggestion for debugging the error your posted above. Take a look at the dataset script and or reader function to see if there is a pace where the data directory is set or pointed to. The error you posted seems to be looking for a file/directory it cannot find. Let me know if that helped.
", "rawMarkdown": "Thanks @William Green, I am going to try out Hengs code next weekend so will get in touch if I need any help by then. I have not looked at his code at all at this point but my suggestion for debugging the error your posted above. Take a look at the dataset script and or reader function to see if there is a pace where the data directory is set or pointed to. The error you posted seems to be looking for a file/directory it cannot find. Let me know if that helped." } ] }, { "id": 301286, "postDate": "2018-03-22T15:17:04.893Z", "content": "@Heng \nDo we have to use pytorch 0.4.0 to run your code? Will this work with 0.3.0?
", "rawMarkdown": "@Heng \nDo we have to use pytorch 0.4.0 to run your code? Will this work with 0.3.0?" }, { "id": 301158, "postDate": "2018-03-22T12:04:13.593Z", "content": "If anyone gets stuck while training the dummy model on the disk0_ids_dummy_9 data, what you'll see is training seems to stall on the first update and when you ctrl c it is consistently stuck on the data loader. To fix this simply decrease batch_size = 4 in train_0.py like Heng did here since there are only 9 images in the dataset we cannot have 16 > 9 batch_size.
", "rawMarkdown": "If anyone gets stuck while training the dummy model on the disk0_ids_dummy_9 data, what you'll see is training seems to stall on the first update and when you ctrl c it is consistently stuck on the data loader. To fix this simply decrease batch_size = 4 in train_0.py like Heng did [here][1] since there are only 9 images in the dataset we cannot have 16 > 9 batch_size.\n\n\n [1]: https://drive.google.com/drive/folders/1Dd7Mwn-VYkf1uqrvNqC2GM5BVwKlGwE8" }, { "id": 295274, "postDate": "2018-03-13T11:56:20.547Z", "content": "Hi, how can I retrain the model, I can not find 00014500_model.pth and 00014500_optimizer.pth.\nThx in advance
", "rawMarkdown": "Hi, how can I retrain the model, I can not find 00014500_model.pth and 00014500_optimizer.pth.\nThx in advance", "replies": [ { "id": 295700, "postDate": "2018-03-14T03:07:37.070Z", "content": "Start from scratch. Set to initial checkpoint to None
", "rawMarkdown": "Start from scratch. Set to initial checkpoint to None", "votes": 2 }, { "id": 295877, "postDate": "2018-03-14T10:56:11.713Z", "content": "THX，but training from scratch will be easily overfitting.
", "rawMarkdown": "THX，but training from scratch will be easily overfitting." } ] }, { "id": 295199, "postDate": "2018-03-13T08:42:52.650Z", "content": "Hi, why 'test1_ids_gray_only_53' and 'test1_ids_all_65' get the same LB score 0.419?\nI have already changed the test file
\n\n\n test_dataset = ScienceDataset(\n 'test1_ids_all_65', mode='test',\n transform = submit_augment)\n
i got an error like this. Can't get attribute '_rebuild_tensor_v2' on module 'torch._utils' from '/home/k/anaconda3/lib/python3.6/site-packages/torch/_utils.py'
", "rawMarkdown": "i got an error like this. Can't get attribute '_rebuild_tensor_v2' on module 'torch._utils' from '/home/k/anaconda3/lib/python3.6/site-packages/torch/_utils.py'", "replies": [ { "id": 303795, "postDate": "2018-03-26T17:09:34.310Z", "content": "Are you solve this problem? I meet the the same problem with you. I not sure if it's because I am not use pytorch v4.0. What pytorch version are you use?
", "rawMarkdown": "Are you solve this problem? I meet the the same problem with you. I not sure if it's because I am not use pytorch v4.0. What pytorch version are you use?" }, { "id": 303871, "postDate": "2018-03-26T18:28:51.550Z", "content": "I had solved the problem. Just update pytorch to v4.0.
", "rawMarkdown": "I had solved the problem. Just update pytorch to v4.0." }, { "id": 307388, "postDate": "2018-04-01T14:01:06.287Z", "content": "can you tell me how to update pytorch to v4.0?
", "rawMarkdown": "can you tell me how to update pytorch to v4.0?" }, { "id": 307390, "postDate": "2018-04-01T14:10:16.817Z", "content": "I use pytorch v3.0 .
", "rawMarkdown": "I use pytorch v3.0 . " }, { "id": 307770, "postDate": "2018-04-02T12:15:35.763Z", "content": "Use source code to install pytorch v4.0.
", "rawMarkdown": "Use source code to install pytorch v4.0." } ] }, { "id": 294480, "postDate": "2018-03-12T02:21:50.987Z", "content": "Hi Heng:
\n\nplease provide requirements.txt for your google folder 20180306 conda setup, currently the following error occurred when execute submit.py (train.py was executed without error)
\n\n\"index = (net.detections[:,0]==b).nonzero().view(-1) IndexError: too many indices for tensor of dimension 1\"
", "rawMarkdown": "Hi Heng:\n\nplease provide requirements.txt for your google folder 20180306 conda setup, currently the following error occurred when execute submit.py (train.py was executed without error)\n \n\"index = (net.detections[:,0]==b).nonzero().view(-1) IndexError: too many indices for tensor of dimension 1\"\n", "replies": [ { "id": 294484, "postDate": "2018-03-12T02:27:53.050Z", "content": "You should print out the size of detections. Or you can use pycharm visual debugger to set a breakpoint at that line of code and inspect the values.
\n\nI suspect it is zero dim , i.e no detection at all
", "rawMarkdown": "You should print out the size of detections. Or you can use pycharm visual debugger to set a breakpoint at that line of code and inspect the values.\n\nI suspect it is zero dim , i.e no detection at all", "votes": 1 }, { "id": 294489, "postDate": "2018-03-12T02:29:26.463Z", "content": "For this bug, just check the dimension of output mask. If it is not equal to input image, then ignore it.
", "rawMarkdown": "For this bug, just check the dimension of output mask. If it is not equal to input image, then ignore it. " } ] }, { "id": 294437, "postDate": "2018-03-12T00:40:57.730Z", "content": "I used Detectron's mask-rcnn but it seems have serious problem, It marked signal nuclei multiple times which leads a submission error: one pixel should not belong to different objects. It confusing me a lot. Little help please.
", "rawMarkdown": "I used Detectron's mask-rcnn but it seems have serious problem, It marked signal nuclei multiple times which leads a submission error: one pixel should not belong to different objects. It confusing me a lot. Little help please." }, { "id": 293382, "postDate": "2018-03-09T18:54:14.697Z", "content": "Thanks for the code, there is so much to learn.\nProbably I am doing something very silly, but when I run train0, it keeps running w/o any error and CPU utilization is 100% but GPU is completely free. Any idea whats going wrong.
", "rawMarkdown": "Thanks for the code, there is so much to learn.\nProbably I am doing something very silly, but when I run train0, it keeps running w/o any error and CPU utilization is 100% but GPU is completely free. Any idea whats going wrong.", "replies": [ { "id": 293742, "postDate": "2018-03-10T13:51:33.407Z", "content": "You may want to check the path given to cv2.imread
", "rawMarkdown": "You may want to check the path given to cv2.imread" }, { "id": 293776, "postDate": "2018-03-10T15:34:40.333Z", "content": "thanks, where should I look for that?\nFYI - trying the sample first and run_check_nms() is giving the right output
", "rawMarkdown": "thanks, where should I look for that?\nFYI - trying the sample first and run_check_nms() is giving the right output" }, { "id": 293809, "postDate": "2018-03-10T17:04:54.080Z", "content": "I guess your code is stuck in some while loop in train0, which is caused by len(some data loader)==0.
the batch_size must be smaller than len(some data loader)
", "rawMarkdown": "the batch_size must be smaller than len(some data loader)", "votes": 1 }, { "id": 296136, "postDate": "2018-03-14T19:14:35.107Z", "content": "Thanks Yan & Heng, it worked.\nwhile Submit I am getting below error\n File \"submit.py\", line 361, in \n run_npy_to_sumbit_csv()\n File \"submit.py\", line 334, in run_npy_to_sumbit_csv\n all = np.hstack((image, contour_overlay, color1_overlay)).astype(np.uint8)\n File \"/home/Py3.6-ENV/lib/python3.6/site-packages/numpy/core/shape_base.py\", line 286, in hstack\n return _nx.concatenate(arrs, 0)\nValueError: all the input arrays must have same number of dimensions
", "rawMarkdown": "Thanks Yan & Heng, it worked.\nwhile Submit I am getting below error\n File \"submit.py\", line 361, in " }, { "id": 312940, "postDate": "2018-04-12T14:58:14.687Z", "content": "@MayankKant I got a similar problem running submit.py for the stage 2 test set, did you manage to solve your problem and if so how did you do it ?
", "rawMarkdown": "@MayankKant I got a similar problem running submit.py for the stage 2 test set, did you manage to solve your problem and if so how did you do it ? " } ] }, { "id": 292889, "postDate": "2018-03-08T20:57:04.347Z", "content": "Hi Heng,
\n\nThank you for your code. I use 14.7, but when I try to train the model, it said cuda runtime error (2) : out of memory. I found the memory usage of GPU increases as the epochs goes. It goes up and up. What is wrong with it?
\n\nAnother question is how to avoid truth_labels is None? Thank you very much.
", "rawMarkdown": "Hi Heng,\n\nThank you for your code. I use 14.7, but when I try to train the model, it said cuda runtime error (2) : out of memory. I found the memory usage of GPU increases as the epochs goes. It goes up and up. What is wrong with it? \n\nAnother question is how to avoid truth_labels is None? Thank you very much.", "replies": [ { "id": 292899, "postDate": "2018-03-08T21:12:44.340Z", "content": "The gradient takes up memory along with other things. You either need to rescale the pictures or lower the batch size.
", "rawMarkdown": "The gradient takes up memory along with other things. You either need to rescale the pictures or lower the batch size." }, { "id": 292959, "postDate": "2018-03-09T00:12:28.213Z", "content": "suggest you use the latest version. this has been more vigorously tested and the truth labels is None should be taken care of.
", "rawMarkdown": "suggest you use the latest version. this has been more vigorously tested and the truth labels is None should be taken care of. \n\n", "votes": 2 }, { "id": 295105, "postDate": "2018-03-13T04:15:11.877Z", "content": "Thanks , the newest one is nice. I have 3 questions.
\n\nI saw in your training log, you use single-shot-resnet34-fpn, so you use single-shot instead of rcnn?
Your training sample is 500 and valid sample 43. Why don't you use all the samples. Which sample do you delete?
When I try to inference with model which training by myself. There is an error. I use mask-rcnn-resnet50-fpn.
File \"submit.py\", line 375, in \n run_submit()\n File \"submit.py\", line 213, in run_submit\n revert(net, images) #unpad, undo test-time augment etc ....\n File \"submit.py\", line 110, in revert\n index = (net.detections[:,0]==b).nonzero().view(-1)\nIndexError: too many indices for tensor of dimension 1
", "rawMarkdown": "Thanks , the newest one is nice. I have 3 questions.\n\n1. I saw in your training log, you use single-shot-resnet34-fpn, so you use single-shot instead of rcnn?\n\n2. Your training sample is 500 and valid sample 43. Why don't you use all the samples. Which sample do you delete?\n\n3. When I try to inference with model which training by myself. There is an error. I use mask-rcnn-resnet50-fpn.\n\nFile \"submit.py\", line 375, in " }, { "id": 296754, "postDate": "2018-03-15T18:00:37.430Z", "content": "\"single-shot-resnet34-fpn,\" is typo error.\nThe log files in google grive is resnet50-fpn
", "rawMarkdown": "\"single-shot-resnet34-fpn,\" is typo error.\nThe log files in google grive is resnet50-fpn" } ] }, { "id": 291904, "postDate": "2018-03-07T04:03:41.317Z", "content": "Very helpful, appreciate your sharing!
", "rawMarkdown": "Very helpful, appreciate your sharing!" }, { "id": 291880, "postDate": "2018-03-07T02:54:17.573Z", "content": "Thanks for your latest code! It's very clear and fast. And the results is perfect!
", "rawMarkdown": "Thanks for your latest code! It's very clear and fast. And the results is perfect!", "replies": [ { "id": 292389, "postDate": "2018-03-07T21:59:09.833Z", "content": "Hi, can you reveal which code you use to obtain 0.504?
", "rawMarkdown": "Hi, can you reveal which code you use to obtain 0.504?", "votes": 1 }, { "id": 292453, "postDate": "2018-03-08T01:20:15.847Z", "content": "Use his latest code is enough.
", "rawMarkdown": "Use his latest code is enough." }, { "id": 292974, "postDate": "2018-03-09T00:53:51.073Z", "content": "In the latest code, when i run the box/process.py run_check_nms() function, i get a strange error
\n\nfor gpu_nms i get : [ 5, 55, 37, 27, 22, 38, 35, 33, 20, 54, 44, 2, 50, 30, 0, 34, 21, 23, 28, 39, 43, 51, 3, 10, 16, 56, 8, 32, 45, 13, 52, 19, 36, 6, 29, 12, 9, 31, 41].... which is different than what is expected\nfor cython_nms i get : [ 5, 55, 37, 27, 35, 20, 0, 45, 52, 19, 29, 24, 4, 11].... which is what is expected
\n\nthen i get a error THcudacheck Fail file = /home/mldl/pytorch/aten/arc/THC/generic/THCTensorCopy.c line=20 erro=78 : a ptx jit compilation failed\nros = torch.from_numpy(rois).cuda()\nRuntimeError : cuda runtime error ( 78) : a ptx jit compilation failed a /home/mldl/pytorch/aten/src/THC/generic/THCTensorcopy.c :20
\n\nAny pointers will be much appreciated
", "rawMarkdown": "In the latest code, when i run the box/process.py run_check_nms() function, i get a strange error \n\nfor gpu_nms i get : [ 5, 55, 37, 27, 22, 38, 35, 33, 20, 54, 44, 2, 50, 30, 0, 34, 21, 23, 28, 39, 43, 51, 3, 10, 16, 56, 8, 32, 45, 13, 52, 19, 36, 6, 29, 12, 9, 31, 41].... which is different than what is expected\nfor cython_nms i get : [ 5, 55, 37, 27, 35, 20, 0, 45, 52, 19, 29, 24, 4, 11].... which is what is expected\n\nthen i get a error THcudacheck Fail file = /home/mldl/pytorch/aten/arc/THC/generic/THCTensorCopy.c line=20 erro=78 : a ptx jit compilation failed\nros = torch.from_numpy(rois).cuda()\nRuntimeError : cuda runtime error ( 78) : a ptx jit compilation failed a /home/mldl/pytorch/aten/src/THC/generic/THCTensorcopy.c :20\n\nAny pointers will be much appreciated\n" }, { "id": 292993, "postDate": "2018-03-09T02:08:00.507Z", "content": "which gpu are you using? I meet this kind of error when I use K80
\n\nUpdate: just make it right for the process.py results. Now checking the training of entire network.\nBasically, I suggest you to delete all the .so files and rebuild them. Change arch=sm_xx to proper setting, in all setup.py or makefile, and run them again. At last, pay attention to the two .so files in the nms directory.\nMake a backup for the entire folder.
", "rawMarkdown": "which gpu are you using? I meet this kind of error when I use K80\n\nUpdate: just make it right for the process.py results. Now checking the training of entire network.\nBasically, I suggest you to delete all the .so files and rebuild them. Change arch=sm_xx to proper setting, in all setup.py or makefile, and run them again. At last, pay attention to the two .so files in the nms directory.\nMake a backup for the entire folder." }, { "id": 293549, "postDate": "2018-03-10T05:07:13.880Z", "content": "Thank you Chen for the suggestion. I have deleted all the .so files and rebuilt them. I have also moved to python 3.6 as opposed to 3.5. It seems to have no effect. I have also tried using the libraries which came packaged as part of the build but the error remains. I have two questions for you\n1. Can you please elaborate more on arch=sm_xx, what should i change?\n2. Also one thing which i have observed that when i import \nthe statement from net.lib.box.nms import gpu_nms it does not work. I get the error that you cannot call module directly.\nI need to put it as\nfrom net.lib.box.nms.gpu_nms.gpu_nms import *
\n\nHave you observed some thing similiar?
", "rawMarkdown": "Thank you Chen for the suggestion. I have deleted all the .so files and rebuilt them. I have also moved to python 3.6 as opposed to 3.5. It seems to have no effect. I have also tried using the libraries which came packaged as part of the build but the error remains. I have two questions for you\n1. Can you please elaborate more on arch=sm_xx, what should i change?\n2. Also one thing which i have observed that when i import \nthe statement from net.lib.box.nms import gpu_nms it does not work. I get the error that you cannot call module directly.\nI need to put it as\nfrom net.lib.box.nms.gpu_nms.gpu_nms import * \n\nHave you observed some thing similiar? \n\n" }, { "id": 293553, "postDate": "2018-03-10T05:28:07.327Z", "content": "Again let me make sure, which gpu are you using?
", "rawMarkdown": "Again let me make sure, which gpu are you using?" }, { "id": 293871, "postDate": "2018-03-10T19:24:34.837Z", "content": "It is GeForce 1080
", "rawMarkdown": "It is GeForce 1080" }, { "id": 294119, "postDate": "2018-03-11T10:21:14.047Z", "content": "Then that might be a different story. For me, arch option of K80 should be sm_30 or sm_37. When I changed that and rebuild all, it works. 1080 should be ok for sm_52. My last advice would be, remember all the .so files, delete them, and build all. Some of them may be copied to certain location different from the dir they are created.
", "rawMarkdown": "Then that might be a different story. For me, arch option of K80 should be sm_30 or sm_37. When I changed that and rebuild all, it works. 1080 should be ok for sm_52. My last advice would be, remember all the .so files, delete them, and build all. Some of them may be copied to certain location different from the dir they are created. " } ] }, { "id": 290811, "postDate": "2018-03-05T01:10:35.520Z", "content": "I generated the submission for the rcnn, but I ran into this error trying to submit to Kaggle: Evaluation Exception: The same pixel may not be assigned to two different objects.. Has anyone else run into this issue?
Yes, I believe it happens when two masks overlap to an identical pixel.
", "rawMarkdown": "Yes, I believe it happens when two masks overlap to an identical pixel.", "votes": 1 }, { "id": 290822, "postDate": "2018-03-05T01:48:17.243Z", "content": "Did you fix it by just modifying the submission file?
", "rawMarkdown": "Did you fix it by just modifying the submission file?" }, { "id": 290823, "postDate": "2018-03-05T01:51:40.253Z", "content": "Yes, I wrote a filter to resolve mask intersections before writing the submission file. That made the problem go away.
", "rawMarkdown": "Yes, I wrote a filter to resolve mask intersections before writing the submission file. That made the problem go away. ", "votes": 1 }, { "id": 290832, "postDate": "2018-03-05T02:33:24.213Z", "content": "How does that work? Shouldn't the RLE script encode each pixel to a single nuclei if I'm using a 256x256 numpy array?
", "rawMarkdown": "How does that work? Shouldn't the RLE script encode each pixel to a single nuclei if I'm using a 256x256 numpy array?" }, { "id": 290838, "postDate": "2018-03-05T02:52:04.033Z", "content": "I just did something dumb to have a valid submission file. 1) I compute the union of all masks and then 2) I go one mask at a time and make sure I don't cover the union twice by deducting the intersection with the current union and then assigning the new union. I've seen others that reported the same issue. They did something more intelligent. In the second step they sort the masks by size or reliability first.
\n\nWhat I'm not sure is if mask-rcnn is supposed to deliver masks that they are already cleaned up but somehow it is getting messed because I do some extra zooming. I did scaling of the images (shortage of GPU memory) and I won't be surprised edges end up colliding when expanding masks back.
\n\nI didn't look at the RLE script to see if it takes care of this issue. I was just assuming it was encoding each mask individually but you have a good point. I think I should look if the script is also supposed to resolve collisions (although that might be suboptimal). I liked the sorting/ranking idea.
", "rawMarkdown": "I just did something dumb to have a valid submission file. 1) I compute the union of all masks and then 2) I go one mask at a time and make sure I don't cover the union twice by deducting the intersection with the current union and then assigning the new union. I've seen others that reported the same issue. They did something more intelligent. In the second step they sort the masks by size or reliability first. \n\nWhat I'm not sure is if mask-rcnn is supposed to deliver masks that they are already cleaned up but somehow it is getting messed because I do some extra zooming. I did scaling of the images (shortage of GPU memory) and I won't be surprised edges end up colliding when expanding masks back. \n\nI didn't look at the RLE script to see if it takes care of this issue. I was just assuming it was encoding each mask individually but you have a good point. I think I should look if the script is also supposed to resolve collisions (although that might be suboptimal). I liked the sorting/ranking idea.", "votes": 1 }, { "id": 291082, "postDate": "2018-03-05T15:45:31.537Z", "content": "Thanks Oscar! I'm not sure how I would get a collision if the expanded mask assigns each pixel in the test image to a nuclei, but I'll probably have to look at the RLE script. I haven't actually fully trained the model yet; I'm still debugging. ;)
", "rawMarkdown": "Thanks Oscar! I'm not sure how I would get a collision if the expanded mask assigns each pixel in the test image to a nuclei, but I'll probably have to look at the RLE script. I haven't actually fully trained the model yet; I'm still debugging. ;)" }, { "id": 291089, "postDate": "2018-03-05T16:00:30.453Z", "content": "No problem Steven. Good luck.
", "rawMarkdown": "No problem Steven. Good luck. " } ] }, { "id": 289294, "postDate": "2018-03-01T10:11:13.447Z", "content": "@Heng CherKeng, pls delight me again:) Instead of fixing the backbone network (FPN based ResNet101) I would like to train entire network, however after 40 epoch, the model starts to overfit (loss continuously decreasing but val_loss increasing). Since it is ResNet based, so batch norm already applied, i wonder if you have any tips to further prevent overfitting? Thank you in advance!
", "rawMarkdown": "@Heng CherKeng, pls delight me again:) Instead of fixing the backbone network (FPN based ResNet101) I would like to train entire network, however after 40 epoch, the model starts to overfit (loss continuously decreasing but val_loss increasing). Since it is ResNet based, so batch norm already applied, i wonder if you have any tips to further prevent overfitting? Thank you in advance! " }, { "id": 289262, "postDate": "2018-03-01T08:48:32.980Z", "content": "I find you use pytorch 0.4, but official pytorch lastare 0.3.1, can you tell me how to install pytorch 0.4? Thanks.
", "rawMarkdown": "I find you use pytorch 0.4, but official pytorch lastare 0.3.1, can you tell me how to install pytorch 0.4? Thanks.", "replies": [ { "id": 289265, "postDate": "2018-03-01T09:01:37.137Z", "content": "install from https://github.com/pytorch/pytorch
\n\nFrom Source\nIf you are installing from source, we highly recommend installing an Anaconda environment. You will get a high-quality BLAS library (MKL) and you get a controlled compiler version regardless of your Linux distro.
\n\nOnce you have Anaconda installed, here are the instructions.
\n\n....
\n\nInstall optional dependencies\nOn Linux
\n\n....
", "rawMarkdown": "install from https://github.com/pytorch/pytorch\n\n\nFrom Source\nIf you are installing from source, we highly recommend installing an Anaconda environment. You will get a high-quality BLAS library (MKL) and you get a controlled compiler version regardless of your Linux distro.\n\nOnce you have Anaconda installed, here are the instructions.\n\n....\n\nInstall optional dependencies\nOn Linux\n\n....", "votes": 1 } ] }, { "id": 288788, "postDate": "2018-02-28T07:18:12.460Z", "content": "I keep encounter \"AssertionError: Torch not compiled with CUDA enabled\" when I run common.py, is there anyway to fix it? (I am running on MacOS) thanks!
", "rawMarkdown": "I keep encounter \"AssertionError: Torch not compiled with CUDA enabled\" when I run common.py, is there anyway to fix it? (I am running on MacOS) thanks!", "replies": [ { "id": 303217, "postDate": "2018-03-25T19:08:35.810Z", "content": "Hi. Have you installed PyTorch from source. And have you installed the prerequisites (CUDA and cuDNN) ? If yes, when you install in mac os, you have to specify : MACOSX_DEPLOYMENT_TARGET=10.13 CC=clang CXX=clang++ python setup.py install . Note 10.13 if you have High Sierra os. This did the trick for me.
", "rawMarkdown": "Hi. Have you installed PyTorch from source. And have you installed the prerequisites (CUDA and cuDNN) ? If yes, when you install in mac os, you have to specify : MACOSX_DEPLOYMENT_TARGET=10.13 CC=clang CXX=clang++ python setup.py install . Note 10.13 if you have High Sierra os. This did the trick for me." } ] }, { "id": 288787, "postDate": "2018-02-28T07:17:18.310Z", "content": "Thank you very much for your code. I found you have 15.4. Is that the newest version?
", "rawMarkdown": "Thank you very much for your code. I found you have 15.4. Is that the newest version?" }, { "id": 288752, "postDate": "2018-02-28T05:55:09.390Z", "content": "Hello @Heng CherKeng: As you said, maskrcnn is fully convolutional so it can take any size for inference. However, when using FPN that used fully connected layers, so that we cannot take any size as input. Do you also modify Fully connected layer to 1x1 convolution in FPN to work with arbitrary size in inference? Thanks
", "rawMarkdown": "Hello @Heng CherKeng: As you said, maskrcnn is fully convolutional so it can take any size for inference. However, when using FPN that used fully connected layers, so that we cannot take any size as input. Do you also modify Fully connected layer to 1x1 convolution in FPN to work with arbitrary size in inference? Thanks\n", "replies": [ { "id": 288755, "postDate": "2018-02-28T06:01:37.203Z", "content": "You should understand “fully convolutional” as the results doesn’t change regardless of image. “Fully convolutional” can use fully connected layer too.
\n\nIn case of mask rcnn( and also faster rcnn) we crop a patch and apply fully connected layer. Regardless of image size, we will make the same crop. Becuase the crop is the same, the fully connected layer gives the same results.
\n\nYou don’t have to make any changes. Just input image of different size.
", "rawMarkdown": "You should understand “fully convolutional” as the results doesn’t change regardless of image. “Fully convolutional” can use fully connected layer too.\n\nIn case of mask rcnn( and also faster rcnn) we crop a patch and apply fully connected layer. Regardless of image size, we will make the same crop. Becuase the crop is the same, the fully connected layer gives the same results.\n\nYou don’t have to make any changes. Just input image of different size.", "votes": 1 }, { "id": 288759, "postDate": "2018-02-28T06:12:30.893Z", "content": "Thanks for your explanation. If I understand correctly, you did not crop raw image size. You cropped the features of resnet-50 in both training and inference. Thus you do not care about input size. Am I right?\n Sorry for that, I thought that you crop input image into fixed size 128x128 and feed it to the network for training.
", "rawMarkdown": "Thanks for your explanation. If I understand correctly, you did not crop raw image size. You cropped the features of resnet-50 in both training and inference. Thus you do not care about input size. Am I right?\n Sorry for that, I thought that you crop input image into fixed size 128x128 and feed it to the network for training. " }, { "id": 289268, "postDate": "2018-03-01T09:09:48.277Z", "content": "Hi @Heng CherKeng. Very impressed with your score. Could you tell me some tips to achieve your score from the previous baseline (0.46 LB)? Does the random crop step achieve a big gain? Thanks
", "rawMarkdown": "Hi @Heng CherKeng. Very impressed with your score. Could you tell me some tips to achieve your score from the previous baseline (0.46 LB)? Does the random crop step achieve a big gain? Thanks\n" }, { "id": 289272, "postDate": "2018-03-01T09:16:12.747Z", "content": "i started off with Unet on all images(color and gray)
\n\nthen i switch to mask rcnn. i first replace unet gray images results with that of mask rcnn gray images and get 0.46.
\n\nFinally i replace use all mask rcnn gray and color images and get about 0.52.
\n\ni will write a post on how to train mask rcnn \"properly\" later. but basically , baseline results of mask rcnn is about 0.50.
", "rawMarkdown": "i started off with Unet on all images(color and gray)\n\nthen i switch to mask rcnn. i first replace unet gray images results with that of mask rcnn gray images and get 0.46.\n\nFinally i replace use all mask rcnn gray and color images and get about 0.52.\n\ni will write a post on how to train mask rcnn \"properly\" later. but basically , baseline results of mask rcnn is about 0.50.", "votes": 7 }, { "id": 289280, "postDate": "2018-03-01T09:32:17.750Z", "content": "Great process. I think the way to implementation of mask-rcnn is very importance. For me, I used the Keras based (matterport's implementation), so I got about 0.41 LB without separate training data into some cluster. The idea to handle with different size is that it resized all input to a fixed size (with larger enough such as 512x512 by zero padding). However, the weakness is that it loss the image distribution, so I purpose a way to randomly crop image to 128x128, but it does not work for inference phase if the image size different with the crop size. Do you think crop image will help to improve performance?
", "rawMarkdown": "Great process. I think the way to implementation of mask-rcnn is very importance. For me, I used the Keras based (matterport's implementation), so I got about 0.41 LB without separate training data into some cluster. The idea to handle with different size is that it resized all input to a fixed size (with larger enough such as 512x512 by zero padding). However, the weakness is that it loss the image distribution, so I purpose a way to randomly crop image to 128x128, but it does not work for inference phase if the image size different with the crop size. Do you think crop image will help to improve performance?" }, { "id": 289292, "postDate": "2018-03-01T10:08:12.813Z", "content": "@john1231983 not sure crop will help. I am also using matterport's implementation, instead of cropping, i tried different data augmentation, like flip up/down, rotation 90 degree etc, which doesn't improve the performance at all. The loss still decrease, but the val_loss remains quite high after 40-60epoch. So far my best score is 0.448 with default augmentation (only flip left/right).
", "rawMarkdown": "@john1231983 not sure crop will help. I am also using matterport's implementation, instead of cropping, i tried different data augmentation, like flip up/down, rotation 90 degree etc, which doesn't improve the performance at all. The loss still decrease, but the val_loss remains quite high after 40-60epoch. So far my best score is 0.448 with default augmentation (only flip left/right). " }, { "id": 289296, "postDate": "2018-03-01T10:17:53.290Z", "content": "Hi Keven Wang, thanks for your information. The author of this thread also use crop and it shows very good performance. So I guess it may be converted to matterport's implementation to improve performance. For your score, I think you can improve the performance by running the code for submission https://github.com/YanWang2014/Mask_RCNN/blob/master/Spot-Nuclei2/val_test2.py and also try the refineMasks function in that link.
\n\nDo you change something in matterport's implementation the to obtain 0.448 score? This is my current setting\n\n IMAGES_PER_GPU = 2\n GPU_COUNT = 1\n NUM_CLASSES = 1 + 1 # Background + Nuclei\n USE_MINI_MASK=True\n MINI_MASK_SHAPE = (56,56)\n RPN_ANCHOR_SCALES = (8, 16, 32, 64,128)\n RPN_ANCHOR_RATIOS = [0.25, 0.5, 1, 2, 4]\n RPN_NMS_THRESHOLD = 0.7\n IMAGE_MIN_DIM =512\n IMAGE_MAX_DIM=512\n DETECTION_MAX_INSTANCES = 200\n RPN_TRAIN_ANCHORS_PER_IMAGE = 320\n RPN_NMS_THRESHOLD = 0.6\n MAX_GT_INSTANCES = 200\n DETECTION_MIN_CONFIDENCE = 0.6\n
hmm, i did quite some experimentation with data augmentation, they only seem to have negative effect, must be sth wrong.
\n\nRegarding configuration matterport's implementation, not much different than yours, but you might want to increase TRAIN_ROIS_PER_IMAGE since in some images there are lots of masks. Again, each training is random process, for me it is also hard to reproduce the same good result even if using the same configuration.
@Keven Wang: Thanks. Do you seperate the training data to some clusters likes: gray, color, ...and train on each cluster individually? Or just use all training image together with gray and color in a signle model? Or convert all training image to gray channel? I use second case
", "rawMarkdown": "@Keven Wang: Thanks. Do you seperate the training data to some clusters likes: gray, color, ...and train on each cluster individually? Or just use all training image together with gray and color in a signle model? Or convert all training image to gray channel? I use second case" }, { "id": 290006, "postDate": "2018-03-02T19:30:36.180Z", "content": "@Heng CherKeng. It is pretty cool idea since unet outputs will be smooth and normalized. Did you retrain whole model from scrach when you change all images to unet output? Or did you train using your pretrained weights that gave you LB=0.46?
", "rawMarkdown": "@Heng CherKeng. It is pretty cool idea since unet outputs will be smooth and normalized. Did you retrain whole model from scrach when you change all images to unet output? Or did you train using your pretrained weights that gave you LB=0.46? " }, { "id": 295879, "postDate": "2018-03-14T10:58:48.813Z", "content": "Hi, I tried this configuration, but I was unable to get a score higher than 0.37. Is there any configuration that I am missing? Some pre-processing or post-processing?
", "rawMarkdown": "Hi, I tried this configuration, but I was unable to get a score higher than 0.37. Is there any configuration that I am missing? Some pre-processing or post-processing?" } ] }, { "id": 288709, "postDate": "2018-02-28T03:11:14.167Z", "content": "Hi, @Heng CherKeng.\nVery nice work.\nI runned the process.py sucessed.But output is:\ngpu_nms : [5, 55, 37, 27, 22, 38, 35, 33, 20, 54, 44, 2, 50, 30, 0, 34, 21, 23, 28, 39, 43, 51, 3, 10, 16, 56, 8, 32, 45, 13, 52, 19, 36, 6, 29, 12, 9, 31, 41, 40, 25, 17, 24, 4, 53, 11, 14, 42, 46, 1, 15, 49, 47, 7, 48, 26, 18]\ncython_nms : [5, 55, 37, 27, 35, 20, 0, 45, 52, 19, 29, 24, 4, 11]\ntorch_nms : [ 5 55 37 27 35 20 0 45 52 19 29 24 4 11]\nThe gpu_nms is longer than cython_nms and torch_nms.Is it error?\nThanks!
", "rawMarkdown": "Hi, @Heng CherKeng.\nVery nice work.\nI runned the process.py sucessed.But output is:\ngpu_nms : [5, 55, 37, 27, 22, 38, 35, 33, 20, 54, 44, 2, 50, 30, 0, 34, 21, 23, 28, 39, 43, 51, 3, 10, 16, 56, 8, 32, 45, 13, 52, 19, 36, 6, 29, 12, 9, 31, 41, 40, 25, 17, 24, 4, 53, 11, 14, 42, 46, 1, 15, 49, 47, 7, 48, 26, 18]\ncython_nms : [5, 55, 37, 27, 35, 20, 0, 45, 52, 19, 29, 24, 4, 11]\ntorch_nms : [ 5 55 37 27 35 20 0 45 52 19 29 24 4 11]\nThe gpu_nms is longer than cython_nms and torch_nms.Is it error?\nThanks!", "replies": [ { "id": 288753, "postDate": "2018-02-28T05:56:04.237Z", "content": "I the results of gpu mns is wrong
", "rawMarkdown": "I the results of gpu mns is wrong", "votes": 1 } ] }, { "id": 288327, "postDate": "2018-02-27T08:20:37.170Z", "content": "Thanks for your wonderful code @Heng CherKeng. Appreciate it! I want to reproduce your work on my own computer.
\n\nI have Anaconda python 3.6, CUDA 8.0 , CuDnn 7 , pytorch 0.3.1. I run the pytorch tutorial . The pytorch and cuda work fine and I can use the code net.cuda() . When I come to your dummy14.2 code, I'm stuck in the network line net = MaskRcnnNet(cfg).cuda(),
The Error is shown below:\nRuntimeError: cuda runtime error (78) : a PTX JIT compilation failed at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THC/generic/THCStorage.cu:58\n
It seems the error comes from .cuda().After I change the .cuda() into cpu(), the line works fine!
\n\nI'm new to the pytorch, and I cannot find the exact answer in google. Can anyone help me? Thx very much!
", "rawMarkdown": "Thanks for your wonderful code @Heng CherKeng. Appreciate it! I want to reproduce your work on my own computer.\n\nI have Anaconda python 3.6, CUDA 8.0 , CuDnn 7 , pytorch 0.3.1. I run the [pytorch tutorial][1] . The pytorch and cuda work fine and I can use the code `net.cuda()` . When I come to your dummy14.2 code, I'm stuck in the network line `net = MaskRcnnNet(cfg).cuda()`, \n\nThe Error is shown below:`\nRuntimeError: cuda runtime error (78) : a PTX JIT compilation failed at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THC/generic/THCStorage.cu:58\n`\n\nIt seems the error comes from .cuda().After I change the .cuda() into cpu(), the line works fine! \n\nI'm new to the pytorch, and I cannot find the exact answer in google. Can anyone help me? Thx very much!\n\n [1]: http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html", "replies": [ { "id": 288329, "postDate": "2018-02-27T08:24:38.933Z", "content": "could be due wrong cuda library, etc?
\n\nyou should verify your pytorch installation with other example problem like mnist or cifar10
", "rawMarkdown": "could be due wrong cuda library, etc?\n\nyou should verify your pytorch installation with other example problem like mnist or cifar10\n", "votes": 1 }, { "id": 288337, "postDate": "2018-02-27T08:36:30.350Z", "content": "Thx for your reply. The pytorch & cuda works fine when I train a cifar-10 classifier. When I add from train import * in front the cifar-10 code, the code stuck in the .cuda() again!(without the line , it works fine) I want to find out whether the train.py have something to do with the cuda?
Hi, @Heng CherKeng . Maybe the problem comes from the *.so? I didn't rebuild the *.so as the .pptx says.
", "rawMarkdown": "Hi, @Heng CherKeng . Maybe the problem comes from the *.so? I didn't rebuild the *.so as the .pptx says." }, { "id": 288350, "postDate": "2018-02-27T09:18:57.213Z", "content": "you need to rebuild
", "rawMarkdown": "you need to rebuild\n", "votes": 1 }, { "id": 288424, "postDate": "2018-02-27T13:01:04.307Z", "content": "Hi, @Heng CherKeng. I've tried rebulid the *.so as the .pptx says, but the problem doesn't go. Maybe the problem comes from the cuda version . I have CUDA8.0 , CuDnn 7. In your code the CUDA version is 9.1. The different nvcc version may complie different *.so. Therefore, my code didn't work. Do you think it's the cuda version matters? Thx！
", "rawMarkdown": "Hi, @Heng CherKeng. I've tried rebulid the *.so as the .pptx says, but the problem doesn't go. Maybe the problem comes from the cuda version . I have CUDA8.0 , CuDnn 7. In your code the CUDA version is 9.1. The different nvcc version may complie different *.so. Therefore, my code didn't work. Do you think it's the cuda version matters? Thx！" }, { "id": 288954, "postDate": "2018-02-28T16:14:01.520Z", "content": "this error is only in the dummy .7 version. The error does not come in .2 version
", "rawMarkdown": "this error is only in the dummy .7 version. The error does not come in .2 version" }, { "id": 292791, "postDate": "2018-03-08T16:07:17.747Z", "content": "you must remove *.so and then rebuild
", "rawMarkdown": "you must remove *.so and then rebuild" }, { "id": 295480, "postDate": "2018-03-13T17:45:06.040Z", "content": "@Liang Jeff: Can we use Cuda 8.0 and cudnn 6.0. I don't want to upgrade to cuda 9.0 and cudnn 7.1
", "rawMarkdown": "@Liang Jeff: Can we use Cuda 8.0 and cudnn 6.0. I don't want to upgrade to cuda 9.0 and cudnn 7.1", "votes": 1 }, { "id": 295708, "postDate": "2018-03-14T03:20:13.527Z", "content": "Anyone here know cuda coding? I am curious about what prevents the code from running on cuda 8?
", "rawMarkdown": "Anyone here know cuda coding? I am curious about what prevents the code from running on cuda 8? " } ] }, { "id": 287921, "postDate": "2018-02-26T08:28:29.973Z", "content": "Hi, you have provide a wonderful work.\nMay I ask how can I run your code on Python3.5 and Cuda8.0? Thx
", "rawMarkdown": "Hi, you have provide a wonderful work.\nMay I ask how can I run your code on Python3.5 and Cuda8.0? Thx" }, { "id": 287873, "postDate": "2018-02-26T05:48:02.687Z", "content": "Has anyone encountered the following the error while training?
\n\n```
\n\n0.0000 0.0 k 0.00 0.0 m | 2.184 0.69 0.09 0.70 0.01 0.69 | 0.000 0.00 0.00 0.00 0.00 0.00 | 0.000 0.00 0.00 0.00 0.00 0.00 | 0 hr 01 min\n0.0010 0.1 k 0.16 0.0 m | 2.213 0.58 0.16 0.83 0.08 0.56 | 1.872 0.53 0.10 0.48 0.07 0.69 | 1.935 0.50 0.06 0.60 0.08 0.70 | 0 hr 07 min\n0.0010 0.2 k 0.25 0.0 m | 2.213 0.58 0.16 0.83 0.08 0.56 | 1.689 0.36 0.11 0.46 0.07 0.69 | 1.645 0.36 0.19 0.36 0.04 0.70 | 0 hr 09 min 160,160,
\n\nTraceback (most recent call last):\nFile \"/media/ext1/Kaggle/DataScienceBowl2018/HengCherKeng/20180214/dummy-14.7/train.py\", line 572, in \n run_train()\n File \"/media/ext1/Kaggle/DataScienceBowl2018/HengCherKeng/20180214/dummy-14.7/train.py\", line 416, in run_train\n net( inputs, truth_boxes, truth_labels, truth_instances )\n File \"/home/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py\", line 357, in call\n result = self.forward(*input, **kwargs)\n File \"/media/ext1/Kaggle/DataScienceBowl2018/HengCherKeng/20180214/dummy-14.7/model/mask_rcnn_resnet50_fpn_net.py\", line 399, in forward\n make_rpn_target(cfg, inputs, self.rpn_windows, truth_boxes, truth_labels )\n File \"/media/ext1/Kaggle/DataScienceBowl2018/HengCherKeng/20180214/dummy-14.7/model/mask_rcnn_lib/rpn_target.py\", line 269, in make_rpn_target\n index = truth_labels[b]>0\nTypeError: unorderable types: NoneType() > int()\n```
", "rawMarkdown": "Has anyone encountered the following the error while training?\n\n```\n rate iter epoch num | valid_loss | train_loss | batch_loss | time \n------------------------------------------------------------------------------------------------------------------------------------------------------------------\n0.0000 0.0 k 0.00 0.0 m | 2.184 0.69 0.09 0.70 0.01 0.69 | 0.000 0.00 0.00 0.00 0.00 0.00 | 0.000 0.00 0.00 0.00 0.00 0.00 | 0 hr 01 min\n0.0010 0.1 k 0.16 0.0 m | 2.213 0.58 0.16 0.83 0.08 0.56 | 1.872 0.53 0.10 0.48 0.07 0.69 | 1.935 0.50 0.06 0.60 0.08 0.70 | 0 hr 07 min\n0.0010 0.2 k 0.25 0.0 m | 2.213 0.58 0.16 0.83 0.08 0.56 | 1.689 0.36 0.11 0.46 0.07 0.69 | 1.645 0.36 0.19 0.36 0.04 0.70 | 0 hr 09 min 160,160,\n\nTraceback (most recent call last):\nFile \"/media/ext1/Kaggle/DataScienceBowl2018/HengCherKeng/20180214/dummy-14.7/train.py\", line 572, in ", "replies": [ { "id": 287916, "postDate": "2018-02-26T08:08:38.617Z", "content": "truth_labels is None.
\n\nit is better to filter the crop in your batch train data to make sure that there is at least one ground truth box.
\n\n(you can also try dummy-14.2, which may be a more stable version)
", "rawMarkdown": "truth_labels is None.\n\nit is better to filter the crop in your batch train data to make sure that there is at least one ground truth box.\n\n(you can also try dummy-14.2, which may be a more stable version)", "votes": 1 }, { "id": 288032, "postDate": "2018-02-26T14:45:05.540Z", "content": "Thanks, I will try the version 14.2. BTW, thanks for sharing the code! I learned a lot. :)
", "rawMarkdown": "Thanks, I will try the version 14.2. BTW, thanks for sharing the code! I learned a lot. :)" } ] }, { "id": 287485, "postDate": "2018-02-25T00:40:08.380Z", "content": "HI, amazing work and thanks for sharing :D \nI’m new in computer vision, and I have a question on how to deal with different size images. On your experience , what is the best way to “normalize” and preprocess images of different sizes in order to train a nice neural network? I think resizing to a fixed size can hurt performance.
\n\nThanks again for sharing :D
", "rawMarkdown": "HI, amazing work and thanks for sharing :D \nI’m new in computer vision, and I have a question on how to deal with different size images. On your experience , what is the best way to “normalize” and preprocess images of different sizes in order to train a nice neural network? I think resizing to a fixed size can hurt performance. \n\nThanks again for sharing :D" }, { "id": 287251, "postDate": "2018-02-24T04:18:30.770Z", "content": "Gread job!!!I want to run the dummy-14.7.Would you tell me where is the train1_ids_remove_error_668_meta.csv file?
", "rawMarkdown": "Gread job!!!I want to run the dummy-14.7.Would you tell me where is the train1_ids_remove_error_668_meta.csv file?", "replies": [ { "id": 290205, "postDate": "2018-03-03T10:14:03.817Z", "content": "did you got that file? just run into the same problem
", "rawMarkdown": "did you got that file? just run into the same problem" }, { "id": 290226, "postDate": "2018-03-03T13:06:04.643Z", "content": "\"train1_ids_remove_error_668_meta.csv\" this file is not required. just modify the code not to read this file and set meta=''\"
", "rawMarkdown": "\"train1_ids_remove_error_668_meta.csv\" this file is not required. just modify the code not to read this file and set meta=''\"", "votes": 1 }, { "id": 290382, "postDate": "2018-03-03T23:05:34.583Z", "content": "Thank you, Heng. Now I am able to run 14.7. But only batch size = 1 on my 6GB memory gpu to avoid oom. \nDid you try transfer learning? My understanding was you train all parameters using the training data in this competition.
", "rawMarkdown": "Thank you, Heng. Now I am able to run 14.7. But only batch size = 1 on my 6GB memory gpu to avoid oom. \nDid you try transfer learning? My understanding was you train all parameters using the training data in this competition." }, { "id": 290442, "postDate": "2018-03-04T03:51:49.767Z", "content": "Hi @Heng CherKeng, @Chen Zhang, I downloaded dummy-14.7.tar.gz from this post and did everything in cython_libs.pptx but still got this error: \n File \"/dummy-14.7/model/mask_rcnn_lib/nms/torch_nms/init.py\", line 44, in torch_nms\n gpu_nms(keep, num_out, dets, thresh)\nNameError: name 'gpu_nms' is not defined
\n\nDid you meet this problem and do you know how to solve it? I did rebuild extension of torch_nms. Thanks.
", "rawMarkdown": "Hi @Heng CherKeng, @Chen Zhang, I downloaded dummy-14.7.tar.gz from this post and did everything in cython_libs.pptx but still got this error: \n File \"/dummy-14.7/model/mask_rcnn_lib/nms/torch_nms/__init__.py\", line 44, in torch_nms\n gpu_nms(keep, num_out, dets, thresh)\nNameError: name 'gpu_nms' is not defined\n\nDid you meet this problem and do you know how to solve it? I did rebuild extension of torch_nms. Thanks." }, { "id": 290469, "postDate": "2018-03-04T06:55:59.507Z", "content": "I did a couple of things to get this solved. I am not sure which makes it work.\nThe one I would recommand is to change the last line in the setup.py in gpu_nms, add \"cythonize\".\nJust like the last line in setup.py in cython_nms.
", "rawMarkdown": "I did a couple of things to get this solved. I am not sure which makes it work.\nThe one I would recommand is to change the last line in the setup.py in gpu_nms, add \"cythonize\".\nJust like the last line in setup.py in cython_nms. \n" }, { "id": 290486, "postDate": "2018-03-04T08:21:03.410Z", "content": "Thanks, Chen Zhang. I changed as you mentioned but saw the same error.
", "rawMarkdown": "Thanks, Chen Zhang. I changed as you mentioned but saw the same error. " } ] }, { "id": 287207, "postDate": "2018-02-23T21:40:56.743Z", "content": "Hi Heng, thanks for sharing so much useful information! I have a question regarding overlapping bounding boxes. If multiple boxes and their corresponding masks both classify a pixel as belonging to their instance, how is this conflict resolved? I.e., how to determine which instance to assign the pixel to if it belongs to multiple instances.
", "rawMarkdown": "Hi Heng, thanks for sharing so much useful information! I have a question regarding overlapping bounding boxes. If multiple boxes and their corresponding masks both classify a pixel as belonging to their instance, how is this conflict resolved? I.e., how to determine which instance to assign the pixel to if it belongs to multiple instances." }, { "id": 286956, "postDate": "2018-02-23T04:42:53.847Z", "content": "Thanks for sharing, @hengck23! Looking through your code has been very educational in various ways. Inspired me to check out how to create custom PyTorch CUDA extensions, etc.
", "rawMarkdown": "Thanks for sharing, @hengck23! Looking through your code has been very educational in various ways. Inspired me to check out how to create custom PyTorch CUDA extensions, etc." }, { "id": 286878, "postDate": "2018-02-23T00:31:19.467Z", "content": "Thanks for the code brother. I have a question in data preparation stage. The multi_mask.npy is a collection of masks in the form of [HxWxI] where I is the number of the masks ?
Great job ! thank you very much for the awesome code !
\n\nI met some problem:
\n\n1) I have 2 1080ti but still encountered oom with batch_size = 4
\n\n2) Strangely, the memory and GPU of second card on my device was not used, I think the memory should be shared even if 'mult_gpus' is not set
\n\nCould you give me some advice?
", "rawMarkdown": "Great job ! thank you very much for the awesome code !\n\nI met some problem:\n\n1) I have 2 1080ti but still encountered oom with batch_size = 4 \n\n2) Strangely, the memory and GPU of second card on my device was not used, I think the memory should be shared even if 'mult_gpus' is not set\n\nCould you give me some advice? ", "replies": [ { "id": 286949, "postDate": "2018-02-23T04:30:30.917Z", "content": "In train.py, did you modify the following line:\nos.environ['CUDA_VISIBLE_DEVICES'] = '0' <- needs to be set to '0,1' if you want to utilize two GPUs.
", "rawMarkdown": "In train.py, did you modify the following line:\nos.environ['CUDA_VISIBLE_DEVICES'] = '0' <- needs to be set to '0,1' if you want to utilize two GPUs.\n" }, { "id": 286964, "postDate": "2018-02-23T05:02:20.810Z", "content": "THX a lot！I am too careless to see it
", "rawMarkdown": "THX a lot！I am too careless to see it" }, { "id": 286967, "postDate": "2018-02-23T05:06:53.140Z", "content": "May I ask one more question : In the submit.py I found this line :
\n\ntest_dataset = ScienceDataset(\n #'test1_ids_gray_only_53', mode='test',\n #'train1_ids_gray_only1_500', mode='test',\n 'valid1_ids_gray_only1_43', mode='test',\n transform = submit_augment)\nI think it means I did not predict the test data but predicted the verification data，What the correct code should be?
", "rawMarkdown": "May I ask one more question : In the submit.py I found this line :\n\n test_dataset = ScienceDataset(\n #'test1_ids_gray_only_53', mode='test',\n #'train1_ids_gray_only1_500', mode='test',\n 'valid1_ids_gray_only1_43', mode='test',\n transform = submit_augment)\n\nI think it means I did not predict the test data but predicted the verification data，What the correct code should be?" } ] }, { "id": 285784, "postDate": "2018-02-21T01:07:04.327Z", "content": "its a very interesting topic. i new here sou i will folow this project for now.
", "rawMarkdown": "its a very interesting topic. i new here sou i will folow this project for now." }, { "id": 285559, "postDate": "2018-02-20T15:35:40.067Z", "content": "Thanks @Heng for your great works. Is there anything I can do to make train.py run on my only GTX 1070? I encountered the not enough memory problem. I did try to increase iter_accum but it didn't work.
\n\nThanks
", "rawMarkdown": "Thanks @Heng for your great works. Is there anything I can do to make train.py run on my only GTX 1070? I encountered the not enough memory problem. I did try to increase iter_accum but it didn't work.\n\nThanks" }, { "id": 285546, "postDate": "2018-02-20T15:09:54.623Z", "content": "Hi Heng, thx for sharing this! I am also using Mask RCNN based approach, not exactly your implementation but quite similar. One challenge i have is two or several instance masks overlap with each other and the prediction of submission is that there is NO single pixel belongs to more than one instance mask. So i wonder what kind of approach you took to solve this pb? Do you think the pb is related with network (and corresponding parameter) or we have to do some post process like go through each masks in the end and if any part overlap with another masks, remove the overlapping part from one of the mask? Thank you and looking forward for your suggestions!
", "rawMarkdown": "Hi Heng, thx for sharing this! I am also using Mask RCNN based approach, not exactly your implementation but quite similar. One challenge i have is two or several instance masks overlap with each other and the prediction of submission is that there is NO single pixel belongs to more than one instance mask. So i wonder what kind of approach you took to solve this pb? Do you think the pb is related with network (and corresponding parameter) or we have to do some post process like go through each masks in the end and if any part overlap with another masks, remove the overlapping part from one of the mask? Thank you and looking forward for your suggestions!", "replies": [ { "id": 285549, "postDate": "2018-02-20T15:12:51.220Z", "content": "i am solving this issue later.
\n\ncurrently, just use the mask from the higher score detection box. it will be placed on \"top\" and overwrite any previous pixel label.
", "rawMarkdown": "i am solving this issue later. \n\ncurrently, just use the mask from the higher score detection box. it will be placed on \"top\" and overwrite any previous pixel label.", "votes": 3 }, { "id": 285575, "postDate": "2018-02-20T16:22:37.770Z", "content": "Here is my overlap cleaning function. Hope it helps!
\n\ndef clean_overlap(masks, scores, height, width):\nif masks.shape[0]==0:\n masks = np.zeros( [height, width,1] )\n masks[0,0,0] = 1\n\nmasks=np.moveaxis( masks, [0,1,2] , [1,2,0] )\nsort_ind = np.argsort(scores)[::-1]\nmasks = masks[sort_ind]\noverlap = np.zeros( [height, width ] )\nfor mm in range( len( masks) ):\n mask = masks[mm]\n overlap +=mask\n mask[overlap>1] = 0\n masks[mm] = mask\n\ndel_ind = np.where( np.sum(masks, axis=(1,2) )<1)[0]\nif len(del_ind)>0:\n if len(del_ind)<len(masks):\n print('Empty mask, deleting', len(del_ind), 'masks')\n masks = np.delete(masks, del_ind, axis=0)\n else:\n masks = np.zeros( [1, height, width ] )\n masks[0,0,0] = 1\n\nreturn masks\nThx both Heng and panserbjorne! I am a little bit stuck with family matters these days, will try out your approaches later! Thank you again!
", "rawMarkdown": "Thx both Heng and panserbjorne! I am a little bit stuck with family matters these days, will try out your approaches later! Thank you again!\n" }, { "id": 303926, "postDate": "2018-03-26T19:30:42.653Z", "content": "@Panserbjorne thanks for sharing your code. What is the scores?
", "rawMarkdown": "@Panserbjorne thanks for sharing your code. What is the scores?" } ] }, { "id": 285488, "postDate": "2018-02-20T12:34:04.257Z", "content": "This is great. Out of interest, how long did you train it for?
", "rawMarkdown": "This is great. Out of interest, how long did you train it for?" }, { "id": 285432, "postDate": "2018-02-20T08:26:43.623Z", "content": "I can't understand how to build *.so libraries. Please help me, Thanks!
\n\nFellow this:\ncd dummy-14.2/model/mask_rcnn_lib/nms/torch_nms/extension\nnvcc -c -o nms_kernel nms_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52
\n\nerror:\ngcc: error: nms_kernel.cu: No such file or directory\ngcc: warning: ‘-x c++’ after last input file has no effect\ngcc: fatal error: no input files
", "rawMarkdown": "I can't understand how to build *.so libraries. Please help me, Thanks!\n\n Fellow this:\ncd dummy-14.2/model/mask_rcnn_lib/nms/torch_nms/extension\nnvcc -c -o nms_kernel nms_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52\n\nerror:\ngcc: error: nms_kernel.cu: No such file or directory\ngcc: warning: ‘-x c++’ after last input file has no effect\ngcc: fatal error: no input files", "replies": [ { "id": 285504, "postDate": "2018-02-20T13:13:36.397Z", "content": "You have to cd into src, not extension folder:
\n\ncd dummy-14.2/model/mask_rcnn_lib/nms/torch_nms/src\nthen:
\n\nnvcc -c -o nms_kernel nms_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52\nThanks, fix this. I'm not familiar with pytorch.\nmeet another error:
\n\n/mask_rcnn_lib/rpn_nms.py\", line 80, in torch_rpn_nms\nboxes = torch_box_transform_inv(windows.view(-1,4) , deltas.view(-1,4))\nNameError: name 'torch_box_transform_inv' is not defined
", "rawMarkdown": "Thanks, fix this. I'm not familiar with pytorch.\nmeet another error:\n\n/mask_rcnn_lib/rpn_nms.py\", line 80, in torch_rpn_nms\nboxes = torch_box_transform_inv(windows.view(-1,4) , deltas.view(-1,4))\nNameError: name 'torch_box_transform_inv' is not defined\n" }, { "id": 285544, "postDate": "2018-02-20T15:09:24.397Z", "content": "torch_box_transform_inv() is in box.py.
\n\nsome path setting issues. I suggest you use pycharm IDE (which i use), then there is no problem. The whole project uploaded in google drive is in pychram.
", "rawMarkdown": "torch_box_transform_inv() is in box.py. \n\nsome path setting issues. I suggest you use pycharm IDE (which i use), then there is no problem. The whole project uploaded in google drive is in pychram.", "votes": 1 }, { "id": 285789, "postDate": "2018-02-21T01:26:09.763Z", "content": "Thanks very much! I have change to use pycharm, build *.so libraries.\nWhile, meet such error, how can rebuild gpu_mns by your attach box.tar.gz ?
\n\nfrom model.mask_rcnn_lib.nms.gpu_nms import gpu_nms\nImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
", "rawMarkdown": "Thanks very much! I have change to use pycharm, build *.so libraries.\nWhile, meet such error, how can rebuild gpu_mns by your attach box.tar.gz ?\n\nfrom model.mask_rcnn_lib.nms.gpu_nms import gpu_nms\nImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory" }, { "id": 285989, "postDate": "2018-02-21T12:37:46.017Z", "content": "In my case, with Ubuntu 16.04LTS, python2.7.12/python3.5.2, cuda 9.1,\nI avoided rebuilding .so libraries by trying to follow at my best @Heng CherKeng's software setup requirements (readme.pptx, slide 2).
\n\nSteps I took:
\n\nInstalled python 3.6.2 and created a virtual environment for python 3.6
Installed PyCharm CE, loaded dummy-14.2 project and assigned python3.6 from the virtual environment to this project.\nThen I made adjustments for the project, removed .idea and reindexed the project.
Installed necessary python packages into the virtual environment, either from terminal using pip or PyCharm interface.
PS PyCharm has an issue with handling \\r (carriage return), I was not able to find a solution. Still, what a great IDE!\nThanks @Heng CherKeng not only for sharing his outstanding Mask-RCNN implemenation but for enabling us to use better tools as well!
", "rawMarkdown": "In my case, with Ubuntu 16.04LTS, python2.7.12/python3.5.2, cuda 9.1,\nI avoided rebuilding .so libraries by trying to follow at my best @Heng CherKeng's software setup requirements (readme.pptx, slide 2). \n\nSteps I took:\n\n1. Installed python 3.6.2 and created a virtual environment for python 3.6\n\n2. Installed PyCharm CE, loaded dummy-14.2 project and assigned python3.6 from the virtual environment to this project.\n Then I made adjustments for the project, removed .idea and reindexed the project.\n\n3. Installed necessary python packages into the virtual environment, either from terminal using pip or PyCharm interface.\n\nPS PyCharm has an issue with handling \\r (carriage return), I was not able to find a solution. Still, what a great IDE!\nThanks @Heng CherKeng not only for sharing his outstanding Mask-RCNN implemenation but for enabling us to use better tools as well!\n", "votes": 2 }, { "id": 286430, "postDate": "2018-02-22T06:35:48.920Z", "content": "I use Ubuntu 16.04LTS, miniconda python3.6, cuda 9.0, have't fix this error.
\nYou have't meet this error?\nfrom model.mask_rcnn_lib.nms.gpu_nms import gpu_nms ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
You should rebuild it ,see:
\n\n\n\n\nfor those who need to rebuild gpu_mns:
\n \nyou need to cythonize \"'gpu_nms.pyx'\" from the attached.
\n \nin box, there is a test code \"def run_check_nms():\", try keep =\n gpu_nms(rois, 0.5), keep = py_nms(rois, 0.5), keep = torch_nms(rois,\n 0.5), you should get same results
\n
Heng have already mentioned it above
", "rawMarkdown": "You should rebuild it ,see:\n\n> for those who need to rebuild gpu_mns:\n> \n> you need to cythonize \"'gpu_nms.pyx'\" from the attached.\n> \n> in box, there is a test code \"def run_check_nms():\", try keep =\n> gpu_nms(rois, 0.5), keep = py_nms(rois, 0.5), keep = torch_nms(rois,\n> 0.5), you should get same results\n\nHeng have already mentioned it above" }, { "id": 292367, "postDate": "2018-03-07T21:23:53.823Z", "rawMarkdown": "" } ] }, { "id": 284998, "postDate": "2018-02-19T02:49:43.613Z", "content": "Thanks! This is really interesting stuff.
", "rawMarkdown": "Thanks! This is really interesting stuff." }, { "id": 284727, "postDate": "2018-02-18T05:31:37.600Z", "content": "Thank you for your great works, @Heng CherKeng. I want to run dummy14.2/train.py. May I ask two things?
\n\nBest,
", "rawMarkdown": "Thank you for your great works, @Heng CherKeng. I want to run dummy14.2/train.py. May I ask two things?\n\n1. If we want to start to train without initial checkpoint, we need to modify top of the run_train() function as \"initial_checkpoint=None\". Is my understanding correct?\n2. If so, I have an error at __getitem__ function in reader.py because of lacking \"multi_mask.npy\" files. Could you tell me how to get \"multi_mask.npy\" files?\n\nBest,", "replies": [ { "id": 284751, "postDate": "2018-02-18T07:34:45.080Z", "content": "set to None is correct
see run_make_annotation() in dataset/process.py
Thank you for your quick reply. I could start training and your training code works well :)
\n\nIf possible, could you tell us your training strategy? For example, do you train gray images and colored images separately?
", "rawMarkdown": "Thank you for your quick reply. I could start training and your training code works well :)\n\nIf possible, could you tell us your training strategy? For example, do you train gray images and colored images separately?", "votes": 1 }, { "id": 284845, "postDate": "2018-02-18T14:56:42.247Z", "content": "treat them as different class. More details will be given later
", "rawMarkdown": "treat them as different class. More details will be given later" }, { "id": 284853, "postDate": "2018-02-18T15:17:27.060Z", "content": "Thank you very much.
", "rawMarkdown": "Thank you very much." }, { "id": 285372, "postDate": "2018-02-20T04:22:46.067Z", "content": "you run the dataset/process.py，sovle the “multi_mask.npy” error？
", "rawMarkdown": "you run the dataset/process.py，sovle the “multi_mask.npy” error？" }, { "id": 286542, "postDate": "2018-02-22T10:59:42.070Z", "content": "Sorry for late reply. The \"multi_mask.npy\" was created by running dataset/process.py.
", "rawMarkdown": "Sorry for late reply. The \"multi_mask.npy\" was created by running dataset/process.py." }, { "id": 286552, "postDate": "2018-02-22T11:18:47.947Z", "content": "Thanks, I have solved this error. \nWhile, I have't fix the \"gpu_nms\", don't understand how to cythonize \"'gpu_nms.pyx'\" from the attached \"box.tar.gz\"
", "rawMarkdown": "Thanks, I have solved this error. \nWhile, I have't fix the \"gpu_nms\", don't understand how to cythonize \"'gpu_nms.pyx'\" from the attached \"box.tar.gz\"" } ] }, { "id": 284191, "postDate": "2018-02-16T18:16:22.640Z", "content": "@Heng CherKeng. Hey! Nice work. I am using this version of Mask R-CNN https://github.com/matterport/Mask_RCNN. I peeked at your code and I liked the idea of random crops and learning rate scheduling. You start with lr=0.01 and keep dividing by iter_acum at each epoch, right? If so, I am confused since you have iter_acum=1. Is iter_acum supposed to increase over time?
", "rawMarkdown": "@Heng CherKeng. Hey! Nice work. I am using this version of Mask R-CNN https://github.com/matterport/Mask_RCNN. I peeked at your code and I liked the idea of random crops and learning rate scheduling. You start with lr=0.01 and keep dividing by iter_acum at each epoch, right? If so, I am confused since you have iter_acum=1. Is iter_acum supposed to increase over time?", "replies": [ { "id": 284195, "postDate": "2018-02-16T18:24:13.520Z", "content": "\"You start with lr=0.01 and keep dividing by iter_acum at each epoch, right?\"
\n\nNo. \"iter_acum\" is not used here. It is for gradient accumulation when your batch cannot fit into memory. Since my patch is 128x128 and i have 4 gpu, they can be fitted into memory. So i do not use \"iter_acum\"
\n\nif \"iter_acum\" is used, say iter_acum=4, then the code would runs as:
\n\n - for each epoch:\n\n - accumulated_gradient = 0\n\n - for i in range(iter_acum):\n\n - subsample a batch x\n\n - compute gradient\n\n - accumulated_gradient += gradient\n\n - update cnn weights using accumulated_gradient\nI see. So you are not changing learning rate during training? Also, are you using some pre-trained weights for initializing the backbone or just training with random initialization?
", "rawMarkdown": "I see. So you are not changing learning rate during training? Also, are you using some pre-trained weights for initializing the backbone or just training with random initialization?" }, { "id": 284199, "postDate": "2018-02-16T18:35:14.003Z", "content": "for the current code, automatic rate schedule is disabled (see \" LR = None \").
\n\ni use manual schedule instead. i.e set rate =0.01 and then run and monitor the loss. when the loss does not improve, i restart the training with rate=0.001, loading the previous weights as initialization.
\n\ni need to check all 4 loss of mask rcnn are balanced and compare each of the 4 loss for train and validation set. So it is easier to use manual schedule.
", "rawMarkdown": "for the current code, automatic rate schedule is disabled (see \" LR = None \").\n\ni use manual schedule instead. i.e set rate =0.01 and then run and monitor the loss. when the loss does not improve, i restart the training with rate=0.001, loading the previous weights as initialization.\n\ni need to check all 4 loss of mask rcnn are balanced and compare each of the 4 loss for train and validation set. So it is easier to use manual schedule. \n", "votes": 3 }, { "id": 284203, "postDate": "2018-02-16T18:41:25.443Z", "content": "That's a great idea. Appreciate the help!
", "rawMarkdown": "That's a great idea. Appreciate the help!" } ] }, { "id": 284078, "postDate": "2018-02-16T15:22:08.107Z", "content": "Great job. I have a question after final step-active contour. After using active contour, you can achieve a smooth boundary. Then, you have to convert it to RLE for submission. For input of RLE, we have two options: using one mask for whole image-like unet segmentation, or individual masks for each cell-like instance segmentation. If you use the second option, could you tell me how do you do?
", "rawMarkdown": "Great job. I have a question after final step-active contour. After using active contour, you can achieve a smooth boundary. Then, you have to convert it to RLE for submission. For input of RLE, we have two options: using one mask for whole image-like unet segmentation, or individual masks for each cell-like instance segmentation. If you use the second option, could you tell me how do you do?", "replies": [ { "id": 284107, "postDate": "2018-02-16T16:11:57.290Z", "content": "there is no active contour step. the contour are just visualization of the instance segmentation mask.
\n\nthe instance segmentation mask is used for submission. for one image, assume each instance segmentation is binary (like the ones given in the train data), you can just encode them individually, while sharing the same image_id.
", "rawMarkdown": "there is no active contour step. the contour are just visualization of the instance segmentation mask. \n\nthe instance segmentation mask is used for submission. for one image, assume each instance segmentation is binary (like the ones given in the train data), you can just encode them individually, while sharing the same image_id. \n\n", "votes": 1 }, { "id": 284281, "postDate": "2018-02-16T21:03:17.797Z", "content": "I see. Regarding to random crop, you randomly crop the original image into 256x256 images then feed it to the network for trainin. For inference, do you resize the testing image to 256x256 images or crop to 256x256 sub-images? If you use first way, the resize may loss the information. If you use the second way, how we can summarize the final image result from result of sub-images? I am having bottleneck in it
", "rawMarkdown": "I see. Regarding to random crop, you randomly crop the original image into 256x256 images then feed it to the network for trainin. For inference, do you resize the testing image to 256x256 images or crop to 256x256 sub-images? If you use first way, the resize may loss the information. If you use the second way, how we can summarize the final image result from result of sub-images? I am having bottleneck in it" }, { "id": 284401, "postDate": "2018-02-17T03:20:43.437Z", "content": "i trained with crops of 128x128. Mask Rcnn is fully convolution, it can take test and train inputs of various sizes. At test test, the full image is input into the network.
\n\ni cannot say about other people code, but for mine, you can take a look at submit.py for inference at full image.
", "rawMarkdown": "i trained with crops of 128x128. Mask Rcnn is fully convolution, it can take test and train inputs of various sizes. At test test, the full image is input into the network.\n\ni cannot say about other people code, but for mine, you can take a look at submit.py for inference at full image.", "votes": 1 }, { "id": 284555, "postDate": "2018-02-17T14:30:56.200Z", "content": "In traditional maskrcnn paper, it used an identical faster rcnn which used fully connected layer. So, do you change the fully connected to fully conv in the original paper?
", "rawMarkdown": "In traditional maskrcnn paper, it used an identical faster rcnn which used fully connected layer. So, do you change the fully connected to fully conv in the original paper? " }, { "id": 287985, "postDate": "2018-02-26T12:03:30.160Z", "content": "For encoding masks, how can I encode individual mask in the same row with image id? Does each mask seperate by comma or something like that?
", "rawMarkdown": "For encoding masks, how can I encode individual mask in the same row with image id? Does each mask seperate by comma or something like that?" } ] }, { "id": 284045, "postDate": "2018-02-16T14:11:05.273Z", "content": "Good work. Thanks for sharing.
\n\nAlthough I may use the tensorflow implementation, I guessI still can use your code for data pre/post processing
", "rawMarkdown": "Good work. Thanks for sharing.\n\nAlthough I may use the tensorflow implementation, I guessI still can use your code for data pre/post processing" }, { "id": 283202, "postDate": "2018-02-15T03:33:40.210Z", "content": "For the ones lacking a GPU and would like to try this one, seems like I am able to run this in Google Colab (https://drive.google.com/open?id=1uE2M_EHYkGlfj2Q7NY45C5TyQJwt2RLQ)
\n\nI am running train.py now and so far no error and I have no idea how long it will take. Anyway, feel free to try it.
\n\nI still have to figure out how to get the result back from Colab but that is for another day.
\n\nUpdate: I might have spoken too soon. It is running but I am not seeing any output also. Will see how long it takes. I have similar issue with onepanel.io
", "rawMarkdown": "For the ones lacking a GPU and would like to try this one, seems like I am able to run this in Google Colab (https://drive.google.com/open?id=1uE2M_EHYkGlfj2Q7NY45C5TyQJwt2RLQ) \n\nI am running train.py now and so far no error and I have no idea how long it will take. Anyway, feel free to try it. \n\nI still have to figure out how to get the result back from Colab but that is for another day.\n\nUpdate: I might have spoken too soon. It is running but I am not seeing any output also. Will see how long it takes. I have similar issue with onepanel.io", "replies": [ { "id": 283204, "postDate": "2018-02-15T03:43:03.767Z", "content": "@Pip Pirrip, I am able to run it on colab as well. Lets see how long...
", "rawMarkdown": "@Pip Pirrip, I am able to run it on colab as well. Lets see how long..." }, { "id": 283280, "postDate": "2018-02-15T06:40:05.683Z", "content": "My suggestion is try to train on a few image first, say 16 to 32 images.Then see if works before scaling up.
\n\nThis will take short time.
\n\nYou should be able to see results on log file and on screen. trained models are saved in \"checkpoint\" folder.
\n\nNote that there is a \"def debug_and_draw(net, inputs, truth_boxes, truth_labels, truth_instances, mode='test')\" in train.py that you can use to draw your results.
\n\nYou may also want to adjust how often the checkpoints (intermediate trained models) are saved. THey are eating up my harddisk space fast!
", "rawMarkdown": "My suggestion is try to train on a few image first, say 16 to 32 images.Then see if works before scaling up.\n\nThis will take short time.\n\nYou should be able to see results on log file and on screen. trained models are saved in \"checkpoint\" folder.\n\nNote that there is a \"def debug_and_draw(net, inputs, truth_boxes, truth_labels, truth_instances, mode='test')\" in train.py that you can use to draw your results.\n\n\nYou may also want to adjust how often the checkpoints (intermediate trained models) are saved. THey are eating up my harddisk space fast!", "votes": 1 }, { "id": 283706, "postDate": "2018-02-15T21:26:58.873Z", "content": "I spoken too soon. Colab doesn't complain even thou I forgot to run \"run_make_annotation()\" before training. Anyone know where I can get the file \"train1_ids_all_670\"? I have \"train_ids_all_670\" (no '1' after train) from an earlier version but not the latest one
", "rawMarkdown": "I spoken too soon. Colab doesn't complain even thou I forgot to run \"run_make_annotation()\" before training. Anyone know where I can get the file \"train1_ids_all_670\"? I have \"train_ids_all_670\" (no '1' after train) from an earlier version but not the latest one" }, { "id": 283779, "postDate": "2018-02-16T00:25:58.313Z", "content": "The split files are in the example_results folder of the google drive. Or you can use your file. Both file ( with and without “1” are the same)
", "rawMarkdown": "The split files are in the example_results folder of the google drive. Or you can use your file. Both file ( with and without “1” are the same)", "votes": 1 }, { "id": 283780, "postDate": "2018-02-16T00:29:38.907Z", "content": "Note that you have to r compile some of the *.so lib for your machine. They are in the mask_rcnn_lib folder, namely nms, overlap, align_roi
", "rawMarkdown": "Note that you have to r compile some of the *.so lib for your machine. They are in the mask_rcnn_lib folder, namely nms, overlap, align_roi", "votes": 1 }, { "id": 283817, "postDate": "2018-02-16T02:33:22.730Z", "content": "Thanks @Heng CherKeng! I always wanted to try you model but only recently I figured out how to run yours with some free GPU services. Not really working yet but at least I am making progress.
", "rawMarkdown": "Thanks @Heng CherKeng! I always wanted to try you model but only recently I figured out how to run yours with some free GPU services. Not really working yet but at least I am making progress. " }, { "id": 284740, "postDate": "2018-02-18T06:42:46.623Z", "content": "Thanks @Heng CherKeng. Since I use python3.5, I managed to build .so libs however seem some file missing to build .../nms/gpu_nms.cpython-36m-x86_64-linux-gnu.so? \nHow do you build this so file?
", "rawMarkdown": "Thanks @Heng CherKeng. Since I use python3.5, I managed to build .so libs however seem some file missing to build .../nms/gpu_nms.cpython-36m-x86_64-linux-gnu.so? \nHow do you build this so file? " }, { "id": 284748, "postDate": "2018-02-18T07:24:18.673Z", "content": "i think this file (gpu_nms.cpython-36m-x86_64-linux-gnu.so) is not used. I will confirm again when i am back in office tmr.
\n\ncython_box_overlap-xxx.so and _extension.so (there are two pytorch extension) are used. see initial setup document: fpn50_maskrcnn.pptx as attached at top of this thread
", "rawMarkdown": "i think this file (gpu_nms.cpython-36m-x86_64-linux-gnu.so) is not used. I will confirm again when i am back in office tmr.\n\ncython_box_overlap-xxx.so and _extension.so (there are two pytorch extension) are used. see initial setup document: fpn50_maskrcnn.pptx as attached at top of this thread" }, { "id": 284754, "postDate": "2018-02-18T07:50:35.187Z", "content": "Thanks @Heng CherKeng, it wasn't needed however model/mask_rcnn_lib/box.py was importing on line 4, once commented out, fixed it. Now I see issue @RK posted. Will check it tomorrow. Thanks a lot for the quick reply.
", "rawMarkdown": "Thanks @Heng CherKeng, it wasn't needed however model/mask_rcnn_lib/box.py was importing on line 4, once commented out, fixed it. Now I see issue @RK posted. Will check it tomorrow. Thanks a lot for the quick reply." }, { "id": 285576, "postDate": "2018-02-20T16:23:33.897Z", "content": "Google Colab has limited on the session time, only 12 hours per session.
", "rawMarkdown": "Google Colab has limited on the session time, only 12 hours per session." }, { "id": 285653, "postDate": "2018-02-20T19:44:42.850Z", "content": "seems like a few ppl looking at my colab notebook. The seperated the \"run_make_annotation()\" into another notebook (https://drive.google.com/open?id=1nNbNlQxIsYwhqQXkebfunpImou58_O6I) I believe it works, but I have to figure out how to get the mask file generated into the main notebook (or just merge the two). Without a decent GPU rig (or willing to pay for GPU time) is relatively painful.
", "rawMarkdown": "seems like a few ppl looking at my colab notebook. The seperated the \"run_make_annotation()\" into another notebook (https://drive.google.com/open?id=1nNbNlQxIsYwhqQXkebfunpImou58_O6I) I believe it works, but I have to figure out how to get the mask file generated into the main notebook (or just merge the two). Without a decent GPU rig (or willing to pay for GPU time) is relatively painful." }, { "id": 290380, "postDate": "2018-03-03T22:59:41.167Z", "content": "@Pip Pirrip, were you able to generate the mask files and successfully train? If so, can you share your process? I am looking at Hengs code to try and do the same.
", "rawMarkdown": "@Pip Pirrip, were you able to generate the mask files and successfully train? If so, can you share your process? I am looking at Hengs code to try and do the same. " }, { "id": 290383, "postDate": "2018-03-03T23:10:15.390Z", "content": "no - I thought I solved the pytorch issue with colab, but I end up with CUDA issue (can't compile CUDA there). I am switching to keras now
", "rawMarkdown": "no - I thought I solved the pytorch issue with colab, but I end up with CUDA issue (can't compile CUDA there). I am switching to keras now" } ] }, { "id": 282722, "postDate": "2018-02-14T13:03:10.693Z", "content": "congrats, great work, guess the score will be pretty high\nwill compare with MaskRCNN and Detectron as soon as I have time
", "rawMarkdown": "congrats, great work, guess the score will be pretty high\nwill compare with MaskRCNN and Detectron as soon as I have time" }, { "id": 429355, "postDate": "2018-11-28T18:24:56.500Z", "content": "thanks
", "rawMarkdown": "thanks" }, { "id": 307508, "postDate": "2018-04-01T21:37:10.530Z", "content": "Thanks! Great kernel.
", "rawMarkdown": "Thanks! Great kernel." }, { "id": 300827, "postDate": "2018-03-22T00:43:52.920Z", "content": "thank you
", "rawMarkdown": "thank you" }, { "id": 284194, "postDate": "2018-02-16T18:22:06.563Z", "content": "Great job. Thanks for kernel!
", "rawMarkdown": "Great job. Thanks for kernel! " }, { "id": 283213, "postDate": "2018-02-15T04:00:57.243Z", "content": "Thanks @Heng CherKeng as always.
", "rawMarkdown": "Thanks @Heng CherKeng as always." } ], "comments": [ { "id": 284625, "author_name": "Cut Onion", "author_url": "", "post_date": "2018-02-17T19:35:41", "content": "Thanks again Heng for providing great baseline!\nbtw, you're famous now (spotted somewhere in the russian internet): \n
For reference, this meme is not really known outside Russia, so here is my attempt at translating the original: https://i.imgur.com/gClRuVH.jpg
", "votes": 28, "replies": [] }, { "id": 284655, "author_name": "nicapotato", "author_url": "", "post_date": "2018-02-17T21:18:07.757000", "content": "yay
", "votes": 0, "replies": [] } ] }, { "id": 284072, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-16T15:12:08.790000", "content": "Github version will be much better.
", "votes": 16, "replies": [] }, { "id": 284217, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-16T19:07:24.817000", "content": "suggested way to train mask rcnn:
\n\nThere is no training, just let prediction = target ground truth for each of the rpn, rcnn and mask heads. Then measure the metric accuracy. We expect about 0.95 to 1.00 accuracy. There are several cropping operations and down-sizing/up-sizing of the mask etc. When we define a bounding box as x0,y0,x1,y1, are these coordinates inclusive or exclusive? should i predict instance mask at 14x14, 16x16 or 28x28? These may affects results. This step verify that such operations do not result in significant drop in accuracy.
Train with small set, say 32 images. valid with the same 32 train images and measure the metric accuracy. There is no generalization problem and we expect about 0.85 to 0.95 accuracy. Because i am using cropped patches, we do not see all samples at one iteration during training. Use this step to check the loss of different rpn, rcnn and mask heads. Are they balanced? What loss is required for a metric of 0.95? Can your network overfit you data?
Now this is the actual training! Use the loss values in step.2 above to decide if there is generalization problem now. if so, will dropout helps? how about data augmentation?
Task loss balancing. Training should proceed such that rpn loss decrease first, then followed by rcnn and then mask. This is because rcnn depends on rpn proposals and mask depends on rcnn proposals.
Class loss balancing. Especially for rcnn part, you may need to use OHEM or focal loss. for rpn part you may need focal loss.
in summary, ask this question: \"what is the upper bound accuracy of your pipeline?\"
", "votes": 12, "replies": [ { "id": 284226, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-02-16T19:17:27.963000", "content": "Hmm... So you are only training the heads not the backbone. And for the backbone you are using resnet50, right?
\n\nI like the idea of training on a smaller sample first. That way I don't have to wait for hours to get an idea of how the model is doing.
\n\nAs for the augmentation, I am only using random flips left to right and up to down.
", "votes": 1, "replies": [] }, { "id": 284227, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-16T19:19:02.437000", "content": "both the backbone and heads are trained.
", "votes": 3, "replies": [] }, { "id": 287931, "author_name": "Keven Wang", "author_url": "", "post_date": "2018-02-26T08:56:49.493000", "content": "Hi Heng, this is one of the most important tip i heard, the training strategy as i don't have much GPU resource, instead of waiting for a day, this is really speed up things. When train with small set (step 2), will network suffer from overfitting? The reason i am asking is that, i am using another Mask RCNN implementation (https://github.com/matterport/Mask_RCNN) and trying to train entire backbone network, even if using full training set, network starts to overfit after 40 epoch (only flip left/right argumentation), loss continuously decreasing while val_loss starts to increase. So i am afraid just taking 32 images will be the same pb, or did i miss anything?
", "votes": 0, "replies": [] } ] }, { "id": 286555, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-22T11:33:47.773000", "content": "for cython_mns.so and gpu_mns.so, please refer to attachment here. (you can ignore the rest of unrelated files here. i will release dummy-14.7 later)
", "votes": 7, "replies": [ { "id": 286872, "author_name": "avsanjay", "author_url": "", "post_date": "2018-02-23T00:07:53.587000", "content": "thanks for the code. It has been a great learning experience. Where can i find resize_to_fractor or pad_to_factor? Also i run into this strange error when i am running the test data set about illegal memory access at THCTensorcopy.c. This seems to get trigerred at mask_rcnn_resnet_50_fpn_net.py/forward/index.append(ids[mask]. The origin seems to be \nrcnn_crops = self.crop(features, self.rpn.proposals). Any pointers would be much appreciated
", "votes": 1, "replies": [] }, { "id": 286951, "author_name": "avsanjay", "author_url": "", "post_date": "2018-02-23T04:35:18.920000", "content": "Thank you. I found the functions in transform.py
", "votes": 0, "replies": [] }, { "id": 290089, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2018-03-02T22:40:38.660000", "content": "Hi Heng!
\n\nWhat hardware did you use to run the dummy-14.7 code with batch_size=2?
open source pytorch mask rcnn:
\n\nhttps://github.com/soeaver/Pytorch_Mask_RCNN
", "votes": 8, "replies": [] }, { "id": 282671, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-14T11:32:19.687000", "content": "block diagrams
\n\n
Great job! The original paper used parallel pipeline to predict detection class and mask in parallel. But you do not seem to follow it. Is it the result of your experiments?
", "votes": 0, "replies": [] }, { "id": 284908, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-18T19:00:31.407000", "content": "the paper mention:
\n\n\"Inference: At test time, the proposal number is 300 for the\nC4 backbone (as in [36]) and 1000 for FPN (as in [27]). We\nrun the box prediction branch on these proposals, followed\nby non-maximum suppression [14]. The mask branch is\nthen applied to the highest scoring 100 detection boxes. Although\nthis differs from the parallel computation used in\ntraining, it speeds up inference and improves accuracy (due\nto the use of fewer, more accurate RoIs)\"
", "votes": 2, "replies": [] }, { "id": 285023, "author_name": "delta_K", "author_url": "", "post_date": "2018-02-19T04:53:45.267000", "content": "Oh sorry it's my mistake. If you don't mind, I still have another question. The resnet 50 use stride 2 conv and maxpooling on the first stage, and stride 1 on the second stage, which seems different from your implementation. Is that true?
", "votes": 0, "replies": [] }, { "id": 285025, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-19T04:57:15.027000", "content": "yes. this is to detect small nuclei of radius 4 pixel
", "votes": 2, "replies": [] }, { "id": 302512, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2018-03-24T08:04:29.940000", "content": "Asian people always love diagrams! It's a hieroglyphics mind style.
", "votes": -3, "replies": [] } ] }, { "id": 306198, "author_name": "William Green", "author_url": "", "post_date": "2018-03-30T02:49:16.570000", "content": "Has anyone encountered ModuleNotFoundError: No module named 'net' ? I seemed to be stuck there. Any assistance would be greatly appreciated.
\n\n---------------------------------------------------------------------------\n ModuleNotFoundError Traceback (most recent call last)\n <ipython-input-4-daf7d55d62fc> in <module>()\n 1 #from common import *\n ----> 2 from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\n 3 from net.lib.box.nms.torch_nms import torch_nms\n 4 from net.lib.box.nms.gpu_nms import gpu_nms\n 5 from net.lib.box.nms.cython_nms import cython_nms\n\n ModuleNotFoundError: No module named 'net'\nDid y get any error when execute \"from common import * \" ?
", "votes": 0, "replies": [] }, { "id": 306211, "author_name": "William Green", "author_url": "", "post_date": "2018-03-30T03:22:15.883000", "content": "Yes, I get no module found.
", "votes": 0, "replies": [] }, { "id": 306218, "author_name": "ianchan1990", "author_url": "", "post_date": "2018-03-30T03:53:09.830000", "content": "Add this two lines in the front of file:\nimport sys\nsys.path.append('../')
", "votes": 0, "replies": [] }, { "id": 306224, "author_name": "William Green", "author_url": "", "post_date": "2018-03-30T04:14:42.113000", "content": "same error
", "votes": 0, "replies": [] }, { "id": 307166, "author_name": "kirk", "author_url": "", "post_date": "2018-04-01T01:00:21.417000", "content": "cd into the mask_rcnn_xxx.a folder\nthen execute\nexport PYTHONPATH=\"pwd\"
\n\nReplace \" with ` in pwd command.
", "votes": 0, "replies": [] }, { "id": 308892, "author_name": "john1231983", "author_url": "", "post_date": "2018-04-04T09:05:07.047000", "content": "I got same error when install in the TitanXp. I changed PYTHONPATH and sys but it does not help. Any solution? Thanks
\n\n set random seed\n SEED=35202\n set cuda environment\n torch.__version__ = 0.4.0a0+b125033\n torch.version.cuda = 9.1.85\n torch.backends.cudnn.version() = 7005\n os['CUDA_VISIBLE_DEVICES'] = 0\n torch.cuda.device_count() = 1\n torch.cuda.current_device() = 0\n\nTraceback (most recent call last):\n File \"train_3.py\", line 6, in <module>\n from dataset.reader import *\n File \"/home/john/maskrcnn/build/mask-rcnn-resnet50-ver-01.a/dataset/reader.py\", line 11, in <module>\n from net.lib.box.process import *\n File \"/home/john/maskrcnn/build/mask-rcnn-resnet50-ver-01.a/net/lib/box/process.py\", line 7, in <module>\n from net.lib.box.overlap.cython_box_overlap import cython_box_overlap\nModuleNotFoundError: No module named 'net.lib.box.overlap.cython_box_overlap'\nI haven't figured it out. How ever, 城璐 has a solution that allows you to add your cuda (gpu) version. \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/53588#308848
", "votes": 0, "replies": [] }, { "id": 309291, "author_name": "john1231983", "author_url": "", "post_date": "2018-04-05T01:21:15.173000", "content": "I solved it. The reason is that after rebuild it, you have to copy *.so to root folder as the figure in the document of Heng. For example, cython_box_overlap.cpython-36m-x86_64-linux-gnu.so to overlap folder. That all
", "votes": 0, "replies": [] } ] }, { "id": 297598, "author_name": "Yuhsuan", "author_url": "", "post_date": "2018-03-17T14:02:01.520000", "content": "Hello Heng CherKeng, i have tried to run your code, but i got error when compiling. I use Pytorch 0.3.0 with Anaconda. After running python build.py, the error is below:\ncffi.error.VerificationError: CompileError: command 'gcc' failed with exit status 1
\n\nIs it because of the version of Pytorch? Is Pytorch 0.4.0 necessary?
", "votes": 3, "replies": [] }, { "id": 294171, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-11T13:05:59.993000", "content": "some tip:
\n\nchanging from resnet50 to se-resnext50 backbone improve the gray image score from 0.419 to 0.430.
\n\nnote:
\n\nif you want to get high LB score, you most improve the mask non-max suppression provided. Together with other improvement like class balancing, adjusting network structure, etc you can above 0.450 just with the gray images alone.
", "votes": 3, "replies": [ { "id": 294381, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2018-03-11T20:58:50.623000", "content": "Thank you very much Heng! for all the ideas, suggestions and open-source code! \n I am running your latest code on a K80 on google cloud compute. I was able to compile everything and follow all the steps in your pdf :-). \n i am running the 1st experiment with grayscale images to make 00016500_model.pth.\nFor me 17k iterations took 16hrs with image size of 128-by-128:
\n\n17.0 k 544.0 0.3 m | 0.335 0.02 0.02 0.17 0.01 0.12 | 0.440 0.01 0.02 0.23 0.01 0.16 | 0.270 0.00 0.02 0.11 0.01 0.13 | 16 hr 07 min 16999,16999,
\n\nOne basic question: I saw in train_0.py that num_iters = 1000 *1000; does this mean 1000k iterations ? can i use any early stopping criteria or experiment with num_iters ?
\n\nThanks\nMehul
", "votes": 0, "replies": [] }, { "id": 294391, "author_name": "Chen Zhang", "author_url": "", "post_date": "2018-03-11T21:51:25.637000", "content": "Hi Heng, I was trying to modify the network structure but getting some error when running submit.py, as I mentioned two days ago. I just changed some parameters in the model.py. It seems the net.detections has a size of 0. Do you have any comments on this issue?
", "votes": 1, "replies": [] } ] }, { "id": 291540, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-06T10:19:14.473000", "content": "the latest version (mask-rcnn-resnet50-ver-01.a) is up! See folder 20180306 and readme.pdf
\n\nsubmission code and sample image/csv results are provided.
sample model and training log (train parameters and loss at each iteration) are provided
Hi. Could you tell me the gain do you achieve when you use focal loss? How do you compute the label weight for the loss. I currently use binary softmax loss. Thanks
", "votes": 0, "replies": [] }, { "id": 293455, "author_name": "hirviö", "author_url": "", "post_date": "2018-03-09T22:05:11.877000", "content": "focal loss is used for single step (stage) object detection, one doesnt need to use focal loss if you you're using a two stage network like MaskRCNN which has the region proposal network (RPN) as 1st stage.
", "votes": 0, "replies": [] } ] }, { "id": 285163, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-19T13:13:37.690000", "content": "during training, you can ignore prediction boxes (for rpn, rcnn and mask) that are too small or those at the boundary of the image. you can try to modify my code (see make_rpn_target(), make_rcnn_target(), make_mask_target()). i.e. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\".
\n\nAttached is some modification i made for rpn. I will post the rest of complete solution for rcnn and mask later.
\n\nspend some time to write code to visualize results, e.g. mark the hit/miss true-positive/false-positive detections of rcnn. you can discover many improvement points. Draw the predicted and ground truth mask as well.
\n\nthis is the reason why i write my own mask rcnn code instead of using the open source. it is to have complete control and visualize results. I think after several observations of the results, you can think of ways to get LB in the range of 0.50 to 0.55 range.
", "votes": 4, "replies": [ { "id": 285235, "author_name": "xuhuaren", "author_url": "", "post_date": "2018-02-19T16:59:12.333000", "content": "Thanks for your wonderful work. Appreciate it! I really like your pytorch maskrcnn code. While I have some questions:\n1. I finished the training processing and find all the testing color images are failed, the results are poor, no bounding box got. So, I must training the model separate by color and gray? Is it enough for training samples.\n2. I submit the results, only using (masks = net.masks) results, while the performance only have 0.33. Anything wrong of my inference processing. I know four rgb image are failed, but this may lead score drop from 0.46 to 0.33?\n3. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\". So you mean I must delete tiny object in the training processing, it lead to great improvement? Interesting, why?
\n\nThanks for your code again. I really like the visualize results. And it has the same performance compared with caffe2 maskrcnn.
", "votes": 1, "replies": [] }, { "id": 285330, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-20T00:29:44.770000", "content": "i visualize the errors of rpn and rcnn. Many at are the boundary of the images and are ambiguous (i.e. if you were annotate a nucleus by hand, it is sometimes hard to decide if you want to mark it when say only 10% of it is visible). In fact in academia dataset like VOC or Caltech Pedestrian dataset, the ground truth box is given the attribute \"difficult\" or \"don't care\", which means that loss is not computed for these cases.
\n\nfrom the faster rcnn paper:
\n\n\"The anchor boxes that cross image boundaries need to be handled with care. During training, we\nignore all cross-boundary anchors so they do not contribute to the loss. For a typical 1000 × 600\nimage, there will be roughly 20k (≈ 60 × 40 × 9) anchors in total. With the cross-boundary anchors\nignored, there are about 6k anchors per image for training. If the boundary-crossing outliers are not\nignored in training, they introduce large, difficult to correct error terms in the objective, and training\ndoes not converge\"
", "votes": 3, "replies": [] }, { "id": 293756, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-10T14:21:36.440000", "content": "Training with mixed gray and color images seems to be difficult, can you give me some suggestions on how to deal with different colors?
", "votes": 0, "replies": [] } ] }, { "id": 282783, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-02-14T15:35:00.847000", "content": "Cool. Mask R-CNN in PyTorch is still rare today. Placing the code on github will make it easier for people to follow and take part in.
", "votes": 4, "replies": [] }, { "id": 309590, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-05T16:29:34.247000", "content": "You should try to understand and adjust the parameters in the configuration.py. E.g. score threshold, overlap threshold of rpn, rcnn and mask
\n\nYou should draw some visualization to see effect of the visualization. It is possible to achieve 0.45 for gray 53 images with se-resnext50 (and maybe also resnet50?)
", "votes": 1, "replies": [] }, { "id": 309558, "author_name": "AngMingliang", "author_url": "", "post_date": "2018-04-05T15:05:26.763000", "content": "Did anyone get this error msg when they tried to train their model
\n\nTHCudaCheck FAIL file=/home/uesu/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory\nTraceback (most recent call last):\n File \"train_0.py\", line 515, in <module>\n run_train()\n File \"train_0.py\", line 347, in run_train\n net( inputs, truth_boxes, truth_labels, truth_instances )\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n result = self.forward(*input, **kwargs)\n File \"/home/uesu/20180306/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/model.py\", line 400, in forward\n self.mask_logits = data_parallel(self.mask_head, mask_crops)\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py\", line 142, in data_parallel\n return module(*inputs[0], **module_kwargs[0])\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 371, in __call__\n result = self.forward(*input, **kwargs)\n File \"/home/uesu/20180306/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/model.py\", line 329, in forward\n x = F.relu(self.bn1(self.conv1(crops)),inplace=True)\n File \"/home/uesu/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py\", line 621, in relu\n return torch.relu_(input)\nRuntimeError: cuda runtime error (2) : out of memory at /home/uesu/pytorch/aten/src/THC/generic/THCStorage.cu:58\nI am training this inside a nvidia enabled docker container on a Titian xp, based on what i have found this error is caused by the lack of shared memory. Does anybody know what is the shared memory used by the model and how to address this issue?
", "votes": 1, "replies": [ { "id": 309928, "author_name": "AngMingliang", "author_url": "", "post_date": "2018-04-06T07:58:38.410000", "content": "Hi guys, managed to solve this issue this morning. I reduced the batch size to 8 and got the model to work. If anybody has any other suggestion you can post on this thread.
", "votes": 2, "replies": [] }, { "id": 361121, "author_name": "JinningLi", "author_url": "", "post_date": "2018-07-23T21:51:31.297000", "content": "I got the same problem in my code. And reducing batch size also solved the problem! \nDo you know why this happens?
", "votes": 0, "replies": [] } ] }, { "id": 308711, "author_name": "john1231983", "author_url": "", "post_date": "2018-04-04T00:51:29.880000", "content": "Hello Heng, I have a question with pretrain model in seresnetxt. I have successful to convert caffe model to pytorch model. It worked as expected in classification. Now, I apply it to your code, I found something:
\n\nThe original seresnetxt-50 uses max-pooling in layer_c0, but you did not use it. As my opinion, your crop size is 256, so if we use it, the resolution will divide by 4, thus we lost many information and some small mask will be come zeros. Hence, the original seresnetxt-50 may not good in this challenge. And you remove the pooling for the purpose. Am I right?
To use pretrain model, we have to insert the max-pooling in the layer_c0, I get error at the first epoch in evaluate. Have you success to use pretrain model?
\n\n\n valid_loss, valid_acc = evaluate(net, valid_loader)\n File \"train_2.py\", line 106, in evaluate\n net(inputs, truth_boxes, truth_labels, truth_instances )\nFile \"/home/john/heng_pytorch/build/mask-rcnn-resnet50-ver-01.a/net/resnet50_mask_rcnn/layer/rpn_multi_target.py\", line 248, in make_one_rpn_target\n label_weight[bg_index] = num_fg/num_bg\n ZeroDivisionError: division by zero\n
i haven't make comparison, so i cannot say for sure. if you still one to use the original seresnetxt-50, you can always up scale your image for input
\n\ni think you have to change your anchor size, etc ... in the configuration file. currently it is assumed that the first feature layer (of the fpn) is 1/2 of input image, second feature is 1/4 , etc ... You have to change the configuration.py
", "votes": 0, "replies": [] } ] }, { "id": 302345, "author_name": "", "author_url": "", "post_date": "2018-03-23T23:44:04.630000", "content": "While training with multi-gpu i get this error
\n\nloss.backward()\n File \"/home/ashwin/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py\", line 120, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph)\n File \"/home/ashwin/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py\", line 81, in backward\n variables, grad_variables, retain_graph, create_graph)\nRuntimeError: arguments are located on different GPUs at /home/ashwin/pytorch/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:231\nAny idea how to resolve this ?
", "votes": 1, "replies": [ { "id": 302349, "author_name": "Yibing Wu", "author_url": "", "post_date": "2018-03-24T00:03:33.443000", "content": "try to comment out \" os.environ['CUDA_VISIBLE_DEVICES'] = 0 # '3,2,1,0' \".
", "votes": 0, "replies": [] }, { "id": 302441, "author_name": "", "author_url": "", "post_date": "2018-03-24T03:48:23.960000", "content": "I did comment this line but still getting the error
", "votes": 0, "replies": [] }, { "id": 302455, "author_name": "Yibing Wu", "author_url": "", "post_date": "2018-03-24T04:43:42.730000", "content": "I followed the instructions. I first saw the same error as you posted and it was gone after I commented out the line.
", "votes": 0, "replies": [] }, { "id": 304884, "author_name": "Dawid Rymarczyk", "author_url": "", "post_date": "2018-03-28T07:05:39.413000", "content": "I've got the same problem. I found that we should use self.register_buffer('variable', variable), but still no idea where put it in the code.
Thanks, Heng, I am trying run your code. It is super fast than Keras based. I will try to convert my scheme to your code to run it. I have some questions:
\n\nHow do you separate purple and gray image into two subsets? Using K-means?
Does your code have any license? Because we have to submit the source code to Kaggle, thus, if I modified it and achieve top-5 performance (Hope!!!), Is it fine to use your code as the baseline and modify from it?
Why not using pre-train for backbone as resnet-50? In my Keras code, pre-train is much better. The training schedule likes: train on heads, then train all. You are train all in current code
Hi, I was trying to run Heng's code, too. But I encounter a problem when running 'train_0.py',
\n\nTraceback (most recent call last):\n File \"/data/yiwei/fun/Bowl/pytorch-maskrcnn/20180306/build/mask-rcnn-resnet50-ver-01.a/train_0.py\", line 515, in <module>\n run_train()\n File \"/data/yiwei/fun/Bowl/pytorch-maskrcnn/20180306/build/mask-rcnn-resnet50-ver-01.a/train_0.py\", line 351, in run_train\n loss.backward()\n File \"/usr/local/lib/python3.5/dist-packages/torch/autograd/variable.py\", line 120, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph)\n File \"/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py\", line 81, in backward\n variables, grad_variables, retain_graph, create_graph)\nRuntimeError: element 0 of variables does not require grad and does not have a grad_fn\nThe error points to the loss.backward(), do you have any ideas?\nThank you very much
Hi Heng! Thank you for your opensource contribution to Kaggle community. I have a little question - I started training using all data, not only grayscale but for color images I get very bad results after 12k iterations, for gray they are ok and close to yours. Can you pls help to undrstand what Iam missing? I use your last code.
", "votes": 1, "replies": [ { "id": 294170, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-11T13:02:00.730000", "content": "for color images, you need to treat it as a separate class.
\n\nif you treat all images as single class, then the common thing for all of them is only outline, which is very weak feature.
\n\nif you treat them as separate classes, you will have more common things within a class (e.g. the interior is also the same within the class)
\n\nI train two separate mask-rcnn, one on gray images and another on purple images (converted to gray scale). both perform reasonably well.
", "votes": 4, "replies": [] }, { "id": 294440, "author_name": "Paul Chen", "author_url": "", "post_date": "2018-03-12T00:48:42.320000", "content": "Hi Heng, how much improvement do you think when you add focal loss to the rpn class graph?
", "votes": 0, "replies": [] } ] }, { "id": 290097, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-02T22:52:20.517000", "content": "Hi Heng, can you comment on how to reproduce LB 0.463 using dummy-14.2? We run your code, but feels it not easy to reproduce the LB score.
", "votes": 1, "replies": [ { "id": 290129, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-03T02:23:18.247000", "content": "what score are you getting? should be in the 4.0 to 4.5 range?
", "votes": 1, "replies": [] }, { "id": 290362, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-03T21:47:46.797000", "content": "So far we got 0.377
", "votes": 0, "replies": [] }, { "id": 291486, "author_name": "Ashish Lal", "author_url": "", "post_date": "2018-03-06T08:16:06.573000", "content": "I am also using dummy-14.2. I get a validation loss of 0.193 using train1_ids_gray_only1_500 and valid1_ids_gray_only1_43 after training for num_iters=1000*50. Hang is correct, removing small boxes improves validation loss isgnificantly. Is this good validation loss that can get you in the 0.4 to 0.45 range?
", "votes": 0, "replies": [] }, { "id": 291489, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-06T08:21:27.117000", "content": "Please wait for a while. I am releasing the final version mask-rcnn-resnet 50 end of today, together with train logfiles, trained models and submission CVS. Hopefully, you don't have to tune parameters and run this version as it is.
\n\nAll trained model and results are on gray images only. By submitting 53 gray images, you get LB score of 0.419 (i.e. 0.5139 = 0.419/65*53 per image on average)
", "votes": 6, "replies": [] }, { "id": 291530, "author_name": "yyll008", "author_url": "", "post_date": "2018-03-06T09:45:45.883000", "content": "Thanks, you do lots of great work!
", "votes": 0, "replies": [] }, { "id": 291536, "author_name": "Ashish Lal", "author_url": "", "post_date": "2018-03-06T10:06:16.413000", "content": "Heng, \nThanks! I will figure it out. You have given enough information already. Thanks for the excellent work. Hope to see you reach the #1 position.
\n\nYou write all this code from scratch. Awesome!
", "votes": 0, "replies": [] } ] }, { "id": 285539, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-20T14:54:10.170000", "content": "How do deal with different color schemes? Do you convert to grayscale or use original images?
", "votes": 1, "replies": [ { "id": 285541, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-20T14:57:01.937000", "content": "i haven't decide for mask rcnn yet. i intend to use both gray-scale and color. i.e. three class:
\n\nwhite on black
black on white
color
in my previous unet results, i just use color.
", "votes": 1, "replies": [] } ] }, { "id": 285537, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-20T14:46:15.007000", "content": "error analysis on dummy images:
\n\ntop image:
\n\nmiddle image:
\n\nbottom image:
\n\nconclusion:
\n\nthese errors should be corrected
\n\nsome pixel shift from roi align
16x16 mask prediction is not sufficient. maybe i should try 24x24 or 28x28
\n\n
there is a bug in crop_instance() for make_mask_target().
\n\n\"crop = instance[y0:y1,x0:x1]\" is wrong because the box coordinates are inclusive. I estimate this case a loss of about 0.02 loss in the LB score. Hope the the LB score is higher after this fix.
\n\nAlso i changed the implementation of roi_align to this: https://github.com/longcw/RoIAlign.pytorch
\n\nI will release all fix later.
\n\ncorrected code:
\n\ndef crop_instance(instance, box, size, threshold=0.5):\nH,W = instance.shape\nx0,y0,x1,y1 = np.rint(box).astype(np.int32)\n\n# x0 = max(0,x0) #input should already be clipped\n# y0 = max(0,y0)\n# x1 = min(W,x1)\n# y1 = min(H,y1)\n#print(x0,y0,x1,y1)\n\ncrop = instance[y0:y1+1,x0:x1+1]\ncrop = cv2.resize(crop,(size,size))\ncrop = (crop>threshold).astype(np.float32)\nreturn crop\nThanks for your wonderful work. Appreciate it! I really like your pytorch maskrcnn code. While I have some questions: \n1. I finished the training processing and find all the testing color images are failed, the results are poor, no bounding box got. So, I must training the model separate by color and gray? Is it enough for training samples. \n2. I submit the results, only using (masks = net.masks) results, while the performance only have 0.33. Anything wrong of my inference processing. I know four rgb image are failed, but this may lead score drop from 0.46 to 0.33?\n 3. there will be no loss to back-propagate from these boxes. It will improve results \"significantly\". So you mean I must delete tiny object in the training processing, it lead to great improvement? Interesting, why?\nThanks for your code again. I really like the visualize results. And it has the same performance compared with caffe2 maskrcnn.
", "votes": 1, "replies": [ { "id": 286968, "author_name": "bintan_FDU", "author_url": "", "post_date": "2018-02-23T05:08:27.297000", "content": "May I ask a question about the submit.py? In the submit file I found this line :
\n\ntest_dataset = ScienceDataset(\n #'test1_ids_gray_only_53', mode='test',\n #'train1_ids_gray_only1_500', mode='test',\n 'valid1_ids_gray_only1_43', mode='test',\n transform = submit_augment)
\n\nI think it means I did not predict the test data but predicted the verification data，What the correct code should be?
", "votes": 0, "replies": [] } ] }, { "id": 283750, "author_name": "Brasnold", "author_url": "", "post_date": "2018-02-15T23:18:17.827000", "content": "Great work!
\n\nAs the trunk of the model you have chosen the resnet50, pretrained I suppose. I don't know in details this architecture but if I'm not mistaken, it's part of a classifier model. Is it so directly shareable with a segmentation based task? Recent works such as google's Multimodel show us how multitasking / weight sharing can find improvements in \"hopeless places\", so I'm not surprised that it works very well, but I'm curious to know the reasons for this choice.
\n\nEdit: looking at the papers https://arxiv.org/pdf/1512.03385.pdf and https://arxiv.org/pdf/1610.02915.pdf i've understood that atleast the architecture fits the problem, it preserves the input dimensions and low level informations, necessary for segmentation. There's also a U-net like version https://link.springer.com/chapter/10.1007/978-3-319-70096-0_38?no-access=true .
", "votes": 1, "replies": [] }, { "id": 283286, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-15T07:04:26.967000", "content": "please see \"20180214/dummy-14.2\" and \"20180214/example_results/14.2\".
\n\ni added a trained model for your experiment, debugging, etc you can produce results like this on the whole image:
\n\n
Hi Heng, I managed to run your 20180314 version. However, I think I am not running it with full function because of your modifications in mask_nms, which is called here in the model.py:
self.masks, self.mask_proposals = mask_nms(cfg, mode, inputs, self.rcnn_proposals, self.mask_logits)\n\nself.detections = self.mask_proposals\nWhat is self.mask_proposals? In my current running, I just change the above two lines back to the 20180306 version.
Also I modified crop = self.crops[l](fs[l], proposals[index].view(-1,8)) to crop = self.crops[l](fs[l], proposals[index].view(-1,7)).
@Yan: Me too. I also got the error and thanks for your points to solve it. I am running it for training from scratch. What is your performance when ran the se-resnext version? Do you train with full images (color+gray)? For version of 20180306, I achieved 0.44 LB on full image set (too low than what I expected). I think train with pretrain model will be better. Do you try with pretrain model?
", "votes": 0, "replies": [] }, { "id": 296620, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-15T14:28:44.803000", "content": "I am running se-resnext with only gray images now, not finished yet. Also I do not use pretrained weights. Using imagenet weight for resnet is possible from the official pytorch weights, I am not sure about resnext or se-resnext. You are using cuda 8 or cuda 9? I am using cuda 9, but want to know if anyone succeeds with cuda 8.
", "votes": 0, "replies": [] }, { "id": 296642, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-15T14:49:58.267000", "content": "Good. I am also running it in cuda 9.0 and cudnn 7.0.5 using gray only. My step size is [0.01 in 10k, 0.001 in 20k, 0.0001 in 25k] I hope it will complete tomorrow. Actually, it takes more GPU resource than resnet50, so I can only run with batch size of 8. For resnet50, Do you achieve similar result as author reported (0.41LB on gray)? I only got 0.445 LB but on both gray and color image. I guess if I use gray image only, this score will lower than 0.41. How about you?
", "votes": 0, "replies": [] }, { "id": 296661, "author_name": "xuan", "author_url": "", "post_date": "2018-03-15T15:25:48.420000", "content": "I also got ~0.445 on both gray and color image using the ResNeXt model, but I only ran for about 5k iters, seems the loss (especially rcnn cls loss) cannot get down for more iters. Is that right? I saw you've ran it for 25k iters.\nBtw, the model has poor performance on color image & slim nuclei, as well as some dark images. May be we neet to find other ways to handle the color ones using pre/post process
", "votes": 0, "replies": [] }, { "id": 296666, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-15T15:35:57.050000", "content": "To run resneXt, Have you change the code? Because the code only has two options: Resnet-50 and SE-resneXt. The score 0.445LB which I achieved that used Resnet-50 and trained with Adam optimization on 20k iterations. I am run SE-ResnetXt now, I did not complete yet.
\n\nYou are right about color image. The author suggested that we can convert it to gray and purple image and trained it with two models: 1 for gray and 1 for purple. Then concatenate the result at the end. For post-processing, he did it with removing small cell and dilation. I think it is fine
", "votes": 0, "replies": [] }, { "id": 296675, "author_name": "xuan", "author_url": "", "post_date": "2018-03-15T15:51:53.207000", "content": "I modified the code with similar process as Yan Wang posted up there.\nI had trained separate model using gray/color images, but the result is no better than current version. Model trained on gray images performs well, but color one is too poor and I'm afraid of overfitting with such few train/valid images.
\n\nAnyway, thanks for feedback! There's still time and I have some trials to struggle :D
", "votes": 0, "replies": [] }, { "id": 296678, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-15T15:57:47.017000", "content": "You are welcome. Actually, I got 0.47 LB on Keras version (https://github.com/matterport/Mask_RCNN). but it takes too too much time for training (1 epoch ~6 min, while Heng code ~2 min) . So I moved to pytorch code. The different one is Keras based uses pretrained model (resnet-50) and I was used it to achieved the score. The strategy for training: train on head and then train all. Because we want to keep the good features from pretrained. However, I did not find any way to use pretrain in this implementation. If you find it, please let me know.
", "votes": 0, "replies": [] }, { "id": 296685, "author_name": "xuan", "author_url": "", "post_date": "2018-03-15T16:12:33.320000", "content": "Since the model has been made a little complicated, I'm not sure if a pretrained model based on other code (i.e. models from the official torchvision repo) could be fitted into this model. Will let u know if I have some work-around. And thanks for sharing training strategy!
", "votes": 0, "replies": [] }, { "id": 296759, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-15T18:11:30.440000", "content": "please compare your loss with the log files in the google drive. A good loss at the end should be:
\n\nrate = 0.001 (using SGD)
\n\ntotal = 0.393 (variable, becuase of rcnn,mask sampling of samples)
\n\nrpn = 0.01 0.01 (you should this number)
\n\nrcnn = 0.25 0.01 (rcnn reg loss should be 0.01, rcnn cls loss should be 0.0.25)
\n\nmask = 0.11 (mask loss is about 0.12 to 0.11)
\n\nWith this , you should get LB = 0.42 on gray images only for resnet50-fpn\nthere are some easy mask post processing you can do. try to draw the mask one by one, one at a time. you will see some single pixel noise, try to clean them out.
\n\ntry to train longer iterations. the mask prediction needs long iteration to get to pixel accuracy (i train 29K iterations (that is about 300 epoches!) which takes 24hr). Don't use epoch, use iterations to count.
\n\nthere are 12 color images.
\n\nfor the 8 (purple on yellow), convert to gray. these 8 images give LB score of bout 0.03.
\n\nthe remaining 4 images are H&E stain. train this with external labelled images (you can find this at the kernel thread). The LB score should be about 0.010 to 0.015.
\n\nremove scale augmentation at the last training iterations with small learning rate of 0.001
", "votes": 2, "replies": [] }, { "id": 296788, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-15T18:59:15.987000", "content": "Thanks Heng. It is a very nice log. Could you tell me why did not you use pretrain model? I used keras with pretrain and it provides better than without pretrain. Using pretrain model we can have a good strategy for training likes paper mentioned. Any problem if we implement model.py with pretrain?
\n\nFYI, I completed training using 500 gray and evaluate in 53 testing gray, validate in 43 grays. It achieved 0.429 after 25k iteration with learning schedule as LR = StepLR([ (0, 0.01), (10000, 0.001), (20000, 0.0001), (25000, 0.00001)])\nThe loss is
\n 0.0001 25.0 k 400.0 0.2 m | 0.366 0.01 0.01 0.22 0.01 0.11 | 0.498 0.02 0.02 0.29 0.01 0.16 | 0.547 0.04 0.03 0.31 0.01 0.15 | 12 hr 07 min\n
Update: After training on other color image (12 images) without modifying the code, Then merge result with gray (have trained on gray image), then my current score is 0.47LB after 3.5K. I think the result will improve if I train on three networks, instead of two as I did. Three network means: 1 for gray, 1 for gray of color and 1 for purple convered from color image.\nThis is log of training on color
\n\n\n0.0001 3.5 k 247.8 0.0 m | 0.792 0.04 0.04 0.48 0.01 0.22 | 0.707 0.03 0.04 0.40 0.02 0.21 | 0.677 0.01 0.04 0.40 0.03 0.20 | 1 hr 51 min\n
@john1231983 ... 0.47LB after 3.5K
\n\nYes this should be around the baseline of the code. The rest are just augmentation (e.g. using merged mosiac images), sample balancing, learning rate, etc ... should be about to go up a little more to 0.48 to 0.49 i think.
\n\nThe rest are heavy mask post processing to break the 0.500 mark.
", "votes": 3, "replies": [] }, { "id": 297286, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-03-16T17:06:08.023000", "content": "the 20180306 code got me 0.427 after training in the gray500 - nice!
\n\nI downloaded the 20180314 model and made the changes you suggested but am getting an error...
\n\n File \"/home/james/Dropbox/LINUX/remote/HKC_c/net/resnet50_mask_rcnn/modelSE.py\", line 297, in forward\n crop = self.crops[l](fs[l], proposals[index].view(-1,7))\n File \"/home/james/anaconda3/envs/HKC/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 365, in __call__\n result = self.forward(*input, **kwargs)\n File \"/home/james/Dropbox/LINUX/remote/HKC_c/net/lib/roi_align_pool_tf/module.py\", line 40, in forward\n ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1)\nValueError: too many values to unpack (expected 5)\nAny idea what's causing that?
\n\nit's happening just after the line...
\n\ncrop = self.crops[l](fs[l], proposals[index].view(-1,7))\n@Jame: You should you the lines of the old version to fix it.
\n\n@Heng: We have a total of 664 images, then we separate them into 2 classes: gray and color. The gray will use as you did (543 images). The remaining 121 color images will be converted to gray and purple and train on two separate networks. But you said
\n\n\n\n\nor the 8 (purple on yellow), convert to gray. these 8 images give LB score of bout 0.03.\n the remaining 4 images are H&E stain. train this with external labelled images (you can find this at the kernel thread). The LB score should be about 0.010 to 0.015.
\n
It looks that we have to separate the color training set into two classes: purple on yellow and H&E, while H&E using augmentation data, and purple using above scheme (convert gray+purple). That means we have three networks for color images. Please correct me if I was wrong? Thanks
", "votes": 0, "replies": [] }, { "id": 297370, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-03-16T23:15:22.923000", "content": "@James: You should use the lines of the old version to fix it.
\n\nThanks, @john
\n\nThe good news is that it's training. My entire CropRoi class is now the same as the in original model. \nSpecifically...
\n\ncrop = self.crops[l](fs[l], proposals[index].view(-1,7))\nis now back to
\n\ncrop = self.crops[l](fs[l], rois[index].view(-1,5))\nand I (re)inserted
\n\nrois = proposals.detach().data[:,0:5]\nrois = Variable(rois)\nAND I had to reduce my batch size from 16 to 8 because I was getting oom errors.
\n\nAnyone else had to reduce the batch size? I have two GPUs a 1080Ti and a 1080. So I guess they are running a batch size of 4 on each.
\n\nI am still curious to know how @Yan got this line working:
\n\ncrop = self.crops[l](fs[l], proposals[index].view(-1,7))\nI'll post my new score when I get it.
", "votes": 0, "replies": [] }, { "id": 297380, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-17T00:15:43.613000", "content": "@Jame You are welcome. For my score, I got 0.473 now without modify the code. 0.03 got from extra H&E dataset. I group all color images in same group and train with single network. I did not seperate them to two network as Henh mentioned. Do you did same thing?
", "votes": 0, "replies": [] }, { "id": 297420, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-17T03:09:53.337000", "content": "@James: Yes, this line will cause some error, we can fix the error by adding two extra return values for this line: ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1) in module.py
@Yan what is your score now? I got 0.429 on 500 gray and 0.47 on all gray and color.
", "votes": 0, "replies": [] }, { "id": 297442, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2018-03-17T03:53:41.790000", "content": "Hi john1231983! \nCould you tell me how many iterations it took to get to 0.429 and 0.47 \nI only have access to K80 GPU on google cloud compute so training takes a long time for me.
\n\nthanks\nMehul
", "votes": 0, "replies": [] }, { "id": 297597, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-17T13:53:45.510000", "content": "@John, We are using a K80 for this code too, not finished yet. I guess your score is good enough for this code. We are trying to run the code on our own server now, which has more GPUs.
", "votes": 0, "replies": [] }, { "id": 297601, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-17T14:18:20.453000", "content": "Hi mehul. I have posted it in this thread. Please read it in above\n@yan I see, FYI. I got 0.473 when using HE external dataset. I also finding the way to use pretrained model ( convert from caffe). I guess the performance increases if we use it. But I did not try
", "votes": 2, "replies": [] }, { "id": 297847, "author_name": "jamesluo1991", "author_url": "", "post_date": "2018-03-18T05:25:17.630000", "content": "@Heng CherKeng\n Hello Heng, how do you calculate accuracy?
", "votes": 0, "replies": [] }, { "id": 297861, "author_name": "HAOYU-LI", "author_url": "", "post_date": "2018-03-18T06:38:54.270000", "content": "Hi @John. What's the extra H&E dataset you used? Possible to give a link here? Thanks a lot!
", "votes": 0, "replies": [] }, { "id": 297869, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-18T07:17:23.293000", "content": "This one https://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations Let me know if it improve your perfomance. I got 0.002 improve
", "votes": 0, "replies": [] }, { "id": 297972, "author_name": "HAOYU-LI", "author_url": "", "post_date": "2018-03-18T12:53:59.693000", "content": "Sure. I will let you know when I get the result. Thanks!
", "votes": 0, "replies": [] }, { "id": 298322, "author_name": "FullySelfDrivingTang", "author_url": "", "post_date": "2018-03-19T11:02:22.923000", "content": "Hi John, As you mentioned, After training on other color image (12 images). Are these 12 images are in training data set? I am thinking these 12 images in test data set, correct? If so , which training set and valid set do you use to create the model? I am trying Heng's code to create model to predict 12 purple images. I am training on 108(random split into 100 in train set, 8 in valid set) colorful images , the coding running with 2 1080ti GPU for 9 hours. but the loss still high. Any idea?\n0.0100 7.9 k 1258.4 0.1 m | 0.816 0.04 0.03 0.55 0.01 0.18 | 0.691 0.01 0.03 0.44 0.02 0.20 | 0.647 0.04 0.03 0.37 0.02 0.18 | 9 hr 12 min
", "votes": 0, "replies": [] }, { "id": 298340, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-19T11:54:50.260000", "content": "Sorry. It was wrong. It means train with color image ime 113 image and 30 augment image ( link above) . You can get some loss as me ( as above) . After that, i test with 12 image and got 0.473. How about your current score now? For your problem , let reduce learning rate to 10 times after each 3k iters
", "votes": 0, "replies": [] }, { "id": 298365, "author_name": "FullySelfDrivingTang", "author_url": "", "post_date": "2018-03-19T13:02:28.473000", "content": "thanks john, I random picked up 150 train images (gray and purple) and predict on the 8 purple test images, then merged with the gray result(0.417) . my score is 0.439 at that time. I am trying to train a colorful images model then predict on the 8 purple to see the result. will test the learning rate 0.001.
", "votes": 0, "replies": [] }, { "id": 298375, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-19T13:24:27.833000", "content": "Maybe something wrong. I have trained on 500 gray images and testing on 53 gray images, it must be around 0.429. Then the 113 color image will combined with 30 extra image to make training color set. Then I train it and test on 12 color image. Combined RLE with 53 testing above image, I got 0.473 LB. What is your batch size? I am using 7 because my GPU resource.
", "votes": 0, "replies": [] }, { "id": 298437, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-03-19T15:13:38.247000", "content": "thanks @Yan. The extra two params nailed it.
\n\nBefore I fixed that, I got the following results on the SE-ResnetXt.
\n\nTraining on gray500 only and submitting on gray53. Straight out the box, in other words...
\n\n12000 iterations: 0.423
\n\n14500 iterations: 0.424
\n\n15000 iterations: 0.436*
\n\n16000 iterations: 0.419
\n\n18000 iterations: 0.414
\n\nWith the regular Res50 model I got 0.427 using 16500 iterations (didn't try and other iterations)
\n\nSeems odd to me that 15000 should spike up like that. What's going on?
", "votes": 0, "replies": [] }, { "id": 298441, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-19T15:18:55.470000", "content": "Impressive with 0.436 LB. Could you tell me your learning schedule? How do you set the learning rate schedule? I am using 0.01 from 0 to 10k, 0.001 from 10k to 15k and 0.0001 from 15k to 30k
", "votes": 0, "replies": [] }, { "id": 298445, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-03-19T15:27:03.187000", "content": "learning rate was 0.01 all the way (out of the box).\nI am running a version now with [(0, 0.01), (10000, 0.001), (20000, 0.0001), (25000, 0.00001)]
", "votes": 0, "replies": [] }, { "id": 298454, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-19T15:37:28.743000", "content": "@Jam: What is return should I change ids, x0, y0, x1, y1= torch.split(rois, 1, dim=1) in module.py? Does it like ids, x0, y0, x1, y1,_,_= torch.split(rois, 1, dim=1) I have corrected based on the previous version, but for safety, I would like to change as small as possible comparsion with current model.py
yes...
\n\nids, x0, y0, x1, y1,_,_ = torch.split(rois, 1, dim=1)
", "votes": 0, "replies": [] }, { "id": 302469, "author_name": "yyll008", "author_url": "", "post_date": "2018-03-24T05:09:00.123000", "content": "Thanks for sharing.Can you tell me how to merge the submissions？
", "votes": 0, "replies": [] }, { "id": 305010, "author_name": "yyll008", "author_url": "", "post_date": "2018-03-28T10:59:54.333000", "content": "Thanks !\n@Yan Wang
\n\nWhat is self.mask_proposals? How do you solve this :
\n\nself.masks, self.mask_proposals = mask_nms(cfg, mode, inputs, self.rcnn_proposals, self.mask_logits)\nself.detections = self.mask_proposals
", "votes": 0, "replies": [] }, { "id": 307410, "author_name": "YL", "author_url": "", "post_date": "2018-04-01T14:52:24.213000", "content": "Hi John. May I ask what configuration are you using with matterport's keras+tensorflow implementation? I using set the image shape to 512 x 512 and it takes me about an hour to train an epoch on a Titan X pascal. ......
", "votes": 0, "replies": [] }, { "id": 307411, "author_name": "YL", "author_url": "", "post_date": "2018-04-01T14:57:11.620000", "content": "John, also may i ask what score are you getting for rpn class loss and rpn bbox loss ... for me these two scores remain unchanged at around 0.7 and 2.1 respectively, which seems a bit odd to me...
", "votes": 0, "replies": [] }, { "id": 310691, "author_name": "Ashish Lal", "author_url": "", "post_date": "2018-04-08T10:08:33.970000", "content": "John, I am getting a similar spike around 15K. Were you able to figure out the cause?
", "votes": 0, "replies": [] } ] }, { "id": 295550, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-03-13T20:51:54.267000", "content": "Hey @Heng CherKeng, a first time Kaggler here, just found this discussion and I have a quick question: it seems you only submit 53 gray images. If we also submit the rest of (color) images and don't do well on those, will we get 0 or get penalized by doing bad on those images? I didn't know we could submit just a subset of images.
", "votes": 2, "replies": [ { "id": 295634, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-14T00:16:28.270000", "content": "I believe we will get 0 for bad predictions, you may try it, since we have 5 submissions every day.
", "votes": 1, "replies": [] }, { "id": 295699, "author_name": "hengck23", "author_url": "", "post_date": "2018-03-14T03:06:48.877000", "content": "The min score for an image is zero. It will not be negative. The purpose of submitting separately is for me to find out the weakness of my algo and the difficulty of the test images.
", "votes": 3, "replies": [] }, { "id": 295935, "author_name": "john1231983", "author_url": "", "post_date": "2018-03-14T12:46:23.053000", "content": "FYI, I have ran the code with gray only and test with gray only. I achieve 0.36LB after 35k iterations ~2hours.
", "votes": 0, "replies": [] } ] }, { "id": 293471, "author_name": "Chen Zhang", "author_url": "", "post_date": "2018-03-09T22:54:07.453000", "content": "Hi Heng, \nThanks again for sharing. I meet some problems when I'm trying to modify some parameters of the network in net/resnet../model.py. My case is, running model.py check is fine, then train the network is fine, but when I use the new .ptb to run submit.py, not a single image can pass. The errors are in\n\nrevert(net, images) #unpad, undo test-time augment etc ....\n\n, then in this \"revert\" function \n\nnet.detections[:,0]\n\ndimension error. Actually, the net.detections is of size 0 for every image.
How could I make modifications to the structure of the network? Am I miss something?
", "votes": 2, "replies": [ { "id": 294012, "author_name": "danieleewww", "author_url": "", "post_date": "2018-03-11T03:43:20.410000", "content": "Hit the same error as yours when execute submit.py:
\n\n\n\n\nindex = (net.detections[:,0]==b).nonzero().view(-1)\n IndexError: too many indices for tensor of dimension 1
\n
Link here may help https://github.com/numpy/numpy/issues/10447. If true, still i did not figure out how to modify .nonzero(). to make it work.
", "votes": 0, "replies": [] }, { "id": 294128, "author_name": "Chen Zhang", "author_url": "", "post_date": "2018-03-11T10:50:07.420000", "content": "If you print net.detections, you can see it is a variable of size 0, for all images. I think some other part code should also be modified together. Haven't figure it out yet.
", "votes": 0, "replies": [] }, { "id": 312855, "author_name": "AngMingliang", "author_url": "", "post_date": "2018-04-12T13:20:36.263000", "content": "Hi, i got this error when using submit.py for stage2 dataset. Did anybody managed to find out what causes this error ?
", "votes": 0, "replies": [] }, { "id": 312858, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-12T13:24:38.543000", "content": "you can modify the code.
\n\n net.set_mode('test')\n with torch.no_grad():\n input = torch.from_numpy(augment_image.transpose((2,0,1))).float().div(255).unsqueeze(0)\n input = Variable(input).cuda()\n net.forward(input)\n\n if (len(net.detections)==0):\n ... save a zero array or just ignore .....\n continue\n else:\n\n rcnn_proposal, detection, mask, instance = undo_test_augment(net, image, **params)\nThanks @Heng CherKeng for your help, however because i am using your 03062018 version of your code i don't have the undo_test_augment function. Instead I found out that by modifying the submit.py code at line 221 like so
\n\n with torch.no_grad():\n inputs = Variable(inputs).cuda()\n net(inputs )\n if (len(net.detections)==0):\n revert(net, images)\nthe error resolved
", "votes": 0, "replies": [] } ] }, { "id": 289000, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2018-02-28T18:03:21.750000", "content": "I'm trying to run the mask-rcnn on a tesla k80 but I'm running into this error:
\n\nRuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1518244507981/work/torch/lib/THC/generic/THCTensorMath.cu:15\nIs the problem just my hardware? I can run the Unet code with the same setup.
\n\nEDIT:\nIf you're using cloud you need to use a p100/v100 gpu to run the mask-rcnns.
", "votes": 2, "replies": [ { "id": 290423, "author_name": "Blueocean", "author_url": "", "post_date": "2018-03-04T02:33:26.980000", "content": "Yes I have the exact same issue with you. Have you fixed this problem?
", "votes": 0, "replies": [] }, { "id": 290471, "author_name": "Chen Zhang", "author_url": "", "post_date": "2018-03-04T07:05:19.040000", "content": "Hi Steven, I think change the setting of arch sm_52 when build .so might make a difference. This is what I found from a readme file of a pytorch mask rcnn on github\nWhich CUDA capabilities do we want to pre-build for?\nhttps://developer.nvidia.com/cuda-gpus\nCompute/shader model Cards
\n\n```\n# Which CUDA capabilities do we want to pre-build for?\n# https://developer.nvidia.com/cuda-gpus\n# Compute/shader model Cards\n# 6.1 P4, P40, Titan Xp, GTX 1080 Ti, GTX 1080\n# 6.0 P100\n# 5.2 M40, Titan X, GTX 980\n# 3.7 K80\n# 3.5 K40, K20\n# 3.0 K10, Grid K520 (AWS G2)\n```\nP100 is way too expensive on cloud
", "votes": 0, "replies": [] }, { "id": 290639, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2018-03-04T15:54:38.327000", "content": "Hi Chen, I'm using preemptible vms on gcp for my p100s, which end up being about $1/hr. With checkpoints there really much of an issue.
", "votes": 0, "replies": [] }, { "id": 292344, "author_name": "MPWARE", "author_url": "", "post_date": "2018-03-07T20:56:10.860000", "content": "Use -arch=sm_30 for Tesla K80. Pay attention sm_52 is hardcoded in gpu_nms setup.py, so update it manually too. All arch mapping here:\nhttp://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
", "votes": 3, "replies": [] } ] }, { "id": 286978, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-23T05:57:14.347000", "content": "the correct making of the anchor boxes in rpn_mns.py should be corrected as below:
\n\ndef make_bases(base_size, base_apsect_ratios):\nbases = []\nfor ratio in base_apsect_ratios:\n w = base_size/math.sqrt(ratio)\n h = w*ratio\n rw = round(w/2)\n rh = round(h/2)\n base =(-rw, -rh, rw, rh, )\n bases.append(base)\n\nbases = np.array(bases,np.float32)\nreturn bases\n