![\"dip](\"http://file///C:/Users/mehul.sampat/Desktop/validation_loss_dip.png\")
Hi all,\nThere are several implementations of mask rcnn available now, but they are not the same. Which one you use, and what score it gives you?\nWe use the matterport implementation and get LB 0.436.
", "messages": [ { "id": "288225", "postDate": "02/27/2018 01:18:50", "content": "Hi all,\nThere are several implementations of mask rcnn available now, but they are not the same. Which one you use, and what score it gives you?\nWe use the matterport implementation and get LB 0.436.
", "rawMarkdown": "Hi all,\nThere are several implementations of mask rcnn available now, but they are not the same. Which one you use, and what score it gives you?\nWe use the matterport implementation and get LB 0.436.", "votes": null }, { "id": "288261", "postDate": "02/27/2018 03:52:07", "content": "Hi Yang, Nice to see your result. I am also using matterport implementation (because the other implementation has the own license-that I think we can not use in this challenge). When you submit the result, do you combine all masks into one mask then encode RLE or encode individually. If you encode individually, could you tell me how can encode each mask individually?
", "rawMarkdown": "Hi Yang, Nice to see your result. I am also using matterport implementation (because the other implementation has the own license-that I think we can not use in this challenge). When you submit the result, do you combine all masks into one mask then encode RLE or encode individually. If you encode individually, could you tell me how can encode each mask individually?", "votes": null }, { "id": "288268", "postDate": "02/27/2018 04:21:48", "content": "Hi John, masks are encoded individually. \nMy full code is available here enter link description here
", "rawMarkdown": "Hi John, masks are encoded individually. \nMy full code is available here [enter link description here][1]\n\n [1]: https://github.com/YanWang2014/Mask_RCNN", "votes": null }, { "id": "288270", "postDate": "02/27/2018 04:31:44", "content": "Thanks. Do you separate the training set into 3 clusters and train each cluster individually? I used 3 clusters and this is loss for 1 cluster at Epoch 152 and crop size 128x128
\n394s 394ms/step - loss: 0.7758 - rpn_class_loss: 0.0184 - rpn_bbox_loss: 0.2962 - mrcnn_class_loss: 0.0454 - mrcnn_bbox_loss: 0.1637 - mrcnn_mask_loss: 0.2522 - val_loss: 1.0752 - val_rpn_class_loss: 0.0406 - val_rpn_bbox_loss: 0.3560 - val_mrcnn_class_loss: 0.0823 - val_mrcnn_bbox_loss: 0.2309 - val_mrcnn_mask_loss: 0.3653\n\nI trained with resnet-50 pretrain model and train all. I will report my performance when it completed. For now, I have to think the way to peform inference with arbitrary size (instead of zero padding). One person have convert fully connected to fully convolution and the LB increased to 0.46. Hope it can help you some idea to improve performance
Hi All,\nI'm not expert at all of computer vision and this is the first approach. I'm using this implementation of mask rcnn\nhttps://github.com/multimodallearning/pytorch-mask-rcnn \nand at the moment I'm stuck at 0.360. I used a smaller mini mask (16,16) or (32,32). I will try with the standard values (56,56)
", "rawMarkdown": "Hi All,\nI'm not expert at all of computer vision and this is the first approach. I'm using this implementation of mask rcnn\nhttps://github.com/multimodallearning/pytorch-mask-rcnn \nand at the moment I'm stuck at 0.360. I used a smaller mini mask (16,16) or (32,32). I will try with the standard values (56,56)", "votes": null }, { "id": "288360", "postDate": "02/27/2018 10:04:51", "content": "Hey Yan!\nI was trying to get the original matterport code to work, but failed.
\n\nData loading is fine, but when I try to train the model, it doesn't seem to do anything.\n(the corresponding notebook cell has the asteriks and it only prints the first epoch and stays like that for hours)
\n\nHave you come across a situation like this?
", "rawMarkdown": "Hey Yan!\nI was trying to get the original matterport code to work, but failed.\n\nData loading is fine, but when I try to train the model, it doesn't seem to do anything.\n(the corresponding notebook cell has the asteriks and it only prints the first epoch and stays like that for hours)\n\nHave you come across a situation like this?", "votes": null }, { "id": "288387", "postDate": "02/27/2018 11:18:59", "content": "How much using 3 clusters improved compared to not using them? How do you use models trained on clusters for test, given the modalities are not the same as in train?
", "rawMarkdown": "How much using 3 clusters improved compared to not using them? How do you use models trained on clusters for test, given the modalities are not the same as in train?", "votes": null }, { "id": "288467", "postDate": "02/27/2018 15:06:18", "content": "Hi Yang, you've updated SGD optimizer to Adam optimizer. Does it really improve LB score?
", "rawMarkdown": "Hi Yang, you've updated SGD optimizer to Adam optimizer. Does it really improve LB score?", "votes": null }, { "id": "288483", "postDate": "02/27/2018 16:01:21", "content": "It's faster
", "rawMarkdown": "It's faster", "votes": null }, { "id": "288484", "postDate": "02/27/2018 16:01:53", "content": "I did not meet this problem
", "rawMarkdown": "I did not meet this problem", "votes": null }, { "id": "288521", "postDate": "02/27/2018 17:39:00", "content": "\n\n\n", "rawMarkdown": "https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/\n\n> “We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.”", "votes": null }, { "id": "288538", "postDate": "02/27/2018 18:07:30", "content": "“We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.”
\n
Good point.
", "rawMarkdown": "Good point.", "votes": null }, { "id": "288617", "postDate": "02/27/2018 22:01:04", "content": "Thanks! For inference, you used same IMAGE_MIN_DIM/IMAGE_MAX_DIM as in training or 1024? I've tried different and LB is not better.
", "rawMarkdown": "Thanks! For inference, you used same IMAGE_MIN_DIM/IMAGE_MAX_DIM as in training or 1024? I've tried different and LB is not better.", "votes": null }, { "id": "288656", "postDate": "02/28/2018 00:35:15", "content": "Hi Yan, how many epoch of training do you get to LB 4.3? What do you think the most important factor is to get the score?
", "rawMarkdown": "Hi Yan, how many epoch of training do you get to LB 4.3? What do you think the most important factor is to get the score?", "votes": null }, { "id": "288695", "postDate": "02/28/2018 02:40:31", "content": "The score is obtained by one of my team members, with some modifications to the nn. As far as I know, the baseline (without any additional work, just tune the parameters) for matterport mask rcnn should be around 0.411.
", "rawMarkdown": "The score is obtained by one of my team members, with some modifications to the nn. As far as I know, the baseline (without any additional work, just tune the parameters) for matterport mask rcnn should be around 0.411.", "votes": null }, { "id": "288717", "postDate": "02/28/2018 04:11:51", "content": "Same here. I ran this submission with python3 val_test.py but I got a low result of LB. It is about 0.33. Anyone has same score when ran this code? This is my final loss\n\n 300/300 [==============================] - 282s 940ms/step - loss: 0.2732 - rpn_class_loss: 0.0034 - rpn_bbox_loss: 0.0332 - mrcnn_class_loss: 0.0284 - mrcnn_bbox_loss: 0.0374 - mrcnn_mask_loss: 0.1708 - val_loss: 0.7583 - val_rpn_class_loss: 0.0035 - val_rpn_bbox_loss: 0.2083 - val_mrcnn_class_loss: 0.0190 - val_mrcnn_bbox_loss: 0.2052 - val_mrcnn_mask_loss: 0.3224\n
I had the problem when using my own code. \nI have tested your code and the problem is gone. Thanks!
", "rawMarkdown": "I had the problem when using my own code. \nI have tested your code and the problem is gone. Thanks!", "votes": null }, { "id": "288941", "postDate": "02/28/2018 15:41:33", "content": "Hi!
\n\nIn val_test2.py, I came across this line:
\n\ndataset.add_nuclei('../../Spot-Nuclei-master/data/stage1_test/','test')\nIs that referring to the normal test data or something else?
\n\nCheers!
", "rawMarkdown": "Hi!\n\nIn val_test2.py, I came across this line:\n\n dataset.add_nuclei('../../Spot-Nuclei-master/data/stage1_test/','test')\n\nIs that referring to the normal test data or something else?\n\nCheers!", "votes": null }, { "id": "288944", "postDate": "02/28/2018 15:54:48", "content": "Yes, the normal test data
", "rawMarkdown": "Yes, the normal test data", "votes": null }, { "id": "288963", "postDate": "02/28/2018 16:42:28", "content": "Hi Yan, your code has some file with the same name, for example, val_test1, val_test2... I know that train_val.py is for training, but which function is for making RLE to obtain 0.43 LB?
", "rawMarkdown": "Hi Yan, your code has some file with the same name, for example, val_test1, val_test2... I know that train_val.py is for training, but which function is for making RLE to obtain 0.43 LB?", "votes": null }, { "id": "289021", "postDate": "02/28/2018 19:00:21", "content": "This is a basic code, which gives around 0.411. The val_test version does not affect too much.
", "rawMarkdown": "This is a basic code, which gives around 0.411. The val_test version does not affect too much.", "votes": null }, { "id": "289051", "postDate": "02/28/2018 21:23:16", "content": "Thanks for the link!\nThe only other PyTorch Mask RCNN implementation I am aware of is incomplete and inference only.
", "rawMarkdown": "Thanks for the link!\nThe only other PyTorch Mask RCNN implementation I am aware of is incomplete and inference only.", "votes": null }, { "id": "289090", "postDate": "02/28/2018 22:16:29", "content": "Hi Geoff, your score is impressive. Which code base you are using?
", "rawMarkdown": "Hi Geoff, your score is impressive. Which code base you are using?", "votes": null }, { "id": "289289", "postDate": "03/01/2018 09:59:57", "content": "Hi Geoff, this pytorch version works very well. It is a translation of keras matterport. Anyhow I did not reach the results people wrote in the forum, and I'm stuck at 0.360. I've tried also image augmentation, but with mini_mask and rotations it fails. Did you get your score with any version of mask rcnn?
", "rawMarkdown": "Hi Geoff, this pytorch version works very well. It is a translation of keras matterport. Anyhow I did not reach the results people wrote in the forum, and I'm stuck at 0.360. I've tried also image augmentation, but with mini_mask and rotations it fails. Did you get your score with any version of mask rcnn?", "votes": null }, { "id": "289344", "postDate": "03/01/2018 12:51:48", "content": "I'm using my own algorithm, not Mask R-CNN.
\n\nI may well switch over to Mask R-CNN as I have used pretty much all the tricks I know to achieve the score I have and I don't really know how to improve matters!
\n\nThanks for pointing this implementation out as I think I will use it as a start.
", "rawMarkdown": "I'm using my own algorithm, not Mask R-CNN.\n\nI may well switch over to Mask R-CNN as I have used pretty much all the tricks I know to achieve the score I have and I don't really know how to improve matters!\n\nThanks for pointing this implementation out as I think I will use it as a start.", "votes": null }, { "id": "289346", "postDate": "03/01/2018 13:00:36", "content": "Your great score is encouraging for those of us who try their own stuff rather than parameterize someone else implementation!
", "rawMarkdown": "Your great score is encouraging for those of us who try their own stuff rather than parameterize someone else implementation!", "votes": null }, { "id": "289912", "postDate": "03/02/2018 15:37:07", "content": "", "rawMarkdown": "", "votes": null }, { "id": "289989", "postDate": "03/02/2018 18:57:48", "content": "Hi Yan. Quick question. Clipping norm on the optimizers seems to improve performance. How did you guys decide on the clipnorm value of 5.0?
", "rawMarkdown": "Hi Yan. Quick question. Clipping norm on the optimizers seems to improve performance. How did you guys decide on the clipnorm value of 5.0?", "votes": null }, { "id": "290021", "postDate": "03/02/2018 19:54:00", "content": "It's used in the original matterport code
", "rawMarkdown": "It's used in the original matterport code", "votes": null }, { "id": "290122", "postDate": "03/03/2018 02:08:59", "content": "I had the same problem when I used the original matterport code. When I tried to run Yan's code, I am getting OOM error. Did you run into anything like that?
", "rawMarkdown": "I had the same problem when I used the original matterport code. When I tried to run Yan's code, I am getting OOM error. Did you run into anything like that?", "votes": null }, { "id": "290124", "postDate": "03/03/2018 02:12:35", "content": "@ananthan31 OOM is (out of memory). Your GPU memory may not fit the code as is.
", "rawMarkdown": "ananthan31 OOM is (out of memory). Your GPU memory may not fit the code as is.", "votes": null }, { "id": "290132", "postDate": "03/03/2018 02:35:22", "content": "I figured out the error. The config file has GPU count as 4. When I changed it to 1 (since I have a single GPU), it worked. Thanks for your help!
", "rawMarkdown": "I figured out the error. The config file has GPU count as 4. When I changed it to 1 (since I have a single GPU), it worked. Thanks for your help!", "votes": null }, { "id": "290133", "postDate": "03/03/2018 02:37:26", "content": "@anantham31 Nice. Good luck and have fun.
", "rawMarkdown": "anantham31 Nice. Good luck and have fun.", "votes": null }, { "id": "290171", "postDate": "03/03/2018 06:38:49", "content": "Hi Yan, Thanks for sharing your code. I tried it but I ran into Out of Memory with my GPU:
\n\n\n\n\nResourceExhaustedError (see above for traceback): OOM when allocating\n tensor with shape[1024,256,14,14] and type float on\n /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
\n
I have a 6GB GPU. I tried to reduce memory use by modifying the config with:
\n\nGPU_COUNT = 1\nIMAGES_PER_GPU = 2\n...\nIMAGE_MIN_DIM = 256\nIMAGE_MAX_DIM = 256\nBut it still ran out of GPU memory. Do you have any suggestion?
", "rawMarkdown": "Hi Yan, Thanks for sharing your code. I tried it but I ran into Out of Memory with my GPU:\n\n> ResourceExhaustedError (see above for traceback): OOM when allocating\n> tensor with shape[1024,256,14,14] and type float on\n> /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n\nI have a 6GB GPU. I tried to reduce memory use by modifying the config with:\n\n GPU_COUNT = 1\n IMAGES_PER_GPU = 2\n ...\n IMAGE_MIN_DIM = 256\n IMAGE_MAX_DIM = 256\n\nBut it still ran out of GPU memory. Do you have any suggestion?", "votes": null }, { "id": "290172", "postDate": "03/03/2018 06:40:17", "content": "Reduce the IMAGE_MIN_DIM to 128, USE_MINI_MASK=True, IMAGES_PER_GPU=1. Hope it help
Hi Yan,
\n\nThanks for sharing your code. I see there are lot of visualize*.py files. I am running on a remote server. Which of these files can I run to visualize the results? Will this save the data to some folder since I am running on a headless server?
", "rawMarkdown": "Hi Yan,\n\nThanks for sharing your code. I see there are lot of visualize*.py files. I am running on a remote server. Which of these files can I run to visualize the results? Will this save the data to some folder since I am running on a headless server?", "votes": null }, { "id": "290360", "postDate": "03/03/2018 21:41:58", "content": "Results will be saved in plot_dir. The difference of all the visualize*.py files is subtle (either using different datasets like val or test, or using different configurations or postprocessing). Their codes are very similar, you'd better check them yourself.
Hi everyone,\nThanks for sharing :) I have a question. Is it possible to train on multiple CPU ? I tried something like this :
\n\nCPU_COUNT=8\nIMAGES_PER_CPU = 1\nSTEPS_PER_EPOCH = 600//(CPU_COUNT*IMAGES_PER_CPU)\nBut the result is worse than using 1CPU and 100 steps per epoch, for the same number of epochs.\nDoes it mean that not all images are read ? Should I keep 600 steps per epoch, or train longer ? Or should I adapt something in the code ? I would like to understand how it works in fact.
\n\nThank you very much for your help.
", "rawMarkdown": "Hi everyone,\nThanks for sharing :) I have a question. Is it possible to train on multiple CPU ? I tried something like this :\n\n CPU_COUNT=8\n IMAGES_PER_CPU = 1\n STEPS_PER_EPOCH = 600//(CPU_COUNT*IMAGES_PER_CPU)\n\nBut the result is worse than using 1CPU and 100 steps per epoch, for the same number of epochs.\nDoes it mean that not all images are read ? Should I keep 600 steps per epoch, or train longer ? Or should I adapt something in the code ? I would like to understand how it works in fact.\n\nThank you very much for your help.", "votes": null }, { "id": "290831", "postDate": "03/05/2018 02:33:05", "content": "Thanks for sharing your code, which help me a lot！
", "rawMarkdown": "Thanks for sharing your code, which help me a lot！", "votes": null }, { "id": "290977", "postDate": "03/05/2018 10:19:19", "content": "Ditto @CPMP and @Geoff French. I am still stretching my own algorithm and hoping to get round to Mask RCNN at some point. Both of your results are encouraging. Thanks.
", "rawMarkdown": "Ditto @CPMP and @Geoff French. I am still stretching my own algorithm and hoping to get round to Mask RCNN at some point. Both of your results are encouraging. Thanks.", "votes": null }, { "id": "293128", "postDate": "03/09/2018 08:07:29", "content": "The biggest memory consumption comes from saving all those weights at each epoch. What I do is manually delete saved weights from each prior epochs using os.remove().
\n\nFor example: os.remove('checkpoints/nuclei20180308T0559/mask_rcnn_nuclei_0077.h5').
\n\nYou will have to remove several epochs after they are no longer used. For example, if you run for 80 epochs, it is advised to delete the weights from 79 epochs, keep the 80th. It would be better if you could figure out how to use 'save_best_only' instead of 'save_weights_only'. I tried to set 'save_best_only' to True, erasing the latter, but the model did not work beyond the first epoch.
", "rawMarkdown": "The biggest memory consumption comes from saving all those weights at each epoch. What I do is manually delete saved weights from each prior epochs using os.remove(). \n\nFor example: os.remove('checkpoints/nuclei20180308T0559/mask_rcnn_nuclei_0077.h5'). \n\nYou will have to remove several epochs after they are no longer used. For example, if you run for 80 epochs, it is advised to delete the weights from 79 epochs, keep the 80th. It would be better if you could figure out how to use 'save_best_only' instead of 'save_weights_only'. I tried to set 'save_best_only' to True, erasing the latter, but the model did not work beyond the first epoch.", "votes": null }, { "id": "297620", "postDate": "03/17/2018 14:53:16", "content": "I ran this code also get around a low result of LB. About 0.31.
\n\n@john1231983 How did you solve this problem?
", "rawMarkdown": "I ran this code also get around a low result of LB. About 0.31.\n\n\n@john1231983 How did you solve this problem?", "votes": null }, { "id": "297621", "postDate": "03/17/2018 14:55:02", "content": "I used pytorch version. If you use Keras. I think you should you pretrain model with training schedule as coco. Train head and then train all
", "rawMarkdown": "I used pytorch version. If you use Keras. I think you should you pretrain model with training schedule as coco. Train head and then train all", "votes": null }, { "id": "297690", "postDate": "03/17/2018 18:56:59", "content": "You need to have 'save_best_only' = True and 'save_weights_only' = True (otherwise it will try to save the whole model which it can't do because of how the gt_boxes Lambda layer is structured).
", "rawMarkdown": "You need to have 'save_best_only' = True and 'save_weights_only' = True (otherwise it will try to save the whole model which it can't do because of how the gt_boxes Lambda layer is structured).", "votes": null }, { "id": "297813", "postDate": "03/18/2018 01:47:59", "content": "With this code I obtain 0.41+ frequently. @John, your previous scores on matterport code are really impressive, I think it's around 0.47?
", "rawMarkdown": "With this code I obtain 0.41+ frequently. @John, your previous scores on matterport code are really impressive, I think it's around 0.47?", "votes": null }, { "id": "297814", "postDate": "03/18/2018 01:55:23", "content": "Yes. But it used postprocesing like dilation. I trained both gray and color together in 1 network. For Heng's code, I trained on 2 network and result is same. But if I apply post processing for Heng code. The accuracy reduced :(
", "rawMarkdown": "Yes. But it used postprocesing like dilation. I trained both gray and color together in 1 network. For Heng's code, I trained on 2 network and result is same. But if I apply post processing for Heng code. The accuracy reduced :(", "votes": null }, { "id": "297987", "postDate": "03/18/2018 13:47:56", "content": "Any tips about postprocessing you are applying?
", "rawMarkdown": "Any tips about postprocessing you are applying?", "votes": null }, { "id": "298133", "postDate": "03/18/2018 22:32:32", "content": "Also using matterport code here! Heng is correct that mask rcnn only gets you so far (~4.7/4.8), light post processing got me to .49 (nms on flips, binary dilation)
", "rawMarkdown": "Also using matterport code here! Heng is correct that mask rcnn only gets you so far (~4.7/4.8), light post processing got me to .49 (nms on flips, binary dilation)", "votes": null }, { "id": "298770", "postDate": "03/20/2018 02:16:12", "content": "UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
\n\nHi, I use your code and find these warning, and the train is extremly slow, here are some train log \n349/349 [==============================] - 9686s - loss: 0.8279 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.2299 - mrcnn_class_loss: 0.1493 - mrcnn_bbox_loss: 0.1719 - mrcnn_mask_loss: 0.2530 - val_loss: 0.8191 - val_rpn_class_loss: 0.0330 - val_rpn_bbox_loss: 0.1929 - val_mrcnn_class_loss: 0.2293 - val_mrcnn_bbox_loss: 0.1340 - val_mrcnn_mask_loss: 0.2298\nkeras：2.0.8\ntf：1.3.0\nGPU: K40\nhave you meet this problem? and how much time you spend in one epoch?
", "rawMarkdown": "*UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.*\n\nHi, I use your code and find these warning, and the train is extremly slow, here are some train log \n349/349 [==============================] - 9686s - loss: 0.8279 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.2299 - mrcnn_class_loss: 0.1493 - mrcnn_bbox_loss: 0.1719 - mrcnn_mask_loss: 0.2530 - val_loss: 0.8191 - val_rpn_class_loss: 0.0330 - val_rpn_bbox_loss: 0.1929 - val_mrcnn_class_loss: 0.2293 - val_mrcnn_bbox_loss: 0.1340 - val_mrcnn_mask_loss: 0.2298\nkeras：2.0.8\ntf：1.3.0\nGPU: K40\nhave you meet this problem? and how much time you spend in one epoch?", "votes": null }, { "id": "298785", "postDate": "03/20/2018 02:37:53", "content": "Commonly I can run in 3-6 min per epoch with 4 titan x. The speed is heavily affected by the configuration, which may last 3 min to 1 h per epoch.
", "rawMarkdown": "Commonly I can run in 3-6 min per epoch with 4 titan x. The speed is heavily affected by the configuration, which may last 3 min to 1 h per epoch.", "votes": null }, { "id": "298793", "postDate": "03/20/2018 02:49:17", "content": "Hi Yan, I have used Heng code and I found that matterport is more accurate than Heng code. The reason is that it did not support pretrain now (we have to change in the source code to support it). What is your score using Heng code?
", "rawMarkdown": "Hi Yan, I have used Heng code and I found that matterport is more accurate than Heng code. The reason is that it did not support pretrain now (we have to change in the source code to support it). What is your score using Heng code?", "votes": null }, { "id": "298799", "postDate": "03/20/2018 02:57:51", "content": "so you get this warning too?
", "rawMarkdown": "so you get this warning too?", "votes": null }, { "id": "299300", "postDate": "03/20/2018 19:09:29", "content": "Does anyone have any idea why the training time of Matterport Mask RCNN varies so much? For me, each epoch usually takes 15 minutes but after a while (usually after about 13/14 epochs) the training time goes up and takes 30 minutes or more. I noticed, when this happens, CPUs are not busy. It does not make sense to me that training time would change after running for a while, unless there are some bugs in the code. If I stop the training and rerun using 'last' model, training time goes back down to usual 15 minutes for a few epochs and increase again.
\n\n@Yan Wang \nDo you mean that certain configuration takes longer than others? or are you also observing that training time change while training a single model.
", "rawMarkdown": "Does anyone have any idea why the training time of Matterport Mask RCNN varies so much? For me, each epoch usually takes 15 minutes but after a while (usually after about 13/14 epochs) the training time goes up and takes 30 minutes or more. I noticed, when this happens, CPUs are not busy. It does not make sense to me that training time would change after running for a while, unless there are some bugs in the code. If I stop the training and rerun using 'last' model, training time goes back down to usual 15 minutes for a few epochs and increase again.\n\n@Yan Wang \nDo you mean that certain configuration takes longer than others? or are you also observing that training time change while training a single model.", "votes": null }, { "id": "299321", "postDate": "03/20/2018 19:41:31", "content": "Ya this concerned me too; but it’s related to the image size and number of masks (among potentially real performance bugs). I noticed that certain image modalities (e.g. The drosophila nuclei images) take ~ an order of mag longer to train.
\n\nDoes your network training remain permanently slow, or does it become fast again once it’s recycled the through the training images (assuming you haven’t played with the data_generator)?
", "rawMarkdown": "Ya this concerned me too; but it’s related to the image size and number of masks (among potentially real performance bugs). I noticed that certain image modalities (e.g. The drosophila nuclei images) take ~ an order of mag longer to train.\n\nDoes your network training remain permanently slow, or does it become fast again once it’s recycled the through the training images (assuming you haven’t played with the data_generator)?", "votes": null }, { "id": "299357", "postDate": "03/20/2018 20:16:49", "content": "I could not wait long enough to see if it gets faster again. I usually kill it and start again from the last saved model. What's interesting is that the problem seem to show up when I increase the image augmentation (ex. rotate up to 45 degrees instead of up to 10 degrees)\nFor now, I'm just putting up with it since I was not able to get any other Mask RCNN implementation to produce good result.
", "rawMarkdown": "I could not wait long enough to see if it gets faster again. I usually kill it and start again from the last saved model. What's interesting is that the problem seem to show up when I increase the image augmentation (ex. rotate up to 45 degrees instead of up to 10 degrees)\nFor now, I'm just putting up with it since I was not able to get any other Mask RCNN implementation to produce good result.", "votes": null }, { "id": "299437", "postDate": "03/20/2018 21:44:05", "content": "@bsp2020 Sure the speed depends on the configuration. Also speed may change during training.
\n @John So which code gives you higher score so far? I am doing some testing recently, haven`t submitted full csv yet.
I used matterport that give me 0.49 lb now. The Heng code is 0.47. The reason is that te Heng code did not use pretrain model and frozen batch norm. That is what I guess. So I did not use Heng code . I use matterport again
", "rawMarkdown": "I used matterport that give me 0.49 lb now. The Heng code is 0.47. The reason is that te Heng code did not use pretrain model and frozen batch norm. That is what I guess. So I did not use Heng code . I use matterport again", "votes": null }, { "id": "299475", "postDate": "03/20/2018 22:17:17", "content": "@bsp, caching masks in a numpy array helps quite a bit, and cuts down on the costly reading of masks from disk (I profiled and lots of time spent in disk IO, hence no CPU ticks, and also no GPU copy ticks).
\n\nIn my load_image/load_mask code I cache if an image has > 50 masks. Of course you will hit a few slow iterations while you read the images for the first time!
", "rawMarkdown": "bsp, caching masks in a numpy array helps quite a bit, and cuts down on the costly reading of masks from disk (I profiled and lots of time spent in disk IO, hence no CPU ticks, and also no GPU copy ticks). \n\nIn my load_image/load_mask code I cache if an image has > 50 masks. Of course you will hit a few slow iterations while you read the images for the first time!", "votes": null }, { "id": "299483", "postDate": "03/20/2018 22:23:32", "content": "@John you mean pretrain on coco?
", "rawMarkdown": "John you mean pretrain on coco?", "votes": null }, { "id": "299484", "postDate": "03/20/2018 22:25:02", "content": "@Spencer very helpful
", "rawMarkdown": "Spencer very helpful", "votes": null }, { "id": "299500", "postDate": "03/20/2018 22:45:08", "content": "@yan imagenet or coco. The current code of Heng cannot use anh pretrain. I hope he can release next version with pretrain imagenet. To do it, i guess the featurenet function need to change to follow the rule of resnet in pytorchvision. Have you try with imagebet pretrain on Heng code?
", "rawMarkdown": "yan imagenet or coco. The current code of Heng cannot use anh pretrain. I hope he can release next version with pretrain imagenet. To do it, i guess the featurenet function need to change to follow the rule of resnet in pytorchvision. Have you try with imagebet pretrain on Heng code?", "votes": null }, { "id": "299508", "postDate": "03/20/2018 22:52:15", "content": "It's not difficult to use the official pretrained weight in pytorch if using resnet as the feature net.
", "rawMarkdown": "It's not difficult to use the official pretrained weight in pytorch if using resnet as the feature net.", "votes": null }, { "id": "299547", "postDate": "03/20/2018 23:58:21", "content": "Hi yan. Have you ever try the pretrain for Heng code using resnet50? Does it improve performance. I have try but i got the error. If you was success. Could you share the modified featurenet function to use pretrain model?
", "rawMarkdown": "Hi yan. Have you ever try the pretrain for Heng code using resnet50? Does it improve performance. I have try but i got the error. If you was success. Could you share the modified featurenet function to use pretrain model?", "votes": null }, { "id": "299775", "postDate": "03/21/2018 03:45:17", "content": "to those using matterport, have you seen this issue: https://github.com/matterport/Mask_RCNN/issues/89
\n\nbasically the validation metrics are a lie :)
", "rawMarkdown": "to those using matterport, have you seen this issue: https://github.com/matterport/Mask_RCNN/issues/89\n\nbasically the validation metrics are a lie :)", "votes": null }, { "id": "299836", "postDate": "03/21/2018 04:47:30", "content": "Thanks. How do you fix it? remove next?
", "rawMarkdown": "Thanks. How do you fix it? remove next?", "votes": null }, { "id": "299847", "postDate": "03/21/2018 04:54:39", "content": "the fix is in the issue a little further down in jremillard‘s comment with the thumb’s up—it requires editing keras code unfortunately (but it works).
\n\nI’m considering creating a custom callback (further hacking the matterport train call to accept user def’d callbacks) that does validation.
", "rawMarkdown": "the fix is in the issue a little further down in jremillard‘s comment with the thumb’s up—it requires editing keras code unfortunately (but it works).\n\nI’m considering creating a custom callback (further hacking the matterport train call to accept user def’d callbacks) that does validation.", "votes": null }, { "id": "299850", "postDate": "03/21/2018 04:56:33", "content": "Does it affect to performance. I am based on the matterport and the performance is not stable. I cannot recovert my best performance when i rerun :(
", "rawMarkdown": "Does it affect to performance. I am based on the matterport and the performance is not stable. I cannot recovert my best performance when i rerun :(", "votes": null }, { "id": "299858", "postDate": "03/21/2018 05:05:14", "content": "my initial model was using val_rpn_bbox_loss to drive logic to update learning rate and model weights saving.
\n\nReproducibility has been an issue for me too; I’ve been studying the matterport code pretty closely these last few days and I’m trying to reproduce my best model.
\n\nSo far I’ve set the numpy seed and be tensorflow graph seed, and using only a single GPU. I’ll report on reproducibility as I the week goes on. :)
", "rawMarkdown": "my initial model was using val_rpn_bbox_loss to drive logic to update learning rate and model weights saving.\n\nReproducibility has been an issue for me too; I’ve been studying the matterport code pretty closely these last few days and I’m trying to reproduce my best model.\n\nSo far I’ve set the numpy seed and be tensorflow graph seed, and using only a single GPU. I’ll report on reproducibility as I the week goes on. :)", "votes": null }, { "id": "299861", "postDate": "03/21/2018 05:06:22", "content": "another proposal by uncommenting \"loss = K.reshape(loss, [1, 1])\" also does fix the error but does it have side effect?
", "rawMarkdown": "another proposal by uncommenting \"loss = K.reshape(loss, [1, 1])\" also does fix the error but does it have side effect?", "votes": null }, { "id": "300454", "postDate": "03/21/2018 16:05:34", "content": "Thanks for pointing that out. I was losing my mind trying to figure out why validation loss is not improving.
", "rawMarkdown": "Thanks for pointing that out. I was losing my mind trying to figure out why validation loss is not improving.", "votes": null }, { "id": "301125", "postDate": "03/22/2018 10:52:21", "content": "Very interesting, thank you for sharing. Btw, I am also using matterport and get about 0.49 on LB
", "rawMarkdown": "Very interesting, thank you for sharing. Btw, I am also using matterport and get about 0.49 on LB", "votes": null }, { "id": "301232", "postDate": "03/22/2018 14:10:29", "content": "@Keven Wang, impressive, are you using post processing? What's the score without post-processing?
", "rawMarkdown": "Keven Wang, impressive, are you using post processing? What's the score without post-processing?", "votes": null }, { "id": "301350", "postDate": "03/22/2018 16:48:18", "content": "@Yan Wang well, i have not applied any post processing except remove the overlapping between two mask instance. But my next step would be post processing, any suggestions?
", "rawMarkdown": "Yan Wang well, i have not applied any post processing except remove the overlapping between two mask instance. But my next step would be post processing, any suggestions?", "votes": null }, { "id": "301385", "postDate": "03/22/2018 17:23:09", "content": "Hey Keven. Did you get this LB score after fixing the validation generator issue? If so, did you train from scratch? Or just, train on top of your previous pretrained model?
", "rawMarkdown": "Hey Keven. Did you get this LB score after fixing the validation generator issue? If so, did you train from scratch? Or just, train on top of your previous pretrained model?", "votes": null }, { "id": "301386", "postDate": "03/22/2018 17:23:30", "content": "I have not tried any post processing now.
", "rawMarkdown": "I have not tried any post processing now.", "votes": null }, { "id": "301403", "postDate": "03/22/2018 17:46:11", "content": "Very impressive Keven. could you tell us how many epochs and if you used default SGD optimizer ?
\nWith the config below, I am stuck as 0.415-0.418 .. \nI choose MEAN_PIXEL = np.array([0.,0.,0.]) because i think putting a specific value for MEAN_PIXEL will not generalize well.. Am I wrong about this ?
==========================================================================\nLEARNING_RATE = 1e-2
\n\n# If enabled, resizes instance masks to a smaller size to reduce\n# memory load. Recommended when using high-resolution image\nUSE_MINI_MASK = True\nMINI_MASK_SHAPE = (56, 56) # (height, width) of the mini-mask\n\n# Train on 1 GPU and 8 images per GPU. Batch size is GPUs * images/GPU.\nGPU_COUNT = 1\nIMAGES_PER_GPU = 2\n# Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch.\n# typically be equal to the number of samples of your dataset divided by the batch size\nSTEPS_PER_EPOCH = 300\nVALIDATION_STEPS = 70\n\n# Number of classes (including background)\nNUM_CLASSES = 1 + 1 # background + nucleis\nIMAGE_MIN_DIM = 512\nIMAGE_MAX_DIM = 512\nIMAGE_PADDING = True # currently, the False option is not supported\nRPN_ANCHOR_SCALES = (8, 16, 32, 64, 128) # anchor side in pixels, maybe add a 256?\n# The strides of each layer of the FPN Pyramid. These values\n# are based on a Resnet101 backbone.\nBACKBONE_STRIDES = [4, 8, 16, 32, 64]\n# How many anchors per image to use for RPN training\nRPN_TRAIN_ANCHORS_PER_IMAGE = 320 #300\n\n# ROIs kept after non-maximum supression (training and inference)\nPOST_NMS_ROIS_TRAINING = 2000\nPOST_NMS_ROIS_INFERENCE = 2000\nPOOL_SIZE = 7\nMASK_POOL_SIZE = 14\nMASK_SHAPE = [28, 28]\nTRAIN_ROIS_PER_IMAGE = 512\nRPN_NMS_THRESHOLD = 0.7\nMAX_GT_INSTANCES = 256\nDETECTION_MAX_INSTANCES = 400 \n# Minimum probability value to accept a detected instance\n# ROIs below this threshold are skipped\nDETECTION_MIN_CONFIDENCE = 0.7 # may be smaller?\n# Non-maximum suppression threshold for detection\nDETECTION_NMS_THRESHOLD = 0.3 # 0.3\n\n\nMEAN_PIXEL = np.array([0.,0.,0.])\n\n# Weight decay regularization\nWEIGHT_DECAY = 0.0001\nhere is a nice discussion on post-processing:\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51476
", "rawMarkdown": "here is a nice discussion on post-processing:\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51476", "votes": null }, { "id": "301590", "postDate": "03/23/2018 01:12:37", "content": "@Mehul Sampat, I also set the MEAN_PIXEL to np.array([0.,0.,0.]) but when I use the coco pre-trained model I got loss:nan, any suggestions?
", "rawMarkdown": "Mehul Sampat, I also set the MEAN_PIXEL to np.array([0.,0.,0.]) but when I use the coco pre-trained model I got loss:nan, any suggestions?", "votes": null }, { "id": "301965", "postDate": "03/23/2018 14:22:52", "content": "@Mehul Sampat same thing stuck at 0.413-0.418 with pure maskrcnn. I don't know how john n kevin manage to get 0.49 without any post processing :o, thats amazing.
", "rawMarkdown": "Mehul Sampat same thing stuck at 0.413-0.418 with pure maskrcnn. I don't know how john n kevin manage to get 0.49 without any post processing :o, thats amazing.", "votes": null }, { "id": "301979", "postDate": "03/23/2018 14:39:05", "content": "hi all, i used data augmentation a lot, i think especially random cropping helps with overfitting. also, i used SGD instead of ADAM since there are some research paper shows that SGD usually can get better local optimal than adaptive optimiser like adam, also mask r-cnn, resnet paper all use SGD instead of adam.
\n\nAlso i achieve above result before val loss fix, i would say i am pretty luck to find the local optimal epoch but also i did find the result was not repeatable before val loss fix so i did many times, now the result is more predictable.
\n\nbtw, i have not changed MEAN_PIXEL so far, maybe i shall take a try.
\n\nlast but not least, i also tried to classify different image with different background/foreground into different class, that doesn't help much in term of performance, so i didn't do it any more (maybe this is another strange result thx to val loss bug, i probably should try again now)
", "rawMarkdown": "hi all, i used data augmentation a lot, i think especially random cropping helps with overfitting. also, i used SGD instead of ADAM since there are some research paper shows that SGD usually can get better local optimal than adaptive optimiser like adam, also mask r-cnn, resnet paper all use SGD instead of adam. \n\nAlso i achieve above result before val loss fix, i would say i am pretty luck to find the local optimal epoch but also i did find the result was not repeatable before val loss fix so i did many times, now the result is more predictable. \n\nbtw, i have not changed MEAN_PIXEL so far, maybe i shall take a try.\n\nlast but not least, i also tried to classify different image with different background/foreground into different class, that doesn't help much in term of performance, so i didn't do it any more (maybe this is another strange result thx to val loss bug, i probably should try again now)", "votes": null }, { "id": "301986", "postDate": "03/23/2018 14:45:42", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "302003", "postDate": "03/23/2018 15:12:49", "content": "Thanks Keven for the tips about data augmentation! I will try it out soon. \nAlso MEAN_PIXEL is interesting. I am thinking for better generalization, I dont want to pick a fixed value but use [0,0,0]. To match this option i thought to use this normalization for each images. \nnormalized_img = ( img - mean_of_img )/ stdev_of_img; in both train and testing; this ensure each image has zero-mean and unit standard deviation. Still need to compare if this better than no normalization or just dividing by 255.
", "rawMarkdown": "Thanks Keven for the tips about data augmentation! I will try it out soon. \nAlso MEAN_PIXEL is interesting. I am thinking for better generalization, I dont want to pick a fixed value but use [0,0,0]. To match this option i thought to use this normalization for each images. \nnormalized_img = ( img - mean_of_img )/ stdev_of_img; in both train and testing; this ensure each image has zero-mean and unit standard deviation. Still need to compare if this better than no normalization or just dividing by 255.", "votes": null }, { "id": "302008", "postDate": "03/23/2018 15:14:08", "content": "@IAMLegend. when I use image size 128-by-128. I had that issue. can you try 256-by-256 and see if the nan issue goes away ? I think in some other thread also another suggestion was given. i will try and see if i can find it.
", "rawMarkdown": "IAMLegend. when I use image size 128-by-128. I had that issue. can you try 256-by-256 and see if the nan issue goes away ? I think in some other thread also another suggestion was given. i will try and see if i can find it.", "votes": null }, { "id": "302012", "postDate": "03/23/2018 15:17:13", "content": "@Mehul yes, in general normalisation is good, so don't take my experience as ground truth:) It is just i didn't have time to try it yet, i will play with it too.
", "rawMarkdown": "Mehul yes, in general normalisation is good, so don't take my experience as ground truth:) It is just i didn't have time to try it yet, i will play with it too.", "votes": null }, { "id": "302014", "postDate": "03/23/2018 15:23:27", "content": "Hi Keven. How do you fix val loss? You uncomment the line loss = K.reshape(loss, [1, 1]), right? I have tried sgd with resnet50 pretrain but it provided lower performancem do you use clip 0.5 in sgd?
", "rawMarkdown": "Hi Keven. How do you fix val loss? You uncomment the line loss = K.reshape(loss, [1, 1]), right? I have tried sgd with resnet50 pretrain but it provided lower performancem do you use clip 0.5 in sgd?", "votes": null }, { "id": "302018", "postDate": "03/23/2018 15:28:05", "content": "Hi John, i took another approach suggested in GitHub, modified keras. It might be a pb when submit the model at stage 1 deadline, but we will see.
\n\nFor me, this approach seems ok, with the same configuration, i still get about 0.486 pretty close to highest score before change. yes, i used clip 0.5, the default configuration in code base.
\n\nTo me, the biggest challenge is (1) high variance, because train loss is about 8% better than val loss. (2) maybe this is the same reason as (1), but i see when IoU threshold is 0.9 or 0.95, the mAP is always 0 or very close to 0 (like 0.000x), not sure if there is any parameter we can tune.
", "rawMarkdown": "Hi John, i took another approach suggested in GitHub, modified keras. It might be a pb when submit the model at stage 1 deadline, but we will see.\n\nFor me, this approach seems ok, with the same configuration, i still get about 0.486 pretty close to highest score before change. yes, i used clip 0.5, the default configuration in code base.\n\nTo me, the biggest challenge is (1) high variance, because train loss is about 8% better than val loss. (2) maybe this is the same reason as (1), but i see when IoU threshold is 0.9 or 0.95, the mAP is always 0 or very close to 0 (like 0.000x), not sure if there is any parameter we can tune.", "votes": null }, { "id": "302022", "postDate": "03/23/2018 15:33:12", "content": "Are you use val for control early stop? Because i did not use it so i think it will not effect to my performance. I am using Adam and I got 0.494 but i cannot reproduce the result with same setting. I guess the reason is from Adam. As you said, you can reproduce the performance when using SGD. Am I right?
", "rawMarkdown": "Are you use val for control early stop? Because i did not use it so i think it will not effect to my performance. I am using Adam and I got 0.494 but i cannot reproduce the result with same setting. I guess the reason is from Adam. As you said, you can reproduce the performance when using SGD. Am I right?", "votes": null }, { "id": "302284", "postDate": "03/23/2018 21:29:11", "content": "No, i didn't use early stop, only use tensor board to identify the optimal epoch. regarding reproducibility, with same configuration, first time i got 0.492, second time i got 0.486, pretty close i would say.
", "rawMarkdown": "No, i didn't use early stop, only use tensor board to identify the optimal epoch. regarding reproducibility, with same configuration, first time i got 0.492, second time i got 0.486, pretty close i would say.", "votes": null }, { "id": "302532", "postDate": "03/24/2018 08:51:58", "content": "@john1231983, did you see improvement with resnet50 over resnet101? And have you figured out pre-trained weight for resnet50?
", "rawMarkdown": "john1231983, did you see improvement with resnet50 over resnet101? And have you figured out pre-trained weight for resnet50?", "votes": null }, { "id": "302534", "postDate": "03/24/2018 08:57:53", "content": "@keven: i don't think resnet101 is good choice. I am still use resnet50 . Itried resnet101 but performance is similar. The pretrained weight of resnet50 is from keras. I used SGD but performance is 0.44. I switched to adam and check it again. I remember only adam can give me 0.49LB. For sgd, i used learning rate 0.001 for 60 epochs for training heads, 0.0001 for 40 epoch training heads, then 100epochs for training all with learning rate 0.0001. Performance shows 0.44lb is too bad. Any tips for training with sgd?
", "rawMarkdown": "keven: i don't think resnet101 is good choice. I am still use resnet50 . Itried resnet101 but performance is similar. The pretrained weight of resnet50 is from keras. I used SGD but performance is 0.44. I switched to adam and check it again. I remember only adam can give me 0.49LB. For sgd, i used learning rate 0.001 for 60 epochs for training heads, 0.0001 for 40 epoch training heads, then 100epochs for training all with learning rate 0.0001. Performance shows 0.44lb is too bad. Any tips for training with sgd?", "votes": null }, { "id": "302542", "postDate": "03/24/2018 09:14:56", "content": "Oh, I just realised by default matterport downloads resnet50 weight instead of 101, so I guess I just need to switch architecture to resnet101 right?
Regarding train with SGD, I tried both approaches, train all from beginning to end and train heads then train all, actually only train all gives slightly better result. But as I mentioned, I didn't use early stop, I go through the loss and val loss in tensor board and pick the most promising one. Actually, usually epoch <50 gives me better result, I guess above that the model starts to overfit. Again, it is result from resnet101.
Yes. You have change the resnet101 to resnet50 in model.py or in config if you are using newest version
", "rawMarkdown": "Yes. You have change the resnet101 to resnet50 in model.py or in config if you are using newest version", "votes": null }, { "id": "302943", "postDate": "03/25/2018 03:10:28", "content": "Hi Keven What are the augmentations that you are using? I ave tried random flips, zoom and crop but still no luck with improving the score.
", "rawMarkdown": "Hi Keven What are the augmentations that you are using? I ave tried random flips, zoom and crop but still no luck with improving the score.", "votes": null }, { "id": "302985", "postDate": "03/25/2018 06:25:16", "content": "@Mungos, @Keven, my score improved from 0.415 to 0.432 after normalization (no data augmentation)\nnormalized_img = ( img - mean_of_img )/ stdev_of_img
\n\nI look at val los in tensorboard and i have a dip (minima) at 12th epoch and then it keeps getting worse. \ni will try and lower learning rate as others suggest.
", "rawMarkdown": "Mungos, @Keven, my score improved from 0.415 to 0.432 after normalization (no data augmentation)\nnormalized_img = ( img - mean_of_img )/ stdev_of_img\n\nI look at val los in tensorboard and i have a dip (minima) at 12th epoch and then it keeps getting worse. \ni will try and lower learning rate as others suggest.", "votes": null }, { "id": "302986", "postDate": "03/25/2018 06:27:28", "content": "
Hi Mehul, that is a good result. Can you tell me how to implement this normalization? Where should this function be placed? Thanks Zheng
", "rawMarkdown": "Hi Mehul, that is a good result. Can you tell me how to implement this normalization? Where should this function be placed? Thanks Zheng", "votes": null }, { "id": "303723", "postDate": "03/26/2018 15:33:37", "content": "@Zheng, you can do it at the place you load your train and test images. I did this: \nmean = np.mean(img)\nstd = np.std(img)\nimg = (img-mean)/std\nAlso i found resnet50 give better result than resnet101 (which is default for matterport rcnn) . this can be selected by putting the following line in the config file: BACKBONE = \"resnet50\"
", "rawMarkdown": "Zheng, you can do it at the place you load your train and test images. I did this: \nmean = np.mean(img)\nstd = np.std(img)\nimg = (img-mean)/std\nAlso i found resnet50 give better result than resnet101 (which is default for matterport rcnn) . this can be selected by putting the following line in the config file: BACKBONE = \"resnet50\"", "votes": null }, { "id": "303764", "postDate": "03/26/2018 16:25:55", "content": "@Mehul, that is so great. Thanks a lot. I will let you know the result after I finish. Have you ever tried training based on imagenet pretrained model? Sometimes I can’t figure out what’s the difference between the imagenet pretrained model and coco. It seems we just choose it by random. Is there some theories can explain why we should choose this model while others shouldn’t? Thanks Zheng
", "rawMarkdown": "Mehul, that is so great. Thanks a lot. I will let you know the result after I finish. Have you ever tried training based on imagenet pretrained model? Sometimes I can’t figure out what’s the difference between the imagenet pretrained model and coco. It seems we just choose it by random. Is there some theories can explain why we should choose this model while others shouldn’t? Thanks Zheng", "votes": null }, { "id": "303777", "postDate": "03/26/2018 16:46:04", "content": "mold_inputs in model.py already subtract MEAN_PIXEL (mean of the training data of the pre-trained weight) from the input. So, normalizing the input with it's per image mean/std seems counter productive. I'd be interested in hearing about your result.
", "rawMarkdown": "mold_inputs in model.py already subtract MEAN_PIXEL (mean of the training data of the pre-trained weight) from the input. So, normalizing the input with it's per image mean/std seems counter productive. I'd be interested in hearing about your result.", "votes": null }, { "id": "303786", "postDate": "03/26/2018 16:57:38", "content": "@bsp2020 is right. I did normalization by subtraction of the mean/std when using unet but when I tried it with matterport, it got the masks out of bounds. I then checked the code and saw it is done in config.py
", "rawMarkdown": "bsp2020 is right. I did normalization by subtraction of the mean/std when using unet but when I tried it with matterport, it got the masks out of bounds. I then checked the code and saw it is done in config.py", "votes": null }, { "id": "303789", "postDate": "03/26/2018 17:01:48", "content": "This may be a silly question. Where do I change the architecture to resnet50 in the code?
", "rawMarkdown": "This may be a silly question. Where do I change the architecture to resnet50 in the code?", "votes": null }, { "id": "303805", "postDate": "03/26/2018 17:16:24", "content": "@William Green you change the line (BACKBONE = \"resnet101\") to resnet50 in the config.py file or add it if it does not exist.
", "rawMarkdown": "William Green you change the line (BACKBONE = \"resnet101\") to resnet50 in the config.py file or add it if it does not exist.", "votes": null }, { "id": "303815", "postDate": "03/26/2018 17:30:50", "content": "architecture is set in resnet_graph function in model.py. Search for resnet101. It should be the line with block_count
", "rawMarkdown": "architecture is set in resnet_graph function in model.py. Search for resnet101. It should be the line with block_count", "votes": null }, { "id": "303828", "postDate": "03/26/2018 17:38:47", "content": "@YanGana and Panserbjorne\nThank you.
", "rawMarkdown": "YanGana and Panserbjorne\nThank you.", "votes": null }, { "id": "304318", "postDate": "03/27/2018 10:58:15", "content": "@bsp2020 your score is very impressive. Can you share some tips about data augmentation?
", "rawMarkdown": "bsp2020 your score is very impressive. Can you share some tips about data augmentation?", "votes": null }, { "id": "304585", "postDate": "03/27/2018 19:15:48", "content": "I took Keras image augmentation class and modified it a bit to fit it into MatterPort Mask RCNN shapes demo code. So, basically, common image augmentation such as rotation, scaling, etc. Rotating +-5 degrees on top of flip & 90 degree rotation made a big difference for me.
\n\nI think I've hit the limit of my current approach. So, can't give out too much more information. :)
", "rawMarkdown": "I took Keras image augmentation class and modified it a bit to fit it into MatterPort Mask RCNN shapes demo code. So, basically, common image augmentation such as rotation, scaling, etc. Rotating +-5 degrees on top of flip & 90 degree rotation made a big difference for me.\n\nI think I've hit the limit of my current approach. So, can't give out too much more information. :)", "votes": null }, { "id": "304727", "postDate": "03/28/2018 00:06:19", "content": "do you used any pre-processing to reach 0.47 (crop, flip or morphology）with matterport code? I only got around 0.4 without any pre-processing or post-processing.
", "rawMarkdown": "do you used any pre-processing to reach 0.47 (crop, flip or morphology）with matterport code? I only got around 0.4 without any pre-processing or post-processing.", "votes": null }, { "id": "304852", "postDate": "03/28/2018 05:45:50", "content": "Thanks for your reply. I found that rotation doesn't improve my performance. My score drops from 0.424 to 0.409. Have you ever encountered the same situation before?
", "rawMarkdown": "Thanks for your reply. I found that rotation doesn't improve my performance. My score drops from 0.424 to 0.409. Have you ever encountered the same situation before?", "votes": null }, { "id": "304853", "postDate": "03/28/2018 05:58:22", "content": "@bsp2020 My code is def data_augmentation(image,masks,rotation=360)\nrotate_times=random.randint(0,rotation/90)\nfor r in range(0, rotate_times):\nimage=np.rot90(image)\nmasks=np.rot90(masks)It hurt my performance. Can you give me some advises about rotation? Thanks Zheng
Hi Zheng, \nI think the issue might be with np.rot90 function. this is rotating by 90 degrees which is too much. \ncould you please try and rotate by smaller amount like +/- 15 degrees ? Also congrats on the score improvement to 0.440. are you using the Matterport mask rcnn code too ?
", "rawMarkdown": "Hi Zheng, \nI think the issue might be with np.rot90 function. this is rotating by 90 degrees which is too much. \ncould you please try and rotate by smaller amount like +/- 15 degrees ? Also congrats on the score improvement to 0.440. are you using the Matterport mask rcnn code too ?", "votes": null }, { "id": "304873", "postDate": "03/28/2018 06:37:42", "content": "Hi Mehul, thank you. Yes, I am using the Matterport mask rcnn code.
", "rawMarkdown": "Hi Mehul, thank you. Yes, I am using the Matterport mask rcnn code.", "votes": null }, { "id": "308801", "postDate": "04/04/2018 06:06:39", "content": "@Mehul Hi,Mehul. I have tried to implement general normalization. It only got 0.389 without any post processing.Learning schedule is 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50. I can't reproduce your result. Can you give me some more details about general normalization?
", "rawMarkdown": "Mehul Hi,Mehul. I have tried to implement general normalization. It only got 0.389 without any post processing.Learning schedule is 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50. I can't reproduce your result. Can you give me some more details about general normalization?", "votes": null }, { "id": "308910", "postDate": "04/04/2018 10:14:23", "content": "Hi @Zheng Li, were you able to get a better result with rotation of +_15 as suggested above?
\n\nAbout normalization, I also got a lower result, i.e. it worsened my LB score. I tried 3 different types of normalization and none of them helped.
", "rawMarkdown": "Hi @Zheng Li, were you able to get a better result with rotation of +_15 as suggested above? \n\nAbout normalization, I also got a lower result, i.e. it worsened my LB score. I tried 3 different types of normalization and none of them helped.", "votes": null }, { "id": "308966", "postDate": "04/04/2018 12:48:03", "content": "Hi, YaGana. I have tried to rotate +- 5 degrees, but no better result achieved.
", "rawMarkdown": "Hi, YaGana. I have tried to rotate +- 5 degrees, but no better result achieved.", "votes": null }, { "id": "309048", "postDate": "04/04/2018 15:00:53", "content": "@ZhengLi, I also finally got to 0.440 ; I got it in 11 epochs (LR = 1e-3). train on all instead of header. i only flipped images for img-aug. This is with resnet101 and no scaling and Mean Pixel =[0 0 0]. I am trying now if scaling improves this score; Also just checking you are using the latest Matterport Mask RCNN; there are some new updates in it. \nTo split into train/validation dataset, I followed the procedure shown in this notebook. \nhttps://github.com/neptune-ml/open-solution-data-science-bowl-2018/blob/mask_rcnn_notebook/main.ipynb (not sure yet if this is the best strategy to split into train and validation yet). Let me know if you need more information on it.
", "rawMarkdown": "ZhengLi, I also finally got to 0.440 ; I got it in 11 epochs (LR = 1e-3). train on all instead of header. i only flipped images for img-aug. This is with resnet101 and no scaling and Mean Pixel =[0 0 0]. I am trying now if scaling improves this score; Also just checking you are using the latest Matterport Mask RCNN; there are some new updates in it. \nTo split into train/validation dataset, I followed the procedure shown in this notebook. \nhttps://github.com/neptune-ml/open-solution-data-science-bowl-2018/blob/mask_rcnn_notebook/main.ipynb (not sure yet if this is the best strategy to split into train and validation yet). Let me know if you need more information on it.", "votes": null }, { "id": "309068", "postDate": "04/04/2018 15:27:00", "content": "@ZhengLi and @YaGana, did you add the external H&E data to your training set. I just start doing this. \nhttps://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations/notebook .I think this might be more helpful than augmentation and normalization since we have very few color training images.
", "rawMarkdown": "ZhengLi and @YaGana, did you add the external H&E data to your training set. I just start doing this. \nhttps://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations/notebook .I think this might be more helpful than augmentation and normalization since we have very few color training images.", "votes": null }, { "id": "309093", "postDate": "04/04/2018 16:23:43", "content": "@Mehul Sampat
\nHave looked into other cloud provider that has GPU? I'm trying GPUeater.com since they have AMD GPU instances. Their instances NVidia instances seem cheaper than GCP. Fast.ai teacher also mentioned Paperspace.com but I have not tried it.
If K80 works for you, you can get it for free using Google Colaboratory. :)
", "rawMarkdown": "Mehul Sampat \nHave looked into other cloud provider that has GPU? I'm trying GPUeater.com since they have AMD GPU instances. Their instances NVidia instances seem cheaper than GCP. Fast.ai teacher also mentioned Paperspace.com but I have not tried it.\n\nIf K80 works for you, you can get it for free using Google Colaboratory. :)", "votes": null }, { "id": "309366", "postDate": "04/05/2018 06:44:48", "content": "@ZhengLi, \nyou mentioned \" 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50\"; where did you find the weights for coco and resnet50. i thought coco weights were for resnet101 only and for resnet50 we need to init with imagenet (please correct me if i am wrong).
\n\nalso did you check val_loss for earlier epochs ? sometime when i run 20 epochs my best validation result is not the last epoch but somewhere in middle like 13th or 14th epoch. the train_loss and other train variables are not much helpful. they maybe going down but maybe the model is just memorizing the small dataset (please correct me if i am wrong here also.). I use tensorboard to try and pick the best weights.
", "rawMarkdown": "ZhengLi, \nyou mentioned \" 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50\"; where did you find the weights for coco and resnet50. i thought coco weights were for resnet101 only and for resnet50 we need to init with imagenet (please correct me if i am wrong). \n\nalso did you check val_loss for earlier epochs ? sometime when i run 20 epochs my best validation result is not the last epoch but somewhere in middle like 13th or 14th epoch. the train_loss and other train variables are not much helpful. they maybe going down but maybe the model is just memorizing the small dataset (please correct me if i am wrong here also.). I use tensorboard to try and pick the best weights.", "votes": null }, { "id": "309405", "postDate": "04/05/2018 08:43:56", "content": "Hi Mehul. I have tried your method. 12epochs all+LR(1e-3)+resnet101+fliplr&flipud+general normalization. It only got LB 0.414. still can't reproduce your result. :( \nDoes \"flipped images\" means fliplr&flipud? \nval_loss sometimes can't reflect the performance of your model. It depends on your testset.I chose the lowest point which val_loss is 0.22. It only got 0.230...I chose the last point and it got 0.440.\nCurrently I am using YanWang's code.https://github.com/YanWang2014/Mask_RCNN\nI have used h&e dataset but only got lower result....
", "rawMarkdown": "Hi Mehul. I have tried your method. 12epochs all+LR(1e-3)+resnet101+fliplr&flipud+general normalization. It only got LB 0.414. still can't reproduce your result. :( \nDoes \"flipped images\" means fliplr&flipud? \nval_loss sometimes can't reflect the performance of your model. It depends on your testset.I chose the lowest point which val_loss is 0.22. It only got 0.230...I chose the last point and it got 0.440.\nCurrently I am using YanWang's code.https://github.com/YanWang2014/Mask_RCNN\nI have used h&e dataset but only got lower result....", "votes": null }, { "id": "309562", "postDate": "04/05/2018 15:15:13", "content": "Hi Zheng, \ncan we disuss via skype ? My skype id is mpsampat. I also want to ask what changes helped you increase performance from 0.44 to 0.476; would you be able to give us some hints please :-) ?
", "rawMarkdown": "Hi Zheng, \ncan we disuss via skype ? My skype id is mpsampat. I also want to ask what changes helped you increase performance from 0.44 to 0.476; would you be able to give us some hints please :-) ?", "votes": null }, { "id": "309881", "postDate": "04/06/2018 05:37:50", "content": "Hi Mehul. You can contact me by email. mdlszhengli@@gmail.com. Mosacis may help you to improve your result. https://www.kaggle.com/bonlime/train-test-image-mosaic/notebook
", "rawMarkdown": "Hi Mehul. You can contact me by email. mdlszhengli@@gmail.com. Mosacis may help you to improve your result. https://www.kaggle.com/bonlime/train-test-image-mosaic/notebook", "votes": null } ], "comments": [ { "id": 288261, "author_name": "john1231983", "author_url": "", "post_date": "02/27/2018 03:52:07", "content": "Hi Yang, Nice to see your result. I am also using matterport implementation (because the other implementation has the own license-that I think we can not use in this challenge). When you submit the result, do you combine all masks into one mask then encode RLE or encode individually. If you encode individually, could you tell me how can encode each mask individually?
", "votes": null, "replies": [ { "id": 288268, "author_name": "wayne999", "author_url": "", "post_date": "02/27/2018 04:21:48", "content": "Hi John, masks are encoded individually. \nMy full code is available here enter link description here
", "votes": null, "replies": [] }, { "id": 288270, "author_name": "john1231983", "author_url": "", "post_date": "02/27/2018 04:31:44", "content": "Thanks. Do you separate the training set into 3 clusters and train each cluster individually? I used 3 clusters and this is loss for 1 cluster at Epoch 152 and crop size 128x128
\n394s 394ms/step - loss: 0.7758 - rpn_class_loss: 0.0184 - rpn_bbox_loss: 0.2962 - mrcnn_class_loss: 0.0454 - mrcnn_bbox_loss: 0.1637 - mrcnn_mask_loss: 0.2522 - val_loss: 1.0752 - val_rpn_class_loss: 0.0406 - val_rpn_bbox_loss: 0.3560 - val_mrcnn_class_loss: 0.0823 - val_mrcnn_bbox_loss: 0.2309 - val_mrcnn_mask_loss: 0.3653\n\nI trained with resnet-50 pretrain model and train all. I will report my performance when it completed. For now, I have to think the way to peform inference with arbitrary size (instead of zero padding). One person have convert fully connected to fully convolution and the LB increased to 0.46. Hope it can help you some idea to improve performance
How much using 3 clusters improved compared to not using them? How do you use models trained on clusters for test, given the modalities are not the same as in train?
", "votes": null, "replies": [] }, { "id": 288656, "author_name": "dxchen", "author_url": "", "post_date": "02/28/2018 00:35:15", "content": "Hi Yan, how many epoch of training do you get to LB 4.3? What do you think the most important factor is to get the score?
", "votes": null, "replies": [] }, { "id": 288695, "author_name": "wayne999", "author_url": "", "post_date": "02/28/2018 02:40:31", "content": "The score is obtained by one of my team members, with some modifications to the nn. As far as I know, the baseline (without any additional work, just tune the parameters) for matterport mask rcnn should be around 0.411.
", "votes": null, "replies": [] } ] }, { "id": 288331, "author_name": "elenacuoco", "author_url": "", "post_date": "02/27/2018 08:28:25", "content": "Hi All,\nI'm not expert at all of computer vision and this is the first approach. I'm using this implementation of mask rcnn\nhttps://github.com/multimodallearning/pytorch-mask-rcnn \nand at the moment I'm stuck at 0.360. I used a smaller mini mask (16,16) or (32,32). I will try with the standard values (56,56)
", "votes": null, "replies": [ { "id": 289051, "author_name": "britefury", "author_url": "", "post_date": "02/28/2018 21:23:16", "content": "Thanks for the link!\nThe only other PyTorch Mask RCNN implementation I am aware of is incomplete and inference only.
", "votes": null, "replies": [] }, { "id": 289090, "author_name": "wayne999", "author_url": "", "post_date": "02/28/2018 22:16:29", "content": "Hi Geoff, your score is impressive. Which code base you are using?
", "votes": null, "replies": [] }, { "id": 289289, "author_name": "elenacuoco", "author_url": "", "post_date": "03/01/2018 09:59:57", "content": "Hi Geoff, this pytorch version works very well. It is a translation of keras matterport. Anyhow I did not reach the results people wrote in the forum, and I'm stuck at 0.360. I've tried also image augmentation, but with mini_mask and rotations it fails. Did you get your score with any version of mask rcnn?
", "votes": null, "replies": [] }, { "id": 289344, "author_name": "britefury", "author_url": "", "post_date": "03/01/2018 12:51:48", "content": "I'm using my own algorithm, not Mask R-CNN.
\n\nI may well switch over to Mask R-CNN as I have used pretty much all the tricks I know to achieve the score I have and I don't really know how to improve matters!
\n\nThanks for pointing this implementation out as I think I will use it as a start.
", "votes": null, "replies": [] }, { "id": 289346, "author_name": "cpmpml", "author_url": "", "post_date": "03/01/2018 13:00:36", "content": "Your great score is encouraging for those of us who try their own stuff rather than parameterize someone else implementation!
", "votes": null, "replies": [] }, { "id": 290977, "author_name": "sheriytm", "author_url": "", "post_date": "03/05/2018 10:19:19", "content": "Ditto @CPMP and @Geoff French. I am still stretching my own algorithm and hoping to get round to Mask RCNN at some point. Both of your results are encouraging. Thanks.
", "votes": null, "replies": [] } ] }, { "id": 288360, "author_name": "fabsta", "author_url": "", "post_date": "02/27/2018 10:04:51", "content": "Hey Yan!\nI was trying to get the original matterport code to work, but failed.
\n\nData loading is fine, but when I try to train the model, it doesn't seem to do anything.\n(the corresponding notebook cell has the asteriks and it only prints the first epoch and stays like that for hours)
\n\nHave you come across a situation like this?
", "votes": null, "replies": [ { "id": 288484, "author_name": "wayne999", "author_url": "", "post_date": "02/27/2018 16:01:53", "content": "I did not meet this problem
", "votes": null, "replies": [] }, { "id": 288939, "author_name": "fabsta", "author_url": "", "post_date": "02/28/2018 15:32:22", "content": "I had the problem when using my own code. \nI have tested your code and the problem is gone. Thanks!
", "votes": null, "replies": [] }, { "id": 290122, "author_name": "", "author_url": "", "post_date": "03/03/2018 02:08:59", "content": "I had the same problem when I used the original matterport code. When I tried to run Yan's code, I am getting OOM error. Did you run into anything like that?
", "votes": null, "replies": [] }, { "id": 290124, "author_name": "pliptor", "author_url": "", "post_date": "03/03/2018 02:12:35", "content": "@ananthan31 OOM is (out of memory). Your GPU memory may not fit the code as is.
", "votes": null, "replies": [] }, { "id": 290132, "author_name": "", "author_url": "", "post_date": "03/03/2018 02:35:22", "content": "I figured out the error. The config file has GPU count as 4. When I changed it to 1 (since I have a single GPU), it worked. Thanks for your help!
", "votes": null, "replies": [] }, { "id": 290133, "author_name": "pliptor", "author_url": "", "post_date": "03/03/2018 02:37:26", "content": "@anantham31 Nice. Good luck and have fun.
", "votes": null, "replies": [] } ] }, { "id": 288467, "author_name": "mpware", "author_url": "", "post_date": "02/27/2018 15:06:18", "content": "Hi Yang, you've updated SGD optimizer to Adam optimizer. Does it really improve LB score?
", "votes": null, "replies": [ { "id": 288483, "author_name": "wayne999", "author_url": "", "post_date": "02/27/2018 16:01:21", "content": "It's faster
", "votes": null, "replies": [] }, { "id": 288521, "author_name": "bsp2020", "author_url": "", "post_date": "02/27/2018 17:39:00", "content": "\n\n\n", "votes": null, "replies": [] }, { "id": 288538, "author_name": "wayne999", "author_url": "", "post_date": "02/27/2018 18:07:30", "content": "“We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.”
\n
Good point.
", "votes": null, "replies": [] }, { "id": 288617, "author_name": "mpware", "author_url": "", "post_date": "02/27/2018 22:01:04", "content": "Thanks! For inference, you used same IMAGE_MIN_DIM/IMAGE_MAX_DIM as in training or 1024? I've tried different and LB is not better.
", "votes": null, "replies": [] }, { "id": 288717, "author_name": "john1231983", "author_url": "", "post_date": "02/28/2018 04:11:51", "content": "Same here. I ran this submission with python3 val_test.py but I got a low result of LB. It is about 0.33. Anyone has same score when ran this code? This is my final loss\n\n 300/300 [==============================] - 282s 940ms/step - loss: 0.2732 - rpn_class_loss: 0.0034 - rpn_bbox_loss: 0.0332 - mrcnn_class_loss: 0.0284 - mrcnn_bbox_loss: 0.0374 - mrcnn_mask_loss: 0.1708 - val_loss: 0.7583 - val_rpn_class_loss: 0.0035 - val_rpn_bbox_loss: 0.2083 - val_mrcnn_class_loss: 0.0190 - val_mrcnn_bbox_loss: 0.2052 - val_mrcnn_mask_loss: 0.3224\n
I ran this code also get around a low result of LB. About 0.31.
\n\n@john1231983 How did you solve this problem?
", "votes": null, "replies": [] }, { "id": 297621, "author_name": "john1231983", "author_url": "", "post_date": "03/17/2018 14:55:02", "content": "I used pytorch version. If you use Keras. I think you should you pretrain model with training schedule as coco. Train head and then train all
", "votes": null, "replies": [] }, { "id": 297813, "author_name": "wayne999", "author_url": "", "post_date": "03/18/2018 01:47:59", "content": "With this code I obtain 0.41+ frequently. @John, your previous scores on matterport code are really impressive, I think it's around 0.47?
", "votes": null, "replies": [] }, { "id": 297814, "author_name": "john1231983", "author_url": "", "post_date": "03/18/2018 01:55:23", "content": "Yes. But it used postprocesing like dilation. I trained both gray and color together in 1 network. For Heng's code, I trained on 2 network and result is same. But if I apply post processing for Heng code. The accuracy reduced :(
", "votes": null, "replies": [] }, { "id": 297987, "author_name": "panserbjorne", "author_url": "", "post_date": "03/18/2018 13:47:56", "content": "Any tips about postprocessing you are applying?
", "votes": null, "replies": [] }, { "id": 298133, "author_name": "spnrpa", "author_url": "", "post_date": "03/18/2018 22:32:32", "content": "Also using matterport code here! Heng is correct that mask rcnn only gets you so far (~4.7/4.8), light post processing got me to .49 (nms on flips, binary dilation)
", "votes": null, "replies": [] }, { "id": 304727, "author_name": "johnsondata", "author_url": "", "post_date": "03/28/2018 00:06:19", "content": "do you used any pre-processing to reach 0.47 (crop, flip or morphology）with matterport code? I only got around 0.4 without any pre-processing or post-processing.
", "votes": null, "replies": [] } ] }, { "id": 288941, "author_name": "fabsta", "author_url": "", "post_date": "02/28/2018 15:41:33", "content": "Hi!
\n\nIn val_test2.py, I came across this line:
\n\ndataset.add_nuclei('../../Spot-Nuclei-master/data/stage1_test/','test')\nIs that referring to the normal test data or something else?
\n\nCheers!
", "votes": null, "replies": [ { "id": 288944, "author_name": "wayne999", "author_url": "", "post_date": "02/28/2018 15:54:48", "content": "Yes, the normal test data
", "votes": null, "replies": [] }, { "id": 288963, "author_name": "john1231983", "author_url": "", "post_date": "02/28/2018 16:42:28", "content": "Hi Yan, your code has some file with the same name, for example, val_test1, val_test2... I know that train_val.py is for training, but which function is for making RLE to obtain 0.43 LB?
", "votes": null, "replies": [] }, { "id": 289021, "author_name": "wayne999", "author_url": "", "post_date": "02/28/2018 19:00:21", "content": "This is a basic code, which gives around 0.411. The val_test version does not affect too much.
", "votes": null, "replies": [] } ] }, { "id": 289989, "author_name": "panserbjorne", "author_url": "", "post_date": "03/02/2018 18:57:48", "content": "Hi Yan. Quick question. Clipping norm on the optimizers seems to improve performance. How did you guys decide on the clipnorm value of 5.0?
", "votes": null, "replies": [ { "id": 290021, "author_name": "wayne999", "author_url": "", "post_date": "03/02/2018 19:54:00", "content": "It's used in the original matterport code
", "votes": null, "replies": [] } ] }, { "id": 290171, "author_name": "jackkwok", "author_url": "", "post_date": "03/03/2018 06:38:49", "content": "Hi Yan, Thanks for sharing your code. I tried it but I ran into Out of Memory with my GPU:
\n\n\n\n\nResourceExhaustedError (see above for traceback): OOM when allocating\n tensor with shape[1024,256,14,14] and type float on\n /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
\n
I have a 6GB GPU. I tried to reduce memory use by modifying the config with:
\n\nGPU_COUNT = 1\nIMAGES_PER_GPU = 2\n...\nIMAGE_MIN_DIM = 256\nIMAGE_MAX_DIM = 256\nBut it still ran out of GPU memory. Do you have any suggestion?
", "votes": null, "replies": [ { "id": 290172, "author_name": "john1231983", "author_url": "", "post_date": "03/03/2018 06:40:17", "content": "Reduce the IMAGE_MIN_DIM to 128, USE_MINI_MASK=True, IMAGES_PER_GPU=1. Hope it help
The biggest memory consumption comes from saving all those weights at each epoch. What I do is manually delete saved weights from each prior epochs using os.remove().
\n\nFor example: os.remove('checkpoints/nuclei20180308T0559/mask_rcnn_nuclei_0077.h5').
\n\nYou will have to remove several epochs after they are no longer used. For example, if you run for 80 epochs, it is advised to delete the weights from 79 epochs, keep the 80th. It would be better if you could figure out how to use 'save_best_only' instead of 'save_weights_only'. I tried to set 'save_best_only' to True, erasing the latter, but the model did not work beyond the first epoch.
", "votes": null, "replies": [] }, { "id": 297690, "author_name": "fergusoci", "author_url": "", "post_date": "03/17/2018 18:56:59", "content": "You need to have 'save_best_only' = True and 'save_weights_only' = True (otherwise it will try to save the whole model which it can't do because of how the gt_boxes Lambda layer is structured).
", "votes": null, "replies": [] } ] }, { "id": 290352, "author_name": "", "author_url": "", "post_date": "03/03/2018 20:55:10", "content": "Hi Yan,
\n\nThanks for sharing your code. I see there are lot of visualize*.py files. I am running on a remote server. Which of these files can I run to visualize the results? Will this save the data to some folder since I am running on a headless server?
", "votes": null, "replies": [ { "id": 290360, "author_name": "wayne999", "author_url": "", "post_date": "03/03/2018 21:41:58", "content": "Results will be saved in plot_dir. The difference of all the visualize*.py files is subtle (either using different datasets like val or test, or using different configurations or postprocessing). Their codes are very similar, you'd better check them yourself.
Hi everyone,\nThanks for sharing :) I have a question. Is it possible to train on multiple CPU ? I tried something like this :
\n\nCPU_COUNT=8\nIMAGES_PER_CPU = 1\nSTEPS_PER_EPOCH = 600//(CPU_COUNT*IMAGES_PER_CPU)\nBut the result is worse than using 1CPU and 100 steps per epoch, for the same number of epochs.\nDoes it mean that not all images are read ? Should I keep 600 steps per epoch, or train longer ? Or should I adapt something in the code ? I would like to understand how it works in fact.
\n\nThank you very much for your help.
", "votes": null, "replies": [] }, { "id": 290831, "author_name": "lincurry", "author_url": "", "post_date": "03/05/2018 02:33:05", "content": "Thanks for sharing your code, which help me a lot！
", "votes": null, "replies": [] }, { "id": 298770, "author_name": "lincurry", "author_url": "", "post_date": "03/20/2018 02:16:12", "content": "UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
\n\nHi, I use your code and find these warning, and the train is extremly slow, here are some train log \n349/349 [==============================] - 9686s - loss: 0.8279 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.2299 - mrcnn_class_loss: 0.1493 - mrcnn_bbox_loss: 0.1719 - mrcnn_mask_loss: 0.2530 - val_loss: 0.8191 - val_rpn_class_loss: 0.0330 - val_rpn_bbox_loss: 0.1929 - val_mrcnn_class_loss: 0.2293 - val_mrcnn_bbox_loss: 0.1340 - val_mrcnn_mask_loss: 0.2298\nkeras：2.0.8\ntf：1.3.0\nGPU: K40\nhave you meet this problem? and how much time you spend in one epoch?
", "votes": null, "replies": [ { "id": 298785, "author_name": "wayne999", "author_url": "", "post_date": "03/20/2018 02:37:53", "content": "Commonly I can run in 3-6 min per epoch with 4 titan x. The speed is heavily affected by the configuration, which may last 3 min to 1 h per epoch.
", "votes": null, "replies": [] }, { "id": 298793, "author_name": "john1231983", "author_url": "", "post_date": "03/20/2018 02:49:17", "content": "Hi Yan, I have used Heng code and I found that matterport is more accurate than Heng code. The reason is that it did not support pretrain now (we have to change in the source code to support it). What is your score using Heng code?
", "votes": null, "replies": [] }, { "id": 298799, "author_name": "lincurry", "author_url": "", "post_date": "03/20/2018 02:57:51", "content": "so you get this warning too?
", "votes": null, "replies": [] }, { "id": 299300, "author_name": "bsp2020", "author_url": "", "post_date": "03/20/2018 19:09:29", "content": "Does anyone have any idea why the training time of Matterport Mask RCNN varies so much? For me, each epoch usually takes 15 minutes but after a while (usually after about 13/14 epochs) the training time goes up and takes 30 minutes or more. I noticed, when this happens, CPUs are not busy. It does not make sense to me that training time would change after running for a while, unless there are some bugs in the code. If I stop the training and rerun using 'last' model, training time goes back down to usual 15 minutes for a few epochs and increase again.
\n\n@Yan Wang \nDo you mean that certain configuration takes longer than others? or are you also observing that training time change while training a single model.
", "votes": null, "replies": [] }, { "id": 299321, "author_name": "spnrpa", "author_url": "", "post_date": "03/20/2018 19:41:31", "content": "Ya this concerned me too; but it’s related to the image size and number of masks (among potentially real performance bugs). I noticed that certain image modalities (e.g. The drosophila nuclei images) take ~ an order of mag longer to train.
\n\nDoes your network training remain permanently slow, or does it become fast again once it’s recycled the through the training images (assuming you haven’t played with the data_generator)?
", "votes": null, "replies": [] }, { "id": 299357, "author_name": "bsp2020", "author_url": "", "post_date": "03/20/2018 20:16:49", "content": "I could not wait long enough to see if it gets faster again. I usually kill it and start again from the last saved model. What's interesting is that the problem seem to show up when I increase the image augmentation (ex. rotate up to 45 degrees instead of up to 10 degrees)\nFor now, I'm just putting up with it since I was not able to get any other Mask RCNN implementation to produce good result.
", "votes": null, "replies": [] }, { "id": 299437, "author_name": "wayne999", "author_url": "", "post_date": "03/20/2018 21:44:05", "content": "@bsp2020 Sure the speed depends on the configuration. Also speed may change during training.
\n @John So which code gives you higher score so far? I am doing some testing recently, haven`t submitted full csv yet.
I used matterport that give me 0.49 lb now. The Heng code is 0.47. The reason is that te Heng code did not use pretrain model and frozen batch norm. That is what I guess. So I did not use Heng code . I use matterport again
", "votes": null, "replies": [] }, { "id": 299475, "author_name": "spnrpa", "author_url": "", "post_date": "03/20/2018 22:17:17", "content": "@bsp, caching masks in a numpy array helps quite a bit, and cuts down on the costly reading of masks from disk (I profiled and lots of time spent in disk IO, hence no CPU ticks, and also no GPU copy ticks).
\n\nIn my load_image/load_mask code I cache if an image has > 50 masks. Of course you will hit a few slow iterations while you read the images for the first time!
", "votes": null, "replies": [] }, { "id": 299483, "author_name": "wayne999", "author_url": "", "post_date": "03/20/2018 22:23:32", "content": "@John you mean pretrain on coco?
", "votes": null, "replies": [] }, { "id": 299484, "author_name": "wayne999", "author_url": "", "post_date": "03/20/2018 22:25:02", "content": "@Spencer very helpful
", "votes": null, "replies": [] }, { "id": 299500, "author_name": "john1231983", "author_url": "", "post_date": "03/20/2018 22:45:08", "content": "@yan imagenet or coco. The current code of Heng cannot use anh pretrain. I hope he can release next version with pretrain imagenet. To do it, i guess the featurenet function need to change to follow the rule of resnet in pytorchvision. Have you try with imagebet pretrain on Heng code?
", "votes": null, "replies": [] }, { "id": 299508, "author_name": "wayne999", "author_url": "", "post_date": "03/20/2018 22:52:15", "content": "It's not difficult to use the official pretrained weight in pytorch if using resnet as the feature net.
", "votes": null, "replies": [] }, { "id": 299547, "author_name": "john1231983", "author_url": "", "post_date": "03/20/2018 23:58:21", "content": "Hi yan. Have you ever try the pretrain for Heng code using resnet50? Does it improve performance. I have try but i got the error. If you was success. Could you share the modified featurenet function to use pretrain model?
", "votes": null, "replies": [] } ] }, { "id": 299775, "author_name": "spnrpa", "author_url": "", "post_date": "03/21/2018 03:45:17", "content": "to those using matterport, have you seen this issue: https://github.com/matterport/Mask_RCNN/issues/89
\n\nbasically the validation metrics are a lie :)
", "votes": null, "replies": [ { "id": 299836, "author_name": "dxchen", "author_url": "", "post_date": "03/21/2018 04:47:30", "content": "Thanks. How do you fix it? remove next?
", "votes": null, "replies": [] }, { "id": 299847, "author_name": "spnrpa", "author_url": "", "post_date": "03/21/2018 04:54:39", "content": "the fix is in the issue a little further down in jremillard‘s comment with the thumb’s up—it requires editing keras code unfortunately (but it works).
\n\nI’m considering creating a custom callback (further hacking the matterport train call to accept user def’d callbacks) that does validation.
", "votes": null, "replies": [] }, { "id": 299850, "author_name": "john1231983", "author_url": "", "post_date": "03/21/2018 04:56:33", "content": "Does it affect to performance. I am based on the matterport and the performance is not stable. I cannot recovert my best performance when i rerun :(
", "votes": null, "replies": [] }, { "id": 299858, "author_name": "spnrpa", "author_url": "", "post_date": "03/21/2018 05:05:14", "content": "my initial model was using val_rpn_bbox_loss to drive logic to update learning rate and model weights saving.
\n\nReproducibility has been an issue for me too; I’ve been studying the matterport code pretty closely these last few days and I’m trying to reproduce my best model.
\n\nSo far I’ve set the numpy seed and be tensorflow graph seed, and using only a single GPU. I’ll report on reproducibility as I the week goes on. :)
", "votes": null, "replies": [] }, { "id": 299861, "author_name": "dxchen", "author_url": "", "post_date": "03/21/2018 05:06:22", "content": "another proposal by uncommenting \"loss = K.reshape(loss, [1, 1])\" also does fix the error but does it have side effect?
", "votes": null, "replies": [] }, { "id": 300454, "author_name": "panserbjorne", "author_url": "", "post_date": "03/21/2018 16:05:34", "content": "Thanks for pointing that out. I was losing my mind trying to figure out why validation loss is not improving.
", "votes": null, "replies": [] }, { "id": 301125, "author_name": "theone4ever", "author_url": "", "post_date": "03/22/2018 10:52:21", "content": "Very interesting, thank you for sharing. Btw, I am also using matterport and get about 0.49 on LB
", "votes": null, "replies": [] }, { "id": 301232, "author_name": "wayne999", "author_url": "", "post_date": "03/22/2018 14:10:29", "content": "@Keven Wang, impressive, are you using post processing? What's the score without post-processing?
", "votes": null, "replies": [] }, { "id": 301350, "author_name": "theone4ever", "author_url": "", "post_date": "03/22/2018 16:48:18", "content": "@Yan Wang well, i have not applied any post processing except remove the overlapping between two mask instance. But my next step would be post processing, any suggestions?
", "votes": null, "replies": [] }, { "id": 301385, "author_name": "panserbjorne", "author_url": "", "post_date": "03/22/2018 17:23:09", "content": "Hey Keven. Did you get this LB score after fixing the validation generator issue? If so, did you train from scratch? Or just, train on top of your previous pretrained model?
", "votes": null, "replies": [] }, { "id": 301386, "author_name": "wayne999", "author_url": "", "post_date": "03/22/2018 17:23:30", "content": "I have not tried any post processing now.
", "votes": null, "replies": [] }, { "id": 301403, "author_name": "mpsampat", "author_url": "", "post_date": "03/22/2018 17:46:11", "content": "Very impressive Keven. could you tell us how many epochs and if you used default SGD optimizer ?
\nWith the config below, I am stuck as 0.415-0.418 .. \nI choose MEAN_PIXEL = np.array([0.,0.,0.]) because i think putting a specific value for MEAN_PIXEL will not generalize well.. Am I wrong about this ?
==========================================================================\nLEARNING_RATE = 1e-2
\n\n# If enabled, resizes instance masks to a smaller size to reduce\n# memory load. Recommended when using high-resolution image\nUSE_MINI_MASK = True\nMINI_MASK_SHAPE = (56, 56) # (height, width) of the mini-mask\n\n# Train on 1 GPU and 8 images per GPU. Batch size is GPUs * images/GPU.\nGPU_COUNT = 1\nIMAGES_PER_GPU = 2\n# Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch.\n# typically be equal to the number of samples of your dataset divided by the batch size\nSTEPS_PER_EPOCH = 300\nVALIDATION_STEPS = 70\n\n# Number of classes (including background)\nNUM_CLASSES = 1 + 1 # background + nucleis\nIMAGE_MIN_DIM = 512\nIMAGE_MAX_DIM = 512\nIMAGE_PADDING = True # currently, the False option is not supported\nRPN_ANCHOR_SCALES = (8, 16, 32, 64, 128) # anchor side in pixels, maybe add a 256?\n# The strides of each layer of the FPN Pyramid. These values\n# are based on a Resnet101 backbone.\nBACKBONE_STRIDES = [4, 8, 16, 32, 64]\n# How many anchors per image to use for RPN training\nRPN_TRAIN_ANCHORS_PER_IMAGE = 320 #300\n\n# ROIs kept after non-maximum supression (training and inference)\nPOST_NMS_ROIS_TRAINING = 2000\nPOST_NMS_ROIS_INFERENCE = 2000\nPOOL_SIZE = 7\nMASK_POOL_SIZE = 14\nMASK_SHAPE = [28, 28]\nTRAIN_ROIS_PER_IMAGE = 512\nRPN_NMS_THRESHOLD = 0.7\nMAX_GT_INSTANCES = 256\nDETECTION_MAX_INSTANCES = 400 \n# Minimum probability value to accept a detected instance\n# ROIs below this threshold are skipped\nDETECTION_MIN_CONFIDENCE = 0.7 # may be smaller?\n# Non-maximum suppression threshold for detection\nDETECTION_NMS_THRESHOLD = 0.3 # 0.3\n\n\nMEAN_PIXEL = np.array([0.,0.,0.])\n\n# Weight decay regularization\nWEIGHT_DECAY = 0.0001\nhere is a nice discussion on post-processing:\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51476
", "votes": null, "replies": [] }, { "id": 301590, "author_name": "chiaosun", "author_url": "", "post_date": "03/23/2018 01:12:37", "content": "@Mehul Sampat, I also set the MEAN_PIXEL to np.array([0.,0.,0.]) but when I use the coco pre-trained model I got loss:nan, any suggestions?
", "votes": null, "replies": [] }, { "id": 301965, "author_name": "mungos", "author_url": "", "post_date": "03/23/2018 14:22:52", "content": "@Mehul Sampat same thing stuck at 0.413-0.418 with pure maskrcnn. I don't know how john n kevin manage to get 0.49 without any post processing :o, thats amazing.
", "votes": null, "replies": [] }, { "id": 301979, "author_name": "theone4ever", "author_url": "", "post_date": "03/23/2018 14:39:05", "content": "hi all, i used data augmentation a lot, i think especially random cropping helps with overfitting. also, i used SGD instead of ADAM since there are some research paper shows that SGD usually can get better local optimal than adaptive optimiser like adam, also mask r-cnn, resnet paper all use SGD instead of adam.
\n\nAlso i achieve above result before val loss fix, i would say i am pretty luck to find the local optimal epoch but also i did find the result was not repeatable before val loss fix so i did many times, now the result is more predictable.
\n\nbtw, i have not changed MEAN_PIXEL so far, maybe i shall take a try.
\n\nlast but not least, i also tried to classify different image with different background/foreground into different class, that doesn't help much in term of performance, so i didn't do it any more (maybe this is another strange result thx to val loss bug, i probably should try again now)
", "votes": null, "replies": [] }, { "id": 302003, "author_name": "mpsampat", "author_url": "", "post_date": "03/23/2018 15:12:49", "content": "Thanks Keven for the tips about data augmentation! I will try it out soon. \nAlso MEAN_PIXEL is interesting. I am thinking for better generalization, I dont want to pick a fixed value but use [0,0,0]. To match this option i thought to use this normalization for each images. \nnormalized_img = ( img - mean_of_img )/ stdev_of_img; in both train and testing; this ensure each image has zero-mean and unit standard deviation. Still need to compare if this better than no normalization or just dividing by 255.
", "votes": null, "replies": [] }, { "id": 302008, "author_name": "mpsampat", "author_url": "", "post_date": "03/23/2018 15:14:08", "content": "@IAMLegend. when I use image size 128-by-128. I had that issue. can you try 256-by-256 and see if the nan issue goes away ? I think in some other thread also another suggestion was given. i will try and see if i can find it.
", "votes": null, "replies": [] }, { "id": 302012, "author_name": "theone4ever", "author_url": "", "post_date": "03/23/2018 15:17:13", "content": "@Mehul yes, in general normalisation is good, so don't take my experience as ground truth:) It is just i didn't have time to try it yet, i will play with it too.
", "votes": null, "replies": [] }, { "id": 302014, "author_name": "john1231983", "author_url": "", "post_date": "03/23/2018 15:23:27", "content": "Hi Keven. How do you fix val loss? You uncomment the line loss = K.reshape(loss, [1, 1]), right? I have tried sgd with resnet50 pretrain but it provided lower performancem do you use clip 0.5 in sgd?
", "votes": null, "replies": [] }, { "id": 302018, "author_name": "theone4ever", "author_url": "", "post_date": "03/23/2018 15:28:05", "content": "Hi John, i took another approach suggested in GitHub, modified keras. It might be a pb when submit the model at stage 1 deadline, but we will see.
\n\nFor me, this approach seems ok, with the same configuration, i still get about 0.486 pretty close to highest score before change. yes, i used clip 0.5, the default configuration in code base.
\n\nTo me, the biggest challenge is (1) high variance, because train loss is about 8% better than val loss. (2) maybe this is the same reason as (1), but i see when IoU threshold is 0.9 or 0.95, the mAP is always 0 or very close to 0 (like 0.000x), not sure if there is any parameter we can tune.
", "votes": null, "replies": [] }, { "id": 302022, "author_name": "john1231983", "author_url": "", "post_date": "03/23/2018 15:33:12", "content": "Are you use val for control early stop? Because i did not use it so i think it will not effect to my performance. I am using Adam and I got 0.494 but i cannot reproduce the result with same setting. I guess the reason is from Adam. As you said, you can reproduce the performance when using SGD. Am I right?
", "votes": null, "replies": [] }, { "id": 302284, "author_name": "theone4ever", "author_url": "", "post_date": "03/23/2018 21:29:11", "content": "No, i didn't use early stop, only use tensor board to identify the optimal epoch. regarding reproducibility, with same configuration, first time i got 0.492, second time i got 0.486, pretty close i would say.
", "votes": null, "replies": [] }, { "id": 302532, "author_name": "theone4ever", "author_url": "", "post_date": "03/24/2018 08:51:58", "content": "@john1231983, did you see improvement with resnet50 over resnet101? And have you figured out pre-trained weight for resnet50?
", "votes": null, "replies": [] }, { "id": 302534, "author_name": "john1231983", "author_url": "", "post_date": "03/24/2018 08:57:53", "content": "@keven: i don't think resnet101 is good choice. I am still use resnet50 . Itried resnet101 but performance is similar. The pretrained weight of resnet50 is from keras. I used SGD but performance is 0.44. I switched to adam and check it again. I remember only adam can give me 0.49LB. For sgd, i used learning rate 0.001 for 60 epochs for training heads, 0.0001 for 40 epoch training heads, then 100epochs for training all with learning rate 0.0001. Performance shows 0.44lb is too bad. Any tips for training with sgd?
", "votes": null, "replies": [] }, { "id": 302542, "author_name": "theone4ever", "author_url": "", "post_date": "03/24/2018 09:14:56", "content": "Oh, I just realised by default matterport downloads resnet50 weight instead of 101, so I guess I just need to switch architecture to resnet101 right?
Regarding train with SGD, I tried both approaches, train all from beginning to end and train heads then train all, actually only train all gives slightly better result. But as I mentioned, I didn't use early stop, I go through the loss and val loss in tensor board and pick the most promising one. Actually, usually epoch <50 gives me better result, I guess above that the model starts to overfit. Again, it is result from resnet101.
Yes. You have change the resnet101 to resnet50 in model.py or in config if you are using newest version
", "votes": null, "replies": [] }, { "id": 302943, "author_name": "akilaw", "author_url": "", "post_date": "03/25/2018 03:10:28", "content": "Hi Keven What are the augmentations that you are using? I ave tried random flips, zoom and crop but still no luck with improving the score.
", "votes": null, "replies": [] }, { "id": 302985, "author_name": "mpsampat", "author_url": "", "post_date": "03/25/2018 06:25:16", "content": "@Mungos, @Keven, my score improved from 0.415 to 0.432 after normalization (no data augmentation)\nnormalized_img = ( img - mean_of_img )/ stdev_of_img
\n\nI look at val los in tensorboard and i have a dip (minima) at 12th epoch and then it keeps getting worse. \ni will try and lower learning rate as others suggest.
", "votes": null, "replies": [] }, { "id": 302986, "author_name": "mpsampat", "author_url": "", "post_date": "03/25/2018 06:27:28", "content": "
Hi Mehul, that is a good result. Can you tell me how to implement this normalization? Where should this function be placed? Thanks Zheng
", "votes": null, "replies": [] }, { "id": 303723, "author_name": "mpsampat", "author_url": "", "post_date": "03/26/2018 15:33:37", "content": "@Zheng, you can do it at the place you load your train and test images. I did this: \nmean = np.mean(img)\nstd = np.std(img)\nimg = (img-mean)/std\nAlso i found resnet50 give better result than resnet101 (which is default for matterport rcnn) . this can be selected by putting the following line in the config file: BACKBONE = \"resnet50\"
", "votes": null, "replies": [] }, { "id": 303764, "author_name": "mdlszhengli", "author_url": "", "post_date": "03/26/2018 16:25:55", "content": "@Mehul, that is so great. Thanks a lot. I will let you know the result after I finish. Have you ever tried training based on imagenet pretrained model? Sometimes I can’t figure out what’s the difference between the imagenet pretrained model and coco. It seems we just choose it by random. Is there some theories can explain why we should choose this model while others shouldn’t? Thanks Zheng
", "votes": null, "replies": [] }, { "id": 303777, "author_name": "bsp2020", "author_url": "", "post_date": "03/26/2018 16:46:04", "content": "mold_inputs in model.py already subtract MEAN_PIXEL (mean of the training data of the pre-trained weight) from the input. So, normalizing the input with it's per image mean/std seems counter productive. I'd be interested in hearing about your result.
", "votes": null, "replies": [] }, { "id": 303786, "author_name": "sheriytm", "author_url": "", "post_date": "03/26/2018 16:57:38", "content": "@bsp2020 is right. I did normalization by subtraction of the mean/std when using unet but when I tried it with matterport, it got the masks out of bounds. I then checked the code and saw it is done in config.py
", "votes": null, "replies": [] }, { "id": 304318, "author_name": "mdlszhengli", "author_url": "", "post_date": "03/27/2018 10:58:15", "content": "@bsp2020 your score is very impressive. Can you share some tips about data augmentation?
", "votes": null, "replies": [] }, { "id": 304585, "author_name": "bsp2020", "author_url": "", "post_date": "03/27/2018 19:15:48", "content": "I took Keras image augmentation class and modified it a bit to fit it into MatterPort Mask RCNN shapes demo code. So, basically, common image augmentation such as rotation, scaling, etc. Rotating +-5 degrees on top of flip & 90 degree rotation made a big difference for me.
\n\nI think I've hit the limit of my current approach. So, can't give out too much more information. :)
", "votes": null, "replies": [] }, { "id": 304852, "author_name": "mdlszhengli", "author_url": "", "post_date": "03/28/2018 05:45:50", "content": "Thanks for your reply. I found that rotation doesn't improve my performance. My score drops from 0.424 to 0.409. Have you ever encountered the same situation before?
", "votes": null, "replies": [] }, { "id": 304853, "author_name": "mdlszhengli", "author_url": "", "post_date": "03/28/2018 05:58:22", "content": "@bsp2020 My code is def data_augmentation(image,masks,rotation=360)\nrotate_times=random.randint(0,rotation/90)\nfor r in range(0, rotate_times):\nimage=np.rot90(image)\nmasks=np.rot90(masks)It hurt my performance. Can you give me some advises about rotation? Thanks Zheng
Hi Zheng, \nI think the issue might be with np.rot90 function. this is rotating by 90 degrees which is too much. \ncould you please try and rotate by smaller amount like +/- 15 degrees ? Also congrats on the score improvement to 0.440. are you using the Matterport mask rcnn code too ?
", "votes": null, "replies": [] }, { "id": 304873, "author_name": "mdlszhengli", "author_url": "", "post_date": "03/28/2018 06:37:42", "content": "Hi Mehul, thank you. Yes, I am using the Matterport mask rcnn code.
", "votes": null, "replies": [] }, { "id": 308801, "author_name": "mdlszhengli", "author_url": "", "post_date": "04/04/2018 06:06:39", "content": "@Mehul Hi,Mehul. I have tried to implement general normalization. It only got 0.389 without any post processing.Learning schedule is 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50. I can't reproduce your result. Can you give me some more details about general normalization?
", "votes": null, "replies": [] }, { "id": 308910, "author_name": "sheriytm", "author_url": "", "post_date": "04/04/2018 10:14:23", "content": "Hi @Zheng Li, were you able to get a better result with rotation of +_15 as suggested above?
\n\nAbout normalization, I also got a lower result, i.e. it worsened my LB score. I tried 3 different types of normalization and none of them helped.
", "votes": null, "replies": [] }, { "id": 308966, "author_name": "mdlszhengli", "author_url": "", "post_date": "04/04/2018 12:48:03", "content": "Hi, YaGana. I have tried to rotate +- 5 degrees, but no better result achieved.
", "votes": null, "replies": [] }, { "id": 309048, "author_name": "mpsampat", "author_url": "", "post_date": "04/04/2018 15:00:53", "content": "@ZhengLi, I also finally got to 0.440 ; I got it in 11 epochs (LR = 1e-3). train on all instead of header. i only flipped images for img-aug. This is with resnet101 and no scaling and Mean Pixel =[0 0 0]. I am trying now if scaling improves this score; Also just checking you are using the latest Matterport Mask RCNN; there are some new updates in it. \nTo split into train/validation dataset, I followed the procedure shown in this notebook. \nhttps://github.com/neptune-ml/open-solution-data-science-bowl-2018/blob/mask_rcnn_notebook/main.ipynb (not sure yet if this is the best strategy to split into train and validation yet). Let me know if you need more information on it.
", "votes": null, "replies": [] }, { "id": 309068, "author_name": "mpsampat", "author_url": "", "post_date": "04/04/2018 15:27:00", "content": "@ZhengLi and @YaGana, did you add the external H&E data to your training set. I just start doing this. \nhttps://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations/notebook .I think this might be more helpful than augmentation and normalization since we have very few color training images.
", "votes": null, "replies": [] }, { "id": 309093, "author_name": "bsp2020", "author_url": "", "post_date": "04/04/2018 16:23:43", "content": "@Mehul Sampat
\nHave looked into other cloud provider that has GPU? I'm trying GPUeater.com since they have AMD GPU instances. Their instances NVidia instances seem cheaper than GCP. Fast.ai teacher also mentioned Paperspace.com but I have not tried it.
If K80 works for you, you can get it for free using Google Colaboratory. :)
", "votes": null, "replies": [] }, { "id": 309366, "author_name": "mpsampat", "author_url": "", "post_date": "04/05/2018 06:44:48", "content": "@ZhengLi, \nyou mentioned \" 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50\"; where did you find the weights for coco and resnet50. i thought coco weights were for resnet101 only and for resnet50 we need to init with imagenet (please correct me if i am wrong).
\n\nalso did you check val_loss for earlier epochs ? sometime when i run 20 epochs my best validation result is not the last epoch but somewhere in middle like 13th or 14th epoch. the train_loss and other train variables are not much helpful. they maybe going down but maybe the model is just memorizing the small dataset (please correct me if i am wrong here also.). I use tensorboard to try and pick the best weights.
", "votes": null, "replies": [] }, { "id": 309405, "author_name": "mdlszhengli", "author_url": "", "post_date": "04/05/2018 08:43:56", "content": "Hi Mehul. I have tried your method. 12epochs all+LR(1e-3)+resnet101+fliplr&flipud+general normalization. It only got LB 0.414. still can't reproduce your result. :( \nDoes \"flipped images\" means fliplr&flipud? \nval_loss sometimes can't reflect the performance of your model. It depends on your testset.I chose the lowest point which val_loss is 0.22. It only got 0.230...I chose the last point and it got 0.440.\nCurrently I am using YanWang's code.https://github.com/YanWang2014/Mask_RCNN\nI have used h&e dataset but only got lower result....
", "votes": null, "replies": [] }, { "id": 309562, "author_name": "mpsampat", "author_url": "", "post_date": "04/05/2018 15:15:13", "content": "Hi Zheng, \ncan we disuss via skype ? My skype id is mpsampat. I also want to ask what changes helped you increase performance from 0.44 to 0.476; would you be able to give us some hints please :-) ?
", "votes": null, "replies": [] }, { "id": 309881, "author_name": "mdlszhengli", "author_url": "", "post_date": "04/06/2018 05:37:50", "content": "Hi Mehul. You can contact me by email. mdlszhengli@@gmail.com. Mosacis may help you to improve your result. https://www.kaggle.com/bonlime/train-test-image-mosaic/notebook
", "votes": null, "replies": [] } ] }, { "id": 301986, "author_name": "hasanemreari", "author_url": "", "post_date": "03/23/2018 14:45:42", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 303789, "author_name": "dskswu", "author_url": "", "post_date": "03/26/2018 17:01:48", "content": "This may be a silly question. Where do I change the architecture to resnet50 in the code?
", "votes": null, "replies": [ { "id": 303805, "author_name": "sheriytm", "author_url": "", "post_date": "03/26/2018 17:16:24", "content": "@William Green you change the line (BACKBONE = \"resnet101\") to resnet50 in the config.py file or add it if it does not exist.
", "votes": null, "replies": [] }, { "id": 303815, "author_name": "panserbjorne", "author_url": "", "post_date": "03/26/2018 17:30:50", "content": "architecture is set in resnet_graph function in model.py. Search for resnet101. It should be the line with block_count
", "votes": null, "replies": [] }, { "id": 303828, "author_name": "dskswu", "author_url": "", "post_date": "03/26/2018 17:38:47", "content": "@YanGana and Panserbjorne\nThank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "288225": "Hi all,\nThere are several implementations of mask rcnn available now, but they are not the same. Which one you use, and what score it gives you?\nWe use the matterport implementation and get LB 0.436.", "288261": "Hi Yang, Nice to see your result. I am also using matterport implementation (because the other implementation has the own license-that I think we can not use in this challenge). When you submit the result, do you combine all masks into one mask then encode RLE or encode individually. If you encode individually, could you tell me how can encode each mask individually?", "288268": "Hi John, masks are encoded individually. \nMy full code is available here [enter link description here][1]\n\n [1]: https://github.com/YanWang2014/Mask_RCNN", "288270": "Thanks. Do you separate the training set into 3 clusters and train each cluster individually? I used 3 clusters and this is loss for 1 cluster at Epoch 152 and crop size `128x128`\n\n```\n394s 394ms/step - loss: 0.7758 - rpn_class_loss: 0.0184 - rpn_bbox_loss: 0.2962 - mrcnn_class_loss: 0.0454 - mrcnn_bbox_loss: 0.1637 - mrcnn_mask_loss: 0.2522 - val_loss: 1.0752 - val_rpn_class_loss: 0.0406 - val_rpn_bbox_loss: 0.3560 - val_mrcnn_class_loss: 0.0823 - val_mrcnn_bbox_loss: 0.2309 - val_mrcnn_mask_loss: 0.3653\n```\nI trained with resnet-50 pretrain model and train `all`. I will report my performance when it completed. For now, I have to think the way to peform inference with arbitrary size (instead of zero padding). One person have convert fully connected to fully convolution and the LB increased to 0.46. Hope it can help you some idea to improve performance", "288331": "Hi All,\nI'm not expert at all of computer vision and this is the first approach. I'm using this implementation of mask rcnn\nhttps://github.com/multimodallearning/pytorch-mask-rcnn \nand at the moment I'm stuck at 0.360. I used a smaller mini mask (16,16) or (32,32). I will try with the standard values (56,56)", "288360": "Hey Yan!\nI was trying to get the original matterport code to work, but failed.\n\nData loading is fine, but when I try to train the model, it doesn't seem to do anything.\n(the corresponding notebook cell has the asteriks and it only prints the first epoch and stays like that for hours)\n\nHave you come across a situation like this?", "288387": "How much using 3 clusters improved compared to not using them? How do you use models trained on clusters for test, given the modalities are not the same as in train?", "288467": "Hi Yang, you've updated SGD optimizer to Adam optimizer. Does it really improve LB score?", "288483": "It's faster", "288484": "I did not meet this problem", "288521": "https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/\n\n> “We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.”", "288538": "Good point.", "288617": "Thanks! For inference, you used same IMAGE_MIN_DIM/IMAGE_MAX_DIM as in training or 1024? I've tried different and LB is not better.", "288656": "Hi Yan, how many epoch of training do you get to LB 4.3? What do you think the most important factor is to get the score?", "288695": "The score is obtained by one of my team members, with some modifications to the nn. As far as I know, the baseline (without any additional work, just tune the parameters) for matterport mask rcnn should be around 0.411.", "288717": "Same here. I ran this submission with `python3 val_test.py` but I got a low result of LB. It is about 0.33. Anyone has same score when ran this code? This is my final loss\n```\n 300/300 [==============================] - 282s 940ms/step - loss: 0.2732 - rpn_class_loss: 0.0034 - rpn_bbox_loss: 0.0332 - mrcnn_class_loss: 0.0284 - mrcnn_bbox_loss: 0.0374 - mrcnn_mask_loss: 0.1708 - val_loss: 0.7583 - val_rpn_class_loss: 0.0035 - val_rpn_bbox_loss: 0.2083 - val_mrcnn_class_loss: 0.0190 - val_mrcnn_bbox_loss: 0.2052 - val_mrcnn_mask_loss: 0.3224\n```", "288939": "I had the problem when using my own code. \nI have tested your code and the problem is gone. Thanks!", "288941": "Hi!\n\nIn val_test2.py, I came across this line:\n\n dataset.add_nuclei('../../Spot-Nuclei-master/data/stage1_test/','test')\n\nIs that referring to the normal test data or something else?\n\nCheers!", "288944": "Yes, the normal test data", "288963": "Hi Yan, your code has some file with the same name, for example, val_test1, val_test2... I know that train_val.py is for training, but which function is for making RLE to obtain 0.43 LB?", "289021": "This is a basic code, which gives around 0.411. The val_test version does not affect too much.", "289051": "Thanks for the link!\nThe only other PyTorch Mask RCNN implementation I am aware of is incomplete and inference only.", "289090": "Hi Geoff, your score is impressive. Which code base you are using?", "289289": "Hi Geoff, this pytorch version works very well. It is a translation of keras matterport. Anyhow I did not reach the results people wrote in the forum, and I'm stuck at 0.360. I've tried also image augmentation, but with mini_mask and rotations it fails. Did you get your score with any version of mask rcnn?", "289344": "I'm using my own algorithm, not Mask R-CNN.\n\nI may well switch over to Mask R-CNN as I have used pretty much all the tricks I know to achieve the score I have and I don't really know how to improve matters!\n\nThanks for pointing this implementation out as I think I will use it as a start.", "289346": "Your great score is encouraging for those of us who try their own stuff rather than parameterize someone else implementation!", "289912": "", "289989": "Hi Yan. Quick question. Clipping norm on the optimizers seems to improve performance. How did you guys decide on the clipnorm value of 5.0?", "290021": "It's used in the original matterport code", "290122": "I had the same problem when I used the original matterport code. When I tried to run Yan's code, I am getting OOM error. Did you run into anything like that?", "290124": "ananthan31 OOM is (out of memory). Your GPU memory may not fit the code as is.", "290132": "I figured out the error. The config file has GPU count as 4. When I changed it to 1 (since I have a single GPU), it worked. Thanks for your help!", "290133": "anantham31 Nice. Good luck and have fun.", "290171": "Hi Yan, Thanks for sharing your code. I tried it but I ran into Out of Memory with my GPU:\n\n> ResourceExhaustedError (see above for traceback): OOM when allocating\n> tensor with shape[1024,256,14,14] and type float on\n> /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n\nI have a 6GB GPU. I tried to reduce memory use by modifying the config with:\n\n GPU_COUNT = 1\n IMAGES_PER_GPU = 2\n ...\n IMAGE_MIN_DIM = 256\n IMAGE_MAX_DIM = 256\n\nBut it still ran out of GPU memory. Do you have any suggestion?", "290172": "Reduce the `IMAGE_MIN_DIM` to 128, `USE_MINI_MASK=True, IMAGES_PER_GPU=1`. Hope it help", "290352": "Hi Yan,\n\nThanks for sharing your code. I see there are lot of visualize*.py files. I am running on a remote server. Which of these files can I run to visualize the results? Will this save the data to some folder since I am running on a headless server?", "290360": "Results will be saved in `plot_dir`. The difference of all the visualize*.py files is subtle (either using different datasets like val or test, or using different configurations or postprocessing). Their codes are very similar, you'd better check them yourself.", "290603": "Hi everyone,\nThanks for sharing :) I have a question. Is it possible to train on multiple CPU ? I tried something like this :\n\n CPU_COUNT=8\n IMAGES_PER_CPU = 1\n STEPS_PER_EPOCH = 600//(CPU_COUNT*IMAGES_PER_CPU)\n\nBut the result is worse than using 1CPU and 100 steps per epoch, for the same number of epochs.\nDoes it mean that not all images are read ? Should I keep 600 steps per epoch, or train longer ? Or should I adapt something in the code ? I would like to understand how it works in fact.\n\nThank you very much for your help.", "290831": "Thanks for sharing your code, which help me a lot！", "290977": "Ditto @CPMP and @Geoff French. I am still stretching my own algorithm and hoping to get round to Mask RCNN at some point. Both of your results are encouraging. Thanks.", "293128": "The biggest memory consumption comes from saving all those weights at each epoch. What I do is manually delete saved weights from each prior epochs using os.remove(). \n\nFor example: os.remove('checkpoints/nuclei20180308T0559/mask_rcnn_nuclei_0077.h5'). \n\nYou will have to remove several epochs after they are no longer used. For example, if you run for 80 epochs, it is advised to delete the weights from 79 epochs, keep the 80th. It would be better if you could figure out how to use 'save_best_only' instead of 'save_weights_only'. I tried to set 'save_best_only' to True, erasing the latter, but the model did not work beyond the first epoch.", "297620": "I ran this code also get around a low result of LB. About 0.31.\n\n\n@john1231983 How did you solve this problem?", "297621": "I used pytorch version. If you use Keras. I think you should you pretrain model with training schedule as coco. Train head and then train all", "297690": "You need to have 'save_best_only' = True and 'save_weights_only' = True (otherwise it will try to save the whole model which it can't do because of how the gt_boxes Lambda layer is structured).", "297813": "With this code I obtain 0.41+ frequently. @John, your previous scores on matterport code are really impressive, I think it's around 0.47?", "297814": "Yes. But it used postprocesing like dilation. I trained both gray and color together in 1 network. For Heng's code, I trained on 2 network and result is same. But if I apply post processing for Heng code. The accuracy reduced :(", "297987": "Any tips about postprocessing you are applying?", "298133": "Also using matterport code here! Heng is correct that mask rcnn only gets you so far (~4.7/4.8), light post processing got me to .49 (nms on flips, binary dilation)", "298770": "*UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.*\n\nHi, I use your code and find these warning, and the train is extremly slow, here are some train log \n349/349 [==============================] - 9686s - loss: 0.8279 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.2299 - mrcnn_class_loss: 0.1493 - mrcnn_bbox_loss: 0.1719 - mrcnn_mask_loss: 0.2530 - val_loss: 0.8191 - val_rpn_class_loss: 0.0330 - val_rpn_bbox_loss: 0.1929 - val_mrcnn_class_loss: 0.2293 - val_mrcnn_bbox_loss: 0.1340 - val_mrcnn_mask_loss: 0.2298\nkeras：2.0.8\ntf：1.3.0\nGPU: K40\nhave you meet this problem? and how much time you spend in one epoch?", "298785": "Commonly I can run in 3-6 min per epoch with 4 titan x. The speed is heavily affected by the configuration, which may last 3 min to 1 h per epoch.", "298793": "Hi Yan, I have used Heng code and I found that matterport is more accurate than Heng code. The reason is that it did not support pretrain now (we have to change in the source code to support it). What is your score using Heng code?", "298799": "so you get this warning too?", "299300": "Does anyone have any idea why the training time of Matterport Mask RCNN varies so much? For me, each epoch usually takes 15 minutes but after a while (usually after about 13/14 epochs) the training time goes up and takes 30 minutes or more. I noticed, when this happens, CPUs are not busy. It does not make sense to me that training time would change after running for a while, unless there are some bugs in the code. If I stop the training and rerun using 'last' model, training time goes back down to usual 15 minutes for a few epochs and increase again.\n\n@Yan Wang \nDo you mean that certain configuration takes longer than others? or are you also observing that training time change while training a single model.", "299321": "Ya this concerned me too; but it’s related to the image size and number of masks (among potentially real performance bugs). I noticed that certain image modalities (e.g. The drosophila nuclei images) take ~ an order of mag longer to train.\n\nDoes your network training remain permanently slow, or does it become fast again once it’s recycled the through the training images (assuming you haven’t played with the data_generator)?", "299357": "I could not wait long enough to see if it gets faster again. I usually kill it and start again from the last saved model. What's interesting is that the problem seem to show up when I increase the image augmentation (ex. rotate up to 45 degrees instead of up to 10 degrees)\nFor now, I'm just putting up with it since I was not able to get any other Mask RCNN implementation to produce good result.", "299437": "bsp2020 Sure the speed depends on the configuration. Also speed may change during training. \n @John So which code gives you higher score so far? I am doing some testing recently, haven`t submitted full csv yet.", "299464": "I used matterport that give me 0.49 lb now. The Heng code is 0.47. The reason is that te Heng code did not use pretrain model and frozen batch norm. That is what I guess. So I did not use Heng code . I use matterport again", "299475": "bsp, caching masks in a numpy array helps quite a bit, and cuts down on the costly reading of masks from disk (I profiled and lots of time spent in disk IO, hence no CPU ticks, and also no GPU copy ticks). \n\nIn my load_image/load_mask code I cache if an image has > 50 masks. Of course you will hit a few slow iterations while you read the images for the first time!", "299483": "John you mean pretrain on coco?", "299484": "Spencer very helpful", "299500": "yan imagenet or coco. The current code of Heng cannot use anh pretrain. I hope he can release next version with pretrain imagenet. To do it, i guess the featurenet function need to change to follow the rule of resnet in pytorchvision. Have you try with imagebet pretrain on Heng code?", "299508": "It's not difficult to use the official pretrained weight in pytorch if using resnet as the feature net.", "299547": "Hi yan. Have you ever try the pretrain for Heng code using resnet50? Does it improve performance. I have try but i got the error. If you was success. Could you share the modified featurenet function to use pretrain model?", "299775": "to those using matterport, have you seen this issue: https://github.com/matterport/Mask_RCNN/issues/89\n\nbasically the validation metrics are a lie :)", "299836": "Thanks. How do you fix it? remove next?", "299847": "the fix is in the issue a little further down in jremillard‘s comment with the thumb’s up—it requires editing keras code unfortunately (but it works).\n\nI’m considering creating a custom callback (further hacking the matterport train call to accept user def’d callbacks) that does validation.", "299850": "Does it affect to performance. I am based on the matterport and the performance is not stable. I cannot recovert my best performance when i rerun :(", "299858": "my initial model was using val_rpn_bbox_loss to drive logic to update learning rate and model weights saving.\n\nReproducibility has been an issue for me too; I’ve been studying the matterport code pretty closely these last few days and I’m trying to reproduce my best model.\n\nSo far I’ve set the numpy seed and be tensorflow graph seed, and using only a single GPU. I’ll report on reproducibility as I the week goes on. :)", "299861": "another proposal by uncommenting \"loss = K.reshape(loss, [1, 1])\" also does fix the error but does it have side effect?", "300454": "Thanks for pointing that out. I was losing my mind trying to figure out why validation loss is not improving.", "301125": "Very interesting, thank you for sharing. Btw, I am also using matterport and get about 0.49 on LB", "301232": "Keven Wang, impressive, are you using post processing? What's the score without post-processing?", "301350": "Yan Wang well, i have not applied any post processing except remove the overlapping between two mask instance. But my next step would be post processing, any suggestions?", "301385": "Hey Keven. Did you get this LB score after fixing the validation generator issue? If so, did you train from scratch? Or just, train on top of your previous pretrained model?", "301386": "I have not tried any post processing now.", "301403": "Very impressive Keven. could you tell us how many epochs and if you used default SGD optimizer ? \nWith the config below, I am stuck as 0.415-0.418 .. \nI choose MEAN_PIXEL = np.array([0.,0.,0.]) because i think putting a specific value for MEAN_PIXEL will not generalize well.. Am I wrong about this ?\n\n==========================================================================\nLEARNING_RATE = 1e-2\n \n # If enabled, resizes instance masks to a smaller size to reduce\n # memory load. Recommended when using high-resolution image\n USE_MINI_MASK = True\n MINI_MASK_SHAPE = (56, 56) # (height, width) of the mini-mask\n \n # Train on 1 GPU and 8 images per GPU. Batch size is GPUs * images/GPU.\n GPU_COUNT = 1\n IMAGES_PER_GPU = 2\n # Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch.\n # typically be equal to the number of samples of your dataset divided by the batch size\n STEPS_PER_EPOCH = 300\n VALIDATION_STEPS = 70\n\n # Number of classes (including background)\n NUM_CLASSES = 1 + 1 # background + nucleis\n IMAGE_MIN_DIM = 512\n IMAGE_MAX_DIM = 512\n IMAGE_PADDING = True # currently, the False option is not supported\n RPN_ANCHOR_SCALES = (8, 16, 32, 64, 128) # anchor side in pixels, maybe add a 256?\n # The strides of each layer of the FPN Pyramid. These values\n # are based on a Resnet101 backbone.\n BACKBONE_STRIDES = [4, 8, 16, 32, 64]\n # How many anchors per image to use for RPN training\n RPN_TRAIN_ANCHORS_PER_IMAGE = 320 #300\n \n # ROIs kept after non-maximum supression (training and inference)\n POST_NMS_ROIS_TRAINING = 2000\n POST_NMS_ROIS_INFERENCE = 2000\n POOL_SIZE = 7\n MASK_POOL_SIZE = 14\n MASK_SHAPE = [28, 28]\n TRAIN_ROIS_PER_IMAGE = 512\n RPN_NMS_THRESHOLD = 0.7\n MAX_GT_INSTANCES = 256\n DETECTION_MAX_INSTANCES = 400 \n # Minimum probability value to accept a detected instance\n # ROIs below this threshold are skipped\n DETECTION_MIN_CONFIDENCE = 0.7 # may be smaller?\n # Non-maximum suppression threshold for detection\n DETECTION_NMS_THRESHOLD = 0.3 # 0.3\n \n \n MEAN_PIXEL = np.array([0.,0.,0.])\n \n # Weight decay regularization\n WEIGHT_DECAY = 0.0001", "301411": "here is a nice discussion on post-processing:\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51476", "301590": "Mehul Sampat, I also set the MEAN_PIXEL to np.array([0.,0.,0.]) but when I use the coco pre-trained model I got loss:nan, any suggestions?", "301965": "Mehul Sampat same thing stuck at 0.413-0.418 with pure maskrcnn. I don't know how john n kevin manage to get 0.49 without any post processing :o, thats amazing.", "301979": "hi all, i used data augmentation a lot, i think especially random cropping helps with overfitting. also, i used SGD instead of ADAM since there are some research paper shows that SGD usually can get better local optimal than adaptive optimiser like adam, also mask r-cnn, resnet paper all use SGD instead of adam. \n\nAlso i achieve above result before val loss fix, i would say i am pretty luck to find the local optimal epoch but also i did find the result was not repeatable before val loss fix so i did many times, now the result is more predictable. \n\nbtw, i have not changed MEAN_PIXEL so far, maybe i shall take a try.\n\nlast but not least, i also tried to classify different image with different background/foreground into different class, that doesn't help much in term of performance, so i didn't do it any more (maybe this is another strange result thx to val loss bug, i probably should try again now)", "301986": "Thanks", "302003": "Thanks Keven for the tips about data augmentation! I will try it out soon. \nAlso MEAN_PIXEL is interesting. I am thinking for better generalization, I dont want to pick a fixed value but use [0,0,0]. To match this option i thought to use this normalization for each images. \nnormalized_img = ( img - mean_of_img )/ stdev_of_img; in both train and testing; this ensure each image has zero-mean and unit standard deviation. Still need to compare if this better than no normalization or just dividing by 255.", "302008": "IAMLegend. when I use image size 128-by-128. I had that issue. can you try 256-by-256 and see if the nan issue goes away ? I think in some other thread also another suggestion was given. i will try and see if i can find it.", "302012": "Mehul yes, in general normalisation is good, so don't take my experience as ground truth:) It is just i didn't have time to try it yet, i will play with it too.", "302014": "Hi Keven. How do you fix val loss? You uncomment the line loss = K.reshape(loss, [1, 1]), right? I have tried sgd with resnet50 pretrain but it provided lower performancem do you use clip 0.5 in sgd?", "302018": "Hi John, i took another approach suggested in GitHub, modified keras. It might be a pb when submit the model at stage 1 deadline, but we will see.\n\nFor me, this approach seems ok, with the same configuration, i still get about 0.486 pretty close to highest score before change. yes, i used clip 0.5, the default configuration in code base.\n\nTo me, the biggest challenge is (1) high variance, because train loss is about 8% better than val loss. (2) maybe this is the same reason as (1), but i see when IoU threshold is 0.9 or 0.95, the mAP is always 0 or very close to 0 (like 0.000x), not sure if there is any parameter we can tune.", "302022": "Are you use val for control early stop? Because i did not use it so i think it will not effect to my performance. I am using Adam and I got 0.494 but i cannot reproduce the result with same setting. I guess the reason is from Adam. As you said, you can reproduce the performance when using SGD. Am I right?", "302284": "No, i didn't use early stop, only use tensor board to identify the optimal epoch. regarding reproducibility, with same configuration, first time i got 0.492, second time i got 0.486, pretty close i would say.", "302532": "john1231983, did you see improvement with resnet50 over resnet101? And have you figured out pre-trained weight for resnet50?", "302534": "keven: i don't think resnet101 is good choice. I am still use resnet50 . Itried resnet101 but performance is similar. The pretrained weight of resnet50 is from keras. I used SGD but performance is 0.44. I switched to adam and check it again. I remember only adam can give me 0.49LB. For sgd, i used learning rate 0.001 for 60 epochs for training heads, 0.0001 for 40 epoch training heads, then 100epochs for training all with learning rate 0.0001. Performance shows 0.44lb is too bad. Any tips for training with sgd?", "302542": "Oh, I just realised by default matterport downloads resnet50 weight instead of 101, so I guess I just need to switch architecture to `resnet101` right?\n\nRegarding train with SGD, I tried both approaches, train `all` from beginning to end and train `heads` then train `all`, actually only train `all` gives slightly better result. But as I mentioned, I didn't use early stop, I go through the loss and val loss in tensor board and pick the most promising one. Actually, usually epoch <50 gives me better result, I guess above that the model starts to overfit. Again, it is result from resnet101.", "302564": "Yes. You have change the resnet101 to resnet50 in model.py or in config if you are using newest version", "302943": "Hi Keven What are the augmentations that you are using? I ave tried random flips, zoom and crop but still no luck with improving the score.", "302985": "Mungos, @Keven, my score improved from 0.415 to 0.432 after normalization (no data augmentation)\nnormalized_img = ( img - mean_of_img )/ stdev_of_img\n\nI look at val los in tensorboard and i have a dip (minima) at 12th epoch and then it keeps getting worse. \ni will try and lower learning rate as others suggest.", "302986": "![dip in val_loss around 11/12th epoch][1]\n\n\n [1]: http://file:///C:/Users/mehul.sampat/Desktop/validation_loss_dip.png", "303628": "Hi Mehul, that is a good result. Can you tell me how to implement this normalization? Where should this function be placed? Thanks Zheng", "303723": "Zheng, you can do it at the place you load your train and test images. I did this: \nmean = np.mean(img)\nstd = np.std(img)\nimg = (img-mean)/std\nAlso i found resnet50 give better result than resnet101 (which is default for matterport rcnn) . this can be selected by putting the following line in the config file: BACKBONE = \"resnet50\"", "303764": "Mehul, that is so great. Thanks a lot. I will let you know the result after I finish. Have you ever tried training based on imagenet pretrained model? Sometimes I can’t figure out what’s the difference between the imagenet pretrained model and coco. It seems we just choose it by random. Is there some theories can explain why we should choose this model while others shouldn’t? Thanks Zheng", "303777": "mold_inputs in model.py already subtract MEAN_PIXEL (mean of the training data of the pre-trained weight) from the input. So, normalizing the input with it's per image mean/std seems counter productive. I'd be interested in hearing about your result.", "303786": "bsp2020 is right. I did normalization by subtraction of the mean/std when using unet but when I tried it with matterport, it got the masks out of bounds. I then checked the code and saw it is done in config.py", "303789": "This may be a silly question. Where do I change the architecture to resnet50 in the code?", "303805": "William Green you change the line (BACKBONE = \"resnet101\") to resnet50 in the config.py file or add it if it does not exist.", "303815": "architecture is set in resnet_graph function in model.py. Search for resnet101. It should be the line with block_count", "303828": "YanGana and Panserbjorne\nThank you.", "304318": "bsp2020 your score is very impressive. Can you share some tips about data augmentation?", "304585": "I took Keras image augmentation class and modified it a bit to fit it into MatterPort Mask RCNN shapes demo code. So, basically, common image augmentation such as rotation, scaling, etc. Rotating +-5 degrees on top of flip & 90 degree rotation made a big difference for me.\n\nI think I've hit the limit of my current approach. So, can't give out too much more information. :)", "304727": "do you used any pre-processing to reach 0.47 (crop, flip or morphology）with matterport code? I only got around 0.4 without any pre-processing or post-processing.", "304852": "Thanks for your reply. I found that rotation doesn't improve my performance. My score drops from 0.424 to 0.409. Have you ever encountered the same situation before?", "304853": "bsp2020 My code is `def data_augmentation(image,masks,rotation=360)\nrotate_times=random.randint(0,rotation/90)\nfor r in range(0, rotate_times):\nimage=np.rot90(image)\nmasks=np.rot90(masks)`It hurt my performance. Can you give me some advises about rotation? Thanks Zheng", "304856": "Hi Zheng, \nI think the issue might be with np.rot90 function. this is rotating by 90 degrees which is too much. \ncould you please try and rotate by smaller amount like +/- 15 degrees ? Also congrats on the score improvement to 0.440. are you using the Matterport mask rcnn code too ?", "304873": "Hi Mehul, thank you. Yes, I am using the Matterport mask rcnn code.", "308801": "Mehul Hi,Mehul. I have tried to implement general normalization. It only got 0.389 without any post processing.Learning schedule is 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50. I can't reproduce your result. Can you give me some more details about general normalization?", "308910": "Hi @Zheng Li, were you able to get a better result with rotation of +_15 as suggested above? \n\nAbout normalization, I also got a lower result, i.e. it worsened my LB score. I tried 3 different types of normalization and none of them helped.", "308966": "Hi, YaGana. I have tried to rotate +- 5 degrees, but no better result achieved.", "309048": "I just finished my experiment with rotation (-10, 10) and (-15 to 15). I use imgaug. \n - I also get worse results with rotation.\n - For normalization, xiapidan who is in top-5 is also using img = img-mean(img)/std(img). \n - The main reason is in model.py, there is a line which does subtract MEAN_PIXEL. \n - But we know the original MEAN_PIXEL is for Imagenet so intuitively, it seemed incorrect for cells.\n - I use MEAN_PIXEL = [0,0,0]; But with more experiments it looks like it does not matter too much if i do normalization or not. Will try and report back on it. I use 1 K-80 on GCP so the processing is slow :(", "309065": "ZhengLi, I also finally got to 0.440 ; I got it in 11 epochs (LR = 1e-3). train on all instead of header. i only flipped images for img-aug. This is with resnet101 and no scaling and Mean Pixel =[0 0 0]. I am trying now if scaling improves this score; Also just checking you are using the latest Matterport Mask RCNN; there are some new updates in it. \nTo split into train/validation dataset, I followed the procedure shown in this notebook. \nhttps://github.com/neptune-ml/open-solution-data-science-bowl-2018/blob/mask_rcnn_notebook/main.ipynb (not sure yet if this is the best strategy to split into train and validation yet). Let me know if you need more information on it.", "309068": "ZhengLi and @YaGana, did you add the external H&E data to your training set. I just start doing this. \nhttps://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations/notebook .I think this might be more helpful than augmentation and normalization since we have very few color training images.", "309093": "Mehul Sampat \nHave looked into other cloud provider that has GPU? I'm trying GPUeater.com since they have AMD GPU instances. Their instances NVidia instances seem cheaper than GCP. Fast.ai teacher also mentioned Paperspace.com but I have not tried it.\n\nIf K80 works for you, you can get it for free using Google Colaboratory. :)", "309366": "ZhengLi, \nyou mentioned \" 50all(LR)+25all(LR/10) LR=1e-4.coco pretrained model+resnet50\"; where did you find the weights for coco and resnet50. i thought coco weights were for resnet101 only and for resnet50 we need to init with imagenet (please correct me if i am wrong). \n\nalso did you check val_loss for earlier epochs ? sometime when i run 20 epochs my best validation result is not the last epoch but somewhere in middle like 13th or 14th epoch. the train_loss and other train variables are not much helpful. they maybe going down but maybe the model is just memorizing the small dataset (please correct me if i am wrong here also.). I use tensorboard to try and pick the best weights.", "309405": "Hi Mehul. I have tried your method. 12epochs all+LR(1e-3)+resnet101+fliplr&flipud+general normalization. It only got LB 0.414. still can't reproduce your result. :( \nDoes \"flipped images\" means fliplr&flipud? \nval_loss sometimes can't reflect the performance of your model. It depends on your testset.I chose the lowest point which val_loss is 0.22. It only got 0.230...I chose the last point and it got 0.440.\nCurrently I am using YanWang's code.https://github.com/YanWang2014/Mask_RCNN\nI have used h&e dataset but only got lower result....", "309562": "Hi Zheng, \ncan we disuss via skype ? My skype id is mpsampat. I also want to ask what changes helped you increase performance from 0.44 to 0.476; would you be able to give us some hints please :-) ?", "309881": "Hi Mehul. You can contact me by email. mdlszhengli@@gmail.com. Mosacis may help you to improve your result. https://www.kaggle.com/bonlime/train-test-image-mosaic/notebook" }, "source": "meta" }