Hi all!\nBelow I have posted config that gives chance to achieve 0.362 LB score only on 8 epochs on 1 1070 with training only 'heads' of the network ONLY. Using this config you can achieve approximately 0.4 on LB using more epochs.
\n\nAlso I used next processing - convert images to grayscale and fill holes in masks, then select 600 a 70 subsets by random seed 5964 and after prediction I process overlaps in a next way - I cut part from the smaller instance.
\n\nHope this will help.
\n\nfrom config import Config
\n\nclass BowlConfig(Config):\n NAME = \"nucleos\"\n GPU_COUNT = 1\n IMAGES_PER_GPU = 1\n NUM_CLASSES = 1 + 1 \n IMAGE_MIN_DIM = 256\n IMAGE_MAX_DIM = 512\n RPN_ANCHOR_SCALES = (4, 8, 16, 32, 64)
\n TRAIN_ROIS_PER_IMAGE = 500\n STEPS_PER_EPOCH = 600 // (IMAGES_PER_GPU * GPU_COUNT)\n VALIDATION_STEPS = 70 // (IMAGES_PER_GPU * GPU_COUNT)\n MEAN_PIXEL = [0, 0, 0]\n LEARNING_RATE = 0.01\n USE_MINI_MASK = True\n MAX_GT_INSTANCES = 500
Hi all!\nBelow I have posted config that gives chance to achieve 0.362 LB score only on 8 epochs on 1 1070 with training only 'heads' of the network ONLY. Using this config you can achieve approximately 0.4 on LB using more epochs.
\n\nAlso I used next processing - convert images to grayscale and fill holes in masks, then select 600 a 70 subsets by random seed 5964 and after prediction I process overlaps in a next way - I cut part from the smaller instance.
\n\nHope this will help.
\n\nfrom config import Config
\n\nclass BowlConfig(Config):\n NAME = \"nucleos\"\n GPU_COUNT = 1\n IMAGES_PER_GPU = 1\n NUM_CLASSES = 1 + 1 \n IMAGE_MIN_DIM = 256\n IMAGE_MAX_DIM = 512\n RPN_ANCHOR_SCALES = (4, 8, 16, 32, 64)
\n TRAIN_ROIS_PER_IMAGE = 500\n STEPS_PER_EPOCH = 600 // (IMAGES_PER_GPU * GPU_COUNT)\n VALIDATION_STEPS = 70 // (IMAGES_PER_GPU * GPU_COUNT)\n MEAN_PIXEL = [0, 0, 0]\n LEARNING_RATE = 0.01\n USE_MINI_MASK = True\n MAX_GT_INSTANCES = 500
Thanks for sharing, Konstantin. I'm still being tortured by nan's in the mrcnn_class_loss. mask and bbox related losses are fine. I thought it might have had something to do with there just being one class_id output, but I tried with multiple class_id values and I'm still having the same issue. Did you run into any nan problems with the class losses, and if so was there something besides the config that solved them?
", "rawMarkdown": "Thanks for sharing, Konstantin. I'm still being tortured by nan's in the mrcnn_class_loss. mask and bbox related losses are fine. I thought it might have had something to do with there just being one class_id output, but I tried with multiple class_id values and I'm still having the same issue. Did you run into any nan problems with the class losses, and if so was there something besides the config that solved them?", "votes": 3, "replies": [ { "id": 283756, "postDate": "2018-02-15T23:30:35.020Z", "content": "Hi, I encountered the same issue at first. Took me a while to figure it out, but I guess it had something to do with my class labels. If you add two classes (0: 'background', 1: 'nucleus'), and then fill an array with 1's of shape mask[-1] (read: the amount of mask instances for one image), it should work (at least it did in my case). Hope this helps!
", "rawMarkdown": "Hi, I encountered the same issue at first. Took me a while to figure it out, but I guess it had something to do with my class labels. If you add two classes (0: 'background', 1: 'nucleus'), and then fill an array with 1's of shape mask[-1] (read: the amount of mask instances for one image), it should work (at least it did in my case). Hope this helps! ", "votes": 1 }, { "id": 283757, "postDate": "2018-02-15T23:33:37.153Z", "content": "Thanks - I'm doing both of those, but it's still cropping up. I think I must have a deeper bug in it somewhere. Going back to basics and re-downloading / re-writing the nuclei adaptation from scratch. Hopefully I can find what's causing the issue.
", "rawMarkdown": "Thanks - I'm doing both of those, but it's still cropping up. I think I must have a deeper bug in it somewhere. Going back to basics and re-downloading / re-writing the nuclei adaptation from scratch. Hopefully I can find what's causing the issue.", "votes": 1 }, { "id": 283761, "postDate": "2018-02-15T23:44:45.953Z", "content": "Just a small check: when you display the top masks (as given in the train_shapes notebook), does it output all the masks in the image next to the original image along with the class as title?
", "rawMarkdown": "Just a small check: when you display the top masks (as given in the train_shapes notebook), does it output all the masks in the image next to the original image along with the class as title? ", "votes": 1 }, { "id": 283763, "postDate": "2018-02-15T23:54:09.197Z", "content": "I've actually just gone back to run a train_shapes notebook replica. The code that I had before was integrated with another pipeline. Working through it now - the check that you gave above works, so hopefully it trains ok and I'll have a working code base to compare to... Thanks for the input.
", "rawMarkdown": "I've actually just gone back to run a train_shapes notebook replica. The code that I had before was integrated with another pipeline. Working through it now - the check that you gave above works, so hopefully it trains ok and I'll have a working code base to compare to... Thanks for the input.", "votes": 1 }, { "id": 283906, "postDate": "2018-02-16T07:01:15.020Z", "content": "Hi fergusoci! I figured out that this issue happens when your class probability goes to 1, that cause situation when your croosentropy logariphm becomes NAN, so i modified code of loss and added Keras epsilon to all crossentropies and that helped.
", "rawMarkdown": "Hi fergusoci! I figured out that this issue happens when your class probability goes to 1, that cause situation when your croosentropy logariphm becomes NAN, so i modified code of loss and added Keras epsilon to all crossentropies and that helped.", "votes": 5 }, { "id": 283951, "postDate": "2018-02-16T09:33:17.833Z", "content": "Thanks, Konstantin... Still no joy for me with that. Clipped the probs with epsilon but still getting the same nan. No worries, I've resigned myself to the fact that I've done something stupid somewhere and need to start again from scratch. Going to have a go at that later today. Cheers.
", "rawMarkdown": "Thanks, Konstantin... Still no joy for me with that. Clipped the probs with epsilon but still getting the same nan. No worries, I've resigned myself to the fact that I've done something stupid somewhere and need to start again from scratch. Going to have a go at that later today. Cheers.", "votes": 1 }, { "id": 284016, "postDate": "2018-02-16T12:47:23.093Z", "content": "Some special-ness on my part. I had active_class_ids set to [0, 1] rather than [1, 1]....
", "rawMarkdown": "Some special-ness on my part. I had active_class_ids set to [0, 1] rather than [1, 1]....", "votes": 1 }, { "id": 286259, "postDate": "2018-02-21T22:18:35.933Z", "content": "Hi fergusoci, I am having the same problem as you did. Did you manage to find a solution to this?
\n\n@ Konstantin Maksimov, can you please explain a little more on how you solved the overflow NAN issue?\nis what you did same as gradient Clipping?
", "rawMarkdown": "Hi fergusoci, I am having the same problem as you did. Did you manage to find a solution to this?\n\n@ Konstantin Maksimov, can you please explain a little more on how you solved the overflow NAN issue?\nis what you did same as gradient Clipping? " }, { "id": 286449, "postDate": "2018-02-22T07:10:08.543Z", "content": "Hi Konstantin Maksimov, can you please explain a little more as to how you solved the NAN issue?\nis what you did similar to Gradient Clipping?
", "rawMarkdown": "Hi Konstantin Maksimov, can you please explain a little more as to how you solved the NAN issue?\nis what you did similar to Gradient Clipping?" }, { "id": 286456, "postDate": "2018-02-22T07:16:02.440Z", "content": "Hi! Cliipping works for gradients, not for loss, and in the model you have losses that use cross entropy, in my case I have 1 class and background and model learns quickly to classify, that leads to situation when one class has 1 or near 1 probability, another(background) has probability vvvverrry close 0 and when you take logarithm of the 0 you get NaN, see example how I deal with it below
\n\ndef rpn_class_loss_graph(rpn_match, rpn_class_logits):\n \"\"\"RPN anchor classifier loss.
\n\nrpn_match: [batch, anchors, 1]. Anchor match type. 1=positive,\n -1=negative, 0=neutral anchor.\nrpn_class_logits: [batch, anchors, 2]. RPN classifier logits for FG/BG.\n\"\"\"\n# Squeeze last dim to simplify\nrpn_match = tf.squeeze(rpn_match, -1)\n# Get anchor classes. Convert the -1/+1 match to 0/1 values.\nanchor_class = K.cast(K.equal(rpn_match, 1), tf.int32)\n# Positive and Negative anchors contribute to the loss,\n# but neutral anchors (match value = 0) don't.\nindices = tf.where(K.not_equal(rpn_match, 0))\n# Pick rows that contribute to the loss and filter out the rest.\nrpn_class_logits = tf.gather_nd(rpn_class_logits, indices) + K.epsilon()\nanchor_class = tf.gather_nd(anchor_class, indices)\n# Crossentropy loss\nloss = K.sparse_categorical_crossentropy(target=anchor_class,\n output=rpn_class_logits,\n from_logits=True)\nloss = K.switch(tf.size(loss) > 0, K.mean(loss), tf.constant(0.0))\nreturn loss\nHi Abishek. I had the active_class_ids variable inadvertently set to np.array([0, 1]) rather than np.array([1, 1]) and that was what was causing my issue. Another thing to check is to make sure that you are screening out all zero masks before you feed them into training (some may end up with all zeros depending on how you resize them, or what augmentations you are doing).
", "rawMarkdown": "Hi Abishek. I had the active_class_ids variable inadvertently set to np.array([0, 1]) rather than np.array([1, 1]) and that was what was causing my issue. Another thing to check is to make sure that you are screening out all zero masks before you feed them into training (some may end up with all zeros depending on how you resize them, or what augmentations you are doing)." }, { "id": 286639, "postDate": "2018-02-22T14:59:47.953Z", "content": "thanks for the immediate response. I had put 0, 1 in the classes, and added a check to make sure the mask is not zero. i was still getting the NAN issue. Also implemented the change suggested by @Konstantin Maksimov, still was getting the issue. Then in the maskRCNN link under the differences from original paper, there is
\n\n\n\n\nLearning Rate: The paper uses a learning rate of 0.02, but we found that to be too high, and often causes the weights to explode, especially when using a small batch size. It might be related to differences between how Caffe and TensorFlow compute gradients (sum vs mean across batches and GPUs). Or, maybe the official model uses gradient clipping to avoid this issue. We do use gradient clipping, but don't set it too aggressively. We found that smaller learning rates converge faster anyway so we go with that.
\n
so i reduced the learning rate and it started working better. in the 10 epochs that i ran i did not get the NAN issue. i also changed the optimiser to Adam. so i am seeing better results
", "rawMarkdown": "thanks for the immediate response. I had put 0, 1 in the classes, and added a check to make sure the mask is not zero. i was still getting the NAN issue. Also implemented the change suggested by @Konstantin Maksimov, still was getting the issue. Then in the [maskRCNN][1] link under the differences from original paper, there is \n\n> Learning Rate: The paper uses a learning rate of 0.02, but we found that to be too high, and often causes the weights to explode, especially when using a small batch size. It might be related to differences between how Caffe and TensorFlow compute gradients (sum vs mean across batches and GPUs). Or, maybe the official model uses gradient clipping to avoid this issue. We do use gradient clipping, but don't set it too aggressively. We found that smaller learning rates converge faster anyway so we go with that.\n\nso i reduced the learning rate and it started working better. in the 10 epochs that i ran i did not get the NAN issue. i also changed the optimiser to Adam. so i am seeing better results\n\n\n [1]: https://github.com/matterport/Mask_RCNN", "votes": 1 }, { "id": 289277, "postDate": "2018-03-01T09:24:32.273Z", "content": "Once there is a nan in your class loss you're going to have issues computing loss for the rest of your training. The nans are occurring when np.log(0) occurs in computing your rpn_bbox, which you can clip in your models.py file like so.
rpn_bbox[ix] = [\n (gt_center_y - a_center_y) / a_h,\n (gt_center_x - a_center_x) / a_w,\n np.log(np.clip(gt_h / a_h, K.epsilon(), None)),\n np.log(np.clip(gt_w / a_w, K.epsilon(), None)),\n]\nHope that helps
", "rawMarkdown": "Once there is a `nan` in your class loss you're going to have issues computing loss for the rest of your training. The `nan`s are occurring when `np.log(0)` occurs in computing your `rpn_bbox`, which you can clip in your `models.py` file like so.\n\n rpn_bbox[ix] = [\n (gt_center_y - a_center_y) / a_h,\n (gt_center_x - a_center_x) / a_w,\n np.log(np.clip(gt_h / a_h, K.epsilon(), None)),\n np.log(np.clip(gt_w / a_w, K.epsilon(), None)),\n ]\n\nHope that helps", "votes": 3 } ] }, { "id": 284280, "postDate": "2018-02-16T21:02:26.780Z", "content": "For anyone using MaskRCNN, were you able to use it with keras ModelCheckpoint? i.e. did MaskRCNN.keras_model.save() work for you?
\n\nGetting a problem with gt_boxes = KL.Lambda(lambda x: x / image_scale)(input_gt_boxes). Because the image_scale input is a tensor the keras.save() won't work.
\n\nLooks like the same problem as here: https://github.com/keras-team/keras/issues/8343
\n\nI tried packaging it into its own custom layer but then get a ('Not JSON Serializable:', tf.float32) error.
\n\nIf anyone happens to have come across the same problem and can share a way around it it would be much appreciated!
\n\nEDIT: Ignore - I'll just save the weights rather than the model.
", "rawMarkdown": "For anyone using MaskRCNN, were you able to use it with keras ModelCheckpoint? i.e. did MaskRCNN.keras_model.save() work for you? \n\nGetting a problem with gt_boxes = KL.Lambda(lambda x: x / image_scale)(input_gt_boxes). Because the image_scale input is a tensor the keras.save() won't work.\n\nLooks like the same problem as here: https://github.com/keras-team/keras/issues/8343\n\nI tried packaging it into its own custom layer but then get a ('Not JSON Serializable:', tf.float32) error.\n\nIf anyone happens to have come across the same problem and can share a way around it it would be much appreciated!\n\nEDIT: Ignore - I'll just save the weights rather than the model.", "votes": 1 }, { "id": 283956, "postDate": "2018-02-16T09:47:17.977Z", "content": "Hi Konstantin, what are the preprocessing steps you are doing?
", "rawMarkdown": "Hi Konstantin, what are the preprocessing steps you are doing?", "votes": 1, "replies": [ { "id": 283964, "postDate": "2018-02-16T10:16:23.250Z", "content": "Hi!
\n\n1 - convert images to grayscale\n2 - fill holes in some broken train masks
\n\nWhat I plan to add - move issues with data from official thread and try to train on different channels.
", "rawMarkdown": "Hi!\n\n1 - convert images to grayscale\n2 - fill holes in some broken train masks\n\nWhat I plan to add - move issues with data from official thread and try to train on different channels.", "votes": 3 } ] }, { "id": 288139, "postDate": "2018-02-26T19:28:28.143Z", "content": "Anyone who is using MaskRCNN and is willing to share some of the evolution of their training/val loss output? Would like to know what types of values are 'normal' for this dataset. Thanks!
", "rawMarkdown": "Anyone who is using MaskRCNN and is willing to share some of the evolution of their training/val loss output? Would like to know what types of values are 'normal' for this dataset. Thanks!", "votes": 2, "replies": [ { "id": 288333, "postDate": "2018-02-27T08:29:19.127Z", "content": "Some discussion here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/50610#latest-288331
", "rawMarkdown": "Some discussion here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/50610#latest-288331", "votes": 1 }, { "id": 291185, "postDate": "2018-03-05T18:54:13.203Z", "content": "this is one of my:\n664/664 [==============================] - 500s 753ms/step - loss: 1.0058 - rpn_class_loss: 0.0685 - rpn_bbox_loss: 0.3533 - mrcnn_class_loss: 0.1389 - mrcnn_bbox_loss: 0.1958 - mrcnn_mask_loss: 0.2493 - val_loss: 2.3675 - val_rpn_class_loss: 0.2059 - val_rpn_bbox_loss: 0.9577 - val_mrcnn_class_loss: 0.2721 - val_mrcnn_bbox_loss: 0.4767 - val_mrcnn_mask_loss: 0.4550
\n\nNow i am not sure which of these metrics are most important ? perhaps val_loss ? \nIn this one i used 2 epochs and got a LB score of 0.358.
", "rawMarkdown": "this is one of my:\n664/664 [==============================] - 500s 753ms/step - loss: 1.0058 - rpn_class_loss: 0.0685 - rpn_bbox_loss: 0.3533 - mrcnn_class_loss: 0.1389 - mrcnn_bbox_loss: 0.1958 - mrcnn_mask_loss: 0.2493 - val_loss: 2.3675 - val_rpn_class_loss: 0.2059 - val_rpn_bbox_loss: 0.9577 - val_mrcnn_class_loss: 0.2721 - val_mrcnn_bbox_loss: 0.4767 - val_mrcnn_mask_loss: 0.4550\n\nNow i am not sure which of these metrics are most important ? perhaps val_loss ? \nIn this one i used 2 epochs and got a LB score of 0.358. \n\n" }, { "id": 291228, "postDate": "2018-03-05T20:48:26.173Z", "rawMarkdown": "" }, { "id": 292025, "postDate": "2018-03-07T08:54:32.167Z", "content": "hey, Mehul, i use this mask rcnn too ,and i got LB score unnormally low , i think my encoding process has some issues , i mean from masks to submission . and my encoding codes is this :
\n\ndef rle_encoding(x):\n dots = np.where(x.T.flatten() == 1)[0]\n run_lengths = []\n prev = -2\n for b in dots:\n if (b>prev+1): run_lengths.extend((b + 1, 0))\n run_lengths[-1] += 1\n prev = b\n return run_lengths
\n\ndef prob_to_rles(x, cutoff=0.5):\n lab_img = label(x > cutoff)\n for i in range(1, lab_img.max() + 1):\n yield rle_encoding(lab_img == i)
\n\nis this right ? \ni would be more than very appreciate , if u could respond to me .thanks so much .
", "rawMarkdown": "hey, Mehul, i use this mask rcnn too ,and i got LB score unnormally low , i think my encoding process has some issues , i mean from masks to submission . and my encoding codes is this :\n\ndef rle_encoding(x):\n dots = np.where(x.T.flatten() == 1)[0]\n run_lengths = []\n prev = -2\n for b in dots:\n if (b>prev+1): run_lengths.extend((b + 1, 0))\n run_lengths[-1] += 1\n prev = b\n return run_lengths\n\ndef prob_to_rles(x, cutoff=0.5):\n lab_img = label(x > cutoff)\n for i in range(1, lab_img.max() + 1):\n yield rle_encoding(lab_img == i)\n\nis this right ? \ni would be more than very appreciate , if u could respond to me .thanks so much ." } ] }, { "id": 306178, "postDate": "2018-03-30T01:59:10.280Z", "content": "Hi Konstantin, I have a silly question about the config file.
\n\nI set GPU_COUNT = 1 and IMAGES_PER_GPU = 1 when I used one 1070 to train the model before.
\n\nIn my opinion, I have two 1080ti now so I can set GPU_COUNT to 2, 1080ti has more memory than 1070 so I can change IMAGES_PER_GPU more than 1.
\n\nNow, I modify GPU_COUNT = 2, IMAGES_PER_GPU = 2 but I always get OOM and CUDA error even I set IMAGES_PER_GPU = 1.
\n\nIt only works when I set GPU_COUNT = 1 and IMAGES_PER_GPU = 1, any suggestion? Thanks.
", "rawMarkdown": "Hi Konstantin, I have a silly question about the config file.\n\nI set GPU_COUNT = 1 and IMAGES_PER_GPU = 1 when I used one 1070 to train the model before.\n\nIn my opinion, I have two 1080ti now so I can set GPU_COUNT to 2, 1080ti has more memory than 1070 so I can change IMAGES_PER_GPU more than 1.\n\nNow, I modify GPU_COUNT = 2, IMAGES_PER_GPU = 2 but I always get OOM and CUDA error even I set IMAGES_PER_GPU = 1.\n\nIt only works when I set GPU_COUNT = 1 and IMAGES_PER_GPU = 1, any suggestion? Thanks.", "replies": [ { "id": 306693, "postDate": "2018-03-30T20:44:31.077Z", "content": "I have same config with one 1070 and other with 2 1080 tI, for me chang to 2 per image was ok, can you provide error tracebak for better understandig the type of problem?
", "rawMarkdown": "I have same config with one 1070 and other with 2 1080 tI, for me chang to 2 per image was ok, can you provide error tracebak for better understandig the type of problem?" }, { "id": 309051, "postDate": "2018-04-04T15:01:27.957Z", "content": "Sorry for the late reply. I solved the CUDA error but still have the OOM problem.
\n\nIt always OOM when batch_size > 1 and I always get ~1000s per epoch(600 samples per epoch).
\n\nI check other threads who has 2 1080tis, they get about ~5 mins per epoch, any suggestion?
", "rawMarkdown": "Sorry for the late reply. I solved the CUDA error but still have the OOM problem. \n\nIt always OOM when batch_size > 1 and I always get ~1000s per epoch(600 samples per epoch).\n\nI check other threads who has 2 1080tis, they get about ~5 mins per epoch, any suggestion?" }, { "id": 309347, "postDate": "2018-04-05T05:02:57.273Z", "content": "Maybe you have old CUDA and CuDNN ? Because I use tensorflow 1.6 with CUDA 9.0 and CuDNN 7 - it gave almost 2x speed up
", "rawMarkdown": "Maybe you have old CUDA and CuDNN ? Because I use tensorflow 1.6 with CUDA 9.0 and CuDNN 7 - it gave almost 2x speed up ", "votes": 1 } ] }, { "id": 290455, "postDate": "2018-03-04T05:15:12.550Z", "content": "I use a very similar configuration but my network is taking much longer to train than 8 epochs. I've trained for 40 epochs and still get a loss around 1.2. Any ideas on how to improve the convergence?
\n\nHere's a link to my code so maybe you can see where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py
", "rawMarkdown": "I use a very similar configuration but my network is taking much longer to train than 8 epochs. I've trained for 40 epochs and still get a loss around 1.2. Any ideas on how to improve the convergence?\n\nHere's a link to my code so maybe you can see where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py" }, { "id": 287819, "postDate": "2018-02-26T02:24:09.300Z", "content": "Hi Konstantin, Thank you for sharing your code. It provides an opportunity for beginners like me to jump into the code. Can you provide the reason for deciding to fill holes in the mask? Is it a very common pre-processing step?
", "rawMarkdown": "Hi Konstantin, Thank you for sharing your code. It provides an opportunity for beginners like me to jump into the code. Can you provide the reason for deciding to fill holes in the mask? Is it a very common pre-processing step?", "replies": [ { "id": 287876, "postDate": "2018-02-26T05:58:41.827Z", "content": "It is not common step, it is an error that was posted in official data quality discussion, see forum for more details.
", "rawMarkdown": "It is not common step, it is an error that was posted in official data quality discussion, see forum for more details.", "votes": 3 }, { "id": 287904, "postDate": "2018-02-26T07:19:15.070Z", "content": "Thank you for helping out and pointing me to the discussions!
", "rawMarkdown": "Thank you for helping out and pointing me to the discussions!" } ] }, { "id": 287377, "postDate": "2018-02-24T15:26:01.760Z", "content": "Thanks for the provided info. Isnt it possible to share your code? I stuck in the data preparation and feeding to network. How do you handle the missing bounding box?
", "rawMarkdown": "Thanks for the provided info. Isnt it possible to share your code? I stuck in the data preparation and feeding to network. How do you handle the missing bounding box?", "replies": [ { "id": 287423, "postDate": "2018-02-24T18:55:00.547Z", "content": "I scaled input images smaller than 512x512 to 512x512 before starting the training process. Then, the error message went away.
", "rawMarkdown": "I scaled input images smaller than 512x512 to 512x512 before starting the training process. Then, the error message went away.", "votes": 1 }, { "id": 287437, "postDate": "2018-02-24T19:36:13.867Z", "content": "Do you get your current LB score with Mask RCNN or something else? Because I stucked at 0.411, but I don't use any augmentations.
", "rawMarkdown": "Do you get your current LB score with Mask RCNN or something else? Because I stucked at 0.411, but I don't use any augmentations." }, { "id": 287439, "postDate": "2018-02-24T19:37:01.827Z", "content": "I plan to share code when I will move at least to top 20.
", "rawMarkdown": "I plan to share code when I will move at least to top 20." }, { "id": 287456, "postDate": "2018-02-24T21:13:16.287Z", "content": "I use Mask RCNN and augmentation. I got stuck at .42 when I was just flipping and rotating 90 degrees. The current score is with 5 degrees rotation on top of flipping/90 rotation.
", "rawMarkdown": "I use Mask RCNN and augmentation. I got stuck at .42 when I was just flipping and rotating 90 degrees. The current score is with 5 degrees rotation on top of flipping/90 rotation.", "votes": 1 }, { "id": 287459, "postDate": "2018-02-24T21:23:50.973Z", "content": "Thanks a lot! Your config will be also great to see)))
", "rawMarkdown": "Thanks a lot! Your config will be also great to see)))" }, { "id": 287617, "postDate": "2018-02-25T13:04:33.547Z", "content": "Did you use mini_mask also with augmentation and arbitrary rotation?
", "rawMarkdown": "Did you use mini_mask also with augmentation and arbitrary rotation? " }, { "id": 289284, "postDate": "2018-03-01T09:48:24.193Z", "content": "Elena, what is the idea behind mini_mask?
\n\nI see it being mentioned in several places, but don't understand it yet.
", "rawMarkdown": "Elena, what is the idea behind mini_mask? \n\nI see it being mentioned in several places, but don't understand it yet." }, { "id": 289287, "postDate": "2018-03-01T09:56:45.643Z", "content": "Hi, it helps for high resolution images, when you are looking for small object. it is well explained here in this notebook https://github.com/matterport/Mask_RCNN/blob/master/inspect_data.ipynb
", "rawMarkdown": "Hi, it helps for high resolution images, when you are looking for small object. it is well explained here in this notebook https://github.com/matterport/Mask_RCNN/blob/master/inspect_data.ipynb ", "votes": 2 }, { "id": 289288, "postDate": "2018-03-01T09:58:38.990Z", "content": "Thanks, Elena!
", "rawMarkdown": "Thanks, Elena!" } ] }, { "id": 286607, "postDate": "2018-02-22T13:24:30.727Z", "content": "Thanks for your solution. I have two questions:\n1. Why not seperate training data into 3 clusters and make 3 models for these clusters?\n2. Why not use imagenet pretrained model and train on \"all\" instead of coco and 'heads' as you did? I found that train \"all\" give better performance but I did not submit yet.\nDo you have any update LB?
", "rawMarkdown": "Thanks for your solution. I have two questions:\n1. Why not seperate training data into 3 clusters and make 3 models for these clusters?\n2. Why not use imagenet pretrained model and train on \"all\" instead of coco and 'heads' as you did? I found that train \"all\" give better performance but I did not submit yet.\nDo you have any update LB?", "replies": [ { "id": 286621, "postDate": "2018-02-22T13:59:07.987Z", "content": "1) Because we don't know how many types we will have in stage2, so we need good generalisation, npt tweaking \n2) Imagenet showed worth result for my config, training heads is common technique, it helps to reduce backprop error\nI am working on improvement my model, my local validation is broken so no any good movements from 0.426 for 1.5 weeks already
", "rawMarkdown": "1) Because we don't know how many types we will have in stage2, so we need good generalisation, npt tweaking \n2) Imagenet showed worth result for my config, training heads is common technique, it helps to reduce backprop error\nI am working on improvement my model, my local validation is broken so no any good movements from 0.426 for 1.5 weeks already", "votes": 1 }, { "id": 286688, "postDate": "2018-02-22T16:49:05.170Z", "content": "Hi Konstantin. Is there a particular reason you are converting the images into grayscale? Did you not get good results with color images?
", "rawMarkdown": "Hi Konstantin. Is there a particular reason you are converting the images into grayscale? Did you not get good results with color images?" }, { "id": 286771, "postDate": "2018-02-22T18:52:38.897Z", "content": "I see. Do you try random crop image into 128x128 for training? It can increase the training set size. Then replace fully connected to fully convolution for making mask rcnn working in arbitrary size, hence you can run testing with original size without crop. It may improve your LB. I guess the current maskrcnn implementation cannot work in arbitrary size
", "rawMarkdown": "I see. Do you try random crop image into 128x128 for training? It can increase the training set size. Then replace fully connected to fully convolution for making mask rcnn working in arbitrary size, hence you can run testing with original size without crop. It may improve your LB. I guess the current maskrcnn implementation cannot work in arbitrary size" }, { "id": 287016, "postDate": "2018-02-23T09:26:51.177Z", "content": "Hi @panserbjorne! The only reason is that it was first idea how to deal with different color schemes, actually no any analysis behind it.
", "rawMarkdown": "Hi @panserbjorne! The only reason is that it was first idea how to deal with different color schemes, actually no any analysis behind it.", "votes": 1 }, { "id": 287106, "postDate": "2018-02-23T15:11:29.837Z", "content": "Hi Konstantin,
\n\nYou mentioned you achieve 0.407 by training on your pretrained network. Can you share the method to achieve 0.426? Thanks!
", "rawMarkdown": "Hi Konstantin,\n\nYou mentioned you achieve 0.407 by training on your pretrained network. Can you share the method to achieve 0.426? Thanks!" }, { "id": 287216, "postDate": "2018-02-23T22:14:36.480Z", "content": "The problem is that I deleted accidentally 0.426 result((( Now my best submission is 0.413 actually. As soon as will crack at least 0.430 I will provide update how to get 0.413
", "rawMarkdown": "The problem is that I deleted accidentally 0.426 result((( Now my best submission is 0.413 actually. As soon as will crack at least 0.430 I will provide update how to get 0.413" }, { "id": 287315, "postDate": "2018-02-24T10:17:05.030Z", "content": "For the masks, did you use the default masks where each mask image contains one segmented nucleus, or did you use masks with all the segmented nuclei in single mask images with a 1:1 correspondence to the original training images?
\n\nI used the latter, with a single corresponding mask image with all segmented nuclei. However, ran into a problem after training for one epoch. The error: 'TypeError: can't pickle _thread.lock objects'
\n\nIt also said the following: \"UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.\"
\n\nAfter using code from inspect_data.ipynb, I noticed that there was only one bounding box cropping all the segmented nuclei when there should be a single bounding box for each instance or each nucleus, right?
", "rawMarkdown": "For the masks, did you use the default masks where each mask image contains one segmented nucleus, or did you use masks with all the segmented nuclei in single mask images with a 1:1 correspondence to the original training images? \n\nI used the latter, with a single corresponding mask image with all segmented nuclei. However, ran into a problem after training for one epoch. The error: 'TypeError: can't pickle _thread.lock objects' \n\nIt also said the following: \"UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.\" \n\nAfter using code from inspect_data.ipynb, I noticed that there was only one bounding box cropping all the segmented nuclei when there should be a single bounding box for each instance or each nucleus, right? " } ] }, { "id": 284479, "postDate": "2018-02-17T08:28:56.557Z", "content": "Hi Konstantin, have you think about modify the pre-trained weights to fit 1 channel grayscale image? I thought it will accelerate the training speed pretty much.
", "rawMarkdown": "Hi Konstantin, have you think about modify the pre-trained weights to fit 1 channel grayscale image? I thought it will accelerate the training speed pretty much.", "replies": [ { "id": 284531, "postDate": "2018-02-17T13:17:15.570Z", "content": "No, because my stratgy is to get new pre-train for grayscale and then try diffrent ideas on new pretrained.
", "rawMarkdown": "No, because my stratgy is to get new pre-train for grayscale and then try diffrent ideas on new pretrained." }, { "id": 284686, "postDate": "2018-02-17T23:27:57.333Z", "content": "Where do you find the new pretrain weights based on grayscale image?
", "rawMarkdown": "Where do you find the new pretrain weights based on grayscale image?" }, { "id": 284804, "postDate": "2018-02-18T12:11:36.137Z", "content": "I train for 50 epochs on coco using grayscle - and it in now my new pretrined network.
", "rawMarkdown": "I train for 50 epochs on coco using grayscle - and it in now my new pretrined network." }, { "id": 284823, "postDate": "2018-02-18T13:32:05.193Z", "content": "How much does it improve your performance?
", "rawMarkdown": "How much does it improve your performance?" }, { "id": 284911, "postDate": "2018-02-18T19:27:14.023Z", "content": "from 0.362 to 0.407 on LB
", "rawMarkdown": "from 0.362 to 0.407 on LB" }, { "id": 284995, "postDate": "2018-02-19T02:25:07.790Z", "content": "The performance seems to get worse after training 200 epochs. What kind of val_loss and other loss metrics do you get for 0.362?
", "rawMarkdown": "The performance seems to get worse after training 200 epochs. What kind of val_loss and other loss metrics do you get for 0.362?" }, { "id": 285056, "postDate": "2018-02-19T07:11:00.573Z", "content": "Hi! val_loss 0.7 mAP on val dataset around 0.48
", "rawMarkdown": "Hi! val_loss 0.7 mAP on val dataset around 0.48" }, { "id": 285104, "postDate": "2018-02-19T10:11:23.367Z", "rawMarkdown": "", "isDeleted": true }, { "id": 285376, "postDate": "2018-02-20T04:53:24.240Z", "content": "Hi Konstantin Maksimov,\nabout to ask a silly question:
\n\nHow did you subclass the nuclei dataset to run it through MatterPort's model?
\n\nMatterport instructs the user to create a derived class from the Dataset class in utils.py.\nSo, so far, I only got this:
\n\nclass NucleiDataset(utils.Dataset):\n def load_nuclei(self):\n pass\n def load_mask(self, image_id):\n pass\n def image_reference(self, image_id):\n pass\nI have the data prep code from this guy. \nCould you give me a hint as to how to incorporate this data-prep code into the derived class NucleiDataset?
", "rawMarkdown": "Hi Konstantin Maksimov,\nabout to ask a silly question: \n\nHow did you subclass the nuclei dataset to run it through MatterPort's model? \n\nMatterport instructs the user to create a derived class from the Dataset class in utils.py.\nSo, so far, I only got this: \n\n class NucleiDataset(utils.Dataset):\n def load_nuclei(self):\n pass\n def load_mask(self, image_id):\n pass\n def image_reference(self, image_id):\n pass\n\nI have the data prep code from this [guy][1]. \nCould you give me a hint as to how to incorporate this data-prep code into the derived class NucleiDataset? \n\n [1]: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277" }, { "id": 285416, "postDate": "2018-02-20T07:34:40.010Z", "content": "Here is my example (some code is not in use and still need refactoring
\n\nclass BowlDataset(utils.Dataset):\n def load_data(self, root_path, mode='train', filter_ids=None):
\n\n # Add classes\n\n self.add_class(\"nucleos\", 1, \"nucleo\")\n if mode == 'test':\n files = os.listdir(os.path.join(root_path, 'images_test'))\n else:\n files = os.listdir(os.path.join(root_path, 'images'))\n\n if filter_ids is not None:\n\n files = [item for item in files if item.split('.')[0] in filter_ids]\n # print(len(files))\n for i, ffile in enumerate(tqdm(files)):\n # pool = Pool(cpu_count() - 2)\n\n\n if mode == 'test':\n data_path = os.path.join(root_path, 'images_test', ffile)\n mask_path = 'No masks'\n else:\n data_path = os.path.join(root_path, 'images', ffile)\n\n\n bg_color = random.randint(0, 255)\n original_id = data_path.split('/')[-1].split('.')[-2]\n # TODO change later maybe\n\n data = cv2.imread(data_path)[:, :, 0]\n height, width = data.shape\n\n self.add_image(\"nucleos\", image_id=i, path=data_path,\n width=width, height=height,\n bg_color=bg_color, original_id=original_id,\n data_path=data_path,\n root_path=root_path)\n\n image = skimage.io.imread(data_path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n\n # self.data_structure[i] = {}\n # self.data_structure[i]['image'] = image[:]\n # self.data_structure[i]['mask'] = self.add_mask_data(original_id, root_path,\n # width=width, height=height)[:]\n\ndef load_image(self, image_id):\n info = self.get_info(image_id)\n path = info['data_path']\n image = skimage.io.imread(path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n return image\n\ndef load_mask(self, image_id):\n info = self.get_info(image_id)\n original_id = info['original_id']\n root_path = info['root_path']\n width = info['width']\n height = info['height']\n return self.add_mask_data(original_id, root_path, width=width, height=height)[:]\n\ndef get_info(self, image_id):\n return self.image_info[image_id]\n\ndef add_mask_data(self, original_id, root_path, width=256, height=256):\n # info = self.image_info[image_id]\n orifinal_id = original_id\n root_path = root_path\n all_masks = os.listdir(os.path.join(root_path, 'masks'))\n # print(original_id)\n # all_masks = [element\n # for element in all_masks if element.split('!')[0] == orifinal_id]\n all_masks = [element for element in all_masks if element.split('_')[0] == orifinal_id]\n\n\n num_labels = 0\n for i, mask_file in enumerate(all_masks):\n\n subpath = os.path.join(root_path, 'masks', mask_file)\n # print(subpath)\n data = skimage.io.imread(subpath)[:, :]\n # print(data.shape)\n # print(np.sum(data))\n\n if np.sum(data) != 0:\n # print('SHIT MASK DETECTED')\n\n num_labels += 1\n\n mask = np.zeros([height,\n width,\n num_labels],\n dtype=np.bool)\n num_labels = 0\n for i, mask_file in enumerate(all_masks):\n subpath = os.path.join(root_path, 'masks', mask_file)\n # print(subpath)\n # data = cv2.imread(subpath)[:, :, 0]\n data = skimage.io.imread(subpath)[:, :]\n # print(np.max(data))\n if np.sum(data) != 0:\n # print('SHIT MASK DETECTED')\n\n # height, width = data.shape\n\n # if height != info['height'] and width != info['width']:\n # data = cv2.resize(data, (info['height'], info['width']))\n # change translation\n # data = (data == 255)\n data = (data != 0)\n # print(np.sum(data))\n mask[:, :, num_labels] = data\n num_labels += 1\n # class_ids = [1]*num_labels\n class_ids = np.array([1] * num_labels)\n class_ids = class_ids.astype(np.int32)\n return mask, class_ids\nYou're awesome! Thank you!
", "rawMarkdown": "You're awesome! Thank you! " }, { "id": 285879, "postDate": "2018-02-21T06:42:05.303Z", "content": "Another silly question: the argument 'filter_ids', does that refer to the id's of the images such as '0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732' as one example?
\n\nI see these lines of code below
\n\nif filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids] \nand interpret the second line as catching this long id string just before the '.png'. \nIs this interpretation incorrect?
\n\nFurther, my file paths are the following: \"root_path/id/images/id.png\" to the images.\nThe path to the masks are the following: \"root_path/id/masks/id/mask_image.png\"\nIt looks like you had to change your file paths for this code to work.
", "rawMarkdown": "Another silly question: the argument 'filter_ids', does that refer to the id's of the images such as '0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732' as one example?\n\nI see these lines of code below \n \n if filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids] \n\nand interpret the second line as catching this long id string just before the '.png'. \nIs this interpretation incorrect? \n\nFurther, my file paths are the following: \"root_path/id/images/id.png\" to the images.\nThe path to the masks are the following: \"root_path/id/masks/id/mask_image.png\"\nIt looks like you had to change your file paths for this code to work. " }, { "id": 285915, "postDate": "2018-02-21T08:22:57.597Z", "content": "filter_ids is used for separation of val and train, and yes - my structure is different from defailt one
", "rawMarkdown": "filter_ids is used for separation of val and train, and yes - my structure is different from defailt one" }, { "id": 286304, "postDate": "2018-02-22T00:47:26.763Z", "content": "Hi Konstantin, I tried your solution and I trained a raw model on the grayscale coco dataset. I'm wondering how much loss do you get from this basemodel as my model has nearly 1.9 on val_loss.
", "rawMarkdown": "Hi Konstantin, I tried your solution and I trained a raw model on the grayscale coco dataset. I'm wondering how much loss do you get from this basemodel as my model has nearly 1.9 on val_loss." }, { "id": 286453, "postDate": "2018-02-22T07:11:41.513Z", "content": "don't train on coco, take coco pretrained model and then train for 8 epochs, that will give val loss around 0.7 - 1
", "rawMarkdown": "don't train on coco, take coco pretrained model and then train for 8 epochs, that will give val loss around 0.7 - 1" } ] }, { "id": 283562, "postDate": "2018-02-15T17:47:13.027Z", "content": "the images are (w,h,1) or (w,h,3)? if it is 1 do u need to change something in maskrcnn code?\nhow did u done a hole in the masks and how do u fill the hole after?
", "rawMarkdown": "the images are (w,h,1) or (w,h,3)? if it is 1 do u need to change something in maskrcnn code?\nhow did u done a hole in the masks and how do u fill the hole after?", "replies": [ { "id": 283617, "postDate": "2018-02-15T19:19:50.663Z", "content": "Hi! images are grayscale (w,h,3) with 3 equal channels, so no any changes to the code. Holes are present in some train masks. I used this for holes fill - https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.ndimage.morphology.binary_fill_holes.html#scipy-ndimage-morphology-binary-fill-holes
", "rawMarkdown": "Hi! images are grayscale (w,h,3) with 3 equal channels, so no any changes to the code. Holes are present in some train masks. I used this for holes fill - https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.ndimage.morphology.binary_fill_holes.html#scipy-ndimage-morphology-binary-fill-holes", "votes": 1 }, { "id": 283743, "postDate": "2018-02-15T22:42:19.770Z", "content": "did u fill holes to all images or u chosen them?
", "rawMarkdown": "did u fill holes to all images or u chosen them?", "votes": 1 }, { "id": 283908, "postDate": "2018-02-16T07:02:37.577Z", "content": "I filled all masks, if there is no holes - nothing happens.
", "rawMarkdown": "I filled all masks, if there is no holes - nothing happens.", "votes": 1 } ] } ], "comments": [ { "id": 283728, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-15T22:15:41.107000", "content": "Thanks for sharing, Konstantin. I'm still being tortured by nan's in the mrcnn_class_loss. mask and bbox related losses are fine. I thought it might have had something to do with there just being one class_id output, but I tried with multiple class_id values and I'm still having the same issue. Did you run into any nan problems with the class losses, and if so was there something besides the config that solved them?
", "votes": 3, "replies": [ { "id": 283756, "author_name": "rubenhx", "author_url": "", "post_date": "2018-02-15T23:30:35.020000", "content": "Hi, I encountered the same issue at first. Took me a while to figure it out, but I guess it had something to do with my class labels. If you add two classes (0: 'background', 1: 'nucleus'), and then fill an array with 1's of shape mask[-1] (read: the amount of mask instances for one image), it should work (at least it did in my case). Hope this helps!
", "votes": 1, "replies": [] }, { "id": 283757, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-15T23:33:37.153000", "content": "Thanks - I'm doing both of those, but it's still cropping up. I think I must have a deeper bug in it somewhere. Going back to basics and re-downloading / re-writing the nuclei adaptation from scratch. Hopefully I can find what's causing the issue.
", "votes": 1, "replies": [] }, { "id": 283761, "author_name": "rubenhx", "author_url": "", "post_date": "2018-02-15T23:44:45.953000", "content": "Just a small check: when you display the top masks (as given in the train_shapes notebook), does it output all the masks in the image next to the original image along with the class as title?
", "votes": 1, "replies": [] }, { "id": 283763, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-15T23:54:09.197000", "content": "I've actually just gone back to run a train_shapes notebook replica. The code that I had before was integrated with another pipeline. Working through it now - the check that you gave above works, so hopefully it trains ok and I'll have a working code base to compare to... Thanks for the input.
", "votes": 1, "replies": [] }, { "id": 283906, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-16T07:01:15.020000", "content": "Hi fergusoci! I figured out that this issue happens when your class probability goes to 1, that cause situation when your croosentropy logariphm becomes NAN, so i modified code of loss and added Keras epsilon to all crossentropies and that helped.
", "votes": 5, "replies": [] }, { "id": 283951, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-16T09:33:17.833000", "content": "Thanks, Konstantin... Still no joy for me with that. Clipped the probs with epsilon but still getting the same nan. No worries, I've resigned myself to the fact that I've done something stupid somewhere and need to start again from scratch. Going to have a go at that later today. Cheers.
", "votes": 1, "replies": [] }, { "id": 284016, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-16T12:47:23.093000", "content": "Some special-ness on my part. I had active_class_ids set to [0, 1] rather than [1, 1]....
", "votes": 1, "replies": [] }, { "id": 286259, "author_name": "Abhishek Venkataram", "author_url": "", "post_date": "2018-02-21T22:18:35.933000", "content": "Hi fergusoci, I am having the same problem as you did. Did you manage to find a solution to this?
\n\n@ Konstantin Maksimov, can you please explain a little more on how you solved the overflow NAN issue?\nis what you did same as gradient Clipping?
", "votes": 0, "replies": [] }, { "id": 286449, "author_name": "Abhishek Venkataram", "author_url": "", "post_date": "2018-02-22T07:10:08.543000", "content": "Hi Konstantin Maksimov, can you please explain a little more as to how you solved the NAN issue?\nis what you did similar to Gradient Clipping?
", "votes": 0, "replies": [] }, { "id": 286456, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-22T07:16:02.440000", "content": "Hi! Cliipping works for gradients, not for loss, and in the model you have losses that use cross entropy, in my case I have 1 class and background and model learns quickly to classify, that leads to situation when one class has 1 or near 1 probability, another(background) has probability vvvverrry close 0 and when you take logarithm of the 0 you get NaN, see example how I deal with it below
\n\ndef rpn_class_loss_graph(rpn_match, rpn_class_logits):\n \"\"\"RPN anchor classifier loss.
\n\nrpn_match: [batch, anchors, 1]. Anchor match type. 1=positive,\n -1=negative, 0=neutral anchor.\nrpn_class_logits: [batch, anchors, 2]. RPN classifier logits for FG/BG.\n\"\"\"\n# Squeeze last dim to simplify\nrpn_match = tf.squeeze(rpn_match, -1)\n# Get anchor classes. Convert the -1/+1 match to 0/1 values.\nanchor_class = K.cast(K.equal(rpn_match, 1), tf.int32)\n# Positive and Negative anchors contribute to the loss,\n# but neutral anchors (match value = 0) don't.\nindices = tf.where(K.not_equal(rpn_match, 0))\n# Pick rows that contribute to the loss and filter out the rest.\nrpn_class_logits = tf.gather_nd(rpn_class_logits, indices) + K.epsilon()\nanchor_class = tf.gather_nd(anchor_class, indices)\n# Crossentropy loss\nloss = K.sparse_categorical_crossentropy(target=anchor_class,\n output=rpn_class_logits,\n from_logits=True)\nloss = K.switch(tf.size(loss) > 0, K.mean(loss), tf.constant(0.0))\nreturn loss\nHi Abishek. I had the active_class_ids variable inadvertently set to np.array([0, 1]) rather than np.array([1, 1]) and that was what was causing my issue. Another thing to check is to make sure that you are screening out all zero masks before you feed them into training (some may end up with all zeros depending on how you resize them, or what augmentations you are doing).
", "votes": 0, "replies": [] }, { "id": 286639, "author_name": "Abhishek Venkataram", "author_url": "", "post_date": "2018-02-22T14:59:47.953000", "content": "thanks for the immediate response. I had put 0, 1 in the classes, and added a check to make sure the mask is not zero. i was still getting the NAN issue. Also implemented the change suggested by @Konstantin Maksimov, still was getting the issue. Then in the maskRCNN link under the differences from original paper, there is
\n\n\n\n\nLearning Rate: The paper uses a learning rate of 0.02, but we found that to be too high, and often causes the weights to explode, especially when using a small batch size. It might be related to differences between how Caffe and TensorFlow compute gradients (sum vs mean across batches and GPUs). Or, maybe the official model uses gradient clipping to avoid this issue. We do use gradient clipping, but don't set it too aggressively. We found that smaller learning rates converge faster anyway so we go with that.
\n
so i reduced the learning rate and it started working better. in the 10 epochs that i ran i did not get the NAN issue. i also changed the optimiser to Adam. so i am seeing better results
", "votes": 1, "replies": [] }, { "id": 289277, "author_name": "Sang Han", "author_url": "", "post_date": "2018-03-01T09:24:32.273000", "content": "Once there is a nan in your class loss you're going to have issues computing loss for the rest of your training. The nans are occurring when np.log(0) occurs in computing your rpn_bbox, which you can clip in your models.py file like so.
rpn_bbox[ix] = [\n (gt_center_y - a_center_y) / a_h,\n (gt_center_x - a_center_x) / a_w,\n np.log(np.clip(gt_h / a_h, K.epsilon(), None)),\n np.log(np.clip(gt_w / a_w, K.epsilon(), None)),\n]\nHope that helps
", "votes": 3, "replies": [] } ] }, { "id": 284280, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-16T21:02:26.780000", "content": "For anyone using MaskRCNN, were you able to use it with keras ModelCheckpoint? i.e. did MaskRCNN.keras_model.save() work for you?
\n\nGetting a problem with gt_boxes = KL.Lambda(lambda x: x / image_scale)(input_gt_boxes). Because the image_scale input is a tensor the keras.save() won't work.
\n\nLooks like the same problem as here: https://github.com/keras-team/keras/issues/8343
\n\nI tried packaging it into its own custom layer but then get a ('Not JSON Serializable:', tf.float32) error.
\n\nIf anyone happens to have come across the same problem and can share a way around it it would be much appreciated!
\n\nEDIT: Ignore - I'll just save the weights rather than the model.
", "votes": 1, "replies": [] }, { "id": 283956, "author_name": "Nishant Singh", "author_url": "", "post_date": "2018-02-16T09:47:17.977000", "content": "Hi Konstantin, what are the preprocessing steps you are doing?
", "votes": 1, "replies": [ { "id": 283964, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-16T10:16:23.250000", "content": "Hi!
\n\n1 - convert images to grayscale\n2 - fill holes in some broken train masks
\n\nWhat I plan to add - move issues with data from official thread and try to train on different channels.
", "votes": 3, "replies": [] } ] }, { "id": 288139, "author_name": "fergusoci", "author_url": "", "post_date": "2018-02-26T19:28:28.143000", "content": "Anyone who is using MaskRCNN and is willing to share some of the evolution of their training/val loss output? Would like to know what types of values are 'normal' for this dataset. Thanks!
", "votes": 2, "replies": [ { "id": 288333, "author_name": "Elena Cuoco", "author_url": "", "post_date": "2018-02-27T08:29:19.127000", "content": "Some discussion here: \nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/50610#latest-288331
", "votes": 1, "replies": [] }, { "id": 291185, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2018-03-05T18:54:13.203000", "content": "this is one of my:\n664/664 [==============================] - 500s 753ms/step - loss: 1.0058 - rpn_class_loss: 0.0685 - rpn_bbox_loss: 0.3533 - mrcnn_class_loss: 0.1389 - mrcnn_bbox_loss: 0.1958 - mrcnn_mask_loss: 0.2493 - val_loss: 2.3675 - val_rpn_class_loss: 0.2059 - val_rpn_bbox_loss: 0.9577 - val_mrcnn_class_loss: 0.2721 - val_mrcnn_bbox_loss: 0.4767 - val_mrcnn_mask_loss: 0.4550
\n\nNow i am not sure which of these metrics are most important ? perhaps val_loss ? \nIn this one i used 2 epochs and got a LB score of 0.358.
", "votes": 0, "replies": [] }, { "id": 291228, "author_name": "Matt", "author_url": "", "post_date": "2018-03-05T20:48:26.173000", "content": "", "votes": 0, "replies": [] }, { "id": 292025, "author_name": "Mr.zhenshuo", "author_url": "", "post_date": "2018-03-07T08:54:32.167000", "content": "hey, Mehul, i use this mask rcnn too ,and i got LB score unnormally low , i think my encoding process has some issues , i mean from masks to submission . and my encoding codes is this :
\n\ndef rle_encoding(x):\n dots = np.where(x.T.flatten() == 1)[0]\n run_lengths = []\n prev = -2\n for b in dots:\n if (b>prev+1): run_lengths.extend((b + 1, 0))\n run_lengths[-1] += 1\n prev = b\n return run_lengths
\n\ndef prob_to_rles(x, cutoff=0.5):\n lab_img = label(x > cutoff)\n for i in range(1, lab_img.max() + 1):\n yield rle_encoding(lab_img == i)
\n\nis this right ? \ni would be more than very appreciate , if u could respond to me .thanks so much .
", "votes": 0, "replies": [] } ] }, { "id": 306178, "author_name": "IAmLegend", "author_url": "", "post_date": "2018-03-30T01:59:10.280000", "content": "Hi Konstantin, I have a silly question about the config file.
\n\nI set GPU_COUNT = 1 and IMAGES_PER_GPU = 1 when I used one 1070 to train the model before.
\n\nIn my opinion, I have two 1080ti now so I can set GPU_COUNT to 2, 1080ti has more memory than 1070 so I can change IMAGES_PER_GPU more than 1.
\n\nNow, I modify GPU_COUNT = 2, IMAGES_PER_GPU = 2 but I always get OOM and CUDA error even I set IMAGES_PER_GPU = 1.
\n\nIt only works when I set GPU_COUNT = 1 and IMAGES_PER_GPU = 1, any suggestion? Thanks.
", "votes": 0, "replies": [ { "id": 306693, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-03-30T20:44:31.077000", "content": "I have same config with one 1070 and other with 2 1080 tI, for me chang to 2 per image was ok, can you provide error tracebak for better understandig the type of problem?
", "votes": 0, "replies": [] }, { "id": 309051, "author_name": "IAmLegend", "author_url": "", "post_date": "2018-04-04T15:01:27.957000", "content": "Sorry for the late reply. I solved the CUDA error but still have the OOM problem.
\n\nIt always OOM when batch_size > 1 and I always get ~1000s per epoch(600 samples per epoch).
\n\nI check other threads who has 2 1080tis, they get about ~5 mins per epoch, any suggestion?
", "votes": 0, "replies": [] }, { "id": 309347, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-04-05T05:02:57.273000", "content": "Maybe you have old CUDA and CuDNN ? Because I use tensorflow 1.6 with CUDA 9.0 and CuDNN 7 - it gave almost 2x speed up
", "votes": 1, "replies": [] } ] }, { "id": 290455, "author_name": "Matt", "author_url": "", "post_date": "2018-03-04T05:15:12.550000", "content": "I use a very similar configuration but my network is taking much longer to train than 8 epochs. I've trained for 40 epochs and still get a loss around 1.2. Any ideas on how to improve the convergence?
\n\nHere's a link to my code so maybe you can see where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py
", "votes": 0, "replies": [] }, { "id": 287819, "author_name": "", "author_url": "", "post_date": "2018-02-26T02:24:09.300000", "content": "Hi Konstantin, Thank you for sharing your code. It provides an opportunity for beginners like me to jump into the code. Can you provide the reason for deciding to fill holes in the mask? Is it a very common pre-processing step?
", "votes": 0, "replies": [ { "id": 287876, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-26T05:58:41.827000", "content": "It is not common step, it is an error that was posted in official data quality discussion, see forum for more details.
", "votes": 3, "replies": [] }, { "id": 287904, "author_name": "", "author_url": "", "post_date": "2018-02-26T07:19:15.070000", "content": "Thank you for helping out and pointing me to the discussions!
", "votes": 0, "replies": [] } ] }, { "id": 287377, "author_name": "mary", "author_url": "", "post_date": "2018-02-24T15:26:01.760000", "content": "Thanks for the provided info. Isnt it possible to share your code? I stuck in the data preparation and feeding to network. How do you handle the missing bounding box?
", "votes": 0, "replies": [ { "id": 287423, "author_name": "bsp2020", "author_url": "", "post_date": "2018-02-24T18:55:00.547000", "content": "I scaled input images smaller than 512x512 to 512x512 before starting the training process. Then, the error message went away.
", "votes": 1, "replies": [] }, { "id": 287437, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-24T19:36:13.867000", "content": "Do you get your current LB score with Mask RCNN or something else? Because I stucked at 0.411, but I don't use any augmentations.
", "votes": 0, "replies": [] }, { "id": 287439, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-24T19:37:01.827000", "content": "I plan to share code when I will move at least to top 20.
", "votes": 0, "replies": [] }, { "id": 287456, "author_name": "bsp2020", "author_url": "", "post_date": "2018-02-24T21:13:16.287000", "content": "I use Mask RCNN and augmentation. I got stuck at .42 when I was just flipping and rotating 90 degrees. The current score is with 5 degrees rotation on top of flipping/90 rotation.
", "votes": 1, "replies": [] }, { "id": 287459, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-24T21:23:50.973000", "content": "Thanks a lot! Your config will be also great to see)))
", "votes": 0, "replies": [] }, { "id": 287617, "author_name": "Elena Cuoco", "author_url": "", "post_date": "2018-02-25T13:04:33.547000", "content": "Did you use mini_mask also with augmentation and arbitrary rotation?
", "votes": 0, "replies": [] }, { "id": 289284, "author_name": "FabSchreiber", "author_url": "", "post_date": "2018-03-01T09:48:24.193000", "content": "Elena, what is the idea behind mini_mask?
\n\nI see it being mentioned in several places, but don't understand it yet.
", "votes": 0, "replies": [] }, { "id": 289287, "author_name": "Elena Cuoco", "author_url": "", "post_date": "2018-03-01T09:56:45.643000", "content": "Hi, it helps for high resolution images, when you are looking for small object. it is well explained here in this notebook https://github.com/matterport/Mask_RCNN/blob/master/inspect_data.ipynb
", "votes": 2, "replies": [] }, { "id": 289288, "author_name": "FabSchreiber", "author_url": "", "post_date": "2018-03-01T09:58:38.990000", "content": "Thanks, Elena!
", "votes": 0, "replies": [] } ] }, { "id": 286607, "author_name": "john1231983", "author_url": "", "post_date": "2018-02-22T13:24:30.727000", "content": "Thanks for your solution. I have two questions:\n1. Why not seperate training data into 3 clusters and make 3 models for these clusters?\n2. Why not use imagenet pretrained model and train on \"all\" instead of coco and 'heads' as you did? I found that train \"all\" give better performance but I did not submit yet.\nDo you have any update LB?
", "votes": 0, "replies": [ { "id": 286621, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-22T13:59:07.987000", "content": "1) Because we don't know how many types we will have in stage2, so we need good generalisation, npt tweaking \n2) Imagenet showed worth result for my config, training heads is common technique, it helps to reduce backprop error\nI am working on improvement my model, my local validation is broken so no any good movements from 0.426 for 1.5 weeks already
", "votes": 1, "replies": [] }, { "id": 286688, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-02-22T16:49:05.170000", "content": "Hi Konstantin. Is there a particular reason you are converting the images into grayscale? Did you not get good results with color images?
", "votes": 0, "replies": [] }, { "id": 286771, "author_name": "john1231983", "author_url": "", "post_date": "2018-02-22T18:52:38.897000", "content": "I see. Do you try random crop image into 128x128 for training? It can increase the training set size. Then replace fully connected to fully convolution for making mask rcnn working in arbitrary size, hence you can run testing with original size without crop. It may improve your LB. I guess the current maskrcnn implementation cannot work in arbitrary size
", "votes": 0, "replies": [] }, { "id": 287016, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-23T09:26:51.177000", "content": "Hi @panserbjorne! The only reason is that it was first idea how to deal with different color schemes, actually no any analysis behind it.
", "votes": 1, "replies": [] }, { "id": 287106, "author_name": "zmingen", "author_url": "", "post_date": "2018-02-23T15:11:29.837000", "content": "Hi Konstantin,
\n\nYou mentioned you achieve 0.407 by training on your pretrained network. Can you share the method to achieve 0.426? Thanks!
", "votes": 0, "replies": [] }, { "id": 287216, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-23T22:14:36.480000", "content": "The problem is that I deleted accidentally 0.426 result((( Now my best submission is 0.413 actually. As soon as will crack at least 0.430 I will provide update how to get 0.413
", "votes": 0, "replies": [] }, { "id": 287315, "author_name": "AndrewJWolf", "author_url": "", "post_date": "2018-02-24T10:17:05.030000", "content": "For the masks, did you use the default masks where each mask image contains one segmented nucleus, or did you use masks with all the segmented nuclei in single mask images with a 1:1 correspondence to the original training images?
\n\nI used the latter, with a single corresponding mask image with all segmented nuclei. However, ran into a problem after training for one epoch. The error: 'TypeError: can't pickle _thread.lock objects'
\n\nIt also said the following: \"UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.\"
\n\nAfter using code from inspect_data.ipynb, I noticed that there was only one bounding box cropping all the segmented nuclei when there should be a single bounding box for each instance or each nucleus, right?
", "votes": 0, "replies": [] } ] }, { "id": 284479, "author_name": "Paul Chen", "author_url": "", "post_date": "2018-02-17T08:28:56.557000", "content": "Hi Konstantin, have you think about modify the pre-trained weights to fit 1 channel grayscale image? I thought it will accelerate the training speed pretty much.
", "votes": 0, "replies": [ { "id": 284531, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-17T13:17:15.570000", "content": "No, because my stratgy is to get new pre-train for grayscale and then try diffrent ideas on new pretrained.
", "votes": 0, "replies": [] }, { "id": 284686, "author_name": "Paul Chen", "author_url": "", "post_date": "2018-02-17T23:27:57.333000", "content": "Where do you find the new pretrain weights based on grayscale image?
", "votes": 0, "replies": [] }, { "id": 284804, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-18T12:11:36.137000", "content": "I train for 50 epochs on coco using grayscle - and it in now my new pretrined network.
", "votes": 0, "replies": [] }, { "id": 284823, "author_name": "Paul Chen", "author_url": "", "post_date": "2018-02-18T13:32:05.193000", "content": "How much does it improve your performance?
", "votes": 0, "replies": [] }, { "id": 284911, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-18T19:27:14.023000", "content": "from 0.362 to 0.407 on LB
", "votes": 0, "replies": [] }, { "id": 284995, "author_name": "Paul Chen", "author_url": "", "post_date": "2018-02-19T02:25:07.790000", "content": "The performance seems to get worse after training 200 epochs. What kind of val_loss and other loss metrics do you get for 0.362?
", "votes": 0, "replies": [] }, { "id": 285056, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-19T07:11:00.573000", "content": "Hi! val_loss 0.7 mAP on val dataset around 0.48
", "votes": 0, "replies": [] }, { "id": 285104, "author_name": "", "author_url": "", "post_date": "2018-02-19T10:11:23.367000", "content": "", "votes": 0, "replies": [] }, { "id": 285376, "author_name": "AndrewJWolf", "author_url": "", "post_date": "2018-02-20T04:53:24.240000", "content": "Hi Konstantin Maksimov,\nabout to ask a silly question:
\n\nHow did you subclass the nuclei dataset to run it through MatterPort's model?
\n\nMatterport instructs the user to create a derived class from the Dataset class in utils.py.\nSo, so far, I only got this:
\n\nclass NucleiDataset(utils.Dataset):\n def load_nuclei(self):\n pass\n def load_mask(self, image_id):\n pass\n def image_reference(self, image_id):\n pass\nI have the data prep code from this guy. \nCould you give me a hint as to how to incorporate this data-prep code into the derived class NucleiDataset?
", "votes": 0, "replies": [] }, { "id": 285416, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-20T07:34:40.010000", "content": "Here is my example (some code is not in use and still need refactoring
\n\nclass BowlDataset(utils.Dataset):\n def load_data(self, root_path, mode='train', filter_ids=None):
\n\n # Add classes\n\n self.add_class(\"nucleos\", 1, \"nucleo\")\n if mode == 'test':\n files = os.listdir(os.path.join(root_path, 'images_test'))\n else:\n files = os.listdir(os.path.join(root_path, 'images'))\n\n if filter_ids is not None:\n\n files = [item for item in files if item.split('.')[0] in filter_ids]\n # print(len(files))\n for i, ffile in enumerate(tqdm(files)):\n # pool = Pool(cpu_count() - 2)\n\n\n if mode == 'test':\n data_path = os.path.join(root_path, 'images_test', ffile)\n mask_path = 'No masks'\n else:\n data_path = os.path.join(root_path, 'images', ffile)\n\n\n bg_color = random.randint(0, 255)\n original_id = data_path.split('/')[-1].split('.')[-2]\n # TODO change later maybe\n\n data = cv2.imread(data_path)[:, :, 0]\n height, width = data.shape\n\n self.add_image(\"nucleos\", image_id=i, path=data_path,\n width=width, height=height,\n bg_color=bg_color, original_id=original_id,\n data_path=data_path,\n root_path=root_path)\n\n image = skimage.io.imread(data_path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n\n # self.data_structure[i] = {}\n # self.data_structure[i]['image'] = image[:]\n # self.data_structure[i]['mask'] = self.add_mask_data(original_id, root_path,\n # width=width, height=height)[:]\n\ndef load_image(self, image_id):\n info = self.get_info(image_id)\n path = info['data_path']\n image = skimage.io.imread(path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n return image\n\ndef load_mask(self, image_id):\n info = self.get_info(image_id)\n original_id = info['original_id']\n root_path = info['root_path']\n width = info['width']\n height = info['height']\n return self.add_mask_data(original_id, root_path, width=width, height=height)[:]\n\ndef get_info(self, image_id):\n return self.image_info[image_id]\n\ndef add_mask_data(self, original_id, root_path, width=256, height=256):\n # info = self.image_info[image_id]\n orifinal_id = original_id\n root_path = root_path\n all_masks = os.listdir(os.path.join(root_path, 'masks'))\n # print(original_id)\n # all_masks = [element\n # for element in all_masks if element.split('!')[0] == orifinal_id]\n all_masks = [element for element in all_masks if element.split('_')[0] == orifinal_id]\n\n\n num_labels = 0\n for i, mask_file in enumerate(all_masks):\n\n subpath = os.path.join(root_path, 'masks', mask_file)\n # print(subpath)\n data = skimage.io.imread(subpath)[:, :]\n # print(data.shape)\n # print(np.sum(data))\n\n if np.sum(data) != 0:\n # print('SHIT MASK DETECTED')\n\n num_labels += 1\n\n mask = np.zeros([height,\n width,\n num_labels],\n dtype=np.bool)\n num_labels = 0\n for i, mask_file in enumerate(all_masks):\n subpath = os.path.join(root_path, 'masks', mask_file)\n # print(subpath)\n # data = cv2.imread(subpath)[:, :, 0]\n data = skimage.io.imread(subpath)[:, :]\n # print(np.max(data))\n if np.sum(data) != 0:\n # print('SHIT MASK DETECTED')\n\n # height, width = data.shape\n\n # if height != info['height'] and width != info['width']:\n # data = cv2.resize(data, (info['height'], info['width']))\n # change translation\n # data = (data == 255)\n data = (data != 0)\n # print(np.sum(data))\n mask[:, :, num_labels] = data\n num_labels += 1\n # class_ids = [1]*num_labels\n class_ids = np.array([1] * num_labels)\n class_ids = class_ids.astype(np.int32)\n return mask, class_ids\nYou're awesome! Thank you!
", "votes": 0, "replies": [] }, { "id": 285879, "author_name": "AndrewJWolf", "author_url": "", "post_date": "2018-02-21T06:42:05.303000", "content": "Another silly question: the argument 'filter_ids', does that refer to the id's of the images such as '0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732' as one example?
\n\nI see these lines of code below
\n\nif filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids] \nand interpret the second line as catching this long id string just before the '.png'. \nIs this interpretation incorrect?
\n\nFurther, my file paths are the following: \"root_path/id/images/id.png\" to the images.\nThe path to the masks are the following: \"root_path/id/masks/id/mask_image.png\"\nIt looks like you had to change your file paths for this code to work.
", "votes": 0, "replies": [] }, { "id": 285915, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-21T08:22:57.597000", "content": "filter_ids is used for separation of val and train, and yes - my structure is different from defailt one
", "votes": 0, "replies": [] }, { "id": 286304, "author_name": "Paul Chen", "author_url": "", "post_date": "2018-02-22T00:47:26.763000", "content": "Hi Konstantin, I tried your solution and I trained a raw model on the grayscale coco dataset. I'm wondering how much loss do you get from this basemodel as my model has nearly 1.9 on val_loss.
", "votes": 0, "replies": [] }, { "id": 286453, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-22T07:11:41.513000", "content": "don't train on coco, take coco pretrained model and then train for 8 epochs, that will give val loss around 0.7 - 1
", "votes": 0, "replies": [] } ] }, { "id": 283562, "author_name": "AviWolfson", "author_url": "", "post_date": "2018-02-15T17:47:13.027000", "content": "the images are (w,h,1) or (w,h,3)? if it is 1 do u need to change something in maskrcnn code?\nhow did u done a hole in the masks and how do u fill the hole after?
", "votes": 0, "replies": [ { "id": 283617, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-15T19:19:50.663000", "content": "Hi! images are grayscale (w,h,3) with 3 equal channels, so no any changes to the code. Holes are present in some train masks. I used this for holes fill - https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.ndimage.morphology.binary_fill_holes.html#scipy-ndimage-morphology-binary-fill-holes
", "votes": 1, "replies": [] }, { "id": 283743, "author_name": "AviWolfson", "author_url": "", "post_date": "2018-02-15T22:42:19.770000", "content": "did u fill holes to all images or u chosen them?
", "votes": 1, "replies": [] }, { "id": 283908, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-02-16T07:02:37.577000", "content": "I filled all masks, if there is no holes - nothing happens.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "283463": "Hi all!\nBelow I have posted config that gives chance to achieve 0.362 LB score only on 8 epochs on 1 1070 with training only 'heads' of the network ONLY. Using this config you can achieve approximately 0.4 on LB using more epochs.\n\nAlso I used next processing - convert images to grayscale and fill holes in masks, then select 600 a 70 subsets by random seed 5964 and after prediction I process overlaps in a next way - I cut part from the smaller instance.\n\nHope this will help. \n\nfrom config import Config\n\nclass BowlConfig(Config):\n NAME = \"nucleos\"\n GPU_COUNT = 1\n IMAGES_PER_GPU = 1\n NUM_CLASSES = 1 + 1 \n IMAGE_MIN_DIM = 256\n IMAGE_MAX_DIM = 512\n RPN_ANCHOR_SCALES = (4, 8, 16, 32, 64) \n TRAIN_ROIS_PER_IMAGE = 500\n STEPS_PER_EPOCH = 600 // (IMAGES_PER_GPU * GPU_COUNT)\n VALIDATION_STEPS = 70 // (IMAGES_PER_GPU * GPU_COUNT)\n MEAN_PIXEL = [0, 0, 0]\n LEARNING_RATE = 0.01\n USE_MINI_MASK = True\n MAX_GT_INSTANCES = 500\n", "283728": "Thanks for sharing, Konstantin. I'm still being tortured by nan's in the mrcnn_class_loss. mask and bbox related losses are fine. I thought it might have had something to do with there just being one class_id output, but I tried with multiple class_id values and I'm still having the same issue. Did you run into any nan problems with the class losses, and if so was there something besides the config that solved them?", "284280": "For anyone using MaskRCNN, were you able to use it with keras ModelCheckpoint? i.e. did MaskRCNN.keras_model.save() work for you? \n\nGetting a problem with gt_boxes = KL.Lambda(lambda x: x / image_scale)(input_gt_boxes). Because the image_scale input is a tensor the keras.save() won't work.\n\nLooks like the same problem as here: https://github.com/keras-team/keras/issues/8343\n\nI tried packaging it into its own custom layer but then get a ('Not JSON Serializable:', tf.float32) error.\n\nIf anyone happens to have come across the same problem and can share a way around it it would be much appreciated!\n\nEDIT: Ignore - I'll just save the weights rather than the model.", "283956": "Hi Konstantin, what are the preprocessing steps you are doing?", "288139": "Anyone who is using MaskRCNN and is willing to share some of the evolution of their training/val loss output? Would like to know what types of values are 'normal' for this dataset. Thanks!", "306178": "Hi Konstantin, I have a silly question about the config file.\n\nI set GPU_COUNT = 1 and IMAGES_PER_GPU = 1 when I used one 1070 to train the model before.\n\nIn my opinion, I have two 1080ti now so I can set GPU_COUNT to 2, 1080ti has more memory than 1070 so I can change IMAGES_PER_GPU more than 1.\n\nNow, I modify GPU_COUNT = 2, IMAGES_PER_GPU = 2 but I always get OOM and CUDA error even I set IMAGES_PER_GPU = 1.\n\nIt only works when I set GPU_COUNT = 1 and IMAGES_PER_GPU = 1, any suggestion? Thanks.", "290455": "I use a very similar configuration but my network is taking much longer to train than 8 epochs. I've trained for 40 epochs and still get a loss around 1.2. Any ideas on how to improve the convergence?\n\nHere's a link to my code so maybe you can see where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py", "287819": "Hi Konstantin, Thank you for sharing your code. It provides an opportunity for beginners like me to jump into the code. Can you provide the reason for deciding to fill holes in the mask? Is it a very common pre-processing step?", "287377": "Thanks for the provided info. Isnt it possible to share your code? I stuck in the data preparation and feeding to network. How do you handle the missing bounding box?", "286607": "Thanks for your solution. I have two questions:\n1. Why not seperate training data into 3 clusters and make 3 models for these clusters?\n2. Why not use imagenet pretrained model and train on \"all\" instead of coco and 'heads' as you did? I found that train \"all\" give better performance but I did not submit yet.\nDo you have any update LB?", "284479": "Hi Konstantin, have you think about modify the pre-trained weights to fit 1 channel grayscale image? I thought it will accelerate the training speed pretty much.", "283562": "the images are (w,h,1) or (w,h,3)? if it is 1 do u need to change something in maskrcnn code?\nhow did u done a hole in the masks and how do u fill the hole after?" } }