I've seen several competitors on here who are able to achieve LB scores > 0.3 after training less than ten epochs. With my implementation, the training seems really slow and even after 40 epochs my LB score is only 0.15. The model still sometimes guesses nuclei in area where there are clearly none.
\n\nHere's a link to my code so perhaps someone can tell me where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py\nI'm doing all the training on Google Colab
", "messages": [ { "id": "291230", "postDate": "03/05/2018 20:53:53", "content": "I've seen several competitors on here who are able to achieve LB scores > 0.3 after training less than ten epochs. With my implementation, the training seems really slow and even after 40 epochs my LB score is only 0.15. The model still sometimes guesses nuclei in area where there are clearly none.
\n\nHere's a link to my code so perhaps someone can tell me where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py\nI'm doing all the training on Google Colab
", "rawMarkdown": "I've seen several competitors on here who are able to achieve LB scores > 0.3 after training less than ten epochs. With my implementation, the training seems really slow and even after 40 epochs my LB score is only 0.15. The model still sometimes guesses nuclei in area where there are clearly none.\n\nHere's a link to my code so perhaps someone can tell me where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py\nI'm doing all the training on Google Colab", "votes": null }, { "id": "291235", "postDate": "03/05/2018 21:02:55", "content": "Hi! I also use Maatterport and you can achieve 0.43+ with this implementation. See my forum discussion with config that gives 0.362 on LB on 8 epochs.
", "rawMarkdown": "Hi! I also use Maatterport and you can achieve 0.43+ with this implementation. See my forum discussion with config that gives 0.362 on LB on 8 epochs.", "votes": null }, { "id": "291239", "postDate": "03/05/2018 21:04:27", "content": "Hi Matthew, \nI got config parameters from repos shared by others. I ran 3 experiments and i got 0.348, 0.335 and 0.352 (2 epoch). \nWill start a detailed discussion with all experiment conditions. these parameters worked for me. let me know if you this dont work
\n\ninit_with = \"coco\"\n LEARNING_RATE = 0.001
\n\n# Train on 1 GPU and 8 images per GPU. We can put multiple images on each\n # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).\n GPU_COUNT = 1\n IMAGES_PER_GPU = 1
\n\n# Number of classes (including background)\nNUM_CLASSES = 1 + 1 # background + nuclei\n\n# Use small images for faster training. Set the limits of the small side\n# the large side, and that determines the image shape.\nIMAGE_MIN_DIM = 256\nIMAGE_MAX_DIM = 256\n\n# Use smaller anchors because our image and objects are small\nRPN_ANCHOR_SCALES = (8, 16, 32, 64) # anchor side in pixels\n\n# Reduce training ROIs per image because the images are small and have\n# few objects. Aim to allow ROI sampling to pick 33% positive ROIs.\nTRAIN_ROIS_PER_IMAGE = 500\n\nSTEPS_PER_EPOCH = None\n\n# use small validation steps since the epoch is small\nVALIDATION_STEPS = 5\n\nUSE_MINI_MASK = False\nI saw your configuration. For some reason I get even worse results when I use this. Something else must be causing the issue. I've shared my code above if you care to take a look
", "rawMarkdown": "I saw your configuration. For some reason I get even worse results when I use this. Something else must be causing the issue. I've shared my code above if you care to take a look", "votes": null }, { "id": "291243", "postDate": "03/05/2018 21:07:44", "content": "I will test with these configuration but I do not think it is my configurations causing the slowdown.
", "rawMarkdown": "I will test with these configuration but I do not think it is my configurations causing the slowdown.", "votes": null }, { "id": "291247", "postDate": "03/05/2018 21:13:14", "content": "why do you have \"ignore_masks \" section. did you find them to be incorrect masks ?
", "rawMarkdown": "why do you have \"ignore_masks \" section. did you find them to be incorrect masks ?", "votes": null }, { "id": "291249", "postDate": "03/05/2018 21:16:58", "content": "I see you have both:
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=20, \n layers='heads')
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE / 10,\n epochs=20, \n layers=\"all\")
\n\nI only did with layers=\"all\"; doing both \"head\" and \"all\" will slow it down
", "rawMarkdown": "I see you have both: \n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=20, \n layers='heads')\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE / 10,\n epochs=20, \n layers=\"all\")\n\nI only did with **layers=\"all\"**; doing both \"head\" and \"all\" will slow it down", "votes": null }, { "id": "291250", "postDate": "03/05/2018 21:18:22", "content": "I found that these masks had no area. They were just single pixel wide lines. I figured they would be harmful for training.
", "rawMarkdown": "I found that these masks had no area. They were just single pixel wide lines. I figured they would be harmful for training.", "votes": null }, { "id": "291268", "postDate": "03/05/2018 21:51:58", "content": "Hi Matthew, \nI think i found your issue. for init with coco you have:
\n\nelif init_with == \"coco\":\n model.load_weights(\"mask_rcnn_nuclei.h5\", by_name=True,\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])
\n\nyou need to use \"\"mask_rcnn_coco.h5\"\" . please try that and let us know if it improved
", "rawMarkdown": "Hi Matthew, \nI think i found your issue. for init with coco you have: \n\n \nelif init_with == \"coco\":\n **model.load_weights(\"mask_rcnn_nuclei.h5\", by_name=True,**\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])\n\nyou need to use **\"\"mask_rcnn_coco.h5\"\"** . please try that and let us know if it improved", "votes": null }, { "id": "292508", "postDate": "03/08/2018 04:03:33", "content": "Hi Matthew, just curious to know what fixed your issue with the low score ?
", "rawMarkdown": "Hi Matthew, just curious to know what fixed your issue with the low score ?", "votes": null }, { "id": "294147", "postDate": "03/11/2018 11:59:09", "content": "Hi, Matthew, you should only add one class \"nuclei/nucleus\" using self.add_class('nuclei', 1, 'nucleus'), and delete the background, because the original code have already contained it.\nAnd I can't see your code on github now, it helped me a lot.
Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. \nHi Matthew have meet the warning above, and how much time you spend in one epoch?
", "rawMarkdown": "*Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.* \nHi Matthew have meet the warning above, and how much time you spend in one epoch?", "votes": null }, { "id": "298813", "postDate": "03/20/2018 03:13:44", "content": "Hi Konstantin ,how much time do you spend in one epoch when you use Maatterport's code
", "rawMarkdown": "Hi Konstantin ,how much time do you spend in one epoch when you use Maatterport's code", "votes": null }, { "id": "308740", "postDate": "04/04/2018 02:09:55", "content": "Hi linCurry,
\n\nCan I have your email? I have some questions about the competition. My email address is zhulinchao7@gmail.com.
", "rawMarkdown": "Hi linCurry,\n\nCan I have your email? I have some questions about the competition. My email address is zhulinchao7@gmail.com.", "votes": null }, { "id": "308753", "postDate": "04/04/2018 02:58:18", "content": "Sure, my email address is linsocurry@163.com
", "rawMarkdown": "Sure, my email address is linsocurry@163.com", "votes": null } ], "comments": [ { "id": 291235, "author_name": "maksimovka", "author_url": "", "post_date": "03/05/2018 21:02:55", "content": "Hi! I also use Maatterport and you can achieve 0.43+ with this implementation. See my forum discussion with config that gives 0.362 on LB on 8 epochs.
", "votes": null, "replies": [ { "id": 291241, "author_name": "matthewmasters", "author_url": "", "post_date": "03/05/2018 21:05:26", "content": "I saw your configuration. For some reason I get even worse results when I use this. Something else must be causing the issue. I've shared my code above if you care to take a look
", "votes": null, "replies": [] }, { "id": 291247, "author_name": "mpsampat", "author_url": "", "post_date": "03/05/2018 21:13:14", "content": "why do you have \"ignore_masks \" section. did you find them to be incorrect masks ?
", "votes": null, "replies": [] }, { "id": 291250, "author_name": "matthewmasters", "author_url": "", "post_date": "03/05/2018 21:18:22", "content": "I found that these masks had no area. They were just single pixel wide lines. I figured they would be harmful for training.
", "votes": null, "replies": [] }, { "id": 298813, "author_name": "lincurry", "author_url": "", "post_date": "03/20/2018 03:13:44", "content": "Hi Konstantin ,how much time do you spend in one epoch when you use Maatterport's code
", "votes": null, "replies": [] }, { "id": 308740, "author_name": "ffmpbgrnn", "author_url": "", "post_date": "04/04/2018 02:09:55", "content": "Hi linCurry,
\n\nCan I have your email? I have some questions about the competition. My email address is zhulinchao7@gmail.com.
", "votes": null, "replies": [] }, { "id": 308753, "author_name": "lincurry", "author_url": "", "post_date": "04/04/2018 02:58:18", "content": "Sure, my email address is linsocurry@163.com
", "votes": null, "replies": [] } ] }, { "id": 291239, "author_name": "mpsampat", "author_url": "", "post_date": "03/05/2018 21:04:27", "content": "Hi Matthew, \nI got config parameters from repos shared by others. I ran 3 experiments and i got 0.348, 0.335 and 0.352 (2 epoch). \nWill start a detailed discussion with all experiment conditions. these parameters worked for me. let me know if you this dont work
\n\ninit_with = \"coco\"\n LEARNING_RATE = 0.001
\n\n# Train on 1 GPU and 8 images per GPU. We can put multiple images on each\n # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).\n GPU_COUNT = 1\n IMAGES_PER_GPU = 1
\n\n# Number of classes (including background)\nNUM_CLASSES = 1 + 1 # background + nuclei\n\n# Use small images for faster training. Set the limits of the small side\n# the large side, and that determines the image shape.\nIMAGE_MIN_DIM = 256\nIMAGE_MAX_DIM = 256\n\n# Use smaller anchors because our image and objects are small\nRPN_ANCHOR_SCALES = (8, 16, 32, 64) # anchor side in pixels\n\n# Reduce training ROIs per image because the images are small and have\n# few objects. Aim to allow ROI sampling to pick 33% positive ROIs.\nTRAIN_ROIS_PER_IMAGE = 500\n\nSTEPS_PER_EPOCH = None\n\n# use small validation steps since the epoch is small\nVALIDATION_STEPS = 5\n\nUSE_MINI_MASK = False\nI will test with these configuration but I do not think it is my configurations causing the slowdown.
", "votes": null, "replies": [] }, { "id": 291249, "author_name": "mpsampat", "author_url": "", "post_date": "03/05/2018 21:16:58", "content": "I see you have both:
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=20, \n layers='heads')
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE / 10,\n epochs=20, \n layers=\"all\")
\n\nI only did with layers=\"all\"; doing both \"head\" and \"all\" will slow it down
", "votes": null, "replies": [] }, { "id": 291268, "author_name": "mpsampat", "author_url": "", "post_date": "03/05/2018 21:51:58", "content": "Hi Matthew, \nI think i found your issue. for init with coco you have:
\n\nelif init_with == \"coco\":\n model.load_weights(\"mask_rcnn_nuclei.h5\", by_name=True,\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])
\n\nyou need to use \"\"mask_rcnn_coco.h5\"\" . please try that and let us know if it improved
", "votes": null, "replies": [] }, { "id": 292508, "author_name": "mpsampat", "author_url": "", "post_date": "03/08/2018 04:03:33", "content": "Hi Matthew, just curious to know what fixed your issue with the low score ?
", "votes": null, "replies": [] } ] }, { "id": 294147, "author_name": "dexterlei", "author_url": "", "post_date": "03/11/2018 11:59:09", "content": "Hi, Matthew, you should only add one class \"nuclei/nucleus\" using self.add_class('nuclei', 1, 'nucleus'), and delete the background, because the original code have already contained it.\nAnd I can't see your code on github now, it helped me a lot.
Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. \nHi Matthew have meet the warning above, and how much time you spend in one epoch?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "291230": "I've seen several competitors on here who are able to achieve LB scores > 0.3 after training less than ten epochs. With my implementation, the training seems really slow and even after 40 epochs my LB score is only 0.15. The model still sometimes guesses nuclei in area where there are clearly none.\n\nHere's a link to my code so perhaps someone can tell me where I'm going wrong\nhttps://github.com/MatthewMasters/DSB2018/blob/master/r_cnn_data_science_bowl_2018.py\nI'm doing all the training on Google Colab", "291235": "Hi! I also use Maatterport and you can achieve 0.43+ with this implementation. See my forum discussion with config that gives 0.362 on LB on 8 epochs.", "291239": "Hi Matthew, \nI got config parameters from repos shared by others. I ran 3 experiments and i got 0.348, 0.335 and 0.352 (2 epoch). \nWill start a detailed discussion with all experiment conditions. these parameters worked for me. let me know if you this dont work\n\n**init_with = \"coco\"**\n **LEARNING_RATE = 0.001**\n\n # Train on 1 GPU and 8 images per GPU. We can put multiple images on each\n # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).\n GPU_COUNT = 1\n IMAGES_PER_GPU = 1\n\n # Number of classes (including background)\n NUM_CLASSES = 1 + 1 # background + nuclei\n\n # Use small images for faster training. Set the limits of the small side\n # the large side, and that determines the image shape.\n IMAGE_MIN_DIM = 256\n IMAGE_MAX_DIM = 256\n\n # Use smaller anchors because our image and objects are small\n RPN_ANCHOR_SCALES = (8, 16, 32, 64) # anchor side in pixels\n\n # Reduce training ROIs per image because the images are small and have\n # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.\n TRAIN_ROIS_PER_IMAGE = 500\n\n STEPS_PER_EPOCH = None\n\n # use small validation steps since the epoch is small\n VALIDATION_STEPS = 5\n\n USE_MINI_MASK = False", "291241": "I saw your configuration. For some reason I get even worse results when I use this. Something else must be causing the issue. I've shared my code above if you care to take a look", "291243": "I will test with these configuration but I do not think it is my configurations causing the slowdown.", "291247": "why do you have \"ignore_masks \" section. did you find them to be incorrect masks ?", "291249": "I see you have both: \n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=20, \n layers='heads')\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE / 10,\n epochs=20, \n layers=\"all\")\n\nI only did with **layers=\"all\"**; doing both \"head\" and \"all\" will slow it down", "291250": "I found that these masks had no area. They were just single pixel wide lines. I figured they would be harmful for training.", "291268": "Hi Matthew, \nI think i found your issue. for init with coco you have: \n\n \nelif init_with == \"coco\":\n **model.load_weights(\"mask_rcnn_nuclei.h5\", by_name=True,**\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])\n\nyou need to use **\"\"mask_rcnn_coco.h5\"\"** . please try that and let us know if it improved", "292508": "Hi Matthew, just curious to know what fixed your issue with the low score ?", "294147": "Hi, Matthew, you should only add one class \"nuclei/nucleus\" using `self.add_class('nuclei', 1, 'nucleus')`, and delete the background, because the original code have already contained it.\nAnd I can't see your code on github now, it helped me a lot.", "298811": "*Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.* \nHi Matthew have meet the warning above, and how much time you spend in one epoch?", "298813": "Hi Konstantin ,how much time do you spend in one epoch when you use Maatterport's code", "308740": "Hi linCurry,\n\nCan I have your email? I have some questions about the competition. My email address is zhulinchao7@gmail.com.", "308753": "Sure, my email address is linsocurry@163.com" }, "source": "meta" }