Hi All,
\n\nI am trying to learn which are the most significant parameters for Matterport Mask RCNN. \nI made the following document to compare the parameters across contestants.
\n\n\n\nSome things are still missing like learning schedule and image pre-processing. I will continue updating it. If you would like to add your parameters you are most welcome to do so! You can paste your parameters here or send me a pull-request.
\n\nThanks\nMehul
", "messages": [ { "id": "316698", "postDate": "04/19/2018 17:32:20", "content": "Hi All,
\n\nI am trying to learn which are the most significant parameters for Matterport Mask RCNN. \nI made the following document to compare the parameters across contestants.
\n\n\n\nSome things are still missing like learning schedule and image pre-processing. I will continue updating it. If you would like to add your parameters you are most welcome to do so! You can paste your parameters here or send me a pull-request.
\n\nThanks\nMehul
", "rawMarkdown": "Hi All,\n\nI am trying to learn which are the most significant parameters for Matterport Mask RCNN. \nI made the following document to compare the parameters across contestants. \n\nhttps://github.com/mpsampat/DSB_2018_Parameter_Selection_For_Matterport_Mask_RCNN/blob/master/comparison_of_Matterport_Mask_RCNN_parameters.md\n\nSome things are still missing like learning schedule and image pre-processing. I will continue updating it. If you would like to add your parameters you are most welcome to do so! You can paste your parameters here or send me a pull-request.\n\nThanks\nMehul", "votes": null }, { "id": "316824", "postDate": "04/20/2018 02:15:13", "content": "Hi, Mehul. Nice work! It's so amazing! Avi's parameter is so different.
", "rawMarkdown": "Hi, Mehul. Nice work! It's so amazing! Avi's parameter is so different.", "votes": null }, { "id": "316991", "postDate": "04/20/2018 11:20:11", "content": "Hi Mehul, I just added my parameters as a PR. For stage 2 score, I added the late submission score from the same model that generated my stage 1 LB score (so that they are comparable), rather than the lower stage 2 LB score after retraining with stage 1 test labels. The table is getting crowded, though...
", "rawMarkdown": "Hi Mehul, I just added my parameters as a PR. For stage 2 score, I added the late submission score from the same model that generated my stage 1 LB score (so that they are comparable), rather than the lower stage 2 LB score after retraining with stage 1 test labels. The table is getting crowded, though...", "votes": null }, { "id": "317227", "postDate": "04/21/2018 02:40:53", "content": "Thanks Mehul! Suggest adding \"Image Preprocessing / Normalization\" and \"Image Postprocessing\" rows. I suspect Zheng's Z-Score style preprocessing may have made a positive difference because of variations in dimness, and his binary dilation postprocessing may have helped better match ground truth human annotation.
", "rawMarkdown": "Thanks Mehul! Suggest adding \"Image Preprocessing / Normalization\" and \"Image Postprocessing\" rows. I suspect Zheng's Z-Score style preprocessing may have made a positive difference because of variations in dimness, and his binary dilation postprocessing may have helped better match ground truth human annotation.", "votes": null }, { "id": "317251", "postDate": "04/21/2018 03:54:16", "content": "Good idea Russ! Will add both items. Yes i think the binary dilation definitely helped. I think there one other contestant who got 0.591 with Mask RCNN and i think he also used binary dilation.
\n\nAlso, Waleed who wrote Matterport Mask RCNN is my friend. He made various updates to the code during the competition. One other weird trend from my table is that contestants who used the older version of the code seem to have a higher score (0.53+). I am still trying to figure out if it is due to parameter selection or due to some specific updates in the code.
\n\nLastly, I think many of us had a hard time with reproducibility across Mask RCNN experiments. \nI think the solution from the MPWARE team (ensembling of Mask-RCNN) might be the most elegant solution to handle this issue.
", "rawMarkdown": "Good idea Russ! Will add both items. Yes i think the binary dilation definitely helped. I think there one other contestant who got 0.591 with Mask RCNN and i think he also used binary dilation. \n\nAlso, Waleed who wrote Matterport Mask RCNN is my friend. He made various updates to the code during the competition. One other weird trend from my table is that contestants who used the older version of the code seem to have a higher score (0.53+). I am still trying to figure out if it is due to parameter selection or due to some specific updates in the code. \n\nLastly, I think many of us had a hard time with reproducibility across Mask RCNN experiments. \nI think the solution from the MPWARE team (ensembling of Mask-RCNN) might be the most elegant solution to handle this issue.", "votes": null }, { "id": "317400", "postDate": "04/21/2018 12:48:02", "content": "Thank you! This will help us learn more about the config
", "rawMarkdown": "Thank you! This will help us learn more about the config", "votes": null }, { "id": "317434", "postDate": "04/21/2018 15:07:54", "content": "Thanks Mehul for the comparison, also interested to better understand how to tune the maskRcnn implementation.
", "rawMarkdown": "Thanks Mehul for the comparison, also interested to better understand how to tune the maskRcnn implementation.", "votes": null }, { "id": "317545", "postDate": "04/21/2018 20:35:55", "content": "Agreed. I think post-processing is especially important for LB scores. Heng and many others suggested that filtering out FP is key to high LB scores, and this is also part of Zheng's pipeline. The performance gain from pre- and post-processing may have larger effects than fine-tuning the parameters of Mask RCNN.
\n\nBy visualizing the masks generated from Mask RCNN, I think minimal tuning of the model parameters can give us a set of very good mask candidates. Although I did not implement successful post-processing techniques, I think setting a low detection confidence cutoff to ensure low FN + sophisticated post-processing to filter out FP is probably the way to go (of course there is another step to process the mask shapes).
\n\nIn general, nuclei types that are over-represented in the training set or are easy to segment gets very high confidence score (0.99+) from the model, and the ones that are under-represented in the training set (some purple nuclei) and those with complicated morphology gets assigned lower confidence scores (0.7-0.9). The detection confidence cutoff sets a trade-off between FP and FN. If we set the cutoff at 0.5 - 0.6, we have very low FN for nuclei with complicated morphology, but also high FP. A cutoff at 0.85 - 0.9 gives much lower FP and overall good LB scores, but also high FN for some nuclei types. So a fixed detection confidence cutoff without post-processing cannot guarantee that all different types of seen and unseen nuclei images are segmented equally well. And when the test set data distribution is different from training and validation set, relying on the model itself to decide which nuclei are true positives may lead to the high variability of Mask-RCNN LB scores. If I had more time for the competition, I would've invested a lot more on post-processing:)
", "rawMarkdown": "Agreed. I think post-processing is especially important for LB scores. Heng and many others suggested that filtering out FP is key to high LB scores, and this is also part of Zheng's pipeline. The performance gain from pre- and post-processing may have larger effects than fine-tuning the parameters of Mask RCNN. \n\nBy visualizing the masks generated from Mask RCNN, I think minimal tuning of the model parameters can give us a set of very good mask candidates. Although I did not implement successful post-processing techniques, I think setting a low detection confidence cutoff to ensure low FN + sophisticated post-processing to filter out FP is probably the way to go (of course there is another step to process the mask shapes). \n\nIn general, nuclei types that are over-represented in the training set or are easy to segment gets very high confidence score (0.99+) from the model, and the ones that are under-represented in the training set (some purple nuclei) and those with complicated morphology gets assigned lower confidence scores (0.7-0.9). The detection confidence cutoff sets a trade-off between FP and FN. If we set the cutoff at 0.5 - 0.6, we have very low FN for nuclei with complicated morphology, but also high FP. A cutoff at 0.85 - 0.9 gives much lower FP and overall good LB scores, but also high FN for some nuclei types. So a fixed detection confidence cutoff without post-processing cannot guarantee that all different types of seen and unseen nuclei images are segmented equally well. And when the test set data distribution is different from training and validation set, relying on the model itself to decide which nuclei are true positives may lead to the high variability of Mask-RCNN LB scores. If I had more time for the competition, I would've invested a lot more on post-processing:)", "votes": null } ], "comments": [ { "id": 316824, "author_name": "mdlszhengli", "author_url": "", "post_date": "04/20/2018 02:15:13", "content": "Hi, Mehul. Nice work! It's so amazing! Avi's parameter is so different.
", "votes": null, "replies": [] }, { "id": 316991, "author_name": "yueqiw", "author_url": "", "post_date": "04/20/2018 11:20:11", "content": "Hi Mehul, I just added my parameters as a PR. For stage 2 score, I added the late submission score from the same model that generated my stage 1 LB score (so that they are comparable), rather than the lower stage 2 LB score after retraining with stage 1 test labels. The table is getting crowded, though...
", "votes": null, "replies": [] }, { "id": 317227, "author_name": "sasrdw", "author_url": "", "post_date": "04/21/2018 02:40:53", "content": "Thanks Mehul! Suggest adding \"Image Preprocessing / Normalization\" and \"Image Postprocessing\" rows. I suspect Zheng's Z-Score style preprocessing may have made a positive difference because of variations in dimness, and his binary dilation postprocessing may have helped better match ground truth human annotation.
", "votes": null, "replies": [ { "id": 317251, "author_name": "mpsampat", "author_url": "", "post_date": "04/21/2018 03:54:16", "content": "Good idea Russ! Will add both items. Yes i think the binary dilation definitely helped. I think there one other contestant who got 0.591 with Mask RCNN and i think he also used binary dilation.
\n\nAlso, Waleed who wrote Matterport Mask RCNN is my friend. He made various updates to the code during the competition. One other weird trend from my table is that contestants who used the older version of the code seem to have a higher score (0.53+). I am still trying to figure out if it is due to parameter selection or due to some specific updates in the code.
\n\nLastly, I think many of us had a hard time with reproducibility across Mask RCNN experiments. \nI think the solution from the MPWARE team (ensembling of Mask-RCNN) might be the most elegant solution to handle this issue.
", "votes": null, "replies": [] }, { "id": 317545, "author_name": "yueqiw", "author_url": "", "post_date": "04/21/2018 20:35:55", "content": "Agreed. I think post-processing is especially important for LB scores. Heng and many others suggested that filtering out FP is key to high LB scores, and this is also part of Zheng's pipeline. The performance gain from pre- and post-processing may have larger effects than fine-tuning the parameters of Mask RCNN.
\n\nBy visualizing the masks generated from Mask RCNN, I think minimal tuning of the model parameters can give us a set of very good mask candidates. Although I did not implement successful post-processing techniques, I think setting a low detection confidence cutoff to ensure low FN + sophisticated post-processing to filter out FP is probably the way to go (of course there is another step to process the mask shapes).
\n\nIn general, nuclei types that are over-represented in the training set or are easy to segment gets very high confidence score (0.99+) from the model, and the ones that are under-represented in the training set (some purple nuclei) and those with complicated morphology gets assigned lower confidence scores (0.7-0.9). The detection confidence cutoff sets a trade-off between FP and FN. If we set the cutoff at 0.5 - 0.6, we have very low FN for nuclei with complicated morphology, but also high FP. A cutoff at 0.85 - 0.9 gives much lower FP and overall good LB scores, but also high FN for some nuclei types. So a fixed detection confidence cutoff without post-processing cannot guarantee that all different types of seen and unseen nuclei images are segmented equally well. And when the test set data distribution is different from training and validation set, relying on the model itself to decide which nuclei are true positives may lead to the high variability of Mask-RCNN LB scores. If I had more time for the competition, I would've invested a lot more on post-processing:)
", "votes": null, "replies": [] } ] }, { "id": 317400, "author_name": "joek47", "author_url": "", "post_date": "04/21/2018 12:48:02", "content": "Thank you! This will help us learn more about the config
", "votes": null, "replies": [] }, { "id": 317434, "author_name": "alexec", "author_url": "", "post_date": "04/21/2018 15:07:54", "content": "Thanks Mehul for the comparison, also interested to better understand how to tune the maskRcnn implementation.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "316698": "Hi All,\n\nI am trying to learn which are the most significant parameters for Matterport Mask RCNN. \nI made the following document to compare the parameters across contestants. \n\nhttps://github.com/mpsampat/DSB_2018_Parameter_Selection_For_Matterport_Mask_RCNN/blob/master/comparison_of_Matterport_Mask_RCNN_parameters.md\n\nSome things are still missing like learning schedule and image pre-processing. I will continue updating it. If you would like to add your parameters you are most welcome to do so! You can paste your parameters here or send me a pull-request.\n\nThanks\nMehul", "316824": "Hi, Mehul. Nice work! It's so amazing! Avi's parameter is so different.", "316991": "Hi Mehul, I just added my parameters as a PR. For stage 2 score, I added the late submission score from the same model that generated my stage 1 LB score (so that they are comparable), rather than the lower stage 2 LB score after retraining with stage 1 test labels. The table is getting crowded, though...", "317227": "Thanks Mehul! Suggest adding \"Image Preprocessing / Normalization\" and \"Image Postprocessing\" rows. I suspect Zheng's Z-Score style preprocessing may have made a positive difference because of variations in dimness, and his binary dilation postprocessing may have helped better match ground truth human annotation.", "317251": "Good idea Russ! Will add both items. Yes i think the binary dilation definitely helped. I think there one other contestant who got 0.591 with Mask RCNN and i think he also used binary dilation. \n\nAlso, Waleed who wrote Matterport Mask RCNN is my friend. He made various updates to the code during the competition. One other weird trend from my table is that contestants who used the older version of the code seem to have a higher score (0.53+). I am still trying to figure out if it is due to parameter selection or due to some specific updates in the code. \n\nLastly, I think many of us had a hard time with reproducibility across Mask RCNN experiments. \nI think the solution from the MPWARE team (ensembling of Mask-RCNN) might be the most elegant solution to handle this issue.", "317400": "Thank you! This will help us learn more about the config", "317434": "Thanks Mehul for the comparison, also interested to better understand how to tune the maskRcnn implementation.", "317545": "Agreed. I think post-processing is especially important for LB scores. Heng and many others suggested that filtering out FP is key to high LB scores, and this is also part of Zheng's pipeline. The performance gain from pre- and post-processing may have larger effects than fine-tuning the parameters of Mask RCNN. \n\nBy visualizing the masks generated from Mask RCNN, I think minimal tuning of the model parameters can give us a set of very good mask candidates. Although I did not implement successful post-processing techniques, I think setting a low detection confidence cutoff to ensure low FN + sophisticated post-processing to filter out FP is probably the way to go (of course there is another step to process the mask shapes). \n\nIn general, nuclei types that are over-represented in the training set or are easy to segment gets very high confidence score (0.99+) from the model, and the ones that are under-represented in the training set (some purple nuclei) and those with complicated morphology gets assigned lower confidence scores (0.7-0.9). The detection confidence cutoff sets a trade-off between FP and FN. If we set the cutoff at 0.5 - 0.6, we have very low FN for nuclei with complicated morphology, but also high FP. A cutoff at 0.85 - 0.9 gives much lower FP and overall good LB scores, but also high FN for some nuclei types. So a fixed detection confidence cutoff without post-processing cannot guarantee that all different types of seen and unseen nuclei images are segmented equally well. And when the test set data distribution is different from training and validation set, relying on the model itself to decide which nuclei are true positives may lead to the high variability of Mask-RCNN LB scores. If I had more time for the competition, I would've invested a lot more on post-processing:)" }, "source": "meta" }