{
  "id": 56238,
  "title": "10th place Code+Datasets (LB: 0.591) Mask R-CNN single model  ",
  "url": "/competitions/data-science-bowl-2018/writeups/gangadhar-payyavula-10th-place-code-datasets-lb-0-",
  "author_name": "",
  "post_date": "2018-05-08T00:08:32.020092500Z",
  "votes": 27,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Note: (currently 7th place while leaderboard is being updated) </p>\n\n<h2>TLDR:  Resources related to this project</h2>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/gangadhar/nuclei-segmentation-in-microscope-cell-images\">Data used to build the models</a></li>\n<li><a href=\"https://github.com/gangadhar-p/NucleiDetectron\">Code</a></li>\n<li><a href=\"https://github.com/gangadhar-p/NucleiDetectron\">Detailed Writeup</a></li>\n<li><a href=\"https://www.kaggle.com/gangadhar/nuclei-detectron-models-for-2018-data-science-bowl\">Pretrained model, predictions and visualization of submission</a></li>\n</ul>\n\n<h2>Preview of training data</h2>\n\n\n  <img src=\"https://i.imgur.com/YBdPS79.jpg\">\n  <p></p>\n\n\n<h2>Dataset preparation</h2>\n\n<ul>\n<li>There were several nuclei datasets with outlines as annotations.\n<ul><li>Applied classical computer vision techniques to convert ground truth from outlines to masks.</li>\n<li>This involved adding boundary pixels to the image so all contours are closed.</li>\n<li>Given outlines of cells with overlaps/touching or at border,\n<ul><li>Mark an outer contour to encompass contours that are at image edges.</li>\n<li>then do cv2.findContours to get the polygons of mask.</li>\n<li>Ref <a href=\"https://github.com/gangadhar-p/NucleiDetectron/blob/master/lib/datasets/nuclei/mask_encoding.py#L184\">parse_segments_from_outlines</a></li></ul></li></ul></li>\n<li>Standardized all datasets into COCO mask RLE JSON file format.\n<ul><li>You can use <a href=\"https://github.com/cocodataset/cocoapi\">cocoapi</a> to load the annotations.</li></ul></li>\n<li>Cut image into tiles when images are bigger than 1000 pixels\n<ul><li>This was necessary since large image features did not fit in GPU memory.</li></ul></li>\n</ul>\n\n<h2>Preprocessing</h2>\n\n<ul>\n<li>Cluster images into classes based on the color statistics.</li>\n<li>Normalize classes size\n<ul><li>Oversample/undersample images from clusters to a constant number of images per class in each epoch.</li></ul></li>\n<li>Fill holes in masks</li>\n<li>Split nuclei masks that are fused\n<ul><li>Applied morphological Erosion and Dilation to seperate fused cells</li>\n<li>Use statistics of nuclie sizes in an image to find outliers</li></ul></li>\n<li><a href=\"http://ufldl.stanford.edu/wiki/index.php/Whitening\">ZCA whitening of images</a></li>\n<li>Zero mean unit variance normalization</li>\n<li>Grey scale: <a href=\"http://tdlc.ucsd.edu/SV2013/Kanan_Cottrell_PLOS_Color_2012.pdf\">Color-to-Grayscale: Does the Method Matter in Image Recognition</a>.\n<ul><li>Very important how you convert to grey scale. Many algorithms for the conversion, loss of potential data.</li>\n<li>Luminous</li>\n<li>Intensity</li>\n<li>Value: This is the method I used.</li></ul></li>\n<li><a href=\"https://docs.opencv.org/3.1.0/d5/daf/tutorial_py_histogram_equalization.html\">Contrast Limited Adaptive Histogram Equalization</a></li>\n</ul>\n\n<h2>Augmentation</h2>\n\n<p>Data augmentation is one of the key to achieve good generalization in this challenge.</p>\n\n<h3>Training time augmentation</h3>\n\n<ul>\n<li>Invert\n<ul><li>This augmentation helped in reducing generalization error significantly</li>\n<li>Randomly choosing to invert caused the models to generalize across all kids of backgrounds in the local validation set.</li></ul></li>\n<li>Geometric\n<ul><li>PerspectiveTransform</li>\n<li>This is very useful to make the circular looking cells to look stretched</li>\n<li>PiecewiseAffine</li>\n<li>Flip</li>\n<li>Rotate (0, 90, 180, 270)</li>\n<li>Crop</li></ul></li>\n<li>Alpha blending\n<ul><li>Create geometrical blur by affine operation</li>\n<li>Shear, rotate, translate, scale</li></ul></li>\n<li>Pixel\n<ul><li>AddToHueAndSaturation</li>\n<li>Multiply</li>\n<li>Dropout, CoarseDropout</li>\n<li>ContrastNormalization</li></ul></li>\n<li>Noise\n<ul><li>AdditiveGaussianNoise</li>\n<li>SimplexNoiseAlpha</li>\n<li>FrequencyNoiseAlpha</li></ul></li>\n<li>Blur\n<ul><li>GaussianBlur</li>\n<li>AverageBlur</li>\n<li>MedianBlur</li>\n<li>BilateralBlur</li></ul></li>\n<li>Texture\n<ul><li>Superpixels</li>\n<li>Sharpen</li>\n<li>Emboss</li>\n<li>EdgeDetect</li>\n<li>DirectedEdgeDetect</li>\n<li>ElasticTransformation</li></ul></li>\n</ul>\n\n<h3>Test time augmentation</h3>\n\n<ol>\n<li>Invert: Have improved the performance a lot</li>\n<li>Multiple Scales 900, 1000, 1100</li>\n<li>Flip left right</li>\n</ol>\n\n<h2>Architecture changes to baseline Detectron</h2>\n\n<p>Detectron network configuration changes from the baseline e2e_mask_rcnn_X-152-32x8d-FPN-IN5k_1.44x.yaml are:</p>\n\n<ol>\n<li>Create small anchor sizes for small nuclei. RPN_ANCHOR_START_SIZE: 8 # default 32</li>\n<li>Add more aspect rations for nuclei that are close but in cylindrical structure. RPN_ASPECT_RATIOS: (0.2, 0.5, 1, 2, 5)</li>\n<li>Increase the ROI resolution. ROI_XFORM_RESOLUTION: 14</li>\n<li>Increase the number of detections per image from default 100. DETECTIONS_PER_IM: 500</li>\n</ol>\n\n<h2>Training</h2>\n\n<ol>\n<li>Decreased warmup fraction to 0.01</li>\n<li>Increased warmup iterations to 10,000</li>\n<li>Gave mask loss more weight WEIGHT_LOSS_MASK: 1.2</li>\n</ol>\n\n<h2>Segmentation Post processing</h2>\n\n<ul>\n<li>Threshold on area to remove masks below area of 15 pixels</li>\n<li>Threshold on BBox confidence of 0.9</li>\n<li>Mask NMS\n<ul><li>On decreasing order of confidence, simple union-mask strategy to remove overlapping segments or cut segments at overlaps if overlap is below 30% of the mask.</li></ul></li>\n</ul>\n\n<h2>What worked most</h2>\n\n<ol>\n<li>Inversion in augmentation</li>\n<li>Blurring and frequency noise</li>\n<li>Additional datasets, even though they caused a drop on the public leaderboard, I noticed no drop in local validation set.</li>\n</ol>\n\n<h2>What did not work</h2>\n\n<ol>\n<li>Mask dilations and erosions\n<ul><li>This did not have any improvement in the segmentation in my experiments</li></ul></li>\n<li>Use contour approximations in place of original masks\n<ul><li>This did not have any improvement either. Maybe this could add a boost if using light augmentations.</li></ul></li>\n<li>Randomly apply structuring like open-close</li>\n<li>Soft NMS thresh\n<ul><li>Did not improve accuracy</li></ul></li>\n<li>Color images\n<ul><li>Did not perform as well as grey images after augmentations</li></ul></li>\n<li>Color style transfer. Take a source image and apply the color style to target image.</li>\n<li>Style transfer: Was losing a lot of details on some nuclei but looked good on very few images.</li>\n<li>Dilation of masks in post processing, this drastically increased error because the model masks are already good.</li>\n<li>Distance transform and split masks during training.</li>\n</ol>\n\n<h2>Things I didn't have time to try</h2>\n\n<ol>\n<li>Ensemble multiple Mask R-CNN's</li>\n<li>Two stage predictions with U-Net after box proposals.</li>\n<li>Augmentation smoothing during training\n<ul><li>Increase the noise and augmentation slowly during the training phase, like from 10% to 50%</li>\n<li>Reduce the augmentation from 90% to 20% during training, for generalization and fitting.</li></ul></li>\n<li>Experiment with different levels of augmentation individually across, noise, blur, texture, alpha blending.</li>\n<li>Different layer normalization techniques, with batch size more than one image at a time. Need bigger GPU.</li>\n<li>Little bit of hyperparameter search on thresholds and network architecture.</li>\n</ol>\n\n<h2>Things I did not think of</h2>\n\n<p>U-Net with watershed, did not think this approach would outperform Mask R-CNN</p>\n\n<h2>Acknowledgements:</h2>\n\n<p>Kaggle community was a great source of inspiration and the discussions are very useful. Special thanks to Discussion Gradmaster <a href=\"https://www.kaggle.com/hengck23\">hengck23</a>.</p>\n\n<h2>Code References</h2>\n\n<ul>\n<li><p><a href=\"https://github.com/facebookresearch/detectron\">Detectron</a>.\nRoss Girshick and Ilija Radosavovic. Georgia Gkioxari. Piotr Doll\\'{a}r. Kaiming He.\nGithub, Jan. 2018.</p></li>\n<li><p><a href=\"https://github.com/aleju/imgaug\">Image augmentation for machine learning experiments</a>.\nAlexander Jung.\nGithub, Jan. 2015.</p></li>\n<li><p><a href=\"https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence\">Normalizing brightfield, stained and fluorescence</a>.\nKevin Mader.\nKaggle Notebook, Apr. 2018.</p></li>\n<li><p><a href=\"https://www.kaggle.com/stainsby/fast-tested-rle-and-input-routines\">Fast, tested RLE and input routines</a>.\nSam Stainsby.\nKaggle Notebook, Apr. 2018.</p></li>\n<li><p><a href=\"https://www.kaggle.com/wcukierski/example-metric-implementation\">Example Metric Implementation</a>.\nWilliam Cukierski.\nKaggle Notebook, Apr. 2018.</p></li>\n</ul>",
  "messages": [
    {
      "id": "324794",
      "postDate": "05/08/2018 00:08:32",
      "content": "<p>Note: (currently 7th place while leaderboard is being updated) </p>\n\n<h2>TLDR:  Resources related to this project</h2>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/gangadhar/nuclei-segmentation-in-microscope-cell-images\">Data used to build the models</a></li>\n<li><a href=\"https://github.com/gangadhar-p/NucleiDetectron\">Code</a></li>\n<li><a href=\"https://github.com/gangadhar-p/NucleiDetectron\">Detailed Writeup</a></li>\n<li><a href=\"https://www.kaggle.com/gangadhar/nuclei-detectron-models-for-2018-data-science-bowl\">Pretrained model, predictions and visualization of submission</a></li>\n</ul>\n\n<h2>Preview of training data</h2>\n\n\n  <img src=\"https://i.imgur.com/YBdPS79.jpg\">\n  <p></p>\n\n\n<h2>Dataset preparation</h2>\n\n<ul>\n<li>There were several nuclei datasets with outlines as annotations.\n<ul><li>Applied classical computer vision techniques to convert ground truth from outlines to masks.</li>\n<li>This involved adding boundary pixels to the image so all contours are closed.</li>\n<li>Given outlines of cells with overlaps/touching or at border,\n<ul><li>Mark an outer contour to encompass contours that are at image edges.</li>\n<li>then do cv2.findContours to get the polygons of mask.</li>\n<li>Ref <a href=\"https://github.com/gangadhar-p/NucleiDetectron/blob/master/lib/datasets/nuclei/mask_encoding.py#L184\">parse_segments_from_outlines</a></li></ul></li></ul></li>\n<li>Standardized all datasets into COCO mask RLE JSON file format.\n<ul><li>You can use <a href=\"https://github.com/cocodataset/cocoapi\">cocoapi</a> to load the annotations.</li></ul></li>\n<li>Cut image into tiles when images are bigger than 1000 pixels\n<ul><li>This was necessary since large image features did not fit in GPU memory.</li></ul></li>\n</ul>\n\n<h2>Preprocessing</h2>\n\n<ul>\n<li>Cluster images into classes based on the color statistics.</li>\n<li>Normalize classes size\n<ul><li>Oversample/undersample images from clusters to a constant number of images per class in each epoch.</li></ul></li>\n<li>Fill holes in masks</li>\n<li>Split nuclei masks that are fused\n<ul><li>Applied morphological Erosion and Dilation to seperate fused cells</li>\n<li>Use statistics of nuclie sizes in an image to find outliers</li></ul></li>\n<li><a href=\"http://ufldl.stanford.edu/wiki/index.php/Whitening\">ZCA whitening of images</a></li>\n<li>Zero mean unit variance normalization</li>\n<li>Grey scale: <a href=\"http://tdlc.ucsd.edu/SV2013/Kanan_Cottrell_PLOS_Color_2012.pdf\">Color-to-Grayscale: Does the Method Matter in Image Recognition</a>.\n<ul><li>Very important how you convert to grey scale. Many algorithms for the conversion, loss of potential data.</li>\n<li>Luminous</li>\n<li>Intensity</li>\n<li>Value: This is the method I used.</li></ul></li>\n<li><a href=\"https://docs.opencv.org/3.1.0/d5/daf/tutorial_py_histogram_equalization.html\">Contrast Limited Adaptive Histogram Equalization</a></li>\n</ul>\n\n<h2>Augmentation</h2>\n\n<p>Data augmentation is one of the key to achieve good generalization in this challenge.</p>\n\n<h3>Training time augmentation</h3>\n\n<ul>\n<li>Invert\n<ul><li>This augmentation helped in reducing generalization error significantly</li>\n<li>Randomly choosing to invert caused the models to generalize across all kids of backgrounds in the local validation set.</li></ul></li>\n<li>Geometric\n<ul><li>PerspectiveTransform</li>\n<li>This is very useful to make the circular looking cells to look stretched</li>\n<li>PiecewiseAffine</li>\n<li>Flip</li>\n<li>Rotate (0, 90, 180, 270)</li>\n<li>Crop</li></ul></li>\n<li>Alpha blending\n<ul><li>Create geometrical blur by affine operation</li>\n<li>Shear, rotate, translate, scale</li></ul></li>\n<li>Pixel\n<ul><li>AddToHueAndSaturation</li>\n<li>Multiply</li>\n<li>Dropout, CoarseDropout</li>\n<li>ContrastNormalization</li></ul></li>\n<li>Noise\n<ul><li>AdditiveGaussianNoise</li>\n<li>SimplexNoiseAlpha</li>\n<li>FrequencyNoiseAlpha</li></ul></li>\n<li>Blur\n<ul><li>GaussianBlur</li>\n<li>AverageBlur</li>\n<li>MedianBlur</li>\n<li>BilateralBlur</li></ul></li>\n<li>Texture\n<ul><li>Superpixels</li>\n<li>Sharpen</li>\n<li>Emboss</li>\n<li>EdgeDetect</li>\n<li>DirectedEdgeDetect</li>\n<li>ElasticTransformation</li></ul></li>\n</ul>\n\n<h3>Test time augmentation</h3>\n\n<ol>\n<li>Invert: Have improved the performance a lot</li>\n<li>Multiple Scales 900, 1000, 1100</li>\n<li>Flip left right</li>\n</ol>\n\n<h2>Architecture changes to baseline Detectron</h2>\n\n<p>Detectron network configuration changes from the baseline e2e_mask_rcnn_X-152-32x8d-FPN-IN5k_1.44x.yaml are:</p>\n\n<ol>\n<li>Create small anchor sizes for small nuclei. RPN_ANCHOR_START_SIZE: 8 # default 32</li>\n<li>Add more aspect rations for nuclei that are close but in cylindrical structure. RPN_ASPECT_RATIOS: (0.2, 0.5, 1, 2, 5)</li>\n<li>Increase the ROI resolution. ROI_XFORM_RESOLUTION: 14</li>\n<li>Increase the number of detections per image from default 100. DETECTIONS_PER_IM: 500</li>\n</ol>\n\n<h2>Training</h2>\n\n<ol>\n<li>Decreased warmup fraction to 0.01</li>\n<li>Increased warmup iterations to 10,000</li>\n<li>Gave mask loss more weight WEIGHT_LOSS_MASK: 1.2</li>\n</ol>\n\n<h2>Segmentation Post processing</h2>\n\n<ul>\n<li>Threshold on area to remove masks below area of 15 pixels</li>\n<li>Threshold on BBox confidence of 0.9</li>\n<li>Mask NMS\n<ul><li>On decreasing order of confidence, simple union-mask strategy to remove overlapping segments or cut segments at overlaps if overlap is below 30% of the mask.</li></ul></li>\n</ul>\n\n<h2>What worked most</h2>\n\n<ol>\n<li>Inversion in augmentation</li>\n<li>Blurring and frequency noise</li>\n<li>Additional datasets, even though they caused a drop on the public leaderboard, I noticed no drop in local validation set.</li>\n</ol>\n\n<h2>What did not work</h2>\n\n<ol>\n<li>Mask dilations and erosions\n<ul><li>This did not have any improvement in the segmentation in my experiments</li></ul></li>\n<li>Use contour approximations in place of original masks\n<ul><li>This did not have any improvement either. Maybe this could add a boost if using light augmentations.</li></ul></li>\n<li>Randomly apply structuring like open-close</li>\n<li>Soft NMS thresh\n<ul><li>Did not improve accuracy</li></ul></li>\n<li>Color images\n<ul><li>Did not perform as well as grey images after augmentations</li></ul></li>\n<li>Color style transfer. Take a source image and apply the color style to target image.</li>\n<li>Style transfer: Was losing a lot of details on some nuclei but looked good on very few images.</li>\n<li>Dilation of masks in post processing, this drastically increased error because the model masks are already good.</li>\n<li>Distance transform and split masks during training.</li>\n</ol>\n\n<h2>Things I didn't have time to try</h2>\n\n<ol>\n<li>Ensemble multiple Mask R-CNN's</li>\n<li>Two stage predictions with U-Net after box proposals.</li>\n<li>Augmentation smoothing during training\n<ul><li>Increase the noise and augmentation slowly during the training phase, like from 10% to 50%</li>\n<li>Reduce the augmentation from 90% to 20% during training, for generalization and fitting.</li></ul></li>\n<li>Experiment with different levels of augmentation individually across, noise, blur, texture, alpha blending.</li>\n<li>Different layer normalization techniques, with batch size more than one image at a time. Need bigger GPU.</li>\n<li>Little bit of hyperparameter search on thresholds and network architecture.</li>\n</ol>\n\n<h2>Things I did not think of</h2>\n\n<p>U-Net with watershed, did not think this approach would outperform Mask R-CNN</p>\n\n<h2>Acknowledgements:</h2>\n\n<p>Kaggle community was a great source of inspiration and the discussions are very useful. Special thanks to Discussion Gradmaster <a href=\"https://www.kaggle.com/hengck23\">hengck23</a>.</p>\n\n<h2>Code References</h2>\n\n<ul>\n<li><p><a href=\"https://github.com/facebookresearch/detectron\">Detectron</a>.\nRoss Girshick and Ilija Radosavovic. Georgia Gkioxari. Piotr Doll\\'{a}r. Kaiming He.\nGithub, Jan. 2018.</p></li>\n<li><p><a href=\"https://github.com/aleju/imgaug\">Image augmentation for machine learning experiments</a>.\nAlexander Jung.\nGithub, Jan. 2015.</p></li>\n<li><p><a href=\"https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence\">Normalizing brightfield, stained and fluorescence</a>.\nKevin Mader.\nKaggle Notebook, Apr. 2018.</p></li>\n<li><p><a href=\"https://www.kaggle.com/stainsby/fast-tested-rle-and-input-routines\">Fast, tested RLE and input routines</a>.\nSam Stainsby.\nKaggle Notebook, Apr. 2018.</p></li>\n<li><p><a href=\"https://www.kaggle.com/wcukierski/example-metric-implementation\">Example Metric Implementation</a>.\nWilliam Cukierski.\nKaggle Notebook, Apr. 2018.</p></li>\n</ul>",
      "rawMarkdown": "Note: (currently 7th place while leaderboard is being updated) \n\n## TLDR:  Resources related to this project\n* [Data used to build the models](https://www.kaggle.com/gangadhar/nuclei-segmentation-in-microscope-cell-images)\n* [Code](https://github.com/gangadhar-p/NucleiDetectron)\n* [Detailed Writeup](https://github.com/gangadhar-p/NucleiDetectron)\n* [Pretrained model, predictions and visualization of submission](https://www.kaggle.com/gangadhar/nuclei-detectron-models-for-2018-data-science-bowl)\n\n## Preview of training data\n<div>\n  <img src=\"https://i.imgur.com/YBdPS79.jpg\">\n  <p></p>\n</div>\n\n\n## Dataset preparation\n* There were several nuclei datasets with outlines as annotations.\n   * Applied classical computer vision techniques to convert ground truth from outlines to masks.\n   * This involved adding boundary pixels to the image so all contours are closed.\n   * Given outlines of cells with overlaps/touching or at border,\n      * Mark an outer contour to encompass contours that are at image edges.\n      * then do cv2.findContours to get the polygons of mask.\n      * Ref [parse_segments_from_outlines](https://github.com/gangadhar-p/NucleiDetectron/blob/master/lib/datasets/nuclei/mask_encoding.py#L184)\n* Standardized all datasets into COCO mask RLE JSON file format.\n   * You can use [cocoapi](https://github.com/cocodataset/cocoapi) to load the annotations.\n* Cut image into tiles when images are bigger than 1000 pixels\n   * This was necessary since large image features did not fit in GPU memory.\n\n## Preprocessing\n* Cluster images into classes based on the color statistics.\n* Normalize classes size\n   * Oversample/undersample images from clusters to a constant number of images per class in each epoch.\n* Fill holes in masks\n* Split nuclei masks that are fused\n   - Applied morphological Erosion and Dilation to seperate fused cells\n   - Use statistics of nuclie sizes in an image to find outliers\n*  [ZCA whitening of images](http://ufldl.stanford.edu/wiki/index.php/Whitening)\n*  Zero mean unit variance normalization\n*  Grey scale: [Color-to-Grayscale: Does the Method Matter in Image Recognition](http://tdlc.ucsd.edu/SV2013/Kanan_Cottrell_PLOS_Color_2012.pdf).\n   - Very important how you convert to grey scale. Many algorithms for the conversion, loss of potential data.\n   - Luminous\n   - Intensity\n   - Value: This is the method I used.\n*  [Contrast Limited Adaptive Histogram Equalization](https://docs.opencv.org/3.1.0/d5/daf/tutorial_py_histogram_equalization.html)\n\n## Augmentation\n\nData augmentation is one of the key to achieve good generalization in this challenge.\n\n### Training time augmentation\n\n* Invert\n  * This augmentation helped in reducing generalization error significantly\n  * Randomly choosing to invert caused the models to generalize across all kids of backgrounds in the local validation set.\n* Geometric\n  * PerspectiveTransform\n    * This is very useful to make the circular looking cells to look stretched\n  * PiecewiseAffine\n  * Flip\n  * Rotate (0, 90, 180, 270)\n  * Crop\n* Alpha blending\n  * Create geometrical blur by affine operation\n  * Shear, rotate, translate, scale\n* Pixel\n  * AddToHueAndSaturation\n  * Multiply\n  * Dropout, CoarseDropout\n  * ContrastNormalization\n* Noise\n  * AdditiveGaussianNoise\n  * SimplexNoiseAlpha\n  * FrequencyNoiseAlpha\n* Blur\n  * GaussianBlur\n  * AverageBlur\n  * MedianBlur\n  * BilateralBlur\n* Texture\n  * Superpixels\n  * Sharpen\n  * Emboss\n  * EdgeDetect\n  * DirectedEdgeDetect\n  * ElasticTransformation\n\n\n### Test time augmentation\n1. Invert: Have improved the performance a lot\n2. Multiple Scales 900, 1000, 1100\n3. Flip left right\n\n\n## Architecture changes to baseline Detectron\n\nDetectron network configuration changes from the baseline e2e_mask_rcnn_X-152-32x8d-FPN-IN5k_1.44x.yaml are:\n\n1. Create small anchor sizes for small nuclei. RPN_ANCHOR_START_SIZE: 8 # default 32\n2. Add more aspect rations for nuclei that are close but in cylindrical structure. RPN_ASPECT_RATIOS: (0.2, 0.5, 1, 2, 5)\n3. Increase the ROI resolution. ROI_XFORM_RESOLUTION: 14\n4. Increase the number of detections per image from default 100. DETECTIONS_PER_IM: 500\n\n## Training\n1. Decreased warmup fraction to 0.01\n2. Increased warmup iterations to 10,000\n3. Gave mask loss more weight WEIGHT_LOSS_MASK: 1.2\n\n## Segmentation Post processing\n  * Threshold on area to remove masks below area of 15 pixels\n  * Threshold on BBox confidence of 0.9\n  * Mask NMS\n    * On decreasing order of confidence, simple union-mask strategy to remove overlapping segments or cut segments at overlaps if overlap is below 30% of the mask.\n\n## What worked most\n1. Inversion in augmentation\n2. Blurring and frequency noise\n3. Additional datasets, even though they caused a drop on the public leaderboard, I noticed no drop in local validation set.\n\n## What did not work\n1. Mask dilations and erosions\n   * This did not have any improvement in the segmentation in my experiments\n2. Use contour approximations in place of original masks\n   * This did not have any improvement either. Maybe this could add a boost if using light augmentations.\n3. Randomly apply structuring like open-close\n4. Soft NMS thresh\n   * Did not improve accuracy\n5. Color images\n   * Did not perform as well as grey images after augmentations\n6. Color style transfer. Take a source image and apply the color style to target image.\n7. Style transfer: Was losing a lot of details on some nuclei but looked good on very few images.\n8. Dilation of masks in post processing, this drastically increased error because the model masks are already good.\n9. Distance transform and split masks during training.\n\n## Things I didn't have time to try\n1. Ensemble multiple Mask R-CNN's\n2. Two stage predictions with U-Net after box proposals.\n3. Augmentation smoothing during training\n   * Increase the noise and augmentation slowly during the training phase, like from 10% to 50%\n   * Reduce the augmentation from 90% to 20% during training, for generalization and fitting.\n4. Experiment with different levels of augmentation individually across, noise, blur, texture, alpha blending.\n5. Different layer normalization techniques, with batch size more than one image at a time. Need bigger GPU.\n6. Little bit of hyperparameter search on thresholds and network architecture.\n\n## Things I did not think of\nU-Net with watershed, did not think this approach would outperform Mask R-CNN\n\n## Acknowledgements:\nKaggle community was a great source of inspiration and the discussions are very useful. Special thanks to Discussion Gradmaster [hengck23](https://www.kaggle.com/hengck23).\n\n## Code References\n- [Detectron](https://github.com/facebookresearch/detectron).\n  Ross Girshick and Ilija Radosavovic. Georgia Gkioxari. Piotr Doll\\'{a}r. Kaiming He.\n  Github, Jan. 2018.\n\n- [Image augmentation for machine learning experiments](https://github.com/aleju/imgaug).\n  Alexander Jung.\n  Github, Jan. 2015.\n\n- [Normalizing brightfield, stained and fluorescence](https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence).\n  Kevin Mader.\n  Kaggle Notebook, Apr. 2018.\n\n- [Fast, tested RLE and input routines](https://www.kaggle.com/stainsby/fast-tested-rle-and-input-routines).\n  Sam Stainsby.\n  Kaggle Notebook, Apr. 2018.\n\n- [Example Metric Implementation](https://www.kaggle.com/wcukierski/example-metric-implementation).\n  William Cukierski.\n  Kaggle Notebook, Apr. 2018.",
      "votes": null
    },
    {
      "id": "324872",
      "postDate": "05/08/2018 00:59:30",
      "content": "<p>Thanks @Gangadhar Payyavula for sharing your solution. I have not had time to reproduce any of the top-10 solutions but when I get round to doing that, yours will be in my list. Thanks again.</p>",
      "rawMarkdown": "Thanks @Gangadhar Payyavula for sharing your solution. I have not had time to reproduce any of the top-10 solutions but when I get round to doing that, yours will be in my list. Thanks again.",
      "votes": null
    },
    {
      "id": "325118",
      "postDate": "05/08/2018 06:21:05",
      "content": "<p>Thanks @Gangadhar Payyavula for sharing!! Great summary!</p>",
      "rawMarkdown": "Thanks @Gangadhar Payyavula for sharing!! Great summary!",
      "votes": null
    },
    {
      "id": "325474",
      "postDate": "05/08/2018 12:59:55",
      "content": "<p>Thank you for your solution and detailed explanation. </p>",
      "rawMarkdown": "Thank you for your solution and detailed explanation.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 324872,
      "author_name": "sheriytm",
      "author_url": "",
      "post_date": "05/08/2018 00:59:30",
      "content": "<p>Thanks @Gangadhar Payyavula for sharing your solution. I have not had time to reproduce any of the top-10 solutions but when I get round to doing that, yours will be in my list. Thanks again.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 325118,
      "author_name": "petkovacs",
      "author_url": "",
      "post_date": "05/08/2018 06:21:05",
      "content": "<p>Thanks @Gangadhar Payyavula for sharing!! Great summary!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 325474,
      "author_name": "adslwang4603",
      "author_url": "",
      "post_date": "05/08/2018 12:59:55",
      "content": "<p>Thank you for your solution and detailed explanation. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "324794": "Note: (currently 7th place while leaderboard is being updated) \n\n## TLDR:  Resources related to this project\n* [Data used to build the models](https://www.kaggle.com/gangadhar/nuclei-segmentation-in-microscope-cell-images)\n* [Code](https://github.com/gangadhar-p/NucleiDetectron)\n* [Detailed Writeup](https://github.com/gangadhar-p/NucleiDetectron)\n* [Pretrained model, predictions and visualization of submission](https://www.kaggle.com/gangadhar/nuclei-detectron-models-for-2018-data-science-bowl)\n\n## Preview of training data\n<div>\n  <img src=\"https://i.imgur.com/YBdPS79.jpg\">\n  <p></p>\n</div>\n\n\n## Dataset preparation\n* There were several nuclei datasets with outlines as annotations.\n   * Applied classical computer vision techniques to convert ground truth from outlines to masks.\n   * This involved adding boundary pixels to the image so all contours are closed.\n   * Given outlines of cells with overlaps/touching or at border,\n      * Mark an outer contour to encompass contours that are at image edges.\n      * then do cv2.findContours to get the polygons of mask.\n      * Ref [parse_segments_from_outlines](https://github.com/gangadhar-p/NucleiDetectron/blob/master/lib/datasets/nuclei/mask_encoding.py#L184)\n* Standardized all datasets into COCO mask RLE JSON file format.\n   * You can use [cocoapi](https://github.com/cocodataset/cocoapi) to load the annotations.\n* Cut image into tiles when images are bigger than 1000 pixels\n   * This was necessary since large image features did not fit in GPU memory.\n\n## Preprocessing\n* Cluster images into classes based on the color statistics.\n* Normalize classes size\n   * Oversample/undersample images from clusters to a constant number of images per class in each epoch.\n* Fill holes in masks\n* Split nuclei masks that are fused\n   - Applied morphological Erosion and Dilation to seperate fused cells\n   - Use statistics of nuclie sizes in an image to find outliers\n*  [ZCA whitening of images](http://ufldl.stanford.edu/wiki/index.php/Whitening)\n*  Zero mean unit variance normalization\n*  Grey scale: [Color-to-Grayscale: Does the Method Matter in Image Recognition](http://tdlc.ucsd.edu/SV2013/Kanan_Cottrell_PLOS_Color_2012.pdf).\n   - Very important how you convert to grey scale. Many algorithms for the conversion, loss of potential data.\n   - Luminous\n   - Intensity\n   - Value: This is the method I used.\n*  [Contrast Limited Adaptive Histogram Equalization](https://docs.opencv.org/3.1.0/d5/daf/tutorial_py_histogram_equalization.html)\n\n## Augmentation\n\nData augmentation is one of the key to achieve good generalization in this challenge.\n\n### Training time augmentation\n\n* Invert\n  * This augmentation helped in reducing generalization error significantly\n  * Randomly choosing to invert caused the models to generalize across all kids of backgrounds in the local validation set.\n* Geometric\n  * PerspectiveTransform\n    * This is very useful to make the circular looking cells to look stretched\n  * PiecewiseAffine\n  * Flip\n  * Rotate (0, 90, 180, 270)\n  * Crop\n* Alpha blending\n  * Create geometrical blur by affine operation\n  * Shear, rotate, translate, scale\n* Pixel\n  * AddToHueAndSaturation\n  * Multiply\n  * Dropout, CoarseDropout\n  * ContrastNormalization\n* Noise\n  * AdditiveGaussianNoise\n  * SimplexNoiseAlpha\n  * FrequencyNoiseAlpha\n* Blur\n  * GaussianBlur\n  * AverageBlur\n  * MedianBlur\n  * BilateralBlur\n* Texture\n  * Superpixels\n  * Sharpen\n  * Emboss\n  * EdgeDetect\n  * DirectedEdgeDetect\n  * ElasticTransformation\n\n\n### Test time augmentation\n1. Invert: Have improved the performance a lot\n2. Multiple Scales 900, 1000, 1100\n3. Flip left right\n\n\n## Architecture changes to baseline Detectron\n\nDetectron network configuration changes from the baseline e2e_mask_rcnn_X-152-32x8d-FPN-IN5k_1.44x.yaml are:\n\n1. Create small anchor sizes for small nuclei. RPN_ANCHOR_START_SIZE: 8 # default 32\n2. Add more aspect rations for nuclei that are close but in cylindrical structure. RPN_ASPECT_RATIOS: (0.2, 0.5, 1, 2, 5)\n3. Increase the ROI resolution. ROI_XFORM_RESOLUTION: 14\n4. Increase the number of detections per image from default 100. DETECTIONS_PER_IM: 500\n\n## Training\n1. Decreased warmup fraction to 0.01\n2. Increased warmup iterations to 10,000\n3. Gave mask loss more weight WEIGHT_LOSS_MASK: 1.2\n\n## Segmentation Post processing\n  * Threshold on area to remove masks below area of 15 pixels\n  * Threshold on BBox confidence of 0.9\n  * Mask NMS\n    * On decreasing order of confidence, simple union-mask strategy to remove overlapping segments or cut segments at overlaps if overlap is below 30% of the mask.\n\n## What worked most\n1. Inversion in augmentation\n2. Blurring and frequency noise\n3. Additional datasets, even though they caused a drop on the public leaderboard, I noticed no drop in local validation set.\n\n## What did not work\n1. Mask dilations and erosions\n   * This did not have any improvement in the segmentation in my experiments\n2. Use contour approximations in place of original masks\n   * This did not have any improvement either. Maybe this could add a boost if using light augmentations.\n3. Randomly apply structuring like open-close\n4. Soft NMS thresh\n   * Did not improve accuracy\n5. Color images\n   * Did not perform as well as grey images after augmentations\n6. Color style transfer. Take a source image and apply the color style to target image.\n7. Style transfer: Was losing a lot of details on some nuclei but looked good on very few images.\n8. Dilation of masks in post processing, this drastically increased error because the model masks are already good.\n9. Distance transform and split masks during training.\n\n## Things I didn't have time to try\n1. Ensemble multiple Mask R-CNN's\n2. Two stage predictions with U-Net after box proposals.\n3. Augmentation smoothing during training\n   * Increase the noise and augmentation slowly during the training phase, like from 10% to 50%\n   * Reduce the augmentation from 90% to 20% during training, for generalization and fitting.\n4. Experiment with different levels of augmentation individually across, noise, blur, texture, alpha blending.\n5. Different layer normalization techniques, with batch size more than one image at a time. Need bigger GPU.\n6. Little bit of hyperparameter search on thresholds and network architecture.\n\n## Things I did not think of\nU-Net with watershed, did not think this approach would outperform Mask R-CNN\n\n## Acknowledgements:\nKaggle community was a great source of inspiration and the discussions are very useful. Special thanks to Discussion Gradmaster [hengck23](https://www.kaggle.com/hengck23).\n\n## Code References\n- [Detectron](https://github.com/facebookresearch/detectron).\n  Ross Girshick and Ilija Radosavovic. Georgia Gkioxari. Piotr Doll\\'{a}r. Kaiming He.\n  Github, Jan. 2018.\n\n- [Image augmentation for machine learning experiments](https://github.com/aleju/imgaug).\n  Alexander Jung.\n  Github, Jan. 2015.\n\n- [Normalizing brightfield, stained and fluorescence](https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence).\n  Kevin Mader.\n  Kaggle Notebook, Apr. 2018.\n\n- [Fast, tested RLE and input routines](https://www.kaggle.com/stainsby/fast-tested-rle-and-input-routines).\n  Sam Stainsby.\n  Kaggle Notebook, Apr. 2018.\n\n- [Example Metric Implementation](https://www.kaggle.com/wcukierski/example-metric-implementation).\n  William Cukierski.\n  Kaggle Notebook, Apr. 2018.",
    "324872": "Thanks @Gangadhar Payyavula for sharing your solution. I have not had time to reproduce any of the top-10 solutions but when I get round to doing that, yours will be in my list. Thanks again.",
    "325118": "Thanks @Gangadhar Payyavula for sharing!! Great summary!",
    "325474": "Thank you for your solution and detailed explanation."
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 8089,
  "competition_slug": "data-science-bowl-2018",
  "team_id": 1264614,
  "solution_url": "/competitions/data-science-bowl-2018/writeups/gangadhar-payyavula-10th-place-code-datasets-lb-0-"
}