![\"weighted](\"https://raw.githubusercontent.com/samuelschen/DSB2018/master/docs/weight_map.jpg\")
First of all, congrats to the DSB 2018 winners! This is our first ever Kaggle competition, and we indeed learned so much through out the whole event. Although our result is not among the best, but we are satisfied with our top 1.5% finish as the first Kaggle attempt. (Team name: 42) Would like to conclude this journey with the sharing of our learning in hope that it might be useful for future novice contestants like us. We documented our journey in a sequential development log style. [Github Code] (https://github.com/samuelschen/DSB2018)
\n\nIn this competition, we struggled on lack of validation criteria during the stage 2, and ended up submitting worse results than we should. A few days after the competition deadline, we were able to achieve 0.623 LB score using the same model with the exception of changing to pre-trained ResNet and fixing our broken TTA codes. I honestly don't think we would have chosen these best results even if we had it before stage 2 ended. So our question is: how did the leading teams choose which 2 results to submit during stage 2 without knowing the real final test2 dataset? Would love to hear everyone's opinion.
\n\nAfter finishing several machine learning online courses and trying out past Kaggle competitions, we are excited about the promise of deep learning. We later decided to participate DSB 2018 in order to benchmark our own capability in machine learning. It led us to participate Data Science Bowl 2018 as our 1st Kaggle competiton.
\n\nIt's emphasized that 'generalizability' is the key goal of this competition. Given the small amount of training set and its diversity, we decided that we'll need lots of data augmentation in data pipeline. Furthermore, easy debugging and integration with existing Python modules would also be important. PyTorch sounds to us a better choice than TensorFlow for this matter; we decided to use DSB 2018 as our first PyTorch experiment.
\n\nBoth MaskRCNN and UNet appear to be reasonable approaches for the segmentation problem. In favor of its simplicity, we picked UNet and built our own code from scratch, also did experiments of data augmentation (random crop, flip, rotate, resize, color jitter, elasitc distortion, color invert, clahe, and gaussian noise). We believed our heavy data augmentation can be helpful in general, but we also learned that some effects did not yield desired improvements, e.g. color invert, clahe, and gaussian noise.
\n\nThis vanilla UNet produced Public LB score of 0.34, trained without external dataset, used Binary Cross-Entropy (BCE) loss function. Then loss function was changed to Jaccard/IoU + BCE, it helped to improve Public LB score to 0.36.
\n\nReviewed the output visualization of our vanilla UNet, single binary output obviously not very effective against separating the overlapping nuclei. We started to look for methods to improve vanilla UNet model.
\n\nThe first idea was transfer learning. A well pre-trained model should mitigate data quantity issue and provide richer low level features as first part of UNet model. We tried VGG16, but unfortunately, no improvement was seen at that time. (note: we revisited transfer learning after competition with the adoption of ResNet)
\n\nIn early experiments of computer vision morphology algorithms, watershed with peak local max, score reached Public LB 0.41. This confirmed that the competition was indeed an instance segmentation problem.
\n\nWe found the model performed poorly in visualized prediction of histology images, besides model improvement, external datasets were added to training. At this moment Public LB score reached 0.44
\n\nInspired by DCAN, we revised the UNet model to be multitasked. One head is for semantic prediction, another head is for contour prediction. However, a naive (semantic - contour) as instance segmentation prediction is not good, so we instead used (semantic - contour) as the marker of watershed algorithm, and it brought the Public LB score to 0.45
\n\nRevisited UNet paper, and felt that using 'weight map' to force the network to learn the border pixels could be helpful, we then implemented a weight map, which emphasizes on borders & centroids (especially for small cells), and it brought us to Public LB 0.47.
\n\n
Comparing the partitioning results of watershed and random walker for touching objects, we felt that the result of \"random walker\" was more natural from our perspective. Changed the post-processing to \"random walker\", it improved Public LB score to 0.49.
\n\nReviewed aforementioned markers (semantic - contour) visually, there are always some poor performing cases and it's hard to design rules for them. Therefore, we decided to let the model learn and predict the markers by itself. The third head was added to predict markers, whose ground truths are derived from shrunk cell masks. We used ('markers head' - 'contour head') as the final markers for random walker to partition the 'semantic head'.
\n\n![\"3head](\"https://raw.githubusercontent.com/samuelschen/DSB2018/master/docs/model_output.jpg\")
In order to address the data imbalance issue, we also implemented Focal loss in addition to aforementioned Jaccard/IoU loss function and weight map mechanism. It further improved our Public LB score to 0.50.
\n\nBy adding synthesized images of touching/overlapping cells to training set, the Public LB improved to 0.52, ranked #22 at that moment.
\n\nWe analyzed error cases of stage 1 test data, one of major issues was large cells (scaling), the other one was lack of similar images in Kaggle training dataset.
\n\nTo address scaling issue, we tried to expand the receptive field of UNet's encoder part with \"dilated convolution.\" We found dilated and non-dilated encoder seems to be complement each other well, and we then ensembled these two models by averaging their pixel-wise prediction of three heads.
\n\nBack and forth, we spent lots effort in collecting and validating external dataset to be included. With data manipulation, it brought the scrore to Public LB 0.534, but it's tedious and required some biology knowledge which we didn't have. :-(
\n\nAlso Test Time Augmentation (TTA) was experimented by horizontal flip, vertical flip, horizontal and vertical flip. However, an implementation error on reflection padding & flip pipeline misled us to drop TTA before end of competition (Ouch! we revisited and fixed TTA after the competition)
\n\n![\"data](\"https://raw.githubusercontent.com/samuelschen/DSB2018/master/docs/data_variation.jpg\")
Surprised by the huge differences between stage1 and stage 2 test datasets, we panicked! Several mistakes were made during the final few days, unfortunately. We added additional BBBC018, BBBC020, and stage 1 test to further train the models. That was the 1st mistake to put everything in training, resulting no objective local cross validataion to judge overfitting, but relied on human evaluation of visualized predictions. (note: see how we learned after competition)
\n\nWe thought the longer our model was trained the better score it could be based on the learning curve observed in stage 1, (:facepalm: another mistake mentioned in post-competition), yet it turned out that we chose the worse ones on final submission. Our major struggle in final week was that no validation set similar to the test set (Andrew Ng had a great talk about Nuts and Bolts of Applying Deep Learning in this regard), so we solely relied on human to evaluate the visualized predictions (and many of them are on-purposed poisoned!).
\n\nReading some nice writeups of top winning UNet-based solutions when competition ended (hats off to all winners for sharing!) Indeed, we had many technical approaches in common, yet a few practices we did not do right or well. For example:
\n\nLast but not least, we used stage 1 test set as validation set (totally isolated from training set), and saw the high variance of Private LB score (0.56 ~ 0.623) along the training epochs.
\n\n![\"stage](\"https://raw.githubusercontent.com/samuelschen/DSB2018/master/docs/overfit-stage-2.jpg\")
The figure suggests that the data distribution of stage 1 test set is likely very different than stage 2 test set, it's somehow like a lottery here since we can't have a reasonable local validation set for stage 2 test set (with tons of on-purposed poison images). The fluctuation might also hint the number of images used in stage 2 scoring can be relatively small.
\n\nUnfortunately, this remains to be a puzzle for us. We felt a 'generalized' model should perform well on both stage 1 test set and stage 2 test set. Without an objective validation criteria, we don't believe that we could have chosen our best 0.623 result to be our final submission.
\n\nLove to hear from experienced Kagglers for any suggestions/practices in this kind of 2-stage competition! Good luck to everyone for future Kaggle competitions.
\n\nIf you are interested in our code -\n- Github Code
", "messages": [ { "id": "334622", "postDate": "05/28/2018 03:25:33", "content": "First of all, congrats to the DSB 2018 winners! This is our first ever Kaggle competition, and we indeed learned so much through out the whole event. Although our result is not among the best, but we are satisfied with our top 1.5% finish as the first Kaggle attempt. (Team name: 42) Would like to conclude this journey with the sharing of our learning in hope that it might be useful for future novice contestants like us. We documented our journey in a sequential development log style. [Github Code] (https://github.com/samuelschen/DSB2018)
\n\nIn this competition, we struggled on lack of validation criteria during the stage 2, and ended up submitting worse results than we should. A few days after the competition deadline, we were able to achieve 0.623 LB score using the same model with the exception of changing to pre-trained ResNet and fixing our broken TTA codes. I honestly don't think we would have chosen these best results even if we had it before stage 2 ended. So our question is: how did the leading teams choose which 2 results to submit during stage 2 without knowing the real final test2 dataset? Would love to hear everyone's opinion.
\n\nAfter finishing several machine learning online courses and trying out past Kaggle competitions, we are excited about the promise of deep learning. We later decided to participate DSB 2018 in order to benchmark our own capability in machine learning. It led us to participate Data Science Bowl 2018 as our 1st Kaggle competiton.
\n\nIt's emphasized that 'generalizability' is the key goal of this competition. Given the small amount of training set and its diversity, we decided that we'll need lots of data augmentation in data pipeline. Furthermore, easy debugging and integration with existing Python modules would also be important. PyTorch sounds to us a better choice than TensorFlow for this matter; we decided to use DSB 2018 as our first PyTorch experiment.
\n\nBoth MaskRCNN and UNet appear to be reasonable approaches for the segmentation problem. In favor of its simplicity, we picked UNet and built our own code from scratch, also did experiments of data augmentation (random crop, flip, rotate, resize, color jitter, elasitc distortion, color invert, clahe, and gaussian noise). We believed our heavy data augmentation can be helpful in general, but we also learned that some effects did not yield desired improvements, e.g. color invert, clahe, and gaussian noise.
\n\nThis vanilla UNet produced Public LB score of 0.34, trained without external dataset, used Binary Cross-Entropy (BCE) loss function. Then loss function was changed to Jaccard/IoU + BCE, it helped to improve Public LB score to 0.36.
\n\nReviewed the output visualization of our vanilla UNet, single binary output obviously not very effective against separating the overlapping nuclei. We started to look for methods to improve vanilla UNet model.
\n\nThe first idea was transfer learning. A well pre-trained model should mitigate data quantity issue and provide richer low level features as first part of UNet model. We tried VGG16, but unfortunately, no improvement was seen at that time. (note: we revisited transfer learning after competition with the adoption of ResNet)
\n\nIn early experiments of computer vision morphology algorithms, watershed with peak local max, score reached Public LB 0.41. This confirmed that the competition was indeed an instance segmentation problem.
\n\nWe found the model performed poorly in visualized prediction of histology images, besides model improvement, external datasets were added to training. At this moment Public LB score reached 0.44
\n\nInspired by DCAN, we revised the UNet model to be multitasked. One head is for semantic prediction, another head is for contour prediction. However, a naive (semantic - contour) as instance segmentation prediction is not good, so we instead used (semantic - contour) as the marker of watershed algorithm, and it brought the Public LB score to 0.45
\n\nRevisited UNet paper, and felt that using 'weight map' to force the network to learn the border pixels could be helpful, we then implemented a weight map, which emphasizes on borders & centroids (especially for small cells), and it brought us to Public LB 0.47.
\n\n
Comparing the partitioning results of watershed and random walker for touching objects, we felt that the result of \"random walker\" was more natural from our perspective. Changed the post-processing to \"random walker\", it improved Public LB score to 0.49.
\n\nReviewed aforementioned markers (semantic - contour) visually, there are always some poor performing cases and it's hard to design rules for them. Therefore, we decided to let the model learn and predict the markers by itself. The third head was added to predict markers, whose ground truths are derived from shrunk cell masks. We used ('markers head' - 'contour head') as the final markers for random walker to partition the 'semantic head'.
\n\n
In order to address the data imbalance issue, we also implemented Focal loss in addition to aforementioned Jaccard/IoU loss function and weight map mechanism. It further improved our Public LB score to 0.50.
\n\nBy adding synthesized images of touching/overlapping cells to training set, the Public LB improved to 0.52, ranked #22 at that moment.
\n\nWe analyzed error cases of stage 1 test data, one of major issues was large cells (scaling), the other one was lack of similar images in Kaggle training dataset.
\n\nTo address scaling issue, we tried to expand the receptive field of UNet's encoder part with \"dilated convolution.\" We found dilated and non-dilated encoder seems to be complement each other well, and we then ensembled these two models by averaging their pixel-wise prediction of three heads.
\n\nBack and forth, we spent lots effort in collecting and validating external dataset to be included. With data manipulation, it brought the scrore to Public LB 0.534, but it's tedious and required some biology knowledge which we didn't have. :-(
\n\nAlso Test Time Augmentation (TTA) was experimented by horizontal flip, vertical flip, horizontal and vertical flip. However, an implementation error on reflection padding & flip pipeline misled us to drop TTA before end of competition (Ouch! we revisited and fixed TTA after the competition)
\n\n
Surprised by the huge differences between stage1 and stage 2 test datasets, we panicked! Several mistakes were made during the final few days, unfortunately. We added additional BBBC018, BBBC020, and stage 1 test to further train the models. That was the 1st mistake to put everything in training, resulting no objective local cross validataion to judge overfitting, but relied on human evaluation of visualized predictions. (note: see how we learned after competition)
\n\nWe thought the longer our model was trained the better score it could be based on the learning curve observed in stage 1, (:facepalm: another mistake mentioned in post-competition), yet it turned out that we chose the worse ones on final submission. Our major struggle in final week was that no validation set similar to the test set (Andrew Ng had a great talk about Nuts and Bolts of Applying Deep Learning in this regard), so we solely relied on human to evaluate the visualized predictions (and many of them are on-purposed poisoned!).
\n\nReading some nice writeups of top winning UNet-based solutions when competition ended (hats off to all winners for sharing!) Indeed, we had many technical approaches in common, yet a few practices we did not do right or well. For example:
\n\nLast but not least, we used stage 1 test set as validation set (totally isolated from training set), and saw the high variance of Private LB score (0.56 ~ 0.623) along the training epochs.
\n\n
The figure suggests that the data distribution of stage 1 test set is likely very different than stage 2 test set, it's somehow like a lottery here since we can't have a reasonable local validation set for stage 2 test set (with tons of on-purposed poison images). The fluctuation might also hint the number of images used in stage 2 scoring can be relatively small.
\n\nUnfortunately, this remains to be a puzzle for us. We felt a 'generalized' model should perform well on both stage 1 test set and stage 2 test set. Without an objective validation criteria, we don't believe that we could have chosen our best 0.623 result to be our final submission.
\n\nLove to hear from experienced Kagglers for any suggestions/practices in this kind of 2-stage competition! Good luck to everyone for future Kaggle competitions.
\n\nIf you are interested in our code -\n- Github Code
", "rawMarkdown": "## First Kaggle competition, and first Kaggle sharing\n\nFirst of all, congrats to the DSB 2018 winners! This is our first ever Kaggle competition, and we indeed learned so much through out the whole event. Although our result is not among the best, but we are satisfied with our top 1.5% finish as the first Kaggle attempt. (Team name: 42) Would like to conclude this journey with the sharing of our learning in hope that it might be useful for future novice contestants like us. We documented our journey in a sequential development log style. [Github Code] (https://github.com/samuelschen/DSB2018) \n\nIn this competition, we struggled on lack of validation criteria during the stage 2, and ended up submitting worse results than we should. A few days after the competition deadline, we were able to achieve **0.623** LB score using the same model with the exception of changing to pre-trained ResNet and fixing our broken TTA codes. I honestly don't think we would have chosen these best results even if we had it before stage 2 ended. So our question is: how did the leading teams choose which 2 results to submit during stage 2 without knowing the real final test2 dataset? Would love to hear everyone's opinion.\n\n## Motivation \n\nAfter finishing several machine learning online courses and trying out past Kaggle competitions, we are excited about the promise of deep learning. We later decided to participate DSB 2018 in order to benchmark our own capability in machine learning. It led us to participate Data Science Bowl 2018 as our 1st Kaggle competiton.\n\n\n## The first couple weeks, skeleton and data pipeline \n\nIt's emphasized that 'generalizability' is the key goal of this [competition](https://www.kaggle.com/c/data-science-bowl-2018/). Given the small amount of training set and its diversity, we decided that we'll need lots of data augmentation in data pipeline. Furthermore, easy debugging and integration with existing Python modules would also be important. PyTorch sounds to us a better choice than TensorFlow for this matter; we decided to use DSB 2018 as our first PyTorch experiment.\n\nBoth [MaskRCNN](https://arxiv.org/abs/1703.06870) and [UNet](https://arxiv.org/abs/1505.04597) appear to be reasonable approaches for the segmentation problem. In favor of its simplicity, we picked **UNet** and built our own code from scratch, also did experiments of data augmentation (random crop, flip, rotate, resize, color jitter, elasitc distortion, color invert, clahe, and gaussian noise). We believed our heavy data augmentation can be helpful in general, but we also learned that some effects did not yield desired improvements, e.g. color invert, clahe, and gaussian noise.\n\nThis vanilla UNet produced Public LB score of **0.34**, trained without external dataset, used **Binary Cross-Entropy (BCE)** loss function. Then loss function was changed to **Jaccard/IoU + BCE**, it helped to improve Public LB score to **0.36**.\n\n## One month on model exploration after the first two weeks\n\nReviewed the output visualization of our vanilla UNet, single binary output obviously not very effective against separating the overlapping nuclei. We started to look for methods to improve vanilla UNet model.\n\nThe first idea was transfer learning. A well pre-trained model should mitigate data quantity issue and provide richer low level features as first part of UNet model. We tried VGG16, but unfortunately, no improvement was seen at that time. (note: we revisited transfer learning after competition with the adoption of ResNet)\n\nIn early experiments of computer vision morphology algorithms, watershed with peak local max, score reached Public LB **0.41**. This confirmed that the competition was indeed an instance segmentation problem.\n\nWe found the model performed poorly in visualized prediction of histology images, besides model improvement, external datasets were added to training. At this moment Public LB score reached **0.44**\n\nInspired by [DCAN](https://arxiv.org/abs/1604.02677), we revised the UNet model to be **multitasked**. One head is for semantic prediction, another head is for contour prediction. However, a naive (semantic - contour) as instance segmentation prediction is not good, so we instead used (semantic - contour) as the marker of [watershed algorithm](http://scikit-image.org/docs/dev/auto_examples/segmentation/plot_watershed.html), and it brought the Public LB score to **0.45**\n\nRevisited UNet paper, and felt that using 'weight map' to force the network to learn the border pixels could be helpful, we then implemented a weight map, which emphasizes on borders & centroids (especially for small cells), and it brought us to Public LB **0.47**.\n\n\n\nComparing the partitioning results of watershed and [random walker](http://scikit-image.org/docs/dev/auto_examples/segmentation/plot_random_walker_segmentation.html) for touching objects, we felt that the result of \"random walker\" was more natural from our perspective. Changed the post-processing to \"random walker\", it improved Public LB score to **0.49**.\n\nReviewed aforementioned markers (semantic - contour) visually, there are always some poor performing cases and it's hard to design rules for them. Therefore, we decided to let the model learn and predict the markers by itself. The third head was added to predict markers, whose ground truths are derived from shrunk cell masks. We used ('markers head' - 'contour head') as the final markers for random walker to partition the 'semantic head'.\n\n\n\nIn order to address the data imbalance issue, we also implemented [Focal loss](https://arxiv.org/abs/1708.02002) in addition to aforementioned Jaccard/IoU loss function and weight map mechanism. It further improved our Public LB score to **0.50**.\n\nBy adding synthesized images of touching/overlapping cells to training set, the Public LB improved to **0.52**, ranked #22 at that moment.\n\n\n## Struggled in last month or so\n\nWe analyzed error cases of stage 1 test data, one of major issues was large cells (scaling), the other one was lack of similar images in Kaggle training dataset.\n\nTo address scaling issue, we tried to expand the receptive field of UNet's encoder part with \"dilated convolution.\" We found dilated and non-dilated encoder seems to be complement each other well, and we then ensembled these two models by averaging their pixel-wise prediction of three heads.\n\nBack and forth, we spent lots effort in collecting and validating external dataset to be included. With data manipulation, it brought the scrore to Public LB **0.534**, but it's tedious and required some biology knowledge which we didn't have. :-(\n\nAlso Test Time Augmentation (TTA) was experimented by horizontal flip, vertical flip, horizontal and vertical flip. However, an implementation error on reflection padding & flip pipeline misled us to drop TTA before end of competition (Ouch! we revisited and fixed TTA after the competition)\n\n\n\n\n## Stage 2 Test Set Released - PANIC mode\n\nSurprised by the huge differences between stage1 and stage 2 test datasets, we panicked! Several mistakes were made during the final few days, unfortunately. We added additional BBBC018, BBBC020, and stage 1 test to further train the models. That was the 1st mistake to put everything in training, resulting no objective local cross validataion to judge overfitting, but relied on human evaluation of visualized predictions. (note: see how we learned after competition)\n\nWe thought the longer our model was trained the better score it could be based on the learning curve observed in stage 1, (:facepalm: another mistake mentioned in post-competition), yet it turned out that we chose the worse ones on final submission. Our major struggle in final week was that no validation set similar to the test set (Andrew Ng had a great talk about [Nuts and Bolts of Applying Deep Learning](https://www.youtube.com/watch?v=F1ka6a13S9I) in this regard), so we solely relied on human to evaluate the visualized predictions (and many of them are on-purposed poisoned!).\n\n\n## Post-Competition Submissions and Reflections\n\nReading some nice writeups of top winning UNet-based solutions when competition ended (hats off to all winners for sharing!) Indeed, we had many technical approaches in common, yet a few practices we did not do right or well. For example:\n\n- Data Augmentation: channel shuffle and rgb/gray color space transformation (it didn't show improvement in our experiments somehow)\n- Implement TTA correctly, which turns out help the score a lot (Private LB **0.569** -> **0.580**)\n- Transfer learning with deep neural network (Resnet), which speed up the training and improved accuracy.\n- Try shared decoder and separated decoders for multitasks, and ensemble them for final prediction.\n\nLast but not least, we used stage 1 test set as validation set (totally isolated from training set), and saw the high variance of Private LB score (**0.56** ~ **0.623**) along the training epochs.\n\n\n\nThe figure suggests that the data distribution of stage 1 test set is likely very different than stage 2 test set, it's somehow like a lottery here since we can't have a reasonable local validation set for stage 2 test set (with tons of on-purposed poison images). The fluctuation might also hint the number of images used in stage 2 scoring can be relatively small.\n\nUnfortunately, this remains to be a puzzle for us. We felt a 'generalized' model should perform well on both stage 1 test set and stage 2 test set. Without an objective validation criteria, we don't believe that we could have chosen our best **0.623** result to be our final submission.\n\nLove to hear from experienced Kagglers for any suggestions/practices in this kind of 2-stage competition! Good luck to everyone for future Kaggle competitions.\n\n\n## Reference\n\nIf you are interested in our code -\n- [Github Code](https://github.com/samuelschen/DSB2018)", "votes": null }, { "id": "339889", "postDate": "06/07/2018 21:28:18", "content": "Wonderful work!! Congratulations!\nDo you mind if I ask some questions?
\n\nThen loss function was changed to Jaccard/IoU + BCE, it helped to improve Public LB score to 0.36. By that you mean that you literally computed each one individually and then added them up?Thanks a lot!
", "rawMarkdown": "Wonderful work!! Congratulations!\nDo you mind if I ask some questions?\n\n 1. `Then loss function was changed to Jaccard/IoU + BCE, it helped to improve Public LB score to 0.36`. By that you mean that you literally computed each one individually and then added them up?\n 2. When you used pre-trained encoders, did you first freeze the encoder layers and trained only the decoder part, followed by a fine-tune?\n 3. I saw from your code that in the decoder part you used 2x2 kernels for conv_transposed followed by 3x3 convs. Why did you chosen this combination?\n 4. Did you try to use deeper decoders? \n\nThanks a lot!", "votes": null }, { "id": "339890", "postDate": "06/07/2018 21:36:09", "content": "This is wonderful. Great job.
", "rawMarkdown": "This is wonderful. Great job.", "votes": null }, { "id": "339929", "postDate": "06/08/2018 01:31:16", "content": "Great, thanks a lot! Just a few more questions, if I may..
\n\nThanks for these great insights!
\n\nSince you mentioned focal loss, I need to ask this (maybe stupid) question. How did you incorporate the focal loss exactly?\nI understand the application of it in a one-stage detector where the proposals get essentially weighted differently but I'm absolute clueless how to incorporate that into a architecture like u-net.\nThanks for the clarification!
", "rawMarkdown": "Thanks for these great insights!\n\nSince you mentioned focal loss, I need to ask this (maybe stupid) question. How did you incorporate the focal loss exactly?\nI understand the application of it in a one-stage detector where the proposals get essentially weighted differently but I'm absolute clueless how to incorporate that into a architecture like u-net.\nThanks for the clarification!", "votes": null }, { "id": "341085", "postDate": "06/11/2018 01:00:12", "content": "Fundamental concept of Focal Loss is \"to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples\". In segmentation case, majority of background pixels were well-classified ones which should be prevented to overwhelm model training.
\n\nImplementation is relatively easy if your model has been guided by cross entropy loss. Refer to code
", "rawMarkdown": "Fundamental concept of [Focal Loss](https://arxiv.org/abs/1708.02002) is \"to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples\". In segmentation case, majority of background pixels were well-classified ones which should be prevented to overwhelm model training. \n\nImplementation is relatively easy if your model has been guided by cross entropy loss. Refer to [code](https://github.com/samuelschen/DSB2018/blob/f3cb0048e8eed5746b5ae387437dce38f60c943b/loss.py#L47)", "votes": null }, { "id": "346961", "postDate": "06/22/2018 22:22:29", "content": "Thanks! About the first item, I'm having some problems when using original dimensions during inference since the dimensions must be multiple of 2**n where n is the number of poolings/conv_stride2 in the encoder, right? Otherwise, the network breaks because dimensions does not match in the u-net skip (encoder to decoder) connections.\nHow did you overcome this issue?
We added padding to the input, and then cropped the output accordingly to get the final prediction.
", "rawMarkdown": "We added padding to the input, and then cropped the output accordingly to get the final prediction.", "votes": null }, { "id": "348049", "postDate": "06/26/2018 01:28:18", "content": "I have implemented zero padding as suggested but it was actually harmful, leading to worse results than a simple resize. Did you also use zero paddings or symmetric/reflect?
", "rawMarkdown": "I have implemented zero padding as suggested but it was actually harmful, leading to worse results than a simple resize. Did you also use zero paddings or symmetric/reflect?", "votes": null }, { "id": "348092", "postDate": "06/26/2018 04:23:01", "content": "Due to nature of training/testing dataset, reflect padding would perform better than constant padding. Background is not always black(zero), see the difference between Histology and Brightfield. Also expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation.
", "rawMarkdown": "Due to nature of training/testing dataset, reflect padding would perform better than constant padding. Background is not always black(zero), see the difference between Histology and Brightfield. Also expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation.", "votes": null }, { "id": "348093", "postDate": "06/26/2018 04:31:49", "content": "Thanks for the help, I'll try the reflect padding.
\n\n\n\n\nAlso expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation.
\n
By this you mean TTA, right? But shouldn't original size predictions instead of resizing improve the results regardless of TTA? Sorry, I got a bit confused here.
", "rawMarkdown": "Thanks for the help, I'll try the reflect padding.\n\n> Also expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation. \n\nBy this you mean TTA, right? But shouldn't original size predictions instead of resizing improve the results regardless of TTA? Sorry, I got a bit confused here.", "votes": null } ], "comments": [ { "id": 339889, "author_name": "arc144", "author_url": "", "post_date": "06/07/2018 21:28:18", "content": "Wonderful work!! Congratulations!\nDo you mind if I ask some questions?
\n\nThen loss function was changed to Jaccard/IoU + BCE, it helped to improve Public LB score to 0.36. By that you mean that you literally computed each one individually and then added them up?Thanks a lot!
", "votes": null, "replies": [ { "id": 339929, "author_name": "rainbean", "author_url": "", "post_date": "06/08/2018 01:31:16", "content": "Great, thanks a lot! Just a few more questions, if I may..
\n\nThanks! About the first item, I'm having some problems when using original dimensions during inference since the dimensions must be multiple of 2**n where n is the number of poolings/conv_stride2 in the encoder, right? Otherwise, the network breaks because dimensions does not match in the u-net skip (encoder to decoder) connections.\nHow did you overcome this issue?
We added padding to the input, and then cropped the output accordingly to get the final prediction.
", "votes": null, "replies": [] }, { "id": 348049, "author_name": "arc144", "author_url": "", "post_date": "06/26/2018 01:28:18", "content": "I have implemented zero padding as suggested but it was actually harmful, leading to worse results than a simple resize. Did you also use zero paddings or symmetric/reflect?
", "votes": null, "replies": [] }, { "id": 348092, "author_name": "rainbean", "author_url": "", "post_date": "06/26/2018 04:23:01", "content": "Due to nature of training/testing dataset, reflect padding would perform better than constant padding. Background is not always black(zero), see the difference between Histology and Brightfield. Also expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation.
", "votes": null, "replies": [] }, { "id": 348093, "author_name": "arc144", "author_url": "", "post_date": "06/26/2018 04:31:49", "content": "Thanks for the help, I'll try the reflect padding.
\n\n\n\n\nAlso expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation.
\n
By this you mean TTA, right? But shouldn't original size predictions instead of resizing improve the results regardless of TTA? Sorry, I got a bit confused here.
", "votes": null, "replies": [] } ] }, { "id": 339890, "author_name": "dskswu", "author_url": "", "post_date": "06/07/2018 21:36:09", "content": "This is wonderful. Great job.
", "votes": null, "replies": [] }, { "id": 340828, "author_name": "chriskkk", "author_url": "", "post_date": "06/10/2018 12:25:59", "content": "Thanks for these great insights!
\n\nSince you mentioned focal loss, I need to ask this (maybe stupid) question. How did you incorporate the focal loss exactly?\nI understand the application of it in a one-stage detector where the proposals get essentially weighted differently but I'm absolute clueless how to incorporate that into a architecture like u-net.\nThanks for the clarification!
", "votes": null, "replies": [ { "id": 341089, "author_name": "rainbean", "author_url": "", "post_date": "06/11/2018 01:13:51", "content": "Fundamental concept of Focal Loss is \"to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples\". In segmentation case, majority of background pixels were well-classified ones which should be prevented to overwhelm model training.
\n\nImplementation is relatively easy if your model has been guided by cross entropy loss. Refer to code
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "334622": "## First Kaggle competition, and first Kaggle sharing\n\nFirst of all, congrats to the DSB 2018 winners! This is our first ever Kaggle competition, and we indeed learned so much through out the whole event. Although our result is not among the best, but we are satisfied with our top 1.5% finish as the first Kaggle attempt. (Team name: 42) Would like to conclude this journey with the sharing of our learning in hope that it might be useful for future novice contestants like us. We documented our journey in a sequential development log style. [Github Code] (https://github.com/samuelschen/DSB2018) \n\nIn this competition, we struggled on lack of validation criteria during the stage 2, and ended up submitting worse results than we should. A few days after the competition deadline, we were able to achieve **0.623** LB score using the same model with the exception of changing to pre-trained ResNet and fixing our broken TTA codes. I honestly don't think we would have chosen these best results even if we had it before stage 2 ended. So our question is: how did the leading teams choose which 2 results to submit during stage 2 without knowing the real final test2 dataset? Would love to hear everyone's opinion.\n\n## Motivation \n\nAfter finishing several machine learning online courses and trying out past Kaggle competitions, we are excited about the promise of deep learning. We later decided to participate DSB 2018 in order to benchmark our own capability in machine learning. It led us to participate Data Science Bowl 2018 as our 1st Kaggle competiton.\n\n\n## The first couple weeks, skeleton and data pipeline \n\nIt's emphasized that 'generalizability' is the key goal of this [competition](https://www.kaggle.com/c/data-science-bowl-2018/). Given the small amount of training set and its diversity, we decided that we'll need lots of data augmentation in data pipeline. Furthermore, easy debugging and integration with existing Python modules would also be important. PyTorch sounds to us a better choice than TensorFlow for this matter; we decided to use DSB 2018 as our first PyTorch experiment.\n\nBoth [MaskRCNN](https://arxiv.org/abs/1703.06870) and [UNet](https://arxiv.org/abs/1505.04597) appear to be reasonable approaches for the segmentation problem. In favor of its simplicity, we picked **UNet** and built our own code from scratch, also did experiments of data augmentation (random crop, flip, rotate, resize, color jitter, elasitc distortion, color invert, clahe, and gaussian noise). We believed our heavy data augmentation can be helpful in general, but we also learned that some effects did not yield desired improvements, e.g. color invert, clahe, and gaussian noise.\n\nThis vanilla UNet produced Public LB score of **0.34**, trained without external dataset, used **Binary Cross-Entropy (BCE)** loss function. Then loss function was changed to **Jaccard/IoU + BCE**, it helped to improve Public LB score to **0.36**.\n\n## One month on model exploration after the first two weeks\n\nReviewed the output visualization of our vanilla UNet, single binary output obviously not very effective against separating the overlapping nuclei. We started to look for methods to improve vanilla UNet model.\n\nThe first idea was transfer learning. A well pre-trained model should mitigate data quantity issue and provide richer low level features as first part of UNet model. We tried VGG16, but unfortunately, no improvement was seen at that time. (note: we revisited transfer learning after competition with the adoption of ResNet)\n\nIn early experiments of computer vision morphology algorithms, watershed with peak local max, score reached Public LB **0.41**. This confirmed that the competition was indeed an instance segmentation problem.\n\nWe found the model performed poorly in visualized prediction of histology images, besides model improvement, external datasets were added to training. At this moment Public LB score reached **0.44**\n\nInspired by [DCAN](https://arxiv.org/abs/1604.02677), we revised the UNet model to be **multitasked**. One head is for semantic prediction, another head is for contour prediction. However, a naive (semantic - contour) as instance segmentation prediction is not good, so we instead used (semantic - contour) as the marker of [watershed algorithm](http://scikit-image.org/docs/dev/auto_examples/segmentation/plot_watershed.html), and it brought the Public LB score to **0.45**\n\nRevisited UNet paper, and felt that using 'weight map' to force the network to learn the border pixels could be helpful, we then implemented a weight map, which emphasizes on borders & centroids (especially for small cells), and it brought us to Public LB **0.47**.\n\n\n\nComparing the partitioning results of watershed and [random walker](http://scikit-image.org/docs/dev/auto_examples/segmentation/plot_random_walker_segmentation.html) for touching objects, we felt that the result of \"random walker\" was more natural from our perspective. Changed the post-processing to \"random walker\", it improved Public LB score to **0.49**.\n\nReviewed aforementioned markers (semantic - contour) visually, there are always some poor performing cases and it's hard to design rules for them. Therefore, we decided to let the model learn and predict the markers by itself. The third head was added to predict markers, whose ground truths are derived from shrunk cell masks. We used ('markers head' - 'contour head') as the final markers for random walker to partition the 'semantic head'.\n\n\n\nIn order to address the data imbalance issue, we also implemented [Focal loss](https://arxiv.org/abs/1708.02002) in addition to aforementioned Jaccard/IoU loss function and weight map mechanism. It further improved our Public LB score to **0.50**.\n\nBy adding synthesized images of touching/overlapping cells to training set, the Public LB improved to **0.52**, ranked #22 at that moment.\n\n\n## Struggled in last month or so\n\nWe analyzed error cases of stage 1 test data, one of major issues was large cells (scaling), the other one was lack of similar images in Kaggle training dataset.\n\nTo address scaling issue, we tried to expand the receptive field of UNet's encoder part with \"dilated convolution.\" We found dilated and non-dilated encoder seems to be complement each other well, and we then ensembled these two models by averaging their pixel-wise prediction of three heads.\n\nBack and forth, we spent lots effort in collecting and validating external dataset to be included. With data manipulation, it brought the scrore to Public LB **0.534**, but it's tedious and required some biology knowledge which we didn't have. :-(\n\nAlso Test Time Augmentation (TTA) was experimented by horizontal flip, vertical flip, horizontal and vertical flip. However, an implementation error on reflection padding & flip pipeline misled us to drop TTA before end of competition (Ouch! we revisited and fixed TTA after the competition)\n\n\n\n\n## Stage 2 Test Set Released - PANIC mode\n\nSurprised by the huge differences between stage1 and stage 2 test datasets, we panicked! Several mistakes were made during the final few days, unfortunately. We added additional BBBC018, BBBC020, and stage 1 test to further train the models. That was the 1st mistake to put everything in training, resulting no objective local cross validataion to judge overfitting, but relied on human evaluation of visualized predictions. (note: see how we learned after competition)\n\nWe thought the longer our model was trained the better score it could be based on the learning curve observed in stage 1, (:facepalm: another mistake mentioned in post-competition), yet it turned out that we chose the worse ones on final submission. Our major struggle in final week was that no validation set similar to the test set (Andrew Ng had a great talk about [Nuts and Bolts of Applying Deep Learning](https://www.youtube.com/watch?v=F1ka6a13S9I) in this regard), so we solely relied on human to evaluate the visualized predictions (and many of them are on-purposed poisoned!).\n\n\n## Post-Competition Submissions and Reflections\n\nReading some nice writeups of top winning UNet-based solutions when competition ended (hats off to all winners for sharing!) Indeed, we had many technical approaches in common, yet a few practices we did not do right or well. For example:\n\n- Data Augmentation: channel shuffle and rgb/gray color space transformation (it didn't show improvement in our experiments somehow)\n- Implement TTA correctly, which turns out help the score a lot (Private LB **0.569** -> **0.580**)\n- Transfer learning with deep neural network (Resnet), which speed up the training and improved accuracy.\n- Try shared decoder and separated decoders for multitasks, and ensemble them for final prediction.\n\nLast but not least, we used stage 1 test set as validation set (totally isolated from training set), and saw the high variance of Private LB score (**0.56** ~ **0.623**) along the training epochs.\n\n\n\nThe figure suggests that the data distribution of stage 1 test set is likely very different than stage 2 test set, it's somehow like a lottery here since we can't have a reasonable local validation set for stage 2 test set (with tons of on-purposed poison images). The fluctuation might also hint the number of images used in stage 2 scoring can be relatively small.\n\nUnfortunately, this remains to be a puzzle for us. We felt a 'generalized' model should perform well on both stage 1 test set and stage 2 test set. Without an objective validation criteria, we don't believe that we could have chosen our best **0.623** result to be our final submission.\n\nLove to hear from experienced Kagglers for any suggestions/practices in this kind of 2-stage competition! Good luck to everyone for future Kaggle competitions.\n\n\n## Reference\n\nIf you are interested in our code -\n- [Github Code](https://github.com/samuelschen/DSB2018)", "339889": "Wonderful work!! Congratulations!\nDo you mind if I ask some questions?\n\n 1. `Then loss function was changed to Jaccard/IoU + BCE, it helped to improve Public LB score to 0.36`. By that you mean that you literally computed each one individually and then added them up?\n 2. When you used pre-trained encoders, did you first freeze the encoder layers and trained only the decoder part, followed by a fine-tune?\n 3. I saw from your code that in the decoder part you used 2x2 kernels for conv_transposed followed by 3x3 convs. Why did you chosen this combination?\n 4. Did you try to use deeper decoders? \n\nThanks a lot!", "339890": "This is wonderful. Great job.", "339929": "1. Yes, calculated individual loss value of each batch, aggregate value and process back propagation at once.\n2. Yes, encoder parts were frozen and trained on decoder part til the network was stable. Encoders fine-tune was experimented after network stable, unfortunately due to high variance between cross-validation and test, we could not endorse effectiveness. \n3. The kernel filter size adopted original design of UNet paper, refer to Network Architecture session of arXiv:1505.04597. \n4. Good question. Variance of multiple model architecture was explored, included number of down/up layers, building blocks of each layer block, pre-trained encoders, and composition of decoders. No obvious result saw in deeper decoders in previous experiments, again, it highly depends on dataset (train and test).", "340297": "Great, thanks a lot! Just a few more questions, if I may..\n\n 1. You did not mention anything about image sizes. Did you train using crops? If yes, there was overlap? What about inference?\n 2. You mentioned that you used an ensemble of models that used both shared and individual decoders for each channel. But separately, which one performed better?\n 3. Do you have an idea how much is improved from using an pre-trained encoder like ResNet, compared to the vanilla U-net?", "340828": "Thanks for these great insights!\n\nSince you mentioned focal loss, I need to ask this (maybe stupid) question. How did you incorporate the focal loss exactly?\nI understand the application of it in a one-stage detector where the proposals get essentially weighted differently but I'm absolute clueless how to incorporate that into a architecture like u-net.\nThanks for the clarification!", "341085": "1. Training data using random resize and crop to align dimension within batch, overlap was nature. Image size kept original dimension is suggested at inference phase. \n2. No constant winner model, refer to [ensemble table](https://github.com/samuelschen/DSB2018#learning-curve)\n3. Pre-train model boosted a lot. refer [learning rate of pre-train model](https://github.com/samuelschen/DSB2018#learning-curve)", "341089": "Fundamental concept of [Focal Loss](https://arxiv.org/abs/1708.02002) is \"to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples\". In segmentation case, majority of background pixels were well-classified ones which should be prevented to overwhelm model training. \n\nImplementation is relatively easy if your model has been guided by cross entropy loss. Refer to [code](https://github.com/samuelschen/DSB2018/blob/f3cb0048e8eed5746b5ae387437dce38f60c943b/loss.py#L47)", "346961": "Thanks! About the first item, I'm having some problems when using original dimensions during inference since the dimensions must be multiple of `2**n` where n is the number of poolings/conv_stride2 in the encoder, right? Otherwise, the network breaks because dimensions does not match in the u-net skip (encoder to decoder) connections.\nHow did you overcome this issue?", "347478": "We added padding to the input, and then cropped the output accordingly to get the final prediction.", "348049": "I have implemented zero padding as suggested but it was actually harmful, leading to worse results than a simple resize. Did you also use zero paddings or symmetric/reflect?", "348092": "Due to nature of training/testing dataset, reflect padding would perform better than constant padding. Background is not always black(zero), see the difference between Histology and Brightfield. Also expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation.", "348093": "Thanks for the help, I'll try the reflect padding.\n\n> Also expect to spot different behavior on various scale of images, which should be countered by model and data generation, e.g. runtime augmentation. \n\nBy this you mean TTA, right? But shouldn't original size predictions instead of resizing improve the results regardless of TTA? Sorry, I got a bit confused here." }, "source": "meta" }