This is an usual kaggle challenge for me. It is the first time i took part in 2 stage challenge. Basically i do commercial algorithm development and i love the idea of \"total black-box\" testing.
\n\nThe usual one stage kaggle competition is \"gray-box\" testing, you see that data, but not the labels. But do note that, whether \"black or grey box\", this will affect everyone and it is still a fair playing ground.
\n\n[things that I did wrong]
\n\nEvery \"wrong thing\" is a learning experience for me. Here, what I have learned is that when there is data label quality issue, we see \"same type of error \" in the training iterations, no matter what i do (e.g. change network structure, learning parameters).
\n\nDuring the search for better architecture, I built a one stage ssd-mask-instance segmentation. I find that 2 stage rcnn is indeed better than one stage ssd method. The rcnn helps to filter wrong results of rpn. In ssd , there is no such mechanism.
\n\nI also just trained the faster-rcnn only. Joint loss of detection and segmentation of mask rcnn is better than faster-rcnn alone. The reduction in classification loss (cross entropy) is quite obvious.
\n\n[if i would to start again, what would i do?]
\n\nSpend more time on clustering and ensemble. Despite limited by data quality, ensemble always work for me. But this is tricky. Because although results visually look better, it not necessary have better LB score (which i think due to some missing annotation in ground truth of LB set).
Spend more time on UNet. It turns out that Unet has smaller segmentation error (if you just consider semantic segmentation and not the instance separation). The best solution is combination of two, though i haven't find a way to do it efficiently and effectively.
Surprisingly, simple unet+post processing can work almost the same as mask-rcnn from LB score point of view. It is not always necessarily to use the latest tool, but you do have to be good at your tool.
\n\nCollect more data. When I read about nerve cells, I thought they look very different from the stage1 dataset and think that it wouldn't appear in stage2. I should have include that in my external data, or at least test it.
More visualization tool. This will help to analysis the error.
[Final thoughts]
\n\nThis may be a case where we don't have to trust the LB too much. Human annotation error is about 0.01 and 0.03. small improvement like 0.001 may or may not real improvement at all. When you compare algorithms, do consider the meaning of \"statistical significance\"
This is also one of the rare case where you get to test a dataset and then you are given the ground truth label (stage1 test). Spend some time to analyse what has gone right or wrong during your \"competition process\" for stage1. We always don't know why the LB score is like this or like that, now you can!
This is also a rare case where you get to test your algorithm in \"unexpected\" scope. Use this chance to understand why algorithm break down. There will also be surprise that it actually works in some unusual cases. It is a good chance for you to understand what is \"robustness\".
\n\nUnexpectedness bring about breakthrough. Sit down, think about it and make a summary write up.
This is an usual kaggle challenge for me. It is the first time i took part in 2 stage challenge. Basically i do commercial algorithm development and i love the idea of \"total black-box\" testing.
\n\nThe usual one stage kaggle competition is \"gray-box\" testing, you see that data, but not the labels. But do note that, whether \"black or grey box\", this will affect everyone and it is still a fair playing ground.
\n\n[things that I did wrong]
\n\nEvery \"wrong thing\" is a learning experience for me. Here, what I have learned is that when there is data label quality issue, we see \"same type of error \" in the training iterations, no matter what i do (e.g. change network structure, learning parameters).
\n\nDuring the search for better architecture, I built a one stage ssd-mask-instance segmentation. I find that 2 stage rcnn is indeed better than one stage ssd method. The rcnn helps to filter wrong results of rpn. In ssd , there is no such mechanism.
\n\nI also just trained the faster-rcnn only. Joint loss of detection and segmentation of mask rcnn is better than faster-rcnn alone. The reduction in classification loss (cross entropy) is quite obvious.
\n\n[if i would to start again, what would i do?]
\n\nSpend more time on clustering and ensemble. Despite limited by data quality, ensemble always work for me. But this is tricky. Because although results visually look better, it not necessary have better LB score (which i think due to some missing annotation in ground truth of LB set).
Spend more time on UNet. It turns out that Unet has smaller segmentation error (if you just consider semantic segmentation and not the instance separation). The best solution is combination of two, though i haven't find a way to do it efficiently and effectively.
Surprisingly, simple unet+post processing can work almost the same as mask-rcnn from LB score point of view. It is not always necessarily to use the latest tool, but you do have to be good at your tool.
\n\nCollect more data. When I read about nerve cells, I thought they look very different from the stage1 dataset and think that it wouldn't appear in stage2. I should have include that in my external data, or at least test it.
More visualization tool. This will help to analysis the error.
[Final thoughts]
\n\nThis may be a case where we don't have to trust the LB too much. Human annotation error is about 0.01 and 0.03. small improvement like 0.001 may or may not real improvement at all. When you compare algorithms, do consider the meaning of \"statistical significance\"
This is also one of the rare case where you get to test a dataset and then you are given the ground truth label (stage1 test). Spend some time to analyse what has gone right or wrong during your \"competition process\" for stage1. We always don't know why the LB score is like this or like that, now you can!
This is also a rare case where you get to test your algorithm in \"unexpected\" scope. Use this chance to understand why algorithm break down. There will also be surprise that it actually works in some unusual cases. It is a good chance for you to understand what is \"robustness\".
\n\nUnexpectedness bring about breakthrough. Sit down, think about it and make a summary write up.
Wow. Awesome survey, Heng.
\n\nI’ve really enjoyed your posts throughout the competition and this was no exception.
", "rawMarkdown": "Wow. Awesome survey, Heng. \n\nI’ve **really** enjoyed your posts throughout the competition and this was no exception. ", "votes": 10 }, { "id": 313176, "postDate": "2018-04-12T21:28:23.163Z", "content": "FYI, working 3 days with a fresh team we could get Unet score at 0.504, and I'm sure we could move higher with few more days. And this is using the simple Unet starter NN, without pretrained weights. I'm convinced Unet can yield way beter results here.
", "rawMarkdown": "FYI, working 3 days with a fresh team we could get Unet score at 0.504, and I'm sure we could move higher with few more days. And this is using the simple Unet starter NN, without pretrained weights. I'm convinced Unet can yield way beter results here.", "votes": 6, "replies": [ { "id": 313204, "postDate": "2018-04-12T23:05:40.980Z", "content": "Will you share your unet solution after the competition?
", "rawMarkdown": "Will you share your unet solution after the competition? " }, { "id": 313308, "postDate": "2018-04-13T05:01:13.513Z", "content": "If my teammates agree to it, yes.
", "rawMarkdown": "If my teammates agree to it, yes.", "votes": 1 }, { "id": 313327, "postDate": "2018-04-13T05:44:39.200Z", "content": "@jiafan Sure thing, we didn't know we had to submit the final models at the end of the first stage (instead of the 2nd stage) until our new team mate @CPMP told us \"What are you guys doing? you have to submit your models you're gonna use for the 2nd stage today!!\" and then we got into the panic mode, totally misunderstood the 2-stage rules as first time Kagglers, so we only got a chance to train 3 of our ideas in the final 2 days and only submitted those. I too believe a single U-net can do better than 0.504, though @Heng mentioned a couple months ago you can't get better than 0.45 using a U-Net and that made us start trying Heng's mask RCNN and matterport's mask RCNN, however, we didn't have enough time and went back to a simple U-net. We didn't tune any hyper parameters but used the same U-net starter, we'll see what the 2nd stage has in store for us.
", "rawMarkdown": "@jiafan Sure thing, we didn't know we had to submit the final models at the end of the first stage (instead of the 2nd stage) until our new team mate @CPMP told us \"What are you guys doing? you have to submit your models you're gonna use for the 2nd stage today!!\" and then we got into the panic mode, totally misunderstood the 2-stage rules as first time Kagglers, so we only got a chance to train 3 of our ideas in the final 2 days and only submitted those. I too believe a single U-net can do better than 0.504, though @Heng mentioned a couple months ago you can't get better than 0.45 using a U-Net and that made us start trying Heng's mask RCNN and matterport's mask RCNN, however, we didn't have enough time and went back to a simple U-net. We didn't tune any hyper parameters but used the same U-net starter, we'll see what the 2nd stage has in store for us.", "votes": 3 }, { "id": 313495, "postDate": "2018-04-13T10:44:38.157Z", "content": "\n\n\nI'm convinced Unet can yield way beter results here.
\n
This discussion prove the point: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54426#latest-313493
\n\nI'm sure we'll learn lots of very interesting stuff after competition ends. Heng shared so much so far that one could wrongly believe there is nothing more to learn....
", "rawMarkdown": "\n\n> I'm convinced Unet can yield way beter results here.\n\nThis discussion prove the point: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54426#latest-313493\n\nI'm sure we'll learn lots of very interesting stuff after competition ends. Heng shared so much so far that one could wrongly believe there is nothing more to learn...." } ] }, { "id": 313289, "postDate": "2018-04-13T03:51:36.217Z", "content": "Hello @Heng, thank you for your posts on the challenge. I have learned a lot of things from your posts. Although my topic is semantic segmentation, it is the good chance for me to try instance segmentation in this challenge.
\n\nIn this challenge, we have to solve many problems: limited dataset, noisy label, overlap nucleus, multi-modalities... Hence, external data, the strategy to train network (against overfitting) post-processing, new network are key components to improve performance. In the stage-1 test, I achieved 0.553 LB with my end-to-end mask-rcnn (modified mask-rcnn) without any post-processing. I think the score may be better when I apply post-processing (likes your ensembling).
\n\nI will post some results on stage2 test in your another thread
", "rawMarkdown": "Hello @Heng, thank you for your posts on the challenge. I have learned a lot of things from your posts. Although my topic is semantic segmentation, it is the good chance for me to try instance segmentation in this challenge.\n \nIn this challenge, we have to solve many problems: limited dataset, noisy label, overlap nucleus, multi-modalities... Hence, external data, the strategy to train network (against overfitting) post-processing, new network are key components to improve performance. In the stage-1 test, I achieved 0.553 LB with my end-to-end mask-rcnn (modified mask-rcnn) without any post-processing. I think the score may be better when I apply post-processing (likes your ensembling).\n\n I will post some results on stage2 test in your another thread", "votes": 3, "replies": [ { "id": 313303, "postDate": "2018-04-13T04:39:56.203Z", "content": "Very impressed result using a single model. Nice work. It will be very helpful if you share details after the competition.
", "rawMarkdown": "Very impressed result using a single model. Nice work. It will be very helpful if you share details after the competition. " }, { "id": 313309, "postDate": "2018-04-13T05:05:59.623Z", "content": "You can check some results on the stage2 at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54353
", "rawMarkdown": "You can check some results on the stage2 at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54353" } ] }, { "id": 313030, "postDate": "2018-04-12T17:24:40.680Z", "content": "(There’s a lot in this post I want to comment on, but I’m going to wait until the competition is over …)
", "rawMarkdown": "(There’s a lot in this post I want to comment on, but I’m going to wait until the competition is over …) ", "votes": 3 }, { "id": 313102, "postDate": "2018-04-12T19:23:18.140Z", "content": "Heng CherKeng, you did an amazing job in this competition. You really deserve to win it !
\n\nI am also developping commercial products in my everyday job ; and I think that you pointed out the real important thing:\n\"It is not always necessarily to use the latest tool, but you do have to be good at your tool\"
\n\nActually I wish that I had used more classical approaches like shape constrained level sets. The temptation with Kaggle competition is always to developp an end-to-end solution, spend too much time on retraining, hyperparmaters tuning, etc.
\n\nI used a cascade of unets: unet on original image -> resize -> second unet on scale normalized image -> third unet on predicted cores to get their centers. This approach gave me a score around 0.54 on the LB without too much post processing (actually only removing small cells).
\n\nI expected new color images but it did not append. I guess the model will perform pretty badly on the new images.
", "rawMarkdown": "Heng CherKeng, you did an amazing job in this competition. You really deserve to win it !\n\nI am also developping commercial products in my everyday job ; and I think that you pointed out the real important thing:\n\"It is not always necessarily to use the latest tool, but you do have to be good at your tool\"\n\nActually I wish that I had used more classical approaches like shape constrained level sets. The temptation with Kaggle competition is always to developp an end-to-end solution, spend too much time on retraining, hyperparmaters tuning, etc. \n\nI used a cascade of unets: unet on original image -> resize -> second unet on scale normalized image -> third unet on predicted cores to get their centers. This approach gave me a score around 0.54 on the LB without too much post processing (actually only removing small cells). \n\nI expected new color images but it did not append. I guess the model will perform pretty badly on the new images.\n", "votes": 1, "replies": [ { "id": 313201, "postDate": "2018-04-12T23:01:04.947Z", "content": "Do you train those unets altogether or separately?
", "rawMarkdown": "Do you train those unets altogether or separately?" }, { "id": 313335, "postDate": "2018-04-13T05:53:15.667Z", "content": "@jeandebleau, quite impressive you got LB=0.54 using a cascade of U-nets, it'd be great to share your core ideas, input and output classes after the competition then. Resizing hurts us, so we didn't resize our pictures for train/test. We haven't started training our new u-net models but after having seen Heng's results, I don't think our simple U-net models can handle these images well, we'll see.
", "rawMarkdown": "@jeandebleau, quite impressive you got LB=0.54 using a cascade of U-nets, it'd be great to share your core ideas, input and output classes after the competition then. Resizing hurts us, so we didn't resize our pictures for train/test. We haven't started training our new u-net models but after having seen Heng's results, I don't think our simple U-net models can handle these images well, we'll see. " }, { "id": 313346, "postDate": "2018-04-13T06:12:40.867Z", "content": "@Jiafan: for memory reasons, yes I had to train the unets separately.
\n\n@Nicole: Resizing was necessary for the unet that predicts the centers of the cells. Moreover this unet performs also much better by directly working on the predicted core, at this point the original image is not considered anymore; thus I can speak of a cascade of unets. Finally, I also removed small particles from the ground truth after that the images are scale-normalized. This permits to obtain smooth and very robust cells detection results.
\n\nFinally the unets are derived from the work presented here: https://arxiv.org/abs/1803.02579\nThis is the only \"new thing\" I could try ; and I like the results of this architecture using concurrent spatial and channel squeeze excitation.
", "rawMarkdown": "@Jiafan: for memory reasons, yes I had to train the unets separately. \n\n@Nicole: Resizing was necessary for the unet that predicts the centers of the cells. Moreover this unet performs also much better by directly working on the predicted core, at this point the original image is not considered anymore; thus I can speak of a cascade of unets. Finally, I also removed small particles from the ground truth after that the images are scale-normalized. This permits to obtain smooth and very robust cells detection results.\n\nFinally the unets are derived from the work presented here: https://arxiv.org/abs/1803.02579\nThis is the only \"new thing\" I could try ; and I like the results of this architecture using concurrent spatial and channel squeeze excitation. \n\n", "votes": 1 }, { "id": 313349, "postDate": "2018-04-13T06:15:10.943Z", "content": "thank you for the comments. for your information, spatial squeeze excitation don't work on mask rcnn in my experiments. But channel squeeze excitation was great.
", "rawMarkdown": "thank you for the comments. for your information, spatial squeeze excitation don't work on mask rcnn in my experiments. But channel squeeze excitation was great.", "votes": 1 }, { "id": 313355, "postDate": "2018-04-13T06:35:34.573Z", "content": "Actually, I see spatial squeeze simply as a variant of deep supervision in the way that you force intermediate layers to produce spatially consistent maps which is important for unet archtitectures. \nI do not know mask rcnn well enough, Did you use FPN ? in my point of view FPN is already implementing this kind of ideas so it should not be necessary to add additional spatial squeezing there.
", "rawMarkdown": "Actually, I see spatial squeeze simply as a variant of deep supervision in the way that you force intermediate layers to produce spatially consistent maps which is important for unet archtitectures. \nI do not know mask rcnn well enough, Did you use FPN ? in my point of view FPN is already implementing this kind of ideas so it should not be necessary to add additional spatial squeezing there. ", "votes": 2 }, { "id": 313359, "postDate": "2018-04-13T06:44:07.240Z", "content": "I see squeeze channel as agate or switch. Channelwise excitation will turn on and off channels based on information of the whole feature maps. Spatialwise will turn off signal at locations.
", "rawMarkdown": "I see squeeze channel as agate or switch. Channelwise excitation will turn on and off channels based on information of the whole feature maps. Spatialwise will turn off signal at locations.", "votes": 1 }, { "id": 313401, "postDate": "2018-04-13T07:57:37.827Z", "content": "@Jeandebleau thanks for sharing as I have not seen this paper before.
", "rawMarkdown": "@Jeandebleau thanks for sharing as I have not seen this paper before." }, { "id": 313429, "postDate": "2018-04-13T08:46:51.493Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 313076, "postDate": "2018-04-12T18:50:50.207Z", "content": "I put UNET down very early. I always meant to go back to it. I have learned a lot from Heng as well. I did not use your code or your methodology per se, but you made me think about many issues I would have passed over but for your sharing. Thanks!
", "rawMarkdown": "I put UNET down very early. I always meant to go back to it. I have learned a lot from Heng as well. I did not use your code or your methodology per se, but you made me think about many issues I would have passed over but for your sharing. Thanks!", "votes": 1 }, { "id": 313068, "postDate": "2018-04-12T18:33:49.393Z", "content": "Thank you for all the contribution in this competition, I learned a lot from you!
", "rawMarkdown": "Thank you for all the contribution in this competition, I learned a lot from you!", "votes": 1 }, { "id": 313669, "postDate": "2018-04-13T17:18:29.740Z", "content": "Yeah, I got the understanding that no amount of tinkering with any nets, especially those learning boundaries is gonna work here right away when I looked at the data. It was quite a frustration having to revision all of those annotations https://github.com/ibmua/data-science-bowl-2018-train-set --- btw, do use those for final sub. I didn't look much at losses, only at visualizations of what actually my net displays. Qualitative analysis, not quantitative. That one's not gonna work for a task like this. And it also seemed apparent that mask rcnn is something way too fancy for this task.
\n\nBlack box was quite a gamble. At the end it's like the more you took Anne's advice to heart the harder you were punched. If I didn't, I'd have a really good submission by now. But I chased the wind instead.
", "rawMarkdown": "Yeah, I got the understanding that no amount of tinkering with any nets, especially those learning boundaries is gonna work here right away when I looked at the data. It was quite a frustration having to revision all of those annotations https://github.com/ibmua/data-science-bowl-2018-train-set --- btw, do use those for final sub. I didn't look much at losses, only at visualizations of what actually my net displays. Qualitative analysis, not quantitative. That one's not gonna work for a task like this. And it also seemed apparent that mask rcnn is something way too fancy for this task.\n\nBlack box was quite a gamble. At the end it's like the more you took Anne's advice to heart the harder you were punched. If I didn't, I'd have a really good submission by now. But I chased the wind instead.", "votes": 2, "replies": [ { "id": 314283, "postDate": "2018-04-15T06:04:55.787Z", "content": ":)
", "rawMarkdown": ":)" } ] }, { "id": 316488, "postDate": "2018-04-19T07:34:54.423Z", "content": "It is a brilliant post from the experienced kaggler! Thanks a lot!
", "rawMarkdown": "It is a brilliant post from the experienced kaggler! Thanks a lot!" }, { "id": 315832, "postDate": "2018-04-17T16:36:33.037Z", "content": "Thanks, Heng! Really tremendous effort on the forum. I can't imagine how you combine kaggling and primary work =)
", "rawMarkdown": "Thanks, Heng! Really tremendous effort on the forum. I can't imagine how you combine kaggling and primary work =)" }, { "id": 314298, "postDate": "2018-04-15T07:22:46.283Z", "content": "Thx for sharing
", "rawMarkdown": "Thx for sharing" }, { "id": 314111, "postDate": "2018-04-14T16:16:43.250Z", "content": "@Heng Thanks for all the comments. I didn't get to spend enough time on this competition but I certainly look forward to following your posts on the next one! :)
", "rawMarkdown": "@Heng Thanks for all the comments. I didn't get to spend enough time on this competition but I certainly look forward to following your posts on the next one! :)" }, { "id": 313621, "postDate": "2018-04-13T15:26:51.763Z", "content": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?
", "rawMarkdown": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?", "replies": [ { "id": 313752, "postDate": "2018-04-13T19:36:47.583Z", "content": "I output \"1 1\" for the image without nuclei detected.
", "rawMarkdown": "I output \"1 1\" for the image without nuclei detected." } ] }, { "id": 313307, "postDate": "2018-04-13T04:52:28.907Z", "content": "Thank you very much! We all learned a lot from your ideas and codes! It is my first time to compete KAGGLE. Bros here are very nice, say good words and I like this place very much! Doge
", "rawMarkdown": "Thank you very much! We all learned a lot from your ideas and codes! It is my first time to compete KAGGLE. Bros here are very nice, say good words and I like this place very much! Doge" }, { "id": 313215, "postDate": "2018-04-12T23:45:52.610Z", "content": "@Heng thanks for this post and all you have shared in this competition. I did not use pytorch but certainly learnt from your codes.
\n\nOur mask RCNN performed much better than UNET but we were able to get the UNET model from LB=0.353 to 0.408 within 3 days with fresh set of eyes after teaming up. Not as impressive as CPMP's 0.504 but we are proud of it.
", "rawMarkdown": "@Heng thanks for this post and all you have shared in this competition. I did not use pytorch but certainly learnt from your codes.\n\nOur mask RCNN performed much better than UNET but we were able to get the UNET model from LB=0.353 to 0.408 within 3 days with fresh set of eyes after teaming up. Not as impressive as CPMP's 0.504 but we are proud of it. " }, { "id": 313155, "postDate": "2018-04-12T21:00:21.920Z", "content": "Heng, nice posts throughout the competition. I learned a lot of new image augmentation and ensembling methods from your posts (which was my goal for this competition). Thanks for being awesome!
", "rawMarkdown": "Heng, nice posts throughout the competition. I learned a lot of new image augmentation and ensembling methods from your posts (which was my goal for this competition). Thanks for being awesome!" }, { "id": 313130, "postDate": "2018-04-12T20:11:59.297Z", "content": "Thank you, Heng. I learned a lot from you, not just from this competition.
", "rawMarkdown": "Thank you, Heng. I learned a lot from you, not just from this competition. " }, { "id": 313122, "postDate": "2018-04-12T19:58:23.253Z", "content": "Thanks, Heng. For me the thoughts and code you provided is the most valuable thing I earned from this competition. Looking forward to seeing you in the next competition
", "rawMarkdown": "Thanks, Heng. For me the thoughts and code you provided is the most valuable thing I earned from this competition. Looking forward to seeing you in the next competition" }, { "id": 313119, "postDate": "2018-04-12T19:51:10.633Z", "content": "Hi Heng\nI just want to say that you are awesome, thank you for sharing the high quality code and clear ideas, I learned a lot from you, thank you
", "rawMarkdown": "Hi Heng\nI just want to say that you are awesome, thank you for sharing the high quality code and clear ideas, I learned a lot from you, thank you" }, { "id": 314271, "postDate": "2018-04-15T04:39:49.413Z", "content": "I'm quite fresh to Kaggle, but so far the best thing for me about this competition has been learning from the discussions and insights being shared by yourself and all the other teams. Thank you!
", "rawMarkdown": "I'm quite fresh to Kaggle, but so far the best thing for me about this competition has been learning from the discussions and insights being shared by yourself and all the other teams. Thank you!", "isDeleted": true }, { "id": 314329, "postDate": "2018-04-15T08:58:40.003Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 313024, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-12T17:14:36.780000", "content": "Wow. Awesome survey, Heng.
\n\nI’ve really enjoyed your posts throughout the competition and this was no exception.
", "votes": 10, "replies": [] }, { "id": 313176, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-12T21:28:23.163000", "content": "FYI, working 3 days with a fresh team we could get Unet score at 0.504, and I'm sure we could move higher with few more days. And this is using the simple Unet starter NN, without pretrained weights. I'm convinced Unet can yield way beter results here.
", "votes": 6, "replies": [ { "id": 313204, "author_name": "Jiafan", "author_url": "", "post_date": "2018-04-12T23:05:40.980000", "content": "Will you share your unet solution after the competition?
", "votes": 0, "replies": [] }, { "id": 313308, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-13T05:01:13.513000", "content": "If my teammates agree to it, yes.
", "votes": 1, "replies": [] }, { "id": 313327, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-13T05:44:39.200000", "content": "@jiafan Sure thing, we didn't know we had to submit the final models at the end of the first stage (instead of the 2nd stage) until our new team mate @CPMP told us \"What are you guys doing? you have to submit your models you're gonna use for the 2nd stage today!!\" and then we got into the panic mode, totally misunderstood the 2-stage rules as first time Kagglers, so we only got a chance to train 3 of our ideas in the final 2 days and only submitted those. I too believe a single U-net can do better than 0.504, though @Heng mentioned a couple months ago you can't get better than 0.45 using a U-Net and that made us start trying Heng's mask RCNN and matterport's mask RCNN, however, we didn't have enough time and went back to a simple U-net. We didn't tune any hyper parameters but used the same U-net starter, we'll see what the 2nd stage has in store for us.
", "votes": 3, "replies": [] }, { "id": 313495, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-13T10:44:38.157000", "content": "\n\n\nI'm convinced Unet can yield way beter results here.
\n
This discussion prove the point: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54426#latest-313493
\n\nI'm sure we'll learn lots of very interesting stuff after competition ends. Heng shared so much so far that one could wrongly believe there is nothing more to learn....
", "votes": 0, "replies": [] } ] }, { "id": 313289, "author_name": "Toan Duc Bui", "author_url": "", "post_date": "2018-04-13T03:51:36.217000", "content": "Hello @Heng, thank you for your posts on the challenge. I have learned a lot of things from your posts. Although my topic is semantic segmentation, it is the good chance for me to try instance segmentation in this challenge.
\n\nIn this challenge, we have to solve many problems: limited dataset, noisy label, overlap nucleus, multi-modalities... Hence, external data, the strategy to train network (against overfitting) post-processing, new network are key components to improve performance. In the stage-1 test, I achieved 0.553 LB with my end-to-end mask-rcnn (modified mask-rcnn) without any post-processing. I think the score may be better when I apply post-processing (likes your ensembling).
\n\nI will post some results on stage2 test in your another thread
", "votes": 3, "replies": [ { "id": 313303, "author_name": "Ildoo Kim", "author_url": "", "post_date": "2018-04-13T04:39:56.203000", "content": "Very impressed result using a single model. Nice work. It will be very helpful if you share details after the competition.
", "votes": 0, "replies": [] }, { "id": 313309, "author_name": "Toan Duc Bui", "author_url": "", "post_date": "2018-04-13T05:05:59.623000", "content": "You can check some results on the stage2 at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54353
", "votes": 0, "replies": [] } ] }, { "id": 313030, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-12T17:24:40.680000", "content": "(There’s a lot in this post I want to comment on, but I’m going to wait until the competition is over …)
", "votes": 3, "replies": [] }, { "id": 313102, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-12T19:23:18.140000", "content": "Heng CherKeng, you did an amazing job in this competition. You really deserve to win it !
\n\nI am also developping commercial products in my everyday job ; and I think that you pointed out the real important thing:\n\"It is not always necessarily to use the latest tool, but you do have to be good at your tool\"
\n\nActually I wish that I had used more classical approaches like shape constrained level sets. The temptation with Kaggle competition is always to developp an end-to-end solution, spend too much time on retraining, hyperparmaters tuning, etc.
\n\nI used a cascade of unets: unet on original image -> resize -> second unet on scale normalized image -> third unet on predicted cores to get their centers. This approach gave me a score around 0.54 on the LB without too much post processing (actually only removing small cells).
\n\nI expected new color images but it did not append. I guess the model will perform pretty badly on the new images.
", "votes": 1, "replies": [ { "id": 313201, "author_name": "Jiafan", "author_url": "", "post_date": "2018-04-12T23:01:04.947000", "content": "Do you train those unets altogether or separately?
", "votes": 0, "replies": [] }, { "id": 313335, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-13T05:53:15.667000", "content": "@jeandebleau, quite impressive you got LB=0.54 using a cascade of U-nets, it'd be great to share your core ideas, input and output classes after the competition then. Resizing hurts us, so we didn't resize our pictures for train/test. We haven't started training our new u-net models but after having seen Heng's results, I don't think our simple U-net models can handle these images well, we'll see.
", "votes": 0, "replies": [] }, { "id": 313346, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-13T06:12:40.867000", "content": "@Jiafan: for memory reasons, yes I had to train the unets separately.
\n\n@Nicole: Resizing was necessary for the unet that predicts the centers of the cells. Moreover this unet performs also much better by directly working on the predicted core, at this point the original image is not considered anymore; thus I can speak of a cascade of unets. Finally, I also removed small particles from the ground truth after that the images are scale-normalized. This permits to obtain smooth and very robust cells detection results.
\n\nFinally the unets are derived from the work presented here: https://arxiv.org/abs/1803.02579\nThis is the only \"new thing\" I could try ; and I like the results of this architecture using concurrent spatial and channel squeeze excitation.
", "votes": 1, "replies": [] }, { "id": 313349, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-13T06:15:10.943000", "content": "thank you for the comments. for your information, spatial squeeze excitation don't work on mask rcnn in my experiments. But channel squeeze excitation was great.
", "votes": 1, "replies": [] }, { "id": 313355, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-13T06:35:34.573000", "content": "Actually, I see spatial squeeze simply as a variant of deep supervision in the way that you force intermediate layers to produce spatially consistent maps which is important for unet archtitectures. \nI do not know mask rcnn well enough, Did you use FPN ? in my point of view FPN is already implementing this kind of ideas so it should not be necessary to add additional spatial squeezing there.
", "votes": 2, "replies": [] }, { "id": 313359, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-13T06:44:07.240000", "content": "I see squeeze channel as agate or switch. Channelwise excitation will turn on and off channels based on information of the whole feature maps. Spatialwise will turn off signal at locations.
", "votes": 1, "replies": [] }, { "id": 313401, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-04-13T07:57:37.827000", "content": "@Jeandebleau thanks for sharing as I have not seen this paper before.
", "votes": 0, "replies": [] }, { "id": 313429, "author_name": "", "author_url": "", "post_date": "2018-04-13T08:46:51.493000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 313076, "author_name": "Brian Farrar", "author_url": "", "post_date": "2018-04-12T18:50:50.207000", "content": "I put UNET down very early. I always meant to go back to it. I have learned a lot from Heng as well. I did not use your code or your methodology per se, but you made me think about many issues I would have passed over but for your sharing. Thanks!
", "votes": 1, "replies": [] }, { "id": 313068, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-04-12T18:33:49.393000", "content": "Thank you for all the contribution in this competition, I learned a lot from you!
", "votes": 1, "replies": [] }, { "id": 313669, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-13T17:18:29.740000", "content": "Yeah, I got the understanding that no amount of tinkering with any nets, especially those learning boundaries is gonna work here right away when I looked at the data. It was quite a frustration having to revision all of those annotations https://github.com/ibmua/data-science-bowl-2018-train-set --- btw, do use those for final sub. I didn't look much at losses, only at visualizations of what actually my net displays. Qualitative analysis, not quantitative. That one's not gonna work for a task like this. And it also seemed apparent that mask rcnn is something way too fancy for this task.
\n\nBlack box was quite a gamble. At the end it's like the more you took Anne's advice to heart the harder you were punched. If I didn't, I'd have a really good submission by now. But I chased the wind instead.
", "votes": 2, "replies": [ { "id": 314283, "author_name": "Nut Kan", "author_url": "", "post_date": "2018-04-15T06:04:55.787000", "content": ":)
", "votes": 0, "replies": [] } ] }, { "id": 316488, "author_name": "HongKee", "author_url": "", "post_date": "2018-04-19T07:34:54.423000", "content": "It is a brilliant post from the experienced kaggler! Thanks a lot!
", "votes": 0, "replies": [] }, { "id": 315832, "author_name": "harshml", "author_url": "", "post_date": "2018-04-17T16:36:33.037000", "content": "Thanks, Heng! Really tremendous effort on the forum. I can't imagine how you combine kaggling and primary work =)
", "votes": 0, "replies": [] }, { "id": 314298, "author_name": "Yu Di", "author_url": "", "post_date": "2018-04-15T07:22:46.283000", "content": "Thx for sharing
", "votes": 0, "replies": [] }, { "id": 314111, "author_name": "HansMelo", "author_url": "", "post_date": "2018-04-14T16:16:43.250000", "content": "@Heng Thanks for all the comments. I didn't get to spend enough time on this competition but I certainly look forward to following your posts on the next one! :)
", "votes": 0, "replies": [] }, { "id": 313621, "author_name": "Robert_luo", "author_url": "", "post_date": "2018-04-13T15:26:51.763000", "content": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?
", "votes": 0, "replies": [ { "id": 313752, "author_name": "Jiafan", "author_url": "", "post_date": "2018-04-13T19:36:47.583000", "content": "I output \"1 1\" for the image without nuclei detected.
", "votes": 0, "replies": [] } ] }, { "id": 313307, "author_name": "lyz_aaa", "author_url": "", "post_date": "2018-04-13T04:52:28.907000", "content": "Thank you very much! We all learned a lot from your ideas and codes! It is my first time to compete KAGGLE. Bros here are very nice, say good words and I like this place very much! Doge
", "votes": 0, "replies": [] }, { "id": 313215, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-04-12T23:45:52.610000", "content": "@Heng thanks for this post and all you have shared in this competition. I did not use pytorch but certainly learnt from your codes.
\n\nOur mask RCNN performed much better than UNET but we were able to get the UNET model from LB=0.353 to 0.408 within 3 days with fresh set of eyes after teaming up. Not as impressive as CPMP's 0.504 but we are proud of it.
", "votes": 0, "replies": [] }, { "id": 313155, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-04-12T21:00:21.920000", "content": "Heng, nice posts throughout the competition. I learned a lot of new image augmentation and ensembling methods from your posts (which was my goal for this competition). Thanks for being awesome!
", "votes": 0, "replies": [] }, { "id": 313130, "author_name": "Yibing Wu", "author_url": "", "post_date": "2018-04-12T20:11:59.297000", "content": "Thank you, Heng. I learned a lot from you, not just from this competition.
", "votes": 0, "replies": [] }, { "id": 313122, "author_name": "xuan", "author_url": "", "post_date": "2018-04-12T19:58:23.253000", "content": "Thanks, Heng. For me the thoughts and code you provided is the most valuable thing I earned from this competition. Looking forward to seeing you in the next competition
", "votes": 0, "replies": [] }, { "id": 313119, "author_name": "bintan_FDU", "author_url": "", "post_date": "2018-04-12T19:51:10.633000", "content": "Hi Heng\nI just want to say that you are awesome, thank you for sharing the high quality code and clear ideas, I learned a lot from you, thank you
", "votes": 0, "replies": [] }, { "id": 314271, "author_name": "", "author_url": "", "post_date": "2018-04-15T04:39:49.413000", "content": "I'm quite fresh to Kaggle, but so far the best thing for me about this competition has been learning from the discussions and insights being shared by yourself and all the other teams. Thank you!
", "votes": 0, "replies": [] }, { "id": 314329, "author_name": "flash_light1", "author_url": "", "post_date": "2018-04-15T08:58:40.003000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "312993": "This is an usual kaggle challenge for me. It is the first time i took part in 2 stage challenge. Basically i do commercial algorithm development and i love the idea of \"total black-box\" testing.\n\nThe usual one stage kaggle competition is \"gray-box\" testing, you see that data, but not the labels. But do note that, whether \"black or grey box\", this will affect everyone and it is still a fair playing ground. \n\n\n[things that I did wrong]\n\n1. Spend too much time, trying to find the \"perfect\" mask rcnn. I though that everything can be learned (including post processing). However, end-to-end solution doesn't exist here. I think here performance is limited by data quality, so trying different architecture, learning parameters etc doesn't seem to work.\n\nEvery \"wrong thing\" is a learning experience for me. Here, what I have learned is that when there is data label quality issue, we see \"same type of error \" in the training iterations, no matter what i do (e.g. change network structure, learning parameters). \n \nDuring the search for better architecture, I built a one stage ssd-mask-instance segmentation. I find that 2 stage rcnn is indeed better than one stage ssd method. The rcnn helps to filter wrong results of rpn. In ssd , there is no such mechanism. \n\nI also just trained the faster-rcnn only. Joint loss of detection and segmentation of mask rcnn is better than faster-rcnn alone. The reduction in classification loss (cross entropy) is quite obvious.\n\n\n\n[if i would to start again, what would i do?]\n\n1. Spend more time on clustering and ensemble. Despite limited by data quality, ensemble always work for me. But this is tricky. Because although results visually look better, it not necessary have better LB score (which i think due to some missing annotation in ground truth of LB set).\n\n2. Spend more time on UNet. It turns out that Unet has smaller segmentation error (if you just consider semantic segmentation and not the instance separation). The best solution is combination of two, though i haven't find a way to do it efficiently and effectively.\n\nSurprisingly, simple unet+post processing can work almost the same as mask-rcnn from LB score point of view. It is not always necessarily to use the latest tool, but you do have to be good at your tool.\n\n3. Collect more data. When I read about nerve cells, I thought they look very different from the stage1 dataset and think that it wouldn't appear in stage2. I should have include that in my external data, or at least test it.\n\n4. More visualization tool. This will help to analysis the error.\n\n\n---\n[Final thoughts]\n\n1. This may be a case where we don't have to trust the LB too much. Human annotation error is about 0.01 and 0.03. small improvement like 0.001 may or may not real improvement at all. When you compare algorithms, do consider the meaning of \"statistical significance\"\n\n2. This is also one of the rare case where you get to test a dataset and then you are given the ground truth label (stage1 test). Spend some time to analyse what has gone right or wrong during your \"competition process\" for stage1. We always don't know why the LB score is like this or like that, now you can!\n\n3. This is also a rare case where you get to test your algorithm in \"unexpected\" scope. Use this chance to understand why algorithm break down. There will also be surprise that it actually works in some unusual cases. It is a good chance for you to understand what is \"robustness\".\n\n Unexpectedness bring about breakthrough. Sit down, think about it and make a summary write up. \n", "313024": "Wow. Awesome survey, Heng. \n\nI’ve **really** enjoyed your posts throughout the competition and this was no exception. ", "313176": "FYI, working 3 days with a fresh team we could get Unet score at 0.504, and I'm sure we could move higher with few more days. And this is using the simple Unet starter NN, without pretrained weights. I'm convinced Unet can yield way beter results here.", "313289": "Hello @Heng, thank you for your posts on the challenge. I have learned a lot of things from your posts. Although my topic is semantic segmentation, it is the good chance for me to try instance segmentation in this challenge.\n \nIn this challenge, we have to solve many problems: limited dataset, noisy label, overlap nucleus, multi-modalities... Hence, external data, the strategy to train network (against overfitting) post-processing, new network are key components to improve performance. In the stage-1 test, I achieved 0.553 LB with my end-to-end mask-rcnn (modified mask-rcnn) without any post-processing. I think the score may be better when I apply post-processing (likes your ensembling).\n\n I will post some results on stage2 test in your another thread", "313030": "(There’s a lot in this post I want to comment on, but I’m going to wait until the competition is over …) ", "313102": "Heng CherKeng, you did an amazing job in this competition. You really deserve to win it !\n\nI am also developping commercial products in my everyday job ; and I think that you pointed out the real important thing:\n\"It is not always necessarily to use the latest tool, but you do have to be good at your tool\"\n\nActually I wish that I had used more classical approaches like shape constrained level sets. The temptation with Kaggle competition is always to developp an end-to-end solution, spend too much time on retraining, hyperparmaters tuning, etc. \n\nI used a cascade of unets: unet on original image -> resize -> second unet on scale normalized image -> third unet on predicted cores to get their centers. This approach gave me a score around 0.54 on the LB without too much post processing (actually only removing small cells). \n\nI expected new color images but it did not append. I guess the model will perform pretty badly on the new images.\n", "313076": "I put UNET down very early. I always meant to go back to it. I have learned a lot from Heng as well. I did not use your code or your methodology per se, but you made me think about many issues I would have passed over but for your sharing. Thanks!", "313068": "Thank you for all the contribution in this competition, I learned a lot from you!", "313669": "Yeah, I got the understanding that no amount of tinkering with any nets, especially those learning boundaries is gonna work here right away when I looked at the data. It was quite a frustration having to revision all of those annotations https://github.com/ibmua/data-science-bowl-2018-train-set --- btw, do use those for final sub. I didn't look much at losses, only at visualizations of what actually my net displays. Qualitative analysis, not quantitative. That one's not gonna work for a task like this. And it also seemed apparent that mask rcnn is something way too fancy for this task.\n\nBlack box was quite a gamble. At the end it's like the more you took Anne's advice to heart the harder you were punched. If I didn't, I'd have a really good submission by now. But I chased the wind instead.", "316488": "It is a brilliant post from the experienced kaggler! Thanks a lot!", "315832": "Thanks, Heng! Really tremendous effort on the forum. I can't imagine how you combine kaggling and primary work =)", "314298": "Thx for sharing", "314111": "@Heng Thanks for all the comments. I didn't get to spend enough time on this competition but I certainly look forward to following your posts on the next one! :)", "313621": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?", "313307": "Thank you very much! We all learned a lot from your ideas and codes! It is my first time to compete KAGGLE. Bros here are very nice, say good words and I like this place very much! Doge", "313215": "@Heng thanks for this post and all you have shared in this competition. I did not use pytorch but certainly learnt from your codes.\n\nOur mask RCNN performed much better than UNET but we were able to get the UNET model from LB=0.353 to 0.408 within 3 days with fresh set of eyes after teaming up. Not as impressive as CPMP's 0.504 but we are proud of it. ", "313155": "Heng, nice posts throughout the competition. I learned a lot of new image augmentation and ensembling methods from your posts (which was my goal for this competition). Thanks for being awesome!", "313130": "Thank you, Heng. I learned a lot from you, not just from this competition. ", "313122": "Thanks, Heng. For me the thoughts and code you provided is the most valuable thing I earned from this competition. Looking forward to seeing you in the next competition", "313119": "Hi Heng\nI just want to say that you are awesome, thank you for sharing the high quality code and clear ideas, I learned a lot from you, thank you", "314271": "I'm quite fresh to Kaggle, but so far the best thing for me about this competition has been learning from the discussions and insights being shared by yourself and all the other teams. Thank you!", "314329": "Thanks for sharing!" } }