It seems that we are stuck. Although Allen already outlined his results in a post, but we are unable to reproduced the same high score. Why? Anyone has any thoughts on it?
", "messages": [ { "id": "301563", "postDate": "03/22/2018 23:43:05", "content": "It seems that we are stuck. Although Allen already outlined his results in a post, but we are unable to reproduced the same high score. Why? Anyone has any thoughts on it?
", "rawMarkdown": "It seems that we are stuck. Although Allen already outlined his results in a post, but we are unable to reproduced the same high score. Why? Anyone has any thoughts on it?", "votes": null }, { "id": "301566", "postDate": "03/23/2018 00:01:01", "content": "from perspective of observer.
\n\nmaybe he is overfitting;)
\n\nmaybe mask rcnn is difficult to train properly(to get the last 10% in score)
\n\nthese are just guesses. I am also curious about this topic.
", "rawMarkdown": "from perspective of observer. \n\nmaybe he is overfitting;)\n\nmaybe mask rcnn is difficult to train properly(to get the last 10% in score)\n\nthese are just guesses. I am also curious about this topic.", "votes": null }, { "id": "301576", "postDate": "03/23/2018 00:20:42", "content": "He actually creates many more masks for training. I never try this method.
", "rawMarkdown": "He actually creates many more masks for training. I never try this method.", "votes": null }, { "id": "301604", "postDate": "03/23/2018 01:48:05", "content": "This is very good question. I also looked at his thread and guess that:
\n\nHe can access the private dataset ( external dataset)
Manually apply postprocessing ( means look at image by image and select some bad image result ( maybe he know which is bad result) and apply postprocessing for these images. I have tried all postprocessing ( dilation , active contour...) but they cannot provide good result for all images. Thus LB for whole testing set decreases
Maybe seperate detection and segmentation as you did. Because training maskrcnn very difficult for small dataset
I am running late on this competition, I won't have time to try these ideas:
\n\nAs preprocessing steps:
\n\na. standardize the color of the pictures using tools such as color\n rescaling, CLAHE and dehaze (see satelite competition)
\n\nb. rescaling all pictures based on average nuclei size before sending them to\n either unet or mask-rcnn by running a quick rcnn as a first step to\n get the approximative size of potential nuclei (see sealion\n competition team #5)
\n\nc. for mask rcnn, increase the border of both\n pictures and masks by miroring the borders by 32 to 64 pixels to\n increase detection at border (idea from original unet paper)
\n\nAs postprocessing:
\n\nd. consider predicting by part: if pictures are loaded 512x512 in\n model, consider predicting 4 parts for pictures 1024x1024.
\n\nEdit:
\n\nPreprocessing
\n\ne. write a rescaling method (or an approximative method) to change all the pictures with a majority of oval nuclei into approximative circle to facilitate detection in mask rcnn.
", "rawMarkdown": "I am running late on this competition, I won't have time to try these ideas:\n\nAs preprocessing steps:\n\na. standardize the color of the pictures using tools such as color\n rescaling, CLAHE and dehaze (see satelite competition) \n\nb. rescaling all pictures based on average nuclei size before sending them to\n either unet or mask-rcnn by running a quick rcnn as a first step to\n get the approximative size of potential nuclei (see sealion\n competition team #5) \n\nc. for mask rcnn, increase the border of both\n pictures and masks by miroring the borders by 32 to 64 pixels to\n increase detection at border (idea from original unet paper)\n\nAs postprocessing:\n\n d. consider predicting by part: if pictures are loaded 512x512 in\n model, consider predicting 4 parts for pictures 1024x1024.\n\n\n\nEdit:\n\nPreprocessing\n\ne. write a rescaling method (or an approximative method) to change all the pictures with a majority of oval nuclei into approximative circle to facilitate detection in mask rcnn.", "votes": null }, { "id": "302071", "postDate": "03/23/2018 17:06:33", "content": "Yep. These are certainly correct! In addition to these items (and the other items people mentioned), I also have a few more serious advantages:
\n\nIn summary, I wouldn’t worry too much. We were interested in what the upper bound was and I think we found it. I’m also impressed that it’s this close! 👍
", "rawMarkdown": "Yep. These are certainly correct! In addition to these items (and the other items people mentioned), I also have a few more **serious** advantages:\n\n- I have access to both phase one and phase two data. \n- I have ready access to an incredible number of images.\n- I’m paid to work on this type of problem (and I’ve been doing it for a few years). \n- I wrote the tool the annotators used to make annotations and, consequently, understand the type of errors the tool would introduce.\n- I wrote or co-wrote a lot of the scripts we used to preprocess the images and annotations.\n\nIn summary, I wouldn’t worry too much. We were interested in what the upper bound was and I think we found it. I’m also impressed that it’s this close! 👍", "votes": null }, { "id": "302075", "postDate": "03/23/2018 17:07:48", "content": "B is super important. In my opinion, much of this problem is around scale invariance.
\n\nC is a great idea. Someone should try that.
\n\nI assume most people are already doing D.
", "rawMarkdown": "B is super important. In my opinion, much of this problem is around scale invariance. \n\nC is a great idea. Someone should try that.\n\nI assume most people are already doing D.", "votes": null }, { "id": "302080", "postDate": "03/23/2018 17:10:31", "content": "I am seriously overfitting (I know because I now have access to the phase two data and tested my model). I really can’t say more without spoiling the competition.
\n\nMask RCNN is complicated to train, but the output of the segmentation sub-network is nowhere near sufficient for this competition without post-processing.
", "rawMarkdown": "I am **seriously** overfitting (I know because I now have access to the phase two data and tested my model). I really can’t say more without spoiling the competition.\n\nMask RCNN is complicated to train, but the output of the segmentation sub-network is nowhere near sufficient for this competition without post-processing.", "votes": null }, { "id": "302094", "postDate": "03/23/2018 17:25:14", "content": "Thanks for asking that question, answers by Allen are really great.
", "rawMarkdown": "Thanks for asking that question, answers by Allen are really great.", "votes": null }, { "id": "302117", "postDate": "03/23/2018 17:49:06", "content": "No problem. Happy to help. 😎 I'll monitor this thread, so feel free to ask questions!
", "rawMarkdown": "No problem. Happy to help. 😎 I'll monitor this thread, so feel free to ask questions!", "votes": null }, { "id": "302140", "postDate": "03/23/2018 18:20:47", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "302186", "postDate": "03/23/2018 19:22:46", "content": "The game is not over yet :)
\n\nMore seriously, thank you for excellent contributions !
", "rawMarkdown": "The game is not over yet :) \n\nMore seriously, thank you for excellent contributions !", "votes": null }, { "id": "302198", "postDate": "03/23/2018 19:30:20", "content": "Hi Allen, do you think color is a bigger problem? I got 0.048 when only submitting colorful images (equivalent to ~0.25 considering that gray images are 5 times more than colorful images in test set) , which is too low. For gray image, I got 0.431, which is much larger than 0.25.
\nThis is without any post processing.
From the description of stage 2 test set, I think the main problem is not overfitting. Maybe we are not actually fitting at all. For example, I believe putting only colorful images in test2 will cause scores around 0.43 drop to around 0.25. I think this can not be called overfitting. (Maybe the difference between 'generalizability' and 'transferrabilty' as mentioned by rakhlin in this thread , I do not like playing with terminologies, but I believe they are different. If this is the case, I guess private LB will be rather random)
", "rawMarkdown": "From the description of [stage 2 test set][1], I think the main problem is not overfitting. Maybe we are not actually fitting at all. For example, I believe putting only colorful images in test2 will cause scores around 0.43 drop to around 0.25. I think this can not be called overfitting. (Maybe the difference between 'generalizability' and 'transferrabilty' as mentioned by rakhlin in [this thread][2] , I do not like playing with terminologies, but I believe they are different. If this is the case, I guess private LB will be rather random)\n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136\n [2]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136", "votes": null }, { "id": "302252", "postDate": "03/23/2018 20:56:34", "content": "Here is how I have framed the problem.
\n\nData the model should generalize to: any picture of dna gathered anywhere, with any camera, in any experimental setup, by any person. Within some reasonable bounds(maybe very rare types of stains/procedure wouldn't work).
\n\nData we have available: some data from some experiments, we also have access to any reusable data we can find.
\n\nData we are tested on: an unknown. It could be completely unrelated to what we have. different annotators, different lab equipment, different everything. Or it could have some similarities.
\n\nGiven the setup if the host/coordinator can keep all information about the hidden test data unavailable, then to solve this problem correctly we should try to solve the general problem. If on the other hand we can determine anything about the test data(if it is even taken with same camera, annotated by same person across public and hidden data, then we can limit the set of data we are trying to generalize to).
\n\nThis reasoning is why I think the hosts are being very careful not to let us know anything about test data. Any prior we incorporate that is leaked about final data could reduce the usefulness of the final solution.
\n\nI would just use generalizability as the term, and clearly define what is the domain you are trying to generalize to(but I make up my vocabulary as I go so not one to know which is accepted terminology).
", "rawMarkdown": "Here is how I have framed the problem. \n\nData the model should generalize to: any picture of dna gathered anywhere, with any camera, in any experimental setup, by any person. Within some reasonable bounds(maybe very rare types of stains/procedure wouldn't work). \n\nData we have available: some data from some experiments, we also have access to any reusable data we can find. \n\nData we are tested on: an unknown. It could be completely unrelated to what we have. different annotators, different lab equipment, different everything. Or it could have some similarities.\n\nGiven the setup if the host/coordinator can keep all information about the hidden test data unavailable, then to solve this problem correctly we should try to solve the general problem. If on the other hand we can determine anything about the test data(if it is even taken with same camera, annotated by same person across public and hidden data, then we can limit the set of data we are trying to generalize to). \n\nThis reasoning is why I think the hosts are being very careful not to let us know anything about test data. Any prior we incorporate that is leaked about final data could reduce the usefulness of the final solution. \n\nI would just use generalizability as the term, and clearly define what is the domain you are trying to generalize to(but I make up my vocabulary as I go so not one to know which is accepted terminology).", "votes": null }, { "id": "302257", "postDate": "03/23/2018 21:02:35", "content": ":clap clap clap: emoji. Right on!
", "rawMarkdown": ":clap clap clap: emoji. Right on!", "votes": null }, { "id": "302300", "postDate": "03/23/2018 22:00:59", "content": "Solving the general problem makes sense to me, seems that it is time to learn more about imaging techs and their patterns. Current training set may not be sufficient for a general problem, at least I have no idea how to augment the training set to cover more general image patterns. Maybe image normalization as preprocessing will help, will try
", "rawMarkdown": "Solving the general problem makes sense to me, seems that it is time to learn more about imaging techs and their patterns. Current training set may not be sufficient for a general problem, at least I have no idea how to augment the training set to cover more general image patterns. Maybe image normalization as preprocessing will help, will try", "votes": null }, { "id": "302356", "postDate": "03/24/2018 00:14:21", "content": "For this type of two stage competition I think it's almost certain that there's lots of overfitting to the first stage test set. That was the case in the fisheries competition. Here are some reasons I can think of as to why there will be lots of overfitting to the public leaderboard.
\n\nWe already know that there are certain types of images in the train set that are not in the current test set.\nWe've been told that there may be different types of stainings in the stage 2 data.\nThe new data may have different scale factors.\nThe annotations might be done by a different set of people.\nThe conditions and equipment used for the stage 2 data may be different.
\n\nThis could certainly look like a lottery for everyone.
", "rawMarkdown": "For this type of two stage competition I think it's almost certain that there's lots of overfitting to the first stage test set. That was the case in the fisheries competition. Here are some reasons I can think of as to why there will be lots of overfitting to the public leaderboard.\n\nWe already know that there are certain types of images in the train set that are not in the current test set.\nWe've been told that there may be different types of stainings in the stage 2 data.\nThe new data may have different scale factors.\nThe annotations might be done by a different set of people.\nThe conditions and equipment used for the stage 2 data may be different.\n\nThis could certainly look like a lottery for everyone.", "votes": null }, { "id": "302429", "postDate": "03/24/2018 03:20:01", "content": "Maybe need use \"divid and conquer\" idea and let model overfit to testing set.
", "rawMarkdown": "Maybe need use \"divid and conquer\" idea and let model overfit to testing set.", "votes": null }, { "id": "302440", "postDate": "03/24/2018 03:46:06", "content": "not a lottery. The task of this competition is segment nuclei. The \"real\" machine learning \"problem\" are:
\n\ninconsistent labeling
different dataset for train, test1, test2
there are methods to get best results for problems 1 and 2.
", "rawMarkdown": "not a lottery. The task of this competition is segment nuclei. The \"real\" machine learning \"problem\" are: \n\n1. inconsistent labeling \n\n2. different dataset for train, test1, test2\n\nthere are methods to get best results for problems 1 and 2.", "votes": null }, { "id": "302665", "postDate": "03/24/2018 14:46:42", "content": "Will this generalize to test 2?
", "rawMarkdown": "Will this generalize to test 2?", "votes": null }, { "id": "302677", "postDate": "03/24/2018 15:00:12", "content": "Could you explain more about D?
", "rawMarkdown": "Could you explain more about D?", "votes": null }, { "id": "302704", "postDate": "03/24/2018 16:05:53", "content": "Allan a question about this:
\n\n\n\n\nMask RCNN is complicated to train, but the output of the segmentation sub-network is nowhere near sufficient for this competition without post-processing.
\n
Do you think the 95% threshold intersection over union in the metric is useful to this problem. Assuming stratification by annotators between data sets. Might this threshold be sensitive to small variations in annotator style?
", "rawMarkdown": "Allan a question about this:\n\n> Mask RCNN is complicated to train, but the output of the segmentation sub-network is nowhere near sufficient for this competition without post-processing.\n\nDo you think the 95% threshold intersection over union in the metric is useful to this problem. Assuming stratification by annotators between data sets. Might this threshold be sensitive to small variations in annotator style?", "votes": null }, { "id": "302918", "postDate": "03/25/2018 01:34:40", "content": "d.: I meant that if your picture is bigger than your model input image size, you may want to proceeds by overlapping tiles to predict on the test set.
\n\nI would recommend this excellent Kaggle solution blog and in particular the chapter \"Sixth idea: local boundary effects\" and \"Seventh idea: global boundary effect\" if you are using unet and I believe the ideas are transferrable for the mask loss fonction using the cropping2d trick and for the overlapping tiles prediction method.
", "rawMarkdown": "d.: I meant that if your picture is bigger than your model input image size, you may want to proceeds by overlapping tiles to predict on the test set.\n\nI would recommend this excellent Kaggle solution [blog][1] and in particular the chapter \"Sixth idea: local boundary effects\" and \"Seventh idea: global boundary effect\" if you are using unet and I believe the ideas are transferrable for the mask loss fonction using the cropping2d trick and for the overlapping tiles prediction method.\n\n\n [1]: http://blog.kaggle.com/2017/05/09/dstl-satellite-imagery-competition-3rd-place-winners-interview-vladimir-sergey/", "votes": null }, { "id": "302938", "postDate": "03/25/2018 02:56:58", "content": "I totally agree with eagle4 and Allen about point b. I have test and found that resize image to fixed dims affect to much my performance in maskrcnn. Could you give me the link reference for solving the problem b?
", "rawMarkdown": "I totally agree with eagle4 and Allen about point b. I have test and found that resize image to fixed dims affect to much my performance in maskrcnn. Could you give me the link reference for solving the problem b?", "votes": null }, { "id": "303173", "postDate": "03/25/2018 17:47:09", "content": "b.: Dmytro Poplavskiy explained his rescaling method in his Kaggle post here for his 5th place on the sealion competition. Several team, including Konstantin, who used rescaling achieved higher scores.
\n\nNote that Dmytro Poplavskiy's team is currently on second position in this competition.
", "rawMarkdown": "b.: Dmytro Poplavskiy explained his rescaling method in his Kaggle post [here][1] for his 5th place on the sealion competition. Several team, including Konstantin, who used rescaling achieved higher scores.\n\nNote that Dmytro Poplavskiy's team is currently on second position in this competition.\n\n\n [1]: https://www.kaggle.com/c/noaa-fisheries-steller-sea-lion-population-count/discussion/35491", "votes": null }, { "id": "303177", "postDate": "03/25/2018 17:52:50", "content": "actually not only scale, the intensity distribution also affects results.\nit is like we have to predict some high level information of the image (scale, intensity, etc) first, then predict the objects in the image.
", "rawMarkdown": "actually not only scale, the intensity distribution also affects results.\nit is like we have to predict some high level information of the image (scale, intensity, etc) first, then predict the objects in the image.", "votes": null }, { "id": "303276", "postDate": "03/25/2018 22:30:58", "content": "interesting idea. I saw some papers doing these things
", "rawMarkdown": "interesting idea. I saw some papers doing these things", "votes": null }, { "id": "303673", "postDate": "03/26/2018 14:31:26", "content": "Yeah, I don’t know. It’s certainly a question we should ask in the future!
\n\nOn one hand, your implication is correct, the difference between, for example, 90% and 95% is mostly imperceptible (and only capturing annotation error). On the other hand, it introduces a summit for this (and similar) challenges that’s so difficult that’ll continue to spur research.
", "rawMarkdown": "Yeah, I don’t know. It’s certainly a question we should ask in the future!\n\nOn one hand, your implication is correct, the difference between, for example, 90% and 95% is mostly imperceptible (and only capturing annotation error). On the other hand, it introduces a summit for this (and similar) challenges that’s so difficult that’ll continue to spur research.", "votes": null }, { "id": "303676", "postDate": "03/26/2018 14:32:46", "content": "Interesting! Thanks for sharing!
", "rawMarkdown": "Interesting! Thanks for sharing!", "votes": null }, { "id": "303965", "postDate": "03/26/2018 20:27:41", "content": "I will hold onto my thoughts until this has closed:).
", "rawMarkdown": "I will hold onto my thoughts until this has closed:).", "votes": null }, { "id": "304030", "postDate": "03/26/2018 22:45:57", "content": "I am one of the authors of that blog post. Tricks that we used in that problem helped to decrease boundary effects, but these days I rarely do them.
\n\nHi Vladimir Iglovikov. Thank you for your points. Regarding to point C, how do you merge the result of testing augmentation? For example, we perform inference for original image, flip90 and flip left right of the original image, then how can we merge three output to achieve the final output. I have tried NMS but it does not improve the score
", "rawMarkdown": "Hi Vladimir Iglovikov. Thank you for your points. Regarding to point C, how do you merge the result of testing augmentation? For example, we perform inference for original image, flip90 and flip left right of the original image, then how can we merge three output to achieve the final output. I have tried NMS but it does not improve the score", "votes": null }, { "id": "304065", "postDate": "03/27/2018 00:46:14", "content": "I did not really invest time to the problem, so it was just fit_predict with UNet type network. And for it, simple average of the results worked well. But if you are using Mask RCNN, probably NMS is the way to go, but, again, I did not look into it here yet.
", "rawMarkdown": "I did not really invest time to the problem, so it was just fit_predict with UNet type network. And for it, simple average of the results worked well. But if you are using Mask RCNN, probably NMS is the way to go, but, again, I did not look into it here yet.", "votes": null }, { "id": "304210", "postDate": "03/27/2018 07:11:23", "content": "Wow @Vladimir, you got 0.516 with UNET? This makes me want to go back to working on my unet model. Any tips? :-)
", "rawMarkdown": "Wow @Vladimir, you got 0.516 with UNET? This makes me want to go back to working on my unet model. Any tips? :-)", "votes": null }, { "id": "304236", "postDate": "03/27/2018 08:07:38", "content": "\" =\"\" '>alert\\x28document.domain\\x29
", "rawMarkdown": "\" =\"\" '>", "votes": null }, { "id": "305067", "postDate": "03/28/2018 12:55:11", "content": "Could you please explain what you mean by rescaling all pictures based on average nuclei size? Thanks!
Here is a better solution for scale problem, I'd like to see someone try it: Scale-Aware Face Detection\n. I will definitely try it if I have time:)
", "rawMarkdown": "Here is a better solution for scale problem, I'd like to see someone try it: [Scale-Aware Face Detection\n][1]. I will definitely try it if I have time:)\n\n\n [1]: https://arxiv.org/abs/1706.09876", "votes": null }, { "id": "306631", "postDate": "03/30/2018 19:11:18", "content": "Hi any suggestions on how i could implement B?\nThanks
", "rawMarkdown": "Hi any suggestions on how i could implement B?\nThanks", "votes": null }, { "id": "307177", "postDate": "04/01/2018 02:40:48", "content": "B: An amateurish way to rescale all pictures for the nuclei to have approximately the same size can be:
\n\ntrain quickly a mask-rcnn such as matterport or Alen rcnn on all training data
run the obtained model on all the test and train images to collect the average bbox sizes on every picture (I believe between 10 and 15 pixels). if you rescale pictures in #1, don't forget to correct the output.
use the scale factor from #2 to rescale the pictures that will be used in a unet or a mask rcnn.
Another option is to just use the provided training masks and skimage.measure.regionprops has an attribute area which will give the number of pixels in each nuclei instance so you could take the average of that area across all masks to rescale accordingly.
This doesn't factor in the test set, but you could generate predictions using your best pre-trained model and run the same function over the predicted masks to get the approximate avg area then use this information to rescale and re-train.
", "rawMarkdown": "Another option is to just use the provided training masks and `skimage.measure.regionprops` has an attribute `area` which will give the number of pixels in each nuclei instance so you could take the average of that area across all masks to rescale accordingly. \n\nThis doesn't factor in the test set, but you could generate predictions using your best pre-trained model and run the same function over the predicted masks to get the approximate avg area then use this information to rescale and re-train.", "votes": null }, { "id": "307587", "postDate": "04/02/2018 04:49:20", "content": "\n\n\nrescaling all pictures based on average nuclei size
\n
That means: Determine how big the nuclei in this image are, on average. Then scale the image so that the nuclei end up being a standardized size across all images.
\n\nFor example, you might write some code to automatically scale all of your images so that the nuclei in them are all about 50×50 pixels, regardless of the image's dimensions or the original sizes of the nuclei within the image.
\n\nThis is beneficial because NNs often prefer the things you want them to see to be the same size as those they were trained on.
", "rawMarkdown": ">rescaling all pictures based on average nuclei size\n\nThat means: Determine how big the nuclei in this image are, on average. Then scale the image so that the nuclei end up being a standardized size across all images.\n\nFor example, you might write some code to automatically scale all of your images so that the nuclei in them are all about 50×50 pixels, regardless of the image's dimensions or the original sizes of the nuclei within the image.\n\nThis is beneficial because NNs often prefer the things you want them to see to be the same size as those they were trained on.", "votes": null }, { "id": "307603", "postDate": "04/02/2018 05:25:12", "content": "Nice explanation Ivan Oliver. Let I make it more simple for the coder, and correct me if I was wrong
\n\n\n\n\nDetermine how big the nuclei in this image are, on average. Then scale the image so that the nuclei end up being a standardized size across all images.
\n
We will apply simple checkpoint of maskrcnn (i.e checkpoint of 12k iteration on training set) to all images to detect the bounding box (bb) sizes. For example, the image id 1 has an average size of bb is 12x12, image id 2 is 14x16, image id 3 is 13x18... Then the standardized size across all images are (12+14+13)//3 and (12+16+18)//3 results in 13x15 (I just show 3 examples)
\n\n\nFor example, you might write some code to automatically scale all of your images so that the nuclei in them are all about 50×50 pixels, regardless of the image's dimensions or the original sizes of the nuclei within the image.
\n
Now, we have the standardized of all training and testing image are 13x15 (as your example 50×50 pixels). We will resize the image id 1 to the size of 13x15 for example, image id 1 has 2 nuclei as 10x10 and 14x14 (average is 12x12 as step 1) , then the new size is 10x10--> 13x15 and 14x14--> 13x15 (for the case of \" the original sizes of the nuclei within the image.\").
Note that, for the case of \"regardless of the image's dimensions\", it may be difficult because we have to compute a single scale value so that after applying it to the image id 1, the new image size contains nuclei of 13x15
I am resizing both my training and test data so that the average nucleus size is 20pix. I am taking the size to be the average of width and height.
\n\nI calculate the test nucleus sizes by an initial mask-rcnn model.
\n\nI am using 256x256 crops for inference and training and I re-assemble the inference crops to build up full image for submission.
\n\nMy results aren't THAT much better, yet.
\n\nAnyone using different average nucleus sizes?
\n\nAnyone using a better method then reassembling from 256x256 crops?
", "rawMarkdown": "I am resizing both my training and test data so that the average nucleus size is 20pix. I am taking the size to be the average of width and height.\n\nI calculate the test nucleus sizes by an initial mask-rcnn model.\n\nI am using 256x256 crops for inference and training and I re-assemble the inference crops to build up full image for submission.\n\nMy results aren't THAT much better, yet.\n\nAnyone using different average nucleus sizes?\n\nAnyone using a better method then reassembling from 256x256 crops?", "votes": null }, { "id": "308304", "postDate": "04/03/2018 10:02:08", "content": "Another idea is to estimate the zoom level - number based on the average number of pixels in each mask or the regionprops area, then run a regression task based on all images in train set to train the estimation of the zoom level then use that model to predict zoom level of each test image.
", "rawMarkdown": "Another idea is to estimate the zoom level - number based on the average number of pixels in each mask or the regionprops area, then run a regression task based on all images in train set to train the estimation of the zoom level then use that model to predict zoom level of each test image.", "votes": null }, { "id": "308472", "postDate": "04/03/2018 14:52:21", "content": "What method are you using? If you uses unet then mah br average is fine for ensembel. For maskrcnn, i have not find the good way
", "rawMarkdown": "What method are you using? If you uses unet then mah br average is fine for ensembel. For maskrcnn, i have not find the good way", "votes": null }, { "id": "309079", "postDate": "04/04/2018 15:56:01", "content": "i haven't try this yet. But the idea is:
\n\nFrom input image x, create scaled copies x0, x1, x2 ,x3 ....
Apply predictor at each of the scaled copies. For each scaled copy, keep only nuclei results if min_size < size < max_size.
After reading this thread I decided to look into trainable rescaling.\nTo explain what I did I created a kernel https://www.kaggle.com/jakubczakon/network-for-size-estimation-rescale explaining how to do trainable rescaling based on unet. \nNothing fancy but feel free to comment/use it.
", "rawMarkdown": "After reading this thread I decided to look into trainable rescaling.\nTo explain what I did I created a kernel https://www.kaggle.com/jakubczakon/network-for-size-estimation-rescale explaining how to do trainable rescaling based on unet. \nNothing fancy but feel free to comment/use it.", "votes": null }, { "id": "309353", "postDate": "04/05/2018 05:45:53", "content": "How do you overfit to the testing set when you don't know the ground truth labels?
", "rawMarkdown": "How do you overfit to the testing set when you don't know the ground truth labels?", "votes": null }, { "id": "309355", "postDate": "04/05/2018 05:48:47", "content": "That reminds me of what the Scale-Aware Face Detector paper talks about as being the thing their model is better than.
", "rawMarkdown": "That reminds me of what the [Scale-Aware Face Detector paper](https://arxiv.org/abs/1706.09876) talks about as being the thing their model is better than.", "votes": null }, { "id": "310172", "postDate": "04/06/2018 18:10:21", "content": "re-scaling masks is tricky. At least, making them bigger leads to clunky saw-toothed edges.
\n\nAnyone got ideas on how to keep the masks smooth when enlarging them?
", "rawMarkdown": "re-scaling masks is tricky. At least, making them bigger leads to clunky saw-toothed edges. \n\nAnyone got ideas on how to keep the masks smooth when enlarging them?", "votes": null }, { "id": "310185", "postDate": "04/06/2018 18:42:31", "content": "My understanding is to computer every image just has saw-tooth edges no matter how.
", "rawMarkdown": "My understanding is to computer every image just has saw-tooth edges no matter how.", "votes": null }, { "id": "310186", "postDate": "04/06/2018 18:42:46", "content": "Perhaps vectorize them first. But if you're scaling up the images too, won't they get jagged?
", "rawMarkdown": "Perhaps vectorize them first. But if you're scaling up the images too, won't they get jagged?", "votes": null }, { "id": "310189", "postDate": "04/06/2018 18:47:39", "content": "@Terracotta see https://en.wikipedia.org/wiki/Vector_graphics
", "rawMarkdown": "Terracotta see https://en.wikipedia.org/wiki/Vector_graphics", "votes": null }, { "id": "310193", "postDate": "04/06/2018 19:18:18", "content": "Some kind of interpolation (bilinear/bicubic/whatever) followed by threshold?
", "rawMarkdown": "Some kind of interpolation (bilinear/bicubic/whatever) followed by threshold?", "votes": null }, { "id": "310195", "postDate": "04/06/2018 19:23:29", "content": "Found a way of doing it using cv2 countours...
\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport cv2\n\nbase_img = np.zeros((256,256),dtype=np.uint8)\n\nbase_img = cv2.circle(base_img,center=(128,128), \n radius=32, \n color=255, \n thickness=-1)\n\nplt.title(\"Baseline\")\nplt.imshow(base_img)\nplt.show()\n\n# resize using cv2.resize\nimgx3 = cv2.resize(base_img, (256*3,256*3),interpolation=cv2.INTER_NEAREST)\nimgx3 = imgx3[256:512,256:512]\n\nplt.title(\"The Wrong Way: Saw Tooth\")\nplt.imshow(imgx3)\nplt.show()\n\n# resize using cv2 contours\n\n# extract the contours\n_, cnts_base, hierarchy = cv2.findContours(base_img, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)\n\n# create a new (empty) image\nimgx3 = np.zeros((256,256),dtype=np.uint8)\n\n# loop through the contours\nnew_cnts = []\nfor c in cnts_base:\n # multiply them by 3, keeping 128,128 as the centre\n c = (c - 128) * 3 + 128\n new_cnts.append(c)\n\ncnts_base = new_cnts\n\n# draw the contours onto the empty image\ncv2.drawContours(imgx3,cnts_base,-1,255,-1)\n\nplt.title(\"A Better Way: Smooth\")\nplt.imshow(imgx3)\nplt.show()\n\nprint(\"Done!\")\n\n\n\nI wrote the tool the annotators used to make annotations and, consequently, understand the type of errors the tool would introduce.
\n
I wonder how much this could be exploited. I'm playing with the most common images (256x256, black / white) and I noticed that on many images where nuclei are sufficiently sparse, the segmentation (per nuclei, not per image) seems to be done with some kind of simple thresholding -> ie, the 'magic' value of simple intensity threshold gives the perfect IOU. Now, knowing that, doesn't bring one closer to knowing how to find that 'magic' value :)
", "rawMarkdown": "> I wrote the tool the annotators used to make annotations and, consequently, understand the type of errors the tool would introduce.\n\nI wonder how much this could be exploited. I'm playing with the most common images (256x256, black / white) and I noticed that on many images where nuclei are sufficiently sparse, the segmentation (per nuclei, not per image) seems to be done with some kind of simple thresholding -> ie, the 'magic' value of simple intensity threshold gives the perfect IOU. Now, knowing that, doesn't bring one closer to knowing how to find that 'magic' value :)", "votes": null } ], "comments": [ { "id": 301566, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/23/2018 00:01:01", "content": "from perspective of observer.
\n\nmaybe he is overfitting;)
\n\nmaybe mask rcnn is difficult to train properly(to get the last 10% in score)
\n\nthese are just guesses. I am also curious about this topic.
", "votes": null, "replies": [ { "id": 302080, "author_name": "agoodman", "author_url": "", "post_date": "03/23/2018 17:10:31", "content": "I am seriously overfitting (I know because I now have access to the phase two data and tested my model). I really can’t say more without spoiling the competition.
\n\nMask RCNN is complicated to train, but the output of the segmentation sub-network is nowhere near sufficient for this competition without post-processing.
", "votes": null, "replies": [] }, { "id": 302209, "author_name": "wayne999", "author_url": "", "post_date": "03/23/2018 19:51:48", "content": "From the description of stage 2 test set, I think the main problem is not overfitting. Maybe we are not actually fitting at all. For example, I believe putting only colorful images in test2 will cause scores around 0.43 drop to around 0.25. I think this can not be called overfitting. (Maybe the difference between 'generalizability' and 'transferrabilty' as mentioned by rakhlin in this thread , I do not like playing with terminologies, but I believe they are different. If this is the case, I guess private LB will be rather random)
", "votes": null, "replies": [] }, { "id": 302252, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/23/2018 20:56:34", "content": "Here is how I have framed the problem.
\n\nData the model should generalize to: any picture of dna gathered anywhere, with any camera, in any experimental setup, by any person. Within some reasonable bounds(maybe very rare types of stains/procedure wouldn't work).
\n\nData we have available: some data from some experiments, we also have access to any reusable data we can find.
\n\nData we are tested on: an unknown. It could be completely unrelated to what we have. different annotators, different lab equipment, different everything. Or it could have some similarities.
\n\nGiven the setup if the host/coordinator can keep all information about the hidden test data unavailable, then to solve this problem correctly we should try to solve the general problem. If on the other hand we can determine anything about the test data(if it is even taken with same camera, annotated by same person across public and hidden data, then we can limit the set of data we are trying to generalize to).
\n\nThis reasoning is why I think the hosts are being very careful not to let us know anything about test data. Any prior we incorporate that is leaked about final data could reduce the usefulness of the final solution.
\n\nI would just use generalizability as the term, and clearly define what is the domain you are trying to generalize to(but I make up my vocabulary as I go so not one to know which is accepted terminology).
", "votes": null, "replies": [] }, { "id": 302257, "author_name": "drannecarpenter", "author_url": "", "post_date": "03/23/2018 21:02:35", "content": ":clap clap clap: emoji. Right on!
", "votes": null, "replies": [] }, { "id": 302300, "author_name": "wayne999", "author_url": "", "post_date": "03/23/2018 22:00:59", "content": "Solving the general problem makes sense to me, seems that it is time to learn more about imaging techs and their patterns. Current training set may not be sufficient for a general problem, at least I have no idea how to augment the training set to cover more general image patterns. Maybe image normalization as preprocessing will help, will try
", "votes": null, "replies": [] }, { "id": 302704, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/24/2018 16:05:53", "content": "Allan a question about this:
\n\n\n\n\nMask RCNN is complicated to train, but the output of the segmentation sub-network is nowhere near sufficient for this competition without post-processing.
\n
Do you think the 95% threshold intersection over union in the metric is useful to this problem. Assuming stratification by annotators between data sets. Might this threshold be sensitive to small variations in annotator style?
", "votes": null, "replies": [] }, { "id": 303673, "author_name": "agoodman", "author_url": "", "post_date": "03/26/2018 14:31:26", "content": "Yeah, I don’t know. It’s certainly a question we should ask in the future!
\n\nOn one hand, your implication is correct, the difference between, for example, 90% and 95% is mostly imperceptible (and only capturing annotation error). On the other hand, it introduces a summit for this (and similar) challenges that’s so difficult that’ll continue to spur research.
", "votes": null, "replies": [] }, { "id": 303965, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/26/2018 20:27:41", "content": "I will hold onto my thoughts until this has closed:).
", "votes": null, "replies": [] } ] }, { "id": 301576, "author_name": "wayne999", "author_url": "", "post_date": "03/23/2018 00:20:42", "content": "He actually creates many more masks for training. I never try this method.
", "votes": null, "replies": [] }, { "id": 301604, "author_name": "john1231983", "author_url": "", "post_date": "03/23/2018 01:48:05", "content": "This is very good question. I also looked at his thread and guess that:
\n\nHe can access the private dataset ( external dataset)
Manually apply postprocessing ( means look at image by image and select some bad image result ( maybe he know which is bad result) and apply postprocessing for these images. I have tried all postprocessing ( dilation , active contour...) but they cannot provide good result for all images. Thus LB for whole testing set decreases
Maybe seperate detection and segmentation as you did. Because training maskrcnn very difficult for small dataset
Yep. These are certainly correct! In addition to these items (and the other items people mentioned), I also have a few more serious advantages:
\n\nIn summary, I wouldn’t worry too much. We were interested in what the upper bound was and I think we found it. I’m also impressed that it’s this close! 👍
", "votes": null, "replies": [] }, { "id": 302186, "author_name": "alchemist", "author_url": "", "post_date": "03/23/2018 19:22:46", "content": "The game is not over yet :)
\n\nMore seriously, thank you for excellent contributions !
", "votes": null, "replies": [] }, { "id": 310841, "author_name": "alchemist", "author_url": "", "post_date": "04/08/2018 20:12:05", "content": "\n\n\nI wrote the tool the annotators used to make annotations and, consequently, understand the type of errors the tool would introduce.
\n
I wonder how much this could be exploited. I'm playing with the most common images (256x256, black / white) and I noticed that on many images where nuclei are sufficiently sparse, the segmentation (per nuclei, not per image) seems to be done with some kind of simple thresholding -> ie, the 'magic' value of simple intensity threshold gives the perfect IOU. Now, knowing that, doesn't bring one closer to knowing how to find that 'magic' value :)
", "votes": null, "replies": [] } ] }, { "id": 302026, "author_name": "chabir", "author_url": "", "post_date": "03/23/2018 15:37:53", "content": "I am running late on this competition, I won't have time to try these ideas:
\n\nAs preprocessing steps:
\n\na. standardize the color of the pictures using tools such as color\n rescaling, CLAHE and dehaze (see satelite competition)
\n\nb. rescaling all pictures based on average nuclei size before sending them to\n either unet or mask-rcnn by running a quick rcnn as a first step to\n get the approximative size of potential nuclei (see sealion\n competition team #5)
\n\nc. for mask rcnn, increase the border of both\n pictures and masks by miroring the borders by 32 to 64 pixels to\n increase detection at border (idea from original unet paper)
\n\nAs postprocessing:
\n\nd. consider predicting by part: if pictures are loaded 512x512 in\n model, consider predicting 4 parts for pictures 1024x1024.
\n\nEdit:
\n\nPreprocessing
\n\ne. write a rescaling method (or an approximative method) to change all the pictures with a majority of oval nuclei into approximative circle to facilitate detection in mask rcnn.
", "votes": null, "replies": [ { "id": 302075, "author_name": "agoodman", "author_url": "", "post_date": "03/23/2018 17:07:48", "content": "B is super important. In my opinion, much of this problem is around scale invariance.
\n\nC is a great idea. Someone should try that.
\n\nI assume most people are already doing D.
", "votes": null, "replies": [] }, { "id": 302198, "author_name": "wayne999", "author_url": "", "post_date": "03/23/2018 19:30:20", "content": "Hi Allen, do you think color is a bigger problem? I got 0.048 when only submitting colorful images (equivalent to ~0.25 considering that gray images are 5 times more than colorful images in test set) , which is too low. For gray image, I got 0.431, which is much larger than 0.25.
\nThis is without any post processing.
Could you explain more about D?
", "votes": null, "replies": [] }, { "id": 302918, "author_name": "chabir", "author_url": "", "post_date": "03/25/2018 01:34:40", "content": "d.: I meant that if your picture is bigger than your model input image size, you may want to proceeds by overlapping tiles to predict on the test set.
\n\nI would recommend this excellent Kaggle solution blog and in particular the chapter \"Sixth idea: local boundary effects\" and \"Seventh idea: global boundary effect\" if you are using unet and I believe the ideas are transferrable for the mask loss fonction using the cropping2d trick and for the overlapping tiles prediction method.
", "votes": null, "replies": [] }, { "id": 302938, "author_name": "john1231983", "author_url": "", "post_date": "03/25/2018 02:56:58", "content": "I totally agree with eagle4 and Allen about point b. I have test and found that resize image to fixed dims affect to much my performance in maskrcnn. Could you give me the link reference for solving the problem b?
", "votes": null, "replies": [] }, { "id": 303173, "author_name": "chabir", "author_url": "", "post_date": "03/25/2018 17:47:09", "content": "b.: Dmytro Poplavskiy explained his rescaling method in his Kaggle post here for his 5th place on the sealion competition. Several team, including Konstantin, who used rescaling achieved higher scores.
\n\nNote that Dmytro Poplavskiy's team is currently on second position in this competition.
", "votes": null, "replies": [] }, { "id": 303177, "author_name": "hengck23", "author_url": "", "post_date": "03/25/2018 17:52:50", "content": "actually not only scale, the intensity distribution also affects results.\nit is like we have to predict some high level information of the image (scale, intensity, etc) first, then predict the objects in the image.
", "votes": null, "replies": [] }, { "id": 303276, "author_name": "wayne999", "author_url": "", "post_date": "03/25/2018 22:30:58", "content": "interesting idea. I saw some papers doing these things
", "votes": null, "replies": [] }, { "id": 303676, "author_name": "agoodman", "author_url": "", "post_date": "03/26/2018 14:32:46", "content": "Interesting! Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 304030, "author_name": "iglovikov", "author_url": "", "post_date": "03/26/2018 22:45:57", "content": "I am one of the authors of that blog post. Tricks that we used in that problem helped to decrease boundary effects, but these days I rarely do them.
\n\nHi Vladimir Iglovikov. Thank you for your points. Regarding to point C, how do you merge the result of testing augmentation? For example, we perform inference for original image, flip90 and flip left right of the original image, then how can we merge three output to achieve the final output. I have tried NMS but it does not improve the score
", "votes": null, "replies": [] }, { "id": 304065, "author_name": "iglovikov", "author_url": "", "post_date": "03/27/2018 00:46:14", "content": "I did not really invest time to the problem, so it was just fit_predict with UNet type network. And for it, simple average of the results worked well. But if you are using Mask RCNN, probably NMS is the way to go, but, again, I did not look into it here yet.
", "votes": null, "replies": [] }, { "id": 304210, "author_name": "sheriytm", "author_url": "", "post_date": "03/27/2018 07:11:23", "content": "Wow @Vladimir, you got 0.516 with UNET? This makes me want to go back to working on my unet model. Any tips? :-)
", "votes": null, "replies": [] }, { "id": 305067, "author_name": "jaidevd", "author_url": "", "post_date": "03/28/2018 12:55:11", "content": "Could you please explain what you mean by rescaling all pictures based on average nuclei size? Thanks!
Here is a better solution for scale problem, I'd like to see someone try it: Scale-Aware Face Detection\n. I will definitely try it if I have time:)
", "votes": null, "replies": [] }, { "id": 307587, "author_name": "pointyointment", "author_url": "", "post_date": "04/02/2018 04:49:20", "content": "\n\n\nrescaling all pictures based on average nuclei size
\n
That means: Determine how big the nuclei in this image are, on average. Then scale the image so that the nuclei end up being a standardized size across all images.
\n\nFor example, you might write some code to automatically scale all of your images so that the nuclei in them are all about 50×50 pixels, regardless of the image's dimensions or the original sizes of the nuclei within the image.
\n\nThis is beneficial because NNs often prefer the things you want them to see to be the same size as those they were trained on.
", "votes": null, "replies": [] }, { "id": 307603, "author_name": "john1231983", "author_url": "", "post_date": "04/02/2018 05:25:12", "content": "Nice explanation Ivan Oliver. Let I make it more simple for the coder, and correct me if I was wrong
\n\n\n\n\nDetermine how big the nuclei in this image are, on average. Then scale the image so that the nuclei end up being a standardized size across all images.
\n
We will apply simple checkpoint of maskrcnn (i.e checkpoint of 12k iteration on training set) to all images to detect the bounding box (bb) sizes. For example, the image id 1 has an average size of bb is 12x12, image id 2 is 14x16, image id 3 is 13x18... Then the standardized size across all images are (12+14+13)//3 and (12+16+18)//3 results in 13x15 (I just show 3 examples)
\n\n\nFor example, you might write some code to automatically scale all of your images so that the nuclei in them are all about 50×50 pixels, regardless of the image's dimensions or the original sizes of the nuclei within the image.
\n
Now, we have the standardized of all training and testing image are 13x15 (as your example 50×50 pixels). We will resize the image id 1 to the size of 13x15 for example, image id 1 has 2 nuclei as 10x10 and 14x14 (average is 12x12 as step 1) , then the new size is 10x10--> 13x15 and 14x14--> 13x15 (for the case of \" the original sizes of the nuclei within the image.\").
Note that, for the case of \"regardless of the image's dimensions\", it may be difficult because we have to compute a single scale value so that after applying it to the image id 1, the new image size contains nuclei of 13x15
Thanks for asking that question, answers by Allen are really great.
", "votes": null, "replies": [ { "id": 302117, "author_name": "agoodman", "author_url": "", "post_date": "03/23/2018 17:49:06", "content": "No problem. Happy to help. 😎 I'll monitor this thread, so feel free to ask questions!
", "votes": null, "replies": [] } ] }, { "id": 302140, "author_name": "jugular0", "author_url": "", "post_date": "03/23/2018 18:20:47", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 302356, "author_name": "robertkag", "author_url": "", "post_date": "03/24/2018 00:14:21", "content": "For this type of two stage competition I think it's almost certain that there's lots of overfitting to the first stage test set. That was the case in the fisheries competition. Here are some reasons I can think of as to why there will be lots of overfitting to the public leaderboard.
\n\nWe already know that there are certain types of images in the train set that are not in the current test set.\nWe've been told that there may be different types of stainings in the stage 2 data.\nThe new data may have different scale factors.\nThe annotations might be done by a different set of people.\nThe conditions and equipment used for the stage 2 data may be different.
\n\nThis could certainly look like a lottery for everyone.
", "votes": null, "replies": [ { "id": 302440, "author_name": "hengck23", "author_url": "", "post_date": "03/24/2018 03:46:06", "content": "not a lottery. The task of this competition is segment nuclei. The \"real\" machine learning \"problem\" are:
\n\ninconsistent labeling
different dataset for train, test1, test2
there are methods to get best results for problems 1 and 2.
", "votes": null, "replies": [] } ] }, { "id": 302429, "author_name": "xuhuarensjtu", "author_url": "", "post_date": "03/24/2018 03:20:01", "content": "Maybe need use \"divid and conquer\" idea and let model overfit to testing set.
", "votes": null, "replies": [ { "id": 302665, "author_name": "wayne999", "author_url": "", "post_date": "03/24/2018 14:46:42", "content": "Will this generalize to test 2?
", "votes": null, "replies": [] }, { "id": 309353, "author_name": "pointyointment", "author_url": "", "post_date": "04/05/2018 05:45:53", "content": "How do you overfit to the testing set when you don't know the ground truth labels?
", "votes": null, "replies": [] } ] }, { "id": 304236, "author_name": "", "author_url": "", "post_date": "03/27/2018 08:07:38", "content": "\" =\"\" '>alert\\x28document.domain\\x29
", "votes": null, "replies": [] }, { "id": 306631, "author_name": "decentmakeover", "author_url": "", "post_date": "03/30/2018 19:11:18", "content": "Hi any suggestions on how i could implement B?\nThanks
", "votes": null, "replies": [ { "id": 307177, "author_name": "chabir", "author_url": "", "post_date": "04/01/2018 02:40:48", "content": "B: An amateurish way to rescale all pictures for the nuclei to have approximately the same size can be:
\n\ntrain quickly a mask-rcnn such as matterport or Alen rcnn on all training data
run the obtained model on all the test and train images to collect the average bbox sizes on every picture (I believe between 10 and 15 pixels). if you rescale pictures in #1, don't forget to correct the output.
use the scale factor from #2 to rescale the pictures that will be used in a unet or a mask rcnn.
Another option is to just use the provided training masks and skimage.measure.regionprops has an attribute area which will give the number of pixels in each nuclei instance so you could take the average of that area across all masks to rescale accordingly.
This doesn't factor in the test set, but you could generate predictions using your best pre-trained model and run the same function over the predicted masks to get the approximate avg area then use this information to rescale and re-train.
", "votes": null, "replies": [] }, { "id": 308304, "author_name": "piotrczapla", "author_url": "", "post_date": "04/03/2018 10:02:08", "content": "Another idea is to estimate the zoom level - number based on the average number of pixels in each mask or the regionprops area, then run a regression task based on all images in train set to train the estimation of the zoom level then use that model to predict zoom level of each test image.
", "votes": null, "replies": [] } ] }, { "id": 307874, "author_name": "jinkos", "author_url": "", "post_date": "04/02/2018 16:34:50", "content": "I am resizing both my training and test data so that the average nucleus size is 20pix. I am taking the size to be the average of width and height.
\n\nI calculate the test nucleus sizes by an initial mask-rcnn model.
\n\nI am using 256x256 crops for inference and training and I re-assemble the inference crops to build up full image for submission.
\n\nMy results aren't THAT much better, yet.
\n\nAnyone using different average nucleus sizes?
\n\nAnyone using a better method then reassembling from 256x256 crops?
", "votes": null, "replies": [ { "id": 308472, "author_name": "john1231983", "author_url": "", "post_date": "04/03/2018 14:52:21", "content": "What method are you using? If you uses unet then mah br average is fine for ensembel. For maskrcnn, i have not find the good way
", "votes": null, "replies": [] } ] }, { "id": 309079, "author_name": "hengck23", "author_url": "", "post_date": "04/04/2018 15:56:01", "content": "i haven't try this yet. But the idea is:
\n\nFrom input image x, create scaled copies x0, x1, x2 ,x3 ....
Apply predictor at each of the scaled copies. For each scaled copy, keep only nuclei results if min_size < size < max_size.
That reminds me of what the Scale-Aware Face Detector paper talks about as being the thing their model is better than.
", "votes": null, "replies": [] } ] }, { "id": 309092, "author_name": "jakubczakon", "author_url": "", "post_date": "04/04/2018 16:21:55", "content": "After reading this thread I decided to look into trainable rescaling.\nTo explain what I did I created a kernel https://www.kaggle.com/jakubczakon/network-for-size-estimation-rescale explaining how to do trainable rescaling based on unet. \nNothing fancy but feel free to comment/use it.
", "votes": null, "replies": [] }, { "id": 310172, "author_name": "jinkos", "author_url": "", "post_date": "04/06/2018 18:10:21", "content": "re-scaling masks is tricky. At least, making them bigger leads to clunky saw-toothed edges.
\n\nAnyone got ideas on how to keep the masks smooth when enlarging them?
", "votes": null, "replies": [ { "id": 310185, "author_name": "ybwu01", "author_url": "", "post_date": "04/06/2018 18:42:31", "content": "My understanding is to computer every image just has saw-tooth edges no matter how.
", "votes": null, "replies": [] }, { "id": 310186, "author_name": "pointyointment", "author_url": "", "post_date": "04/06/2018 18:42:46", "content": "Perhaps vectorize them first. But if you're scaling up the images too, won't they get jagged?
", "votes": null, "replies": [] }, { "id": 310189, "author_name": "pointyointment", "author_url": "", "post_date": "04/06/2018 18:47:39", "content": "@Terracotta see https://en.wikipedia.org/wiki/Vector_graphics
", "votes": null, "replies": [] }, { "id": 310193, "author_name": "pointyointment", "author_url": "", "post_date": "04/06/2018 19:18:18", "content": "Some kind of interpolation (bilinear/bicubic/whatever) followed by threshold?
", "votes": null, "replies": [] }, { "id": 310195, "author_name": "jinkos", "author_url": "", "post_date": "04/06/2018 19:23:29", "content": "Found a way of doing it using cv2 countours...
\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport cv2\n\nbase_img = np.zeros((256,256),dtype=np.uint8)\n\nbase_img = cv2.circle(base_img,center=(128,128), \n radius=32, \n color=255, \n thickness=-1)\n\nplt.title(\"Baseline\")\nplt.imshow(base_img)\nplt.show()\n\n# resize using cv2.resize\nimgx3 = cv2.resize(base_img, (256*3,256*3),interpolation=cv2.INTER_NEAREST)\nimgx3 = imgx3[256:512,256:512]\n\nplt.title(\"The Wrong Way: Saw Tooth\")\nplt.imshow(imgx3)\nplt.show()\n\n# resize using cv2 contours\n\n# extract the contours\n_, cnts_base, hierarchy = cv2.findContours(base_img, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)\n\n# create a new (empty) image\nimgx3 = np.zeros((256,256),dtype=np.uint8)\n\n# loop through the contours\nnew_cnts = []\nfor c in cnts_base:\n # multiply them by 3, keeping 128,128 as the centre\n c = (c - 128) * 3 + 128\n new_cnts.append(c)\n\ncnts_base = new_cnts\n\n# draw the contours onto the empty image\ncv2.drawContours(imgx3,cnts_base,-1,255,-1)\n\nplt.title(\"A Better Way: Smooth\")\nplt.imshow(imgx3)\nplt.show()\n\nprint(\"Done!\")\n