![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/273049/8353/b1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe.png\")
The hosts had a nice idea to create a consolidated thread where everyone can share minor data quality issues and other miscellaneous \"gotchas\" that you hit when getting started on this problem. Feel free to sound off below if you find something and want to share it with the group. We suggest you show each potentially problematic image as a separate post unless they are highly related.
", "messages": [ { "id": 270642, "postDate": "2018-01-18T16:55:04.820Z", "content": "The hosts had a nice idea to create a consolidated thread where everyone can share minor data quality issues and other miscellaneous \"gotchas\" that you hit when getting started on this problem. Feel free to sound off below if you find something and want to share it with the group. We suggest you show each potentially problematic image as a separate post unless they are highly related.
", "rawMarkdown": "The hosts had a nice idea to create a consolidated thread where everyone can share minor data quality issues and other miscellaneous \"gotchas\" that you hit when getting started on this problem. Feel free to sound off below if you find something and want to share it with the group. We suggest you show each potentially problematic image as a separate post unless they are highly related.", "votes": 46 }, { "id": 273049, "postDate": "2018-01-24T03:45:27.140Z", "content": "Several possible miss-annotations:
\n\nb1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe:
\n\nmissing some of the nucleis;\n
19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a:
\n\nMost neighbor nucleis were annotated as one. Is it on purpose?\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/273049/8354/19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a.png\")
9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e:
\n\nA strange bar-like object at the lower right corner\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/273049/8355/9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e.png\")
Question about the rule to separate neighbor nucleis:
\n\ne.g. 1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca:
\n\nThe yellow and green nuclei in the middle were marked as 2 different nucleis. But note that to their left (yellow) and to their right (blue) adjacent nucleis were marked as one nuclei. I do not understand why, and I have tried adjusting display window but nothing showed up. \n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/273049/8356/1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca.png\")
1) I agree b1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe is missing two nuclei off to the right. I'm also disturbed that there are tiny black dots, indicating holes in the nuclei which should not be there. (I cannot swear to it, but in general nuclei as we have annotated them should NOT have any holes)\n2) For 19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a I guess it depends how you generated the image you are showing on the right. You are right that in general they ought to be multi-colored whereas your result indicates the masks are improperly merged. Just not sure whether that is happening in the annotations or in your code that generates the multicolor visualization (I assume the former of course, in which case it's definitely an error). \n3) For 9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e indeed, that object is an error. I caught many of these before the training set went out but apparently not all!\n4) For 1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca: Those pairs of nuclei should indeed be split. Those are errors.
\n\nThanks for pointing these out!
", "rawMarkdown": "1) I agree b1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe is missing two nuclei off to the right. I'm also disturbed that there are tiny black dots, indicating holes in the nuclei which should not be there. (I cannot swear to it, but in general nuclei as we have annotated them should NOT have any holes)\n2) For 19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a I guess it depends how you generated the image you are showing on the right. You are right that in general they ought to be multi-colored whereas your result indicates the masks are improperly merged. Just not sure whether that is happening in the annotations or in your code that generates the multicolor visualization (I assume the former of course, in which case it's definitely an error). \n3) For 9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e indeed, that object is an error. I caught many of these before the training set went out but apparently not all!\n4) For 1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca: Those pairs of nuclei should indeed be split. Those are errors.\n\nThanks for pointing these out! ", "votes": 5 }, { "id": 273384, "postDate": "2018-01-24T14:51:41.743Z", "content": "Hi Anne,
\n\nThanks for the detailed reply.
\n\nBlack dots in image 1: There is indeed single pixel holes in the original masks.
\n\nSecond image: we have checked the original images and confirmed that the original masks are like these.
\n\nThe last issue: this is a little problematic, because I found multiple cases with something like this (especially for 360*360 resolution images) I will try to find all cases and post them here.
", "rawMarkdown": "Hi Anne,\n\nThanks for the detailed reply. \n\n**Black dots in image 1**: There is indeed single pixel holes in the original masks. \n\n**Second image**: we have checked the original images and confirmed that the original masks are like these. \n\n**The last issue**: this is a little problematic, because I found multiple cases with something like this (especially for 360*360 resolution images) I will try to find all cases and post them here. ", "votes": 1 }, { "id": 273541, "postDate": "2018-01-24T18:35:01.957Z", "content": "Thanks for confirming 1 & 2! Definitely happy to hear about errors.
", "rawMarkdown": "Thanks for confirming 1 & 2! Definitely happy to hear about errors. ", "votes": 1 }, { "id": 277673, "postDate": "2018-02-03T22:14:21.163Z", "content": "The ID of the mask with the vertical bar is a09ae9559f817b104e049e5bb01cf4ad1b87e3ef7fad3dcf932b6585ca3edc05 if anyone wants to exclude just that mask.
", "rawMarkdown": "The ID of the mask with the vertical bar is a09ae9559f817b104e049e5bb01cf4ad1b87e3ef7fad3dcf932b6585ca3edc05 if anyone wants to exclude just that mask.", "votes": 4 }, { "id": 278050, "postDate": "2018-02-05T08:12:35.853Z", "content": "Hi Anne,
\n\nI found 436 images where the masks have holes. One example is below. \nstage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png
\n\nThere are also 11 other masks where there is a gap between the annotated masks making them appear as two parts. One example is\nstage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png
\n\nAre these valid masks or do these need to be fixed?
", "rawMarkdown": "Hi Anne,\n\nI found 436 images where the masks have holes. One example is below. \nstage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png\n\nThere are also 11 other masks where there is a gap between the annotated masks making them appear as two parts. One example is\nstage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png\n\nAre these valid masks or do these need to be fixed?", "votes": 1 }, { "id": 278331, "postDate": "2018-02-05T23:24:02.643Z", "content": "Hi there, Jerry. Thanks for noting these. Allen here had it on his list to check for tiny holes resulting from our annotation tool but hadn't gotten to it yet!
\n\nI would guess that those indeed need to be fixed to fill in the holes (as long as a hole is entirely surrounded by a single nucleus' label). Is the size is generally a single pixel? Terribly sorry this is an issue! Luckily it's an easy fix to correct the training data (and we are checking the stage 2 test set to ensure it won't be a problem there).
\n\nIt would be great if you can post the actual images in the post so I can provide guidance on the second issue; I see there is a single white mask in that image but without the corresponding image side by side I'm not sure what to make of it.
", "rawMarkdown": "Hi there, Jerry. Thanks for noting these. Allen here had it on his list to check for tiny holes resulting from our annotation tool but hadn't gotten to it yet!\n\nI would guess that those indeed need to be fixed to fill in the holes (as long as a hole is entirely surrounded by a single nucleus' label). Is the size is generally a single pixel? Terribly sorry this is an issue! Luckily it's an easy fix to correct the training data (and we are checking the stage 2 test set to ensure it won't be a problem there).\n\nIt would be great if you can post the actual images in the post so I can provide guidance on the second issue; I see there is a single white mask in that image but without the corresponding image side by side I'm not sure what to make of it.", "votes": 1 }, { "id": 278642, "postDate": "2018-02-06T16:46:58.107Z", "content": "I accidentally replied twice and attached the images in separate reply. But I had not included the original image. Here are the masks and the corresponding images. I have cropped both image and mask to the mask area.
\n\nMask with holes and corresponding image.\n![\"Mask](\"https://www.kaggle.io/svf/2357892/9ec4f3ca4132c19296b2c63ed7f924df/__results___files/__results___4_0.png\")
Mask with a single pixel noise to the side. This looks more like an incorrect annotation, although I am not sure. This is on the bottom edge of the image\n![\"mask](\"https://www.kaggle.io/svf/2357892/9ec4f3ca4132c19296b2c63ed7f924df/__results___files/__results___6_0.png\")
Mask with gap/Two nuclei annotated as one? This is on the bottom edge of the image\n![\"two](\"https://www.kaggle.io/svf/2359003/4a7901ca80e0619a44a2995a431e4e3b/__results___files/__results___8_0.png\")
Thanks! For the first image, yes, those speckly holes ought to be filled in. We noticed too late the annotation tool used on some of the images didn't fill in completely (ugh!)
\n\nFor the second image \"Mask with a single pixel noise to the side.\" - indeed that looks like an error! The left-most masked nucleus should not be connected to the remaining pixels (which should have been background). This is a common artifact of using our tools, which we mostly caught and fixed manually, but not this one!
\n\nFor the third image those are definitely two separate nuclei... your question is why are two separate nuclei's masks present in a single slice of the annotations, right? Yeah, that shouldn't happen - that's another check we can add, to be sure that all masks are contiguous (it's hard for me to imagine how this happened with either of our annotation approaches, TBH!).
", "rawMarkdown": "Thanks! For the first image, yes, those speckly holes ought to be filled in. We noticed too late the annotation tool used on some of the images didn't fill in completely (ugh!)\n\nFor the second image \"Mask with a single pixel noise to the side.\" - indeed that looks like an error! The left-most masked nucleus should not be connected to the remaining pixels (which should have been background). This is a common artifact of using our tools, which we mostly caught and fixed manually, but not this one!\n\nFor the third image those are definitely two separate nuclei... your question is why are two separate nuclei's masks present in a single slice of the annotations, right? Yeah, that shouldn't happen - that's another check we can add, to be sure that all masks are contiguous (it's hard for me to imagine how this happened with either of our annotation approaches, TBH!).", "votes": 1 } ] }, { "id": 271184, "postDate": "2018-01-19T21:08:37.690Z", "content": "stage1_train/58c593bcb98386e7fd42a1d34e291db93477624b164e83ab2afa3caa90d1d921
\n\n![\"img\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/271184/8317/58c593.png\")
Looks like it misses one mask on the right bottom corner. Please confirm.
", "rawMarkdown": "stage1_train/58c593bcb98386e7fd42a1d34e291db93477624b164e83ab2afa3caa90d1d921\n\n![img][1]\n\nLooks like it misses one mask on the right bottom corner. Please confirm.\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/271184/8317/58c593.png", "votes": 5, "replies": [ { "id": 271192, "postDate": "2018-01-19T21:31:35.493Z", "content": "Hello, an organizer here. I agree that nucleus ought to have been identified. Great catch!
", "rawMarkdown": "Hello, an organizer here. I agree that nucleus ought to have been identified. Great catch!", "votes": 6 }, { "id": 289116, "postDate": "2018-02-28T23:59:33.113Z", "content": "I just downloaded the training data and that nucleus is still unannotated. I take it the official training dataset isn't being updated with these corrections, and we're expected to apply them ourselves? (I'm new around here but I think I've read that that's how it works.) [Edited to add: Indeed, you've confirmed this below and apparently it's a Kaggle rule as I thought.]
\n\nA further error with the same image is that the two nuclei on the left seem to be annotated as one. At least, they're in the same mask file, which I think is supposed to be one per nucleus.
", "rawMarkdown": "I just downloaded the training data and that nucleus is still unannotated. I take it the official training dataset isn't being updated with these corrections, and we're expected to apply them ourselves? (I'm new around here but I think I've read that that's how it works.) [Edited to add: Indeed, you've confirmed this below and apparently it's a Kaggle rule as I thought.]\n\nA further error with the same image is that the two nuclei on the left seem to be annotated as one. At least, they're in the same mask file, which I think is supposed to be one per nucleus." }, { "id": 289875, "postDate": "2018-03-02T14:22:28.793Z", "content": "re 2 nuclei annotated as one - Sort of depends how the person made this visualization. They are using the same color (yellow) for all nuclei so we've no idea whether the underlying annotations are distinct nuclei there or a single one.
", "rawMarkdown": "re 2 nuclei annotated as one - Sort of depends how the person made this visualization. They are using the same color (yellow) for all nuclei so we've no idea whether the underlying annotations are distinct nuclei there or a single one." } ] }, { "id": 283933, "postDate": "2018-02-16T08:12:48.577Z", "content": "This list of 55 masks look like they should be separated
", "rawMarkdown": "This list of 55 masks look like they should be separated", "votes": 3 }, { "id": 271186, "postDate": "2018-01-19T21:15:25.977Z", "content": "Found another:
\n\nstage1_train/b1e3aeb0c56261c17eb71c747d116057b8da7e8c8a6845bdc01b2b3ee2299229
\n\n![\"img\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/271186/8318/b1e3ae.png\")
Some of them I could barely see when zooming in, but most are just black. Please confirm.
", "rawMarkdown": "Found another:\n\nstage1_train/b1e3aeb0c56261c17eb71c747d116057b8da7e8c8a6845bdc01b2b3ee2299229\n\n![img][1]\n\nSome of them I could barely see when zooming in, but most are just black. Please confirm.\n\n\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/271186/8318/b1e3ae.png", "votes": 3, "replies": [ { "id": 271191, "postDate": "2018-01-19T21:30:35.790Z", "content": "Hello, an organizer here. There are often very dim nuclei and this is such a case. Adjust the brightness on your monitor, or adjust the contrast of the images and you will see the nuclei that correspond to those masks.
\n\nStill, it should be noted that the annotators each had their own monitor and style so how dim something would be in order to be ignored will vary from image to image.
", "rawMarkdown": "Hello, an organizer here. There are often *very* dim nuclei and this is such a case. Adjust the brightness on your monitor, or adjust the contrast of the images and you will see the nuclei that correspond to those masks. \n\nStill, it should be noted that the annotators each had their own monitor and style so how dim something would be in order to be ignored will vary from image to image. ", "votes": 2 }, { "id": 271198, "postDate": "2018-01-19T21:42:37.497Z", "content": "Thanks, it clears my question. :)
", "rawMarkdown": "Thanks, it clears my question. :)" }, { "id": 303084, "postDate": "2018-03-25T13:17:53.900Z", "content": "The really sad part is that these cases are only annotated sometimes. At many times, I'm guessing most, they weren't seen by the annotators. Fingers crossed that you guys have located all of these in the evaluation cases.
", "rawMarkdown": "The really sad part is that these cases are only annotated sometimes. At many times, I'm guessing most, they weren't seen by the annotators. Fingers crossed that you guys have located all of these in the evaluation cases.", "votes": -1 } ] }, { "id": 280340, "postDate": "2018-02-09T18:31:57.290Z", "content": "I think that there are a couple of issues with image\n1bd0f2b3000b7c7723f25335fabfcdddcdf4595dd7de1b142d52bb7a186885f0.png
\n\n![\"image\"](\"http://th.if.uj.edu.pl/~romuald/1bd0f.png\")
\nFirst look at the top - it seems that the first two nuclei should be separated, while the next one on the top edge which is marked as two should be indeed one.
Then the two yellow ones in the center should be separate but are marked as one object. \nIs it correct?
\n\nBtw. suppose that in principle we correct some labelling - what should then be the procedure for submitting code before stage 2?
", "rawMarkdown": "I think that there are a couple of issues with image\n1bd0f2b3000b7c7723f25335fabfcdddcdf4595dd7de1b142d52bb7a186885f0.png\n\n![image][1]\nFirst look at the top - it seems that the first two nuclei should be separated, while the next one on the top edge which is marked as two should be indeed one.\n\nThen the two yellow ones in the center should be separate but are marked as one object. \nIs it correct?\n\nBtw. suppose that in principle we correct some labelling - what should then be the procedure for submitting code before stage 2?\n\n [1]: http://th.if.uj.edu.pl/~romuald/1bd0f.png", "votes": 4, "replies": [ { "id": 280401, "postDate": "2018-02-09T22:26:36.790Z", "content": "I agree with your assessment of this one (although maybe the top edge that is two could be one - hard to see without zooming in/adjusting contrast on my screen which I can't do at the moment!)
\n\nThat is a great question about the procedure for submitting code since indeed you can't submit a fully reproducible pipeline without also submitting annotation fixes along with it. I will see if one of the other organizers know how this is handled.
\n\nIn the meantime, thanks for pointing these out!
", "rawMarkdown": "I agree with your assessment of this one (although maybe the top edge that is two *could* be one - hard to see without zooming in/adjusting contrast on my screen which I can't do at the moment!)\n\nThat is a great question about the procedure for submitting code since indeed you can't submit a fully reproducible pipeline without also submitting annotation fixes along with it. I will see if one of the other organizers know how this is handled. \n\nIn the meantime, thanks for pointing these out!", "votes": 3 }, { "id": 282324, "postDate": "2018-02-13T21:24:50.530Z", "content": "The code submission should include some documentation that you have curated the annotation. Either include the manual curated data or the code that generate those fixes are both acceptable. As long as the code is reproducible for the final test set, you're fine.
", "rawMarkdown": "The code submission should include some documentation that you have curated the annotation. Either include the manual curated data or the code that generate those fixes are both acceptable. As long as the code is reproducible for the final test set, you're fine. \n", "votes": 1 }, { "id": 303219, "postDate": "2018-03-25T19:15:29.003Z", "content": "I also wonder abount the tiny spec at the centre d99c71ab55a7d103eeef63f60cf946d6318e533c893ae107919160fc08845599 - it doesn't seem any better than the one below blue at mid-bottom, but it's marked, while the other one isn't. And this, of course, is not the only case, there are tons of similar cases all along the dataset. If organizers were to state their policy on these tiny specs that is going to be enforced in the final evaluation dataset, that would be most helpful. That matters a lot, because there are tons of these specs all over images, so it's a potential question of mislabeling a ton of objects.
\n\nI would advise against them, because with such a small amount of pixels getting 50% IOU is freaking impossible - for example even in this case if the annotators got the mere fact of labeling that spec as a hit, the question whether their own mask's IOU over actual spec is at least 50% is up to debate and I'd say it's no more than 70%. It's not a problem that can be solved with precision, unless you define what that precision has to be.
", "rawMarkdown": "I also wonder abount the tiny spec at the centre d99c71ab55a7d103eeef63f60cf946d6318e533c893ae107919160fc08845599 - it doesn't seem any better than the one below blue at mid-bottom, but it's marked, while the other one isn't. And this, of course, is not the only case, there are tons of similar cases all along the dataset. If organizers were to state their policy on these tiny specs that is going to be enforced in the final evaluation dataset, that would be most helpful. That matters a lot, because there are tons of these specs all over images, so it's a potential question of mislabeling a ton of objects. \n\nI would advise against them, because with such a small amount of pixels getting 50% IOU is freaking impossible - for example even in this case if the annotators got the mere fact of labeling that spec as a hit, the question whether their own mask's IOU over actual spec is at least 50% is up to debate and I'd say it's no more than 70%. It's not a problem that can be solved with precision, unless you define what that precision has to be.", "votes": 1 } ] }, { "id": 276364, "postDate": "2018-01-31T10:34:40.520Z", "content": "This image should either be annotated or excluded from training set\n7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80 \n![\"Image\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/276364/8448/7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80.png\")
\n![\"Annotation\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/276364/8449/5ffe1e4987f987f45f716bbeb749f378bb76ab235be18b5fc4d7468523a8e005.png\")
Indeed!
", "rawMarkdown": "Indeed!", "votes": 2 } ] }, { "id": 312867, "postDate": "2018-04-12T13:36:19.423Z", "content": "Just out of curiosity: what is the reason for images like 0aa80, 0f48f, 0a9b6, 0f4b0, 0b2f4 and some more?
", "rawMarkdown": "Just out of curiosity: what is the reason for images like 0aa80, 0f48f, 0a9b6, 0f4b0, 0b2f4 and some more?", "votes": 1, "replies": [ { "id": 314947, "postDate": "2018-04-16T14:38:19.693Z", "content": "If it's a film, I want to see it...
", "rawMarkdown": "If it's a film, I want to see it...", "votes": 1 } ] }, { "id": 312802, "postDate": "2018-04-12T11:08:52.027Z", "content": "As I know I'm definitely not gonna win now, I make my revision of the training set and my annotation tool available at https://github.com/ibmua/data-science-bowl-2018-train-set . I think the training set is probably the best there is, as I went through each white-on-black sample and fixed it by hand. Annotation tool is also pretty useful, though, probably limitedly so now after the first stage is over. Maybe, for some future competitions, I donno.
\n\nP.S. uploaded this pic just because I can't upload it in a comment. Not related.
", "rawMarkdown": "As I know I'm definitely not gonna win now, I make my revision of the training set and my annotation tool available at https://github.com/ibmua/data-science-bowl-2018-train-set . I think the training set is probably the best there is, as I went through each white-on-black sample and fixed it by hand. Annotation tool is also pretty useful, though, probably limitedly so now after the first stage is over. Maybe, for some future competitions, I donno.\n\nP.S. uploaded this pic just because I can't upload it in a comment. Not related.", "votes": 1 }, { "id": 278052, "postDate": "2018-02-05T08:21:54.317Z", "content": "Hi Anne,
\n\nI found a few masks which had holes in them. One example is below. I am not sure if all of them follow the same pattern. \n../input/stage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png
\n\nI also found 11 others where there is a break in the mask making them appear as two items.\n../input/stage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png
\n\nAre these valid masks? You have mentioned earlier that masks are not expected to have holes in them. Is the same applicable to these. I will double check on the list of masks I found with holes.
", "rawMarkdown": "Hi Anne,\n\nI found a few masks which had holes in them. One example is below. I am not sure if all of them follow the same pattern. \n../input/stage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png\n\nI also found 11 others where there is a break in the mask making them appear as two items.\n../input/stage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png\n\nAre these valid masks? You have mentioned earlier that masks are not expected to have holes in them. Is the same applicable to these. I will double check on the list of masks I found with holes.", "votes": 1, "replies": [ { "id": 279198, "postDate": "2018-02-07T15:19:34.840Z", "content": "These Qs are addressed in another posting above, but this is nice because it gives the image numbers.
", "rawMarkdown": "These Qs are addressed in another posting above, but this is nice because it gives the image numbers. " } ] }, { "id": 276985, "postDate": "2018-02-01T23:54:43.093Z", "content": "Reading the \"Evaluation Rules\"
\n\n\n\n\n.... It also checks that no two predicted masks for the same image are overlapping.
\n
however there are enough images in the training set where nucleus seems to be overlapping (for example in the 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80). Wouldn't be more meaningful (natural) in such cases to strive for the real necleus masks (i.e. even if they overlap)?
\n\nAnd a second question: assuming the test set would have similar segmentation errors like the training set, if we imagine a model able of doing a \"perfect\" segmentation, it will probably get worse score than a model making the same segmentation errors as in the test set used for the final evaluation.
\n\nI'm wondering are there any chances that this might happen (here), i.e. penalizing models doing better segmentation than the one in the test set?
", "rawMarkdown": "Reading the \"Evaluation Rules\" \n\n> .... It also checks that no two predicted masks for the same image are overlapping.\n\nhowever there are enough images in the training set where nucleus seems to be overlapping (for example in the 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80). Wouldn't be more meaningful (natural) in such cases to strive for the real necleus masks (i.e. even if they overlap)?\n\nAnd a second question: assuming the test set would have similar segmentation errors like the training set, if we imagine a model able of doing a \"perfect\" segmentation, it will probably get worse score than a model making the same segmentation errors as in the test set used for the final evaluation. \n\nI'm wondering are there any chances that this might happen (here), i.e. penalizing models doing better segmentation than the one in the test set? ", "votes": 1, "replies": [ { "id": 277278, "postDate": "2018-02-02T17:29:37.937Z", "content": "We certainly debated overlapping nuclei a lot. In the end we decided it would make annotation substantially harder to the point where we wouldn't be able to do the competition. We would need new tools that allow for overlap. Thankfully, for most real world applications the percentage of affected nuclei is low... but this is definitely an area for improvement.
\n\nI don't really understand the questions about whether to attempt a model that produces errors. For example, a few training images had a random line in them that obviously isn't nucleus. Whether the test set has errors or not (NB: we checked the test set more carefully than the training set), I don't really see how you'd design a model that tried to reproduce errors: like, what, the model would place random lines in random places on the image? That is certainly not going to raise your score. That's an obvious case but even for the other types of errors, it's hard for me to see how it could possibly help to do anything other than ignore or fix the incorrectly annotated training images.
", "rawMarkdown": "We certainly debated overlapping nuclei a lot. In the end we decided it would make annotation substantially harder to the point where we wouldn't be able to do the competition. We would need new tools that allow for overlap. Thankfully, for most real world applications the percentage of affected nuclei is low... but this is definitely an area for improvement. \n\nI don't really understand the questions about whether to attempt a model that produces errors. For example, a few training images had a random line in them that obviously isn't nucleus. Whether the test set has errors or not (NB: we checked the test set more carefully than the training set), I don't really see how you'd design a model that tried to reproduce errors: like, what, the model would place random lines in random places on the image? That is certainly not going to raise your score. That's an obvious case but even for the other types of errors, it's hard for me to see how it could possibly help to do anything other than ignore or fix the incorrectly annotated training images.", "votes": 1 }, { "id": 277337, "postDate": "2018-02-02T20:50:00.850Z", "content": "My labmate pointed out that maybe your second question was directly related to the first.
\n\nAFAIK, nuclei are not allowed to overlap when submitted as results.
\n\nPlus, the annotations used for in evaluation for both test stages (1 & 2) do not have overlap, again AFAIK. So it will be in your best interest to not produce a model that yields overlap.
", "rawMarkdown": "My labmate pointed out that maybe your second question was directly related to the first. \n\nAFAIK, nuclei are not allowed to overlap when submitted as results.\n\nPlus, the annotations used for in evaluation for both test stages (1 & 2) do not have overlap, again AFAIK. So it will be in your best interest to not produce a model that yields overlap.", "votes": 2 } ] }, { "id": 275305, "postDate": "2018-01-28T18:29:59.297Z", "content": "I'm a bit confused here. Does this mean its our responsibility to monitor this thread and adjust out images set or will updated versions of the data be released?
", "rawMarkdown": "I'm a bit confused here. Does this mean its our responsibility to monitor this thread and adjust out images set or will updated versions of the data be released?", "votes": 1, "replies": [ { "id": 275311, "postDate": "2018-01-28T19:18:49.353Z", "content": "According to our Kaggler people-in-charge, updated versions will not be released. I agree it's a hassle but it's outside my sphere of knowledge. The number of instances with actual errors is quite low (maybe 4 or 5 listed here so far, out of hundreds of images).
", "rawMarkdown": "According to our Kaggler people-in-charge, updated versions will not be released. I agree it's a hassle but it's outside my sphere of knowledge. The number of instances with actual errors is quite low (maybe 4 or 5 listed here so far, out of hundreds of images).", "votes": 1 }, { "id": 275708, "postDate": "2018-01-29T20:38:18.310Z", "content": "Ok, Thanks!
", "rawMarkdown": "Ok, Thanks!" } ] }, { "id": 312633, "postDate": "2018-04-12T04:21:32.550Z", "content": "Not a big deal, but in the stage2 test set this image:
\n\n646f5e00a2db3add97fb80a83ef3c07edd1b17b1b0d47c2bd650cdcab9f322c0
\n\nis a 2d grayscale. All other grayscale images have a channel dimension.
", "rawMarkdown": "Not a big deal, but in the stage2 test set this image:\n\n646f5e00a2db3add97fb80a83ef3c07edd1b17b1b0d47c2bd650cdcab9f322c0\n\nis a 2d grayscale. All other grayscale images have a channel dimension.", "votes": 2 }, { "id": 312608, "postDate": "2018-04-12T03:14:21.690Z", "content": "In stage2_test, this image:
\n\n170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece
\n\nHas a .png extension but is actually a jpg file. It is markedly different than any other file in the set. Was this unintentionally added to the stage 2 set?
", "rawMarkdown": "In stage2_test, this image:\n\n170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece\n\nHas a .png extension but is actually a jpg file. It is markedly different than any other file in the set. Was this unintentionally added to the stage 2 set?", "votes": 2, "replies": [ { "id": 312610, "postDate": "2018-04-12T03:23:06.110Z", "content": "Nice catch. Are there more jpgs or other than png format in the mix?
", "rawMarkdown": "Nice catch. Are there more jpgs or other than png format in the mix?" }, { "id": 312613, "postDate": "2018-04-12T03:33:20.250Z", "content": "Answering my own question:\n5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png\nis actually a TIFF image. Yours and this are the only exception. All others have the PNG magic number.
", "rawMarkdown": "Answering my own question:\n5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png\nis actually a TIFF image. Yours and this are the only exception. All others have the PNG magic number." }, { "id": 312621, "postDate": "2018-04-12T03:54:29.320Z", "content": "Yep, I just discovered that myself and posted it in another thread. It has a caption on it like it was taken straight out of a textbook.
", "rawMarkdown": "Yep, I just discovered that myself and posted it in another thread. It has a caption on it like it was taken straight out of a textbook." }, { "id": 312662, "postDate": "2018-04-12T05:31:12.007Z", "content": "Anyway, the structures in those 2 images look more like mitochondria than nuclei ... π
", "rawMarkdown": "Anyway, the structures in those 2 images look more like mitochondria than nuclei ... π" }, { "id": 312665, "postDate": "2018-04-12T05:35:20.950Z", "content": "The filename that begins with ffbc47f... also has an issue when read as a PNG - please verify.
", "rawMarkdown": "The filename that begins with **ffbc47f...** also has an issue when read as a PNG - please verify." } ] }, { "id": 281894, "postDate": "2018-02-13T08:33:33.620Z", "content": "While filling the holes in some of the masks I came across a new issue. Once the hole is filled the mask overlaps with another mask. It looks like the annotators were annotating two overlapping nuclei. Should these be excluded?
\n\nThe images left to right
\n\nThis one has a very bright circle and lighter background circle. Seems like they are supposed to be two nuclei. Since they are overlapping shouldn't they be excluded?
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/281894/8557/df53.png\")
Similar case\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/281894/8556/8efe.png\")
Multiple overlapping nuclei? Somehow the annotation looks odd.\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/281894/8558/da31.png\")
Another overlapping nuclei annotated as separate images, one with hole that matches the other.\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/281894/8559/0827.png\")
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/281894/8560/9bb6.png\")
Hi Jerry - indeed those look very intentionally annotated. We did as a team agree that whenever two cells overlapped, the top-most cell would be annotated first, then we'd do the second one. I didn't personally realize that sometimes this meant the second nucleus would be a ring.
\n\nI suspect this doesn't happen often (is it only these 5 you found in the training/test?) As for how to handle it, I guess the answer is that this is indeed a valid annotation (by our agreed-upon custom, which aim for non-overlapping nuclei at all costs).
", "rawMarkdown": "Hi Jerry - indeed those look very intentionally annotated. We did as a team agree that whenever two cells overlapped, the top-most cell would be annotated first, then we'd do the second one. I didn't personally realize that sometimes this meant the second nucleus would be a ring.\n\nI suspect this doesn't happen often (is it only these 5 you found in the training/test?) As for how to handle it, I guess the answer is that this is indeed a valid annotation (by our agreed-upon custom, which aim for non-overlapping nuclei at all costs). " }, { "id": 282027, "postDate": "2018-02-13T13:51:41.167Z", "content": "Hi Jerry, the examples you show reflect a general rule the annotators used to draw masks around the nuclei: if more than one nuclei overlap then the nuclei that appears to be the top most nuclei should be given preference during annotation.
", "rawMarkdown": "Hi Jerry, the examples you show reflect a general rule the annotators used to draw masks around the nuclei: if more than one nuclei overlap then the nuclei that appears to be the top most nuclei should be given preference during annotation." }, { "id": 282070, "postDate": "2018-02-13T15:32:25.153Z", "content": "Hi, Jerry. Why would you want to exclude the overlapping objects?
\n\nThe masks (the last two images in each row) look fantastic IMO!
", "rawMarkdown": "Hi, Jerry. Why would you want to exclude the overlapping objects?\n\nThe masks (the last two images in each row) look fantastic IMO!" }, { "id": 282202, "postDate": "2018-02-13T18:57:43.493Z", "content": "Our team chatted about it Jerry, and we think that with respect to the scoring, you would want to fill holes as you've done in the second image in each set. We think this will have VERY little (possibly zero) impact because I don't think there are cases like this in stage 2 test, but I will keep an eye out for it in our final look-over.
\n\n( Allen, the last two images are the provided ground truth :D )
", "rawMarkdown": "Our team chatted about it Jerry, and we think that with respect to the scoring, you would want to fill holes as you've done in the second image in each set. We think this will have VERY little (possibly zero) impact because I don't think there are cases like this in stage 2 test, but I will keep an eye out for it in our final look-over.\n\n( Allen, the last two images are the provided ground truth :D )" }, { "id": 282649, "postDate": "2018-02-14T10:28:34.163Z", "content": "Thanks for the response Anne, Allen & Kyle.
\n\n@Anne I found 8 such cases. Though not all are completely similar. These are the ones where the hole in one fits another. The others are ones where a part of one overlaps the other.
\n\n@Allen: I agree that the annotations look great! I was trying to understand the annotations. I remember reading a rule that overlapping nuclei are supposed to be excluded. The way I had interpreted that statement was that only nuclei which are non overlapping are to be identified and the examples I posted above were cases where one nucleus overlaps another. Hence the question, should I exclude them.
\n\n@Anne: Given two nuclei that overlap there are two possibilities. Smaller one on to of a larger one like the ones above in which case you have clarified that even the larger one should be identified as a mask without a hole. The second possibility is of a partial overlap. In this case should both be detected as separate nuclei, one with the overlapping region included in it's mask and one without the overlapping region in it's mask? Or is the area of overlap in the mask(s) not significant as in the case of the holes?
\n\nSorry for repeating the question, I am trying to make sure I understood the rules on overlapping nuclei correctly.
", "rawMarkdown": "Thanks for the response Anne, Allen & Kyle. \n\n@Anne I found 8 such cases. Though not all are completely similar. These are the ones where the hole in one fits another. The others are ones where a part of one overlaps the other. \n\n@Allen: I agree that the annotations look great! I was trying to understand the annotations. I remember reading a rule that overlapping nuclei are supposed to be excluded. The way I had interpreted that statement was that only nuclei which are non overlapping are to be identified and the examples I posted above were cases where one nucleus overlaps another. Hence the question, should I exclude them. \n\n@Anne: Given two nuclei that overlap there are two possibilities. Smaller one on to of a larger one like the ones above in which case you have clarified that even the larger one should be identified as a mask without a hole. The second possibility is of a partial overlap. In this case should both be detected as separate nuclei, one with the overlapping region included in it's mask and one without the overlapping region in it's mask? Or is the area of overlap in the mask(s) not significant as in the case of the holes?\n\nSorry for repeating the question, I am trying to make sure I understood the rules on overlapping nuclei correctly." }, { "id": 282832, "postDate": "2018-02-14T17:15:44.513Z", "content": "Hi Jerry, if I knew for sure I would answer your question clearly. I know how I personally annotated images, but it was a team that worked together on this so I don't want to give you concrete guidance and then find out in the test set there is something contradicting my advice!
\n\nI really appreciate you confirming this is a rare case - 8 out of the thousands we annotated, apparently. Again, I would just say that I don't believe there are any cases in the test set of nuclei that overlap in such a way as to make one enclosing the other fully - but just in case I'd advise \"you would want to fill holes as you've done in the second image in each set.\" which means ignoring the smaller enclosed one and leaving just the larger enclosing one.
", "rawMarkdown": "Hi Jerry, if I knew for sure I would answer your question clearly. I know how I personally annotated images, but it was a team that worked together on this so I don't want to give you concrete guidance and then find out in the test set there is something contradicting my advice!\n\nI really appreciate you confirming this is a rare case - 8 out of the thousands we annotated, apparently. Again, I would just say that I don't believe there are any cases in the test set of nuclei that overlap in such a way as to make one enclosing the other fully - but just in case I'd advise \"you would want to fill holes as you've done in the second image in each set.\" which means ignoring the smaller enclosed one and leaving just the larger enclosing one.", "votes": 1 }, { "id": 303075, "postDate": "2018-03-25T12:51:12.403Z", "content": "πΎππ ππππ πππππππ πππππ ππ πΉππππ’, πππ π π πππππ ππππ π πππ πππππππ ππ πππ πππππππ, π’ππ π ππππ π πππ ππ ππππ πππππ ππ π’ππ'ππ ππππ ππ πππ ππππππ πππππ ππ ππππ πππ. ππ πππππ ππππ π πππ ππππ π π΄ππ ππππππ (ππππππππ’ π£πππ) ππππππ πππππππ πΈ πππ'π πππππ πππππ πππ πππππ ππππ ππππ ππ πππππ πΈ ππππ, πππ πΈ π πππ ππππ ππ ππ’π πππ πππ ππ ππ πππ πππππ ππππ-ππππ.
\n\nWhat? Look at https://www.kaggle.com/c/data-science-bowl-2018#evaluation
\n\nπππ ππππππ ππππππ ππππ πππ πππππ πππ ππππππ, ππππππππ, πππ πππ πππππππ\n πππ‘ππ ππππππ πππ πππ ππππππππππ. πΈπ ππππ ππππππ ππππ ππ ππ π πππππππππ\nπππππ πππ πππ ππππ πππππ πππ πππππππππππ.
\n\nThen do you mean that in such cases we need to annotate two nucleis as one, which is completely unlike that's out there in the training data, or that we need to submit two masks as sharing the same pixels, which goes against what I've just quoted from the evaluation metrics page?
\n\nIMHO, you need to get your evaluation masks up to the standards described in the training data. Although, I must say, it's not easy to understand what those are at many times. For example, some tiny specs are marked as nuclei, while others aren't with no apparent logic. I don't think a human would be able to score better than the algorithms on this task and I'm guessing that winner's neural nets are probably going to do a better, more consistent job with annotation than what we can find in the training set.
", "rawMarkdown": "πΎππ ππππ πππππππ πππππ ππ πΉππππ’, πππ π π πππππ ππππ π πππ πππππππ ππ πππ πππππππ, π’ππ π ππππ π πππ ππ ππππ πππππ ππ π’ππ'ππ ππππ ππ πππ ππππππ πππππ ππ ππππ πππ. ππ πππππ ππππ π πππ ππππ π π΄ππ ππππππ (ππππππππ’ π£πππ) ππππππ πππππππ πΈ πππ'π πππππ πππππ πππ πππππ ππππ ππππ ππ πππππ πΈ ππππ, πππ πΈ π πππ ππππ ππ ππ’π πππ πππ ππ ππ πππ πππππ ππππ-ππππ.\n\nWhat? Look at https://www.kaggle.com/c/data-science-bowl-2018#evaluation\n\nπππ ππππππ ππππππ ππππ πππ πππππ πππ ππππππ, ππππππππ, πππ πππ πππππππ\n πππ‘ππ ππππππ πππ πππ ππππππππππ. πΈπ ππππ ππππππ ππππ ππ ππ π πππππππππ\nπππππ πππ πππ ππππ πππππ πππ πππππππππππ.\n\nThen do you mean that in such cases we need to annotate two nucleis as one, which is completely unlike that's out there in the training data, or that we need to submit two masks as sharing the same pixels, which goes against what I've just quoted from the evaluation metrics page?\n\nIMHO, you need to get your evaluation masks up to the standards described in the training data. Although, I must say, it's not easy to understand what those are at many times. For example, some tiny specs are marked as nuclei, while others aren't with no apparent logic. I don't think a human would be able to score better than the algorithms on this task and I'm guessing that winner's neural nets are probably going to do a better, more consistent job with annotation than what we can find in the training set." }, { "id": 303147, "postDate": "2018-03-25T16:07:05.150Z", "content": "Hi there, we are talking here about an issue that affects 8 nuclei out of the ~20,000 that we annotated by hand. As I mentioned, the former: \"annotate two nuclei as one\" is my recommendation.
\n\nWith respect to you not understanding the logic of how the expert biologists annotated the images: that is what makes us expert biologists. You are most welcome to organize your own challenge and annotate however you like.
", "rawMarkdown": "Hi there, we are talking here about an issue that affects 8 nuclei out of the ~20,000 that we annotated by hand. As I mentioned, the former: \"annotate two nuclei as one\" is my recommendation.\n\nWith respect to you not understanding the logic of how the expert biologists annotated the images: that is what makes us expert biologists. You are most welcome to organize your own challenge and annotate however you like." }, { "id": 303201, "postDate": "2018-03-25T18:34:45.683Z", "content": "Just read this comment
\n\nπΏπππ, πππ πππππππππππ ππππ πππ ππ ππππππππππ πππ ππππ ππππ ππππππ (π· &\nπΈ) ππ πππ ππππ πππππππ, πππππ π°π΅π°πΈπΊ. ππ ππ π πππ ππ ππ π’πππ ππππ\nππππππππ ππ πππ πππππππ π πππππ ππππ π’πππππ πππππππ.
\n\nSo I guess that's the answer.
\n\nEdit: Thanks for the reply, haven't saw it before posting the above.
", "rawMarkdown": "Just read this comment\n\nπΏπππ, πππ πππππππππππ ππππ πππ ππ ππππππππππ πππ ππππ ππππ ππππππ (π· &\nπΈ) ππ πππ ππππ πππππππ, πππππ π°π΅π°πΈπΊ. ππ ππ π πππ ππ ππ π’πππ ππππ\nππππππππ ππ πππ πππππππ π πππππ ππππ π’πππππ πππππππ.\n\nSo I guess that's the answer.\n\nEdit: Thanks for the reply, haven't saw it before posting the above." }, { "id": 303230, "postDate": "2018-03-25T20:05:12.203Z", "content": "The major problem is not with 8 nuclei in the training set, but with any potential ones in the final evaluation set which might make a difference of, say, getting into top 5 vs getting into to 10.
", "rawMarkdown": "The major problem is not with 8 nuclei in the training set, but with any potential ones in the final evaluation set which might make a difference of, say, getting into top 5 vs getting into to 10. ", "votes": 1 } ] }, { "id": 278622, "postDate": "2018-02-06T15:50:44.920Z", "content": "Image af576e8ec3a8d0b57eb6a311299e9e4fd2047970d3dd9d6f52e54ea6a91109da seem to miss one mask.
", "rawMarkdown": "Image af576e8ec3a8d0b57eb6a311299e9e4fd2047970d3dd9d6f52e54ea6a91109da seem to miss one mask.", "votes": 2, "replies": [ { "id": 279199, "postDate": "2018-02-07T15:20:20.043Z", "content": "Could you please be more specific?
", "rawMarkdown": "Could you please be more specific? " }, { "id": 304723, "postDate": "2018-03-27T23:47:04.090Z", "content": "It seems to contain up to 6 nuclei, pretty sure more than the labeled two.
", "rawMarkdown": "It seems to contain up to 6 nuclei, pretty sure more than the labeled two." } ] }, { "id": 277952, "postDate": "2018-02-04T22:30:12.063Z", "content": "Am I missing masks for adc315bd40d699fd4e4effbcce81cd7162851007f485d754ad3b0472f73a86df ?\n(green = my attempts, yellow = agreement with masks, red = masks)
", "rawMarkdown": "Am I missing masks for adc315bd40d699fd4e4effbcce81cd7162851007f485d754ad3b0472f73a86df ?\n(green = my attempts, yellow = agreement with masks, red = masks)", "votes": 2, "replies": [ { "id": 279192, "postDate": "2018-02-07T15:13:30.950Z", "content": "Indeed, your green guys are correct!
", "rawMarkdown": "Indeed, your green guys are correct!" } ] }, { "id": 273386, "postDate": "2018-01-24T14:54:25.930Z", "content": "For this band-like object (what is it?) I can see some \"nuclei-like\" things beneath it. They are not annotated in the mask. So those things are not considered as nucleis?\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/273386/8365/193ffaa5272d5c421ae02130a64d98ad120ec70e4ed97a72cdcd4801ce93b066.png\")
This might be a case where annotators disagree. For me, if I could see that something was a nucleus, even behind a piece of debris like this, I would annotate it. Thankfully, debris like this is not common.
", "rawMarkdown": "This might be a case where annotators disagree. For me, if I could see that something was a nucleus, even behind a piece of debris like this, I would annotate it. Thankfully, debris like this is not common.", "votes": 4 } ] }, { "id": 271427, "postDate": "2018-01-20T12:26:50.860Z", "content": "I am wondering if train data is generated by a software or not. As you can see in the following pictures, the blue rectangles show areas with masks containing only a few pixels where on the original image, no stain is visible. However, this behavior is not consistent as you can see in the third picture that shows a stain which has no corresponding mask.
\n\nMoreover, In the second image, the red rectangles have masks but the green ones do not. To me, they all look the same and I cannot distinguish between them. Would anyone please clarify?
\n\nFirst Image:\n![\"enter](\"https://s14.postimg.org/5x24bdrsh/5479.png\")
\nSecond Image:\n![\"enter](\"https://s14.postimg.org/3shra0fsh/74a77.png\")
\nThird image:\n![\"enter](\"https://s14.postimg.org/8sf7iadsx/a7f767.png\")
Hello there. None of these are errors per se, as far as I can tell.
\n\nMy guess is that on the first image if you remove the black outline around the image, the nucleus stain would be more visible. It's also possible the image was cropped after annotating and those pixels would have been more obviously part of a nucleus if more nucleus was in the field of view.
\n\nRegardless, it seems that you are more interested in the general principles than specific instances, so let me elaborate on that. Although the team discussed our criteria frequently and agreed upon strategy* together, the annotators each had their own monitor and style so how dim something would be in order to be ignored will vary from image to image. In the case of the second & third images, if they appeared on the annotator's monitor the way I'm seeing it now, they should have outlined far more of those dim burry nuclei. Hope this helps!
\n\nThanks for the explanation. It makes sense for few-pixel nucleus that the images are cropped after the annotation.
\n\nStill, I think there are some errors, if error is excluding a nuclei in the masks. I understand that every monitor is slightly different and annotators could miss some blurry nucleus. Taking image 2, the red rectangles show nucleus that are as blurry as the one just right to the 200 tick. But this one is not annotated and it cannot be attributed to differences in monitors assuming a slide is annotated on a single monitor. Even the red and green rectangles show nucleus with the same intensity and they are to be missed all together.
", "rawMarkdown": "Thanks for the explanation. It makes sense for few-pixel nucleus that the images are cropped after the annotation.\n\nStill, I think there are some errors, if error is excluding a nuclei in the masks. I understand that every monitor is slightly different and annotators could miss some blurry nucleus. Taking image 2, the red rectangles show nucleus that are as blurry as the one just right to the 200 tick. But this one is not annotated and it cannot be attributed to differences in monitors assuming a slide is annotated on a single monitor. Even the red and green rectangles show nucleus with the same intensity and they are to be missed all together.", "votes": 1 }, { "id": 272208, "postDate": "2018-01-22T13:52:38.747Z", "content": "Personally, I agree that those nuclei should have been annotated, given the images how I see them on my monitor now. I just want to be clear that individual annotators and their monitors will vary.
", "rawMarkdown": "Personally, I agree that those nuclei should have been annotated, given the images how I see them on my monitor now. I just want to be clear that individual annotators and their monitors will vary.", "votes": 2 }, { "id": 313221, "postDate": "2018-04-13T00:18:27.313Z", "content": "After scaling the image RGB values to (0..1), I displayed np.sqrt(img) alongside the original image (to reduce the dynamic range). The dark images all come up beautifully clearly - exposing lots of invisible nucleii (and various interesting background artifacts).
", "rawMarkdown": "After scaling the image RGB values to (0..1), I displayed np.sqrt(img) alongside the original image (to reduce the dynamic range). The dark images all come up beautifully clearly - exposing lots of invisible nucleii (and various interesting background artifacts)." }, { "id": 313230, "postDate": "2018-04-13T00:52:29.663Z", "content": "That's interesting! Did you use this in your solution?
", "rawMarkdown": "That's interesting! Did you use this in your solution?" }, { "id": 313234, "postDate": "2018-04-13T01:13:27.943Z", "content": "For my stage1 submission, I used sqrt on the monochrome images and left the color ones alone.
\n\nTo be honest, I have no idea whether it improved matters - I wrote the final bits of code shortly before the stage1 deadline, so didn't have any time to play with it. This is also my first ever (non-toy non-MATLAB) neural net, so any opinions I could offer would probably be laughably inaccurate :)
", "rawMarkdown": "For my stage1 submission, I used sqrt on the monochrome images and left the color ones alone.\n\nTo be honest, I have no idea whether it improved matters - I wrote the final bits of code shortly before the stage1 deadline, so didn't have any time to play with it. This is also my first ever (non-toy non-MATLAB) neural net, so any opinions I could offer would probably be laughably inaccurate :)" }, { "id": 313236, "postDate": "2018-04-13T01:23:08.560Z", "content": "Original, sqrt and log in different channels.\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/312802/9132/download%20(6).png\")
I don't think your idea was stupid. Actually, probably pretty great. Might be one of the best ones this contest. But needs testing.
", "rawMarkdown": "I don't think your idea was stupid. Actually, probably pretty great. Might be one of the best ones this contest. But needs testing." }, { "id": 313247, "postDate": "2018-04-13T01:49:47.433Z", "content": "As, you see, weights are linear, even after ReLU. So we're helping the net by making some adjjustments for it with convexity. It usually hurts perf when you put convex activation functions deep into the net - networks easily overfit as the actual data and logic, it seems, is far more linear and specific non-linearities can be learned via combining different linear stuff and ReLUs, or other simple more linear activations, MaxOut, is a notable example. But here we're talking about stuff we know about our data, it's lighting conditions and inherent specific stuff about it, it's not deep inside the net, so I don't think this will trigger any overfitting. May depend on an amount of training data, though.
", "rawMarkdown": "As, you see, weights are linear, even after ReLU. So we're helping the net by making some adjjustments for it with convexity. It usually hurts perf when you put convex activation functions deep into the net - networks easily overfit as the actual data and logic, it seems, is far more linear and specific non-linearities can be learned via combining different linear stuff and ReLUs, or other simple more linear activations, MaxOut, is a notable example. But here we're talking about stuff we know about our data, it's lighting conditions and inherent specific stuff about it, it's not deep inside the net, so I don't think this will trigger any overfitting. May depend on an amount of training data, though." }, { "id": 313248, "postDate": "2018-04-13T01:54:55.997Z", "content": "Thanks :)
", "rawMarkdown": "Thanks :)" }, { "id": 313255, "postDate": "2018-04-13T02:16:54.433Z", "content": "May also take some skills to make it work. I'll report my findings. Thus far seems promising after one nice little trick I love in the nets. My favourite one. ;) But I only started the learning process. Got to pass more iterations first.
", "rawMarkdown": "May also take some skills to make it work. I'll report my findings. Thus far seems promising after one nice little trick I love in the nets. My favourite one. ;) But I only started the learning process. Got to pass more iterations first." } ] }, { "id": 271020, "postDate": "2018-01-19T13:45:32.097Z", "content": "stage1_train/12aeefb1b522b283819b12e4cfaf6b13c1264c0aadac3412b4edd2ace304cb40
", "rawMarkdown": "stage1_train/12aeefb1b522b283819b12e4cfaf6b13c1264c0aadac3412b4edd2ace304cb40", "votes": 2, "replies": [ { "id": 271193, "postDate": "2018-01-19T21:32:44.973Z", "content": "Hello, organizer here. I agree that image was badly annotated.
", "rawMarkdown": "Hello, organizer here. I agree that image was badly annotated." } ] }, { "id": 270666, "postDate": "2018-01-18T17:48:56.497Z", "content": "Just re-post @heng's finding, the error train image:\n Stage1_train/7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80
\n\nRef: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47739
", "rawMarkdown": "Just re-post @heng's finding, the error train image:\n Stage1_train/7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80\n\nRef: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47739", "votes": 2 }, { "id": 313622, "postDate": "2018-04-13T15:27:17.700Z", "content": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?
", "rawMarkdown": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?" }, { "id": 312536, "postDate": "2018-04-12T00:38:49.463Z", "content": "Is the stage2_test/15cdf102276bc0edc7491e15e6960e74f7bb47e1f0d7685cb7cbdc3f914c56c5 image all black ?
", "rawMarkdown": "Is the stage2_test/15cdf102276bc0edc7491e15e6960e74f7bb47e1f0d7685cb7cbdc3f914c56c5 image all black ?" }, { "id": 309795, "postDate": "2018-04-06T02:01:03.740Z", "content": "Left is contrast enhanced image # 'f8e74d4006dd68c1dbe68df7be905835e00d8ba4916f3b18884509a15fdc0b55'
\n\nRight is the image labeled using the mask from the training set
\n\nAre these small objects really nuclei?\n![\"bad](\"https://i.imgur.com/X7rdSRC.png\")
Yes, those are what are called \"micronuclei\" and we decided to include them in the challenge (in part because if a model finds them all properly, then a biologist can filter them downstream simply based on size)
", "rawMarkdown": "Yes, those are what are called \"micronuclei\" and we decided to include them in the challenge (in part because if a model finds them all properly, then a biologist can filter them downstream simply based on size)" } ] }, { "id": 306822, "postDate": "2018-03-31T04:44:23.937Z", "content": "Hi,\n Could you please look into the mask of the image 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80? I think the mask files are missing, only one mask.....
", "rawMarkdown": "Hi,\n Could you please look into the mask of the image 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80? I think the mask files are missing, only one mask....." }, { "id": 305148, "postDate": "2018-03-28T14:30:06.010Z", "content": "In d256b32adda37f2301c9e46f34b7f9a36cce273256369ceb5dc2c73c3007e3c4 are you certain 0a13899f3cd5597920b9cdb078c90a48f9bd63ca4a2d413414bf83bc647ea126.png is not the same nucleus as as the one marked right below it - b4cd6644b0975c0603e3023119614653c7e379d6bec71e267fa7f07bc93830bc.png ?
", "rawMarkdown": "In d256b32adda37f2301c9e46f34b7f9a36cce273256369ceb5dc2c73c3007e3c4 are you certain 0a13899f3cd5597920b9cdb078c90a48f9bd63ca4a2d413414bf83bc647ea126.png is not the same nucleus as as the one marked right below it - b4cd6644b0975c0603e3023119614653c7e379d6bec71e267fa7f07bc93830bc.png ?" }, { "id": 304753, "postDate": "2018-03-28T01:13:43.017Z", "content": "As has been reported in another thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/53134), 20b20ab049372d184c705acebe7af026d3580f5fd5a72ed796e3622e1685af2f and cb4df20a83b2f38b394c67f1d9d4aef29f9794d5345da3576318374ec3a11490 are the same image in two contrast/lighting conditions. Annotated with a vastly different amount of masks. If organizers would at least tell which annotations to trust, that would be most helpful.
", "rawMarkdown": "As has been reported in another thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/53134), 20b20ab049372d184c705acebe7af026d3580f5fd5a72ed796e3622e1685af2f and cb4df20a83b2f38b394c67f1d9d4aef29f9794d5345da3576318374ec3a11490 are the same image in two contrast/lighting conditions. Annotated with a vastly different amount of masks. If organizers would at least tell which annotations to trust, that would be most helpful.", "replies": [ { "id": 308053, "postDate": "2018-04-02T21:46:47.840Z", "content": "Hi there, see that thread for a reply (spoiler alert: probably an unsatisfying reply!).
", "rawMarkdown": "Hi there, see that thread for a reply (spoiler alert: probably an unsatisfying reply!). " } ] }, { "id": 304010, "postDate": "2018-03-26T22:04:37.427Z", "content": "243443ae303cc09cfbea85bfd22b0c4f026342f3dfc3aa1076f27867910d025b seems like there are 3 nucleuses that are merged, although they could be annotated as separate. The ones that are of avarage brightness and the ones that shine very brightly on this picture. Or should they be/are they a single instance?
", "rawMarkdown": "243443ae303cc09cfbea85bfd22b0c4f026342f3dfc3aa1076f27867910d025b seems like there are 3 nucleuses that are merged, although they could be annotated as separate. The ones that are of avarage brightness and the ones that shine very brightly on this picture. Or should they be/are they a single instance?", "replies": [ { "id": 304320, "postDate": "2018-03-27T11:05:49.753Z", "content": "Could you be more specific about which nuclei? Thanks!
", "rawMarkdown": "Could you be more specific about which nuclei? Thanks!" }, { "id": 304744, "postDate": "2018-03-28T00:59:58.193Z", "content": "7ca7532afd7d57016562238823cb26abd955c2085ec4d1c343af416555be561a and the one above that. I don't have any tools to easily find the mask name of the one above.
", "rawMarkdown": "7ca7532afd7d57016562238823cb26abd955c2085ec4d1c343af416555be561a and the one above that. I don't have any tools to easily find the mask name of the one above.\n" } ] }, { "id": 303232, "postDate": "2018-03-25T20:12:45.153Z", "content": "Michael Kazachok's kernel https://www.kaggle.com/miklgr500/data-analysis raises a point that the test set #1 doesn't contain images the likes of 8d05fb18ee0cda107d56735cafa6197a31884e0a5092dc6d41760fb92ae23ab4. Am I correct to assume that the final test set is going to contain those types of images?
", "rawMarkdown": "Michael Kazachok's kernel https://www.kaggle.com/miklgr500/data-analysis raises a point that the test set #1 doesn't contain images the likes of 8d05fb18ee0cda107d56735cafa6197a31884e0a5092dc6d41760fb92ae23ab4. Am I correct to assume that the final test set is going to contain those types of images?", "replies": [ { "id": 304322, "postDate": "2018-03-27T11:11:16.130Z", "content": "I cannot answer that (for fear of giving too much away, and also I don't recall, and also whether an image is of the same \"type\" or not is a bit subjective).
\n\nBut what I can say is that this challenge is all about generalization, so what is public knowledge is the fact that the stage 2 test set contains image sets/experiments that are mutually exclusive from training set and stage 1 test set. Do expect variety, and unseen image types.
", "rawMarkdown": "I cannot answer that (for fear of giving too much away, and also I don't recall, and also whether an image is of the same \"type\" or not is a bit subjective). \n\nBut what I *can* say is that this challenge is all about generalization, so what *is* public knowledge is the fact that the stage 2 test set contains image sets/experiments that are mutually exclusive from training set and stage 1 test set. Do expect variety, and unseen image types. ", "votes": 1 }, { "id": 304323, "postDate": "2018-03-27T11:12:00.887Z", "content": "Take a look here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#290093
", "rawMarkdown": "Take a look here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#290093", "votes": 1 } ] }, { "id": 295901, "postDate": "2018-03-14T12:00:45.547Z", "content": "Dear all, i had a look at the trainingset and at the discussion here and would like to ask: the trainingset contains a lot of errors, it seems as if some semi-automated tool was used to create the annotation masks. What i understood from the discussion, the errors won't be corrected within the trainingset of stage 1, we can do it by ourselves if we like to, but for the stage2, there won't be those errors true? What about the evaluation of stage 1 - if we come up with a solution able to distinguish most of the touching nuclei, but the testset contains many errors, our evaluation score will be low although we propose good results, true?
", "rawMarkdown": "Dear all, i had a look at the trainingset and at the discussion here and would like to ask: the trainingset contains a lot of errors, it seems as if some semi-automated tool was used to create the annotation masks. What i understood from the discussion, the errors won't be corrected within the trainingset of stage 1, we can do it by ourselves if we like to, but for the stage2, there won't be those errors true? What about the evaluation of stage 1 - if we come up with a solution able to distinguish most of the touching nuclei, but the testset contains many errors, our evaluation score will be low although we propose good results, true?", "replies": [ { "id": 303212, "postDate": "2018-03-25T18:56:59.563Z", "content": "As far as organizers are saying, thay did a much better job annotating the evaluation set.
", "rawMarkdown": "As far as organizers are saying, thay did a much better job annotating the evaluation set.", "votes": 1 } ] }, { "id": 293441, "postDate": "2018-03-09T20:51:07.287Z", "content": "stage1_train/9facc652efe19f634639585d692a53dd6c2a8e2f0c9baebdfd85b9b41ec58851/masks/9551f2dad01c71f81d5e2386a99bc963457338d0c79fda0e90e9226f50b5dada.png
\n\nzero width annotation. can just be ignored i guess (posted in case someone is collecting a error less dataset)
", "rawMarkdown": "stage1_train/9facc652efe19f634639585d692a53dd6c2a8e2f0c9baebdfd85b9b41ec58851/masks/9551f2dad01c71f81d5e2386a99bc963457338d0c79fda0e90e9226f50b5dada.png\n\nzero width annotation. can just be ignored i guess (posted in case someone is collecting a error less dataset)" }, { "id": 288978, "postDate": "2018-02-28T17:14:10.177Z", "content": "0d2bf916cc8de90d02f4cd4c23ea79b227dbc45d845b4124ffea380c92d34c8c -> 1c344631e35e9e282cf3cab642cafadff0745944a81b0715ab0a13944b18cfb9 merged nuclei in mask.
", "rawMarkdown": "0d2bf916cc8de90d02f4cd4c23ea79b227dbc45d845b4124ffea380c92d34c8c -> 1c344631e35e9e282cf3cab642cafadff0745944a81b0715ab0a13944b18cfb9 merged nuclei in mask." }, { "id": 283810, "postDate": "2018-02-16T02:01:11.460Z", "content": "Have any of the suggested errors been fixed in the training dataset that is available for download ? I am just wondering if I should be downloading the training dataset again or manually fixing these issues on my local copy
", "rawMarkdown": "Have any of the suggested errors been fixed in the training dataset that is available for download ? I am just wondering if I should be downloading the training dataset again or manually fixing these issues on my local copy", "replies": [ { "id": 284181, "postDate": "2018-02-16T17:55:35.573Z", "content": "The organiser has replied elsewhere in this thread: competition dataset will not be changed. You can fix it for yourself in your local copy.
", "rawMarkdown": "The organiser has replied elsewhere in this thread: competition dataset will not be changed. You can fix it for yourself in your local copy." }, { "id": 284204, "postDate": "2018-02-16T18:41:56.343Z", "content": "\n\n\nThe organiser has replied elsewhere in this thread: competition dataset will not be changed. You can fix it for yourself in your local copy.
\n
This statement needs a disclaimer: if you decide to \"fix\" your local copy, it will probably get you to the better ground truth - assuming you know how to annotate nuclei. Getting a better ground truth, however, will almost certainly result in lower LB score. That wouldn't be necessarily bad if it meant that the model will generalize better, yet that is assuming that the SAME annotators that did stage 1 date will not be making ANY of the mistakes that have been reported so far.
", "rawMarkdown": "> The organiser has replied elsewhere in this thread: competition dataset will not be changed. You can fix it for yourself in your local copy.\n\nThis statement needs a disclaimer: if you decide to \"fix\" your local copy, it will probably get you to the better ground truth - assuming you know how to annotate nuclei. Getting a better ground truth, however, will almost certainly result in lower LB score. That wouldn't be necessarily bad if it meant that the model will generalize better, yet that is assuming that the SAME annotators that did stage 1 date will not be making ANY of the mistakes that have been reported so far.", "votes": 1 }, { "id": 285637, "postDate": "2018-02-20T19:12:33.970Z", "content": "I personally disagree that fixing ground truth would give a lower LB score.
\n\nThe only way that could be the case is if there were systematic errors you hoped that your model would replicate. The majority of what people are pointing out are random one-off sorts of mistakes, like a missed nucleus or merged nuclei here and there. It's hard to imagine wanting your model to attempt to learn that sort of error to get a better score.
\n\nFurthermore, we are looking over the stage 2 test set much more carefully than the training/validation set.
", "rawMarkdown": "I personally disagree that fixing ground truth would give a lower LB score. \n\nThe only way that could be the case is if there were systematic errors you hoped that your model would replicate. The majority of what people are pointing out are random one-off sorts of mistakes, like a missed nucleus or merged nuclei here and there. It's hard to imagine wanting your model to attempt to learn that sort of error to get a better score. \n\nFurthermore, we are looking over the stage 2 test set much more carefully than the training/validation set.\n\n", "votes": 4 }, { "id": 303308, "postDate": "2018-03-26T01:22:14.187Z", "content": "Sadly, there is definitely some amount of truth to the statement about lowering estimated accuracy. The tool you guys used for annotations, with those flood fills (\"paint bucket tool\" kind of \"fill\") very consistently gives slightly wrong masks. And I'm guessing you're going to use this tool to annotate the eval set as well. Areas that are obviously not a part of a nucleus, but which are lightened due to proximity of another nucleus nearby are considered to be a part of nucleus in the dataset. You can find a ton of evidence of this in c304a1fdf3bca2f4b4580d2cac59942e2224a7678001bf5ed9d9852f57708932, for example. The effect of this on a score, I'm guessing, is going to be quite huge. Huge not in absolute terms, but in relative, as an absolutely small difference is going to mean a large difference in rating.
\n\nThe only way to solve this would be best if you'd ensure pixel-perfect annotation. It would be more beneficial to have a smaller eval set which is pixel-perfect than a large one that has systemic flaws. It's a question of the actual best solution winning the contest, or a dice throw.
", "rawMarkdown": "Sadly, there is definitely some amount of truth to the statement about lowering estimated accuracy. The tool you guys used for annotations, with those flood fills (\"paint bucket tool\" kind of \"fill\") very consistently gives slightly wrong masks. And I'm guessing you're going to use this tool to annotate the eval set as well. Areas that are obviously not a part of a nucleus, but which are lightened due to proximity of another nucleus nearby are considered to be a part of nucleus in the dataset. You can find a ton of evidence of this in c304a1fdf3bca2f4b4580d2cac59942e2224a7678001bf5ed9d9852f57708932, for example. The effect of this on a score, I'm guessing, is going to be quite huge. Huge not in absolute terms, but in relative, as an absolutely small difference is going to mean a large difference in rating.\n\nThe only way to solve this would be best if you'd ensure pixel-perfect annotation. It would be more beneficial to have a smaller eval set which is pixel-perfect than a large one that has systemic flaws. It's a question of the actual best solution winning the contest, or a dice throw." }, { "id": 303310, "postDate": "2018-03-26T01:25:34.793Z", "content": "Please, reply whether you guys are going to get rid of that systemic flaw in the final eval, as I'd like to know if I should fix my data, or let my models learn this and other flaws.
\n\nWhat I'd understand as a \"much more careful\" nearly pixel-perfect approach to annotation takes, I'm guessing, around 30 seconds per nucleus with no flood fills and plainly pixel and multipixel brushes used over the original image. Doesn't seem too hard to do for a $100k contest with 3000 participants. I'm guessing one person could do that in a day, or two.
", "rawMarkdown": "Please, reply whether you guys are going to get rid of that systemic flaw in the final eval, as I'd like to know if I should fix my data, or let my models learn this and other flaws.\n\nWhat I'd understand as a \"much more careful\" nearly pixel-perfect approach to annotation takes, I'm guessing, around 30 seconds per nucleus with no flood fills and plainly pixel and multipixel brushes used over the original image. Doesn't seem too hard to do for a $100k contest with 3000 participants. I'm guessing one person could do that in a day, or two." }, { "id": 303814, "postDate": "2018-03-26T17:29:45.197Z", "content": "@theNickName, the hosts are working on better ground truth annotations closely with Kaggle to ensure the quality of the 2nd stage test data.
\n\nAs everyone knows, image segmentation data/labeling is extremely labor intensive and expensive. While we can't guarantee that the masks will be pixel-perfect, we appreciate everyone's input and will do the best we can.
", "rawMarkdown": "@theNickName, the hosts are working on better ground truth annotations closely with Kaggle to ensure the quality of the 2nd stage test data. \n\nAs everyone knows, image segmentation data/labeling is extremely labor intensive and expensive. While we can't guarantee that the masks will be pixel-perfect, we appreciate everyone's input and will do the best we can. " } ] }, { "id": 277533, "postDate": "2018-02-03T13:37:15.253Z", "content": "apoptotic stuff : \nstage 1 test\nd616d323a9eeb9da1b66f8d5df671d63c092c9919cb2c0b223e29c63257c944d
\n\nnuclei outside imaging plane:\nstage 1 test da6c593410340b19bb212b9f6d274f95b08c0fc8f2570cd66bc5ed42c560acab\n550450e4bff4036fd671decdc5d42fec23578198d6a2fd79179c4368b9d6da18
\n\nmetaphase/mitotic \nstage 1 test 0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac\n912a679e4b9b1d1a75170254fd675b8c24b664d80ad7ea7e460241a23535a406\n8b59819fbc92eefe45b1db95c0cc3a467ddcfc755684c7f2ba2f6ccb9ad740ab
\n\nczy liczyc jadra komorkowe znajdujace sie pod tym wloknem?\ndo we count nuclei behind this fiber?\nstage 1 test 336d3e4105766f8ad328a7ee9571e743f376f8cbcf6a969ca7e353fe3235c523
\n\nfunny stuff of different kind (don't know what is shining there)\n52b267e20519174e3ce1e1994b5d677804b16bc670aa5f6ffb6344a0fdf63fde
\n\noversaturated fluorsescent image (pixel grey value over 255, like in overexposed photo)\n43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af
\n\nI understand this is a tissue fragment, but image quality is really low...\n4f949bd8d914bbfa06f40d6a0e2b5b75c38bf53dbcbafc48c97f105bee4f8fac
", "rawMarkdown": "apoptotic stuff : \nstage 1 test\nd616d323a9eeb9da1b66f8d5df671d63c092c9919cb2c0b223e29c63257c944d \n\nnuclei outside imaging plane:\nstage 1 test da6c593410340b19bb212b9f6d274f95b08c0fc8f2570cd66bc5ed42c560acab\n550450e4bff4036fd671decdc5d42fec23578198d6a2fd79179c4368b9d6da18 \n\nmetaphase/mitotic \nstage 1 test 0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac\n912a679e4b9b1d1a75170254fd675b8c24b664d80ad7ea7e460241a23535a406\n8b59819fbc92eefe45b1db95c0cc3a467ddcfc755684c7f2ba2f6ccb9ad740ab\n\nczy liczyc jadra komorkowe znajdujace sie pod tym wloknem?\ndo we count nuclei behind this fiber?\nstage 1 test 336d3e4105766f8ad328a7ee9571e743f376f8cbcf6a969ca7e353fe3235c523\n\nfunny stuff of different kind (don't know what is shining there)\n52b267e20519174e3ce1e1994b5d677804b16bc670aa5f6ffb6344a0fdf63fde\n\noversaturated fluorsescent image (pixel grey value over 255, like in overexposed photo)\n43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af\n\nI understand this is a tissue fragment, but image quality is really low...\n4f949bd8d914bbfa06f40d6a0e2b5b75c38bf53dbcbafc48c97f105bee4f8fac", "replies": [ { "id": 277570, "postDate": "2018-02-03T15:37:05.207Z", "content": "Hi there! \nIt's a bit hard to respond to the message, maybe giving letters or numbers to each item would be easier. I'm going to go through your issues by letter:\nA) Yes, there is indeed some weird stuff happening in the image. \nB) Yes, some nuclei are out of focus but we want to identify them anyway. \nC) Yes, some nuclei are mitotic stage, we want to identify them. \nD) This type of debris occurs in images sometimes. As you can see from the annotation the annotator acted as if the debris was the edge of the image and only annotated the parts of nuclei that are visible. I don't recall whether all annotators were consistent about this, though. \nE) Some of the nuclei are very skinny, is that what you mean?\nF) Yes, overexposed image; we still want to identify nuclei in the image.\nG) Yes, image is dim/low-quality.
\n\nAs far as I can see, there are not annotation errors in these examples (please point them out if you see something specific!) You've indeed found examples of non-ideal images but that is the point of the challenge. For various technical reasons a biologist may need to collect and use non-ideal images and we want the model to be robust to these.
", "rawMarkdown": "Hi there! \nIt's a bit hard to respond to the message, maybe giving letters or numbers to each item would be easier. I'm going to go through your issues by letter:\nA) Yes, there is indeed some weird stuff happening in the image. \nB) Yes, some nuclei are out of focus but we want to identify them anyway. \nC) Yes, some nuclei are mitotic stage, we want to identify them. \nD) This type of debris occurs in images sometimes. As you can see from the annotation the annotator acted as if the debris was the edge of the image and only annotated the parts of nuclei that are visible. I don't recall whether all annotators were consistent about this, though. \nE) Some of the nuclei are very skinny, is that what you mean?\nF) Yes, overexposed image; we still want to identify nuclei in the image.\nG) Yes, image is dim/low-quality. \n\nAs far as I can see, there are not annotation errors in these examples (please point them out if you see something specific!) You've indeed found examples of non-ideal images but that is the point of the challenge. For various technical reasons a biologist may need to collect and use non-ideal images and we want the model to be robust to these. ", "votes": 1 } ] }, { "id": 276063, "postDate": "2018-01-30T16:27:47.033Z", "content": "I need some clarification regarding to 0121d6759c5adb290c8e828fc882f37dfaf3663ec885c663859948c154a443ed \nOn the middle image I marked couple of interesting points.
\n\n![\"Img\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/276063/8440/0121d6759c5adb290c8e828fc882f37dfaf3663ec885c663859948c154a443ed.jpg\")
Let's start from the Red circle. The shape of masks looks like two thick lines, but original image clearly shows one roundish shape.
\n\nYellow mask have a hole inside, is that and error or expected mask?
\n\nGreen mask have 4 (one outside) 'stains' but reference image has different shape (one or two?, hard to say)
\n\nBlue circle similarly to green have few 'stains' instead of one.
\n\nThere are also masks that looks like a square, it seems to be not really realistic. Could you give me some comment about it? Are you going to update dataset regarding the previously found mistakes in masks?
", "rawMarkdown": "I need some clarification regarding to 0121d6759c5adb290c8e828fc882f37dfaf3663ec885c663859948c154a443ed \nOn the middle image I marked couple of interesting points. \n\n![Img][1]\n\nLet's start from the Red circle. The shape of masks looks like two thick lines, but original image clearly shows one roundish shape. \n\nYellow mask have a hole inside, is that and error or expected mask?\n\nGreen mask have 4 (one outside) 'stains' but reference image has different shape (one or two?, hard to say)\n\nBlue circle similarly to green have few 'stains' instead of one. \n\n\nThere are also masks that looks like a square, it seems to be not really realistic. Could you give me some comment about it? Are you going to update dataset regarding the previously found mistakes in masks? \n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/276063/8440/0121d6759c5adb290c8e828fc882f37dfaf3663ec885c663859948c154a443ed.jpg", "replies": [ { "id": 276611, "postDate": "2018-01-31T22:18:30.767Z", "content": "Red: I am afraid the image you uploaded is a bit too low-res and compressed to see whether that is one or two nuclei.
\n\nYellow: definitely an error, good catch!
\n\nGreen: definitely there are some errors there, there should be much more white across those regions (however, this could be a function of the annotator's screen being dim), and I would call that two nuclei. Also, two \"masked objects\" above the green circle, the mask has strange parts missing from it.
\n\nBlue: I would agree this probably should have been annotated as a single nucleus - however, this is quite possibly a function of brightness of the screen for the annotator: I could imagine that if the screen was dim you would see those three objects.
\n\nAfter the competition is over, our lab will omit images or fix annotations, but we have been advised not to do so during the competition itself.
", "rawMarkdown": "Red: I am afraid the image you uploaded is a bit too low-res and compressed to see whether that is one or two nuclei.\n\nYellow: definitely an error, good catch!\n\nGreen: definitely there are some errors there, there should be much more white across those regions (however, this could be a function of the annotator's screen being dim), and I would call that two nuclei. Also, two \"masked objects\" above the green circle, the mask has strange parts missing from it.\n\nBlue: I would agree this probably should have been annotated as a single nucleus - however, this is quite possibly a function of brightness of the screen for the annotator: I could imagine that if the screen was dim you would see those three objects.\n\nAfter the competition is over, our lab will omit images or fix annotations, but we have been advised not to do so during the competition itself.", "votes": 1 }, { "id": 281338, "postDate": "2018-02-12T15:13:08.317Z", "content": "Another question about the annotations, did the annotators use any algorithm to help segmentation? I find this image in particular very very bad.\nI have a high confidence that it is an algorithm who annotated this one, indeed a human would not take time to create holes within a nuclei. Am i mistaken?
", "rawMarkdown": "Another question about the annotations, did the annotators use any algorithm to help segmentation? I find this image in particular very very bad.\nI have a high confidence that it is an algorithm who annotated this one, indeed a human would not take time to create holes within a nuclei. Am i mistaken? " }, { "id": 281751, "postDate": "2018-02-13T01:52:52.473Z", "content": "Indeed, as noted in some of the other responses, we used an annotation tool which resulted in some holes (that were not visible/obvious to the humans annotating at the time).
\n\n\"yes, those speckly holes ought to be filled in. We noticed too late the annotation tool used on some of the images didn't fill in completely (ugh!)\"
", "rawMarkdown": "Indeed, as noted in some of the other responses, we used an annotation tool which resulted in some holes (that were not visible/obvious to the humans annotating at the time). \n\n\"yes, those speckly holes ought to be filled in. We noticed too late the annotation tool used on some of the images didn't fill in completely (ugh!)\"", "votes": 1 } ] }, { "id": 275728, "postDate": "2018-01-29T21:33:56.397Z", "content": "Curious: What is this image with 375 masks?\nI've done some basic EDA with stage1 train dataset and the one with most masks is a102535b0e88374bea4a1cfd9ee7cb3822ff54f4ab2a9845d428ec22f9ee2288. It looks quite different from the other. I've displayed it at the end of a notebook. It looks like labels are around a big ellipse.
", "rawMarkdown": "Curious: What is this image with 375 masks?\nI've done some basic EDA with stage1 train dataset and the one with most masks is a102535b0e88374bea4a1cfd9ee7cb3822ff54f4ab2a9845d428ec22f9ee2288. It looks quite different from the other. I've displayed it at the end of a [notebook][1]. It looks like labels are around a big ellipse.\n\n\n [1]: https://www.kaggle.com/mpware/stage1-eda-microscope-image-types-clustering", "replies": [ { "id": 276612, "postDate": "2018-01-31T22:19:29.513Z", "content": "That is a very beautiful Drosophila embryo. Its nuclei are around the edge of the embryo, which has not yet divided into multiple cells yet.
", "rawMarkdown": "That is a very beautiful Drosophila embryo. Its nuclei are around the edge of the embryo, which has not yet divided into multiple cells yet.", "votes": 2 }, { "id": 277300, "postDate": "2018-02-02T18:33:22.437Z", "content": "Thanks
", "rawMarkdown": "Thanks" } ] }, { "id": 274037, "postDate": "2018-01-25T17:57:13.370Z", "content": "Training example 0a7d30b252359a10fd298b638b90cb9ada3acced4e0c0e5a3692013f432ee4e9.\nSingle mask seems to cover two objects: masks/8246970b0d108736b3e8a1a45872737a30a23.png
", "rawMarkdown": "Training example 0a7d30b252359a10fd298b638b90cb9ada3acced4e0c0e5a3692013f432ee4e9.\nSingle mask seems to cover two objects: masks/8246970b0d108736b3e8a1a45872737a30a23.png", "replies": [ { "id": 274063, "postDate": "2018-01-25T20:16:17.683Z", "content": "Agree that should be separated into two!
", "rawMarkdown": "Agree that should be separated into two!" } ] }, { "id": 272358, "postDate": "2018-01-22T21:47:49.383Z", "content": "Is it possible to offer an identifier indicating the annotator, so that we may be able to identify potential biases?
", "rawMarkdown": "Is it possible to offer an identifier indicating the annotator, so that we may be able to identify potential biases? ", "replies": [ { "id": 272376, "postDate": "2018-01-22T23:37:43.370Z", "content": "Hello! Funny you should ask, we debated this one back and forth for quite a while. In the end we decided to withhold that information intentionally.
", "rawMarkdown": "Hello! Funny you should ask, we debated this one back and forth for quite a while. In the end we decided to withhold that information intentionally.", "votes": 1 }, { "id": 272557, "postDate": "2018-01-23T08:22:49.247Z", "content": "Great, an extra challenge. I understand, it may cause tension in your group when 'worst-annotator' threads are popping up ;-).
", "rawMarkdown": "Great, an extra challenge. I understand, it may cause tension in your group when 'worst-annotator' threads are popping up ;-)." }, { "id": 272675, "postDate": "2018-01-23T13:23:11.057Z", "content": "Ha, ha - well, it is true that I do highly value my team's happiness, but more importantly we want to ensure that methods are useful to our end goal: doing this task on a huge variety of images from the biological community out in the wild, long after this challenge is over.
\n\nWe didn't want people to expend energy fitting their solutions to a particular person's annotation style (or monitor settings) especially given that competitors will not know who annotated each image in the stage 2 test set anyway.
", "rawMarkdown": "Ha, ha - well, it is true that I do highly value my team's happiness, but more importantly we want to ensure that methods are useful to our end goal: doing this task on a huge variety of images from the biological community out in the wild, long after this challenge is over. \n\nWe didn't want people to expend energy fitting their solutions to a particular person's annotation style (or monitor settings) especially given that competitors will not know who annotated each image in the stage 2 test set anyway.", "votes": 2 }, { "id": 272682, "postDate": "2018-01-23T13:37:45.217Z", "rawMarkdown": "", "votes": 9, "isDeleted": true }, { "id": 272755, "postDate": "2018-01-23T16:32:14.643Z", "content": "We are intentionally withholding the answers to some of your questions, but I can say that the stage 2 data goes through a much more rigorous checking process than the stage 1 training and test data.
", "rawMarkdown": "We are intentionally withholding the answers to some of your questions, but I can say that the stage 2 data goes through a much more rigorous checking process than the stage 1 training and test data. ", "votes": 7 }, { "id": 272757, "postDate": "2018-01-23T16:37:27.573Z", "rawMarkdown": "", "votes": 4, "isDeleted": true } ] }, { "id": 271178, "postDate": "2018-01-19T20:54:30.427Z", "content": "Unless our job is to identify cells rather than nuclei, there are many images with incorrect nuclei masks. There are also many images with strong non-cellular signals that were included in masks. I identified several non-representative examples here.
", "rawMarkdown": "Unless our job is to identify cells rather than nuclei, there are many images with incorrect nuclei masks. There are also many images with strong non-cellular signals that were included in masks. I identified several non-representative examples [__here__](https://www.kaggle.com/c/data-science-bowl-2018/discussion/47857).", "replies": [ { "id": 271195, "postDate": "2018-01-19T21:33:48.790Z", "content": "See my reply to that discussion thread. Apart from Heng's image already mentioned above, the others are not errors according to how we biologists decided to carry out the challenge.
", "rawMarkdown": "See my reply to that discussion thread. Apart from Heng's image already mentioned above, the others are not errors according to how we biologists decided to carry out the challenge.", "votes": 1 } ] }, { "id": 270816, "postDate": "2018-01-19T02:21:38.270Z", "content": "Good catch, @Heng!
", "rawMarkdown": "Good catch, @Heng!" }, { "id": 309793, "postDate": "2018-04-06T01:56:16.790Z", "rawMarkdown": "", "isDeleted": true }, { "id": 309791, "postDate": "2018-04-06T01:55:03.630Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 273049, "author_name": "Dufan Wu", "author_url": "", "post_date": "2018-01-24T03:45:27.140000", "content": "Several possible miss-annotations:
\n\nb1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe:
\n\nmissing some of the nucleis;\n
19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a:
\n\nMost neighbor nucleis were annotated as one. Is it on purpose?\n
9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e:
\n\nA strange bar-like object at the lower right corner\n
Question about the rule to separate neighbor nucleis:
\n\ne.g. 1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca:
\n\nThe yellow and green nuclei in the middle were marked as 2 different nucleis. But note that to their left (yellow) and to their right (blue) adjacent nucleis were marked as one nuclei. I do not understand why, and I have tried adjusting display window but nothing showed up. \n
1) I agree b1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe is missing two nuclei off to the right. I'm also disturbed that there are tiny black dots, indicating holes in the nuclei which should not be there. (I cannot swear to it, but in general nuclei as we have annotated them should NOT have any holes)\n2) For 19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a I guess it depends how you generated the image you are showing on the right. You are right that in general they ought to be multi-colored whereas your result indicates the masks are improperly merged. Just not sure whether that is happening in the annotations or in your code that generates the multicolor visualization (I assume the former of course, in which case it's definitely an error). \n3) For 9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e indeed, that object is an error. I caught many of these before the training set went out but apparently not all!\n4) For 1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca: Those pairs of nuclei should indeed be split. Those are errors.
\n\nThanks for pointing these out!
", "votes": 5, "replies": [] }, { "id": 273384, "author_name": "Dufan Wu", "author_url": "", "post_date": "2018-01-24T14:51:41.743000", "content": "Hi Anne,
\n\nThanks for the detailed reply.
\n\nBlack dots in image 1: There is indeed single pixel holes in the original masks.
\n\nSecond image: we have checked the original images and confirmed that the original masks are like these.
\n\nThe last issue: this is a little problematic, because I found multiple cases with something like this (especially for 360*360 resolution images) I will try to find all cases and post them here.
", "votes": 1, "replies": [] }, { "id": 273541, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-01-24T18:35:01.957000", "content": "Thanks for confirming 1 & 2! Definitely happy to hear about errors.
", "votes": 1, "replies": [] }, { "id": 277673, "author_name": "Tim Hochberg", "author_url": "", "post_date": "2018-02-03T22:14:21.163000", "content": "The ID of the mask with the vertical bar is a09ae9559f817b104e049e5bb01cf4ad1b87e3ef7fad3dcf932b6585ca3edc05 if anyone wants to exclude just that mask.
", "votes": 4, "replies": [] }, { "id": 278050, "author_name": "Jerry Thomas", "author_url": "", "post_date": "2018-02-05T08:12:35.853000", "content": "Hi Anne,
\n\nI found 436 images where the masks have holes. One example is below. \nstage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png
\n\nThere are also 11 other masks where there is a gap between the annotated masks making them appear as two parts. One example is\nstage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png
\n\nAre these valid masks or do these need to be fixed?
", "votes": 1, "replies": [] }, { "id": 278331, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-05T23:24:02.643000", "content": "Hi there, Jerry. Thanks for noting these. Allen here had it on his list to check for tiny holes resulting from our annotation tool but hadn't gotten to it yet!
\n\nI would guess that those indeed need to be fixed to fill in the holes (as long as a hole is entirely surrounded by a single nucleus' label). Is the size is generally a single pixel? Terribly sorry this is an issue! Luckily it's an easy fix to correct the training data (and we are checking the stage 2 test set to ensure it won't be a problem there).
\n\nIt would be great if you can post the actual images in the post so I can provide guidance on the second issue; I see there is a single white mask in that image but without the corresponding image side by side I'm not sure what to make of it.
", "votes": 1, "replies": [] }, { "id": 278642, "author_name": "Jerry Thomas", "author_url": "", "post_date": "2018-02-06T16:46:58.107000", "content": "I accidentally replied twice and attached the images in separate reply. But I had not included the original image. Here are the masks and the corresponding images. I have cropped both image and mask to the mask area.
\n\nMask with holes and corresponding image.\n
Mask with a single pixel noise to the side. This looks more like an incorrect annotation, although I am not sure. This is on the bottom edge of the image\n
Mask with gap/Two nuclei annotated as one? This is on the bottom edge of the image\n
Thanks! For the first image, yes, those speckly holes ought to be filled in. We noticed too late the annotation tool used on some of the images didn't fill in completely (ugh!)
\n\nFor the second image \"Mask with a single pixel noise to the side.\" - indeed that looks like an error! The left-most masked nucleus should not be connected to the remaining pixels (which should have been background). This is a common artifact of using our tools, which we mostly caught and fixed manually, but not this one!
\n\nFor the third image those are definitely two separate nuclei... your question is why are two separate nuclei's masks present in a single slice of the annotations, right? Yeah, that shouldn't happen - that's another check we can add, to be sure that all masks are contiguous (it's hard for me to imagine how this happened with either of our annotation approaches, TBH!).
", "votes": 1, "replies": [] } ] }, { "id": 271184, "author_name": "Josan", "author_url": "", "post_date": "2018-01-19T21:08:37.690000", "content": "stage1_train/58c593bcb98386e7fd42a1d34e291db93477624b164e83ab2afa3caa90d1d921
\n\n
Looks like it misses one mask on the right bottom corner. Please confirm.
", "votes": 5, "replies": [ { "id": 271192, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-01-19T21:31:35.493000", "content": "Hello, an organizer here. I agree that nucleus ought to have been identified. Great catch!
", "votes": 6, "replies": [] }, { "id": 289116, "author_name": "Ian Oliver", "author_url": "", "post_date": "2018-02-28T23:59:33.113000", "content": "I just downloaded the training data and that nucleus is still unannotated. I take it the official training dataset isn't being updated with these corrections, and we're expected to apply them ourselves? (I'm new around here but I think I've read that that's how it works.) [Edited to add: Indeed, you've confirmed this below and apparently it's a Kaggle rule as I thought.]
\n\nA further error with the same image is that the two nuclei on the left seem to be annotated as one. At least, they're in the same mask file, which I think is supposed to be one per nucleus.
", "votes": 0, "replies": [] }, { "id": 289875, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-02T14:22:28.793000", "content": "re 2 nuclei annotated as one - Sort of depends how the person made this visualization. They are using the same color (yellow) for all nuclei so we've no idea whether the underlying annotations are distinct nuclei there or a single one.
", "votes": 0, "replies": [] } ] }, { "id": 283933, "author_name": "Spencer Aiello", "author_url": "", "post_date": "2018-02-16T08:12:48.577000", "content": "This list of 55 masks look like they should be separated
", "votes": 3, "replies": [] }, { "id": 271186, "author_name": "Josan", "author_url": "", "post_date": "2018-01-19T21:15:25.977000", "content": "Found another:
\n\nstage1_train/b1e3aeb0c56261c17eb71c747d116057b8da7e8c8a6845bdc01b2b3ee2299229
\n\n
Some of them I could barely see when zooming in, but most are just black. Please confirm.
", "votes": 3, "replies": [ { "id": 271191, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-01-19T21:30:35.790000", "content": "Hello, an organizer here. There are often very dim nuclei and this is such a case. Adjust the brightness on your monitor, or adjust the contrast of the images and you will see the nuclei that correspond to those masks.
\n\nStill, it should be noted that the annotators each had their own monitor and style so how dim something would be in order to be ignored will vary from image to image.
", "votes": 2, "replies": [] }, { "id": 271198, "author_name": "Josan", "author_url": "", "post_date": "2018-01-19T21:42:37.497000", "content": "Thanks, it clears my question. :)
", "votes": 0, "replies": [] }, { "id": 303084, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T13:17:53.900000", "content": "The really sad part is that these cases are only annotated sometimes. At many times, I'm guessing most, they weren't seen by the annotators. Fingers crossed that you guys have located all of these in the evaluation cases.
", "votes": -1, "replies": [] } ] }, { "id": 280340, "author_name": "rmldj", "author_url": "", "post_date": "2018-02-09T18:31:57.290000", "content": "I think that there are a couple of issues with image\n1bd0f2b3000b7c7723f25335fabfcdddcdf4595dd7de1b142d52bb7a186885f0.png
\n\n\nFirst look at the top - it seems that the first two nuclei should be separated, while the next one on the top edge which is marked as two should be indeed one.
Then the two yellow ones in the center should be separate but are marked as one object. \nIs it correct?
\n\nBtw. suppose that in principle we correct some labelling - what should then be the procedure for submitting code before stage 2?
", "votes": 4, "replies": [ { "id": 280401, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-09T22:26:36.790000", "content": "I agree with your assessment of this one (although maybe the top edge that is two could be one - hard to see without zooming in/adjusting contrast on my screen which I can't do at the moment!)
\n\nThat is a great question about the procedure for submitting code since indeed you can't submit a fully reproducible pipeline without also submitting annotation fixes along with it. I will see if one of the other organizers know how this is handled.
\n\nIn the meantime, thanks for pointing these out!
", "votes": 3, "replies": [] }, { "id": 282324, "author_name": "Wendy Kan", "author_url": "", "post_date": "2018-02-13T21:24:50.530000", "content": "The code submission should include some documentation that you have curated the annotation. Either include the manual curated data or the code that generate those fixes are both acceptable. As long as the code is reproducible for the final test set, you're fine.
", "votes": 1, "replies": [] }, { "id": 303219, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T19:15:29.003000", "content": "I also wonder abount the tiny spec at the centre d99c71ab55a7d103eeef63f60cf946d6318e533c893ae107919160fc08845599 - it doesn't seem any better than the one below blue at mid-bottom, but it's marked, while the other one isn't. And this, of course, is not the only case, there are tons of similar cases all along the dataset. If organizers were to state their policy on these tiny specs that is going to be enforced in the final evaluation dataset, that would be most helpful. That matters a lot, because there are tons of these specs all over images, so it's a potential question of mislabeling a ton of objects.
\n\nI would advise against them, because with such a small amount of pixels getting 50% IOU is freaking impossible - for example even in this case if the annotators got the mere fact of labeling that spec as a hit, the question whether their own mask's IOU over actual spec is at least 50% is up to debate and I'd say it's no more than 70%. It's not a problem that can be solved with precision, unless you define what that precision has to be.
", "votes": 1, "replies": [] } ] }, { "id": 276364, "author_name": "Abdelaali HassaΓ―ne", "author_url": "", "post_date": "2018-01-31T10:34:40.520000", "content": "This image should either be annotated or excluded from training set\n7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80 \n\n
Indeed!
", "votes": 2, "replies": [] } ] }, { "id": 312867, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2018-04-12T13:36:19.423000", "content": "Just out of curiosity: what is the reason for images like 0aa80, 0f48f, 0a9b6, 0f4b0, 0b2f4 and some more?
", "votes": 1, "replies": [ { "id": 314947, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2018-04-16T14:38:19.693000", "content": "If it's a film, I want to see it...
", "votes": 1, "replies": [] } ] }, { "id": 312802, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-12T11:08:52.027000", "content": "As I know I'm definitely not gonna win now, I make my revision of the training set and my annotation tool available at https://github.com/ibmua/data-science-bowl-2018-train-set . I think the training set is probably the best there is, as I went through each white-on-black sample and fixed it by hand. Annotation tool is also pretty useful, though, probably limitedly so now after the first stage is over. Maybe, for some future competitions, I donno.
\n\nP.S. uploaded this pic just because I can't upload it in a comment. Not related.
", "votes": 1, "replies": [] }, { "id": 278052, "author_name": "Jerry Thomas", "author_url": "", "post_date": "2018-02-05T08:21:54.317000", "content": "Hi Anne,
\n\nI found a few masks which had holes in them. One example is below. I am not sure if all of them follow the same pattern. \n../input/stage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png
\n\nI also found 11 others where there is a break in the mask making them appear as two items.\n../input/stage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png
\n\nAre these valid masks? You have mentioned earlier that masks are not expected to have holes in them. Is the same applicable to these. I will double check on the list of masks I found with holes.
", "votes": 1, "replies": [ { "id": 279198, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-07T15:19:34.840000", "content": "These Qs are addressed in another posting above, but this is nice because it gives the image numbers.
", "votes": 0, "replies": [] } ] }, { "id": 276985, "author_name": "kpe", "author_url": "", "post_date": "2018-02-01T23:54:43.093000", "content": "Reading the \"Evaluation Rules\"
\n\n\n\n\n.... It also checks that no two predicted masks for the same image are overlapping.
\n
however there are enough images in the training set where nucleus seems to be overlapping (for example in the 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80). Wouldn't be more meaningful (natural) in such cases to strive for the real necleus masks (i.e. even if they overlap)?
\n\nAnd a second question: assuming the test set would have similar segmentation errors like the training set, if we imagine a model able of doing a \"perfect\" segmentation, it will probably get worse score than a model making the same segmentation errors as in the test set used for the final evaluation.
\n\nI'm wondering are there any chances that this might happen (here), i.e. penalizing models doing better segmentation than the one in the test set?
", "votes": 1, "replies": [ { "id": 277278, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-02T17:29:37.937000", "content": "We certainly debated overlapping nuclei a lot. In the end we decided it would make annotation substantially harder to the point where we wouldn't be able to do the competition. We would need new tools that allow for overlap. Thankfully, for most real world applications the percentage of affected nuclei is low... but this is definitely an area for improvement.
\n\nI don't really understand the questions about whether to attempt a model that produces errors. For example, a few training images had a random line in them that obviously isn't nucleus. Whether the test set has errors or not (NB: we checked the test set more carefully than the training set), I don't really see how you'd design a model that tried to reproduce errors: like, what, the model would place random lines in random places on the image? That is certainly not going to raise your score. That's an obvious case but even for the other types of errors, it's hard for me to see how it could possibly help to do anything other than ignore or fix the incorrectly annotated training images.
", "votes": 1, "replies": [] }, { "id": 277337, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-02T20:50:00.850000", "content": "My labmate pointed out that maybe your second question was directly related to the first.
\n\nAFAIK, nuclei are not allowed to overlap when submitted as results.
\n\nPlus, the annotations used for in evaluation for both test stages (1 & 2) do not have overlap, again AFAIK. So it will be in your best interest to not produce a model that yields overlap.
", "votes": 2, "replies": [] } ] }, { "id": 275305, "author_name": "Zero", "author_url": "", "post_date": "2018-01-28T18:29:59.297000", "content": "I'm a bit confused here. Does this mean its our responsibility to monitor this thread and adjust out images set or will updated versions of the data be released?
", "votes": 1, "replies": [ { "id": 275311, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-01-28T19:18:49.353000", "content": "According to our Kaggler people-in-charge, updated versions will not be released. I agree it's a hassle but it's outside my sphere of knowledge. The number of instances with actual errors is quite low (maybe 4 or 5 listed here so far, out of hundreds of images).
", "votes": 1, "replies": [] }, { "id": 275708, "author_name": "Zero", "author_url": "", "post_date": "2018-01-29T20:38:18.310000", "content": "Ok, Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 312633, "author_name": "emergent complexity", "author_url": "", "post_date": "2018-04-12T04:21:32.550000", "content": "Not a big deal, but in the stage2 test set this image:
\n\n646f5e00a2db3add97fb80a83ef3c07edd1b17b1b0d47c2bd650cdcab9f322c0
\n\nis a 2d grayscale. All other grayscale images have a channel dimension.
", "votes": 2, "replies": [] }, { "id": 312608, "author_name": "emergent complexity", "author_url": "", "post_date": "2018-04-12T03:14:21.690000", "content": "In stage2_test, this image:
\n\n170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece
\n\nHas a .png extension but is actually a jpg file. It is markedly different than any other file in the set. Was this unintentionally added to the stage 2 set?
", "votes": 2, "replies": [ { "id": 312610, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-12T03:23:06.110000", "content": "Nice catch. Are there more jpgs or other than png format in the mix?
", "votes": 0, "replies": [] }, { "id": 312613, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-12T03:33:20.250000", "content": "Answering my own question:\n5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png\nis actually a TIFF image. Yours and this are the only exception. All others have the PNG magic number.
", "votes": 0, "replies": [] }, { "id": 312621, "author_name": "emergent complexity", "author_url": "", "post_date": "2018-04-12T03:54:29.320000", "content": "Yep, I just discovered that myself and posted it in another thread. It has a caption on it like it was taken straight out of a textbook.
", "votes": 0, "replies": [] }, { "id": 312662, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-12T05:31:12.007000", "content": "Anyway, the structures in those 2 images look more like mitochondria than nuclei ... π
", "votes": 0, "replies": [] }, { "id": 312665, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-12T05:35:20.950000", "content": "The filename that begins with ffbc47f... also has an issue when read as a PNG - please verify.
", "votes": 0, "replies": [] } ] }, { "id": 281894, "author_name": "Jerry Thomas", "author_url": "", "post_date": "2018-02-13T08:33:33.620000", "content": "While filling the holes in some of the masks I came across a new issue. Once the hole is filled the mask overlaps with another mask. It looks like the annotators were annotating two overlapping nuclei. Should these be excluded?
\n\nThe images left to right
\n\nThis one has a very bright circle and lighter background circle. Seems like they are supposed to be two nuclei. Since they are overlapping shouldn't they be excluded?
\n\n
Similar case\n
Multiple overlapping nuclei? Somehow the annotation looks odd.\n
Another overlapping nuclei annotated as separate images, one with hole that matches the other.\n
Hi Jerry - indeed those look very intentionally annotated. We did as a team agree that whenever two cells overlapped, the top-most cell would be annotated first, then we'd do the second one. I didn't personally realize that sometimes this meant the second nucleus would be a ring.
\n\nI suspect this doesn't happen often (is it only these 5 you found in the training/test?) As for how to handle it, I guess the answer is that this is indeed a valid annotation (by our agreed-upon custom, which aim for non-overlapping nuclei at all costs).
", "votes": 0, "replies": [] }, { "id": 282027, "author_name": "Kyle", "author_url": "", "post_date": "2018-02-13T13:51:41.167000", "content": "Hi Jerry, the examples you show reflect a general rule the annotators used to draw masks around the nuclei: if more than one nuclei overlap then the nuclei that appears to be the top most nuclei should be given preference during annotation.
", "votes": 0, "replies": [] }, { "id": 282070, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-02-13T15:32:25.153000", "content": "Hi, Jerry. Why would you want to exclude the overlapping objects?
\n\nThe masks (the last two images in each row) look fantastic IMO!
", "votes": 0, "replies": [] }, { "id": 282202, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-13T18:57:43.493000", "content": "Our team chatted about it Jerry, and we think that with respect to the scoring, you would want to fill holes as you've done in the second image in each set. We think this will have VERY little (possibly zero) impact because I don't think there are cases like this in stage 2 test, but I will keep an eye out for it in our final look-over.
\n\n( Allen, the last two images are the provided ground truth :D )
", "votes": 0, "replies": [] }, { "id": 282649, "author_name": "Jerry Thomas", "author_url": "", "post_date": "2018-02-14T10:28:34.163000", "content": "Thanks for the response Anne, Allen & Kyle.
\n\n@Anne I found 8 such cases. Though not all are completely similar. These are the ones where the hole in one fits another. The others are ones where a part of one overlaps the other.
\n\n@Allen: I agree that the annotations look great! I was trying to understand the annotations. I remember reading a rule that overlapping nuclei are supposed to be excluded. The way I had interpreted that statement was that only nuclei which are non overlapping are to be identified and the examples I posted above were cases where one nucleus overlaps another. Hence the question, should I exclude them.
\n\n@Anne: Given two nuclei that overlap there are two possibilities. Smaller one on to of a larger one like the ones above in which case you have clarified that even the larger one should be identified as a mask without a hole. The second possibility is of a partial overlap. In this case should both be detected as separate nuclei, one with the overlapping region included in it's mask and one without the overlapping region in it's mask? Or is the area of overlap in the mask(s) not significant as in the case of the holes?
\n\nSorry for repeating the question, I am trying to make sure I understood the rules on overlapping nuclei correctly.
", "votes": 0, "replies": [] }, { "id": 282832, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-14T17:15:44.513000", "content": "Hi Jerry, if I knew for sure I would answer your question clearly. I know how I personally annotated images, but it was a team that worked together on this so I don't want to give you concrete guidance and then find out in the test set there is something contradicting my advice!
\n\nI really appreciate you confirming this is a rare case - 8 out of the thousands we annotated, apparently. Again, I would just say that I don't believe there are any cases in the test set of nuclei that overlap in such a way as to make one enclosing the other fully - but just in case I'd advise \"you would want to fill holes as you've done in the second image in each set.\" which means ignoring the smaller enclosed one and leaving just the larger enclosing one.
", "votes": 1, "replies": [] }, { "id": 303075, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T12:51:12.403000", "content": "πΎππ ππππ πππππππ πππππ ππ πΉππππ’, πππ π π πππππ ππππ π πππ πππππππ ππ πππ πππππππ, π’ππ π ππππ π πππ ππ ππππ πππππ ππ π’ππ'ππ ππππ ππ πππ ππππππ πππππ ππ ππππ πππ. ππ πππππ ππππ π πππ ππππ π π΄ππ ππππππ (ππππππππ’ π£πππ) ππππππ πππππππ πΈ πππ'π πππππ πππππ πππ πππππ ππππ ππππ ππ πππππ πΈ ππππ, πππ πΈ π πππ ππππ ππ ππ’π πππ πππ ππ ππ πππ πππππ ππππ-ππππ.
\n\nWhat? Look at https://www.kaggle.com/c/data-science-bowl-2018#evaluation
\n\nπππ ππππππ ππππππ ππππ πππ πππππ πππ ππππππ, ππππππππ, πππ πππ πππππππ\n πππ‘ππ ππππππ πππ πππ ππππππππππ. πΈπ ππππ ππππππ ππππ ππ ππ π πππππππππ\nπππππ πππ πππ ππππ πππππ πππ πππππππππππ.
\n\nThen do you mean that in such cases we need to annotate two nucleis as one, which is completely unlike that's out there in the training data, or that we need to submit two masks as sharing the same pixels, which goes against what I've just quoted from the evaluation metrics page?
\n\nIMHO, you need to get your evaluation masks up to the standards described in the training data. Although, I must say, it's not easy to understand what those are at many times. For example, some tiny specs are marked as nuclei, while others aren't with no apparent logic. I don't think a human would be able to score better than the algorithms on this task and I'm guessing that winner's neural nets are probably going to do a better, more consistent job with annotation than what we can find in the training set.
", "votes": 0, "replies": [] }, { "id": 303147, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-25T16:07:05.150000", "content": "Hi there, we are talking here about an issue that affects 8 nuclei out of the ~20,000 that we annotated by hand. As I mentioned, the former: \"annotate two nuclei as one\" is my recommendation.
\n\nWith respect to you not understanding the logic of how the expert biologists annotated the images: that is what makes us expert biologists. You are most welcome to organize your own challenge and annotate however you like.
", "votes": 0, "replies": [] }, { "id": 303201, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T18:34:45.683000", "content": "Just read this comment
\n\nπΏπππ, πππ πππππππππππ ππππ πππ ππ ππππππππππ πππ ππππ ππππ ππππππ (π· &\nπΈ) ππ πππ ππππ πππππππ, πππππ π°π΅π°πΈπΊ. ππ ππ π πππ ππ ππ π’πππ ππππ\nππππππππ ππ πππ πππππππ π πππππ ππππ π’πππππ πππππππ.
\n\nSo I guess that's the answer.
\n\nEdit: Thanks for the reply, haven't saw it before posting the above.
", "votes": 0, "replies": [] }, { "id": 303230, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T20:05:12.203000", "content": "The major problem is not with 8 nuclei in the training set, but with any potential ones in the final evaluation set which might make a difference of, say, getting into top 5 vs getting into to 10.
", "votes": 1, "replies": [] } ] }, { "id": 278622, "author_name": "Oscar Asterkrans", "author_url": "", "post_date": "2018-02-06T15:50:44.920000", "content": "Image af576e8ec3a8d0b57eb6a311299e9e4fd2047970d3dd9d6f52e54ea6a91109da seem to miss one mask.
", "votes": 2, "replies": [ { "id": 279199, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-07T15:20:20.043000", "content": "Could you please be more specific?
", "votes": 0, "replies": [] }, { "id": 304723, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-27T23:47:04.090000", "content": "It seems to contain up to 6 nuclei, pretty sure more than the labeled two.
", "votes": 0, "replies": [] } ] }, { "id": 277952, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2018-02-04T22:30:12.063000", "content": "Am I missing masks for adc315bd40d699fd4e4effbcce81cd7162851007f485d754ad3b0472f73a86df ?\n(green = my attempts, yellow = agreement with masks, red = masks)
", "votes": 2, "replies": [ { "id": 279192, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-07T15:13:30.950000", "content": "Indeed, your green guys are correct!
", "votes": 0, "replies": [] } ] }, { "id": 273386, "author_name": "Dufan Wu", "author_url": "", "post_date": "2018-01-24T14:54:25.930000", "content": "For this band-like object (what is it?) I can see some \"nuclei-like\" things beneath it. They are not annotated in the mask. So those things are not considered as nucleis?\n
This might be a case where annotators disagree. For me, if I could see that something was a nucleus, even behind a piece of debris like this, I would annotate it. Thankfully, debris like this is not common.
", "votes": 4, "replies": [] } ] }, { "id": 271427, "author_name": "Morteza Milani", "author_url": "", "post_date": "2018-01-20T12:26:50.860000", "content": "I am wondering if train data is generated by a software or not. As you can see in the following pictures, the blue rectangles show areas with masks containing only a few pixels where on the original image, no stain is visible. However, this behavior is not consistent as you can see in the third picture that shows a stain which has no corresponding mask.
\n\nMoreover, In the second image, the red rectangles have masks but the green ones do not. To me, they all look the same and I cannot distinguish between them. Would anyone please clarify?
\n\nFirst Image:\n\nSecond Image:\n\nThird image:\n
Hello there. None of these are errors per se, as far as I can tell.
\n\nMy guess is that on the first image if you remove the black outline around the image, the nucleus stain would be more visible. It's also possible the image was cropped after annotating and those pixels would have been more obviously part of a nucleus if more nucleus was in the field of view.
\n\nRegardless, it seems that you are more interested in the general principles than specific instances, so let me elaborate on that. Although the team discussed our criteria frequently and agreed upon strategy* together, the annotators each had their own monitor and style so how dim something would be in order to be ignored will vary from image to image. In the case of the second & third images, if they appeared on the annotator's monitor the way I'm seeing it now, they should have outlined far more of those dim burry nuclei. Hope this helps!
\n\nThanks for the explanation. It makes sense for few-pixel nucleus that the images are cropped after the annotation.
\n\nStill, I think there are some errors, if error is excluding a nuclei in the masks. I understand that every monitor is slightly different and annotators could miss some blurry nucleus. Taking image 2, the red rectangles show nucleus that are as blurry as the one just right to the 200 tick. But this one is not annotated and it cannot be attributed to differences in monitors assuming a slide is annotated on a single monitor. Even the red and green rectangles show nucleus with the same intensity and they are to be missed all together.
", "votes": 1, "replies": [] }, { "id": 272208, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-01-22T13:52:38.747000", "content": "Personally, I agree that those nuclei should have been annotated, given the images how I see them on my monitor now. I just want to be clear that individual annotators and their monitors will vary.
", "votes": 2, "replies": [] }, { "id": 313221, "author_name": "Snorkle", "author_url": "", "post_date": "2018-04-13T00:18:27.313000", "content": "After scaling the image RGB values to (0..1), I displayed np.sqrt(img) alongside the original image (to reduce the dynamic range). The dark images all come up beautifully clearly - exposing lots of invisible nucleii (and various interesting background artifacts).
", "votes": 0, "replies": [] }, { "id": 313230, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-13T00:52:29.663000", "content": "That's interesting! Did you use this in your solution?
", "votes": 0, "replies": [] }, { "id": 313234, "author_name": "Snorkle", "author_url": "", "post_date": "2018-04-13T01:13:27.943000", "content": "For my stage1 submission, I used sqrt on the monochrome images and left the color ones alone.
\n\nTo be honest, I have no idea whether it improved matters - I wrote the final bits of code shortly before the stage1 deadline, so didn't have any time to play with it. This is also my first ever (non-toy non-MATLAB) neural net, so any opinions I could offer would probably be laughably inaccurate :)
", "votes": 0, "replies": [] }, { "id": 313236, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-13T01:23:08.560000", "content": "Original, sqrt and log in different channels.\n
I don't think your idea was stupid. Actually, probably pretty great. Might be one of the best ones this contest. But needs testing.
", "votes": 0, "replies": [] }, { "id": 313247, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-13T01:49:47.433000", "content": "As, you see, weights are linear, even after ReLU. So we're helping the net by making some adjjustments for it with convexity. It usually hurts perf when you put convex activation functions deep into the net - networks easily overfit as the actual data and logic, it seems, is far more linear and specific non-linearities can be learned via combining different linear stuff and ReLUs, or other simple more linear activations, MaxOut, is a notable example. But here we're talking about stuff we know about our data, it's lighting conditions and inherent specific stuff about it, it's not deep inside the net, so I don't think this will trigger any overfitting. May depend on an amount of training data, though.
", "votes": 0, "replies": [] }, { "id": 313248, "author_name": "Snorkle", "author_url": "", "post_date": "2018-04-13T01:54:55.997000", "content": "Thanks :)
", "votes": 0, "replies": [] }, { "id": 313255, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-13T02:16:54.433000", "content": "May also take some skills to make it work. I'll report my findings. Thus far seems promising after one nice little trick I love in the nets. My favourite one. ;) But I only started the learning process. Got to pass more iterations first.
", "votes": 0, "replies": [] } ] }, { "id": 271020, "author_name": "hengck23", "author_url": "", "post_date": "2018-01-19T13:45:32.097000", "content": "stage1_train/12aeefb1b522b283819b12e4cfaf6b13c1264c0aadac3412b4edd2ace304cb40
", "votes": 2, "replies": [ { "id": 271193, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-01-19T21:32:44.973000", "content": "Hello, organizer here. I agree that image was badly annotated.
", "votes": 0, "replies": [] } ] }, { "id": 270666, "author_name": "Josan", "author_url": "", "post_date": "2018-01-18T17:48:56.497000", "content": "Just re-post @heng's finding, the error train image:\n Stage1_train/7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80
\n\nRef: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47739
", "votes": 2, "replies": [] }, { "id": 313622, "author_name": "Robert_luo", "author_url": "", "post_date": "2018-04-13T15:27:17.700000", "content": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?
", "votes": 0, "replies": [] }, { "id": 312536, "author_name": "aglotero", "author_url": "", "post_date": "2018-04-12T00:38:49.463000", "content": "Is the stage2_test/15cdf102276bc0edc7491e15e6960e74f7bb47e1f0d7685cb7cbdc3f914c56c5 image all black ?
", "votes": 0, "replies": [] }, { "id": 309795, "author_name": "featureselection", "author_url": "", "post_date": "2018-04-06T02:01:03.740000", "content": "Left is contrast enhanced image # 'f8e74d4006dd68c1dbe68df7be905835e00d8ba4916f3b18884509a15fdc0b55'
\n\nRight is the image labeled using the mask from the training set
\n\nAre these small objects really nuclei?\n
Yes, those are what are called \"micronuclei\" and we decided to include them in the challenge (in part because if a model finds them all properly, then a biologist can filter them downstream simply based on size)
", "votes": 0, "replies": [] } ] }, { "id": 306822, "author_name": "UtpalDatta", "author_url": "", "post_date": "2018-03-31T04:44:23.937000", "content": "Hi,\n Could you please look into the mask of the image 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80? I think the mask files are missing, only one mask.....
", "votes": 0, "replies": [] }, { "id": 305148, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-28T14:30:06.010000", "content": "In d256b32adda37f2301c9e46f34b7f9a36cce273256369ceb5dc2c73c3007e3c4 are you certain 0a13899f3cd5597920b9cdb078c90a48f9bd63ca4a2d413414bf83bc647ea126.png is not the same nucleus as as the one marked right below it - b4cd6644b0975c0603e3023119614653c7e379d6bec71e267fa7f07bc93830bc.png ?
", "votes": 0, "replies": [] }, { "id": 304753, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-28T01:13:43.017000", "content": "As has been reported in another thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/53134), 20b20ab049372d184c705acebe7af026d3580f5fd5a72ed796e3622e1685af2f and cb4df20a83b2f38b394c67f1d9d4aef29f9794d5345da3576318374ec3a11490 are the same image in two contrast/lighting conditions. Annotated with a vastly different amount of masks. If organizers would at least tell which annotations to trust, that would be most helpful.
", "votes": 0, "replies": [ { "id": 308053, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-02T21:46:47.840000", "content": "Hi there, see that thread for a reply (spoiler alert: probably an unsatisfying reply!).
", "votes": 0, "replies": [] } ] }, { "id": 304010, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-26T22:04:37.427000", "content": "243443ae303cc09cfbea85bfd22b0c4f026342f3dfc3aa1076f27867910d025b seems like there are 3 nucleuses that are merged, although they could be annotated as separate. The ones that are of avarage brightness and the ones that shine very brightly on this picture. Or should they be/are they a single instance?
", "votes": 0, "replies": [ { "id": 304320, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-27T11:05:49.753000", "content": "Could you be more specific about which nuclei? Thanks!
", "votes": 0, "replies": [] }, { "id": 304744, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-28T00:59:58.193000", "content": "7ca7532afd7d57016562238823cb26abd955c2085ec4d1c343af416555be561a and the one above that. I don't have any tools to easily find the mask name of the one above.
", "votes": 0, "replies": [] } ] }, { "id": 303232, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T20:12:45.153000", "content": "Michael Kazachok's kernel https://www.kaggle.com/miklgr500/data-analysis raises a point that the test set #1 doesn't contain images the likes of 8d05fb18ee0cda107d56735cafa6197a31884e0a5092dc6d41760fb92ae23ab4. Am I correct to assume that the final test set is going to contain those types of images?
", "votes": 0, "replies": [ { "id": 304322, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-27T11:11:16.130000", "content": "I cannot answer that (for fear of giving too much away, and also I don't recall, and also whether an image is of the same \"type\" or not is a bit subjective).
\n\nBut what I can say is that this challenge is all about generalization, so what is public knowledge is the fact that the stage 2 test set contains image sets/experiments that are mutually exclusive from training set and stage 1 test set. Do expect variety, and unseen image types.
", "votes": 1, "replies": [] }, { "id": 304323, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-27T11:12:00.887000", "content": "Take a look here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#290093
", "votes": 1, "replies": [] } ] }, { "id": 295901, "author_name": "Florian Kromp", "author_url": "", "post_date": "2018-03-14T12:00:45.547000", "content": "Dear all, i had a look at the trainingset and at the discussion here and would like to ask: the trainingset contains a lot of errors, it seems as if some semi-automated tool was used to create the annotation masks. What i understood from the discussion, the errors won't be corrected within the trainingset of stage 1, we can do it by ourselves if we like to, but for the stage2, there won't be those errors true? What about the evaluation of stage 1 - if we come up with a solution able to distinguish most of the touching nuclei, but the testset contains many errors, our evaluation score will be low although we propose good results, true?
", "votes": 0, "replies": [ { "id": 303212, "author_name": "theNickName", "author_url": "", "post_date": "2018-03-25T18:56:59.563000", "content": "As far as organizers are saying, thay did a much better job annotating the evaluation set.
", "votes": 1, "replies": [] } ] }, { "id": 293441, "author_name": "", "author_url": "", "post_date": "2018-03-09T20:51:07.287000", "content": "", "votes": 0, "replies": [] }, { "id": 288978, "author_name": "", "author_url": "", "post_date": "2018-02-28T17:14:10.177000", "content": "", "votes": 0, "replies": [] }, { "id": 283810, "author_name": "", "author_url": "", "post_date": "2018-02-16T02:01:11.460000", "content": "", "votes": 0, "replies": [ { "id": 284181, "author_name": "", "author_url": "", "post_date": "2018-02-16T17:55:35.573000", "content": "", "votes": 0, "replies": [] }, { "id": 284204, "author_name": "", "author_url": "", "post_date": "2018-02-16T18:41:56.343000", "content": "", "votes": 1, "replies": [] }, { "id": 285637, "author_name": "", "author_url": "", "post_date": "2018-02-20T19:12:33.970000", "content": "", "votes": 4, "replies": [] }, { "id": 303308, "author_name": "", "author_url": "", "post_date": "2018-03-26T01:22:14.187000", "content": "", "votes": 0, "replies": [] }, { "id": 303310, "author_name": "", "author_url": "", "post_date": "2018-03-26T01:25:34.793000", "content": "", "votes": 0, "replies": [] }, { "id": 303814, "author_name": "", "author_url": "", "post_date": "2018-03-26T17:29:45.197000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 277533, "author_name": "", "author_url": "", "post_date": "2018-02-03T13:37:15.253000", "content": "", "votes": 0, "replies": [ { "id": 277570, "author_name": "", "author_url": "", "post_date": "2018-02-03T15:37:05.207000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 276063, "author_name": "", "author_url": "", "post_date": "2018-01-30T16:27:47.033000", "content": "", "votes": 0, "replies": [ { "id": 276611, "author_name": "", "author_url": "", "post_date": "2018-01-31T22:18:30.767000", "content": "", "votes": 1, "replies": [] }, { "id": 281338, "author_name": "", "author_url": "", "post_date": "2018-02-12T15:13:08.317000", "content": "", "votes": 0, "replies": [] }, { "id": 281751, "author_name": "", "author_url": "", "post_date": "2018-02-13T01:52:52.473000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 275728, "author_name": "", "author_url": "", "post_date": "2018-01-29T21:33:56.397000", "content": "", "votes": 0, "replies": [ { "id": 276612, "author_name": "", "author_url": "", "post_date": "2018-01-31T22:19:29.513000", "content": "", "votes": 2, "replies": [] }, { "id": 277300, "author_name": "", "author_url": "", "post_date": "2018-02-02T18:33:22.437000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 274037, "author_name": "", "author_url": "", "post_date": "2018-01-25T17:57:13.370000", "content": "", "votes": 0, "replies": [ { "id": 274063, "author_name": "", "author_url": "", "post_date": "2018-01-25T20:16:17.683000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 272358, "author_name": "", "author_url": "", "post_date": "2018-01-22T21:47:49.383000", "content": "", "votes": 0, "replies": [ { "id": 272376, "author_name": "", "author_url": "", "post_date": "2018-01-22T23:37:43.370000", "content": "", "votes": 1, "replies": [] }, { "id": 272557, "author_name": "", "author_url": "", "post_date": "2018-01-23T08:22:49.247000", "content": "", "votes": 0, "replies": [] }, { "id": 272675, "author_name": "", "author_url": "", "post_date": "2018-01-23T13:23:11.057000", "content": "", "votes": 2, "replies": [] }, { "id": 272682, "author_name": "", "author_url": "", "post_date": "2018-01-23T13:37:45.217000", "content": "", "votes": 9, "replies": [] }, { "id": 272755, "author_name": "", "author_url": "", "post_date": "2018-01-23T16:32:14.643000", "content": "", "votes": 7, "replies": [] }, { "id": 272757, "author_name": "", "author_url": "", "post_date": "2018-01-23T16:37:27.573000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 271178, "author_name": "", "author_url": "", "post_date": "2018-01-19T20:54:30.427000", "content": "", "votes": 0, "replies": [ { "id": 271195, "author_name": "", "author_url": "", "post_date": "2018-01-19T21:33:48.790000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 270816, "author_name": "", "author_url": "", "post_date": "2018-01-19T02:21:38.270000", "content": "", "votes": 0, "replies": [] }, { "id": 309793, "author_name": "", "author_url": "", "post_date": "2018-04-06T01:56:16.790000", "content": "", "votes": 0, "replies": [] }, { "id": 309791, "author_name": "", "author_url": "", "post_date": "2018-04-06T01:55:03.630000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "270642": "The hosts had a nice idea to create a consolidated thread where everyone can share minor data quality issues and other miscellaneous \"gotchas\" that you hit when getting started on this problem. Feel free to sound off below if you find something and want to share it with the group. We suggest you show each potentially problematic image as a separate post unless they are highly related.", "273049": "**Several possible miss-annotations:**\n\nb1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe: \n\nmissing some of the nucleis;\n![][1]\n\n19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a:\n\nMost neighbor nucleis were annotated as one. Is it on purpose?\n![][2]\n\n9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e:\n\nA strange bar-like object at the lower right corner\n![][3]\n\n**Question about the rule to separate neighbor nucleis:**\n\ne.g. 1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca:\n\nThe yellow and green nuclei in the middle were marked as 2 different nucleis. But note that to their left (yellow) and to their right (blue) adjacent nucleis were marked as one nuclei. I do not understand why, and I have tried adjusting display window but nothing showed up. \n![][4]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/273049/8353/b1eb0123fe2d8c825694b193efb7b923d95effac9558ee4eaf3116374c2c94fe.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/273049/8354/19f0653c33982a416feed56e5d1ce6849fd83314fd19dfa1c5b23c6b66e9868a.png\n [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/273049/8355/9bb6e39d5f4415bc7554842ee5d1280403a602f2ba56122b87f453a62d37c06e.png\n [4]: https://kaggle2.blob.core.windows.net/forum-message-attachments/273049/8356/1f0008060150b5b93084ae2e4dabd160ab80a95ce8071a321b80ec4e33b58aca.png", "271184": "stage1_train/58c593bcb98386e7fd42a1d34e291db93477624b164e83ab2afa3caa90d1d921\n\n![img][1]\n\nLooks like it misses one mask on the right bottom corner. Please confirm.\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/271184/8317/58c593.png", "283933": "This list of 55 masks look like they should be separated", "271186": "Found another:\n\nstage1_train/b1e3aeb0c56261c17eb71c747d116057b8da7e8c8a6845bdc01b2b3ee2299229\n\n![img][1]\n\nSome of them I could barely see when zooming in, but most are just black. Please confirm.\n\n\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/271186/8318/b1e3ae.png", "280340": "I think that there are a couple of issues with image\n1bd0f2b3000b7c7723f25335fabfcdddcdf4595dd7de1b142d52bb7a186885f0.png\n\n![image][1]\nFirst look at the top - it seems that the first two nuclei should be separated, while the next one on the top edge which is marked as two should be indeed one.\n\nThen the two yellow ones in the center should be separate but are marked as one object. \nIs it correct?\n\nBtw. suppose that in principle we correct some labelling - what should then be the procedure for submitting code before stage 2?\n\n [1]: http://th.if.uj.edu.pl/~romuald/1bd0f.png", "276364": "This image should either be annotated or excluded from training set\n7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80 \n![Image][1]\n![Annotation][2]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/276364/8448/7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/276364/8449/5ffe1e4987f987f45f716bbeb749f378bb76ab235be18b5fc4d7468523a8e005.png", "312867": "Just out of curiosity: what is the reason for images like 0aa80, 0f48f, 0a9b6, 0f4b0, 0b2f4 and some more?", "312802": "As I know I'm definitely not gonna win now, I make my revision of the training set and my annotation tool available at https://github.com/ibmua/data-science-bowl-2018-train-set . I think the training set is probably the best there is, as I went through each white-on-black sample and fixed it by hand. Annotation tool is also pretty useful, though, probably limitedly so now after the first stage is over. Maybe, for some future competitions, I donno.\n\nP.S. uploaded this pic just because I can't upload it in a comment. Not related.", "278052": "Hi Anne,\n\nI found a few masks which had holes in them. One example is below. I am not sure if all of them follow the same pattern. \n../input/stage1_train/5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974/masks/5e6e650a28e22f651817b2edeacbf93a960adf633f1dbef69ecea585ef35d544.png\n\nI also found 11 others where there is a break in the mask making them appear as two items.\n../input/stage1_train/55f98f43c152aa0dc8bea513f8ba558cc57494b81ae4ee816977816e79629c50/masks/e5b2747c30db016c8318c1df1391708b85c290d8a80d62e013b14ceb759c998e.png\n\nAre these valid masks? You have mentioned earlier that masks are not expected to have holes in them. Is the same applicable to these. I will double check on the list of masks I found with holes.", "276985": "Reading the \"Evaluation Rules\" \n\n> .... It also checks that no two predicted masks for the same image are overlapping.\n\nhowever there are enough images in the training set where nucleus seems to be overlapping (for example in the 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80). Wouldn't be more meaningful (natural) in such cases to strive for the real necleus masks (i.e. even if they overlap)?\n\nAnd a second question: assuming the test set would have similar segmentation errors like the training set, if we imagine a model able of doing a \"perfect\" segmentation, it will probably get worse score than a model making the same segmentation errors as in the test set used for the final evaluation. \n\nI'm wondering are there any chances that this might happen (here), i.e. penalizing models doing better segmentation than the one in the test set? ", "275305": "I'm a bit confused here. Does this mean its our responsibility to monitor this thread and adjust out images set or will updated versions of the data be released?", "312633": "Not a big deal, but in the stage2 test set this image:\n\n646f5e00a2db3add97fb80a83ef3c07edd1b17b1b0d47c2bd650cdcab9f322c0\n\nis a 2d grayscale. All other grayscale images have a channel dimension.", "312608": "In stage2_test, this image:\n\n170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece\n\nHas a .png extension but is actually a jpg file. It is markedly different than any other file in the set. Was this unintentionally added to the stage 2 set?", "281894": "While filling the holes in some of the masks I came across a new issue. Once the hole is filled the mask overlaps with another mask. It looks like the annotators were annotating two overlapping nuclei. Should these be excluded?\n\nThe images left to right\n\n - Original image\n - Mask with hole filled\n - 2 original masks that overlap if the holes are filled.\n\nThis one has a very bright circle and lighter background circle. Seems like they are supposed to be two nuclei. Since they are overlapping shouldn't they be excluded?\n\n![enter image description here][1]\n\nSimilar case\n![enter image description here][2]\n\nMultiple overlapping nuclei? Somehow the annotation looks odd.\n![enter image description here][3]\n\nAnother overlapping nuclei annotated as separate images, one with hole that matches the other.\n![enter image description here][4]\n\n![enter image description here][5]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/281894/8557/df53.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/281894/8556/8efe.png\n [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/281894/8558/da31.png\n [4]: https://kaggle2.blob.core.windows.net/forum-message-attachments/281894/8559/0827.png\n [5]: https://kaggle2.blob.core.windows.net/forum-message-attachments/281894/8560/9bb6.png", "278622": "Image af576e8ec3a8d0b57eb6a311299e9e4fd2047970d3dd9d6f52e54ea6a91109da seem to miss one mask.", "277952": "Am I missing masks for adc315bd40d699fd4e4effbcce81cd7162851007f485d754ad3b0472f73a86df ?\n(green = my attempts, yellow = agreement with masks, red = masks)", "273386": "For this band-like object (what is it?) I can see some \"nuclei-like\" things beneath it. They are not annotated in the mask. So those things are not considered as nucleis?\n![][1]\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/273386/8365/193ffaa5272d5c421ae02130a64d98ad120ec70e4ed97a72cdcd4801ce93b066.png", "271427": "I am wondering if train data is generated by a software or not. As you can see in the following pictures, the blue rectangles show areas with masks containing only a few pixels where on the original image, no stain is visible. However, this behavior is not consistent as you can see in the third picture that shows a stain which has no corresponding mask.\n\nMoreover, In the second image, the red rectangles have masks but the green ones do not. To me, they all look the same and I cannot distinguish between them. Would anyone please clarify? \n\nFirst Image:\n![enter image description here][1]\nSecond Image:\n![enter image description here][2]\nThird image:\n![enter image description here][3]\n\n\n [1]: https://s14.postimg.org/5x24bdrsh/5479.png\n [2]: https://s14.postimg.org/3shra0fsh/74a77.png\n [3]: https://s14.postimg.org/8sf7iadsx/a7f767.png", "271020": "stage1_train/12aeefb1b522b283819b12e4cfaf6b13c1264c0aadac3412b4edd2ace304cb40", "270666": "Just re-post @heng's finding, the error train image:\n Stage1_train/7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80\n\nRef: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47739", "313622": "if one image, there is no cell, how to submit results? whar's the csv style when a image has on instance?", "312536": "Is the stage2_test/15cdf102276bc0edc7491e15e6960e74f7bb47e1f0d7685cb7cbdc3f914c56c5 image all black ?", "309795": "Left is contrast enhanced image # 'f8e74d4006dd68c1dbe68df7be905835e00d8ba4916f3b18884509a15fdc0b55'\n\nRight is the image labeled using the mask from the training set\n\nAre these small objects really nuclei?\n![bad annotation?][1]\n [1]: https://i.imgur.com/X7rdSRC.png", "306822": "Hi,\n Could you please look into the mask of the image 7b38c9173ebe69b4c6ba7e703c0c27f39305d9b2910f46405993d2ea7a963b80? I think the mask files are missing, only one mask.....", "305148": "In d256b32adda37f2301c9e46f34b7f9a36cce273256369ceb5dc2c73c3007e3c4 are you certain 0a13899f3cd5597920b9cdb078c90a48f9bd63ca4a2d413414bf83bc647ea126.png is not the same nucleus as as the one marked right below it - b4cd6644b0975c0603e3023119614653c7e379d6bec71e267fa7f07bc93830bc.png ?", "304753": "As has been reported in another thread (https://www.kaggle.com/c/data-science-bowl-2018/discussion/53134), 20b20ab049372d184c705acebe7af026d3580f5fd5a72ed796e3622e1685af2f and cb4df20a83b2f38b394c67f1d9d4aef29f9794d5345da3576318374ec3a11490 are the same image in two contrast/lighting conditions. Annotated with a vastly different amount of masks. If organizers would at least tell which annotations to trust, that would be most helpful.", "304010": "243443ae303cc09cfbea85bfd22b0c4f026342f3dfc3aa1076f27867910d025b seems like there are 3 nucleuses that are merged, although they could be annotated as separate. The ones that are of avarage brightness and the ones that shine very brightly on this picture. Or should they be/are they a single instance?", "303232": "Michael Kazachok's kernel https://www.kaggle.com/miklgr500/data-analysis raises a point that the test set #1 doesn't contain images the likes of 8d05fb18ee0cda107d56735cafa6197a31884e0a5092dc6d41760fb92ae23ab4. Am I correct to assume that the final test set is going to contain those types of images?", "295901": "Dear all, i had a look at the trainingset and at the discussion here and would like to ask: the trainingset contains a lot of errors, it seems as if some semi-automated tool was used to create the annotation masks. What i understood from the discussion, the errors won't be corrected within the trainingset of stage 1, we can do it by ourselves if we like to, but for the stage2, there won't be those errors true? What about the evaluation of stage 1 - if we come up with a solution able to distinguish most of the touching nuclei, but the testset contains many errors, our evaluation score will be low although we propose good results, true?", "293441": "stage1_train/9facc652efe19f634639585d692a53dd6c2a8e2f0c9baebdfd85b9b41ec58851/masks/9551f2dad01c71f81d5e2386a99bc963457338d0c79fda0e90e9226f50b5dada.png\n\nzero width annotation. can just be ignored i guess (posted in case someone is collecting a error less dataset)", "288978": "0d2bf916cc8de90d02f4cd4c23ea79b227dbc45d845b4124ffea380c92d34c8c -> 1c344631e35e9e282cf3cab642cafadff0745944a81b0715ab0a13944b18cfb9 merged nuclei in mask.", "283810": "Have any of the suggested errors been fixed in the training dataset that is available for download ? I am just wondering if I should be downloading the training dataset again or manually fixing these issues on my local copy", "277533": "apoptotic stuff : \nstage 1 test\nd616d323a9eeb9da1b66f8d5df671d63c092c9919cb2c0b223e29c63257c944d \n\nnuclei outside imaging plane:\nstage 1 test da6c593410340b19bb212b9f6d274f95b08c0fc8f2570cd66bc5ed42c560acab\n550450e4bff4036fd671decdc5d42fec23578198d6a2fd79179c4368b9d6da18 \n\nmetaphase/mitotic \nstage 1 test 0999dab07b11bc85fb8464fc36c947fbd8b5d6ec49817361cb780659ca805eac\n912a679e4b9b1d1a75170254fd675b8c24b664d80ad7ea7e460241a23535a406\n8b59819fbc92eefe45b1db95c0cc3a467ddcfc755684c7f2ba2f6ccb9ad740ab\n\nczy liczyc jadra komorkowe znajdujace sie pod tym wloknem?\ndo we count nuclei behind this fiber?\nstage 1 test 336d3e4105766f8ad328a7ee9571e743f376f8cbcf6a969ca7e353fe3235c523\n\nfunny stuff of different kind (don't know what is shining there)\n52b267e20519174e3ce1e1994b5d677804b16bc670aa5f6ffb6344a0fdf63fde\n\noversaturated fluorsescent image (pixel grey value over 255, like in overexposed photo)\n43a71aeb641faa18742cb826772a8566c6c947d7050f9ab15459de6cc2b3b6af\n\nI understand this is a tissue fragment, but image quality is really low...\n4f949bd8d914bbfa06f40d6a0e2b5b75c38bf53dbcbafc48c97f105bee4f8fac", "276063": "I need some clarification regarding to 0121d6759c5adb290c8e828fc882f37dfaf3663ec885c663859948c154a443ed \nOn the middle image I marked couple of interesting points. \n\n![Img][1]\n\nLet's start from the Red circle. The shape of masks looks like two thick lines, but original image clearly shows one roundish shape. \n\nYellow mask have a hole inside, is that and error or expected mask?\n\nGreen mask have 4 (one outside) 'stains' but reference image has different shape (one or two?, hard to say)\n\nBlue circle similarly to green have few 'stains' instead of one. \n\n\nThere are also masks that looks like a square, it seems to be not really realistic. Could you give me some comment about it? Are you going to update dataset regarding the previously found mistakes in masks? \n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/276063/8440/0121d6759c5adb290c8e828fc882f37dfaf3663ec885c663859948c154a443ed.jpg", "275728": "Curious: What is this image with 375 masks?\nI've done some basic EDA with stage1 train dataset and the one with most masks is a102535b0e88374bea4a1cfd9ee7cb3822ff54f4ab2a9845d428ec22f9ee2288. It looks quite different from the other. I've displayed it at the end of a [notebook][1]. It looks like labels are around a big ellipse.\n\n\n [1]: https://www.kaggle.com/mpware/stage1-eda-microscope-image-types-clustering", "274037": "Training example 0a7d30b252359a10fd298b638b90cb9ada3acced4e0c0e5a3692013f432ee4e9.\nSingle mask seems to cover two objects: masks/8246970b0d108736b3e8a1a45872737a30a23.png", "272358": "Is it possible to offer an identifier indicating the annotator, so that we may be able to identify potential biases? ", "271178": "Unless our job is to identify cells rather than nuclei, there are many images with incorrect nuclei masks. There are also many images with strong non-cellular signals that were included in masks. I identified several non-representative examples [__here__](https://www.kaggle.com/c/data-science-bowl-2018/discussion/47857).", "270816": "Good catch, @Heng!", "309793": "", "309791": "" } }