The question has come up in various threads, so I will answer it here because I think it will affect the real-world utility of solutions you create.
\n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged are generally from different experiments or cell types than those included in the stage 1 training and test data provided to you. In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images.
\n\nSo generalizability is a key goal for your solution. I honestly don't recall at this point which image sets went into training vs stage 1 test vs stage 2 test but it was not systematic or intended to trick you (i.e., our goal is not to have the image types in stage 2 test be crazy different than training or stage 1 test, though I imagine the definition of \"crazy different\" might vary from person to person). Changes in stain colors are quite common in biology, for example, so you should expect images with different color schemes.
", "messages": [ { "id": 272874, "postDate": "2018-01-23T22:23:13.657Z", "content": "The question has come up in various threads, so I will answer it here because I think it will affect the real-world utility of solutions you create.
\n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged are generally from different experiments or cell types than those included in the stage 1 training and test data provided to you. In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images.
\n\nSo generalizability is a key goal for your solution. I honestly don't recall at this point which image sets went into training vs stage 1 test vs stage 2 test but it was not systematic or intended to trick you (i.e., our goal is not to have the image types in stage 2 test be crazy different than training or stage 1 test, though I imagine the definition of \"crazy different\" might vary from person to person). Changes in stain colors are quite common in biology, for example, so you should expect images with different color schemes.
", "rawMarkdown": "The question has come up in various threads, so I will answer it here because I think it will affect the real-world utility of solutions you create. \n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged are generally from different experiments or cell types than those included in the stage 1 training and test data provided to you. **In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images.** \n \nSo generalizability is a key goal for your solution. I honestly don't recall at this point which image sets went into training vs stage 1 test vs stage 2 test but it was not systematic or intended to trick you (i.e., our goal is not to have the image types in stage 2 test be crazy different than training or stage 1 test, though I imagine the definition of \"crazy different\" might vary from person to person). Changes in stain colors are quite common in biology, for example, so you should expect images with different color schemes.", "votes": 26 }, { "id": 284577, "postDate": "2018-02-17T16:25:10.100Z", "content": "What you call generalizability must be called transferrabilty, b/c you expect to transfer a model trained on some type of data to different data. Generalizability is ability to perform on the same data distribution. i.i.d. assumption about training/test data is the cornerstone of ML, and automatic transferrabilty you are anticipating does not exist. By automatic I mean no additional retraining or other supervised procedure to adjust distributions. The results will be random to degree of distribution difference.
", "rawMarkdown": "What you call generalizability must be called transferrabilty, b/c you expect to transfer a model trained on some type of data to different data. Generalizability is ability to perform on the same data distribution. i.i.d. assumption about training/test data is the cornerstone of ML, and *automatic* transferrabilty you are anticipating does not exist. By automatic I mean no additional retraining or other supervised procedure to adjust distributions. The results will be random to degree of distribution difference.", "votes": 11, "replies": [ { "id": 285640, "postDate": "2018-02-20T19:16:14.473Z", "content": "Well, it's an interesting question of semantics that you raise here.
\n\nSay I train a model on 100 different data types and then I test it on 100 additional data types, distinct from the first set. Would you call that generalizability or transferability? I can see arguments both ways.
\n\nNote that in our competition, we only have a handful of image types in each stage rather than 100.
", "rawMarkdown": "Well, it's an interesting question of semantics that you raise here. \n\nSay I train a model on 100 different data types and then I test it on 100 additional data types, distinct from the first set. Would you call that generalizability or transferability? I can see arguments both ways.\n\nNote that in our competition, we only have a handful of image types in each stage rather than 100. ", "votes": -1 }, { "id": 285704, "postDate": "2018-02-20T21:39:32.973Z", "content": "It's not about semantics. One can not adequately test a [statistically trained] model on another and unseen data distribution. No miracles.
", "rawMarkdown": "It's not about semantics. One can not adequately test a [statistically trained] model on another and unseen data distribution. No miracles.", "votes": 3 }, { "id": 285992, "postDate": "2018-02-21T12:47:40.680Z", "content": "I disagree. The semantic question is what data distributions you consider \"similar\" vs \"not similar\". We seem to agree that: \nsimilar data distributions -> generalizability\nnot similar data distributions -> transferability
\n\nI would argue that two collections wherein each contains multiple image types are \"similar\" and thus this is a problem of generalizability.
\n\nI'm not sure why you are talking about miracles and adequate statistical testing. I thought we were talking about which term to use, not expected results or appropriate statistical testing.
", "rawMarkdown": "I disagree. The semantic question is what data distributions you consider \"similar\" vs \"not similar\". We seem to agree that: \nsimilar data distributions -> generalizability\nnot similar data distributions -> transferability\n\nI would argue that two collections wherein each contains multiple image types are \"similar\" and thus this is a problem of generalizability.\n\nI'm not sure why you are talking about miracles and adequate statistical testing. I thought we were talking about which term to use, not expected results or appropriate statistical testing.", "votes": -2 }, { "id": 286018, "postDate": "2018-02-21T13:55:25.197Z", "content": "Here is your quote I was referring to, highlighting is mine:
\n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged are generally from different experiments or cell types than those included in the stage 1 training and test data provided to you. In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images.
\n\nI read this as the test data comes from different distribution and is dissimilar to training data. How many image types are contained in training/test sets is irrelevant from this point of view. Multiple image types is still okay for generalization, provided they are equally represented in both sets.
", "rawMarkdown": "Here is your quote I was referring to, highlighting is mine:\n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged **are generally from different experiments or cell types than those included in the stage 1 training and test data** provided to you. In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images. \n\nI read this as the test data comes from different distribution and is dissimilar to training data. How many image types are contained in training/test sets is irrelevant from this point of view. Multiple image types is still okay for generalization, provided they are equally represented in both sets.", "votes": 1 }, { "id": 286602, "postDate": "2018-02-22T13:15:02.283Z", "content": "I guess we will not agree on the terminology, then. I think this style of training set is quite unusual so most people are not used to thinking of training/test sets where we have a variety of image types in both but no cross-representation. Our goal is to prevent Voodoo machine learning where methods do great during testing but fail to generalize: https://www.biorxiv.org/content/early/2016/06/19/059774
", "rawMarkdown": "I guess we will not agree on the terminology, then. I think this style of training set is quite unusual so most people are not used to thinking of training/test sets where we have a variety of image types in both but no cross-representation. Our goal is to prevent Voodoo machine learning where methods do great during testing but fail to generalize: https://www.biorxiv.org/content/early/2016/06/19/059774" }, { "id": 286659, "postDate": "2018-02-22T15:49:43.153Z", "content": "Failure to transfer ML models inter subjects has been known and addressed for years. Look at any Kaggle EEG competition for instance. What the authors of this 2016 paper seemingly don't understand is that it is expected and follows from stats learning theory.
", "rawMarkdown": "Failure to transfer ML models inter subjects has been known and addressed for years. Look at any Kaggle EEG competition for instance. What the authors of this 2016 paper seemingly don't understand is that it is expected and follows from stats learning theory.", "votes": 1 }, { "id": 290091, "postDate": "2018-03-02T22:43:03.697Z", "content": "On the contrary, IMO the authors aim to make these facts more widely known and are a bit appalled that the lesson needs to be re-learned over and over.
", "rawMarkdown": "On the contrary, IMO the authors aim to make these facts more widely known and are a bit appalled that the lesson needs to be re-learned over and over.", "votes": -2 }, { "id": 302220, "postDate": "2018-03-23T20:04:28.233Z", "content": "I think we need consider this, this is the real difficulty for test 2.
", "rawMarkdown": "I think we need consider this, this is the real difficulty for test 2." } ] }, { "id": 281306, "postDate": "2018-02-12T13:58:57.957Z", "content": "Can we know in advance the maximum size of the images of second stage?
\nFor some architectures it is not trivial to work with images of different size.
Agree on this. I tried to predict some images that are significantly different in size from the training set and the predictions were very bad. When I create a cropped image from the original image and do prediction again results were better.
", "rawMarkdown": "Agree on this. I tried to predict some images that are significantly different in size from the training set and the predictions were very bad. When I create a cropped image from the original image and do prediction again results were better. " }, { "id": 284799, "postDate": "2018-02-18T11:49:38.887Z", "content": "\n\n\nAkila Wajirasena wrote
\n \n\n\nAgree on this. I tried to predict some images that are significantly different in size from the training set and the predictions were very bad. When I create a cropped image from the original image and do prediction again results were better.
\n
Thanks Akila, I think it's a important question, however the host didn't even answer...
", "rawMarkdown": "\n> **Akila Wajirasena wrote**\n> \n> > Agree on this. I tried to predict some images that are significantly different in size from the training set and the predictions were very bad. When I create a cropped image from the original image and do prediction again results were better. \n\nThanks Akila, I think it's a important question, however the host didn't even answer..." }, { "id": 290090, "postDate": "2018-03-02T22:42:15.693Z", "content": "Hello, to the best of my knowledge the max individual size of images in the stage 2 test set, in pixels is 1388 wide and max height is 1040 (not from the same image). Wendy Kan is working on finding out the total number of images to be predicted in the stage 2 test set.
", "rawMarkdown": "Hello, to the best of my knowledge the max individual size of images in the stage 2 test set, in pixels is 1388 wide and max height is 1040 (not from the same image). Wendy Kan is working on finding out the total number of images to be predicted in the stage 2 test set.", "votes": 3 }, { "id": 290093, "postDate": "2018-03-02T22:45:48.293Z", "content": "Wendy is speedy! Apparently there are ~3000 images in the stage 2 test set.
", "rawMarkdown": "Wendy is speedy! Apparently there are ~3000 images in the stage 2 test set. ", "votes": 5 } ] }, { "id": 273906, "postDate": "2018-01-25T13:28:28.517Z", "content": "Thanks for the heads up on the test set possibilities! Stain colors are easy to generalize for but I was rather distressed when people said there were different, experimental conditions on the test set. It would be very difficult to build a model that could read things like phase contrast images without showing them it.
", "rawMarkdown": "Thanks for the heads up on the test set possibilities! Stain colors are easy to generalize for but I was rather distressed when people said there were different, experimental conditions on the test set. It would be very difficult to build a model that could read things like phase contrast images without showing them it.", "votes": 3 }, { "id": 312768, "postDate": "2018-04-12T09:57:00.373Z", "content": "Okay, so everyone who actually believed that \"generalizability is a key goal\", used lots of external data and heavy augmentation and expected colorful images much unlike those in the datasets is actually going to come at the very bottom.
", "rawMarkdown": "Okay, so everyone who actually believed that \"generalizability is a key goal\", used lots of external data and heavy augmentation and expected colorful images much unlike those in the datasets is actually going to come at the very bottom.", "replies": [ { "id": 312892, "postDate": "2018-04-12T14:03:59.750Z", "content": "I’m curious, why did you expect “colorful images?”
", "rawMarkdown": "I’m curious, why did you expect “colorful images?” " }, { "id": 312900, "postDate": "2018-04-12T14:13:24.297Z", "content": "Quoting from the first post:
\n\n\n", "rawMarkdown": "Quoting from the first post:\n\n> Changes in stain colors are quite common in biology, for example, so\n> you should expect images with different color schemes." }, { "id": 312905, "postDate": "2018-04-12T14:18:04.447Z", "content": "Changes in stain colors are quite common in biology, for example, so\n you should expect images with different color schemes.
\n
Interesting. I read that differently. My interpretation was a stain that’s stuffed into one channel could be stuffed into another for a different image. Thanks for pointing that out.
", "rawMarkdown": "Interesting. I read that differently. My interpretation was a stain that’s stuffed into one channel could be stuffed into another for a different image. Thanks for pointing that out. \n" }, { "id": 313158, "postDate": "2018-04-12T21:02:07.910Z", "content": "Well, I read and seen more, evidently, so I expected both and much much more. I first, basically scrambled a correct solution with a u, or w-type (u-net followed by u-net) grouped resnet-type network engineered in a smart manner, that could probably go to the top of the chart on what we now know the data to be, one of the things I also did there was just inverting images that had >50% avg light and subtracting some light from each channel, so no learning of both sides of the lighting conditions was required of nets, but then I started reading discussions, viewing images and more and more came up. And I was dragged into figuring out how to solve a task that is 100 times harder to solve and even when solved couldn't possibly produce results as good as the easy specialized solution. So it's not much of a pitty I wasn't able to produce a great solution for that task, as no doubt I wouldn't get into top 5 with such a solution.
\n\nRead Anne's
\n\n\n\n\nThe images to be used in the final test phase (aka, stage 2 test) on\n which you will be judged are generally from different experiments or\n cell types than those included in the stage 1 training and test data\n provided to you. In other words, we generally tried to hold out entire\n image sets for the stage 2 test, rather than holding out just\n individual images.
\n \nSo generalizability is a key goal for your solution. I honestly don't\n recall at this point which image sets went into training vs stage 1\n test vs stage 2 test but it was not systematic or intended to trick\n you (i.e., our goal is not to have the image types in stage 2 test be\n crazy different than training or stage 1 test, though I imagine the\n definition of \"crazy different\" might vary from person to person).\n Changes in stain colors are quite common in biology, for example, so\n you should expect images with different color schemes.
\n
Basically all of the data posted in \"external data\" thread, with a notable exception of BBBC6 was vastly different forms of nuclei images that nobody said were any different from what we should expect the stage 2 dataset to be and those are only going to hinder perf. When you guys say we should expect whole lotta different datasets being included and there are tons of datasets of conceptually similar stuff and many are alike and we post that and you say nothing that gives us very reasonable expectations that that data is going to be helpful. When you look for cell nuclei on the net, most images you find, even those on https://en.wikipedia.org/wiki/Cell_nucleus are much different. But with enough effort and thought, I guess it's possible to make a solution that would be able to reasonably well work all across those.
\n\nhttps://www.youtube.com/watch?v=Dbiq6l50zO8 here are also some hints, for example at 3:12 , 3:36, 3:50 that images like in external data and with per-channel randomly permutated images are to be expected. Considering the later I took data from BBBC6, fixed a small part of it - I needed to review the data quality anyway, btw it seems to be quite as poor boundary-wise as the training set, and I combined images from 2 of the filters, or I don't know what that thing is, maybe originally 2 different parts of spectre (different colors) into single images with one channel being one image, other being the other and yet another being both - I'm guessing some of the stage 2 test samples are like the later one. Which I later randomly permutated during training. Oh, man, does permutation hurt. Not as much as the inversion, and all of the color shifting I did to account from different lighting conditions, though!
\n\nhttps://www.youtube.com/watch?v=Dbiq6l50zO8 Especially misleading statements start at 4:00.
\n\nMore misleading comments by Anne:
\n\n\n\n\nAgree! Besides, the problem here is generalizability across a variety\n of image/cell/microscope types. Solving any one image type, one might\n achieve high accuracy but the variation is what makes this challenge a\n challenge.
\n \nWe want a single model that just works across all kinds of image\n modalities, no matter the size of the nuclei or the color scheme. Such\n a model could be built into software that biologists use with all\n kinds of microscopes and eliminate the need for them to train on their\n individual data or provide metadata about their cell type, microscope,\n resolution, etc.
\n \nAn organizer here. You could take two views: (1) Annoyance that this\n challenge is hard because we are intentionally not giving you the full\n range of all possible image types. (2) Excitement that this challenge\n is about generalizability and robustness.
\n \nThe advantage of perspective (2) is it produces meaningful solutions\n that help in the real world. I also think this problem is a solvable\n one: biologists can detect nuclei accurately no matter the size,\n species, or microscopy type (so long as they agree on some ground\n rules, like whether to count micronuclei). I'm not saying that I\n expect >0.90 in this challenge, perhaps we need much more training\n data to get to that point, but I just mean that I think it will be\n solved sooner or later.
\n \nIf we gave you a defined set of image types as training data, the\n problem would be, well, perhaps not trivial but at least a lot easier,\n where differences between competitors come down to not-so-interesting\n choices in models. As we have planned the challenge, we expect a lot\n of variety in how teams choose to handle it.
\n \nBonus: This one will be a nail-biter at the end, as we see how\n generalizable the models end up being!
\n \nThe most important thing to understand, which I believe will resolve\n all your well-thought-out concerns, is that the goal of the challenge\n is not to count nuclei. It is to identify nuclei and fragments of DNA\n (as long as they were not too obviously streaky debris but still had\n relatively roundish shapes), as the starting point for a workflow the\n biologist will choose. If their goal does happen to be to count\n nuclei, they would want to follow the filtering procedures you\n describe (this is relatively rare in the high-throughput case, because\n there are cheaper and easier ways to measure cell proliferation). But\n most often, their goal will be to use the nuclei as the starting point\n for a more complex analysis that often includes identifying cell\n boundaries using a separate stain visible in a different channel. For\n those who want to eliminate small fragments of DNA it is pretty\n trivial to filter by size or shape criteria. But some may want to\n include them because they want to specifically measure micronuclei.\n This is why we decided to include all DNA fragments in the\n annotations.
\n \nStep 1: Neural Network (NN) for foreground/background (nuclei vs\n non-nuclei). One could almost get away with adaptive thresholding\n here, but the potential for different color schemes makes it hard to\n know which channel (or combos of channels) to use for this so I assume\n NNs magically figure this out). This yields a foreground/background\n probability map.
\n
Annes comment a little above my question
\n\n\n\n\nI'm not aware of major differences. Both sets are mixtures of images\n from different experimental sets, so you should expect a variety.
\n
is also as misleading. Though, it hurt more when I asked her my question about vastly different pics from BBBC6 made with different staining and then later read that comment not here, but in an email notification and thought it was a response to my question. Although, I don't regret that, as those images are actually pretty useful on the contrary.
\n\n\n\n\nIndeed. I'm an organizer. The second stage (test) dataset will indeed\n have image sets that are experimentally distinct from anything seen in\n the training and validation (stage 1) datasets. The goal of this\n competition is generalizability and robustness so the only way to test\n this well is to be sure that the final test image sets are distinct\n relative to the training data.
\n
Again.. Basically, just go here https://www.kaggle.com/drannecarpenter/discussion to find all the misleading info you need to understand what I thought. Allen, your comment about how your model was overfit vs stage 2 data also contributed to my perception that that dataset is gonna be hell lotta different. As did the fact that train set contained black-on-white images not present in test set... Turns out in either one. Fools were those who tried to fit that data, especially those who did'nt invert colors. It definitely was a hit to perf.
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47615#271047 all of this discussion generally makes one think that cancerous tissues are some images that are probably gonna be there in the test 2.
\n\nI definitely wasn't alone in being mislead about what the data is gonna be and I wasn't even the most crazy person https://www.kaggle.com/c/data-science-bowl-2018/discussion/53337 - at least I assumed my model should just take a lot of info from context and should know a learn features from different sample settings and should just work on any nuclei images, not eyeballs and yolks.
\n\nI expected different info stuffed into different channels, as specialized software produces such things, I expected different lighting conditions, for example fluorescence and backlighting, I expected cancerous tissues, binucleate stuff, vastly different zoom levels, some images with hugely packed density of nuclei, different sort of crap being overlayed over the nuclei, some light being reflected, different focus and sharpness, etc.
\n\nAs the solution phase is already over and I won't get any prize anyway, I'll share some things about my solution. It was somewhat like Anne and many others thought up with boundaries and masks. Except I didn't annotate boundaries, like most - that was not a very wise move, rly. Especially given the data quality. And one should not, of course, only ask the network to show where boundaries between nuclei are, as it's only going to be harder, again, especially given that the ground truth quality is far from pixel-level. One thing that came really obvious to me early on, again, also largely accounting for ground truth quality and the basic lack of precision in such a task - is that one shouldn't ask the net to annotate boundaries, but should instead focus on meaningful nuclei inner parts. So a mask that the net should generate should contain both - boundries and parts of the picture that are outside of nuclei, but the emphasis of loss weighting should be on mostly on touch zones and bounds that are close (say 5 pixels close) to that of any other nucleus (say weight of 1), boundaries (say, weight of 0.25) and inner part of the nuclei (say, weight of 0.2), with the parts of the picture outside of these zones being weighted at only, say, 0.02-0.01. And, because the annotations are not really all that precise, the inner parts of inner nuclei that are exactly adjecent to border are also weighted as unimportant if they're not crucial. They're crucial if, for example, a nuclei is small, or they are part of a thin stretch. We only disreguard whatever our network is going to produce for those pixels isn't going to hurt our understanding of this nucleuses bounds. One important way it should help is that at places where two nuclei meet we don't have pixel-precise annotations. So giving the net a wider range of pixels that it can label as a boundary that we don't penalise it for is better. And it won't hurt us by creating holes inside of a nucleus, or spreading it into two parts - we only allow this imprecision on pixels where it won't.
\n\nWhy we need to learn stuff outside of nuclei? Well, firstly, of course, if you really don't care about what it shows for the outside pixels, you should at least not apply weights to the outside zones, as otherwise you're forcing the net to learn that outside of nuclei it should produce a result same as inside and only the thin stretch of boundary matters. Which is extremely hard and confusing, especially, again, given the imprecision of ground truth info. And it forces your net to learn info which is hard to learn and which is going to massively hit it's ability to see boundaries, because if it doubts whether a pixel is a boundary, or if it's outside it may choose to opt for saying it's outside instead. But that's not really enough, it's better to make it learn that outside of nuclei it should produce same result as for boundaries, because annotations are not precise and sometimes what is known to us as a boundary from ground truth, actually lays outside of a nucleus and even some of the annotated insides also lay outside of actual nucleus. And when you label actual boundary as being inside, you mislead and confuse the net hurting its understanding of the subject. So you're telling it what stuff outside looks like at one place and annotate it same as bounds at other places.
\n\nI had quite an amount of frustration with the train set and BBBC6, especially with the train set, where some nuclei are missed, random pixel-wide holes in are present in anotations and hundreds (as far as I remember) nucleuses are merged into one that obviously aren't. In order for my model to ever work, I of course, had to have a decent precise dataset. So I went through each sample and fixed it. https://github.com/ibmua/data-science-bowl-2018-train-set First simply in a MS Paint-like KolourPaint app, but then when I saw that there's way too many mistakes I decided to make a special tool I now openly published. It's pretty laggy, but it's magic compared to other means. BBBC6 dataset has decent coverage and nice images on the other hand, but when it comes to boundaries, annotations seem quite insane. But I didn't bother annotating BBBC6, because I was running late. At the end, with all of the misleading info I wasn't able to make it on time and have an appropriate model, so I definitely don't regret failing to make that investment. Only fixed some small part. Processed into appropriate format it's now available at https://drive.google.com/file/d/16KVLYeKBkk_Y0W4Fra_2AhjmW-p6JjZu/view?usp=sharing
\n\nMy network was a U-net style network, also tried double-U-net, with a pretty equal amount of channels all around the scales (not widening as at deepens) with 3x3->3x3[grouped] residual blocks on all scales. I wasn't able to produce a really decent \"general nuclei discovery network\" by deadline, so I can't say for sure, but it seemed, like groups seemed especially highly beneficial for the generalizeable network - in the best one I had I used 16 groups for 64 channels. It's seemingly not that important for the non-generalizeable solution for stage 2 data, but for general \"any type\", especially including channel permutations data I guess it's important to, at least at the largest scale, process each channel with a single subnetwork - sharing weights. Then, maybe, combine results with a 1x1 conv. Adding resulting feature layers from all colors together also works, at least somewhat, but I think 1x1 is probably going to be much better.
\n\nOne more really important, I think, tip is to first train a inner nuclei map net and then repurpose it into a general map net. That saves time, the net is better and works better with the net it's based on. I mean using boundary map net as a pretrain. Then just making it learn that boundaries should be included into what's expected of it, not just inner parts of nuclei.
", "rawMarkdown": "Well, I read and seen more, evidently, so I expected both and much much more. I first, basically scrambled a correct solution with a u, or w-type (u-net followed by u-net) grouped resnet-type network engineered in a smart manner, that could probably go to the top of the chart on what we now know the data to be, one of the things I also did there was just inverting images that had >50% avg light and subtracting some light from each channel, so no learning of both sides of the lighting conditions was required of nets, but then I started reading discussions, viewing images and more and more came up. And I was dragged into figuring out how to solve a task that is 100 times harder to solve and even when solved couldn't possibly produce results as good as the easy specialized solution. So it's not much of a pitty I wasn't able to produce a great solution for that task, as no doubt I wouldn't get into top 5 with such a solution.\n\nRead Anne's\n\n> The images to be used in the final test phase (aka, stage 2 test) on\n> which you will be judged are generally from different experiments or\n> cell types than those included in the stage 1 training and test data\n> provided to you. In other words, we generally tried to hold out entire\n> image sets for the stage 2 test, rather than holding out just\n> individual images.\n> \n> So generalizability is a key goal for your solution. I honestly don't\n> recall at this point which image sets went into training vs stage 1\n> test vs stage 2 test but it was not systematic or intended to trick\n> you (i.e., our goal is not to have the image types in stage 2 test be\n> crazy different than training or stage 1 test, though I imagine the\n> definition of \"crazy different\" might vary from person to person).\n> Changes in stain colors are quite common in biology, for example, so\n> you should expect images with different color schemes.\n\nBasically all of the data posted in \"external data\" thread, with a notable exception of BBBC6 was vastly different forms of nuclei images that nobody said were any different from what we should expect the stage 2 dataset to be and those are only going to hinder perf. When you guys say we should expect whole lotta different datasets being included and there are tons of datasets of conceptually similar stuff and many are alike and we post that and you say nothing that gives us very reasonable expectations that that data is going to be helpful. When you look for cell nuclei on the net, most images you find, even those on https://en.wikipedia.org/wiki/Cell_nucleus are much different. But with enough effort and thought, I guess it's possible to make a solution that would be able to reasonably well work all across those.\n\nhttps://www.youtube.com/watch?v=Dbiq6l50zO8 here are also some hints, for example at 3:12 , 3:36, 3:50 that images like in external data and with per-channel randomly permutated images are to be expected. Considering the later I took data from BBBC6, fixed a small part of it - I needed to review the data quality anyway, btw it seems to be quite as poor boundary-wise as the training set, and I combined images from 2 of the filters, or I don't know what that thing is, maybe originally 2 different parts of spectre (different colors) into single images with one channel being one image, other being the other and yet another being both - I'm guessing some of the stage 2 test samples are like the later one. Which I later randomly permutated during training. Oh, man, does permutation hurt. Not as much as the inversion, and all of the color shifting I did to account from different lighting conditions, though!\n\nhttps://www.youtube.com/watch?v=Dbiq6l50zO8 Especially misleading statements start at 4:00.\n\nMore misleading comments by Anne:\n\n> Agree! Besides, the problem here is generalizability across a variety\n> of image/cell/microscope types. Solving any one image type, one might\n> achieve high accuracy but the variation is what makes this challenge a\n> challenge.\n> \n> We want a single model that just works across all kinds of image\n> modalities, no matter the size of the nuclei or the color scheme. Such\n> a model could be built into software that biologists use with all\n> kinds of microscopes and eliminate the need for them to train on their\n> individual data or provide metadata about their cell type, microscope,\n> resolution, etc.\n> \n> An organizer here. You could take two views: (1) Annoyance that this\n> challenge is hard because we are intentionally not giving you the full\n> range of all possible image types. (2) Excitement that this challenge\n> is about generalizability and robustness.\n> \n> The advantage of perspective (2) is it produces meaningful solutions\n> that help in the real world. I also think this problem is a solvable\n> one: biologists can detect nuclei accurately no matter the size,\n> species, or microscopy type (so long as they agree on some ground\n> rules, like whether to count micronuclei). I'm not saying that I\n> expect >0.90 in this challenge, perhaps we need much more training\n> data to get to that point, but I just mean that I think it will be\n> solved sooner or later.\n> \n> If we gave you a defined set of image types as training data, the\n> problem would be, well, perhaps not trivial but at least a lot easier,\n> where differences between competitors come down to not-so-interesting\n> choices in models. As we have planned the challenge, we expect a lot\n> of variety in how teams choose to handle it.\n> \n> Bonus: This one will be a nail-biter at the end, as we see how\n> generalizable the models end up being!\n> \n> The most important thing to understand, which I believe will resolve\n> all your well-thought-out concerns, is that the goal of the challenge\n> is not to count nuclei. It is to identify nuclei and fragments of DNA\n> (as long as they were not too obviously streaky debris but still had\n> relatively roundish shapes), as the starting point for a workflow the\n> biologist will choose. If their goal does happen to be to count\n> nuclei, they would want to follow the filtering procedures you\n> describe (this is relatively rare in the high-throughput case, because\n> there are cheaper and easier ways to measure cell proliferation). But\n> most often, their goal will be to use the nuclei as the starting point\n> for a more complex analysis that often includes identifying cell\n> boundaries using a separate stain visible in a different channel. For\n> those who want to eliminate small fragments of DNA it is pretty\n> trivial to filter by size or shape criteria. But some may want to\n> include them because they want to specifically measure micronuclei.\n> This is why we decided to include all DNA fragments in the\n> annotations.\n> \n> Step 1: Neural Network (NN) for foreground/background (nuclei vs\n> non-nuclei). One could almost get away with adaptive thresholding\n> here, but the potential for different color schemes makes it hard to\n> know which channel (or combos of channels) to use for this so I assume\n> NNs magically figure this out). This yields a foreground/background\n> probability map.\n\nAnnes comment a little above my question\n\n> I'm not aware of major differences. Both sets are mixtures of images\n> from different experimental sets, so you should expect a variety.\n\nis also as misleading. Though, it hurt more when I asked her my question about vastly different pics from BBBC6 made with different staining and then later read that comment not here, but in an email notification and thought it was a response to my question. Although, I don't regret that, as those images are actually pretty useful on the contrary.\n\n> Indeed. I'm an organizer. The second stage (test) dataset will indeed\n> have image sets that are experimentally distinct from anything seen in\n> the training and validation (stage 1) datasets. The goal of this\n> competition is generalizability and robustness so the only way to test\n> this well is to be sure that the final test image sets are distinct\n> relative to the training data.\n\nAgain.. Basically, just go here https://www.kaggle.com/drannecarpenter/discussion to find all the misleading info you need to understand what I thought. Allen, your comment about how your model was overfit vs stage 2 data also contributed to my perception that that dataset is gonna be hell lotta different. As did the fact that train set contained black-on-white images not present in test set... Turns out in either one. Fools were those who tried to fit that data, especially those who did'nt invert colors. It definitely was a hit to perf.\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47615#271047 all of this discussion generally makes one think that cancerous tissues are some images that are probably gonna be there in the test 2.\n\nI definitely wasn't alone in being mislead about what the data is gonna be and I wasn't even the most crazy person https://www.kaggle.com/c/data-science-bowl-2018/discussion/53337 - at least I assumed my model should just take a lot of info from context and should know a learn features from different sample settings and should just work on any nuclei images, not eyeballs and yolks.\n\nI expected different info stuffed into different channels, as specialized software produces such things, I expected different lighting conditions, for example fluorescence and backlighting, I expected cancerous tissues, binucleate stuff, vastly different zoom levels, some images with hugely packed density of nuclei, different sort of crap being overlayed over the nuclei, some light being reflected, different focus and sharpness, etc.\n\nAs the solution phase is already over and I won't get any prize anyway, I'll share some things about my solution. It was somewhat like Anne and many others thought up with boundaries and masks. Except I didn't annotate boundaries, like most - that was not a very wise move, rly. Especially given the data quality. And one should not, of course, only ask the network to show where boundaries between nuclei are, as it's only going to be harder, again, especially given that the ground truth quality is far from pixel-level. One thing that came really obvious to me early on, again, also largely accounting for ground truth quality and the basic lack of precision in such a task - is that one shouldn't ask the net to annotate boundaries, but should instead focus on meaningful nuclei inner parts. So a mask that the net should generate should contain both - boundries and parts of the picture that are outside of nuclei, but the emphasis of loss weighting should be on mostly on touch zones and bounds that are close (say 5 pixels close) to that of any other nucleus (say weight of 1), boundaries (say, weight of 0.25) and inner part of the nuclei (say, weight of 0.2), with the parts of the picture outside of these zones being weighted at only, say, 0.02-0.01. And, because the annotations are not really all that precise, the inner parts of inner nuclei that are exactly adjecent to border are also weighted as unimportant if they're not crucial. They're crucial if, for example, a nuclei is small, or they are part of a thin stretch. We only disreguard whatever our network is going to produce for those pixels isn't going to hurt our understanding of this nucleuses bounds. One important way it should help is that at places where two nuclei meet we don't have pixel-precise annotations. So giving the net a wider range of pixels that it can label as a boundary that we don't penalise it for is better. And it won't hurt us by creating holes inside of a nucleus, or spreading it into two parts - we only allow this imprecision on pixels where it won't.\n\nWhy we need to learn stuff outside of nuclei? Well, firstly, of course, if you really don't care about what it shows for the outside pixels, you should at least not apply weights to the outside zones, as otherwise you're forcing the net to learn that outside of nuclei it should produce a result same as inside and only the thin stretch of boundary matters. Which is extremely hard and confusing, especially, again, given the imprecision of ground truth info. And it forces your net to learn info which is hard to learn and which is going to massively hit it's ability to see boundaries, because if it doubts whether a pixel is a boundary, or if it's outside it may choose to opt for saying it's outside instead. But that's not really enough, it's better to make it learn that outside of nuclei it should produce same result as for boundaries, because annotations are not precise and sometimes what is known to us as a boundary from ground truth, actually lays outside of a nucleus and even some of the annotated insides also lay outside of actual nucleus. And when you label actual boundary as being inside, you mislead and confuse the net hurting its understanding of the subject. So you're telling it what stuff outside looks like at one place and annotate it same as bounds at other places.\n\nI had quite an amount of frustration with the train set and BBBC6, especially with the train set, where some nuclei are missed, random pixel-wide holes in are present in anotations and hundreds (as far as I remember) nucleuses are merged into one that obviously aren't. In order for my model to ever work, I of course, had to have a decent precise dataset. So I went through each sample and fixed it. https://github.com/ibmua/data-science-bowl-2018-train-set First simply in a MS Paint-like KolourPaint app, but then when I saw that there's way too many mistakes I decided to make a special tool I now openly published. It's pretty laggy, but it's magic compared to other means. BBBC6 dataset has decent coverage and nice images on the other hand, but when it comes to boundaries, annotations seem quite insane. But I didn't bother annotating BBBC6, because I was running late. At the end, with all of the misleading info I wasn't able to make it on time and have an appropriate model, so I definitely don't regret failing to make that investment. Only fixed some small part. Processed into appropriate format it's now available at https://drive.google.com/file/d/16KVLYeKBkk_Y0W4Fra_2AhjmW-p6JjZu/view?usp=sharing\n\nMy network was a U-net style network, also tried double-U-net, with a pretty equal amount of channels all around the scales (not widening as at deepens) with 3x3->3x3[grouped] residual blocks on all scales. I wasn't able to produce a really decent \"general nuclei discovery network\" by deadline, so I can't say for sure, but it seemed, like groups seemed especially highly beneficial for the generalizeable network - in the best one I had I used 16 groups for 64 channels. It's seemingly not that important for the non-generalizeable solution for stage 2 data, but for general \"any type\", especially including channel permutations data I guess it's important to, at least at the largest scale, process each channel with a single subnetwork - sharing weights. Then, maybe, combine results with a 1x1 conv. Adding resulting feature layers from all colors together also works, at least somewhat, but I think 1x1 is probably going to be much better.\n\nOne more really important, I think, tip is to first train a inner nuclei map net and then repurpose it into a general map net. That saves time, the net is better and works better with the net it's based on. I mean using boundary map net as a pretrain. Then just making it learn that boundaries should be included into what's expected of it, not just inner parts of nuclei." }, { "id": 313197, "postDate": "2018-04-12T22:28:12.327Z", "content": "One general reason to look for external data was also that there were very fiew colored images, they weren't well annotated and I had no clue how they should be annotated. I tried annotating the test set and as I saw stage 1 test's ground truth upon stage 1's end I was shocked that I was totally wrong about my guesses on all colored images and also some grayscale ones. Especially those that looked markedly different from any of the ones in the train set.
", "rawMarkdown": "One general reason to look for external data was also that there were very fiew colored images, they weren't well annotated and I had no clue how they should be annotated. I tried annotating the test set and as I saw stage 1 test's ground truth upon stage 1's end I was shocked that I was totally wrong about my guesses on all colored images and also some grayscale ones. Especially those that looked markedly different from any of the ones in the train set." } ] }, { "id": 307465, "postDate": "2018-04-01T17:07:34.210Z", "content": "Are our models expected to identify nucleuses on hardcore types of images like these? If direct answer is not possible, any indirect ones are appreciated. Like how desireable it is. \"Phalloidin\"-type image taken from your dataset https://data.broadinstitute.org/bbbc/BBBC006/
", "rawMarkdown": "Are our models expected to identify nucleuses on hardcore types of images like these? If direct answer is not possible, any indirect ones are appreciated. Like how desireable it is. \"Phalloidin\"-type image taken from your dataset https://data.broadinstitute.org/bbbc/BBBC006/", "replies": [ { "id": 308059, "postDate": "2018-04-02T21:57:30.123Z", "content": "We debated whether to include images that had relatively indirect labeling of nuclei and decided to not make the competition THAT hard. \"Indirect\" is maybe a bad word choice because many histology images use H&E staining which is not exactly a direct DNA stain. But anyway, generally speaking, to the best of my recollection, we only included images where the nucleus itself looked pretty stained relative to its surroundings.
", "rawMarkdown": "We debated whether to include images that had relatively indirect labeling of nuclei and decided to not make the competition THAT hard. \"Indirect\" is maybe a bad word choice because many histology images use H&E staining which is not exactly a direct DNA stain. But anyway, generally speaking, to the best of my recollection, we only included images where the nucleus itself looked pretty stained relative to its surroundings.", "votes": 1 }, { "id": 313192, "postDate": "2018-04-12T22:11:09.920Z", "rawMarkdown": "", "isDeleted": true }, { "id": 313196, "postDate": "2018-04-12T22:20:37.840Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 305209, "postDate": "2018-03-28T15:41:50.090Z", "content": "Hi Anne, \nWould the distribution of number of nuclei in each image be way different in stage 2 test (compared to the training set and stage 1 test) ?
", "rawMarkdown": "Hi Anne, \nWould the distribution of number of nuclei in each image be way different in stage 2 test (compared to the training set and stage 1 test) ?", "replies": [ { "id": 308040, "postDate": "2018-04-02T20:53:49.617Z", "content": "I'm not aware of major differences. Both sets are mixtures of images from different experimental sets, so you should expect a variety.
", "rawMarkdown": "I'm not aware of major differences. Both sets are mixtures of images from different experimental sets, so you should expect a variety." } ] }, { "id": 283610, "postDate": "2018-02-15T19:09:20.030Z", "content": "It'll be very helpful if the following 2 things can happen :\n[1] Update us with a CLEANED up train set, get rid of all the problem images.\n[2] Increase the daily submit limit to 20 or 30 for each person, although not everyone will submit that many each day, so it shouldn't be too much of a burden to the server.
", "rawMarkdown": "It'll be very helpful if the following 2 things can happen :\n[1] Update us with a CLEANED up train set, get rid of all the problem images.\n[2] Increase the daily submit limit to 20 or 30 for each person, although not everyone will submit that many each day, so it shouldn't be too much of a burden to the server.", "replies": [ { "id": 283670, "postDate": "2018-02-15T20:37:06.753Z", "content": "Do you mind submitting this comment to a new, or relevant thread? It may receive more attention there. I personally do not control the rules about either of these.
", "rawMarkdown": "Do you mind submitting this comment to a new, or relevant thread? It may receive more attention there. I personally do not control the rules about either of these. " } ] }, { "id": 282083, "postDate": "2018-02-13T15:52:28.783Z", "content": "Hi Anne, may we assume that we will also have a corresponding new training set in Phase2? \nThanks in Advance!!
", "rawMarkdown": "Hi Anne, may we assume that we will also have a corresponding new training set in Phase2? \nThanks in Advance!!", "replies": [ { "id": 282212, "postDate": "2018-02-13T19:08:06.070Z", "content": "Nope. Unless you count the stage 1 test's annotations which will be provided at that point. See timeline page:
\n\n\n", "rawMarkdown": "Nope. Unless you count the stage 1 test's annotations which will be provided at that point. See timeline page: \n\n> April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set.", "votes": 1 }, { "id": 282457, "postDate": "2018-02-14T02:24:41.970Z", "content": "April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set.
\n
Can we retrain the model after April 11, 2018 with the validation set?
", "rawMarkdown": "Can we retrain the model after April 11, 2018 with the validation set?", "votes": 7 }, { "id": 282604, "postDate": "2018-02-14T08:34:12.193Z", "content": "Thanks Anne!!
", "rawMarkdown": "Thanks Anne!!" } ] } ], "comments": [ { "id": 284577, "author_name": "rakhlin", "author_url": "", "post_date": "2018-02-17T16:25:10.100000", "content": "What you call generalizability must be called transferrabilty, b/c you expect to transfer a model trained on some type of data to different data. Generalizability is ability to perform on the same data distribution. i.i.d. assumption about training/test data is the cornerstone of ML, and automatic transferrabilty you are anticipating does not exist. By automatic I mean no additional retraining or other supervised procedure to adjust distributions. The results will be random to degree of distribution difference.
", "votes": 11, "replies": [ { "id": 285640, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-20T19:16:14.473000", "content": "Well, it's an interesting question of semantics that you raise here.
\n\nSay I train a model on 100 different data types and then I test it on 100 additional data types, distinct from the first set. Would you call that generalizability or transferability? I can see arguments both ways.
\n\nNote that in our competition, we only have a handful of image types in each stage rather than 100.
", "votes": -1, "replies": [] }, { "id": 285704, "author_name": "rakhlin", "author_url": "", "post_date": "2018-02-20T21:39:32.973000", "content": "It's not about semantics. One can not adequately test a [statistically trained] model on another and unseen data distribution. No miracles.
", "votes": 3, "replies": [] }, { "id": 285992, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-21T12:47:40.680000", "content": "I disagree. The semantic question is what data distributions you consider \"similar\" vs \"not similar\". We seem to agree that: \nsimilar data distributions -> generalizability\nnot similar data distributions -> transferability
\n\nI would argue that two collections wherein each contains multiple image types are \"similar\" and thus this is a problem of generalizability.
\n\nI'm not sure why you are talking about miracles and adequate statistical testing. I thought we were talking about which term to use, not expected results or appropriate statistical testing.
", "votes": -2, "replies": [] }, { "id": 286018, "author_name": "rakhlin", "author_url": "", "post_date": "2018-02-21T13:55:25.197000", "content": "Here is your quote I was referring to, highlighting is mine:
\n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged are generally from different experiments or cell types than those included in the stage 1 training and test data provided to you. In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images.
\n\nI read this as the test data comes from different distribution and is dissimilar to training data. How many image types are contained in training/test sets is irrelevant from this point of view. Multiple image types is still okay for generalization, provided they are equally represented in both sets.
", "votes": 1, "replies": [] }, { "id": 286602, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-22T13:15:02.283000", "content": "I guess we will not agree on the terminology, then. I think this style of training set is quite unusual so most people are not used to thinking of training/test sets where we have a variety of image types in both but no cross-representation. Our goal is to prevent Voodoo machine learning where methods do great during testing but fail to generalize: https://www.biorxiv.org/content/early/2016/06/19/059774
", "votes": 0, "replies": [] }, { "id": 286659, "author_name": "rakhlin", "author_url": "", "post_date": "2018-02-22T15:49:43.153000", "content": "Failure to transfer ML models inter subjects has been known and addressed for years. Look at any Kaggle EEG competition for instance. What the authors of this 2016 paper seemingly don't understand is that it is expected and follows from stats learning theory.
", "votes": 1, "replies": [] }, { "id": 290091, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-02T22:43:03.697000", "content": "On the contrary, IMO the authors aim to make these facts more widely known and are a bit appalled that the lesson needs to be re-learned over and over.
", "votes": -2, "replies": [] }, { "id": 302220, "author_name": "Yan Wang", "author_url": "", "post_date": "2018-03-23T20:04:28.233000", "content": "I think we need consider this, this is the real difficulty for test 2.
", "votes": 0, "replies": [] } ] }, { "id": 281306, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2018-02-12T13:58:57.957000", "content": "Can we know in advance the maximum size of the images of second stage?
\nFor some architectures it is not trivial to work with images of different size.
Agree on this. I tried to predict some images that are significantly different in size from the training set and the predictions were very bad. When I create a cropped image from the original image and do prediction again results were better.
", "votes": 0, "replies": [] }, { "id": 284799, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2018-02-18T11:49:38.887000", "content": "\n\n\nAkila Wajirasena wrote
\n \n\n\nAgree on this. I tried to predict some images that are significantly different in size from the training set and the predictions were very bad. When I create a cropped image from the original image and do prediction again results were better.
\n
Thanks Akila, I think it's a important question, however the host didn't even answer...
", "votes": 0, "replies": [] }, { "id": 290090, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-02T22:42:15.693000", "content": "Hello, to the best of my knowledge the max individual size of images in the stage 2 test set, in pixels is 1388 wide and max height is 1040 (not from the same image). Wendy Kan is working on finding out the total number of images to be predicted in the stage 2 test set.
", "votes": 3, "replies": [] }, { "id": 290093, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-03-02T22:45:48.293000", "content": "Wendy is speedy! Apparently there are ~3000 images in the stage 2 test set.
", "votes": 5, "replies": [] } ] }, { "id": 273906, "author_name": "Arpan Dhatt", "author_url": "", "post_date": "2018-01-25T13:28:28.517000", "content": "Thanks for the heads up on the test set possibilities! Stain colors are easy to generalize for but I was rather distressed when people said there were different, experimental conditions on the test set. It would be very difficult to build a model that could read things like phase contrast images without showing them it.
", "votes": 3, "replies": [] }, { "id": 312768, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-12T09:57:00.373000", "content": "Okay, so everyone who actually believed that \"generalizability is a key goal\", used lots of external data and heavy augmentation and expected colorful images much unlike those in the datasets is actually going to come at the very bottom.
", "votes": 0, "replies": [ { "id": 312892, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-12T14:03:59.750000", "content": "I’m curious, why did you expect “colorful images?”
", "votes": 0, "replies": [] }, { "id": 312900, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2018-04-12T14:13:24.297000", "content": "Quoting from the first post:
\n\n\n", "votes": 0, "replies": [] }, { "id": 312905, "author_name": "Allen Goodman", "author_url": "", "post_date": "2018-04-12T14:18:04.447000", "content": "Changes in stain colors are quite common in biology, for example, so\n you should expect images with different color schemes.
\n
Interesting. I read that differently. My interpretation was a stain that’s stuffed into one channel could be stuffed into another for a different image. Thanks for pointing that out.
", "votes": 0, "replies": [] }, { "id": 313158, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-12T21:02:07.910000", "content": "Well, I read and seen more, evidently, so I expected both and much much more. I first, basically scrambled a correct solution with a u, or w-type (u-net followed by u-net) grouped resnet-type network engineered in a smart manner, that could probably go to the top of the chart on what we now know the data to be, one of the things I also did there was just inverting images that had >50% avg light and subtracting some light from each channel, so no learning of both sides of the lighting conditions was required of nets, but then I started reading discussions, viewing images and more and more came up. And I was dragged into figuring out how to solve a task that is 100 times harder to solve and even when solved couldn't possibly produce results as good as the easy specialized solution. So it's not much of a pitty I wasn't able to produce a great solution for that task, as no doubt I wouldn't get into top 5 with such a solution.
\n\nRead Anne's
\n\n\n\n\nThe images to be used in the final test phase (aka, stage 2 test) on\n which you will be judged are generally from different experiments or\n cell types than those included in the stage 1 training and test data\n provided to you. In other words, we generally tried to hold out entire\n image sets for the stage 2 test, rather than holding out just\n individual images.
\n \nSo generalizability is a key goal for your solution. I honestly don't\n recall at this point which image sets went into training vs stage 1\n test vs stage 2 test but it was not systematic or intended to trick\n you (i.e., our goal is not to have the image types in stage 2 test be\n crazy different than training or stage 1 test, though I imagine the\n definition of \"crazy different\" might vary from person to person).\n Changes in stain colors are quite common in biology, for example, so\n you should expect images with different color schemes.
\n
Basically all of the data posted in \"external data\" thread, with a notable exception of BBBC6 was vastly different forms of nuclei images that nobody said were any different from what we should expect the stage 2 dataset to be and those are only going to hinder perf. When you guys say we should expect whole lotta different datasets being included and there are tons of datasets of conceptually similar stuff and many are alike and we post that and you say nothing that gives us very reasonable expectations that that data is going to be helpful. When you look for cell nuclei on the net, most images you find, even those on https://en.wikipedia.org/wiki/Cell_nucleus are much different. But with enough effort and thought, I guess it's possible to make a solution that would be able to reasonably well work all across those.
\n\nhttps://www.youtube.com/watch?v=Dbiq6l50zO8 here are also some hints, for example at 3:12 , 3:36, 3:50 that images like in external data and with per-channel randomly permutated images are to be expected. Considering the later I took data from BBBC6, fixed a small part of it - I needed to review the data quality anyway, btw it seems to be quite as poor boundary-wise as the training set, and I combined images from 2 of the filters, or I don't know what that thing is, maybe originally 2 different parts of spectre (different colors) into single images with one channel being one image, other being the other and yet another being both - I'm guessing some of the stage 2 test samples are like the later one. Which I later randomly permutated during training. Oh, man, does permutation hurt. Not as much as the inversion, and all of the color shifting I did to account from different lighting conditions, though!
\n\nhttps://www.youtube.com/watch?v=Dbiq6l50zO8 Especially misleading statements start at 4:00.
\n\nMore misleading comments by Anne:
\n\n\n\n\nAgree! Besides, the problem here is generalizability across a variety\n of image/cell/microscope types. Solving any one image type, one might\n achieve high accuracy but the variation is what makes this challenge a\n challenge.
\n \nWe want a single model that just works across all kinds of image\n modalities, no matter the size of the nuclei or the color scheme. Such\n a model could be built into software that biologists use with all\n kinds of microscopes and eliminate the need for them to train on their\n individual data or provide metadata about their cell type, microscope,\n resolution, etc.
\n \nAn organizer here. You could take two views: (1) Annoyance that this\n challenge is hard because we are intentionally not giving you the full\n range of all possible image types. (2) Excitement that this challenge\n is about generalizability and robustness.
\n \nThe advantage of perspective (2) is it produces meaningful solutions\n that help in the real world. I also think this problem is a solvable\n one: biologists can detect nuclei accurately no matter the size,\n species, or microscopy type (so long as they agree on some ground\n rules, like whether to count micronuclei). I'm not saying that I\n expect >0.90 in this challenge, perhaps we need much more training\n data to get to that point, but I just mean that I think it will be\n solved sooner or later.
\n \nIf we gave you a defined set of image types as training data, the\n problem would be, well, perhaps not trivial but at least a lot easier,\n where differences between competitors come down to not-so-interesting\n choices in models. As we have planned the challenge, we expect a lot\n of variety in how teams choose to handle it.
\n \nBonus: This one will be a nail-biter at the end, as we see how\n generalizable the models end up being!
\n \nThe most important thing to understand, which I believe will resolve\n all your well-thought-out concerns, is that the goal of the challenge\n is not to count nuclei. It is to identify nuclei and fragments of DNA\n (as long as they were not too obviously streaky debris but still had\n relatively roundish shapes), as the starting point for a workflow the\n biologist will choose. If their goal does happen to be to count\n nuclei, they would want to follow the filtering procedures you\n describe (this is relatively rare in the high-throughput case, because\n there are cheaper and easier ways to measure cell proliferation). But\n most often, their goal will be to use the nuclei as the starting point\n for a more complex analysis that often includes identifying cell\n boundaries using a separate stain visible in a different channel. For\n those who want to eliminate small fragments of DNA it is pretty\n trivial to filter by size or shape criteria. But some may want to\n include them because they want to specifically measure micronuclei.\n This is why we decided to include all DNA fragments in the\n annotations.
\n \nStep 1: Neural Network (NN) for foreground/background (nuclei vs\n non-nuclei). One could almost get away with adaptive thresholding\n here, but the potential for different color schemes makes it hard to\n know which channel (or combos of channels) to use for this so I assume\n NNs magically figure this out). This yields a foreground/background\n probability map.
\n
Annes comment a little above my question
\n\n\n\n\nI'm not aware of major differences. Both sets are mixtures of images\n from different experimental sets, so you should expect a variety.
\n
is also as misleading. Though, it hurt more when I asked her my question about vastly different pics from BBBC6 made with different staining and then later read that comment not here, but in an email notification and thought it was a response to my question. Although, I don't regret that, as those images are actually pretty useful on the contrary.
\n\n\n\n\nIndeed. I'm an organizer. The second stage (test) dataset will indeed\n have image sets that are experimentally distinct from anything seen in\n the training and validation (stage 1) datasets. The goal of this\n competition is generalizability and robustness so the only way to test\n this well is to be sure that the final test image sets are distinct\n relative to the training data.
\n
Again.. Basically, just go here https://www.kaggle.com/drannecarpenter/discussion to find all the misleading info you need to understand what I thought. Allen, your comment about how your model was overfit vs stage 2 data also contributed to my perception that that dataset is gonna be hell lotta different. As did the fact that train set contained black-on-white images not present in test set... Turns out in either one. Fools were those who tried to fit that data, especially those who did'nt invert colors. It definitely was a hit to perf.
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/47615#271047 all of this discussion generally makes one think that cancerous tissues are some images that are probably gonna be there in the test 2.
\n\nI definitely wasn't alone in being mislead about what the data is gonna be and I wasn't even the most crazy person https://www.kaggle.com/c/data-science-bowl-2018/discussion/53337 - at least I assumed my model should just take a lot of info from context and should know a learn features from different sample settings and should just work on any nuclei images, not eyeballs and yolks.
\n\nI expected different info stuffed into different channels, as specialized software produces such things, I expected different lighting conditions, for example fluorescence and backlighting, I expected cancerous tissues, binucleate stuff, vastly different zoom levels, some images with hugely packed density of nuclei, different sort of crap being overlayed over the nuclei, some light being reflected, different focus and sharpness, etc.
\n\nAs the solution phase is already over and I won't get any prize anyway, I'll share some things about my solution. It was somewhat like Anne and many others thought up with boundaries and masks. Except I didn't annotate boundaries, like most - that was not a very wise move, rly. Especially given the data quality. And one should not, of course, only ask the network to show where boundaries between nuclei are, as it's only going to be harder, again, especially given that the ground truth quality is far from pixel-level. One thing that came really obvious to me early on, again, also largely accounting for ground truth quality and the basic lack of precision in such a task - is that one shouldn't ask the net to annotate boundaries, but should instead focus on meaningful nuclei inner parts. So a mask that the net should generate should contain both - boundries and parts of the picture that are outside of nuclei, but the emphasis of loss weighting should be on mostly on touch zones and bounds that are close (say 5 pixels close) to that of any other nucleus (say weight of 1), boundaries (say, weight of 0.25) and inner part of the nuclei (say, weight of 0.2), with the parts of the picture outside of these zones being weighted at only, say, 0.02-0.01. And, because the annotations are not really all that precise, the inner parts of inner nuclei that are exactly adjecent to border are also weighted as unimportant if they're not crucial. They're crucial if, for example, a nuclei is small, or they are part of a thin stretch. We only disreguard whatever our network is going to produce for those pixels isn't going to hurt our understanding of this nucleuses bounds. One important way it should help is that at places where two nuclei meet we don't have pixel-precise annotations. So giving the net a wider range of pixels that it can label as a boundary that we don't penalise it for is better. And it won't hurt us by creating holes inside of a nucleus, or spreading it into two parts - we only allow this imprecision on pixels where it won't.
\n\nWhy we need to learn stuff outside of nuclei? Well, firstly, of course, if you really don't care about what it shows for the outside pixels, you should at least not apply weights to the outside zones, as otherwise you're forcing the net to learn that outside of nuclei it should produce a result same as inside and only the thin stretch of boundary matters. Which is extremely hard and confusing, especially, again, given the imprecision of ground truth info. And it forces your net to learn info which is hard to learn and which is going to massively hit it's ability to see boundaries, because if it doubts whether a pixel is a boundary, or if it's outside it may choose to opt for saying it's outside instead. But that's not really enough, it's better to make it learn that outside of nuclei it should produce same result as for boundaries, because annotations are not precise and sometimes what is known to us as a boundary from ground truth, actually lays outside of a nucleus and even some of the annotated insides also lay outside of actual nucleus. And when you label actual boundary as being inside, you mislead and confuse the net hurting its understanding of the subject. So you're telling it what stuff outside looks like at one place and annotate it same as bounds at other places.
\n\nI had quite an amount of frustration with the train set and BBBC6, especially with the train set, where some nuclei are missed, random pixel-wide holes in are present in anotations and hundreds (as far as I remember) nucleuses are merged into one that obviously aren't. In order for my model to ever work, I of course, had to have a decent precise dataset. So I went through each sample and fixed it. https://github.com/ibmua/data-science-bowl-2018-train-set First simply in a MS Paint-like KolourPaint app, but then when I saw that there's way too many mistakes I decided to make a special tool I now openly published. It's pretty laggy, but it's magic compared to other means. BBBC6 dataset has decent coverage and nice images on the other hand, but when it comes to boundaries, annotations seem quite insane. But I didn't bother annotating BBBC6, because I was running late. At the end, with all of the misleading info I wasn't able to make it on time and have an appropriate model, so I definitely don't regret failing to make that investment. Only fixed some small part. Processed into appropriate format it's now available at https://drive.google.com/file/d/16KVLYeKBkk_Y0W4Fra_2AhjmW-p6JjZu/view?usp=sharing
\n\nMy network was a U-net style network, also tried double-U-net, with a pretty equal amount of channels all around the scales (not widening as at deepens) with 3x3->3x3[grouped] residual blocks on all scales. I wasn't able to produce a really decent \"general nuclei discovery network\" by deadline, so I can't say for sure, but it seemed, like groups seemed especially highly beneficial for the generalizeable network - in the best one I had I used 16 groups for 64 channels. It's seemingly not that important for the non-generalizeable solution for stage 2 data, but for general \"any type\", especially including channel permutations data I guess it's important to, at least at the largest scale, process each channel with a single subnetwork - sharing weights. Then, maybe, combine results with a 1x1 conv. Adding resulting feature layers from all colors together also works, at least somewhat, but I think 1x1 is probably going to be much better.
\n\nOne more really important, I think, tip is to first train a inner nuclei map net and then repurpose it into a general map net. That saves time, the net is better and works better with the net it's based on. I mean using boundary map net as a pretrain. Then just making it learn that boundaries should be included into what's expected of it, not just inner parts of nuclei.
", "votes": 0, "replies": [] }, { "id": 313197, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-12T22:28:12.327000", "content": "One general reason to look for external data was also that there were very fiew colored images, they weren't well annotated and I had no clue how they should be annotated. I tried annotating the test set and as I saw stage 1 test's ground truth upon stage 1's end I was shocked that I was totally wrong about my guesses on all colored images and also some grayscale ones. Especially those that looked markedly different from any of the ones in the train set.
", "votes": 0, "replies": [] } ] }, { "id": 307465, "author_name": "theNickName", "author_url": "", "post_date": "2018-04-01T17:07:34.210000", "content": "Are our models expected to identify nucleuses on hardcore types of images like these? If direct answer is not possible, any indirect ones are appreciated. Like how desireable it is. \"Phalloidin\"-type image taken from your dataset https://data.broadinstitute.org/bbbc/BBBC006/
", "votes": 0, "replies": [ { "id": 308059, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-02T21:57:30.123000", "content": "We debated whether to include images that had relatively indirect labeling of nuclei and decided to not make the competition THAT hard. \"Indirect\" is maybe a bad word choice because many histology images use H&E staining which is not exactly a direct DNA stain. But anyway, generally speaking, to the best of my recollection, we only included images where the nucleus itself looked pretty stained relative to its surroundings.
", "votes": 1, "replies": [] }, { "id": 313192, "author_name": "", "author_url": "", "post_date": "2018-04-12T22:11:09.920000", "content": "", "votes": 0, "replies": [] }, { "id": 313196, "author_name": "", "author_url": "", "post_date": "2018-04-12T22:20:37.840000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 305209, "author_name": "Burhan ", "author_url": "", "post_date": "2018-03-28T15:41:50.090000", "content": "Hi Anne, \nWould the distribution of number of nuclei in each image be way different in stage 2 test (compared to the training set and stage 1 test) ?
", "votes": 0, "replies": [ { "id": 308040, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-02T20:53:49.617000", "content": "I'm not aware of major differences. Both sets are mixtures of images from different experimental sets, so you should expect a variety.
", "votes": 0, "replies": [] } ] }, { "id": 283610, "author_name": "Frank Ni", "author_url": "", "post_date": "2018-02-15T19:09:20.030000", "content": "It'll be very helpful if the following 2 things can happen :\n[1] Update us with a CLEANED up train set, get rid of all the problem images.\n[2] Increase the daily submit limit to 20 or 30 for each person, although not everyone will submit that many each day, so it shouldn't be too much of a burden to the server.
", "votes": 0, "replies": [ { "id": 283670, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-15T20:37:06.753000", "content": "Do you mind submitting this comment to a new, or relevant thread? It may receive more attention there. I personally do not control the rules about either of these.
", "votes": 0, "replies": [] } ] }, { "id": 282083, "author_name": "Andrea Rapuzzi", "author_url": "", "post_date": "2018-02-13T15:52:28.783000", "content": "Hi Anne, may we assume that we will also have a corresponding new training set in Phase2? \nThanks in Advance!!
", "votes": 0, "replies": [ { "id": 282212, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-13T19:08:06.070000", "content": "Nope. Unless you count the stage 1 test's annotations which will be provided at that point. See timeline page:
\n\n\n", "votes": 1, "replies": [] }, { "id": 282457, "author_name": "Akila Wajirasena", "author_url": "", "post_date": "2018-02-14T02:24:41.970000", "content": "April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set.
\n
Can we retrain the model after April 11, 2018 with the validation set?
", "votes": 7, "replies": [] }, { "id": 282604, "author_name": "Andrea Rapuzzi", "author_url": "", "post_date": "2018-02-14T08:34:12.193000", "content": "Thanks Anne!!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "272874": "The question has come up in various threads, so I will answer it here because I think it will affect the real-world utility of solutions you create. \n\nThe images to be used in the final test phase (aka, stage 2 test) on which you will be judged are generally from different experiments or cell types than those included in the stage 1 training and test data provided to you. **In other words, we generally tried to hold out entire image sets for the stage 2 test, rather than holding out just individual images.** \n \nSo generalizability is a key goal for your solution. I honestly don't recall at this point which image sets went into training vs stage 1 test vs stage 2 test but it was not systematic or intended to trick you (i.e., our goal is not to have the image types in stage 2 test be crazy different than training or stage 1 test, though I imagine the definition of \"crazy different\" might vary from person to person). Changes in stain colors are quite common in biology, for example, so you should expect images with different color schemes.", "284577": "What you call generalizability must be called transferrabilty, b/c you expect to transfer a model trained on some type of data to different data. Generalizability is ability to perform on the same data distribution. i.i.d. assumption about training/test data is the cornerstone of ML, and *automatic* transferrabilty you are anticipating does not exist. By automatic I mean no additional retraining or other supervised procedure to adjust distributions. The results will be random to degree of distribution difference.", "281306": "Can we know in advance the maximum size of the images of second stage? \nFor some architectures it is not trivial to work with images of different size.", "273906": "Thanks for the heads up on the test set possibilities! Stain colors are easy to generalize for but I was rather distressed when people said there were different, experimental conditions on the test set. It would be very difficult to build a model that could read things like phase contrast images without showing them it.", "312768": "Okay, so everyone who actually believed that \"generalizability is a key goal\", used lots of external data and heavy augmentation and expected colorful images much unlike those in the datasets is actually going to come at the very bottom.", "307465": "Are our models expected to identify nucleuses on hardcore types of images like these? If direct answer is not possible, any indirect ones are appreciated. Like how desireable it is. \"Phalloidin\"-type image taken from your dataset https://data.broadinstitute.org/bbbc/BBBC006/", "305209": "Hi Anne, \nWould the distribution of number of nuclei in each image be way different in stage 2 test (compared to the training set and stage 1 test) ?", "283610": "It'll be very helpful if the following 2 things can happen :\n[1] Update us with a CLEANED up train set, get rid of all the problem images.\n[2] Increase the daily submit limit to 20 or 30 for each person, although not everyone will submit that many each day, so it shouldn't be too much of a burden to the server.", "282083": "Hi Anne, may we assume that we will also have a corresponding new training set in Phase2? \nThanks in Advance!!" } }