I apologize if the nature of this post because it questions the very idea of this competition but I'm very concerned about the premises of it:
\n\nI don't really understand how this competition makes sense without having any measurement of human annotation?
\n\nThe process of annotating nuclei is very subjective: I've done it, seen it done by others and it's very variable. It depends wildly on the details of the microscopy technique used to obtain a dataset: calibration of illumination for fluorescence microscopy for instance, sensor sensitivity, depth of field etc.. not to mention sample preparation, or in general, the 'biological' part of the process.
\n\nOn top of that, you don't have a real ground truth. Take two 'experts' and ask them to annotate the same dataset, using your method for evaluation, what is the score of an annotated set if you use the other one as ground truth?
\n\nI'll go further and say: you don't become an expert at annotating nuclei. The reason for this is that you most likely don't have any sort of feedback to adjust your performance when it's bad and become better at it.\nBest case scenario (assuming you use only one technique in a very controlled way for a long time), you just become more consistent with your own subjective impression of where a nucleus is. You can't ever check how good your annotation really is. Maybe you can argue with another ‘expert' about what pixel is in or not but there is no objective way to decide. In other words the ground truth segmentation provided is not really a ground truth.
\n\n\"Ground truth is a term used in various fields to refer to information provided by direct observation (i.e. empirical evidence) as opposed to information provided by inference.\"
\n\nThis is all strictly inference here, made by different people and no metrics to measure their level of expertise.
\n\nIn another thread (Discussion #53134) someone mentions the same image annotated twice with two different number of nuclei.
\n\nThey said one picture was annotated twice and one person found 250 nuclei for an image and another person found 289 nuclei for the same image with a different contrast. I find it quite a big difference.
\n\nWe're gonna declare a winner who is gonna have marginally better performance than the #2 and #3 and #4 when the results are actually sensitive to which undergrad annotated which image.
\n\nI don't think you can get a very good (let alone state of the art) nucleus detector with a competition with terms so loosely defined.
\n\nFinally another implicit assumption made by the premises of this challenge is that \"microscopy data has to be a mess let a computers deal with it\".
\n\nI strongly believe that it would be more valuable to come up with strict guidelines to prepare these images and teach people how to calibrate their acquisition rather than trying to find a universal nucleus detector.
\n\nIf for each picture we had the scale (µm/pixel) the technique used (fluorescence microscopy, the fluorescent marker DAPI etc...) the tissue from which the sample is extracted (mouse liver?) etc... I feel like the nuclei detection would be dramatically better than trying to create a detector that deals with every possible situation.
", "messages": [ { "id": "308075", "postDate": "04/02/2018 22:54:18", "content": "I apologize if the nature of this post because it questions the very idea of this competition but I'm very concerned about the premises of it:
\n\nI don't really understand how this competition makes sense without having any measurement of human annotation?
\n\nThe process of annotating nuclei is very subjective: I've done it, seen it done by others and it's very variable. It depends wildly on the details of the microscopy technique used to obtain a dataset: calibration of illumination for fluorescence microscopy for instance, sensor sensitivity, depth of field etc.. not to mention sample preparation, or in general, the 'biological' part of the process.
\n\nOn top of that, you don't have a real ground truth. Take two 'experts' and ask them to annotate the same dataset, using your method for evaluation, what is the score of an annotated set if you use the other one as ground truth?
\n\nI'll go further and say: you don't become an expert at annotating nuclei. The reason for this is that you most likely don't have any sort of feedback to adjust your performance when it's bad and become better at it.\nBest case scenario (assuming you use only one technique in a very controlled way for a long time), you just become more consistent with your own subjective impression of where a nucleus is. You can't ever check how good your annotation really is. Maybe you can argue with another ‘expert' about what pixel is in or not but there is no objective way to decide. In other words the ground truth segmentation provided is not really a ground truth.
\n\n\"Ground truth is a term used in various fields to refer to information provided by direct observation (i.e. empirical evidence) as opposed to information provided by inference.\"
\n\nThis is all strictly inference here, made by different people and no metrics to measure their level of expertise.
\n\nIn another thread (Discussion #53134) someone mentions the same image annotated twice with two different number of nuclei.
\n\nThey said one picture was annotated twice and one person found 250 nuclei for an image and another person found 289 nuclei for the same image with a different contrast. I find it quite a big difference.
\n\nWe're gonna declare a winner who is gonna have marginally better performance than the #2 and #3 and #4 when the results are actually sensitive to which undergrad annotated which image.
\n\nI don't think you can get a very good (let alone state of the art) nucleus detector with a competition with terms so loosely defined.
\n\nFinally another implicit assumption made by the premises of this challenge is that \"microscopy data has to be a mess let a computers deal with it\".
\n\nI strongly believe that it would be more valuable to come up with strict guidelines to prepare these images and teach people how to calibrate their acquisition rather than trying to find a universal nucleus detector.
\n\nIf for each picture we had the scale (µm/pixel) the technique used (fluorescence microscopy, the fluorescent marker DAPI etc...) the tissue from which the sample is extracted (mouse liver?) etc... I feel like the nuclei detection would be dramatically better than trying to create a detector that deals with every possible situation.
", "rawMarkdown": "I apologize if the nature of this post because it questions the very idea of this competition but I'm very concerned about the premises of it:\n\nI don't really understand how this competition makes sense without having any measurement of human annotation?\n\nThe process of annotating nuclei is very subjective: I've done it, seen it done by others and it's very variable. It depends wildly on the details of the microscopy technique used to obtain a dataset: calibration of illumination for fluorescence microscopy for instance, sensor sensitivity, depth of field etc.. not to mention sample preparation, or in general, the 'biological' part of the process.\n\nOn top of that, you don't have a real ground truth. Take two 'experts' and ask them to annotate the same dataset, using your method for evaluation, what is the score of an annotated set if you use the other one as ground truth? \n\nI'll go further and say: you don't become an expert at annotating nuclei. The reason for this is that you most likely don't have any sort of feedback to adjust your performance when it's bad and become better at it.\nBest case scenario (assuming you use only one technique in a very controlled way for a long time), you just become more consistent with your own subjective impression of where a nucleus is. You can't ever check how good your annotation really is. Maybe you can argue with another ‘expert' about what pixel is in or not but there is no objective way to decide. In other words the ground truth segmentation provided is not really a ground truth.\n\n\"Ground truth is a term used in various fields to refer to information provided by direct observation (i.e. empirical evidence) as opposed to information provided by inference.\"\n\nThis is all strictly inference here, made by different people and no metrics to measure their level of expertise.\n\nIn another thread ([Discussion #53134](https://www.kaggle.com/c/data-science-bowl-2018/discussion/53134)) someone mentions the same image annotated twice with two different number of nuclei.\n\nThey said one picture was annotated twice and one person found 250 nuclei for an image and another person found 289 nuclei for the same image with a different contrast. I find it quite a big difference.\n\nWe're gonna declare a winner who is gonna have marginally better performance than the #2 and #3 and #4 when the results are actually sensitive to which undergrad annotated which image.\n\nI don't think you can get a very good (let alone state of the art) nucleus detector with a competition with terms so loosely defined.\n\nFinally another implicit assumption made by the premises of this challenge is that \"microscopy data has to be a mess let a computers deal with it\".\n\nI strongly believe that it would be more valuable to come up with strict guidelines to prepare these images and teach people how to calibrate their acquisition rather than trying to find a universal nucleus detector.\n\nIf for each picture we had the scale (µm/pixel) the technique used (fluorescence microscopy, the fluorescent marker DAPI etc...) the tissue from which the sample is extracted (mouse liver?) etc... I feel like the nuclei detection would be dramatically better than trying to create a detector that deals with every possible situation.", "votes": null }, { "id": "308115", "postDate": "04/03/2018 00:49:23", "content": "Label noise and label errors are \"real\" in application development. In fact data scientists or algorithm developers (like me) spent a great deal amount of time to clean up or think a way to handle label issues.
\n\nIn Kaggle Challenge, it is not just about designing algorithm. It is also about how you handle such label issues. Hence, the machine learning problem here is: \"Given a set of noisy labels with label noise= x, how to maximise your algorithm performance on a test set, which has ground truth label noise = y\".
\n\nIt is interesting to note that, in case of label inconsistency, the performance of machine learning is \"better\" than human annotation. Machine learning will learn the \"average annotation\", which will be more consistent and perform better in most cases (but not all cases). For example, learned algorithm perform better for the 2 leaked LB test images in my post.
\n\nIn summary,
\n\nI think it is all right that label noise is present in data science competition. The competitors will have the handled it.
\n\nI wonder if another way to solve this problem is have a fast learning neural network, that can be fine tuned with few annotated images for each new dataset. How many images is too many to annotate, and what would be an acceptable computation budget for the fine tuning?
\n\nThis would not work if the goal was to use a large amount of data already available online, but for specific labs it could be more feasible(any case where there are not too many data sources).
", "rawMarkdown": "I wonder if another way to solve this problem is have a fast learning neural network, that can be fine tuned with few annotated images for each new dataset. How many images is too many to annotate, and what would be an acceptable computation budget for the fine tuning?\n\nThis would not work if the goal was to use a large amount of data already available online, but for specific labs it could be more feasible(any case where there are not too many data sources).", "votes": null }, { "id": "308120", "postDate": "04/03/2018 01:25:24", "content": "Indeed, although such a solution doesn't work for the competition's rules, we expect that to be a practical approach for the real application. That is, if we cannot find a model that works well for most nuclei, our backup plan is to allow fine-tuning a network with a bit of annotation. IIRC, one can annotate ~100 nuclei in a minute.
", "rawMarkdown": "Indeed, although such a solution doesn't work for the competition's rules, we expect that to be a practical approach for the real application. That is, if we cannot find a model that works well for most nuclei, our backup plan is to allow fine-tuning a network with a bit of annotation. IIRC, one can annotate ~100 nuclei in a minute.", "votes": null }, { "id": "308121", "postDate": "04/03/2018 01:25:47", "content": "I wish I had access to emojis, because I would clap very loudly here.
", "rawMarkdown": "I wish I had access to emojis, because I would clap very loudly here.", "votes": null }, { "id": "308124", "postDate": "04/03/2018 01:30:12", "content": "I agree with most of what the original poster says!
\n\nI can only provide the following rationale: yes, the competition would be more consistent/fair if we had more perfect ground truth with carefully optimized image acquisition and experimental settings. However, such a competition would be useless towards our end goal of making a model that can identify nuclei in images as well as a biologist can (even if it's a biologist with different opinions and a brighter screen than yours).
\n\nI agree that in practical use it wouldn't be too much trouble for the biologist to state a typical nucleus diameter. Then again, it's probably also pretty easy to scan a few images to find nuclei in some rough way, then take the mode of their sizes and use that for the network. Same for color scheme, a biologist could click on some pixels indicating where nuclei are but that also seems like a solvable problem.
\n\nPlease note: to clarify, no undergraduates were harmed in the creation of this annotated data set.
", "rawMarkdown": "I agree with most of what the original poster says! \n\nI can only provide the following rationale: yes, the competition would be more consistent/fair if we had more perfect ground truth with carefully optimized image acquisition and experimental settings. However, such a competition would be useless towards our end goal of making a model that can identify nuclei in images as well as a biologist can (even if it's a biologist with different opinions and a brighter screen than yours). \n\nI agree that in practical use it wouldn't be too much trouble for the biologist to state a typical nucleus diameter. Then again, it's probably also pretty easy to scan a few images to find nuclei in some rough way, then take the mode of their sizes and use that for the network. Same for color scheme, a biologist could click on some pixels indicating where nuclei are but that also seems like a solvable problem.\n\n*Please note: to clarify, no undergraduates were harmed in the creation of this annotated data set.*", "votes": null }, { "id": "308192", "postDate": "04/03/2018 05:22:49", "content": "\n\n\nI think it is all right that label noise is present in data science competition. The competitors will have the handled it.
\n
Sure no dataset is perfect, but going through the discussions I feel like there is no obvious consensus about how to best annotate the nuclei, especially given the diversity of situations. so 'handling it' seems rather obscure to me.
\n\n\n\n\nIn real life applications, if you can come up with better ground truth, you should do it. But sometime if that was difficult, then one would have to deal with it.
\n
Well the thing is that 'dealing with it' is affecting the results of the competition.\nI saw in many discussion people talking about spurious detections (tiny false negatives) and holes in nuclei etc... and the organizers telling them what the intended annotation was but if there is no way to test that in the (vast) majority of these cases the original annotations followed the guidelines explained a posteriori, if you build an algorithm using these latest additions to the rules, you're not necessarily improving your score in the sense of the evaluation metrics while you're trying to follow these guidelines. So the best algorithm might not actually come first which is a bummer because you might not end up using it.
\n\nI feel like this could have been \"easily\" (SO MUCH work, I know, I know) avoided by having multiple people annotating the same images and figuring out where they can mostly agree.
", "rawMarkdown": "> I think it is all right that label noise is present in data science competition. The competitors will have the handled it.\n\nSure no dataset is perfect, but going through the discussions I feel like there is no obvious consensus about how to best annotate the nuclei, especially given the diversity of situations. so 'handling it' seems rather obscure to me.\n\n> In real life applications, if you can come up with better ground truth, you should do it. But sometime if that was difficult, then one would have to deal with it.\n\nWell the thing is that 'dealing with it' is affecting the results of the competition.\nI saw in many discussion people talking about spurious detections (tiny false negatives) and holes in nuclei etc... and the organizers telling them what the intended annotation was but if there is no way to test that in the (vast) majority of these cases the original annotations followed the guidelines explained a posteriori, if you build an algorithm using these latest additions to the rules, you're not necessarily improving your score in the sense of the evaluation metrics while you're trying to follow these guidelines. So the best algorithm might not actually come first which is a bummer because you might not end up using it.\n\nI feel like this could have been \"easily\" (SO MUCH work, I know, I know) avoided by having multiple people annotating the same images and figuring out where they can mostly agree.", "votes": null }, { "id": "308199", "postDate": "04/03/2018 05:38:42", "content": "I'm glad to learn that no undergraduates were harmed. Maybe just a little traumatized? :p\nThanks for your patience responding to my post. I really appreciate it.
\n\nTo summarize, my fear is that someone will come up with an algorithm declared 'the best algorithm to detect nuclei' and it will become this black box that everyone use without having a way to evaluate the performance of the algorithm in their own situation.
\n\nGiven how physically constrained a nucleus is (size, shape etc...) I think a semi-supervised application that learns from user input would have been more safe and maybe more powerful.
\n\nYou could imagine a website that stores different sets of images with nuclei examples and the parameters used to detect them. You could reuse these parameters or create your own.
\n\nThis way it would force people to check the performance of the algorithm in their situations.
\n\nBut sure this would limit the volume of images you could treat to some extent.
", "rawMarkdown": "I'm glad to learn that no undergraduates were harmed. Maybe just a little traumatized? :p\nThanks for your patience responding to my post. I really appreciate it.\n\nTo summarize, my fear is that someone will come up with an algorithm declared 'the best algorithm to detect nuclei' and it will become this black box that everyone use without having a way to evaluate the performance of the algorithm in their own situation.\n\nGiven how physically constrained a nucleus is (size, shape etc...) I think a semi-supervised application that learns from user input would have been more safe and maybe more powerful.\n\nYou could imagine a website that stores different sets of images with nuclei examples and the parameters used to detect them. You could reuse these parameters or create your own.\n\nThis way it would force people to check the performance of the algorithm in their situations.\n\nBut sure this would limit the volume of images you could treat to some extent.", "votes": null }, { "id": "308990", "postDate": "04/04/2018 13:20:36", "content": "We believe that better models will be produced by what we provided, vs. half the number of images but with a consensus annotation on each. In other words, the set we provide shows some variation from biologist to biologist and screen to screen but at least each one reflects someone's best attempt, which seemed better to us than providing half the number of images, with a consensus annotation for each.
", "rawMarkdown": "We believe that better models will be produced by what we provided, vs. half the number of images but with a consensus annotation on each. In other words, the set we provide shows some variation from biologist to biologist and screen to screen but at least each one reflects someone's best attempt, which seemed better to us than providing half the number of images, with a consensus annotation for each.", "votes": null }, { "id": "309426", "postDate": "04/05/2018 09:53:02", "content": "I am clapping for @Heng because I agree with all the points he made.
\n\nWe are kind of spoiled here at Kaggle because we usually get reasonably cleaned up and labelled data. In real life ML projects, one have to do both data cleaning as well as labelling before you even think about modelling. This just reflects a real world problem in my humble opinion.
", "rawMarkdown": "I am clapping for @Heng because I agree with all the points he made.\n\nWe are kind of spoiled here at Kaggle because we usually get reasonably cleaned up and labelled data. In real life ML projects, one have to do both data cleaning as well as labelling before you even think about modelling. This just reflects a real world problem in my humble opinion.", "votes": null }, { "id": "309429", "postDate": "04/05/2018 09:58:37", "content": "@Devin, the best solution might end up being what you described.
\n\nThanks @Anne for answering our questions so promptly in this competition in the various discussion threads.
", "rawMarkdown": "Devin, the best solution might end up being what you described.\n\nThanks @Anne for answering our questions so promptly in this competition in the various discussion threads.", "votes": null }, { "id": "309785", "postDate": "04/06/2018 01:34:11", "content": "Well yeah from the point of view of the people who just want to train themselves in a messy real-life situation sure, that's a good training. But from the point of view of the organizers and their sponsor they're not really setting themselves to obtain a very powerful detector. It's a definition problem: if there is no consensus on what is nucleus among the expert who label them, there is no reason to believe that ML algorithm will figure it out. In my opinion, everyone is just working on either improving the dataset (not really productive for this platform) or building algorithm 'as bad' as humans for this task. And again, there is no quantitive reason to believe that asking 10 experts to work together to annotate a dataset would solve this issue because they have no way to check if what they end up deciding is actually true.
", "rawMarkdown": "Well yeah from the point of view of the people who just want to train themselves in a messy real-life situation sure, that's a good training. But from the point of view of the organizers and their sponsor they're not really setting themselves to obtain a very powerful detector. It's a definition problem: if there is no consensus on what is nucleus among the expert who label them, there is no reason to believe that ML algorithm will figure it out. In my opinion, everyone is just working on either improving the dataset (not really productive for this platform) or building algorithm 'as bad' as humans for this task. And again, there is no quantitive reason to believe that asking 10 experts to work together to annotate a dataset would solve this issue because they have no way to check if what they end up deciding is actually true.", "votes": null }, { "id": "309822", "postDate": "04/06/2018 03:00:45", "content": "@Featureselection, you made a valid point.
", "rawMarkdown": "Featureselection, you made a valid point.", "votes": null }, { "id": "310215", "postDate": "04/06/2018 20:31:14", "content": "It's a common situation in supervised learning. Sometimes you can't assess the ground truth and you have human-labeled data. In this case, building a model on the top of it will create a substitute for the human, it automates the task of human labeling.
\n\nIt's like in a diagnosis case. Building a model that uses as label diagnosis given by doctors can be used to replace doctors diagnosis and it's only a proxy for the real cause of the symptoms. If it was possible to know the ground truth, we would be able to build a model to map symptoms to their real cause. Though the later is the desired situation always, when we only have the first it stills can be useful.
", "rawMarkdown": "It's a common situation in supervised learning. Sometimes you can't assess the ground truth and you have human-labeled data. In this case, building a model on the top of it will create a substitute for the human, it automates the task of human labeling. \n\nIt's like in a diagnosis case. Building a model that uses as label diagnosis given by doctors can be used to replace doctors diagnosis and it's only a proxy for the real cause of the symptoms. If it was possible to know the ground truth, we would be able to build a model to map symptoms to their real cause. Though the later is the desired situation always, when we only have the first it stills can be useful.", "votes": null } ], "comments": [ { "id": 308115, "author_name": "hengck23", "author_url": "", "post_date": "04/03/2018 00:49:23", "content": "Label noise and label errors are \"real\" in application development. In fact data scientists or algorithm developers (like me) spent a great deal amount of time to clean up or think a way to handle label issues.
\n\nIn Kaggle Challenge, it is not just about designing algorithm. It is also about how you handle such label issues. Hence, the machine learning problem here is: \"Given a set of noisy labels with label noise= x, how to maximise your algorithm performance on a test set, which has ground truth label noise = y\".
\n\nIt is interesting to note that, in case of label inconsistency, the performance of machine learning is \"better\" than human annotation. Machine learning will learn the \"average annotation\", which will be more consistent and perform better in most cases (but not all cases). For example, learned algorithm perform better for the 2 leaked LB test images in my post.
\n\nIn summary,
\n\nI think it is all right that label noise is present in data science competition. The competitors will have the handled it.
\n\nI wish I had access to emojis, because I would clap very loudly here.
", "votes": null, "replies": [] }, { "id": 308192, "author_name": "josarago", "author_url": "", "post_date": "04/03/2018 05:22:49", "content": "\n\n\nI think it is all right that label noise is present in data science competition. The competitors will have the handled it.
\n
Sure no dataset is perfect, but going through the discussions I feel like there is no obvious consensus about how to best annotate the nuclei, especially given the diversity of situations. so 'handling it' seems rather obscure to me.
\n\n\n\n\nIn real life applications, if you can come up with better ground truth, you should do it. But sometime if that was difficult, then one would have to deal with it.
\n
Well the thing is that 'dealing with it' is affecting the results of the competition.\nI saw in many discussion people talking about spurious detections (tiny false negatives) and holes in nuclei etc... and the organizers telling them what the intended annotation was but if there is no way to test that in the (vast) majority of these cases the original annotations followed the guidelines explained a posteriori, if you build an algorithm using these latest additions to the rules, you're not necessarily improving your score in the sense of the evaluation metrics while you're trying to follow these guidelines. So the best algorithm might not actually come first which is a bummer because you might not end up using it.
\n\nI feel like this could have been \"easily\" (SO MUCH work, I know, I know) avoided by having multiple people annotating the same images and figuring out where they can mostly agree.
", "votes": null, "replies": [] }, { "id": 308990, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/04/2018 13:20:36", "content": "We believe that better models will be produced by what we provided, vs. half the number of images but with a consensus annotation on each. In other words, the set we provide shows some variation from biologist to biologist and screen to screen but at least each one reflects someone's best attempt, which seemed better to us than providing half the number of images, with a consensus annotation for each.
", "votes": null, "replies": [] }, { "id": 309426, "author_name": "sheriytm", "author_url": "", "post_date": "04/05/2018 09:53:02", "content": "I am clapping for @Heng because I agree with all the points he made.
\n\nWe are kind of spoiled here at Kaggle because we usually get reasonably cleaned up and labelled data. In real life ML projects, one have to do both data cleaning as well as labelling before you even think about modelling. This just reflects a real world problem in my humble opinion.
", "votes": null, "replies": [] }, { "id": 309785, "author_name": "josarago", "author_url": "", "post_date": "04/06/2018 01:34:11", "content": "Well yeah from the point of view of the people who just want to train themselves in a messy real-life situation sure, that's a good training. But from the point of view of the organizers and their sponsor they're not really setting themselves to obtain a very powerful detector. It's a definition problem: if there is no consensus on what is nucleus among the expert who label them, there is no reason to believe that ML algorithm will figure it out. In my opinion, everyone is just working on either improving the dataset (not really productive for this platform) or building algorithm 'as bad' as humans for this task. And again, there is no quantitive reason to believe that asking 10 experts to work together to annotate a dataset would solve this issue because they have no way to check if what they end up deciding is actually true.
", "votes": null, "replies": [] } ] }, { "id": 308117, "author_name": "devinanzelmo", "author_url": "", "post_date": "04/03/2018 00:53:08", "content": "I wonder if another way to solve this problem is have a fast learning neural network, that can be fine tuned with few annotated images for each new dataset. How many images is too many to annotate, and what would be an acceptable computation budget for the fine tuning?
\n\nThis would not work if the goal was to use a large amount of data already available online, but for specific labs it could be more feasible(any case where there are not too many data sources).
", "votes": null, "replies": [ { "id": 308120, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/03/2018 01:25:24", "content": "Indeed, although such a solution doesn't work for the competition's rules, we expect that to be a practical approach for the real application. That is, if we cannot find a model that works well for most nuclei, our backup plan is to allow fine-tuning a network with a bit of annotation. IIRC, one can annotate ~100 nuclei in a minute.
", "votes": null, "replies": [] }, { "id": 309429, "author_name": "sheriytm", "author_url": "", "post_date": "04/05/2018 09:58:37", "content": "@Devin, the best solution might end up being what you described.
\n\nThanks @Anne for answering our questions so promptly in this competition in the various discussion threads.
", "votes": null, "replies": [] } ] }, { "id": 308124, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/03/2018 01:30:12", "content": "I agree with most of what the original poster says!
\n\nI can only provide the following rationale: yes, the competition would be more consistent/fair if we had more perfect ground truth with carefully optimized image acquisition and experimental settings. However, such a competition would be useless towards our end goal of making a model that can identify nuclei in images as well as a biologist can (even if it's a biologist with different opinions and a brighter screen than yours).
\n\nI agree that in practical use it wouldn't be too much trouble for the biologist to state a typical nucleus diameter. Then again, it's probably also pretty easy to scan a few images to find nuclei in some rough way, then take the mode of their sizes and use that for the network. Same for color scheme, a biologist could click on some pixels indicating where nuclei are but that also seems like a solvable problem.
\n\nPlease note: to clarify, no undergraduates were harmed in the creation of this annotated data set.
", "votes": null, "replies": [ { "id": 308199, "author_name": "josarago", "author_url": "", "post_date": "04/03/2018 05:38:42", "content": "I'm glad to learn that no undergraduates were harmed. Maybe just a little traumatized? :p\nThanks for your patience responding to my post. I really appreciate it.
\n\nTo summarize, my fear is that someone will come up with an algorithm declared 'the best algorithm to detect nuclei' and it will become this black box that everyone use without having a way to evaluate the performance of the algorithm in their own situation.
\n\nGiven how physically constrained a nucleus is (size, shape etc...) I think a semi-supervised application that learns from user input would have been more safe and maybe more powerful.
\n\nYou could imagine a website that stores different sets of images with nuclei examples and the parameters used to detect them. You could reuse these parameters or create your own.
\n\nThis way it would force people to check the performance of the algorithm in their situations.
\n\nBut sure this would limit the volume of images you could treat to some extent.
", "votes": null, "replies": [] }, { "id": 309822, "author_name": "sheriytm", "author_url": "", "post_date": "04/06/2018 03:00:45", "content": "@Featureselection, you made a valid point.
", "votes": null, "replies": [] } ] }, { "id": 310215, "author_name": "lgmoneda", "author_url": "", "post_date": "04/06/2018 20:31:14", "content": "It's a common situation in supervised learning. Sometimes you can't assess the ground truth and you have human-labeled data. In this case, building a model on the top of it will create a substitute for the human, it automates the task of human labeling.
\n\nIt's like in a diagnosis case. Building a model that uses as label diagnosis given by doctors can be used to replace doctors diagnosis and it's only a proxy for the real cause of the symptoms. If it was possible to know the ground truth, we would be able to build a model to map symptoms to their real cause. Though the later is the desired situation always, when we only have the first it stills can be useful.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "308075": "I apologize if the nature of this post because it questions the very idea of this competition but I'm very concerned about the premises of it:\n\nI don't really understand how this competition makes sense without having any measurement of human annotation?\n\nThe process of annotating nuclei is very subjective: I've done it, seen it done by others and it's very variable. It depends wildly on the details of the microscopy technique used to obtain a dataset: calibration of illumination for fluorescence microscopy for instance, sensor sensitivity, depth of field etc.. not to mention sample preparation, or in general, the 'biological' part of the process.\n\nOn top of that, you don't have a real ground truth. Take two 'experts' and ask them to annotate the same dataset, using your method for evaluation, what is the score of an annotated set if you use the other one as ground truth? \n\nI'll go further and say: you don't become an expert at annotating nuclei. The reason for this is that you most likely don't have any sort of feedback to adjust your performance when it's bad and become better at it.\nBest case scenario (assuming you use only one technique in a very controlled way for a long time), you just become more consistent with your own subjective impression of where a nucleus is. You can't ever check how good your annotation really is. Maybe you can argue with another ‘expert' about what pixel is in or not but there is no objective way to decide. In other words the ground truth segmentation provided is not really a ground truth.\n\n\"Ground truth is a term used in various fields to refer to information provided by direct observation (i.e. empirical evidence) as opposed to information provided by inference.\"\n\nThis is all strictly inference here, made by different people and no metrics to measure their level of expertise.\n\nIn another thread ([Discussion #53134](https://www.kaggle.com/c/data-science-bowl-2018/discussion/53134)) someone mentions the same image annotated twice with two different number of nuclei.\n\nThey said one picture was annotated twice and one person found 250 nuclei for an image and another person found 289 nuclei for the same image with a different contrast. I find it quite a big difference.\n\nWe're gonna declare a winner who is gonna have marginally better performance than the #2 and #3 and #4 when the results are actually sensitive to which undergrad annotated which image.\n\nI don't think you can get a very good (let alone state of the art) nucleus detector with a competition with terms so loosely defined.\n\nFinally another implicit assumption made by the premises of this challenge is that \"microscopy data has to be a mess let a computers deal with it\".\n\nI strongly believe that it would be more valuable to come up with strict guidelines to prepare these images and teach people how to calibrate their acquisition rather than trying to find a universal nucleus detector.\n\nIf for each picture we had the scale (µm/pixel) the technique used (fluorescence microscopy, the fluorescent marker DAPI etc...) the tissue from which the sample is extracted (mouse liver?) etc... I feel like the nuclei detection would be dramatically better than trying to create a detector that deals with every possible situation.", "308115": "Label noise and label errors are \"real\" in application development. In fact data scientists or algorithm developers (like me) spent a great deal amount of time to clean up or think a way to handle label issues.\n\nIn Kaggle Challenge, it is not just about designing algorithm. It is also about how you handle such label issues. Hence, the machine learning problem here is: \"Given a set of noisy labels with label noise= x, how to maximise your algorithm performance on a test set, which has ground truth label noise = y\".\n\nIt is interesting to note that, in case of label inconsistency, the performance of machine learning is \"better\" than human annotation. Machine learning will learn the \"average annotation\", which will be more consistent and perform better in most cases (but not all cases). For example, learned algorithm perform better for the 2 leaked LB test images in my post. \n\n\nIn summary,\n\n - I think it is all right that label noise is present in data science competition. The competitors will have the handled it.\n\n - In real life applications, if you can come up with better ground truth, you should do it. But sometime if that was difficult, then one would have to deal with it.", "308117": "I wonder if another way to solve this problem is have a fast learning neural network, that can be fine tuned with few annotated images for each new dataset. How many images is too many to annotate, and what would be an acceptable computation budget for the fine tuning?\n\nThis would not work if the goal was to use a large amount of data already available online, but for specific labs it could be more feasible(any case where there are not too many data sources).", "308120": "Indeed, although such a solution doesn't work for the competition's rules, we expect that to be a practical approach for the real application. That is, if we cannot find a model that works well for most nuclei, our backup plan is to allow fine-tuning a network with a bit of annotation. IIRC, one can annotate ~100 nuclei in a minute.", "308121": "I wish I had access to emojis, because I would clap very loudly here.", "308124": "I agree with most of what the original poster says! \n\nI can only provide the following rationale: yes, the competition would be more consistent/fair if we had more perfect ground truth with carefully optimized image acquisition and experimental settings. However, such a competition would be useless towards our end goal of making a model that can identify nuclei in images as well as a biologist can (even if it's a biologist with different opinions and a brighter screen than yours). \n\nI agree that in practical use it wouldn't be too much trouble for the biologist to state a typical nucleus diameter. Then again, it's probably also pretty easy to scan a few images to find nuclei in some rough way, then take the mode of their sizes and use that for the network. Same for color scheme, a biologist could click on some pixels indicating where nuclei are but that also seems like a solvable problem.\n\n*Please note: to clarify, no undergraduates were harmed in the creation of this annotated data set.*", "308192": "> I think it is all right that label noise is present in data science competition. The competitors will have the handled it.\n\nSure no dataset is perfect, but going through the discussions I feel like there is no obvious consensus about how to best annotate the nuclei, especially given the diversity of situations. so 'handling it' seems rather obscure to me.\n\n> In real life applications, if you can come up with better ground truth, you should do it. But sometime if that was difficult, then one would have to deal with it.\n\nWell the thing is that 'dealing with it' is affecting the results of the competition.\nI saw in many discussion people talking about spurious detections (tiny false negatives) and holes in nuclei etc... and the organizers telling them what the intended annotation was but if there is no way to test that in the (vast) majority of these cases the original annotations followed the guidelines explained a posteriori, if you build an algorithm using these latest additions to the rules, you're not necessarily improving your score in the sense of the evaluation metrics while you're trying to follow these guidelines. So the best algorithm might not actually come first which is a bummer because you might not end up using it.\n\nI feel like this could have been \"easily\" (SO MUCH work, I know, I know) avoided by having multiple people annotating the same images and figuring out where they can mostly agree.", "308199": "I'm glad to learn that no undergraduates were harmed. Maybe just a little traumatized? :p\nThanks for your patience responding to my post. I really appreciate it.\n\nTo summarize, my fear is that someone will come up with an algorithm declared 'the best algorithm to detect nuclei' and it will become this black box that everyone use without having a way to evaluate the performance of the algorithm in their own situation.\n\nGiven how physically constrained a nucleus is (size, shape etc...) I think a semi-supervised application that learns from user input would have been more safe and maybe more powerful.\n\nYou could imagine a website that stores different sets of images with nuclei examples and the parameters used to detect them. You could reuse these parameters or create your own.\n\nThis way it would force people to check the performance of the algorithm in their situations.\n\nBut sure this would limit the volume of images you could treat to some extent.", "308990": "We believe that better models will be produced by what we provided, vs. half the number of images but with a consensus annotation on each. In other words, the set we provide shows some variation from biologist to biologist and screen to screen but at least each one reflects someone's best attempt, which seemed better to us than providing half the number of images, with a consensus annotation for each.", "309426": "I am clapping for @Heng because I agree with all the points he made.\n\nWe are kind of spoiled here at Kaggle because we usually get reasonably cleaned up and labelled data. In real life ML projects, one have to do both data cleaning as well as labelling before you even think about modelling. This just reflects a real world problem in my humble opinion.", "309429": "Devin, the best solution might end up being what you described.\n\nThanks @Anne for answering our questions so promptly in this competition in the various discussion threads.", "309785": "Well yeah from the point of view of the people who just want to train themselves in a messy real-life situation sure, that's a good training. But from the point of view of the organizers and their sponsor they're not really setting themselves to obtain a very powerful detector. It's a definition problem: if there is no consensus on what is nucleus among the expert who label them, there is no reason to believe that ML algorithm will figure it out. In my opinion, everyone is just working on either improving the dataset (not really productive for this platform) or building algorithm 'as bad' as humans for this task. And again, there is no quantitive reason to believe that asking 10 experts to work together to annotate a dataset would solve this issue because they have no way to check if what they end up deciding is actually true.", "309822": "Featureselection, you made a valid point.", "310215": "It's a common situation in supervised learning. Sometimes you can't assess the ground truth and you have human-labeled data. In this case, building a model on the top of it will create a substitute for the human, it automates the task of human labeling. \n\nIt's like in a diagnosis case. Building a model that uses as label diagnosis given by doctors can be used to replace doctors diagnosis and it's only a proxy for the real cause of the symptoms. If it was possible to know the ground truth, we would be able to build a model to map symptoms to their real cause. Though the later is the desired situation always, when we only have the first it stills can be useful." }, "source": "meta" }