I want to understand the purpose of the competition. I understand the rule is to find the largest overlap between my answer and test set. However, I have a few concerns from a biology point of view. From my experience of manually counting cells/nuclei as a graduate student, I feel the current training label may not generate beneficial model for the academic research community.
\n\nIt is most useful for a biologist to:
\n\nThe current label have these issues:
\n\nI hope you could at least let people know the current training data labeling is controversial. We cannot expect a biologist to be able to modify the code or re-train the model according to their liking. At the very least I hope you can give a prize incentive for people who train alternative models using labels different than what's in the given training data.
\n\nThank you for your time.
\n\nPreference:\nAbercrombie, M., M. L. Johnson 1946 Quantitative histology of Wallerian degeneration I. Nuclear population in rabbit sciatic nerve. J. Anat. Lond. vol. 80, pp. 37-50.
", "messages": [ { "id": "271931", "postDate": "01/21/2018 22:19:25", "content": "I want to understand the purpose of the competition. I understand the rule is to find the largest overlap between my answer and test set. However, I have a few concerns from a biology point of view. From my experience of manually counting cells/nuclei as a graduate student, I feel the current training label may not generate beneficial model for the academic research community.
\n\nIt is most useful for a biologist to:
\n\nThe current label have these issues:
\n\nI hope you could at least let people know the current training data labeling is controversial. We cannot expect a biologist to be able to modify the code or re-train the model according to their liking. At the very least I hope you can give a prize incentive for people who train alternative models using labels different than what's in the given training data.
\n\nThank you for your time.
\n\nPreference:\nAbercrombie, M., M. L. Johnson 1946 Quantitative histology of Wallerian degeneration I. Nuclear population in rabbit sciatic nerve. J. Anat. Lond. vol. 80, pp. 37-50.
", "rawMarkdown": "I want to understand the purpose of the competition. I understand the rule is to find the largest overlap between my answer and test set. However, I have a few concerns from a biology point of view. From my experience of manually counting cells/nuclei as a graduate student, I feel the current training label may not generate beneficial model for the academic research community.\n\nIt is most useful for a biologist to:\n\n 1. Count the number of nuclei \n 2. Find the average cross-sectional area of\n nuclei \n 3. Find the presence and ratio of abnormal nuclei, which are\n usually not roundish shape.\n\nThe current label have these issues:\n\n 2. Partially cut-off nuclei on all 4 edges are counted. This will\n overestimate the number of nuclei and underestimate the average\n nucleus size. First talked about by Abercrombie (1946),\n counting/sampling this way make you double count half nuclei and\n introduce systematic errors. Counting half nuclei will obviously\n lower the average nucleus size. A better way is only count 2 edges\n systematically, e.g. only count top and right edge nuclei.\n Alternative approach is to not count edge nuclei for more accurate\n size estimation.\n 3. There are smaller polygons inside of larger polygons. Again, this\n will overestimate nuclei number and underestimate nuclei size (ONUS).\n For fluorescent images, smaller circles inside larger circles should\n be ignored.\n 4. Multiple nuclei clumped together are counted as separate smaller\n nuclei. ONUS. The clumps are mostly likely one nuclei atrophy/broke\n down into smaller pieces. They should be counted as one small\n nucleus.\n\nI hope you could at least let people know the current training data labeling is controversial. We cannot expect a biologist to be able to modify the code or re-train the model according to their liking. At the very least I hope you can give a prize incentive for people who train alternative models using labels different than what's in the given training data.\n\nThank you for your time.\n\nPreference:\nAbercrombie, M., M. L. Johnson 1946 Quantitative histology of Wallerian degeneration I. Nuclear population in rabbit sciatic nerve. J. Anat. Lond. vol. 80, pp. 37-50.", "votes": null }, { "id": "272210", "postDate": "01/22/2018 14:00:12", "content": "Hello, I am happy to resolve the confusion.
\n\nThe most important thing to understand, which I believe will resolve all your well-thought-out concerns, is that the goal of the challenge is not to count nuclei. It is to identify nuclei and fragments of DNA (as long as they were not too obviously streaky debris but still had relatively roundish shapes), as the starting point for a workflow the biologist will choose. If their goal does happen to be to count nuclei, they would want to follow the filtering procedures you describe (this is relatively rare in the high-throughput case, because there are cheaper and easier ways to measure cell proliferation). But most often, their goal will be to use the nuclei as the starting point for a more complex analysis that often includes identifying cell boundaries using a separate stain visible in a different channel. For those who want to eliminate small fragments of DNA it is pretty trivial to filter by size or shape criteria. But some may want to include them because they want to specifically measure micronuclei. This is why we decided to include all DNA fragments in the annotations.
\n\nAs you can imagine, we did not want to call the challenge \"Find-the-nuclei-and-various-DNA-fragments challenge\" so we simplified a bit :)
\n\nYou can understand more about the motivations of the challenge in the competition video introduction here: https://www.youtube.com/watch?v=eHwkfhmJexs&feature=youtu.be
", "rawMarkdown": "Hello, I am happy to resolve the confusion. \n\nThe most important thing to understand, which I believe will resolve all your well-thought-out concerns, is that the goal of the challenge is **not** to count nuclei. It is to identify nuclei **and** fragments of DNA (as long as they were not too obviously streaky debris but still had relatively roundish shapes), as the starting point for a workflow the biologist will choose. If their goal does happen to be to count nuclei, they would want to follow the filtering procedures you describe (this is relatively rare in the high-throughput case, because there are cheaper and easier ways to measure cell proliferation). But most often, their goal will be to use the nuclei as the starting point for a more complex analysis that often includes identifying cell boundaries using a separate stain visible in a different channel. For those who want to eliminate small fragments of DNA it is pretty trivial to filter by size or shape criteria. But some may want to include them because they want to specifically measure micronuclei. This is why we decided to include all DNA fragments in the annotations.\n\nAs you can imagine, we did not want to call the challenge \"Find-the-nuclei-and-various-DNA-fragments challenge\" so we simplified a bit :)\n\nYou can understand more about the motivations of the challenge in the competition video introduction here: https://www.youtube.com/watch?v=eHwkfhmJexs&feature=youtu.be", "votes": null }, { "id": "272273", "postDate": "01/22/2018 17:35:32", "content": "Thanks for the clarification. Let me make sure I understand the goal here. Do you want to identify all areas with potential DNA in the images? If your goal is to faithfully get all areas and do post-processing later, then the current label and verification is probably what you want. Does this mean that the evaluation only cares about whether the masked pixels are present in the test label? So we can think of the test label as a single, large, non-overlapping mask? If the count doesn't matter, does it matter whether we have 10 or 2 masks? I feel the evaluation method does not reflect this fact. Or do you still want to separate the different nuclei into separate masks?
", "rawMarkdown": "Thanks for the clarification. Let me make sure I understand the goal here. **Do you want to identify all areas with potential DNA in the images?** If your goal is to faithfully get all areas and do post-processing later, then the current label and verification is probably what you want. Does this mean that the evaluation only cares about whether the masked pixels are present in the test label? So we can think of the test label as a single, large, non-overlapping mask? If the count doesn't matter, does it matter whether we have 10 or 2 masks? I feel the evaluation method does not reflect this fact. Or do you still want to separate the different nuclei into separate masks?", "votes": null } ], "comments": [ { "id": 272210, "author_name": "drannecarpenter", "author_url": "", "post_date": "01/22/2018 14:00:12", "content": "Hello, I am happy to resolve the confusion.
\n\nThe most important thing to understand, which I believe will resolve all your well-thought-out concerns, is that the goal of the challenge is not to count nuclei. It is to identify nuclei and fragments of DNA (as long as they were not too obviously streaky debris but still had relatively roundish shapes), as the starting point for a workflow the biologist will choose. If their goal does happen to be to count nuclei, they would want to follow the filtering procedures you describe (this is relatively rare in the high-throughput case, because there are cheaper and easier ways to measure cell proliferation). But most often, their goal will be to use the nuclei as the starting point for a more complex analysis that often includes identifying cell boundaries using a separate stain visible in a different channel. For those who want to eliminate small fragments of DNA it is pretty trivial to filter by size or shape criteria. But some may want to include them because they want to specifically measure micronuclei. This is why we decided to include all DNA fragments in the annotations.
\n\nAs you can imagine, we did not want to call the challenge \"Find-the-nuclei-and-various-DNA-fragments challenge\" so we simplified a bit :)
\n\nYou can understand more about the motivations of the challenge in the competition video introduction here: https://www.youtube.com/watch?v=eHwkfhmJexs&feature=youtu.be
", "votes": null, "replies": [ { "id": 272273, "author_name": "yangliu2", "author_url": "", "post_date": "01/22/2018 17:35:32", "content": "Thanks for the clarification. Let me make sure I understand the goal here. Do you want to identify all areas with potential DNA in the images? If your goal is to faithfully get all areas and do post-processing later, then the current label and verification is probably what you want. Does this mean that the evaluation only cares about whether the masked pixels are present in the test label? So we can think of the test label as a single, large, non-overlapping mask? If the count doesn't matter, does it matter whether we have 10 or 2 masks? I feel the evaluation method does not reflect this fact. Or do you still want to separate the different nuclei into separate masks?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "271931": "I want to understand the purpose of the competition. I understand the rule is to find the largest overlap between my answer and test set. However, I have a few concerns from a biology point of view. From my experience of manually counting cells/nuclei as a graduate student, I feel the current training label may not generate beneficial model for the academic research community.\n\nIt is most useful for a biologist to:\n\n 1. Count the number of nuclei \n 2. Find the average cross-sectional area of\n nuclei \n 3. Find the presence and ratio of abnormal nuclei, which are\n usually not roundish shape.\n\nThe current label have these issues:\n\n 2. Partially cut-off nuclei on all 4 edges are counted. This will\n overestimate the number of nuclei and underestimate the average\n nucleus size. First talked about by Abercrombie (1946),\n counting/sampling this way make you double count half nuclei and\n introduce systematic errors. Counting half nuclei will obviously\n lower the average nucleus size. A better way is only count 2 edges\n systematically, e.g. only count top and right edge nuclei.\n Alternative approach is to not count edge nuclei for more accurate\n size estimation.\n 3. There are smaller polygons inside of larger polygons. Again, this\n will overestimate nuclei number and underestimate nuclei size (ONUS).\n For fluorescent images, smaller circles inside larger circles should\n be ignored.\n 4. Multiple nuclei clumped together are counted as separate smaller\n nuclei. ONUS. The clumps are mostly likely one nuclei atrophy/broke\n down into smaller pieces. They should be counted as one small\n nucleus.\n\nI hope you could at least let people know the current training data labeling is controversial. We cannot expect a biologist to be able to modify the code or re-train the model according to their liking. At the very least I hope you can give a prize incentive for people who train alternative models using labels different than what's in the given training data.\n\nThank you for your time.\n\nPreference:\nAbercrombie, M., M. L. Johnson 1946 Quantitative histology of Wallerian degeneration I. Nuclear population in rabbit sciatic nerve. J. Anat. Lond. vol. 80, pp. 37-50.", "272210": "Hello, I am happy to resolve the confusion. \n\nThe most important thing to understand, which I believe will resolve all your well-thought-out concerns, is that the goal of the challenge is **not** to count nuclei. It is to identify nuclei **and** fragments of DNA (as long as they were not too obviously streaky debris but still had relatively roundish shapes), as the starting point for a workflow the biologist will choose. If their goal does happen to be to count nuclei, they would want to follow the filtering procedures you describe (this is relatively rare in the high-throughput case, because there are cheaper and easier ways to measure cell proliferation). But most often, their goal will be to use the nuclei as the starting point for a more complex analysis that often includes identifying cell boundaries using a separate stain visible in a different channel. For those who want to eliminate small fragments of DNA it is pretty trivial to filter by size or shape criteria. But some may want to include them because they want to specifically measure micronuclei. This is why we decided to include all DNA fragments in the annotations.\n\nAs you can imagine, we did not want to call the challenge \"Find-the-nuclei-and-various-DNA-fragments challenge\" so we simplified a bit :)\n\nYou can understand more about the motivations of the challenge in the competition video introduction here: https://www.youtube.com/watch?v=eHwkfhmJexs&feature=youtu.be", "272273": "Thanks for the clarification. Let me make sure I understand the goal here. **Do you want to identify all areas with potential DNA in the images?** If your goal is to faithfully get all areas and do post-processing later, then the current label and verification is probably what you want. Does this mean that the evaluation only cares about whether the masked pixels are present in the test label? So we can think of the test label as a single, large, non-overlapping mask? If the count doesn't matter, does it matter whether we have 10 or 2 masks? I feel the evaluation method does not reflect this fact. Or do you still want to separate the different nuclei into separate masks?" }, "source": "meta" }