{
  "id": 54309,
  "title": "Questions on stage1 test labels",
  "url": "/competitions/data-science-bowl-2018/discussion/54309",
  "author_name": "",
  "post_date": "2018-04-12T04:00:33.416403200Z",
  "votes": 2,
  "comment_count": 10,
  "views": 0,
  "content": "<p>I want to find out why is it that during the phase1, some of my images looked visually better but ended with \"much lower\" LB score.</p>\n\n<p>I also want to estimate the error of human annotation and machined learn algorithm.\n(e.g. a +0.01 in LB not necessarily mean you improve your algorithm because of ground truth noise). What is limit of +x% that we can claim there is statistical improvement in public LB?</p>\n\n<p>For the image below, this is ground truth correct? For example, there is dark purple oval surrounded by light purple. I thought only the dark purple is the nuclei, while the other light purple is not.</p>\n\n<p>Can any biologist confirm this?\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/312623/9095/pic2.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/312623/9096/pic1.png\" alt=\"enter image description here\"></p>",
  "messages": [
    {
      "id": "312623",
      "postDate": "04/12/2018 04:00:33",
      "content": "<p>I want to find out why is it that during the phase1, some of my images looked visually better but ended with \"much lower\" LB score.</p>\n\n<p>I also want to estimate the error of human annotation and machined learn algorithm.\n(e.g. a +0.01 in LB not necessarily mean you improve your algorithm because of ground truth noise). What is limit of +x% that we can claim there is statistical improvement in public LB?</p>\n\n<p>For the image below, this is ground truth correct? For example, there is dark purple oval surrounded by light purple. I thought only the dark purple is the nuclei, while the other light purple is not.</p>\n\n<p>Can any biologist confirm this?\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/312623/9095/pic2.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/312623/9096/pic1.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "I want to find out why is it that during the phase1, some of my images looked visually better but ended with \"much lower\" LB score.\n\nI also want to estimate the error of human annotation and machined learn algorithm.\n(e.g. a +0.01 in LB not necessarily mean you improve your algorithm because of ground truth noise). What is limit of +x% that we can claim there is statistical improvement in public LB?\n\nFor the image below, this is ground truth correct? For example, there is dark purple oval surrounded by light purple. I thought only the dark purple is the nuclei, while the other light purple is not.\n\nCan any biologist confirm this?\n  ![enter image description here][1]\n\n  ![enter image description here][2]\n  \n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/312623/9095/pic2.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/312623/9096/pic1.png",
      "votes": null
    },
    {
      "id": "312628",
      "postDate": "04/12/2018 04:15:17",
      "content": "<p>I check my results. The easiest nuclei are usually detected and well segmented. If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.</p>\n\n<p>There is a couple of missed annotation. Ensemble actually will have more detection (machine is better than human) and this may be one of the reason why ensemble perform \"worse on metric score\"</p>\n\n<p>There are some clear post processing tricks:</p>\n\n<ul>\n<li><p>when nuclei overlap, the overlapped region is sometime brighter. This is a good clue to decide if a confusing clump of ovals should be separated or not </p></li>\n<li><p>overlapped nuclei with over saturated value (255) are mostly labelled as single one. (They may be actually separate nuclei. Mask rcnn will separate them. So you may need to fuse them back to improve LB score, though they may be actually separate nuclei. Of course the labeling scheme may be different in stage2)</p></li>\n</ul>",
      "rawMarkdown": "I check my results. The easiest nuclei are usually detected and well segmented. If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.\n\nThere is a couple of missed annotation. Ensemble actually will have more detection (machine is better than human) and this may be one of the reason why ensemble perform \"worse on metric score\"\n\nThere are some clear post processing tricks:\n\n-  when nuclei overlap, the overlapped region is sometime brighter. This is a good clue to decide if a confusing clump of ovals should be separated or not \n\n- overlapped nuclei with over saturated value (255) are mostly labelled as single one. (They may be actually separate nuclei. Mask rcnn will separate them. So you may need to fuse them back to improve LB score, though they may be actually separate nuclei. Of course the labeling scheme may be different in stage2)",
      "votes": null
    },
    {
      "id": "312717",
      "postDate": "04/12/2018 07:17:14",
      "content": "<blockquote>\n  <p>If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.</p>\n</blockquote>\n\n<p>Great insight!  I wish I had thought of this before.  I observed that my best scores were obtained with submissions containing less nuclei, but did not di in it.</p>",
      "rawMarkdown": "&gt; If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.\n\nGreat insight!  I wish I had thought of this before.  I observed that my best scores were obtained with submissions containing less nuclei, but did not di in it.",
      "votes": null
    },
    {
      "id": "312996",
      "postDate": "04/12/2018 16:39:28",
      "content": "<p>Indeed, only the dark purple is the nucleus, while the light purple is not.</p>",
      "rawMarkdown": "Indeed, only the dark purple is the nucleus, while the light purple is not.",
      "votes": null
    },
    {
      "id": "312998",
      "postDate": "04/12/2018 16:41:34",
      "content": "<p>thanks! </p>",
      "rawMarkdown": "thanks!",
      "votes": null
    },
    {
      "id": "313019",
      "postDate": "04/12/2018 17:07:10",
      "content": "<p>I was curious about this picture. What are the orange regions? What are the light purple too?</p>",
      "rawMarkdown": "I was curious about this picture. What are the orange regions? What are the light purple too?",
      "votes": null
    },
    {
      "id": "313213",
      "postDate": "04/12/2018 23:35:19",
      "content": "<p>I actually don't know! Need a pathologist to answer, it's a tissue sample.</p>",
      "rawMarkdown": "I actually don't know! Need a pathologist to answer, it's a tissue sample.",
      "votes": null
    },
    {
      "id": "313219",
      "postDate": "04/13/2018 00:00:16",
      "content": "<p>@Anne Thanks for the answer. </p>",
      "rawMarkdown": "Anne Thanks for the answer.",
      "votes": null
    },
    {
      "id": "313831",
      "postDate": "04/13/2018 22:27:44",
      "content": "<p>@Heng, I agree with you. ( Btw, how did you embed your image in the comment. )</p>\n\n<p>@Anne Just visualised the ground truth of the stage 1 test images. Our neural net can detect the nuclei behind the debris. However, none of them are annotated by the biologist. Is it a human error or they're not nuclei?  (See the attached image)</p>",
      "rawMarkdown": "Heng, I agree with you. ( Btw, how did you embed your image in the comment. )\n\n@Anne Just visualised the ground truth of the stage 1 test images. Our neural net can detect the nuclei behind the debris. However, none of them are annotated by the biologist. Is it a human error or they're not nuclei?  (See the attached image)",
      "votes": null
    },
    {
      "id": "313864",
      "postDate": "04/14/2018 00:58:18",
      "content": "<p>i think such case are ambiguous.  There is one training image that has this fibre debris and the nuclei behind are not annotated.</p>\n\n<p>In  dataset like VOC, caltech pedestrian, some ground truth are marked as \"difficult\" or \"don't care\". During evaluation, groudth truth mak as \"difficult\" or \"don't care\" are not evaluated. Meaning that if they are detected by algorithm, it would not penalised as false positive error. If they are missed by algorithm, it would not be penalised as false negative error either.</p>",
      "rawMarkdown": "i think such case are ambiguous.  There is one training image that has this fibre debris and the nuclei behind are not annotated.\n\nIn  dataset like VOC, caltech pedestrian, some ground truth are marked as \"difficult\" or \"don't care\". During evaluation, groudth truth mak as \"difficult\" or \"don't care\" are not evaluated. Meaning that if they are detected by algorithm, it would not penalised as false positive error. If they are missed by algorithm, it would not be penalised as false negative error either.",
      "votes": null
    },
    {
      "id": "313865",
      "postDate": "04/14/2018 01:06:59",
      "content": "<p>I wouldn't worry about such cases, I don't think there are any of that sort that are scored in this stage. But it's cool your model did a decent job of it! </p>",
      "rawMarkdown": "I wouldn't worry about such cases, I don't think there are any of that sort that are scored in this stage. But it's cool your model did a decent job of it!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 312628,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "04/12/2018 04:15:17",
      "content": "<p>I check my results. The easiest nuclei are usually detected and well segmented. If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.</p>\n\n<p>There is a couple of missed annotation. Ensemble actually will have more detection (machine is better than human) and this may be one of the reason why ensemble perform \"worse on metric score\"</p>\n\n<p>There are some clear post processing tricks:</p>\n\n<ul>\n<li><p>when nuclei overlap, the overlapped region is sometime brighter. This is a good clue to decide if a confusing clump of ovals should be separated or not </p></li>\n<li><p>overlapped nuclei with over saturated value (255) are mostly labelled as single one. (They may be actually separate nuclei. Mask rcnn will separate them. So you may need to fuse them back to improve LB score, though they may be actually separate nuclei. Of course the labeling scheme may be different in stage2)</p></li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 312717,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "04/12/2018 07:17:14",
          "content": "<blockquote>\n  <p>If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.</p>\n</blockquote>\n\n<p>Great insight!  I wish I had thought of this before.  I observed that my best scores were obtained with submissions containing less nuclei, but did not di in it.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 312996,
      "author_name": "drannecarpenter",
      "author_url": "",
      "post_date": "04/12/2018 16:39:28",
      "content": "<p>Indeed, only the dark purple is the nucleus, while the light purple is not.</p>",
      "votes": null,
      "replies": [
        {
          "id": 312998,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "04/12/2018 16:41:34",
          "content": "<p>thanks! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 313019,
          "author_name": "pliptor",
          "author_url": "",
          "post_date": "04/12/2018 17:07:10",
          "content": "<p>I was curious about this picture. What are the orange regions? What are the light purple too?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 313213,
          "author_name": "drannecarpenter",
          "author_url": "",
          "post_date": "04/12/2018 23:35:19",
          "content": "<p>I actually don't know! Need a pathologist to answer, it's a tissue sample.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 313219,
          "author_name": "pliptor",
          "author_url": "",
          "post_date": "04/13/2018 00:00:16",
          "content": "<p>@Anne Thanks for the answer. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 313831,
      "author_name": "nicolefinnie",
      "author_url": "",
      "post_date": "04/13/2018 22:27:44",
      "content": "<p>@Heng, I agree with you. ( Btw, how did you embed your image in the comment. )</p>\n\n<p>@Anne Just visualised the ground truth of the stage 1 test images. Our neural net can detect the nuclei behind the debris. However, none of them are annotated by the biologist. Is it a human error or they're not nuclei?  (See the attached image)</p>",
      "votes": null,
      "replies": [
        {
          "id": 313864,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "04/14/2018 00:58:18",
          "content": "<p>i think such case are ambiguous.  There is one training image that has this fibre debris and the nuclei behind are not annotated.</p>\n\n<p>In  dataset like VOC, caltech pedestrian, some ground truth are marked as \"difficult\" or \"don't care\". During evaluation, groudth truth mak as \"difficult\" or \"don't care\" are not evaluated. Meaning that if they are detected by algorithm, it would not penalised as false positive error. If they are missed by algorithm, it would not be penalised as false negative error either.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 313865,
          "author_name": "drannecarpenter",
          "author_url": "",
          "post_date": "04/14/2018 01:06:59",
          "content": "<p>I wouldn't worry about such cases, I don't think there are any of that sort that are scored in this stage. But it's cool your model did a decent job of it! </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "312623": "I want to find out why is it that during the phase1, some of my images looked visually better but ended with \"much lower\" LB score.\n\nI also want to estimate the error of human annotation and machined learn algorithm.\n(e.g. a +0.01 in LB not necessarily mean you improve your algorithm because of ground truth noise). What is limit of +x% that we can claim there is statistical improvement in public LB?\n\nFor the image below, this is ground truth correct? For example, there is dark purple oval surrounded by light purple. I thought only the dark purple is the nuclei, while the other light purple is not.\n\nCan any biologist confirm this?\n  ![enter image description here][1]\n\n  ![enter image description here][2]\n  \n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/312623/9095/pic2.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/312623/9096/pic1.png",
    "312628": "I check my results. The easiest nuclei are usually detected and well segmented. If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.\n\nThere is a couple of missed annotation. Ensemble actually will have more detection (machine is better than human) and this may be one of the reason why ensemble perform \"worse on metric score\"\n\nThere are some clear post processing tricks:\n\n-  when nuclei overlap, the overlapped region is sometime brighter. This is a good clue to decide if a confusing clump of ovals should be separated or not \n\n- overlapped nuclei with over saturated value (255) are mostly labelled as single one. (They may be actually separate nuclei. Mask rcnn will separate them. So you may need to fuse them back to improve LB score, though they may be actually separate nuclei. Of course the labeling scheme may be different in stage2)",
    "312717": "&gt; If we detected new nuclei and their iou is not high, the score actually decreases. So actually it is better to detect less.\n\nGreat insight!  I wish I had thought of this before.  I observed that my best scores were obtained with submissions containing less nuclei, but did not di in it.",
    "312996": "Indeed, only the dark purple is the nucleus, while the light purple is not.",
    "312998": "thanks!",
    "313019": "I was curious about this picture. What are the orange regions? What are the light purple too?",
    "313213": "I actually don't know! Need a pathologist to answer, it's a tissue sample.",
    "313219": "Anne Thanks for the answer.",
    "313831": "Heng, I agree with you. ( Btw, how did you embed your image in the comment. )\n\n@Anne Just visualised the ground truth of the stage 1 test images. Our neural net can detect the nuclei behind the debris. However, none of them are annotated by the biologist. Is it a human error or they're not nuclei?  (See the attached image)",
    "313864": "i think such case are ambiguous.  There is one training image that has this fibre debris and the nuclei behind are not annotated.\n\nIn  dataset like VOC, caltech pedestrian, some ground truth are marked as \"difficult\" or \"don't care\". During evaluation, groudth truth mak as \"difficult\" or \"don't care\" are not evaluated. Meaning that if they are detected by algorithm, it would not penalised as false positive error. If they are missed by algorithm, it would not be penalised as false negative error either.",
    "313865": "I wouldn't worry about such cases, I don't think there are any of that sort that are scored in this stage. But it's cool your model did a decent job of it!"
  },
  "source": "meta"
}