{
  "id": 53134,
  "title": "data leak ???",
  "url": "/competitions/data-science-bowl-2018/discussion/53134",
  "author_name": "",
  "post_date": "2018-03-27T13:08:11.519144900Z",
  "votes": 31,
  "comment_count": 20,
  "views": 0,
  "content": "<p>same images ????</p>\n\n<p>LB test:  697a05c6fe4a07c601d46da80885645ad574ea19b47ee795ccff216c9f1f1808</p>\n\n<p>train:       0287e7ee5b007c91ae2bd7628d09735e70496bc6127ecb7f3dd043e04ce37426</p>\n\n<ul>\n<li><p>but submission of the ground-truth label gives lower score.</p>\n\n<p>This proves that the annotation is inconsistent and your solution may need to handle this.</p></li>\n<li><p>also the contrast has been changed. hence the some LB test1 set may be modified version of the train set (e.g. rescale, or change contast).</p></li>\n</ul>",
  "messages": [
    {
      "id": "304363",
      "postDate": "03/27/2018 13:08:11",
      "content": "<p>same images ????</p>\n\n<p>LB test:  697a05c6fe4a07c601d46da80885645ad574ea19b47ee795ccff216c9f1f1808</p>\n\n<p>train:       0287e7ee5b007c91ae2bd7628d09735e70496bc6127ecb7f3dd043e04ce37426</p>\n\n<ul>\n<li><p>but submission of the ground-truth label gives lower score.</p>\n\n<p>This proves that the annotation is inconsistent and your solution may need to handle this.</p></li>\n<li><p>also the contrast has been changed. hence the some LB test1 set may be modified version of the train set (e.g. rescale, or change contast).</p></li>\n</ul>",
      "rawMarkdown": "same images ????\n\nLB test:  697a05c6fe4a07c601d46da80885645ad574ea19b47ee795ccff216c9f1f1808\n\ntrain:       0287e7ee5b007c91ae2bd7628d09735e70496bc6127ecb7f3dd043e04ce37426\n\n\n - but submission of the ground-truth label gives lower score.\n\n   This proves that the annotation is inconsistent and your solution may need to handle this.\n\n - also the contrast has been changed. hence the some LB test1 set may be modified version of the train set (e.g. rescale, or change contast).",
      "votes": null
    },
    {
      "id": "304369",
      "postDate": "03/27/2018 13:23:53",
      "content": "<p>interesting</p>",
      "rawMarkdown": "interesting",
      "votes": null
    },
    {
      "id": "304432",
      "postDate": "03/27/2018 15:52:52",
      "content": "<p>Crud! Wish I had known earlier. Also, in train cb4d &amp; 20b2 are similar, but annotations not.</p>",
      "rawMarkdown": "Crud! Wish I had known earlier. Also, in train cb4d &amp; 20b2 are similar, but annotations not.",
      "votes": null
    },
    {
      "id": "304437",
      "postDate": "03/27/2018 16:07:34",
      "content": "<p>They are also the same sample at different contrasts.</p>",
      "rawMarkdown": "They are also the same sample at different contrasts.",
      "votes": null
    },
    {
      "id": "304442",
      "postDate": "03/27/2018 16:12:33",
      "content": "<p>It almost seems that from Heng's and Glimmungs' examples that performing some form of contrast normalization may actually hurt score.</p>",
      "rawMarkdown": "It almost seems that from Heng's and Glimmungs' examples that performing some form of contrast normalization may actually hurt score.",
      "votes": null
    },
    {
      "id": "304503",
      "postDate": "03/27/2018 17:31:27",
      "content": "<p>Good find !! I assume the annotations were done based on what was visible in human eye. Contrast/color/brightness augmentation may hurt the algorithm as the ground truth masks will not be the same at that time. However, little augmentation may help as we might expect some similar images with contrast/color/brightness variation in stage 2.</p>",
      "rawMarkdown": "Good find !! I assume the annotations were done based on what was visible in human eye. Contrast/color/brightness augmentation may hurt the algorithm as the ground truth masks will not be the same at that time. However, little augmentation may help as we might expect some similar images with contrast/color/brightness variation in stage 2.",
      "votes": null
    },
    {
      "id": "304748",
      "postDate": "03/28/2018 01:08:08",
      "content": "<p>LOL, even the number of detected masks is 250 for one image and 289 for the other!</p>",
      "rawMarkdown": "LOL, even the number of detected masks is 250 for one image and 289 for the other!",
      "votes": null
    },
    {
      "id": "305128",
      "postDate": "03/28/2018 14:12:16",
      "content": "<p>Interesting finding, thanks for posting.</p>",
      "rawMarkdown": "Interesting finding, thanks for posting.",
      "votes": null
    },
    {
      "id": "307574",
      "postDate": "04/02/2018 03:10:42",
      "content": "<p>Another leak:</p>\n\n<p>train: 7c318172e976ae5a962c9c7a4e9fe46d7fb985765ddd3a3e2108e893a90b92b2</p>\n\n<p>test: 0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732</p>",
      "rawMarkdown": "Another leak:\n\n\ntrain: 7c318172e976ae5a962c9c7a4e9fe46d7fb985765ddd3a3e2108e893a90b92b2\n\n\ntest: 0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732",
      "votes": null
    },
    {
      "id": "307576",
      "postDate": "04/02/2018 03:14:30",
      "content": "<p>thanks!</p>",
      "rawMarkdown": "thanks!",
      "votes": null
    },
    {
      "id": "307735",
      "postDate": "04/02/2018 10:35:19",
      "content": "<p>Take your best submission. Include these 2 leaks 697a, 0a84 (copy/paste from matching train identifiers). Submit and LB is not better? It means annotations on test for the same 2 images are different from train.</p>",
      "rawMarkdown": "Take your best submission. Include these 2 leaks 697a, 0a84 (copy/paste from matching train identifiers). Submit and LB is not better? It means annotations on test for the same 2 images are different from train.",
      "votes": null
    },
    {
      "id": "307736",
      "postDate": "04/02/2018 10:39:08",
      "content": "<p>697a is not better (i lost 0.002). I didn't try 0a84</p>",
      "rawMarkdown": "697a is not better (i lost 0.002). I didn't try 0a84",
      "votes": null
    },
    {
      "id": "307737",
      "postDate": "04/02/2018 10:40:42",
      "content": "<p>Tried both, I've lost 0.005.\nAfter 04/11, we're allowed to re-train (but not modify model logic) with stage1 release. I think we should drop these 2 images in training as it would hurt learning (same image with different annotations).</p>",
      "rawMarkdown": "Tried both, I've lost 0.005.\nAfter 04/11, we're allowed to re-train (but not modify model logic) with stage1 release. I think we should drop these 2 images in training as it would hurt learning (same image with different annotations).",
      "votes": null
    },
    {
      "id": "307738",
      "postDate": "04/02/2018 10:48:20",
      "content": "<p>this means that test and train annotation are different.</p>\n\n<p>In fact, I hand annotate some train data and compare it with given ground truth. I think human error is quite large. e.g.</p>\n\n<ul>\n<li><p>for cases where we have same detection ( a nuclei is detected if iou &gt;0.5) with zero miss and false negative, human annotation error can be in the range of 0.01 (depends on the size of the nuclei). This is because we perceive \"outline\" of the nuclei differently, resulting in about one to two pixel shift.</p></li>\n<li><p>in some cases, the detection is different. E.g. I disagree with the ground truth if faint dark nuclei should be marked or not. Other cases includes those small fragmented  (micro-nucleus) and also those at the boundary.</p></li>\n</ul>\n\n<p>in summary, human error is large! </p>",
      "rawMarkdown": "this means that test and train annotation are different.\n\nIn fact, I hand annotate some train data and compare it with given ground truth. I think human error is quite large. e.g.\n \n- for cases where we have same detection ( a nuclei is detected if iou &gt;0.5) with zero miss and false negative, human annotation error can be in the range of 0.01 (depends on the size of the nuclei). This is because we perceive \"outline\" of the nuclei differently, resulting in about one to two pixel shift.\n\n- in some cases, the detection is different. E.g. I disagree with the ground truth if faint dark nuclei should be marked or not. Other cases includes those small fragmented  (micro-nucleus) and also those at the boundary.\n\nin summary, human error is large!",
      "votes": null
    },
    {
      "id": "307840",
      "postDate": "04/02/2018 15:25:58",
      "content": "<p>amazing Thanks you</p>",
      "rawMarkdown": "amazing Thanks you",
      "votes": null
    },
    {
      "id": "308051",
      "postDate": "04/02/2018 21:45:44",
      "content": "<blockquote>\n  <p>in summary, human error is large!</p>\n</blockquote>\n\n<p>Yes, yes it probably is! </p>\n\n<p>The biggest source of variation (at least for a case like this) is likely how bright the annotator's screen was during annotation. We considered carefully how to handle this issue. The choices were to carefully calibrate all the screens to be identical - this was practically impossible because we were not just on different tablets/laptops/spare monitors but also because we each were subjected to different lighting conditions. So we decided instead to let the variation reflect the 'natural' variation real biologists would have when examining their images in the wild. It's easy to see how such a situation would yield different annotations for the same images. Sorry that it's not a satisfying answer, but it's what we've got to work with!</p>",
      "rawMarkdown": "&gt; in summary, human error is large!\n\nYes, yes it probably is! \n\nThe biggest source of variation (at least for a case like this) is likely how bright the annotator's screen was during annotation. We considered carefully how to handle this issue. The choices were to carefully calibrate all the screens to be identical - this was practically impossible because we were not just on different tablets/laptops/spare monitors but also because we each were subjected to different lighting conditions. So we decided instead to let the variation reflect the 'natural' variation real biologists would have when examining their images in the wild. It's easy to see how such a situation would yield different annotations for the same images. Sorry that it's not a satisfying answer, but it's what we've got to work with!",
      "votes": null
    },
    {
      "id": "308219",
      "postDate": "04/03/2018 06:43:22",
      "content": "<p>Anne, thank you for your answer, but the question here might be why did you decide to include same images into train and test sets on the first place? Because this would inevitably harm the algorithm performance - was it to make sure there are no 'perfect' solutions?</p>",
      "rawMarkdown": "Anne, thank you for your answer, but the question here might be why did you decide to include same images into train and test sets on the first place? Because this would inevitably harm the algorithm performance - was it to make sure there are no 'perfect' solutions?",
      "votes": null
    },
    {
      "id": "308987",
      "postDate": "04/04/2018 13:16:57",
      "content": "<p>Pretty sure it was unintentional.</p>",
      "rawMarkdown": "Pretty sure it was unintentional.",
      "votes": null
    },
    {
      "id": "309748",
      "postDate": "04/05/2018 22:42:09",
      "content": "<p>@Heng thanks for posting this.</p>\n\n<p>@MPWARE Team, removing those 2 images from the training data hurt my scores by 0.003, any thoughts?</p>",
      "rawMarkdown": "Heng thanks for posting this.\n\n@MPWARE Team, removing those 2 images from the training data hurt my scores by 0.003, any thoughts?",
      "votes": null
    },
    {
      "id": "309809",
      "postDate": "04/06/2018 02:30:31",
      "content": "<blockquote>\n  <blockquote>\n    <p>removing those 2 images from the training data hurt my scores by 0.003</p>\n  </blockquote>\n</blockquote>\n\n<p>so it means that:</p>\n\n<ul>\n<li><p>training with inaccurate labels (within certain error) is better than training without them at all</p></li>\n<li><p>prediction from trained model is better than different human annotation</p></li>\n</ul>",
      "rawMarkdown": "&gt;&gt;removing those 2 images from the training data hurt my scores by 0.003\n\nso it means that:\n\n-  training with inaccurate labels (within certain error) is better than training without them at all\n\n- prediction from trained model is better than different human annotation",
      "votes": null
    },
    {
      "id": "309888",
      "postDate": "04/06/2018 05:49:46",
      "content": "<p>Thanks @Heng that is what I inferred from it but wanted to know what others think.</p>",
      "rawMarkdown": "Thanks @Heng that is what I inferred from it but wanted to know what others think.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 304369,
      "author_name": "wayne999",
      "author_url": "",
      "post_date": "03/27/2018 13:23:53",
      "content": "<p>interesting</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 304432,
      "author_name": "glimmung",
      "author_url": "",
      "post_date": "03/27/2018 15:52:52",
      "content": "<p>Crud! Wish I had known earlier. Also, in train cb4d &amp; 20b2 are similar, but annotations not.</p>",
      "votes": null,
      "replies": [
        {
          "id": 304437,
          "author_name": "pliptor",
          "author_url": "",
          "post_date": "03/27/2018 16:07:34",
          "content": "<p>They are also the same sample at different contrasts.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 304748,
          "author_name": "thenickname",
          "author_url": "",
          "post_date": "03/28/2018 01:08:08",
          "content": "<p>LOL, even the number of detected masks is 250 for one image and 289 for the other!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 304442,
      "author_name": "pliptor",
      "author_url": "",
      "post_date": "03/27/2018 16:12:33",
      "content": "<p>It almost seems that from Heng's and Glimmungs' examples that performing some form of contrast normalization may actually hurt score.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 304503,
      "author_name": "sgalib",
      "author_url": "",
      "post_date": "03/27/2018 17:31:27",
      "content": "<p>Good find !! I assume the annotations were done based on what was visible in human eye. Contrast/color/brightness augmentation may hurt the algorithm as the ground truth masks will not be the same at that time. However, little augmentation may help as we might expect some similar images with contrast/color/brightness variation in stage 2.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 305128,
      "author_name": "hel111",
      "author_url": "",
      "post_date": "03/28/2018 14:12:16",
      "content": "<p>Interesting finding, thanks for posting.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 307574,
      "author_name": "spnrpa",
      "author_url": "",
      "post_date": "04/02/2018 03:10:42",
      "content": "<p>Another leak:</p>\n\n<p>train: 7c318172e976ae5a962c9c7a4e9fe46d7fb985765ddd3a3e2108e893a90b92b2</p>\n\n<p>test: 0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732</p>",
      "votes": null,
      "replies": [
        {
          "id": 307576,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "04/02/2018 03:14:30",
          "content": "<p>thanks!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 307735,
          "author_name": "mpware",
          "author_url": "",
          "post_date": "04/02/2018 10:35:19",
          "content": "<p>Take your best submission. Include these 2 leaks 697a, 0a84 (copy/paste from matching train identifiers). Submit and LB is not better? It means annotations on test for the same 2 images are different from train.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 307736,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "04/02/2018 10:39:08",
          "content": "<p>697a is not better (i lost 0.002). I didn't try 0a84</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 307737,
          "author_name": "mpware",
          "author_url": "",
          "post_date": "04/02/2018 10:40:42",
          "content": "<p>Tried both, I've lost 0.005.\nAfter 04/11, we're allowed to re-train (but not modify model logic) with stage1 release. I think we should drop these 2 images in training as it would hurt learning (same image with different annotations).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 307738,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "04/02/2018 10:48:20",
          "content": "<p>this means that test and train annotation are different.</p>\n\n<p>In fact, I hand annotate some train data and compare it with given ground truth. I think human error is quite large. e.g.</p>\n\n<ul>\n<li><p>for cases where we have same detection ( a nuclei is detected if iou &gt;0.5) with zero miss and false negative, human annotation error can be in the range of 0.01 (depends on the size of the nuclei). This is because we perceive \"outline\" of the nuclei differently, resulting in about one to two pixel shift.</p></li>\n<li><p>in some cases, the detection is different. E.g. I disagree with the ground truth if faint dark nuclei should be marked or not. Other cases includes those small fragmented  (micro-nucleus) and also those at the boundary.</p></li>\n</ul>\n\n<p>in summary, human error is large! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 308051,
          "author_name": "drannecarpenter",
          "author_url": "",
          "post_date": "04/02/2018 21:45:44",
          "content": "<blockquote>\n  <p>in summary, human error is large!</p>\n</blockquote>\n\n<p>Yes, yes it probably is! </p>\n\n<p>The biggest source of variation (at least for a case like this) is likely how bright the annotator's screen was during annotation. We considered carefully how to handle this issue. The choices were to carefully calibrate all the screens to be identical - this was practically impossible because we were not just on different tablets/laptops/spare monitors but also because we each were subjected to different lighting conditions. So we decided instead to let the variation reflect the 'natural' variation real biologists would have when examining their images in the wild. It's easy to see how such a situation would yield different annotations for the same images. Sorry that it's not a satisfying answer, but it's what we've got to work with!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 308219,
          "author_name": "skyfallen",
          "author_url": "",
          "post_date": "04/03/2018 06:43:22",
          "content": "<p>Anne, thank you for your answer, but the question here might be why did you decide to include same images into train and test sets on the first place? Because this would inevitably harm the algorithm performance - was it to make sure there are no 'perfect' solutions?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 308987,
          "author_name": "drannecarpenter",
          "author_url": "",
          "post_date": "04/04/2018 13:16:57",
          "content": "<p>Pretty sure it was unintentional.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 309748,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "04/05/2018 22:42:09",
          "content": "<p>@Heng thanks for posting this.</p>\n\n<p>@MPWARE Team, removing those 2 images from the training data hurt my scores by 0.003, any thoughts?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 309809,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "04/06/2018 02:30:31",
          "content": "<blockquote>\n  <blockquote>\n    <p>removing those 2 images from the training data hurt my scores by 0.003</p>\n  </blockquote>\n</blockquote>\n\n<p>so it means that:</p>\n\n<ul>\n<li><p>training with inaccurate labels (within certain error) is better than training without them at all</p></li>\n<li><p>prediction from trained model is better than different human annotation</p></li>\n</ul>",
          "votes": null,
          "replies": []
        },
        {
          "id": 309888,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "04/06/2018 05:49:46",
          "content": "<p>Thanks @Heng that is what I inferred from it but wanted to know what others think.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 307840,
      "author_name": "uzzawal",
      "author_url": "",
      "post_date": "04/02/2018 15:25:58",
      "content": "<p>amazing Thanks you</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "304363": "same images ????\n\nLB test:  697a05c6fe4a07c601d46da80885645ad574ea19b47ee795ccff216c9f1f1808\n\ntrain:       0287e7ee5b007c91ae2bd7628d09735e70496bc6127ecb7f3dd043e04ce37426\n\n\n - but submission of the ground-truth label gives lower score.\n\n   This proves that the annotation is inconsistent and your solution may need to handle this.\n\n - also the contrast has been changed. hence the some LB test1 set may be modified version of the train set (e.g. rescale, or change contast).",
    "304369": "interesting",
    "304432": "Crud! Wish I had known earlier. Also, in train cb4d &amp; 20b2 are similar, but annotations not.",
    "304437": "They are also the same sample at different contrasts.",
    "304442": "It almost seems that from Heng's and Glimmungs' examples that performing some form of contrast normalization may actually hurt score.",
    "304503": "Good find !! I assume the annotations were done based on what was visible in human eye. Contrast/color/brightness augmentation may hurt the algorithm as the ground truth masks will not be the same at that time. However, little augmentation may help as we might expect some similar images with contrast/color/brightness variation in stage 2.",
    "304748": "LOL, even the number of detected masks is 250 for one image and 289 for the other!",
    "305128": "Interesting finding, thanks for posting.",
    "307574": "Another leak:\n\n\ntrain: 7c318172e976ae5a962c9c7a4e9fe46d7fb985765ddd3a3e2108e893a90b92b2\n\n\ntest: 0a849e0eb15faa8a6d7329c3dd66aabe9a294cccb52ed30a90c8ca99092ae732",
    "307576": "thanks!",
    "307735": "Take your best submission. Include these 2 leaks 697a, 0a84 (copy/paste from matching train identifiers). Submit and LB is not better? It means annotations on test for the same 2 images are different from train.",
    "307736": "697a is not better (i lost 0.002). I didn't try 0a84",
    "307737": "Tried both, I've lost 0.005.\nAfter 04/11, we're allowed to re-train (but not modify model logic) with stage1 release. I think we should drop these 2 images in training as it would hurt learning (same image with different annotations).",
    "307738": "this means that test and train annotation are different.\n\nIn fact, I hand annotate some train data and compare it with given ground truth. I think human error is quite large. e.g.\n \n- for cases where we have same detection ( a nuclei is detected if iou &gt;0.5) with zero miss and false negative, human annotation error can be in the range of 0.01 (depends on the size of the nuclei). This is because we perceive \"outline\" of the nuclei differently, resulting in about one to two pixel shift.\n\n- in some cases, the detection is different. E.g. I disagree with the ground truth if faint dark nuclei should be marked or not. Other cases includes those small fragmented  (micro-nucleus) and also those at the boundary.\n\nin summary, human error is large!",
    "307840": "amazing Thanks you",
    "308051": "&gt; in summary, human error is large!\n\nYes, yes it probably is! \n\nThe biggest source of variation (at least for a case like this) is likely how bright the annotator's screen was during annotation. We considered carefully how to handle this issue. The choices were to carefully calibrate all the screens to be identical - this was practically impossible because we were not just on different tablets/laptops/spare monitors but also because we each were subjected to different lighting conditions. So we decided instead to let the variation reflect the 'natural' variation real biologists would have when examining their images in the wild. It's easy to see how such a situation would yield different annotations for the same images. Sorry that it's not a satisfying answer, but it's what we've got to work with!",
    "308219": "Anne, thank you for your answer, but the question here might be why did you decide to include same images into train and test sets on the first place? Because this would inevitably harm the algorithm performance - was it to make sure there are no 'perfect' solutions?",
    "308987": "Pretty sure it was unintentional.",
    "309748": "Heng thanks for posting this.\n\n@MPWARE Team, removing those 2 images from the training data hurt my scores by 0.003, any thoughts?",
    "309809": "&gt;&gt;removing those 2 images from the training data hurt my scores by 0.003\n\nso it means that:\n\n-  training with inaccurate labels (within certain error) is better than training without them at all\n\n- prediction from trained model is better than different human annotation",
    "309888": "Thanks @Heng that is what I inferred from it but wanted to know what others think."
  },
  "source": "meta"
}