{
  "id": 53831,
  "title": "How to prepare external  BBBC006 dataset ?",
  "url": "/competitions/data-science-bowl-2018/discussion/53831",
  "author_name": "",
  "post_date": "2018-04-05T16:12:59.075120200Z",
  "votes": null,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hello all, I know that many of you are using the BBBC006 dataset  <a href=\"https://data.broadinstitute.org/bbbc/BBBC006/\">https://data.broadinstitute.org/bbbc/BBBC006/</a>   as external data for training. I have downloaded it to my computer with raw images( 1536 images ) and ground-truth 768 images. For 1 ground-truth image, we have an on focus raw image and an out of focus image. To use the dataset, I think we have to select a right raw image for each ground truth image results in 768 raw image and 768 ground truth image. However, I did not find any rule to find the right raw image in the dataset corresponding to a given ground truth image. Please share your way to prepare the dataset for training maskrcnn. Thank you so much</p>",
  "messages": [
    {
      "id": "309585",
      "postDate": "04/05/2018 16:12:59",
      "content": "<p>Hello all, I know that many of you are using the BBBC006 dataset  <a href=\"https://data.broadinstitute.org/bbbc/BBBC006/\">https://data.broadinstitute.org/bbbc/BBBC006/</a>   as external data for training. I have downloaded it to my computer with raw images( 1536 images ) and ground-truth 768 images. For 1 ground-truth image, we have an on focus raw image and an out of focus image. To use the dataset, I think we have to select a right raw image for each ground truth image results in 768 raw image and 768 ground truth image. However, I did not find any rule to find the right raw image in the dataset corresponding to a given ground truth image. Please share your way to prepare the dataset for training maskrcnn. Thank you so much</p>",
      "rawMarkdown": "Hello all, I know that many of you are using the BBBC006 dataset  https://data.broadinstitute.org/bbbc/BBBC006/   as external data for training. I have downloaded it to my computer with raw images( 1536 images ) and ground-truth 768 images. For 1 ground-truth image, we have an on focus raw image and an out of focus image. To use the dataset, I think we have to select a right raw image for each ground truth image results in 768 raw image and 768 ground truth image. However, I did not find any rule to find the right raw image in the dataset corresponding to a given ground truth image. Please share your way to prepare the dataset for training maskrcnn. Thank you so much",
      "votes": null
    },
    {
      "id": "309588",
      "postDate": "04/05/2018 16:25:28",
      "content": "<p>the ground truth provided is not accurate. I suggest not to use it.</p>",
      "rawMarkdown": "the ground truth provided is not accurate. I suggest not to use it.",
      "votes": null
    },
    {
      "id": "310034",
      "postDate": "04/06/2018 12:47:09",
      "content": "<p>Thanks Heng for value information. I have no time to refine it. I will hse ground truth from it and test if it helps improve performance. </p>\n\n<p>However, my main question is not ground truth quality. I want to find the raw images corresponds to given ground truth. I have download bbbc006_z16 and found some images and its ground truth. However, some ground truth does not have raw image in the folder. I guess we have to download all .zip file to find it but it take a lot of time. I also look at the csv file but did not find any zip file have the image id in the csv file. Do you know which folder name makes groundtruth and image id in csv file? Or how to find raw image ( in focus) from the given ground truth</p>",
      "rawMarkdown": "Thanks Heng for value information. I have no time to refine it. I will hse ground truth from it and test if it helps improve performance. \n\nHowever, my main question is not ground truth quality. I want to find the raw images corresponds to given ground truth. I have download bbbc006_z16 and found some images and its ground truth. However, some ground truth does not have raw image in the folder. I guess we have to download all .zip file to find it but it take a lot of time. I also look at the csv file but did not find any zip file have the image id in the csv file. Do you know which folder name makes groundtruth and image id in csv file? Or how to find raw image ( in focus) from the given ground truth",
      "votes": null
    },
    {
      "id": "310048",
      "postDate": "04/06/2018 13:18:02",
      "content": "<p>I downloaded the data. The images are sorted alphabetically. you can take every two images and you will find the corresponding ground truth also alphabetically. \nexample : </p>\n\n<p>original image name: mcf-z-stacks-03212011_a01_s1_w1.....</p>\n\n<p>segmentation image: mcf-z-stacks-03212011_a01_s1.png\netc.</p>\n\n<p>The quality is not great. Many nuclei are not well separated. </p>",
      "rawMarkdown": "I downloaded the data. The images are sorted alphabetically. you can take every two images and you will find the corresponding ground truth also alphabetically. \nexample : \n\noriginal image name: mcf-z-stacks-03212011_a01_s1_w1.....\n\nsegmentation image: mcf-z-stacks-03212011_a01_s1.png\netc.\n\nThe quality is not great. Many nuclei are not well separated.",
      "votes": null
    },
    {
      "id": "310055",
      "postDate": "04/06/2018 14:05:33",
      "content": "<p>Great thanks. Do you download BBC006_z16.zip or download all from z00 to z33? I just download z16 (center in focus) and did not find some raw image corresponds to some ground truth. And the image name in csv cannot find in the zip file like mcf-z-stacks-03212011_a02_s1_w1ebe36734-a690-47f3-a2f9-83b58c8b620f.tif</p>",
      "rawMarkdown": "Great thanks. Do you download BBC006_z16.zip or download all from z00 to z33? I just download z16 (center in focus) and did not find some raw image corresponds to some ground truth. And the image name in csv cannot find in the zip file like mcf-z-stacks-03212011_a02_s1_w1ebe36734-a690-47f3-a2f9-83b58c8b620f.tif",
      "votes": null
    },
    {
      "id": "310443",
      "postDate": "04/07/2018 14:46:51",
      "content": "<p>some the the labelled images (automatic label from my model and correct by hand, not all are correct)</p>\n\n<pre><code>def image_to_mask(image):\nH,W = image.shape[:2]\n\nmulti_mask = np.zeros((H,W),np.int32)\nunique_color = set( tuple(v) for m in image for v in m )\n\ncount=0\n#print(len(unique_color))\nfor color in unique_color:\n    #print(color)\n    if color ==(0,0,0): continue\n    #if color ==(10,10,10): continue\n\n    thresh = (image==color).all(axis=2)\n    label  = skimage.morphology.label(thresh)\n\n    index = [label!=0]\n    count = multi_mask.max()\n    multi_mask[index] =  label[index]+count\n\n\nreturn multi_mask\n</code></pre>",
      "rawMarkdown": "some the the labelled images (automatic label from my model and correct by hand, not all are correct)\n\n    def image_to_mask(image):\n    H,W = image.shape[:2]\n\n    multi_mask = np.zeros((H,W),np.int32)\n    unique_color = set( tuple(v) for m in image for v in m )\n\n    count=0\n    #print(len(unique_color))\n    for color in unique_color:\n        #print(color)\n        if color ==(0,0,0): continue\n        #if color ==(10,10,10): continue\n\n        thresh = (image==color).all(axis=2)\n        label  = skimage.morphology.label(thresh)\n\n        index = [label!=0]\n        count = multi_mask.max()\n        multi_mask[index] =  label[index]+count\n\n\n    return multi_mask",
      "votes": null
    },
    {
      "id": "310490",
      "postDate": "04/07/2018 17:27:37",
      "content": "<p>Many thanks for your hard work to correct these masks. it is very useful. </p>",
      "rawMarkdown": "Many thanks for your hard work to correct these masks. it is very useful.",
      "votes": null
    },
    {
      "id": "310558",
      "postDate": "04/08/2018 00:59:00",
      "content": "<p>@Heng: I found that you only gives 30 fixed images. Do you only use these 30 images as external dataset? Because the BBBC006 has 768 images in total</p>",
      "rawMarkdown": "Heng: I found that you only gives 30 fixed images. Do you only use these 30 images as external dataset? Because the BBBC006 has 768 images in total",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 309588,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "04/05/2018 16:25:28",
      "content": "<p>the ground truth provided is not accurate. I suggest not to use it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 310034,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "04/06/2018 12:47:09",
          "content": "<p>Thanks Heng for value information. I have no time to refine it. I will hse ground truth from it and test if it helps improve performance. </p>\n\n<p>However, my main question is not ground truth quality. I want to find the raw images corresponds to given ground truth. I have download bbbc006_z16 and found some images and its ground truth. However, some ground truth does not have raw image in the folder. I guess we have to download all .zip file to find it but it take a lot of time. I also look at the csv file but did not find any zip file have the image id in the csv file. Do you know which folder name makes groundtruth and image id in csv file? Or how to find raw image ( in focus) from the given ground truth</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310048,
          "author_name": "jeandebleau",
          "author_url": "",
          "post_date": "04/06/2018 13:18:02",
          "content": "<p>I downloaded the data. The images are sorted alphabetically. you can take every two images and you will find the corresponding ground truth also alphabetically. \nexample : </p>\n\n<p>original image name: mcf-z-stacks-03212011_a01_s1_w1.....</p>\n\n<p>segmentation image: mcf-z-stacks-03212011_a01_s1.png\netc.</p>\n\n<p>The quality is not great. Many nuclei are not well separated. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310055,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "04/06/2018 14:05:33",
          "content": "<p>Great thanks. Do you download BBC006_z16.zip or download all from z00 to z33? I just download z16 (center in focus) and did not find some raw image corresponds to some ground truth. And the image name in csv cannot find in the zip file like mcf-z-stacks-03212011_a02_s1_w1ebe36734-a690-47f3-a2f9-83b58c8b620f.tif</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 310443,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "04/07/2018 14:46:51",
      "content": "<p>some the the labelled images (automatic label from my model and correct by hand, not all are correct)</p>\n\n<pre><code>def image_to_mask(image):\nH,W = image.shape[:2]\n\nmulti_mask = np.zeros((H,W),np.int32)\nunique_color = set( tuple(v) for m in image for v in m )\n\ncount=0\n#print(len(unique_color))\nfor color in unique_color:\n    #print(color)\n    if color ==(0,0,0): continue\n    #if color ==(10,10,10): continue\n\n    thresh = (image==color).all(axis=2)\n    label  = skimage.morphology.label(thresh)\n\n    index = [label!=0]\n    count = multi_mask.max()\n    multi_mask[index] =  label[index]+count\n\n\nreturn multi_mask\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 310490,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "04/07/2018 17:27:37",
          "content": "<p>Many thanks for your hard work to correct these masks. it is very useful. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310558,
          "author_name": "john1231983",
          "author_url": "",
          "post_date": "04/08/2018 00:59:00",
          "content": "<p>@Heng: I found that you only gives 30 fixed images. Do you only use these 30 images as external dataset? Because the BBBC006 has 768 images in total</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "309585": "Hello all, I know that many of you are using the BBBC006 dataset  https://data.broadinstitute.org/bbbc/BBBC006/   as external data for training. I have downloaded it to my computer with raw images( 1536 images ) and ground-truth 768 images. For 1 ground-truth image, we have an on focus raw image and an out of focus image. To use the dataset, I think we have to select a right raw image for each ground truth image results in 768 raw image and 768 ground truth image. However, I did not find any rule to find the right raw image in the dataset corresponding to a given ground truth image. Please share your way to prepare the dataset for training maskrcnn. Thank you so much",
    "309588": "the ground truth provided is not accurate. I suggest not to use it.",
    "310034": "Thanks Heng for value information. I have no time to refine it. I will hse ground truth from it and test if it helps improve performance. \n\nHowever, my main question is not ground truth quality. I want to find the raw images corresponds to given ground truth. I have download bbbc006_z16 and found some images and its ground truth. However, some ground truth does not have raw image in the folder. I guess we have to download all .zip file to find it but it take a lot of time. I also look at the csv file but did not find any zip file have the image id in the csv file. Do you know which folder name makes groundtruth and image id in csv file? Or how to find raw image ( in focus) from the given ground truth",
    "310048": "I downloaded the data. The images are sorted alphabetically. you can take every two images and you will find the corresponding ground truth also alphabetically. \nexample : \n\noriginal image name: mcf-z-stacks-03212011_a01_s1_w1.....\n\nsegmentation image: mcf-z-stacks-03212011_a01_s1.png\netc.\n\nThe quality is not great. Many nuclei are not well separated.",
    "310055": "Great thanks. Do you download BBC006_z16.zip or download all from z00 to z33? I just download z16 (center in focus) and did not find some raw image corresponds to some ground truth. And the image name in csv cannot find in the zip file like mcf-z-stacks-03212011_a02_s1_w1ebe36734-a690-47f3-a2f9-83b58c8b620f.tif",
    "310443": "some the the labelled images (automatic label from my model and correct by hand, not all are correct)\n\n    def image_to_mask(image):\n    H,W = image.shape[:2]\n\n    multi_mask = np.zeros((H,W),np.int32)\n    unique_color = set( tuple(v) for m in image for v in m )\n\n    count=0\n    #print(len(unique_color))\n    for color in unique_color:\n        #print(color)\n        if color ==(0,0,0): continue\n        #if color ==(10,10,10): continue\n\n        thresh = (image==color).all(axis=2)\n        label  = skimage.morphology.label(thresh)\n\n        index = [label!=0]\n        count = multi_mask.max()\n        multi_mask[index] =  label[index]+count\n\n\n    return multi_mask",
    "310490": "Many thanks for your hard work to correct these masks. it is very useful.",
    "310558": "Heng: I found that you only gives 30 fixed images. Do you only use these 30 images as external dataset? Because the BBBC006 has 768 images in total"
  },
  "source": "meta"
}