I'd like to posts fixed masks and images, ideally to a public github repo (with a link to the competition, and commiting changes separately), so that we can avoid wasting everyone's time fixing the same issues, and hopefully achieve better quality. But according to general kaggle competition rules (7B), this is not allowed. But maybe it's fine to do it for this particular dataset and competition?
\n\nWhy github and not drop a zip archive? Because github supports pull requests, showing what's changed, diffs for images, etc. - so it's much more useful than dropping a zip archive IMO.
\n\nUpdate: The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, many thanks to Kaggle and Carpenter lab!
", "messages": [ { "id": 287712, "postDate": "2018-02-25T19:15:02.980Z", "content": "I'd like to posts fixed masks and images, ideally to a public github repo (with a link to the competition, and commiting changes separately), so that we can avoid wasting everyone's time fixing the same issues, and hopefully achieve better quality. But according to general kaggle competition rules (7B), this is not allowed. But maybe it's fine to do it for this particular dataset and competition?
\n\nWhy github and not drop a zip archive? Because github supports pull requests, showing what's changed, diffs for images, etc. - so it's much more useful than dropping a zip archive IMO.
\n\nUpdate: The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, many thanks to Kaggle and Carpenter lab!
", "rawMarkdown": "I'd like to posts fixed masks and images, ideally to a public github repo (with a link to the competition, and commiting changes separately), so that we can avoid wasting everyone's time fixing the same issues, and hopefully achieve better quality. But according to general kaggle competition rules (7B), this is not allowed. But maybe it's fine to do it for this particular dataset and competition?\n\nWhy github and not drop a zip archive? Because github supports pull requests, showing what's changed, diffs for images, etc. - so it's much more useful than dropping a zip archive IMO.\n\n**Update:** The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, many thanks to Kaggle and Carpenter lab!", "votes": 123 }, { "id": 288558, "postDate": "2018-02-27T18:40:29.403Z", "content": "It is fine with us if you create a public repo for the curated masks. We can treat it as external public data. Please post a link in the external data thread so the community can see it.
", "rawMarkdown": "It is fine with us if you create a public repo for the curated masks. We can treat it as external public data. Please post a link in the external data thread so the community can see it. ", "votes": 20, "replies": [ { "id": 288583, "postDate": "2018-02-27T20:03:14.623Z", "content": "Awesome! I was going to suggest submitting just diff binary patches but a repo would be even easier!
Great, thanks! The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, will post to the external data thread as well.
", "rawMarkdown": "Great, thanks! The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, will post to the external data thread as well.", "votes": 10 }, { "id": 288904, "postDate": "2018-02-28T13:40:43.530Z", "content": "@Konstantin, \nJust for clarification, do we have to download the data and run the script? Or, is the data already processed and ready to use?
", "rawMarkdown": "@Konstantin, \nJust for clarification, do we have to download the data and run the script? Or, is the data already processed and ready to use?", "votes": 1 }, { "id": 288918, "postDate": "2018-02-28T14:28:26.653Z", "content": "@William Green the data is already processed and ready to use, the script was used to prepare some annotations. I added an extra clarification to the README.
", "rawMarkdown": "@William Green the data is already processed and ready to use, the script was used to prepare some annotations. I added an extra clarification to the README.", "votes": 3 }, { "id": 288931, "postDate": "2018-02-28T15:04:54.660Z", "content": "Thank you Konstantin. :)
", "rawMarkdown": "Thank you Konstantin. :)" } ] }, { "id": 289462, "postDate": "2018-03-01T18:36:21.050Z", "content": "Hi, I made a small check yesterday and found other 394 \"broken\" masks (out of 28993, but still). To see such masks, please take a look at images attached. I will try to fix it, but perhaps someone else can do it earlier as I am leaving tonight and won't have an access to my PC for a week.
\n\nThere are some tips to find these masks and to fix them:
\n\nFirst of all, I use the following code to detect masks with holes. To be honest, I just wanted to detect all possible contours in order to create bounding boxes... :)
\n\ndata_base = '../data/stage1_train'\nfor img_id in tqdm(os.listdir(data_base)):\n images_folder = '{}/{}/images/'.format(data_base, img_id)\n masks_folder = '{}/{}/masks/'.format(data_base, img_id)\n img_path = '{}/{}'.format(images_folder, os.listdir(images_folder)[0])\n\n for mask_id in os.listdir(masks_folder):\n counter_all += 1\n mask_path = '{}/{}'.format(masks_folder, mask_id)\n mask_img = cv2.imread(mask_path)\n mask = cv2.cvtColor(mask_img, cv2.COLOR_RGB2GRAY )\n\n _, contours, _ = cv2.findContours(\n mask,\n cv2.RETR_TREE,\n cv2.CHAIN_APPROX_SIMPLE\n )\n\n for cnt_num, c in enumerate(contours):\n cnt = contours[cnt_num]\n x, y, w, h = cv2.boundingRect(cnt)\n cv2.rectangle(mask_img, (x, y), (x + w, y + h), (255, 0, 0), 2)\n\n if len(contours) > 1:\n plt.imshow(mask_img)\n plt.show()\nThis gives us the following statistics:
\n\nBad masks: 394\nAll masks: 28993\nBad masks in %: 0.01359\nAnd a dictionary with a number of masks for each amount of contours per mask:
\n\n1: 28599,\n2: 243,\n3: 57,\n4: 29,\n5: 17,\n6: 16,\n7: 7,\n8: 9,\n9: 7,\n10: 2,\n11: 5,\n12: 1,\n13: 1\nNot certainly clear how to fix, but there are two main methods that I'd like to try first of all
\n\nWe'll keep posted if our team have any updates. But we don't mind anyone can do it before we do ;)
", "rawMarkdown": "Hi, I made a small check yesterday and found other 394 \"broken\" masks (out of 28993, but still). To see such masks, please take a look at images attached. I will try to fix it, but perhaps someone else can do it earlier as I am leaving tonight and won't have an access to my PC for a week. \n\nThere are some tips to find these masks and to fix them:\n\n1. Find problematic masks\n-------------------------\n\nFirst of all, I use the following code to detect masks with holes. To be honest, I just wanted to detect all possible contours in order to create bounding boxes... :)\n\n data_base = '../data/stage1_train'\n for img_id in tqdm(os.listdir(data_base)):\n images_folder = '{}/{}/images/'.format(data_base, img_id)\n masks_folder = '{}/{}/masks/'.format(data_base, img_id)\n img_path = '{}/{}'.format(images_folder, os.listdir(images_folder)[0])\n\n for mask_id in os.listdir(masks_folder):\n counter_all += 1\n mask_path = '{}/{}'.format(masks_folder, mask_id)\n mask_img = cv2.imread(mask_path)\n mask = cv2.cvtColor(mask_img, cv2.COLOR_RGB2GRAY )\n\n _, contours, _ = cv2.findContours(\n mask,\n cv2.RETR_TREE,\n cv2.CHAIN_APPROX_SIMPLE\n )\n\n for cnt_num, c in enumerate(contours):\n cnt = contours[cnt_num]\n x, y, w, h = cv2.boundingRect(cnt)\n cv2.rectangle(mask_img, (x, y), (x + w, y + h), (255, 0, 0), 2)\n\n if len(contours) > 1:\n plt.imshow(mask_img)\n plt.show()\n\nThis gives us the following statistics:\n\n Bad masks: 394\n All masks: 28993\n Bad masks in %: 0.01359\n\nAnd a dictionary with a number of masks for each amount of contours per mask:\n\n 1: 28599,\n 2: 243,\n 3: 57,\n 4: 29,\n 5: 17,\n 6: 16,\n 7: 7,\n 8: 9,\n 9: 7,\n 10: 2,\n 11: 5,\n 12: 1,\n 13: 1\n\n2. Fix problematic masks\n-------------------------\n\nNot certainly clear how to fix, but there are two main methods that I'd like to try first of all\n\n - https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html#closing\n - https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.ndimage.morphology.binary_fill_holes.html#scipy-ndimage-morphology-binary-fill-holes\n\nWe'll keep posted if our team have any updates. But we don't mind anyone can do it before we do ;)\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/289462/8661/bad_mask_001.PNG", "votes": 8, "replies": [ { "id": 289474, "postDate": "2018-03-01T19:06:56.123Z", "content": "Hi, thanks for checking! FYI, there is a PR https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/pull/2 that, among other things, fixes a lot of masks with holes, which I'll review in the next few days.
", "rawMarkdown": "Hi, thanks for checking! FYI, there is a PR https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/pull/2 that, among other things, fixes a lot of masks with holes, which I'll review in the next few days.", "votes": 4 }, { "id": 289483, "postDate": "2018-03-01T19:27:40.853Z", "content": "Yeah... saw it at the time of creating a comment :D Anyway I decided that someone might want to see the code of finding and fixing mask holes..
\n\nBut I see the PR is about changes in 395 files, so I am pretty sure it fixes all masks my code finds.
", "rawMarkdown": "Yeah... saw it at the time of creating a comment :D Anyway I decided that someone might want to see the code of finding and fixing mask holes..\n\nBut I see the PR is about changes in 395 files, so I am pretty sure it fixes all masks my code finds.", "votes": 5 } ] }, { "id": 288450, "postDate": "2018-02-27T14:19:01.543Z", "content": "I will see if I can get an official answer from the Kaggle folks on this.
\n\nCertainly I do not object, and the images and annotations are licensed Creative Commons 0 (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864) although it is true that general kaggle competition rules 7B seems to over-ride that at least while the competition is in process. You'd only be releasing a tiny subset of the images anyway.
", "rawMarkdown": "I will see if I can get an official answer from the Kaggle folks on this. \n\nCertainly I do not object, and the images and annotations are licensed Creative Commons 0 (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864) although it is true that general kaggle competition rules 7B seems to over-ride that at least while the competition is in process. You'd only be releasing a tiny subset of the images anyway.", "votes": 8, "replies": [ { "id": 288459, "postDate": "2018-02-27T14:47:11.530Z", "content": "Thanks a lot Anne!
\n\n\n\n\nYou'd only be releasing a tiny subset of the images anyway.
\n
Do you mean that data released on kaggle is a tiny subset, or that I would be putting only the images with fixes? If the latter, than ideally I wanted to put them all, because there were a couple of cases where the easiest \"fix\" is removing the whole image from the training set, so it's more convenient to have all images, not just the fixes.
", "rawMarkdown": "Thanks a lot Anne!\n\n> You'd only be releasing a tiny subset of the images anyway.\n\nDo you mean that data released on kaggle is a tiny subset, or that I would be putting only the images with fixes? If the latter, than ideally I wanted to put them all, because there were a couple of cases where the easiest \"fix\" is removing the whole image from the training set, so it's more convenient to have all images, not just the fixes.", "votes": 8 }, { "id": 288620, "postDate": "2018-02-27T22:06:02.930Z", "content": "Definitely agree, and I'm excited that Wendy says it's all a thumbs up! That is even better/easier.
", "rawMarkdown": "Definitely agree, and I'm excited that Wendy says it's all a thumbs up! That is even better/easier." } ] }, { "id": 287910, "postDate": "2018-02-26T07:31:04.970Z", "content": "Awesome!!!
", "rawMarkdown": "Awesome!!!", "votes": 3 }, { "id": 287793, "postDate": "2018-02-26T00:41:43.107Z", "content": "This idea is great!
", "rawMarkdown": "This idea is great!", "votes": 1, "replies": [ { "id": 287800, "postDate": "2018-02-26T00:54:44.777Z", "content": "Agreed!
", "rawMarkdown": "Agreed!" } ] }, { "id": 287937, "postDate": "2018-02-26T09:17:03.993Z", "content": "The bad annotation do affect results. You work will help to our results. Thanks!
", "rawMarkdown": "The bad annotation do affect results. You work will help to our results. Thanks!", "votes": 2 }, { "id": 3501401, "postDate": "2026-07-21T23:34:25.997Z", "content": "This is the exact reason why this GitHub repo is a game-changer over a standard ZIP file:\nVisual Image Diffs: We can see the exact pixel changes between the original and fixed masks directly in the browser.
\nNo Duplicated Effort: Everyone benefits from the same clean data instead of hundreds of teams fixing the same images independently.
\nOpen Collaboration: Anyone can submit a Pull Request if they find and fix more bad masks.
\nChange Tracking: The commit history keeps a clear log of what was changed, by whom, and why
", "rawMarkdown": "This is the exact reason why this GitHub repo is a game-changer over a standard ZIP file:\nVisual Image Diffs: We can see the exact pixel changes between the original and fixed masks directly in the browser.\n\nNo Duplicated Effort: Everyone benefits from the same clean data instead of hundreds of teams fixing the same images independently.\n\nOpen Collaboration: Anyone can submit a Pull Request if they find and fix more bad masks.\n\nChange Tracking: The commit history keeps a clear log of what was changed, by whom, and why" }, { "id": 311426, "postDate": "2018-04-10T03:10:52.557Z", "content": "@Konstantin, is mask_tools.py the only pre-processing code you used so that one can use it on the current test data in stage 2? If not one can plan on not using this data for final model submission in 2 days. Thanks.
", "rawMarkdown": "@Konstantin, is mask_tools.py the only pre-processing code you used so that one can use it on the current test data in stage 2? If not one can plan on not using this data for final model submission in 2 days. Thanks.", "replies": [ { "id": 311428, "postDate": "2018-04-10T03:24:49.530Z", "content": "I didn't change images in any way, mask_tools.py is only for converting masks into the original dataset format. The masks in the test set of stage 1 are already in the correct format, you don't need to run mask_tools.py.
", "rawMarkdown": "I didn't change images in any way, mask_tools.py is only for converting masks into the original dataset format. The masks in the test set of stage 1 are already in the correct format, you don't need to run mask_tools.py." } ] }, { "id": 308225, "postDate": "2018-04-03T07:15:05.373Z", "content": "@Konstantin Lopuhin , It seems that the mask in red don't exist in the original image \nebc18868864ad075548cc1784f4f9a237bb98335f9645ee727dac8332a3e3716
", "rawMarkdown": "@Konstantin Lopuhin , It seems that the mask in red don't exist in the original image \nebc18868864ad075548cc1784f4f9a237bb98335f9645ee727dac8332a3e3716", "replies": [ { "id": 308310, "postDate": "2018-04-03T10:10:24.430Z", "content": "Thanks for the catch, removed in https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/commit/30354551efef351c1241da7c6d48cf938d76f02a
", "rawMarkdown": "Thanks for the catch, removed in https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/commit/30354551efef351c1241da7c6d48cf938d76f02a" } ] }, { "id": 301195, "postDate": "2018-03-22T13:19:59.853Z", "content": "Hi Konstantin,
\n\nThanks so much for providing this prepared data. I used it last time on March 6th but today using it with the same script I get an error. Do you have any idea why?
\n\n> > Getting and resizing train images and masks ... 25/664\n> [>.............................] - ETA: 2:31 ValueError
\n> Traceback (most recent call last)\n2
--->3 # get test_data
\n\n4 test_img,test_img_sizes = read_test_data()
", "rawMarkdown": "Hi Konstantin,\n\nThanks so much for providing this prepared data. I used it last time on March 6th but today using it with the same script I get an error. Do you have any idea why?\n\n#Here is the error I get when I try to read it\n\n> > Getting and resizing train images and masks ... 25/664\n> [>.............................] - ETA: 2:31 ValueError \n> Traceback (most recent call last)\n2\n\n--->3 # get test_data\n\n 4 test_img,test_img_sizes = read_test_data()\n ", "replies": [ { "id": 301390, "postDate": "2018-03-22T17:31:39.043Z", "content": "@sheriytm Sorry for the issue, some masks had bad shapes, fixed in https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/commit/38ce5aa81a7752dfbeb0d6a8e681e933d309a96f - please tell if you still have any issues.
", "rawMarkdown": "@sheriytm Sorry for the issue, some masks had bad shapes, fixed in https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/commit/38ce5aa81a7752dfbeb0d6a8e681e933d309a96f - please tell if you still have any issues.", "votes": 2 }, { "id": 301585, "postDate": "2018-03-23T00:59:11.423Z", "content": "Thanks @Konstantin, it is working fine now.
", "rawMarkdown": "Thanks @Konstantin, it is working fine now. ", "votes": 1 } ] }, { "id": 299240, "postDate": "2018-03-20T17:36:10.517Z", "content": "Thanks for your great job. I have a question. The official train dataset has 670 images but you only have 664. Why do your delete 6 images?
", "rawMarkdown": "Thanks for your great job. I have a question. The official train dataset has 670 images but you only have 664. Why do your delete 6 images? ", "replies": [ { "id": 301392, "postDate": "2018-03-22T17:32:15.403Z", "content": "Please see below: https://www.kaggle.com/c/data-science-bowl-2018/discussion/50518#290915
", "rawMarkdown": "Please see below: https://www.kaggle.com/c/data-science-bowl-2018/discussion/50518#290915", "votes": 1 } ] }, { "id": 290854, "postDate": "2018-03-05T03:32:39.247Z", "content": "Thx for your awesome work @Konstantin Lopuhin . I download your repo from github. I find there are only 664 pictures in the stage1_train, but there are 670 pictures in the raw dataset. Did you delete 6 \"wrong pictures\"? I'm looking forward to your reply, thx.
", "rawMarkdown": "Thx for your awesome work @Konstantin Lopuhin . I download your repo from github. I find there are only 664 pictures in the stage1_train, but there are 670 pictures in the raw dataset. Did you delete 6 \"wrong pictures\"? I'm looking forward to your reply, thx.", "replies": [ { "id": 290915, "postDate": "2018-03-05T07:01:42.743Z", "content": "Yes, I removed some pictures - there was nothing wrong with the pictures themselves, but some of them didn't have annotations at all, and some had annotations that looked wrong and it was easier to exclude them then to re-annotate them.
\n\nBut now there is a PR (not merged yet) which brings back these pictures by providing the annotations.
", "rawMarkdown": "Yes, I removed some pictures - there was nothing wrong with the pictures themselves, but some of them didn't have annotations at all, and some had annotations that looked wrong and it was easier to exclude them then to re-annotate them.\n\nBut now there is a PR (not merged yet) which brings back these pictures by providing the annotations.", "votes": 2 }, { "id": 291061, "postDate": "2018-03-05T14:58:33.580Z", "content": "Very appreciate！
", "rawMarkdown": "Very appreciate！" } ] }, { "id": 289046, "postDate": "2018-02-28T20:51:59.273Z", "content": "Thanks a lot!
\n\nHowever, I thought all small holes in each mask should be filled in for image 5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974?\nFor instance, for mask 888a9bf5e5db6a92cdd9556069010b50d4faa869c225708181e3914c5e592fdd. What do you think about it?
\n\nAlso, is 1 pixel width (short line) masks really valid? For instance, for image 02903040e19ddf92f452907644ad3822918f54af41dd85e5a3fe3e1b6d6f9339 and mask: ee43930956917b35e8ea9950119904ee43eeaf297ae2d0999a96bce06465119b
", "rawMarkdown": "Thanks a lot! \n\nHowever, I thought all small holes in each mask should be filled in for image 5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974?\nFor instance, for mask 888a9bf5e5db6a92cdd9556069010b50d4faa869c225708181e3914c5e592fdd. What do you think about it?\n\nAlso, is 1 pixel width (short line) masks really valid? For instance, for image 02903040e19ddf92f452907644ad3822918f54af41dd85e5a3fe3e1b6d6f9339 and mask: ee43930956917b35e8ea9950119904ee43eeaf297ae2d0999a96bce06465119b" }, { "id": 288444, "postDate": "2018-02-27T13:51:49.667Z", "content": "This would be really nice!
", "rawMarkdown": "This would be really nice!" }, { "id": 287925, "postDate": "2018-02-26T08:36:39.867Z", "content": "It would be very helpful to all; I don't see a downside.
", "rawMarkdown": "It would be very helpful to all; I don't see a downside. ", "replies": [ { "id": 287928, "postDate": "2018-02-26T08:42:36.100Z", "content": "A potential downside could be if organizers are not comfortable sharing annotations publicly yet. In this case I'll share my fixes on the forum, which would be less convenient to use and collaborate, but still better than nothing.
", "rawMarkdown": "A potential downside could be if organizers are not comfortable sharing annotations publicly yet. In this case I'll share my fixes on the forum, which would be less convenient to use and collaborate, but still better than nothing.", "votes": 7 } ] }, { "id": 297383, "postDate": "2018-03-17T00:27:33.317Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 301391, "postDate": "2018-03-22T17:32:04.793Z", "content": "Please see below: https://www.kaggle.com/c/data-science-bowl-2018/discussion/50518#290915
", "rawMarkdown": "Please see below: https://www.kaggle.com/c/data-science-bowl-2018/discussion/50518#290915", "votes": 1 } ] }, { "id": 289589, "postDate": "2018-03-01T22:59:36.017Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ], "comments": [ { "id": 288558, "author_name": "Wendy Kan", "author_url": "", "post_date": "2018-02-27T18:40:29.403000", "content": "It is fine with us if you create a public repo for the curated masks. We can treat it as external public data. Please post a link in the external data thread so the community can see it.
", "votes": 20, "replies": [ { "id": 288583, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2018-02-27T20:03:14.623000", "content": "Awesome! I was going to suggest submitting just diff binary patches but a repo would be even easier!
Great, thanks! The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, will post to the external data thread as well.
", "votes": 10, "replies": [] }, { "id": 288904, "author_name": "William Green", "author_url": "", "post_date": "2018-02-28T13:40:43.530000", "content": "@Konstantin, \nJust for clarification, do we have to download the data and run the script? Or, is the data already processed and ready to use?
", "votes": 1, "replies": [] }, { "id": 288918, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-02-28T14:28:26.653000", "content": "@William Green the data is already processed and ready to use, the script was used to prepare some annotations. I added an extra clarification to the README.
", "votes": 3, "replies": [] }, { "id": 288931, "author_name": "William Green", "author_url": "", "post_date": "2018-02-28T15:04:54.660000", "content": "Thank you Konstantin. :)
", "votes": 0, "replies": [] } ] }, { "id": 289462, "author_name": "true_pk", "author_url": "", "post_date": "2018-03-01T18:36:21.050000", "content": "Hi, I made a small check yesterday and found other 394 \"broken\" masks (out of 28993, but still). To see such masks, please take a look at images attached. I will try to fix it, but perhaps someone else can do it earlier as I am leaving tonight and won't have an access to my PC for a week.
\n\nThere are some tips to find these masks and to fix them:
\n\nFirst of all, I use the following code to detect masks with holes. To be honest, I just wanted to detect all possible contours in order to create bounding boxes... :)
\n\ndata_base = '../data/stage1_train'\nfor img_id in tqdm(os.listdir(data_base)):\n images_folder = '{}/{}/images/'.format(data_base, img_id)\n masks_folder = '{}/{}/masks/'.format(data_base, img_id)\n img_path = '{}/{}'.format(images_folder, os.listdir(images_folder)[0])\n\n for mask_id in os.listdir(masks_folder):\n counter_all += 1\n mask_path = '{}/{}'.format(masks_folder, mask_id)\n mask_img = cv2.imread(mask_path)\n mask = cv2.cvtColor(mask_img, cv2.COLOR_RGB2GRAY )\n\n _, contours, _ = cv2.findContours(\n mask,\n cv2.RETR_TREE,\n cv2.CHAIN_APPROX_SIMPLE\n )\n\n for cnt_num, c in enumerate(contours):\n cnt = contours[cnt_num]\n x, y, w, h = cv2.boundingRect(cnt)\n cv2.rectangle(mask_img, (x, y), (x + w, y + h), (255, 0, 0), 2)\n\n if len(contours) > 1:\n plt.imshow(mask_img)\n plt.show()\nThis gives us the following statistics:
\n\nBad masks: 394\nAll masks: 28993\nBad masks in %: 0.01359\nAnd a dictionary with a number of masks for each amount of contours per mask:
\n\n1: 28599,\n2: 243,\n3: 57,\n4: 29,\n5: 17,\n6: 16,\n7: 7,\n8: 9,\n9: 7,\n10: 2,\n11: 5,\n12: 1,\n13: 1\nNot certainly clear how to fix, but there are two main methods that I'd like to try first of all
\n\nWe'll keep posted if our team have any updates. But we don't mind anyone can do it before we do ;)
", "votes": 8, "replies": [ { "id": 289474, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-03-01T19:06:56.123000", "content": "Hi, thanks for checking! FYI, there is a PR https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/pull/2 that, among other things, fixes a lot of masks with holes, which I'll review in the next few days.
", "votes": 4, "replies": [] }, { "id": 289483, "author_name": "true_pk", "author_url": "", "post_date": "2018-03-01T19:27:40.853000", "content": "Yeah... saw it at the time of creating a comment :D Anyway I decided that someone might want to see the code of finding and fixing mask holes..
\n\nBut I see the PR is about changes in 395 files, so I am pretty sure it fixes all masks my code finds.
", "votes": 5, "replies": [] } ] }, { "id": 288450, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-27T14:19:01.543000", "content": "I will see if I can get an official answer from the Kaggle folks on this.
\n\nCertainly I do not object, and the images and annotations are licensed Creative Commons 0 (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864) although it is true that general kaggle competition rules 7B seems to over-ride that at least while the competition is in process. You'd only be releasing a tiny subset of the images anyway.
", "votes": 8, "replies": [ { "id": 288459, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-02-27T14:47:11.530000", "content": "Thanks a lot Anne!
\n\n\n\n\nYou'd only be releasing a tiny subset of the images anyway.
\n
Do you mean that data released on kaggle is a tiny subset, or that I would be putting only the images with fixes? If the latter, than ideally I wanted to put them all, because there were a couple of cases where the easiest \"fix\" is removing the whole image from the training set, so it's more convenient to have all images, not just the fixes.
", "votes": 8, "replies": [] }, { "id": 288620, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-02-27T22:06:02.930000", "content": "Definitely agree, and I'm excited that Wendy says it's all a thumbs up! That is even better/easier.
", "votes": 0, "replies": [] } ] }, { "id": 287910, "author_name": "Alex Donchuk", "author_url": "", "post_date": "2018-02-26T07:31:04.970000", "content": "Awesome!!!
", "votes": 3, "replies": [] }, { "id": 287793, "author_name": "Oleksandr Bailo", "author_url": "", "post_date": "2018-02-26T00:41:43.107000", "content": "This idea is great!
", "votes": 1, "replies": [ { "id": 287800, "author_name": "BrandonPippin", "author_url": "", "post_date": "2018-02-26T00:54:44.777000", "content": "Agreed!
", "votes": 0, "replies": [] } ] }, { "id": 287937, "author_name": "hengck23", "author_url": "", "post_date": "2018-02-26T09:17:03.993000", "content": "The bad annotation do affect results. You work will help to our results. Thanks!
", "votes": 2, "replies": [] }, { "id": 3501401, "author_name": "", "author_url": "", "post_date": "2026-07-21T23:34:25.997000", "content": "This is the exact reason why this GitHub repo is a game-changer over a standard ZIP file:\nVisual Image Diffs: We can see the exact pixel changes between the original and fixed masks directly in the browser.
\nNo Duplicated Effort: Everyone benefits from the same clean data instead of hundreds of teams fixing the same images independently.
\nOpen Collaboration: Anyone can submit a Pull Request if they find and fix more bad masks.
\nChange Tracking: The commit history keeps a clear log of what was changed, by whom, and why
", "votes": 0, "replies": [] }, { "id": 311426, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-04-10T03:10:52.557000", "content": "@Konstantin, is mask_tools.py the only pre-processing code you used so that one can use it on the current test data in stage 2? If not one can plan on not using this data for final model submission in 2 days. Thanks.
", "votes": 0, "replies": [ { "id": 311428, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-04-10T03:24:49.530000", "content": "I didn't change images in any way, mask_tools.py is only for converting masks into the original dataset format. The masks in the test set of stage 1 are already in the correct format, you don't need to run mask_tools.py.
", "votes": 0, "replies": [] } ] }, { "id": 308225, "author_name": "Zhiran Ye", "author_url": "", "post_date": "2018-04-03T07:15:05.373000", "content": "@Konstantin Lopuhin , It seems that the mask in red don't exist in the original image \nebc18868864ad075548cc1784f4f9a237bb98335f9645ee727dac8332a3e3716
", "votes": 0, "replies": [ { "id": 308310, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-04-03T10:10:24.430000", "content": "Thanks for the catch, removed in https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/commit/30354551efef351c1241da7c6d48cf938d76f02a
", "votes": 0, "replies": [] } ] }, { "id": 301195, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-03-22T13:19:59.853000", "content": "Hi Konstantin,
\n\nThanks so much for providing this prepared data. I used it last time on March 6th but today using it with the same script I get an error. Do you have any idea why?
\n\n> > Getting and resizing train images and masks ... 25/664\n> [>.............................] - ETA: 2:31 ValueError
\n> Traceback (most recent call last)\n2
--->3 # get test_data
\n\n4 test_img,test_img_sizes = read_test_data()
", "votes": 0, "replies": [ { "id": 301390, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-03-22T17:31:39.043000", "content": "@sheriytm Sorry for the issue, some masks had bad shapes, fixed in https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes/commit/38ce5aa81a7752dfbeb0d6a8e681e933d309a96f - please tell if you still have any issues.
", "votes": 2, "replies": [] }, { "id": 301585, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-03-23T00:59:11.423000", "content": "Thanks @Konstantin, it is working fine now.
", "votes": 1, "replies": [] } ] }, { "id": 299240, "author_name": "Zhen Chen", "author_url": "", "post_date": "2018-03-20T17:36:10.517000", "content": "Thanks for your great job. I have a question. The official train dataset has 670 images but you only have 664. Why do your delete 6 images?
", "votes": 0, "replies": [ { "id": 301392, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-03-22T17:32:15.403000", "content": "Please see below: https://www.kaggle.com/c/data-science-bowl-2018/discussion/50518#290915
", "votes": 1, "replies": [] } ] }, { "id": 290854, "author_name": "Liang Jeff", "author_url": "", "post_date": "2018-03-05T03:32:39.247000", "content": "Thx for your awesome work @Konstantin Lopuhin . I download your repo from github. I find there are only 664 pictures in the stage1_train, but there are 670 pictures in the raw dataset. Did you delete 6 \"wrong pictures\"? I'm looking forward to your reply, thx.
", "votes": 0, "replies": [ { "id": 290915, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-03-05T07:01:42.743000", "content": "Yes, I removed some pictures - there was nothing wrong with the pictures themselves, but some of them didn't have annotations at all, and some had annotations that looked wrong and it was easier to exclude them then to re-annotate them.
\n\nBut now there is a PR (not merged yet) which brings back these pictures by providing the annotations.
", "votes": 2, "replies": [] }, { "id": 291061, "author_name": "Liang Jeff", "author_url": "", "post_date": "2018-03-05T14:58:33.580000", "content": "Very appreciate！
", "votes": 0, "replies": [] } ] }, { "id": 289046, "author_name": "MPWARE", "author_url": "", "post_date": "2018-02-28T20:51:59.273000", "content": "Thanks a lot!
\n\nHowever, I thought all small holes in each mask should be filled in for image 5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974?\nFor instance, for mask 888a9bf5e5db6a92cdd9556069010b50d4faa869c225708181e3914c5e592fdd. What do you think about it?
\n\nAlso, is 1 pixel width (short line) masks really valid? For instance, for image 02903040e19ddf92f452907644ad3822918f54af41dd85e5a3fe3e1b6d6f9339 and mask: ee43930956917b35e8ea9950119904ee43eeaf297ae2d0999a96bce06465119b
", "votes": 0, "replies": [] }, { "id": 288444, "author_name": "Gábor Vecsei", "author_url": "", "post_date": "2018-02-27T13:51:49.667000", "content": "This would be really nice!
", "votes": 0, "replies": [] }, { "id": 287925, "author_name": "MichaelCarlon", "author_url": "", "post_date": "2018-02-26T08:36:39.867000", "content": "It would be very helpful to all; I don't see a downside.
", "votes": 0, "replies": [ { "id": 287928, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-02-26T08:42:36.100000", "content": "A potential downside could be if organizers are not comfortable sharing annotations publicly yet. In this case I'll share my fixes on the forum, which would be less convenient to use and collaborate, but still better than nothing.
", "votes": 7, "replies": [] } ] }, { "id": 297383, "author_name": "", "author_url": "", "post_date": "2018-03-17T00:27:33.317000", "content": "", "votes": 0, "replies": [ { "id": 301391, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-03-22T17:32:04.793000", "content": "Please see below: https://www.kaggle.com/c/data-science-bowl-2018/discussion/50518#290915
", "votes": 1, "replies": [] } ] }, { "id": 289589, "author_name": "DStjhb", "author_url": "", "post_date": "2018-03-01T22:59:36.017000", "content": "Thank you!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "287712": "I'd like to posts fixed masks and images, ideally to a public github repo (with a link to the competition, and commiting changes separately), so that we can avoid wasting everyone's time fixing the same issues, and hopefully achieve better quality. But according to general kaggle competition rules (7B), this is not allowed. But maybe it's fine to do it for this particular dataset and competition?\n\nWhy github and not drop a zip archive? Because github supports pull requests, showing what's changed, diffs for images, etc. - so it's much more useful than dropping a zip archive IMO.\n\n**Update:** The repo is up: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes, many thanks to Kaggle and Carpenter lab!", "288558": "It is fine with us if you create a public repo for the curated masks. We can treat it as external public data. Please post a link in the external data thread so the community can see it. ", "289462": "Hi, I made a small check yesterday and found other 394 \"broken\" masks (out of 28993, but still). To see such masks, please take a look at images attached. I will try to fix it, but perhaps someone else can do it earlier as I am leaving tonight and won't have an access to my PC for a week. \n\nThere are some tips to find these masks and to fix them:\n\n1. Find problematic masks\n-------------------------\n\nFirst of all, I use the following code to detect masks with holes. To be honest, I just wanted to detect all possible contours in order to create bounding boxes... :)\n\n data_base = '../data/stage1_train'\n for img_id in tqdm(os.listdir(data_base)):\n images_folder = '{}/{}/images/'.format(data_base, img_id)\n masks_folder = '{}/{}/masks/'.format(data_base, img_id)\n img_path = '{}/{}'.format(images_folder, os.listdir(images_folder)[0])\n\n for mask_id in os.listdir(masks_folder):\n counter_all += 1\n mask_path = '{}/{}'.format(masks_folder, mask_id)\n mask_img = cv2.imread(mask_path)\n mask = cv2.cvtColor(mask_img, cv2.COLOR_RGB2GRAY )\n\n _, contours, _ = cv2.findContours(\n mask,\n cv2.RETR_TREE,\n cv2.CHAIN_APPROX_SIMPLE\n )\n\n for cnt_num, c in enumerate(contours):\n cnt = contours[cnt_num]\n x, y, w, h = cv2.boundingRect(cnt)\n cv2.rectangle(mask_img, (x, y), (x + w, y + h), (255, 0, 0), 2)\n\n if len(contours) > 1:\n plt.imshow(mask_img)\n plt.show()\n\nThis gives us the following statistics:\n\n Bad masks: 394\n All masks: 28993\n Bad masks in %: 0.01359\n\nAnd a dictionary with a number of masks for each amount of contours per mask:\n\n 1: 28599,\n 2: 243,\n 3: 57,\n 4: 29,\n 5: 17,\n 6: 16,\n 7: 7,\n 8: 9,\n 9: 7,\n 10: 2,\n 11: 5,\n 12: 1,\n 13: 1\n\n2. Fix problematic masks\n-------------------------\n\nNot certainly clear how to fix, but there are two main methods that I'd like to try first of all\n\n - https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html#closing\n - https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.ndimage.morphology.binary_fill_holes.html#scipy-ndimage-morphology-binary-fill-holes\n\nWe'll keep posted if our team have any updates. But we don't mind anyone can do it before we do ;)\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/289462/8661/bad_mask_001.PNG", "288450": "I will see if I can get an official answer from the Kaggle folks on this. \n\nCertainly I do not object, and the images and annotations are licensed Creative Commons 0 (https://www.kaggle.com/c/data-science-bowl-2018/discussion/47864) although it is true that general kaggle competition rules 7B seems to over-ride that at least while the competition is in process. You'd only be releasing a tiny subset of the images anyway.", "287910": "Awesome!!!", "287793": "This idea is great!", "287937": "The bad annotation do affect results. You work will help to our results. Thanks!", "3501401": "This is the exact reason why this GitHub repo is a game-changer over a standard ZIP file:\nVisual Image Diffs: We can see the exact pixel changes between the original and fixed masks directly in the browser.\n\nNo Duplicated Effort: Everyone benefits from the same clean data instead of hundreds of teams fixing the same images independently.\n\nOpen Collaboration: Anyone can submit a Pull Request if they find and fix more bad masks.\n\nChange Tracking: The commit history keeps a clear log of what was changed, by whom, and why", "311426": "@Konstantin, is mask_tools.py the only pre-processing code you used so that one can use it on the current test data in stage 2? If not one can plan on not using this data for final model submission in 2 days. Thanks.", "308225": "@Konstantin Lopuhin , It seems that the mask in red don't exist in the original image \nebc18868864ad075548cc1784f4f9a237bb98335f9645ee727dac8332a3e3716", "301195": "Hi Konstantin,\n\nThanks so much for providing this prepared data. I used it last time on March 6th but today using it with the same script I get an error. Do you have any idea why?\n\n#Here is the error I get when I try to read it\n\n> > Getting and resizing train images and masks ... 25/664\n> [>.............................] - ETA: 2:31 ValueError \n> Traceback (most recent call last)\n2\n\n--->3 # get test_data\n\n 4 test_img,test_img_sizes = read_test_data()\n ", "299240": "Thanks for your great job. I have a question. The official train dataset has 670 images but you only have 664. Why do your delete 6 images? ", "290854": "Thx for your awesome work @Konstantin Lopuhin . I download your repo from github. I find there are only 664 pictures in the stage1_train, but there are 670 pictures in the raw dataset. Did you delete 6 \"wrong pictures\"? I'm looking forward to your reply, thx.", "289046": "Thanks a lot! \n\nHowever, I thought all small holes in each mask should be filled in for image 5d21acedb3015c1208b31778561f8b1079cca7487399300390c3947f691e3974?\nFor instance, for mask 888a9bf5e5db6a92cdd9556069010b50d4faa869c225708181e3914c5e592fdd. What do you think about it?\n\nAlso, is 1 pixel width (short line) masks really valid? For instance, for image 02903040e19ddf92f452907644ad3822918f54af41dd85e5a3fe3e1b6d6f9339 and mask: ee43930956917b35e8ea9950119904ee43eeaf297ae2d0999a96bce06465119b", "288444": "This would be really nice!", "287925": "It would be very helpful to all; I don't see a downside. ", "297383": "", "289589": "Thank you!" } }