I've released the code to process Emil's mosaics which you can use to augment your dataset:
\n\nhttps://www.kaggle.com/killthekitten/assemble-mosaics
\n\nThis code assembles mosaics using the two CSV files generated with Emil's kernel (you should look for files in comments but not in the topic body):
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51184
\n\nPlease, consider sending a PR or commenting here if you fix something:
\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/rebuild_mosaics.py
\n\nmask = np.load('42.npy').stage1_train/mosaics/*.(png|npy) and stage1_test/mosaics/*.png.I've released the code to process Emil's mosaics which you can use to augment your dataset:
\n\nhttps://www.kaggle.com/killthekitten/assemble-mosaics
\n\nThis code assembles mosaics using the two CSV files generated with Emil's kernel (you should look for files in comments but not in the topic body):
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51184
\n\nPlease, consider sending a PR or commenting here if you fix something:
\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/rebuild_mosaics.py
\n\nmask = np.load('42.npy').stage1_train/mosaics/*.(png|npy) and stage1_test/mosaics/*.png.Thanks for your code. However, I got Memory error while running at 17it (it's large and contains lots of masks) in train:
\n\n\nmasks = np.zeros((*mosaic_shape, channels_count), dtype=np.bool)\nMemoryError\n
Is the np array too large? I didn't see any significant change in memory usage while running.
", "rawMarkdown": "Thanks for your code. However, I got Memory error while running at 17it (it's large and contains lots of masks) in train:\n\n```\nmasks = np.zeros((*mosaic_shape, channels_count), dtype=np.bool)\nMemoryError\n```\n\nIs the np array too large? I didn't see any significant change in memory usage while running.", "votes": null }, { "id": "294005", "postDate": "03/11/2018 03:18:45", "content": "Chances are, you were running on the edge of your memory limit before the mosaics and everything was fine. Now, each mosaic has 4 times bigger area than most of the pictures in the original dataset, but more importantly, the mask depth may go up to 800-900 layers for some of the densely populated mosaics.
\n\nI prefer to use them with small random crops to generate more diverse samples. If you still want to train on full pictures, quick tup would be to decrease your queue size (it's equal to 100 if you base on the matterport mask rcnn code).
", "rawMarkdown": "Chances are, you were running on the edge of your memory limit before the mosaics and everything was fine. Now, each mosaic has 4 times bigger area than most of the pictures in the original dataset, but more importantly, the mask depth may go up to 800-900 layers for some of the densely populated mosaics.\n\nI prefer to use them with small random crops to generate more diverse samples. If you still want to train on full pictures, quick tup would be to decrease your queue size (it's equal to 100 if you base on the matterport mask rcnn code).", "votes": null }, { "id": "294007", "postDate": "03/11/2018 03:24:20", "content": "Oh, sorry, now I get it. Yeah, around 17it there was one of those 800+ layers beasts. How much memory do you have? I've tested the code on a 16 gb machine.
", "rawMarkdown": "Oh, sorry, now I get it. Yeah, around 17it there was one of those 800+ layers beasts. How much memory do you have? I've tested the code on a 16 gb machine.", "votes": null }, { "id": "294096", "postDate": "03/11/2018 09:04:50", "content": "16 GB on my machine. I was thinking it should be a problem but that just happened. Currently I just skip those images.
", "rawMarkdown": "16 GB on my machine. I was thinking it should be a problem but that just happened. Currently I just skip those images.", "votes": null }, { "id": "294197", "postDate": "03/11/2018 14:53:03", "content": "Should definitely work then.
", "rawMarkdown": "Should definitely work then.", "votes": null }, { "id": "299031", "postDate": "03/20/2018 10:53:40", "content": "Works great for me.
\n\nAre you happy about this 'glued together if they touch with at least a single pixel' assumption?
\n\nif I wanted to try with a say, 4 pixel overlap, would I change
\n\nif intersection_map.shape[0] == 0 or intersection_map.shape[1] == 0:\n return mask, 0
\n\nto
\n\nif intersection_map.shape[0] < 4 or intersection_map.shape[1] < 4:\n return mask, 0
\n\n?
", "rawMarkdown": "Works great for me.\n\nAre you happy about this 'glued together if they touch with at least a single pixel' assumption? \n\nif I wanted to try with a say, 4 pixel overlap, would I change\n\nif intersection_map.shape[0] == 0 or intersection_map.shape[1] == 0:\n return mask, 0\n\nto\n\nif intersection_map.shape[0] < 4 or intersection_map.shape[1] < 4:\n return mask, 0\n\n?", "votes": null }, { "id": "299050", "postDate": "03/20/2018 11:54:56", "content": "I haven't been involved in the competition since I published this, so had no chance to try out different thresholds.
\n\nTo change the threshold, I believe you need to hack around this line https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/rebuild_mosaics.py#L75. The intersection map contains the pixel overlap and this line defines what is the minimal overlap to consider a layer pair for merge:
\n\n\n if intersection_map[i].max() == 0:\n
You need to change this to:
\n\n\n if intersection_map[i].max() >= 4:\n
Hi Nikolay, thanks for sharing this code! Did you use these newly generated images as additional training data, which could be added to the old dataset? Or you used them as a separate training dataset? What was an improvement in LB score in either case?
", "rawMarkdown": "Hi Nikolay, thanks for sharing this code! Did you use these newly generated images as additional training data, which could be added to the old dataset? Or you used them as a separate training dataset? What was an improvement in LB score in either case?", "votes": null }, { "id": "300177", "postDate": "03/21/2018 11:32:34", "content": "I've used them as a replacement for the original dataset, mainly to generate richer augmentations. Can't tell much about the effect on LB since I haven't tried to submit since.
", "rawMarkdown": "I've used them as a replacement for the original dataset, mainly to generate richer augmentations. Can't tell much about the effect on LB since I haven't tried to submit since.", "votes": null } ], "comments": [ { "id": 293974, "author_name": "czhangcn", "author_url": "", "post_date": "03/11/2018 01:37:16", "content": "Thanks for your code. However, I got Memory error while running at 17it (it's large and contains lots of masks) in train:
\n\n\nmasks = np.zeros((*mosaic_shape, channels_count), dtype=np.bool)\nMemoryError\n
Is the np array too large? I didn't see any significant change in memory usage while running.
", "votes": null, "replies": [ { "id": 294005, "author_name": "killthekitten", "author_url": "", "post_date": "03/11/2018 03:18:45", "content": "Chances are, you were running on the edge of your memory limit before the mosaics and everything was fine. Now, each mosaic has 4 times bigger area than most of the pictures in the original dataset, but more importantly, the mask depth may go up to 800-900 layers for some of the densely populated mosaics.
\n\nI prefer to use them with small random crops to generate more diverse samples. If you still want to train on full pictures, quick tup would be to decrease your queue size (it's equal to 100 if you base on the matterport mask rcnn code).
", "votes": null, "replies": [] }, { "id": 294007, "author_name": "killthekitten", "author_url": "", "post_date": "03/11/2018 03:24:20", "content": "Oh, sorry, now I get it. Yeah, around 17it there was one of those 800+ layers beasts. How much memory do you have? I've tested the code on a 16 gb machine.
", "votes": null, "replies": [] }, { "id": 294096, "author_name": "czhangcn", "author_url": "", "post_date": "03/11/2018 09:04:50", "content": "16 GB on my machine. I was thinking it should be a problem but that just happened. Currently I just skip those images.
", "votes": null, "replies": [] }, { "id": 294197, "author_name": "killthekitten", "author_url": "", "post_date": "03/11/2018 14:53:03", "content": "Should definitely work then.
", "votes": null, "replies": [] } ] }, { "id": 299031, "author_name": "jinkos", "author_url": "", "post_date": "03/20/2018 10:53:40", "content": "Works great for me.
\n\nAre you happy about this 'glued together if they touch with at least a single pixel' assumption?
\n\nif I wanted to try with a say, 4 pixel overlap, would I change
\n\nif intersection_map.shape[0] == 0 or intersection_map.shape[1] == 0:\n return mask, 0
\n\nto
\n\nif intersection_map.shape[0] < 4 or intersection_map.shape[1] < 4:\n return mask, 0
\n\n?
", "votes": null, "replies": [ { "id": 299050, "author_name": "killthekitten", "author_url": "", "post_date": "03/20/2018 11:54:56", "content": "I haven't been involved in the competition since I published this, so had no chance to try out different thresholds.
\n\nTo change the threshold, I believe you need to hack around this line https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/rebuild_mosaics.py#L75. The intersection map contains the pixel overlap and this line defines what is the minimal overlap to consider a layer pair for merge:
\n\n\n if intersection_map[i].max() == 0:\n
You need to change this to:
\n\n\n if intersection_map[i].max() >= 4:\n
Hi Nikolay, thanks for sharing this code! Did you use these newly generated images as additional training data, which could be added to the old dataset? Or you used them as a separate training dataset? What was an improvement in LB score in either case?
", "votes": null, "replies": [ { "id": 300177, "author_name": "killthekitten", "author_url": "", "post_date": "03/21/2018 11:32:34", "content": "I've used them as a replacement for the original dataset, mainly to generate richer augmentations. Can't tell much about the effect on LB since I haven't tried to submit since.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "293255": "I've released the code to process Emil's mosaics which you can use to augment your dataset:\n\nhttps://www.kaggle.com/killthekitten/assemble-mosaics\n\nThis code assembles mosaics using the two CSV files generated with Emil's kernel (you should look for files in comments but not in the topic body):\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/51184\n\nPlease, consider sending a PR or commenting here if you fix something:\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/rebuild_mosaics.py\n\n* Instances on the edges of mosaic patches are glued together if they touch with at least a single pixel.\n* Masks are saved as a single numpy file which can be then loaded with `mask = np.load('42.npy')`.\n* For simplicity, even non-mosaic images are processed and saved to the new place.\n* Output is saved to `stage1_train/mosaics/*.(png|npy)` and `stage1_test/mosaics/*.png`.\n* One obvious refinement to the numpy files would be to merge them into a single layer and use non-binary encoding (i.e. assign each instance a separate number instead of 0s and 1s), but I didn't do it since it's not compatible with the matterport pipeline. Feel free to PR.", "293974": "Thanks for your code. However, I got Memory error while running at 17it (it's large and contains lots of masks) in train:\n\n```\nmasks = np.zeros((*mosaic_shape, channels_count), dtype=np.bool)\nMemoryError\n```\n\nIs the np array too large? I didn't see any significant change in memory usage while running.", "294005": "Chances are, you were running on the edge of your memory limit before the mosaics and everything was fine. Now, each mosaic has 4 times bigger area than most of the pictures in the original dataset, but more importantly, the mask depth may go up to 800-900 layers for some of the densely populated mosaics.\n\nI prefer to use them with small random crops to generate more diverse samples. If you still want to train on full pictures, quick tup would be to decrease your queue size (it's equal to 100 if you base on the matterport mask rcnn code).", "294007": "Oh, sorry, now I get it. Yeah, around 17it there was one of those 800+ layers beasts. How much memory do you have? I've tested the code on a 16 gb machine.", "294096": "16 GB on my machine. I was thinking it should be a problem but that just happened. Currently I just skip those images.", "294197": "Should definitely work then.", "299031": "Works great for me.\n\nAre you happy about this 'glued together if they touch with at least a single pixel' assumption? \n\nif I wanted to try with a say, 4 pixel overlap, would I change\n\nif intersection_map.shape[0] == 0 or intersection_map.shape[1] == 0:\n return mask, 0\n\nto\n\nif intersection_map.shape[0] < 4 or intersection_map.shape[1] < 4:\n return mask, 0\n\n?", "299050": "I haven't been involved in the competition since I published this, so had no chance to try out different thresholds.\n\nTo change the threshold, I believe you need to hack around this line https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/rebuild_mosaics.py#L75. The intersection map contains the pixel overlap and this line defines what is the minimal overlap to consider a layer pair for merge:\n\n```\n if intersection_map[i].max() == 0:\n```\n\nYou need to change this to:\n\n```\n if intersection_map[i].max() >= 4:\n```", "300168": "Hi Nikolay, thanks for sharing this code! Did you use these newly generated images as additional training data, which could be added to the old dataset? Or you used them as a separate training dataset? What was an improvement in LB score in either case?", "300177": "I've used them as a replacement for the original dataset, mainly to generate richer augmentations. Can't tell much about the effect on LB since I haven't tried to submit since." }, "source": "meta" }