For those using Matterport's MaskRCNN you may want to be aware of this:
\n\nutils uses scipy.misc.imresize() to resize any images with scale != 1. scipy.misc.imresize outputs type uint8 in the range 0 - 255 as default. This means that you can be sending in images that are float 0-1 range, say, but the result of the resizing will transform them to 0 -255 scale.
\n\nThe problem manifests most severely when you have some images that are being resized (scale != 1) and some that are not (scale == 1). In this instance if the images you feed in are in a range that is not 0 - 255 you are going to get a mismatch in images being fed into the model.
\n\nThis may be a problem if you are using greyscale images via skimage.color.rgb2gray(), for example. skimage.color.rgb2gray outputs type float in the range 0 - 1 as default.
\n\nIn most cases in this competition the image sizes are such that this doesn't cause issues as they all end up with scale != 1, but depending if you are cropping specific sizes, or if you are using something like 256 x 256 as you image input dims, you could run into trouble.
", "messages": [ { "id": "309409", "postDate": "04/05/2018 09:02:57", "content": "For those using Matterport's MaskRCNN you may want to be aware of this:
\n\nutils uses scipy.misc.imresize() to resize any images with scale != 1. scipy.misc.imresize outputs type uint8 in the range 0 - 255 as default. This means that you can be sending in images that are float 0-1 range, say, but the result of the resizing will transform them to 0 -255 scale.
\n\nThe problem manifests most severely when you have some images that are being resized (scale != 1) and some that are not (scale == 1). In this instance if the images you feed in are in a range that is not 0 - 255 you are going to get a mismatch in images being fed into the model.
\n\nThis may be a problem if you are using greyscale images via skimage.color.rgb2gray(), for example. skimage.color.rgb2gray outputs type float in the range 0 - 1 as default.
\n\nIn most cases in this competition the image sizes are such that this doesn't cause issues as they all end up with scale != 1, but depending if you are cropping specific sizes, or if you are using something like 256 x 256 as you image input dims, you could run into trouble.
", "rawMarkdown": "For those using Matterport's MaskRCNN you may want to be aware of this:\n\nutils uses scipy.misc.imresize() to resize any images with scale != 1. scipy.misc.imresize outputs type uint8 in the range 0 - 255 as default. This means that you can be sending in images that are float 0-1 range, say, but the result of the resizing will transform them to 0 -255 scale. \n\nThe problem manifests most severely when you have some images that are being resized (scale != 1) and some that are not (scale == 1). In this instance if the images you feed in are in a range that is not 0 - 255 you are going to get a mismatch in images being fed into the model.\n\nThis may be a problem if you are using greyscale images via skimage.color.rgb2gray(), for example. skimage.color.rgb2gray outputs type float in the range 0 - 1 as default. \n\nIn most cases in this competition the image sizes are such that this doesn't cause issues as they all end up with scale != 1, but depending if you are cropping specific sizes, or if you are using something like 256 x 256 as you image input dims, you could run into trouble.", "votes": null }, { "id": "309721", "postDate": "04/05/2018 21:20:26", "content": "Thanks @fergusoci for the catch. Is this in utils.py ? I have used a modified matterport's mask_RCNN code before but working unet at the moment and plan on going back to it tomorrow. I used it with all my images resized to 256x256 but ran without any error. Did you get an error message as a result of this bug?
", "rawMarkdown": "Thanks @fergusoci for the catch. Is this in utils.py ? I have used a modified matterport's mask_RCNN code before but working unet at the moment and plan on going back to it tomorrow. I used it with all my images resized to 256x256 but ran without any error. Did you get an error message as a result of this bug?", "votes": null }, { "id": "309726", "postDate": "04/05/2018 21:34:00", "content": "No error, the code runs fine, it just might not be doing what you want it to.
\n\nIf you were using greyscale images scaled on a 0-1 range (say as a result of converting to greyscale via skimage.color.rgb2gray(), for example), and you were using 256 x 256 inputs then I think this could be very problematic. I think about half of your training images would be flowing through your model on a 0-1 scale and the remainder on a 0-255 scale.
\n\nIf your images inputs were on a 0 - 255 scale then you should be ok, this will not have affected anything.
", "rawMarkdown": "No error, the code runs fine, it just might not be doing what you want it to. \n\nIf you were using greyscale images scaled on a 0-1 range (say as a result of converting to greyscale via skimage.color.rgb2gray(), for example), and you were using 256 x 256 inputs then I think this could be very problematic. I think about half of your training images would be flowing through your model on a 0-1 scale and the remainder on a 0-255 scale.\n\nIf your images inputs were on a 0 - 255 scale then you should be ok, this will not have affected anything.", "votes": null }, { "id": "309730", "postDate": "04/05/2018 21:42:52", "content": "In which function is that resizing done? I have downloaded his latest code but yet to use it. Are you using the latest?
", "rawMarkdown": "In which function is that resizing done? I have downloaded his latest code but yet to use it. Are you using the latest?", "votes": null }, { "id": "309731", "postDate": "04/05/2018 21:48:04", "content": "thanks @fergusoci! I saw someone filed an issue for this on matterport Mask-RCNN github.
\n\nWe were trying a normalization like this: img = (img - mean(img)) /std(img)\n@YaGana, and @ZhengLi, this bug might be the reason why the normalization did not help.
", "rawMarkdown": "thanks @fergusoci! I saw someone filed an issue for this on matterport Mask-RCNN github.\n\nWe were trying a normalization like this: img = (img - mean(img)) /std(img)\n@YaGana, and @ZhengLi, this bug might be the reason why the normalization did not help.", "votes": null }, { "id": "309740", "postDate": "04/05/2018 22:19:40", "content": "Thanks @fergusoci , I found where it is. I will just comment it out for now.
\n\n@Mehul, let me know if you try the scaling again without this bug. I will just comment it out for now.
", "rawMarkdown": "Thanks @fergusoci , I found where it is. I will just comment it out for now. \n\n@Mehul, let me know if you try the scaling again without this bug. I will just comment it out for now.", "votes": null }, { "id": "309749", "postDate": "04/05/2018 22:44:14", "content": "I saw Waleed fixed this issue in the latest Mask RCNN code. the commit is here \nhttps://github.com/matterport/Mask_RCNN/commit/7920f6faaf78145fe1c35036059f71dd63448868#diff-04d2c9518498da64bfa9db44c6211645
\n\nyou can git clone https://github.com/matterport/Mask_RCNN.git to get the latest code. \nWaleed made a couple of updates int he last few days;
", "rawMarkdown": "I saw Waleed fixed this issue in the latest Mask RCNN code. the commit is here \nhttps://github.com/matterport/Mask_RCNN/commit/7920f6faaf78145fe1c35036059f71dd63448868#diff-04d2c9518498da64bfa9db44c6211645\n\nyou can git clone https://github.com/matterport/Mask_RCNN.git to get the latest code. \nWaleed made a couple of updates int he last few days;", "votes": null }, { "id": "309759", "postDate": "04/05/2018 23:09:28", "content": "Thanks @Mehul, I just checked it was updated in the last hour.
", "rawMarkdown": "Thanks @Mehul, I just checked it was updated in the last hour.", "votes": null }, { "id": "309761", "postDate": "04/05/2018 23:20:58", "content": "Yup you are right. Actually, he is my friend and he is awesome at fixing the issues quickly! :-) I posted a comment in the relevant issue and he just fixed it
", "rawMarkdown": "Yup you are right. Actually, he is my friend and he is awesome at fixing the issues quickly! :-) I posted a comment in the relevant issue and he just fixed it", "votes": null }, { "id": "310343", "postDate": "04/07/2018 07:56:21", "content": "It seems matterport new code, using skimage will hurt detection result a bit. For me it lower down lb by 0.001, and local validation map 0.002.
\nUsing skimage resize
\nimage shape: (519, 253, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8
\nimage resizeshape: (1024, 499, 3) min: 10.00000 max: 224.49437 mean: 27.53918 unique: 427313 float64
Using scipy resize
\nimage shape: (519, 253, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8
\nimage resizeshape: (1024, 499, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8
@goldenlock , I just found out this may be why the validation loss is gone up by almost a factor of 2 when I loaded a saved model to continue training. So when I saw your post I reverted back to the previous files, continued training and the validation losses are now as expected.
\n\nHaving said that, I just saw a post by @Waleed whre he mentioned a couple of fixes to do with resizing in this thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/53705
\n\nTake a look at his latest fixes, I have downloaded but not used it yet.
", "rawMarkdown": "goldenlock , I just found out this may be why the validation loss is gone up by almost a factor of 2 when I loaded a saved model to continue training. So when I saw your post I reverted back to the previous files, continued training and the validation losses are now as expected.\n\nHaving said that, I just saw a post by @Waleed whre he mentioned a couple of fixes to do with resizing in this thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/53705\n\nTake a look at his latest fixes, I have downloaded but not used it yet.", "votes": null }, { "id": "310453", "postDate": "04/07/2018 15:26:42", "content": "Thanks! I will try latest code and feed back here.
", "rawMarkdown": "Thanks! I will try latest code and feed back here.", "votes": null }, { "id": "310468", "postDate": "04/07/2018 16:32:49", "content": "Well I tried new code, still the same, wether you train or inference, I would suggest you use latest code but in resize_image change to use old way scipy image resize not using skimage resize. I tested on model using old code training, it has map 57926, using latest code it get 5756 but if change to use scipy image resize I got 5807, I still think scipy resize better option wether train or inference )\nAfter retrain I can not get same lb score, 0.001 lower, so I change back to use old matterport version.
", "rawMarkdown": "Well I tried new code, still the same, wether you train or inference, I would suggest you use latest code but in resize_image change to use old way scipy image resize not using skimage resize. I tested on model using old code training, it has map 57926, using latest code it get 5756 but if change to use scipy image resize I got 5807, I still think scipy resize better option wether train or inference )\nAfter retrain I can not get same lb score, 0.001 lower, so I change back to use old matterport version.", "votes": null }, { "id": "311956", "postDate": "04/11/2018 01:21:21", "content": "Just trying to revisit the resizing issue, and what my understanding is:
\n\nI think we probably should use scipy.misc.imresize. Based on https://github.com/python-pillow/Pillow/issues/2865, and http://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.resize, it seems that skimage.transform.resize is not using anti-aliasing by default, which will create artifact when down sampling. It seems scipy.misc.imresize has that implemented. I think 0.14 version of skimage.transform.resize will have anti-aliasing implemented, but we need to install from their source. scipy.misc.imresize will deprecate soon, so skimage.transform.resize will be the future and hope it catch up in time.
For now I think we can detect the image range before using scipy.misc.imresize. If the range is [0,1], we divide the resized image by 255. This will manually preserve the range ( in case it is necessary).
I think the best way is trying to avoid resizing as much as possible.
I ended up doing something similar to what you mentioned. I used the following:
\n\nimage_dtype = image.dtype\nimage_max = np.max(image)
\n\nimage = scipy.misc.imresize(image, (round(h * scale), round(w * scale)))\nimage = image.astype(image_dtype)\nif image_max <= 1:\n image = image / 255.
", "rawMarkdown": "I ended up doing something similar to what you mentioned. I used the following:\n\nimage_dtype = image.dtype\nimage_max = np.max(image)\n\nimage = scipy.misc.imresize(image, (round(h * scale), round(w * scale)))\nimage = image.astype(image_dtype)\nif image_max <= 1:\n image = image / 255.", "votes": null } ], "comments": [ { "id": 309721, "author_name": "sheriytm", "author_url": "", "post_date": "04/05/2018 21:20:26", "content": "Thanks @fergusoci for the catch. Is this in utils.py ? I have used a modified matterport's mask_RCNN code before but working unet at the moment and plan on going back to it tomorrow. I used it with all my images resized to 256x256 but ran without any error. Did you get an error message as a result of this bug?
", "votes": null, "replies": [ { "id": 309726, "author_name": "fergusoci", "author_url": "", "post_date": "04/05/2018 21:34:00", "content": "No error, the code runs fine, it just might not be doing what you want it to.
\n\nIf you were using greyscale images scaled on a 0-1 range (say as a result of converting to greyscale via skimage.color.rgb2gray(), for example), and you were using 256 x 256 inputs then I think this could be very problematic. I think about half of your training images would be flowing through your model on a 0-1 scale and the remainder on a 0-255 scale.
\n\nIf your images inputs were on a 0 - 255 scale then you should be ok, this will not have affected anything.
", "votes": null, "replies": [] }, { "id": 309730, "author_name": "sheriytm", "author_url": "", "post_date": "04/05/2018 21:42:52", "content": "In which function is that resizing done? I have downloaded his latest code but yet to use it. Are you using the latest?
", "votes": null, "replies": [] }, { "id": 309731, "author_name": "mpsampat", "author_url": "", "post_date": "04/05/2018 21:48:04", "content": "thanks @fergusoci! I saw someone filed an issue for this on matterport Mask-RCNN github.
\n\nWe were trying a normalization like this: img = (img - mean(img)) /std(img)\n@YaGana, and @ZhengLi, this bug might be the reason why the normalization did not help.
", "votes": null, "replies": [] }, { "id": 309740, "author_name": "sheriytm", "author_url": "", "post_date": "04/05/2018 22:19:40", "content": "Thanks @fergusoci , I found where it is. I will just comment it out for now.
\n\n@Mehul, let me know if you try the scaling again without this bug. I will just comment it out for now.
", "votes": null, "replies": [] }, { "id": 309749, "author_name": "mpsampat", "author_url": "", "post_date": "04/05/2018 22:44:14", "content": "I saw Waleed fixed this issue in the latest Mask RCNN code. the commit is here \nhttps://github.com/matterport/Mask_RCNN/commit/7920f6faaf78145fe1c35036059f71dd63448868#diff-04d2c9518498da64bfa9db44c6211645
\n\nyou can git clone https://github.com/matterport/Mask_RCNN.git to get the latest code. \nWaleed made a couple of updates int he last few days;
", "votes": null, "replies": [] }, { "id": 309759, "author_name": "sheriytm", "author_url": "", "post_date": "04/05/2018 23:09:28", "content": "Thanks @Mehul, I just checked it was updated in the last hour.
", "votes": null, "replies": [] }, { "id": 309761, "author_name": "mpsampat", "author_url": "", "post_date": "04/05/2018 23:20:58", "content": "Yup you are right. Actually, he is my friend and he is awesome at fixing the issues quickly! :-) I posted a comment in the relevant issue and he just fixed it
", "votes": null, "replies": [] } ] }, { "id": 310343, "author_name": "goldenlock", "author_url": "", "post_date": "04/07/2018 07:56:21", "content": "It seems matterport new code, using skimage will hurt detection result a bit. For me it lower down lb by 0.001, and local validation map 0.002.
\nUsing skimage resize
\nimage shape: (519, 253, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8
\nimage resizeshape: (1024, 499, 3) min: 10.00000 max: 224.49437 mean: 27.53918 unique: 427313 float64
Using scipy resize
\nimage shape: (519, 253, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8
\nimage resizeshape: (1024, 499, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8
@goldenlock , I just found out this may be why the validation loss is gone up by almost a factor of 2 when I loaded a saved model to continue training. So when I saw your post I reverted back to the previous files, continued training and the validation losses are now as expected.
\n\nHaving said that, I just saw a post by @Waleed whre he mentioned a couple of fixes to do with resizing in this thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/53705
\n\nTake a look at his latest fixes, I have downloaded but not used it yet.
", "votes": null, "replies": [] }, { "id": 310453, "author_name": "goldenlock", "author_url": "", "post_date": "04/07/2018 15:26:42", "content": "Thanks! I will try latest code and feed back here.
", "votes": null, "replies": [] }, { "id": 310468, "author_name": "goldenlock", "author_url": "", "post_date": "04/07/2018 16:32:49", "content": "Well I tried new code, still the same, wether you train or inference, I would suggest you use latest code but in resize_image change to use old way scipy image resize not using skimage resize. I tested on model using old code training, it has map 57926, using latest code it get 5756 but if change to use scipy image resize I got 5807, I still think scipy resize better option wether train or inference )\nAfter retrain I can not get same lb score, 0.001 lower, so I change back to use old matterport version.
", "votes": null, "replies": [] }, { "id": 311956, "author_name": "ruiforecasting", "author_url": "", "post_date": "04/11/2018 01:21:21", "content": "Just trying to revisit the resizing issue, and what my understanding is:
\n\nI think we probably should use scipy.misc.imresize. Based on https://github.com/python-pillow/Pillow/issues/2865, and http://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.resize, it seems that skimage.transform.resize is not using anti-aliasing by default, which will create artifact when down sampling. It seems scipy.misc.imresize has that implemented. I think 0.14 version of skimage.transform.resize will have anti-aliasing implemented, but we need to install from their source. scipy.misc.imresize will deprecate soon, so skimage.transform.resize will be the future and hope it catch up in time.
For now I think we can detect the image range before using scipy.misc.imresize. If the range is [0,1], we divide the resized image by 255. This will manually preserve the range ( in case it is necessary).
I think the best way is trying to avoid resizing as much as possible.
I ended up doing something similar to what you mentioned. I used the following:
\n\nimage_dtype = image.dtype\nimage_max = np.max(image)
\n\nimage = scipy.misc.imresize(image, (round(h * scale), round(w * scale)))\nimage = image.astype(image_dtype)\nif image_max <= 1:\n image = image / 255.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "309409": "For those using Matterport's MaskRCNN you may want to be aware of this:\n\nutils uses scipy.misc.imresize() to resize any images with scale != 1. scipy.misc.imresize outputs type uint8 in the range 0 - 255 as default. This means that you can be sending in images that are float 0-1 range, say, but the result of the resizing will transform them to 0 -255 scale. \n\nThe problem manifests most severely when you have some images that are being resized (scale != 1) and some that are not (scale == 1). In this instance if the images you feed in are in a range that is not 0 - 255 you are going to get a mismatch in images being fed into the model.\n\nThis may be a problem if you are using greyscale images via skimage.color.rgb2gray(), for example. skimage.color.rgb2gray outputs type float in the range 0 - 1 as default. \n\nIn most cases in this competition the image sizes are such that this doesn't cause issues as they all end up with scale != 1, but depending if you are cropping specific sizes, or if you are using something like 256 x 256 as you image input dims, you could run into trouble.", "309721": "Thanks @fergusoci for the catch. Is this in utils.py ? I have used a modified matterport's mask_RCNN code before but working unet at the moment and plan on going back to it tomorrow. I used it with all my images resized to 256x256 but ran without any error. Did you get an error message as a result of this bug?", "309726": "No error, the code runs fine, it just might not be doing what you want it to. \n\nIf you were using greyscale images scaled on a 0-1 range (say as a result of converting to greyscale via skimage.color.rgb2gray(), for example), and you were using 256 x 256 inputs then I think this could be very problematic. I think about half of your training images would be flowing through your model on a 0-1 scale and the remainder on a 0-255 scale.\n\nIf your images inputs were on a 0 - 255 scale then you should be ok, this will not have affected anything.", "309730": "In which function is that resizing done? I have downloaded his latest code but yet to use it. Are you using the latest?", "309731": "thanks @fergusoci! I saw someone filed an issue for this on matterport Mask-RCNN github.\n\nWe were trying a normalization like this: img = (img - mean(img)) /std(img)\n@YaGana, and @ZhengLi, this bug might be the reason why the normalization did not help.", "309740": "Thanks @fergusoci , I found where it is. I will just comment it out for now. \n\n@Mehul, let me know if you try the scaling again without this bug. I will just comment it out for now.", "309749": "I saw Waleed fixed this issue in the latest Mask RCNN code. the commit is here \nhttps://github.com/matterport/Mask_RCNN/commit/7920f6faaf78145fe1c35036059f71dd63448868#diff-04d2c9518498da64bfa9db44c6211645\n\nyou can git clone https://github.com/matterport/Mask_RCNN.git to get the latest code. \nWaleed made a couple of updates int he last few days;", "309759": "Thanks @Mehul, I just checked it was updated in the last hour.", "309761": "Yup you are right. Actually, he is my friend and he is awesome at fixing the issues quickly! :-) I posted a comment in the relevant issue and he just fixed it", "310343": "It seems matterport new code, using skimage will hurt detection result a bit. For me it lower down lb by 0.001, and local validation map 0.002. \nUsing skimage resize \nimage shape: (519, 253, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8 \nimage resizeshape: (1024, 499, 3) min: 10.00000 max: 224.49437 mean: 27.53918 unique: 427313 float64 \n\nUsing scipy resize \nimage shape: (519, 253, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8 \nimage resizeshape: (1024, 499, 3) min: 10.00000 max: 225.00000 mean: 27.57925 unique: 216 uint8", "310395": "goldenlock , I just found out this may be why the validation loss is gone up by almost a factor of 2 when I loaded a saved model to continue training. So when I saw your post I reverted back to the previous files, continued training and the validation losses are now as expected.\n\nHaving said that, I just saw a post by @Waleed whre he mentioned a couple of fixes to do with resizing in this thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/53705\n\nTake a look at his latest fixes, I have downloaded but not used it yet.", "310453": "Thanks! I will try latest code and feed back here.", "310468": "Well I tried new code, still the same, wether you train or inference, I would suggest you use latest code but in resize_image change to use old way scipy image resize not using skimage resize. I tested on model using old code training, it has map 57926, using latest code it get 5756 but if change to use scipy image resize I got 5807, I still think scipy resize better option wether train or inference )\nAfter retrain I can not get same lb score, 0.001 lower, so I change back to use old matterport version.", "311956": "Just trying to revisit the resizing issue, and what my understanding is:\n\n1. I think we probably should use scipy.misc.imresize. Based on https://github.com/python-pillow/Pillow/issues/2865, and http://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.resize, it seems that skimage.transform.resize is not using anti-aliasing by default, which will create artifact when down sampling. It seems scipy.misc.imresize has that implemented. I think 0.14 version of skimage.transform.resize will have anti-aliasing implemented, but we need to install from their source. scipy.misc.imresize will deprecate soon, so skimage.transform.resize will be the future and hope it catch up in time. \n\n2. For now I think we can detect the image range before using scipy.misc.imresize. If the range is [0,1], we divide the resized image by 255. This will manually preserve the range ( in case it is necessary). \n\n3. I think the best way is trying to avoid resizing as much as possible.", "312117": "I ended up doing something similar to what you mentioned. I used the following:\n\nimage_dtype = image.dtype\nimage_max = np.max(image)\n\nimage = scipy.misc.imresize(image, (round(h * scale), round(w * scale)))\nimage = image.astype(image_dtype)\nif image_max <= 1:\n image = image / 255." }, "source": "meta" }