I have been trying to make data augmentation work for a long time, but it has repeatedly hurt my LB score. The way I do augmentation is following: For each (image, list of masks) pair, I generate 8 (image, list of masks) pairs by rotating the image and masks by 90, 180 and 270 degrees, flipping horizontally, flipping vertically and combining these operations to generate 8 different possibilities (A square has eight symmetries).
\n\nHowever, my LB score always drops when I use augmentation. FWIW I am using Mask RCNN. Has anyone else faced a similar problem?
\n\nI initially suspected that I may be doing augmentation incorrectly, but I have checked augmented image and masks: they seem to be alright.
", "messages": [ { "id": "282566", "postDate": "02/14/2018 07:08:09", "content": "I have been trying to make data augmentation work for a long time, but it has repeatedly hurt my LB score. The way I do augmentation is following: For each (image, list of masks) pair, I generate 8 (image, list of masks) pairs by rotating the image and masks by 90, 180 and 270 degrees, flipping horizontally, flipping vertically and combining these operations to generate 8 different possibilities (A square has eight symmetries).
\n\nHowever, my LB score always drops when I use augmentation. FWIW I am using Mask RCNN. Has anyone else faced a similar problem?
\n\nI initially suspected that I may be doing augmentation incorrectly, but I have checked augmented image and masks: they seem to be alright.
", "rawMarkdown": "I have been trying to make data augmentation work for a long time, but it has repeatedly hurt my LB score. The way I do augmentation is following: For each (image, list of masks) pair, I generate 8 (image, list of masks) pairs by rotating the image and masks by 90, 180 and 270 degrees, flipping horizontally, flipping vertically and combining these operations to generate 8 different possibilities (A square has eight symmetries).\n\nHowever, my LB score always drops when I use augmentation. FWIW I am using Mask RCNN. Has anyone else faced a similar problem?\n\nI initially suspected that I may be doing augmentation incorrectly, but I have checked augmented image and masks: they seem to be alright.", "votes": null }, { "id": "282569", "postDate": "02/14/2018 07:16:54", "content": "Did you make sure masks match the image after augmentation? And did you check not doing augmentation for test set?
", "rawMarkdown": "Did you make sure masks match the image after augmentation? And did you check not doing augmentation for test set?", "votes": null }, { "id": "282634", "postDate": "02/14/2018 09:45:12", "content": "Yes, I had checked (image, mask) pairs after augmentation, and they appeared to be correct. I will check again, more throughly.
\n\nI am not doing augmentation at test time.
\n\nAt this point I have two hypotheses:\n- Maybe I am doing augmentation incorrectly. I have visually seen (image, mask) pairs after augmentation and they seem alright, but I'll do more through checking.\n- May be, I just need to train more, since there is more data. I am currently in the process of training much more and will report if I see any boost in the score.
", "rawMarkdown": "Yes, I had checked (image, mask) pairs after augmentation, and they appeared to be correct. I will check again, more throughly. \n\nI am not doing augmentation at test time.\n\nAt this point I have two hypotheses:\n- Maybe I am doing augmentation incorrectly. I have visually seen (image, mask) pairs after augmentation and they seem alright, but I'll do more through checking.\n- May be, I just need to train more, since there is more data. I am currently in the process of training much more and will report if I see any boost in the score.", "votes": null }, { "id": "282706", "postDate": "02/14/2018 12:38:39", "content": "Are you using Mask RCNN library to build the model? There is a random horizontal flip as augmentation I use the code with that and I was able to achieve 0.409 in the LB.
", "rawMarkdown": "Are you using [Mask RCNN][1] library to build the model? There is a random horizontal flip as augmentation I use the code with that and I was able to achieve 0.409 in the LB.\n\n [1]: http://github.com/matterport/Mask_RCNN", "votes": null }, { "id": "282822", "postDate": "02/14/2018 16:46:39", "content": "Yes, I am using the library you referred. I know about random horizontal flip, but I wanted to add more augmentations. I have also got 0.412 with that method, but can't go beyond that even with more augmentations.
", "rawMarkdown": "Yes, I am using the library you referred. I know about random horizontal flip, but I wanted to add more augmentations. I have also got 0.412 with that method, but can't go beyond that even with more augmentations.", "votes": null }, { "id": "282946", "postDate": "02/14/2018 19:55:30", "content": "Check if the augmented images have the same scale than the original. When I was testing some augmentation, the original images are in [0-255] range, after a resize they became floats [0-1]
", "rawMarkdown": "Check if the augmented images have the same scale than the original. When I was testing some augmentation, the original images are in [0-255] range, after a resize they became floats [0-1]", "votes": null }, { "id": "282951", "postDate": "02/14/2018 20:00:09", "content": "Yes, specially if one uses skimage.transform.resize. To avoid that, I use preserve_range=True option and convert the returned image to uint.
Yep. This is a crucial tip (and could likely be responsible for issue)!
\n\nI’m curious, are you augmenting both your training and validation set?
\n\nBut it’s also worth noting that additional augmentation will require additional training (and possibly additional parameter capacity). I’d also avoid comparing augmented scores to non-augmented scores since one model is likely overfit and the other is likely underfit).
", "rawMarkdown": "Yep. This is a crucial tip (and could likely be responsible for issue)!\n\nI’m curious, are you augmenting both your training and validation set?\n\n*But it’s also worth noting that additional augmentation will require additional training (and possibly additional parameter capacity). I’d also avoid comparing augmented scores to non-augmented scores since one model is likely overfit and the other is likely underfit).*", "votes": null }, { "id": "283168", "postDate": "02/15/2018 01:34:21", "content": "I made a submission with 90 degrees rotation horizontal and vertical flips (I used numpy functions flipud rot90 and fliplr) But it still got 0.409 and I think the issue is with detecting the color images (histology). I think we should try some of the color augmentations that Allen mentioned in his post https://www.kaggle.com/c/data-science-bowl-2018/discussion/48130#282959
", "rawMarkdown": "I made a submission with 90 degrees rotation horizontal and vertical flips (I used numpy functions flipud rot90 and fliplr) But it still got 0.409 and I think the issue is with detecting the color images (histology). I think we should try some of the color augmentations that Allen mentioned in his post https://www.kaggle.com/c/data-science-bowl-2018/discussion/48130#282959", "votes": null }, { "id": "283218", "postDate": "02/15/2018 04:30:35", "content": "That's such nice tip. However, I use np.tranpose() and np.flip() to do my augmentations, so there is no chance of this phenomena.
", "rawMarkdown": "That's such nice tip. However, I use np.tranpose() and np.flip() to do my augmentations, so there is no chance of this phenomena.", "votes": null }, { "id": "283271", "postDate": "02/15/2018 06:16:37", "content": "@Allen Goodman, comparing train loss and validation loss will help us to identify if the model is overfitted or not, am I right? If this is the case, we can still compare the two models.
Can it be a result of incorrect splitting between training and validation images?
", "rawMarkdown": "Can it be a result of incorrect splitting between training and validation images?", "votes": null }, { "id": "283395", "postDate": "02/15/2018 13:08:03", "content": "I use one image set for training, and other for validation.
", "rawMarkdown": "I use one image set for training, and other for validation.", "votes": null }, { "id": "283398", "postDate": "02/15/2018 13:11:27", "content": "By now, I have given up on making simple flip/rotation augmentations. Maybe, it is incorrect to expect an improvement by using these augmentations: convolution filters scan over whole images and they will learn same features whether they are located in this part of the image or that part of the image.
\n\nPerhaps augmentations like zoom/crop/brightness/noise will help, but not simple rotation and flip.
\n\nIf someone finds their scores improved by rotation/flip, kindly let me know.
\n\nThanks,\nVivek
", "rawMarkdown": "By now, I have given up on making simple flip/rotation augmentations. Maybe, it is incorrect to expect an improvement by using these augmentations: convolution filters scan over whole images and they will learn same features whether they are located in this part of the image or that part of the image. \n\nPerhaps augmentations like zoom/crop/brightness/noise will help, but not simple rotation and flip.\n\nIf someone finds their scores improved by rotation/flip, kindly let me know.\n\nThanks,\nVivek", "votes": null }, { "id": "283871", "postDate": "02/16/2018 05:22:31", "content": "i haven't try LB submission yet. My experiments showed better results for validation loss as follows:
\n\n.
\n\naugmentation-0: no augmentation
\n\naugmentation-1: horizontal-flip, vertical-flip, various 90-degree rotation
\n\naugmentation-2: augmentation-1 and scale/shift/rotate
\n\naugmentation-3: augmentation-1 and contrast/brightness shift
\n\n.
\n\nin mask-rcnn, there are 4 loss:
\n\nrpn_cls_loss/rpn_reg_loss : no so much affected by augmentation-1/2/3
rcnn_cls_loss : augmentation-1/2 is better than augmentation-0. augmentation-2 is the best. augmentation-3 get worst
rcnn_reg_loss : no so much affected by augmentation-1/2/3
mask_loss : no so much affected by augmentation-1/2/3
Are you using the standard configuration that came with that model? Or are you changing some of the parameters?
", "rawMarkdown": "Are you using the standard configuration that came with that model? Or are you changing some of the parameters?", "votes": null }, { "id": "286325", "postDate": "02/22/2018 01:43:26", "content": "I have changed the image size mask size and rpn anchor scales from the config. I am also doing horizontal flipping and rotation as augmentation.
", "rawMarkdown": "I have changed the image size mask size and rpn anchor scales from the config. I am also doing horizontal flipping and rotation as augmentation.", "votes": null } ], "comments": [ { "id": 282569, "author_name": "mrtzmilani", "author_url": "", "post_date": "02/14/2018 07:16:54", "content": "Did you make sure masks match the image after augmentation? And did you check not doing augmentation for test set?
", "votes": null, "replies": [ { "id": 282634, "author_name": "thevivekpandey", "author_url": "", "post_date": "02/14/2018 09:45:12", "content": "Yes, I had checked (image, mask) pairs after augmentation, and they appeared to be correct. I will check again, more throughly.
\n\nI am not doing augmentation at test time.
\n\nAt this point I have two hypotheses:\n- Maybe I am doing augmentation incorrectly. I have visually seen (image, mask) pairs after augmentation and they seem alright, but I'll do more through checking.\n- May be, I just need to train more, since there is more data. I am currently in the process of training much more and will report if I see any boost in the score.
", "votes": null, "replies": [] } ] }, { "id": 282706, "author_name": "akilaw", "author_url": "", "post_date": "02/14/2018 12:38:39", "content": "Are you using Mask RCNN library to build the model? There is a random horizontal flip as augmentation I use the code with that and I was able to achieve 0.409 in the LB.
", "votes": null, "replies": [ { "id": 282822, "author_name": "thevivekpandey", "author_url": "", "post_date": "02/14/2018 16:46:39", "content": "Yes, I am using the library you referred. I know about random horizontal flip, but I wanted to add more augmentations. I have also got 0.412 with that method, but can't go beyond that even with more augmentations.
", "votes": null, "replies": [] }, { "id": 283168, "author_name": "akilaw", "author_url": "", "post_date": "02/15/2018 01:34:21", "content": "I made a submission with 90 degrees rotation horizontal and vertical flips (I used numpy functions flipud rot90 and fliplr) But it still got 0.409 and I think the issue is with detecting the color images (histology). I think we should try some of the color augmentations that Allen mentioned in his post https://www.kaggle.com/c/data-science-bowl-2018/discussion/48130#282959
", "votes": null, "replies": [] }, { "id": 286221, "author_name": "panserbjorne", "author_url": "", "post_date": "02/21/2018 21:23:28", "content": "Are you using the standard configuration that came with that model? Or are you changing some of the parameters?
", "votes": null, "replies": [] }, { "id": 286325, "author_name": "akilaw", "author_url": "", "post_date": "02/22/2018 01:43:26", "content": "I have changed the image size mask size and rpn anchor scales from the config. I am also doing horizontal flipping and rotation as augmentation.
", "votes": null, "replies": [] } ] }, { "id": 282946, "author_name": "aglotero", "author_url": "", "post_date": "02/14/2018 19:55:30", "content": "Check if the augmented images have the same scale than the original. When I was testing some augmentation, the original images are in [0-255] range, after a resize they became floats [0-1]
", "votes": null, "replies": [ { "id": 282951, "author_name": "mrtzmilani", "author_url": "", "post_date": "02/14/2018 20:00:09", "content": "Yes, specially if one uses skimage.transform.resize. To avoid that, I use preserve_range=True option and convert the returned image to uint.
Yep. This is a crucial tip (and could likely be responsible for issue)!
\n\nI’m curious, are you augmenting both your training and validation set?
\n\nBut it’s also worth noting that additional augmentation will require additional training (and possibly additional parameter capacity). I’d also avoid comparing augmented scores to non-augmented scores since one model is likely overfit and the other is likely underfit).
", "votes": null, "replies": [] }, { "id": 283218, "author_name": "thevivekpandey", "author_url": "", "post_date": "02/15/2018 04:30:35", "content": "That's such nice tip. However, I use np.tranpose() and np.flip() to do my augmentations, so there is no chance of this phenomena.
", "votes": null, "replies": [] }, { "id": 283271, "author_name": "mrtzmilani", "author_url": "", "post_date": "02/15/2018 06:16:37", "content": "@Allen Goodman, comparing train loss and validation loss will help us to identify if the model is overfitted or not, am I right? If this is the case, we can still compare the two models.
Can it be a result of incorrect splitting between training and validation images?
", "votes": null, "replies": [ { "id": 283395, "author_name": "thevivekpandey", "author_url": "", "post_date": "02/15/2018 13:08:03", "content": "I use one image set for training, and other for validation.
", "votes": null, "replies": [] } ] }, { "id": 283398, "author_name": "thevivekpandey", "author_url": "", "post_date": "02/15/2018 13:11:27", "content": "By now, I have given up on making simple flip/rotation augmentations. Maybe, it is incorrect to expect an improvement by using these augmentations: convolution filters scan over whole images and they will learn same features whether they are located in this part of the image or that part of the image.
\n\nPerhaps augmentations like zoom/crop/brightness/noise will help, but not simple rotation and flip.
\n\nIf someone finds their scores improved by rotation/flip, kindly let me know.
\n\nThanks,\nVivek
", "votes": null, "replies": [ { "id": 283871, "author_name": "hengck23", "author_url": "", "post_date": "02/16/2018 05:22:31", "content": "i haven't try LB submission yet. My experiments showed better results for validation loss as follows:
\n\n.
\n\naugmentation-0: no augmentation
\n\naugmentation-1: horizontal-flip, vertical-flip, various 90-degree rotation
\n\naugmentation-2: augmentation-1 and scale/shift/rotate
\n\naugmentation-3: augmentation-1 and contrast/brightness shift
\n\n.
\n\nin mask-rcnn, there are 4 loss:
\n\nrpn_cls_loss/rpn_reg_loss : no so much affected by augmentation-1/2/3
rcnn_cls_loss : augmentation-1/2 is better than augmentation-0. augmentation-2 is the best. augmentation-3 get worst
rcnn_reg_loss : no so much affected by augmentation-1/2/3
mask_loss : no so much affected by augmentation-1/2/3