Please excuse the question from a newbie in CNN and image competition. I am currently overfitting to the training dataset with a modified Unet model. To be sure, I degraded a bit my CV score in various ways, and it led to better LB scores. I am doing 6 fold CV, and CV score is not really related to LB score.
\n\nThe cause is known: too few images to train from, and a different distribution in train vs test. I am not using external data as I would not be able to annotate it correctly.
\n\nWith other kind of data and other types of ML techniques I would have ideas about what to do, but here the only thing I can think of is to limit number of epochs and do LB probing. Those following me know how much I hate LB probing, hence my question is: is there any other way to limit overfit and evaluate it without resorting to LB probing?
\n\nEdits: I do some data augmentation (eg flipping images), it helps but is not sufficient.
", "messages": [ { "id": "290989", "postDate": "03/05/2018 10:57:03", "content": "Please excuse the question from a newbie in CNN and image competition. I am currently overfitting to the training dataset with a modified Unet model. To be sure, I degraded a bit my CV score in various ways, and it led to better LB scores. I am doing 6 fold CV, and CV score is not really related to LB score.
\n\nThe cause is known: too few images to train from, and a different distribution in train vs test. I am not using external data as I would not be able to annotate it correctly.
\n\nWith other kind of data and other types of ML techniques I would have ideas about what to do, but here the only thing I can think of is to limit number of epochs and do LB probing. Those following me know how much I hate LB probing, hence my question is: is there any other way to limit overfit and evaluate it without resorting to LB probing?
\n\nEdits: I do some data augmentation (eg flipping images), it helps but is not sufficient.
", "rawMarkdown": "Please excuse the question from a newbie in CNN and image competition. I am currently overfitting to the training dataset with a modified Unet model. To be sure, I degraded a bit my CV score in various ways, and it led to better LB scores. I am doing 6 fold CV, and CV score is not really related to LB score.\n\nThe cause is known: too few images to train from, and a different distribution in train vs test. I am not using external data as I would not be able to annotate it correctly.\n\nWith other kind of data and other types of ML techniques I would have ideas about what to do, but here the only thing I can think of is to limit number of epochs and do LB probing. Those following me know how much I hate LB probing, hence my question is: is there any other way to limit overfit and evaluate it without resorting to LB probing?\n\nEdits: I do some data augmentation (eg flipping images), it helps but is not sufficient.", "votes": null }, { "id": "290992", "postDate": "03/05/2018 11:09:29", "content": "Yes, it's annoying. I spent days improving my data augmentation and ended up with poorer results. Now I am having a re-think.\nI don't have the answer, but how about some kind of unsupervised learning approach?\nMaybe some kind of pre-processing that standardises all images?\nOr a GAN?
", "rawMarkdown": "Yes, it's annoying. I spent days improving my data augmentation and ended up with poorer results. Now I am having a re-think.\nI don't have the answer, but how about some kind of unsupervised learning approach?\nMaybe some kind of pre-processing that standardises all images?\nOr a GAN?", "votes": null }, { "id": "290998", "postDate": "03/05/2018 11:40:52", "content": "I do some postprocessing along what you describe (kind of standardization), it helps, but I'm still overfitting. I just tried limiting epoch, and got a small LB improvement (+ 0.003) with a larger CV degradation (- 0.008).
\n\nI also do some data augmentation (eg flipping images), it also helps.
", "rawMarkdown": "I do some postprocessing along what you describe (kind of standardization), it helps, but I'm still overfitting. I just tried limiting epoch, and got a small LB improvement (+ 0.003) with a larger CV degradation (- 0.008).\n\nI also do some data augmentation (eg flipping images), it also helps.", "votes": null }, { "id": "291019", "postDate": "03/05/2018 13:14:57", "content": "How big is the standard deviation of your CV? If yours is as large as mine (about .05) then I feel that a change of .008 doesn't mean that much.
", "rawMarkdown": "How big is the standard deviation of your CV? If yours is as large as mine (about .05) then I feel that a change of .008 doesn't mean that much.", "votes": null }, { "id": "291022", "postDate": "03/05/2018 13:29:26", "content": "Until 2 days ago, my CV and my LB evolved the same way. They aren't anymore, which is why I say I overfit. Magnitude of variation is not what matters, sign of variation matters.
", "rawMarkdown": "Until 2 days ago, my CV and my LB evolved the same way. They aren't anymore, which is why I say I overfit. Magnitude of variation is not what matters, sign of variation matters.", "votes": null }, { "id": "291048", "postDate": "03/05/2018 14:18:11", "content": "@CPMP aside from flipping you can also do rotations, scaling, translation, etc for augmentation. The cv2 and ndimage packages have functions for doing this.
", "rawMarkdown": "CPMP aside from flipping you can also do rotations, scaling, translation, etc for augmentation. The cv2 and ndimage packages have functions for doing this.", "votes": null }, { "id": "291058", "postDate": "03/05/2018 14:50:20", "content": "Thanks. Seems you are saying that more data augmentation is one way to fight overfitting.
\n\nI have tried rotations, it did not help. Seems others reported the same. Scaling, translation, cropping are on my to do list ;) I' using ndimage mostly, but need to learn cv2.
\n\nI also don't do anything with color space, and we know train and test don't have the same color spaces. I'll work on it too.
", "rawMarkdown": "Thanks. Seems you are saying that more data augmentation is one way to fight overfitting. \n\nI have tried rotations, it did not help. Seems others reported the same. Scaling, translation, cropping are on my to do list ;) I' using ndimage mostly, but need to learn cv2.\n\nI also don't do anything with color space, and we know train and test don't have the same color spaces. I'll work on it too.", "votes": null }, { "id": "291074", "postDate": "03/05/2018 15:31:14", "content": "I know I don't have enough credits to interfere with this discussion. But I am just curious that there is rare mention of batch normalisation. Is it because it's so trivial to mention? Actually I got much better results with batch normalisation.
", "rawMarkdown": "I know I don't have enough credits to interfere with this discussion. But I am just curious that there is rare mention of batch normalisation. Is it because it's so trivial to mention? Actually I got much better results with batch normalisation.", "votes": null }, { "id": "291078", "postDate": "03/05/2018 15:37:59", "content": "I don't have that much credit either when it comes to deep learning ;) Yes, in my limited experience batch normalization helps.
", "rawMarkdown": "I don't have that much credit either when it comes to deep learning ;) Yes, in my limited experience batch normalization helps.", "votes": null }, { "id": "291113", "postDate": "03/05/2018 16:42:21", "content": "Yes, the augmentations create more data, so that would help with reducing overfitting. In terms of colorspaces there is also the possibility of varying the brightness of each image. I didn't have much success with that.
", "rawMarkdown": "Yes, the augmentations create more data, so that would help with reducing overfitting. In terms of colorspaces there is also the possibility of varying the brightness of each image. I didn't have much success with that.", "votes": null }, { "id": "291150", "postDate": "03/05/2018 18:04:25", "content": "FWIW, here are the groups I tried to use for local validation, with \"b\" and \"f\" going into train and \"a\", \"c\", \"d\", \"e\" going into validation.
", "rawMarkdown": "FWIW, here are the groups I tried to use for local validation, with \"b\" and \"f\" going into train and \"a\", \"c\", \"d\", \"e\" going into validation.", "votes": null }, { "id": "292167", "postDate": "03/07/2018 15:10:12", "content": "@CPMP - a first time Kaggler here. With this amount of data, your LB is quite impressive, maybe you don't even need to do data augmentation then. :p (please teach us how to do data reduction to achieve your LB please :D ) In our case, applying histogram + only keeping on L-channel + data augmentation/reflection/rotation helped a lot and our model is underfit (due to a lack of GPUs) since our CV IoU score is higher, and as you said, post-processing of the predicted masks is necessary to remove noises (false positives). More importantly, we need a machine with 100 GPUs :D I saw a top ranked guy trained his model for 14k iterations. He must have a Lamborghini machine then.
", "rawMarkdown": "CPMP - a first time Kaggler here. With this amount of data, your LB is quite impressive, maybe you don't even need to do data augmentation then. :p (please teach us how to do data reduction to achieve your LB please :D ) In our case, applying histogram + only keeping on L-channel + data augmentation/reflection/rotation helped a lot and our model is underfit (due to a lack of GPUs) since our CV IoU score is higher, and as you said, post-processing of the predicted masks is necessary to remove noises (false positives). More importantly, we need a machine with 100 GPUs :D I saw a top ranked guy trained his model for 14k iterations. He must have a Lamborghini machine then.", "votes": null }, { "id": "292195", "postDate": "03/07/2018 15:59:16", "content": "Thanks for the praise! ;) I really need to do more data augmentation it seems...
\n\nMy CV IoU score is also way higher than my LB. That is to be expected given the difference in train/test distributions of images.
\n\nRe number of epochs, I use aAdam optimizer. I read that SGD with carefully designed learning rate decay can be better, but I'll keep this for later, if I have time. Adam converges quite rapidly here.
", "rawMarkdown": "Thanks for the praise! ;) I really need to do more data augmentation it seems...\n\nMy CV IoU score is also way higher than my LB. That is to be expected given the difference in train/test distributions of images. \n\nRe number of epochs, I use aAdam optimizer. I read that SGD with carefully designed learning rate decay can be better, but I'll keep this for later, if I have time. Adam converges quite rapidly here.", "votes": null }, { "id": "292208", "postDate": "03/07/2018 16:19:35", "content": "Hey @CPMP, for us the CV IoU score is still higher than the training IoU score, an indicator of underfitting sometimes :) we plan to run more epochs. I did notice when the loss converges slowly, our model performs generally better. When it converges rapidly, our model usually performs poorly. We use a reduced U-net as well, 9 layers with reduced neurons, each layer has a dropout layer with a low dropout rate, and we were playing with dropout layers and didn't see much of an improvement there. Thanks for your information, maybe we should try SGD instead of Adam.
\n\nWe found a Stanford paper for cell segmentation and counting using 9600 cell images downloaded from the Broad Institute, they used VGG-11. I got a feeling there's no best CNN in this competition. Who can create more data + try enough models and train the model long enough can get a better score, given how little image data we have. Still need to back up this theory by doing more experiments. \nhttps://arxiv.org/pdf/1802.10548.pdf
", "rawMarkdown": "Hey @CPMP, for us the CV IoU score is still higher than the training IoU score, an indicator of underfitting sometimes :) we plan to run more epochs. I did notice when the loss converges slowly, our model performs generally better. When it converges rapidly, our model usually performs poorly. We use a reduced U-net as well, 9 layers with reduced neurons, each layer has a dropout layer with a low dropout rate, and we were playing with dropout layers and didn't see much of an improvement there. Thanks for your information, maybe we should try SGD instead of Adam. \n\nWe found a Stanford paper for cell segmentation and counting using 9600 cell images downloaded from the Broad Institute, they used VGG-11. I got a feeling there's no best CNN in this competition. Who can create more data + try enough models and train the model long enough can get a better score, given how little image data we have. Still need to back up this theory by doing more experiments. \nhttps://arxiv.org/pdf/1802.10548.pdf", "votes": null }, { "id": "292481", "postDate": "03/08/2018 02:32:54", "content": "Thanks.
\n\nI misread you. I agree you seem to underfit.
\n\nI have not computed train IoU, but the binary cross entropy loss for train is a bit lower than the val loss.
\n\nI use early stopping or 20 epochs, whichever is smaller.
\n\nI started from this version of unet: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277
", "rawMarkdown": "Thanks.\n\nI misread you. I agree you seem to underfit.\n\nI have not computed train IoU, but the binary cross entropy loss for train is a bit lower than the val loss.\n\nI use early stopping or 20 epochs, whichever is smaller.\n\nI started from this version of unet: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277", "votes": null }, { "id": "292699", "postDate": "03/08/2018 13:20:17", "content": "@CPMP, yes, we started with the starter too. Only up 20 epochs with so few images? Wow, we've probably done something wrong then, if our model training early stopped before 30 epochs, we usually get a horrible IoU score of training/validation data and the model didn't learn much from its training. May I ask if your labelled data are contours or masks? With U-net, we can only output either masks or contours, if we want to output 3 classes such as masks + contours (or bounding boxes) + centres, we'd need to try another CNN model. I know some top scorers do so using either a mask RCNN (for bounding boxes) or a ResNet (@agoodman). Anyway, having a contour output class will be the next thing we want to try to see if we can improve the boundaries of adjacent / touching cells.
", "rawMarkdown": "CPMP, yes, we started with the starter too. Only up 20 epochs with so few images? Wow, we've probably done something wrong then, if our model training early stopped before 30 epochs, we usually get a horrible IoU score of training/validation data and the model didn't learn much from its training. May I ask if your labelled data are contours or masks? With U-net, we can only output either masks or contours, if we want to output 3 classes such as masks + contours (or bounding boxes) + centres, we'd need to try another CNN model. I know some top scorers do so using either a mask RCNN (for bounding boxes) or a ResNet (@agoodman). Anyway, having a contour output class will be the next thing we want to try to see if we can improve the boundaries of adjacent / touching cells.", "votes": null }, { "id": "292715", "postDate": "03/08/2018 13:51:50", "content": "I am getting better IoU CV score with more epochs, but lower LB. As indicated above, reducing the number of epochs was my way to improve generalization power of the model.
\n\nI discuss briefly my labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48825#289906
", "rawMarkdown": "I am getting better IoU CV score with more epochs, but lower LB. As indicated above, reducing the number of epochs was my way to improve generalization power of the model.\n\nI discuss briefly my labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48825#289906", "votes": null }, { "id": "292724", "postDate": "03/08/2018 14:07:45", "content": "@CPMP - That's very helpful. Thanks for sharing the information and re-ensuring this idea. (thumbs-up!)
", "rawMarkdown": "CPMP - That's very helpful. Thanks for sharing the information and re-ensuring this idea. (thumbs-up!)", "votes": null }, { "id": "292726", "postDate": "03/08/2018 14:16:29", "content": "Didn't say it before, but you're doing great for a first time Kaggler!
", "rawMarkdown": "Didn't say it before, but you're doing great for a first time Kaggler!", "votes": null }, { "id": "292728", "postDate": "03/08/2018 14:20:16", "content": "Let me also say I use early stopping on the loss (binary crossentropy), not on IoU. Reason is heavy post processing. I cannot afford to run it after each epoch.
", "rawMarkdown": "Let me also say I use early stopping on the loss (binary crossentropy), not on IoU. Reason is heavy post processing. I cannot afford to run it after each epoch.", "votes": null }, { "id": "292732", "postDate": "03/08/2018 14:24:52", "content": "@CPMP thanks! yes, we use binary crossentropy and we compare the IoU score after training, since IoU is in general not differentiate you know that and cannot be used for the loss function. Anyway, that's a paper for implementing IoU as a loss function but I don't see it would work.
", "rawMarkdown": "CPMP thanks! yes, we use binary crossentropy and we compare the IoU score after training, since IoU is in general not differentiate you know that and cannot be used for the loss function. Anyway, that's a paper for implementing IoU as a loss function but I don't see it would work.", "votes": null }, { "id": "293611", "postDate": "03/10/2018 08:34:47", "content": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553
", "rawMarkdown": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553", "votes": null }, { "id": "309862", "postDate": "04/06/2018 04:26:44", "content": "@CPMP, this thread just popped into my head last night, so I'm responding to this old thread, yes, you're right about 20 epochs, if there's enough amount of data, like just 150k after data augmentation, even training 10 epochs (U-net) is enough for us to create a similar public LB. We must have overfit before (training 70 epochs 6-fold, not doable without a powerful machine). We're still on track to augment more data. The more data we have, the more generic our model will be, hopefully.
", "rawMarkdown": "CPMP, this thread just popped into my head last night, so I'm responding to this old thread, yes, you're right about 20 epochs, if there's enough amount of data, like just 150k after data augmentation, even training 10 epochs (U-net) is enough for us to create a similar public LB. We must have overfit before (training 70 epochs 6-fold, not doable without a powerful machine). We're still on track to augment more data. The more data we have, the more generic our model will be, hopefully.", "votes": null } ], "comments": [ { "id": 290992, "author_name": "jinkos", "author_url": "", "post_date": "03/05/2018 11:09:29", "content": "Yes, it's annoying. I spent days improving my data augmentation and ended up with poorer results. Now I am having a re-think.\nI don't have the answer, but how about some kind of unsupervised learning approach?\nMaybe some kind of pre-processing that standardises all images?\nOr a GAN?
", "votes": null, "replies": [ { "id": 290998, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 11:40:52", "content": "I do some postprocessing along what you describe (kind of standardization), it helps, but I'm still overfitting. I just tried limiting epoch, and got a small LB improvement (+ 0.003) with a larger CV degradation (- 0.008).
\n\nI also do some data augmentation (eg flipping images), it also helps.
", "votes": null, "replies": [] } ] }, { "id": 291019, "author_name": "sangxia", "author_url": "", "post_date": "03/05/2018 13:14:57", "content": "How big is the standard deviation of your CV? If yours is as large as mine (about .05) then I feel that a change of .008 doesn't mean that much.
", "votes": null, "replies": [ { "id": 291022, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 13:29:26", "content": "Until 2 days ago, my CV and my LB evolved the same way. They aren't anymore, which is why I say I overfit. Magnitude of variation is not what matters, sign of variation matters.
", "votes": null, "replies": [] } ] }, { "id": 291048, "author_name": "robertkag", "author_url": "", "post_date": "03/05/2018 14:18:11", "content": "@CPMP aside from flipping you can also do rotations, scaling, translation, etc for augmentation. The cv2 and ndimage packages have functions for doing this.
", "votes": null, "replies": [ { "id": 291058, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 14:50:20", "content": "Thanks. Seems you are saying that more data augmentation is one way to fight overfitting.
\n\nI have tried rotations, it did not help. Seems others reported the same. Scaling, translation, cropping are on my to do list ;) I' using ndimage mostly, but need to learn cv2.
\n\nI also don't do anything with color space, and we know train and test don't have the same color spaces. I'll work on it too.
", "votes": null, "replies": [] }, { "id": 291113, "author_name": "robertkag", "author_url": "", "post_date": "03/05/2018 16:42:21", "content": "Yes, the augmentations create more data, so that would help with reducing overfitting. In terms of colorspaces there is also the possibility of varying the brightness of each image. I didn't have much success with that.
", "votes": null, "replies": [] } ] }, { "id": 291074, "author_name": "vijaybj", "author_url": "", "post_date": "03/05/2018 15:31:14", "content": "I know I don't have enough credits to interfere with this discussion. But I am just curious that there is rare mention of batch normalisation. Is it because it's so trivial to mention? Actually I got much better results with batch normalisation.
", "votes": null, "replies": [ { "id": 291078, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 15:37:59", "content": "I don't have that much credit either when it comes to deep learning ;) Yes, in my limited experience batch normalization helps.
", "votes": null, "replies": [] } ] }, { "id": 291150, "author_name": "lopuhin", "author_url": "", "post_date": "03/05/2018 18:04:25", "content": "FWIW, here are the groups I tried to use for local validation, with \"b\" and \"f\" going into train and \"a\", \"c\", \"d\", \"e\" going into validation.
", "votes": null, "replies": [] }, { "id": 292167, "author_name": "nicolefinnie", "author_url": "", "post_date": "03/07/2018 15:10:12", "content": "@CPMP - a first time Kaggler here. With this amount of data, your LB is quite impressive, maybe you don't even need to do data augmentation then. :p (please teach us how to do data reduction to achieve your LB please :D ) In our case, applying histogram + only keeping on L-channel + data augmentation/reflection/rotation helped a lot and our model is underfit (due to a lack of GPUs) since our CV IoU score is higher, and as you said, post-processing of the predicted masks is necessary to remove noises (false positives). More importantly, we need a machine with 100 GPUs :D I saw a top ranked guy trained his model for 14k iterations. He must have a Lamborghini machine then.
", "votes": null, "replies": [ { "id": 292195, "author_name": "cpmpml", "author_url": "", "post_date": "03/07/2018 15:59:16", "content": "Thanks for the praise! ;) I really need to do more data augmentation it seems...
\n\nMy CV IoU score is also way higher than my LB. That is to be expected given the difference in train/test distributions of images.
\n\nRe number of epochs, I use aAdam optimizer. I read that SGD with carefully designed learning rate decay can be better, but I'll keep this for later, if I have time. Adam converges quite rapidly here.
", "votes": null, "replies": [] }, { "id": 292208, "author_name": "nicolefinnie", "author_url": "", "post_date": "03/07/2018 16:19:35", "content": "Hey @CPMP, for us the CV IoU score is still higher than the training IoU score, an indicator of underfitting sometimes :) we plan to run more epochs. I did notice when the loss converges slowly, our model performs generally better. When it converges rapidly, our model usually performs poorly. We use a reduced U-net as well, 9 layers with reduced neurons, each layer has a dropout layer with a low dropout rate, and we were playing with dropout layers and didn't see much of an improvement there. Thanks for your information, maybe we should try SGD instead of Adam.
\n\nWe found a Stanford paper for cell segmentation and counting using 9600 cell images downloaded from the Broad Institute, they used VGG-11. I got a feeling there's no best CNN in this competition. Who can create more data + try enough models and train the model long enough can get a better score, given how little image data we have. Still need to back up this theory by doing more experiments. \nhttps://arxiv.org/pdf/1802.10548.pdf
", "votes": null, "replies": [] }, { "id": 292481, "author_name": "cpmpml", "author_url": "", "post_date": "03/08/2018 02:32:54", "content": "Thanks.
\n\nI misread you. I agree you seem to underfit.
\n\nI have not computed train IoU, but the binary cross entropy loss for train is a bit lower than the val loss.
\n\nI use early stopping or 20 epochs, whichever is smaller.
\n\nI started from this version of unet: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277
", "votes": null, "replies": [] }, { "id": 292699, "author_name": "nicolefinnie", "author_url": "", "post_date": "03/08/2018 13:20:17", "content": "@CPMP, yes, we started with the starter too. Only up 20 epochs with so few images? Wow, we've probably done something wrong then, if our model training early stopped before 30 epochs, we usually get a horrible IoU score of training/validation data and the model didn't learn much from its training. May I ask if your labelled data are contours or masks? With U-net, we can only output either masks or contours, if we want to output 3 classes such as masks + contours (or bounding boxes) + centres, we'd need to try another CNN model. I know some top scorers do so using either a mask RCNN (for bounding boxes) or a ResNet (@agoodman). Anyway, having a contour output class will be the next thing we want to try to see if we can improve the boundaries of adjacent / touching cells.
", "votes": null, "replies": [] }, { "id": 292715, "author_name": "cpmpml", "author_url": "", "post_date": "03/08/2018 13:51:50", "content": "I am getting better IoU CV score with more epochs, but lower LB. As indicated above, reducing the number of epochs was my way to improve generalization power of the model.
\n\nI discuss briefly my labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48825#289906
", "votes": null, "replies": [] }, { "id": 292724, "author_name": "nicolefinnie", "author_url": "", "post_date": "03/08/2018 14:07:45", "content": "@CPMP - That's very helpful. Thanks for sharing the information and re-ensuring this idea. (thumbs-up!)
", "votes": null, "replies": [] }, { "id": 292726, "author_name": "cpmpml", "author_url": "", "post_date": "03/08/2018 14:16:29", "content": "Didn't say it before, but you're doing great for a first time Kaggler!
", "votes": null, "replies": [] }, { "id": 292728, "author_name": "cpmpml", "author_url": "", "post_date": "03/08/2018 14:20:16", "content": "Let me also say I use early stopping on the loss (binary crossentropy), not on IoU. Reason is heavy post processing. I cannot afford to run it after each epoch.
", "votes": null, "replies": [] }, { "id": 292732, "author_name": "nicolefinnie", "author_url": "", "post_date": "03/08/2018 14:24:52", "content": "@CPMP thanks! yes, we use binary crossentropy and we compare the IoU score after training, since IoU is in general not differentiate you know that and cannot be used for the loss function. Anyway, that's a paper for implementing IoU as a loss function but I don't see it would work.
", "votes": null, "replies": [] }, { "id": 293611, "author_name": "cpmpml", "author_url": "", "post_date": "03/10/2018 08:34:47", "content": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553
", "votes": null, "replies": [] }, { "id": 309862, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/06/2018 04:26:44", "content": "@CPMP, this thread just popped into my head last night, so I'm responding to this old thread, yes, you're right about 20 epochs, if there's enough amount of data, like just 150k after data augmentation, even training 10 epochs (U-net) is enough for us to create a similar public LB. We must have overfit before (training 70 epochs 6-fold, not doable without a powerful machine). We're still on track to augment more data. The more data we have, the more generic our model will be, hopefully.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "290989": "Please excuse the question from a newbie in CNN and image competition. I am currently overfitting to the training dataset with a modified Unet model. To be sure, I degraded a bit my CV score in various ways, and it led to better LB scores. I am doing 6 fold CV, and CV score is not really related to LB score.\n\nThe cause is known: too few images to train from, and a different distribution in train vs test. I am not using external data as I would not be able to annotate it correctly.\n\nWith other kind of data and other types of ML techniques I would have ideas about what to do, but here the only thing I can think of is to limit number of epochs and do LB probing. Those following me know how much I hate LB probing, hence my question is: is there any other way to limit overfit and evaluate it without resorting to LB probing?\n\nEdits: I do some data augmentation (eg flipping images), it helps but is not sufficient.", "290992": "Yes, it's annoying. I spent days improving my data augmentation and ended up with poorer results. Now I am having a re-think.\nI don't have the answer, but how about some kind of unsupervised learning approach?\nMaybe some kind of pre-processing that standardises all images?\nOr a GAN?", "290998": "I do some postprocessing along what you describe (kind of standardization), it helps, but I'm still overfitting. I just tried limiting epoch, and got a small LB improvement (+ 0.003) with a larger CV degradation (- 0.008).\n\nI also do some data augmentation (eg flipping images), it also helps.", "291019": "How big is the standard deviation of your CV? If yours is as large as mine (about .05) then I feel that a change of .008 doesn't mean that much.", "291022": "Until 2 days ago, my CV and my LB evolved the same way. They aren't anymore, which is why I say I overfit. Magnitude of variation is not what matters, sign of variation matters.", "291048": "CPMP aside from flipping you can also do rotations, scaling, translation, etc for augmentation. The cv2 and ndimage packages have functions for doing this.", "291058": "Thanks. Seems you are saying that more data augmentation is one way to fight overfitting. \n\nI have tried rotations, it did not help. Seems others reported the same. Scaling, translation, cropping are on my to do list ;) I' using ndimage mostly, but need to learn cv2.\n\nI also don't do anything with color space, and we know train and test don't have the same color spaces. I'll work on it too.", "291074": "I know I don't have enough credits to interfere with this discussion. But I am just curious that there is rare mention of batch normalisation. Is it because it's so trivial to mention? Actually I got much better results with batch normalisation.", "291078": "I don't have that much credit either when it comes to deep learning ;) Yes, in my limited experience batch normalization helps.", "291113": "Yes, the augmentations create more data, so that would help with reducing overfitting. In terms of colorspaces there is also the possibility of varying the brightness of each image. I didn't have much success with that.", "291150": "FWIW, here are the groups I tried to use for local validation, with \"b\" and \"f\" going into train and \"a\", \"c\", \"d\", \"e\" going into validation.", "292167": "CPMP - a first time Kaggler here. With this amount of data, your LB is quite impressive, maybe you don't even need to do data augmentation then. :p (please teach us how to do data reduction to achieve your LB please :D ) In our case, applying histogram + only keeping on L-channel + data augmentation/reflection/rotation helped a lot and our model is underfit (due to a lack of GPUs) since our CV IoU score is higher, and as you said, post-processing of the predicted masks is necessary to remove noises (false positives). More importantly, we need a machine with 100 GPUs :D I saw a top ranked guy trained his model for 14k iterations. He must have a Lamborghini machine then.", "292195": "Thanks for the praise! ;) I really need to do more data augmentation it seems...\n\nMy CV IoU score is also way higher than my LB. That is to be expected given the difference in train/test distributions of images. \n\nRe number of epochs, I use aAdam optimizer. I read that SGD with carefully designed learning rate decay can be better, but I'll keep this for later, if I have time. Adam converges quite rapidly here.", "292208": "Hey @CPMP, for us the CV IoU score is still higher than the training IoU score, an indicator of underfitting sometimes :) we plan to run more epochs. I did notice when the loss converges slowly, our model performs generally better. When it converges rapidly, our model usually performs poorly. We use a reduced U-net as well, 9 layers with reduced neurons, each layer has a dropout layer with a low dropout rate, and we were playing with dropout layers and didn't see much of an improvement there. Thanks for your information, maybe we should try SGD instead of Adam. \n\nWe found a Stanford paper for cell segmentation and counting using 9600 cell images downloaded from the Broad Institute, they used VGG-11. I got a feeling there's no best CNN in this competition. Who can create more data + try enough models and train the model long enough can get a better score, given how little image data we have. Still need to back up this theory by doing more experiments. \nhttps://arxiv.org/pdf/1802.10548.pdf", "292481": "Thanks.\n\nI misread you. I agree you seem to underfit.\n\nI have not computed train IoU, but the binary cross entropy loss for train is a bit lower than the val loss.\n\nI use early stopping or 20 epochs, whichever is smaller.\n\nI started from this version of unet: https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277", "292699": "CPMP, yes, we started with the starter too. Only up 20 epochs with so few images? Wow, we've probably done something wrong then, if our model training early stopped before 30 epochs, we usually get a horrible IoU score of training/validation data and the model didn't learn much from its training. May I ask if your labelled data are contours or masks? With U-net, we can only output either masks or contours, if we want to output 3 classes such as masks + contours (or bounding boxes) + centres, we'd need to try another CNN model. I know some top scorers do so using either a mask RCNN (for bounding boxes) or a ResNet (@agoodman). Anyway, having a contour output class will be the next thing we want to try to see if we can improve the boundaries of adjacent / touching cells.", "292715": "I am getting better IoU CV score with more epochs, but lower LB. As indicated above, reducing the number of epochs was my way to improve generalization power of the model.\n\nI discuss briefly my labels here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48825#289906", "292724": "CPMP - That's very helpful. Thanks for sharing the information and re-ensuring this idea. (thumbs-up!)", "292726": "Didn't say it before, but you're doing great for a first time Kaggler!", "292728": "Let me also say I use early stopping on the loss (binary crossentropy), not on IoU. Reason is heavy post processing. I cannot afford to run it after each epoch.", "292732": "CPMP thanks! yes, we use binary crossentropy and we compare the IoU score after training, since IoU is in general not differentiate you know that and cannot be used for the loss function. Anyway, that's a paper for implementing IoU as a loss function but I don't see it would work.", "293611": "This may help: https://www.kaggle.com/c/data-science-bowl-2018/discussion/51553", "309862": "CPMP, this thread just popped into my head last night, so I'm responding to this old thread, yes, you're right about 20 epochs, if there's enough amount of data, like just 150k after data augmentation, even training 10 epochs (U-net) is enough for us to create a similar public LB. We must have overfit before (training 70 epochs 6-fold, not doable without a powerful machine). We're still on track to augment more data. The more data we have, the more generic our model will be, hopefully." }, "source": "meta" }