Out of curiosity, what is your gap between local validation set and the LB? I have a terrible difference between the two that I am working on solving before moving on to trying other things to improve the score itself. So far my best validation score is over 0.80 locally but only 0.3 on LB.
\n\nAny ideas on what may be causing this issue for this particular dataset?
", "messages": [ { "id": "271239", "postDate": "01/20/2018 00:24:14", "content": "Out of curiosity, what is your gap between local validation set and the LB? I have a terrible difference between the two that I am working on solving before moving on to trying other things to improve the score itself. So far my best validation score is over 0.80 locally but only 0.3 on LB.
\n\nAny ideas on what may be causing this issue for this particular dataset?
", "rawMarkdown": "Out of curiosity, what is your gap between local validation set and the LB? I have a terrible difference between the two that I am working on solving before moving on to trying other things to improve the score itself. So far my best validation score is over 0.80 locally but only 0.3 on LB.\n\nAny ideas on what may be causing this issue for this particular dataset?", "votes": null }, { "id": "271253", "postDate": "01/20/2018 01:06:24", "content": "Visually inspect the results
", "rawMarkdown": "Visually inspect the results", "votes": null }, { "id": "271313", "postDate": "01/20/2018 06:00:47", "content": "I guess you used this kernel https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277 .\nThe poster of kernel said that mean_iou implementation is wrong.\nThis might help, https://www.kaggle.com/wcukierski/example-metric-implementation
", "rawMarkdown": "I guess you used this kernel https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277 .\nThe poster of kernel said that mean_iou implementation is wrong.\nThis might help, https://www.kaggle.com/wcukierski/example-metric-implementation", "votes": null }, { "id": "271892", "postDate": "01/21/2018 20:20:33", "content": "Thanks lyakaap. That is correct, I borrowed the mean_iou implementation from there. I later tinkered a bitt with tf.metrics.mean_iou, dropping the rest of the code but that was not fruitful either. I am now looking into implementing the post you reference in pure tensorflow without having to use numpy (for speed reasons), I am still figuring out the per-object intersection instead of using pixels like the Jaccard index would.
", "rawMarkdown": "Thanks lyakaap. That is correct, I borrowed the mean_iou implementation from there. I later tinkered a bitt with tf.metrics.mean_iou, dropping the rest of the code but that was not fruitful either. I am now looking into implementing the post you reference in pure tensorflow without having to use numpy (for speed reasons), I am still figuring out the per-object intersection instead of using pixels like the Jaccard index would.", "votes": null }, { "id": "271893", "postDate": "01/21/2018 20:23:54", "content": "Thanks Heng. I had inspected the train predictions (maybe 12 of them) and validation predictions results (half of them) and most were good. When it comes to test, there are big hits and misses, which can explain a portion of the gap but I think it is a combination of a bad metric implementation as lyakaap pointed out and maybe the train set not being entirely representative of the test one. I am looking into preprocessing beyond normalizing to zero mean and unit variance.
", "rawMarkdown": "Thanks Heng. I had inspected the train predictions (maybe 12 of them) and validation predictions results (half of them) and most were good. When it comes to test, there are big hits and misses, which can explain a portion of the gap but I think it is a combination of a bad metric implementation as lyakaap pointed out and maybe the train set not being entirely representative of the test one. I am looking into preprocessing beyond normalizing to zero mean and unit variance.", "votes": null }, { "id": "272019", "postDate": "01/22/2018 04:15:40", "content": "Same with me. 0.9 in local but only 0.33 in LB. Some assumption: (1) training and testing set have different distribution. (2) the submit code has something wrong?
", "rawMarkdown": "Same with me. 0.9 in local but only 0.33 in LB. Some assumption: (1) training and testing set have different distribution. (2) the submit code has something wrong?", "votes": null }, { "id": "272029", "postDate": "01/22/2018 04:56:12", "content": "you should implement the metric. then try it on the train image ground truth and simulate predicted results. then you can understand how the metric behave and how each errors affect score.
\n\ne.g. if we do not separate overlapping cell, what is the best metric score we expect (i.e. create deep net for semantic segmentation rather than instance segmentation)
", "rawMarkdown": "you should implement the metric. then try it on the train image ground truth and simulate predicted results. then you can understand how the metric behave and how each errors affect score. \n\n\ne.g. if we do not separate overlapping cell, what is the best metric score we expect (i.e. create deep net for semantic segmentation rather than instance segmentation)", "votes": null }, { "id": "272252", "postDate": "01/22/2018 16:26:03", "content": "It’ll be tricky to implement since TensorFlow lacks an equivalent to argmax. If you’re using a neural network I’d recommend thinking carefully about your loss function (e.g. use something like Huber loss, et al.) and only validating with mean average precision after you’ve trained your model.
We have a conversation on the Keras-RCNN repository about this. It might be worth reading.
", "rawMarkdown": "We have a conversation on the Keras-RCNN repository about this. It might be worth reading.", "votes": null }, { "id": "272268", "postDate": "01/22/2018 17:18:03", "content": "Hi @Allen, could you please share the link to the Keras-RCNN repo?
", "rawMarkdown": "Hi @Allen, could you please share the link to the Keras-RCNN repo?", "votes": null }, { "id": "272361", "postDate": "01/22/2018 21:53:21", "content": "Hi, Do you find a way to implement mean-iou with pure kears or tensorflow?
", "rawMarkdown": "Hi, Do you find a way to implement mean-iou with pure kears or tensorflow?", "votes": null }, { "id": "272791", "postDate": "01/23/2018 18:13:41", "content": "You can find the issue here. However, the implementations you'll see are slightly more complex since they compute IOU for bounding boxes rather than masks.
", "rawMarkdown": "You can find the issue [here][1]. However, the implementations you'll see are slightly more complex since they compute IOU for bounding boxes rather than masks. \n\n\n [1]: https://github.com/broadinstitute/keras-rcnn/issues/6", "votes": null }, { "id": "272838", "postDate": "01/23/2018 20:23:31", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null } ], "comments": [ { "id": 271253, "author_name": "hengck23", "author_url": "", "post_date": "01/20/2018 01:06:24", "content": "Visually inspect the results
", "votes": null, "replies": [ { "id": 271893, "author_name": "malr87", "author_url": "", "post_date": "01/21/2018 20:23:54", "content": "Thanks Heng. I had inspected the train predictions (maybe 12 of them) and validation predictions results (half of them) and most were good. When it comes to test, there are big hits and misses, which can explain a portion of the gap but I think it is a combination of a bad metric implementation as lyakaap pointed out and maybe the train set not being entirely representative of the test one. I am looking into preprocessing beyond normalizing to zero mean and unit variance.
", "votes": null, "replies": [] } ] }, { "id": 271313, "author_name": "lyakaap", "author_url": "", "post_date": "01/20/2018 06:00:47", "content": "I guess you used this kernel https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277 .\nThe poster of kernel said that mean_iou implementation is wrong.\nThis might help, https://www.kaggle.com/wcukierski/example-metric-implementation
", "votes": null, "replies": [ { "id": 271892, "author_name": "malr87", "author_url": "", "post_date": "01/21/2018 20:20:33", "content": "Thanks lyakaap. That is correct, I borrowed the mean_iou implementation from there. I later tinkered a bitt with tf.metrics.mean_iou, dropping the rest of the code but that was not fruitful either. I am now looking into implementing the post you reference in pure tensorflow without having to use numpy (for speed reasons), I am still figuring out the per-object intersection instead of using pixels like the Jaccard index would.
", "votes": null, "replies": [] }, { "id": 272361, "author_name": "zhijianli", "author_url": "", "post_date": "01/22/2018 21:53:21", "content": "Hi, Do you find a way to implement mean-iou with pure kears or tensorflow?
", "votes": null, "replies": [] } ] }, { "id": 272019, "author_name": "renxuhua", "author_url": "", "post_date": "01/22/2018 04:15:40", "content": "Same with me. 0.9 in local but only 0.33 in LB. Some assumption: (1) training and testing set have different distribution. (2) the submit code has something wrong?
", "votes": null, "replies": [] }, { "id": 272029, "author_name": "hengck23", "author_url": "", "post_date": "01/22/2018 04:56:12", "content": "you should implement the metric. then try it on the train image ground truth and simulate predicted results. then you can understand how the metric behave and how each errors affect score.
\n\ne.g. if we do not separate overlapping cell, what is the best metric score we expect (i.e. create deep net for semantic segmentation rather than instance segmentation)
", "votes": null, "replies": [] }, { "id": 272252, "author_name": "agoodman", "author_url": "", "post_date": "01/22/2018 16:26:03", "content": "It’ll be tricky to implement since TensorFlow lacks an equivalent to argmax. If you’re using a neural network I’d recommend thinking carefully about your loss function (e.g. use something like Huber loss, et al.) and only validating with mean average precision after you’ve trained your model.
We have a conversation on the Keras-RCNN repository about this. It might be worth reading.
", "votes": null, "replies": [] }, { "id": 272268, "author_name": "abnerchou", "author_url": "", "post_date": "01/22/2018 17:18:03", "content": "Hi @Allen, could you please share the link to the Keras-RCNN repo?
", "votes": null, "replies": [] }, { "id": 272791, "author_name": "agoodman", "author_url": "", "post_date": "01/23/2018 18:13:41", "content": "You can find the issue here. However, the implementations you'll see are slightly more complex since they compute IOU for bounding boxes rather than masks.
", "votes": null, "replies": [] }, { "id": 272838, "author_name": "abnerchou", "author_url": "", "post_date": "01/23/2018 20:23:31", "content": "Thank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "271239": "Out of curiosity, what is your gap between local validation set and the LB? I have a terrible difference between the two that I am working on solving before moving on to trying other things to improve the score itself. So far my best validation score is over 0.80 locally but only 0.3 on LB.\n\nAny ideas on what may be causing this issue for this particular dataset?", "271253": "Visually inspect the results", "271313": "I guess you used this kernel https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277 .\nThe poster of kernel said that mean_iou implementation is wrong.\nThis might help, https://www.kaggle.com/wcukierski/example-metric-implementation", "271892": "Thanks lyakaap. That is correct, I borrowed the mean_iou implementation from there. I later tinkered a bitt with tf.metrics.mean_iou, dropping the rest of the code but that was not fruitful either. I am now looking into implementing the post you reference in pure tensorflow without having to use numpy (for speed reasons), I am still figuring out the per-object intersection instead of using pixels like the Jaccard index would.", "271893": "Thanks Heng. I had inspected the train predictions (maybe 12 of them) and validation predictions results (half of them) and most were good. When it comes to test, there are big hits and misses, which can explain a portion of the gap but I think it is a combination of a bad metric implementation as lyakaap pointed out and maybe the train set not being entirely representative of the test one. I am looking into preprocessing beyond normalizing to zero mean and unit variance.", "272019": "Same with me. 0.9 in local but only 0.33 in LB. Some assumption: (1) training and testing set have different distribution. (2) the submit code has something wrong?", "272029": "you should implement the metric. then try it on the train image ground truth and simulate predicted results. then you can understand how the metric behave and how each errors affect score. \n\n\ne.g. if we do not separate overlapping cell, what is the best metric score we expect (i.e. create deep net for semantic segmentation rather than instance segmentation)", "272252": "It’ll be tricky to implement since TensorFlow lacks an equivalent to `argmax`. If you’re using a neural network I’d recommend thinking carefully about your loss function (e.g. use something like Huber loss, et al.) and only validating with mean average precision after you’ve trained your model.", "272253": "We have a conversation on the Keras-RCNN repository about this. It might be worth reading.", "272268": "Hi @Allen, could you please share the link to the Keras-RCNN repo?", "272361": "Hi, Do you find a way to implement mean-iou with pure kears or tensorflow?", "272791": "You can find the issue [here][1]. However, the implementations you'll see are slightly more complex since they compute IOU for bounding boxes rather than masks. \n\n\n [1]: https://github.com/broadinstitute/keras-rcnn/issues/6", "272838": "Thank you." }, "source": "meta" }