It's strange. My local IOU could achieve 0.89 even 0.92 IOU. While in testing sets it could only approach to 0.32-0.33. So, it means the training and testing sets has different distribution? If they have different distribution, nonsense to tune hyper-parameter in training sets. Anyone can give suggestion why there is a gap between training and testing. My IOU score just copied Discussion (using tensorflow api).
", "messages": [ { "id": "271970", "postDate": "01/22/2018 01:46:58", "content": "It's strange. My local IOU could achieve 0.89 even 0.92 IOU. While in testing sets it could only approach to 0.32-0.33. So, it means the training and testing sets has different distribution? If they have different distribution, nonsense to tune hyper-parameter in training sets. Anyone can give suggestion why there is a gap between training and testing. My IOU score just copied Discussion (using tensorflow api).
", "rawMarkdown": "It's strange. My local IOU could achieve 0.89 even 0.92 IOU. While in testing sets it could only approach to 0.32-0.33. So, it means the training and testing sets has different distribution? If they have different distribution, nonsense to tune hyper-parameter in training sets. Anyone can give suggestion why there is a gap between training and testing. My IOU score just copied Discussion (using tensorflow api).", "votes": null }, { "id": "271973", "postDate": "01/22/2018 02:01:41", "content": "I have looked at the images of the test set. They are very similar to the train set. I believe it might be an incorrect implementation of IoU. I believe I am suffering from the same problem
", "rawMarkdown": "I have looked at the images of the test set. They are very similar to the train set. I believe it might be an incorrect implementation of IoU. I believe I am suffering from the same problem", "votes": null }, { "id": "271984", "postDate": "01/22/2018 02:44:11", "content": "By IoU, you mean semantic level or instance level? Submissions are evaluated using results of instance segmentations.
", "rawMarkdown": "By IoU, you mean semantic level or instance level? Submissions are evaluated using results of instance segmentations.", "votes": null }, { "id": "272013", "postDate": "01/22/2018 04:10:03", "content": "While. I think training and testing mask by my model looks good. At least not only 0.33. Considering most of the cell has been denoted by CNN, so the score is confusing for me.
", "rawMarkdown": "While. I think training and testing mask by my model looks good. At least not only 0.33. Considering most of the cell has been denoted by CNN, so the score is confusing for me.", "votes": null }, { "id": "272016", "postDate": "01/22/2018 04:11:14", "content": "instance level, the definition is iou=tp/(tp+fp+fn). I think my calculation is right.
", "rawMarkdown": "instance level, the definition is iou=tp/(tp+fp+fn). I think my calculation is right.", "votes": null }, { "id": "272023", "postDate": "01/22/2018 04:49:05", "content": "Thanks for your reply. So, this challenge just calculate the fp tp fn by cell num but not pixel level segmentation.
", "rawMarkdown": "Thanks for your reply. So, this challenge just calculate the fp tp fn by cell num but not pixel level segmentation.", "votes": null }, { "id": "272025", "postDate": "01/22/2018 04:49:45", "content": "Just different evaluation metric: https://www.kaggle.com/xuhuarensjtu/example-metric-implementation/editnb
", "rawMarkdown": "Just different evaluation metric: https://www.kaggle.com/xuhuarensjtu/example-metric-implementation/editnb", "votes": null }, { "id": "272126", "postDate": "01/22/2018 10:33:38", "content": "Because the score calculate IOU for each object you detect. A great part of the gap is resulted by mistaking several closing objects as one single object.
", "rawMarkdown": "Because the score calculate IOU for each object you detect. A great part of the gap is resulted by mistaking several closing objects as one single object.", "votes": null } ], "comments": [ { "id": 271973, "author_name": "arpandhatt", "author_url": "", "post_date": "01/22/2018 02:01:41", "content": "I have looked at the images of the test set. They are very similar to the train set. I believe it might be an incorrect implementation of IoU. I believe I am suffering from the same problem
", "votes": null, "replies": [ { "id": 272013, "author_name": "renxuhua", "author_url": "", "post_date": "01/22/2018 04:10:03", "content": "While. I think training and testing mask by my model looks good. At least not only 0.33. Considering most of the cell has been denoted by CNN, so the score is confusing for me.
", "votes": null, "replies": [] }, { "id": 272025, "author_name": "xuhuarensjtu", "author_url": "", "post_date": "01/22/2018 04:49:45", "content": "Just different evaluation metric: https://www.kaggle.com/xuhuarensjtu/example-metric-implementation/editnb
", "votes": null, "replies": [] } ] }, { "id": 271984, "author_name": "wayne999", "author_url": "", "post_date": "01/22/2018 02:44:11", "content": "By IoU, you mean semantic level or instance level? Submissions are evaluated using results of instance segmentations.
", "votes": null, "replies": [ { "id": 272016, "author_name": "renxuhua", "author_url": "", "post_date": "01/22/2018 04:11:14", "content": "instance level, the definition is iou=tp/(tp+fp+fn). I think my calculation is right.
", "votes": null, "replies": [] }, { "id": 272023, "author_name": "xuhuarensjtu", "author_url": "", "post_date": "01/22/2018 04:49:05", "content": "Thanks for your reply. So, this challenge just calculate the fp tp fn by cell num but not pixel level segmentation.
", "votes": null, "replies": [] } ] }, { "id": 272126, "author_name": "dingkun", "author_url": "", "post_date": "01/22/2018 10:33:38", "content": "Because the score calculate IOU for each object you detect. A great part of the gap is resulted by mistaking several closing objects as one single object.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "271970": "It's strange. My local IOU could achieve 0.89 even 0.92 IOU. While in testing sets it could only approach to 0.32-0.33. So, it means the training and testing sets has different distribution? If they have different distribution, nonsense to tune hyper-parameter in training sets. Anyone can give suggestion why there is a gap between training and testing. My IOU score just copied Discussion (using tensorflow api).", "271973": "I have looked at the images of the test set. They are very similar to the train set. I believe it might be an incorrect implementation of IoU. I believe I am suffering from the same problem", "271984": "By IoU, you mean semantic level or instance level? Submissions are evaluated using results of instance segmentations.", "272013": "While. I think training and testing mask by my model looks good. At least not only 0.33. Considering most of the cell has been denoted by CNN, so the score is confusing for me.", "272016": "instance level, the definition is iou=tp/(tp+fp+fn). I think my calculation is right.", "272023": "Thanks for your reply. So, this challenge just calculate the fp tp fn by cell num but not pixel level segmentation.", "272025": "Just different evaluation metric: https://www.kaggle.com/xuhuarensjtu/example-metric-implementation/editnb", "272126": "Because the score calculate IOU for each object you detect. A great part of the gap is resulted by mistaking several closing objects as one single object." }, "source": "meta" }