![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4256010%2F752dc301c06a0616dd8d2fa7fe9073f3%2Fexample-include.png?generation=1589366225178771&alt=media\")
I'd like to suggest to my fellow competitors, and competition organisers, that a slight modification be made to the evaluation metric.
\n\nRight now we take the average of the AP evaluated at IoU thresholds: 0.50, 0.55, 0.60, 0.65, 0.70, 0.75
\n\nI argue that we should only evaluate AP at the lower few thresholds or even just at 0.50. Here are the reasons:
\n\nThe proposed metric will be less sensitive to inconsistencies in labelling - I have noticed that the labelling in the dataset has some inconsistency over whether the awns are included in the bounding boxes or not (see below for image examples).
It is more sensible in the context of the real world problem being solved - From the competition description: \"These images are used to estimate the density and size of wheat heads in different varieties”. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall.
If the competition outcome is indeed impacted by the aforementioned labelling inconsistency, it's possible that a final evaluation will not put the \"best\" solutions at the top. Furthermore, it could be more difficult for competitors to zone in on the \"best\" solutions using the competition metric for feedback. This wouldn't do any favours to the hosts either!
\n\nIt should be further noted that the benchmark set up in the GWHD paper only uses the 0.50 IOU threshold.
\n\nPlease upvote if you agree.
\n\nExample of labelling where awns are included. Notice that my bounding boxes (blue) tend to be smaller than the ground truths (green)\nMy measured AP: @0.50 = 0.33, @0.55 = 0.22, @0.60 = 0.09, @0.65 = 0.06, @0.70 = 0.05, @0.75 = 0.04
Example of labelling where awns are not included. Notice that my bounding boxes (blue) tend to be larger than the ground truths (green).\nMy measured AP: @0.50 = 0.56, @0.55 = 0.34, @0.60 = 0.23, @0.65 = 0.19, @0.70 = 0.14, @0.75 = 0.07
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4256010%2Fe49c41543583ab8b14efcae054a74adb%2Fexample-not-include.png?generation=1589366417661117&alt=media\")
Finally, examine the difference between the @0.50 scores and @0.75 scores. Of course these are supposed to be different, but we're talking a factor of nearly 10.
I'd like to suggest to my fellow competitors, and competition organisers, that a slight modification be made to the evaluation metric.
\n\nRight now we take the average of the AP evaluated at IoU thresholds: 0.50, 0.55, 0.60, 0.65, 0.70, 0.75
\n\nI argue that we should only evaluate AP at the lower few thresholds or even just at 0.50. Here are the reasons:
\n\nThe proposed metric will be less sensitive to inconsistencies in labelling - I have noticed that the labelling in the dataset has some inconsistency over whether the awns are included in the bounding boxes or not (see below for image examples).
It is more sensible in the context of the real world problem being solved - From the competition description: \"These images are used to estimate the density and size of wheat heads in different varieties”. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall.
If the competition outcome is indeed impacted by the aforementioned labelling inconsistency, it's possible that a final evaluation will not put the \"best\" solutions at the top. Furthermore, it could be more difficult for competitors to zone in on the \"best\" solutions using the competition metric for feedback. This wouldn't do any favours to the hosts either!
\n\nIt should be further noted that the benchmark set up in the GWHD paper only uses the 0.50 IOU threshold.
\n\nPlease upvote if you agree.
\n\nExample of labelling where awns are included. Notice that my bounding boxes (blue) tend to be smaller than the ground truths (green)\nMy measured AP: @0.50 = 0.33, @0.55 = 0.22, @0.60 = 0.09, @0.65 = 0.06, @0.70 = 0.05, @0.75 = 0.04
Example of labelling where awns are not included. Notice that my bounding boxes (blue) tend to be larger than the ground truths (green).\nMy measured AP: @0.50 = 0.56, @0.55 = 0.34, @0.60 = 0.23, @0.65 = 0.19, @0.70 = 0.14, @0.75 = 0.07
Finally, examine the difference between the @0.50 scores and @0.75 scores. Of course these are supposed to be different, but we're talking a factor of nearly 10.
I understand your point that the score decreases abruptly and makes the good score from @0.5 less important. But the evaluation metric is usually selected carefully by the organizers for specific reasons that matter the most for them.
For example, maybe in this competition, they care more about the @0.75 than @0.5.
Also there is a long way toward the end of this competition, and until then, there will probably be good discussions for the higher thresholds.
", "rawMarkdown": "Also there is a long way toward the end of this competition, and until then, there will probably be good discussions for the higher thresholds.", "votes": null }, { "id": "845864", "postDate": "05/13/2020 13:12:35", "content": "Hmm, I think my point here is less about finding a way to get a good score at the higher threshold and more about making the competition more aligned towards the end goal.
\n\nDon't get me wrong, I think the organizers are doing a fantastic job, but it's the competitors who are most likely to catch out issues like these - after all that's what we are for! And it won't be the first time a metric has been modified during a Kaggle competition.
", "rawMarkdown": "Hmm, I think my point here is less about finding a way to get a good score at the higher threshold and more about making the competition more aligned towards the end goal.\n\nDon't get me wrong, I think the organizers are doing a fantastic job, but it's the competitors who are most likely to catch out issues like these - after all that's what we are for! And it won't be the first time a metric has been modified during a Kaggle competition.", "votes": null }, { "id": "884658", "postDate": "06/13/2020 14:24:53", "content": "So now you're the second and probably don't want to change metric anymore. :)
", "rawMarkdown": "So now you're the second and probably don't want to change metric anymore. :)", "votes": null }, { "id": "884688", "postDate": "06/13/2020 14:56:25", "content": "If that were true, it would make me kind of a scumbag wouldn't it
", "rawMarkdown": "If that were true, it would make me kind of a scumbag wouldn't it", "votes": null }, { "id": "885358", "postDate": "06/14/2020 06:07:11", "content": "Sorry, that was just a joke.\nI think good models would be still at the top. But you're right, comparing them can be noisy because of the noise bboxes.
", "rawMarkdown": "Sorry, that was just a joke.\nI think good models would be still at the top. But you're right, comparing them can be noisy because of the noise bboxes.", "votes": null }, { "id": "896963", "postDate": "06/22/2020 14:42:33", "content": "Thank you so much for your feedback @alexandersoare
\n\nWe choose 0.75 to be more discriminative. I acknowledge the fact that this dataset is labeled slightly differently than the rest. In the test set, the bbox are tight to the wheat head (you need to trust me on this one ^^).
\n\nCould changing the metric to 0.5 only allow noisy models to win against more accurate ones ?
", "rawMarkdown": "Thank you so much for your feedback @alexandersoare \n\nWe choose 0.75 to be more discriminative. I acknowledge the fact that this dataset is labeled slightly differently than the rest. In the test set, the bbox are tight to the wheat head (you need to trust me on this one ^^). \n\nCould changing the metric to 0.5 only allow noisy models to win against more accurate ones ?", "votes": null }, { "id": "897068", "postDate": "06/22/2020 15:45:25", "content": "I see what you're saying and it makes sense. I wouldn't change it to just 0.5 but I would raise the question of whether to remove 0.75 and 0.70.
\n\nBecause of the label noise, if I have enough time on my hands I can figure out some heuristic which will help me fix it. And if you hadn't said anything about the test set, I might have even made one of my two final submissions alter the bbox size of wheat heads with awns - maybe I could win by blind luck. No one here wants to spend their time doing that as it doesn't add real value.
\n\nIn any case, without an overwhelming number of upvotes on this topic, I'll leave it at that.
", "rawMarkdown": "I see what you're saying and it makes sense. I wouldn't change it to just 0.5 but I would raise the question of whether to remove 0.75 and 0.70.\n\nBecause of the label noise, if I have enough time on my hands I can figure out some heuristic which will help me fix it. And if you hadn't said anything about the test set, I might have even made one of my two final submissions alter the bbox size of wheat heads with awns - maybe I could win by blind luck. No one here wants to spend their time doing that as it doesn't add real value.\n\nIn any case, without an overwhelming number of upvotes on this topic, I'll leave it at that.", "votes": null } ], "comments": [ { "id": 845836, "author_name": "", "author_url": "", "post_date": "05/13/2020 12:57:41", "content": "I understand your point that the score decreases abruptly and makes the good score from @0.5 less important. But the evaluation metric is usually selected carefully by the organizers for specific reasons that matter the most for them.
For example, maybe in this competition, they care more about the @0.75 than @0.5.
Also there is a long way toward the end of this competition, and until then, there will probably be good discussions for the higher thresholds.
", "votes": null, "replies": [] }, { "id": 845864, "author_name": "alexandersoare", "author_url": "", "post_date": "05/13/2020 13:12:35", "content": "Hmm, I think my point here is less about finding a way to get a good score at the higher threshold and more about making the competition more aligned towards the end goal.
\n\nDon't get me wrong, I think the organizers are doing a fantastic job, but it's the competitors who are most likely to catch out issues like these - after all that's what we are for! And it won't be the first time a metric has been modified during a Kaggle competition.
", "votes": null, "replies": [] } ] }, { "id": 884658, "author_name": "sergeyzlobin", "author_url": "", "post_date": "06/13/2020 14:24:53", "content": "So now you're the second and probably don't want to change metric anymore. :)
", "votes": null, "replies": [ { "id": 884688, "author_name": "alexandersoare", "author_url": "", "post_date": "06/13/2020 14:56:25", "content": "If that were true, it would make me kind of a scumbag wouldn't it
", "votes": null, "replies": [] }, { "id": 885358, "author_name": "sergeyzlobin", "author_url": "", "post_date": "06/14/2020 06:07:11", "content": "Sorry, that was just a joke.\nI think good models would be still at the top. But you're right, comparing them can be noisy because of the noise bboxes.
", "votes": null, "replies": [] } ] }, { "id": 896963, "author_name": "bendvd", "author_url": "", "post_date": "06/22/2020 14:42:33", "content": "Thank you so much for your feedback @alexandersoare
\n\nWe choose 0.75 to be more discriminative. I acknowledge the fact that this dataset is labeled slightly differently than the rest. In the test set, the bbox are tight to the wheat head (you need to trust me on this one ^^).
\n\nCould changing the metric to 0.5 only allow noisy models to win against more accurate ones ?
", "votes": null, "replies": [ { "id": 897068, "author_name": "alexandersoare", "author_url": "", "post_date": "06/22/2020 15:45:25", "content": "I see what you're saying and it makes sense. I wouldn't change it to just 0.5 but I would raise the question of whether to remove 0.75 and 0.70.
\n\nBecause of the label noise, if I have enough time on my hands I can figure out some heuristic which will help me fix it. And if you hadn't said anything about the test set, I might have even made one of my two final submissions alter the bbox size of wheat heads with awns - maybe I could win by blind luck. No one here wants to spend their time doing that as it doesn't add real value.
\n\nIn any case, without an overwhelming number of upvotes on this topic, I'll leave it at that.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "845645": "I'd like to suggest to my fellow competitors, and competition organisers, that a slight modification be made to the evaluation metric.\n\nRight now we take the average of the AP evaluated at IoU thresholds: 0.50, 0.55, 0.60, 0.65, 0.70, 0.75\n\nI argue that we should only evaluate AP at the lower few thresholds or even just at 0.50. Here are the reasons:\n\n1. **The proposed metric will be less sensitive to inconsistencies in labelling** - I have noticed that the labelling in the dataset has some inconsistency over whether the awns are included in the bounding boxes or not (see below for image examples).\n\n2. **It is more sensible in the context of the real world problem being solved** - From the competition description: _\"These images are used to estimate the density and size of wheat heads in different varieties”_. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall. \n\n**If the competition outcome is indeed impacted by the aforementioned labelling inconsistency, it's possible that a final evaluation will not put the \"best\" solutions at the top. Furthermore, it could be more difficult for competitors to zone in on the \"best\" solutions using the competition metric for feedback. This wouldn't do any favours to the hosts either!**\n\nIt should be further noted that the benchmark set up in the [GWHD paper](https://arxiv.org/pdf/2005.02162.pdf) only uses the 0.50 IOU threshold. \n\nPlease upvote if you agree.\n\n## Supporting examples\n\nExample of labelling where awns are included. Notice that my bounding boxes (blue) tend to be smaller than the ground truths (green)\nMy measured AP: `@0.50` = 0.33, `@0.55` = 0.22, `@0.60` = 0.09, `@0.65` = 0.06, `@0.70` = 0.05, `@0.75` = 0.04\n\n\n\nExample of labelling where awns are not included. Notice that my bounding boxes (blue) tend to be larger than the ground truths (green).\nMy measured AP: `@0.50` = 0.56, `@0.55` = 0.34, `@0.60` = 0.23, `@0.65` = 0.19, `@0.70` = 0.14, `@0.75` = 0.07\n\n\n\nFinally, examine the difference between the `@0.50` scores and `@0.75` scores. Of course these are supposed to be different, but we're talking a factor of nearly 10.", "845836": "I understand your point that the score decreases abruptly and makes the good score from `@0.5` less important. But the evaluation metric is usually selected carefully by the organizers for specific reasons that matter the most for them.\n\nFor example, maybe in this competition, they care more about the `@0.75` than `@0.5`.", "845841": "Also there is a long way toward the end of this competition, and until then, there will probably be good discussions for the higher thresholds.", "845864": "Hmm, I think my point here is less about finding a way to get a good score at the higher threshold and more about making the competition more aligned towards the end goal.\n\nDon't get me wrong, I think the organizers are doing a fantastic job, but it's the competitors who are most likely to catch out issues like these - after all that's what we are for! And it won't be the first time a metric has been modified during a Kaggle competition.", "884658": "So now you're the second and probably don't want to change metric anymore. :)", "884688": "If that were true, it would make me kind of a scumbag wouldn't it", "885358": "Sorry, that was just a joke.\nI think good models would be still at the top. But you're right, comparing them can be noisy because of the noise bboxes.", "896963": "Thank you so much for your feedback @alexandersoare \n\nWe choose 0.75 to be more discriminative. I acknowledge the fact that this dataset is labeled slightly differently than the rest. In the test set, the bbox are tight to the wheat head (you need to trust me on this one ^^). \n\nCould changing the metric to 0.5 only allow noisy models to win against more accurate ones ?", "897068": "I see what you're saying and it makes sense. I wouldn't change it to just 0.5 but I would raise the question of whether to remove 0.75 and 0.70.\n\nBecause of the label noise, if I have enough time on my hands I can figure out some heuristic which will help me fix it. And if you hadn't said anything about the test set, I might have even made one of my two final submissions alter the bbox size of wheat heads with awns - maybe I could win by blind luck. No one here wants to spend their time doing that as it doesn't add real value.\n\nIn any case, without an overwhelming number of upvotes on this topic, I'll leave it at that." }, "source": "meta" }