![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F864684%2F45a1b465f4850bc6357c828c0b9b0325%2Flabel_noise_1.jpg?generation=1591654163217084&alt=media\")
I've found another type of label noise. Besides, what @artgor posted earlier (too large, too small GT boxes), there is a third type of issue.
\n\nMy model found (IMO almost perfect) wheat heads, but the GT is much larger/smaller, which results in a low IoU (< 0.5).
\n\n
Left: Image with GT boxes\n*Right: Predicted, GT boxes (True positives are excluded), IoU threshold: 0.5*
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F864684%2F1a71a1af2f70063be07e190d9b0b11aa%2Flabel_nosie_2.jpg?generation=1591654182556080&alt=media\")
These results are one of my model's OOF predictions (5-folds). I only counted as noise if the IoU is less then 0.5, and the GT is inside of the predicted box, or the predicted box is inside of the GT.
\n\n| source | GT | Noisy | % |\n|:---------:|:-----:|-------|-------|\n| inrae_1 | 3701 | 22 | 0.59% |\n| rres_1 | 20236 | 137 | 0.68% |\n| ethz_1 | 51489 | 579 | 1.12% |\n| arvalis_3 | 16665 | 191 | 1.14% |\n| arvalis_1 | 45716 | 653 | 1.43% |\n| arvalis_2 | 4179 | 114 | 2.73% |\n| usask_1 | 5807 | 160 | 2.76% |
\n\nNote: The correct number of noisy labels are a bit less because in some cases the model failed
", "messages": [ { "id": "878829", "postDate": "06/08/2020 22:26:59", "content": "I've found another type of label noise. Besides, what @artgor posted earlier (too large, too small GT boxes), there is a third type of issue.
\n\nMy model found (IMO almost perfect) wheat heads, but the GT is much larger/smaller, which results in a low IoU (< 0.5).
\n\n
Left: Image with GT boxes\n*Right: Predicted, GT boxes (True positives are excluded), IoU threshold: 0.5*
\n\n
These results are one of my model's OOF predictions (5-folds). I only counted as noise if the IoU is less then 0.5, and the GT is inside of the predicted box, or the predicted box is inside of the GT.
\n\n| source | GT | Noisy | % |\n|:---------:|:-----:|-------|-------|\n| inrae_1 | 3701 | 22 | 0.59% |\n| rres_1 | 20236 | 137 | 0.68% |\n| ethz_1 | 51489 | 579 | 1.12% |\n| arvalis_3 | 16665 | 191 | 1.14% |\n| arvalis_1 | 45716 | 653 | 1.43% |\n| arvalis_2 | 4179 | 114 | 2.73% |\n| usask_1 | 5807 | 160 | 2.76% |
\n\nNote: The correct number of noisy labels are a bit less because in some cases the model failed
", "rawMarkdown": "I've found another type of label noise. Besides, what [@artgor posted](https://www.kaggle.com/c/global-wheat-detection/discussion/154685) earlier (too large, too small GT boxes), there is a third type of issue.\n\nMy model found (IMO almost perfect) wheat heads, but the GT is much larger/smaller, which results in a low IoU (< 0.5).\n\n### Examples\n\n\n\n*Left: Image with GT boxes*\n*Right: Predicted, GT boxes (True positives are excluded), IoU threshold: 0.5*\n\n\n### Magnified sample\n\n\n\n### Results\nThese results are one of my model's OOF predictions (5-folds). I only counted as noise if the IoU is less then 0.5, and the GT is inside of the predicted box, or the predicted box is inside of the GT.\n\n| source | GT | Noisy | % |\n|:---------:|:-----:|-------|-------|\n| inrae_1 | 3701 | 22 | 0.59% |\n| rres_1 | 20236 | 137 | 0.68% |\n| ethz_1 | 51489 | 579 | 1.12% |\n| arvalis_3 | 16665 | 191 | 1.14% |\n| arvalis_1 | 45716 | 653 | 1.43% |\n| arvalis_2 | 4179 | 114 | 2.73% |\n| usask_1 | 5807 | 160 | 2.76% |\n\n*Note: The correct number of noisy labels are a bit less because in some cases the model failed*", "votes": null }, { "id": "878850", "postDate": "06/08/2020 23:32:44", "content": "I've also investigated this issue. There are many cases where the box is extended indiscriminately beyond the wheat head. I get that there will always be some noise due to the human labeling but some seem just lazy. There is also a surprising amount of missing/incorrect labels which harshly penalizes the accuracy when the model predicts them correctly. In most cases, I am more confident in my models predictions than the ground truth at this point. This makes me concerned for the test set as there is a maximum performance we can hit before overfitting. I think we've probably already hit that point at the top of the LB.
\n\nYou could in theory have a second person label some of the data, then compute the mAP between the two human labels to estimate the max accuracy you should be able to obtain with this set.
", "rawMarkdown": "I've also investigated this issue. There are many cases where the box is extended indiscriminately beyond the wheat head. I get that there will always be some noise due to the human labeling but some seem just lazy. There is also a surprising amount of missing/incorrect labels which harshly penalizes the accuracy when the model predicts them correctly. In most cases, I am more confident in my models predictions than the ground truth at this point. This makes me concerned for the test set as there is a maximum performance we can hit before overfitting. I think we've probably already hit that point at the top of the LB.\n\nYou could in theory have a second person label some of the data, then compute the mAP between the two human labels to estimate the max accuracy you should be able to obtain with this set.", "votes": null }, { "id": "879411", "postDate": "06/09/2020 13:04:26", "content": "Something similar is in this thread:\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/150781
\n\nProbably this is due to different labellers and their understanding how to draw boxes.
", "rawMarkdown": "Something similar is in this thread:\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/150781\n\nProbably this is due to different labellers and their understanding how to draw boxes.", "votes": null }, { "id": "879571", "postDate": "06/09/2020 15:00:35", "content": "There is a note in the paper (section 2.4):
\n\n\n\n\nWhen the bounding box was too large to include the awns, it was restricted to the head\n only.
\n
And another about weakly supervision:
\n\n\n\n\nFor the datasets not labeled, we used a “weakly supervised deep learning framework” [27] to label images efficiently for these sub-datasets. A YoloV3 model [28] was trained over
\nUTokyo_1andArvalis_1sub-datasets and then applied to the un-labeled sub-datasets.
I think one of these (or both) is the cause of the wrong labels.
", "rawMarkdown": "There is a note in the [paper](https://arxiv.org/pdf/2005.02162.pdf) (section 2.4):\n\n> When the bounding box was too large to include the awns, it was restricted to the head\nonly.\n\nAnd another about weakly supervision:\n> For the datasets not labeled, we used a “weakly supervised deep learning framework” [27] to label images efficiently for these sub-datasets. A YoloV3 model [28] was trained over `UTokyo_1` and `Arvalis_1` sub-datasets and then applied to the un-labeled sub-datasets. \n\nI think one of these (or both) is the cause of the wrong labels.", "votes": null }, { "id": "879849", "postDate": "06/09/2020 19:11:32", "content": "Yep, this is why I think any easy fix could be to slightly modify the competition metric. The agriculture companies mostly need wheat head counts, so I don't see a point in demanding 75% IOU.
\n\nAnd the last thing the competition funders would want is for us to concern ourselves with this unnecessary detail, rather than focus on the main task at hand.
", "rawMarkdown": "Yep, this is why I think any easy fix could be to [slightly modify the competition metric](https://www.kaggle.com/c/global-wheat-detection/discussion/150781). The agriculture companies mostly need wheat head counts, so I don't see a point in demanding 75% IOU.\n\nAnd the last thing the competition funders would want is for us to concern ourselves with this unnecessary detail, rather than focus on the main task at hand.", "votes": null }, { "id": "879876", "postDate": "06/09/2020 19:44:32", "content": "That's a good point, I didn't fully understand this before. I would also love to see the metric changed
", "rawMarkdown": "That's a good point, I didn't fully understand this before. I would also love to see the metric changed", "votes": null }, { "id": "880038", "postDate": "06/10/2020 00:22:54", "content": "A follow up question perhaps the organizers ( @bendvd ) or kaggle ( @maggiemd ) could answer. If we find examples of incorrectly labelled heads and would like to correct them manually, is that allowed? Or does it break the rule which states \"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records\"?
", "rawMarkdown": "A follow up question perhaps the organizers ( @bendvd ) or kaggle ( @maggiemd ) could answer. If we find examples of incorrectly labelled heads and would like to correct them manually, is that allowed? Or does it break the rule which states \"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records\"?", "votes": null }, { "id": "880233", "postDate": "06/10/2020 06:02:48", "content": "Usually at Kaggle you can do anything with the train set.\nThe rule says that you cannot manually do something with the Test set (only automatic things like pseudo labels). In this competition the Test set is hidden, so it doesn't matter. It's probably matter only for that visible 10 images.
", "rawMarkdown": "Usually at Kaggle you can do anything with the train set.\nThe rule says that you cannot manually do something with the Test set (only automatic things like pseudo labels). In this competition the Test set is hidden, so it doesn't matter. It's probably matter only for that visible 10 images.", "votes": null }, { "id": "880794", "postDate": "06/10/2020 15:16:47", "content": "I Also think that their could be better metrics like the wheat head counts as you said and/or their positions
", "rawMarkdown": "I Also think that their could be better metrics like the wheat head counts as you said and/or their positions", "votes": null }, { "id": "896969", "postDate": "06/22/2020 14:44:29", "content": "I think the only conditions would be to make it an open, external dataset even if the problem comes more from the organizers than the competitor !
", "rawMarkdown": "I think the only conditions would be to make it an open, external dataset even if the problem comes more from the organizers than the competitor !", "votes": null }, { "id": "897055", "postDate": "06/22/2020 15:39:20", "content": "@bendvd thanks for coming back on these topics. To clarify, are you saying that if we tweak the bounding boxes on the training set we need to share it? I know many people are filtering some of the unrealistic ones out. And one might even try to correct for the label noise by applying some heuristic (and that heuristic may be general or quite specific - verging on \"hand tweaked\").
", "rawMarkdown": "bendvd thanks for coming back on these topics. To clarify, are you saying that if we tweak the bounding boxes on the training set we need to share it? I know many people are filtering some of the unrealistic ones out. And one might even try to correct for the label noise by applying some heuristic (and that heuristic may be general or quite specific - verging on \"hand tweaked\").", "votes": null }, { "id": "897080", "postDate": "06/22/2020 15:52:41", "content": "I would also like to know as I have hand-removed some specific bounding boxes (like this one). I'd like to share my modified dataset after the final deadline, but is it required I share it before the entry deadline?
\n\nThank you
", "rawMarkdown": "I would also like to know as I have hand-removed some specific bounding boxes (like [this one](https://www.kaggle.com/c/global-wheat-detection/discussion/160353)). I'd like to share my modified dataset after the final deadline, but is it required I share it before the entry deadline?\n\nThank you", "votes": null }, { "id": "897096", "postDate": "06/22/2020 16:04:16", "content": "There are important issues, and I acknowledge the fact that we need to set a clear rule on this topic. Let me see with the other organizers what they think on this topic. I will be much more present until the end of the challenge as I am freee from other duties.
", "rawMarkdown": "There are important issues, and I acknowledge the fact that we need to set a clear rule on this topic. Let me see with the other organizers what they think on this topic. I will be much more present until the end of the challenge as I am freee from other duties.", "votes": null }, { "id": "901727", "postDate": "06/25/2020 16:42:24", "content": "Any update on this? It would be nice to know so I can allocate my time most effectively. I’m currently operating under the assumption we do not have to share “hand labels” as external data, since this has been allowed in previous competitions under the same rules
", "rawMarkdown": "Any update on this? It would be nice to know so I can allocate my time most effectively. I’m currently operating under the assumption we do not have to share “hand labels” as external data, since this has been allowed in previous competitions under the same rules", "votes": null }, { "id": "902528", "postDate": "06/26/2020 07:47:01", "content": "We will bring a clear update in a couple of days, sorry for the frustrating waiting!
", "rawMarkdown": "We will bring a clear update in a couple of days, sorry for the frustrating waiting!", "votes": null }, { "id": "907262", "postDate": "06/29/2020 19:51:18", "content": "Any update..?
", "rawMarkdown": "Any update..?", "votes": null }, { "id": "911194", "postDate": "07/01/2020 15:54:17", "content": "Edit: I got an answer! 😄
", "rawMarkdown": "Edit: [I got an answer](https://www.kaggle.com/c/global-wheat-detection/discussion/163291#911265)! 😄", "votes": null } ], "comments": [ { "id": 878850, "author_name": "matthewmasters", "author_url": "", "post_date": "06/08/2020 23:32:44", "content": "I've also investigated this issue. There are many cases where the box is extended indiscriminately beyond the wheat head. I get that there will always be some noise due to the human labeling but some seem just lazy. There is also a surprising amount of missing/incorrect labels which harshly penalizes the accuracy when the model predicts them correctly. In most cases, I am more confident in my models predictions than the ground truth at this point. This makes me concerned for the test set as there is a maximum performance we can hit before overfitting. I think we've probably already hit that point at the top of the LB.
\n\nYou could in theory have a second person label some of the data, then compute the mAP between the two human labels to estimate the max accuracy you should be able to obtain with this set.
", "votes": null, "replies": [] }, { "id": 879411, "author_name": "sergeyzlobin", "author_url": "", "post_date": "06/09/2020 13:04:26", "content": "Something similar is in this thread:\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/150781
\n\nProbably this is due to different labellers and their understanding how to draw boxes.
", "votes": null, "replies": [ { "id": 879571, "author_name": "pestipeti", "author_url": "", "post_date": "06/09/2020 15:00:35", "content": "There is a note in the paper (section 2.4):
\n\n\n\n\nWhen the bounding box was too large to include the awns, it was restricted to the head\n only.
\n
And another about weakly supervision:
\n\n\n\n\nFor the datasets not labeled, we used a “weakly supervised deep learning framework” [27] to label images efficiently for these sub-datasets. A YoloV3 model [28] was trained over
\nUTokyo_1andArvalis_1sub-datasets and then applied to the un-labeled sub-datasets.
I think one of these (or both) is the cause of the wrong labels.
", "votes": null, "replies": [] } ] }, { "id": 879849, "author_name": "alexandersoare", "author_url": "", "post_date": "06/09/2020 19:11:32", "content": "Yep, this is why I think any easy fix could be to slightly modify the competition metric. The agriculture companies mostly need wheat head counts, so I don't see a point in demanding 75% IOU.
\n\nAnd the last thing the competition funders would want is for us to concern ourselves with this unnecessary detail, rather than focus on the main task at hand.
", "votes": null, "replies": [ { "id": 879876, "author_name": "matthewmasters", "author_url": "", "post_date": "06/09/2020 19:44:32", "content": "That's a good point, I didn't fully understand this before. I would also love to see the metric changed
", "votes": null, "replies": [] }, { "id": 880794, "author_name": "layediop", "author_url": "", "post_date": "06/10/2020 15:16:47", "content": "I Also think that their could be better metrics like the wheat head counts as you said and/or their positions
", "votes": null, "replies": [] } ] }, { "id": 880038, "author_name": "matthewmasters", "author_url": "", "post_date": "06/10/2020 00:22:54", "content": "A follow up question perhaps the organizers ( @bendvd ) or kaggle ( @maggiemd ) could answer. If we find examples of incorrectly labelled heads and would like to correct them manually, is that allowed? Or does it break the rule which states \"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records\"?
", "votes": null, "replies": [ { "id": 880233, "author_name": "sergeyzlobin", "author_url": "", "post_date": "06/10/2020 06:02:48", "content": "Usually at Kaggle you can do anything with the train set.\nThe rule says that you cannot manually do something with the Test set (only automatic things like pseudo labels). In this competition the Test set is hidden, so it doesn't matter. It's probably matter only for that visible 10 images.
", "votes": null, "replies": [] }, { "id": 896969, "author_name": "bendvd", "author_url": "", "post_date": "06/22/2020 14:44:29", "content": "I think the only conditions would be to make it an open, external dataset even if the problem comes more from the organizers than the competitor !
", "votes": null, "replies": [] }, { "id": 897055, "author_name": "alexandersoare", "author_url": "", "post_date": "06/22/2020 15:39:20", "content": "@bendvd thanks for coming back on these topics. To clarify, are you saying that if we tweak the bounding boxes on the training set we need to share it? I know many people are filtering some of the unrealistic ones out. And one might even try to correct for the label noise by applying some heuristic (and that heuristic may be general or quite specific - verging on \"hand tweaked\").
", "votes": null, "replies": [] }, { "id": 897080, "author_name": "matthewmasters", "author_url": "", "post_date": "06/22/2020 15:52:41", "content": "I would also like to know as I have hand-removed some specific bounding boxes (like this one). I'd like to share my modified dataset after the final deadline, but is it required I share it before the entry deadline?
\n\nThank you
", "votes": null, "replies": [] }, { "id": 897096, "author_name": "bendvd", "author_url": "", "post_date": "06/22/2020 16:04:16", "content": "There are important issues, and I acknowledge the fact that we need to set a clear rule on this topic. Let me see with the other organizers what they think on this topic. I will be much more present until the end of the challenge as I am freee from other duties.
", "votes": null, "replies": [] }, { "id": 901727, "author_name": "matthewmasters", "author_url": "", "post_date": "06/25/2020 16:42:24", "content": "Any update on this? It would be nice to know so I can allocate my time most effectively. I’m currently operating under the assumption we do not have to share “hand labels” as external data, since this has been allowed in previous competitions under the same rules
", "votes": null, "replies": [] }, { "id": 902528, "author_name": "bendvd", "author_url": "", "post_date": "06/26/2020 07:47:01", "content": "We will bring a clear update in a couple of days, sorry for the frustrating waiting!
", "votes": null, "replies": [] }, { "id": 907262, "author_name": "matthewmasters", "author_url": "", "post_date": "06/29/2020 19:51:18", "content": "Any update..?
", "votes": null, "replies": [] }, { "id": 911194, "author_name": "matthewmasters", "author_url": "", "post_date": "07/01/2020 15:54:17", "content": "Edit: I got an answer! 😄
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "878829": "I've found another type of label noise. Besides, what [@artgor posted](https://www.kaggle.com/c/global-wheat-detection/discussion/154685) earlier (too large, too small GT boxes), there is a third type of issue.\n\nMy model found (IMO almost perfect) wheat heads, but the GT is much larger/smaller, which results in a low IoU (< 0.5).\n\n### Examples\n\n\n\n*Left: Image with GT boxes*\n*Right: Predicted, GT boxes (True positives are excluded), IoU threshold: 0.5*\n\n\n### Magnified sample\n\n\n\n### Results\nThese results are one of my model's OOF predictions (5-folds). I only counted as noise if the IoU is less then 0.5, and the GT is inside of the predicted box, or the predicted box is inside of the GT.\n\n| source | GT | Noisy | % |\n|:---------:|:-----:|-------|-------|\n| inrae_1 | 3701 | 22 | 0.59% |\n| rres_1 | 20236 | 137 | 0.68% |\n| ethz_1 | 51489 | 579 | 1.12% |\n| arvalis_3 | 16665 | 191 | 1.14% |\n| arvalis_1 | 45716 | 653 | 1.43% |\n| arvalis_2 | 4179 | 114 | 2.73% |\n| usask_1 | 5807 | 160 | 2.76% |\n\n*Note: The correct number of noisy labels are a bit less because in some cases the model failed*", "878850": "I've also investigated this issue. There are many cases where the box is extended indiscriminately beyond the wheat head. I get that there will always be some noise due to the human labeling but some seem just lazy. There is also a surprising amount of missing/incorrect labels which harshly penalizes the accuracy when the model predicts them correctly. In most cases, I am more confident in my models predictions than the ground truth at this point. This makes me concerned for the test set as there is a maximum performance we can hit before overfitting. I think we've probably already hit that point at the top of the LB.\n\nYou could in theory have a second person label some of the data, then compute the mAP between the two human labels to estimate the max accuracy you should be able to obtain with this set.", "879411": "Something similar is in this thread:\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/150781\n\nProbably this is due to different labellers and their understanding how to draw boxes.", "879571": "There is a note in the [paper](https://arxiv.org/pdf/2005.02162.pdf) (section 2.4):\n\n> When the bounding box was too large to include the awns, it was restricted to the head\nonly.\n\nAnd another about weakly supervision:\n> For the datasets not labeled, we used a “weakly supervised deep learning framework” [27] to label images efficiently for these sub-datasets. A YoloV3 model [28] was trained over `UTokyo_1` and `Arvalis_1` sub-datasets and then applied to the un-labeled sub-datasets. \n\nI think one of these (or both) is the cause of the wrong labels.", "879849": "Yep, this is why I think any easy fix could be to [slightly modify the competition metric](https://www.kaggle.com/c/global-wheat-detection/discussion/150781). The agriculture companies mostly need wheat head counts, so I don't see a point in demanding 75% IOU.\n\nAnd the last thing the competition funders would want is for us to concern ourselves with this unnecessary detail, rather than focus on the main task at hand.", "879876": "That's a good point, I didn't fully understand this before. I would also love to see the metric changed", "880038": "A follow up question perhaps the organizers ( @bendvd ) or kaggle ( @maggiemd ) could answer. If we find examples of incorrectly labelled heads and would like to correct them manually, is that allowed? Or does it break the rule which states \"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records\"?", "880233": "Usually at Kaggle you can do anything with the train set.\nThe rule says that you cannot manually do something with the Test set (only automatic things like pseudo labels). In this competition the Test set is hidden, so it doesn't matter. It's probably matter only for that visible 10 images.", "880794": "I Also think that their could be better metrics like the wheat head counts as you said and/or their positions", "896969": "I think the only conditions would be to make it an open, external dataset even if the problem comes more from the organizers than the competitor !", "897055": "bendvd thanks for coming back on these topics. To clarify, are you saying that if we tweak the bounding boxes on the training set we need to share it? I know many people are filtering some of the unrealistic ones out. And one might even try to correct for the label noise by applying some heuristic (and that heuristic may be general or quite specific - verging on \"hand tweaked\").", "897080": "I would also like to know as I have hand-removed some specific bounding boxes (like [this one](https://www.kaggle.com/c/global-wheat-detection/discussion/160353)). I'd like to share my modified dataset after the final deadline, but is it required I share it before the entry deadline?\n\nThank you", "897096": "There are important issues, and I acknowledge the fact that we need to set a clear rule on this topic. Let me see with the other organizers what they think on this topic. I will be much more present until the end of the challenge as I am freee from other duties.", "901727": "Any update on this? It would be nice to know so I can allocate my time most effectively. I’m currently operating under the assumption we do not have to share “hand labels” as external data, since this has been allowed in previous competitions under the same rules", "902528": "We will bring a clear update in a couple of days, sorry for the frustrating waiting!", "907262": "Any update..?", "911194": "Edit: [I got an answer](https://www.kaggle.com/c/global-wheat-detection/discussion/163291#911265)! 😄" }, "source": "meta" }