A feeling of the magic power of Yolo model began to haunt me after reading the Wheat comp discussions and seeing the PL scores of the public notebooks. I imagine many students who like me have just started with object detection were overwhelmed with similar sensation.
\n\nThis comment from @amiras in the post initiated by @kalyanpichuka explains the most likely cause of the super-performance of Yolo in this competition: YoloV3 model was initially trained on manually labeled part of the data and then applied to unlabeled part to predict bounding boxes, and now these automatic predictions are the bounding boxes we have to find.
\n\nIf the automatically labeled data is large enough within the test set, the winning solutions will most likely inherit the automatic labeling biases and add to the generalization error when applied to the real life wheat images.
\n\nTHE QUESTION is \"how can we use this information to train better models without using Yolo models (not MIT-licensed) ?\"
\n\nPOST DISCUSSIONS NOTE: Based on comments below, my opinion might have been too strong or wrong with respect to reasons of exceptional Yolo performance on this dataset.
", "messages": [ { "id": "937736", "postDate": "07/21/2020 06:27:20", "content": "A feeling of the magic power of Yolo model began to haunt me after reading the Wheat comp discussions and seeing the PL scores of the public notebooks. I imagine many students who like me have just started with object detection were overwhelmed with similar sensation.
\n\nThis comment from @amiras in the post initiated by @kalyanpichuka explains the most likely cause of the super-performance of Yolo in this competition: YoloV3 model was initially trained on manually labeled part of the data and then applied to unlabeled part to predict bounding boxes, and now these automatic predictions are the bounding boxes we have to find.
\n\nIf the automatically labeled data is large enough within the test set, the winning solutions will most likely inherit the automatic labeling biases and add to the generalization error when applied to the real life wheat images.
\n\nTHE QUESTION is \"how can we use this information to train better models without using Yolo models (not MIT-licensed) ?\"
\n\nPOST DISCUSSIONS NOTE: Based on comments below, my opinion might have been too strong or wrong with respect to reasons of exceptional Yolo performance on this dataset.
", "rawMarkdown": "A feeling of the magic power of Yolo model began to haunt me after reading the Wheat comp discussions and seeing the PL scores of the public notebooks. I imagine many students who like me have just started with object detection were overwhelmed with similar sensation.\n\n[This comment](https://www.kaggle.com/c/global-wheat-detection/discussion/167554#933057) from @amiras in the post initiated by @kalyanpichuka explains the most likely cause of the super-performance of Yolo in this competition: YoloV3 model was initially trained on manually labeled part of the data and then applied to unlabeled part to predict bounding boxes, and now these automatic predictions are the bounding boxes we have to find.\n\nIf the automatically labeled data is large enough within the test set, the winning solutions will most likely inherit the automatic labeling biases and add to the generalization error when applied to the real life wheat images.\n\nTHE QUESTION is \"how can we use this information to train better models without using Yolo models (not MIT-licensed) ?\"\n\nPOST DISCUSSIONS NOTE: Based on comments below, my opinion might have been too strong or wrong with respect to reasons of exceptional Yolo performance on this dataset.", "votes": null }, { "id": "937755", "postDate": "07/21/2020 06:37:49", "content": "I wouldn’t put too much weight on that in my opinion. If you read it again, you’ll notice that they only do this for the “unlabeled sub datasets”. This means only the UTokyo_2 images in the test dataset are affected (<10%).
Also, remark that the organizers relabeled the private test set and made sure that \"all bounding boxes are close to wheat head.\" So we shall see after the end of the competition.
", "rawMarkdown": "Also, remark that the organizers relabeled the private test set and made sure that \"all bounding boxes are close to wheat head.\" So we shall see after the end of the competition.", "votes": null }, { "id": "938033", "postDate": "07/21/2020 10:01:27", "content": "Thanks @matthewmasters for pointing this out. This then leaves the question open as for the performance of Yolo on this particular dataset. I have not found the mentions on the ratio of automatically labelled boxes in the paper. Only that UTokyo_1 was used for training - so 72% of heads (boxes) are expected to be hand-labeled.
\n\nTest set (Australia, Japan, and China) from Table 3 in the paper:\n Num of labelled heads pct of total test set\nUTokyo_1 29,174 72%\nUTokyo_2 3,263 8%\nNAU_1 1,250 3%\nUQ_1 7,035 17%\nTotal 40,722
", "rawMarkdown": "Thanks @matthewmasters for pointing this out. This then leaves the question open as for the performance of Yolo on this particular dataset. I have not found the mentions on the ratio of automatically labelled boxes in the paper. Only that UTokyo_1 was used for training - so 72% of heads (boxes) are expected to be hand-labeled.\n\nTest set (Australia, Japan, and China) from Table 3 in the paper:\n Num of labelled heads\tpct of total test set\nUTokyo_1\t29,174\t 72%\nUTokyo_2\t3,263\t 8%\nNAU_1\t 1,250\t 3%\nUQ_1\t 7,035\t 17%\nTotal\t 40,722", "votes": null }, { "id": "938583", "postDate": "07/21/2020 16:02:35", "content": "\n\nYolo models (not MIT-licensed)
\n
Only yolo v5 has \"bad\" license.
\nv3 and v4 are ok, but I didn't see good results here (I didn't try it also). I don't know why, because organizers used v3.
Thanks @sergeyzlobin for the info. This made me rethink the matter.
", "rawMarkdown": "Thanks @sergeyzlobin for the info. This made me rethink the matter.", "votes": null }, { "id": "939425", "postDate": "07/22/2020 08:32:04", "content": "Thanks @alexanderliao. This seems to further complicate the competition.
", "rawMarkdown": "Thanks @alexanderliao. This seems to further complicate the competition.", "votes": null } ], "comments": [ { "id": 937996, "author_name": "alexanderliao", "author_url": "", "post_date": "07/21/2020 09:38:19", "content": "Also, remark that the organizers relabeled the private test set and made sure that \"all bounding boxes are close to wheat head.\" So we shall see after the end of the competition.
", "votes": null, "replies": [ { "id": 939425, "author_name": "isakev", "author_url": "", "post_date": "07/22/2020 08:32:04", "content": "Thanks @alexanderliao. This seems to further complicate the competition.
", "votes": null, "replies": [] } ] }, { "id": 938583, "author_name": "sergeyzlobin", "author_url": "", "post_date": "07/21/2020 16:02:35", "content": "\n\nYolo models (not MIT-licensed)
\n
Only yolo v5 has \"bad\" license.
\nv3 and v4 are ok, but I didn't see good results here (I didn't try it also). I don't know why, because organizers used v3.
Thanks @sergeyzlobin for the info. This made me rethink the matter.
", "votes": null, "replies": [] } ] }, { "id": 937755, "author_name": "matthewmasters", "author_url": "", "post_date": "07/21/2020 06:37:49", "content": "I wouldn’t put too much weight on that in my opinion. If you read it again, you’ll notice that they only do this for the “unlabeled sub datasets”. This means only the UTokyo_2 images in the test dataset are affected (<10%).
Thanks @matthewmasters for pointing this out. This then leaves the question open as for the performance of Yolo on this particular dataset. I have not found the mentions on the ratio of automatically labelled boxes in the paper. Only that UTokyo_1 was used for training - so 72% of heads (boxes) are expected to be hand-labeled.
\n\nTest set (Australia, Japan, and China) from Table 3 in the paper:\n Num of labelled heads pct of total test set\nUTokyo_1 29,174 72%\nUTokyo_2 3,263 8%\nNAU_1 1,250 3%\nUQ_1 7,035 17%\nTotal 40,722
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "937736": "A feeling of the magic power of Yolo model began to haunt me after reading the Wheat comp discussions and seeing the PL scores of the public notebooks. I imagine many students who like me have just started with object detection were overwhelmed with similar sensation.\n\n[This comment](https://www.kaggle.com/c/global-wheat-detection/discussion/167554#933057) from @amiras in the post initiated by @kalyanpichuka explains the most likely cause of the super-performance of Yolo in this competition: YoloV3 model was initially trained on manually labeled part of the data and then applied to unlabeled part to predict bounding boxes, and now these automatic predictions are the bounding boxes we have to find.\n\nIf the automatically labeled data is large enough within the test set, the winning solutions will most likely inherit the automatic labeling biases and add to the generalization error when applied to the real life wheat images.\n\nTHE QUESTION is \"how can we use this information to train better models without using Yolo models (not MIT-licensed) ?\"\n\nPOST DISCUSSIONS NOTE: Based on comments below, my opinion might have been too strong or wrong with respect to reasons of exceptional Yolo performance on this dataset.", "937755": "I wouldn’t put too much weight on that in my opinion. If you read it again, you’ll notice that they only do this for the “unlabeled sub datasets”. This means only the `UTokyo_2` images in the test dataset are affected (<10%).", "937996": "Also, remark that the organizers relabeled the private test set and made sure that \"all bounding boxes are close to wheat head.\" So we shall see after the end of the competition.", "938033": "Thanks @matthewmasters for pointing this out. This then leaves the question open as for the performance of Yolo on this particular dataset. I have not found the mentions on the ratio of automatically labelled boxes in the paper. Only that UTokyo_1 was used for training - so 72% of heads (boxes) are expected to be hand-labeled.\n\nTest set (Australia, Japan, and China) from Table 3 in the paper:\n Num of labelled heads\tpct of total test set\nUTokyo_1\t29,174\t 72%\nUTokyo_2\t3,263\t 8%\nNAU_1\t 1,250\t 3%\nUQ_1\t 7,035\t 17%\nTotal\t 40,722", "938583": "> Yolo models (not MIT-licensed) \n\nOnly yolo v5 has \"bad\" license. \nv3 and v4 are ok, but I didn't see good results here (I didn't try it also). I don't know why, because organizers used v3.", "939422": "Thanks @sergeyzlobin for the info. This made me rethink the matter.", "939425": "Thanks @alexanderliao. This seems to further complicate the competition." }, "source": "meta" }