![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2259429%2Fdde3bd0ede184bc381f52c308d022466%2Fyolov5_ablation_study_200823.png?generation=1598108960636900&alt=media\")
Congratulations to all and thanks to the competition organizers!
\n(I don't use YOLOv5 as my final submission and this is not my solution write-up)
\nSince YOLOv5 emerged in this competition, my interest has been in what in the method makes so much difference.
\nLet me share my insights about the method.
\nPlease point out if you find something wrong, and share your insights too :)
[update] ablation study results for training on pseudo labels (based on nvnn's notebook)
\nEach component has been dropped by modifying the source code on the notebook. YOLOv5 v1.0 was used.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2259429%2F2dd41ab58f405904cdccb5956a13ff6f%2Ftarget_yolo.png?generation=1596598166960469&alt=media\")
(#): dropping this component during training on pseudo labels degrades the score
\nThank you!
", "messages": [ { "id": "958563", "postDate": "08/05/2020 04:01:24", "content": "Congratulations to all and thanks to the competition organizers!
\n(I don't use YOLOv5 as my final submission and this is not my solution write-up)
\nSince YOLOv5 emerged in this competition, my interest has been in what in the method makes so much difference.
\nLet me share my insights about the method.
\nPlease point out if you find something wrong, and share your insights too :)
[update] ablation study results for training on pseudo labels (based on nvnn's notebook)
\nEach component has been dropped by modifying the source code on the notebook. YOLOv5 v1.0 was used.
(#): dropping this component during training on pseudo labels degrades the score
\nThank you!
", "rawMarkdown": "Congratulations to all and thanks to the competition organizers!\n \n(I don't use YOLOv5 as my final submission and this is not my solution write-up) \n \nSince YOLOv5 emerged in this competition, my interest has been in what in the method makes so much difference. \nLet me share my insights about the method.\nPlease point out if you find something wrong, and share your insights too :)\n\n**[update]** ablation study results for training on pseudo labels (based on [nvnn's notebook](https://www.kaggle.com/nvnnghia/yolov5-pseudo-labeling))\nEach component has been dropped by modifying the source code on the notebook. YOLOv5 v1.0 was used. \n\n- Target assignment matters for both private and public datasets\n- Optimal confidence threshold is higher on the private dataset than public\n\n\n\n \n## YOLOv5 components\n\n### Backbone\n- Focus : pixel un-shuffling at the first layer (instead of stride=2 conv) \n- SPP : concat of (5, 9, 13) max-pooled features to enlarge the receptive field \n- BottleNeckCSP : the module proposed in Cross Stage Partial Networks\n\n### Detection (YOLO) layer\n- anchors: same as yolov3\n- xy output range: 2.0 pixels from the center (yolov3: 1 pixel)\n- wh output range: 0-4 * output * anchor sizes (yolov3: exp(output) * anchor sizes)\n- (#) target assignment : <-this seems to be essential\n - all the anchors that belong to the layer and whose size ratio to the target (w and h resp.) is less than 4 (yolov3: only one anchor with max iou with the target across three layers)\n - (#) plus neighboring 2 pixels\n\n\n\n- Box GIOU loss (yolov3: xywh regression) \n- GIOU objectness:\n - (#) Use GIOU (pred vs target) results as confidence target (like yolov2), instead of 1.0 (yolov3)\n- loss balance:\n - Box GIOU (0.05) vs objectness (1.00) , reduction=mean (yolov3: 1:1, reduction=sum)\n\n### Optimization\n- warmup + cosine\n- gradient accumulation : same as yolov3\n- exponential moving average\n\n### Data Augmentation\n- Mosaic\n- Affine : only random scaling (x0.5-1.5) by default\n- Final box filtering: w and h larger than 2.0, aspect ratio smaller than 20\n- Random HSV (not much different from yolov3)\n\n### Others\n- COCO-pretrained weights <- this should matter. Do all the ablation experiments need coco pretraining...?\n- Pseudo Labeling <- Training Faster-RCNN on YOLOv3 pseudo labels didn't work :(\n\n(#): dropping this component during training on pseudo labels degrades the score\n\nThank you!", "votes": null }, { "id": "958628", "postDate": "08/05/2020 04:59:10", "content": "I believe YOLOv5 uses EMA by default
", "rawMarkdown": "I believe YOLOv5 uses EMA by default", "votes": null }, { "id": "958634", "postDate": "08/05/2020 05:03:25", "content": "It was interesting that someone mentioned YOLOv5 version 2 does not get high LB as version 1. I forgot which thread, but the differences maybe the secret sauce of its performance.
", "rawMarkdown": "It was interesting that someone mentioned YOLOv5 version 2 does not get high LB as version 1. I forgot which thread, but the differences maybe the secret sauce of its performance.", "votes": null }, { "id": "958639", "postDate": "08/05/2020 05:11:30", "content": "There were also some rumors that yoloV5 version 1 was used to label the dataset, hence the high performance (especially compared to other models that perform better on COCO), but I guess we'll have to see.
", "rawMarkdown": "There were also some rumors that yoloV5 version 1 was used to label the dataset, hence the high performance (especially compared to other models that perform better on COCO), but I guess we'll have to see.", "votes": null }, { "id": "958644", "postDate": "08/05/2020 05:16:48", "content": "thanks! interesting...\nOne of the many differences I noticed is that version 2 introduces different loss weights (called 'balance') on three yolo layers. Larger weight is imposed on yolo layer with finer grids.
", "rawMarkdown": "thanks! interesting...\nOne of the many differences I noticed is that version 2 introduces different loss weights (called 'balance') on three yolo layers. Larger weight is imposed on yolo layer with finer grids.", "votes": null }, { "id": "958646", "postDate": "08/05/2020 05:17:57", "content": "Thanks! it could make difference!
", "rawMarkdown": "Thanks! it could make difference!", "votes": null }, { "id": "958647", "postDate": "08/05/2020 05:18:40", "content": "Yeah there was some discussion about that here. I personally don't think that was the reason YoloV5 excelled though. Only Tokyo_2 (~10%) in the test set was labelled using Yolo and it was v3 not v5.
EMA is better than w/o using EfficientDet in my experiment, maybe yolov5 is same situation.
", "rawMarkdown": "EMA is better than w/o using EfficientDet in my experiment, maybe yolov5 is same situation.", "votes": null }, { "id": "958875", "postDate": "08/05/2020 07:37:47", "content": "EMA didn't work for my CenterNet btw
", "rawMarkdown": "EMA didn't work for my CenterNet btw", "votes": null }, { "id": "958968", "postDate": "08/05/2020 08:48:17", "content": "Agree with that. \nYOLOv5's detection layer is quite different from v3's, from target assignment to output tensor processing.
", "rawMarkdown": "Agree with that. \nYOLOv5's detection layer is quite different from v3's, from target assignment to output tensor processing.", "votes": null }, { "id": "958979", "postDate": "08/05/2020 09:01:22", "content": "I don't know EMA.. could somebody point to me a paper I should read? ;)
", "rawMarkdown": "I don't know EMA.. could somebody point to me a paper I should read? ;)", "votes": null }, { "id": "959014", "postDate": "08/05/2020 09:36:08", "content": "I don't know in which paper it appeared but YOLOv5 uses rwightman's EMA module, which refers to this doc.
", "rawMarkdown": "I don't know in which paper it appeared but YOLOv5 uses [rwightman's EMA module](https://github.com/rwightman/pytorch-image-models/blob/master/timm/utils.py#L234-L297), which refers to [this doc](https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage).", "votes": null }, { "id": "959106", "postDate": "08/05/2020 10:53:37", "content": "EMA paper\nThe Unusual Effectiveness of Averaging in GAN Training\nhttps://arxiv.org/abs/1806.04498
", "rawMarkdown": "EMA paper\nThe Unusual Effectiveness of Averaging in GAN Training\nhttps://arxiv.org/abs/1806.04498", "votes": null }, { "id": "959981", "postDate": "08/06/2020 04:16:03", "content": "Thanks for the pointers @hirotoschwert @tereka.\nI will look into it
", "rawMarkdown": "Thanks for the pointers @hirotoschwert @tereka.\nI will look into it", "votes": null }, { "id": "959986", "postDate": "08/06/2020 04:19:36", "content": "FYI: I did an ablation study by dropping EMA during pseudo-label training, but the score did not degrade. (see the table above)
", "rawMarkdown": "FYI: I did an ablation study by dropping EMA during pseudo-label training, but the score did not degrade. (see the table above)", "votes": null }, { "id": "959994", "postDate": "08/06/2020 04:26:25", "content": "Wow, great ablation study!
\n\nI've never used YOLO and not sure if I understand (#) target assignment : <-this seems to be essential correctly.\ndo this stand for the regression of the box centers?
OK, I read your article and I guess it's about the training process assigning detector outputs to the GT.\n@hirotoschwert/reproducing-training-performance-of-yolov3-in-pytorch-part2-78be277a7faa\">https://medium.com/@hirotoschwert/reproducing-training-performance-of-yolov3-in-pytorch-part2-78be277a7faa
", "rawMarkdown": "OK, I read your article and I guess it's about the training process assigning detector outputs to the GT.\nhttps://medium.com/@hirotoschwert/reproducing-training-performance-of-yolov3-in-pytorch-part2-78be277a7faa", "votes": null }, { "id": "959999", "postDate": "08/06/2020 04:33:18", "content": "Yes this is target grid point (pixel) selection as 'foreground' points during training. Box regression loss is calculated only on the selected grid points and confidence target is set to non-zero. YOLOv5 selects multiple grid points as foreground (unlike YOLOv3 or Faster RCNN), that could be the reason why it's robust to noisy labels...
", "rawMarkdown": "Yes this is target grid point (pixel) selection as 'foreground' points during training. Box regression loss is calculated only on the selected grid points and confidence target is set to non-zero. YOLOv5 selects multiple grid points as foreground (unlike YOLOv3 or Faster RCNN), that could be the reason why it's robust to noisy labels...", "votes": null }, { "id": "965688", "postDate": "08/10/2020 19:04:35", "content": "@hirotoschwert thanks for the feedback!! 👍
\n\nIn your experience from this competition, what changes would you recommend to YOLOv5 to improve the performance and the user experience / user interface (UI/UX)? Any ideas or feedback would be greatly appreciated so we can fold it into future YOLOv5 updates for the benefit of everyone. 😃
", "rawMarkdown": "hirotoschwert thanks for the feedback!! 👍 \n\nIn your experience from this competition, what changes would you recommend to YOLOv5 to improve the performance and the user experience / user interface (UI/UX)? Any ideas or feedback would be greatly appreciated so we can fold it into future YOLOv5 updates for the benefit of everyone. 😃", "votes": null }, { "id": "965819", "postDate": "08/10/2020 21:54:27", "content": "Hi, thank you for your comment! I've been happy to do experiments using your great repo. \nI think it would be helpful (at least for research purposes) if v1.0 settings could be reproduced more easily.\nI understand one of the differences between v2 and v1 is the balancing factors imposed on the 3 detection layers, which is hard-coded in v2.0. I have not tried an ablation study regarding the balance yet, but it could affect the AP results differently depending on dataset characteristics (e.g. box size distribution).\nThank you!
", "rawMarkdown": "Hi, thank you for your comment! I've been happy to do experiments using your great repo. \nI think it would be helpful (at least for research purposes) if v1.0 settings could be reproduced more easily.\nI understand one of the differences between v2 and v1 is [the balancing factors](https://github.com/ultralytics/yolov5/blob/9ae868364a2d98bd03cecb8ba8f6310c0d11b482/utils/general.py#L483) imposed on the 3 detection layers, which is hard-coded in v2.0. I have not tried an ablation study regarding the balance yet, but it could affect the AP results differently depending on dataset characteristics (e.g. box size distribution).\nThank you!", "votes": null }, { "id": "976420", "postDate": "08/18/2020 20:48:32", "content": "Yes, you are correct. The balancing terms were based on the obj losses seen at the 3 output layers. We simply averaged them over a few epochs of early training and set them to their present values. The smaller output layers have more imbalance than the larger object output layers in COCO, and probably in many other datasets as well, but performance will naturally vary by dataset, so I'm not sure if the balancing helps or hurt wheat detection. This question is interrelated to the number of output layers, and the anchors used.
\nUnfortunately we've been spending all of our time on development, and have not had time to properly isolate and characterize the effects of individual changes as you have here.
\nWe've since released a v3.0 update which primarily substitutes Hardswish() activations for LeakyReLU(0.1), which helps all models, but helps smaller models more than larger models. The change also causes earlier overtraining, and now our current issue (particularly for the largest models) seems to be how to delay overfitting. See https://github.com/ultralytics/yolov5/issues/746
", "rawMarkdown": "Yes, you are correct. The balancing terms were based on the obj losses seen at the 3 output layers. We simply averaged them over a few epochs of early training and set them to their present values. The smaller output layers have more imbalance than the larger object output layers in COCO, and probably in many other datasets as well, but performance will naturally vary by dataset, so I'm not sure if the balancing helps or hurt wheat detection. This question is interrelated to the number of output layers, and the anchors used. \n\nUnfortunately we've been spending all of our time on development, and have not had time to properly isolate and characterize the effects of individual changes as you have here.\n\nWe've since released a v3.0 update which primarily substitutes Hardswish() activations for LeakyReLU(0.1), which helps all models, but helps smaller models more than larger models. The change also causes earlier overtraining, and now our current issue (particularly for the largest models) seems to be how to delay overfitting. See https://github.com/ultralytics/yolov5/issues/746", "votes": null }, { "id": "976424", "postDate": "08/18/2020 20:52:15", "content": "Another very interesting item is that we are seeing quite different behavior when training from scratch than when finetuning. Finetuning seems to benefit greatly from normalization options like mixup, higher scale jitter, dropout, while when training from scratch these actually appear to harm our best mAP.
\nThe official YOLOv5 models are all trained to 300 epochs on COCO from scratch, and the hyps we've set are tuned for that. It's likely there are a completely separate set of hyps better suited for say, finetuning to 50 epochs, so that is another avenue of improvement for finetuning applications like this competition.
", "rawMarkdown": "Another very interesting item is that we are seeing quite different behavior when training from scratch than when finetuning. Finetuning seems to benefit greatly from normalization options like mixup, higher scale jitter, dropout, while when training from scratch these actually appear to harm our best mAP.\n\nThe official YOLOv5 models are all trained to 300 epochs on COCO from scratch, and the hyps we've set are tuned for that. It's likely there are a completely separate set of hyps better suited for say, finetuning to 50 epochs, so that is another avenue of improvement for finetuning applications like this competition.", "votes": null }, { "id": "981589", "postDate": "08/22/2020 15:22:00", "content": "Thank you for your comments and information on YOLOv5 experiments.
\nYes we should find a totally different hyper-parameter setting (including anchors and obj-loss balancing factors) to fine-tune the model for this wheat dataset.
\nIt takes so much time to train detectors on COCO, so the hyps you have optimized through many experiments are so valuable for us!
I've just updated the ablation study table above, which contains the late-sub experiment results on the final private dataset.
\nstill, kaggle such a great place.
", "rawMarkdown": "still, kaggle such a great place.", "votes": null }, { "id": "1513258", "postDate": "09/15/2021 02:35:44", "content": "When i use yolov5, i can feel it absorb lots of competition tricks from kaggle.
", "rawMarkdown": "When i use yolov5, i can feel it absorb lots of competition tricks from kaggle.", "votes": null } ], "comments": [ { "id": 981600, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/22/2020 15:28:08", "content": "I've just updated the ablation study table above, which contains the late-sub experiment results on the final private dataset.
\nstill, kaggle such a great place.
", "votes": null, "replies": [] }, { "id": 1513258, "author_name": "need4data", "author_url": "", "post_date": "09/15/2021 02:35:44", "content": "When i use yolov5, i can feel it absorb lots of competition tricks from kaggle.
", "votes": null, "replies": [] }, { "id": 958628, "author_name": "jihangz", "author_url": "", "post_date": "08/05/2020 04:59:10", "content": "I believe YOLOv5 uses EMA by default
", "votes": null, "replies": [ { "id": 958646, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/05/2020 05:17:57", "content": "Thanks! it could make difference!
", "votes": null, "replies": [] }, { "id": 958750, "author_name": "tereka", "author_url": "", "post_date": "08/05/2020 06:09:58", "content": "EMA is better than w/o using EfficientDet in my experiment, maybe yolov5 is same situation.
", "votes": null, "replies": [] }, { "id": 958875, "author_name": "jihangz", "author_url": "", "post_date": "08/05/2020 07:37:47", "content": "EMA didn't work for my CenterNet btw
", "votes": null, "replies": [] }, { "id": 958979, "author_name": "kyoshioka47", "author_url": "", "post_date": "08/05/2020 09:01:22", "content": "I don't know EMA.. could somebody point to me a paper I should read? ;)
", "votes": null, "replies": [] }, { "id": 959014, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/05/2020 09:36:08", "content": "I don't know in which paper it appeared but YOLOv5 uses rwightman's EMA module, which refers to this doc.
", "votes": null, "replies": [] }, { "id": 959106, "author_name": "tereka", "author_url": "", "post_date": "08/05/2020 10:53:37", "content": "EMA paper\nThe Unusual Effectiveness of Averaging in GAN Training\nhttps://arxiv.org/abs/1806.04498
", "votes": null, "replies": [] }, { "id": 959981, "author_name": "kyoshioka47", "author_url": "", "post_date": "08/06/2020 04:16:03", "content": "Thanks for the pointers @hirotoschwert @tereka.\nI will look into it
", "votes": null, "replies": [] }, { "id": 959986, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/06/2020 04:19:36", "content": "FYI: I did an ablation study by dropping EMA during pseudo-label training, but the score did not degrade. (see the table above)
", "votes": null, "replies": [] }, { "id": 959994, "author_name": "kyoshioka47", "author_url": "", "post_date": "08/06/2020 04:26:25", "content": "Wow, great ablation study!
\n\nI've never used YOLO and not sure if I understand (#) target assignment : <-this seems to be essential correctly.\ndo this stand for the regression of the box centers?
OK, I read your article and I guess it's about the training process assigning detector outputs to the GT.\n@hirotoschwert/reproducing-training-performance-of-yolov3-in-pytorch-part2-78be277a7faa\">https://medium.com/@hirotoschwert/reproducing-training-performance-of-yolov3-in-pytorch-part2-78be277a7faa
", "votes": null, "replies": [] }, { "id": 959999, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/06/2020 04:33:18", "content": "Yes this is target grid point (pixel) selection as 'foreground' points during training. Box regression loss is calculated only on the selected grid points and confidence target is set to non-zero. YOLOv5 selects multiple grid points as foreground (unlike YOLOv3 or Faster RCNN), that could be the reason why it's robust to noisy labels...
", "votes": null, "replies": [] } ] }, { "id": 958634, "author_name": "kyoshioka47", "author_url": "", "post_date": "08/05/2020 05:03:25", "content": "It was interesting that someone mentioned YOLOv5 version 2 does not get high LB as version 1. I forgot which thread, but the differences maybe the secret sauce of its performance.
", "votes": null, "replies": [ { "id": 958639, "author_name": "stanleyjzheng", "author_url": "", "post_date": "08/05/2020 05:11:30", "content": "There were also some rumors that yoloV5 version 1 was used to label the dataset, hence the high performance (especially compared to other models that perform better on COCO), but I guess we'll have to see.
", "votes": null, "replies": [] }, { "id": 958644, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/05/2020 05:16:48", "content": "thanks! interesting...\nOne of the many differences I noticed is that version 2 introduces different loss weights (called 'balance') on three yolo layers. Larger weight is imposed on yolo layer with finer grids.
", "votes": null, "replies": [] }, { "id": 958647, "author_name": "matthewmasters", "author_url": "", "post_date": "08/05/2020 05:18:40", "content": "Yeah there was some discussion about that here. I personally don't think that was the reason YoloV5 excelled though. Only Tokyo_2 (~10%) in the test set was labelled using Yolo and it was v3 not v5.
Agree with that. \nYOLOv5's detection layer is quite different from v3's, from target assignment to output tensor processing.
", "votes": null, "replies": [] } ] }, { "id": 965688, "author_name": "glennjocherultralytics", "author_url": "", "post_date": "08/10/2020 19:04:35", "content": "@hirotoschwert thanks for the feedback!! 👍
\n\nIn your experience from this competition, what changes would you recommend to YOLOv5 to improve the performance and the user experience / user interface (UI/UX)? Any ideas or feedback would be greatly appreciated so we can fold it into future YOLOv5 updates for the benefit of everyone. 😃
", "votes": null, "replies": [ { "id": 965819, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/10/2020 21:54:27", "content": "Hi, thank you for your comment! I've been happy to do experiments using your great repo. \nI think it would be helpful (at least for research purposes) if v1.0 settings could be reproduced more easily.\nI understand one of the differences between v2 and v1 is the balancing factors imposed on the 3 detection layers, which is hard-coded in v2.0. I have not tried an ablation study regarding the balance yet, but it could affect the AP results differently depending on dataset characteristics (e.g. box size distribution).\nThank you!
", "votes": null, "replies": [] }, { "id": 976420, "author_name": "glennjocherultralytics", "author_url": "", "post_date": "08/18/2020 20:48:32", "content": "Yes, you are correct. The balancing terms were based on the obj losses seen at the 3 output layers. We simply averaged them over a few epochs of early training and set them to their present values. The smaller output layers have more imbalance than the larger object output layers in COCO, and probably in many other datasets as well, but performance will naturally vary by dataset, so I'm not sure if the balancing helps or hurt wheat detection. This question is interrelated to the number of output layers, and the anchors used.
\nUnfortunately we've been spending all of our time on development, and have not had time to properly isolate and characterize the effects of individual changes as you have here.
\nWe've since released a v3.0 update which primarily substitutes Hardswish() activations for LeakyReLU(0.1), which helps all models, but helps smaller models more than larger models. The change also causes earlier overtraining, and now our current issue (particularly for the largest models) seems to be how to delay overfitting. See https://github.com/ultralytics/yolov5/issues/746
", "votes": null, "replies": [] }, { "id": 976424, "author_name": "glennjocherultralytics", "author_url": "", "post_date": "08/18/2020 20:52:15", "content": "Another very interesting item is that we are seeing quite different behavior when training from scratch than when finetuning. Finetuning seems to benefit greatly from normalization options like mixup, higher scale jitter, dropout, while when training from scratch these actually appear to harm our best mAP.
\nThe official YOLOv5 models are all trained to 300 epochs on COCO from scratch, and the hyps we've set are tuned for that. It's likely there are a completely separate set of hyps better suited for say, finetuning to 50 epochs, so that is another avenue of improvement for finetuning applications like this competition.
", "votes": null, "replies": [] }, { "id": 981589, "author_name": "hirotoschwert", "author_url": "", "post_date": "08/22/2020 15:22:00", "content": "Thank you for your comments and information on YOLOv5 experiments.
\nYes we should find a totally different hyper-parameter setting (including anchors and obj-loss balancing factors) to fine-tune the model for this wheat dataset.
\nIt takes so much time to train detectors on COCO, so the hyps you have optimized through many experiments are so valuable for us!