[update] The code is now available at https://github.com/liaopeiyuan/TransferDet .
\nNow that the results are out, I've decided to share a bit more than my previous post.
\nFirst of all, huge thanks to @rwightman for his PyTorch implementation of EfficientDet and @shonenkov for his amazing starter kernel. I've also learned a lot from the discussion posts.
\nThis is an interesting challenge with a very noisy dataset and huge domain shift between train/test, and within train/test as well. The kernel running time limit also gives possibility to do semi-supervised learning, which can in a way reduce the aforementioned problem.
\nThe first choice I made was the ways I validate and perform model selection. This is a tough choice because 1. object detection is inherently a hard task and 2. the training data are noisy and unstable.
\nEventually my decision was \"relative detection loss\" on a 20% stratified split with bad boxes removed (by manual inspection), as hinted here. My reasoning is as follows:
We know that train and test distributions are wildly different, so mAP to me would make little sense because it's a heavy processed result (with hyperparameters in between of nms, and which mAP to look at, etc.) And detection loss we use to train our networks can capture the overall classification and regression performance. In addition, I don't use this loss to compare across models, and instead I accumulated a list of procedures that I considered as \"comparable under loss,\" by pure empirical results and a little bit of instinct, and used them to select minor choices such as augmentations. For model selection overall, I took the public LB with a grain of salt, and in a way because I ran out of time to train diverse models, I didn't have to make those huge choices of model ensemble. I believe that I can investigate into this more once the entire set is released and cleaned.
\nNow, on to data insights. We can all agree that the data, even with blatantly bad boxes removed, are really noisy, in two forms: noisy anchor and noisy boxes.
\nTherefore, all the models I developed are trying to address these issues. I first tried DetectorRS, but only managed to get pass 0.72 LB. So I moved on to the EfficientDet pipeline. I ran some simple tests from D4 to D7, and concluded that D6 with COCO-pretrained weight is the best. Here are my reasonings.
\nD0-D5: not large enough
\nD7: Remark that D7 got its precision by larger input size (and hence larger anchors). I really couldn't fit 1536-resized images on my GPU and get reasonable results.
\nD6 with noisy student/AdvProp pretrained cls backbone and random-initialized FPN/box, cls net: Object detection is quite hard. I couldn't make these models converge.
The next thing I did was trying to improve baseline performance of my EfficientDet-D6 model. I tried over 10 different techniques by dissecting the timm-efficientdet repo line by line and hacking into them, but turned out simplicity and elegance prevailed. The best performance is achieved by the default setting of the D6 model (with anchors unchanged; I tried KNN clustering of anchors in train set but it gave worse results), with default huber loss and focal loss. Augmentation wise, it's just an empirically deduced recipe:
\n A.Compose(\n [\n A.RandomSizedCrop(min_max_height=(800, 1024), height=1024, width=1024, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.ToGray(p=0.01),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.Transpose(p=0.5),\n A.JpegCompression(quality_lower=85, quality_upper=95, p=0.2),\n A.OneOf([\n A.Blur(blur_limit=3, p=1.0),\n A.MedianBlur(blur_limit=3, p=1.0)\n ],p=0.1),\n A.Resize(height=1024, width=1024, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\nand for __getitem__:
if self.test or random.random() > 0.33:\n image, boxes = self.load_image_and_boxes(index)\nelif random.random() > 0.5:\n image, boxes = self.load_cutmix_image_and_boxes(index)\nelse:\n image, boxes = self.load_mixup_image_and_boxes(index)\nRemark that mixup is the 1:1 version; I didn't get the beta version to work (I tried to reweight the boxes during loss calculation; I got worse results). Then I tuned the WBF over TTA (8x, because 16x has duplicates) pipeline a bit with basic sanity checks and exclusion of boxes that have low score or are too big or small. The overall baseline gave public LB of around 0.753.
\nThen, I realized that similar to the TGS competition I participated few years back, we can get more data by doing jigsaws. You can check out one of the popular public kernels for how to restore the original full-sized images. After this, I realized two more problems:
\nBoxes over the edge are fractured in the sense that if a wheat head appears in half in one image and in half in another, there will be two boxes in the final image. This will generate huge amount of noise and will certainly degrade model performance.
This is more subtle. Remark that in the data insight section I talked about the conundrum of boxes over the edge. Now, if we are now cropping 1024 patches from the larger image, the resulting distribution is actually different than the one represented by what we now can see as crops at special positions. I didn't really document the changes in model performance, but it is indeed significant enough for training.
My solutions is as follows:
\nI tried to correct every single label manually but got bored after 30 minutes. So I wrote an automatic procedure that does that for me:
\nWhat I did here is basically treating images obtained from jigsaw as \"pseudo-labels,\" or simply images that come from different distributions.
\nI also designed a detection variant of FixMatch, but I couldn't fit it in my GPU with image size = 1024.
\nFinally the semi-supervised learning part was pretty mundane, as fancy techniques (STAC, SplitBatchNorm, Pi-Model, Mean Teacher, etc) did not fit under the kernel eval time frame. All I did was to freeze the backbone to allow training with 16GB vram, and different hyperparameters to generate the pseudolabels. You can see the details in my open-sourced kernels. Most choices are more empirical than insightful.
\nSorry for this messy solution journal as working alone gives you a bad habit of trying too many stuff and not documenting well enough. Also, since I'm working alone, I didn't have enough time to investigate in depth some of the ideas I brought up above. I will seek to improve the solution journal in the upcoming days, and maybe write a tech report if I have time.
", "messages": [ { "id": "978155", "postDate": "08/20/2020 01:47:14", "content": "[update] The code is now available at https://github.com/liaopeiyuan/TransferDet .
\nNow that the results are out, I've decided to share a bit more than my previous post.
\nFirst of all, huge thanks to @rwightman for his PyTorch implementation of EfficientDet and @shonenkov for his amazing starter kernel. I've also learned a lot from the discussion posts.
\nThis is an interesting challenge with a very noisy dataset and huge domain shift between train/test, and within train/test as well. The kernel running time limit also gives possibility to do semi-supervised learning, which can in a way reduce the aforementioned problem.
\nThe first choice I made was the ways I validate and perform model selection. This is a tough choice because 1. object detection is inherently a hard task and 2. the training data are noisy and unstable.
\nEventually my decision was \"relative detection loss\" on a 20% stratified split with bad boxes removed (by manual inspection), as hinted here. My reasoning is as follows:
We know that train and test distributions are wildly different, so mAP to me would make little sense because it's a heavy processed result (with hyperparameters in between of nms, and which mAP to look at, etc.) And detection loss we use to train our networks can capture the overall classification and regression performance. In addition, I don't use this loss to compare across models, and instead I accumulated a list of procedures that I considered as \"comparable under loss,\" by pure empirical results and a little bit of instinct, and used them to select minor choices such as augmentations. For model selection overall, I took the public LB with a grain of salt, and in a way because I ran out of time to train diverse models, I didn't have to make those huge choices of model ensemble. I believe that I can investigate into this more once the entire set is released and cleaned.
\nNow, on to data insights. We can all agree that the data, even with blatantly bad boxes removed, are really noisy, in two forms: noisy anchor and noisy boxes.
\nTherefore, all the models I developed are trying to address these issues. I first tried DetectorRS, but only managed to get pass 0.72 LB. So I moved on to the EfficientDet pipeline. I ran some simple tests from D4 to D7, and concluded that D6 with COCO-pretrained weight is the best. Here are my reasonings.
\nD0-D5: not large enough
\nD7: Remark that D7 got its precision by larger input size (and hence larger anchors). I really couldn't fit 1536-resized images on my GPU and get reasonable results.
\nD6 with noisy student/AdvProp pretrained cls backbone and random-initialized FPN/box, cls net: Object detection is quite hard. I couldn't make these models converge.
The next thing I did was trying to improve baseline performance of my EfficientDet-D6 model. I tried over 10 different techniques by dissecting the timm-efficientdet repo line by line and hacking into them, but turned out simplicity and elegance prevailed. The best performance is achieved by the default setting of the D6 model (with anchors unchanged; I tried KNN clustering of anchors in train set but it gave worse results), with default huber loss and focal loss. Augmentation wise, it's just an empirically deduced recipe:
\n A.Compose(\n [\n A.RandomSizedCrop(min_max_height=(800, 1024), height=1024, width=1024, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.ToGray(p=0.01),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.Transpose(p=0.5),\n A.JpegCompression(quality_lower=85, quality_upper=95, p=0.2),\n A.OneOf([\n A.Blur(blur_limit=3, p=1.0),\n A.MedianBlur(blur_limit=3, p=1.0)\n ],p=0.1),\n A.Resize(height=1024, width=1024, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\nand for __getitem__:
if self.test or random.random() > 0.33:\n image, boxes = self.load_image_and_boxes(index)\nelif random.random() > 0.5:\n image, boxes = self.load_cutmix_image_and_boxes(index)\nelse:\n image, boxes = self.load_mixup_image_and_boxes(index)\nRemark that mixup is the 1:1 version; I didn't get the beta version to work (I tried to reweight the boxes during loss calculation; I got worse results). Then I tuned the WBF over TTA (8x, because 16x has duplicates) pipeline a bit with basic sanity checks and exclusion of boxes that have low score or are too big or small. The overall baseline gave public LB of around 0.753.
\nThen, I realized that similar to the TGS competition I participated few years back, we can get more data by doing jigsaws. You can check out one of the popular public kernels for how to restore the original full-sized images. After this, I realized two more problems:
\nBoxes over the edge are fractured in the sense that if a wheat head appears in half in one image and in half in another, there will be two boxes in the final image. This will generate huge amount of noise and will certainly degrade model performance.
This is more subtle. Remark that in the data insight section I talked about the conundrum of boxes over the edge. Now, if we are now cropping 1024 patches from the larger image, the resulting distribution is actually different than the one represented by what we now can see as crops at special positions. I didn't really document the changes in model performance, but it is indeed significant enough for training.
My solutions is as follows:
\nI tried to correct every single label manually but got bored after 30 minutes. So I wrote an automatic procedure that does that for me:
\nWhat I did here is basically treating images obtained from jigsaw as \"pseudo-labels,\" or simply images that come from different distributions.
\nI also designed a detection variant of FixMatch, but I couldn't fit it in my GPU with image size = 1024.
\nFinally the semi-supervised learning part was pretty mundane, as fancy techniques (STAC, SplitBatchNorm, Pi-Model, Mean Teacher, etc) did not fit under the kernel eval time frame. All I did was to freeze the backbone to allow training with 16GB vram, and different hyperparameters to generate the pseudolabels. You can see the details in my open-sourced kernels. Most choices are more empirical than insightful.
\nSorry for this messy solution journal as working alone gives you a bad habit of trying too many stuff and not documenting well enough. Also, since I'm working alone, I didn't have enough time to investigate in depth some of the ideas I brought up above. I will seek to improve the solution journal in the upcoming days, and maybe write a tech report if I have time.
", "rawMarkdown": "[update] The code is now available at https://github.com/liaopeiyuan/TransferDet .\n\nNow that the results are out, I've decided to share a bit more than [my previous post](https://www.kaggle.com/c/global-wheat-detection/discussion/172433).\n\nFirst of all, huge thanks to @rwightman for his PyTorch implementation of EfficientDet and @shonenkov for his amazing starter kernel. I've also learned a lot from the discussion posts.\n\nThis is an interesting challenge with a very noisy dataset and huge domain shift between train/test, and within train/test as well. The kernel running time limit also gives possibility to do semi-supervised learning, which can in a way reduce the aforementioned problem.\n\nThe first choice I made was the ways I validate and perform model selection. This is a tough choice because 1. object detection is inherently a hard task and 2. the training data are noisy and unstable.\nEventually my decision was \"relative detection loss\" on a 20% stratified split with bad boxes removed (by manual inspection), as hinted [here](https://www.kaggle.com/c/global-wheat-detection/discussion/166564#927359). My reasoning is as follows:\n\nWe know that train and test distributions are wildly different, so mAP to me would make little sense because it's a heavy processed result (with hyperparameters in between of nms, and which mAP to look at, etc.) And detection loss we use to train our networks can capture the overall classification and regression performance. In addition, I don't use this loss to compare across models, and instead I accumulated a list of procedures that I considered as \"comparable under loss,\" by pure empirical results and a little bit of instinct, and used them to select minor choices such as augmentations. For model selection overall, I took the public LB with a grain of salt, and in a way because I ran out of time to train diverse models, I didn't have to make those huge choices of model ensemble. I believe that I can investigate into this more once the entire set is released and cleaned.\n\nNow, on to data insights. We can all agree that the data, even with blatantly bad boxes removed, are really noisy, in two forms: noisy anchor and noisy boxes. \n- For noisy boxes, this is the direct result of different understanding of wheat heads by annotators (including an automatic one, Yolov3). So I ran a few experiments to ascertain the noise. I degraded the boxes' x,y coordinates by 10% and 5% respectively (kind of the norm on articles robust object detection), and trained with my previously best pipeline. Interestingly, for both degradation we've seen a drop in model performance, so in a way the data is more informative than a 5% shift. My models were extremely costly to run, so I didn't run any more tests or designed experiments that could reveal more about bbox noise, but I had a basic idea of how to proceed. \n- Noisy anchors, on the other hand, is a little tricky, because the problem is not just about disagreements between labellers. Remark that our training data are 1024 crops of larger pictures, and (correct me if I'm mistaken) annotation is only done on crops, with the wheat heads on edges only getting annotated if 1/3 of the head is visible. This is kind of crucial because APs in this competition has a huge impact on the resulting mAP performance. Models that are too confident over the border (identifying all wheat heads no matter if it's over 1/3 or not, sometimes even leaves mistakenly) or too pessimistic both degrades final performance. This also explains my jigsaw trick, which I will explain in the following paragraphs. Finally, the source images are really different. One could train a simple CNN classifier and classify the source with validation accuracy 99.99%. Even so, you can learn a metric learning model on say 4 sources and achieve 80%+ homogeneity when extracting embedding on the remaining 3 sources that the model has never seen.\n\nTherefore, all the models I developed are trying to address these issues. I first tried DetectorRS, but only managed to get pass 0.72 LB. So I moved on to the EfficientDet pipeline. I ran some simple tests from D4 to D7, and concluded that D6 with COCO-pretrained weight is the best. Here are my reasonings.\n\nD0-D5: not large enough\nD7: Remark that D7 got its precision by larger input size (and hence larger anchors). I really couldn't fit 1536-resized images on my GPU and get reasonable results.\nD6 with noisy student/AdvProp pretrained cls backbone and random-initialized FPN/box, cls net: Object detection is quite hard. I couldn't make these models converge.\n\nThe next thing I did was trying to improve baseline performance of my EfficientDet-D6 model. I tried over 10 different techniques by dissecting the timm-efficientdet repo line by line and hacking into them, but turned out simplicity and elegance prevailed. The best performance is achieved by the default setting of the D6 model (with anchors unchanged; I tried KNN clustering of anchors in train set but it gave worse results), with default huber loss and focal loss. Augmentation wise, it's just an empirically deduced recipe:\n\n```\n A.Compose(\n [\n A.RandomSizedCrop(min_max_height=(800, 1024), height=1024, width=1024, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.ToGray(p=0.01),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n\t A.Transpose(p=0.5),\n\t A.JpegCompression(quality_lower=85, quality_upper=95, p=0.2),\n\t A.OneOf([\n\t\tA.Blur(blur_limit=3, p=1.0),\n\t\tA.MedianBlur(blur_limit=3, p=1.0)\n\t ],p=0.1),\n A.Resize(height=1024, width=1024, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\n```\n\nand for `__getitem__`:\n\n```\nif self.test or random.random() > 0.33:\n image, boxes = self.load_image_and_boxes(index)\nelif random.random() > 0.5:\n image, boxes = self.load_cutmix_image_and_boxes(index)\nelse:\n image, boxes = self.load_mixup_image_and_boxes(index)\n```\n\nRemark that mixup is the 1:1 version; I didn't get the beta version to work (I tried to reweight the boxes during loss calculation; I got worse results). Then I tuned the WBF over TTA (8x, because 16x has duplicates) pipeline a bit with basic sanity checks and exclusion of boxes that have low score or are too big or small. The overall baseline gave public LB of around 0.753.\n\nThen, I realized that similar to the TGS competition I participated few years back, we can get more data by doing jigsaws. You can check out one of the popular public kernels for how to restore the original full-sized images. After this, I realized two more problems: \n\n1. Boxes over the edge are fractured in the sense that if a wheat head appears in half in one image and in half in another, there will be two boxes in the final image. This will generate huge amount of noise and will certainly degrade model performance.\n\n2. This is more subtle. Remark that in the data insight section I talked about the conundrum of boxes over the edge. Now, if we are now cropping 1024 patches from the larger image, the resulting distribution is actually different than the one represented by what we now can see as crops at special positions. I didn't really document the changes in model performance, but it is indeed significant enough for training.\n\nMy solutions is as follows:\n\n1. I tried to correct every single label manually but got bored after 30 minutes. So I wrote an automatic procedure that does that for me:\n - Get all boxes that have one of the coordinates divisible by 1024\n - Prune those over the edge of the big picture\n - Generate the segments over the edge and match it to box (aka we only care about the side on the edge)\n - Calculate pariwise-IoU over edges\n - Pair boxes using a greedy approach\n - Fuse boxes\nI then used the corrected data to train my models. And, to encourage APs gain for my models, The images have a 0.5 chance of feeding into the network without cropping, e.g., resizing an 2048x2048 down to 1024x1024 instead of cropping it from the larger image. I did some ablation experiments and this does improve model performance.\n\n2. What I did here is basically treating images obtained from jigsaw as \"pseudo-labels,\" or simply images that come from different distributions.\n - The first model I developed is based on the paper \"A Simple Semi-Supervised Learning Framework for Object Detection\". Basically, the resulting loss is a weighted sum of loss calculated by original image and jigsaw image separately. \n - Then, I remembered the trick introduced in the paper \"Adversarial Examples Improve Image Recognition\", where images of different distributions goes through different BatchNorms. So I copied the batchnorm stats from baseline and replicate it twice, and during 2-nd stage training original data and jigsaw data passes through different norms, and the stats are calculated separately. During inference, I use the distribution of the original data. I tried extended scenario such as a 7-split norm by source, or average of different splits of norms during inference, all of which gave improvements locally, but not on public LB, so I didn't investigate further.\n\nI also designed a detection variant of FixMatch, but I couldn't fit it in my GPU with image size = 1024.\n\nFinally the semi-supervised learning part was pretty mundane, as fancy techniques (STAC, SplitBatchNorm, Pi-Model, Mean Teacher, etc) did not fit under the kernel eval time frame. All I did was to freeze the backbone to allow training with 16GB vram, and different hyperparameters to generate the pseudolabels. You can see the details in my open-sourced kernels. Most choices are more empirical than insightful.\n\nSorry for this messy solution journal as working alone gives you a bad habit of trying too many stuff and not documenting well enough. Also, since I'm working alone, I didn't have enough time to investigate in depth some of the ideas I brought up above. I will seek to improve the solution journal in the upcoming days, and maybe write a tech report if I have time.", "votes": null }, { "id": "978168", "postDate": "08/20/2020 02:12:10", "content": "Congrats on becoming GM @alexanderliao and thanks for the writeup
", "rawMarkdown": "Congrats on becoming GM @alexanderliao and thanks for the writeup", "votes": null }, { "id": "978221", "postDate": "08/20/2020 03:20:58", "content": "Congrats on GM
", "rawMarkdown": "Congrats on GM", "votes": null }, { "id": "978259", "postDate": "08/20/2020 04:07:15", "content": "Congrats @alexanderliao and Thanks for sharing your solution in details.
\nYour solution seems single model using pseudo labeling.
\nIt means your solution is powerful and it can be applied other objects not wheat head.
I am really happy to have good results for you.
\nCongrats again and enjoy!
@alexanderliao How are you so good at such a young age đź’Ż
\nJust a quick question, I noticed the common augmentation pipeline is as described above, but, in particular, \nA.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9)\n
\ncaught my eyes, if i'm not wrong, the HSV in Albumentation has default value of around 20. I experimented and think that 0.2 for each hsv did almost nothing to the image.
You are right. I must've misread the albumentation APIs. It still changes the pixel values by 1-2 on average though. I could run a few more experiments to determine if this part is crucial to my solution.
", "rawMarkdown": "You are right. I must've misread the albumentation APIs. It still changes the pixel values by 1-2 on average though. I could run a few more experiments to determine if this part is crucial to my solution.", "votes": null }, { "id": "978354", "postDate": "08/20/2020 06:03:08", "content": "@alexanderliao Thanks for the reply! And congrats on achieving \"possibly the youngest\" GM status haha
", "rawMarkdown": "alexanderliao Thanks for the reply! And congrats on achieving \"possibly the youngest\" GM status haha", "votes": null }, { "id": "978368", "postDate": "08/20/2020 06:18:52", "content": "@anokas Mikel is way earlier than I am when he got his GM status :)
", "rawMarkdown": "anokas Mikel is way earlier than I am when he got his GM status :)", "votes": null }, { "id": "978374", "postDate": "08/20/2020 06:25:17", "content": "@alexanderliao Oh dear… I'm way behind haha - just got in touch with ML/DL1 year ago!
", "rawMarkdown": "alexanderliao Oh dear... I'm way behind haha - just got in touch with ML/DL1 year ago!", "votes": null }, { "id": "978456", "postDate": "08/20/2020 07:33:23", "content": "And congrats @alexanderliao for Competitions Grandmaster!
\nđź‘Ť
Congrats @alexanderliao on your placement!
", "rawMarkdown": "Congrats @alexanderliao on your placement!", "votes": null }, { "id": "983696", "postDate": "08/24/2020 14:18:54", "content": "Congrats @alexanderliao!
", "rawMarkdown": "Congrats @alexanderliao!", "votes": null }, { "id": "984034", "postDate": "08/24/2020 19:24:26", "content": "You definitely did a lot of trial and error to get the solution. Great persistence! It's how I'm learning, and to my knowledge, the only way to truly become proficient. Congrat!
", "rawMarkdown": "You definitely did a lot of trial and error to get the solution. Great persistence! It's how I'm learning, and to my knowledge, the only way to truly become proficient. Congrat!", "votes": null }, { "id": "986401", "postDate": "08/26/2020 13:03:30", "content": "Update: the code is released at https://github.com/liaopeiyuan/TransferDet .
", "rawMarkdown": "Update: the code is released at https://github.com/liaopeiyuan/TransferDet .", "votes": null }, { "id": "988397", "postDate": "08/28/2020 02:44:54", "content": "Congratulations on your result and well deserved GM title. Thank you for opensourcing your code. It would be helpful for people like me who wants to replicate your result if you could add a README describing step-by-step process to replicate your solution. [BTW, you can edit this title to 3rd place now :)]
", "rawMarkdown": "Congratulations on your result and well deserved GM title. Thank you for opensourcing your code. It would be helpful for people like me who wants to replicate your result if you could add a README describing step-by-step process to replicate your solution. [BTW, you can edit this title to 3rd place now :)]", "votes": null }, { "id": "988603", "postDate": "08/28/2020 06:38:29", "content": "Dear @alexanderliao, it's me again. This question might sound a bit out of context, but would you mind enlightening me over some code snippets on how to modify an efficientdet when we are predicting more than 1 class? For instance, in this comp, we only have 1 class, what if there are multiple classes? How should we go about modifying the getitem method and others? Currently, the labels look like this.
# there is only one class
\nlabels = torch.ones((boxes.shape[0],), dtype=torch.int64)
Hi @alexanderliao, congratz on your work ! Thanks for sharing so much, I really learn a lot from your posts & code. I particularly appreciate your quality of writing, which is not so common in tech reporting !
\nI did not find any clue about your hardware settings, though. I would be interested in knowing a bit more about it, especially on the GPU you used for experimenting so much.
\nDid you rely on several computing solutions (local single/multi gpu, kaggle kernels, cloud instances …) for training ?
\nYou mentioned a few times the 16 vram gpu you’ve worked with. It would be very instructive to know more about your working process, as those hardware limitations are closely related to design choices and ideas you’ve tried.
\nDid you try some mixed-precision training ?
\nIt is of course much more impressive (and informative) to see some results obtained with affordable hardware settings, rather than insanely expensive ones.
\nThanks again !
", "rawMarkdown": "Hi @alexanderliao, congratz on your work ! Thanks for sharing so much, I really learn a lot from your posts & code. I particularly appreciate your quality of writing, which is not so common in tech reporting !\n\nI did not find any clue about your hardware settings, though. I would be interested in knowing a bit more about it, especially on the GPU you used for experimenting so much.\n\nDid you rely on several computing solutions (local single/multi gpu, kaggle kernels, cloud instances …) for training ?\n\nYou mentioned a few times the 16 vram gpu you’ve worked with. It would be very instructive to know more about your working process, as those hardware limitations are closely related to design choices and ideas you’ve tried. \n\nDid you try some mixed-precision training ?\n\nIt is of course much more impressive (and informative) to see some results obtained with affordable hardware settings, rather than insanely expensive ones.\n\nThanks again !", "votes": null }, { "id": "1074114", "postDate": "11/10/2020 09:14:07", "content": "ReadonlyConfigError Traceback (most recent call last)
\n in
\n 11 return DetBenchTrain(net, config)
\n 12
\n---> 13 net = get_net()
\n 14 run_training()
I am receiving this error. Can anyone help?
", "rawMarkdown": "ReadonlyConfigError Traceback (most recent call last)\ncongrats @alexanderliao for Competitions Grandmaster
", "rawMarkdown": "congrats @alexanderliao for Competitions Grandmaster", "votes": null }, { "id": "1197893", "postDate": "02/12/2021 14:16:42", "content": "Thank you sharing your work đź‘Ť
", "rawMarkdown": "Thank you sharing your work đź‘Ť", "votes": null } ], "comments": [ { "id": 978168, "author_name": "duykhanh99", "author_url": "", "post_date": "08/20/2020 02:12:10", "content": "Congrats on becoming GM @alexanderliao and thanks for the writeup
", "votes": null, "replies": [] }, { "id": 978221, "author_name": "bopengiowa", "author_url": "", "post_date": "08/20/2020 03:20:58", "content": "Congrats on GM
", "votes": null, "replies": [] }, { "id": 978259, "author_name": "piantic", "author_url": "", "post_date": "08/20/2020 04:07:15", "content": "Congrats @alexanderliao and Thanks for sharing your solution in details.
\nYour solution seems single model using pseudo labeling.
\nIt means your solution is powerful and it can be applied other objects not wheat head.
I am really happy to have good results for you.
\nCongrats again and enjoy!
@alexanderliao How are you so good at such a young age đź’Ż
\nJust a quick question, I noticed the common augmentation pipeline is as described above, but, in particular, \nA.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9)\n
\ncaught my eyes, if i'm not wrong, the HSV in Albumentation has default value of around 20. I experimented and think that 0.2 for each hsv did almost nothing to the image.
You are right. I must've misread the albumentation APIs. It still changes the pixel values by 1-2 on average though. I could run a few more experiments to determine if this part is crucial to my solution.
", "votes": null, "replies": [] }, { "id": 978354, "author_name": "reighns", "author_url": "", "post_date": "08/20/2020 06:03:08", "content": "@alexanderliao Thanks for the reply! And congrats on achieving \"possibly the youngest\" GM status haha
", "votes": null, "replies": [] }, { "id": 978368, "author_name": "alexanderliao", "author_url": "", "post_date": "08/20/2020 06:18:52", "content": "@anokas Mikel is way earlier than I am when he got his GM status :)
", "votes": null, "replies": [] }, { "id": 978374, "author_name": "reighns", "author_url": "", "post_date": "08/20/2020 06:25:17", "content": "@alexanderliao Oh dear… I'm way behind haha - just got in touch with ML/DL1 year ago!
", "votes": null, "replies": [] } ] }, { "id": 978456, "author_name": "piantic", "author_url": "", "post_date": "08/20/2020 07:33:23", "content": "And congrats @alexanderliao for Competitions Grandmaster!
\nđź‘Ť
Congrats @alexanderliao on your placement!
", "votes": null, "replies": [] }, { "id": 983696, "author_name": "arunprakashfrtech", "author_url": "", "post_date": "08/24/2020 14:18:54", "content": "Congrats @alexanderliao!
", "votes": null, "replies": [] }, { "id": 984034, "author_name": "baronsmith", "author_url": "", "post_date": "08/24/2020 19:24:26", "content": "You definitely did a lot of trial and error to get the solution. Great persistence! It's how I'm learning, and to my knowledge, the only way to truly become proficient. Congrat!
", "votes": null, "replies": [] }, { "id": 986401, "author_name": "alexanderliao", "author_url": "", "post_date": "08/26/2020 13:03:30", "content": "Update: the code is released at https://github.com/liaopeiyuan/TransferDet .
", "votes": null, "replies": [] }, { "id": 988397, "author_name": "bibek777", "author_url": "", "post_date": "08/28/2020 02:44:54", "content": "Congratulations on your result and well deserved GM title. Thank you for opensourcing your code. It would be helpful for people like me who wants to replicate your result if you could add a README describing step-by-step process to replicate your solution. [BTW, you can edit this title to 3rd place now :)]
", "votes": null, "replies": [] }, { "id": 988603, "author_name": "reighns", "author_url": "", "post_date": "08/28/2020 06:38:29", "content": "Dear @alexanderliao, it's me again. This question might sound a bit out of context, but would you mind enlightening me over some code snippets on how to modify an efficientdet when we are predicting more than 1 class? For instance, in this comp, we only have 1 class, what if there are multiple classes? How should we go about modifying the getitem method and others? Currently, the labels look like this.
# there is only one class
\nlabels = torch.ones((boxes.shape[0],), dtype=torch.int64)
Hi @alexanderliao, congratz on your work ! Thanks for sharing so much, I really learn a lot from your posts & code. I particularly appreciate your quality of writing, which is not so common in tech reporting !
\nI did not find any clue about your hardware settings, though. I would be interested in knowing a bit more about it, especially on the GPU you used for experimenting so much.
\nDid you rely on several computing solutions (local single/multi gpu, kaggle kernels, cloud instances …) for training ?
\nYou mentioned a few times the 16 vram gpu you’ve worked with. It would be very instructive to know more about your working process, as those hardware limitations are closely related to design choices and ideas you’ve tried.
\nDid you try some mixed-precision training ?
\nIt is of course much more impressive (and informative) to see some results obtained with affordable hardware settings, rather than insanely expensive ones.
\nThanks again !
", "votes": null, "replies": [] }, { "id": 1074114, "author_name": "srishtis1258", "author_url": "", "post_date": "11/10/2020 09:14:07", "content": "ReadonlyConfigError Traceback (most recent call last)
\n in
\n 11 return DetBenchTrain(net, config)
\n 12
\n---> 13 net = get_net()
\n 14 run_training()
I am receiving this error. Can anyone help?
", "votes": null, "replies": [] }, { "id": 1172712, "author_name": "caojamie", "author_url": "", "post_date": "01/27/2021 14:13:22", "content": "congrats @alexanderliao for Competitions Grandmaster
", "votes": null, "replies": [] }, { "id": 1197893, "author_name": "yogeshkumarpilli", "author_url": "", "post_date": "02/12/2021 14:16:42", "content": "Thank you sharing your work đź‘Ť
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "978155": "[update] The code is now available at https://github.com/liaopeiyuan/TransferDet .\n\nNow that the results are out, I've decided to share a bit more than [my previous post](https://www.kaggle.com/c/global-wheat-detection/discussion/172433).\n\nFirst of all, huge thanks to @rwightman for his PyTorch implementation of EfficientDet and @shonenkov for his amazing starter kernel. I've also learned a lot from the discussion posts.\n\nThis is an interesting challenge with a very noisy dataset and huge domain shift between train/test, and within train/test as well. The kernel running time limit also gives possibility to do semi-supervised learning, which can in a way reduce the aforementioned problem.\n\nThe first choice I made was the ways I validate and perform model selection. This is a tough choice because 1. object detection is inherently a hard task and 2. the training data are noisy and unstable.\nEventually my decision was \"relative detection loss\" on a 20% stratified split with bad boxes removed (by manual inspection), as hinted [here](https://www.kaggle.com/c/global-wheat-detection/discussion/166564#927359). My reasoning is as follows:\n\nWe know that train and test distributions are wildly different, so mAP to me would make little sense because it's a heavy processed result (with hyperparameters in between of nms, and which mAP to look at, etc.) And detection loss we use to train our networks can capture the overall classification and regression performance. In addition, I don't use this loss to compare across models, and instead I accumulated a list of procedures that I considered as \"comparable under loss,\" by pure empirical results and a little bit of instinct, and used them to select minor choices such as augmentations. For model selection overall, I took the public LB with a grain of salt, and in a way because I ran out of time to train diverse models, I didn't have to make those huge choices of model ensemble. I believe that I can investigate into this more once the entire set is released and cleaned.\n\nNow, on to data insights. We can all agree that the data, even with blatantly bad boxes removed, are really noisy, in two forms: noisy anchor and noisy boxes. \n- For noisy boxes, this is the direct result of different understanding of wheat heads by annotators (including an automatic one, Yolov3). So I ran a few experiments to ascertain the noise. I degraded the boxes' x,y coordinates by 10% and 5% respectively (kind of the norm on articles robust object detection), and trained with my previously best pipeline. Interestingly, for both degradation we've seen a drop in model performance, so in a way the data is more informative than a 5% shift. My models were extremely costly to run, so I didn't run any more tests or designed experiments that could reveal more about bbox noise, but I had a basic idea of how to proceed. \n- Noisy anchors, on the other hand, is a little tricky, because the problem is not just about disagreements between labellers. Remark that our training data are 1024 crops of larger pictures, and (correct me if I'm mistaken) annotation is only done on crops, with the wheat heads on edges only getting annotated if 1/3 of the head is visible. This is kind of crucial because APs in this competition has a huge impact on the resulting mAP performance. Models that are too confident over the border (identifying all wheat heads no matter if it's over 1/3 or not, sometimes even leaves mistakenly) or too pessimistic both degrades final performance. This also explains my jigsaw trick, which I will explain in the following paragraphs. Finally, the source images are really different. One could train a simple CNN classifier and classify the source with validation accuracy 99.99%. Even so, you can learn a metric learning model on say 4 sources and achieve 80%+ homogeneity when extracting embedding on the remaining 3 sources that the model has never seen.\n\nTherefore, all the models I developed are trying to address these issues. I first tried DetectorRS, but only managed to get pass 0.72 LB. So I moved on to the EfficientDet pipeline. I ran some simple tests from D4 to D7, and concluded that D6 with COCO-pretrained weight is the best. Here are my reasonings.\n\nD0-D5: not large enough\nD7: Remark that D7 got its precision by larger input size (and hence larger anchors). I really couldn't fit 1536-resized images on my GPU and get reasonable results.\nD6 with noisy student/AdvProp pretrained cls backbone and random-initialized FPN/box, cls net: Object detection is quite hard. I couldn't make these models converge.\n\nThe next thing I did was trying to improve baseline performance of my EfficientDet-D6 model. I tried over 10 different techniques by dissecting the timm-efficientdet repo line by line and hacking into them, but turned out simplicity and elegance prevailed. The best performance is achieved by the default setting of the D6 model (with anchors unchanged; I tried KNN clustering of anchors in train set but it gave worse results), with default huber loss and focal loss. Augmentation wise, it's just an empirically deduced recipe:\n\n```\n A.Compose(\n [\n A.RandomSizedCrop(min_max_height=(800, 1024), height=1024, width=1024, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.ToGray(p=0.01),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n\t A.Transpose(p=0.5),\n\t A.JpegCompression(quality_lower=85, quality_upper=95, p=0.2),\n\t A.OneOf([\n\t\tA.Blur(blur_limit=3, p=1.0),\n\t\tA.MedianBlur(blur_limit=3, p=1.0)\n\t ],p=0.1),\n A.Resize(height=1024, width=1024, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\n```\n\nand for `__getitem__`:\n\n```\nif self.test or random.random() > 0.33:\n image, boxes = self.load_image_and_boxes(index)\nelif random.random() > 0.5:\n image, boxes = self.load_cutmix_image_and_boxes(index)\nelse:\n image, boxes = self.load_mixup_image_and_boxes(index)\n```\n\nRemark that mixup is the 1:1 version; I didn't get the beta version to work (I tried to reweight the boxes during loss calculation; I got worse results). Then I tuned the WBF over TTA (8x, because 16x has duplicates) pipeline a bit with basic sanity checks and exclusion of boxes that have low score or are too big or small. The overall baseline gave public LB of around 0.753.\n\nThen, I realized that similar to the TGS competition I participated few years back, we can get more data by doing jigsaws. You can check out one of the popular public kernels for how to restore the original full-sized images. After this, I realized two more problems: \n\n1. Boxes over the edge are fractured in the sense that if a wheat head appears in half in one image and in half in another, there will be two boxes in the final image. This will generate huge amount of noise and will certainly degrade model performance.\n\n2. This is more subtle. Remark that in the data insight section I talked about the conundrum of boxes over the edge. Now, if we are now cropping 1024 patches from the larger image, the resulting distribution is actually different than the one represented by what we now can see as crops at special positions. I didn't really document the changes in model performance, but it is indeed significant enough for training.\n\nMy solutions is as follows:\n\n1. I tried to correct every single label manually but got bored after 30 minutes. So I wrote an automatic procedure that does that for me:\n - Get all boxes that have one of the coordinates divisible by 1024\n - Prune those over the edge of the big picture\n - Generate the segments over the edge and match it to box (aka we only care about the side on the edge)\n - Calculate pariwise-IoU over edges\n - Pair boxes using a greedy approach\n - Fuse boxes\nI then used the corrected data to train my models. And, to encourage APs gain for my models, The images have a 0.5 chance of feeding into the network without cropping, e.g., resizing an 2048x2048 down to 1024x1024 instead of cropping it from the larger image. I did some ablation experiments and this does improve model performance.\n\n2. What I did here is basically treating images obtained from jigsaw as \"pseudo-labels,\" or simply images that come from different distributions.\n - The first model I developed is based on the paper \"A Simple Semi-Supervised Learning Framework for Object Detection\". Basically, the resulting loss is a weighted sum of loss calculated by original image and jigsaw image separately. \n - Then, I remembered the trick introduced in the paper \"Adversarial Examples Improve Image Recognition\", where images of different distributions goes through different BatchNorms. So I copied the batchnorm stats from baseline and replicate it twice, and during 2-nd stage training original data and jigsaw data passes through different norms, and the stats are calculated separately. During inference, I use the distribution of the original data. I tried extended scenario such as a 7-split norm by source, or average of different splits of norms during inference, all of which gave improvements locally, but not on public LB, so I didn't investigate further.\n\nI also designed a detection variant of FixMatch, but I couldn't fit it in my GPU with image size = 1024.\n\nFinally the semi-supervised learning part was pretty mundane, as fancy techniques (STAC, SplitBatchNorm, Pi-Model, Mean Teacher, etc) did not fit under the kernel eval time frame. All I did was to freeze the backbone to allow training with 16GB vram, and different hyperparameters to generate the pseudolabels. You can see the details in my open-sourced kernels. Most choices are more empirical than insightful.\n\nSorry for this messy solution journal as working alone gives you a bad habit of trying too many stuff and not documenting well enough. Also, since I'm working alone, I didn't have enough time to investigate in depth some of the ideas I brought up above. I will seek to improve the solution journal in the upcoming days, and maybe write a tech report if I have time.", "978168": "Congrats on becoming GM @alexanderliao and thanks for the writeup", "978221": "Congrats on GM", "978259": "Congrats @alexanderliao and Thanks for sharing your solution in details.\n\nYour solution seems single model using pseudo labeling.\nIt means your solution is powerful and it can be applied other objects not wheat head.\n \nI am really happy to have good results for you.\nCongrats again and enjoy!", "978305": "alexanderliao How are you so good at such a young age 💯\nJust a quick question, I noticed the common augmentation pipeline is as described above, but, in particular, ```\nA.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9)\n```\ncaught my eyes, if i'm not wrong, the HSV in Albumentation has default value of around 20. I experimented and think that 0.2 for each hsv did almost nothing to the image.", "978347": "You are right. I must've misread the albumentation APIs. It still changes the pixel values by 1-2 on average though. I could run a few more experiments to determine if this part is crucial to my solution.", "978354": "alexanderliao Thanks for the reply! And congrats on achieving \"possibly the youngest\" GM status haha", "978368": "anokas Mikel is way earlier than I am when he got his GM status :)", "978374": "alexanderliao Oh dear... I'm way behind haha - just got in touch with ML/DL1 year ago!", "978456": "And congrats @alexanderliao for Competitions Grandmaster!\n👍", "978877": "Congrats @alexanderliao on your placement!", "983696": "Congrats @alexanderliao!", "984034": "You definitely did a lot of trial and error to get the solution. Great persistence! It's how I'm learning, and to my knowledge, the only way to truly become proficient. Congrat!", "986401": "Update: the code is released at https://github.com/liaopeiyuan/TransferDet .", "988397": "Congratulations on your result and well deserved GM title. Thank you for opensourcing your code. It would be helpful for people like me who wants to replicate your result if you could add a README describing step-by-step process to replicate your solution. [BTW, you can edit this title to 3rd place now :)]", "988603": "Dear @alexanderliao, it's me again. This question might sound a bit out of context, but would you mind enlightening me over some code snippets on how to modify an efficientdet when we are predicting more than 1 class? For instance, in this comp, we only have 1 class, what if there are multiple classes? How should we go about modifying the `getitem` method and others? Currently, the labels look like this.\n\n` # there is only one class`\n` labels = torch.ones((boxes.shape[0],), dtype=torch.int64)`", "991611": "Hi @alexanderliao, congratz on your work ! Thanks for sharing so much, I really learn a lot from your posts & code. I particularly appreciate your quality of writing, which is not so common in tech reporting !\n\nI did not find any clue about your hardware settings, though. I would be interested in knowing a bit more about it, especially on the GPU you used for experimenting so much.\n\nDid you rely on several computing solutions (local single/multi gpu, kaggle kernels, cloud instances …) for training ?\n\nYou mentioned a few times the 16 vram gpu you’ve worked with. It would be very instructive to know more about your working process, as those hardware limitations are closely related to design choices and ideas you’ve tried. \n\nDid you try some mixed-precision training ?\n\nIt is of course much more impressive (and informative) to see some results obtained with affordable hardware settings, rather than insanely expensive ones.\n\nThanks again !", "1074114": "ReadonlyConfigError Traceback (most recent call last)\n