![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4256010%2Ffd3e335bf416544a894ff550af5af5eb%2Fdiag1.png?generation=1596729863641670&alt=media\")
Hi all! It's been a real pleasure working alongside you during this competition. Before YOLOv5 came into the mix, I was sitting in 2nd place on the public leaderboard thanks to a neat little trick that gave me a bigger boost than expected.
\nI was working off an EffecientDetB5 model which gave around 0.72 on the public scoreboard without any TTA. With my TTA trick I got a massive boost to 0.7535. I'll explain it below, and would love to get your opinion on why it might have been so useful. Also, here is a notebook with a working example.
\nThe motivation was that I didn't have time to be training 1024x1024, but I still wanted to work with full resolution while training 640x640. So the idea was:
\n1) Break the test images up into 640x640 quadrants (so there would be some overlap between them).
\n2) Do inference on all 4 quadrants and stitch the results together.
\n3) Do inference on the full image resized to 640x640 (to help fill gaps - more on this later).
\n4) Ensemble with WBF.
Here's a diagram explaining the quadrants + full image resized.
\n
About the stitching
\nI didn't want to keep any boxes that got too close to the edge of the quadrants because they could cut through the middle of a wheat head. So I discarded them.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4256010%2Fdef474abb689561acb28b252830e14ca%2Fdiag2.png?generation=1596730533838943&alt=media\")
And now you can see why I also did inference on the rescaled full image. I wanted to capture any boxes that may have been discarded with the method above.
\nAbout the training
\nI found that with this method, mosaic augmentation during training didn't help my score. I stuck with doing random resized crops of the training images. The size of the crops were uniformly distributed in [640, 1024]. That way I had the benefit of training near full resolution a lot of the time.
\nQuestion to Kagglers
\nI'm actually not familiar with the concept of multi-scale inference. I understand it's a TTA technique. What exactly is happening in it? Is it the similar to what I did?
", "messages": [ { "id": "960726", "postDate": "08/06/2020 16:20:41", "content": "Hi all! It's been a real pleasure working alongside you during this competition. Before YOLOv5 came into the mix, I was sitting in 2nd place on the public leaderboard thanks to a neat little trick that gave me a bigger boost than expected.
\nI was working off an EffecientDetB5 model which gave around 0.72 on the public scoreboard without any TTA. With my TTA trick I got a massive boost to 0.7535. I'll explain it below, and would love to get your opinion on why it might have been so useful. Also, here is a notebook with a working example.
\nThe motivation was that I didn't have time to be training 1024x1024, but I still wanted to work with full resolution while training 640x640. So the idea was:
\n1) Break the test images up into 640x640 quadrants (so there would be some overlap between them).
\n2) Do inference on all 4 quadrants and stitch the results together.
\n3) Do inference on the full image resized to 640x640 (to help fill gaps - more on this later).
\n4) Ensemble with WBF.
Here's a diagram explaining the quadrants + full image resized.
\n
About the stitching
\nI didn't want to keep any boxes that got too close to the edge of the quadrants because they could cut through the middle of a wheat head. So I discarded them.
\n
And now you can see why I also did inference on the rescaled full image. I wanted to capture any boxes that may have been discarded with the method above.
\nAbout the training
\nI found that with this method, mosaic augmentation during training didn't help my score. I stuck with doing random resized crops of the training images. The size of the crops were uniformly distributed in [640, 1024]. That way I had the benefit of training near full resolution a lot of the time.
\nQuestion to Kagglers
\nI'm actually not familiar with the concept of multi-scale inference. I understand it's a TTA technique. What exactly is happening in it? Is it the similar to what I did?
", "rawMarkdown": "Hi all! It's been a real pleasure working alongside you during this competition. Before YOLOv5 came into the mix, I was sitting in 2nd place on the public leaderboard thanks to a neat little trick that gave me a bigger boost than expected.\n\nI was working off an EffecientDetB5 model which gave around 0.72 on the public scoreboard without any TTA. With my TTA trick I got a massive boost to 0.7535. I'll explain it below, and would love to get your opinion on why it might have been so useful. Also, here is a [notebook with a working example](https://www.kaggle.com/alexandersoare/gwd-detect-by-quadrant-tta-trick).\n\nThe motivation was that I didn't have time to be training 1024x1024, but I still wanted to work with full resolution while training 640x640. So the idea was:\n\n1) Break the test images up into 640x640 quadrants (so there would be some overlap between them).\n2) Do inference on all 4 quadrants and stitch the results together.\n3) Do inference on the full image resized to 640x640 (to help fill gaps - more on this later).\n4) Ensemble with WBF.\n\nHere's a diagram explaining the quadrants + full image resized.\n\n\n\n**About the stitching**\n\nI didn't want to keep any boxes that got too close to the edge of the quadrants because they could cut through the middle of a wheat head. So I discarded them.\n\n\n\nAnd now you can see why I also did inference on the rescaled full image. I wanted to capture any boxes that may have been discarded with the method above.\n\n**About the training**\n\nI found that with this method, mosaic augmentation during training didn't help my score. I stuck with doing random resized crops of the training images. The size of the crops were uniformly distributed in [640, 1024]. That way I had the benefit of training near full resolution a lot of the time.\n\n**Question to Kagglers**\n\nI'm actually not familiar with the concept of multi-scale inference. I understand it's a TTA technique. What exactly is happening in it? Is it the similar to what I did?", "votes": null }, { "id": "960729", "postDate": "08/06/2020 16:27:08", "content": "This is kind of an idea that I thought in my mind and never do it cause somehow I believed it won't work but seeing your example taught me a lesson, I should have learnt long ago.
", "rawMarkdown": "This is kind of an idea that I thought in my mind and never do it cause somehow I believed it won't work but seeing your example taught me a lesson, I should have learnt long ago.", "votes": null }, { "id": "960731", "postDate": "08/06/2020 16:30:04", "content": "Lol I know what you mean. But hindsight is 20/20 - your first intuition might be right more often than wrong.
", "rawMarkdown": "Lol I know what you mean. But hindsight is 20/20 - your first intuition might be right more often than wrong.", "votes": null }, { "id": "961240", "postDate": "08/07/2020 03:15:35", "content": "This trick I read about the first time in the AlexNet paper … 224px random crops crops of 256px X 10 runs for TTA … Didn't think it would work that well in a detection problem tho :) … Thank you for the insight :) it makes perfect sense though because the feature extractor would fire if it sees a certain shape (I believe this is why the inversion and rotation work too)
\nI believe Multi Scale is basically you inference at different resolutions of your full image or random crops and average the results (because the regressor is outputting a normalized point (0-1) anyway so it's not scale sensitive)
", "rawMarkdown": "This trick I read about the first time in the AlexNet paper ... 224px random crops crops of 256px X 10 runs for TTA ... Didn't think it would work that well in a detection problem tho :) ... Thank you for the insight :) it makes perfect sense though because the feature extractor would fire if it sees a certain shape (I believe this is why the inversion and rotation work too)\n\nI believe Multi Scale is basically you inference at different resolutions of your full image or random crops and average the results (because the regressor is outputting a normalized point (0-1) anyway so it's not scale sensitive)", "votes": null }, { "id": "961372", "postDate": "08/07/2020 06:02:23", "content": "Hi, your approach is correct. I used the same approach one year ago in a Signate competition where we used the same cropping. It elevated us to fourth place in the competition:) We also describe the problem in a paper, but it is not published yet (only accepted) so I cannot post you a link.
\n\nThe point is that with the cropping, you give more information to the network. Of course, there are some rules to be verified, such as the crops should be as big as biggest objects in an image. But the benefits are enormous. We used it in combination with YOLOv3; in that case, the YOLO's problem of rewritten labels is reduced. On the other hand, prototypic anchors have to be adjusted because the relative size of the objects to be detected is increased.
\n\nDuring the prediction, the shift between crops can be smaller, i.e., you can take (x,x) crops with the sift of (x/y, x/y) which gives you more predictions per object and eliminate the problem with objects on borders because the object will always be presented in its non-cropped form in some of the crops. Then, WBF is more beneficial. And yes, that can be viewed as a special kind of TTA:)
", "rawMarkdown": "Hi, your approach is correct. I used the same approach one year ago in a Signate competition where we used the same cropping. It elevated us to fourth place in the competition:) We also describe the problem in a paper, but it is not published yet (only accepted) so I cannot post you a link.\n\nThe point is that with the cropping, you give more information to the network. Of course, there are some rules to be verified, such as the crops should be as big as biggest objects in an image. But the benefits are enormous. We used it in combination with YOLOv3; in that case, the YOLO's problem of rewritten labels is reduced. On the other hand, prototypic anchors have to be adjusted because the relative size of the objects to be detected is increased.\n\nDuring the prediction, the shift between crops can be smaller, i.e., you can take (x,x) crops with the sift of (x/y, x/y) which gives you more predictions per object and eliminate the problem with objects on borders because the object will always be presented in its non-cropped form in some of the crops. Then, WBF is more beneficial. And yes, that can be viewed as a special kind of TTA:)", "votes": null }, { "id": "961378", "postDate": "08/07/2020 06:10:35", "content": "Thanks a lot for the insight! One question: do you know how this is different from the \"multi scale\" TTA used by some of the too kernels?
", "rawMarkdown": "Thanks a lot for the insight! One question: do you know how this is different from the \"multi scale\" TTA used by some of the too kernels?", "votes": null }, { "id": "961511", "postDate": "08/07/2020 08:52:17", "content": "I didnt check the kernels, but generally multiscale means:
\n\nIn the case of training, a crop from the image is taken, upscaled/downscaled, and placed into a blank image. So when the crop is bigger, it is cropped, and when it is smaller, the letterbox is created. The \"issue\" is that an image still has the original resolution, e.g., 1024^2. On the other hand, the cropping, as you describe it, creates a smaller image. That is beneficial when you have limited GPU memory and want a bigger batch size. The second benefit is that you can apply the multiscale augmentation to your crops as well.
\n\nIn the case of inference, it is necessary to have a fully convolutional network. You can then resize your input image n-times and put these various-sized images into the neural network and obtain n-predictions. So, in other words, you change the resolution of the model. The network is trained somehow and may have different sensitivity to a different size of objects (because of an unbalanced dataset). Such different resolution increases the chance that your object will be hit (because NN can fully miss it). Again, this TTA can also be applied to your crops-approach.
", "rawMarkdown": "I didnt check the kernels, but generally multiscale means:\n\nIn the case of training, a crop from the image is taken, upscaled/downscaled, and placed into a blank image. So when the crop is bigger, it is cropped, and when it is smaller, the letterbox is created. The \"issue\" is that an image still has the original resolution, e.g., 1024^2. On the other hand, the cropping, as you describe it, creates a smaller image. That is beneficial when you have limited GPU memory and want a bigger batch size. The second benefit is that you can apply the multiscale augmentation to your crops as well.\n\nIn the case of inference, it is necessary to have a fully convolutional network. You can then resize your input image n-times and put these various-sized images into the neural network and obtain n-predictions. So, in other words, you change the resolution of the model. The network is trained somehow and may have different sensitivity to a different size of objects (because of an unbalanced dataset). Such different resolution increases the chance that your object will be hit (because NN can fully miss it). Again, this TTA can also be applied to your crops-approach.", "votes": null }, { "id": "961549", "postDate": "08/07/2020 09:24:10", "content": "Thanks! Something to try next time.
", "rawMarkdown": "Thanks! Something to try next time.", "votes": null } ], "comments": [ { "id": 961240, "author_name": "ma7moud", "author_url": "", "post_date": "08/07/2020 03:15:35", "content": "This trick I read about the first time in the AlexNet paper … 224px random crops crops of 256px X 10 runs for TTA … Didn't think it would work that well in a detection problem tho :) … Thank you for the insight :) it makes perfect sense though because the feature extractor would fire if it sees a certain shape (I believe this is why the inversion and rotation work too)
\nI believe Multi Scale is basically you inference at different resolutions of your full image or random crops and average the results (because the regressor is outputting a normalized point (0-1) anyway so it's not scale sensitive)
", "votes": null, "replies": [] }, { "id": 960729, "author_name": "ajay19", "author_url": "", "post_date": "08/06/2020 16:27:08", "content": "This is kind of an idea that I thought in my mind and never do it cause somehow I believed it won't work but seeing your example taught me a lesson, I should have learnt long ago.
", "votes": null, "replies": [ { "id": 960731, "author_name": "alexandersoare", "author_url": "", "post_date": "08/06/2020 16:30:04", "content": "Lol I know what you mean. But hindsight is 20/20 - your first intuition might be right more often than wrong.
", "votes": null, "replies": [] } ] }, { "id": 961372, "author_name": "mastemace", "author_url": "", "post_date": "08/07/2020 06:02:23", "content": "Hi, your approach is correct. I used the same approach one year ago in a Signate competition where we used the same cropping. It elevated us to fourth place in the competition:) We also describe the problem in a paper, but it is not published yet (only accepted) so I cannot post you a link.
\n\nThe point is that with the cropping, you give more information to the network. Of course, there are some rules to be verified, such as the crops should be as big as biggest objects in an image. But the benefits are enormous. We used it in combination with YOLOv3; in that case, the YOLO's problem of rewritten labels is reduced. On the other hand, prototypic anchors have to be adjusted because the relative size of the objects to be detected is increased.
\n\nDuring the prediction, the shift between crops can be smaller, i.e., you can take (x,x) crops with the sift of (x/y, x/y) which gives you more predictions per object and eliminate the problem with objects on borders because the object will always be presented in its non-cropped form in some of the crops. Then, WBF is more beneficial. And yes, that can be viewed as a special kind of TTA:)
", "votes": null, "replies": [ { "id": 961378, "author_name": "alexandersoare", "author_url": "", "post_date": "08/07/2020 06:10:35", "content": "Thanks a lot for the insight! One question: do you know how this is different from the \"multi scale\" TTA used by some of the too kernels?
", "votes": null, "replies": [] }, { "id": 961511, "author_name": "mastemace", "author_url": "", "post_date": "08/07/2020 08:52:17", "content": "I didnt check the kernels, but generally multiscale means:
\n\nIn the case of training, a crop from the image is taken, upscaled/downscaled, and placed into a blank image. So when the crop is bigger, it is cropped, and when it is smaller, the letterbox is created. The \"issue\" is that an image still has the original resolution, e.g., 1024^2. On the other hand, the cropping, as you describe it, creates a smaller image. That is beneficial when you have limited GPU memory and want a bigger batch size. The second benefit is that you can apply the multiscale augmentation to your crops as well.
\n\nIn the case of inference, it is necessary to have a fully convolutional network. You can then resize your input image n-times and put these various-sized images into the neural network and obtain n-predictions. So, in other words, you change the resolution of the model. The network is trained somehow and may have different sensitivity to a different size of objects (because of an unbalanced dataset). Such different resolution increases the chance that your object will be hit (because NN can fully miss it). Again, this TTA can also be applied to your crops-approach.
", "votes": null, "replies": [] }, { "id": 961549, "author_name": "alexandersoare", "author_url": "", "post_date": "08/07/2020 09:24:10", "content": "Thanks! Something to try next time.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "960726": "Hi all! It's been a real pleasure working alongside you during this competition. Before YOLOv5 came into the mix, I was sitting in 2nd place on the public leaderboard thanks to a neat little trick that gave me a bigger boost than expected.\n\nI was working off an EffecientDetB5 model which gave around 0.72 on the public scoreboard without any TTA. With my TTA trick I got a massive boost to 0.7535. I'll explain it below, and would love to get your opinion on why it might have been so useful. Also, here is a [notebook with a working example](https://www.kaggle.com/alexandersoare/gwd-detect-by-quadrant-tta-trick).\n\nThe motivation was that I didn't have time to be training 1024x1024, but I still wanted to work with full resolution while training 640x640. So the idea was:\n\n1) Break the test images up into 640x640 quadrants (so there would be some overlap between them).\n2) Do inference on all 4 quadrants and stitch the results together.\n3) Do inference on the full image resized to 640x640 (to help fill gaps - more on this later).\n4) Ensemble with WBF.\n\nHere's a diagram explaining the quadrants + full image resized.\n\n\n\n**About the stitching**\n\nI didn't want to keep any boxes that got too close to the edge of the quadrants because they could cut through the middle of a wheat head. So I discarded them.\n\n\n\nAnd now you can see why I also did inference on the rescaled full image. I wanted to capture any boxes that may have been discarded with the method above.\n\n**About the training**\n\nI found that with this method, mosaic augmentation during training didn't help my score. I stuck with doing random resized crops of the training images. The size of the crops were uniformly distributed in [640, 1024]. That way I had the benefit of training near full resolution a lot of the time.\n\n**Question to Kagglers**\n\nI'm actually not familiar with the concept of multi-scale inference. I understand it's a TTA technique. What exactly is happening in it? Is it the similar to what I did?", "960729": "This is kind of an idea that I thought in my mind and never do it cause somehow I believed it won't work but seeing your example taught me a lesson, I should have learnt long ago.", "960731": "Lol I know what you mean. But hindsight is 20/20 - your first intuition might be right more often than wrong.", "961240": "This trick I read about the first time in the AlexNet paper ... 224px random crops crops of 256px X 10 runs for TTA ... Didn't think it would work that well in a detection problem tho :) ... Thank you for the insight :) it makes perfect sense though because the feature extractor would fire if it sees a certain shape (I believe this is why the inversion and rotation work too)\n\nI believe Multi Scale is basically you inference at different resolutions of your full image or random crops and average the results (because the regressor is outputting a normalized point (0-1) anyway so it's not scale sensitive)", "961372": "Hi, your approach is correct. I used the same approach one year ago in a Signate competition where we used the same cropping. It elevated us to fourth place in the competition:) We also describe the problem in a paper, but it is not published yet (only accepted) so I cannot post you a link.\n\nThe point is that with the cropping, you give more information to the network. Of course, there are some rules to be verified, such as the crops should be as big as biggest objects in an image. But the benefits are enormous. We used it in combination with YOLOv3; in that case, the YOLO's problem of rewritten labels is reduced. On the other hand, prototypic anchors have to be adjusted because the relative size of the objects to be detected is increased.\n\nDuring the prediction, the shift between crops can be smaller, i.e., you can take (x,x) crops with the sift of (x/y, x/y) which gives you more predictions per object and eliminate the problem with objects on borders because the object will always be presented in its non-cropped form in some of the crops. Then, WBF is more beneficial. And yes, that can be viewed as a special kind of TTA:)", "961378": "Thanks a lot for the insight! One question: do you know how this is different from the \"multi scale\" TTA used by some of the too kernels?", "961511": "I didnt check the kernels, but generally multiscale means:\n\nIn the case of training, a crop from the image is taken, upscaled/downscaled, and placed into a blank image. So when the crop is bigger, it is cropped, and when it is smaller, the letterbox is created. The \"issue\" is that an image still has the original resolution, e.g., 1024^2. On the other hand, the cropping, as you describe it, creates a smaller image. That is beneficial when you have limited GPU memory and want a bigger batch size. The second benefit is that you can apply the multiscale augmentation to your crops as well.\n\nIn the case of inference, it is necessary to have a fully convolutional network. You can then resize your input image n-times and put these various-sized images into the neural network and obtain n-predictions. So, in other words, you change the resolution of the model. The network is trained somehow and may have different sensitivity to a different size of objects (because of an unbalanced dataset). Such different resolution increases the chance that your object will be hit (because NN can fully miss it). Again, this TTA can also be applied to your crops-approach.", "961549": "Thanks! Something to try next time." }, "source": "meta" }