# (2nd) Weakly supervised semantic segmentation + Synthetic data

Thanks to the organizers for organizing this competition. It’s been a long time since I last saw a traditional computer vision competition. When this competition first started, I wondered how to tackle the image regression problem besides the traditional techniques I had used before.
1. Cross-validation strategy
In this competition, I use a very simple validation strategy: stratified by State and choosing a seed so that the mean and standard deviation of the labels are similar across all 5 folds. There are a few main reasons: first, the dataset is too small, so if we split it in an extreme way, such as grouping by date or by state, the label distribution would become heavily distorted. Second, the host has confirmed that there are no new species in the public and private test sets. Some states have a certain species while others don’t. Grouping by date also risks missing some species or not having all four seasons represented in the training set. As I understand it, each date group corresponds to a trip to a particular farm to collect samples, and it’s possible that some clover species exist only in certain farms.
Basically, my split is almost certainly suffering from data leakage, and I accept that. To mitigate this, I use the CV–LB correlation and the LB score to evaluate the model.
2. Synthetic dataset
Right from the start of the competition, I focused on methods for generating additional training data. However, almost all of my initial experiments failed: the synthetic data, when used to train the model, consistently caused the Public Leaderboard score to drop.
One of the main challenges in using synthetic datasets for image regression is the ground truth issue; when an image is modified, the biomass value of each target changes accordingly. To address this, I employed a pseudo-labeling strategy: I used a model trained on the original training set to generate labels for the synthetic dataset, and subsequently trained the final model on the combined dataset (original training set + synthetic dataset).
I used
[Qwen Image Edit](https://github.com/QwenLM/Qwen-Image)
to create two types of synthetic datasets:
The synthetic dataset was generated by simulating seasonal weather changes and adding artifacts such as gravel and rocks.
I removed all grass vegetation from the images, retaining only the bare soil, and set the target values for clover, dead, and green to 0. These images were then combined with the training dataset to help the model discriminate between biomass-containing regions and non-biomass regions.
![](assets/2e2d240c11d829a0dd178250.jpg)
3. Data augmentation
Version 1
train_transform = A.ReplayCompose([
        A.RandomResizedCrop(size=(1024, 1024), scale=(0.8, 1.0), interpolation=cv2.INTER_CUBIC, p=1),
        A.HorizontalFlip(p=0.5),
        A.VerticalFlip(p=0.5),
        A.RandomRotate90(p=0.5),
        A.Rotate(limit=(-5, 5), p=0.7),
        A.RandomGamma(gamma_limit=(80, 120), p=0.5),
        A.RandomBrightnessContrast(p=0.75),
        A.HueSaturationValue(hue_shift_limit=10, 
                    sat_shift_limit=30, 
                    val_shift_limit=20, p=0.5),
         A.OneOf([
               A.CLAHE(clip_limit=(1, 4), tile_grid_size=(8, 8), p=0.5),
               A.Equalize(p=0.5),
         ], p=0.25),
         A.OneOf([
               A.Sharpen(p=0.5),
               A.Emboss(p=0.5),
         ], p=0.25),
    ])
Version 2
First, I used these augmentations
train_transform = A.Compose([
        A.RandomResizedCrop(size=(self.train_imgsize, self.train_imgsize), scale=(0.8, 1.0), interpolation=cv2.INTER_CUBIC, p=1),
        A.HorizontalFlip(p=0.5),
        A.RandomRotate90(p=0.5),
        A.VerticalFlip(p=0.5),
   ])
After that, the augmented images are further processed with RandAugment.
Other augmentations
random_erase = timm.data.random_erasing.RandomErasing(probability=0.5)
I also used both Mixup and CutMix with α=0.4; both of these augmentations helped improve the CV and public leaderboard scores.
The most surprising thing here is that
RandomResizedCrop
and
RandomErasing
turned out to play a crucial role in improving both the CV and Public Leaderboard scores. At first, I thought these augmentations would change the image labels and therefore lower the score. But that wasn’t the case. Perhaps the small random crop range reflects the fact that, in reality, the cropped grass region is not always exactly 70×30, but fluctuates slightly around that size.
Image size
I used two types of image sizes.
type
width x height
1
2048 x 1024
2
1024 x 1024
4. Optimizing the objective function
Loss function
I used Smooth L1 loss. Attempts to use MSE, binary cross-entropy loss, or to apply a log transform to the target values were all unsuccessful.
criterion_green = torch.nn.SmoothL1Loss()
criterion_clover = torch.nn.SmoothL1Loss()
criterion_dead = torch.nn.SmoothL1Loss()
Optimizer
I used
[Schedule-Free Optimizers](https://github.com/facebookresearch/schedule_free)
with layer-wise learning rates
hyper parameters
value
lr
0.000017
batch size
2
weight decay
0.02
number of epochs
35
backbone_params = [p for p in model.model.parameters() if p.requires_grad]
backbone_ids = set(map(id, backbone_params))

# Identify other parameters
other_params = [p for p in model.parameters() if p.requires_grad and id(p) not in backbone_ids]

param_groups = []
if backbone_params:
    param_groups.append({
         'params': backbone_params,
         'lr': 1e-5,
         'weight_decay': configs['weight_decay'],
    })
if other_params:
    param_groups.append({
        'params': other_params,
        'lr': configs['lr'],
        'weight_decay': configs['weight_decay'],
    })
optimizer = schedulefree.AdamWScheduleFree(param_groups, warmup_steps=0)
The targets
My model predicts three targets:
clover
,
dead
, and
green
. Attempts using 5 targets
clover
,
dead
,
green
,
total
,
gdm
and using 3 targets
clover
,
total
,
gdm
both resulted in lower public scores.
Early stopping strategy
backbone
selected epoch
dinov3 vitlarge
24-26
dinov3 vithuge
33-34
5. Model
Backbone
The high-resolution dense features of DINOv3 are truly remarkable. In density estimation tasks, detailed features with high discriminative power are crucial. I experimented with various backbones, including modern ones like dinov2, XCiT, SigLIP, and PE spatial (Perception Encoder), but both CV and LB scores were significantly lower compared to DINOv3.
Although I wanted to ensure diversity, I could only select backbones from DINOv3 family, where ConvNeXt-Large performed significantly worse than ViT-Large.
Weakly supervised semantic segmentation
The idea is that instead of treating this merely as a standard image regression problem, I want to approach it as a density estimation problem. Therefore, I used DINOv3 to extract feature maps of size (N, C, H, W). Then, I tried attaching different heads to estimate the density map. The final results would be the target values for clover, dead, and green.
First attempt: Simple head Conv + ReLU
self.model = timm.create_model('vit_large_patch16_dinov3_qkvb.lvd1689m', pretrained=True, features_only=True)
        self.pre_pool_conv = nn.Conv2d(feature_channels, feature_channels, kernel_size=3, padding=1, bias=False)
        self.global_pool = nn.AdaptiveAvgPool2d((1, 1))
        self.reg_head_main = nn.Linear(feature_channels, 3)  # clover, dead, green
    def forward(self, x):
        ...
        features = self.model(x)
        last_feature = features[-1]
        last_feature = self.pre_pool_conv(last_feature)
        last_feature = F.relu(last_feature)
        pooled = self.global_pool(last_feature).flatten(1)
        target_values = self.reg_head_main(pooled)
Second attempt: DPT head
I wanted to design the model as a semantic segmentation model and found
[SegDino](https://github.com/script-Yang/SegDINO)
. By utilizing multiple feature maps and adopting the DPT head from SegDino, I designed the model to output a heatmap where each pixel corresponds to the target value of a 16x16 patch in the image.
Backbone
Extracts intermediate features at indices [4, 11, 17, 23].
DPT Head
Projects the 4 input feature maps to [96, 192, 384, 768] channels.
Refines them with 3x3 convolutions ("Scratch" layers).
Upsamples lower-resolution features and concatenates them.
Final 1x1 convolution reduces channels to 3 (for the 3 classes).
Dual-Path Strategy:
The input image is split into left and right halves.
Each half is processed independently through the backbone and head.
The results are globally pooled and then summed together.
![](assets/8ebb0fe035ac46732bcd1e04.png)
...
    self.model = timm.create_model('vit_large_patch16_dinov3_qkvb.lvd1689m', pretrained=True)
    self.feature_indices = [4, 11, 17, 23]
    self.head = DPTHead(3, self.model.embed_dim, 128, False, out_channels=[96, 192, 384, 768])

    self.IMAGENET_DEFAULT_MEAN = torch.tensor([0.485, 0.456, 0.406]).to(device_id)
    self.IMAGENET_DEFAULT_STD = torch.tensor([0.229, 0.224, 0.225]).to(device_id)

    self.global_pool = nn.AdaptiveAvgPool2d((1, 1))

    def forward(self, x, return_features=False):
        ...
        w = x.shape[3]
        x_l = x[:,:,:,:w//2]
        x_r = x[:,:,:,w//2:]
        outputs_l, features_l = self.model.forward_intermediates(x_l, indices=self.feature_indices)
        outputs_r, features_r = self.model.forward_intermediates(x_r, indices=self.feature_indices)
        features_l = self.head(features_l)
        features_r = self.head(features_r)

        main_logits_l = self.global_pool(features_l).flatten(1)
        main_logits_r = self.global_pool(features_r).flatten(1)
        main_logits = main_logits_l + main_logits_r
I visualized the heatmap for each target. White corresponds to a target value of 0, while darker colors indicate values closer to the maximum. The results look promising.
Heatmap of the clover target
![Heatmap of the clover target](assets/10cdb3304f8c3094f8c647ea.png)
Heatmap of the dead target
![Heatmap of the dead target](assets/2d27b50ce17bf2697174d8fb.png)
Heatmap of the green target
![Heatmap of the green target](assets/fab3519d5b7d65ad9f66aaaf.png)
6. Postprocessing
State-based scaling postprocess
I tried classifying the images by State, and surprisingly the accuracy was over 98%.
At the same time, I also plotted the distribution of the targets by State.
Label Distribution per Target per State
============================================================

Target: Dry_Clover_g
----------------------------------------
       count       mean        std  min       25%       50%        75%      max
State                                                                          
NSW     75.0   0.134641   1.166028  0.0  0.000000   0.00000   0.000000  10.0981
Tas    138.0   6.239966  11.614679  0.0  0.000000   1.08390   6.628400  71.7865
Vic    112.0   7.106464   9.571358  0.0  1.406525   4.05135   8.274350  67.8977
WA      32.0  22.087584  20.214792  0.0  4.302500  15.60500  38.200825  58.8800

Target: Dry_Dead_g
----------------------------------------
       count       mean        std  min       25%       50%        75%      max
State                                                                          
NSW     75.0  14.203379  15.625313  0.0  2.600000   8.36140  20.898100  83.8407
Tas    138.0  15.231128  12.431224  0.4  6.047525  11.07405  21.062300  58.7479
Vic    112.0  10.113879   8.856831  0.0  3.393400   7.22240  13.375525  38.8581
WA      32.0   0.000000   0.000000  0.0  0.000000   0.00000   0.000000   0.0000

Target: Dry_Green_g
----------------------------------------
       count       mean        std  min        25%       50%        75%       max
State                                                                            
NSW     75.0  56.559313  31.228115  7.6  30.224150  53.50000  72.829800  157.9836
Tas    138.0  15.327257  13.107406  0.0   3.953725  12.97535  22.857150   69.6107
Vic    112.0  25.449273  16.016427  3.0  13.882050  21.58675  33.037750   89.7395
WA      32.0   9.299916  18.789542  0.0   0.000000   0.35000   5.413475   72.1000
It’s easy to see that all samples from the state of WA have a dead value of 0. Therefore, I applied a post-processing step, setting the dead value of all samples with wa_prob > 0.75 to 0.
wa_prob = state_probs[:, 2]
dead[wa_prob > 0.75] = 0.0
In the final days of the competition, I was stuck at a score in the low 0.76x range. After reading
[@kevin1742064161](https://www.kaggle.com/kevin1742064161)
’s
[comment](https://www.kaggle.com/competitions/csiro-biomass/discussion/668666#3393901)
, I started to suspect there might be bias in the public dataset, so I tried optimizing a scale factor for each target and for each State. Note that
[my initial attempts](https://www.kaggle.com/competitions/csiro-biomass/discussion/665538#3384888)
to scale the prediction outputs had failed. Surprisingly, the PB score jumped straight from the 0.76822 to 0.77937. And even more surprisingly, this also held on the private leaderboard, where the private score of the ensemble model improved from 0.66153 to 0.67558.
# Public LB optimization: "WA":  {"clover": 1.03->1.0, "clover": 1.0->0.97, "clover": 0.97->0.90, "clover": 0.97->0.95,"green": 1.0->0.97, "dead": 1.0->0.97->0.90->0.80}
# Public LB optimization: "VIC":  {"clover": 1.0->0.97->0.90, "clover": 0.97->0.90}
# Public LB optimization: "TAS":  {"clover": 0.97->1.0, "green": 0.97->1.0}
# Public LB optimization: "NSW":  {"clover": 0.97->1.0, "green": 1.0->1.03}
STATE_TARGET_SCALE = {
    "NSW": {"clover": 1.0, "dead": 1.0, "green": 1.03},
    "Tas": {"clover": 1.0, "dead": 1.0, "green": 1},
    "Vic": {"clover": 0.85, "dead": 1, "green": 1},
    "WA":  {"clover": 0.80, "dead": 0.80, "green": 0.97},
}
Clipping method
Use np.clip to restrict the values of the clover, dead, and green variables to specific ranges. The maximum value computed from the training set.
clover = np.clip(clover, 0, 71.7865)
dead = np.clip(dead, 0, 83.8407)
green = np.clip(green, 0, 157.9836)
7. Inference
Seperate target models
During training, I noticed that the green target usually converged very early and then started to overfit. The RMSE of the green target typically decreased until around epochs 11–15, and then kept increasing. The remaining targets, clover and dead, usually converged later, around epochs 24–30.
Therefore, I used the model at epoch 15 to predict the green target, and the model at epoch 26 to predict the other two targets, clover and dead.
Version
Public leaderboard
Private leaderboard
With the seperate target models
0.76731
0.65615
Without the seperate target models
0.76128
0.64744
List of models used for ensembling
Model
Image size
Backbone
Public leaderboard
Private leaderboard
conv_relu + 1 tiles
2048x1024
dinov3 vit large
0.75401
0.63349
dpt_head + 1 tiles
2048x1024
dinov3 vit large
0.75844
0.62890
conv_relu + 1 tiles
1280x1280
dinov3 vit huge
0.75390
0.65361
dpt_head + 2 tiles
2048x1024
dinov3 vit large
0.76379
0.64781
dpt_head + 8 tiles
2048x1024
dinov3 vit large
0.75036
0.62650
dpt_head + 2 tiles
2048x1024
dinov3 vit huge
0.75994
0.66569
conv_relu + 2 tiles + separate target models
2048x1024
dinov3 vit large
0.76731
0.65615
conv_relu + 2 tiles
2048x1024
dinov3 vit huge
0.75718
0.66731
Looking at the table above, we can see that models with a ViT-Large backbone usually have higher public scores, but the models with a ViT-Huge backbone are the ones that achieve higher private scores.
Selection of submissions
Since I wasn’t sure whether the post-processing technique would also work on the private dataset, I submitted two versions: an ensemble model with the post-processing technique, and an ensemble model without the post-processing technique.
Version
Public leaderboard
Private leaderboard
With the post-processing technique
0.77839
0.67558
Without the post-processing technique
0.76822
0.66153
The interesting thing here is that the ensemble models actually had a lower private score than a single model: 0.66153 < 0.67156 (without the post-processing technique). I’ve attached the notebook link for the single model with a private score of 0.67156.
I really had no way of knowing exactly which single model would perform best on the private dataset, so even though the ensemble’s private score wasn’t perfect, it helped me avoid a shake-up. If I had known exactly which single model had the 0.67156 private score, then with the ensemble method + seperate target models and the post-processing technique I could probably have increased the private score to 0.69 😅
Runtime optimization
I used FP16 quantization and ran two processes on two T4 GPUs. As a result, I was able to reduce the inference time from about
2 hours
to
27 minutes
.
8. Trying to find a different approach but without success
Decoding-based Regression:
[https://www.kaggle.com/competitions/csiro-biomass/discussion/668735](https://www.kaggle.com/competitions/csiro-biomass/discussion/668735)
I attempted to use
[SAM 2](https://github.com/facebookresearch/sam2)
for segmentation. However, since the clover and grass are extremely small and densely intertwined, effective segmentation using SAM 2 proved to be virtually impossible.
I tried using
[DiffuseMix](https://github.com/khawar-islam/diffuseMix)
, but as mentioned earlier, the problem with synthetic data is that when the image changes, the label changes along with it.
Used Excess Green index for segmentation, then used the mask as the 4th channel. Public score did not improve.
Since the biomass estimation problem relies on plant density and height, Height_Ave_cm was used as an auxiliary loss. I also tried using
[Depth Anything v3](https://github.com/ByteDance-Seed/Depth-Anything-3)
+ DINOv3 ViT-Large in a dual-stream model. All methods decreased the public score.
Used Pre_GSHH_NDVI, State, Species, and Height_Ave_cm as auxiliary losses. All resulted in no change or a decrease in the Public score.
Pretrained on an external dataset, then fine-tuned on the training dataset; the public score decreased relatively significantly.
Test time training adaptation: abandoned because it was too time-consuming during inference.
Hflip TTA has no effect or negligible impact with 6-4 weights.