Hello all,
\nFirstly, we wish to thank Kaggle and the host for this competition. I wish to thank the host specifically for being proactive and answering our questions well within the rules of the competition as well. Also, I wish to thank the forum contributors for being generous and sharing excellent discussion ideas and kernels for one and all.
\nLet's delve into the assignment solution and discuss the architecture and model choices below-
\nThis was a 5-target regression image problem to estimate the pasture biomass from training images covering Australian pastures across different seasons, regions, and species mixes, along with metadata including height, NDVI, state and species details. The test data consists of images only without meta-data provided. The host provided 357 training images and nearly 800 test-set images divided into 53% public leaderboard and 47% private leaderboard.
\nMetadata columns
\nProvided targets -
\nThese targets had the below mathematical relationships - \n$$ Total = Dead + GDM $$\n$$ GDM = Green + Clover $$\n$$ Total = Dead + Green + Clover $$
\nThis was one of the most tricky and time consuming components in this competition. I joined this competition early and most of my initial experiments in the first 4-5 weeks focused on an untuned EVA02 base model with the below CV schemes -
\n| CV scheme | \nResults - CV/ LB relation | \nAverage CV-LB gap | \n
|---|---|---|
| K-fold (5-fold) | \nLB and CV were not in sync at all | \n0.2-0.25 | \n
| Stratified K-fold (5-fold) on state | \nLB and CV were not in sync at all | \n0.2-0.25 | \n
| Stratified K-fold (5-fold) on state and species | \nLB and CV were not in sync at all | \n0.15-0.20 | \n
| Group-K-fold (5-fold) on month of sample date | \nLB and CV were not in sync at all | \n0.10- 0.175 | \n
| Stratified Group-K-fold (5-fold) on state, group by month of sample date | \nLB and CV were partially in sync | \n0.025-0.06 | \n
We ultimately chose to proceed with the Stratified Group-5-fold scheme for our later experiments. @kishanvavdara joined the team at this stage and we set out to train models for the final solution.
We focused on the below models thereafter -
\n| Model type | \nCV range | \nLB range | \nComments | \n
|---|---|---|---|
| EVA02 base | \n0.65-0.70 | \n0.55 - 0.60 | \nPoor LB but quick training pipeline - useful for prototyping | \n
| EVA02 large | \n0.68-0.73 | \n0.62 - 0.67 | \nPoor LB but provided diversity in the ensemble and useful for prototyping also | \n
| DINOV2 base | \n0.65-0.69 | \n0.62 - 0.66 | \nGood start point for DINO architectures and quick training | \n
| DINOV2 large | \n0.675-0.72 | \n0.64 - 0.68 | \nProvided a good option to test the freeze-full train structure - quick training as well | \n
| DINOV3 large | \n0.725-0.765 | \n0.69 - 0.72 | \nProvided a good solution with moderate CV-LB relation and allowed quick prototyping for larger models | \n
| DINOV3 huge | \n0.76 - 0.7925 | \n0.71-0.74 | \nCynosure for the final solution | \n
We were well aware of the high usage of DINOV3 models across the top LB solutions. To engender diversity in our solution compared to others, we focused on the below-
| Image splits | \nComment | \n
|---|---|
| 1 | \nProvided no reasonable CV-LB relation and was excluded | \n
| 2 | \nProvided decent CV-LB relation but it provided room for improvement | \n
| 3 | \nProvided improvement over 2-split, image was split into 3 equal parts | \n
| 4 | \nProvided best CV-LB relation, image was split into 4 equal parts | \n
Sample code for this is as below-
H, W = image.shape[:2] # H=1000, W=2000\nquarter = W // 4 # quarter=500\n\npart1 = image[:, :quarter, :] # [0:500]\npart2 = image[:, quarter:2*quarter, :] # [500:1000]\npart3 = image[:, 2*quarter:3*quarter, :] # [1000:1500]\npart4 = image[:, 3*quarter:, :] # [1500:2000]\nAugmentation (training only)
\nResize: 1000×500 → 448×448
\nNormalization: ImageNet statistics
\nFinally, this was converted to PyTorch tensors for the model
\ndef forward(self, part1, part2, part3, part4):\n # Each part: [B, 3, 448, 448]\n\n # Extract features using shared backbone\n feat1 = self.backbone(part1) # [B, 1536]\n feat2 = self.backbone(part2) # [B, 1536]\n feat3 = self.backbone(part3) # [B, 1536]\n feat4 = self.backbone(part4) # [B, 1536]\n\n # Concatenate features\n combined = torch.cat([feat1, feat2, feat3, feat4], dim=1) \nKey Design Choices
\nShared Backbone Weights
\nLate Fusion
\nWe used a SwiGLU module that outperformed a traditional ReLU/GELU. Code for this is as below -
\nclass SwiGLU(nn.Module):\n def __init__(self, input_dim, hidden_dim, output_dim, dropout=0.1):\n super().__init__()\n self.gate_proj = nn.Linear(input_dim, hidden_dim)\n self.value_proj = nn.Linear(input_dim, hidden_dim)\n self.down_proj = nn.Linear(hidden_dim, output_dim)\n self.dropout = nn.Dropout(dropout)\n self.silu = nn.SiLU() # Swish activation\n\n def forward(self, x):\n gate = self.silu(self.gate_proj(x)) # Apply Swish\n value = self.value_proj(x) # Linear transform\n hidden = gate * value # Element-wise multiplication\n hidden = self.dropout(hidden)\n output = self.down_proj(hidden)\n return output\nUsage of swiglu was optional, we tried a conventional approach as well , with the details as below -
\nif use_swiglu:\n hidden_expanded = hidden * 2 # 6144 for hidden_ratio=0.5\n def make_head():\n return SwiGLU(\n input_dim=combined, # 6144\n hidden_dim=hidden_expanded, # 6144\n output_dim=1,\n dropout=dropout\n )\nelse:\n def make_head():\n return nn.Sequential(\n nn.Linear(combined, hidden),\n nn.ReLU(inplace=True),\n nn.Dropout(dropout),\n nn.Linear(hidden, 1),\n )\n\nself.head_green = make_head()\nself.head_clover = make_head() # (if needed)\nself.head_dead = make_head()\n
Swiglu was quite helpful and provided a boost in our CV-LB relation. Our best single models used swiglu.
We used soft-plus activation across our target heads as below -
\nself.softplus = nn.Softplus(beta=1.0)
green = self.softplus(self.head_green(combined))\nclover = self.softplus(self.head_clover(combined))\ndead = self.softplus(self.head_dead(combined))\n$$ Softplus(x, β) = log(1 + exp(βx)) / β $$
\nWe concatenate the outputs from the heads as below -
\noutputs = torch.cat([green, clover, dead], dim=1) # [B, 3] or [B, 5]\nreturn outputs\nOne such case using a simple 3-2 targets prediction / derivation is as below-
\ndef calculate_derived_targets(preds_3, states=None):\n pred_green = preds_3[:, 0]\n pred_clover = preds_3[:, 1]\n pred_dead = preds_3[:, 2]\n\n # Apply ReLU to ensure non-negative\n pred_green = np.maximum(pred_green, 0)\n pred_clover = np.maximum(pred_clover, 0)\n pred_dead = np.maximum(pred_dead, 0)\n\n # Derive aggregate targets\n pred_gdm = pred_green + pred_clover # GDM = Green Dry Matter\n pred_total = pred_gdm + pred_dead # Total = GDM + Dead\n\n # Assemble final 5 targets\n preds_5 = np.zeros((len(preds_3), 5), dtype=np.float32)\n preds_5[:, 0] = pred_green\n preds_5[:, 1] = pred_dead\n preds_5[:, 2] = pred_clover\n preds_5[:, 3] = pred_gdm\n preds_5[:, 4] = pred_total\n\n return preds_5\nFor the 5-target case, we don't derive anything and directly do - all_preds_5 = np.maximum(all_preds_5, 0)
We used a simple L1 smooth loss for our submission. We tried weighted R2 loss, L1 loss, MSE loss but smooth L1 loss was the best
\nclass WeightedSmoothL1Loss(nn.Module):\n def __init__(self, weights):\n super().__init__()\n self.weights = torch.tensor(weights, dtype=torch.float32)\n self.loss_fn = nn.SmoothL1Loss(reduction='none')\n\n def forward(self, y_pred, y_true):\n self.weights = self.weights.to(y_pred.device)\n loss = self.loss_fn(y_pred, y_true) # [B, num_targets]\n weighted_loss = (self.weights * loss).mean(dim=1) # [B]\n return weighted_loss.mean()\nif epoch < freeze_epochs:\n for param in model.backbone.parameters():\n param.requires_grad = False\nelse:\n # Freeze all backbone\n for param in model.backbone.parameters():\n param.requires_grad = False\n\n # Unfreeze last 6 layers\n total_layers = len(model.backbone.blocks)\n start_idx = total_layers - 6\n\n for i in range(start_idx, total_layers):\n for param in model.backbone.blocks[i].parameters():\n param.requires_grad = True\n\n # Unfreeze norm layer\n if hasattr(model.backbone, 'norm'):\n for param in model.backbone.norm.parameters():\n param.requires_grad = True\n\n # Reduce learning rate\n for param_group in optimizer.param_groups:\n param_group['lr'] = lr_stage2\nWe used the competition metric as below to evaluate the model OOF -
\ndef weighted_r2_score(y_true, y_pred, weights):\n weights = np.array(weights)\n y_true_flat = y_true.flatten()\n y_pred_flat = y_pred.flatten()\n weights_flat = np.tile(weights, len(y_true))\n\n ss_res = np.sum(weights_flat * (y_true_flat - y_pred_flat) ** 2)\n y_mean = np.sum(weights_flat * y_true_flat) / np.sum(weights_flat)\n ss_tot = np.sum(weights_flat * (y_true_flat - y_mean) ** 2)\n\n return 1 - ss_res / ss_tot\nWe used the below augmentations during inference -
\nOur ensemble strategy included the below individual model ideas -
\n| Model ID | \nModel idea | \nCV score | \nSingle LB score | \n
|---|---|---|---|
| DINOV7_17 | \n- 3 train targets, 2 derived targets - swiglu - 6-last layers unfrozen - Freeze till epoch 1 | \n0.792425 | \n0.74 | \n
| DINOV7_30 | \n- 3 train targets, 2 derived targets - 5-stream architecture - swiglu - 6-last layers unfrozen - Freeze till epoch 1 | \n0.78498377 | \n0.73 | \n
| DINOV7_34 | \n- 3 train targets, 2 derived targets - 4-stream architecture - swiglu - 7-last layers unfrozen - Dropped bottom 10% image due to higher risk of non-grass components in this region - Freeze till epoch 1 | \n0.78237804 | \n0.73 | \n
| DINOV7_36 | \n- 3 train targets, 2 derived targets - 4-stream architecture - swiglu - 6-last layers unfrozen - Sequential unfreezing after epoch 1 across 4 stages | \n0.78187030 | \n0.73 | \n
| DINOV7_17_5T | \n- 5 train targets - swiglu - 6-last layers unfrozen - Freeze till epoch 1 | \n0.788456 | \n0.74 | \n
| DINOV9_10 | \n- 5-target prediction and derivation/ estimation of dead biomass based on predicted height - swiglu - 6-last layers unfrozen - Freeze till epoch 1 - 4-stream architecture | \n0.79041560 | \n0.72 | \n
| EVA02V8_3 | \n- 3 train targets, 2 derived targets - swiglu - Freeze till epoch 1 | \n0.73975521 | \n0.66 | \n
We blended the models using -
4 of our single models outscored our chosen kernel. All of them were DINOv3 Huge backbones and froze the parameters till epoch-1.
\nThey are as below-
| Model idea | \nPrivate LB score | \n
|---|---|
| - 5 target predictions with residual connections | \n0.64 | \n
| - 3+2 target predictions with linear scheduler instead of cosine | \n0.63 (high end) | \n
| - 4 target predictions with dry/total auxiliary ratio with linear scheduler instead of cosine | \n0.63 (high end) | \n
| - 5 target predictions with estimation of dead biomass based on predicted height | \n0.63 (slightly above chosen kernel) | \n
We learn that good single models can also be of value in small data competitions - a model blend is not always necessary.
Thanks for reading the write-up! Hope we were able to articulate the process with some aplomb!! Let us know your thoughts and we will respond appropriately.
\nBest wishes for your personal and professional/ Kaggle journey ahead!
\nRegards,
\nRavi Ramakrishnan
\nKishan Vavdara
\nTony Li
Hello all,
\nFirstly, we wish to thank Kaggle and the host for this competition. I wish to thank the host specifically for being proactive and answering our questions well within the rules of the competition as well. Also, I wish to thank the forum contributors for being generous and sharing excellent discussion ideas and kernels for one and all.
\nLet's delve into the assignment solution and discuss the architecture and model choices below-
\nThis was a 5-target regression image problem to estimate the pasture biomass from training images covering Australian pastures across different seasons, regions, and species mixes, along with metadata including height, NDVI, state and species details. The test data consists of images only without meta-data provided. The host provided 357 training images and nearly 800 test-set images divided into 53% public leaderboard and 47% private leaderboard.
\nMetadata columns
\nProvided targets -
\nThese targets had the below mathematical relationships - \n$$ Total = Dead + GDM $$\n$$ GDM = Green + Clover $$\n$$ Total = Dead + Green + Clover $$
\nThis was one of the most tricky and time consuming components in this competition. I joined this competition early and most of my initial experiments in the first 4-5 weeks focused on an untuned EVA02 base model with the below CV schemes -
\n| CV scheme | \nResults - CV/ LB relation | \nAverage CV-LB gap | \n
|---|---|---|
| K-fold (5-fold) | \nLB and CV were not in sync at all | \n0.2-0.25 | \n
| Stratified K-fold (5-fold) on state | \nLB and CV were not in sync at all | \n0.2-0.25 | \n
| Stratified K-fold (5-fold) on state and species | \nLB and CV were not in sync at all | \n0.15-0.20 | \n
| Group-K-fold (5-fold) on month of sample date | \nLB and CV were not in sync at all | \n0.10- 0.175 | \n
| Stratified Group-K-fold (5-fold) on state, group by month of sample date | \nLB and CV were partially in sync | \n0.025-0.06 | \n
We ultimately chose to proceed with the Stratified Group-5-fold scheme for our later experiments. @kishanvavdara joined the team at this stage and we set out to train models for the final solution.
We focused on the below models thereafter -
\n| Model type | \nCV range | \nLB range | \nComments | \n
|---|---|---|---|
| EVA02 base | \n0.65-0.70 | \n0.55 - 0.60 | \nPoor LB but quick training pipeline - useful for prototyping | \n
| EVA02 large | \n0.68-0.73 | \n0.62 - 0.67 | \nPoor LB but provided diversity in the ensemble and useful for prototyping also | \n
| DINOV2 base | \n0.65-0.69 | \n0.62 - 0.66 | \nGood start point for DINO architectures and quick training | \n
| DINOV2 large | \n0.675-0.72 | \n0.64 - 0.68 | \nProvided a good option to test the freeze-full train structure - quick training as well | \n
| DINOV3 large | \n0.725-0.765 | \n0.69 - 0.72 | \nProvided a good solution with moderate CV-LB relation and allowed quick prototyping for larger models | \n
| DINOV3 huge | \n0.76 - 0.7925 | \n0.71-0.74 | \nCynosure for the final solution | \n
We were well aware of the high usage of DINOV3 models across the top LB solutions. To engender diversity in our solution compared to others, we focused on the below-
| Image splits | \nComment | \n
|---|---|
| 1 | \nProvided no reasonable CV-LB relation and was excluded | \n
| 2 | \nProvided decent CV-LB relation but it provided room for improvement | \n
| 3 | \nProvided improvement over 2-split, image was split into 3 equal parts | \n
| 4 | \nProvided best CV-LB relation, image was split into 4 equal parts | \n
Sample code for this is as below-
H, W = image.shape[:2] # H=1000, W=2000\nquarter = W // 4 # quarter=500\n\npart1 = image[:, :quarter, :] # [0:500]\npart2 = image[:, quarter:2*quarter, :] # [500:1000]\npart3 = image[:, 2*quarter:3*quarter, :] # [1000:1500]\npart4 = image[:, 3*quarter:, :] # [1500:2000]\nAugmentation (training only)
\nResize: 1000×500 → 448×448
\nNormalization: ImageNet statistics
\nFinally, this was converted to PyTorch tensors for the model
\ndef forward(self, part1, part2, part3, part4):\n # Each part: [B, 3, 448, 448]\n\n # Extract features using shared backbone\n feat1 = self.backbone(part1) # [B, 1536]\n feat2 = self.backbone(part2) # [B, 1536]\n feat3 = self.backbone(part3) # [B, 1536]\n feat4 = self.backbone(part4) # [B, 1536]\n\n # Concatenate features\n combined = torch.cat([feat1, feat2, feat3, feat4], dim=1) \nKey Design Choices
\nShared Backbone Weights
\nLate Fusion
\nWe used a SwiGLU module that outperformed a traditional ReLU/GELU. Code for this is as below -
\nclass SwiGLU(nn.Module):\n def __init__(self, input_dim, hidden_dim, output_dim, dropout=0.1):\n super().__init__()\n self.gate_proj = nn.Linear(input_dim, hidden_dim)\n self.value_proj = nn.Linear(input_dim, hidden_dim)\n self.down_proj = nn.Linear(hidden_dim, output_dim)\n self.dropout = nn.Dropout(dropout)\n self.silu = nn.SiLU() # Swish activation\n\n def forward(self, x):\n gate = self.silu(self.gate_proj(x)) # Apply Swish\n value = self.value_proj(x) # Linear transform\n hidden = gate * value # Element-wise multiplication\n hidden = self.dropout(hidden)\n output = self.down_proj(hidden)\n return output\nUsage of swiglu was optional, we tried a conventional approach as well , with the details as below -
\nif use_swiglu:\n hidden_expanded = hidden * 2 # 6144 for hidden_ratio=0.5\n def make_head():\n return SwiGLU(\n input_dim=combined, # 6144\n hidden_dim=hidden_expanded, # 6144\n output_dim=1,\n dropout=dropout\n )\nelse:\n def make_head():\n return nn.Sequential(\n nn.Linear(combined, hidden),\n nn.ReLU(inplace=True),\n nn.Dropout(dropout),\n nn.Linear(hidden, 1),\n )\n\nself.head_green = make_head()\nself.head_clover = make_head() # (if needed)\nself.head_dead = make_head()\n
Swiglu was quite helpful and provided a boost in our CV-LB relation. Our best single models used swiglu.
We used soft-plus activation across our target heads as below -
\nself.softplus = nn.Softplus(beta=1.0)
green = self.softplus(self.head_green(combined))\nclover = self.softplus(self.head_clover(combined))\ndead = self.softplus(self.head_dead(combined))\n$$ Softplus(x, β) = log(1 + exp(βx)) / β $$
\nWe concatenate the outputs from the heads as below -
\noutputs = torch.cat([green, clover, dead], dim=1) # [B, 3] or [B, 5]\nreturn outputs\nOne such case using a simple 3-2 targets prediction / derivation is as below-
\ndef calculate_derived_targets(preds_3, states=None):\n pred_green = preds_3[:, 0]\n pred_clover = preds_3[:, 1]\n pred_dead = preds_3[:, 2]\n\n # Apply ReLU to ensure non-negative\n pred_green = np.maximum(pred_green, 0)\n pred_clover = np.maximum(pred_clover, 0)\n pred_dead = np.maximum(pred_dead, 0)\n\n # Derive aggregate targets\n pred_gdm = pred_green + pred_clover # GDM = Green Dry Matter\n pred_total = pred_gdm + pred_dead # Total = GDM + Dead\n\n # Assemble final 5 targets\n preds_5 = np.zeros((len(preds_3), 5), dtype=np.float32)\n preds_5[:, 0] = pred_green\n preds_5[:, 1] = pred_dead\n preds_5[:, 2] = pred_clover\n preds_5[:, 3] = pred_gdm\n preds_5[:, 4] = pred_total\n\n return preds_5\nFor the 5-target case, we don't derive anything and directly do - all_preds_5 = np.maximum(all_preds_5, 0)
We used a simple L1 smooth loss for our submission. We tried weighted R2 loss, L1 loss, MSE loss but smooth L1 loss was the best
\nclass WeightedSmoothL1Loss(nn.Module):\n def __init__(self, weights):\n super().__init__()\n self.weights = torch.tensor(weights, dtype=torch.float32)\n self.loss_fn = nn.SmoothL1Loss(reduction='none')\n\n def forward(self, y_pred, y_true):\n self.weights = self.weights.to(y_pred.device)\n loss = self.loss_fn(y_pred, y_true) # [B, num_targets]\n weighted_loss = (self.weights * loss).mean(dim=1) # [B]\n return weighted_loss.mean()\nif epoch < freeze_epochs:\n for param in model.backbone.parameters():\n param.requires_grad = False\nelse:\n # Freeze all backbone\n for param in model.backbone.parameters():\n param.requires_grad = False\n\n # Unfreeze last 6 layers\n total_layers = len(model.backbone.blocks)\n start_idx = total_layers - 6\n\n for i in range(start_idx, total_layers):\n for param in model.backbone.blocks[i].parameters():\n param.requires_grad = True\n\n # Unfreeze norm layer\n if hasattr(model.backbone, 'norm'):\n for param in model.backbone.norm.parameters():\n param.requires_grad = True\n\n # Reduce learning rate\n for param_group in optimizer.param_groups:\n param_group['lr'] = lr_stage2\nWe used the competition metric as below to evaluate the model OOF -
\ndef weighted_r2_score(y_true, y_pred, weights):\n weights = np.array(weights)\n y_true_flat = y_true.flatten()\n y_pred_flat = y_pred.flatten()\n weights_flat = np.tile(weights, len(y_true))\n\n ss_res = np.sum(weights_flat * (y_true_flat - y_pred_flat) ** 2)\n y_mean = np.sum(weights_flat * y_true_flat) / np.sum(weights_flat)\n ss_tot = np.sum(weights_flat * (y_true_flat - y_mean) ** 2)\n\n return 1 - ss_res / ss_tot\nWe used the below augmentations during inference -
\nOur ensemble strategy included the below individual model ideas -
\n| Model ID | \nModel idea | \nCV score | \nSingle LB score | \n
|---|---|---|---|
| DINOV7_17 | \n- 3 train targets, 2 derived targets - swiglu - 6-last layers unfrozen - Freeze till epoch 1 | \n0.792425 | \n0.74 | \n
| DINOV7_30 | \n- 3 train targets, 2 derived targets - 5-stream architecture - swiglu - 6-last layers unfrozen - Freeze till epoch 1 | \n0.78498377 | \n0.73 | \n
| DINOV7_34 | \n- 3 train targets, 2 derived targets - 4-stream architecture - swiglu - 7-last layers unfrozen - Dropped bottom 10% image due to higher risk of non-grass components in this region - Freeze till epoch 1 | \n0.78237804 | \n0.73 | \n
| DINOV7_36 | \n- 3 train targets, 2 derived targets - 4-stream architecture - swiglu - 6-last layers unfrozen - Sequential unfreezing after epoch 1 across 4 stages | \n0.78187030 | \n0.73 | \n
| DINOV7_17_5T | \n- 5 train targets - swiglu - 6-last layers unfrozen - Freeze till epoch 1 | \n0.788456 | \n0.74 | \n
| DINOV9_10 | \n- 5-target prediction and derivation/ estimation of dead biomass based on predicted height - swiglu - 6-last layers unfrozen - Freeze till epoch 1 - 4-stream architecture | \n0.79041560 | \n0.72 | \n
| EVA02V8_3 | \n- 3 train targets, 2 derived targets - swiglu - Freeze till epoch 1 | \n0.73975521 | \n0.66 | \n
We blended the models using -
4 of our single models outscored our chosen kernel. All of them were DINOv3 Huge backbones and froze the parameters till epoch-1.
\nThey are as below-
| Model idea | \nPrivate LB score | \n
|---|---|
| - 5 target predictions with residual connections | \n0.64 | \n
| - 3+2 target predictions with linear scheduler instead of cosine | \n0.63 (high end) | \n
| - 4 target predictions with dry/total auxiliary ratio with linear scheduler instead of cosine | \n0.63 (high end) | \n
| - 5 target predictions with estimation of dead biomass based on predicted height | \n0.63 (slightly above chosen kernel) | \n
We learn that good single models can also be of value in small data competitions - a model blend is not always necessary.
Thanks for reading the write-up! Hope we were able to articulate the process with some aplomb!! Let us know your thoughts and we will respond appropriately.
\nBest wishes for your personal and professional/ Kaggle journey ahead!
\nRegards,
\nRavi Ramakrishnan
\nKishan Vavdara
\nTony Li