Our solution is a weighted ensemble of three model variants, all built on the DINOv3 ViT-Huge backbone with LocalMamba fusion blocks. The key thing that helped was addressing the challenging Dry_Dead_g target through ratio-based prediction and multi-head architecture variations.
All models share the following core architecture:
\nInput: Image (2000×1000) → Split into Left (1000×1000) + Right (1000×1000)\n ↓\nDINOv3 ViT-Huge Backbone (vit_huge_plus_patch16_dinov3.lvd1689m)\n - Pretrained weights from timm\n - Gradient checkpointing enabled for memory efficiency\n - global_pool='' to retain patch tokens\n ↓\nFeature Concatenation: [Left Tokens, Right Tokens] → (B, 2N, D)\n ↓\nLocalMamba Fusion (2 blocks)\n - LayerNorm → Gated Linear → DepthwiseConv1d → Projection\n - Residual connections\n ↓\nAdaptive Average Pooling → (B, D)\n ↓\nMulti-Head Regression with Softplus activation (non-negative outputs)\nKey Innovation: Predict Dry_Dead_g as a ratio of Dry_Total_g
| Head | \nOutput | \nActivation | \n
|---|---|---|
head_green | \nDry_Green_g | \nSoftplus | \n
head_clover | \nDry_Clover_g | \nSoftplus | \n
head_total | \nDry_Total_g | \nSoftplus | \n
head_dead_ratio | \nDead/Total ratio | \nSigmoid [0,1] | \n
| Derivation: | \n\n | \n |
Dry_Dead_g = head_dead_ratio × Dry_Total_gGDM_g = Dry_Green_g + Dry_Clover_g\nSpecial Handling:Architecture: Direct prediction with derived targets
\n| Head | \nPredicted | \nDerived | \n
|---|---|---|
head_green | \nDry_Green_g | \n- | \n
head_dead | \nDry_Dead_g | \n- | \n
head_clover | \nDry_Clover_g | \n- | \n
| - | \nGDM_g | \nGreen + Clover | \n
| - | \nDry_Total_g | \nGDM + Dead | \n
| Training Config: | \n\n | \n |
Key Difference: Predict Total and GDM directly, derive Dead
\n| Head | \nPredicted | \nDerived | \n
|---|---|---|
head_green | \nDry_Green_g | \n- | \n
head_clover | \nDry_Clover_g | \n- | \n
head_gdm | \nGDM_g | \n- | \n
head_total | \nDry_Total_g | \n- | \n
| - | \nDry_Dead_g | \nTotal - GDM | \n
| Training Config: | \n\n | \n |
Image Split: Panoramic images (2000×1000) split into Left half (0:mid) and Right half (mid:end), processed separately through the backbone.
\nThe CV Strategy we used was suggested by someone in the discussion. I am not able to find that thread, but thanks to him.\nwe used StratifiedGroupKFold (4 folds) with a carefully designed stratification and grouping scheme:\nStratification Label: Based on presence/absence of challenging targets
\nstratify_col = f\"{has_clover}_{has_dead}\"\n# Ensures each fold has balanced representation of:\n# - Samples WITH Clover (Dry_Clover_g > 0)\n# - Samples WITH Dead (Dry_Dead_g > 0)\nGroup Key: Prevents data leakage by grouping samples from same collection session
\ngroup_key = f\"{day}/{month}_{state}\"\n# Examples: \"15/3_NSW\", \"22/6_WA\", \"8/11_Vic\"\n# Ensures samples collected on the same date in the same state stay together\nWhy This Matters:
\nDry_Clover_g is often zero (sparse), and Dry_Dead_g has high variance. Stratifying ensures each fold sees similar distributions of these challenging targets.sgkf = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=42)\nfor fold, (_, val_idx) in enumerate(\n sgkf.split(df, df['stratify_col'], groups=df['group_key'])\n):\n df.loc[val_idx, 'fold'] = fold\nA.Resize(IMG_SIZE, IMG_SIZE)\nA.HorizontalFlip(p=0.5)\nA.VerticalFlip(p=0.5)\nA.RandomRotate90(p=0.5)\nA.Rotate(limit=(-15, 15), p=0.3, border_mode=REFLECT)\nA.ColorJitter(brightness=0.15, contrast=0.15, saturation=0.15, hue=0.05, p=0.5)\nA.GaussNoise(var_limit=(5.0, 20.0), p=0.2)\nA.Normalize(ImageNet stats)\n| Parameter | \nValue | \n
|---|---|
| Batch Size | \n2 | \n
| Gradient Accumulation | \n8 (effective batch = 16) | \n
| Epochs | \n100 (early stopping) | \n
| Freeze Epochs | \n10 | \n
| Warmup Epochs | \n3 | \n
| LR Backbone | \n1e-5 | \n
| LR Head | \n1e-4 | \n
| Weight Decay | \n1e-2 | \n
| EMA Decay | \n0.998 | \n
| Patience | \n15 | \n
| Gradient Clipping | \n1.0 | \n
| Mixed Precision | \nFP16 | \n
Weighted Huber Loss (β=5.0):
\nloss = Σ(weight_i × SmoothL1Loss(pred_i, target_i, beta=5.0))\nModel 1 Additional Losses:
\n4-way TTA during validation:
\nWeighted Average with Fold-Specific Weights:
\nweight = {\n fold_0: 1.25,\n fold_1: 0.75,\n fold_2: 1.25,\n fold_3: 0.75\n}\nensemble = Σ(weight × fold_predictions) / Σ(weights)\nif target_name == \"Dry_Clover_g\":\n prediction *= 0.8\nelif target_name == \"Dry_Dead_g\":\n if prediction > 20:\n prediction *= 1.1\n elif prediction < 10:\n prediction *= 0.9\nDry_Dead_g has the highest variance and lowest correlation with other features. Ratio-based prediction (Dead/Total) significantly improved stability.| Model / Ensemble | \nPublic Score | \nPrivate Score | \n
|---|---|---|
| Model 1: Ratio-Based Dead | \n0.76760 | \n0.65329 | \n
| Model 2: Standard Multi-Head | \n0.77157 | \n0.64714 | \n
| Model 3: 4-Heads (Total-First) | \n0.76685 | \n0.64803 | \n
| Ensemble (with fold weights + post-processing) | \n0.77142 | \n0.65179 | \n
| Ensemble (no fold weights, no post-processing) | \n0.76772 | \n0.64855 | \n
Our solution is a weighted ensemble of three model variants, all built on the DINOv3 ViT-Huge backbone with LocalMamba fusion blocks. The key thing that helped was addressing the challenging Dry_Dead_g target through ratio-based prediction and multi-head architecture variations.
All models share the following core architecture:
\nInput: Image (2000×1000) → Split into Left (1000×1000) + Right (1000×1000)\n ↓\nDINOv3 ViT-Huge Backbone (vit_huge_plus_patch16_dinov3.lvd1689m)\n - Pretrained weights from timm\n - Gradient checkpointing enabled for memory efficiency\n - global_pool='' to retain patch tokens\n ↓\nFeature Concatenation: [Left Tokens, Right Tokens] → (B, 2N, D)\n ↓\nLocalMamba Fusion (2 blocks)\n - LayerNorm → Gated Linear → DepthwiseConv1d → Projection\n - Residual connections\n ↓\nAdaptive Average Pooling → (B, D)\n ↓\nMulti-Head Regression with Softplus activation (non-negative outputs)\nKey Innovation: Predict Dry_Dead_g as a ratio of Dry_Total_g
| Head | \nOutput | \nActivation | \n
|---|---|---|
head_green | \nDry_Green_g | \nSoftplus | \n
head_clover | \nDry_Clover_g | \nSoftplus | \n
head_total | \nDry_Total_g | \nSoftplus | \n
head_dead_ratio | \nDead/Total ratio | \nSigmoid [0,1] | \n
| Derivation: | \n\n | \n |
Dry_Dead_g = head_dead_ratio × Dry_Total_gGDM_g = Dry_Green_g + Dry_Clover_g\nSpecial Handling:Architecture: Direct prediction with derived targets
\n| Head | \nPredicted | \nDerived | \n
|---|---|---|
head_green | \nDry_Green_g | \n- | \n
head_dead | \nDry_Dead_g | \n- | \n
head_clover | \nDry_Clover_g | \n- | \n
| - | \nGDM_g | \nGreen + Clover | \n
| - | \nDry_Total_g | \nGDM + Dead | \n
| Training Config: | \n\n | \n |
Key Difference: Predict Total and GDM directly, derive Dead
\n| Head | \nPredicted | \nDerived | \n
|---|---|---|
head_green | \nDry_Green_g | \n- | \n
head_clover | \nDry_Clover_g | \n- | \n
head_gdm | \nGDM_g | \n- | \n
head_total | \nDry_Total_g | \n- | \n
| - | \nDry_Dead_g | \nTotal - GDM | \n
| Training Config: | \n\n | \n |
Image Split: Panoramic images (2000×1000) split into Left half (0:mid) and Right half (mid:end), processed separately through the backbone.
\nThe CV Strategy we used was suggested by someone in the discussion. I am not able to find that thread, but thanks to him.\nwe used StratifiedGroupKFold (4 folds) with a carefully designed stratification and grouping scheme:\nStratification Label: Based on presence/absence of challenging targets
\nstratify_col = f\"{has_clover}_{has_dead}\"\n# Ensures each fold has balanced representation of:\n# - Samples WITH Clover (Dry_Clover_g > 0)\n# - Samples WITH Dead (Dry_Dead_g > 0)\nGroup Key: Prevents data leakage by grouping samples from same collection session
\ngroup_key = f\"{day}/{month}_{state}\"\n# Examples: \"15/3_NSW\", \"22/6_WA\", \"8/11_Vic\"\n# Ensures samples collected on the same date in the same state stay together\nWhy This Matters:
\nDry_Clover_g is often zero (sparse), and Dry_Dead_g has high variance. Stratifying ensures each fold sees similar distributions of these challenging targets.sgkf = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=42)\nfor fold, (_, val_idx) in enumerate(\n sgkf.split(df, df['stratify_col'], groups=df['group_key'])\n):\n df.loc[val_idx, 'fold'] = fold\nA.Resize(IMG_SIZE, IMG_SIZE)\nA.HorizontalFlip(p=0.5)\nA.VerticalFlip(p=0.5)\nA.RandomRotate90(p=0.5)\nA.Rotate(limit=(-15, 15), p=0.3, border_mode=REFLECT)\nA.ColorJitter(brightness=0.15, contrast=0.15, saturation=0.15, hue=0.05, p=0.5)\nA.GaussNoise(var_limit=(5.0, 20.0), p=0.2)\nA.Normalize(ImageNet stats)\n| Parameter | \nValue | \n
|---|---|
| Batch Size | \n2 | \n
| Gradient Accumulation | \n8 (effective batch = 16) | \n
| Epochs | \n100 (early stopping) | \n
| Freeze Epochs | \n10 | \n
| Warmup Epochs | \n3 | \n
| LR Backbone | \n1e-5 | \n
| LR Head | \n1e-4 | \n
| Weight Decay | \n1e-2 | \n
| EMA Decay | \n0.998 | \n
| Patience | \n15 | \n
| Gradient Clipping | \n1.0 | \n
| Mixed Precision | \nFP16 | \n
Weighted Huber Loss (β=5.0):
\nloss = Σ(weight_i × SmoothL1Loss(pred_i, target_i, beta=5.0))\nModel 1 Additional Losses:
\n4-way TTA during validation:
\nWeighted Average with Fold-Specific Weights:
\nweight = {\n fold_0: 1.25,\n fold_1: 0.75,\n fold_2: 1.25,\n fold_3: 0.75\n}\nensemble = Σ(weight × fold_predictions) / Σ(weights)\nif target_name == \"Dry_Clover_g\":\n prediction *= 0.8\nelif target_name == \"Dry_Dead_g\":\n if prediction > 20:\n prediction *= 1.1\n elif prediction < 10:\n prediction *= 0.9\nDry_Dead_g has the highest variance and lowest correlation with other features. Ratio-based prediction (Dead/Total) significantly improved stability.| Model / Ensemble | \nPublic Score | \nPrivate Score | \n
|---|---|---|
| Model 1: Ratio-Based Dead | \n0.76760 | \n0.65329 | \n
| Model 2: Standard Multi-Head | \n0.77157 | \n0.64714 | \n
| Model 3: 4-Heads (Total-First) | \n0.76685 | \n0.64803 | \n
| Ensemble (with fold weights + post-processing) | \n0.77142 | \n0.65179 | \n
| Ensemble (no fold weights, no post-processing) | \n0.76772 | \n0.64855 | \n
Great writeup! This helps to understand very easily the approach u guys used
", "rawMarkdown": "Great writeup! This helps to understand very easily the approach u guys used", "votes": null } ], "comments": [ { "id": 3408487, "author_name": "arshianrax", "author_url": "", "post_date": "02/20/2026 18:13:54", "content": "Great writeup! This helps to understand very easily the approach u guys used
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3399432": "Our solution is a weighted ensemble of **three model variants**, all built on the DINOv3 ViT-Huge backbone with LocalMamba fusion blocks. The key thing that helped was addressing the challenging `Dry_Dead_g` target through ratio-based prediction and multi-head architecture variations.\n**Final Ensemble:** 3 model architectures × 4 folds.\n---\n\n## Model Architecture\n\n### Shared Components\nAll models share the following core architecture:\n```\nInput: Image (2000×1000) → Split into Left (1000×1000) + Right (1000×1000)\n ↓\nDINOv3 ViT-Huge Backbone (vit_huge_plus_patch16_dinov3.lvd1689m)\n - Pretrained weights from timm\n - Gradient checkpointing enabled for memory efficiency\n - global_pool='' to retain patch tokens\n ↓\nFeature Concatenation: [Left Tokens, Right Tokens] → (B, 2N, D)\n ↓\nLocalMamba Fusion (2 blocks)\n - LayerNorm → Gated Linear → DepthwiseConv1d → Projection\n - Residual connections\n ↓\nAdaptive Average Pooling → (B, D)\n ↓\nMulti-Head Regression with Softplus activation (non-negative outputs)\n```\n---\n### Model 1: Improved Dead Prediction (Ratio-Based)\n**Key Innovation:** Predict `Dry_Dead_g` as a ratio of `Dry_Total_g`\n| Head | Output | Activation |\n|------|--------|------------|\n| `head_green` | Dry_Green_g | Softplus |\n| `head_clover` | Dry_Clover_g | Softplus |\n| `head_total` | Dry_Total_g | Softplus |\n| `head_dead_ratio` | Dead/Total ratio | **Sigmoid** [0,1] |\n**Derivation:**\n- `Dry_Dead_g = head_dead_ratio × Dry_Total_g`\n- `GDM_g = Dry_Green_g + Dry_Clover_g`\n**Special Handling:**\n- State-specific Dead adjustment via learnable bias (WA state has Dead≈0)\n- Multi-task loss with auxiliary ratio loss\n- Scale-invariant log-space loss for Dead\n**Training Config:**\n- Image Size: 1024×1024\n- Loss Weights: [0.12, **0.25**, 0.12, 0.16, 0.35] (increased Dead weight)\n\n---\n\n### Model 2: Standard Multi-Head\n**Architecture:** Direct prediction with derived targets\n| Head | Predicted | Derived |\n|------|-----------|---------|\n| `head_green` | Dry_Green_g | - |\n| `head_dead` | Dry_Dead_g | - |\n| `head_clover` | Dry_Clover_g | - |\n| - | GDM_g | Green + Clover |\n| - | Dry_Total_g | GDM + Dead |\n**Training Config:**\n- Image Size: 1024×1024\n- Loss Weights: [0.15, 0.15, 0.15, 0.2, 0.35]\n\n---\n\n### Model 3: Total-First Prediction\n**Key Difference:** Predict Total and GDM directly, derive Dead\n| Head | Predicted | Derived |\n|------|-----------|---------|\n| `head_green` | Dry_Green_g | - |\n| `head_clover` | Dry_Clover_g | - |\n| `head_gdm` | GDM_g | - |\n| `head_total` | Dry_Total_g | - |\n| - | Dry_Dead_g | **Total - GDM** |\n**Training Config:**\n- Image Size: 896×896\n- Loss Weights: [0.15, 0.15, 0.15, 0.2, 0.35]\n\n---\n\n## Training Strategy\n### Data Preprocessing\n**Image Split:** Panoramic images (2000×1000) split into Left half (0:mid) and Right half (mid:end), processed separately through the backbone.\n### Cross-Validation Strategy\nThe CV Strategy we used was suggested by someone in the discussion. I am not able to find that thread, but thanks to him.\nwe used **StratifiedGroupKFold** (4 folds) with a carefully designed stratification and grouping scheme:\n**Stratification Label:** Based on presence/absence of challenging targets\n```python\nstratify_col = f\"{has_clover}_{has_dead}\"\n# Ensures each fold has balanced representation of:\n# - Samples WITH Clover (Dry_Clover_g > 0)\n# - Samples WITH Dead (Dry_Dead_g > 0)\n```\n**Group Key:** Prevents data leakage by grouping samples from same collection session\n```python\ngroup_key = f\"{day}/{month}_{state}\"\n# Examples: \"15/3_NSW\", \"22/6_WA\", \"8/11_Vic\"\n# Ensures samples collected on the same date in the same state stay together\n```\n**Why This Matters:**\n1. **Stratification by Clover/Dead:** `Dry_Clover_g` is often zero (sparse), and `Dry_Dead_g` has high variance. Stratifying ensures each fold sees similar distributions of these challenging targets.\n2. **Grouping by Date+State:** Samples collected on the same day in the same location likely have similar conditions (weather, pasture state). Keeping them in the same fold prevents the model from \"memorizing\" collection-day patterns.\n```python\nsgkf = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=42)\nfor fold, (_, val_idx) in enumerate(\n sgkf.split(df, df['stratify_col'], groups=df['group_key'])\n):\n df.loc[val_idx, 'fold'] = fold\n```\n\n### Augmentations (Training)\n```python\nA.Resize(IMG_SIZE, IMG_SIZE)\nA.HorizontalFlip(p=0.5)\nA.VerticalFlip(p=0.5)\nA.RandomRotate90(p=0.5)\nA.Rotate(limit=(-15, 15), p=0.3, border_mode=REFLECT)\nA.ColorJitter(brightness=0.15, contrast=0.15, saturation=0.15, hue=0.05, p=0.5)\nA.GaussNoise(var_limit=(5.0, 20.0), p=0.2)\nA.Normalize(ImageNet stats)\n```\n\n### Hyperparameters\n| Parameter | Value |\n|-----------|-------|\n| Batch Size | 2 |\n| Gradient Accumulation | 8 (effective batch = 16) |\n| Epochs | 100 (early stopping) |\n| Freeze Epochs | 10 |\n| Warmup Epochs | 3 |\n| LR Backbone | 1e-5 |\n| LR Head | 1e-4 |\n| Weight Decay | 1e-2 |\n| EMA Decay | 0.998 |\n| Patience | 15 |\n| Gradient Clipping | 1.0 |\n| Mixed Precision | FP16 |\n### Loss Function\n**Weighted Huber Loss (β=5.0):**\n```python\nloss = Σ(weight_i × SmoothL1Loss(pred_i, target_i, beta=5.0))\n```\n**Model 1 Additional Losses:**\n- Auxiliary MSE loss on Dead/Total ratio (weight=0.15)\n- Scale-invariant log-space loss for Dead (weight=0.05)\n\n---\n\n## Inference & Ensemble\n### Validation TTA\n4-way TTA during validation:\n1. Original\n2. Horizontal Flip\n3. Vertical Flip\n4. Horizontal + Vertical Flip\n### Ensemble Strategy\n**Weighted Average with Fold-Specific Weights:**\n```python\nweight = {\n fold_0: 1.25,\n fold_1: 0.75,\n fold_2: 1.25,\n fold_3: 0.75\n}\nensemble = Σ(weight × fold_predictions) / Σ(weights)\n```\n### Post-Processing\n```python\nif target_name == \"Dry_Clover_g\":\n prediction *= 0.8\nelif target_name == \"Dry_Dead_g\":\n if prediction > 20:\n prediction *= 1.1\n elif prediction < 10:\n prediction *= 0.9\n```\n\n---\n\n## Key Insights\n1. **Dead Prediction is Hard:** `Dry_Dead_g` has the highest variance and lowest correlation with other features. Ratio-based prediction (Dead/Total) significantly improved stability.\n2. **State Matters for Dead:** WA samples consistently have Dead=0. Learnable state-specific bias helped the model learn this pattern.\n3. **Resolution Trade-off:** 1024×1024 performed slightly better but 896×896 was faster; combining both helped. In fact, for the models trained on 896, inferring them on 1024 performed better than inferring them on 896.\n\n---\n\n## Results\n| Model / Ensemble | Public Score | Private Score |\n|------------------|--------------|---------------|\n| Model 1: Ratio-Based Dead | 0.76760 | 0.65329 |\n| Model 2: Standard Multi-Head | **0.77157** | 0.64714 |\n| Model 3: 4-Heads (Total-First) | 0.76685 | 0.64803 |\n| Ensemble (with fold weights + post-processing) | 0.77142 | **0.65179** |\n| Ensemble (no fold weights, no post-processing) | 0.76772 | 0.64855 |\n\n---\n\n## What Didn't Work\n- Metadata conditioning (State/Season embeddings) did not work\n- We did not use TTA for the submission of ensemble as it was exceeding 9 hrs.\n- Higher image resolutions (>1024) \n- More Mamba blocks (>2) \n\n---", "3408487": "Great writeup! This helps to understand very easily the approach u guys used" }, "source": "meta" }