{
  "id": 671255,
  "title": "76th Private/11th Public - CSIRO Biomass Prediction Pipeline",
  "url": "/competitions/csiro-biomass/discussion/671255",
  "author_name": "stefanoclss",
  "post_date": "2026-01-31T23:27:50.253000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>CSIRO Biomass Prediction Pipeline</h1>\n<h2>Overview</h2>\n<p>This solution achieves <strong>0.790 weighted R²</strong> on the CSIRO Biomass prediction task by ensembling three complementary deep learning models with optimized per-target weights and physical constraints.</p>\n<h2>Architecture Overview</h2>\n<p>The pipeline consists of three diverse models combined through an optimized ensemble:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6213473%2Fa86e71395c4869461f909d157cbe390b%2FScreenshot%202026-01-31%20212753.png?generation=1769902056234242&amp;alt=media\" alt=\"\"></p>\n<h3>Model 1: S5-DINO (ContextAwareMIL)</h3>\n<ul>\n<li><strong>Backbone</strong>: DINOv3 ViT-Huge (1,280-dim embeddings)</li>\n<li><strong>Multi-Scale Crops</strong>: 3×3 grid (9 crops) + 2×2 grid (4 crops) = <strong>13 crops</strong></li>\n<li><strong>Physics Features</strong>: Green Fraction (HSV-based) + Excess Green Index (ExG)</li>\n<li><strong>Sequence Model</strong>: 4-layer S5 (State Space) blocks with residual connections</li>\n<li><strong>Context Aggregation</strong>: Mean + Std pooling across crops</li>\n<li><strong>Output</strong>: Softplus activation → Sum across crops (MIL aggregation)</li>\n</ul>\n<p><strong>Feature dimensions</strong>: 13 crops × (1,280 DINO + 2 physics) = <strong>16,666 features</strong></p>\n<h3>Model 2: S5-SigLIP (ContextAwareMIL + Semantic)</h3>\n<ul>\n<li><strong>Backbone</strong>: DINOv3 ViT-Huge + SigLIP SO400M</li>\n<li><strong>Multi-Scale Crops</strong>: Same 13-crop strategy</li>\n<li><strong>Semantic Features</strong>: 8 concept groups via CLIP-style similarity:<ul>\n<li><code>bare</code>, <code>sparse</code>, <code>medium</code>, <code>dense</code> (coverage)</li>\n<li><code>green</code>, <code>dead</code> (vitality)</li>\n<li><code>clover</code>, <code>grass</code> (species)</li></ul></li>\n<li><strong>Derived Ratios</strong>: greenness, clover fraction, cover ratio</li>\n</ul>\n<p><strong>Feature dimensions</strong>: 13 crops × (1,280 DINO + 11 SigLIP + 2 physics) = <strong>16,809 features</strong></p>\n<h3>Model 3: MIL Uncertainty (DINOv3 7B)</h3>\n<ul>\n<li><strong>Backbone</strong>: DINOv3 ViT-7B (4,096-dim embeddings)</li>\n<li><strong>Tile Strategy</strong>: 3×6 patch-aligned grid = <strong>18 tiles</strong></li>\n<li><strong>Head</strong>: MILUncertaintyHead with attention-based aggregation</li>\n<li><strong>Uncertainty</strong>: Predicts both μ (mean) and σ² (variance) per target</li>\n<li><strong>Ensemble</strong>: 5-fold × 3 members × 2 TTA (orig + hflip) = <strong>30 components</strong></li>\n</ul>\n<hr>\n<h2>Training Strategy</h2>\n<h3>Cross-Validation</h3>\n<ul>\n<li><strong>S5 Models</strong>: 3-fold GroupKFold (grouped by image ID)</li>\n<li><strong>MIL Model</strong>: 5-fold GroupKFold with 3 ensemble members per fold</li>\n</ul>\n<h3>Data Augmentation</h3>\n<ul>\n<li><strong>S5 Models</strong>: Crop shuffle TTA (seeds: None, 42, 123)</li>\n<li><strong>MIL Model</strong>: Horizontal flip TTA</li>\n</ul>\n<h3>Normalization</h3>\n<ul>\n<li><strong>Features</strong>: MinMaxScaler(0, 1) fitted per fold</li>\n<li><strong>Targets</strong>: MinMaxScaler(1e-5, 1) for stable gradients</li>\n</ul>\n<hr>\n<h2>Ensemble Optimization</h2>\n<h3>Weight Search Strategy</h3>\n<p>Brute-force grid search with step=0.01 to find optimal weights:</p>\n<table>\n<thead>\n<tr>\n<th>Target</th>\n<th>w_MIL</th>\n<th>w_S5</th>\n<th>w_SigLIP</th>\n<th>R²</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>Dry_Clover_g</code></td>\n<td>0.33</td>\n<td>0.31</td>\n<td>0.36</td>\n<td>0.832</td>\n</tr>\n<tr>\n<td><code>Dry_Dead_g</code></td>\n<td>0.37</td>\n<td>0.38</td>\n<td>0.25</td>\n<td>0.601</td>\n</tr>\n<tr>\n<td><code>Dry_Green_g</code></td>\n<td>0.31</td>\n<td>0.62</td>\n<td>0.07</td>\n<td>0.850</td>\n</tr>\n<tr>\n<td><code>Dry_Total_g</code></td>\n<td>0.56</td>\n<td>0.36</td>\n<td>0.08</td>\n<td>0.792</td>\n</tr>\n<tr>\n<td><code>GDM_g</code></td>\n<td>0.25</td>\n<td>0.69</td>\n<td>0.06</td>\n<td>0.828</td>\n</tr>\n</tbody>\n</table>\n<h3>Physical Constraints</h3>\n<p>Post-processing to ensure biologically valid predictions:</p>\n<ol>\n<li>Clip all values to [0, 200]</li>\n<li>Enforce: <code>Dry_Total_g ≥ Dry_Green_g + Dry_Dead_g</code></li>\n<li>Enforce: <code>GDM_g ≤ Dry_Total_g</code></li>\n</ol>\n<hr>\n<h2>📊 Results Summary</h2>\n<table>\n<thead>\n<tr>\n<th>Method</th>\n<th>Weighted R²</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>MIL Uncertainty (single)</td>\n<td>0.7496</td>\n</tr>\n<tr>\n<td>S5-SigLIP (single)</td>\n<td>0.7556</td>\n</tr>\n<tr>\n<td>S5-DINO (single)</td>\n<td>0.7631</td>\n</tr>\n<tr>\n<td>Simple Average (3-model)</td>\n<td>0.7865</td>\n</tr>\n<tr>\n<td>Global Optimal (3-model)</td>\n<td>0.7880</td>\n</tr>\n<tr>\n<td><strong>Per-Target + Constraints</strong></td>\n<td><strong>0.7901</strong> ✓</td>\n</tr>\n</tbody>\n</table>\n<hr>",
  "messages": [
    {
      "id": 3400098,
      "postDate": "2026-01-31T23:27:50.253Z",
      "content": "<h1>CSIRO Biomass Prediction Pipeline</h1>\n<h2>Overview</h2>\n<p>This solution achieves <strong>0.790 weighted R²</strong> on the CSIRO Biomass prediction task by ensembling three complementary deep learning models with optimized per-target weights and physical constraints.</p>\n<h2>Architecture Overview</h2>\n<p>The pipeline consists of three diverse models combined through an optimized ensemble:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6213473%2Fa86e71395c4869461f909d157cbe390b%2FScreenshot%202026-01-31%20212753.png?generation=1769902056234242&amp;alt=media\" alt=\"\"></p>\n<h3>Model 1: S5-DINO (ContextAwareMIL)</h3>\n<ul>\n<li><strong>Backbone</strong>: DINOv3 ViT-Huge (1,280-dim embeddings)</li>\n<li><strong>Multi-Scale Crops</strong>: 3×3 grid (9 crops) + 2×2 grid (4 crops) = <strong>13 crops</strong></li>\n<li><strong>Physics Features</strong>: Green Fraction (HSV-based) + Excess Green Index (ExG)</li>\n<li><strong>Sequence Model</strong>: 4-layer S5 (State Space) blocks with residual connections</li>\n<li><strong>Context Aggregation</strong>: Mean + Std pooling across crops</li>\n<li><strong>Output</strong>: Softplus activation → Sum across crops (MIL aggregation)</li>\n</ul>\n<p><strong>Feature dimensions</strong>: 13 crops × (1,280 DINO + 2 physics) = <strong>16,666 features</strong></p>\n<h3>Model 2: S5-SigLIP (ContextAwareMIL + Semantic)</h3>\n<ul>\n<li><strong>Backbone</strong>: DINOv3 ViT-Huge + SigLIP SO400M</li>\n<li><strong>Multi-Scale Crops</strong>: Same 13-crop strategy</li>\n<li><strong>Semantic Features</strong>: 8 concept groups via CLIP-style similarity:<ul>\n<li><code>bare</code>, <code>sparse</code>, <code>medium</code>, <code>dense</code> (coverage)</li>\n<li><code>green</code>, <code>dead</code> (vitality)</li>\n<li><code>clover</code>, <code>grass</code> (species)</li></ul></li>\n<li><strong>Derived Ratios</strong>: greenness, clover fraction, cover ratio</li>\n</ul>\n<p><strong>Feature dimensions</strong>: 13 crops × (1,280 DINO + 11 SigLIP + 2 physics) = <strong>16,809 features</strong></p>\n<h3>Model 3: MIL Uncertainty (DINOv3 7B)</h3>\n<ul>\n<li><strong>Backbone</strong>: DINOv3 ViT-7B (4,096-dim embeddings)</li>\n<li><strong>Tile Strategy</strong>: 3×6 patch-aligned grid = <strong>18 tiles</strong></li>\n<li><strong>Head</strong>: MILUncertaintyHead with attention-based aggregation</li>\n<li><strong>Uncertainty</strong>: Predicts both μ (mean) and σ² (variance) per target</li>\n<li><strong>Ensemble</strong>: 5-fold × 3 members × 2 TTA (orig + hflip) = <strong>30 components</strong></li>\n</ul>\n<hr>\n<h2>Training Strategy</h2>\n<h3>Cross-Validation</h3>\n<ul>\n<li><strong>S5 Models</strong>: 3-fold GroupKFold (grouped by image ID)</li>\n<li><strong>MIL Model</strong>: 5-fold GroupKFold with 3 ensemble members per fold</li>\n</ul>\n<h3>Data Augmentation</h3>\n<ul>\n<li><strong>S5 Models</strong>: Crop shuffle TTA (seeds: None, 42, 123)</li>\n<li><strong>MIL Model</strong>: Horizontal flip TTA</li>\n</ul>\n<h3>Normalization</h3>\n<ul>\n<li><strong>Features</strong>: MinMaxScaler(0, 1) fitted per fold</li>\n<li><strong>Targets</strong>: MinMaxScaler(1e-5, 1) for stable gradients</li>\n</ul>\n<hr>\n<h2>Ensemble Optimization</h2>\n<h3>Weight Search Strategy</h3>\n<p>Brute-force grid search with step=0.01 to find optimal weights:</p>\n<table>\n<thead>\n<tr>\n<th>Target</th>\n<th>w_MIL</th>\n<th>w_S5</th>\n<th>w_SigLIP</th>\n<th>R²</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>Dry_Clover_g</code></td>\n<td>0.33</td>\n<td>0.31</td>\n<td>0.36</td>\n<td>0.832</td>\n</tr>\n<tr>\n<td><code>Dry_Dead_g</code></td>\n<td>0.37</td>\n<td>0.38</td>\n<td>0.25</td>\n<td>0.601</td>\n</tr>\n<tr>\n<td><code>Dry_Green_g</code></td>\n<td>0.31</td>\n<td>0.62</td>\n<td>0.07</td>\n<td>0.850</td>\n</tr>\n<tr>\n<td><code>Dry_Total_g</code></td>\n<td>0.56</td>\n<td>0.36</td>\n<td>0.08</td>\n<td>0.792</td>\n</tr>\n<tr>\n<td><code>GDM_g</code></td>\n<td>0.25</td>\n<td>0.69</td>\n<td>0.06</td>\n<td>0.828</td>\n</tr>\n</tbody>\n</table>\n<h3>Physical Constraints</h3>\n<p>Post-processing to ensure biologically valid predictions:</p>\n<ol>\n<li>Clip all values to [0, 200]</li>\n<li>Enforce: <code>Dry_Total_g ≥ Dry_Green_g + Dry_Dead_g</code></li>\n<li>Enforce: <code>GDM_g ≤ Dry_Total_g</code></li>\n</ol>\n<hr>\n<h2>📊 Results Summary</h2>\n<table>\n<thead>\n<tr>\n<th>Method</th>\n<th>Weighted R²</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>MIL Uncertainty (single)</td>\n<td>0.7496</td>\n</tr>\n<tr>\n<td>S5-SigLIP (single)</td>\n<td>0.7556</td>\n</tr>\n<tr>\n<td>S5-DINO (single)</td>\n<td>0.7631</td>\n</tr>\n<tr>\n<td>Simple Average (3-model)</td>\n<td>0.7865</td>\n</tr>\n<tr>\n<td>Global Optimal (3-model)</td>\n<td>0.7880</td>\n</tr>\n<tr>\n<td><strong>Per-Target + Constraints</strong></td>\n<td><strong>0.7901</strong> ✓</td>\n</tr>\n</tbody>\n</table>\n<hr>",
      "rawMarkdown": "# CSIRO Biomass Prediction Pipeline\n\n## Overview\n\nThis solution achieves **0.790 weighted R²** on the CSIRO Biomass prediction task by ensembling three complementary deep learning models with optimized per-target weights and physical constraints.\n\n\n\n## Architecture Overview\n\nThe pipeline consists of three diverse models combined through an optimized ensemble:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6213473%2Fa86e71395c4869461f909d157cbe390b%2FScreenshot%202026-01-31%20212753.png?generation=1769902056234242&alt=media)\n\n\n### Model 1: S5-DINO (ContextAwareMIL)\n\n- **Backbone**: DINOv3 ViT-Huge (1,280-dim embeddings)\n- **Multi-Scale Crops**: 3×3 grid (9 crops) + 2×2 grid (4 crops) = **13 crops**\n- **Physics Features**: Green Fraction (HSV-based) + Excess Green Index (ExG)\n- **Sequence Model**: 4-layer S5 (State Space) blocks with residual connections\n- **Context Aggregation**: Mean + Std pooling across crops\n- **Output**: Softplus activation → Sum across crops (MIL aggregation)\n\n**Feature dimensions**: 13 crops × (1,280 DINO + 2 physics) = **16,666 features**\n\n### Model 2: S5-SigLIP (ContextAwareMIL + Semantic)\n\n- **Backbone**: DINOv3 ViT-Huge + SigLIP SO400M\n- **Multi-Scale Crops**: Same 13-crop strategy\n- **Semantic Features**: 8 concept groups via CLIP-style similarity:\n  - `bare`, `sparse`, `medium`, `dense` (coverage)\n  - `green`, `dead` (vitality)\n  - `clover`, `grass` (species)\n- **Derived Ratios**: greenness, clover fraction, cover ratio\n\n**Feature dimensions**: 13 crops × (1,280 DINO + 11 SigLIP + 2 physics) = **16,809 features**\n\n### Model 3: MIL Uncertainty (DINOv3 7B)\n\n- **Backbone**: DINOv3 ViT-7B (4,096-dim embeddings)\n- **Tile Strategy**: 3×6 patch-aligned grid = **18 tiles**\n- **Head**: MILUncertaintyHead with attention-based aggregation\n- **Uncertainty**: Predicts both μ (mean) and σ² (variance) per target\n- **Ensemble**: 5-fold × 3 members × 2 TTA (orig + hflip) = **30 components**\n\n---\n\n## Training Strategy\n\n### Cross-Validation\n- **S5 Models**: 3-fold GroupKFold (grouped by image ID)\n- **MIL Model**: 5-fold GroupKFold with 3 ensemble members per fold\n\n### Data Augmentation\n- **S5 Models**: Crop shuffle TTA (seeds: None, 42, 123)\n- **MIL Model**: Horizontal flip TTA\n\n### Normalization\n- **Features**: MinMaxScaler(0, 1) fitted per fold\n- **Targets**: MinMaxScaler(1e-5, 1) for stable gradients\n\n---\n\n## Ensemble Optimization\n\n### Weight Search Strategy\nBrute-force grid search with step=0.01 to find optimal weights:\n\n| Target | w_MIL | w_S5 | w_SigLIP | R² |\n|--------|-------|------|----------|-----|\n| `Dry_Clover_g` | 0.33 | 0.31 | 0.36 | 0.832 |\n| `Dry_Dead_g` | 0.37 | 0.38 | 0.25 | 0.601 |\n| `Dry_Green_g` | 0.31 | 0.62 | 0.07 | 0.850 |\n| `Dry_Total_g` | 0.56 | 0.36 | 0.08 | 0.792 |\n| `GDM_g` | 0.25 | 0.69 | 0.06 | 0.828 |\n\n### Physical Constraints\nPost-processing to ensure biologically valid predictions:\n1. Clip all values to [0, 200]\n2. Enforce: `Dry_Total_g ≥ Dry_Green_g + Dry_Dead_g`\n3. Enforce: `GDM_g ≤ Dry_Total_g`\n\n---\n\n## 📊 Results Summary\n\n| Method | Weighted R² |\n|--------|-------------|\n| MIL Uncertainty (single) | 0.7496 |\n| S5-SigLIP (single) | 0.7556 |\n| S5-DINO (single) | 0.7631 |\n| Simple Average (3-model) | 0.7865 |\n| Global Optimal (3-model) | 0.7880 |\n| **Per-Target + Constraints** | **0.7901** ✓ |\n\n---\n\n",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3400098": "# CSIRO Biomass Prediction Pipeline\n\n## Overview\n\nThis solution achieves **0.790 weighted R²** on the CSIRO Biomass prediction task by ensembling three complementary deep learning models with optimized per-target weights and physical constraints.\n\n\n\n## Architecture Overview\n\nThe pipeline consists of three diverse models combined through an optimized ensemble:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6213473%2Fa86e71395c4869461f909d157cbe390b%2FScreenshot%202026-01-31%20212753.png?generation=1769902056234242&alt=media)\n\n\n### Model 1: S5-DINO (ContextAwareMIL)\n\n- **Backbone**: DINOv3 ViT-Huge (1,280-dim embeddings)\n- **Multi-Scale Crops**: 3×3 grid (9 crops) + 2×2 grid (4 crops) = **13 crops**\n- **Physics Features**: Green Fraction (HSV-based) + Excess Green Index (ExG)\n- **Sequence Model**: 4-layer S5 (State Space) blocks with residual connections\n- **Context Aggregation**: Mean + Std pooling across crops\n- **Output**: Softplus activation → Sum across crops (MIL aggregation)\n\n**Feature dimensions**: 13 crops × (1,280 DINO + 2 physics) = **16,666 features**\n\n### Model 2: S5-SigLIP (ContextAwareMIL + Semantic)\n\n- **Backbone**: DINOv3 ViT-Huge + SigLIP SO400M\n- **Multi-Scale Crops**: Same 13-crop strategy\n- **Semantic Features**: 8 concept groups via CLIP-style similarity:\n  - `bare`, `sparse`, `medium`, `dense` (coverage)\n  - `green`, `dead` (vitality)\n  - `clover`, `grass` (species)\n- **Derived Ratios**: greenness, clover fraction, cover ratio\n\n**Feature dimensions**: 13 crops × (1,280 DINO + 11 SigLIP + 2 physics) = **16,809 features**\n\n### Model 3: MIL Uncertainty (DINOv3 7B)\n\n- **Backbone**: DINOv3 ViT-7B (4,096-dim embeddings)\n- **Tile Strategy**: 3×6 patch-aligned grid = **18 tiles**\n- **Head**: MILUncertaintyHead with attention-based aggregation\n- **Uncertainty**: Predicts both μ (mean) and σ² (variance) per target\n- **Ensemble**: 5-fold × 3 members × 2 TTA (orig + hflip) = **30 components**\n\n---\n\n## Training Strategy\n\n### Cross-Validation\n- **S5 Models**: 3-fold GroupKFold (grouped by image ID)\n- **MIL Model**: 5-fold GroupKFold with 3 ensemble members per fold\n\n### Data Augmentation\n- **S5 Models**: Crop shuffle TTA (seeds: None, 42, 123)\n- **MIL Model**: Horizontal flip TTA\n\n### Normalization\n- **Features**: MinMaxScaler(0, 1) fitted per fold\n- **Targets**: MinMaxScaler(1e-5, 1) for stable gradients\n\n---\n\n## Ensemble Optimization\n\n### Weight Search Strategy\nBrute-force grid search with step=0.01 to find optimal weights:\n\n| Target | w_MIL | w_S5 | w_SigLIP | R² |\n|--------|-------|------|----------|-----|\n| `Dry_Clover_g` | 0.33 | 0.31 | 0.36 | 0.832 |\n| `Dry_Dead_g` | 0.37 | 0.38 | 0.25 | 0.601 |\n| `Dry_Green_g` | 0.31 | 0.62 | 0.07 | 0.850 |\n| `Dry_Total_g` | 0.56 | 0.36 | 0.08 | 0.792 |\n| `GDM_g` | 0.25 | 0.69 | 0.06 | 0.828 |\n\n### Physical Constraints\nPost-processing to ensure biologically valid predictions:\n1. Clip all values to [0, 200]\n2. Enforce: `Dry_Total_g ≥ Dry_Green_g + Dry_Dead_g`\n3. Enforce: `GDM_g ≤ Dry_Total_g`\n\n---\n\n## 📊 Results Summary\n\n| Method | Weighted R² |\n|--------|-------------|\n| MIL Uncertainty (single) | 0.7496 |\n| S5-SigLIP (single) | 0.7556 |\n| S5-DINO (single) | 0.7631 |\n| Simple Average (3-model) | 0.7865 |\n| Global Optimal (3-model) | 0.7880 |\n| **Per-Target + Constraints** | **0.7901** ✓ |\n\n---\n\n"
  }
}