{ "id": 673265, "title": "CSIRO Solution", "url": "/competitions/csiro-biomass/discussion/673265", "author_name": "Ahmed Walid", "post_date": "2026-02-13T13:45:29.802000", "votes": 3, "comment_count": 0, "views": 0, "content": "

Overview

\n

The solution is based on a high capacity DinoV3 Huge backbone fine tuned using LoRA (Low-Rank Adaptation). I treated the problem as a density estimation task using an architecture that processes high-resolution images via a tiling strategy, aggregating predictions across 5 specific biomass targets.

\n

1. Data Pipeline & Augmentation

\n

Resolution & Tiling

\n

I processed images at a target resolution of 1024x2048. Instead of downscaling the entire image and losing granular texture details, I implemented a tiling mechanism:

\n\n

The \"Toroidal Roll\"

\n

To handle the edge continuity of crop fields and prevent the model from overfitting to specific crop locations, I implemented a custom Toroidal Roll augmentation (p=0.7).

\n\n

Other Augmentations:

\n\n

2. Model Architecture:

\n

I utilized vit_huge_plus_patch16_dinov3_qkvb as the backbone.

\n

LoRA Config

\n\n

Multi-Target Density

\n

The solution used a custom head to handle the 5 distinct targets simultaneously:

\n
    \n
  1. Token Extraction: Extract patch tokens from the DinoV3 backbone.
    2. \n
  2. Attention Gating: A shared attention gate (Linear -> Tanh -> Linear -> Sigmoid) computes a weight map for the tokens, determining which parts of the image contain relevant biomass.
    3. \n
  3. 5-Head Output: Use 5 separate heads (one per target).
    4. \n
  4. Quantile Regression: Each head projects to 3 outputs corresponding to quantiles [0.1, 0.5, 0.9].
    5. \n
  5. Aggregation: The model computes density * attention per tile and sums the mass to get the final image-level prediction.
    6. \n
\n
# Core aggregation logic  \ndensity = head(tokens)  \nweighted_map = density * attn  \ntile_mass = torch.sum(weighted_map, dim=1) # Sum over tokens  \nfinal_preds = torch.sum(grouped_preds, dim=1) # Sum over tiles\n
\n

3. Training Strategy

\n\n

4. Hardware

\n", "messages": [ { "id": 3405600, "postDate": "2026-02-13T13:45:29.803Z", "content": "

Overview

\n

The solution is based on a high capacity DinoV3 Huge backbone fine tuned using LoRA (Low-Rank Adaptation). I treated the problem as a density estimation task using an architecture that processes high-resolution images via a tiling strategy, aggregating predictions across 5 specific biomass targets.

\n

1. Data Pipeline & Augmentation

\n

Resolution & Tiling

\n

I processed images at a target resolution of 1024x2048. Instead of downscaling the entire image and losing granular texture details, I implemented a tiling mechanism:

\n\n

The \"Toroidal Roll\"

\n

To handle the edge continuity of crop fields and prevent the model from overfitting to specific crop locations, I implemented a custom Toroidal Roll augmentation (p=0.7).

\n\n

Other Augmentations:

\n\n

2. Model Architecture:

\n

I utilized vit_huge_plus_patch16_dinov3_qkvb as the backbone.

\n

LoRA Config

\n\n

Multi-Target Density

\n

The solution used a custom head to handle the 5 distinct targets simultaneously:

\n
    \n
  1. Token Extraction: Extract patch tokens from the DinoV3 backbone.
    2. \n
  2. Attention Gating: A shared attention gate (Linear -> Tanh -> Linear -> Sigmoid) computes a weight map for the tokens, determining which parts of the image contain relevant biomass.
    3. \n
  3. 5-Head Output: Use 5 separate heads (one per target).
    4. \n
  4. Quantile Regression: Each head projects to 3 outputs corresponding to quantiles [0.1, 0.5, 0.9].
    5. \n
  5. Aggregation: The model computes density * attention per tile and sums the mass to get the final image-level prediction.
    6. \n
\n
# Core aggregation logic  \ndensity = head(tokens)  \nweighted_map = density * attn  \ntile_mass = torch.sum(weighted_map, dim=1) # Sum over tokens  \nfinal_preds = torch.sum(grouped_preds, dim=1) # Sum over tiles\n
\n

3. Training Strategy

\n\n

4. Hardware

\n", "rawMarkdown": "## **Overview**\n\nThe solution is based on a high capacity **DinoV3 Huge** backbone fine tuned using **LoRA (Low-Rank Adaptation)**. I treated the problem as a density estimation task using an architecture that processes high-resolution images via a tiling strategy, aggregating predictions across 5 specific biomass targets.\n\n\n## **1\\. Data Pipeline & Augmentation**\n\n### **Resolution & Tiling**\n\nI processed images at a target resolution of **1024x2048**. Instead of downscaling the entire image and losing granular texture details, I implemented a tiling mechanism:\n\n* The image is sliced into **512x512** non-overlapping tiles. \n* These tiles are passed through the backbone as a batch, preserving local features.\n\n### **The \"Toroidal Roll\"**\n\nTo handle the edge continuity of crop fields and prevent the model from overfitting to specific crop locations, I implemented a custom **Toroidal Roll** augmentation (`p=0.7`).\n\n* This randomly shifts the image pixels vertically and horizontally, wrapping them around (rolling). \n* This mimics an infinite field and ensures the model focuses on texture density rather than absolute position.\n\n**Other Augmentations:**\n\n* Horizontal/Vertical Flip \n* Random Rotate90 \n* Optical Distortion & CLAHE \n* ISO Noise & Color Jitter\n\n## **2\\. Model Architecture:** \n\nI utilized `vit_huge_plus_patch16_dinov3_qkvb` as the backbone. \n\n### **LoRA Config**\n\n* **Rank:** 16 \n* **Alpha:** 32 \n* **Targets:** `qkv`, `proj`, `fc1`, `fc2` \n* **Trainable Params:** I also unfroze normalization layers (`norm`).\n\n### **Multi-Target Density**\n\nThe solution used a custom head to handle the 5 distinct targets simultaneously:\n\n1. **Token Extraction:** Extract patch tokens from the DinoV3 backbone. \n2. **Attention Gating:** A shared attention gate (`Linear -> Tanh -> Linear -> Sigmoid`) computes a weight map for the tokens, determining which parts of the image contain relevant biomass. \n3. **5-Head Output:** Use 5 separate heads (one per target). \n4. **Quantile Regression:** Each head projects to 3 outputs corresponding to quantiles `[0.1, 0.5, 0.9]`. \n5. **Aggregation:** The model computes `density * attention` per tile and sums the mass to get the final image-level prediction.\n\n```python\n# Core aggregation logic \ndensity = head(tokens) \nweighted_map = density * attn \ntile_mass = torch.sum(weighted_map, dim=1) # Sum over tokens \nfinal_preds = torch.sum(grouped_preds, dim=1) # Sum over tiles\n```\n\n## **3\\. Training Strategy**\n\n* **Loss Function:** Quantile Loss (Pinball Loss) on Log-space targets (`log1p`). \n* **Optimizer:** AdamW (`lr=2e-4`). \n* **Schedule:** 80 Epochs with Early Stopping (Patience=20). \n* **Precision:** `bfloat16` mixed precision \n* **Compilation:** `torch.compile(mode=\"default\")` was used to maximize throughput.\n\n## **4\\. Hardware**\n\n* **Compute:** Single NVIDIA A100 (40GB). \n* **Gradient Accumulation:** (steps=4). \n* **Training time**: 10 hours\n", "votes": 3 } ], "comments": [], "raw_markdown_by_id": { "3405600": "## **Overview**\n\nThe solution is based on a high capacity **DinoV3 Huge** backbone fine tuned using **LoRA (Low-Rank Adaptation)**. I treated the problem as a density estimation task using an architecture that processes high-resolution images via a tiling strategy, aggregating predictions across 5 specific biomass targets.\n\n\n## **1\\. Data Pipeline & Augmentation**\n\n### **Resolution & Tiling**\n\nI processed images at a target resolution of **1024x2048**. Instead of downscaling the entire image and losing granular texture details, I implemented a tiling mechanism:\n\n* The image is sliced into **512x512** non-overlapping tiles. \n* These tiles are passed through the backbone as a batch, preserving local features.\n\n### **The \"Toroidal Roll\"**\n\nTo handle the edge continuity of crop fields and prevent the model from overfitting to specific crop locations, I implemented a custom **Toroidal Roll** augmentation (`p=0.7`).\n\n* This randomly shifts the image pixels vertically and horizontally, wrapping them around (rolling). \n* This mimics an infinite field and ensures the model focuses on texture density rather than absolute position.\n\n**Other Augmentations:**\n\n* Horizontal/Vertical Flip \n* Random Rotate90 \n* Optical Distortion & CLAHE \n* ISO Noise & Color Jitter\n\n## **2\\. Model Architecture:** \n\nI utilized `vit_huge_plus_patch16_dinov3_qkvb` as the backbone. \n\n### **LoRA Config**\n\n* **Rank:** 16 \n* **Alpha:** 32 \n* **Targets:** `qkv`, `proj`, `fc1`, `fc2` \n* **Trainable Params:** I also unfroze normalization layers (`norm`).\n\n### **Multi-Target Density**\n\nThe solution used a custom head to handle the 5 distinct targets simultaneously:\n\n1. **Token Extraction:** Extract patch tokens from the DinoV3 backbone. \n2. **Attention Gating:** A shared attention gate (`Linear -> Tanh -> Linear -> Sigmoid`) computes a weight map for the tokens, determining which parts of the image contain relevant biomass. \n3. **5-Head Output:** Use 5 separate heads (one per target). \n4. **Quantile Regression:** Each head projects to 3 outputs corresponding to quantiles `[0.1, 0.5, 0.9]`. \n5. **Aggregation:** The model computes `density * attention` per tile and sums the mass to get the final image-level prediction.\n\n```python\n# Core aggregation logic \ndensity = head(tokens) \nweighted_map = density * attn \ntile_mass = torch.sum(weighted_map, dim=1) # Sum over tokens \nfinal_preds = torch.sum(grouped_preds, dim=1) # Sum over tiles\n```\n\n## **3\\. Training Strategy**\n\n* **Loss Function:** Quantile Loss (Pinball Loss) on Log-space targets (`log1p`). \n* **Optimizer:** AdamW (`lr=2e-4`). \n* **Schedule:** 80 Epochs with Early Stopping (Patience=20). \n* **Precision:** `bfloat16` mixed precision \n* **Compilation:** `torch.compile(mode=\"default\")` was used to maximize throughput.\n\n## **4\\. Hardware**\n\n* **Compute:** Single NVIDIA A100 (40GB). \n* **Gradient Accumulation:** (steps=4). \n* **Training time**: 10 hours\n" } }