# 123rd (public 6th) Solution - Tile-based ViT with 2D Pos-Emb & MIL pooling

TL;DR
Our approach reconstructs the input image into a 32-tile grid and learns the global representation using ViT + 2D positional embeddings + Multi-head MIL attention pooling. Instead of predicting all 5 targets directly, we predict the 3 main targets (
DG
,
GDM
,
DT
) and physically reconstruct the remaining two (
DD
,
DC
) to align with the metric. We also utilized LUPI (Learning Using Privileged Information) via a meta-teacher to distill knowledge from train-only metadata into the vision model, enabling image-only inference.
![Figure 1. Overview (Inference): Tile-based ViT with 2D positional embeddings and MIL pooling](assets/05bb7e2acbd5efea3931c799.png)
Figure 1. Overview (Inference): Tile-based ViT with 2D positional embeddings and MIL pooling
1. Data preprocessing
Features
Meta
:
Pre_GSHH_NDVI
,
Height_Ave_cm
,
Sampling_Date
,
State
,
Species
Derived
Month
: extracted from
Sampling_Date
State_encoded
,
Species_encoded
: label encoding
Quality meta (train-time meta feature)
(Ref.
[https://www.kaggle.com/competitions/csiro-biomass/discussion/667537](https://www.kaggle.com/competitions/csiro-biomass/discussion/667537)
)
q_sharpness
: Laplacian variance (sharpness)
q_noise
: residual mean after Gaussian blur (noise proxy)
q_device_cluster
:
StandardScaler + KMeans(n_clusters=2)
32-tile input construction (4×8 global grid)
Original image:
2000×1000
Crops (each 1000×1000)
Left: x 0–1000
Middle: x 500–1500
Right: x 1000–2000
Build 32 tiles of 250×250 on a global 4×8 grid:
global col 0..1: from
Left
(local col 0..1) → 8 tiles
global col 2..5: from
Middle
(local col 0..3) → 16 tiles
global col 6..7: from
Right
(local col 2..3) → 8 tiles
→ total 32 tiles (row-major order)
Augmentation / transform
Crop-level aug: random H/V flip, ColorJitter
Per-tile transform: Resize(
img_size=448
), Normalize
Tile dropout: Dropout for some tiles
2. Data split strategy
StratifiedGroupKFold
Group key
:
Sampling_Date + State
Stratify
: After quantile binning
Dry_Total_g
using
qcut(n_bins=5)
, stratify
We used 4 folds, and the cross-validation was stable between folds.
3. Method
3.1 Backbone + tile feature extraction
Backbone:
timm/vit_huge_plus_patch16_dinov3...
(
num_classes=0
,
global_pool=avg
)
Input:
(B, 32, C, H, W)
→ reshape
(B*32, C, H, W)
→ backbone →
(B, 32, F)
3.2 2D positional embedding + multi-head MIL attention pooling
Global spatial information is injected by adding 2D positional embeddings
(row_emb + col_emb)
to the tile features.
The attention scorer computes softmax weights over the tile dimension (per head) and aggregates the weighted features.
An output projection maps the result back to
(B, F)
, keeping the downstream heads simple.
3.3 Heads: main(3) + aux(DD/DC)
Main head
: Predicts the three primary targets
(DG, GDM, DT)
.
Aux heads
: Independently predict
DD
and
DC
(with ReLU to prevent negative values).
Not used during inference by default.
3.4 Postprocess + 5-target reconstruction
Enforce constraints:
DG ≥ 0
,
GDM ≥ DG
,
DT ≥ GDM
5-target reconstruction:
DD = DT - GDM
DC = GDM - DG
3.5 Loss
Base
:
SmoothL1Loss
applied to the 3 main targets.
R² Surrogate Loss
:
Normalized MSE based on SST calculated from the train split.
Importance weights: [0.1, 0.2, 1.0] for
(DG, GDM, DT)
.
Hybrid
:
loss = alpha * SmoothL1 + (1-alpha) * R²_surrogate
(with
alpha=0.5
).
Empirically,
SmoothL1
contributed to better training stability.
3.6 Aux + consistency + log loss
Aux
: SmoothL1 on DD/DC.
Log-space L1
: Improves stability for large-scale targets.
Consistency Loss
(
consis_w=0.2
): Enforces alignment between aux predictions and reconstructed values:
Induce
DD_aux_pred
to match
(DT−GDM)
and
DC_aux_pred
to match
(GDM−DG)
.
3.7 Train-time meta teacher (LUPI) — knowledge distillation
Meta input
: NDVI, Height, State/Species, Month + quality meta(sharpness/noise/cluster)
Teacher predicts ratios (sigmoid)
:
DG/DT
,
GDM/DT
,
DD/DT
.
DC/DT
is excluded as redundant (derivable from
GDM/DT − DG/DT
).
Distillation
: Teacher outputs are supervised by target ratios, while student ratios are distilled from the teacher's predictions.
Absolute DD
: We also separately supervise and distill the absolute DD value (
teacher_dd
,
out_dim=1
).
3.8 Two-stage fine-tuning
Stage 1
: Unfreeze last 1 block (
lr=1e-3
) to stably train the head, MIL, and meta components.
Stage 2
: Unfreeze last 3 blocks (
lr=1e-4
) to fine-tune and push the performance ceiling.
4. Results
Best Single Model:
CV (SGKF 4-fold)
:
0.7615
Public/Private LB
:
0.78/0.62
Table: Single-model milestone progression (Baseline → Best)
Step
Tiles / img_size
Key change (what improved)
Mean CV
(4-fold)
Full OOF
Public LB
Δ Full OOF vs Reference†
Reference†
–
early Lightning baseline
0.5802
0.6357
0.64
+0.0000 (
+0.0%
)
Baseline (comparable)
12 / 512
TileAttentionPool
0.6972
0.7044
0.76
+0.0687 (
+10.8%
)
+ Pooling
12 / 512
Multi-head MIL pooling
0.7165
0.7328
0.76
+0.0971 (
+15.3%
)
+ Tiles / scale
32 / 256
32 tiles
0.7224
0.7306
0.77
+0.1144 (
+18.0%
)
+ Quality meta
32 / 256
+ quality meta (q_sharpness/q_noise/cluster)
0.7369
0.7509
0.77
+0.1152 (
+18.1%
)
Best single
32 / 448
2D pos + MH-MIL + quality meta + R² loss
0.7468
0.7615
0.78
+0.1258 (+19.8%)
Mean CV and Full OOF are based on the same 4-fold pipeline. Δ(%) indicates the improvement in Full OOF compared to the Reference†.
5. Discussion & Retrospective
While our solution achieved 6th place on the Public LB, we experienced a significant shake-up, finishing 129th on the Private LB. Below is a summary of our approach's strengths and our retrospective analysis on the drop.
What Worked
2D Positional Embedding + MIL
: Explicitly modeling the spatial relationship of tiles with 2D embeddings significantly outperformed simple averaging or concatenation (LB 0.71 → 0.75).
Target Reconstruction Strategy
: Focusing on the three highly correlated "green biomass" targets (
DG
,
GDM
,
DT
) and reconstructing the targets (
DD
,
DC
) proved effective for ensuring physical constraints (DG <= GDM <= DT) and stabilizing convergence.
Metric-aligned Training
: Mixing the R² surrogate loss helped align the optimization direction with the competition metric.
Retrospective
The Shake-up (Public 6th → Private 129th) I suspect two main factors contributed to the lack of generalization on the Private set:
Over-reliance on Meta-Teacher (LUPI)
: We utilized a Meta-Teacher to distill metadata (NDVI, Season, etc.) into the vision model. While this boosted CV and Public LB, it likely introduced a
bias towards the training/public set's metadata distribution
. As noted in our risks, if the metadata distribution shifted significantly in the Private set, the "internalized priors" from the teacher might have harmed the model's robustness.
CV Strategy vs. Distribution Shift
: We used StratifiedGroupKFold (grouped by Date+State, stratified by Target bins) to maintain a stable CV-LB correlation. While this strategy successfully prevented leakage from same-day samples, it may not have been strict enough to simulate the domain shift present in the Private test set.
6. Conclusion
Our solution demonstrates that combining
2D positional embeddings
with
MIL pooling
is effective for high-resolution biomass estimation. However, the significant shake-up highlights the risk of
over-fitting to metadata priors via LUPI
. While distilling privileged information boosts performance on similar distributions, it requires more rigorous validation to ensure robustness against domain shifts in unseen data.