![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F13143896%2F3faf64ba4b9134cb37becc3432355853%2Fimage%20(2).png?generation=1769698780181835&alt=media\")
\nFigure 1. Overview (Inference): Tile-based ViT with 2D positional embeddings and MIL poolingOur approach reconstructs the input image into a 32-tile grid and learns the global representation using ViT + 2D positional embeddings + Multi-head MIL attention pooling. Instead of predicting all 5 targets directly, we predict the 3 main targets (DG, GDM, DT) and physically reconstruct the remaining two (DD, DC) to align with the metric. We also utilized LUPI (Learning Using Privileged Information) via a meta-teacher to distill knowledge from train-only metadata into the vision model, enabling image-only inference.
\nFigure 1. Overview (Inference): Tile-based ViT with 2D positional embeddings and MIL pooling
Pre_GSHH_NDVI, Height_Ave_cm, Sampling_Date, State, SpeciesMonth: extracted from Sampling_DateState_encoded, Species_encoded: label encoding(Ref. https://www.kaggle.com/competitions/csiro-biomass/discussion/667537)
\nq_sharpness: Laplacian variance (sharpness)q_noise: residual mean after Gaussian blur (noise proxy)q_device_cluster: StandardScaler + KMeans(n_clusters=2)Sampling_Date + StateDry_Total_g using qcut(n_bins=5), stratifyWe used 4 folds, and the cross-validation was stable between folds.
\ntimm/vit_huge_plus_patch16_dinov3... (num_classes=0, global_pool=avg)(B, 32, C, H, W) → reshape (B*32, C, H, W) → backbone → (B, 32, F)(row_emb + col_emb)to the tile features.(B, F), keeping the downstream heads simple.(DG, GDM, DT).DD and DC (with ReLU to prevent negative values).DG ≥ 0, GDM ≥ DG, DT ≥ GDMDD = DT - GDMDC = GDM - DGSmoothL1Loss applied to the 3 main targets.(DG, GDM, DT).loss = alpha * SmoothL1 + (1-alpha) * R²_surrogate (with alpha=0.5).SmoothL1 contributed to better training stability.consis_w=0.2): Enforces alignment between aux predictions and reconstructed values:DD_aux_pred to match (DT−GDM) and DC_aux_pred to match (GDM−DG).DG/DT, GDM/DT, DD/DT.DC/DT is excluded as redundant (derivable from GDM/DT − DG/DT).teacher_dd, out_dim=1).lr=1e-3) to stably train the head, MIL, and meta components.lr=1e-4) to fine-tune and push the performance ceiling.| Step | \nTiles / img_size | \nKey change (what improved) | \nMean CV (4-fold) | \nFull OOF | \nPublic LB | \nΔ Full OOF vs Reference† | \n
|---|---|---|---|---|---|---|
| Reference† | \n– | \nearly Lightning baseline | \n0.5802 | \n0.6357 | \n0.64 | \n+0.0000 (+0.0%) | \n
| Baseline (comparable) | \n12 / 512 | \nTileAttentionPool | \n0.6972 | \n0.7044 | \n0.76 | \n+0.0687 (+10.8%) | \n
| + Pooling | \n12 / 512 | \nMulti-head MIL pooling | \n0.7165 | \n0.7328 | \n0.76 | \n+0.0971 (+15.3%) | \n
| + Tiles / scale | \n32 / 256 | \n32 tiles | \n0.7224 | \n0.7306 | \n0.77 | \n+0.1144 (+18.0%) | \n
| + Quality meta | \n32 / 256 | \n+ quality meta (q_sharpness/q_noise/cluster) | \n0.7369 | \n0.7509 | \n0.77 | \n+0.1152 (+18.1%) | \n
| Best single | \n32 / 448 | \n2D pos + MH-MIL + quality meta + R² loss | \n0.7468 | \n0.7615 | \n0.78 | \n+0.1258 (+19.8%) | \n
Mean CV and Full OOF are based on the same 4-fold pipeline. Δ(%) indicates the improvement in Full OOF compared to the Reference†.
\nWhile our solution achieved 6th place on the Public LB, we experienced a significant shake-up, finishing 129th on the Private LB. Below is a summary of our approach's strengths and our retrospective analysis on the drop.
\nDG, GDM, DT) and reconstructing the targets (DD, DC) proved effective for ensuring physical constraints (DG <= GDM <= DT) and stabilizing convergence.The Shake-up (Public 6th → Private 129th) I suspect two main factors contributed to the lack of generalization on the Private set:
\nOur solution demonstrates that combining 2D positional embeddings with MIL pooling is effective for high-resolution biomass estimation. However, the significant shake-up highlights the risk of over-fitting to metadata priors via LUPI. While distilling privileged information boosts performance on similar distributions, it requires more rigorous validation to ensure robustness against domain shifts in unseen data.
", "messages": [ { "id": "3398702", "postDate": "01/29/2026 15:07:20", "content": "Our approach reconstructs the input image into a 32-tile grid and learns the global representation using ViT + 2D positional embeddings + Multi-head MIL attention pooling. Instead of predicting all 5 targets directly, we predict the 3 main targets (DG, GDM, DT) and physically reconstruct the remaining two (DD, DC) to align with the metric. We also utilized LUPI (Learning Using Privileged Information) via a meta-teacher to distill knowledge from train-only metadata into the vision model, enabling image-only inference.
\nFigure 1. Overview (Inference): Tile-based ViT with 2D positional embeddings and MIL pooling
Pre_GSHH_NDVI, Height_Ave_cm, Sampling_Date, State, SpeciesMonth: extracted from Sampling_DateState_encoded, Species_encoded: label encoding(Ref. https://www.kaggle.com/competitions/csiro-biomass/discussion/667537)
\nq_sharpness: Laplacian variance (sharpness)q_noise: residual mean after Gaussian blur (noise proxy)q_device_cluster: StandardScaler + KMeans(n_clusters=2)Sampling_Date + StateDry_Total_g using qcut(n_bins=5), stratifyWe used 4 folds, and the cross-validation was stable between folds.
\ntimm/vit_huge_plus_patch16_dinov3... (num_classes=0, global_pool=avg)(B, 32, C, H, W) → reshape (B*32, C, H, W) → backbone → (B, 32, F)(row_emb + col_emb)to the tile features.(B, F), keeping the downstream heads simple.(DG, GDM, DT).DD and DC (with ReLU to prevent negative values).DG ≥ 0, GDM ≥ DG, DT ≥ GDMDD = DT - GDMDC = GDM - DGSmoothL1Loss applied to the 3 main targets.(DG, GDM, DT).loss = alpha * SmoothL1 + (1-alpha) * R²_surrogate (with alpha=0.5).SmoothL1 contributed to better training stability.consis_w=0.2): Enforces alignment between aux predictions and reconstructed values:DD_aux_pred to match (DT−GDM) and DC_aux_pred to match (GDM−DG).DG/DT, GDM/DT, DD/DT.DC/DT is excluded as redundant (derivable from GDM/DT − DG/DT).teacher_dd, out_dim=1).lr=1e-3) to stably train the head, MIL, and meta components.lr=1e-4) to fine-tune and push the performance ceiling.| Step | \nTiles / img_size | \nKey change (what improved) | \nMean CV (4-fold) | \nFull OOF | \nPublic LB | \nΔ Full OOF vs Reference† | \n
|---|---|---|---|---|---|---|
| Reference† | \n– | \nearly Lightning baseline | \n0.5802 | \n0.6357 | \n0.64 | \n+0.0000 (+0.0%) | \n
| Baseline (comparable) | \n12 / 512 | \nTileAttentionPool | \n0.6972 | \n0.7044 | \n0.76 | \n+0.0687 (+10.8%) | \n
| + Pooling | \n12 / 512 | \nMulti-head MIL pooling | \n0.7165 | \n0.7328 | \n0.76 | \n+0.0971 (+15.3%) | \n
| + Tiles / scale | \n32 / 256 | \n32 tiles | \n0.7224 | \n0.7306 | \n0.77 | \n+0.1144 (+18.0%) | \n
| + Quality meta | \n32 / 256 | \n+ quality meta (q_sharpness/q_noise/cluster) | \n0.7369 | \n0.7509 | \n0.77 | \n+0.1152 (+18.1%) | \n
| Best single | \n32 / 448 | \n2D pos + MH-MIL + quality meta + R² loss | \n0.7468 | \n0.7615 | \n0.78 | \n+0.1258 (+19.8%) | \n
Mean CV and Full OOF are based on the same 4-fold pipeline. Δ(%) indicates the improvement in Full OOF compared to the Reference†.
\nWhile our solution achieved 6th place on the Public LB, we experienced a significant shake-up, finishing 129th on the Private LB. Below is a summary of our approach's strengths and our retrospective analysis on the drop.
\nDG, GDM, DT) and reconstructing the targets (DD, DC) proved effective for ensuring physical constraints (DG <= GDM <= DT) and stabilizing convergence.The Shake-up (Public 6th → Private 129th) I suspect two main factors contributed to the lack of generalization on the Private set:
\nOur solution demonstrates that combining 2D positional embeddings with MIL pooling is effective for high-resolution biomass estimation. However, the significant shake-up highlights the risk of over-fitting to metadata priors via LUPI. While distilling privileged information boosts performance on similar distributions, it requires more rigorous validation to ensure robustness against domain shifts in unseen data.
", "rawMarkdown": "# TL;DR\nOur approach reconstructs the input image into a 32-tile grid and learns the global representation using ViT + 2D positional embeddings + Multi-head MIL attention pooling. Instead of predicting all 5 targets directly, we predict the 3 main targets (`DG`, `GDM`, `DT`) and physically reconstruct the remaining two (`DD`, `DC`) to align with the metric. We also utilized LUPI (Learning Using Privileged Information) via a meta-teacher to distill knowledge from train-only metadata into the vision model, enabling image-only inference.\n\n\n.png?generation=1769698780181835&alt=media)\n**Figure 1. Overview (Inference): Tile-based ViT with 2D positional embeddings and MIL pooling**\n\n## 1. Data preprocessing\n\n### Features\n\n- **Meta**: `Pre_GSHH_NDVI`, `Height_Ave_cm`, `Sampling_Date`, `State`, `Species`\n- **Derived**\n - `Month`: extracted from `Sampling_Date`\n - `State_encoded`, `Species_encoded`: label encoding\n\n### Quality meta (train-time meta feature)\n(Ref. https://www.kaggle.com/competitions/csiro-biomass/discussion/667537)\n\n- `q_sharpness`: Laplacian variance (sharpness)\n- `q_noise`: residual mean after Gaussian blur (noise proxy)\n- `q_device_cluster`: `StandardScaler + KMeans(n_clusters=2)`\n\n### 32-tile input construction (4×8 global grid)\n- Original image: **2000×1000**\n- Crops (each 1000×1000)\n - Left: x 0–1000\n - Middle: x 500–1500\n - Right: x 1000–2000\n- Build 32 tiles of 250×250 on a global 4×8 grid:\n - global col 0..1: from **Left** (local col 0..1) → 8 tiles\n - global col 2..5: from **Middle** (local col 0..3) → 16 tiles\n - global col 6..7: from **Right** (local col 2..3) → 8 tiles\n → total 32 tiles (row-major order)\n\n### Augmentation / transform\n- Crop-level aug: random H/V flip, ColorJitter\n- Per-tile transform: Resize(**img_size=448**), Normalize\n- Tile dropout: Dropout for some tiles\n\n## 2. Data split strategy\n\n### StratifiedGroupKFold\n- **Group key**: `Sampling_Date + State`\n- **Stratify**: After quantile binning `Dry_Total_g` using `qcut(n_bins=5)`, stratify\n\nWe used 4 folds, and the cross-validation was stable between folds.\n\n## 3. Method\n### 3.1 Backbone + tile feature extraction\n- Backbone: `timm/vit_huge_plus_patch16_dinov3...` (`num_classes=0`, `global_pool=avg`)\n- Input: `(B, 32, C, H, W)` → reshape `(B*32, C, H, W)` → backbone → `(B, 32, F)`\n\n### 3.2 2D positional embedding + multi-head MIL attention pooling\n- Global spatial information is injected by adding 2D positional embeddings `(row_emb + col_emb) `to the tile features.\n- The attention scorer computes softmax weights over the tile dimension (per head) and aggregates the weighted features.\n- An output projection maps the result back to `(B, F)`, keeping the downstream heads simple.\n\n### 3.3 Heads: main(3) + aux(DD/DC)\n\n- **Main head**: Predicts the three primary targets `(DG, GDM, DT)`.\n- **Aux heads**: Independently predict `DD` and `DC` (with ReLU to prevent negative values).\n - Not used during inference by default.\n\n### 3.4 Postprocess + 5-target reconstruction\n\n- Enforce constraints:\n - `DG ≥ 0`, `GDM ≥ DG`, `DT ≥ GDM`\n- 5-target reconstruction:\n - `DD = DT - GDM`\n - `DC = GDM - DG`\n\n### 3.5 Loss\n\n- **Base**: `SmoothL1Loss` applied to the 3 main targets.\n- **R² Surrogate Loss**:\n - Normalized MSE based on SST calculated from the train split.\n - Importance weights: [0.1, 0.2, 1.0] for `(DG, GDM, DT)`.\n- **Hybrid**:\n - `loss = alpha * SmoothL1 + (1-alpha) * R²_surrogate` (with `alpha=0.5`).\n - Empirically, `SmoothL1` contributed to better training stability.\n\n\n### 3.6 Aux + consistency + log loss\n\n- **Aux**: SmoothL1 on DD/DC.\n- **Log-space L1**: Improves stability for large-scale targets.\n- **Consistency Loss** (`consis_w=0.2`): Enforces alignment between aux predictions and reconstructed values:\n - Induce `DD_aux_pred` to match `(DT−GDM)` and `DC_aux_pred` to match `(GDM−DG)`.\n\n### 3.7 Train-time meta teacher (LUPI) — knowledge distillation\n\n- **Meta input**: NDVI, Height, State/Species, Month + quality meta(sharpness/noise/cluster)\n- **Teacher predicts ratios (sigmoid)**:\n - `DG/DT`, `GDM/DT`, `DD/DT`.\n - `DC/DT` is excluded as redundant (derivable from `GDM/DT − DG/DT`).\n- **Distillation**: Teacher outputs are supervised by target ratios, while student ratios are distilled from the teacher's predictions.\n- **Absolute DD**: We also separately supervise and distill the absolute DD value (`teacher_dd`, `out_dim=1`).\n\n### 3.8 Two-stage fine-tuning\n\n- **Stage 1**: Unfreeze last 1 block (`lr=1e-3`) to stably train the head, MIL, and meta components.\n- **Stage 2**: Unfreeze last 3 blocks (`lr=1e-4`) to fine-tune and push the performance ceiling.\n\n## 4. Results\n- **Best Single Model:**\n - **CV (SGKF 4-fold)**: **0.7615**\n - **Public/Private LB**: **0.78/0.62**\n\n### Table: Single-model milestone progression (Baseline → Best)\n\n| Step | Tiles / img_size | Key change (what improved) | Mean CV32-tile input construction (4×8 global grid)\nFeels like this splitting the image into independent tiles was the reason for the shake down. We observed the same. The public went from 0.68 to 0.77 but private fell from 0.61 to 0.58. 500/500 250/250 100/100 or even 16/16 patch token level were all bad (with more tiles, the public and private moved in opposite directions)
\nThe single image which performed trash on both CV and public LB did better on private. Which I felt is quite surprising since I thought 2000x1000 -> 448x448 had significant information loss 😐
\nEdit: the 0.68 wasn’t even with Dino so it’s insane 😂
", "rawMarkdown": "```\n32-tile input construction (4×8 global grid)\n```\n\nFeels like this splitting the image into independent tiles was the reason for the shake down. We observed the same. The public went from 0.68 to 0.77 but private fell from 0.61 to 0.58. 500/500 250/250 100/100 or even 16/16 patch token level were all bad (with more tiles, the public and private moved in opposite directions)\n\nThe single image which performed trash on both CV and public LB did better on private. Which I felt is quite surprising since I thought 2000x1000 -> 448x448 had significant information loss 😐\n\nEdit: the 0.68 wasn’t even with Dino so it’s insane 😂", "votes": null }, { "id": "3398939", "postDate": "01/29/2026 23:55:49", "content": "Thanks for the great insights.
\n\n\nThe public went from 0.68 to 0.77 but private fell from 0.61 to 0.58.
\n
I learned something new here. Empirically, it seems the tile-based approach hurt private LB performance.\nDo you have any tips on how we could have noticed or expected this before the private scores came out? In my experiments, CV and public LB were pretty well correlated.
", "rawMarkdown": "Thanks for the great insights.\n\n> The public went from 0.68 to 0.77 but private fell from 0.61 to 0.58.\n\nI learned something new here. Empirically, it seems the tile-based approach hurt private LB performance.\nDo you have any tips on how we could have noticed or expected this before the private scores came out? In my experiments, CV and public LB were pretty well correlated.", "votes": null }, { "id": "3398948", "postDate": "01/30/2026 00:31:44", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F30633138e205f82348fb403dabbb2ead%2FScreenshot%202026-01-30%20at%2007.29.49.png?generation=1769733019317622&alt=media\")
\nOh, for me, 8 tiles is much worse than 2 tiles on the private leaderboard, even though the public scores are the same.
I guess lots of the top teams on the public leaderboard used the same tile-level chunking method, many of them dropped by similar magnitude.
\nI wonder is it because overprediction of clovers?
", "rawMarkdown": "I guess lots of the top teams on the public leaderboard used the same tile-level chunking method, many of them dropped by similar magnitude.\n\nI wonder is it because overprediction of clovers?", "votes": null }, { "id": "3399013", "postDate": "01/30/2026 03:56:45", "content": "I did not employ any tiles-level feature aggregation, predict each biomass targets for each tile then sum all (hopefully prevent overfitting). But more tiles, CV decreases which I can't find way to recover the accuracy. I mean, global context is still important, model just looking at overall shape/semantic and guess, so DINOv3 VIT (but not ConvNeXT) dominates this competition.\nHere are some local CV results on frozen backbone, did not submit to LB:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F52158752b421ab748f27ac6e6f8bd2b9%2FScreenshot%202026-01-30%20at%2010.52.03.png?generation=1769745330703345&alt=media\")
Yeah, when I submitted the 8-tile solution to the public leaderboard, the score dropped a bit, which is why I didn’t choose the 8-tile solution as my main approach. I used the 2-tile solution as the primary one and added the 8-tile solution into the ensemble, but it actually dragged my ensemble score down 😅
", "rawMarkdown": "Yeah, when I submitted the 8-tile solution to the public leaderboard, the score dropped a bit, which is why I didn’t choose the 8-tile solution as my main approach. I used the 2-tile solution as the primary one and added the 8-tile solution into the ensemble, but it actually dragged my ensemble score down 😅", "votes": null }, { "id": "3399031", "postDate": "01/30/2026 04:46:37", "content": "Sorry to see you guys took such a big drop. Lots of things to learn about small sample size model building. Have not looked at the results the hosts got with meta only data a decade ago, but seems like vision did not add as much as they might have hoped.
\nThe images were pretty crappy - hopefully if you do a vision model you don't generate low quality images and hope they \"generalize\" well. The description of how the weights of each of the labels were done also seemed pretty sketchy to me. Probably involved in several thousand experiments over my working years and I made the comment at the start of this competition that it was a pretty poor dataset.
\nRegardless did learn a lot myself about modeling with such a small number of images and had fun most of the 3 months. Ended up spending almost $3000 to upgrade my computes, so really committed to getting an ROI over the rest of this year :)
", "rawMarkdown": "Sorry to see you guys took such a big drop. Lots of things to learn about small sample size model building. Have not looked at the results the hosts got with meta only data a decade ago, but seems like vision did not add as much as they might have hoped.\n\nThe images were pretty crappy - hopefully if you do a vision model you don't generate low quality images and hope they \"generalize\" well. The description of how the weights of each of the labels were done also seemed pretty sketchy to me. Probably involved in several thousand experiments over my working years and I made the comment at the start of this competition that it was a pretty poor dataset. \n\nRegardless did learn a lot myself about modeling with such a small number of images and had fun most of the 3 months. Ended up spending almost $3000 to upgrade my computes, so really committed to getting an ROI over the rest of this year :)", "votes": null }, { "id": "3399274", "postDate": "01/30/2026 14:44:36", "content": "Ahh was it really tiles ,, we are testing with other folds if it was folds or tiles :( .
", "rawMarkdown": "Ahh was it really tiles ,, we are testing with other folds if it was folds or tiles :( .", "votes": null }, { "id": "3399297", "postDate": "01/30/2026 15:49:30", "content": "In my experiments, 32 tile-base sum apporach improve my lb and cv significantly, but I think this method lost the global feature. I try many method to fuse local and global information, but still can not get good result. My best PB single model use self-supervised learning and global feature to predict the local tile position, but get low LB. I still feel that the dataset for this competition might be relatively small, resulting in a certain level of randomness in the outcomes. Moreover, some biomass information cannot be determined solely from images.\nSo, your method might actually be more effective with larger datasets.
", "rawMarkdown": "In my experiments, 32 tile-base sum apporach improve my lb and cv significantly, but I think this method lost the global feature. I try many method to fuse local and global information, but still can not get good result. My best PB single model use self-supervised learning and global feature to predict the local tile position, but get low LB. I still feel that the dataset for this competition might be relatively small, resulting in a certain level of randomness in the outcomes. Moreover, some biomass information cannot be determined solely from images.\nSo, your method might actually be more effective with larger datasets.", "votes": null }, { "id": "3399310", "postDate": "01/30/2026 16:02:33", "content": "I also tried to do something like this with 8 patches. I got it upto 0.6,1 but I did not understand ViT in its technical depth yet, so I gave up on this early and moved to a simpler structure.
\nI basically tried to create a first transformer structure, which was pretrained and frozen. Then created another transformer layer, which treated the first transformer patch output as input patches and predicted the final feats to which I attached the heads.
\nclass BioMassViTRegressor(nn.Module):\n def __init__(\n self,\n backbone: str,\n weights: Optional[Any] = None,\n num_targets: int = 4,\n drop_rate: float = 0.3,\n patch_w: int = 4,\n patch_h: int = 2,\n cp: bool = True,\n ):\n super().__init__()\n\n # save the attributes\n self.num_targets = num_targets\n self.num_patches = patch_w * patch_h\n self.cp = cp\n\n # build initial patch creator for processing high res image\n self.patchifier = Rearrange(\"n c (h ph) (w pw) -> n (h w) c ph pw\", h=patch_h, w=patch_w)\n\n # build backbone layers\n backbone, out_dim = self.build_backbone(backbone, weights=weights)\n self.backbone = backbone\n\n # build local patch compressor (Attentive Pooling)\n self.compressor = PatchCompressor(out_dim)\n\n # global transformer aggregator (~1.5M params)\n self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches, out_dim))\n encoder_layer = nn.TransformerEncoderLayer(\n d_model=out_dim, nhead=self.num_patches, dim_feedforward=out_dim * 2, dropout=drop_rate, batch_first=True\n )\n self.global_transformer = nn.TransformerEncoder(encoder_layer, num_layers=1)\n\n # 5. Final Regression Head\n self.head = nn.Sequential(\n nn.LayerNorm(out_dim),\n nn.Linear(out_dim, out_dim // 2),\n nn.ReLU(),\n nn.Dropout(drop_rate),\n nn.Linear(out_dim // 2, num_targets),\n )\n\n def freeze_backbone(self, stem: bool = True, stages: int = 4):\n # unfreeze everything first\n for param in self.backbone.parameters():\n param.requires_grad = True\n\n # apply stem freeze if requested\n if stem:\n for param in self.backbone.patch_embed.parameters():\n param.requires_grad = False\n\n # apply stage-wise freeze\n if stages != 0:\n for stage_idx in range(stages):\n block_start_idx = stage_idx * 3\n for block_idx in range(block_start_idx, block_start_idx + 3):\n for param in self.backbone.blocks[block_idx].parameters():\n param.requires_grad = False\n\n def build_backbone(self, backbone: str, weights: Optional[str], **kwargs):\n if backbone.startswith(\"vit\"):\n if weights is None:\n model = timm.create_model(backbone, pretrained=False, num_classes=0, **kwargs)\n elif isinstance(weights, str) and weights.startswith(\"./\") == False:\n model = timm.create_model(f\"{backbone}.{weights}\", pretrained=True, num_classes=0, **kwargs)\n elif isinstance(weights, str) and weights.startswith(\"./\") == True:\n model = timm.create_model(backbone, pretrained=None, checkpoint_path=weights, num_classes=0, **kwargs)\n\n # mask the head and return the backbone\n model.head = nn.Identity()\n return model, model.head_hidden_size\n\n else:\n raise ValueError(\"Unknown backbone requested\")\n\n def forward(self, x: torch.Tensor) -> torch.Tensor:\n batch_size = x.shape[0]\n\n # create patches and extract patch features\n patch_feats = []\n if len(x.shape) == 4:\n x = self.patchifier(x)\n for patch_idx in range(self.num_patches):\n feat_patch = checkpoint(self.backbone.forward_features, x[:, patch_idx, :, :], use_reentrant=False)\n patch_feats.append(feat_patch)\n\n # stitching individual patches in one\n x = patch_feats\n x = torch.stack(patch_feats, dim=1)\n x = x.view(-1, x.shape[2], x.shape[3])\n x = self.compressor(x)\n x = x.view(batch_size, self.num_patches, -1)\n x = x + self.pos_embed\n x = self.global_transformer(x)\n x = x.mean(dim=1)\n return self.head(x)\n32-tile input construction (4×8 global grid)\nFeels like this splitting the image into independent tiles was the reason for the shake down. We observed the same. The public went from 0.68 to 0.77 but private fell from 0.61 to 0.58. 500/500 250/250 100/100 or even 16/16 patch token level were all bad (with more tiles, the public and private moved in opposite directions)
\nThe single image which performed trash on both CV and public LB did better on private. Which I felt is quite surprising since I thought 2000x1000 -> 448x448 had significant information loss 😐
\nEdit: the 0.68 wasn’t even with Dino so it’s insane 😂
", "votes": null, "replies": [ { "id": 3398939, "author_name": "ayaanjang", "author_url": "", "post_date": "01/29/2026 23:55:49", "content": "Thanks for the great insights.
\n\n\nThe public went from 0.68 to 0.77 but private fell from 0.61 to 0.58.
\n
I learned something new here. Empirically, it seems the tile-based approach hurt private LB performance.\nDo you have any tips on how we could have noticed or expected this before the private scores came out? In my experiments, CV and public LB were pretty well correlated.
", "votes": null, "replies": [ { "id": 3398948, "author_name": "quan0095", "author_url": "", "post_date": "01/30/2026 00:31:44", "content": "\nOh, for me, 8 tiles is much worse than 2 tiles on the private leaderboard, even though the public scores are the same.
I guess lots of the top teams on the public leaderboard used the same tile-level chunking method, many of them dropped by similar magnitude.
\nI wonder is it because overprediction of clovers?
", "votes": null, "replies": [] } ] }, { "id": 3399013, "author_name": "dangnh0611", "author_url": "", "post_date": "01/30/2026 03:56:45", "content": "I did not employ any tiles-level feature aggregation, predict each biomass targets for each tile then sum all (hopefully prevent overfitting). But more tiles, CV decreases which I can't find way to recover the accuracy. I mean, global context is still important, model just looking at overall shape/semantic and guess, so DINOv3 VIT (but not ConvNeXT) dominates this competition.\nHere are some local CV results on frozen backbone, did not submit to LB:\n
Yeah, when I submitted the 8-tile solution to the public leaderboard, the score dropped a bit, which is why I didn’t choose the 8-tile solution as my main approach. I used the 2-tile solution as the primary one and added the 8-tile solution into the ensemble, but it actually dragged my ensemble score down 😅
", "votes": null, "replies": [ { "id": 3399274, "author_name": "gauravbrills", "author_url": "", "post_date": "01/30/2026 14:44:36", "content": "Ahh was it really tiles ,, we are testing with other folds if it was folds or tiles :( .
", "votes": null, "replies": [] } ] }, { "id": 3399297, "author_name": "forgakki", "author_url": "", "post_date": "01/30/2026 15:49:30", "content": "In my experiments, 32 tile-base sum apporach improve my lb and cv significantly, but I think this method lost the global feature. I try many method to fuse local and global information, but still can not get good result. My best PB single model use self-supervised learning and global feature to predict the local tile position, but get low LB. I still feel that the dataset for this competition might be relatively small, resulting in a certain level of randomness in the outcomes. Moreover, some biomass information cannot be determined solely from images.\nSo, your method might actually be more effective with larger datasets.
", "votes": null, "replies": [] } ] } ] }, { "id": 3399031, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/30/2026 04:46:37", "content": "Sorry to see you guys took such a big drop. Lots of things to learn about small sample size model building. Have not looked at the results the hosts got with meta only data a decade ago, but seems like vision did not add as much as they might have hoped.
\nThe images were pretty crappy - hopefully if you do a vision model you don't generate low quality images and hope they \"generalize\" well. The description of how the weights of each of the labels were done also seemed pretty sketchy to me. Probably involved in several thousand experiments over my working years and I made the comment at the start of this competition that it was a pretty poor dataset.
\nRegardless did learn a lot myself about modeling with such a small number of images and had fun most of the 3 months. Ended up spending almost $3000 to upgrade my computes, so really committed to getting an ROI over the rest of this year :)
", "votes": null, "replies": [] }, { "id": 3399310, "author_name": "pushpakbhoge", "author_url": "", "post_date": "01/30/2026 16:02:33", "content": "I also tried to do something like this with 8 patches. I got it upto 0.6,1 but I did not understand ViT in its technical depth yet, so I gave up on this early and moved to a simpler structure.
\nI basically tried to create a first transformer structure, which was pretrained and frozen. Then created another transformer layer, which treated the first transformer patch output as input patches and predicted the final feats to which I attached the heads.
\nclass BioMassViTRegressor(nn.Module):\n def __init__(\n self,\n backbone: str,\n weights: Optional[Any] = None,\n num_targets: int = 4,\n drop_rate: float = 0.3,\n patch_w: int = 4,\n patch_h: int = 2,\n cp: bool = True,\n ):\n super().__init__()\n\n # save the attributes\n self.num_targets = num_targets\n self.num_patches = patch_w * patch_h\n self.cp = cp\n\n # build initial patch creator for processing high res image\n self.patchifier = Rearrange(\"n c (h ph) (w pw) -> n (h w) c ph pw\", h=patch_h, w=patch_w)\n\n # build backbone layers\n backbone, out_dim = self.build_backbone(backbone, weights=weights)\n self.backbone = backbone\n\n # build local patch compressor (Attentive Pooling)\n self.compressor = PatchCompressor(out_dim)\n\n # global transformer aggregator (~1.5M params)\n self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches, out_dim))\n encoder_layer = nn.TransformerEncoderLayer(\n d_model=out_dim, nhead=self.num_patches, dim_feedforward=out_dim * 2, dropout=drop_rate, batch_first=True\n )\n self.global_transformer = nn.TransformerEncoder(encoder_layer, num_layers=1)\n\n # 5. Final Regression Head\n self.head = nn.Sequential(\n nn.LayerNorm(out_dim),\n nn.Linear(out_dim, out_dim // 2),\n nn.ReLU(),\n nn.Dropout(drop_rate),\n nn.Linear(out_dim // 2, num_targets),\n )\n\n def freeze_backbone(self, stem: bool = True, stages: int = 4):\n # unfreeze everything first\n for param in self.backbone.parameters():\n param.requires_grad = True\n\n # apply stem freeze if requested\n if stem:\n for param in self.backbone.patch_embed.parameters():\n param.requires_grad = False\n\n # apply stage-wise freeze\n if stages != 0:\n for stage_idx in range(stages):\n block_start_idx = stage_idx * 3\n for block_idx in range(block_start_idx, block_start_idx + 3):\n for param in self.backbone.blocks[block_idx].parameters():\n param.requires_grad = False\n\n def build_backbone(self, backbone: str, weights: Optional[str], **kwargs):\n if backbone.startswith(\"vit\"):\n if weights is None:\n model = timm.create_model(backbone, pretrained=False, num_classes=0, **kwargs)\n elif isinstance(weights, str) and weights.startswith(\"./\") == False:\n model = timm.create_model(f\"{backbone}.{weights}\", pretrained=True, num_classes=0, **kwargs)\n elif isinstance(weights, str) and weights.startswith(\"./\") == True:\n model = timm.create_model(backbone, pretrained=None, checkpoint_path=weights, num_classes=0, **kwargs)\n\n # mask the head and return the backbone\n model.head = nn.Identity()\n return model, model.head_hidden_size\n\n else:\n raise ValueError(\"Unknown backbone requested\")\n\n def forward(self, x: torch.Tensor) -> torch.Tensor:\n batch_size = x.shape[0]\n\n # create patches and extract patch features\n patch_feats = []\n if len(x.shape) == 4:\n x = self.patchifier(x)\n for patch_idx in range(self.num_patches):\n feat_patch = checkpoint(self.backbone.forward_features, x[:, patch_idx, :, :], use_reentrant=False)\n patch_feats.append(feat_patch)\n\n # stitching individual patches in one\n x = patch_feats\n x = torch.stack(patch_feats, dim=1)\n x = x.view(-1, x.shape[2], x.shape[3])\n x = self.compressor(x)\n x = x.view(batch_size, self.num_patches, -1)\n x = x + self.pos_embed\n x = self.global_transformer(x)\n x = x.mean(dim=1)\n return self.head(x)\n