![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16529018%2F958a480e25d2ef209ca2c5746141d259%2Fbaseline.jpg?generation=1769653060849832&alt=media\")
With dinov3 backbone, I create a simple baseline model with lb 0.72. And then based on the baseline model, I desiged 3 types of models : ordinary regression model with softrouter head (just like a kind of MoE head), median regression with section post-processing and total-ratio regression with custom loss. Besides, I also use a head to predict the survival probability of clover, and then use this probability to correct predictions of clover.
\nStratifiedGroupKFold with kf.split(df, y=df.State, groups=df[\"Sampling_Date\"]). It shows relatively good correlationship with public, but seems to have nothing to do with private 😅.
\nvit_huge_plus_patch16_dinov3_qkvb, only finetune last 4 blocks.
\nWith dual streaming, I split the 1000x2000 into 2 images and transform them into 768x768 resolution respectively. After that, I concatenate features from 2 images and input features into 3 linear heads (total, gdm, green). This model get 0.72 in lb and 0.7823 in cv.
\n
For extracting features further, I use Tile to split each image into N grid, and input every grid into backbone one by one. Instead of default avg pooling, I use attention pooling for the output from backbone (B, N_patch, Feat). Then I apply FiLM module into Tile outputs (B, N_grid, Feat) to modulate local features with global context information and finally average N grids feature to get (B, Feat) data for each stream. This model get 0.74 in lb and 0.7847 in cv. When I change the img_size into 960x960, I get 0.76 (lower) in lb and 0.7871 in cv.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16529018%2Fb1439a6651a7b4bebb998137c3659f6f%2Ffilm_tilejpg.jpg?generation=1769654251513432&alt=media\")
In the middle-term stage, I realize that there may be different seasons with various scenes in Australia because of savanna climate. So I designed a soft router regression head based on MoE head, hoping different experts network can solve different climate change. In this head, a lightweight router network dynamically assigns weights to experts based on the input image features. This allows the model to adaptively select the best processing path for different scenarios (e.g., dry vs. wet season, sparse vs. dense vegetation). When I replaced linear heads with this soft router head (num_experts is 4), I get 0.77 in lb and 0.790 in cv.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16529018%2F5649b2798f858bd77ec79ef7593340eb%2Fsoft.jpg?generation=1769654793227735&alt=media\")
What's more, there are about 130 samples whose clover value is 0.0, which will impact model performance badly. Rather than scale the clover value manully, why not let the model learn how to scale it. So I also add another soft router head to predict the probability of existence, then I use the probability to correct clover prediction (prob*clover_pred). I get 0.77 (higher) in lb and 0.795 in cv.
\nTo address the heteroscedasticity and potential outliers in biomass data, I also use median regression to get robuster prediction. For each regression head, it will output 3 value (q0.1, delta_1, delta_2). After that I use torch.cum() to get real quantile values (q0.1, q0.5, q0.9). I think preidcting increment is eaiser than precise values. Then I calculate quantile loss (pinball loss) with 3 quantile values. During inference, I use the predicted median (q0.5) as the final submission value. The interval width (q0.9 - 0.1) serves as an uncertainty measure, which is used for uncertainty-weighted model ensembling. I get 0.77 (lower) in lb and 0.7899 in cv.
\nclass QuantileLoss(nn.Module):\n def __init__(self, quantiles=[0.1, 0.5, 0.9]):\n super().__init__()\n self.quantiles = quantiles\n\n def forward(self, preds, target):\n loss = 0.0\n for i, q in enumerate(self.quantiles):\n # [Target1_q1, Target1_q2... Target2_q1...] \n\n pred_q = preds[:, [0+i, 3+i, 6+i]]\n\n errors = target - pred_q\n # Pinball Loss: max((q-1)*error, q*error)\n loss += torch.max((q-1) * errors, q * errors).mean()\n\n return loss\n\n def uncertainty_weighted_ensemble(predictions_list, uncertainties_list):\n # (M, N, T)\n preds = np.stack(predictions_list, axis=0)\n uncerts = np.stack(uncertainties_list, axis=0)\n\n uncerts = np.maximum(uncerts, 1e-6)\n\n weights = 1.0 / uncerts\n\n weights_norm = weights / weights.sum(axis=0, keepdims=True)\n\n ensemble_pred = (weights_norm * preds).sum(axis=0)\n\n return ensemble_pred\nThis types model only consist 2 regression heads: total_head to predict toal values(B, 1) and ratio_head to predict the ratios of each component (dead, clover, green). The loss function of these output contain 3 parts. SmoothL1 loss in log space for total, KL Divergence between predicted ratio and true ratio, and SmoothL1 loss for true values of 3 components. I get 0.76(higher) in lb and 0.7905 in cv.
\nclass CompositionalBiomassLoss(nn.Module):\n def __init__(self, eps=1e-6):\n super().__init__()\n self.eps = eps\n self.weights = {'green': 0.1, 'dead': 0.1, 'clover': 0.1, 'gdm': 0.2, 'total': 0.5}\n\n def forward(self, pred_total, pred_logits, targets):\n gt_green = targets[:, 1].unsqueeze(1)\n gt_dead = targets[:, 2].unsqueeze(1)\n gt_clover= targets[:, 3].unsqueeze(1)\n gt_total = targets[:, 0].unsqueeze(1)\n\n gt_components = torch.cat([gt_green, gt_dead, gt_clover], dim=1) # (B, 3)\n gt_ratios = gt_components / (gt_total + self.eps)\n\n loss_total = F.smooth_l1_loss(torch.log1p(pred_total), torch.log1p(gt_total))\n\n\n pred_log_probs = F.log_softmax(pred_logits, dim=1)\n loss_dist = F.kl_div(pred_log_probs, gt_ratios, reduction='batchmean')\n\n pred_ratios = F.softmax(pred_logits, dim=1) # (B, 3)\n\n recon_green = pred_total * pred_ratios[:, 0:1]\n recon_dead = pred_total * pred_ratios[:, 1:2]\n recon_clover = pred_total * pred_ratios[:, 2:3]\n\n l_green = F.smooth_l1_loss(recon_green, gt_green)\n l_dead = F.smooth_l1_loss(recon_dead, gt_dead)\n l_clover = F.smooth_l1_loss(recon_clover, gt_clover)\n\n loss_recon = (self.weights['green'] * l_green + \n self.weights['dead'] * l_dead + \n self.weights['clover']* l_clover)\n\n total_loss = 1.0 * loss_total + 0.5 * loss_dist + 1.0 * loss_recon\n\n return total_loss\nA.Compose(\n [\n A.Resize(CFG.IMG_SIZE, CFG.IMG_SIZE),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=1.0),\n A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n ToTensorV2(),\n ]\n)\noptimizer: AdamW; \nscheduler: CosineLRScheduler; \nEMA
\nIMG_SIZE=960\nLR_INIT = 1e-4\nWD = 3e-3\nepochs_warmup = 20\nwarmup_lr_init = 5e-5\nlr_min = 1e-6\ntile_grid=(2,2)\nI use this notebook as my reference.
\nShaking down really frustrate us, but this is kaggle, right? 🥲
\nI generate the images in the article by Nano Banana Pro, so the images might be a little blurry. If you want more details, you can check my training code.I will release them later.
", "messages": [ { "id": "3398375", "postDate": "01/29/2026 03:32:08", "content": "With dinov3 backbone, I create a simple baseline model with lb 0.72. And then based on the baseline model, I desiged 3 types of models : ordinary regression model with softrouter head (just like a kind of MoE head), median regression with section post-processing and total-ratio regression with custom loss. Besides, I also use a head to predict the survival probability of clover, and then use this probability to correct predictions of clover.
\nStratifiedGroupKFold with kf.split(df, y=df.State, groups=df[\"Sampling_Date\"]). It shows relatively good correlationship with public, but seems to have nothing to do with private 😅.
\nvit_huge_plus_patch16_dinov3_qkvb, only finetune last 4 blocks.
\nWith dual streaming, I split the 1000x2000 into 2 images and transform them into 768x768 resolution respectively. After that, I concatenate features from 2 images and input features into 3 linear heads (total, gdm, green). This model get 0.72 in lb and 0.7823 in cv.
\n
For extracting features further, I use Tile to split each image into N grid, and input every grid into backbone one by one. Instead of default avg pooling, I use attention pooling for the output from backbone (B, N_patch, Feat). Then I apply FiLM module into Tile outputs (B, N_grid, Feat) to modulate local features with global context information and finally average N grids feature to get (B, Feat) data for each stream. This model get 0.74 in lb and 0.7847 in cv. When I change the img_size into 960x960, I get 0.76 (lower) in lb and 0.7871 in cv.
\n
In the middle-term stage, I realize that there may be different seasons with various scenes in Australia because of savanna climate. So I designed a soft router regression head based on MoE head, hoping different experts network can solve different climate change. In this head, a lightweight router network dynamically assigns weights to experts based on the input image features. This allows the model to adaptively select the best processing path for different scenarios (e.g., dry vs. wet season, sparse vs. dense vegetation). When I replaced linear heads with this soft router head (num_experts is 4), I get 0.77 in lb and 0.790 in cv.
\n
What's more, there are about 130 samples whose clover value is 0.0, which will impact model performance badly. Rather than scale the clover value manully, why not let the model learn how to scale it. So I also add another soft router head to predict the probability of existence, then I use the probability to correct clover prediction (prob*clover_pred). I get 0.77 (higher) in lb and 0.795 in cv.
\nTo address the heteroscedasticity and potential outliers in biomass data, I also use median regression to get robuster prediction. For each regression head, it will output 3 value (q0.1, delta_1, delta_2). After that I use torch.cum() to get real quantile values (q0.1, q0.5, q0.9). I think preidcting increment is eaiser than precise values. Then I calculate quantile loss (pinball loss) with 3 quantile values. During inference, I use the predicted median (q0.5) as the final submission value. The interval width (q0.9 - 0.1) serves as an uncertainty measure, which is used for uncertainty-weighted model ensembling. I get 0.77 (lower) in lb and 0.7899 in cv.
\nclass QuantileLoss(nn.Module):\n def __init__(self, quantiles=[0.1, 0.5, 0.9]):\n super().__init__()\n self.quantiles = quantiles\n\n def forward(self, preds, target):\n loss = 0.0\n for i, q in enumerate(self.quantiles):\n # [Target1_q1, Target1_q2... Target2_q1...] \n\n pred_q = preds[:, [0+i, 3+i, 6+i]]\n\n errors = target - pred_q\n # Pinball Loss: max((q-1)*error, q*error)\n loss += torch.max((q-1) * errors, q * errors).mean()\n\n return loss\n\n def uncertainty_weighted_ensemble(predictions_list, uncertainties_list):\n # (M, N, T)\n preds = np.stack(predictions_list, axis=0)\n uncerts = np.stack(uncertainties_list, axis=0)\n\n uncerts = np.maximum(uncerts, 1e-6)\n\n weights = 1.0 / uncerts\n\n weights_norm = weights / weights.sum(axis=0, keepdims=True)\n\n ensemble_pred = (weights_norm * preds).sum(axis=0)\n\n return ensemble_pred\nThis types model only consist 2 regression heads: total_head to predict toal values(B, 1) and ratio_head to predict the ratios of each component (dead, clover, green). The loss function of these output contain 3 parts. SmoothL1 loss in log space for total, KL Divergence between predicted ratio and true ratio, and SmoothL1 loss for true values of 3 components. I get 0.76(higher) in lb and 0.7905 in cv.
\nclass CompositionalBiomassLoss(nn.Module):\n def __init__(self, eps=1e-6):\n super().__init__()\n self.eps = eps\n self.weights = {'green': 0.1, 'dead': 0.1, 'clover': 0.1, 'gdm': 0.2, 'total': 0.5}\n\n def forward(self, pred_total, pred_logits, targets):\n gt_green = targets[:, 1].unsqueeze(1)\n gt_dead = targets[:, 2].unsqueeze(1)\n gt_clover= targets[:, 3].unsqueeze(1)\n gt_total = targets[:, 0].unsqueeze(1)\n\n gt_components = torch.cat([gt_green, gt_dead, gt_clover], dim=1) # (B, 3)\n gt_ratios = gt_components / (gt_total + self.eps)\n\n loss_total = F.smooth_l1_loss(torch.log1p(pred_total), torch.log1p(gt_total))\n\n\n pred_log_probs = F.log_softmax(pred_logits, dim=1)\n loss_dist = F.kl_div(pred_log_probs, gt_ratios, reduction='batchmean')\n\n pred_ratios = F.softmax(pred_logits, dim=1) # (B, 3)\n\n recon_green = pred_total * pred_ratios[:, 0:1]\n recon_dead = pred_total * pred_ratios[:, 1:2]\n recon_clover = pred_total * pred_ratios[:, 2:3]\n\n l_green = F.smooth_l1_loss(recon_green, gt_green)\n l_dead = F.smooth_l1_loss(recon_dead, gt_dead)\n l_clover = F.smooth_l1_loss(recon_clover, gt_clover)\n\n loss_recon = (self.weights['green'] * l_green + \n self.weights['dead'] * l_dead + \n self.weights['clover']* l_clover)\n\n total_loss = 1.0 * loss_total + 0.5 * loss_dist + 1.0 * loss_recon\n\n return total_loss\nA.Compose(\n [\n A.Resize(CFG.IMG_SIZE, CFG.IMG_SIZE),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=1.0),\n A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n ToTensorV2(),\n ]\n)\noptimizer: AdamW; \nscheduler: CosineLRScheduler; \nEMA
\nIMG_SIZE=960\nLR_INIT = 1e-4\nWD = 3e-3\nepochs_warmup = 20\nwarmup_lr_init = 5e-5\nlr_min = 1e-6\ntile_grid=(2,2)\nI use this notebook as my reference.
\nShaking down really frustrate us, but this is kaggle, right? 🥲
\nI generate the images in the article by Nano Banana Pro, so the images might be a little blurry. If you want more details, you can check my training code.I will release them later.
", "rawMarkdown": "# TL;DR\nWith dinov3 backbone, I create a simple baseline model with lb 0.72. And then based on the baseline model, I desiged 3 types of models : ordinary regression model with softrouter head (just like a kind of MoE head), median regression with section post-processing and total-ratio regression with custom loss. Besides, I also use a head to predict the survival probability of clover, and then use this probability to correct predictions of clover.\n\n# Cross-validation strategy\nStratifiedGroupKFold with kf.split(df, y=df.State, groups=df[\"Sampling_Date\"]). It shows relatively good correlationship with public, but seems to have nothing to do with private 😅.\n\n# Model Architecture\n\n## Backbone\nvit_huge_plus_patch16_dinov3_qkvb, only finetune last 4 blocks.\n\n## Baseline\nWith dual streaming, I split the 1000x2000 into 2 images and transform them into 768x768 resolution respectively. After that, I concatenate features from 2 images and input features into 3 linear heads (total, gdm, green). This model get 0.72 in lb and 0.7823 in cv.\n\n\n\n## FiLM + Tile + Attention Pooling\nFor extracting features further, I use Tile to split each image into N grid, and input every grid into backbone one by one. Instead of default avg pooling, I use attention pooling for the output from backbone (B, N_patch, Feat). Then I apply FiLM module into Tile outputs (B, N_grid, Feat) to modulate local features with global context information and finally average N grids feature to get (B, Feat) data for each stream. This model get 0.74 in lb and 0.7847 in cv. When I change the img_size into 960x960, I get 0.76 (lower) in lb and 0.7871 in cv.\n\n\n \n## Soft Router MoE Head + Gate probability.\nIn the middle-term stage, I realize that there may be different seasons with various scenes in Australia because of savanna climate. So I designed a soft router regression head based on MoE head, hoping different experts network can solve different climate change. In this head, a lightweight router network dynamically assigns weights to experts based on the input image features. This allows the model to adaptively select the best processing path for different scenarios (e.g., dry vs. wet season, sparse vs. dense vegetation). When I replaced linear heads with this soft router head (num_experts is 4), I get 0.77 in lb and 0.790 in cv.\n\n\n\nWhat's more, there are about 130 samples whose clover value is 0.0, which will impact model performance badly. Rather than scale the clover value manully, why not let the model learn how to scale it. So I also add another soft router head to predict the probability of existence, then I use the probability to correct clover prediction (prob*clover_pred). I get 0.77 (higher) in lb and 0.795 in cv.\n\n## Median Regression+section post-processing\nTo address the heteroscedasticity and potential outliers in biomass data, I also use median regression to get robuster prediction. For each regression head, it will output 3 value (q0.1, delta_1, delta_2). After that I use torch.cum() to get real quantile values (q0.1, q0.5, q0.9). I think preidcting increment is eaiser than precise values. Then I calculate quantile loss (pinball loss) with 3 quantile values. During inference, I use the predicted median (q0.5) as the final submission value. The interval width (q0.9 - 0.1) serves as an uncertainty measure, which is used for uncertainty-weighted model ensembling. I get 0.77 (lower) in lb and 0.7899 in cv.\n\n```\nclass QuantileLoss(nn.Module):\n def __init__(self, quantiles=[0.1, 0.5, 0.9]):\n super().__init__()\n self.quantiles = quantiles\n\n def forward(self, preds, target):\n loss = 0.0\n for i, q in enumerate(self.quantiles):\n # [Target1_q1, Target1_q2... Target2_q1...] \n \n pred_q = preds[:, [0+i, 3+i, 6+i]]\n \n errors = target - pred_q\n # Pinball Loss: max((q-1)*error, q*error)\n loss += torch.max((q-1) * errors, q * errors).mean()\n \n return loss\n\n def uncertainty_weighted_ensemble(predictions_list, uncertainties_list):\n # (M, N, T)\n preds = np.stack(predictions_list, axis=0)\n uncerts = np.stack(uncertainties_list, axis=0)\n \n uncerts = np.maximum(uncerts, 1e-6)\n \n weights = 1.0 / uncerts\n \n weights_norm = weights / weights.sum(axis=0, keepdims=True)\n \n ensemble_pred = (weights_norm * preds).sum(axis=0)\n \n return ensemble_pred\n```\n\n## Total-ratio Regression.\nThis types model only consist 2 regression heads: total_head to predict toal values(B, 1) and ratio_head to predict the ratios of each component (dead, clover, green). The loss function of these output contain 3 parts. SmoothL1 loss in log space for total, KL Divergence between predicted ratio and true ratio, and SmoothL1 loss for true values of 3 components. I get 0.76(higher) in lb and 0.7905 in cv.\n\n```\nclass CompositionalBiomassLoss(nn.Module):\n def __init__(self, eps=1e-6):\n super().__init__()\n self.eps = eps\n self.weights = {'green': 0.1, 'dead': 0.1, 'clover': 0.1, 'gdm': 0.2, 'total': 0.5}\n\n def forward(self, pred_total, pred_logits, targets):\n gt_green = targets[:, 1].unsqueeze(1)\n gt_dead = targets[:, 2].unsqueeze(1)\n gt_clover= targets[:, 3].unsqueeze(1)\n gt_total = targets[:, 0].unsqueeze(1)\n\n gt_components = torch.cat([gt_green, gt_dead, gt_clover], dim=1) # (B, 3)\n gt_ratios = gt_components / (gt_total + self.eps)\n\n loss_total = F.smooth_l1_loss(torch.log1p(pred_total), torch.log1p(gt_total))\n\n\n pred_log_probs = F.log_softmax(pred_logits, dim=1)\n loss_dist = F.kl_div(pred_log_probs, gt_ratios, reduction='batchmean')\n\n pred_ratios = F.softmax(pred_logits, dim=1) # (B, 3)\n\n recon_green = pred_total * pred_ratios[:, 0:1]\n recon_dead = pred_total * pred_ratios[:, 1:2]\n recon_clover = pred_total * pred_ratios[:, 2:3]\n\n l_green = F.smooth_l1_loss(recon_green, gt_green)\n l_dead = F.smooth_l1_loss(recon_dead, gt_dead)\n l_clover = F.smooth_l1_loss(recon_clover, gt_clover)\n \n loss_recon = (self.weights['green'] * l_green + \n self.weights['dead'] * l_dead + \n self.weights['clover']* l_clover)\n\n total_loss = 1.0 * loss_total + 0.5 * loss_dist + 1.0 * loss_recon\n \n return total_loss\n```\n\n## Augmentation\n```\nA.Compose(\n [\n A.Resize(CFG.IMG_SIZE, CFG.IMG_SIZE),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=1.0),\n A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n ToTensorV2(),\n ]\n)\n```\n\n## Other configuration.\noptimizer: AdamW; \nscheduler: CosineLRScheduler; \nEMA\n```\nIMG_SIZE=960\nLR_INIT = 1e-4\nWD = 3e-3\nepochs_warmup = 20\nwarmup_lr_init = 5e-5\nlr_min = 1e-6\ntile_grid=(2,2)\n```\n\nI use this [notebook ](https://www.kaggle.com/code/gothamjocker/lb-0-65-dino-v2-tiled) as my reference.\n\n# What didn't work\n- Larger model (vit_7b_dinov3)\n- LoRA\n- Finetune more blocks\n- Test-time training. Honestly, I think oneline training will be an important trick for this small dataset competition. But no matter how hard I had tried, I can't get an ideal result. Maybe I miss something :(\n- Single Stream\n- Larger image resolution (1024x1024, 1280x1280, 1536x1536)\n- More grids for Tile\n- TTA (So weird)\n\n# Ending\nShaking down really frustrate us, but this is kaggle, right? 🥲\n\n----------------------------------------------------------------------------------------------\n\nI generate the images in the article by Nano Banana Pro, so the images might be a little blurry. If you want more details, you can check my training code.I will release them later.", "votes": null }, { "id": "3398437", "postDate": "01/29/2026 06:28:39", "content": "My architecture is very similar to yours, except that I added classifiers and embeddings for State, Season, and Species (customizable). This might be the reason why my model is severely overfitting😭.
", "rawMarkdown": "My architecture is very similar to yours, except that I added classifiers and embeddings for State, Season, and Species (customizable). This might be the reason why my model is severely overfitting😭.", "votes": null }, { "id": "3398682", "postDate": "01/29/2026 14:43:06", "content": "i tried adding classifiers & embeddings for metadata too. Also tried distill etc. But none worked. Eventually I concluded that metadata are just misleading noise for models to learn. I ended up not using any of them for my submission…
", "rawMarkdown": "i tried adding classifiers & embeddings for metadata too. Also tried distill etc. But none worked. Eventually I concluded that metadata are just misleading noise for models to learn. I ended up not using any of them for my submission…", "votes": null } ], "comments": [ { "id": 3398437, "author_name": "wukaiqi", "author_url": "", "post_date": "01/29/2026 06:28:39", "content": "My architecture is very similar to yours, except that I added classifiers and embeddings for State, Season, and Species (customizable). This might be the reason why my model is severely overfitting😭.
", "votes": null, "replies": [ { "id": 3398682, "author_name": "yannan90", "author_url": "", "post_date": "01/29/2026 14:43:06", "content": "i tried adding classifiers & embeddings for metadata too. Also tried distill etc. But none worked. Eventually I concluded that metadata are just misleading noise for models to learn. I ended up not using any of them for my submission…
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3398375": "# TL;DR\nWith dinov3 backbone, I create a simple baseline model with lb 0.72. And then based on the baseline model, I desiged 3 types of models : ordinary regression model with softrouter head (just like a kind of MoE head), median regression with section post-processing and total-ratio regression with custom loss. Besides, I also use a head to predict the survival probability of clover, and then use this probability to correct predictions of clover.\n\n# Cross-validation strategy\nStratifiedGroupKFold with kf.split(df, y=df.State, groups=df[\"Sampling_Date\"]). It shows relatively good correlationship with public, but seems to have nothing to do with private 😅.\n\n# Model Architecture\n\n## Backbone\nvit_huge_plus_patch16_dinov3_qkvb, only finetune last 4 blocks.\n\n## Baseline\nWith dual streaming, I split the 1000x2000 into 2 images and transform them into 768x768 resolution respectively. After that, I concatenate features from 2 images and input features into 3 linear heads (total, gdm, green). This model get 0.72 in lb and 0.7823 in cv.\n\n\n\n## FiLM + Tile + Attention Pooling\nFor extracting features further, I use Tile to split each image into N grid, and input every grid into backbone one by one. Instead of default avg pooling, I use attention pooling for the output from backbone (B, N_patch, Feat). Then I apply FiLM module into Tile outputs (B, N_grid, Feat) to modulate local features with global context information and finally average N grids feature to get (B, Feat) data for each stream. This model get 0.74 in lb and 0.7847 in cv. When I change the img_size into 960x960, I get 0.76 (lower) in lb and 0.7871 in cv.\n\n\n \n## Soft Router MoE Head + Gate probability.\nIn the middle-term stage, I realize that there may be different seasons with various scenes in Australia because of savanna climate. So I designed a soft router regression head based on MoE head, hoping different experts network can solve different climate change. In this head, a lightweight router network dynamically assigns weights to experts based on the input image features. This allows the model to adaptively select the best processing path for different scenarios (e.g., dry vs. wet season, sparse vs. dense vegetation). When I replaced linear heads with this soft router head (num_experts is 4), I get 0.77 in lb and 0.790 in cv.\n\n\n\nWhat's more, there are about 130 samples whose clover value is 0.0, which will impact model performance badly. Rather than scale the clover value manully, why not let the model learn how to scale it. So I also add another soft router head to predict the probability of existence, then I use the probability to correct clover prediction (prob*clover_pred). I get 0.77 (higher) in lb and 0.795 in cv.\n\n## Median Regression+section post-processing\nTo address the heteroscedasticity and potential outliers in biomass data, I also use median regression to get robuster prediction. For each regression head, it will output 3 value (q0.1, delta_1, delta_2). After that I use torch.cum() to get real quantile values (q0.1, q0.5, q0.9). I think preidcting increment is eaiser than precise values. Then I calculate quantile loss (pinball loss) with 3 quantile values. During inference, I use the predicted median (q0.5) as the final submission value. The interval width (q0.9 - 0.1) serves as an uncertainty measure, which is used for uncertainty-weighted model ensembling. I get 0.77 (lower) in lb and 0.7899 in cv.\n\n```\nclass QuantileLoss(nn.Module):\n def __init__(self, quantiles=[0.1, 0.5, 0.9]):\n super().__init__()\n self.quantiles = quantiles\n\n def forward(self, preds, target):\n loss = 0.0\n for i, q in enumerate(self.quantiles):\n # [Target1_q1, Target1_q2... Target2_q1...] \n \n pred_q = preds[:, [0+i, 3+i, 6+i]]\n \n errors = target - pred_q\n # Pinball Loss: max((q-1)*error, q*error)\n loss += torch.max((q-1) * errors, q * errors).mean()\n \n return loss\n\n def uncertainty_weighted_ensemble(predictions_list, uncertainties_list):\n # (M, N, T)\n preds = np.stack(predictions_list, axis=0)\n uncerts = np.stack(uncertainties_list, axis=0)\n \n uncerts = np.maximum(uncerts, 1e-6)\n \n weights = 1.0 / uncerts\n \n weights_norm = weights / weights.sum(axis=0, keepdims=True)\n \n ensemble_pred = (weights_norm * preds).sum(axis=0)\n \n return ensemble_pred\n```\n\n## Total-ratio Regression.\nThis types model only consist 2 regression heads: total_head to predict toal values(B, 1) and ratio_head to predict the ratios of each component (dead, clover, green). The loss function of these output contain 3 parts. SmoothL1 loss in log space for total, KL Divergence between predicted ratio and true ratio, and SmoothL1 loss for true values of 3 components. I get 0.76(higher) in lb and 0.7905 in cv.\n\n```\nclass CompositionalBiomassLoss(nn.Module):\n def __init__(self, eps=1e-6):\n super().__init__()\n self.eps = eps\n self.weights = {'green': 0.1, 'dead': 0.1, 'clover': 0.1, 'gdm': 0.2, 'total': 0.5}\n\n def forward(self, pred_total, pred_logits, targets):\n gt_green = targets[:, 1].unsqueeze(1)\n gt_dead = targets[:, 2].unsqueeze(1)\n gt_clover= targets[:, 3].unsqueeze(1)\n gt_total = targets[:, 0].unsqueeze(1)\n\n gt_components = torch.cat([gt_green, gt_dead, gt_clover], dim=1) # (B, 3)\n gt_ratios = gt_components / (gt_total + self.eps)\n\n loss_total = F.smooth_l1_loss(torch.log1p(pred_total), torch.log1p(gt_total))\n\n\n pred_log_probs = F.log_softmax(pred_logits, dim=1)\n loss_dist = F.kl_div(pred_log_probs, gt_ratios, reduction='batchmean')\n\n pred_ratios = F.softmax(pred_logits, dim=1) # (B, 3)\n\n recon_green = pred_total * pred_ratios[:, 0:1]\n recon_dead = pred_total * pred_ratios[:, 1:2]\n recon_clover = pred_total * pred_ratios[:, 2:3]\n\n l_green = F.smooth_l1_loss(recon_green, gt_green)\n l_dead = F.smooth_l1_loss(recon_dead, gt_dead)\n l_clover = F.smooth_l1_loss(recon_clover, gt_clover)\n \n loss_recon = (self.weights['green'] * l_green + \n self.weights['dead'] * l_dead + \n self.weights['clover']* l_clover)\n\n total_loss = 1.0 * loss_total + 0.5 * loss_dist + 1.0 * loss_recon\n \n return total_loss\n```\n\n## Augmentation\n```\nA.Compose(\n [\n A.Resize(CFG.IMG_SIZE, CFG.IMG_SIZE),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=1.0),\n A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n ToTensorV2(),\n ]\n)\n```\n\n## Other configuration.\noptimizer: AdamW; \nscheduler: CosineLRScheduler; \nEMA\n```\nIMG_SIZE=960\nLR_INIT = 1e-4\nWD = 3e-3\nepochs_warmup = 20\nwarmup_lr_init = 5e-5\nlr_min = 1e-6\ntile_grid=(2,2)\n```\n\nI use this [notebook ](https://www.kaggle.com/code/gothamjocker/lb-0-65-dino-v2-tiled) as my reference.\n\n# What didn't work\n- Larger model (vit_7b_dinov3)\n- LoRA\n- Finetune more blocks\n- Test-time training. Honestly, I think oneline training will be an important trick for this small dataset competition. But no matter how hard I had tried, I can't get an ideal result. Maybe I miss something :(\n- Single Stream\n- Larger image resolution (1024x1024, 1280x1280, 1536x1536)\n- More grids for Tile\n- TTA (So weird)\n\n# Ending\nShaking down really frustrate us, but this is kaggle, right? 🥲\n\n----------------------------------------------------------------------------------------------\n\nI generate the images in the article by Nano Banana Pro, so the images might be a little blurry. If you want more details, you can check my training code.I will release them later.", "3398437": "My architecture is very similar to yours, except that I added classifiers and embeddings for State, Season, and Species (customizable). This might be the reason why my model is severely overfitting😭.", "3398682": "i tried adding classifiers & embeddings for metadata too. Also tried distill etc. But none worked. Eventually I concluded that metadata are just misleading noise for models to learn. I ended up not using any of them for my submission…" }, "source": "meta" }