{ "id": 670695, "title": "55th Solution", "url": "/competitions/csiro-biomass/discussion/670695", "author_name": "zhiyue666", "post_date": "2026-01-29T08:10:06.666000", "votes": 10, "comment_count": 2, "views": 0, "content": "

These are the links to the two notebooks we submitted:

\n

CSIRO-infer-base-2x4-GW(Public LB: 0.72 | Private LB: 0.63 | Local CV:0.8610)

\n

CSIRO-infer-Ensemble-2p8p(Public LB: 0.71 | Private LB: 0.63 | Local CV:0.8660)

\n

You can view our training code here: Github-zhiyue666-Kaggle-CSIRO-Image2Biomass-Prediction-55th-Solution

\n

We didn't use overly complicated tricks. Let's start by building a Local CV.

\n
    \n
  1. We used StratifiedGroupKFold + rare class handling to construct the CV.
    2. \n
  2. We use Image ID as the group criterion to ensure that all Target rows corresponding to the same image are strictly divided into the same fold, thus eliminating the data leakage problem of \"the same image being in both the training set and the validation set\".
    3. \n
  3. Regarding stratification and handling of long-tail distribution: We used Species as the stratification basis and implemented a Rare Class Aggregation strategy for long-tail distribution. Rare species with fewer than 10 images were uniformly relabeled as \"Other\". This solved the missing class (NaN) problem in the validation set caused by extreme class imbalance, ensuring that each fold evenly covers common species and rare samples (Hard Examples).
    4. \n
  4. This is the CSV file we use after the above processing: CSIRO-my-train-csv-20251223
    5. \n
\n

\n\n\n\n\n

Having discussed the construction of CV, let's talk about the loss function and local evaluation function we used.

\n\n\n\n\n\n\n
class WeightedMSELoss(nn.Module):\n    def __init__(self, feature_weights=None):\n        super().__init__()\n        if feature_weights is None:\n            self.register_buffer('feature_weights', \n                               torch.tensor([0.1, 0.1, 0.1, 0.5, 0.2]))\n        else:\n            self.register_buffer('feature_weights', torch.tensor(feature_weights))\n\n    def forward(self, y_pred, y_true):\n        weights = self.feature_weights.to(device=y_pred.device, dtype=y_pred.dtype)\n        loss = weights * (y_pred - y_true) ** 2\n        return loss.sum(dim=1).mean()\n
\n\n
def Calculate_Weighted_R2(y_true, y_pred):\n    \"\"\"\n    0: Dry_Clover_g (w=0.1)\n    1: Dry_Dead_g   (w=0.1)\n    2: Dry_Green_g  (w=0.1)\n    3: Dry_Total_g  (w=0.5)\n    4: GDM_g        (w=0.2)\n    \"\"\"\n    weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n    y_true = np.array(y_true)\n    y_pred = np.array(y_pred)\n\n    weighted_sum = np.sum(y_true * weights) \n    total_weight = np.sum(weights) * y_true.shape[0] # weights总和 * 样本数\n    y_bar_w = weighted_sum / total_weight\n\n    ss_res = np.sum(weights * (y_true - y_pred)**2)\n    ss_tot = np.sum(weights * (y_true - y_bar_w)**2)\n\n    if ss_tot == 0:\n        return 0.0\n\n    r2 = 1 - (ss_res / ss_tot)\n\n    return r2\n
\n

Next is about model selection and construction.

\n\n
class CSIROHead(nn.Module):\n    def __init__(self, in_features):\n        super(CSIROHead, self).__init__()\n        self.fusion = nn.Sequential(\n            LocalMambaBlock(in_features, kernel_size=5, dropout=0.1),\n            LocalMambaBlock(in_features, kernel_size=5, dropout=0.1)\n        )\n        self.pool = nn.AdaptiveAvgPool1d(1)\n        self.out_head = nn.Sequential(\n            nn.Linear(in_features, in_features // 2),\n            nn.LayerNorm(in_features // 2),\n            nn.ReLU(),\n            nn.Dropout(0.2), \n            nn.Linear(in_features // 2, CONFIG.head_out),\n        )\n\n    def forward(self, x):\n        _tmp = self.fusion(x)\n        _tmp_pool = self.pool(_tmp.transpose(1, 2)).flatten(1)\n        out = self.out_head(_tmp_pool)\n        return out\n
\n\n
class EMA:\n    def __init__(self, model, decay=0.999):\n        self.model = model\n        self.decay = decay\n        self.shadow = {}\n        self.backup = {}\n\n    def register(self):\n        \"\"\"Called before training starts: Initialize shadow variables.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                self.shadow[name] = param.data.clone()\n\n    def update(self):\n        \"\"\"Called after each optimizer.step(): Update shadow variables.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                assert name in self.shadow\n                # Compute exponential moving average: \n                # new_avg = (1 - decay) * current_weight + decay * old_avg\n                new_average = (1.0 - self.decay) * param.data + self.decay * self.shadow[name]\n                self.shadow[name] = new_average.clone()\n\n    def apply_shadow(self):\n        \"\"\"Called before validation: Replace model parameters with shadow parameters.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                self.backup[name] = param.data.clone()\n                param.data = self.shadow[name]\n\n    def restore(self):\n        \"\"\"Called after validation: Restore original parameters to resume training.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                assert name in self.backup\n                param.data = self.backup[name]\n        self.backup = {}\n
\n\n
seed = 308\nn_folds = 5\nepochs = 100\nbackbone = \"vit_large_patch16_dinov3.lvd1689m\"\noptimizer = AdamW\nlr_backbone = warmup --> 1e-5 --> 1e-8\nlr_head = warmup --> 1e-3 -->  1e-6\nuse_mixup = True\nema_decay = 0.999\n\nData Augmentation:\n        A.HorizontalFlip(p=0.5), \n        A.VerticalFlip(p=0.5),\n        A.RandomRotate90(p=0.5),\n\nMixup: alpha=0.4, p=0.8\n
\n

For us, the part that didn't work:

\n
    \n
  1. TTA: We found that using TTA on our local CV did not improve it, but rather decreased it. Therefore, we did not use TTA in our submitted notebooks.
    2. \n
  2. More models fusion: Similarly, when we used more models to fuse them on local CV, we found that the CV improvement was very limited (after all, all our models were trained on the same dataset, and their prediction distributions were very similar). We found that combining more models did not significantly improve cross-validation performance as much as using a single model for post-processing, so we ultimately chose a single model and a fusion model of 2-patch + 8-patch.
    3. \n
  3. Other data augmentations: We also used the following data augmentation methods, but they did not yield good results.
    4. \n
\n
A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=0.5),\nA.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), p=0.5),\nA.CoarseDropout(max_holes=8, max_height=32, max_width=32, fill_value=0, p=0.5),\nA.OneOf([\n    A.RandomBrightnessContrast(p=1.0), \n    A.HueSaturationValue(p=1.0), \n    A.GaussNoise(p=1.0),\n], p=0.5),\n
\n

This is the core of our solution. Thank you for reading. Finally, I want to thanks to Kaggle and CSIRO for organizing this competition, and salute to all the participants.

", "messages": [ { "id": 3398489, "postDate": "2026-01-29T08:10:06.667Z", "content": "

These are the links to the two notebooks we submitted:

\n

CSIRO-infer-base-2x4-GW(Public LB: 0.72 | Private LB: 0.63 | Local CV:0.8610)

\n

CSIRO-infer-Ensemble-2p8p(Public LB: 0.71 | Private LB: 0.63 | Local CV:0.8660)

\n

You can view our training code here: Github-zhiyue666-Kaggle-CSIRO-Image2Biomass-Prediction-55th-Solution

\n

We didn't use overly complicated tricks. Let's start by building a Local CV.

\n
    \n
  1. We used StratifiedGroupKFold + rare class handling to construct the CV.
    2. \n
  2. We use Image ID as the group criterion to ensure that all Target rows corresponding to the same image are strictly divided into the same fold, thus eliminating the data leakage problem of \"the same image being in both the training set and the validation set\".
    3. \n
  3. Regarding stratification and handling of long-tail distribution: We used Species as the stratification basis and implemented a Rare Class Aggregation strategy for long-tail distribution. Rare species with fewer than 10 images were uniformly relabeled as \"Other\". This solved the missing class (NaN) problem in the validation set caused by extreme class imbalance, ensuring that each fold evenly covers common species and rare samples (Hard Examples).
    4. \n
  4. This is the CSV file we use after the above processing: CSIRO-my-train-csv-20251223
    5. \n
\n

\n\n\n\n\n

Having discussed the construction of CV, let's talk about the loss function and local evaluation function we used.

\n\n\n\n\n\n\n
class WeightedMSELoss(nn.Module):\n    def __init__(self, feature_weights=None):\n        super().__init__()\n        if feature_weights is None:\n            self.register_buffer('feature_weights', \n                               torch.tensor([0.1, 0.1, 0.1, 0.5, 0.2]))\n        else:\n            self.register_buffer('feature_weights', torch.tensor(feature_weights))\n\n    def forward(self, y_pred, y_true):\n        weights = self.feature_weights.to(device=y_pred.device, dtype=y_pred.dtype)\n        loss = weights * (y_pred - y_true) ** 2\n        return loss.sum(dim=1).mean()\n
\n\n
def Calculate_Weighted_R2(y_true, y_pred):\n    \"\"\"\n    0: Dry_Clover_g (w=0.1)\n    1: Dry_Dead_g   (w=0.1)\n    2: Dry_Green_g  (w=0.1)\n    3: Dry_Total_g  (w=0.5)\n    4: GDM_g        (w=0.2)\n    \"\"\"\n    weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n    y_true = np.array(y_true)\n    y_pred = np.array(y_pred)\n\n    weighted_sum = np.sum(y_true * weights) \n    total_weight = np.sum(weights) * y_true.shape[0] # weights总和 * 样本数\n    y_bar_w = weighted_sum / total_weight\n\n    ss_res = np.sum(weights * (y_true - y_pred)**2)\n    ss_tot = np.sum(weights * (y_true - y_bar_w)**2)\n\n    if ss_tot == 0:\n        return 0.0\n\n    r2 = 1 - (ss_res / ss_tot)\n\n    return r2\n
\n

Next is about model selection and construction.

\n\n
class CSIROHead(nn.Module):\n    def __init__(self, in_features):\n        super(CSIROHead, self).__init__()\n        self.fusion = nn.Sequential(\n            LocalMambaBlock(in_features, kernel_size=5, dropout=0.1),\n            LocalMambaBlock(in_features, kernel_size=5, dropout=0.1)\n        )\n        self.pool = nn.AdaptiveAvgPool1d(1)\n        self.out_head = nn.Sequential(\n            nn.Linear(in_features, in_features // 2),\n            nn.LayerNorm(in_features // 2),\n            nn.ReLU(),\n            nn.Dropout(0.2), \n            nn.Linear(in_features // 2, CONFIG.head_out),\n        )\n\n    def forward(self, x):\n        _tmp = self.fusion(x)\n        _tmp_pool = self.pool(_tmp.transpose(1, 2)).flatten(1)\n        out = self.out_head(_tmp_pool)\n        return out\n
\n\n
class EMA:\n    def __init__(self, model, decay=0.999):\n        self.model = model\n        self.decay = decay\n        self.shadow = {}\n        self.backup = {}\n\n    def register(self):\n        \"\"\"Called before training starts: Initialize shadow variables.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                self.shadow[name] = param.data.clone()\n\n    def update(self):\n        \"\"\"Called after each optimizer.step(): Update shadow variables.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                assert name in self.shadow\n                # Compute exponential moving average: \n                # new_avg = (1 - decay) * current_weight + decay * old_avg\n                new_average = (1.0 - self.decay) * param.data + self.decay * self.shadow[name]\n                self.shadow[name] = new_average.clone()\n\n    def apply_shadow(self):\n        \"\"\"Called before validation: Replace model parameters with shadow parameters.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                self.backup[name] = param.data.clone()\n                param.data = self.shadow[name]\n\n    def restore(self):\n        \"\"\"Called after validation: Restore original parameters to resume training.\"\"\"\n        for name, param in self.model.named_parameters():\n            if param.requires_grad:\n                assert name in self.backup\n                param.data = self.backup[name]\n        self.backup = {}\n
\n\n
seed = 308\nn_folds = 5\nepochs = 100\nbackbone = \"vit_large_patch16_dinov3.lvd1689m\"\noptimizer = AdamW\nlr_backbone = warmup --> 1e-5 --> 1e-8\nlr_head = warmup --> 1e-3 -->  1e-6\nuse_mixup = True\nema_decay = 0.999\n\nData Augmentation:\n        A.HorizontalFlip(p=0.5), \n        A.VerticalFlip(p=0.5),\n        A.RandomRotate90(p=0.5),\n\nMixup: alpha=0.4, p=0.8\n
\n

For us, the part that didn't work:

\n
    \n
  1. TTA: We found that using TTA on our local CV did not improve it, but rather decreased it. Therefore, we did not use TTA in our submitted notebooks.
    2. \n
  2. More models fusion: Similarly, when we used more models to fuse them on local CV, we found that the CV improvement was very limited (after all, all our models were trained on the same dataset, and their prediction distributions were very similar). We found that combining more models did not significantly improve cross-validation performance as much as using a single model for post-processing, so we ultimately chose a single model and a fusion model of 2-patch + 8-patch.
    3. \n
  3. Other data augmentations: We also used the following data augmentation methods, but they did not yield good results.
    4. \n
\n
A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=0.5),\nA.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), p=0.5),\nA.CoarseDropout(max_holes=8, max_height=32, max_width=32, fill_value=0, p=0.5),\nA.OneOf([\n    A.RandomBrightnessContrast(p=1.0), \n    A.HueSaturationValue(p=1.0), \n    A.GaussNoise(p=1.0),\n], p=0.5),\n
\n

This is the core of our solution. Thank you for reading. Finally, I want to thanks to Kaggle and CSIRO for organizing this competition, and salute to all the participants.

", "rawMarkdown": "**These are the links to the two notebooks we submitted:**\n\n[CSIRO-infer-base-2x4-GW(Public LB: 0.72 | Private LB: 0.63 | Local CV:0.8610)](https://www.kaggle.com/code/zhiyue666/csiro-infer-base-2x4-gw?scriptVersionId=292394299)\n\n[CSIRO-infer-Ensemble-2p8p(Public LB: 0.71 | Private LB: 0.63 | Local CV:0.8660)](https://www.kaggle.com/code/zhiyue666/csiro-infer-ensemble-2p8p?scriptVersionId=293327244)\n\n**You can view our training code here: [Github-zhiyue666-Kaggle-CSIRO-Image2Biomass-Prediction-55th-Solution](https://github.com/zhiyuemouse/Kaggle-CSIRO-Image2Biomass-Prediction-55th-Solution)**\n\n## We didn't use overly complicated tricks. Let's start by building a Local CV.\n1. We used **StratifiedGroupKFold** + **rare class handling** to construct the CV.\n2. We use **Image ID** as the group criterion to ensure that all Target rows corresponding to the same image are strictly divided into the same fold, thus eliminating the data leakage problem of \"the same image being in both the training set and the validation set\".\n3. Regarding **stratification** and **handling of long-tail distribution**: We used **Species** as the stratification basis and implemented a **Rare Class Aggregation** strategy for long-tail distribution. Rare species with fewer than 10 images were uniformly relabeled as **\"Other\"**. This solved the missing class (NaN) problem in the validation set caused by extreme class imbalance, ensuring that each fold evenly covers common species and rare samples (Hard Examples).\n4. This is the CSV file we use after the above processing: [CSIRO-my-train-csv-20251223](https://www.kaggle.com/datasets/zhiyue666/csiro-my-train-csv-20251223)\n- Below is a portion of our CV's corresponding PubLB and PriLB, showing an overall positive correlation.\n|PublicLB|PrivateLB|Local CV↓|\n|-----|-----|-----|\n|**0.71**|**0.63**|**0.8660**|\n|**0.72**|**0.63**|**0.8610**|\n|0.70|0.61|0.8511|\n|0.69|0.59|0.8436|\n|0.65|0.58|0.8423|\n|0.68|0.57|0.8338|\n|0.61|0.56|0.8189|\n|0.58|0.51|0.7758|\n|0.54|0.47|0.7312|\n|0.51|0.45|0.7153|\n|0.28|0.19|0.5809|\n\n\n## Having discussed the construction of CV, let's talk about the loss function and local evaluation function we used.\n- We are using weighted MSELoss, and the following is its code implementation.\n```python\nclass WeightedMSELoss(nn.Module):\n def __init__(self, feature_weights=None):\n super().__init__()\n if feature_weights is None:\n self.register_buffer('feature_weights', \n torch.tensor([0.1, 0.1, 0.1, 0.5, 0.2]))\n else:\n self.register_buffer('feature_weights', torch.tensor(feature_weights))\n \n def forward(self, y_pred, y_true):\n weights = self.feature_weights.to(device=y_pred.device, dtype=y_pred.dtype)\n loss = weights * (y_pred - y_true) ** 2\n return loss.sum(dim=1).mean()\n```\n- The weights above are based on the evaluation method provided by the official competition organizers, so your labels should be sorted in the format of [\"__Dry_Clover_g\", \"__Dry_Dead_g\", \"__Dry_Green_g\", \"__Dry_Total_g\", \"__GDM_g\"].\n- We also tried using weighted L1Loss, and in our experiments, it performed similarly to weighted MSELoss. Therefore, all subsequent training was based on weighted MSELoss.\n- The local evaluation function we used is as follows:\n```python\ndef Calculate_Weighted_R2(y_true, y_pred):\n \"\"\"\n 0: Dry_Clover_g (w=0.1)\n 1: Dry_Dead_g (w=0.1)\n 2: Dry_Green_g (w=0.1)\n 3: Dry_Total_g (w=0.5)\n 4: GDM_g (w=0.2)\n \"\"\"\n weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n y_true = np.array(y_true)\n y_pred = np.array(y_pred)\n \n weighted_sum = np.sum(y_true * weights) \n total_weight = np.sum(weights) * y_true.shape[0] # weights总和 * 样本数\n y_bar_w = weighted_sum / total_weight\n\n ss_res = np.sum(weights * (y_true - y_pred)**2)\n ss_tot = np.sum(weights * (y_true - y_bar_w)**2)\n\n if ss_tot == 0:\n return 0.0\n \n r2 = 1 - (ss_res / ss_tot)\n \n return r2\n```\n\n## Next is about model selection and construction.\n- In the early stages of our participation in the competition, we used many CNN models (EfficientNet, ConvNext, edgenext) because they didn't need to consider whether the image aspect ratio was 1:1. For the 1:2 images in this competition, they could easily run the code without much consideration.\n- After conducting some experiments, we tried using VIT, from Dinov2 to Dinov3. Our experiments showed that VIT consistently outperformed CNNs, perhaps due to VIT's large pre-training capacity. Dinov3 performed better than Dinov2, so in subsequent experiments we used the **\"vit_large_patch16_dinov3.lvd1689m\"** model. It can be directly called via timm, which is very convenient.(I love timm🥰)\n- For the input to this model, like most people in the code area, we resize the image to (512, 1024), then split the image horizontally into two (512, 512) images, input each image into the model, and then perform feature fusion at the head layer.\n- In addition to the **2-patch** approach mentioned above, we also used an **8-patch** approach. The image was resized to (1024, 2048), and then divided into 2×4=8 (512, 512) images, each input into the model for feature fusion.\n- For feature fusion at the head layer, we input the [B, L, D] features output from the backbone into LocalMambaBlock for feature fusion, and finally output the predicted 5 regression values ​​directly through two linear layers.\n```python\nclass CSIROHead(nn.Module):\n def __init__(self, in_features):\n super(CSIROHead, self).__init__()\n self.fusion = nn.Sequential(\n LocalMambaBlock(in_features, kernel_size=5, dropout=0.1),\n LocalMambaBlock(in_features, kernel_size=5, dropout=0.1)\n )\n self.pool = nn.AdaptiveAvgPool1d(1)\n self.out_head = nn.Sequential(\n nn.Linear(in_features, in_features // 2),\n nn.LayerNorm(in_features // 2),\n nn.ReLU(),\n nn.Dropout(0.2), \n nn.Linear(in_features // 2, CONFIG.head_out),\n )\n\n def forward(self, x):\n _tmp = self.fusion(x)\n _tmp_pool = self.pool(_tmp.transpose(1, 2)).flatten(1)\n out = self.out_head(_tmp_pool)\n return out\n```\n- The implementation of LocalMambaBlock comes from open-source code in the code section (**thanks to all kagglers for their open-source contributions**).\n- The following is the process for submitting the two notebooks.\n![process](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16233502%2Fc1bd56adb3f9fad238f0a78e5e4892f4%2FCSIRO.jpg?generation=1769683093724218&alt=media)\n- The post-processing includes Linear Balance and Mass Balance, which you can see in my submission notebook.(Since the results after the first three decimal places are not yet visible, it remains to be seen whether this post-processing is effective.)\n- To ensure training stability, we used EMA for training.\n```python\nclass EMA:\n def __init__(self, model, decay=0.999):\n self.model = model\n self.decay = decay\n self.shadow = {}\n self.backup = {}\n\n def register(self):\n \"\"\"Called before training starts: Initialize shadow variables.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n self.shadow[name] = param.data.clone()\n\n def update(self):\n \"\"\"Called after each optimizer.step(): Update shadow variables.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.shadow\n # Compute exponential moving average: \n # new_avg = (1 - decay) * current_weight + decay * old_avg\n new_average = (1.0 - self.decay) * param.data + self.decay * self.shadow[name]\n self.shadow[name] = new_average.clone()\n\n def apply_shadow(self):\n \"\"\"Called before validation: Replace model parameters with shadow parameters.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n self.backup[name] = param.data.clone()\n param.data = self.shadow[name]\n\n def restore(self):\n \"\"\"Called after validation: Restore original parameters to resume training.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.backup\n param.data = self.backup[name]\n self.backup = {}\n```\n- Our training configuration is as follows:\n```python\nseed = 308\nn_folds = 5\nepochs = 100\nbackbone = \"vit_large_patch16_dinov3.lvd1689m\"\noptimizer = AdamW\nlr_backbone = warmup --> 1e-5 --> 1e-8\nlr_head = warmup --> 1e-3 --> 1e-6\nuse_mixup = True\nema_decay = 0.999\n\nData Augmentation:\n A.HorizontalFlip(p=0.5), \n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n\nMixup: alpha=0.4, p=0.8\n```\n### For us, the part that didn't work:\n1. **TTA**: We found that using TTA on our local CV did not improve it, but rather decreased it. Therefore, we did not use TTA in our submitted notebooks.\n2. **More models fusion**: Similarly, when we used more models to fuse them on local CV, we found that the CV improvement was very limited (after all, all our models were trained on the same dataset, and their prediction distributions were very similar). We found that combining more models did not significantly improve cross-validation performance as much as using a single model for post-processing, so we ultimately chose a single model and a fusion model of 2-patch + 8-patch.\n3. **Other data augmentations**: We also used the following data augmentation methods, but they did not yield good results.\n```python\nA.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=0.5),\nA.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), p=0.5),\nA.CoarseDropout(max_holes=8, max_height=32, max_width=32, fill_value=0, p=0.5),\nA.OneOf([\n A.RandomBrightnessContrast(p=1.0), \n A.HueSaturationValue(p=1.0), \n A.GaussNoise(p=1.0),\n], p=0.5),\n```\n\n\n\n## This is the core of our solution. Thank you for reading. Finally, I want to thanks to Kaggle and CSIRO for organizing this competition, and salute to all the participants.", "votes": 10 }, { "id": 3398502, "postDate": "2026-01-29T08:29:52.963Z", "content": "

its a nice job! zhiyue666!

", "rawMarkdown": "its a nice job! zhiyue666!\n" }, { "id": 3426856, "postDate": "2026-03-23T11:29:15.537Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3398502, "author_name": "yigecaiji", "author_url": "", "post_date": "2026-01-29T08:29:52.963000", "content": "

its a nice job! zhiyue666!

", "votes": 0, "replies": [] }, { "id": 3426856, "author_name": "", "author_url": "", "post_date": "2026-03-23T11:29:15.537000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3398489": "**These are the links to the two notebooks we submitted:**\n\n[CSIRO-infer-base-2x4-GW(Public LB: 0.72 | Private LB: 0.63 | Local CV:0.8610)](https://www.kaggle.com/code/zhiyue666/csiro-infer-base-2x4-gw?scriptVersionId=292394299)\n\n[CSIRO-infer-Ensemble-2p8p(Public LB: 0.71 | Private LB: 0.63 | Local CV:0.8660)](https://www.kaggle.com/code/zhiyue666/csiro-infer-ensemble-2p8p?scriptVersionId=293327244)\n\n**You can view our training code here: [Github-zhiyue666-Kaggle-CSIRO-Image2Biomass-Prediction-55th-Solution](https://github.com/zhiyuemouse/Kaggle-CSIRO-Image2Biomass-Prediction-55th-Solution)**\n\n## We didn't use overly complicated tricks. Let's start by building a Local CV.\n1. We used **StratifiedGroupKFold** + **rare class handling** to construct the CV.\n2. We use **Image ID** as the group criterion to ensure that all Target rows corresponding to the same image are strictly divided into the same fold, thus eliminating the data leakage problem of \"the same image being in both the training set and the validation set\".\n3. Regarding **stratification** and **handling of long-tail distribution**: We used **Species** as the stratification basis and implemented a **Rare Class Aggregation** strategy for long-tail distribution. Rare species with fewer than 10 images were uniformly relabeled as **\"Other\"**. This solved the missing class (NaN) problem in the validation set caused by extreme class imbalance, ensuring that each fold evenly covers common species and rare samples (Hard Examples).\n4. This is the CSV file we use after the above processing: [CSIRO-my-train-csv-20251223](https://www.kaggle.com/datasets/zhiyue666/csiro-my-train-csv-20251223)\n- Below is a portion of our CV's corresponding PubLB and PriLB, showing an overall positive correlation.\n|PublicLB|PrivateLB|Local CV↓|\n|-----|-----|-----|\n|**0.71**|**0.63**|**0.8660**|\n|**0.72**|**0.63**|**0.8610**|\n|0.70|0.61|0.8511|\n|0.69|0.59|0.8436|\n|0.65|0.58|0.8423|\n|0.68|0.57|0.8338|\n|0.61|0.56|0.8189|\n|0.58|0.51|0.7758|\n|0.54|0.47|0.7312|\n|0.51|0.45|0.7153|\n|0.28|0.19|0.5809|\n\n\n## Having discussed the construction of CV, let's talk about the loss function and local evaluation function we used.\n- We are using weighted MSELoss, and the following is its code implementation.\n```python\nclass WeightedMSELoss(nn.Module):\n def __init__(self, feature_weights=None):\n super().__init__()\n if feature_weights is None:\n self.register_buffer('feature_weights', \n torch.tensor([0.1, 0.1, 0.1, 0.5, 0.2]))\n else:\n self.register_buffer('feature_weights', torch.tensor(feature_weights))\n \n def forward(self, y_pred, y_true):\n weights = self.feature_weights.to(device=y_pred.device, dtype=y_pred.dtype)\n loss = weights * (y_pred - y_true) ** 2\n return loss.sum(dim=1).mean()\n```\n- The weights above are based on the evaluation method provided by the official competition organizers, so your labels should be sorted in the format of [\"__Dry_Clover_g\", \"__Dry_Dead_g\", \"__Dry_Green_g\", \"__Dry_Total_g\", \"__GDM_g\"].\n- We also tried using weighted L1Loss, and in our experiments, it performed similarly to weighted MSELoss. Therefore, all subsequent training was based on weighted MSELoss.\n- The local evaluation function we used is as follows:\n```python\ndef Calculate_Weighted_R2(y_true, y_pred):\n \"\"\"\n 0: Dry_Clover_g (w=0.1)\n 1: Dry_Dead_g (w=0.1)\n 2: Dry_Green_g (w=0.1)\n 3: Dry_Total_g (w=0.5)\n 4: GDM_g (w=0.2)\n \"\"\"\n weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n y_true = np.array(y_true)\n y_pred = np.array(y_pred)\n \n weighted_sum = np.sum(y_true * weights) \n total_weight = np.sum(weights) * y_true.shape[0] # weights总和 * 样本数\n y_bar_w = weighted_sum / total_weight\n\n ss_res = np.sum(weights * (y_true - y_pred)**2)\n ss_tot = np.sum(weights * (y_true - y_bar_w)**2)\n\n if ss_tot == 0:\n return 0.0\n \n r2 = 1 - (ss_res / ss_tot)\n \n return r2\n```\n\n## Next is about model selection and construction.\n- In the early stages of our participation in the competition, we used many CNN models (EfficientNet, ConvNext, edgenext) because they didn't need to consider whether the image aspect ratio was 1:1. For the 1:2 images in this competition, they could easily run the code without much consideration.\n- After conducting some experiments, we tried using VIT, from Dinov2 to Dinov3. Our experiments showed that VIT consistently outperformed CNNs, perhaps due to VIT's large pre-training capacity. Dinov3 performed better than Dinov2, so in subsequent experiments we used the **\"vit_large_patch16_dinov3.lvd1689m\"** model. It can be directly called via timm, which is very convenient.(I love timm🥰)\n- For the input to this model, like most people in the code area, we resize the image to (512, 1024), then split the image horizontally into two (512, 512) images, input each image into the model, and then perform feature fusion at the head layer.\n- In addition to the **2-patch** approach mentioned above, we also used an **8-patch** approach. The image was resized to (1024, 2048), and then divided into 2×4=8 (512, 512) images, each input into the model for feature fusion.\n- For feature fusion at the head layer, we input the [B, L, D] features output from the backbone into LocalMambaBlock for feature fusion, and finally output the predicted 5 regression values ​​directly through two linear layers.\n```python\nclass CSIROHead(nn.Module):\n def __init__(self, in_features):\n super(CSIROHead, self).__init__()\n self.fusion = nn.Sequential(\n LocalMambaBlock(in_features, kernel_size=5, dropout=0.1),\n LocalMambaBlock(in_features, kernel_size=5, dropout=0.1)\n )\n self.pool = nn.AdaptiveAvgPool1d(1)\n self.out_head = nn.Sequential(\n nn.Linear(in_features, in_features // 2),\n nn.LayerNorm(in_features // 2),\n nn.ReLU(),\n nn.Dropout(0.2), \n nn.Linear(in_features // 2, CONFIG.head_out),\n )\n\n def forward(self, x):\n _tmp = self.fusion(x)\n _tmp_pool = self.pool(_tmp.transpose(1, 2)).flatten(1)\n out = self.out_head(_tmp_pool)\n return out\n```\n- The implementation of LocalMambaBlock comes from open-source code in the code section (**thanks to all kagglers for their open-source contributions**).\n- The following is the process for submitting the two notebooks.\n![process](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16233502%2Fc1bd56adb3f9fad238f0a78e5e4892f4%2FCSIRO.jpg?generation=1769683093724218&alt=media)\n- The post-processing includes Linear Balance and Mass Balance, which you can see in my submission notebook.(Since the results after the first three decimal places are not yet visible, it remains to be seen whether this post-processing is effective.)\n- To ensure training stability, we used EMA for training.\n```python\nclass EMA:\n def __init__(self, model, decay=0.999):\n self.model = model\n self.decay = decay\n self.shadow = {}\n self.backup = {}\n\n def register(self):\n \"\"\"Called before training starts: Initialize shadow variables.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n self.shadow[name] = param.data.clone()\n\n def update(self):\n \"\"\"Called after each optimizer.step(): Update shadow variables.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.shadow\n # Compute exponential moving average: \n # new_avg = (1 - decay) * current_weight + decay * old_avg\n new_average = (1.0 - self.decay) * param.data + self.decay * self.shadow[name]\n self.shadow[name] = new_average.clone()\n\n def apply_shadow(self):\n \"\"\"Called before validation: Replace model parameters with shadow parameters.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n self.backup[name] = param.data.clone()\n param.data = self.shadow[name]\n\n def restore(self):\n \"\"\"Called after validation: Restore original parameters to resume training.\"\"\"\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.backup\n param.data = self.backup[name]\n self.backup = {}\n```\n- Our training configuration is as follows:\n```python\nseed = 308\nn_folds = 5\nepochs = 100\nbackbone = \"vit_large_patch16_dinov3.lvd1689m\"\noptimizer = AdamW\nlr_backbone = warmup --> 1e-5 --> 1e-8\nlr_head = warmup --> 1e-3 --> 1e-6\nuse_mixup = True\nema_decay = 0.999\n\nData Augmentation:\n A.HorizontalFlip(p=0.5), \n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n\nMixup: alpha=0.4, p=0.8\n```\n### For us, the part that didn't work:\n1. **TTA**: We found that using TTA on our local CV did not improve it, but rather decreased it. Therefore, we did not use TTA in our submitted notebooks.\n2. **More models fusion**: Similarly, when we used more models to fuse them on local CV, we found that the CV improvement was very limited (after all, all our models were trained on the same dataset, and their prediction distributions were very similar). We found that combining more models did not significantly improve cross-validation performance as much as using a single model for post-processing, so we ultimately chose a single model and a fusion model of 2-patch + 8-patch.\n3. **Other data augmentations**: We also used the following data augmentation methods, but they did not yield good results.\n```python\nA.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05, p=0.5),\nA.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), p=0.5),\nA.CoarseDropout(max_holes=8, max_height=32, max_width=32, fill_value=0, p=0.5),\nA.OneOf([\n A.RandomBrightnessContrast(p=1.0), \n A.HueSaturationValue(p=1.0), \n A.GaussNoise(p=1.0),\n], p=0.5),\n```\n\n\n\n## This is the core of our solution. Thank you for reading. Finally, I want to thanks to Kaggle and CSIRO for organizing this competition, and salute to all the participants.", "3398502": "its a nice job! zhiyue666!\n", "3426856": "" } }