Thanks to CSIRO and Kaggle for hosting and others for sharing their ideas throughout!
\nI was honestly surprised to end up so high. I bet on heavy ensemble to safeguard against a shake-up but didn't expect it would be of any help in climbing the leaderboard. My submission was a 19 model ensemble across different model types and CV strategies.
\nI trained two family of models
\nInitially I trained models with vit_huge_plus_patch16_dinov3 backbone where the models have an auxiliary head to predict all the metadata which then feeds into the final heads to predict biomass. The models were trained with an auxiliary metadata loss along with the main loss. Later on I realized, metadata isn't really making much of a difference so I dropped it.
I picked up the popular MambaBlock based model that everyone was using and made it work on 2 backbones and then eventually 3 backbones. First model had vit_huge_plus_patch16_dinov3 and convnextv2_large as they complement well and the second model added vit_base_patch16_224.mae to the mix. From my experience, choice of model made very little improvement to the overall performance as all my models got 0.64+ over a 4/5 fold ensemble. However, the Huge Plus version of DinoV3 always outperformed the Large version for me.
All the models were 2-stream (splitting image into left & right) based except for one model where I tried the whole image as input. It was almost similar in performance.
\nAll the models had 3 heads. GDM and Total were inferred from the 3 predicted values.
\nDuring training, I trained the backbones with 1/10th of the normal learning rate so as to not totally destroy the pre-trained weights.
\nclass BiomassModelV2(nn.Module):\n \"\"\"DINOv3 + Mamba Fusion + Multi-Head Regression\"\"\"\n\n def __init__(self, model1_name: str,\n model2_name: str,\n pretrained: bool = True\n ):\n super().__init__()\n self.model1_name = model1_name\n self.model2_name = model2_name\n\n self.backbone1 = timm.create_model(\n model1_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n self.backbone2 = timm.create_model(\n model2_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n nf1 = self.backbone1.num_features\n nf2 = 256\n nf = nf1 + nf2\n\n self.fusion1 = nn.Sequential(\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1)\n )\n self.fusion2 = nn.Sequential(\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1)\n )\n self.pool = nn.AdaptiveAvgPool1d(1)\n\n self.head_green = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_dead = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_clover = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n\n def forward(self, x):\n # x is a tuple (left, right)\n left, right = x\n\n x_l1 = self.fusion1(self.backbone1(left))\n x_l2 = self.fusion2(self.backbone2(left).flatten(2))\n x_r1 = self.fusion1(self.backbone1(right))\n x_r2 = self.fusion2(self.backbone2(right).flatten(2))\n\n x_l1 = self.pool(x_l1.transpose(1, 2)).flatten(1)\n x_l2 = self.pool(x_l2.transpose(1, 2)).flatten(1)\n x_r1 = self.pool(x_r1.transpose(1, 2)).flatten(1)\n x_r2 = self.pool(x_r2.transpose(1, 2)).flatten(1)\n\n x_cat = torch.cat([x_l1, x_l2, x_r1, x_r2], dim=1)\n\n green = self.head_green(x_cat)\n dead = self.head_dead(x_cat)\n clover = self.head_clover(x_cat)\n gdm = green + clover\n total = gdm + dead\n\n # Return as a single tensor (batch, 5)\n return torch.cat([green, dead, clover, gdm, total], dim=1)\nData was obviously the biggest factor of this competition and from the very beginning my focus was to extract the maximum information out of the training data provided. I used all the major augmentations available. Special mention to random grayscale to ensure the models learn features, not just color.
\nI improvised two extra augmentations which I believe helped
\nI applied the above two augmentations together, not one or the other.
\nI tried 2 CV techniques, or rather I should say fold creation techniques - as after the first month, I gave up on CV scores and solely trusted LB scores. To avoid overfitting, I chose to always use all the models of all the folds when I train a certain architecture. None of the CV strategies actually felt robust - there were always 1-2 folds that did great and 1-2 folds that were bad on LB.
\ndef create_robust_cv(\n df,\n embeddings_col,\n target_names,\n n_splits=5,\n n_clusters=50,\n seed=42,\n k=5,\n distance_metric=\"euclidean\",\n):\n df = df.copy().reset_index(drop=True)\n\n # 1. Embeddings\n X = np.vstack(df[embeddings_col].values).astype(np.float32)\n X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-8\n\n # 2. Visual clustering\n kmeans = KMeans(\n n_clusters=n_clusters,\n random_state=seed,\n n_init=10,\n )\n df[\"visual_cluster\"] = kmeans.fit_predict(X)\n\n # 3. Stratification target\n weights = {\n \"Dry_Green_g\": 0.1,\n \"Dry_Dead_g\": 0.1,\n \"Dry_Clover_g\": 0.1,\n \"GDM_g\": 0.2,\n \"Dry_Total_g\": 0.5,\n }\n\n composite = sum(df[t] * weights.get(t, 0) for t in target_names)\n try:\n df[\"target_bins\"] = pd.qcut(composite, q=10, labels=False, duplicates=\"drop\")\n except ValueError:\n df[\"target_bins\"] = pd.qcut(composite, q=5, labels=False, duplicates=\"drop\")\n\n # 4. CV split\n sgkf = StratifiedGroupKFold(\n n_splits=n_splits,\n shuffle=True,\n random_state=seed,\n )\n df[\"fold\"] = -1\n df[\"val_difficulty\"] = None\n df[\"mean_knn_dist_to_train\"] = np.nan\n\n for f, (train_idx, val_idx) in enumerate(\n sgkf.split(df, df[\"target_bins\"], groups=df[\"visual_cluster\"])\n ):\n X_train, X_val = X[train_idx], X[val_idx]\n\n nn = NearestNeighbors(n_neighbors=k, metric=distance_metric)\n nn.fit(X_train)\n knn_dist, _ = nn.kneighbors(X_val)\n mean_dist = knn_dist.mean(axis=1)\n df.loc[val_idx, \"mean_knn_dist_to_train\"] = mean_dist\n\n # 5. EASY / HARD split\n order = np.argsort(mean_dist)\n mid = len(order) // 2\n\n easy = val_idx[order[:mid]]\n hard = val_idx[order[mid:]]\n\n df.loc[easy, [\"fold\", \"val_difficulty\"]] = [f, \"easy\"]\n df.loc[hard, [\"fold\", \"val_difficulty\"]] = [f + n_splits, \"hard\"]\n return df\nTTA actually made LB performance worse, so I turned it off. With heavy augmentation and ensemble, I didn't need it in the end.
I obviously did use the scaling that I showed in the notebook I shared.. I ended up extending a little bit more. For clover, it really seems the data distribution is very different in test set. For Dry Dead, I feel all the models predict conservatively towards the mean when the actual values are more towards the fringes. The scaling gave me a decent boost (< 0.01) on the Public LB.
Since neither the CV or LB was reliable, my strategy from the beginning was to train multiple architectures and ensemble. I didn't make any drastic changes in my models or training, mostly slow gradual improvements - based on stability of CV scores and improvements in LB scores.
\nMost of my individual architectures were very similar in performance (~0.64+) over 5-folds. The last model set I trained - the 3 backbone non-metadata model was my highest scoring submission with 0.67.
\nOn a final note, two things I tried making work but failed were synthetic dataset and test time training, especially the latter - since the test data was so different. It's kinda exciting to read how some top teams got it to work for them.
", "messages": [ { "id": "3399038", "postDate": "01/30/2026 05:03:08", "content": "Thanks to CSIRO and Kaggle for hosting and others for sharing their ideas throughout!
\nI was honestly surprised to end up so high. I bet on heavy ensemble to safeguard against a shake-up but didn't expect it would be of any help in climbing the leaderboard. My submission was a 19 model ensemble across different model types and CV strategies.
\nI trained two family of models
\nInitially I trained models with vit_huge_plus_patch16_dinov3 backbone where the models have an auxiliary head to predict all the metadata which then feeds into the final heads to predict biomass. The models were trained with an auxiliary metadata loss along with the main loss. Later on I realized, metadata isn't really making much of a difference so I dropped it.
I picked up the popular MambaBlock based model that everyone was using and made it work on 2 backbones and then eventually 3 backbones. First model had vit_huge_plus_patch16_dinov3 and convnextv2_large as they complement well and the second model added vit_base_patch16_224.mae to the mix. From my experience, choice of model made very little improvement to the overall performance as all my models got 0.64+ over a 4/5 fold ensemble. However, the Huge Plus version of DinoV3 always outperformed the Large version for me.
All the models were 2-stream (splitting image into left & right) based except for one model where I tried the whole image as input. It was almost similar in performance.
\nAll the models had 3 heads. GDM and Total were inferred from the 3 predicted values.
\nDuring training, I trained the backbones with 1/10th of the normal learning rate so as to not totally destroy the pre-trained weights.
\nclass BiomassModelV2(nn.Module):\n \"\"\"DINOv3 + Mamba Fusion + Multi-Head Regression\"\"\"\n\n def __init__(self, model1_name: str,\n model2_name: str,\n pretrained: bool = True\n ):\n super().__init__()\n self.model1_name = model1_name\n self.model2_name = model2_name\n\n self.backbone1 = timm.create_model(\n model1_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n self.backbone2 = timm.create_model(\n model2_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n nf1 = self.backbone1.num_features\n nf2 = 256\n nf = nf1 + nf2\n\n self.fusion1 = nn.Sequential(\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1)\n )\n self.fusion2 = nn.Sequential(\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1)\n )\n self.pool = nn.AdaptiveAvgPool1d(1)\n\n self.head_green = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_dead = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_clover = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n\n def forward(self, x):\n # x is a tuple (left, right)\n left, right = x\n\n x_l1 = self.fusion1(self.backbone1(left))\n x_l2 = self.fusion2(self.backbone2(left).flatten(2))\n x_r1 = self.fusion1(self.backbone1(right))\n x_r2 = self.fusion2(self.backbone2(right).flatten(2))\n\n x_l1 = self.pool(x_l1.transpose(1, 2)).flatten(1)\n x_l2 = self.pool(x_l2.transpose(1, 2)).flatten(1)\n x_r1 = self.pool(x_r1.transpose(1, 2)).flatten(1)\n x_r2 = self.pool(x_r2.transpose(1, 2)).flatten(1)\n\n x_cat = torch.cat([x_l1, x_l2, x_r1, x_r2], dim=1)\n\n green = self.head_green(x_cat)\n dead = self.head_dead(x_cat)\n clover = self.head_clover(x_cat)\n gdm = green + clover\n total = gdm + dead\n\n # Return as a single tensor (batch, 5)\n return torch.cat([green, dead, clover, gdm, total], dim=1)\nData was obviously the biggest factor of this competition and from the very beginning my focus was to extract the maximum information out of the training data provided. I used all the major augmentations available. Special mention to random grayscale to ensure the models learn features, not just color.
\nI improvised two extra augmentations which I believe helped
\nI applied the above two augmentations together, not one or the other.
\nI tried 2 CV techniques, or rather I should say fold creation techniques - as after the first month, I gave up on CV scores and solely trusted LB scores. To avoid overfitting, I chose to always use all the models of all the folds when I train a certain architecture. None of the CV strategies actually felt robust - there were always 1-2 folds that did great and 1-2 folds that were bad on LB.
\ndef create_robust_cv(\n df,\n embeddings_col,\n target_names,\n n_splits=5,\n n_clusters=50,\n seed=42,\n k=5,\n distance_metric=\"euclidean\",\n):\n df = df.copy().reset_index(drop=True)\n\n # 1. Embeddings\n X = np.vstack(df[embeddings_col].values).astype(np.float32)\n X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-8\n\n # 2. Visual clustering\n kmeans = KMeans(\n n_clusters=n_clusters,\n random_state=seed,\n n_init=10,\n )\n df[\"visual_cluster\"] = kmeans.fit_predict(X)\n\n # 3. Stratification target\n weights = {\n \"Dry_Green_g\": 0.1,\n \"Dry_Dead_g\": 0.1,\n \"Dry_Clover_g\": 0.1,\n \"GDM_g\": 0.2,\n \"Dry_Total_g\": 0.5,\n }\n\n composite = sum(df[t] * weights.get(t, 0) for t in target_names)\n try:\n df[\"target_bins\"] = pd.qcut(composite, q=10, labels=False, duplicates=\"drop\")\n except ValueError:\n df[\"target_bins\"] = pd.qcut(composite, q=5, labels=False, duplicates=\"drop\")\n\n # 4. CV split\n sgkf = StratifiedGroupKFold(\n n_splits=n_splits,\n shuffle=True,\n random_state=seed,\n )\n df[\"fold\"] = -1\n df[\"val_difficulty\"] = None\n df[\"mean_knn_dist_to_train\"] = np.nan\n\n for f, (train_idx, val_idx) in enumerate(\n sgkf.split(df, df[\"target_bins\"], groups=df[\"visual_cluster\"])\n ):\n X_train, X_val = X[train_idx], X[val_idx]\n\n nn = NearestNeighbors(n_neighbors=k, metric=distance_metric)\n nn.fit(X_train)\n knn_dist, _ = nn.kneighbors(X_val)\n mean_dist = knn_dist.mean(axis=1)\n df.loc[val_idx, \"mean_knn_dist_to_train\"] = mean_dist\n\n # 5. EASY / HARD split\n order = np.argsort(mean_dist)\n mid = len(order) // 2\n\n easy = val_idx[order[:mid]]\n hard = val_idx[order[mid:]]\n\n df.loc[easy, [\"fold\", \"val_difficulty\"]] = [f, \"easy\"]\n df.loc[hard, [\"fold\", \"val_difficulty\"]] = [f + n_splits, \"hard\"]\n return df\nTTA actually made LB performance worse, so I turned it off. With heavy augmentation and ensemble, I didn't need it in the end.
I obviously did use the scaling that I showed in the notebook I shared.. I ended up extending a little bit more. For clover, it really seems the data distribution is very different in test set. For Dry Dead, I feel all the models predict conservatively towards the mean when the actual values are more towards the fringes. The scaling gave me a decent boost (< 0.01) on the Public LB.
Since neither the CV or LB was reliable, my strategy from the beginning was to train multiple architectures and ensemble. I didn't make any drastic changes in my models or training, mostly slow gradual improvements - based on stability of CV scores and improvements in LB scores.
\nMost of my individual architectures were very similar in performance (~0.64+) over 5-folds. The last model set I trained - the 3 backbone non-metadata model was my highest scoring submission with 0.67.
\nOn a final note, two things I tried making work but failed were synthetic dataset and test time training, especially the latter - since the test data was so different. It's kinda exciting to read how some top teams got it to work for them.
", "rawMarkdown": "Thanks to CSIRO and Kaggle for hosting and others for sharing their ideas throughout!\n\nI was honestly surprised to end up so high. I bet on heavy ensemble to safeguard against a shake-up but didn't expect it would be of any help in climbing the leaderboard. My submission was a 19 model ensemble across different model types and CV strategies. \n\n# 1. Model Architecture\n\nI trained two family of models\n- Metadata Models\n- Non Metadata Models\n\n## Metadata Models\n\nInitially I trained models with `vit_huge_plus_patch16_dinov3` backbone where the models have an auxiliary head to predict all the metadata which then feeds into the final heads to predict biomass. The models were trained with an auxiliary metadata loss along with the main loss. Later on I realized, metadata isn't really making much of a difference so I dropped it.\n\n## Non Metadata Models\n\nI picked up the popular MambaBlock based model that everyone was using and made it work on 2 backbones and then eventually 3 backbones. First model had `vit_huge_plus_patch16_dinov3` and `convnextv2_large` as they complement well and the second model added `vit_base_patch16_224.mae` to the mix. From my experience, choice of model made very little improvement to the overall performance as all my models got 0.64+ over a 4/5 fold ensemble. However, the Huge Plus version of DinoV3 always outperformed the Large version for me.\n\nAll the models were 2-stream (splitting image into left & right) based except for one model where I tried the whole image as input. It was almost similar in performance. \n\nAll the models had 3 heads. GDM and Total were inferred from the 3 predicted values.\n\nDuring training, I trained the backbones with 1/10th of the normal learning rate so as to not totally destroy the pre-trained weights.\n```\nclass BiomassModelV2(nn.Module):\n \"\"\"DINOv3 + Mamba Fusion + Multi-Head Regression\"\"\"\n\n def __init__(self, model1_name: str,\n model2_name: str,\n pretrained: bool = True\n ):\n super().__init__()\n self.model1_name = model1_name\n self.model2_name = model2_name\n\n self.backbone1 = timm.create_model(\n model1_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n self.backbone2 = timm.create_model(\n model2_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n nf1 = self.backbone1.num_features\n nf2 = 256\n nf = nf1 + nf2\n\n self.fusion1 = nn.Sequential(\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1)\n )\n self.fusion2 = nn.Sequential(\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1)\n )\n self.pool = nn.AdaptiveAvgPool1d(1)\n\n self.head_green = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_dead = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_clover = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n\n def forward(self, x):\n # x is a tuple (left, right)\n left, right = x\n \n x_l1 = self.fusion1(self.backbone1(left))\n x_l2 = self.fusion2(self.backbone2(left).flatten(2))\n x_r1 = self.fusion1(self.backbone1(right))\n x_r2 = self.fusion2(self.backbone2(right).flatten(2))\n\n x_l1 = self.pool(x_l1.transpose(1, 2)).flatten(1)\n x_l2 = self.pool(x_l2.transpose(1, 2)).flatten(1)\n x_r1 = self.pool(x_r1.transpose(1, 2)).flatten(1)\n x_r2 = self.pool(x_r2.transpose(1, 2)).flatten(1)\n\n x_cat = torch.cat([x_l1, x_l2, x_r1, x_r2], dim=1)\n\n green = self.head_green(x_cat)\n dead = self.head_dead(x_cat)\n clover = self.head_clover(x_cat)\n gdm = green + clover\n total = gdm + dead\n\n # Return as a single tensor (batch, 5)\n return torch.cat([green, dead, clover, gdm, total], dim=1)\n```\n\n# Data\n\nData was obviously the biggest factor of this competition and from the very beginning my focus was to extract the maximum information out of the training data provided. I used all the major augmentations available. Special mention to random grayscale to ensure the models learn features, not just color.\n\nI improvised two extra augmentations which I believe helped\n- One is a simple MixUp with a low lambda of 0.2-0.3.\n- In the other, I made 3 random vertical cuts on the image to divide it into 4 segments and I permute the order of the segments. This is a good augmentation as a jumbled up image still has the same amount of biomass depicted in it. Similar to this, I switched left and right images with 50% probability.\n\nI applied the above two augmentations together, not one or the other.\n\n# CV\n\nI tried 2 CV techniques, or rather I should say fold creation techniques - as after the first month, I gave up on CV scores and solely trusted LB scores. To avoid overfitting, I chose to always use all the models of all the folds when I train a certain architecture. None of the CV strategies actually felt robust - there were always 1-2 folds that did great and 1-2 folds that were bad on LB.\n\n- First CV technique was a GroupKFold one with the dates as groups.\n- Second CV technique was slightly complex. It is based on the embedding distance CV that was open sourced. I extended it to divide each validation set into 2 equal subsets - easy and hard (based on nearest neighbors). At this point, I expected the model weights giving a higher score on the hard subset to align more with LB, but the exact opposite was true. The easy subset and LB had consistently better alignment. My guess is that the hard samples in test are really hard and models can get a higher score by being excellent on easy samples and okayish on hard samples compared to good on both samples.\n\n```\ndef create_robust_cv(\n df,\n embeddings_col,\n target_names,\n n_splits=5,\n n_clusters=50,\n seed=42,\n k=5,\n distance_metric=\"euclidean\",\n):\n df = df.copy().reset_index(drop=True)\n\n # 1. Embeddings\n X = np.vstack(df[embeddings_col].values).astype(np.float32)\n X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-8\n\n # 2. Visual clustering\n kmeans = KMeans(\n n_clusters=n_clusters,\n random_state=seed,\n n_init=10,\n )\n df[\"visual_cluster\"] = kmeans.fit_predict(X)\n\n # 3. Stratification target\n weights = {\n \"Dry_Green_g\": 0.1,\n \"Dry_Dead_g\": 0.1,\n \"Dry_Clover_g\": 0.1,\n \"GDM_g\": 0.2,\n \"Dry_Total_g\": 0.5,\n }\n\n composite = sum(df[t] * weights.get(t, 0) for t in target_names)\n try:\n df[\"target_bins\"] = pd.qcut(composite, q=10, labels=False, duplicates=\"drop\")\n except ValueError:\n df[\"target_bins\"] = pd.qcut(composite, q=5, labels=False, duplicates=\"drop\")\n\n # 4. CV split\n sgkf = StratifiedGroupKFold(\n n_splits=n_splits,\n shuffle=True,\n random_state=seed,\n )\n df[\"fold\"] = -1\n df[\"val_difficulty\"] = None\n df[\"mean_knn_dist_to_train\"] = np.nan\n\n for f, (train_idx, val_idx) in enumerate(\n sgkf.split(df, df[\"target_bins\"], groups=df[\"visual_cluster\"])\n ):\n X_train, X_val = X[train_idx], X[val_idx]\n\n nn = NearestNeighbors(n_neighbors=k, metric=distance_metric)\n nn.fit(X_train)\n knn_dist, _ = nn.kneighbors(X_val)\n mean_dist = knn_dist.mean(axis=1)\n df.loc[val_idx, \"mean_knn_dist_to_train\"] = mean_dist\n\n # 5. EASY / HARD split\n order = np.argsort(mean_dist)\n mid = len(order) // 2\n\n easy = val_idx[order[:mid]]\n hard = val_idx[order[mid:]]\n\n df.loc[easy, [\"fold\", \"val_difficulty\"]] = [f, \"easy\"]\n df.loc[hard, [\"fold\", \"val_difficulty\"]] = [f + n_splits, \"hard\"]\n return df\n```\n\n# Inference\n\n- TTA actually made LB performance worse, so I turned it off. With heavy augmentation and ensemble, I didn't need it in the end.\n\n- I obviously did use the scaling that I showed in the [notebook I shared.](https://www.kaggle.com/code/mayukh18/dinov3-no-tta-postprocess). I ended up extending a little bit more. For clover, it really seems the data distribution is very different in test set. For Dry Dead, I feel all the models predict conservatively towards the mean when the actual values are more towards the fringes. The scaling gave me a decent boost (< 0.01) on the Public LB.\n\n\nSince neither the CV or LB was reliable, my strategy from the beginning was to train multiple architectures and ensemble. I didn't make any drastic changes in my models or training, mostly slow gradual improvements - based on stability of CV scores and improvements in LB scores. \n\nMost of my individual architectures were very similar in performance (~0.64+) over 5-folds. The last model set I trained - the 3 backbone non-metadata model was my highest scoring submission with 0.67.\n\nOn a final note, two things I tried making work but failed were synthetic dataset and test time training, especially the latter - since the test data was so different. It's kinda exciting to read how some top teams got it to work for them.", "votes": null }, { "id": "3399271", "postDate": "01/30/2026 14:41:15", "content": "Congratulations, interesting & simple.\nI really liked the 2nd cv technique, not sure how did you come up with it, difficult to come up with, also thought of dividing into hard and easy.
", "rawMarkdown": "Congratulations, interesting & simple.\nI really liked the 2nd cv technique, not sure how did you come up with it, difficult to come up with, also thought of dividing into hard and easy.", "votes": null }, { "id": "3399418", "postDate": "01/30/2026 20:26:16", "content": "congratulation 2nd place the cv technique realy stands out
", "rawMarkdown": "congratulation 2nd place the cv technique realy stands out", "votes": null } ], "comments": [ { "id": 3399271, "author_name": "ppilania1985", "author_url": "", "post_date": "01/30/2026 14:41:15", "content": "Congratulations, interesting & simple.\nI really liked the 2nd cv technique, not sure how did you come up with it, difficult to come up with, also thought of dividing into hard and easy.
", "votes": null, "replies": [] }, { "id": 3399418, "author_name": "bassetkerouche", "author_url": "", "post_date": "01/30/2026 20:26:16", "content": "congratulation 2nd place the cv technique realy stands out
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3399038": "Thanks to CSIRO and Kaggle for hosting and others for sharing their ideas throughout!\n\nI was honestly surprised to end up so high. I bet on heavy ensemble to safeguard against a shake-up but didn't expect it would be of any help in climbing the leaderboard. My submission was a 19 model ensemble across different model types and CV strategies. \n\n# 1. Model Architecture\n\nI trained two family of models\n- Metadata Models\n- Non Metadata Models\n\n## Metadata Models\n\nInitially I trained models with `vit_huge_plus_patch16_dinov3` backbone where the models have an auxiliary head to predict all the metadata which then feeds into the final heads to predict biomass. The models were trained with an auxiliary metadata loss along with the main loss. Later on I realized, metadata isn't really making much of a difference so I dropped it.\n\n## Non Metadata Models\n\nI picked up the popular MambaBlock based model that everyone was using and made it work on 2 backbones and then eventually 3 backbones. First model had `vit_huge_plus_patch16_dinov3` and `convnextv2_large` as they complement well and the second model added `vit_base_patch16_224.mae` to the mix. From my experience, choice of model made very little improvement to the overall performance as all my models got 0.64+ over a 4/5 fold ensemble. However, the Huge Plus version of DinoV3 always outperformed the Large version for me.\n\nAll the models were 2-stream (splitting image into left & right) based except for one model where I tried the whole image as input. It was almost similar in performance. \n\nAll the models had 3 heads. GDM and Total were inferred from the 3 predicted values.\n\nDuring training, I trained the backbones with 1/10th of the normal learning rate so as to not totally destroy the pre-trained weights.\n```\nclass BiomassModelV2(nn.Module):\n \"\"\"DINOv3 + Mamba Fusion + Multi-Head Regression\"\"\"\n\n def __init__(self, model1_name: str,\n model2_name: str,\n pretrained: bool = True\n ):\n super().__init__()\n self.model1_name = model1_name\n self.model2_name = model2_name\n\n self.backbone1 = timm.create_model(\n model1_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n self.backbone2 = timm.create_model(\n model2_name, pretrained=pretrained, num_classes=0, global_pool=''\n )\n nf1 = self.backbone1.num_features\n nf2 = 256\n nf = nf1 + nf2\n\n self.fusion1 = nn.Sequential(\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf1, kernel_size=5, dropout=0.1)\n )\n self.fusion2 = nn.Sequential(\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1),\n LocalMambaBlock(nf2, kernel_size=5, dropout=0.1)\n )\n self.pool = nn.AdaptiveAvgPool1d(1)\n\n self.head_green = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_dead = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n self.head_clover = nn.Sequential(\n nn.Linear(2*nf, nf // 2), nn.GELU(), nn.Dropout(0.2),\n nn.Linear(nf // 2, 1), nn.Softplus()\n )\n\n def forward(self, x):\n # x is a tuple (left, right)\n left, right = x\n \n x_l1 = self.fusion1(self.backbone1(left))\n x_l2 = self.fusion2(self.backbone2(left).flatten(2))\n x_r1 = self.fusion1(self.backbone1(right))\n x_r2 = self.fusion2(self.backbone2(right).flatten(2))\n\n x_l1 = self.pool(x_l1.transpose(1, 2)).flatten(1)\n x_l2 = self.pool(x_l2.transpose(1, 2)).flatten(1)\n x_r1 = self.pool(x_r1.transpose(1, 2)).flatten(1)\n x_r2 = self.pool(x_r2.transpose(1, 2)).flatten(1)\n\n x_cat = torch.cat([x_l1, x_l2, x_r1, x_r2], dim=1)\n\n green = self.head_green(x_cat)\n dead = self.head_dead(x_cat)\n clover = self.head_clover(x_cat)\n gdm = green + clover\n total = gdm + dead\n\n # Return as a single tensor (batch, 5)\n return torch.cat([green, dead, clover, gdm, total], dim=1)\n```\n\n# Data\n\nData was obviously the biggest factor of this competition and from the very beginning my focus was to extract the maximum information out of the training data provided. I used all the major augmentations available. Special mention to random grayscale to ensure the models learn features, not just color.\n\nI improvised two extra augmentations which I believe helped\n- One is a simple MixUp with a low lambda of 0.2-0.3.\n- In the other, I made 3 random vertical cuts on the image to divide it into 4 segments and I permute the order of the segments. This is a good augmentation as a jumbled up image still has the same amount of biomass depicted in it. Similar to this, I switched left and right images with 50% probability.\n\nI applied the above two augmentations together, not one or the other.\n\n# CV\n\nI tried 2 CV techniques, or rather I should say fold creation techniques - as after the first month, I gave up on CV scores and solely trusted LB scores. To avoid overfitting, I chose to always use all the models of all the folds when I train a certain architecture. None of the CV strategies actually felt robust - there were always 1-2 folds that did great and 1-2 folds that were bad on LB.\n\n- First CV technique was a GroupKFold one with the dates as groups.\n- Second CV technique was slightly complex. It is based on the embedding distance CV that was open sourced. I extended it to divide each validation set into 2 equal subsets - easy and hard (based on nearest neighbors). At this point, I expected the model weights giving a higher score on the hard subset to align more with LB, but the exact opposite was true. The easy subset and LB had consistently better alignment. My guess is that the hard samples in test are really hard and models can get a higher score by being excellent on easy samples and okayish on hard samples compared to good on both samples.\n\n```\ndef create_robust_cv(\n df,\n embeddings_col,\n target_names,\n n_splits=5,\n n_clusters=50,\n seed=42,\n k=5,\n distance_metric=\"euclidean\",\n):\n df = df.copy().reset_index(drop=True)\n\n # 1. Embeddings\n X = np.vstack(df[embeddings_col].values).astype(np.float32)\n X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-8\n\n # 2. Visual clustering\n kmeans = KMeans(\n n_clusters=n_clusters,\n random_state=seed,\n n_init=10,\n )\n df[\"visual_cluster\"] = kmeans.fit_predict(X)\n\n # 3. Stratification target\n weights = {\n \"Dry_Green_g\": 0.1,\n \"Dry_Dead_g\": 0.1,\n \"Dry_Clover_g\": 0.1,\n \"GDM_g\": 0.2,\n \"Dry_Total_g\": 0.5,\n }\n\n composite = sum(df[t] * weights.get(t, 0) for t in target_names)\n try:\n df[\"target_bins\"] = pd.qcut(composite, q=10, labels=False, duplicates=\"drop\")\n except ValueError:\n df[\"target_bins\"] = pd.qcut(composite, q=5, labels=False, duplicates=\"drop\")\n\n # 4. CV split\n sgkf = StratifiedGroupKFold(\n n_splits=n_splits,\n shuffle=True,\n random_state=seed,\n )\n df[\"fold\"] = -1\n df[\"val_difficulty\"] = None\n df[\"mean_knn_dist_to_train\"] = np.nan\n\n for f, (train_idx, val_idx) in enumerate(\n sgkf.split(df, df[\"target_bins\"], groups=df[\"visual_cluster\"])\n ):\n X_train, X_val = X[train_idx], X[val_idx]\n\n nn = NearestNeighbors(n_neighbors=k, metric=distance_metric)\n nn.fit(X_train)\n knn_dist, _ = nn.kneighbors(X_val)\n mean_dist = knn_dist.mean(axis=1)\n df.loc[val_idx, \"mean_knn_dist_to_train\"] = mean_dist\n\n # 5. EASY / HARD split\n order = np.argsort(mean_dist)\n mid = len(order) // 2\n\n easy = val_idx[order[:mid]]\n hard = val_idx[order[mid:]]\n\n df.loc[easy, [\"fold\", \"val_difficulty\"]] = [f, \"easy\"]\n df.loc[hard, [\"fold\", \"val_difficulty\"]] = [f + n_splits, \"hard\"]\n return df\n```\n\n# Inference\n\n- TTA actually made LB performance worse, so I turned it off. With heavy augmentation and ensemble, I didn't need it in the end.\n\n- I obviously did use the scaling that I showed in the [notebook I shared.](https://www.kaggle.com/code/mayukh18/dinov3-no-tta-postprocess). I ended up extending a little bit more. For clover, it really seems the data distribution is very different in test set. For Dry Dead, I feel all the models predict conservatively towards the mean when the actual values are more towards the fringes. The scaling gave me a decent boost (< 0.01) on the Public LB.\n\n\nSince neither the CV or LB was reliable, my strategy from the beginning was to train multiple architectures and ensemble. I didn't make any drastic changes in my models or training, mostly slow gradual improvements - based on stability of CV scores and improvements in LB scores. \n\nMost of my individual architectures were very similar in performance (~0.64+) over 5-folds. The last model set I trained - the 3 backbone non-metadata model was my highest scoring submission with 0.67.\n\nOn a final note, two things I tried making work but failed were synthetic dataset and test time training, especially the latter - since the test data was so different. It's kinda exciting to read how some top teams got it to work for them.", "3399271": "Congratulations, interesting & simple.\nI really liked the 2nd cv technique, not sure how did you come up with it, difficult to come up with, also thought of dividing into hard and easy.", "3399418": "congratulation 2nd place the cv technique realy stands out" }, "source": "meta" }