{
  "id": 671211,
  "title": "5th place solution",
  "url": "/competitions/csiro-biomass/writeups/5th-place-solution",
  "author_name": "",
  "post_date": "2026-02-02T07:41:28.330Z",
  "votes": 25,
  "comment_count": 2,
  "views": 0,
  "content": "<h1>5th Place Solution</h1>\n<p>Thank you to the hosts for this competition. It was a straightforward image task and I really enjoyed participating.</p>\n<h2>Overview</h2>\n<p>DINOv3's dense features turned out to be extremely powerful. My approach focused on leveraging these features effectively while preserving the pretrained weights as much as possible.\nUsing Global Features (CLS) would discard DINOv3's powerful local separation capability, so I designed the model to be as close to a segmentation task as possible.  </p>\n<p></p><ul>\n<li>End-to-end training using DINOv3's dense features</li>\n<li>Gradual Unfreeze up to 50% — no full fine-tuning</li>\n<li>Split 2000x1000 images in half and halve the labels to double the training data</li>\n<li>For shake robustness: ensemble of 6 models x 2 seeds across Large/Huge+ and multiple resolutions</li>\n<li>Inference optimizations (T4x2 parallel inference, FP16) to fit within the 9h time limit<p></p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>resolution</th>\n<th>CV (OOF R²)</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>DinoV3 Huge+ (child-exp012)</td>\n<td>672</td>\n<td>0.809</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Huge+ (child-exp017)</td>\n<td>768</td>\n<td>0.816</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Huge+ (child-exp026)</td>\n<td>672 + PatchDropout</td>\n<td>0.811</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Huge+ (child-exp032)</td>\n<td>864</td>\n<td>0.814</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Large (child-exp020)</td>\n<td>864</td>\n<td>0.818</td>\n<td>0.73</td>\n<td>0.66</td>\n</tr>\n<tr>\n<td>DinoV3 Large (child-exp037)</td>\n<td>960</td>\n<td>0.819</td>\n<td>0.74</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td></td></tr></tbody></table></li></ul>\n\n\n\n\n\n\n<h2>Model</h2>\n\n\n\n\n\n\n\n<h3>Why Density Estimation?</h3>\n<p>When visualizing DINOv3 features with PCA, you can see that even zero-shot, DINOv3 excels at local separation. It can clearly distinguish Green, Clover, Dead (partially), and Soil.\nFocus on Clover:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F7107f60d0070ecc805c896f0e23e253b%2Fimage-2.png?generation=1769875110579258&amp;alt=media\" alt=\"\"><br>\nFocus on Dead:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F6b1373047211edd2b79face09abd206f%2Fimage-3.png?generation=1769875122624262&amp;alt=media\" alt=\"\"><br>\n<em>Note: I intentionally selected samples where the separation is visually clear. Not all samples separate this cleanly.</em><br>\nDespite this excellent local separation ability, attaching a head to the CLS token discards spatial information, which felt wasteful. Instead, I adopted a Density Estimation approach: predict local biomass density from each patch token and sum (integrate) across the entire image.</p>\n<h3>Architecture: Density-Only Head</h3>\n<pre><code>Input image (1000x1000, one half of left-right split)\n   | resize\nInput (672x672x3)\n   |\nDINOv3 ViT-Huge+ backbone (frozen -&gt; gradual unfreeze)\n   |\nPatch tokens [B, 1764, 1280]  (42x42 patches, 1280-dim each)\n   | reshape\n[B, 1280, 42, 42]\n   |\n1x1 Conv (1280 -&gt; 5)   &lt;- Only 6,405 learnable parameters here\n   |\nSoftplus (non-negativity constraint: biomass density is physically non-negative)\n   |\nDensity map [B, 5, 42, 42]  (local density for 5 targets per patch)\n   |\nSum (aggregate all patches)\n   |\nPrediction [B, 5] = [Green, Dead, Clover, GDM, Total]\n</code></pre>\n<p>Key points:</p>\n<ul>\n<li><strong>CLS token is not used</strong>. Prediction relies solely on patch tokens (Density-Only)</li>\n<li><strong>1x1 Conv</strong> linearly transforms each patch's 1280-dim features into 5 biomass density values (no spatial convolution)</li>\n<li><strong>Softplus</strong> enforces non-negativity, with each patch representing \"how much biomass is in this region\"</li>\n<li>The final <strong>sum over all patches</strong> encodes the physical relationship: biomass = density x area\nGetting training to converge was challenging. I tried Conv3x3 to learn relationships between neighboring patches, but couldn't resolve NaN loss issues regardless of learning rate adjustments. Training only stabilized with the simpler Conv1x1. I also had to monitor training for the first 5-8 epochs to check if the validation score stabilized, manually stopping runs that didn't converge.</li>\n</ul>\n<h3>Resolution</h3>\n<p>Resolution was critical. Higher resolution consistently improved both CV and LB scores. I started at 224 and eventually reached 960.\nHigher resolution requires more training time and VRAM, so I started experiments at 256 and gradually increased to 512, then 768 in the final 3 weeks of the competition. The final models used Large at 960px and Huge+ at 864px. Training Huge+ at 864px took 20 hours for 5 folds, using 76GB VRAM at batch size 8 (Colab A100 80GB).</p>\n<h2>Training Strategy</h2>\n<p>Gradual Unfreeze was key. With a model like DINOv3 that has strong pretrained weights, full fine-tuning from the start destroys the valuable pretrained representations.</p>\n<h3>Gradual Unfreeze</h3>\n<ul>\n<li><strong>Epochs 1-5</strong>: Train only the head with a higher learning rate. Backbone is frozen.</li>\n<li><strong>Epochs 6-40</strong>: Gradually unfreeze layers starting from those closest to the head.\nFor example, DINOv3 Large has 24 Transformer blocks in total. With max_unfreeze_ratio=50%, up to 12 blocks are unfrozen by epoch 40.\nAn important design choice was <strong>not using a learning rate scheduler</strong>. Most schedulers apply warmup followed by gradual decay, but with this strategy, layers unfrozen in later epochs wouldn't receive sufficient learning. Instead, I linearly increased the backbone learning rate:</li>\n<li>Start (epoch 5): 1e-5</li>\n<li>End (epoch 40): 3e-5</li>\n</ul>\n<h2>Data Split (Left-Right Halving)</h2>\n<p>Many public notebooks split the 2000x1000 image into left and right halves, then concatenated the final layer outputs before feeding them to the head.\nMy approach was different: split into left and right halves <strong>and halve the labels too</strong>, effectively doubling the training data. This approach consistently outperformed the 2-stream concatenation method across DINOv3, EVA02-CLIP, and SigLIP.</p>\n<h2>Loss Function</h2>\n<p>The base loss was SmoothL1Loss, predicting all 5 targets.\nThe key addition was <strong>ConsistencyLoss</strong>, which enforces the physical relationships:</p>\n<ul>\n<li>Total = Green + Dead + Clover</li>\n<li>GDM = Green + Clover\nI initially thought predicting only 3 independent targets (Green, Dead, Clover) and deriving the rest would be more efficient. However, since there appeared to be some label noise between Clover and Green, predicting all 5 with a consistency constraint served as effective regularization.</li>\n</ul>\n<h2>CV Strategy</h2>\n<p>Adopting the group-based approach from <a href=\"https://www.kaggle.com/competitions/csiro-biomass/discussion/666501#3387810\" target=\"_blank\">this discussion post</a> stabilized the CV-LB correlation.\nSince Clover and Dead were difficult to predict, I used stratification to distribute them evenly across folds:</p>\n<pre><code>df_wide['clover_dead_presence'] = (\n   (df_wide['Dry_Clover_g'] &gt; 0).astype(int).astype(str) + '_' +\n   (df_wide['Dry_Dead_g'] &gt; 0).astype(int).astype(str)\n)\n</code></pre>\n<p>This was used as the stratification variable for StratifiedGroupKFold, ensuring samples with zero Clover or Dead values were evenly distributed across folds. Since many samples have zero values for these targets, naive random splitting could cause certain folds to be skewed.</p>\n<table>\n<thead>\n<tr>\n<th>Value</th>\n<th>Clover</th>\n<th>Dead</th>\n<th>Meaning</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0_0</td>\n<td>None (=0)</td>\n<td>None (=0)</td>\n<td>Both Clover and Dead are zero</td>\n</tr>\n<tr>\n<td>0_1</td>\n<td>None (=0)</td>\n<td>Present (&gt;0)</td>\n<td>Dead only</td>\n</tr>\n<tr>\n<td>1_0</td>\n<td>Present (&gt;0)</td>\n<td>None (=0)</td>\n<td>Clover only</td>\n</tr>\n<tr>\n<td>1_1</td>\n<td>Present (&gt;0)</td>\n<td>Present (&gt;0)</td>\n<td>Both present</td>\n</tr>\n<tr>\n<td>This made the CV-LB correlation very stable.</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2Fd58afcdf3d7ecab204891676dda19b1c%2Fall_exp_cv_lb_scatter.png?generation=1769875478188136&amp;alt=media\" alt=\"\"></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h2>Augmentation</h2>\n<p>The augmentations used:</p>\n<pre><code>HorizontalFlip (p=0.5)\nVerticalFlip (p=0.5)\nRandomRotate90 (p=0.5)\nRotate (limit=10, p=0.3)\nRandomResizedCrop (scale=0.85-1.0, ratio=0.95-1.05, p=0.5)\nColorJitter (brightness=0.3, contrast=0.3, saturation=0.3, hue=0.15, p=0.7)\nRandomGamma (gamma_limit=80-120, p=0.3)\nRandomBrightnessContrast (brightness=0.25, contrast=0.25, p=0.5)\nGaussianBlur (blur_limit=3-5, p=0.2)\nRandomShadow (p=0.1)\nRandomToneCurve (p=0.2)\n</code></pre>\n<p><br>\nRegarding RandomResizedCrop: I initially thought it shouldn't be used since it changes the amount of grass visible in the image. However, it actually improved accuracy. This may be because the camera-to-ground distance varies across photographers, and this augmentation improved robustness to such variation.\nOverly aggressive augmentations degraded performance:</p>\n<ul>\n<li>The host's paper mentioned images were captured with various camera types, so I tried adding diverse noise augmentations, but going too far hurt performance.</li>\n<li>RandomGridShuffle also did not help.</li>\n</ul>\n<h2>Shake Mitigation</h2>\n<h3>2-Seed Ensemble</h3>\n<p>Given the small dataset and regression task, a significant shake was expected. Inspired by <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/writeups/psi-5th-place-solution\" target=\"_blank\">Psi's discussion from the Feedback Prize competition</a>, I trained and evaluated with 2 seeds to verify that models achieved genuine generalization and to improve robustness.</p>\n<h2>Model Comparison</h2>\n<p>I participated from early in the competition, progressively upgrading models: SigLIP -&gt; EVA02-CLIP -&gt; DINOv3.\n<em>I didn't know DINOv3 would be this strong at the start. I usually prefer CLIP-based models, but DINOv3 was overwhelmingly superior in this competition.</em>\nAll models below used the same training pipeline (augmentation, learning rate, gradual unfreeze). Only batch size was adjusted to fit VRAM.</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>SigLIP</td>\n<td>0.75</td>\n<td>0.66</td>\n<td>0.55</td>\n</tr>\n<tr>\n<td>EVA02-CLIP</td>\n<td>0.75</td>\n<td>0.69</td>\n<td>0.584</td>\n</tr>\n<tr>\n<td>DINOv3 [Global Feature]</td>\n<td>0.79</td>\n<td>0.73</td>\n<td>0.64</td>\n</tr>\n<tr>\n<td>DINOv3 [Dense Feature]</td>\n<td>0.81</td>\n<td>0.75</td>\n<td>0.66</td>\n</tr>\n</tbody>\n</table>\n<h2>What Didn't Work</h2>\n<ul>\n<li><strong>TTA</strong> — No improvement</li>\n<li><strong>Sliding Window Inference</strong> — Instead of splitting left/right, sliding by 500px for 4 passes. Did not help.</li>\n<li><strong>Data Cleaning</strong> — The \"WA/2015-8-21\" date had notably noisy labels. Hand-correcting them improved CV but worsened LB. The test data likely contains the same labeling errors.</li>\n<li><strong>Auxiliary Losses</strong> — Using metadata (Height, Species, Season) as auxiliary targets did not help.</li>\n<li><strong>Depth Anything</strong> — Depth maps showed strong correlation with Height in train.csv. I tried multiplying depth with the density map to incorporate per-patch height information, but it didn't improve results.</li>\n<li><strong>EMA</strong> — Did not work, though my implementation may have been incorrect. I didn't investigate further.</li>\n</ul>",
  "messages": [
    {
      "id": "3399968",
      "postDate": "01/31/2026 16:12:49",
      "content": "<h1>5th Place Solution</h1>\n<p>Thank you to the hosts for this competition. It was a straightforward image task and I really enjoyed participating.</p>\n<h2>Overview</h2>\n<p>DINOv3's dense features turned out to be extremely powerful. My approach focused on leveraging these features effectively while preserving the pretrained weights as much as possible.\nUsing Global Features (CLS) would discard DINOv3's powerful local separation capability, so I designed the model to be as close to a segmentation task as possible.  </p>\n<p></p><ul>\n<li>End-to-end training using DINOv3's dense features</li>\n<li>Gradual Unfreeze up to 50% — no full fine-tuning</li>\n<li>Split 2000x1000 images in half and halve the labels to double the training data</li>\n<li>For shake robustness: ensemble of 6 models x 2 seeds across Large/Huge+ and multiple resolutions</li>\n<li>Inference optimizations (T4x2 parallel inference, FP16) to fit within the 9h time limit<p></p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>resolution</th>\n<th>CV (OOF R²)</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>DinoV3 Huge+ (child-exp012)</td>\n<td>672</td>\n<td>0.809</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Huge+ (child-exp017)</td>\n<td>768</td>\n<td>0.816</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Huge+ (child-exp026)</td>\n<td>672 + PatchDropout</td>\n<td>0.811</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Huge+ (child-exp032)</td>\n<td>864</td>\n<td>0.814</td>\n<td>0.75</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td>DinoV3 Large (child-exp020)</td>\n<td>864</td>\n<td>0.818</td>\n<td>0.73</td>\n<td>0.66</td>\n</tr>\n<tr>\n<td>DinoV3 Large (child-exp037)</td>\n<td>960</td>\n<td>0.819</td>\n<td>0.74</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td></td></tr></tbody></table></li></ul>\n\n\n\n\n\n\n<h2>Model</h2>\n\n\n\n\n\n\n\n<h3>Why Density Estimation?</h3>\n<p>When visualizing DINOv3 features with PCA, you can see that even zero-shot, DINOv3 excels at local separation. It can clearly distinguish Green, Clover, Dead (partially), and Soil.\nFocus on Clover:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F7107f60d0070ecc805c896f0e23e253b%2Fimage-2.png?generation=1769875110579258&amp;alt=media\" alt=\"\"><br>\nFocus on Dead:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F6b1373047211edd2b79face09abd206f%2Fimage-3.png?generation=1769875122624262&amp;alt=media\" alt=\"\"><br>\n<em>Note: I intentionally selected samples where the separation is visually clear. Not all samples separate this cleanly.</em><br>\nDespite this excellent local separation ability, attaching a head to the CLS token discards spatial information, which felt wasteful. Instead, I adopted a Density Estimation approach: predict local biomass density from each patch token and sum (integrate) across the entire image.</p>\n<h3>Architecture: Density-Only Head</h3>\n<pre><code>Input image (1000x1000, one half of left-right split)\n   | resize\nInput (672x672x3)\n   |\nDINOv3 ViT-Huge+ backbone (frozen -&gt; gradual unfreeze)\n   |\nPatch tokens [B, 1764, 1280]  (42x42 patches, 1280-dim each)\n   | reshape\n[B, 1280, 42, 42]\n   |\n1x1 Conv (1280 -&gt; 5)   &lt;- Only 6,405 learnable parameters here\n   |\nSoftplus (non-negativity constraint: biomass density is physically non-negative)\n   |\nDensity map [B, 5, 42, 42]  (local density for 5 targets per patch)\n   |\nSum (aggregate all patches)\n   |\nPrediction [B, 5] = [Green, Dead, Clover, GDM, Total]\n</code></pre>\n<p>Key points:</p>\n<ul>\n<li><strong>CLS token is not used</strong>. Prediction relies solely on patch tokens (Density-Only)</li>\n<li><strong>1x1 Conv</strong> linearly transforms each patch's 1280-dim features into 5 biomass density values (no spatial convolution)</li>\n<li><strong>Softplus</strong> enforces non-negativity, with each patch representing \"how much biomass is in this region\"</li>\n<li>The final <strong>sum over all patches</strong> encodes the physical relationship: biomass = density x area\nGetting training to converge was challenging. I tried Conv3x3 to learn relationships between neighboring patches, but couldn't resolve NaN loss issues regardless of learning rate adjustments. Training only stabilized with the simpler Conv1x1. I also had to monitor training for the first 5-8 epochs to check if the validation score stabilized, manually stopping runs that didn't converge.</li>\n</ul>\n<h3>Resolution</h3>\n<p>Resolution was critical. Higher resolution consistently improved both CV and LB scores. I started at 224 and eventually reached 960.\nHigher resolution requires more training time and VRAM, so I started experiments at 256 and gradually increased to 512, then 768 in the final 3 weeks of the competition. The final models used Large at 960px and Huge+ at 864px. Training Huge+ at 864px took 20 hours for 5 folds, using 76GB VRAM at batch size 8 (Colab A100 80GB).</p>\n<h2>Training Strategy</h2>\n<p>Gradual Unfreeze was key. With a model like DINOv3 that has strong pretrained weights, full fine-tuning from the start destroys the valuable pretrained representations.</p>\n<h3>Gradual Unfreeze</h3>\n<ul>\n<li><strong>Epochs 1-5</strong>: Train only the head with a higher learning rate. Backbone is frozen.</li>\n<li><strong>Epochs 6-40</strong>: Gradually unfreeze layers starting from those closest to the head.\nFor example, DINOv3 Large has 24 Transformer blocks in total. With max_unfreeze_ratio=50%, up to 12 blocks are unfrozen by epoch 40.\nAn important design choice was <strong>not using a learning rate scheduler</strong>. Most schedulers apply warmup followed by gradual decay, but with this strategy, layers unfrozen in later epochs wouldn't receive sufficient learning. Instead, I linearly increased the backbone learning rate:</li>\n<li>Start (epoch 5): 1e-5</li>\n<li>End (epoch 40): 3e-5</li>\n</ul>\n<h2>Data Split (Left-Right Halving)</h2>\n<p>Many public notebooks split the 2000x1000 image into left and right halves, then concatenated the final layer outputs before feeding them to the head.\nMy approach was different: split into left and right halves <strong>and halve the labels too</strong>, effectively doubling the training data. This approach consistently outperformed the 2-stream concatenation method across DINOv3, EVA02-CLIP, and SigLIP.</p>\n<h2>Loss Function</h2>\n<p>The base loss was SmoothL1Loss, predicting all 5 targets.\nThe key addition was <strong>ConsistencyLoss</strong>, which enforces the physical relationships:</p>\n<ul>\n<li>Total = Green + Dead + Clover</li>\n<li>GDM = Green + Clover\nI initially thought predicting only 3 independent targets (Green, Dead, Clover) and deriving the rest would be more efficient. However, since there appeared to be some label noise between Clover and Green, predicting all 5 with a consistency constraint served as effective regularization.</li>\n</ul>\n<h2>CV Strategy</h2>\n<p>Adopting the group-based approach from <a href=\"https://www.kaggle.com/competitions/csiro-biomass/discussion/666501#3387810\" target=\"_blank\">this discussion post</a> stabilized the CV-LB correlation.\nSince Clover and Dead were difficult to predict, I used stratification to distribute them evenly across folds:</p>\n<pre><code>df_wide['clover_dead_presence'] = (\n   (df_wide['Dry_Clover_g'] &gt; 0).astype(int).astype(str) + '_' +\n   (df_wide['Dry_Dead_g'] &gt; 0).astype(int).astype(str)\n)\n</code></pre>\n<p>This was used as the stratification variable for StratifiedGroupKFold, ensuring samples with zero Clover or Dead values were evenly distributed across folds. Since many samples have zero values for these targets, naive random splitting could cause certain folds to be skewed.</p>\n<table>\n<thead>\n<tr>\n<th>Value</th>\n<th>Clover</th>\n<th>Dead</th>\n<th>Meaning</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0_0</td>\n<td>None (=0)</td>\n<td>None (=0)</td>\n<td>Both Clover and Dead are zero</td>\n</tr>\n<tr>\n<td>0_1</td>\n<td>None (=0)</td>\n<td>Present (&gt;0)</td>\n<td>Dead only</td>\n</tr>\n<tr>\n<td>1_0</td>\n<td>Present (&gt;0)</td>\n<td>None (=0)</td>\n<td>Clover only</td>\n</tr>\n<tr>\n<td>1_1</td>\n<td>Present (&gt;0)</td>\n<td>Present (&gt;0)</td>\n<td>Both present</td>\n</tr>\n<tr>\n<td>This made the CV-LB correlation very stable.</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2Fd58afcdf3d7ecab204891676dda19b1c%2Fall_exp_cv_lb_scatter.png?generation=1769875478188136&amp;alt=media\" alt=\"\"></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h2>Augmentation</h2>\n<p>The augmentations used:</p>\n<pre><code>HorizontalFlip (p=0.5)\nVerticalFlip (p=0.5)\nRandomRotate90 (p=0.5)\nRotate (limit=10, p=0.3)\nRandomResizedCrop (scale=0.85-1.0, ratio=0.95-1.05, p=0.5)\nColorJitter (brightness=0.3, contrast=0.3, saturation=0.3, hue=0.15, p=0.7)\nRandomGamma (gamma_limit=80-120, p=0.3)\nRandomBrightnessContrast (brightness=0.25, contrast=0.25, p=0.5)\nGaussianBlur (blur_limit=3-5, p=0.2)\nRandomShadow (p=0.1)\nRandomToneCurve (p=0.2)\n</code></pre>\n<p><br>\nRegarding RandomResizedCrop: I initially thought it shouldn't be used since it changes the amount of grass visible in the image. However, it actually improved accuracy. This may be because the camera-to-ground distance varies across photographers, and this augmentation improved robustness to such variation.\nOverly aggressive augmentations degraded performance:</p>\n<ul>\n<li>The host's paper mentioned images were captured with various camera types, so I tried adding diverse noise augmentations, but going too far hurt performance.</li>\n<li>RandomGridShuffle also did not help.</li>\n</ul>\n<h2>Shake Mitigation</h2>\n<h3>2-Seed Ensemble</h3>\n<p>Given the small dataset and regression task, a significant shake was expected. Inspired by <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/writeups/psi-5th-place-solution\" target=\"_blank\">Psi's discussion from the Feedback Prize competition</a>, I trained and evaluated with 2 seeds to verify that models achieved genuine generalization and to improve robustness.</p>\n<h2>Model Comparison</h2>\n<p>I participated from early in the competition, progressively upgrading models: SigLIP -&gt; EVA02-CLIP -&gt; DINOv3.\n<em>I didn't know DINOv3 would be this strong at the start. I usually prefer CLIP-based models, but DINOv3 was overwhelmingly superior in this competition.</em>\nAll models below used the same training pipeline (augmentation, learning rate, gradual unfreeze). Only batch size was adjusted to fit VRAM.</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>SigLIP</td>\n<td>0.75</td>\n<td>0.66</td>\n<td>0.55</td>\n</tr>\n<tr>\n<td>EVA02-CLIP</td>\n<td>0.75</td>\n<td>0.69</td>\n<td>0.584</td>\n</tr>\n<tr>\n<td>DINOv3 [Global Feature]</td>\n<td>0.79</td>\n<td>0.73</td>\n<td>0.64</td>\n</tr>\n<tr>\n<td>DINOv3 [Dense Feature]</td>\n<td>0.81</td>\n<td>0.75</td>\n<td>0.66</td>\n</tr>\n</tbody>\n</table>\n<h2>What Didn't Work</h2>\n<ul>\n<li><strong>TTA</strong> — No improvement</li>\n<li><strong>Sliding Window Inference</strong> — Instead of splitting left/right, sliding by 500px for 4 passes. Did not help.</li>\n<li><strong>Data Cleaning</strong> — The \"WA/2015-8-21\" date had notably noisy labels. Hand-correcting them improved CV but worsened LB. The test data likely contains the same labeling errors.</li>\n<li><strong>Auxiliary Losses</strong> — Using metadata (Height, Species, Season) as auxiliary targets did not help.</li>\n<li><strong>Depth Anything</strong> — Depth maps showed strong correlation with Height in train.csv. I tried multiplying depth with the density map to incorporate per-patch height information, but it didn't improve results.</li>\n<li><strong>EMA</strong> — Did not work, though my implementation may have been incorrect. I didn't investigate further.</li>\n</ul>",
      "rawMarkdown": "# 5th Place Solution\n\nThank you to the hosts for this competition. It was a straightforward image task and I really enjoyed participating.\n\n## Overview\n\nDINOv3's dense features turned out to be extremely powerful. My approach focused on leveraging these features effectively while preserving the pretrained weights as much as possible.\nUsing Global Features (CLS) would discard DINOv3's powerful local separation capability, so I designed the model to be as close to a segmentation task as possible.  \n\n\n- End-to-end training using DINOv3's dense features\n- Gradual Unfreeze up to 50% — no full fine-tuning\n- Split 2000x1000 images in half and halve the labels to double the training data\n- For shake robustness: ensemble of 6 models x 2 seeds across Large/Huge+ and multiple resolutions\n- Inference optimizations (T4x2 parallel inference, FP16) to fit within the 9h time limit\n\n| model | resolution | CV (OOF R²) | Public | Private |\n| --- | --- | --- | --- | --- |\n| DinoV3 Huge+ (child-exp012) | 672 | 0.809 | 0.75 | 0.65 |\n| DinoV3 Huge+ (child-exp017) | 768 | 0.816 | 0.75 | 0.65 |\n| DinoV3 Huge+ (child-exp026) | 672 + PatchDropout | 0.811 | 0.75 | 0.65 |\n| DinoV3 Huge+ (child-exp032) | 864 | 0.814 | 0.75 | 0.65 |\n| DinoV3 Large (child-exp020) | 864 | 0.818 | 0.73 | 0.66 |\n| DinoV3 Large (child-exp037) | 960 | 0.819 | 0.74 | 0.65 |\n\n\n## Model\n### Why Density Estimation?\n\nWhen visualizing DINOv3 features with PCA, you can see that even zero-shot, DINOv3 excels at local separation. It can clearly distinguish Green, Clover, Dead (partially), and Soil.\n\nFocus on Clover:  \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F7107f60d0070ecc805c896f0e23e253b%2Fimage-2.png?generation=1769875110579258&alt=media)  \n\n\nFocus on Dead:  \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F6b1373047211edd2b79face09abd206f%2Fimage-3.png?generation=1769875122624262&alt=media)  \n\n\n*Note: I intentionally selected samples where the separation is visually clear. Not all samples separate this cleanly.*  \n\n\nDespite this excellent local separation ability, attaching a head to the CLS token discards spatial information, which felt wasteful. Instead, I adopted a Density Estimation approach: predict local biomass density from each patch token and sum (integrate) across the entire image.\n\n### Architecture: Density-Only Head\n\n```\nInput image (1000x1000, one half of left-right split)\n    | resize\nInput (672x672x3)\n    |\nDINOv3 ViT-Huge+ backbone (frozen -> gradual unfreeze)\n    |\nPatch tokens [B, 1764, 1280]  (42x42 patches, 1280-dim each)\n    | reshape\n[B, 1280, 42, 42]\n    |\n1x1 Conv (1280 -> 5)   <- Only 6,405 learnable parameters here\n    |\nSoftplus (non-negativity constraint: biomass density is physically non-negative)\n    |\nDensity map [B, 5, 42, 42]  (local density for 5 targets per patch)\n    |\nSum (aggregate all patches)\n    |\nPrediction [B, 5] = [Green, Dead, Clover, GDM, Total]\n```\n\nKey points:\n- **CLS token is not used**. Prediction relies solely on patch tokens (Density-Only)\n- **1x1 Conv** linearly transforms each patch's 1280-dim features into 5 biomass density values (no spatial convolution)\n- **Softplus** enforces non-negativity, with each patch representing \"how much biomass is in this region\"\n- The final **sum over all patches** encodes the physical relationship: biomass = density x area\n\nGetting training to converge was challenging. I tried Conv3x3 to learn relationships between neighboring patches, but couldn't resolve NaN loss issues regardless of learning rate adjustments. Training only stabilized with the simpler Conv1x1. I also had to monitor training for the first 5-8 epochs to check if the validation score stabilized, manually stopping runs that didn't converge.\n\n### Resolution\n\nResolution was critical. Higher resolution consistently improved both CV and LB scores. I started at 224 and eventually reached 960.\n\nHigher resolution requires more training time and VRAM, so I started experiments at 256 and gradually increased to 512, then 768 in the final 3 weeks of the competition. The final models used Large at 960px and Huge+ at 864px. Training Huge+ at 864px took 20 hours for 5 folds, using 76GB VRAM at batch size 8 (Colab A100 80GB).\n\n\n## Training Strategy\n\nGradual Unfreeze was key. With a model like DINOv3 that has strong pretrained weights, full fine-tuning from the start destroys the valuable pretrained representations.\n\n### Gradual Unfreeze\n\n- **Epochs 1-5**: Train only the head with a higher learning rate. Backbone is frozen.\n- **Epochs 6-40**: Gradually unfreeze layers starting from those closest to the head.\n\nFor example, DINOv3 Large has 24 Transformer blocks in total. With max_unfreeze_ratio=50%, up to 12 blocks are unfrozen by epoch 40.\n\nAn important design choice was **not using a learning rate scheduler**. Most schedulers apply warmup followed by gradual decay, but with this strategy, layers unfrozen in later epochs wouldn't receive sufficient learning. Instead, I linearly increased the backbone learning rate:\n- Start (epoch 5): 1e-5\n- End (epoch 40): 3e-5\n\n## Data Split (Left-Right Halving)\n\nMany public notebooks split the 2000x1000 image into left and right halves, then concatenated the final layer outputs before feeding them to the head.\n\nMy approach was different: split into left and right halves **and halve the labels too**, effectively doubling the training data. This approach consistently outperformed the 2-stream concatenation method across DINOv3, EVA02-CLIP, and SigLIP.\n\n\n## Loss Function\n\nThe base loss was SmoothL1Loss, predicting all 5 targets.\n\nThe key addition was **ConsistencyLoss**, which enforces the physical relationships:\n- Total = Green + Dead + Clover\n- GDM = Green + Clover\n\nI initially thought predicting only 3 independent targets (Green, Dead, Clover) and deriving the rest would be more efficient. However, since there appeared to be some label noise between Clover and Green, predicting all 5 with a consistency constraint served as effective regularization.\n\n\n## CV Strategy\n\nAdopting the group-based approach from [this discussion post](https://www.kaggle.com/competitions/csiro-biomass/discussion/666501#3387810) stabilized the CV-LB correlation.\n\nSince Clover and Dead were difficult to predict, I used stratification to distribute them evenly across folds:\n\n```python\ndf_wide['clover_dead_presence'] = (\n    (df_wide['Dry_Clover_g'] > 0).astype(int).astype(str) + '_' +\n    (df_wide['Dry_Dead_g'] > 0).astype(int).astype(str)\n)\n```\n\nThis was used as the stratification variable for StratifiedGroupKFold, ensuring samples with zero Clover or Dead values were evenly distributed across folds. Since many samples have zero values for these targets, naive random splitting could cause certain folds to be skewed.\n\n| Value | Clover | Dead | Meaning |\n|---|---|---|---|\n| 0_0 | None (=0) | None (=0) | Both Clover and Dead are zero |\n| 0_1 | None (=0) | Present (>0) | Dead only |\n| 1_0 | Present (>0) | None (=0) | Clover only |\n| 1_1 | Present (>0) | Present (>0) | Both present |\n\nThis made the CV-LB correlation very stable.  \n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2Fd58afcdf3d7ecab204891676dda19b1c%2Fall_exp_cv_lb_scatter.png?generation=1769875478188136&alt=media)  \n\n\n\n## Augmentation\n\nThe augmentations used:\n```\nHorizontalFlip (p=0.5)\nVerticalFlip (p=0.5)\nRandomRotate90 (p=0.5)\nRotate (limit=10, p=0.3)\nRandomResizedCrop (scale=0.85-1.0, ratio=0.95-1.05, p=0.5)\nColorJitter (brightness=0.3, contrast=0.3, saturation=0.3, hue=0.15, p=0.7)\nRandomGamma (gamma_limit=80-120, p=0.3)\nRandomBrightnessContrast (brightness=0.25, contrast=0.25, p=0.5)\nGaussianBlur (blur_limit=3-5, p=0.2)\nRandomShadow (p=0.1)\nRandomToneCurve (p=0.2)\n```  \n\n\nRegarding RandomResizedCrop: I initially thought it shouldn't be used since it changes the amount of grass visible in the image. However, it actually improved accuracy. This may be because the camera-to-ground distance varies across photographers, and this augmentation improved robustness to such variation.\n\nOverly aggressive augmentations degraded performance:\n- The host's paper mentioned images were captured with various camera types, so I tried adding diverse noise augmentations, but going too far hurt performance.\n- RandomGridShuffle also did not help.\n\n## Shake Mitigation\n\n### 2-Seed Ensemble\n\nGiven the small dataset and regression task, a significant shake was expected. Inspired by [Psi's discussion from the Feedback Prize competition](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/writeups/psi-5th-place-solution), I trained and evaluated with 2 seeds to verify that models achieved genuine generalization and to improve robustness.\n\n\n## Model Comparison\n\nI participated from early in the competition, progressively upgrading models: SigLIP -> EVA02-CLIP -> DINOv3.\n\n*I didn't know DINOv3 would be this strong at the start. I usually prefer CLIP-based models, but DINOv3 was overwhelmingly superior in this competition.*\n\nAll models below used the same training pipeline (augmentation, learning rate, gradual unfreeze). Only batch size was adjusted to fit VRAM.\n\n| Model | CV | Public | Private |\n|-------|----|--------|---------|\n| SigLIP | 0.75 | 0.66 | 0.55 |\n| EVA02-CLIP | 0.75 | 0.69 | 0.584 |\n| DINOv3 [Global Feature] | 0.79 | 0.73 | 0.64 |\n| DINOv3 [Dense Feature] | 0.81 | 0.75 | 0.66 |\n\n\n## What Didn't Work\n\n- **TTA** — No improvement\n- **Sliding Window Inference** — Instead of splitting left/right, sliding by 500px for 4 passes. Did not help.\n- **Data Cleaning** — The \"WA/2015-8-21\" date had notably noisy labels. Hand-correcting them improved CV but worsened LB. The test data likely contains the same labeling errors.\n- **Auxiliary Losses** — Using metadata (Height, Species, Season) as auxiliary targets did not help.\n- **Depth Anything** — Depth maps showed strong correlation with Height in train.csv. I tried multiplying depth with the density map to incorporate per-patch height information, but it didn't improve results.\n- **EMA** — Did not work, though my implementation may have been incorrect. I didn't investigate further.",
      "votes": null
    },
    {
      "id": "3400022",
      "postDate": "01/31/2026 17:49:19",
      "content": "<p>That's a very elegant solution! Congratulations on the gold medal!</p>",
      "rawMarkdown": "That's a very elegant solution! Congratulations on the gold medal!",
      "votes": null
    },
    {
      "id": "3401028",
      "postDate": "02/02/2026 19:28:30",
      "content": "<p>like it, simple yet on point. </p>",
      "rawMarkdown": "like it, simple yet on point.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3400022,
      "author_name": "prgckwb",
      "author_url": "",
      "post_date": "01/31/2026 17:49:19",
      "content": "<p>That's a very elegant solution! Congratulations on the gold medal!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3401028,
      "author_name": "davidvikstrand",
      "author_url": "",
      "post_date": "02/02/2026 19:28:30",
      "content": "<p>like it, simple yet on point. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3399968": "# 5th Place Solution\n\nThank you to the hosts for this competition. It was a straightforward image task and I really enjoyed participating.\n\n## Overview\n\nDINOv3's dense features turned out to be extremely powerful. My approach focused on leveraging these features effectively while preserving the pretrained weights as much as possible.\nUsing Global Features (CLS) would discard DINOv3's powerful local separation capability, so I designed the model to be as close to a segmentation task as possible.  \n\n\n- End-to-end training using DINOv3's dense features\n- Gradual Unfreeze up to 50% — no full fine-tuning\n- Split 2000x1000 images in half and halve the labels to double the training data\n- For shake robustness: ensemble of 6 models x 2 seeds across Large/Huge+ and multiple resolutions\n- Inference optimizations (T4x2 parallel inference, FP16) to fit within the 9h time limit\n\n| model | resolution | CV (OOF R²) | Public | Private |\n| --- | --- | --- | --- | --- |\n| DinoV3 Huge+ (child-exp012) | 672 | 0.809 | 0.75 | 0.65 |\n| DinoV3 Huge+ (child-exp017) | 768 | 0.816 | 0.75 | 0.65 |\n| DinoV3 Huge+ (child-exp026) | 672 + PatchDropout | 0.811 | 0.75 | 0.65 |\n| DinoV3 Huge+ (child-exp032) | 864 | 0.814 | 0.75 | 0.65 |\n| DinoV3 Large (child-exp020) | 864 | 0.818 | 0.73 | 0.66 |\n| DinoV3 Large (child-exp037) | 960 | 0.819 | 0.74 | 0.65 |\n\n\n## Model\n### Why Density Estimation?\n\nWhen visualizing DINOv3 features with PCA, you can see that even zero-shot, DINOv3 excels at local separation. It can clearly distinguish Green, Clover, Dead (partially), and Soil.\n\nFocus on Clover:  \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F7107f60d0070ecc805c896f0e23e253b%2Fimage-2.png?generation=1769875110579258&alt=media)  \n\n\nFocus on Dead:  \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2F6b1373047211edd2b79face09abd206f%2Fimage-3.png?generation=1769875122624262&alt=media)  \n\n\n*Note: I intentionally selected samples where the separation is visually clear. Not all samples separate this cleanly.*  \n\n\nDespite this excellent local separation ability, attaching a head to the CLS token discards spatial information, which felt wasteful. Instead, I adopted a Density Estimation approach: predict local biomass density from each patch token and sum (integrate) across the entire image.\n\n### Architecture: Density-Only Head\n\n```\nInput image (1000x1000, one half of left-right split)\n    | resize\nInput (672x672x3)\n    |\nDINOv3 ViT-Huge+ backbone (frozen -> gradual unfreeze)\n    |\nPatch tokens [B, 1764, 1280]  (42x42 patches, 1280-dim each)\n    | reshape\n[B, 1280, 42, 42]\n    |\n1x1 Conv (1280 -> 5)   <- Only 6,405 learnable parameters here\n    |\nSoftplus (non-negativity constraint: biomass density is physically non-negative)\n    |\nDensity map [B, 5, 42, 42]  (local density for 5 targets per patch)\n    |\nSum (aggregate all patches)\n    |\nPrediction [B, 5] = [Green, Dead, Clover, GDM, Total]\n```\n\nKey points:\n- **CLS token is not used**. Prediction relies solely on patch tokens (Density-Only)\n- **1x1 Conv** linearly transforms each patch's 1280-dim features into 5 biomass density values (no spatial convolution)\n- **Softplus** enforces non-negativity, with each patch representing \"how much biomass is in this region\"\n- The final **sum over all patches** encodes the physical relationship: biomass = density x area\n\nGetting training to converge was challenging. I tried Conv3x3 to learn relationships between neighboring patches, but couldn't resolve NaN loss issues regardless of learning rate adjustments. Training only stabilized with the simpler Conv1x1. I also had to monitor training for the first 5-8 epochs to check if the validation score stabilized, manually stopping runs that didn't converge.\n\n### Resolution\n\nResolution was critical. Higher resolution consistently improved both CV and LB scores. I started at 224 and eventually reached 960.\n\nHigher resolution requires more training time and VRAM, so I started experiments at 256 and gradually increased to 512, then 768 in the final 3 weeks of the competition. The final models used Large at 960px and Huge+ at 864px. Training Huge+ at 864px took 20 hours for 5 folds, using 76GB VRAM at batch size 8 (Colab A100 80GB).\n\n\n## Training Strategy\n\nGradual Unfreeze was key. With a model like DINOv3 that has strong pretrained weights, full fine-tuning from the start destroys the valuable pretrained representations.\n\n### Gradual Unfreeze\n\n- **Epochs 1-5**: Train only the head with a higher learning rate. Backbone is frozen.\n- **Epochs 6-40**: Gradually unfreeze layers starting from those closest to the head.\n\nFor example, DINOv3 Large has 24 Transformer blocks in total. With max_unfreeze_ratio=50%, up to 12 blocks are unfrozen by epoch 40.\n\nAn important design choice was **not using a learning rate scheduler**. Most schedulers apply warmup followed by gradual decay, but with this strategy, layers unfrozen in later epochs wouldn't receive sufficient learning. Instead, I linearly increased the backbone learning rate:\n- Start (epoch 5): 1e-5\n- End (epoch 40): 3e-5\n\n## Data Split (Left-Right Halving)\n\nMany public notebooks split the 2000x1000 image into left and right halves, then concatenated the final layer outputs before feeding them to the head.\n\nMy approach was different: split into left and right halves **and halve the labels too**, effectively doubling the training data. This approach consistently outperformed the 2-stream concatenation method across DINOv3, EVA02-CLIP, and SigLIP.\n\n\n## Loss Function\n\nThe base loss was SmoothL1Loss, predicting all 5 targets.\n\nThe key addition was **ConsistencyLoss**, which enforces the physical relationships:\n- Total = Green + Dead + Clover\n- GDM = Green + Clover\n\nI initially thought predicting only 3 independent targets (Green, Dead, Clover) and deriving the rest would be more efficient. However, since there appeared to be some label noise between Clover and Green, predicting all 5 with a consistency constraint served as effective regularization.\n\n\n## CV Strategy\n\nAdopting the group-based approach from [this discussion post](https://www.kaggle.com/competitions/csiro-biomass/discussion/666501#3387810) stabilized the CV-LB correlation.\n\nSince Clover and Dead were difficult to predict, I used stratification to distribute them evenly across folds:\n\n```python\ndf_wide['clover_dead_presence'] = (\n    (df_wide['Dry_Clover_g'] > 0).astype(int).astype(str) + '_' +\n    (df_wide['Dry_Dead_g'] > 0).astype(int).astype(str)\n)\n```\n\nThis was used as the stratification variable for StratifiedGroupKFold, ensuring samples with zero Clover or Dead values were evenly distributed across folds. Since many samples have zero values for these targets, naive random splitting could cause certain folds to be skewed.\n\n| Value | Clover | Dead | Meaning |\n|---|---|---|---|\n| 0_0 | None (=0) | None (=0) | Both Clover and Dead are zero |\n| 0_1 | None (=0) | Present (>0) | Dead only |\n| 1_0 | Present (>0) | None (=0) | Clover only |\n| 1_1 | Present (>0) | Present (>0) | Both present |\n\nThis made the CV-LB correlation very stable.  \n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3836857%2Fd58afcdf3d7ecab204891676dda19b1c%2Fall_exp_cv_lb_scatter.png?generation=1769875478188136&alt=media)  \n\n\n\n## Augmentation\n\nThe augmentations used:\n```\nHorizontalFlip (p=0.5)\nVerticalFlip (p=0.5)\nRandomRotate90 (p=0.5)\nRotate (limit=10, p=0.3)\nRandomResizedCrop (scale=0.85-1.0, ratio=0.95-1.05, p=0.5)\nColorJitter (brightness=0.3, contrast=0.3, saturation=0.3, hue=0.15, p=0.7)\nRandomGamma (gamma_limit=80-120, p=0.3)\nRandomBrightnessContrast (brightness=0.25, contrast=0.25, p=0.5)\nGaussianBlur (blur_limit=3-5, p=0.2)\nRandomShadow (p=0.1)\nRandomToneCurve (p=0.2)\n```  \n\n\nRegarding RandomResizedCrop: I initially thought it shouldn't be used since it changes the amount of grass visible in the image. However, it actually improved accuracy. This may be because the camera-to-ground distance varies across photographers, and this augmentation improved robustness to such variation.\n\nOverly aggressive augmentations degraded performance:\n- The host's paper mentioned images were captured with various camera types, so I tried adding diverse noise augmentations, but going too far hurt performance.\n- RandomGridShuffle also did not help.\n\n## Shake Mitigation\n\n### 2-Seed Ensemble\n\nGiven the small dataset and regression task, a significant shake was expected. Inspired by [Psi's discussion from the Feedback Prize competition](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/writeups/psi-5th-place-solution), I trained and evaluated with 2 seeds to verify that models achieved genuine generalization and to improve robustness.\n\n\n## Model Comparison\n\nI participated from early in the competition, progressively upgrading models: SigLIP -> EVA02-CLIP -> DINOv3.\n\n*I didn't know DINOv3 would be this strong at the start. I usually prefer CLIP-based models, but DINOv3 was overwhelmingly superior in this competition.*\n\nAll models below used the same training pipeline (augmentation, learning rate, gradual unfreeze). Only batch size was adjusted to fit VRAM.\n\n| Model | CV | Public | Private |\n|-------|----|--------|---------|\n| SigLIP | 0.75 | 0.66 | 0.55 |\n| EVA02-CLIP | 0.75 | 0.69 | 0.584 |\n| DINOv3 [Global Feature] | 0.79 | 0.73 | 0.64 |\n| DINOv3 [Dense Feature] | 0.81 | 0.75 | 0.66 |\n\n\n## What Didn't Work\n\n- **TTA** — No improvement\n- **Sliding Window Inference** — Instead of splitting left/right, sliding by 500px for 4 passes. Did not help.\n- **Data Cleaning** — The \"WA/2015-8-21\" date had notably noisy labels. Hand-correcting them improved CV but worsened LB. The test data likely contains the same labeling errors.\n- **Auxiliary Losses** — Using metadata (Height, Species, Season) as auxiliary targets did not help.\n- **Depth Anything** — Depth maps showed strong correlation with Height in train.csv. I tried multiplying depth with the density map to incorporate per-patch height information, but it didn't improve results.\n- **EMA** — Did not work, though my implementation may have been incorrect. I didn't investigate further.",
    "3400022": "That's a very elegant solution! Congratulations on the gold medal!",
    "3401028": "like it, simple yet on point."
  },
  "source": "meta"
}