{ "id": 670956, "title": "27th Place Solution", "url": "/competitions/csiro-biomass/discussion/670956", "author_name": "Pushpak Bhoge", "post_date": "2026-01-30T13:51:48.809000", "votes": 5, "comment_count": 0, "views": 0, "content": "

First of all, Thanks to the host for organizing such interesting and fun competitions. This was my first serious participation, and I am very happy my solution was actually decent enough to rank 27th. The experiences and knowledge gained during this particiaption has been extremely mind-opening. So thank you for broadening our knowledge horizons. This is also my first time using cross-validation and ensembling strategies.

\n

☘️ EDA

\n

After analysing the data, I noted the following observation

\n\n

☘️ Data Preprocessing and Strategy

\n\n

☘️ Cross Folds Creation

\n

As the dataset was very small, I tried to create data splits where each train se set would see the entire spectrum of values. But this was not happening in single cross-folds as I was always leaking dates, state in train-val splits. So I decided to create two cross validations splits.

\n\n

☘️ Augmentations

\n

First of all, what worked

\n\n

☘️ Architectural Overview

\n\n

☘️ Loss & Optimization

\n\n

☘️ What did not Work

\n\n

I have written quite extensive code for this. You can find the code on my GitHub repository\nThis is my submission notebook I am planning to release an exhaustively detailed article about this, which will take a week or two.

\n

Finally, Why were CNNs consistently inferior to ViTs in this competition? I’d love to hear insights from others.

", "messages": [ { "id": 3399247, "postDate": "2026-01-30T13:51:48.810Z", "content": "

First of all, Thanks to the host for organizing such interesting and fun competitions. This was my first serious participation, and I am very happy my solution was actually decent enough to rank 27th. The experiences and knowledge gained during this particiaption has been extremely mind-opening. So thank you for broadening our knowledge horizons. This is also my first time using cross-validation and ensembling strategies.

\n

☘️ EDA

\n

After analysing the data, I noted the following observation

\n\n

☘️ Data Preprocessing and Strategy

\n\n

☘️ Cross Folds Creation

\n

As the dataset was very small, I tried to create data splits where each train se set would see the entire spectrum of values. But this was not happening in single cross-folds as I was always leaking dates, state in train-val splits. So I decided to create two cross validations splits.

\n\n

☘️ Augmentations

\n

First of all, what worked

\n\n

☘️ Architectural Overview

\n\n

☘️ Loss & Optimization

\n\n

☘️ What did not Work

\n\n

I have written quite extensive code for this. You can find the code on my GitHub repository\nThis is my submission notebook I am planning to release an exhaustively detailed article about this, which will take a week or two.

\n

Finally, Why were CNNs consistently inferior to ViTs in this competition? I’d love to hear insights from others.

", "rawMarkdown": "First of all, Thanks to the host for organizing such interesting and fun competitions. This was my first serious participation, and I am very happy my solution was actually decent enough to rank 27th. The experiences and knowledge gained during this particiaption has been extremely mind-opening. So thank you for broadening our knowledge horizons. This is also my first time using cross-validation and ensembling strategies.\n\n# ☘️ EDA\nAfter analysing the data, I noted the following observation\n* The biomass targets follow clear additive relationships `Dry_Total_g = Dry_Green_g + Dry_Dead_g + Dry_Clover_g` and `GDM_g = Dry_Green_g + Dry_Clover_g` \n* Target-wise distribution characteristics, \n * `Dry_Total_g` and `GDM_g ` were the least noisy, and only these two were roughly Gaussian.\n * `Dry_Green_g` had a power law distribution with a shorter head.\n * `Dry_Dead_g` and `Dry_Clover_g` also had a power law distribution with much taller heads and long tails\n* Species targets were noisy; some species were visually very similar (especially subspecies)\n* The 3 auxiliary targets shown strong correlation with the values, for example\n * If there is no clover or its subspecies present, then `Dry_Clover_g` was often zero.\n * If height was high, the targets also had higher mass, except in the case of the `lucerne`, where the mass values were not scaled in the same proportions\n\n# ☘️ Data Preprocessing and Strategy\n* For biomass targets since `Dry_Dead_g` and `Dry_Clover_g` were noisy and had a lot of zeros, I decided to predict `Dry_Total_g`, `GDM_g`, and `Dry_Green_g` and derive rest two from them. I used log norm transformation for these biomass targets.\n\n ```python\n means = torch.tensor([3.6171854219925024, 3.217874424165861, 2.832479241567281])\n stds = torch.tensor([0.5573042636308412, 0.8185179833598041, 1.2134970356508157])\n normliazed = (torch.log(data + 1) - means) / stds\n data = torch.exp((normliazed * stds) + means) - 1\n ```\n* For auxiliary regression targets, I did the following pre-processing\n * Auxiliary target `Height_Ave_cm` is also normalized and denormalized in the same way with a mean of `1.7949544186979798` and std of `0.7399843934368923`. As the height also have power law distribution with a tall head and long tails\n * Auxiliary target `Pre_GSHH_NDVI` was not transformed and used as it is\n * For Species I preprocessed and grouped similar-looking species, there was also a capitalization mismatch whic were resolved. I reduced 15 categories to 8 categories. I used the following function. This grouping gave a ~0.03 R² boost.\n\n ```\n def reduce_species(species):\n new_species = []\n for one_species in species.split(\"_\"):\n if one_species in (\"BarleyGrass\", \"Barleygrass\", \"Bromegrass\", \"SilverGrass\", \"SpearGrass\"):\n new_species.append(\"grass\")\n elif one_species in (\"SubcloverDalkeith\", \"SubcloverLosa\", \"Clover\"):\n new_species.append(\"clover\")\n elif one_species in (\"Capeweed\", \"CrumbWeed\"):\n new_species.append(\"weed\")\n elif one_species == \"Mixed\":\n new_species.append(\"ryegrass\")\n else:\n new_species.append(one_species.lower())\n return list(set(new_species))\n ```\n\n\n# ☘️ Cross Folds Creation\nAs the dataset was very small, I tried to create data splits where each train se set would see the entire spectrum of values. But this was not happening in single cross-folds as I was always leaking dates, state in train-val splits. So I decided to create two cross validations splits.\n* 5 Cross Folds -> I called this `clean_folds`, where I stratified across `State` and `Sampling_Date`, using the following code \n\n ```\n df = pd.read_csv('./datasets/csiro-biomass/train_unwrapped.csv')\n df['session_id'] = df['State'].astype(str) + \"_\" + df['Sampling_Date'].astype(str)\n df['biomass_bin'] = pd.qcut(df['Dry_Total_g'], q=5, labels=False)\n sgkf = StratifiedGroupKFold(n_splits=5, shuffle=False)\n ```\n* 3 Cross Folds -> Here I simply sorted by `Dry_Total_g` in descending order and created 3 groups so each train set will see a full spectrum of target values. Only 3 folds were used to avoid overly similar train/val splits.\n\n# ☘️ Augmentations\n### First of all, what worked\n* Tested with a combination of image resolution `384, 512, 768, 1024` image size with patch size of `2x1, 4x2`. Found image size of `768x768` with two patches `left and right` worked the best, so that's how the rest of the experiments were conducted.\n* For augmentation, I used some global transforms, which were applied on full image. And local transforms, which were applied patchwise. I wrote a custom compose class for this. I also experimented with switching to a weaker pipeline in the middle of training. At worst, it did nothing, and at best, gave a little bump in metrics, so I used it for the rest of the runs by default, switching the pipeline in the last 5-10 epochs\n\n ```\n NORM_SETTING = {\"type\": \"Normalize\", \"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]}\n STRONG_AUG_TRANSFORMS = {\n \"type\": \"albu\",\n \"compose_cls\": \"tiled_compose\",\n \"global_transforms\": [\n {\n \"type\": \"OneOf\", \"p\": 0.5,\n \"transforms\": [\n {\"type\": \"AutoContrast\", \"method\": \"cdf\", \"p\": 0.5},\n {\"type\": \"RandomBrightnessContrast\", \"brightness_limit\": 0.3, \"contrast_limit\": 0.2, \"p\": 0.5},\n {\"type\": \"RandomToneCurve\", \"scale\": 0.3, \"p\": 0.3},\n {\"type\": \"HueSaturationValue\", \"hue_shift_limit\": 20, \"sat_shift_limit\": 30, \"val_shift_limit\": 20, \"p\": 0.5},\n {\"type\": \"PlanckianJitter\", \"mode\": \"blackbody\", \"temperature_limit\": [4000, 8000], \"sampling_method\": \"gaussian\", \"p\": 0.5},\n {\"type\": \"CLAHE\", \"p\": 0.5},~~~~\n ],\n },\n {\"type\": \"ShiftScaleRotate\", \"shift_limit\": [-0.1, 0.1], \"scale_limit\": [-0.1, 0.1], \"rotate_limit\": [10, 10], \"border_mode\": 4, \"p\": 0.5},\n {\"type\": \"GaussianBlur\", \"blur_limit\": 3, \"sigma_limit\": 1, \"p\": 0.5},\n {\n \"type\": \"OneOf\", \"p\": 0.5,\n \"transforms\": [\n {\"type\": \"GridDistortion\", \"distort_limit\": 0.5, \"p\": 1.0},\n {\"type\": \"OpticalDistortion\", \"p\": 1.0},\n ],\n },\n ],\n \"local_transforms\": [\n {\"type\": \"D4\", \"p\": 1.0},\n {\n \"type\": \"OneOf\", \"p\": 0.5,\n \"transforms\": [\n {\"type\": \"Sharpen\", \"alpha\": [0.1, 0.5], \"lightness\": [1.0, 1.0], \"method\": \"kernel\", \"p\": 0.8},\n {\"type\": \"Emboss\", \"alpha\": [0.4, 0.5], \"strength\": [0.3, 0.7], \"p\": 0.8},\n ],\n },\n ],\n \"cvt_transforms\": [NORM_SETTING, {\"type\": \"ToTensorV2\"}],\n \"n_patch_h\": 1, \"n_patch_w\": 2,\n \"target_size\": [IMAGE_SIZE, IMAGE_SIZE * 2]\n }\n ```\n\n\n# ☘️ Architectural Overview\n* Used DinoV3 ViT large as my backbone, as almost everyone else in this competition did. \n* The image patches were passed to a shared backbone, and the features of shape [N, dim] from both patches were concatenated.\n* First, the height, NDVI, and species were predicted from the backbone features using a simple 2-layer MLP. The same backbone features were passed into an MLP projection network, which produced an embedding. \n* Then, the NDVI, height, and sigmoid-activated species logits were concatenated and then passed to a FiLM layer, which then modulated the embedding produced by the above MLP projection network. Earlier, I was simply concatenating them, but due to the size of the embedding, it dominated. I was looking for better options, so I thought to predict the scale shift parameter and transfrom embedding that way. I ended up designing the FiLM layer. (Fun fact, I initially thought, I discovered a novel idea, as this took me from 0.68 to 0.72 in public LB. Later discovered the FiLM paper 😅)\n* Finally, I predicted 3 biomass targets at once from the modulated embeddings\n\n# ☘️ Loss & Optimization\n* For loss, I used a weighted SmoothL1 loss for biomass targets and a SmoothL1 loss for height and NDVI targets. \n* For the species head, I used a simple BCE loss with the following weights for each category\n ```\n SPECIES = [\"clover\", \"ryegrass\", \"phalaris\", \"grass\", \"fescue\", \"lucerne\", \"weed\", \"whiteclover\"]\n SPECIES_WEIGHTS = [1.68, 1.86, 2.76, 3.59, 4.09, 5.01, 5.89, 7.08] # roots sqrt(N / Nc)\n ```\n* The above 3 losses were combined with weights `{\"biomass\": 3.0, \"aux\": 1.0, \"species\": 0.2}`\n* For training, I used AdamW with lr of 1e-5 and weight decay of 0.3 (I know it's high, but it regularizes the model ), did warmup for 5 epochs, then reduced the lr for the rest of the training using cosine annealing\n* The training was unstable, so I used EMA and gradient clipping to stabilize the training\n* All three heads were trained together on 3CV and 5CV. These two were my final submissions.\n\n# ☘️ What did not Work\n* As the dataset provided in competetion was very small, I found a large dataset (was ~270GB with ~40k images) which I have processed and uploaded to Kaggle. It was not much of a help, though. [here](https://www.kaggle.com/datasets/pushpakbhoge/grass-clover-datasets)\n* **DinoV2 Pretraining**: Fine-tuned DinoV2 (ViT-small) on the largely unlabelled data for 100 epochs. It failed to beat the original weights despite promising attention maps. Halted due to compute limits, though this remains a potential avenue for exploration.\n* **Segmentation**: Trained a 16-class model on synthetic data to create guided attention maps for the downstream CNN [code](https://github.com/PushpakBhoge/csiro_biomass_regression/blob/main/components/bricks.py#L73).\n* **CNN Architecture**: Initially used ConvNeXt-atto (ImageNet1k pretrained), theorizing that a smaller model suited the dataset size. Integrated the segmentation maps for attention and tested various pooling methods.\n* **CNN Outcome**: Despite extensive tuning, the CNN could not outperform ViT. Realizing late that superior features were critical, I tried ConvNeXt-Base with DinoV3 weights but only achieved 0.65 Public LB, suggesting ConvNeXt features were less effective than ViT-base here. These weights for ConvNeXt were distilled from ViT, so maybe that has something to do with it\n* **Meta-Averaging**: Tried meta-averaging auxiliary predictions before biomass inference, neutral effect.\n* **Unfreezing-backbone**: Tried scaled lr across different backbone stages, and training small model from scratch did not work\n* **Biomass Summation**: Attempted to approximate density maps by summing patch biomass. Summing raw channels caused exploding gradients, while 3x3 average pooling stabilized training but yielded no gains.\n* **Spatial Pooling**: Implemented a convolutional spatial pooling layer (reducing 24x24 features to 1x1) followed by MLPs. Summing these outputs improved Public LB by 0.01, peaking at 0.59 [code](https://github.com/PushpakBhoge/csiro_biomass_regression/blob/main/components/bricks.py#L217).\n* **FPN & Multi-Stage Heads**: Tested fusing backbone features at varying resolutions (FPN) and attaching heads directly to Stage 2 and 3 outputs. Neither approach improved the score.\n\n\nI have written quite extensive code for this. You can find the code on my [GitHub repository](https://github.com/PushpakBhoge/csiro_biomass_regression)\nThis is my [submission notebook](https://www.kaggle.com/code/pushpakbhoge/csiro-image2biomas-submission-notebook) I am planning to release an exhaustively detailed article about this, which will take a week or two.\n\nFinally, Why were CNNs consistently inferior to ViTs in this competition? I’d love to hear insights from others.", "votes": 5 } ], "comments": [], "raw_markdown_by_id": { "3399247": "First of all, Thanks to the host for organizing such interesting and fun competitions. This was my first serious participation, and I am very happy my solution was actually decent enough to rank 27th. The experiences and knowledge gained during this particiaption has been extremely mind-opening. So thank you for broadening our knowledge horizons. This is also my first time using cross-validation and ensembling strategies.\n\n# ☘️ EDA\nAfter analysing the data, I noted the following observation\n* The biomass targets follow clear additive relationships `Dry_Total_g = Dry_Green_g + Dry_Dead_g + Dry_Clover_g` and `GDM_g = Dry_Green_g + Dry_Clover_g` \n* Target-wise distribution characteristics, \n * `Dry_Total_g` and `GDM_g ` were the least noisy, and only these two were roughly Gaussian.\n * `Dry_Green_g` had a power law distribution with a shorter head.\n * `Dry_Dead_g` and `Dry_Clover_g` also had a power law distribution with much taller heads and long tails\n* Species targets were noisy; some species were visually very similar (especially subspecies)\n* The 3 auxiliary targets shown strong correlation with the values, for example\n * If there is no clover or its subspecies present, then `Dry_Clover_g` was often zero.\n * If height was high, the targets also had higher mass, except in the case of the `lucerne`, where the mass values were not scaled in the same proportions\n\n# ☘️ Data Preprocessing and Strategy\n* For biomass targets since `Dry_Dead_g` and `Dry_Clover_g` were noisy and had a lot of zeros, I decided to predict `Dry_Total_g`, `GDM_g`, and `Dry_Green_g` and derive rest two from them. I used log norm transformation for these biomass targets.\n\n ```python\n means = torch.tensor([3.6171854219925024, 3.217874424165861, 2.832479241567281])\n stds = torch.tensor([0.5573042636308412, 0.8185179833598041, 1.2134970356508157])\n normliazed = (torch.log(data + 1) - means) / stds\n data = torch.exp((normliazed * stds) + means) - 1\n ```\n* For auxiliary regression targets, I did the following pre-processing\n * Auxiliary target `Height_Ave_cm` is also normalized and denormalized in the same way with a mean of `1.7949544186979798` and std of `0.7399843934368923`. As the height also have power law distribution with a tall head and long tails\n * Auxiliary target `Pre_GSHH_NDVI` was not transformed and used as it is\n * For Species I preprocessed and grouped similar-looking species, there was also a capitalization mismatch whic were resolved. I reduced 15 categories to 8 categories. I used the following function. This grouping gave a ~0.03 R² boost.\n\n ```\n def reduce_species(species):\n new_species = []\n for one_species in species.split(\"_\"):\n if one_species in (\"BarleyGrass\", \"Barleygrass\", \"Bromegrass\", \"SilverGrass\", \"SpearGrass\"):\n new_species.append(\"grass\")\n elif one_species in (\"SubcloverDalkeith\", \"SubcloverLosa\", \"Clover\"):\n new_species.append(\"clover\")\n elif one_species in (\"Capeweed\", \"CrumbWeed\"):\n new_species.append(\"weed\")\n elif one_species == \"Mixed\":\n new_species.append(\"ryegrass\")\n else:\n new_species.append(one_species.lower())\n return list(set(new_species))\n ```\n\n\n# ☘️ Cross Folds Creation\nAs the dataset was very small, I tried to create data splits where each train se set would see the entire spectrum of values. But this was not happening in single cross-folds as I was always leaking dates, state in train-val splits. So I decided to create two cross validations splits.\n* 5 Cross Folds -> I called this `clean_folds`, where I stratified across `State` and `Sampling_Date`, using the following code \n\n ```\n df = pd.read_csv('./datasets/csiro-biomass/train_unwrapped.csv')\n df['session_id'] = df['State'].astype(str) + \"_\" + df['Sampling_Date'].astype(str)\n df['biomass_bin'] = pd.qcut(df['Dry_Total_g'], q=5, labels=False)\n sgkf = StratifiedGroupKFold(n_splits=5, shuffle=False)\n ```\n* 3 Cross Folds -> Here I simply sorted by `Dry_Total_g` in descending order and created 3 groups so each train set will see a full spectrum of target values. Only 3 folds were used to avoid overly similar train/val splits.\n\n# ☘️ Augmentations\n### First of all, what worked\n* Tested with a combination of image resolution `384, 512, 768, 1024` image size with patch size of `2x1, 4x2`. Found image size of `768x768` with two patches `left and right` worked the best, so that's how the rest of the experiments were conducted.\n* For augmentation, I used some global transforms, which were applied on full image. And local transforms, which were applied patchwise. I wrote a custom compose class for this. I also experimented with switching to a weaker pipeline in the middle of training. At worst, it did nothing, and at best, gave a little bump in metrics, so I used it for the rest of the runs by default, switching the pipeline in the last 5-10 epochs\n\n ```\n NORM_SETTING = {\"type\": \"Normalize\", \"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]}\n STRONG_AUG_TRANSFORMS = {\n \"type\": \"albu\",\n \"compose_cls\": \"tiled_compose\",\n \"global_transforms\": [\n {\n \"type\": \"OneOf\", \"p\": 0.5,\n \"transforms\": [\n {\"type\": \"AutoContrast\", \"method\": \"cdf\", \"p\": 0.5},\n {\"type\": \"RandomBrightnessContrast\", \"brightness_limit\": 0.3, \"contrast_limit\": 0.2, \"p\": 0.5},\n {\"type\": \"RandomToneCurve\", \"scale\": 0.3, \"p\": 0.3},\n {\"type\": \"HueSaturationValue\", \"hue_shift_limit\": 20, \"sat_shift_limit\": 30, \"val_shift_limit\": 20, \"p\": 0.5},\n {\"type\": \"PlanckianJitter\", \"mode\": \"blackbody\", \"temperature_limit\": [4000, 8000], \"sampling_method\": \"gaussian\", \"p\": 0.5},\n {\"type\": \"CLAHE\", \"p\": 0.5},~~~~\n ],\n },\n {\"type\": \"ShiftScaleRotate\", \"shift_limit\": [-0.1, 0.1], \"scale_limit\": [-0.1, 0.1], \"rotate_limit\": [10, 10], \"border_mode\": 4, \"p\": 0.5},\n {\"type\": \"GaussianBlur\", \"blur_limit\": 3, \"sigma_limit\": 1, \"p\": 0.5},\n {\n \"type\": \"OneOf\", \"p\": 0.5,\n \"transforms\": [\n {\"type\": \"GridDistortion\", \"distort_limit\": 0.5, \"p\": 1.0},\n {\"type\": \"OpticalDistortion\", \"p\": 1.0},\n ],\n },\n ],\n \"local_transforms\": [\n {\"type\": \"D4\", \"p\": 1.0},\n {\n \"type\": \"OneOf\", \"p\": 0.5,\n \"transforms\": [\n {\"type\": \"Sharpen\", \"alpha\": [0.1, 0.5], \"lightness\": [1.0, 1.0], \"method\": \"kernel\", \"p\": 0.8},\n {\"type\": \"Emboss\", \"alpha\": [0.4, 0.5], \"strength\": [0.3, 0.7], \"p\": 0.8},\n ],\n },\n ],\n \"cvt_transforms\": [NORM_SETTING, {\"type\": \"ToTensorV2\"}],\n \"n_patch_h\": 1, \"n_patch_w\": 2,\n \"target_size\": [IMAGE_SIZE, IMAGE_SIZE * 2]\n }\n ```\n\n\n# ☘️ Architectural Overview\n* Used DinoV3 ViT large as my backbone, as almost everyone else in this competition did. \n* The image patches were passed to a shared backbone, and the features of shape [N, dim] from both patches were concatenated.\n* First, the height, NDVI, and species were predicted from the backbone features using a simple 2-layer MLP. The same backbone features were passed into an MLP projection network, which produced an embedding. \n* Then, the NDVI, height, and sigmoid-activated species logits were concatenated and then passed to a FiLM layer, which then modulated the embedding produced by the above MLP projection network. Earlier, I was simply concatenating them, but due to the size of the embedding, it dominated. I was looking for better options, so I thought to predict the scale shift parameter and transfrom embedding that way. I ended up designing the FiLM layer. (Fun fact, I initially thought, I discovered a novel idea, as this took me from 0.68 to 0.72 in public LB. Later discovered the FiLM paper 😅)\n* Finally, I predicted 3 biomass targets at once from the modulated embeddings\n\n# ☘️ Loss & Optimization\n* For loss, I used a weighted SmoothL1 loss for biomass targets and a SmoothL1 loss for height and NDVI targets. \n* For the species head, I used a simple BCE loss with the following weights for each category\n ```\n SPECIES = [\"clover\", \"ryegrass\", \"phalaris\", \"grass\", \"fescue\", \"lucerne\", \"weed\", \"whiteclover\"]\n SPECIES_WEIGHTS = [1.68, 1.86, 2.76, 3.59, 4.09, 5.01, 5.89, 7.08] # roots sqrt(N / Nc)\n ```\n* The above 3 losses were combined with weights `{\"biomass\": 3.0, \"aux\": 1.0, \"species\": 0.2}`\n* For training, I used AdamW with lr of 1e-5 and weight decay of 0.3 (I know it's high, but it regularizes the model ), did warmup for 5 epochs, then reduced the lr for the rest of the training using cosine annealing\n* The training was unstable, so I used EMA and gradient clipping to stabilize the training\n* All three heads were trained together on 3CV and 5CV. These two were my final submissions.\n\n# ☘️ What did not Work\n* As the dataset provided in competetion was very small, I found a large dataset (was ~270GB with ~40k images) which I have processed and uploaded to Kaggle. It was not much of a help, though. [here](https://www.kaggle.com/datasets/pushpakbhoge/grass-clover-datasets)\n* **DinoV2 Pretraining**: Fine-tuned DinoV2 (ViT-small) on the largely unlabelled data for 100 epochs. It failed to beat the original weights despite promising attention maps. Halted due to compute limits, though this remains a potential avenue for exploration.\n* **Segmentation**: Trained a 16-class model on synthetic data to create guided attention maps for the downstream CNN [code](https://github.com/PushpakBhoge/csiro_biomass_regression/blob/main/components/bricks.py#L73).\n* **CNN Architecture**: Initially used ConvNeXt-atto (ImageNet1k pretrained), theorizing that a smaller model suited the dataset size. Integrated the segmentation maps for attention and tested various pooling methods.\n* **CNN Outcome**: Despite extensive tuning, the CNN could not outperform ViT. Realizing late that superior features were critical, I tried ConvNeXt-Base with DinoV3 weights but only achieved 0.65 Public LB, suggesting ConvNeXt features were less effective than ViT-base here. These weights for ConvNeXt were distilled from ViT, so maybe that has something to do with it\n* **Meta-Averaging**: Tried meta-averaging auxiliary predictions before biomass inference, neutral effect.\n* **Unfreezing-backbone**: Tried scaled lr across different backbone stages, and training small model from scratch did not work\n* **Biomass Summation**: Attempted to approximate density maps by summing patch biomass. Summing raw channels caused exploding gradients, while 3x3 average pooling stabilized training but yielded no gains.\n* **Spatial Pooling**: Implemented a convolutional spatial pooling layer (reducing 24x24 features to 1x1) followed by MLPs. Summing these outputs improved Public LB by 0.01, peaking at 0.59 [code](https://github.com/PushpakBhoge/csiro_biomass_regression/blob/main/components/bricks.py#L217).\n* **FPN & Multi-Stage Heads**: Tested fusing backbone features at varying resolutions (FPN) and attaching heads directly to Stage 2 and 3 outputs. Neither approach improved the score.\n\n\nI have written quite extensive code for this. You can find the code on my [GitHub repository](https://github.com/PushpakBhoge/csiro_biomass_regression)\nThis is my [submission notebook](https://www.kaggle.com/code/pushpakbhoge/csiro-image2biomas-submission-notebook) I am planning to release an exhaustively detailed article about this, which will take a week or two.\n\nFinally, Why were CNNs consistently inferior to ViTs in this competition? I’d love to hear insights from others." } }