{ "id": 654028, "title": "Creating a Dataset of Sub Image Labels", "url": "/competitions/csiro-biomass/discussion/654028", "author_name": "", "post_date": "2025-12-07T08:05:51.255759Z", "votes": 12, "comment_count": 7, "views": 0, "content": "

Notebook: https://www.kaggle.com/code/laodriverayu/dataset-sub-image-labels?scriptVersionId=284424223

\n

Idea

\n

Here is my attempt of augmenting the dataset by labeling crops from original images. Since weight is naturally additive, I can expect that the crop labels sums up to the original image labels. Meanwhile, similar looking crops should receive similar labels. I came up with a constrained optimization approach to achieve the goal. There is no supervision or learning. Similarity is calculated from DINOv2 image embeddings.\nThe end result is effectively solving a quadratic problem:\n$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$

\n

Math below

\n
\n

1. Goal

\n

Label image crops so that the sum over each image is preserved, and similar crops have similar labels

\n

2. Formal Definition:

\n\n

Goals:

\n
    \n

  1. Per-image label conservation:\n$$\n\\sum_{j=1}^{n_p} x_{i,j} = y_i,\\quad \\forall i = 1,\\dots,n_s.\n$$

    2. \n

  2. Smoothness over the patch graph:\n$$\n\\frac{1}{2} \\sum_{(i,j)} \\sum_{(i',j')} \nW_{(i,j),(i',j')} \\,\\big|x_{i,j} - x_{i',j'}\\big|^2\n\\quad \\text{is small}.\n$$

    3. \n
\n

So the constrained optimization problem is:

\n

$$\n\\min_{{x_{i,j} \\in \\mathbb{R}^k}}\n\\;\n\\frac{1}{2} \n\\sum_{(i,j)} \\sum_{(i',j')} \nW_{(i,j),(i',j')} \\,\\big|x_{i,j} - x_{i',j'}\\big|^2 \n\\quad\n\\text{s.t.}\\quad\n\\sum_{j=1}^{n_p} x_{i,j} = y_i, \n\\;\\forall i = 1,\\dots,n_s.\n$$

\n
\n

2. In Matrix Form

\n

Let:

\n\n

Graph Laplacian

\n

Let (W \\in \\mathbb{R}^{N \\times N}) be the patch similarity matrix and define:

\n

$$\nD_{pp} = \\sum_q W_{pq}, \\quad\nL = D - W.\n$$

\n

Then the smoothness term can be written as:

\n

$$\n\\frac{1}{2} \\sum_{p,q} W_{pq} (x_p - x_q)^2\n= x^\\top L x.\n$$

\n

Optionally define a strictly positive-definite matrix:

\n

$$\nQ = L + \\varepsilon I, \\quad \\varepsilon > 0.\n$$

\n

Constraint matrix

\n

Define (A \\in \\mathbb{R}^{n_s \\times N}) such that row (i) selects the patches of image (i):

\n

$$\nA_{i,p} =\n\\begin{cases}\n1 & \\text{if patch } p \\text{ belongs to image } i, \\\n0 & \\text{otherwise}.\n\\end{cases}\n$$

\n

Then the constraints\n$$\n\\sum_{j=1}^{n_p} x_{i,j} = y_i\n$$\nbecome\n$$\nA x = y.\n$$

\n

Final matrix form

\n

$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$

\n

This can be solved in close form

\n

Note

\n

I did not enforce positive label constraint, so some labels can go negative. I did not do post-processing in this notebook

", "messages": [ { "id": "3365541", "postDate": "12/07/2025 08:05:51", "content": "

Notebook: https://www.kaggle.com/code/laodriverayu/dataset-sub-image-labels?scriptVersionId=284424223

\n

Idea

\n

Here is my attempt of augmenting the dataset by labeling crops from original images. Since weight is naturally additive, I can expect that the crop labels sums up to the original image labels. Meanwhile, similar looking crops should receive similar labels. I came up with a constrained optimization approach to achieve the goal. There is no supervision or learning. Similarity is calculated from DINOv2 image embeddings.\nThe end result is effectively solving a quadratic problem:\n$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$

\n

Math below

\n
\n

1. Goal

\n

Label image crops so that the sum over each image is preserved, and similar crops have similar labels

\n

2. Formal Definition:

\n\n

Goals:

\n
    \n

  1. Per-image label conservation:\n$$\n\\sum_{j=1}^{n_p} x_{i,j} = y_i,\\quad \\forall i = 1,\\dots,n_s.\n$$

    2. \n

  2. Smoothness over the patch graph:\n$$\n\\frac{1}{2} \\sum_{(i,j)} \\sum_{(i',j')} \nW_{(i,j),(i',j')} \\,\\big|x_{i,j} - x_{i',j'}\\big|^2\n\\quad \\text{is small}.\n$$

    3. \n
\n

So the constrained optimization problem is:

\n

$$\n\\min_{{x_{i,j} \\in \\mathbb{R}^k}}\n\\;\n\\frac{1}{2} \n\\sum_{(i,j)} \\sum_{(i',j')} \nW_{(i,j),(i',j')} \\,\\big|x_{i,j} - x_{i',j'}\\big|^2 \n\\quad\n\\text{s.t.}\\quad\n\\sum_{j=1}^{n_p} x_{i,j} = y_i, \n\\;\\forall i = 1,\\dots,n_s.\n$$

\n
\n

2. In Matrix Form

\n

Let:

\n\n

Graph Laplacian

\n

Let (W \\in \\mathbb{R}^{N \\times N}) be the patch similarity matrix and define:

\n

$$\nD_{pp} = \\sum_q W_{pq}, \\quad\nL = D - W.\n$$

\n

Then the smoothness term can be written as:

\n

$$\n\\frac{1}{2} \\sum_{p,q} W_{pq} (x_p - x_q)^2\n= x^\\top L x.\n$$

\n

Optionally define a strictly positive-definite matrix:

\n

$$\nQ = L + \\varepsilon I, \\quad \\varepsilon > 0.\n$$

\n

Constraint matrix

\n

Define (A \\in \\mathbb{R}^{n_s \\times N}) such that row (i) selects the patches of image (i):

\n

$$\nA_{i,p} =\n\\begin{cases}\n1 & \\text{if patch } p \\text{ belongs to image } i, \\\n0 & \\text{otherwise}.\n\\end{cases}\n$$

\n

Then the constraints\n$$\n\\sum_{j=1}^{n_p} x_{i,j} = y_i\n$$\nbecome\n$$\nA x = y.\n$$

\n

Final matrix form

\n

$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$

\n

This can be solved in close form

\n

Note

\n

I did not enforce positive label constraint, so some labels can go negative. I did not do post-processing in this notebook

", "rawMarkdown": "Notebook: https://www.kaggle.com/code/laodriverayu/dataset-sub-image-labels?scriptVersionId=284424223\n\n## Idea\nHere is my attempt of augmenting the dataset by labeling crops from original images. Since weight is naturally additive, I can expect that the crop labels sums up to the original image labels. Meanwhile, similar looking crops should receive similar labels. I came up with a constrained optimization approach to achieve the goal. There is no supervision or learning. Similarity is calculated from DINOv2 image embeddings.\nThe end result is effectively solving a quadratic problem:\n$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$\n\nMath below\n\n---\n\n\n\n## 1. Goal\nLabel image crops so that the sum over each image is preserved, and similar crops have similar labels\n\n\n## 2. Formal Definition:\n\n- Have `n_s` images.\n- Each image `i` in `{1, ..., n_s}` is split into `n_p` patches.\n- Each patch `(i, j)` (patch `j` of image `i`) has an unknown label $x_{i,j}$.\n- Build a patch-similarity graph over all patches, with weights $W_{(i,j),(i',j')} ≥ 0$.\n\n\nGoals:\n\n1. **Per-image label conservation**:\n $$\n \\sum_{j=1}^{n_p} x_{i,j} = y_i,\\quad \\forall i = 1,\\dots,n_s.\n $$\n\n2. **Smoothness over the patch graph**:\n $$\n \\frac{1}{2} \\sum_{(i,j)} \\sum_{(i',j')} \n W_{(i,j),(i',j')} \\,\\big\\|x_{i,j} - x_{i',j'}\\big\\|^2\n \\quad \\text{is small}.\n $$\n\nSo the constrained optimization problem is:\n\n$$\n\\min_{\\{x_{i,j} \\in \\mathbb{R}^k\\}}\n\\;\n\\frac{1}{2} \n\\sum_{(i,j)} \\sum_{(i',j')} \nW_{(i,j),(i',j')} \\,\\big\\|x_{i,j} - x_{i',j'}\\big\\|^2 \n\\quad\n\\text{s.t.}\\quad\n\\sum_{j=1}^{n_p} x_{i,j} = y_i, \n\\;\\forall i = 1,\\dots,n_s.\n$$\n\n---\n\n## 2. In Matrix Form\n\nLet:\n\n- Total number of patches:\n $$\n N = n_s \\cdot n_p.\n $$\n- Index patches by a single index \\(p \\in \\{1,\\dots,N\\}\\).\n- For **scalar labels**, collect all patch labels into:\n $$\n x \\in \\mathbb{R}^N,\\quad x_p = \\text{label of patch } p.\n $$\n- Collect image labels into:\n $$\n y \\in \\mathbb{R}^{n_s}.\n $$\n\n### Graph Laplacian\n\nLet \\(W \\in \\mathbb{R}^{N \\times N}\\) be the patch similarity matrix and define:\n\n$$\nD_{pp} = \\sum_q W_{pq}, \\quad\nL = D - W.\n$$\n\nThen the smoothness term can be written as:\n\n$$\n\\frac{1}{2} \\sum_{p,q} W_{pq} (x_p - x_q)^2\n= x^\\top L x.\n$$\n\nOptionally define a strictly positive-definite matrix:\n\n$$\nQ = L + \\varepsilon I, \\quad \\varepsilon > 0.\n$$\n\n### Constraint matrix\n\nDefine \\(A \\in \\mathbb{R}^{n_s \\times N}\\) such that row \\(i\\) selects the patches of image \\(i\\):\n\n$$\nA_{i,p} =\n\\begin{cases}\n1 & \\text{if patch } p \\text{ belongs to image } i, \\\\\n0 & \\text{otherwise}.\n\\end{cases}\n$$\n\nThen the constraints\n$$\n\\sum_{j=1}^{n_p} x_{i,j} = y_i\n$$\nbecome\n$$\nA x = y.\n$$\n\n### Final matrix form\n\n$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$\n\nThis can be solved in close form\n\n### Note\nI did not enforce positive label constraint, so some labels can go negative. I did not do post-processing in this notebook", "votes": null }, { "id": "3366622", "postDate": "12/07/2025 21:03:11", "content": "

Had some minor errors attempting to fork on my local machine, but easy to fix.

\n

Nice work - thanks for the share.

", "rawMarkdown": "Had some minor errors attempting to fork on my local machine, but easy to fix. \n\nNice work - thanks for the share.", "votes": null }, { "id": "3367632", "postDate": "12/08/2025 15:45:20", "content": "

Ran a model training with the crop images on my local machine. Variation between folds was one of my better, cv value was lower at 0.65 than typical but this is in line with what this model has generated over the last couple of days for LB score so I am good with that. Will not likely do a LB submission as I don't see it being the break thru over 0.66 that I need. With more images it is of course a bit slower to train.

\n

It did not help \"fix\" my current model issues. 1) Large prediction errors for high biomass images 2) Poor performance on Dead stuff 3) mis-identification of several plants as Clover when they are just Green.

\n

I do like it and will return to its use in a few weeks when I find the magic for my 3 current issues. I believe it does help produce a less over fitted model.

\n

Thanks again for the share.

", "rawMarkdown": "Ran a model training with the crop images on my local machine. Variation between folds was one of my better, cv value was lower at 0.65 than typical but this is in line with what this model has generated over the last couple of days for LB score so I am good with that. Will not likely do a LB submission as I don't see it being the break thru over 0.66 that I need. With more images it is of course a bit slower to train.\n\nIt did not help \"fix\" my current model issues. 1) Large prediction errors for high biomass images 2) Poor performance on Dead stuff 3) mis-identification of several plants as Clover when they are just Green.\n\nI do like it and will return to its use in a few weeks when I find the magic for my 3 current issues. I believe it does help produce a less over fitted model.\n\nThanks again for the share.", "votes": null }, { "id": "3368016", "postDate": "12/08/2025 22:28:29", "content": "

Thanks for testing it out! The results confirmed my expectations. Since this augmentation does not add new information, the score should be similar to other Dinov2 based approaches. I see this augmented dataset most useful as a test set in detecting overfit. Getting similar scores on full images and cropped images is a sign of robustness

", "rawMarkdown": "Thanks for testing it out! The results confirmed my expectations. Since this augmentation does not add new information, the score should be similar to other Dinov2 based approaches. I see this augmented dataset most useful as a test set in detecting overfit. Getting similar scores on full images and cropped images is a sign of robustness", "votes": null }, { "id": "3368019", "postDate": "12/08/2025 22:35:56", "content": "

Some insights for your issue 1. If you plot biomass against 'ratio of green pixels', you would see some exponential relation. This makes sense because you would need to scatter a ton of grass on the ground to make the photo covered in green. ReLU extrapolates linearly. Maybe calibrate the mass for training, then map to exponential as post-processing

", "rawMarkdown": "Some insights for your issue 1. If you plot biomass against 'ratio of green pixels', you would see some exponential relation. This makes sense because you would need to scatter a ton of grass on the ground to make the photo covered in green. ReLU extrapolates linearly. Maybe calibrate the mass for training, then map to exponential as post-processing", "votes": null }, { "id": "3368695", "postDate": "12/09/2025 12:34:33", "content": "

@laodriverayu What you ar suggesting is similar to training the model on log-transformed values. However, even with that approach, my model still struggles with high-biomass images. I think there are two main reasons for this:

\n", "rawMarkdown": "laodriverayu What you ar suggesting is similar to training the model on log-transformed values. However, even with that approach, my model still struggles with high-biomass images. I think there are two main reasons for this:\n- High biomass often comes from rare species or cases with very few training examples, so the model does not learn them well.\n- The model tends to underestimate grass height (which is well correlated with amount of biomass) because it only sees a 2D image and has no notion of depth. As a result, very tall grass that bends or lies along the surface can look similar to simply having a dense amount of grass", "votes": null }, { "id": "3378058", "postDate": "12/17/2025 12:56:49", "content": "

I don't know how you came up with this but this is impressive :D

", "rawMarkdown": "I don't know how you came up with this but this is impressive :D", "votes": null }, { "id": "3378246", "postDate": "12/17/2025 19:49:16", "content": "

haha Thanks. Do you follow my set up?

", "rawMarkdown": "haha Thanks. Do you follow my set up?", "votes": null } ], "comments": [ { "id": 3366622, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/07/2025 21:03:11", "content": "

Had some minor errors attempting to fork on my local machine, but easy to fix.

\n

Nice work - thanks for the share.

", "votes": null, "replies": [ { "id": 3367632, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/08/2025 15:45:20", "content": "

Ran a model training with the crop images on my local machine. Variation between folds was one of my better, cv value was lower at 0.65 than typical but this is in line with what this model has generated over the last couple of days for LB score so I am good with that. Will not likely do a LB submission as I don't see it being the break thru over 0.66 that I need. With more images it is of course a bit slower to train.

\n

It did not help \"fix\" my current model issues. 1) Large prediction errors for high biomass images 2) Poor performance on Dead stuff 3) mis-identification of several plants as Clover when they are just Green.

\n

I do like it and will return to its use in a few weeks when I find the magic for my 3 current issues. I believe it does help produce a less over fitted model.

\n

Thanks again for the share.

", "votes": null, "replies": [ { "id": 3368016, "author_name": "laodriverayu", "author_url": "", "post_date": "12/08/2025 22:28:29", "content": "

Thanks for testing it out! The results confirmed my expectations. Since this augmentation does not add new information, the score should be similar to other Dinov2 based approaches. I see this augmented dataset most useful as a test set in detecting overfit. Getting similar scores on full images and cropped images is a sign of robustness

", "votes": null, "replies": [] }, { "id": 3368019, "author_name": "laodriverayu", "author_url": "", "post_date": "12/08/2025 22:35:56", "content": "

Some insights for your issue 1. If you plot biomass against 'ratio of green pixels', you would see some exponential relation. This makes sense because you would need to scatter a ton of grass on the ground to make the photo covered in green. ReLU extrapolates linearly. Maybe calibrate the mass for training, then map to exponential as post-processing

", "votes": null, "replies": [ { "id": 3368695, "author_name": "nizarcharrada", "author_url": "", "post_date": "12/09/2025 12:34:33", "content": "

@laodriverayu What you ar suggesting is similar to training the model on log-transformed values. However, even with that approach, my model still struggles with high-biomass images. I think there are two main reasons for this:

\n", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3378058, "author_name": "dendrebeden", "author_url": "", "post_date": "12/17/2025 12:56:49", "content": "

I don't know how you came up with this but this is impressive :D

", "votes": null, "replies": [ { "id": 3378246, "author_name": "laodriverayu", "author_url": "", "post_date": "12/17/2025 19:49:16", "content": "

haha Thanks. Do you follow my set up?

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3365541": "Notebook: https://www.kaggle.com/code/laodriverayu/dataset-sub-image-labels?scriptVersionId=284424223\n\n## Idea\nHere is my attempt of augmenting the dataset by labeling crops from original images. Since weight is naturally additive, I can expect that the crop labels sums up to the original image labels. Meanwhile, similar looking crops should receive similar labels. I came up with a constrained optimization approach to achieve the goal. There is no supervision or learning. Similarity is calculated from DINOv2 image embeddings.\nThe end result is effectively solving a quadratic problem:\n$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$\n\nMath below\n\n---\n\n\n\n## 1. Goal\nLabel image crops so that the sum over each image is preserved, and similar crops have similar labels\n\n\n## 2. Formal Definition:\n\n- Have `n_s` images.\n- Each image `i` in `{1, ..., n_s}` is split into `n_p` patches.\n- Each patch `(i, j)` (patch `j` of image `i`) has an unknown label $x_{i,j}$.\n- Build a patch-similarity graph over all patches, with weights $W_{(i,j),(i',j')} ≥ 0$.\n\n\nGoals:\n\n1. **Per-image label conservation**:\n $$\n \\sum_{j=1}^{n_p} x_{i,j} = y_i,\\quad \\forall i = 1,\\dots,n_s.\n $$\n\n2. **Smoothness over the patch graph**:\n $$\n \\frac{1}{2} \\sum_{(i,j)} \\sum_{(i',j')} \n W_{(i,j),(i',j')} \\,\\big\\|x_{i,j} - x_{i',j'}\\big\\|^2\n \\quad \\text{is small}.\n $$\n\nSo the constrained optimization problem is:\n\n$$\n\\min_{\\{x_{i,j} \\in \\mathbb{R}^k\\}}\n\\;\n\\frac{1}{2} \n\\sum_{(i,j)} \\sum_{(i',j')} \nW_{(i,j),(i',j')} \\,\\big\\|x_{i,j} - x_{i',j'}\\big\\|^2 \n\\quad\n\\text{s.t.}\\quad\n\\sum_{j=1}^{n_p} x_{i,j} = y_i, \n\\;\\forall i = 1,\\dots,n_s.\n$$\n\n---\n\n## 2. In Matrix Form\n\nLet:\n\n- Total number of patches:\n $$\n N = n_s \\cdot n_p.\n $$\n- Index patches by a single index \\(p \\in \\{1,\\dots,N\\}\\).\n- For **scalar labels**, collect all patch labels into:\n $$\n x \\in \\mathbb{R}^N,\\quad x_p = \\text{label of patch } p.\n $$\n- Collect image labels into:\n $$\n y \\in \\mathbb{R}^{n_s}.\n $$\n\n### Graph Laplacian\n\nLet \\(W \\in \\mathbb{R}^{N \\times N}\\) be the patch similarity matrix and define:\n\n$$\nD_{pp} = \\sum_q W_{pq}, \\quad\nL = D - W.\n$$\n\nThen the smoothness term can be written as:\n\n$$\n\\frac{1}{2} \\sum_{p,q} W_{pq} (x_p - x_q)^2\n= x^\\top L x.\n$$\n\nOptionally define a strictly positive-definite matrix:\n\n$$\nQ = L + \\varepsilon I, \\quad \\varepsilon > 0.\n$$\n\n### Constraint matrix\n\nDefine \\(A \\in \\mathbb{R}^{n_s \\times N}\\) such that row \\(i\\) selects the patches of image \\(i\\):\n\n$$\nA_{i,p} =\n\\begin{cases}\n1 & \\text{if patch } p \\text{ belongs to image } i, \\\\\n0 & \\text{otherwise}.\n\\end{cases}\n$$\n\nThen the constraints\n$$\n\\sum_{j=1}^{n_p} x_{i,j} = y_i\n$$\nbecome\n$$\nA x = y.\n$$\n\n### Final matrix form\n\n$$\n\\min_{x \\in \\mathbb{R}^N} \\quad \\frac{1}{2} x^\\top Q x\n\\quad \\text{s.t.} \\quad A x = y.\n$$\n\nThis can be solved in close form\n\n### Note\nI did not enforce positive label constraint, so some labels can go negative. I did not do post-processing in this notebook", "3366622": "Had some minor errors attempting to fork on my local machine, but easy to fix. \n\nNice work - thanks for the share.", "3367632": "Ran a model training with the crop images on my local machine. Variation between folds was one of my better, cv value was lower at 0.65 than typical but this is in line with what this model has generated over the last couple of days for LB score so I am good with that. Will not likely do a LB submission as I don't see it being the break thru over 0.66 that I need. With more images it is of course a bit slower to train.\n\nIt did not help \"fix\" my current model issues. 1) Large prediction errors for high biomass images 2) Poor performance on Dead stuff 3) mis-identification of several plants as Clover when they are just Green.\n\nI do like it and will return to its use in a few weeks when I find the magic for my 3 current issues. I believe it does help produce a less over fitted model.\n\nThanks again for the share.", "3368016": "Thanks for testing it out! The results confirmed my expectations. Since this augmentation does not add new information, the score should be similar to other Dinov2 based approaches. I see this augmented dataset most useful as a test set in detecting overfit. Getting similar scores on full images and cropped images is a sign of robustness", "3368019": "Some insights for your issue 1. If you plot biomass against 'ratio of green pixels', you would see some exponential relation. This makes sense because you would need to scatter a ton of grass on the ground to make the photo covered in green. ReLU extrapolates linearly. Maybe calibrate the mass for training, then map to exponential as post-processing", "3368695": "laodriverayu What you ar suggesting is similar to training the model on log-transformed values. However, even with that approach, my model still struggles with high-biomass images. I think there are two main reasons for this:\n- High biomass often comes from rare species or cases with very few training examples, so the model does not learn them well.\n- The model tends to underestimate grass height (which is well correlated with amount of biomass) because it only sees a 2D image and has no notion of depth. As a result, very tall grass that bends or lies along the surface can look similar to simply having a dense amount of grass", "3378058": "I don't know how you came up with this but this is impressive :D", "3378246": "haha Thanks. Do you follow my set up?" }, "source": "meta" }