{ "id": 541978, "title": "Ranking based on noise level ", "url": "/competitions/data-centric-land-cover-classification-challenge/discussion/541978", "author_name": "", "post_date": "2024-10-22T11:07:28.309105200Z", "votes": 1, "comment_count": 9, "views": 0, "content": "

Hello,
\nI have two questions regarding the ranking of the noise level:

\n

1.) How is noise evaluated? Is it per-pixel agreement between the true and the noisy mask?
\n2.) How are masks ranked that have the same level of noise? (e.g. no noise at all ?)

", "messages": [ { "id": "3025076", "postDate": "10/22/2024 11:07:28", "content": "

Hello,
\nI have two questions regarding the ranking of the noise level:

\n

1.) How is noise evaluated? Is it per-pixel agreement between the true and the noisy mask?
\n2.) How are masks ranked that have the same level of noise? (e.g. no noise at all ?)

", "rawMarkdown": "Hello, \nI have two questions regarding the ranking of the noise level:\n\n1.) How is noise evaluated? Is it per-pixel agreement between the true and the noisy mask? \n2.) How are masks ranked that have the same level of noise? (e.g. no noise at all ?)", "votes": null }, { "id": "3025166", "postDate": "10/22/2024 13:39:02", "content": "

Hi David,

\n

here are the answers to your questions:

\n
    \n
  1. Yes! The process is: we calculate the pixelwise IoU over the original and noisy mask. Then, we generate the rank based on the pixelwise IoU, from least to most noisy.
    2. \n
  2. It should be very rare to have two samples with the same pixelwise IoU score - they will differ at some point, e.g. 0.999759609 != 0.998941279 != 0.997768557. IF there are two or more samples with the same IoU score, then they will be sorted based on the imageid
    3. \n
", "rawMarkdown": "Hi David,\n\nhere are the answers to your questions:\n\n1. Yes! The process is: we calculate the pixelwise IoU over the original and noisy mask. Then, we generate the rank based on the pixelwise IoU, from least to most noisy.\n2. It should be very rare to have two samples with the same pixelwise IoU score - they will differ at some point, e.g. 0.999759609 != 0.998941279 != 0.997768557. IF there are two or more samples with the same IoU score, then they will be sorted based on the `imageid`", "votes": null }, { "id": "3025201", "postDate": "10/22/2024 14:35:43", "content": "

Thank you very much for the fast response :)

", "rawMarkdown": "Thank you very much for the fast response :)", "votes": null }, { "id": "3025202", "postDate": "10/22/2024 14:36:14", "content": "

For 2) What if the noisy-mask and original mask has no building pixel? The IOU would be zero. right?

", "rawMarkdown": "For 2) What if the noisy-mask and original mask has no building pixel? The IOU would be zero. right?", "votes": null }, { "id": "3027083", "postDate": "10/24/2024 14:08:07", "content": "

We removed original samples with no building pixels.

\n

In any case, as I said, if there are more than one sample with the same score, they will be sorted by imageid.

", "rawMarkdown": "We removed original samples with no building pixels.\n\nIn any case, as I said, if there are more than one sample with the same score, they will be sorted by `imageid`.", "votes": null }, { "id": "3028756", "postDate": "10/26/2024 13:34:28", "content": "

IoU is not a bad metric for the task but if talking about real life applications I think possibly making adjustments could be very beneficial for performance.
\nMaking a distinction for example between false positive pixels which encompass a target object in its contour vs. not, would make a more accurate \"noise score\". Give worse noise score to false positive pixels which don't have an object inside the contour, since they could contribute more to bad learning vs. a few extra label pixels oversegmenting the target object.

", "rawMarkdown": "IoU is not a bad metric for the task but if talking about real life applications I think possibly making adjustments could be very beneficial for performance.\nMaking a distinction for example between false positive pixels which encompass a target object in its contour vs. not, would make a more accurate \"noise score\". Give worse noise score to false positive pixels which don't have an object inside the contour, since they could contribute more to bad learning vs. a few extra label pixels oversegmenting the target object.", "votes": null }, { "id": "3028761", "postDate": "10/26/2024 13:38:13", "content": "

Also taking raw pixel number instead of the relative \"IoU\" maybe is another way to optimize it.

", "rawMarkdown": "Also taking raw pixel number instead of the relative \"IoU\" maybe is another way to optimize it.", "votes": null }, { "id": "3029910", "postDate": "10/27/2024 23:10:13", "content": "

To follow up on Rohit's question: let's assume two images, both with buildings but for which the noisy masks do not capture those buildings. Won't the IOUs be both zero?

", "rawMarkdown": "To follow up on Rohit's question: let's assume two images, both with buildings but for which the noisy masks do not capture those buildings. Won't the IOUs be both zero?", "votes": null }, { "id": "3035496", "postDate": "11/03/2024 13:52:12", "content": "

Hi @keiller ,

\n

I will re-post a comment I've made to illustrate my concern regarding the IOU-based ranking.
\nFirst, just to make sure we are on the same line, I compute it as: sum(my_mask & noisy_mask) / sum(my_mask | noisy_mask). Is this what you meant or are you using some class-balancing between foreground and background?

\n

I'm asking because I noticed that some noisy masks contain only zeros (164 in total). The numerator in this case will always be zero, no matter how good or bad my mask is. I'm a bit worried also because this number can increase further more, reaching up to ~700, if we allow a few (i.e. less than 1%) positive pixels in the noisy masks. Given that some of these (already few) pixels are noise, it will again lead to a zero intersection, even if my masks are correct.

", "rawMarkdown": "Hi @keiller ,\n\nI will re-post a comment I've made to illustrate my concern regarding the IOU-based ranking. \nFirst, just to make sure we are on the same line, I compute it as: `sum(my_mask & noisy_mask) / sum(my_mask | noisy_mask)`. Is this what you meant or are you using some class-balancing between foreground and background?\n\nI'm asking because I noticed that some noisy masks contain only zeros (164 in total). The numerator in this case will always be zero, no matter how good or bad my mask is. I'm a bit worried also because this number can increase further more, reaching up to ~700, if we allow a few (i.e. less than 1%) positive pixels in the noisy masks. Given that some of these (already few) pixels are noise, it will again lead to a zero intersection, even if my masks are correct.", "votes": null }, { "id": "3040295", "postDate": "11/09/2024 00:58:10", "content": "

@keiller, could you provide more information on the metric used to rank the noisy labels? If the formula described above by @codrutandrei is used, an educated random approach should, with high probability, already achieve better results than the current submissions in the ranking. Therefore, I would like to make sure that we consider the correct metric and that there is no weighting or anything that should be considered for the iou/ranking (or is considered in the evaluation algorithm).

", "rawMarkdown": "keiller, could you provide more information on the metric used to rank the noisy labels? If the formula described above by @codrutandrei is used, an educated random approach should, with high probability, already achieve better results than the current submissions in the ranking. Therefore, I would like to make sure that we consider the correct metric and that there is no weighting or anything that should be considered for the iou/ranking (or is considered in the evaluation algorithm).", "votes": null } ], "comments": [ { "id": 3025166, "author_name": "keiller", "author_url": "", "post_date": "10/22/2024 13:39:02", "content": "

Hi David,

\n

here are the answers to your questions:

\n
    \n
  1. Yes! The process is: we calculate the pixelwise IoU over the original and noisy mask. Then, we generate the rank based on the pixelwise IoU, from least to most noisy.
    2. \n
  2. It should be very rare to have two samples with the same pixelwise IoU score - they will differ at some point, e.g. 0.999759609 != 0.998941279 != 0.997768557. IF there are two or more samples with the same IoU score, then they will be sorted based on the imageid
    3. \n
", "votes": null, "replies": [ { "id": 3025201, "author_name": "davidhansen", "author_url": "", "post_date": "10/22/2024 14:35:43", "content": "

Thank you very much for the fast response :)

", "votes": null, "replies": [] }, { "id": 3025202, "author_name": "rhtkhati", "author_url": "", "post_date": "10/22/2024 14:36:14", "content": "

For 2) What if the noisy-mask and original mask has no building pixel? The IOU would be zero. right?

", "votes": null, "replies": [ { "id": 3027083, "author_name": "keiller", "author_url": "", "post_date": "10/24/2024 14:08:07", "content": "

We removed original samples with no building pixels.

\n

In any case, as I said, if there are more than one sample with the same score, they will be sorted by imageid.

", "votes": null, "replies": [ { "id": 3029910, "author_name": "codrutandrei", "author_url": "", "post_date": "10/27/2024 23:10:13", "content": "

To follow up on Rohit's question: let's assume two images, both with buildings but for which the noisy masks do not capture those buildings. Won't the IOUs be both zero?

", "votes": null, "replies": [] } ] } ] }, { "id": 3028756, "author_name": "ilyasavenko", "author_url": "", "post_date": "10/26/2024 13:34:28", "content": "

IoU is not a bad metric for the task but if talking about real life applications I think possibly making adjustments could be very beneficial for performance.
\nMaking a distinction for example between false positive pixels which encompass a target object in its contour vs. not, would make a more accurate \"noise score\". Give worse noise score to false positive pixels which don't have an object inside the contour, since they could contribute more to bad learning vs. a few extra label pixels oversegmenting the target object.

", "votes": null, "replies": [ { "id": 3028761, "author_name": "ilyasavenko", "author_url": "", "post_date": "10/26/2024 13:38:13", "content": "

Also taking raw pixel number instead of the relative \"IoU\" maybe is another way to optimize it.

", "votes": null, "replies": [] } ] } ] }, { "id": 3035496, "author_name": "codrutandrei", "author_url": "", "post_date": "11/03/2024 13:52:12", "content": "

Hi @keiller ,

\n

I will re-post a comment I've made to illustrate my concern regarding the IOU-based ranking.
\nFirst, just to make sure we are on the same line, I compute it as: sum(my_mask & noisy_mask) / sum(my_mask | noisy_mask). Is this what you meant or are you using some class-balancing between foreground and background?

\n

I'm asking because I noticed that some noisy masks contain only zeros (164 in total). The numerator in this case will always be zero, no matter how good or bad my mask is. I'm a bit worried also because this number can increase further more, reaching up to ~700, if we allow a few (i.e. less than 1%) positive pixels in the noisy masks. Given that some of these (already few) pixels are noise, it will again lead to a zero intersection, even if my masks are correct.

", "votes": null, "replies": [ { "id": 3040295, "author_name": "empty1990", "author_url": "", "post_date": "11/09/2024 00:58:10", "content": "

@keiller, could you provide more information on the metric used to rank the noisy labels? If the formula described above by @codrutandrei is used, an educated random approach should, with high probability, already achieve better results than the current submissions in the ranking. Therefore, I would like to make sure that we consider the correct metric and that there is no weighting or anything that should be considered for the iou/ranking (or is considered in the evaluation algorithm).

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3025076": "Hello, \nI have two questions regarding the ranking of the noise level:\n\n1.) How is noise evaluated? Is it per-pixel agreement between the true and the noisy mask? \n2.) How are masks ranked that have the same level of noise? (e.g. no noise at all ?)", "3025166": "Hi David,\n\nhere are the answers to your questions:\n\n1. Yes! The process is: we calculate the pixelwise IoU over the original and noisy mask. Then, we generate the rank based on the pixelwise IoU, from least to most noisy.\n2. It should be very rare to have two samples with the same pixelwise IoU score - they will differ at some point, e.g. 0.999759609 != 0.998941279 != 0.997768557. IF there are two or more samples with the same IoU score, then they will be sorted based on the `imageid`", "3025201": "Thank you very much for the fast response :)", "3025202": "For 2) What if the noisy-mask and original mask has no building pixel? The IOU would be zero. right?", "3027083": "We removed original samples with no building pixels.\n\nIn any case, as I said, if there are more than one sample with the same score, they will be sorted by `imageid`.", "3028756": "IoU is not a bad metric for the task but if talking about real life applications I think possibly making adjustments could be very beneficial for performance.\nMaking a distinction for example between false positive pixels which encompass a target object in its contour vs. not, would make a more accurate \"noise score\". Give worse noise score to false positive pixels which don't have an object inside the contour, since they could contribute more to bad learning vs. a few extra label pixels oversegmenting the target object.", "3028761": "Also taking raw pixel number instead of the relative \"IoU\" maybe is another way to optimize it.", "3029910": "To follow up on Rohit's question: let's assume two images, both with buildings but for which the noisy masks do not capture those buildings. Won't the IOUs be both zero?", "3035496": "Hi @keiller ,\n\nI will re-post a comment I've made to illustrate my concern regarding the IOU-based ranking. \nFirst, just to make sure we are on the same line, I compute it as: `sum(my_mask & noisy_mask) / sum(my_mask | noisy_mask)`. Is this what you meant or are you using some class-balancing between foreground and background?\n\nI'm asking because I noticed that some noisy masks contain only zeros (164 in total). The numerator in this case will always be zero, no matter how good or bad my mask is. I'm a bit worried also because this number can increase further more, reaching up to ~700, if we allow a few (i.e. less than 1%) positive pixels in the noisy masks. Given that some of these (already few) pixels are noise, it will again lead to a zero intersection, even if my masks are correct.", "3040295": "keiller, could you provide more information on the metric used to rank the noisy labels? If the formula described above by @codrutandrei is used, an educated random approach should, with high probability, already achieve better results than the current submissions in the ranking. Therefore, I would like to make sure that we consider the correct metric and that there is no weighting or anything that should be considered for the iou/ranking (or is considered in the evaluation algorithm)." }, "source": "meta" }