According to my understanding, the R² score should never exceed 0 when the predictions are constant. (If the prediction is the mean of the targets, R² becomes exactly 0 since SS_tot = SS_res, but otherwise it should always be negative.)
\nHowever, in the following notebook, the author makes constant predictions for each target, yet achieves scores above 0:\nhttps://www.kaggle.com/code/jirkaborovec/csiro-img2bio-simple-stat-predictions
\nIf the R² score is computed separately for each target and then combined with the given weights as described in the Overview tab, I don’t think this should be possible.
\nPlease correct me if my understanding is wrong.
", "messages": [ { "id": "3310190", "postDate": "11/02/2025 10:33:50", "content": "According to my understanding, the R² score should never exceed 0 when the predictions are constant. (If the prediction is the mean of the targets, R² becomes exactly 0 since SS_tot = SS_res, but otherwise it should always be negative.)
\nHowever, in the following notebook, the author makes constant predictions for each target, yet achieves scores above 0:\nhttps://www.kaggle.com/code/jirkaborovec/csiro-img2bio-simple-stat-predictions
\nIf the R² score is computed separately for each target and then combined with the given weights as described in the Overview tab, I don’t think this should be possible.
\nPlease correct me if my understanding is wrong.
", "rawMarkdown": "According to my understanding, the R² score should never exceed 0 when the predictions are constant. (If the prediction is the mean of the targets, R² becomes exactly 0 since SS_tot = SS_res, but otherwise it should always be negative.)\n\nHowever, in the following notebook, the author makes constant predictions for each target, yet achieves scores above 0:\nhttps://www.kaggle.com/code/jirkaborovec/csiro-img2bio-simple-stat-predictions\n\nIf the R² score is computed separately for each target and then combined with the given weights as described in the Overview tab, I don’t think this should be possible.\n\nPlease correct me if my understanding is wrong.", "votes": null }, { "id": "3310200", "postDate": "11/02/2025 11:10:04", "content": "I'm also confused about how to calculate the score metric. I have two ways of calculating it: Suppose the input data has one sample id, and that sample has 5 target names. In the first way, I input an array with shape (5,)—where 5 is the number of target names—then the calculated score is relatively low and doesn't match the results shown on the leaderboard. In the second way, I split the original sample id into 5 sample ids as in the submission, then input an array like [[target[0], 0, 0, 0, 0], [0, target[1], 0, 0, 0], …], and the score calculated is much higher. The host has provided a script to calculate the score, but I haven't had time to verify it yet, especially regarding how to input data into the script.\nhttps://www.kaggle.com/competitions/csiro-biomass/discussion/614102
", "rawMarkdown": "I'm also confused about how to calculate the score metric. I have two ways of calculating it: Suppose the input data has one sample id, and that sample has 5 target names. In the first way, I input an array with shape (5,)—where 5 is the number of target names—then the calculated score is relatively low and doesn't match the results shown on the leaderboard. In the second way, I split the original sample id into 5 sample ids as in the submission, then input an array like [[target[0], 0, 0, 0, 0], [0, target[1], 0, 0, 0], ...], and the score calculated is much higher. The host has provided a script to calculate the score, but I haven't had time to verify it yet, especially regarding how to input data into the script.\nhttps://www.kaggle.com/competitions/csiro-biomass/discussion/614102", "votes": null }, { "id": "3310267", "postDate": "11/02/2025 14:08:13", "content": "R is 0 if the predictions are all equal to the mean of the targets, in that notebook the author uses the median. In any case, we don't know the targets of the test data, so it would be hard to submit the mean of the targets to achieve LB score of 0.
\nIn the evaluation metric code supplied by Kaggle (https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785), it looks like they do not calculate R separately and then compute the weighted sum. Instead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target. Based on the r2_score source code from scikit-learn, it seems they multiply the values by the corresponding weights before doing the calculation: https://github.com/scikit-learn/scikit-learn/blob/c60dae2060/sklearn/metrics/_regression.py#L1292
\n\nInstead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target.
\n
Yes, I also thought that was the most likely explanation.\nBut if that’s true, it seems inconsistent with the description of the evaluation metric, so either the explanation or the metric implementation might need to be fixed.
", "rawMarkdown": "> Instead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target.\n\nYes, I also thought that was the most likely explanation.\nBut if that’s true, it seems inconsistent with the description of the evaluation metric, so either the explanation or the metric implementation might need to be fixed.", "votes": null }, { "id": "3310301", "postDate": "11/02/2025 15:50:12", "content": "@qiyuliaocsiro please check it, thanks!
", "rawMarkdown": "qiyuliaocsiro please check it, thanks!", "votes": null }, { "id": "3310428", "postDate": "11/02/2025 21:59:05", "content": "@martynaplomecka Please help to check this. Thanks.
", "rawMarkdown": "martynaplomecka Please help to check this. Thanks.", "votes": null }, { "id": "3310438", "postDate": "11/02/2025 22:53:04", "content": "Thanks everyone for the thoughtful discussion.
\nWe’ve reviewed the evaluation setup and can confirm that there is no issue with the metric implementation. The leaderboard uses a single, globally weighted coefficient of determination (R²) rather than computing R² separately for each target and then averaging.
\nIn practice, this means all (image, target) pairs are combined into one long list, and each row is weighted according to its target type using the weights shown on the Overview page. \nThe metric_weight column in the solution file provides these per-row weights directly to the evaluator.\nBecause the score is computed globally, the denominator of R² is based on the overall weighted mean of all ground-truth values. As a result, a model that predicts a different constant for each target can achieve a positive R², since it effectively reduces between-target variance relative to the global mean.
\nSo the behavior observed in the linked notebook is expected under this global formulation.
\nWe updated the Overview text to make this clearer.
", "rawMarkdown": "Thanks everyone for the thoughtful discussion.\n\nWe’ve reviewed the evaluation setup and can confirm that there is **no issue with the metric implementation**. The leaderboard uses a **single, globally weighted coefficient of determination (R²)** rather than computing R² separately for each target and then averaging.\n\nIn practice, this means all (image, target) pairs are combined into one long list, and each row is weighted according to its target type using the weights shown on the Overview page. \nThe metric_weight column in the solution file provides these per-row weights directly to the evaluator.\nBecause the score is computed globally, the denominator of R² is based on the overall weighted mean of all ground-truth values. As a result, a model that predicts a different constant for each target can achieve a positive R², since it effectively reduces between-target variance relative to the global mean.\n\nSo the behavior observed in the linked notebook is expected under this global formulation.\n\nWe updated the Overview text to make this clearer.", "votes": null }, { "id": "3310445", "postDate": "11/02/2025 23:38:20", "content": "Thank you for your explanation — it helped me completely understand how the leaderboard metric is actually calculated. That said, I still think the description in the Overview tab can be interpreted as meaning that the five R² scores are computed separately and then combined using a weighted average. (Or maybe the Overview tab just hasn’t been updated yet?)
", "rawMarkdown": "Thank you for your explanation — it helped me completely understand how the leaderboard metric is actually calculated. That said, I still think the description in the Overview tab can be interpreted as meaning that the five R² scores are computed separately and then combined using a weighted average. (Or maybe the Overview tab just hasn’t been updated yet?)", "votes": null }, { "id": "3310451", "postDate": "11/03/2025 00:17:27", "content": "Hi, could you please refresh the competition page?
", "rawMarkdown": "Hi, could you please refresh the competition page?", "votes": null }, { "id": "3310452", "postDate": "11/03/2025 00:25:05", "content": "Thank you! The ambiguity in the metric description has been completely resolved.
", "rawMarkdown": "Thank you! The ambiguity in the metric description has been completely resolved.", "votes": null }, { "id": "3310528", "postDate": "11/03/2025 06:15:23", "content": "Great catch on the metric behavior! It's helpful to see this thorough investigation and how the community worked together to clarify it. Thanks for sharing!
", "rawMarkdown": "Great catch on the metric behavior! It's helpful to see this thorough investigation and how the community worked together to clarify it. Thanks for sharing!", "votes": null } ], "comments": [ { "id": 3310200, "author_name": "quan0095", "author_url": "", "post_date": "11/02/2025 11:10:04", "content": "I'm also confused about how to calculate the score metric. I have two ways of calculating it: Suppose the input data has one sample id, and that sample has 5 target names. In the first way, I input an array with shape (5,)—where 5 is the number of target names—then the calculated score is relatively low and doesn't match the results shown on the leaderboard. In the second way, I split the original sample id into 5 sample ids as in the submission, then input an array like [[target[0], 0, 0, 0, 0], [0, target[1], 0, 0, 0], …], and the score calculated is much higher. The host has provided a script to calculate the score, but I haven't had time to verify it yet, especially regarding how to input data into the script.\nhttps://www.kaggle.com/competitions/csiro-biomass/discussion/614102
", "votes": null, "replies": [] }, { "id": 3310267, "author_name": "vaillant", "author_url": "", "post_date": "11/02/2025 14:08:13", "content": "R is 0 if the predictions are all equal to the mean of the targets, in that notebook the author uses the median. In any case, we don't know the targets of the test data, so it would be hard to submit the mean of the targets to achieve LB score of 0.
\nIn the evaluation metric code supplied by Kaggle (https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785), it looks like they do not calculate R separately and then compute the weighted sum. Instead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target. Based on the r2_score source code from scikit-learn, it seems they multiply the values by the corresponding weights before doing the calculation: https://github.com/scikit-learn/scikit-learn/blob/c60dae2060/sklearn/metrics/_regression.py#L1292
\n\nInstead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target.
\n
Yes, I also thought that was the most likely explanation.\nBut if that’s true, it seems inconsistent with the description of the evaluation metric, so either the explanation or the metric implementation might need to be fixed.
", "votes": null, "replies": [ { "id": 3310301, "author_name": "lizhecheng", "author_url": "", "post_date": "11/02/2025 15:50:12", "content": "@qiyuliaocsiro please check it, thanks!
", "votes": null, "replies": [ { "id": 3310428, "author_name": "qiyuliaocsiro", "author_url": "", "post_date": "11/02/2025 21:59:05", "content": "@martynaplomecka Please help to check this. Thanks.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3310438, "author_name": "martynaplomecka", "author_url": "", "post_date": "11/02/2025 22:53:04", "content": "Thanks everyone for the thoughtful discussion.
\nWe’ve reviewed the evaluation setup and can confirm that there is no issue with the metric implementation. The leaderboard uses a single, globally weighted coefficient of determination (R²) rather than computing R² separately for each target and then averaging.
\nIn practice, this means all (image, target) pairs are combined into one long list, and each row is weighted according to its target type using the weights shown on the Overview page. \nThe metric_weight column in the solution file provides these per-row weights directly to the evaluator.\nBecause the score is computed globally, the denominator of R² is based on the overall weighted mean of all ground-truth values. As a result, a model that predicts a different constant for each target can achieve a positive R², since it effectively reduces between-target variance relative to the global mean.
\nSo the behavior observed in the linked notebook is expected under this global formulation.
\nWe updated the Overview text to make this clearer.
", "votes": null, "replies": [ { "id": 3310445, "author_name": "rsakata", "author_url": "", "post_date": "11/02/2025 23:38:20", "content": "Thank you for your explanation — it helped me completely understand how the leaderboard metric is actually calculated. That said, I still think the description in the Overview tab can be interpreted as meaning that the five R² scores are computed separately and then combined using a weighted average. (Or maybe the Overview tab just hasn’t been updated yet?)
", "votes": null, "replies": [ { "id": 3310451, "author_name": "martynaplomecka", "author_url": "", "post_date": "11/03/2025 00:17:27", "content": "Hi, could you please refresh the competition page?
", "votes": null, "replies": [ { "id": 3310452, "author_name": "rsakata", "author_url": "", "post_date": "11/03/2025 00:25:05", "content": "Thank you! The ambiguity in the metric description has been completely resolved.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3310528, "author_name": "aadityaiscoding", "author_url": "", "post_date": "11/03/2025 06:15:23", "content": "Great catch on the metric behavior! It's helpful to see this thorough investigation and how the community worked together to clarify it. Thanks for sharing!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3310190": "According to my understanding, the R² score should never exceed 0 when the predictions are constant. (If the prediction is the mean of the targets, R² becomes exactly 0 since SS_tot = SS_res, but otherwise it should always be negative.)\n\nHowever, in the following notebook, the author makes constant predictions for each target, yet achieves scores above 0:\nhttps://www.kaggle.com/code/jirkaborovec/csiro-img2bio-simple-stat-predictions\n\nIf the R² score is computed separately for each target and then combined with the given weights as described in the Overview tab, I don’t think this should be possible.\n\nPlease correct me if my understanding is wrong.", "3310200": "I'm also confused about how to calculate the score metric. I have two ways of calculating it: Suppose the input data has one sample id, and that sample has 5 target names. In the first way, I input an array with shape (5,)—where 5 is the number of target names—then the calculated score is relatively low and doesn't match the results shown on the leaderboard. In the second way, I split the original sample id into 5 sample ids as in the submission, then input an array like [[target[0], 0, 0, 0, 0], [0, target[1], 0, 0, 0], ...], and the score calculated is much higher. The host has provided a script to calculate the score, but I haven't had time to verify it yet, especially regarding how to input data into the script.\nhttps://www.kaggle.com/competitions/csiro-biomass/discussion/614102", "3310267": "R2 is 0 if the predictions are all equal to the mean of the targets, in that notebook the author uses the median. In any case, we don't know the targets of the test data, so it would be hard to submit the mean of the targets to achieve LB score of 0.\n\nIn the evaluation metric code supplied by Kaggle (https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785), it looks like they do not calculate R2 separately and then compute the weighted sum. Instead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target. Based on the `r2_score` source code from scikit-learn, it seems they multiply the values by the corresponding weights before doing the calculation: https://github.com/scikit-learn/scikit-learn/blob/c60dae2060/sklearn/metrics/_regression.py#L1292", "3310278": "> Instead, they supply all the targets as a single concatenated array and weight each element using the corresponding weight for that target.\n\nYes, I also thought that was the most likely explanation.\nBut if that’s true, it seems inconsistent with the description of the evaluation metric, so either the explanation or the metric implementation might need to be fixed.", "3310301": "qiyuliaocsiro please check it, thanks!", "3310428": "martynaplomecka Please help to check this. Thanks.", "3310438": "Thanks everyone for the thoughtful discussion.\n\nWe’ve reviewed the evaluation setup and can confirm that there is **no issue with the metric implementation**. The leaderboard uses a **single, globally weighted coefficient of determination (R²)** rather than computing R² separately for each target and then averaging.\n\nIn practice, this means all (image, target) pairs are combined into one long list, and each row is weighted according to its target type using the weights shown on the Overview page. \nThe metric_weight column in the solution file provides these per-row weights directly to the evaluator.\nBecause the score is computed globally, the denominator of R² is based on the overall weighted mean of all ground-truth values. As a result, a model that predicts a different constant for each target can achieve a positive R², since it effectively reduces between-target variance relative to the global mean.\n\nSo the behavior observed in the linked notebook is expected under this global formulation.\n\nWe updated the Overview text to make this clearer.", "3310445": "Thank you for your explanation — it helped me completely understand how the leaderboard metric is actually calculated. That said, I still think the description in the Overview tab can be interpreted as meaning that the five R² scores are computed separately and then combined using a weighted average. (Or maybe the Overview tab just hasn’t been updated yet?)", "3310451": "Hi, could you please refresh the competition page?", "3310452": "Thank you! The ambiguity in the metric description has been completely resolved.", "3310528": "Great catch on the metric behavior! It's helpful to see this thorough investigation and how the community worked together to clarify it. Thanks for sharing!" }, "source": "meta" }