I tried different approaches with proper validation locally. My local validation (including KFold CV) consistently shows scores between 0.72 and 0.7499.
But the best I can do on the public leaderboard is 0.51. Is this normal?
What is your local CV score?
I used the metric proposed by Kaggle: https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785
\nI suspect something is either wrong with the training data or the metric used on the hidden set is somehow different from the one described on the competition page?
", "messages": [ { "id": 3311585, "postDate": "2025-11-05T11:00:14.237Z", "content": "I tried different approaches with proper validation locally. My local validation (including KFold CV) consistently shows scores between 0.72 and 0.7499.
But the best I can do on the public leaderboard is 0.51. Is this normal?
What is your local CV score?
I used the metric proposed by Kaggle: https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785
\nI suspect something is either wrong with the training data or the metric used on the hidden set is somehow different from the one described on the competition page?
", "rawMarkdown": "- I tried different approaches with proper validation locally. My local validation (including KFold CV) consistently shows scores between 0.72 and 0.7499. \n\n- But the best I can do on the public leaderboard is 0.51. Is this normal? \n\n- What is your local CV score?\n\nI used the metric proposed by Kaggle: https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785\n\nI suspect something is either wrong with the training data or the metric used on the hidden set is somehow different from the one described on the competition page?", "votes": 41 }, { "id": 3311600, "postDate": "2025-11-05T11:32:23.777Z", "content": "There are some reasons about this huge gap -:)\nFor me, i have try several approaches available to reduce it.
\nMy best gap so long:\nCV: 0.62, LB: 0.56 (used user takaito metric implemented in his public kernel, before resolved metric issue, for me takaito metric works well and help me align with LB instead of the new approach)
\nThe 2nd user on LB:\nCV: 0.79, LB: 0.71 (based on the new metric after it's resolved)
\nOne of reasons is that the test set contains new location so your models must be able to generalize to these unseen locations instead of learning patterns of pastures from locations in the train dataset. I let you discover the remaining causes.
\nHope, it's helpful.
\n[UPDATE] https://www.kaggle.com/competitions/csiro-biomass/discussion/615003
", "rawMarkdown": "There are some reasons about this huge gap -:)\nFor me, i have try several approaches available to reduce it.\n\nMy best gap so long:\nCV: 0.62, LB: 0.56 (used user takaito metric implemented in his public kernel, before resolved metric issue, for me takaito metric works well and help me align with LB instead of the new approach)\n\nThe 2nd user on LB:\nCV: 0.79, LB: 0.71 (based on the new metric after it's resolved)\n\nOne of reasons is that the test set contains new location so your models must be able to generalize to these unseen locations instead of learning patterns of pastures from locations in the train dataset. I let you discover the remaining causes.\n\nHope, it's helpful.\n\n[UPDATE] https://www.kaggle.com/competitions/csiro-biomass/discussion/615003", "votes": 10, "replies": [ { "id": 3311608, "postDate": "2025-11-05T11:51:19.893Z", "content": "Yes this is helpful, thanks
", "rawMarkdown": "Yes this is helpful, thanks" }, { "id": 3311911, "postDate": "2025-11-06T02:50:46.963Z", "content": "what is takito metric can you provide me any reference regarding that metric ?
", "rawMarkdown": "what is takito metric can you provide me any reference regarding that metric ?", "replies": [ { "id": 3311962, "postDate": "2025-11-06T05:55:52.543Z", "content": "\n ():\n \n weights = np.array([, , , , ])\n r2_scores = []\n\n i ():\n y_t = y_true[:, i]\n y_p = y_pred[:, i]\n\n ss_res = np.((y_t - y_p) ** )\n ss_tot = np.((y_t - np.mean(y_t)) ** )\n r2 = - ss_res / ss_tot ss_tot > \n\n r2_scores.append(r2)\n\n r2_scores = np.array(r2_scores)\n weighted_r2 = np.(r2_scores * weights) / np.(weights)\n\n weighted_r2, r2_scores\nReference: [CSIRO-img2bio] Training Notebook
", "rawMarkdown": "```python\n# ======== Weighted R² ========\ndef weighted_r2_score(y_true: np.ndarray, y_pred: np.ndarray):\n \"\"\"\n y_true, y_pred: shape (N, 5)\n \"\"\"\n weights = np.array([0.1, 0.1, 0.1, 0.2, 0.5])\n r2_scores = []\n\n for i in range(5):\n y_t = y_true[:, i]\n y_p = y_pred[:, i]\n\n ss_res = np.sum((y_t - y_p) ** 2)\n ss_tot = np.sum((y_t - np.mean(y_t)) ** 2)\n r2 = 1 - ss_res / ss_tot if ss_tot > 0 else 0.0\n\n r2_scores.append(r2)\n\n r2_scores = np.array(r2_scores)\n weighted_r2 = np.sum(r2_scores * weights) / np.sum(weights)\n\n return weighted_r2, r2_scores\n```\n\nReference: [[CSIRO-img2bio] Training Notebook](https://www.kaggle.com/code/takaito/csiro-img2bio-training-notebook)", "votes": 3 }, { "id": 3312424, "postDate": "2025-11-07T06:51:06.227Z", "content": "after using this metric still getting huge gap between cv and lb
", "rawMarkdown": "after using this metric still getting huge gap between cv and lb\n" }, { "id": 3312449, "postDate": "2025-11-07T07:38:45.540Z", "content": "Here is a quick snapshot tutorial -:)\nWrite with heart so sorry for my poor english (I'm french speaker)…
\nEx: If you got +0.05 after improving feature engineering during CV. You will know how you will got on LB.\nAnd for me, the @takaito metric implemented in his public kernel helps me do that moving me from 159th to 24th position.
\nEx: tf_efficientnetv2_l (CV: 0.64, LB: 0.56)\ntf_efficientnetv2_s (CV: 0.62, LB: 0.56)
\nWe got same results but reduce a gap from -0.02. It is huge for me.
\nIs this wrong? the evaluation section mentions:
\n\n", "rawMarkdown": "Is this wrong? the evaluation section mentions:\n\n>The model performance is evaluated using a globally weighted coefficient of determination (R²) computed over all (image, target) pairs together.", "votes": 1 }, { "id": 3312850, "postDate": "2025-11-08T05:44:23.307Z", "content": "The model performance is evaluated using a globally weighted coefficient of determination (R²) computed over all (image, target) pairs together.
\n
I’m using a global_weighted_r2 function generated by gpt, but there’s a huge gap between my cv and lb(cv-0.5048, lb-0.35). Could you share your CV-LB score?
\n global_weighted_r2(y_true: np.ndarray, y_pred: np.ndarray):\n\n = np.array([., ., ., ., .])\n = y_true.reshape(-)\n = y_pred.reshape(-)\n = np.tile(weights, y_true.shape[])\n\n = np.average(y_true_flat, weights=w_flat)\n\n = np.sum(w_flat * (y_true_flat - y_pred_flat) ** )\n = np.sum(w_flat * (y_true_flat - y_true_mean) ** )\n\n = - ss_res / ss_tot if ss_tot > else .\n\n r2_global\nYou may want to compare your metric with the one I posted at the top of this discussion. I did that with my own metric and found the exact same score.
", "rawMarkdown": "You may want to compare your metric with the one I posted at the top of this discussion. I did that with my own metric and found the exact same score.", "votes": 1 }, { "id": 3314042, "postDate": "2025-11-10T03:52:51.373Z", "content": "Have you found any reason for the gap? I used this metric too, and same huge gap ~0.3 between CV and LB.
", "rawMarkdown": "Have you found any reason for the gap? I used this metric too, and same huge gap ~0.3 between CV and LB." }, { "id": 3314635, "postDate": "2025-11-10T09:10:54.927Z", "content": "Yes, I normalized the targets during training. After removing the normalization, the CV-LB gap got smaller, but there's still no clear correlation between CV and LB.
\n\n", "rawMarkdown": "Yes, I normalized the targets during training. After removing the normalization, the CV-LB gap got smaller, but there's still no clear correlation between CV and LB.\n\n> Split folds by date + state\n\n> CV: 0.559 vs LB: 0.43\n\n> CV: 0.492 vs LB: 0.46\n\n\n> Split folds by date\n\n> CV: 0.462 vs LB: 0.40\n\n> CV: 0.468 vs LB: 0.39" }, { "id": 3339229, "postDate": "2025-11-19T01:19:59.817Z", "content": "Split folds by date + state
\nCV: 0.559 vs LB: 0.43
\nCV: 0.492 vs LB: 0.46
\nSplit folds by date
\nCV: 0.462 vs LB: 0.40
\nCV: 0.468 vs LB: 0.39
\n
this matches with the paper but not the overview here, the equations in the overview for the competition show the weights being applied within the SS_res and SS_tot computations as well as for the mean target computation -- whereas this code here is as shown in the paper… separate R-squared for each target then the weighted average of those R-squared values
", "rawMarkdown": "this matches with the paper but not the overview here, the equations in the overview for the competition show the weights being applied within the SS_res and SS_tot computations as well as for the mean target computation -- whereas this code here is as shown in the paper... separate R-squared for each target then the weighted average of those R-squared values" } ] }, { "id": 3311940, "postDate": "2025-11-06T04:53:09.547Z", "content": "Woah, now I know one of the factors my model were heavily overfitting. I stratified across the 4 states while grouping across the sampling date. Thanks for sharing that there are unseen locations. So now does it mean I have to GroupKFold on Sampling Date + State?
", "rawMarkdown": "Woah, now I know one of the factors my model were heavily overfitting. I stratified across the 4 states while grouping across the sampling date. Thanks for sharing that there are unseen locations. So now does it mean I have to GroupKFold on Sampling Date + State? ", "votes": 1, "replies": [ { "id": 3311966, "postDate": "2025-11-06T06:04:09.827Z", "content": "I just use the default as said by @lizhecheng : Comment at the top of this public discussion.\nI think working on regularization parameters would work also. Not only at the validation level.
", "rawMarkdown": "I just use the default as said by @lizhecheng : Comment at the top of this public discussion.\nI think working on regularization parameters would work also. Not only at the validation level." } ] }, { "id": 3336452, "postDate": "2025-11-18T13:45:26.260Z", "content": "Hi, I am also struggling with getting any correlation between my local estimate and the lb-score. What has been resolved about the metric? Was there a problem?
", "rawMarkdown": "Hi, I am also struggling with getting any correlation between my local estimate and the lb-score. What has been resolved about the metric? Was there a problem?" } ] }, { "id": 3319796, "postDate": "2025-11-12T03:59:23.820Z", "content": "my local CV is 0.79, and LB 0.74 but can vary
", "rawMarkdown": "my local CV is 0.79, and LB 0.74 but can vary", "votes": 6, "replies": [ { "id": 3320047, "postDate": "2025-11-12T07:42:17.543Z", "content": "I can achieve local cv with 0.763, but the LB only reachs 0.65😂
", "rawMarkdown": "I can achieve local cv with 0.763, but the LB only reachs 0.65😂", "replies": [ { "id": 3322598, "postDate": "2025-11-13T15:42:37.603Z", "content": "thats pretty good, my first couple of submissions I had like 0.81 local CV but then a 0.65 on LB
", "rawMarkdown": "thats pretty good, my first couple of submissions I had like 0.81 local CV but then a 0.65 on LB" } ] }, { "id": 3320166, "postDate": "2025-11-12T09:39:29.220Z", "content": "Congrats on finding the way to go:)
", "rawMarkdown": "Congrats on finding the way to go:)" }, { "id": 3333615, "postDate": "2025-11-17T08:28:45.810Z", "content": "if possible can you tell what split did you use? I am getting cv of around .79 but .51 as lb
", "rawMarkdown": "if possible can you tell what split did you use? I am getting cv of around .79 but .51 as lb", "votes": 3 } ] }, { "id": 3378037, "postDate": "2025-12-17T12:23:39.277Z", "content": "It seems that the amount of training data is small
", "rawMarkdown": "It seems that the amount of training data is small" }, { "id": 3312716, "postDate": "2025-11-07T19:03:47.730Z", "content": "straight facts
", "rawMarkdown": "straight facts", "votes": 3 }, { "id": 3312005, "postDate": "2025-11-06T07:37:49.703Z", "content": "my latest result:
\n~0.85 CV 0.60 LB
", "rawMarkdown": "my latest result:\n\n~0.85 CV 0.60 LB", "votes": 1, "replies": [ { "id": 3322600, "postDate": "2025-11-13T15:44:55.660Z", "content": "Had a similar result from my first submisions
", "rawMarkdown": "Had a similar result from my first submisions", "votes": 1, "replies": [ { "id": 3376799, "postDate": "2025-12-15T06:27:59.113Z", "content": "Do you get any idea why this happened? I am quite curious. Thanks in advance
", "rawMarkdown": "Do you get any idea why this happened? I am quite curious. Thanks in advance" }, { "id": 3376818, "postDate": "2025-12-15T07:09:57.723Z", "content": "Data Leakage. Your model is basically cheating. You are not training your model well for the final exams, you are providing very easy to solve problems to your model. Hence, your model performs poorly on the final exams.
", "rawMarkdown": "Data Leakage. Your model is basically cheating. You are not training your model well for the final exams, you are providing very easy to solve problems to your model. Hence, your model performs poorly on the final exams. " }, { "id": 3376846, "postDate": "2025-12-15T09:03:14.613Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3376847, "postDate": "2025-12-15T09:03:29.053Z", "content": "May I ask what's the metric you used in evaluation?
", "rawMarkdown": "May I ask what's the metric you used in evaluation?" } ] } ] }, { "id": 3311804, "postDate": "2025-11-05T20:07:52.517Z", "content": "I tried different split methods, and based on my current results, a random kfold split after grouping by image id seems to work best.
", "rawMarkdown": "I tried different split methods, and based on my current results, a random kfold split after grouping by image id seems to work best.", "votes": 1, "replies": [ { "id": 3311839, "postDate": "2025-11-05T22:16:10.240Z", "content": "This is my default split but still
", "rawMarkdown": "This is my default split but still " }, { "id": 3311965, "postDate": "2025-11-06T06:02:03.393Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3311967, "postDate": "2025-11-06T06:10:08.247Z", "content": "just got a holdout 10% validation of 0.72 (inner KFold CV of 0.74) but LB is 0.5 haha
", "rawMarkdown": "just got a holdout 10% validation of 0.72 (inner KFold CV of 0.74) but LB is 0.5 haha", "replies": [ { "id": 3311981, "postDate": "2025-11-06T07:04:06.703Z", "content": "And this 0.5 as score is strange. It suggests that ss_res/ss_tot is exactly 0.5 too. I got 0.5 with ViT-large which had a good CV + (training score of 0.91). Looks like overfitting the training data. Validation score didn't matter here
", "rawMarkdown": "And this 0.5 as score is strange. It suggests that ss_res/ss_tot is exactly 0.5 too. I got 0.5 with ViT-large which had a good CV + (training score of 0.91). Looks like overfitting the training data. Validation score didn't matter here" }, { "id": 3312000, "postDate": "2025-11-06T07:22:22.377Z", "content": "its not exactly 0.5000000 right? that would be so strange…
\nIt definitely overfitted on my end, at the beginning, it was 0.48 cv and 0.48 lb, my agent iterated, tested various ideas to improve it to 0.72 cv. But the LB only increased from 0.48 to 0.50
", "rawMarkdown": "its not exactly 0.5000000 right? that would be so strange...\n\nIt definitely overfitted on my end, at the beginning, it was 0.48 cv and 0.48 lb, my agent iterated, tested various ideas to improve it to 0.72 cv. But the LB only increased from 0.48 to 0.50" }, { "id": 3312003, "postDate": "2025-11-06T07:31:21.623Z", "content": "Not exactly 0.500. But I got it at many occasions. Adding that to your own experience makes it strange. One may need to investigate better what ss_res/ss_tot ~ 0.5 translates to.
", "rawMarkdown": "Not exactly 0.500. But I got it at many occasions. Adding that to your own experience makes it strange. One may need to investigate better what ss_res/ss_tot ~ 0.5 translates to." }, { "id": 3313264, "postDate": "2025-11-09T03:02:50.527Z", "content": "I changed my split to stratify by Sampling_Date (Month) and State and the same pipeline only achieved 0.536@10% holdout
", "rawMarkdown": "I changed my split to stratify by Sampling_Date (Month) and State and the same pipeline only achieved 0.536@10% holdout" } ] } ] }, { "id": 3311797, "postDate": "2025-11-05T19:40:33.657Z", "content": "Did you use the metadata for training ? Note the test has no metadata.
\nWhen I trained with metadata, the local CV is ~0.8 but is not reflective since in test these are set to NULLs
", "rawMarkdown": "Did you use the metadata for training ? Note the test has no metadata.\n\nWhen I trained with metadata, the local CV is ~0.8 but is not reflective since in test these are set to NULLs", "votes": 1, "replies": [ { "id": 3311826, "postDate": "2025-11-05T21:36:28.963Z", "content": "Yep, waiting to see how well CV shakes out before I look into it myself.
", "rawMarkdown": "Yep, waiting to see how well CV shakes out before I look into it myself. " }, { "id": 3311928, "postDate": "2025-11-06T04:07:37.157Z", "content": "Why are you using metadata as features exactly even though it is not present in the test data?\nThere are two options here it seems:
\nI am trying the second one, the CV-LB gap worsens. The correct weighting of the losses is crucial and difficult.
", "rawMarkdown": "Why are you using metadata as features exactly even though it is not present in the test data?\nThere are two options here it seems:\n1. Use a separate model and specialize them to predict these features from the image, and then pass their predictions to your original model\n2. Train your model to predict them, using the prediction of metadata as auxiliary tasks.\n\nI am trying the second one, the CV-LB gap worsens. The correct weighting of the losses is crucial and difficult. ", "votes": 2, "replies": [ { "id": 3311933, "postDate": "2025-11-06T04:24:21.340Z", "content": "I am no longer using it, was giving an example which leads to optimistic CV scores…
", "rawMarkdown": "I am no longer using it, was giving an example which leads to optimistic CV scores..." } ] } ] }, { "id": 3312076, "postDate": "2025-11-06T11:49:26.713Z", "content": "I have some models with CV from 0.71 to 0.79, but their performance on LB can vary from 0.50 to 0.66. It's truly interesting. Still trying to find a solid method to verify performance.
", "rawMarkdown": "I have some models with CV from 0.71 to 0.79, but their performance on LB can vary from 0.50 to 0.66. It's truly interesting. Still trying to find a solid method to verify performance.", "votes": 2, "replies": [ { "id": 3312106, "postDate": "2025-11-06T13:17:19.970Z", "content": "I find it quite interesting. I am not seeing any correlation between my local score and my leaderboard score. I am quite sure I know what is happening, but I have not found a good way to replicate it in training…
", "rawMarkdown": "I find it quite interesting. I am not seeing any correlation between my local score and my leaderboard score. I am quite sure I know what is happening, but I have not found a good way to replicate it in training...\n", "votes": 3, "replies": [ { "id": 3314942, "postDate": "2025-11-10T12:32:26.390Z", "content": "With your current rank I guess you have figured out what was wrong:)
", "rawMarkdown": "With your current rank I guess you have figured out what was wrong:)" }, { "id": 3314964, "postDate": "2025-11-10T12:47:48.670Z", "content": "Thanks! But no. I am quite new to this, and I am playing with how to deal with this problem. I think I am getting closer but in all honesty my local CV scores do not represent my lb score. I am nervous about the hidden lb.
", "rawMarkdown": "Thanks! But no. I am quite new to this, and I am playing with how to deal with this problem. I think I am getting closer but in all honesty my local CV scores do not represent my lb score. I am nervous about the hidden lb." } ] }, { "id": 3312431, "postDate": "2025-11-07T07:03:39.910Z", "content": "Climbed the leaderboard at an incredible speed. Amazing!\nYes, Splitting is indeed very crucial here. Some have achieved great results using simple KFold or Stratified KFold, which doesn't seem to be working for me. CV is ~0.76 with these strategies but the LB is ~0.5.\nThe GroupKFold on State+Sampling_Date has been the best for me so far, though not the perfect.
", "rawMarkdown": "Climbed the leaderboard at an incredible speed. Amazing!\nYes, Splitting is indeed very crucial here. Some have achieved great results using simple KFold or Stratified KFold, which doesn't seem to be working for me. CV is ~0.76 with these strategies but the LB is ~0.5.\nThe GroupKFold on State+Sampling_Date has been the best for me so far, though not the perfect.", "votes": 3 }, { "id": 3318651, "postDate": "2025-11-11T09:30:18.133Z", "content": "Did you solve the problem?
", "rawMarkdown": "Did you solve the problem? ", "replies": [ { "id": 3319120, "postDate": "2025-11-11T15:18:01.443Z", "content": "Actually, no. It's pretty like guessing some method can work and submit it, then see how the LB changes.
", "rawMarkdown": "Actually, no. It's pretty like guessing some method can work and submit it, then see how the LB changes.", "votes": 1 }, { "id": 3395453, "postDate": "2026-01-23T02:29:13.323Z", "content": "did you solve this now？ And could u plz tell me how?
", "rawMarkdown": "did you solve this now？ And could u plz tell me how?" } ] }, { "id": 3322599, "postDate": "2025-11-13T15:44:25.713Z", "content": "same here, my CV to LB ranges quite a bit from ~0.5 - 0.10
", "rawMarkdown": "same here, my CV to LB ranges quite a bit from ~0.5 - 0.10", "votes": 1, "replies": [ { "id": 3336583, "postDate": "2025-11-18T14:35:23.720Z", "content": "Ok, so if even the people at the top are seeing this wild variance in the lb scores, then the private lb might be a total different situation. I am seeing huge differences between scores for individual folds as well as between splits with different random seeds. If the data is so unevenly distributed, the upper quarter of the lb will basically be totally reshuffled …
", "rawMarkdown": "Ok, so if even the people at the top are seeing this wild variance in the lb scores, then the private lb might be a total different situation. I am seeing huge differences between scores for individual folds as well as between splits with different random seeds. If the data is so unevenly distributed, the upper quarter of the lb will basically be totally reshuffled ...", "votes": 3 }, { "id": 3336719, "postDate": "2025-11-18T15:51:06.950Z", "content": "Yes, a lot of people are clearly overfitting to the public leaderboard. Most of the public notebooks are using an outdated evaluation approach (computing R² separately and then taking a weighted mean)…\nI’m still working on establishing a reliable validation strategy.
", "rawMarkdown": "Yes, a lot of people are clearly overfitting to the public leaderboard. Most of the public notebooks are using an outdated evaluation approach (computing R² separately and then taking a weighted mean)...\nI’m still working on establishing a reliable validation strategy.", "votes": 2 } ] } ] }, { "id": 3377544, "postDate": "2025-12-16T14:47:10.217Z", "content": "Local CV is 0.69, LB is 0.47. I used random KFold, and I suspect there is data leakage.😭
", "rawMarkdown": "Local CV is 0.69, LB is 0.47. I used random KFold, and I suspect there is data leakage.😭" }, { "id": 3363358, "postDate": "2025-12-05T22:05:47.880Z", "content": "cv0.72 lb0.69
", "rawMarkdown": "cv0.72 lb0.69" }, { "id": 3331733, "postDate": "2025-11-16T14:06:18.557Z", "content": "My local CV score is 0.71, but my LB is only 0.56. I’m hoping to narrow that gap as I continue refining my approach.
", "rawMarkdown": "My local CV score is **0.71**, but my LB is only **0.56**. I’m hoping to narrow that gap as I continue refining my approach." }, { "id": 3320985, "postDate": "2025-11-13T01:12:33.310Z", "content": "CV: 0.67\nLB: 0.62
", "rawMarkdown": "CV: 0.67\nLB: 0.62" }, { "id": 3319672, "postDate": "2025-11-12T01:31:46.060Z", "content": "ignore Dry_Total_g, GDM_g\nIn the prediction
", "rawMarkdown": "ignore Dry_Total_g, GDM_g\nIn the prediction" }, { "id": 3319671, "postDate": "2025-11-12T01:29:29.760Z", "content": "Yes its normal, I got 0.55 on LB with efficientnet B3, but my Locally score was around 0.68.
", "rawMarkdown": "Yes its normal, I got 0.55 on LB with efficientnet B3, but my Locally score was around 0.68." }, { "id": 3312332, "postDate": "2025-11-07T02:48:50.967Z", "content": "cv0.54 lb0.58
", "rawMarkdown": "cv0.54 lb0.58", "replies": [ { "id": 3312932, "postDate": "2025-11-08T09:37:26.190Z", "content": "I think I’ve finally found a fully reliable CV strategy! After several experiments, the results are very consistent: CV 0.519 → LB 0.52, CV 0.54 → LB 0.54, CV 0.57 → LB 0.57.
", "rawMarkdown": "I think I’ve finally found a fully reliable CV strategy! After several experiments, the results are very consistent: CV 0.519 → LB 0.52, CV 0.54 → LB 0.54, CV 0.57 → LB 0.57.\n", "votes": 1, "replies": [ { "id": 3312935, "postDate": "2025-11-08T09:41:56.850Z", "content": "Congratulations -:)
", "rawMarkdown": "Congratulations -:)" }, { "id": 3312988, "postDate": "2025-11-08T12:00:14.183Z", "content": "Great! Did you use a different metric or still the same as competition's?
", "rawMarkdown": "Great! Did you use a different metric or still the same as competition's?" }, { "id": 3313334, "postDate": "2025-11-09T05:32:37.677Z", "content": "the same as competition's
", "rawMarkdown": "the same as competition's" } ] } ] } ], "comments": [ { "id": 3311600, "author_name": "Yehoshua", "author_url": "", "post_date": "2025-11-05T11:32:23.777000", "content": "There are some reasons about this huge gap -:)\nFor me, i have try several approaches available to reduce it.
\nMy best gap so long:\nCV: 0.62, LB: 0.56 (used user takaito metric implemented in his public kernel, before resolved metric issue, for me takaito metric works well and help me align with LB instead of the new approach)
\nThe 2nd user on LB:\nCV: 0.79, LB: 0.71 (based on the new metric after it's resolved)
\nOne of reasons is that the test set contains new location so your models must be able to generalize to these unseen locations instead of learning patterns of pastures from locations in the train dataset. I let you discover the remaining causes.
\nHope, it's helpful.
\n[UPDATE] https://www.kaggle.com/competitions/csiro-biomass/discussion/615003
", "votes": 10, "replies": [ { "id": 3311608, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-05T11:51:19.893000", "content": "Yes this is helpful, thanks
", "votes": 0, "replies": [] }, { "id": 3311911, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2025-11-06T02:50:46.963000", "content": "what is takito metric can you provide me any reference regarding that metric ?
", "votes": 0, "replies": [ { "id": 3311962, "author_name": "Yehoshua", "author_url": "", "post_date": "2025-11-06T05:55:52.543000", "content": "\n ():\n \n weights = np.array([, , , , ])\n r2_scores = []\n\n i ():\n y_t = y_true[:, i]\n y_p = y_pred[:, i]\n\n ss_res = np.((y_t - y_p) ** )\n ss_tot = np.((y_t - np.mean(y_t)) ** )\n r2 = - ss_res / ss_tot ss_tot > \n\n r2_scores.append(r2)\n\n r2_scores = np.array(r2_scores)\n weighted_r2 = np.(r2_scores * weights) / np.(weights)\n\n weighted_r2, r2_scores\nReference: [CSIRO-img2bio] Training Notebook
", "votes": 3, "replies": [] }, { "id": 3312424, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2025-11-07T06:51:06.227000", "content": "after using this metric still getting huge gap between cv and lb
", "votes": 0, "replies": [] }, { "id": 3312449, "author_name": "Yehoshua", "author_url": "", "post_date": "2025-11-07T07:38:45.540000", "content": "Here is a quick snapshot tutorial -:)\nWrite with heart so sorry for my poor english (I'm french speaker)…
\nEx: If you got +0.05 after improving feature engineering during CV. You will know how you will got on LB.\nAnd for me, the @takaito metric implemented in his public kernel helps me do that moving me from 159th to 24th position.
\nEx: tf_efficientnetv2_l (CV: 0.64, LB: 0.56)\ntf_efficientnetv2_s (CV: 0.62, LB: 0.56)
\nWe got same results but reduce a gap from -0.02. It is huge for me.
\nIs this wrong? the evaluation section mentions:
\n\n", "votes": 1, "replies": [] }, { "id": 3312850, "author_name": "Rongchu", "author_url": "", "post_date": "2025-11-08T05:44:23.307000", "content": "The model performance is evaluated using a globally weighted coefficient of determination (R²) computed over all (image, target) pairs together.
\n
I’m using a global_weighted_r2 function generated by gpt, but there’s a huge gap between my cv and lb(cv-0.5048, lb-0.35). Could you share your CV-LB score?
\n global_weighted_r2(y_true: np.ndarray, y_pred: np.ndarray):\n\n = np.array([., ., ., ., .])\n = y_true.reshape(-)\n = y_pred.reshape(-)\n = np.tile(weights, y_true.shape[])\n\n = np.average(y_true_flat, weights=w_flat)\n\n = np.sum(w_flat * (y_true_flat - y_pred_flat) ** )\n = np.sum(w_flat * (y_true_flat - y_true_mean) ** )\n\n = - ss_res / ss_tot if ss_tot > else .\n\n r2_global\nYou may want to compare your metric with the one I posted at the top of this discussion. I did that with my own metric and found the exact same score.
", "votes": 1, "replies": [] }, { "id": 3314042, "author_name": "fruit monk", "author_url": "", "post_date": "2025-11-10T03:52:51.373000", "content": "Have you found any reason for the gap? I used this metric too, and same huge gap ~0.3 between CV and LB.
", "votes": 0, "replies": [] }, { "id": 3314635, "author_name": "Rongchu", "author_url": "", "post_date": "2025-11-10T09:10:54.927000", "content": "Yes, I normalized the targets during training. After removing the normalization, the CV-LB gap got smaller, but there's still no clear correlation between CV and LB.
\n\n", "votes": 0, "replies": [] }, { "id": 3339229, "author_name": "Brian Lynch", "author_url": "", "post_date": "2025-11-19T01:19:59.817000", "content": "Split folds by date + state
\nCV: 0.559 vs LB: 0.43
\nCV: 0.492 vs LB: 0.46
\nSplit folds by date
\nCV: 0.462 vs LB: 0.40
\nCV: 0.468 vs LB: 0.39
\n
this matches with the paper but not the overview here, the equations in the overview for the competition show the weights being applied within the SS_res and SS_tot computations as well as for the mean target computation -- whereas this code here is as shown in the paper… separate R-squared for each target then the weighted average of those R-squared values
", "votes": 0, "replies": [] } ] }, { "id": 3311940, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-11-06T04:53:09.547000", "content": "Woah, now I know one of the factors my model were heavily overfitting. I stratified across the 4 states while grouping across the sampling date. Thanks for sharing that there are unseen locations. So now does it mean I have to GroupKFold on Sampling Date + State?
", "votes": 1, "replies": [ { "id": 3311966, "author_name": "Yehoshua", "author_url": "", "post_date": "2025-11-06T06:04:09.827000", "content": "I just use the default as said by @lizhecheng : Comment at the top of this public discussion.\nI think working on regularization parameters would work also. Not only at the validation level.
", "votes": 0, "replies": [] } ] }, { "id": 3336452, "author_name": "thomas rost", "author_url": "", "post_date": "2025-11-18T13:45:26.260000", "content": "Hi, I am also struggling with getting any correlation between my local estimate and the lb-score. What has been resolved about the metric? Was there a problem?
", "votes": 0, "replies": [] } ] }, { "id": 3319796, "author_name": "evnsnclr", "author_url": "", "post_date": "2025-11-12T03:59:23.820000", "content": "my local CV is 0.79, and LB 0.74 but can vary
", "votes": 6, "replies": [ { "id": 3320047, "author_name": "Wenxuan Ye", "author_url": "", "post_date": "2025-11-12T07:42:17.543000", "content": "I can achieve local cv with 0.763, but the LB only reachs 0.65😂
", "votes": 0, "replies": [ { "id": 3322598, "author_name": "evnsnclr", "author_url": "", "post_date": "2025-11-13T15:42:37.603000", "content": "thats pretty good, my first couple of submissions I had like 0.81 local CV but then a 0.65 on LB
", "votes": 0, "replies": [] } ] }, { "id": 3320166, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-12T09:39:29.220000", "content": "Congrats on finding the way to go:)
", "votes": 0, "replies": [] }, { "id": 3333615, "author_name": "ghatotkachh", "author_url": "", "post_date": "2025-11-17T08:28:45.810000", "content": "if possible can you tell what split did you use? I am getting cv of around .79 but .51 as lb
", "votes": 3, "replies": [] } ] }, { "id": 3378037, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-12-17T12:23:39.277000", "content": "It seems that the amount of training data is small
", "votes": 0, "replies": [] }, { "id": 3312716, "author_name": "ImperfectKitto", "author_url": "", "post_date": "2025-11-07T19:03:47.730000", "content": "straight facts
", "votes": 3, "replies": [] }, { "id": 3312005, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2025-11-06T07:37:49.703000", "content": "my latest result:
\n~0.85 CV 0.60 LB
", "votes": 1, "replies": [ { "id": 3322600, "author_name": "evnsnclr", "author_url": "", "post_date": "2025-11-13T15:44:55.660000", "content": "Had a similar result from my first submisions
", "votes": 1, "replies": [ { "id": 3376799, "author_name": "DJS885", "author_url": "", "post_date": "2025-12-15T06:27:59.113000", "content": "Do you get any idea why this happened? I am quite curious. Thanks in advance
", "votes": 0, "replies": [] }, { "id": 3376818, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-12-15T07:09:57.723000", "content": "Data Leakage. Your model is basically cheating. You are not training your model well for the final exams, you are providing very easy to solve problems to your model. Hence, your model performs poorly on the final exams.
", "votes": 0, "replies": [] }, { "id": 3376846, "author_name": "", "author_url": "", "post_date": "2025-12-15T09:03:14.613000", "content": "", "votes": 0, "replies": [] }, { "id": 3376847, "author_name": "DJS885", "author_url": "", "post_date": "2025-12-15T09:03:29.053000", "content": "May I ask what's the metric you used in evaluation?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3311804, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2025-11-05T20:07:52.517000", "content": "I tried different split methods, and based on my current results, a random kfold split after grouping by image id seems to work best.
", "votes": 1, "replies": [ { "id": 3311839, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-05T22:16:10.240000", "content": "This is my default split but still
", "votes": 0, "replies": [] }, { "id": 3311965, "author_name": "", "author_url": "", "post_date": "2025-11-06T06:02:03.393000", "content": "", "votes": 0, "replies": [] }, { "id": 3311967, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-06T06:10:08.247000", "content": "just got a holdout 10% validation of 0.72 (inner KFold CV of 0.74) but LB is 0.5 haha
", "votes": 0, "replies": [ { "id": 3311981, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-06T07:04:06.703000", "content": "And this 0.5 as score is strange. It suggests that ss_res/ss_tot is exactly 0.5 too. I got 0.5 with ViT-large which had a good CV + (training score of 0.91). Looks like overfitting the training data. Validation score didn't matter here
", "votes": 0, "replies": [] }, { "id": 3312000, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-06T07:22:22.377000", "content": "its not exactly 0.5000000 right? that would be so strange…
\nIt definitely overfitted on my end, at the beginning, it was 0.48 cv and 0.48 lb, my agent iterated, tested various ideas to improve it to 0.72 cv. But the LB only increased from 0.48 to 0.50
", "votes": 0, "replies": [] }, { "id": 3312003, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-06T07:31:21.623000", "content": "Not exactly 0.500. But I got it at many occasions. Adding that to your own experience makes it strange. One may need to investigate better what ss_res/ss_tot ~ 0.5 translates to.
", "votes": 0, "replies": [] }, { "id": 3313264, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-09T03:02:50.527000", "content": "I changed my split to stratify by Sampling_Date (Month) and State and the same pipeline only achieved 0.536@10% holdout
", "votes": 0, "replies": [] } ] } ] }, { "id": 3311797, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-05T19:40:33.657000", "content": "Did you use the metadata for training ? Note the test has no metadata.
\nWhen I trained with metadata, the local CV is ~0.8 but is not reflective since in test these are set to NULLs
", "votes": 1, "replies": [ { "id": 3311826, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-11-05T21:36:28.963000", "content": "Yep, waiting to see how well CV shakes out before I look into it myself.
", "votes": 0, "replies": [] }, { "id": 3311928, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-11-06T04:07:37.157000", "content": "Why are you using metadata as features exactly even though it is not present in the test data?\nThere are two options here it seems:
\nI am trying the second one, the CV-LB gap worsens. The correct weighting of the losses is crucial and difficult.
", "votes": 2, "replies": [ { "id": 3311933, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-06T04:24:21.340000", "content": "I am no longer using it, was giving an example which leads to optimistic CV scores…
", "votes": 0, "replies": [] } ] } ] }, { "id": 3312076, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-11-06T11:49:26.713000", "content": "I have some models with CV from 0.71 to 0.79, but their performance on LB can vary from 0.50 to 0.66. It's truly interesting. Still trying to find a solid method to verify performance.
", "votes": 2, "replies": [ { "id": 3312106, "author_name": "Tim Krige", "author_url": "", "post_date": "2025-11-06T13:17:19.970000", "content": "I find it quite interesting. I am not seeing any correlation between my local score and my leaderboard score. I am quite sure I know what is happening, but I have not found a good way to replicate it in training…
", "votes": 3, "replies": [ { "id": 3314942, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-10T12:32:26.390000", "content": "With your current rank I guess you have figured out what was wrong:)
", "votes": 0, "replies": [] }, { "id": 3314964, "author_name": "Tim Krige", "author_url": "", "post_date": "2025-11-10T12:47:48.670000", "content": "Thanks! But no. I am quite new to this, and I am playing with how to deal with this problem. I think I am getting closer but in all honesty my local CV scores do not represent my lb score. I am nervous about the hidden lb.
", "votes": 0, "replies": [] } ] }, { "id": 3312431, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-11-07T07:03:39.910000", "content": "Climbed the leaderboard at an incredible speed. Amazing!\nYes, Splitting is indeed very crucial here. Some have achieved great results using simple KFold or Stratified KFold, which doesn't seem to be working for me. CV is ~0.76 with these strategies but the LB is ~0.5.\nThe GroupKFold on State+Sampling_Date has been the best for me so far, though not the perfect.
", "votes": 3, "replies": [] }, { "id": 3318651, "author_name": "shanzhong8", "author_url": "", "post_date": "2025-11-11T09:30:18.133000", "content": "Did you solve the problem?
", "votes": 0, "replies": [ { "id": 3319120, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-11-11T15:18:01.443000", "content": "Actually, no. It's pretty like guessing some method can work and submit it, then see how the LB changes.
", "votes": 1, "replies": [] }, { "id": 3395453, "author_name": "sam_the_rice_cake", "author_url": "", "post_date": "2026-01-23T02:29:13.323000", "content": "did you solve this now？ And could u plz tell me how?
", "votes": 0, "replies": [] } ] }, { "id": 3322599, "author_name": "evnsnclr", "author_url": "", "post_date": "2025-11-13T15:44:25.713000", "content": "same here, my CV to LB ranges quite a bit from ~0.5 - 0.10
", "votes": 1, "replies": [ { "id": 3336583, "author_name": "thomas rost", "author_url": "", "post_date": "2025-11-18T14:35:23.720000", "content": "Ok, so if even the people at the top are seeing this wild variance in the lb scores, then the private lb might be a total different situation. I am seeing huge differences between scores for individual folds as well as between splits with different random seeds. If the data is so unevenly distributed, the upper quarter of the lb will basically be totally reshuffled …
", "votes": 3, "replies": [] }, { "id": 3336719, "author_name": "Nizar Charrada", "author_url": "", "post_date": "2025-11-18T15:51:06.950000", "content": "Yes, a lot of people are clearly overfitting to the public leaderboard. Most of the public notebooks are using an outdated evaluation approach (computing R² separately and then taking a weighted mean)…\nI’m still working on establishing a reliable validation strategy.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3377544, "author_name": "YYH", "author_url": "", "post_date": "2025-12-16T14:47:10.217000", "content": "Local CV is 0.69, LB is 0.47. I used random KFold, and I suspect there is data leakage.😭
", "votes": 0, "replies": [] }, { "id": 3363358, "author_name": "RｒＲｒＲ", "author_url": "", "post_date": "2025-12-05T22:05:47.880000", "content": "cv0.72 lb0.69
", "votes": 0, "replies": [] }, { "id": 3331733, "author_name": "Yehoshua", "author_url": "", "post_date": "2025-11-16T14:06:18.557000", "content": "My local CV score is 0.71, but my LB is only 0.56. I’m hoping to narrow that gap as I continue refining my approach.
", "votes": 0, "replies": [] }, { "id": 3320985, "author_name": "阳光开朗大男孩", "author_url": "", "post_date": "2025-11-13T01:12:33.310000", "content": "CV: 0.67\nLB: 0.62
", "votes": 0, "replies": [] }, { "id": 3319672, "author_name": "Muhammad Awais", "author_url": "", "post_date": "2025-11-12T01:31:46.060000", "content": "ignore Dry_Total_g, GDM_g\nIn the prediction
", "votes": 0, "replies": [] }, { "id": 3319671, "author_name": "Muhammad Awais", "author_url": "", "post_date": "2025-11-12T01:29:29.760000", "content": "Yes its normal, I got 0.55 on LB with efficientnet B3, but my Locally score was around 0.68.
", "votes": 0, "replies": [] }, { "id": 3312332, "author_name": "Aic", "author_url": "", "post_date": "2025-11-07T02:48:50.967000", "content": "cv0.54 lb0.58
", "votes": 0, "replies": [ { "id": 3312932, "author_name": "Aic", "author_url": "", "post_date": "2025-11-08T09:37:26.190000", "content": "I think I’ve finally found a fully reliable CV strategy! After several experiments, the results are very consistent: CV 0.519 → LB 0.52, CV 0.54 → LB 0.54, CV 0.57 → LB 0.57.
", "votes": 1, "replies": [ { "id": 3312935, "author_name": "Yehoshua", "author_url": "", "post_date": "2025-11-08T09:41:56.850000", "content": "Congratulations -:)
", "votes": 0, "replies": [] }, { "id": 3312988, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-11-08T12:00:14.183000", "content": "Great! Did you use a different metric or still the same as competition's?
", "votes": 0, "replies": [] }, { "id": 3313334, "author_name": "Aic", "author_url": "", "post_date": "2025-11-09T05:32:37.677000", "content": "the same as competition's
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3311585": "- I tried different approaches with proper validation locally. My local validation (including KFold CV) consistently shows scores between 0.72 and 0.7499. \n\n- But the best I can do on the public leaderboard is 0.51. Is this normal? \n\n- What is your local CV score?\n\nI used the metric proposed by Kaggle: https://www.kaggle.com/code/metric/r2-score?scriptVersionId=151169785\n\nI suspect something is either wrong with the training data or the metric used on the hidden set is somehow different from the one described on the competition page?", "3311600": "There are some reasons about this huge gap -:)\nFor me, i have try several approaches available to reduce it.\n\nMy best gap so long:\nCV: 0.62, LB: 0.56 (used user takaito metric implemented in his public kernel, before resolved metric issue, for me takaito metric works well and help me align with LB instead of the new approach)\n\nThe 2nd user on LB:\nCV: 0.79, LB: 0.71 (based on the new metric after it's resolved)\n\nOne of reasons is that the test set contains new location so your models must be able to generalize to these unseen locations instead of learning patterns of pastures from locations in the train dataset. I let you discover the remaining causes.\n\nHope, it's helpful.\n\n[UPDATE] https://www.kaggle.com/competitions/csiro-biomass/discussion/615003", "3319796": "my local CV is 0.79, and LB 0.74 but can vary", "3378037": "It seems that the amount of training data is small", "3312716": "straight facts", "3312005": "my latest result:\n\n~0.85 CV 0.60 LB", "3311804": "I tried different split methods, and based on my current results, a random kfold split after grouping by image id seems to work best.", "3311797": "Did you use the metadata for training ? Note the test has no metadata.\n\nWhen I trained with metadata, the local CV is ~0.8 but is not reflective since in test these are set to NULLs", "3312076": "I have some models with CV from 0.71 to 0.79, but their performance on LB can vary from 0.50 to 0.66. It's truly interesting. Still trying to find a solid method to verify performance.", "3377544": "Local CV is 0.69, LB is 0.47. I used random KFold, and I suspect there is data leakage.😭", "3363358": "cv0.72 lb0.69", "3331733": "My local CV score is **0.71**, but my LB is only **0.56**. I’m hoping to narrow that gap as I continue refining my approach.", "3320985": "CV: 0.67\nLB: 0.62", "3319672": "ignore Dry_Total_g, GDM_g\nIn the prediction", "3319671": "Yes its normal, I got 0.55 on LB with efficientnet B3, but my Locally score was around 0.68.", "3312332": "cv0.54 lb0.58" } }