![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5256933%2F224a761faf0c3f0fec564e95d93b969e%2FScreenshot%202025-11-26%20at%2012.37.45AM.png?generation=1764088687508531&alt=media\")
[reference discussion] : https://www.kaggle.com/competitions/csiro-biomass/discussion/615003
\nIn the discussion, the competition host stated that while the train and test sets have the same State and Species, the Sampling_Date is not completely the same.\nTherefore, I conducted the experiment by grouping by Sampling_Date and stratifying by State.
| \n | Model | \nStratified target | \ngroup target | \nCV | \nLB | \n(CV-LB) | \n
|---|---|---|---|---|---|---|
| V1 | \nConvNext Tiny | \nState | \n- | \n0.7189 | \n0.60 | \n0.1189 | \n
| V2 | \nConvNext Tiny | \nState | \nSampling_Date | \n0.5837 | \n0.55 | \n0.0337 | \n
| V3 | \nConvNext Tiny | \nState | \nSampling_Date | \n0.5968 | \n0.57 | \n0.0268 | \n
[reference discussion] : https://www.kaggle.com/competitions/csiro-biomass/discussion/615003
\nIn the discussion, the competition host stated that while the train and test sets have the same State and Species, the Sampling_Date is not completely the same.\nTherefore, I conducted the experiment by grouping by Sampling_Date and stratifying by State.
| \n | Model | \nStratified target | \ngroup target | \nCV | \nLB | \n(CV-LB) | \n
|---|---|---|---|---|---|---|
| V1 | \nConvNext Tiny | \nState | \n- | \n0.7189 | \n0.60 | \n0.1189 | \n
| V2 | \nConvNext Tiny | \nState | \nSampling_Date | \n0.5837 | \n0.55 | \n0.0337 | \n
| V3 | \nConvNext Tiny | \nState | \nSampling_Date | \n0.5968 | \n0.57 | \n0.0268 | \n
unfortunately using this split led to huge overfitting for me too. I changed to a stricter split (involved hardcoding samples into folds š)
\n-> CV 0.64 LB 0.68. CV 0.62 LB 0.65. So far its been consistent
", "rawMarkdown": "unfortunately using this split led to huge overfitting for me too. I changed to a stricter split (involved hardcoding samples into folds š) \n\n-> CV 0.64 LB 0.68. CV 0.62 LB 0.65. So far its been consistent ", "votes": 8, "replies": [ { "id": 3347805, "postDate": "2025-11-25T12:24:50.807Z", "content": "I tried using the splitting strategy mentioned above, but it actually gave me worse results compared to a standard split. It seems that at this stage of the competition, the data-splitting approach plays a major role in achieving stable performance.
\nRight now, Iām using a simple train/validation split and getting around 0.77 CV / 0.63 LB. I also tested K-Fold, but the results were slightly lower and very similar on the LB. So for now, the normal split has been more reliable for me.
\nBy the way, did you notice the CVāLB gap change only because of the splitting strategy, or did you also modify something in the model or training setup?
", "rawMarkdown": "I tried using the splitting strategy mentioned above, but it actually gave me worse results compared to a standard split. It seems that at this stage of the competition, the data-splitting approach plays a major role in achieving stable performance.\n\nRight now, Iām using a simple train/validation split and getting around 0.77 CV / 0.63 LB. I also tested K-Fold, but the results were slightly lower and very similar on the LB. So for now, the normal split has been more reliable for me.\n\nBy the way, did you notice the CVāLB gap change only because of the splitting strategy, or did you also modify something in the model or training setup?", "votes": 1, "replies": [ { "id": 3347993, "postDate": "2025-11-25T14:32:09.830Z", "content": "Changing the data split alone affects the leaderboard score because each model ends up training on a different subset of an already small dataset. In my case, stratifying by state gives me a score of 0.61, but switching to the split mentioned above drops it to 0.56 with the exact same model.
", "rawMarkdown": "Changing the data split alone affects the leaderboard score because each model ends up training on a different subset of an already small dataset. In my case, stratifying by state gives me a score of 0.61, but switching to the split mentioned above drops it to 0.56 with the exact same model.", "votes": 5 }, { "id": 3348098, "postDate": "2025-11-25T15:47:23.443Z", "content": "Hmm my team hasn't observed changing CV split alone can swing scores by that much.
", "rawMarkdown": "Hmm my team hasn't observed changing CV split alone can swing scores by that much.", "votes": 2 }, { "id": 3348170, "postDate": "2025-11-25T16:38:24.610Z", "content": "too much overfitting in the training CV score thoughā¦ š
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10701376%2F8e022fb94ddf0a7005c2672cbb302a50%2F__results___0_1511.png?generation=1764098034722578&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10701376%2F504c8605893d45c110668fb61c8265bc%2FScreenshot%202025-11-25%20201208.png?generation=1764098061188223&alt=media\")
same here !
", "rawMarkdown": "\n\n\nsame here !" }, { "id": 3349287, "postDate": "2025-11-26T15:28:45.813Z", "content": "yup that has been the most worrying factor here , maybe folds should be conservative and with most variability . but some folding strategies do help on Public LB .
", "rawMarkdown": "yup that has been the most worrying factor here , maybe folds should be conservative and with most variability . but some folding strategies do help on Public LB .", "votes": 1 }, { "id": 3352438, "postDate": "2025-11-29T07:37:15.130Z", "content": "Same here, my CV-LB is all over the place. I doubt that the public leaderboard has a greater representation from 1 state and a smaller representation from another state causing this problem. Also, the training data is very small and such datasets usually create such problems @nizarcharrada
", "rawMarkdown": "Same here, my CV-LB is all over the place. I doubt that the public leaderboard has a greater representation from 1 state and a smaller representation from another state causing this problem. Also, the training data is very small and such datasets usually create such problems @nizarcharrada ", "votes": 1 }, { "id": 3352444, "postDate": "2025-11-29T07:40:00.913Z", "content": "This is another worry as the public and private LB may not be in sync causing an ICR-like problem
", "rawMarkdown": "This is another worry as the public and private LB may not be in sync causing an [ICR-like problem](https://www.kaggle.com/competitions/icr-identify-age-related-conditions)" }, { "id": 3352963, "postDate": "2025-11-29T13:53:47.480Z", "content": "Your CV-LB gap is relatable, I am also having 3-4 submissions like this\nCV 0.75 LB 0.59\nCV 0.765 LB 0.60\nCV 0.77 LB 0.62
", "rawMarkdown": "Your CV-LB gap is relatable, I am also having 3-4 submissions like this\nCV 0.75 LB 0.59\nCV 0.765 LB 0.60\nCV 0.77 LB 0.62" } ] }, { "id": 3348510, "postDate": "2025-11-26T00:55:42.427Z", "content": "Just wondering if your CV is calculated using the global metric. I think some people use a metric that is not the same as the competition metric and end up with a CV that is close to their LB, which confused me.
\nAlso, may I ask if your CV is always below 0.70? No matter what split I try, my 5-fold average CV is always above 0.70.
", "rawMarkdown": "Just wondering if your CV is calculated using the global metric. I think some people use a metric that is not the same as the competition metric and end up with a CV that is close to their LB, which confused me.\n\nAlso, may I ask if your CV is always below 0.70? No matter what split I try, my 5-fold average CV is always above 0.70.", "votes": 1 }, { "id": 3352440, "postDate": "2025-11-29T07:38:48.867Z", "content": "This is a good way to handle this problem, I think relying on seeds / scikit-learn for model selection is a sub-optimal process given the data-size and the overall fluctuation. \nMy train-loss curves are quite worrying as I am 100% sure I have a bad overfitting there @yeoyunsianggeremie
", "rawMarkdown": "This is a good way to handle this problem, I think relying on seeds / scikit-learn for model selection is a sub-optimal process given the data-size and the overall fluctuation. \nMy train-loss curves are quite worrying as I am 100% sure I have a bad overfitting there @yeoyunsianggeremie " }, { "id": 3393430, "postDate": "2026-01-19T02:43:58.703Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ] }, { "id": 3320361, "postDate": "2025-11-12T13:24:02.180Z", "content": "Monte Carlo Seed Search Strategy Results
\nState Stratified GroupKFold on Sampling Dates
\n\nEarlier, I tried this seed searching strategy but for GroupKFold on both State and Sampling_date because someone mentioned that there are unseen states in the test set. Later, in some other discussion, the competition host clarified that all states are present in the test set; hence indicating that only the sites/places might be unseen.
\n
Then I moved on to this strategy.
\nCV Fold Results
\n| Fold | \nRĀ² Score | \n
|---|---|
| 0 | \n0.6779 | \n
| 1 | \n0.6387 | \n
| 2 | \n0.6214 | \n
| 3 | \n0.5963 | \n
Mean RĀ²: 0.6336
\nStd RĀ²: 0.0297
Leaderboard Score (LB): 0.63
\nSummary:
\nCV ā LB!
But actually, I used #folds = 4 because that gave me slightly better stratification score (I made a simple metric to calculate the goodness of stratification on State) than the best split I got with %folds = 5. But I think this small difference in the Stratification score was relatively and I should have used %folds = 5 so as to get 5 models instead of 4. This increase in the number of models would have been better.
", "rawMarkdown": "**Monte Carlo Seed Search Strategy Results** \n*State Stratified GroupKFold on Sampling Dates*\n\n> Earlier, I tried this seed searching strategy but for GroupKFold on both State and Sampling_date because someone mentioned that there are unseen states in the test set. Later, in some other discussion, the competition host clarified that all states are present in the test set; hence indicating that only the sites/places might be unseen.\n\nThen I moved on to this strategy.\n\n**CV Fold Results**\n\n| Fold | RĀ² Score |\n|---|---|\n| 0 | 0.6779 |\n| 1 | 0.6387 |\n| 2 | 0.6214 |\n| 3 | 0.5963 |\n\n\n**Mean RĀ²:** 0.6336 \n**Std RĀ²:** 0.0297 \n\n**Leaderboard Score (LB):** 0.63 \n**Summary:** \n*CV ā LB!*\n\nBut actually, I used #folds = 4 because that gave me slightly better stratification score (I made a simple metric to calculate the goodness of stratification on State) than the best split I got with %folds = 5. But I think this small difference in the Stratification score was relatively and I should have used %folds = 5 so as to get 5 models instead of 4. This increase in the number of models would have been better. ", "votes": 8, "replies": [ { "id": 3338173, "postDate": "2025-11-18T22:51:09.427Z", "content": "May I ask whether you used strong augmentation to keep the CV quite low, or if you selected a specific seed? I tried this split method, and my CV easily went above 0.70.
", "rawMarkdown": "May I ask whether you used strong augmentation to keep the CV quite low, or if you selected a specific seed? I tried this split method, and my CV easily went above 0.70.", "replies": [ { "id": 3339254, "postDate": "2025-11-19T01:56:43.840Z", "content": "Hi @lizhecheng. I have indeed done a Seed Search as specified. However, the seed I got is no lucky seed. I have tried using the 2nd best and 3rd best seed (according to my metric) and have got nearly identical results.
\nI am not using any strong augmentations as they doesn't seem to give a good gain/compute ratio. Geometric transforms like Flipping, Rotation are already providing a good amount of additional data to work with, with a pinch of ColorJitter.
\nAlso, My CV and LB are still nearly identical at 0.68.
", "rawMarkdown": "Hi @lizhecheng. I have indeed done a Seed Search as specified. However, the seed I got is no lucky seed. I have tried using the 2nd best and 3rd best seed (according to my metric) and have got nearly identical results.\n\nI am not using any strong augmentations as they doesn't seem to give a good gain/compute ratio. Geometric transforms like Flipping, Rotation are already providing a good amount of additional data to work with, with a pinch of ColorJitter.\n\nAlso, My CV and LB are still nearly identical at 0.68.", "votes": 4 }, { "id": 3339523, "postDate": "2025-11-19T05:32:10.113Z", "content": "Thanks for your detailed reply! Youāve come up with a great splitting method!
", "rawMarkdown": "Thanks for your detailed reply! Youāve come up with a great splitting method!" }, { "id": 3340071, "postDate": "2025-11-19T11:39:31.340Z", "content": "May I ask whether youāre calculating the globally weighted RĀ² (the competition metric) or the weighted mean of the RĀ² values?
", "rawMarkdown": "May I ask whether youāre calculating the globally weighted RĀ² (the competition metric) or the weighted mean of the RĀ² values?" }, { "id": 3340105, "postDate": "2025-11-19T11:55:01.540Z", "content": "I use both.
\nI tried this split and got an interesting result:
\nCV: ~0.63\nLB: 0.73
\nI think public LB is very deceptive.
", "rawMarkdown": "I tried this split and got an interesting result:\n\nCV: ~0.63\nLB: 0.73\n\nI think public LB is very deceptive.", "votes": 4, "replies": [ { "id": 3347531, "postDate": "2025-11-25T08:18:23.080Z", "content": "cv:0.82 , lb:0.66 . It makes me mad.Why it would like this.Is the reason of overfitting ?
", "rawMarkdown": "cv:0.82 , lb:0.66 . It makes me mad.Why it would like this.Is the reason of overfitting ?", "votes": 1, "replies": [ { "id": 3347766, "postDate": "2025-11-25T11:53:16.913Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3348105, "postDate": "2025-11-25T15:55:10.700Z", "content": "At least for me, CV can't build a solid method to eval model's performance. For this model and this split method, as CV gets higher, its score goes down continuously on LB.
", "rawMarkdown": "At least for me, CV can't build a solid method to eval model's performance. For this model and this split method, as CV gets higher, its score goes down continuously on LB.", "votes": 2 }, { "id": 3352637, "postDate": "2025-11-29T10:20:07.047Z", "content": "I have this feeling .I tried a lot of methods to find a sold eval.but it doesn't work.I decide to be based on the public test
", "rawMarkdown": "I have this feeling .I tried a lot of methods to find a sold eval.but it doesn't work.I decide to be based on the public test " }, { "id": 3352958, "postDate": "2025-11-29T13:50:58.887Z", "content": "Same for me, increased CV == reduced LB beyond a point @linkinpony
", "rawMarkdown": "Same for me, increased CV == reduced LB beyond a point @linkinpony ", "votes": 3 }, { "id": 3387488, "postDate": "2026-01-07T05:55:06.400Z", "content": "\n\nAt least for me, CV can't build a solid method to eval model's performance. For this model and this split method, as CV gets higher, its score goes down continuously on LB.
\n
Same š
", "rawMarkdown": "> At least for me, CV can't build a solid method to eval model's performance. For this model and this split method, as CV gets higher, its score goes down continuously on LB.\n\nSame š" }, { "id": 3387492, "postDate": "2026-01-07T06:07:00.397Z", "content": "cv 0.67 lb 0.66
\ncv 0.69 lb 0.67
\ncv 0.73 lb 0.69
\ncv 0.75 lb 0.70
\ncv 0.76 lb 0.72
", "rawMarkdown": "cv 0.67 lb 0.66\n\ncv 0.69 lb 0.67\n\ncv 0.73 lb 0.69\n\ncv 0.75 lb 0.70\n\ncv 0.76 lb 0.72", "votes": 2 } ] } ] }, { "id": 3315438, "postDate": "2025-11-10T15:52:30.980Z", "content": "I am seeing much better correlation with StratifiedKFold by State and Sampling_Date (Month) too - but not better than the split shared here. Thanks š
StratifiedKFold? @yeoyunsianggeremie
", "rawMarkdown": "StratifiedKFold? @yeoyunsianggeremie ", "votes": -3 } ] }, { "id": 3315400, "postDate": "2025-11-10T15:32:38.583Z", "content": "Thanks for the insight!
\nI didn't do stratify. I only did group by sampling_date
CV 0.6369 - LB 0.61
\nCV 0.6932 - LB 0.59
I'll try stratify by state now. Will share the update here. I hope we will finally see some correlation
I recommend setting State to Stratified target because the distribution of target is different for each State.
(meanĀ±std)
\n| State | \nDry_Clover_g | \nDry_Dead_g | \nDry_Green_g | \nDry_Total_g | \nGDM_g | \n
|---|---|---|---|---|---|
| Tas | \n6.24 Ā± 11.61 | \n15.23 Ā± 12.43 | \n15.33 Ā± 13.11 | \n36.80 Ā± 20.70 | \n21.57 Ā± 14.36 | \n
| NSW | \n0.13 Ā± 1.17 | \n14.20 Ā± 15.63 | \n56.56 Ā± 31.23 | \n70.90 Ā± 35.58 | \n56.69 Ā± 31.26 | \n
| WA | \n22.09 Ā± 20.21 | \n0.00 Ā± 0.00 | \n9.30 Ā± 18.79 | \n31.39 Ā± 19.62 | \n31.39 Ā± 19.62 | \n
| Vic | \n7.11 Ā± 9.57 | \n10.11 Ā± 8.86 | \n25.45 Ā± 16.02 | \n42.67 Ā± 20.89 | \n32.56 Ā± 20.93 | \n
The best post in this competitions for now.š
", "rawMarkdown": "The best post in this competitions for now.š", "votes": -2 }, { "id": 3320050, "postDate": "2025-11-12T07:44:58.727Z", "content": "The best in the competition
", "rawMarkdown": "The best in the competition ", "votes": -3 }, { "id": 3395391, "postDate": "2026-01-22T22:10:46.783Z", "content": "use seed 29570
", "rawMarkdown": "use seed 29570", "replies": [ { "id": 3395415, "postDate": "2026-01-22T23:43:51.597Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3318255, "postDate": "2025-11-11T04:51:06.810Z", "content": "Appreciate yours insight ! Can you share yours metric code you use ? I also use ConvnextTiny.
\nMy gap between CV and LB is very high:
\nCV. 0.73334 - LB: 0.59
", "rawMarkdown": "Appreciate yours insight ! Can you share yours metric code you use ? I also use ConvnextTiny.\n\nMy gap between CV and LB is very high: \n\nCV. 0.73334 - LB: 0.59 ", "replies": [ { "id": 3318460, "postDate": "2025-11-11T07:17:32.613Z", "content": "here!
\n ():\n \n pred_total = all_preds_3[]\n pred_gdm = all_preds_3[]\n pred_green = all_preds_3[]\n\n \n pred_clover = np.maximum(, pred_gdm - pred_green)\n pred_dead = np.maximum(, pred_total - pred_gdm)\n\n \n y_pred = np.stack([\n pred_green,\n pred_dead,\n pred_clover,\n pred_gdm,\n pred_total\n ], axis=) \n\n y_true = all_targets_5 \n\n weights = np.array([, , , , ])\n r2_scores = []\n\n i ():\n y_t = y_true[:, i]\n y_p = y_pred[:, i]\n\n ss_res = np.((y_t - y_p) ** )\n ss_tot = np.((y_t - np.mean(y_t)) ** )\n r2 = - ss_res / ss_tot ss_tot > \n\n r2_scores.append(r2)\n\n r2_scores = np.array(r2_scores)\n weighted_r2 = np.(r2_scores * weights) / np.(weights)\n\n weighted_r2\nyour metric calculates RĀ² separately for each of the 5 targets then averages them, so its different from the competition metric.\nThe competition actual metric calculates a single global RĀ² across all rows at once, with per-row weights.
\nInterestingly, some people find the per-target approach gives better CV-LB correlation anyway, so it might still be useful for model selection! But just wanted to flag the difference.
", "rawMarkdown": "your metric calculates RĀ² separately for each of the 5 targets then averages them, so its different from the competition metric.\nThe competition actual metric calculates a single global RĀ² across all rows at once, with per-row weights.\n\nInterestingly, some people find the per-target approach gives better CV-LB correlation anyway, so it might still be useful for model selection! But just wanted to flag the difference.", "votes": 10 } ] } ] }, { "id": 3316903, "postDate": "2025-11-10T20:55:25.967Z", "content": "Thanks for sharing these valuable insights! I just wonder if you mean StratifiedGroupKFold. I implement it based on your ideas using 5 folds and find that, in some folds, the ratio of train data to validation data is much larger than 5. I understand this makes sense because of many constraints in the split, just to confirm if my implementation is correct.
I also encountered the same issue with StratifiedGroupKFold. So I changed the random seed until I obtained an appropriate ratio.
Let the guessing game begin!
", "rawMarkdown": "Let the guessing game begin!", "votes": -2 }, { "id": 3316936, "postDate": "2025-11-10T21:39:07.920Z", "content": "Thanks for your answer! Just wondering, do different seeds make a big difference between CV and LB?
", "rawMarkdown": "Thanks for your answer! Just wondering, do different seeds make a big difference between CV and LB?" }, { "id": 3317960, "postDate": "2025-11-10T22:43:18.713Z", "content": "I'm not really sure about thatā¦
", "rawMarkdown": "I'm not really sure about that..." } ] } ] }, { "id": 3315747, "postDate": "2025-11-10T18:43:40.400Z", "content": "I try it but still get a huge gap between LB and CV?
", "rawMarkdown": "I try it but still get a huge gap between LB and CV?", "replies": [ { "id": 3315752, "postDate": "2025-11-10T18:45:05.790Z", "content": "This is not a silver bullet on its own but one of several required techniques that are needed to try and ensure that your model is being \"fair\" to itself during training. If your model overfits or cannot generalize for any other reason a change to fold is unlikely to fix everything.
", "rawMarkdown": "This is not a silver bullet on its own but one of several required techniques that are needed to try and ensure that your model is being \"fair\" to itself during training. If your model overfits or cannot generalize for any other reason a change to fold is unlikely to fix everything.", "votes": 2 } ] }, { "id": 3386544, "postDate": "2026-01-05T13:36:52.733Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3320392, "postDate": "2025-11-12T13:54:28.173Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3315615, "postDate": "2025-11-10T17:30:38.700Z", "content": "Interesting, thanks for the share!
", "rawMarkdown": "Interesting, thanks for the share!", "votes": 1 }, { "id": 3357414, "postDate": "2025-12-01T08:50:36.720Z", "content": "Thank you for sharing. I understand.
", "rawMarkdown": "Thank you for sharing. I understand." }, { "id": 3315511, "postDate": "2025-11-10T16:36:18.250Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!" } ], "comments": [ { "id": 3347793, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-25T12:13:52.683000", "content": "unfortunately using this split led to huge overfitting for me too. I changed to a stricter split (involved hardcoding samples into folds š)
\n-> CV 0.64 LB 0.68. CV 0.62 LB 0.65. So far its been consistent
", "votes": 8, "replies": [ { "id": 3347805, "author_name": "ZAID", "author_url": "", "post_date": "2025-11-25T12:24:50.807000", "content": "I tried using the splitting strategy mentioned above, but it actually gave me worse results compared to a standard split. It seems that at this stage of the competition, the data-splitting approach plays a major role in achieving stable performance.
\nRight now, Iām using a simple train/validation split and getting around 0.77 CV / 0.63 LB. I also tested K-Fold, but the results were slightly lower and very similar on the LB. So for now, the normal split has been more reliable for me.
\nBy the way, did you notice the CVāLB gap change only because of the splitting strategy, or did you also modify something in the model or training setup?
", "votes": 1, "replies": [ { "id": 3347993, "author_name": "Nizar Charrada", "author_url": "", "post_date": "2025-11-25T14:32:09.830000", "content": "Changing the data split alone affects the leaderboard score because each model ends up training on a different subset of an already small dataset. In my case, stratifying by state gives me a score of 0.61, but switching to the split mentioned above drops it to 0.56 with the exact same model.
", "votes": 5, "replies": [] }, { "id": 3348098, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-25T15:47:23.443000", "content": "Hmm my team hasn't observed changing CV split alone can swing scores by that much.
", "votes": 2, "replies": [] }, { "id": 3348170, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-25T16:38:24.610000", "content": "too much overfitting in the training CV score thoughā¦ š
\n
\n
same here !
", "votes": 0, "replies": [] }, { "id": 3349287, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2025-11-26T15:28:45.813000", "content": "yup that has been the most worrying factor here , maybe folds should be conservative and with most variability . but some folding strategies do help on Public LB .
", "votes": 1, "replies": [] }, { "id": 3352438, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-11-29T07:37:15.130000", "content": "Same here, my CV-LB is all over the place. I doubt that the public leaderboard has a greater representation from 1 state and a smaller representation from another state causing this problem. Also, the training data is very small and such datasets usually create such problems @nizarcharrada
", "votes": 1, "replies": [] }, { "id": 3352444, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-11-29T07:40:00.913000", "content": "This is another worry as the public and private LB may not be in sync causing an ICR-like problem
", "votes": 0, "replies": [] }, { "id": 3352963, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-11-29T13:53:47.480000", "content": "Your CV-LB gap is relatable, I am also having 3-4 submissions like this\nCV 0.75 LB 0.59\nCV 0.765 LB 0.60\nCV 0.77 LB 0.62
", "votes": 0, "replies": [] } ] }, { "id": 3348510, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2025-11-26T00:55:42.427000", "content": "Just wondering if your CV is calculated using the global metric. I think some people use a metric that is not the same as the competition metric and end up with a CV that is close to their LB, which confused me.
\nAlso, may I ask if your CV is always below 0.70? No matter what split I try, my 5-fold average CV is always above 0.70.
", "votes": 1, "replies": [] }, { "id": 3352440, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-11-29T07:38:48.867000", "content": "This is a good way to handle this problem, I think relying on seeds / scikit-learn for model selection is a sub-optimal process given the data-size and the overall fluctuation. \nMy train-loss curves are quite worrying as I am 100% sure I have a bad overfitting there @yeoyunsianggeremie
", "votes": 0, "replies": [] }, { "id": 3393430, "author_name": "", "author_url": "", "post_date": "2026-01-19T02:43:58.703000", "content": "", "votes": -1, "replies": [] } ] }, { "id": 3320361, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-11-12T13:24:02.180000", "content": "Monte Carlo Seed Search Strategy Results
\nState Stratified GroupKFold on Sampling Dates
\n\nEarlier, I tried this seed searching strategy but for GroupKFold on both State and Sampling_date because someone mentioned that there are unseen states in the test set. Later, in some other discussion, the competition host clarified that all states are present in the test set; hence indicating that only the sites/places might be unseen.
\n
Then I moved on to this strategy.
\nCV Fold Results
\n| Fold | \nRĀ² Score | \n
|---|---|
| 0 | \n0.6779 | \n
| 1 | \n0.6387 | \n
| 2 | \n0.6214 | \n
| 3 | \n0.5963 | \n
Mean RĀ²: 0.6336
\nStd RĀ²: 0.0297
Leaderboard Score (LB): 0.63
\nSummary:
\nCV ā LB!
But actually, I used #folds = 4 because that gave me slightly better stratification score (I made a simple metric to calculate the goodness of stratification on State) than the best split I got with %folds = 5. But I think this small difference in the Stratification score was relatively and I should have used %folds = 5 so as to get 5 models instead of 4. This increase in the number of models would have been better.
", "votes": 8, "replies": [ { "id": 3338173, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2025-11-18T22:51:09.427000", "content": "May I ask whether you used strong augmentation to keep the CV quite low, or if you selected a specific seed? I tried this split method, and my CV easily went above 0.70.
", "votes": 0, "replies": [ { "id": 3339254, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-11-19T01:56:43.840000", "content": "Hi @lizhecheng. I have indeed done a Seed Search as specified. However, the seed I got is no lucky seed. I have tried using the 2nd best and 3rd best seed (according to my metric) and have got nearly identical results.
\nI am not using any strong augmentations as they doesn't seem to give a good gain/compute ratio. Geometric transforms like Flipping, Rotation are already providing a good amount of additional data to work with, with a pinch of ColorJitter.
\nAlso, My CV and LB are still nearly identical at 0.68.
", "votes": 4, "replies": [] }, { "id": 3339523, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2025-11-19T05:32:10.113000", "content": "Thanks for your detailed reply! Youāve come up with a great splitting method!
", "votes": 0, "replies": [] }, { "id": 3340071, "author_name": "Nizar Charrada", "author_url": "", "post_date": "2025-11-19T11:39:31.340000", "content": "May I ask whether youāre calculating the globally weighted RĀ² (the competition metric) or the weighted mean of the RĀ² values?
", "votes": 0, "replies": [] }, { "id": 3340105, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-11-19T11:55:01.540000", "content": "I use both.
\nI tried this split and got an interesting result:
\nCV: ~0.63\nLB: 0.73
\nI think public LB is very deceptive.
", "votes": 4, "replies": [ { "id": 3347531, "author_name": "", "author_url": "", "post_date": "2025-11-25T08:18:23.080000", "content": "cv:0.82 , lb:0.66 . It makes me mad.Why it would like this.Is the reason of overfitting ?
", "votes": 1, "replies": [ { "id": 3347766, "author_name": "", "author_url": "", "post_date": "2025-11-25T11:53:16.913000", "content": "", "votes": 0, "replies": [] }, { "id": 3348105, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-11-25T15:55:10.700000", "content": "At least for me, CV can't build a solid method to eval model's performance. For this model and this split method, as CV gets higher, its score goes down continuously on LB.
", "votes": 2, "replies": [] }, { "id": 3352637, "author_name": "", "author_url": "", "post_date": "2025-11-29T10:20:07.047000", "content": "I have this feeling .I tried a lot of methods to find a sold eval.but it doesn't work.I decide to be based on the public test
", "votes": 0, "replies": [] }, { "id": 3352958, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-11-29T13:50:58.887000", "content": "Same for me, increased CV == reduced LB beyond a point @linkinpony
", "votes": 3, "replies": [] }, { "id": 3387488, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2026-01-07T05:55:06.400000", "content": "\n\nAt least for me, CV can't build a solid method to eval model's performance. For this model and this split method, as CV gets higher, its score goes down continuously on LB.
\n
Same š
", "votes": 0, "replies": [] }, { "id": 3387492, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2026-01-07T06:07:00.397000", "content": "cv 0.67 lb 0.66
\ncv 0.69 lb 0.67
\ncv 0.73 lb 0.69
\ncv 0.75 lb 0.70
\ncv 0.76 lb 0.72
", "votes": 2, "replies": [] } ] } ] }, { "id": 3315438, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2025-11-10T15:52:30.980000", "content": "I am seeing much better correlation with StratifiedKFold by State and Sampling_Date (Month) too - but not better than the split shared here. Thanks š
StratifiedKFold? @yeoyunsianggeremie
", "votes": -3, "replies": [] } ] }, { "id": 3315400, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2025-11-10T15:32:38.583000", "content": "Thanks for the insight!
\nI didn't do stratify. I only did group by sampling_date
CV 0.6369 - LB 0.61
\nCV 0.6932 - LB 0.59
I'll try stratify by state now. Will share the update here. I hope we will finally see some correlation
I recommend setting State to Stratified target because the distribution of target is different for each State.
(meanĀ±std)
\n| State | \nDry_Clover_g | \nDry_Dead_g | \nDry_Green_g | \nDry_Total_g | \nGDM_g | \n
|---|---|---|---|---|---|
| Tas | \n6.24 Ā± 11.61 | \n15.23 Ā± 12.43 | \n15.33 Ā± 13.11 | \n36.80 Ā± 20.70 | \n21.57 Ā± 14.36 | \n
| NSW | \n0.13 Ā± 1.17 | \n14.20 Ā± 15.63 | \n56.56 Ā± 31.23 | \n70.90 Ā± 35.58 | \n56.69 Ā± 31.26 | \n
| WA | \n22.09 Ā± 20.21 | \n0.00 Ā± 0.00 | \n9.30 Ā± 18.79 | \n31.39 Ā± 19.62 | \n31.39 Ā± 19.62 | \n
| Vic | \n7.11 Ā± 9.57 | \n10.11 Ā± 8.86 | \n25.45 Ā± 16.02 | \n42.67 Ā± 20.89 | \n32.56 Ā± 20.93 | \n
The best post in this competitions for now.š
", "votes": -2, "replies": [] }, { "id": 3320050, "author_name": "Shanks_777", "author_url": "", "post_date": "2025-11-12T07:44:58.727000", "content": "The best in the competition
", "votes": -3, "replies": [] }, { "id": 3395391, "author_name": "Ali Ozan Memetoglu", "author_url": "", "post_date": "2026-01-22T22:10:46.783000", "content": "use seed 29570
", "votes": 0, "replies": [ { "id": 3395415, "author_name": "", "author_url": "", "post_date": "2026-01-22T23:43:51.597000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3318255, "author_name": "Long Nguyį» n HoĆ ng", "author_url": "", "post_date": "2025-11-11T04:51:06.810000", "content": "Appreciate yours insight ! Can you share yours metric code you use ? I also use ConvnextTiny.
\nMy gap between CV and LB is very high:
\nCV. 0.73334 - LB: 0.59
", "votes": 0, "replies": [ { "id": 3318460, "author_name": "EISLab_hwlee", "author_url": "", "post_date": "2025-11-11T07:17:32.613000", "content": "here!
\n ():\n \n pred_total = all_preds_3[]\n pred_gdm = all_preds_3[]\n pred_green = all_preds_3[]\n\n \n pred_clover = np.maximum(, pred_gdm - pred_green)\n pred_dead = np.maximum(, pred_total - pred_gdm)\n\n \n y_pred = np.stack([\n pred_green,\n pred_dead,\n pred_clover,\n pred_gdm,\n pred_total\n ], axis=) \n\n y_true = all_targets_5 \n\n weights = np.array([, , , , ])\n r2_scores = []\n\n i ():\n y_t = y_true[:, i]\n y_p = y_pred[:, i]\n\n ss_res = np.((y_t - y_p) ** )\n ss_tot = np.((y_t - np.mean(y_t)) ** )\n r2 = - ss_res / ss_tot ss_tot > \n\n r2_scores.append(r2)\n\n r2_scores = np.array(r2_scores)\n weighted_r2 = np.(r2_scores * weights) / np.(weights)\n\n weighted_r2\nyour metric calculates RĀ² separately for each of the 5 targets then averages them, so its different from the competition metric.\nThe competition actual metric calculates a single global RĀ² across all rows at once, with per-row weights.
\nInterestingly, some people find the per-target approach gives better CV-LB correlation anyway, so it might still be useful for model selection! But just wanted to flag the difference.
", "votes": 10, "replies": [] } ] } ] }, { "id": 3316903, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2025-11-10T20:55:25.967000", "content": "Thanks for sharing these valuable insights! I just wonder if you mean StratifiedGroupKFold. I implement it based on your ideas using 5 folds and find that, in some folds, the ratio of train data to validation data is much larger than 5. I understand this makes sense because of many constraints in the split, just to confirm if my implementation is correct.
I also encountered the same issue with StratifiedGroupKFold. So I changed the random seed until I obtained an appropriate ratio.
Let the guessing game begin!
", "votes": -2, "replies": [] }, { "id": 3316936, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2025-11-10T21:39:07.920000", "content": "Thanks for your answer! Just wondering, do different seeds make a big difference between CV and LB?
", "votes": 0, "replies": [] }, { "id": 3317960, "author_name": "EISLab_hwlee", "author_url": "", "post_date": "2025-11-10T22:43:18.713000", "content": "I'm not really sure about thatā¦
", "votes": 0, "replies": [] } ] } ] }, { "id": 3315747, "author_name": "Forrest_xlz", "author_url": "", "post_date": "2025-11-10T18:43:40.400000", "content": "I try it but still get a huge gap between LB and CV?
", "votes": 0, "replies": [ { "id": 3315752, "author_name": "Tim Krige", "author_url": "", "post_date": "2025-11-10T18:45:05.790000", "content": "This is not a silver bullet on its own but one of several required techniques that are needed to try and ensure that your model is being \"fair\" to itself during training. If your model overfits or cannot generalize for any other reason a change to fold is unlikely to fix everything.
", "votes": 2, "replies": [] } ] }, { "id": 3386544, "author_name": "", "author_url": "", "post_date": "2026-01-05T13:36:52.733000", "content": "", "votes": 0, "replies": [] }, { "id": 3320392, "author_name": "", "author_url": "", "post_date": "2025-11-12T13:54:28.173000", "content": "", "votes": 0, "replies": [] }, { "id": 3315615, "author_name": "evnsnclr", "author_url": "", "post_date": "2025-11-10T17:30:38.700000", "content": "Interesting, thanks for the share!
", "votes": 1, "replies": [] }, { "id": 3357414, "author_name": "Kango", "author_url": "", "post_date": "2025-12-01T08:50:36.720000", "content": "Thank you for sharing. I understand.
", "votes": 0, "replies": [] }, { "id": 3315511, "author_name": "Forrest_xlz", "author_url": "", "post_date": "2025-11-10T16:36:18.250000", "content": "Thank you very much!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3314932": "[reference discussion] : https://www.kaggle.com/competitions/csiro-biomass/discussion/615003\n\nIn the discussion, the competition host stated that while the train and test sets have the same `State` and `Species`, the `Sampling_Date` is not completely the same.\nTherefore, I conducted the experiment by grouping by `Sampling_Date` and stratifying by `State`.\n\n| | Model | Stratified target | group target | CV | LB | (CV-LB) |\n| --- | --- | --- | --- | --- | --- | --- |\n| V1 | ConvNext Tiny | `State` | - | 0.7189 | 0.60 | 0.1189 |\n| V2 | ConvNext Tiny | `State` | `Sampling_Date` | 0.5837 | 0.55 | 0.0337 |\n| V3 | ConvNext Tiny | `State` | `Sampling_Date` | 0.5968 | 0.57 | 0.0268 |\n", "3347793": "unfortunately using this split led to huge overfitting for me too. I changed to a stricter split (involved hardcoding samples into folds š) \n\n-> CV 0.64 LB 0.68. CV 0.62 LB 0.65. So far its been consistent ", "3320361": "**Monte Carlo Seed Search Strategy Results** \n*State Stratified GroupKFold on Sampling Dates*\n\n> Earlier, I tried this seed searching strategy but for GroupKFold on both State and Sampling_date because someone mentioned that there are unseen states in the test set. Later, in some other discussion, the competition host clarified that all states are present in the test set; hence indicating that only the sites/places might be unseen.\n\nThen I moved on to this strategy.\n\n**CV Fold Results**\n\n| Fold | RĀ² Score |\n|---|---|\n| 0 | 0.6779 |\n| 1 | 0.6387 |\n| 2 | 0.6214 |\n| 3 | 0.5963 |\n\n\n**Mean RĀ²:** 0.6336 \n**Std RĀ²:** 0.0297 \n\n**Leaderboard Score (LB):** 0.63 \n**Summary:** \n*CV ā LB!*\n\nBut actually, I used #folds = 4 because that gave me slightly better stratification score (I made a simple metric to calculate the goodness of stratification on State) than the best split I got with %folds = 5. But I think this small difference in the Stratification score was relatively and I should have used %folds = 5 so as to get 5 models instead of 4. This increase in the number of models would have been better. ", "3347157": "I tried this split and got an interesting result:\n\nCV: ~0.63\nLB: 0.73\n\nI think public LB is very deceptive.", "3315438": "I am seeing much better correlation with StratifiedKFold by `State and Sampling_Date (Month)` too - but not better than the split shared here. Thanks š", "3315400": "Thanks for the insight! \n\nI didn't do stratify. I only did group by `sampling_date`\n\nCV 0.6369 - LB 0.61 \nCV 0.6932 - LB 0.59 \n\nI'll try stratify by `state` now. Will share the update here. I hope we will finally see some correlation\n\n", "3319023": "The best post in this competitions for now.š", "3320050": "The best in the competition ", "3395391": "use seed 29570", "3318255": "Appreciate yours insight ! Can you share yours metric code you use ? I also use ConvnextTiny.\n\nMy gap between CV and LB is very high: \n\nCV. 0.73334 - LB: 0.59 ", "3316903": "Thanks for sharing these valuable insights! I just wonder if you mean ``StratifiedGroupKFold``. I implement it based on your ideas using 5 folds and find that, in some folds, the ratio of train data to validation data is much larger than 5. I understand this makes sense because of many constraints in the split, just to confirm if my implementation is correct.", "3315747": "I try it but still get a huge gap between LB and CV?", "3386544": "", "3320392": "", "3315615": "Interesting, thanks for the share!", "3357414": "Thank you for sharing. I understand.", "3315511": "Thank you very much!" } }