Hi Everyone,
\nWhat a fun competition this has been so far. The small dataset and seasonal distribution shifts makes fitting tricky, but experimenting easy (in my opinion at least).
\nFirstly, I'm genuinely surprised how well a linear model fit on top of some pre-trained NN embeddings (either using partial least squares or principal component regression) performs. I was able to get ~0.52 using this approach, as have others.
\nI haven't gone down this rabbit hole too far, but it also looks like ConvNet -> PCA -> SVR works pretty well! Maybe I should explore things like this more, as it's currently beating my approach :).
\nWith simple NN Models I'm able to get up to about 0.6 with a CV ensemble of a single model, but honestly I'm a little stuck at the moment. I've learned a lot about these models and with the right CV splitting I can get pretty consistent LB results, as have others: Useful CV Thread. It's a little underwhelming to only beat the linear model by a few R squared points though 😄.
\nAll that said, it really feels like I'm missing something obvious 😅. Fine-tuning hasn't been profitable so far; it keeps over-fitting even with warm ups and low learning rates. There's probably a bit more on the table if I do some targeted ensembling, but I feel like there's a lot of missing signal I'm just not seeing that's right in front of me. Finding a way to incorporate the metadata into this problem seems enticing. You can build some pretty decent models just using the metadata, but I haven't really figured out how to translate something we have in the train set, but not the test set, as a feature. Distillation and multitasking come to mind (ChatGPT has been helpful learning about some of these things) and seem to be a pretty common topic of discussion, though I'm not super confident in implementing these yet.
\nAnyone else feel this way? I'm actively looking for that \"aha\" moment and enjoying the competition, I just can't shake the feeling that the thing I'm missing is right in front of me.
\nUPDATE: Nothing I was missing to go from 0.6 -> 0.69 was \"obvious\", but it was all simple. This competition really rewards simple. Getting > 0.7 with a single model is proving quite difficult still. I'm still wondering what I'm missing…Still a month to go!
", "messages": [ { "id": "3373516", "postDate": "12/12/2025 13:34:43", "content": "Hi Everyone,
\nWhat a fun competition this has been so far. The small dataset and seasonal distribution shifts makes fitting tricky, but experimenting easy (in my opinion at least).
\nFirstly, I'm genuinely surprised how well a linear model fit on top of some pre-trained NN embeddings (either using partial least squares or principal component regression) performs. I was able to get ~0.52 using this approach, as have others.
\nI haven't gone down this rabbit hole too far, but it also looks like ConvNet -> PCA -> SVR works pretty well! Maybe I should explore things like this more, as it's currently beating my approach :).
\nWith simple NN Models I'm able to get up to about 0.6 with a CV ensemble of a single model, but honestly I'm a little stuck at the moment. I've learned a lot about these models and with the right CV splitting I can get pretty consistent LB results, as have others: Useful CV Thread. It's a little underwhelming to only beat the linear model by a few R squared points though 😄.
\nAll that said, it really feels like I'm missing something obvious 😅. Fine-tuning hasn't been profitable so far; it keeps over-fitting even with warm ups and low learning rates. There's probably a bit more on the table if I do some targeted ensembling, but I feel like there's a lot of missing signal I'm just not seeing that's right in front of me. Finding a way to incorporate the metadata into this problem seems enticing. You can build some pretty decent models just using the metadata, but I haven't really figured out how to translate something we have in the train set, but not the test set, as a feature. Distillation and multitasking come to mind (ChatGPT has been helpful learning about some of these things) and seem to be a pretty common topic of discussion, though I'm not super confident in implementing these yet.
\nAnyone else feel this way? I'm actively looking for that \"aha\" moment and enjoying the competition, I just can't shake the feeling that the thing I'm missing is right in front of me.
\nUPDATE: Nothing I was missing to go from 0.6 -> 0.69 was \"obvious\", but it was all simple. This competition really rewards simple. Getting > 0.7 with a single model is proving quite difficult still. I'm still wondering what I'm missing…Still a month to go!
", "rawMarkdown": "Hi Everyone,\n\nWhat a fun competition this has been so far. The small dataset and seasonal distribution shifts makes fitting tricky, but experimenting easy (in my opinion at least).\n\nFirstly, I'm genuinely surprised how well a linear model fit on top of some pre-trained NN embeddings (either using partial least squares or principal component regression) performs. I was able to get ~0.52 using this approach, as have others.\n\nI haven't gone down this rabbit hole too far, but it also looks like [ConvNet -> PCA -> SVR](https://www.kaggle.com/code/kospintr/csiro-convnextlarge-pca-svr) works pretty well! Maybe I should explore things like this more, as it's currently beating my approach :).\n\nWith simple NN Models I'm able to get up to about 0.6 with a CV ensemble of a single model, but honestly I'm a little stuck at the moment. I've learned a lot about these models and with the right CV splitting I can get pretty consistent LB results, as have others: [Useful CV Thread](https://www.kaggle.com/competitions/csiro-biomass/discussion/614042). It's a little underwhelming to only beat the linear model by a few R squared points though 😄.\n\nAll that said, it really feels like I'm missing something obvious 😅. Fine-tuning hasn't been profitable so far; it keeps over-fitting even with warm ups and low learning rates. There's probably a bit more on the table if I do some targeted ensembling, but I feel like there's a lot of missing signal I'm just not seeing that's right in front of me. Finding a way to incorporate the metadata into this problem seems enticing. You can build some pretty decent models just using the metadata, but I haven't really figured out how to translate something we have in the train set, but not the test set, as a feature. Distillation and multitasking come to mind (ChatGPT has been helpful learning about some of these things) and seem to be a pretty common topic of discussion, though I'm not super confident in implementing these yet.\n\nAnyone else feel this way? I'm actively looking for that \"aha\" moment and enjoying the competition, I just can't shake the feeling that the thing I'm missing is right in front of me.\n\nUPDATE: Nothing I was missing to go from 0.6 -> 0.69 was \"obvious\", but it was all simple. This competition really rewards simple. Getting > 0.7 with a single model is proving quite difficult still. I'm still wondering what I'm missing...Still a month to go!", "votes": null }, { "id": "3373642", "postDate": "12/12/2025 14:14:12", "content": "I had a lb score of 0.67 at the beginning of November. I then spend about 40 submissions going all over the place, from 0.62 to -11.5. There are fascinating traps all over the place and I really think this is a great competition as it forces you to really be careful about overfitting, feature engineering, and augmentations.
\nThere are discussions about fold strategies. I think these are key. Sometimes you may have a bad (or simple) fold strategy and a good (\"lucky\") seed (@\"Genius Seed is all you need\", I am looking at you…)
\n[Edit: I do not mean to say that a bad fold strategy can be fixed by a great seed - I mean a great seed may be all you need with a basic folding strategy. ]
Be careful with data preprocessing and cleaning. There are threads on this already but the data is messy. Some of this mess is generalization, some of this mess is data leak routes.
Metadata is an interesting point - it is critical information, but also hard to use.
What are your current strategies on these? Have you tried to changes these strategies?
", "rawMarkdown": "I had a lb score of 0.67 at the beginning of November. I then spend about 40 submissions going all over the place, from 0.62 to -11.5. There are fascinating traps all over the place and I really think this is a great competition as it forces you to really be careful about overfitting, feature engineering, and augmentations. \n\n1. There are discussions about fold strategies. I think these are key. Sometimes you may have a bad (or simple) fold strategy and a good (\"lucky\") seed (@\"Genius Seed is all you need\", I am looking at you...) \n[Edit: I do not mean to say that a bad fold strategy can be fixed by a great seed - I mean a great seed may be all you need with a basic folding strategy. ]\n\n2. Be careful with data preprocessing and cleaning. There are threads on this already but the data is messy. Some of this mess is generalization, some of this mess is data leak routes.\n3. Metadata is an interesting point - it is critical information, but also hard to use.\n\nWhat are your current strategies on these? Have you tried to changes these strategies?", "votes": null }, { "id": "3373801", "postDate": "12/12/2025 15:05:02", "content": "Sorry to ask, but could you clarify what you mean by 'bad fold strategy'? Do you mean it's bad for the public leaderboard or just for certain seeds?
", "rawMarkdown": "Sorry to ask, but could you clarify what you mean by 'bad fold strategy'? Do you mean it's bad for the public leaderboard or just for certain seeds?", "votes": null }, { "id": "3373849", "postDate": "12/12/2025 15:20:08", "content": "Decent question: a bad fold strategy is one that is not fair. If you have images that you train on that are too similar to the ones in the validation, this is bad. So, trying to make sure that you have a fold strategy that fairly keeps information away from the training set is key. People have suggested folding based on state, or by date, or by random seed. All will run, but it is hard to develop a robust method of folding whereby your local CV score is indicative of the LB score. People who have found this will keep it a secret most likely.
\nThings to look for: do your folds have comparable val scores? do these scores tend to go in the same direction as the LB scores? (i.e., is a local improvement also a LB improvement?) and finally: are your local scores realistic?
\nEdit: I reread my original post and added some notes to make it clearer
", "rawMarkdown": "Decent question: a bad fold strategy is one that is not fair. If you have images that you train on that are too similar to the ones in the validation, this is bad. So, trying to make sure that you have a fold strategy that fairly keeps information away from the training set is key. People have suggested folding based on state, or by date, or by random seed. All will run, but it is hard to develop a robust method of folding whereby your local CV score is indicative of the LB score. People who have found this will keep it a secret most likely. \n\nThings to look for: do your folds have comparable val scores? do these scores tend to go in the same direction as the LB scores? (i.e., is a local improvement also a LB improvement?) and finally: are your local scores realistic?\n\nEdit: I reread my original post and added some notes to make it clearer", "votes": null }, { "id": "3374406", "postDate": "12/12/2025 17:51:48", "content": "Hey @timkrige thanks for your reply! I'm glad you're enjoying the competition too.
\nThe fold strategy is something I've been looking pretty heavily into. Early on I had a model I was pretty excited about using stratified or random folds. I ended up learning pretty quickly from the LB itself and discussions here that the CV correlation is highly split dependent (I had ~0.75 CV, but ~0.5 LB). I first switched to a fairly naive split just doing GroupKFold with the sampling date being the groups. This got me to a CV that seems to replicate the leaderboard fairly well for my models (LB +/- 0.05 max from CV), but I felt stuck trying to get more performance out of the splits. It looked like some folds were failing to extrapolate and some folds were failing to generalize based on train/test distributions and prediction scatter plots. I converted the dates to months and seasons, then looked at the distributions of different data & metadata during different seasons. I noticed we only have certain states exemplified in certain seasons. The host mentioned here that the data split was not completely random and while all states are exemplified in the hidden set, not all of the dates are. I suspect the missing months might make up part of that. So now I'm trying more custom splitting strategies to account for this sort of thing. On my CV I get a little boost here, but we'll see how it plays out on the leaderboard once I have a model to submit.
Honestly, I've done a bit of work on the preprocessing & augmentation, but not a ton. For the sake of experimentation I've been saving my pre-computed augmentations, but perhaps I'm not capturing a wide enough distribution with them to prevent over-fitting. The big thing that stands out to me now that I'm thinking about it is I haven't done any cleaning…This might just be the \"obvious\" thing I feel like I'm missing.
I haven't spent a ton of time on the metadata yet, at least not in a way that directly uses it in a model. I've mostly used it to try and understand how to get a better split so far. Based on public notebooks, I think I should be able to get LB of ~ 0.65 at least before needing metadata to get further. We'll have to see. The distillation stuff is very new to me, with auxiliary tasks in multitasking being somewhat new, so I'm holding that off for now until after I can find a little more signal just from the images.
I had a lb score of 0.67 at the beginning of November. I then spend about 40 submissions going all over the place, from 0.62 to -11.5. There are fascinating traps all over the place and I really think this is a great competition as it forces you to really be careful about overfitting, feature engineering, and augmentations.
\nThere are discussions about fold strategies. I think these are key. Sometimes you may have a bad (or simple) fold strategy and a good (\"lucky\") seed (@\"Genius Seed is all you need\", I am looking at you…)
\n[Edit: I do not mean to say that a bad fold strategy can be fixed by a great seed - I mean a great seed may be all you need with a basic folding strategy. ]
Be careful with data preprocessing and cleaning. There are threads on this already but the data is messy. Some of this mess is generalization, some of this mess is data leak routes.
Metadata is an interesting point - it is critical information, but also hard to use.
What are your current strategies on these? Have you tried to changes these strategies?
", "votes": null, "replies": [ { "id": 3373801, "author_name": "nizarcharrada", "author_url": "", "post_date": "12/12/2025 15:05:02", "content": "Sorry to ask, but could you clarify what you mean by 'bad fold strategy'? Do you mean it's bad for the public leaderboard or just for certain seeds?
", "votes": null, "replies": [ { "id": 3373849, "author_name": "timkrige", "author_url": "", "post_date": "12/12/2025 15:20:08", "content": "Decent question: a bad fold strategy is one that is not fair. If you have images that you train on that are too similar to the ones in the validation, this is bad. So, trying to make sure that you have a fold strategy that fairly keeps information away from the training set is key. People have suggested folding based on state, or by date, or by random seed. All will run, but it is hard to develop a robust method of folding whereby your local CV score is indicative of the LB score. People who have found this will keep it a secret most likely.
\nThings to look for: do your folds have comparable val scores? do these scores tend to go in the same direction as the LB scores? (i.e., is a local improvement also a LB improvement?) and finally: are your local scores realistic?
\nEdit: I reread my original post and added some notes to make it clearer
", "votes": null, "replies": [] } ] }, { "id": 3374406, "author_name": "chemdatafarmer", "author_url": "", "post_date": "12/12/2025 17:51:48", "content": "Hey @timkrige thanks for your reply! I'm glad you're enjoying the competition too.
\nThe fold strategy is something I've been looking pretty heavily into. Early on I had a model I was pretty excited about using stratified or random folds. I ended up learning pretty quickly from the LB itself and discussions here that the CV correlation is highly split dependent (I had ~0.75 CV, but ~0.5 LB). I first switched to a fairly naive split just doing GroupKFold with the sampling date being the groups. This got me to a CV that seems to replicate the leaderboard fairly well for my models (LB +/- 0.05 max from CV), but I felt stuck trying to get more performance out of the splits. It looked like some folds were failing to extrapolate and some folds were failing to generalize based on train/test distributions and prediction scatter plots. I converted the dates to months and seasons, then looked at the distributions of different data & metadata during different seasons. I noticed we only have certain states exemplified in certain seasons. The host mentioned here that the data split was not completely random and while all states are exemplified in the hidden set, not all of the dates are. I suspect the missing months might make up part of that. So now I'm trying more custom splitting strategies to account for this sort of thing. On my CV I get a little boost here, but we'll see how it plays out on the leaderboard once I have a model to submit.
Honestly, I've done a bit of work on the preprocessing & augmentation, but not a ton. For the sake of experimentation I've been saving my pre-computed augmentations, but perhaps I'm not capturing a wide enough distribution with them to prevent over-fitting. The big thing that stands out to me now that I'm thinking about it is I haven't done any cleaning…This might just be the \"obvious\" thing I feel like I'm missing.
I haven't spent a ton of time on the metadata yet, at least not in a way that directly uses it in a model. I've mostly used it to try and understand how to get a better split so far. Based on public notebooks, I think I should be able to get LB of ~ 0.65 at least before needing metadata to get further. We'll have to see. The distillation stuff is very new to me, with auxiliary tasks in multitasking being somewhat new, so I'm holding that off for now until after I can find a little more signal just from the images.