As we can know, the cv have great variance between folds. In my experiments, just adding one feature, getting good cv on 4 folds but getting worest cv on 1 fold, which got better cv but bad lb. So I think different feature on different folds should be tried. But it will raise more question such as : 1. model input dim 2. Which model should be bigger weights when we are in inference? I would like to hear someone who have tried this.
", "messages": [ { "id": 3245555, "postDate": "2025-07-09T14:45:18.350Z", "content": "I have experienced some disappointing CV/LB correlation lately as well. It's always frustrating to improve CV but get worse LB score.
\nNevertheless, you need to keep in mind why your performed cross validation in the first place: estimate as accurately as possible the performances of ONE set of training/modelling parameters.
\nChanging the parameters for each fold in order to optimize your CV does not make sense at all: you are overfitting individually each fold in order to artificially increase a CV score which does not play it's initial role anymore.
I think the most important part is to understand why folds are behaving differently.
", "rawMarkdown": "I have experienced some disappointing CV/LB correlation lately as well. It's always frustrating to improve CV but get worse LB score.\n\nNevertheless, you need to keep in mind why your performed cross validation in the first place: estimate as accurately as possible the performances of ONE set of training/modelling parameters.\nChanging the parameters for each fold in order to optimize your CV does not make sense at all: you are overfitting individually each fold in order to artificially increase a CV score which does not play it's initial role anymore.\n\n\nI think the most important part is to understand why folds are behaving differently.\n", "votes": 5, "replies": [ { "id": 3245596, "postDate": "2025-07-09T15:18:49.107Z", "content": "You're right, I have the experience as you said. I even tried different seed on folds lol… Thank you for the reminder.
", "rawMarkdown": "You're right, I have the experience as you said. I even tried different seed on folds lol... Thank you for the reminder." }, { "id": 3246190, "postDate": "2025-07-10T14:22:04.673Z", "content": "Are you using grouped folds? If so, have your recent frustrating experiments been an overall CV improvement where already strong folds decreased and weak folds increased?
", "rawMarkdown": "Are you using grouped folds? If so, have your recent frustrating experiments been an overall CV improvement where already strong folds decreased and weak folds increased? ", "replies": [ { "id": 3246281, "postDate": "2025-07-10T16:05:20.557Z", "content": "sadly all the folds increased :)
", "rawMarkdown": "sadly all the folds increased :)" }, { "id": 3246284, "postDate": "2025-07-10T16:07:33.513Z", "content": "Sadly sure is an interesting word in this context 😂 Have you checked confusion matrix? Class representation in train likely doesn’t reflect distribution in test. Orientations definitely add some problems
", "rawMarkdown": "Sadly sure is an interesting word in this context 😂 Have you checked confusion matrix? Class representation in train likely doesn’t reflect distribution in test. Orientations definitely add some problems " } ] } ] }, { "id": 3246017, "postDate": "2025-07-10T07:04:50.910Z", "content": "IMO it's better to aim both increasing the mean and decreasing the std of the CV. I'm not considering doing a submission if both are not checked.
", "rawMarkdown": "IMO it's better to aim both increasing the mean and decreasing the std of the CV. I'm not considering doing a submission if both are not checked.", "votes": 3, "replies": [ { "id": 3246067, "postDate": "2025-07-10T08:35:21.970Z", "content": "In order to figure out whether a new feature need to add, I have to submit it even if both increasing the mean and decreasing the std of the CV, because LB is the third performance standard.
\nIn my feature experiments, such as :
\na:
\nFold 1 Best H-F1:0.8294
\nFold 2 Best H-F1:0.7768
\nFold 3 Best H-F1:0.7808
\nFold 4 Best H-F1:0.7824
\nFold 5 Best H-F1:0.7882
\nMean H-F1 across 5 folds:0.7915 + 0.0193
\nb:
\nFold 1 Best H-F1:0.8325
\nFold 2 Best H-F1:0.7731
\nFold 3 Best H-F1:0.7805
\nFold 4 Best H-F1:0.7777
\nFold 5 Best H-F1:0.7912
\nMean H-F1 across 5 folds:0.7910 + 0.0216
\nb got better LB.
0.0005 is too small of a change to consider, I actually consider trying if I get an improvement >0.003
", "rawMarkdown": "0.0005 is too small of a change to consider, I actually consider trying if I get an improvement >0.003", "votes": 2 } ] } ] }, { "id": 3245502, "postDate": "2025-07-09T13:01:57.777Z", "content": "Maybe we need to use different set of features for some attributes? I think that's a good direction.
", "rawMarkdown": "Maybe we need to use different set of features for some attributes? I think that's a good direction.", "votes": 1, "replies": [ { "id": 3245509, "postDate": "2025-07-09T13:20:56.870Z", "content": "I don't understand, would you mind to explain it? Thanks! I thought it still means a global feature, if data is perfect. But it will get good distinguish when some sequence lose few data(attributes), such as acc_x(attributes) have linear_acc_x(set of features). If so, it would be a good resolution! But it need to make complete EDA on different folds.
", "rawMarkdown": "I don't understand, would you mind to explain it? Thanks! I thought it still means a global feature, if data is perfect. But it will get good distinguish when some sequence lose few data(attributes), such as acc_x(attributes) have linear_acc_x(set of features). If so, it would be a good resolution! But it need to make complete EDA on different folds.", "replies": [ { "id": 3245956, "postDate": "2025-07-10T04:39:14.727Z", "content": "@kurisew like: if a specific physical phenomenon happened, we provide jerk and snap, or other physical measurements.
", "rawMarkdown": "@kurisew like: if a specific physical phenomenon happened, we provide jerk and snap, or other physical measurements. ", "votes": 1 } ] } ] }, { "id": 3245501, "postDate": "2025-07-09T12:59:14.980Z", "content": "As we can know, the cv have great variance between folds. In my experiments, just adding one feature, getting good cv on 4 folds but getting worest cv on 1 fold, which got better cv but bad lb. So I think different feature on different folds should be tried. But it will raise more question such as : 1. model input dim 2. Which model should be bigger weights when we are in inference? I would like to hear someone who have tried this.
", "rawMarkdown": "As we can know, the cv have great variance between folds. In my experiments, just adding one feature, getting good cv on 4 folds but getting worest cv on 1 fold, which got better cv but bad lb. So I think different feature on different folds should be tried. But it will raise more question such as : 1. model input dim 2. Which model should be bigger weights when we are in inference? I would like to hear someone who have tried this." }, { "id": 3246030, "postDate": "2025-07-10T07:38:45.840Z", "rawMarkdown": "", "votes": -3, "isDeleted": true } ], "comments": [ { "id": 3245555, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-09T14:45:18.350000", "content": "I have experienced some disappointing CV/LB correlation lately as well. It's always frustrating to improve CV but get worse LB score.
\nNevertheless, you need to keep in mind why your performed cross validation in the first place: estimate as accurately as possible the performances of ONE set of training/modelling parameters.
\nChanging the parameters for each fold in order to optimize your CV does not make sense at all: you are overfitting individually each fold in order to artificially increase a CV score which does not play it's initial role anymore.
I think the most important part is to understand why folds are behaving differently.
", "votes": 5, "replies": [ { "id": 3245596, "author_name": "Kurise", "author_url": "", "post_date": "2025-07-09T15:18:49.107000", "content": "You're right, I have the experience as you said. I even tried different seed on folds lol… Thank you for the reminder.
", "votes": 0, "replies": [] }, { "id": 3246190, "author_name": "Jack", "author_url": "", "post_date": "2025-07-10T14:22:04.673000", "content": "Are you using grouped folds? If so, have your recent frustrating experiments been an overall CV improvement where already strong folds decreased and weak folds increased?
", "votes": 0, "replies": [ { "id": 3246281, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-10T16:05:20.557000", "content": "sadly all the folds increased :)
", "votes": 0, "replies": [] }, { "id": 3246284, "author_name": "Jack", "author_url": "", "post_date": "2025-07-10T16:07:33.513000", "content": "Sadly sure is an interesting word in this context 😂 Have you checked confusion matrix? Class representation in train likely doesn’t reflect distribution in test. Orientations definitely add some problems
", "votes": 0, "replies": [] } ] } ] }, { "id": 3246017, "author_name": "Anil Ozturk", "author_url": "", "post_date": "2025-07-10T07:04:50.910000", "content": "IMO it's better to aim both increasing the mean and decreasing the std of the CV. I'm not considering doing a submission if both are not checked.
", "votes": 3, "replies": [ { "id": 3246067, "author_name": "Kurise", "author_url": "", "post_date": "2025-07-10T08:35:21.970000", "content": "In order to figure out whether a new feature need to add, I have to submit it even if both increasing the mean and decreasing the std of the CV, because LB is the third performance standard.
\nIn my feature experiments, such as :
\na:
\nFold 1 Best H-F1:0.8294
\nFold 2 Best H-F1:0.7768
\nFold 3 Best H-F1:0.7808
\nFold 4 Best H-F1:0.7824
\nFold 5 Best H-F1:0.7882
\nMean H-F1 across 5 folds:0.7915 + 0.0193
\nb:
\nFold 1 Best H-F1:0.8325
\nFold 2 Best H-F1:0.7731
\nFold 3 Best H-F1:0.7805
\nFold 4 Best H-F1:0.7777
\nFold 5 Best H-F1:0.7912
\nMean H-F1 across 5 folds:0.7910 + 0.0216
\nb got better LB.
0.0005 is too small of a change to consider, I actually consider trying if I get an improvement >0.003
", "votes": 2, "replies": [] } ] } ] }, { "id": 3245502, "author_name": "Tom", "author_url": "", "post_date": "2025-07-09T13:01:57.777000", "content": "Maybe we need to use different set of features for some attributes? I think that's a good direction.
", "votes": 1, "replies": [ { "id": 3245509, "author_name": "Kurise", "author_url": "", "post_date": "2025-07-09T13:20:56.870000", "content": "I don't understand, would you mind to explain it? Thanks! I thought it still means a global feature, if data is perfect. But it will get good distinguish when some sequence lose few data(attributes), such as acc_x(attributes) have linear_acc_x(set of features). If so, it would be a good resolution! But it need to make complete EDA on different folds.
", "votes": 0, "replies": [ { "id": 3245956, "author_name": "Tom", "author_url": "", "post_date": "2025-07-10T04:39:14.727000", "content": "@kurisew like: if a specific physical phenomenon happened, we provide jerk and snap, or other physical measurements.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3246030, "author_name": "", "author_url": "", "post_date": "2025-07-10T07:38:45.840000", "content": "", "votes": -3, "replies": [] } ], "raw_markdown_by_id": { "3245555": "I have experienced some disappointing CV/LB correlation lately as well. It's always frustrating to improve CV but get worse LB score.\n\nNevertheless, you need to keep in mind why your performed cross validation in the first place: estimate as accurately as possible the performances of ONE set of training/modelling parameters.\nChanging the parameters for each fold in order to optimize your CV does not make sense at all: you are overfitting individually each fold in order to artificially increase a CV score which does not play it's initial role anymore.\n\n\nI think the most important part is to understand why folds are behaving differently.\n", "3246017": "IMO it's better to aim both increasing the mean and decreasing the std of the CV. I'm not considering doing a submission if both are not checked.", "3245502": "Maybe we need to use different set of features for some attributes? I think that's a good direction.", "3245501": "As we can know, the cv have great variance between folds. In my experiments, just adding one feature, getting good cv on 4 folds but getting worest cv on 1 fold, which got better cv but bad lb. So I think different feature on different folds should be tried. But it will raise more question such as : 1. model input dim 2. Which model should be bigger weights when we are in inference? I would like to hear someone who have tried this.", "3246030": "" } }