{ "id": 588051, "title": "Fold Performance Consistency", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/588051", "author_name": "Anil Ozturk", "post_date": "2025-07-04T07:18:07.526000", "votes": 35, "comment_count": 28, "views": 0, "content": "

Are you getting distinctly worse scores for some of your CV folds or are all of them similar?

", "messages": [ { "id": 3240733, "postDate": "2025-07-04T07:18:07.527Z", "content": "

Are you getting distinctly worse scores for some of your CV folds or are all of them similar?

", "rawMarkdown": "Are you getting distinctly worse scores for some of your CV folds or are all of them similar?", "votes": 35 }, { "id": 3240789, "postDate": "2025-07-04T08:25:30.920Z", "content": "

There are two subjects who wore their devices on the wrong side of the wrist. They score really low. I checked and this doesn't happen anywhere in the leaderboards.

", "rawMarkdown": "There are two subjects who wore their devices on the wrong side of the wrist. They score really low. I checked and this doesn't happen anywhere in the leaderboards. ", "votes": 27, "replies": [ { "id": 3240805, "postDate": "2025-07-04T08:33:50.373Z", "content": "

Interesting! Would you share how you can spot that? You mean they wore the device like a real watch, not facing the palm of the hand ?

", "rawMarkdown": "Interesting! Would you share how you can spot that? You mean they wore the device like a real watch, not facing the palm of the hand ?", "votes": 4 }, { "id": 3240809, "postDate": "2025-07-04T08:37:48.550Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3240812, "postDate": "2025-07-04T08:40:50.357Z", "content": "

I assume it because they wore it wrong, not sure what else it could be. The average y acceleration is negative while all other subjects have positive average y acceleration.

", "rawMarkdown": "I assume it because they wore it wrong, not sure what else it could be. The average y acceleration is negative while all other subjects have positive average y acceleration. ", "votes": 10, "replies": [ { "id": 3240855, "postDate": "2025-07-04T09:34:31.223Z", "content": "

How can you probe that on the LB since you can't compute statistics at the subject level (only at sequence level) during inference due to the API server ?

", "rawMarkdown": "How can you probe that on the LB since you can't compute statistics at the subject level (only at sequence level) during inference due to the API server ?", "votes": 1 }, { "id": 3240865, "postDate": "2025-07-04T09:45:50.147Z", "content": "

Probably already said too much. But there is at least one work around to get subject level stats.

", "rawMarkdown": "Probably already said too much. But there is at least one work around to get subject level stats. ", "votes": 3 }, { "id": 3241005, "postDate": "2025-07-04T12:40:16.677Z", "content": "

yes, I can confirm that there are two subjects (SUBJ_045235, SUBJ_019262) who wore it like a wrist-watch.
\nI looked at the transformed Z-Vector (which should be facing away from the palm) for gestures where we know that the average direction of this should be approximately \"forward\" (e.g. waving or wirting the name in the air).

", "rawMarkdown": "yes, I can confirm that there are two subjects (SUBJ_045235, SUBJ_019262) who wore it like a wrist-watch.\nI looked at the transformed Z-Vector (which should be facing away from the palm) for gestures where we know that the average direction of this should be approximately \"forward\" (e.g. waving or wirting the name in the air)." }, { "id": 3241033, "postDate": "2025-07-04T13:02:29.200Z", "content": "

These 2 also have an average negative x_acc, there's around 8 subjects in total with negative x_acc

", "rawMarkdown": "These 2 also have an average negative x_acc, there's around 8 subjects in total with negative x_acc" }, { "id": 3241115, "postDate": "2025-07-04T15:09:19.183Z", "content": "

@devinanzelmo @andreizamfir
\nThe real question is.. do you see improvement from removing the subjects with obscure data? I trained 5 folds without (SUBJ_045235, SUBJ_019262) and will be testing that IMU-only submission tonight. When I look at my metric by subject, those two are significantly worse than the rest, so I'm hopeful it will improve LB. If not, I might have to try correcting their data.

", "rawMarkdown": "@devinanzelmo @andreizamfir \nThe real question is.. do you see improvement from removing the subjects with obscure data? I trained 5 folds without (SUBJ_045235, SUBJ_019262) and will be testing that IMU-only submission tonight. When I look at my metric by subject, those two are significantly worse than the rest, so I'm hopeful it will improve LB. If not, I might have to try correcting their data.", "votes": 1 }, { "id": 3241151, "postDate": "2025-07-04T15:32:53.570Z", "content": "

I have not tried, curious to what results your experiment gets, will try it as well. Did you see improvements in CV following their removal?

", "rawMarkdown": "I have not tried, curious to what results your experiment gets, will try it as well. Did you see improvements in CV following their removal?" }, { "id": 3241157, "postDate": "2025-07-04T15:38:04.173Z", "content": "

Well removing it will improve CV because its not decreasing the score anymore. You would have to remove it from the training part of the CV but not the evaluating part in order to tell if it is doing anything. I don't think it will have a noticeable impact on leaderboard score, but I am removing it from my training data for submission models just in case.

", "rawMarkdown": "Well removing it will improve CV because its not decreasing the score anymore. You would have to remove it from the training part of the CV but not the evaluating part in order to tell if it is doing anything. I don't think it will have a noticeable impact on leaderboard score, but I am removing it from my training data for submission models just in case. ", "votes": 1 }, { "id": 3241160, "postDate": "2025-07-04T15:38:57.627Z", "content": "

Yes, CV improved. Edit: LB did not. Leave the people who don’t know how to wear the sensor in!

", "rawMarkdown": "Yes, CV improved. Edit: LB did not. Leave the people who don’t know how to wear the sensor in!", "votes": 1 }, { "id": 3241591, "postDate": "2025-07-05T00:06:54.783Z", "content": "

My cv fell, just my luck haha

", "rawMarkdown": "My cv fell, just my luck haha", "votes": 1 }, { "id": 3241861, "postDate": "2025-07-05T09:36:53.480Z", "content": "

However, the other subjects with avg. negative x_acc are simply due to them being left handed (as expected due to the mirroring of the x-axis with regards to the handedness)

", "rawMarkdown": "However, the other subjects with avg. negative x_acc are simply due to them being left handed (as expected due to the mirroring of the x-axis with regards to the handedness)", "votes": 1 }, { "id": 3246061, "postDate": "2025-07-10T08:25:00.110Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 3247328, "postDate": "2025-07-12T14:57:02.657Z", "content": "

Why would you leave it in though just because LB improved. Isn't it bad data?

", "rawMarkdown": "Why would you leave it in though just because LB improved. Isn't it bad data?" } ] }, { "id": 3240853, "postDate": "2025-07-04T09:30:31.183Z", "content": "

well, I think I need to stop procrastinating on doing EDA. 😅

", "rawMarkdown": "well, I think I need to stop procrastinating on doing EDA. 😅", "votes": 1 } ] }, { "id": 3240768, "postDate": "2025-07-04T08:09:46.987Z", "content": "

my first fold is remarkably worse than the rest of the folds… I haven't investigated properly what is going on but it is a real concern.

\n

Could it be that some cases are mislabeled?

", "rawMarkdown": "my first fold is remarkably worse than the rest of the folds... I haven't investigated properly what is going on but it is a real concern.\n\nCould it be that some cases are mislabeled?", "votes": 3 }, { "id": 3242672, "postDate": "2025-07-06T07:56:28.560Z", "content": "

In our splitting strategy, for imu-only, fold 4 and fold 5 drop about 0.004~0.005 hf1. But fold 4 is great after adding tof + thm. Is it possible that the missing modality is causing this result?

", "rawMarkdown": "In our splitting strategy, for imu-only, fold 4 and fold 5 drop about 0.004~0.005 hf1. But fold 4 is great after adding tof + thm. Is it possible that the missing modality is causing this result? ", "votes": 1 }, { "id": 3240744, "postDate": "2025-07-04T07:34:02.100Z", "content": "

Yes, in my 5-fold cross-validation, there were two results around 0.8-0.82, one particularly poor result at only about 0.76.

", "rawMarkdown": "Yes, in my 5-fold cross-validation, there were two results around 0.8-0.82, one particularly poor result at only about 0.76.", "votes": 1 }, { "id": 3241084, "postDate": "2025-07-04T14:17:12.057Z", "content": "

No matter StratifiedKFold or StratifiedGroupKFold, my fold_4 is worest, fold_1 is best, which gap is more than 0.02.

", "rawMarkdown": "No matter StratifiedKFold or StratifiedGroupKFold, my fold_4 is worest, fold_1 is best, which gap is more than 0.02.", "votes": 2, "replies": [ { "id": 3241100, "postDate": "2025-07-04T14:47:12.110Z", "content": "

me too, I am using random seed 42 for the dataset

", "rawMarkdown": "me too, I am using random seed 42 for the dataset" }, { "id": 3241102, "postDate": "2025-07-04T14:48:12.013Z", "content": "

My gap is more than 0.05 lol

", "rawMarkdown": "My gap is more than 0.05 lol", "votes": 1, "replies": [ { "id": 3241106, "postDate": "2025-07-04T14:59:14.173Z", "content": "

Try to use global scaler as PB? It seems shorten the folds'gap on my experiments but cv is downing too.

", "rawMarkdown": "Try to use global scaler as PB? It seems shorten the folds'gap on my experiments but cv is downing too.", "votes": 1 }, { "id": 3241705, "postDate": "2025-07-05T04:57:12.427Z", "content": "

I didn't use scalers, I tried using standard scaler and it dropped my LB score.

", "rawMarkdown": "I didn't use scalers, I tried using standard scaler and it dropped my LB score.\n" } ] } ] }, { "id": 3246537, "postDate": "2025-07-11T05:33:14.560Z", "content": "

Nice.. Ok….

", "rawMarkdown": "Nice.. Ok.... ", "votes": -8 }, { "id": 3246163, "postDate": "2025-07-10T13:04:43.777Z", "content": "

I’m trying some group on subject trainings and found that some folds improving does not appear to correlate with LB. Now, I don’t want to over fit to public LB, just saying it’s interesting.

", "rawMarkdown": "I’m trying some group on subject trainings and found that some folds improving does not appear to correlate with LB. Now, I don’t want to over fit to public LB, just saying it’s interesting. " }, { "id": 3241872, "postDate": "2025-07-05T09:59:17.590Z", "content": "

I am using GroupKFold and there is a difference of 0.08 between the folds.

", "rawMarkdown": "I am using GroupKFold and there is a difference of 0.08 between the folds." } ], "comments": [ { "id": 3240789, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2025-07-04T08:25:30.920000", "content": "

There are two subjects who wore their devices on the wrong side of the wrist. They score really low. I checked and this doesn't happen anywhere in the leaderboards.

", "votes": 27, "replies": [ { "id": 3240805, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-04T08:33:50.373000", "content": "

Interesting! Would you share how you can spot that? You mean they wore the device like a real watch, not facing the palm of the hand ?

", "votes": 4, "replies": [] }, { "id": 3240809, "author_name": "", "author_url": "", "post_date": "2025-07-04T08:37:48.550000", "content": "", "votes": 0, "replies": [] }, { "id": 3240812, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2025-07-04T08:40:50.357000", "content": "

I assume it because they wore it wrong, not sure what else it could be. The average y acceleration is negative while all other subjects have positive average y acceleration.

", "votes": 10, "replies": [ { "id": 3240855, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-04T09:34:31.223000", "content": "

How can you probe that on the LB since you can't compute statistics at the subject level (only at sequence level) during inference due to the API server ?

", "votes": 1, "replies": [] }, { "id": 3240865, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2025-07-04T09:45:50.147000", "content": "

Probably already said too much. But there is at least one work around to get subject level stats.

", "votes": 3, "replies": [] }, { "id": 3241005, "author_name": "PatrickAIForFun", "author_url": "", "post_date": "2025-07-04T12:40:16.677000", "content": "

yes, I can confirm that there are two subjects (SUBJ_045235, SUBJ_019262) who wore it like a wrist-watch.
\nI looked at the transformed Z-Vector (which should be facing away from the palm) for gestures where we know that the average direction of this should be approximately \"forward\" (e.g. waving or wirting the name in the air).

", "votes": 0, "replies": [] }, { "id": 3241033, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-07-04T13:02:29.200000", "content": "

These 2 also have an average negative x_acc, there's around 8 subjects in total with negative x_acc

", "votes": 0, "replies": [] }, { "id": 3241115, "author_name": "Jack", "author_url": "", "post_date": "2025-07-04T15:09:19.183000", "content": "

@devinanzelmo @andreizamfir
\nThe real question is.. do you see improvement from removing the subjects with obscure data? I trained 5 folds without (SUBJ_045235, SUBJ_019262) and will be testing that IMU-only submission tonight. When I look at my metric by subject, those two are significantly worse than the rest, so I'm hopeful it will improve LB. If not, I might have to try correcting their data.

", "votes": 1, "replies": [] }, { "id": 3241151, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-07-04T15:32:53.570000", "content": "

I have not tried, curious to what results your experiment gets, will try it as well. Did you see improvements in CV following their removal?

", "votes": 0, "replies": [] }, { "id": 3241157, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2025-07-04T15:38:04.173000", "content": "

Well removing it will improve CV because its not decreasing the score anymore. You would have to remove it from the training part of the CV but not the evaluating part in order to tell if it is doing anything. I don't think it will have a noticeable impact on leaderboard score, but I am removing it from my training data for submission models just in case.

", "votes": 1, "replies": [] }, { "id": 3241160, "author_name": "Jack", "author_url": "", "post_date": "2025-07-04T15:38:57.627000", "content": "

Yes, CV improved. Edit: LB did not. Leave the people who don’t know how to wear the sensor in!

", "votes": 1, "replies": [] }, { "id": 3241591, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-07-05T00:06:54.783000", "content": "

My cv fell, just my luck haha

", "votes": 1, "replies": [] }, { "id": 3241861, "author_name": "PatrickAIForFun", "author_url": "", "post_date": "2025-07-05T09:36:53.480000", "content": "

However, the other subjects with avg. negative x_acc are simply due to them being left handed (as expected due to the mirroring of the x-axis with regards to the handedness)

", "votes": 1, "replies": [] }, { "id": 3246061, "author_name": "", "author_url": "", "post_date": "2025-07-10T08:25:00.110000", "content": "", "votes": -1, "replies": [] }, { "id": 3247328, "author_name": "Ravon", "author_url": "", "post_date": "2025-07-12T14:57:02.657000", "content": "

Why would you leave it in though just because LB improved. Isn't it bad data?

", "votes": 0, "replies": [] } ] }, { "id": 3240853, "author_name": "Anil Ozturk", "author_url": "", "post_date": "2025-07-04T09:30:31.183000", "content": "

well, I think I need to stop procrastinating on doing EDA. 😅

", "votes": 1, "replies": [] } ] }, { "id": 3240768, "author_name": "Optimo", "author_url": "", "post_date": "2025-07-04T08:09:46.987000", "content": "

my first fold is remarkably worse than the rest of the folds… I haven't investigated properly what is going on but it is a real concern.

\n

Could it be that some cases are mislabeled?

", "votes": 3, "replies": [] }, { "id": 3242672, "author_name": "Tom", "author_url": "", "post_date": "2025-07-06T07:56:28.560000", "content": "

In our splitting strategy, for imu-only, fold 4 and fold 5 drop about 0.004~0.005 hf1. But fold 4 is great after adding tof + thm. Is it possible that the missing modality is causing this result?

", "votes": 1, "replies": [] }, { "id": 3240744, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-07-04T07:34:02.100000", "content": "

Yes, in my 5-fold cross-validation, there were two results around 0.8-0.82, one particularly poor result at only about 0.76.

", "votes": 1, "replies": [] }, { "id": 3241084, "author_name": "Kurise", "author_url": "", "post_date": "2025-07-04T14:17:12.057000", "content": "

No matter StratifiedKFold or StratifiedGroupKFold, my fold_4 is worest, fold_1 is best, which gap is more than 0.02.

", "votes": 2, "replies": [ { "id": 3241100, "author_name": "menglei", "author_url": "", "post_date": "2025-07-04T14:47:12.110000", "content": "

me too, I am using random seed 42 for the dataset

", "votes": 0, "replies": [] }, { "id": 3241102, "author_name": "menglei", "author_url": "", "post_date": "2025-07-04T14:48:12.013000", "content": "

My gap is more than 0.05 lol

", "votes": 1, "replies": [ { "id": 3241106, "author_name": "Kurise", "author_url": "", "post_date": "2025-07-04T14:59:14.173000", "content": "

Try to use global scaler as PB? It seems shorten the folds'gap on my experiments but cv is downing too.

", "votes": 1, "replies": [] }, { "id": 3241705, "author_name": "menglei", "author_url": "", "post_date": "2025-07-05T04:57:12.427000", "content": "

I didn't use scalers, I tried using standard scaler and it dropped my LB score.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3246537, "author_name": "Mehwish", "author_url": "", "post_date": "2025-07-11T05:33:14.560000", "content": "

Nice.. Ok….

", "votes": -8, "replies": [] }, { "id": 3246163, "author_name": "Jack", "author_url": "", "post_date": "2025-07-10T13:04:43.777000", "content": "

I’m trying some group on subject trainings and found that some folds improving does not appear to correlate with LB. Now, I don’t want to over fit to public LB, just saying it’s interesting.

", "votes": 0, "replies": [] }, { "id": 3241872, "author_name": "t fuku", "author_url": "", "post_date": "2025-07-05T09:59:17.590000", "content": "

I am using GroupKFold and there is a difference of 0.08 between the folds.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3240733": "Are you getting distinctly worse scores for some of your CV folds or are all of them similar?", "3240789": "There are two subjects who wore their devices on the wrong side of the wrist. They score really low. I checked and this doesn't happen anywhere in the leaderboards. ", "3240768": "my first fold is remarkably worse than the rest of the folds... I haven't investigated properly what is going on but it is a real concern.\n\nCould it be that some cases are mislabeled?", "3242672": "In our splitting strategy, for imu-only, fold 4 and fold 5 drop about 0.004~0.005 hf1. But fold 4 is great after adding tof + thm. Is it possible that the missing modality is causing this result? ", "3240744": "Yes, in my 5-fold cross-validation, there were two results around 0.8-0.82, one particularly poor result at only about 0.76.", "3241084": "No matter StratifiedKFold or StratifiedGroupKFold, my fold_4 is worest, fold_1 is best, which gap is more than 0.02.", "3246537": "Nice.. Ok.... ", "3246163": "I’m trying some group on subject trainings and found that some folds improving does not appear to correlate with LB. Now, I don’t want to over fit to public LB, just saying it’s interesting. ", "3241872": "I am using GroupKFold and there is a difference of 0.08 between the folds." } }