{ "id": 587453, "title": "Large gap between CV and LB", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/587453", "author_name": "menglei", "post_date": "2025-07-01T05:47:19.269000", "votes": 0, "comment_count": 5, "views": 0, "content": "

Mine CV is always 0.06 higher than my LB, has anyone has met this, my 5-fold cross validation is:

\n

seq_info = df.groupby('sequence_id')\\\n .agg(gesture=('gesture','first'), subject=('subject','first'))\\\n .reset_index()
\nfollowed by
\n`splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)

\n

for fold, (tr_idx, vl_idx) in enumerate(
\n splitter.split(np.zeros(len(seq_info)), seq_info.gesture, groups=seq_info.subject),
\n start=1
\n ):`

\n

Does anyone know if this is correct? If it is correct where might the gap come from?

", "messages": [ { "id": 3237526, "postDate": "2025-07-01T06:35:32.187Z", "content": "

Half of the test data consists of null tof-thm sensors, that part might harm your performance if you use them actively without considering null-scenario.

", "rawMarkdown": "Half of the test data consists of null tof-thm sensors, that part might harm your performance if you use them actively without considering null-scenario.", "votes": 1, "replies": [ { "id": 3237532, "postDate": "2025-07-01T06:41:15.940Z", "content": "

I used two models, an imu only and a full model, when all of the the tof or thm is absent (nan), I will use the imu only model, but there is still a gap, I'm trying to understand why lol.

", "rawMarkdown": "I used two models, an imu only and a full model, when all of the the tof or thm is absent (nan), I will use the imu only model, but there is still a gap, I'm trying to understand why lol.", "votes": 1, "replies": [ { "id": 3237585, "postDate": "2025-07-01T07:20:00.883Z", "content": "

I have exactly the same. I expect 0.81 LB after evaluating the CV but I'm getting 0.75.

", "rawMarkdown": "I have exactly the same. I expect 0.81 LB after evaluating the CV but I'm getting 0.75." }, { "id": 3237699, "postDate": "2025-07-01T08:47:50.590Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3238148, "postDate": "2025-07-01T15:45:03.463Z", "content": "

Btw I'm wondering what model are you submitting? Are you submitting your best model across all folds according to the validation score, or you retrain them as a whole and submitted it, or you ensemble the models together?

", "rawMarkdown": "Btw I'm wondering what model are you submitting? Are you submitting your best model across all folds according to the validation score, or you retrain them as a whole and submitted it, or you ensemble the models together?", "votes": -2 } ] } ] }, { "id": 3237469, "postDate": "2025-07-01T05:47:19.270Z", "content": "

Mine CV is always 0.06 higher than my LB, has anyone has met this, my 5-fold cross validation is:

\n

seq_info = df.groupby('sequence_id')\\\n .agg(gesture=('gesture','first'), subject=('subject','first'))\\\n .reset_index()
\nfollowed by
\n`splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)

\n

for fold, (tr_idx, vl_idx) in enumerate(
\n splitter.split(np.zeros(len(seq_info)), seq_info.gesture, groups=seq_info.subject),
\n start=1
\n ):`

\n

Does anyone know if this is correct? If it is correct where might the gap come from?

", "rawMarkdown": "Mine CV is always 0.06 higher than my LB, has anyone has met this, my 5-fold cross validation is: \n\n\n\n`seq_info = df.groupby('sequence_id')\\\n .agg(gesture=('gesture','first'), subject=('subject','first'))\\\n .reset_index()`\nfollowed by\n`splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)\n\nfor fold, (tr_idx, vl_idx) in enumerate(\n splitter.split(np.zeros(len(seq_info)), seq_info.gesture, groups=seq_info.subject),\n start=1\n ):`\n\nDoes anyone know if this is correct? If it is correct where might the gap come from?" } ], "comments": [ { "id": 3237526, "author_name": "Anil Ozturk", "author_url": "", "post_date": "2025-07-01T06:35:32.187000", "content": "

Half of the test data consists of null tof-thm sensors, that part might harm your performance if you use them actively without considering null-scenario.

", "votes": 1, "replies": [ { "id": 3237532, "author_name": "menglei", "author_url": "", "post_date": "2025-07-01T06:41:15.940000", "content": "

I used two models, an imu only and a full model, when all of the the tof or thm is absent (nan), I will use the imu only model, but there is still a gap, I'm trying to understand why lol.

", "votes": 1, "replies": [ { "id": 3237585, "author_name": "stefanoclss", "author_url": "", "post_date": "2025-07-01T07:20:00.883000", "content": "

I have exactly the same. I expect 0.81 LB after evaluating the CV but I'm getting 0.75.

", "votes": 0, "replies": [] }, { "id": 3237699, "author_name": "", "author_url": "", "post_date": "2025-07-01T08:47:50.590000", "content": "", "votes": 0, "replies": [] }, { "id": 3238148, "author_name": "menglei", "author_url": "", "post_date": "2025-07-01T15:45:03.463000", "content": "

Btw I'm wondering what model are you submitting? Are you submitting your best model across all folds according to the validation score, or you retrain them as a whole and submitted it, or you ensemble the models together?

", "votes": -2, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3237526": "Half of the test data consists of null tof-thm sensors, that part might harm your performance if you use them actively without considering null-scenario.", "3237469": "Mine CV is always 0.06 higher than my LB, has anyone has met this, my 5-fold cross validation is: \n\n\n\n`seq_info = df.groupby('sequence_id')\\\n .agg(gesture=('gesture','first'), subject=('subject','first'))\\\n .reset_index()`\nfollowed by\n`splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)\n\nfor fold, (tr_idx, vl_idx) in enumerate(\n splitter.split(np.zeros(len(seq_info)), seq_info.gesture, groups=seq_info.subject),\n start=1\n ):`\n\nDoes anyone know if this is correct? If it is correct where might the gap come from?" } }