{ "id": 590520, "title": "Cross Validation/Leader board score mismatch", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/590520", "author_name": "", "post_date": "2025-07-21T12:34:59.130957600Z", "votes": null, "comment_count": 2, "views": 0, "content": "

During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.

\n

I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:

\n
 () -> [[pd.DataFrame, pd.DataFrame]]:\n    sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n    folds = []\n    X = df.drop(columns=[target_col])\n    y = df[target_col]\n    groups = df[group_col]\n\n     train_idx, valid_idx  sgkf.split(X, y, groups):\n        train_df = df.iloc[train_idx].copy()\n        valid_df = df.iloc[valid_idx].copy()\n        folds.append((train_df, valid_df))\n\n     folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\n
\n

And I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:

\n
 () -> :\n    model = model.()\n    model_device = (model.parameters()).device\n    competition_metric = metric_package.Metric()\n\n    total_accuracy = \n    total_test_loss = \n     torch.inference_mode():\n         x, y  validation_loader:\n            x = x.to(model_device)\n            y = y.to(model_device)\n\n            half = BATCH_SIZE //          \n            x_front = x[:half]               \n            x_back  = x[half:].clone()      \n\n            x_back[:, :, non_imu_feats_idx] =     \n            x = torch.cat([x_front, x_back], dim=)  \n\n            y_pred = model(x)\n            total_test_loss += critirion(y_pred, y).item()\n            batch_accuracy = compute_accuracy(y_pred, y)\n            total_accuracy += batch_accuracy\n            competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n     {\n        : total_test_loss / (validation_loader),\n        : total_accuracy / (validation_loader),\n        : competition_metric.score(),\n    }\n
\n

The current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, [training notebook]During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.

\n

I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:

\n
 () -> [[pd.DataFrame, pd.DataFrame]]:\n    sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n    folds = []\n    X = df.drop(columns=[target_col])\n    y = df[target_col]\n    groups = df[group_col]\n\n     train_idx, valid_idx  sgkf.split(X, y, groups):\n        train_df = df.iloc[train_idx].copy()\n        valid_df = df.iloc[valid_idx].copy()\n        folds.append((train_df, valid_df))\n\n     folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\n
\n

And I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:

\n
 () -> :\n    model = model.()\n    model_device = (model.parameters()).device\n    competition_metric = metric_package.Metric()\n\n    total_accuracy = \n    total_test_loss = \n     torch.inference_mode():\n         x, y  validation_loader:\n            x = x.to(model_device)\n            y = y.to(model_device)\n\n            half = BATCH_SIZE //          \n            x_front = x[:half]               \n            x_back  = x[half:].clone()      \n\n            x_back[:, :, non_imu_feats_idx] =     \n            x = torch.cat([x_front, x_back], dim=)  \n\n            y_pred = model(x)\n            total_test_loss += critirion(y_pred, y).item()\n            batch_accuracy = compute_accuracy(y_pred, y)\n            total_accuracy += batch_accuracy\n            competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n     {\n        : total_test_loss / (validation_loader),\n        : total_accuracy / (validation_loader),\n        : competition_metric.score(),\n    }\n
\n

The current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, training notebook and inference notebook.

\n

Do you see something I am doing wrong?
\nReading the entire pipeline would be cumbersome, so let me know if you need extra info to help.
\nEvery answer/advice is welcome.
\nThanks in advance!

", "messages": [ { "id": "3251703", "postDate": "07/21/2025 12:34:59", "content": "

During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.

\n

I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:

\n
 () -> [[pd.DataFrame, pd.DataFrame]]:\n    sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n    folds = []\n    X = df.drop(columns=[target_col])\n    y = df[target_col]\n    groups = df[group_col]\n\n     train_idx, valid_idx  sgkf.split(X, y, groups):\n        train_df = df.iloc[train_idx].copy()\n        valid_df = df.iloc[valid_idx].copy()\n        folds.append((train_df, valid_df))\n\n     folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\n
\n

And I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:

\n
 () -> :\n    model = model.()\n    model_device = (model.parameters()).device\n    competition_metric = metric_package.Metric()\n\n    total_accuracy = \n    total_test_loss = \n     torch.inference_mode():\n         x, y  validation_loader:\n            x = x.to(model_device)\n            y = y.to(model_device)\n\n            half = BATCH_SIZE //          \n            x_front = x[:half]               \n            x_back  = x[half:].clone()      \n\n            x_back[:, :, non_imu_feats_idx] =     \n            x = torch.cat([x_front, x_back], dim=)  \n\n            y_pred = model(x)\n            total_test_loss += critirion(y_pred, y).item()\n            batch_accuracy = compute_accuracy(y_pred, y)\n            total_accuracy += batch_accuracy\n            competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n     {\n        : total_test_loss / (validation_loader),\n        : total_accuracy / (validation_loader),\n        : competition_metric.score(),\n    }\n
\n

The current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, [training notebook]During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.

\n

I am using sklearn's StratifiedGroupKFold to create my CV folds as follow:

\n
 () -> [[pd.DataFrame, pd.DataFrame]]:\n    sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=)\n\n    folds = []\n    X = df.drop(columns=[target_col])\n    y = df[target_col]\n    groups = df[group_col]\n\n     train_idx, valid_idx  sgkf.split(X, y, groups):\n        train_df = df.iloc[train_idx].copy()\n        valid_df = df.iloc[valid_idx].copy()\n        folds.append((train_df, valid_df))\n\n     folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=, group_col=)\n
\n

And I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025
\nI integrate it in my evaluation function as follow:

\n
 () -> :\n    model = model.()\n    model_device = (model.parameters()).device\n    competition_metric = metric_package.Metric()\n\n    total_accuracy = \n    total_test_loss = \n     torch.inference_mode():\n         x, y  validation_loader:\n            x = x.to(model_device)\n            y = y.to(model_device)\n\n            half = BATCH_SIZE //          \n            x_front = x[:half]               \n            x_back  = x[half:].clone()      \n\n            x_back[:, :, non_imu_feats_idx] =     \n            x = torch.cat([x_front, x_back], dim=)  \n\n            y_pred = model(x)\n            total_test_loss += critirion(y_pred, y).item()\n            batch_accuracy = compute_accuracy(y_pred, y)\n            total_accuracy += batch_accuracy\n            competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n     {\n        : total_test_loss / (validation_loader),\n        : total_accuracy / (validation_loader),\n        : competition_metric.score(),\n    }\n
\n

The current version of my preprocessing, training and inference notebooks are attached.
\nHere are the links to my preprocessing notebook, training notebook and inference notebook.

\n

Do you see something I am doing wrong?
\nReading the entire pipeline would be cumbersome, so let me know if you need extra info to help.
\nEvery answer/advice is welcome.
\nThanks in advance!

", "rawMarkdown": "During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.\n\nI am using sklearn's StratifiedGroupKFold to create my CV folds as follow:\n```python\ndef split_dataset_stratified_groupkfold(df: pd.DataFrame, target_col: str, group_col: str) -> list[tuple[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=True)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n for train_idx, valid_idx in sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n return folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=\"gesture\", group_col=\"subject\")\n```\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025\nI integrate it in my evaluation function as follow:\n```python\ndef evaluate_model(\n model: torch.nn.Module,\n critirion:callable,\n validation_loader:DL,\n ) -> dict:\n model = model.eval()\n model_device = next(model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = 0\n total_test_loss = 0\n with torch.inference_mode():\n for x, y in validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // 2 \n x_front = x[:half] \n x_back = x[half:].clone() \n \n x_back[:, :, non_imu_feats_idx] = 0.0 \n x = torch.cat([x_front, x_back], dim=0) # (B, C, T)\n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n return {\n \"validation_loss\": total_test_loss / len(validation_loader),\n \"validation_accuracy\": total_accuracy / len(validation_loader),\n \"competition_score\": competition_metric.score(),\n }\n```\nThe current version of my preprocessing, training and inference notebooks are attached.\nHere are the links to my [preprocessing notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/preprocess_dataset.ipynb), [training notebook]During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.\n\nI am using sklearn's StratifiedGroupKFold to create my CV folds as follow:\n```python\ndef split_dataset_stratified_groupkfold(df: pd.DataFrame, target_col: str, group_col: str) -> list[tuple[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=True)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n for train_idx, valid_idx in sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n return folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=\"gesture\", group_col=\"subject\")\n```\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025\nI integrate it in my evaluation function as follow:\n```python\ndef evaluate_model(\n model: torch.nn.Module,\n critirion:callable,\n validation_loader:DL,\n ) -> dict:\n model = model.eval()\n model_device = next(model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = 0\n total_test_loss = 0\n with torch.inference_mode():\n for x, y in validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // 2 \n x_front = x[:half] \n x_back = x[half:].clone() \n \n x_back[:, :, non_imu_feats_idx] = 0.0 \n x = torch.cat([x_front, x_back], dim=0) # (B, C, T)\n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n return {\n \"validation_loss\": total_test_loss / len(validation_loader),\n \"validation_accuracy\": total_accuracy / len(validation_loader),\n \"competition_score\": competition_metric.score(),\n }\n```\nThe current version of my preprocessing, training and inference notebooks are attached.\nHere are the links to my [preprocessing notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/preprocess_dataset.ipynb), [training notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/training.ipynb) and [inference notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/inference.ipynb).\n\nDo you see something I am doing wrong?\nReading the entire pipeline would be cumbersome, so let me know if you need extra info to help.\nEvery answer/advice is welcome.\nThanks in advance!", "votes": null }, { "id": "3252343", "postDate": "07/22/2025 13:52:52", "content": "

half of the test seqeunce will be imu only, that is probably the cause

", "rawMarkdown": "half of the test seqeunce will be imu only, that is probably the cause", "votes": null }, { "id": "3252441", "postDate": "07/22/2025 18:18:02", "content": "

Hello, thank you for your answer, I null the non imu features in the evaluation function here:

\n
            half = BATCH_SIZE //          \n            x_front = x[:half]               \n            x_back  = x[half:].clone()      \n\n            x_back[:, :, non_imu_feats_idx] =     \n
\n

Unfortunatly that doesn't seem to allow me to have an accurate score.
\nDo you see anything else that I might be doing wrong.
\nAgain, thank you for your answer.

", "rawMarkdown": "Hello, thank you for your answer, I null the non imu features in the evaluation function here:\n```python\n half = BATCH_SIZE // 2 \n x_front = x[:half] \n x_back = x[half:].clone() \n\n x_back[:, :, non_imu_feats_idx] = 0.0 \n```\nUnfortunatly that doesn't seem to allow me to have an accurate score.\nDo you see anything else that I might be doing wrong.\nAgain, thank you for your answer.", "votes": null } ], "comments": [ { "id": 3252343, "author_name": "georgewang73", "author_url": "", "post_date": "07/22/2025 13:52:52", "content": "

half of the test seqeunce will be imu only, that is probably the cause

", "votes": null, "replies": [ { "id": 3252441, "author_name": "mauroabidalcarrer", "author_url": "", "post_date": "07/22/2025 18:18:02", "content": "

Hello, thank you for your answer, I null the non imu features in the evaluation function here:

\n
            half = BATCH_SIZE //          \n            x_front = x[:half]               \n            x_back  = x[half:].clone()      \n\n            x_back[:, :, non_imu_feats_idx] =     \n
\n

Unfortunatly that doesn't seem to allow me to have an accurate score.
\nDo you see anything else that I might be doing wrong.
\nAgain, thank you for your answer.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3251703": "During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.\n\nI am using sklearn's StratifiedGroupKFold to create my CV folds as follow:\n```python\ndef split_dataset_stratified_groupkfold(df: pd.DataFrame, target_col: str, group_col: str) -> list[tuple[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=True)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n for train_idx, valid_idx in sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n return folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=\"gesture\", group_col=\"subject\")\n```\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025\nI integrate it in my evaluation function as follow:\n```python\ndef evaluate_model(\n model: torch.nn.Module,\n critirion:callable,\n validation_loader:DL,\n ) -> dict:\n model = model.eval()\n model_device = next(model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = 0\n total_test_loss = 0\n with torch.inference_mode():\n for x, y in validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // 2 \n x_front = x[:half] \n x_back = x[half:].clone() \n \n x_back[:, :, non_imu_feats_idx] = 0.0 \n x = torch.cat([x_front, x_back], dim=0) # (B, C, T)\n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n return {\n \"validation_loss\": total_test_loss / len(validation_loader),\n \"validation_accuracy\": total_accuracy / len(validation_loader),\n \"competition_score\": competition_metric.score(),\n }\n```\nThe current version of my preprocessing, training and inference notebooks are attached.\nHere are the links to my [preprocessing notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/preprocess_dataset.ipynb), [training notebook]During training I get a CV average of 0.57 but the corresponding Leader board is 0.32.\n\nI am using sklearn's StratifiedGroupKFold to create my CV folds as follow:\n```python\ndef split_dataset_stratified_groupkfold(df: pd.DataFrame, target_col: str, group_col: str) -> list[tuple[pd.DataFrame, pd.DataFrame]]:\n sgkf = StratifiedGroupKFold(n_splits=N_FOLDS, shuffle=True)\n\n folds = []\n X = df.drop(columns=[target_col])\n y = df[target_col]\n groups = df[group_col]\n\n for train_idx, valid_idx in sgkf.split(X, y, groups):\n train_df = df.iloc[train_idx].copy()\n valid_df = df.iloc[valid_idx].copy()\n folds.append((train_df, valid_df))\n\n return folds\n\nfolds = split_dataset_stratified_groupkfold(df, target_col=\"gesture\", group_col=\"subject\")\n```\nAnd I am using this code to measure my score locally: https://www.kaggle.com/code/metric/cmi-2025\nI integrate it in my evaluation function as follow:\n```python\ndef evaluate_model(\n model: torch.nn.Module,\n critirion:callable,\n validation_loader:DL,\n ) -> dict:\n model = model.eval()\n model_device = next(model.parameters()).device\n competition_metric = metric_package.Metric()\n\n total_accuracy = 0\n total_test_loss = 0\n with torch.inference_mode():\n for x, y in validation_loader:\n x = x.to(model_device)\n y = y.to(model_device)\n\n half = BATCH_SIZE // 2 \n x_front = x[:half] \n x_back = x[half:].clone() \n \n x_back[:, :, non_imu_feats_idx] = 0.0 \n x = torch.cat([x_front, x_back], dim=0) # (B, C, T)\n\n y_pred = model(x)\n total_test_loss += critirion(y_pred, y).item()\n batch_accuracy = compute_accuracy(y_pred, y)\n total_accuracy += batch_accuracy\n competition_metric.add(get_str_target(y), get_str_target(y_pred))\n\n return {\n \"validation_loss\": total_test_loss / len(validation_loader),\n \"validation_accuracy\": total_accuracy / len(validation_loader),\n \"competition_score\": competition_metric.score(),\n }\n```\nThe current version of my preprocessing, training and inference notebooks are attached.\nHere are the links to my [preprocessing notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/preprocess_dataset.ipynb), [training notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/training.ipynb) and [inference notebook](https://github.com/MauroAbidalCarrer/CMI-2025-competition/blob/competition_score_metric/notebooks/inference.ipynb).\n\nDo you see something I am doing wrong?\nReading the entire pipeline would be cumbersome, so let me know if you need extra info to help.\nEvery answer/advice is welcome.\nThanks in advance!", "3252343": "half of the test seqeunce will be imu only, that is probably the cause", "3252441": "Hello, thank you for your answer, I null the non imu features in the evaluation function here:\n```python\n half = BATCH_SIZE // 2 \n x_front = x[:half] \n x_back = x[half:].clone() \n\n x_back[:, :, non_imu_feats_idx] = 0.0 \n```\nUnfortunatly that doesn't seem to allow me to have an accurate score.\nDo you see anything else that I might be doing wrong.\nAgain, thank you for your answer." }, "source": "meta" }