{ "id": 598291, "title": "Bert Solo Model Training", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/598291", "author_name": "Cody_Null", "post_date": "2025-08-09T19:05:55.476000", "votes": 11, "comment_count": 16, "views": 0, "content": "

After the release of the 5 fold ensembled bert by @wasupandceacar I tried to replicate the training script. I will share this version of that script: https://www.kaggle.com/code/cody11null/public-bert-training-attempt I did a number of other trials with this but was not able to improve from the public notebook. Only get a near score. I would be curious if anyone else has had any luck and what they have tried?

", "messages": [ { "id": 3266877, "postDate": "2025-08-09T20:58:23.433Z", "content": "

In my training pipeline(only saying the version I published):

\n
    \n
  1. More augment, like cutmix, mixup, time_scale, etc. Mention that imu only augment is also important
    2. \n
  2. I find adamw has some unstable converge than adam, maybe the dataset is too small
    3. \n
  3. ReduceLROnPlateau with parameters tuning
    4. \n
  4. More loss combination
    5. \n
", "rawMarkdown": "In my training pipeline(only saying the version I published):\n1. More augment, like cutmix, mixup, time_scale, etc. Mention that imu only augment is also important\n2. I find adamw has some unstable converge than adam, maybe the dataset is too small\n3. ReduceLROnPlateau with parameters tuning\n4. More loss combination", "votes": 14 }, { "id": 3266830, "postDate": "2025-08-09T19:05:55.477Z", "content": "

After the release of the 5 fold ensembled bert by @wasupandceacar I tried to replicate the training script. I will share this version of that script: https://www.kaggle.com/code/cody11null/public-bert-training-attempt I did a number of other trials with this but was not able to improve from the public notebook. Only get a near score. I would be curious if anyone else has had any luck and what they have tried?

", "rawMarkdown": "After the release of the 5 fold ensembled bert by @wasupandceacar I tried to replicate the training script. I will share this version of that script: https://www.kaggle.com/code/cody11null/public-bert-training-attempt I did a number of other trials with this but was not able to improve from the public notebook. Only get a near score. I would be curious if anyone else has had any luck and what they have tried?", "votes": 11 }, { "id": 3267203, "postDate": "2025-08-10T17:13:25.077Z", "content": "

Thank you for sharing this great code! @cody11null
\nI have a question about the Validation Strategy and would appreciate some feedback.

\n

This train & original use StratifiedKFold

\n
\nStratifiedKFold(=n_folds, =, =random_seed)\n\nself.skf = StratifiedKFold(=n_folds, =, =random_seed)\n
\n

In my opinion, SKF leaks data because data providers are not grouped.
\nTherefore, I believe that SGKF by subject has a better CV/LB correlation.

\n

I look forward to your feedback.

", "rawMarkdown": "Thank you for sharing this great code! @cody11null \nI have a question about the Validation Strategy and would appreciate some feedback.\n\nThis train & original use **StratifiedKFold**\n```\n# Origin\nStratifiedKFold(n_splits=n_folds, shuffle=True, random_state=random_seed)\n# This Train\nself.skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=random_seed)\n```\n\nIn my opinion, **SKF** leaks data because data providers are not grouped.\nTherefore, I believe that **SGKF** by subject has a better CV/LB correlation.\n\nI look forward to your feedback.\n", "votes": 1, "replies": [ { "id": 3267207, "postDate": "2025-08-10T17:16:47.803Z", "content": "

I think you are right and I totally agree!

", "rawMarkdown": "I think you are right and I totally agree!", "votes": 1 } ] }, { "id": 3266884, "postDate": "2025-08-09T21:26:17.850Z", "content": "

Your team has a score of 0.86, surely you have a better single model than 0.82?

", "rawMarkdown": "Your team has a score of 0.86, surely you have a better single model than 0.82?", "votes": 1, "replies": [ { "id": 3266916, "postDate": "2025-08-10T00:38:12.457Z", "content": "

My teams work is very unique in this competition I think. My teammates strategy gives us great boost with ensemble. Currently our best solo model is below .85 and best imu <82.5. We hope to team with someone with better solo models as well as improve our own. We have had quite some issues with a few parts that have recently been resolved.

", "rawMarkdown": "My teams work is very unique in this competition I think. My teammates strategy gives us great boost with ensemble. Currently our best solo model is below .85 and best imu <82.5. We hope to team with someone with better solo models as well as improve our own. We have had quite some issues with a few parts that have recently been resolved.", "replies": [ { "id": 3266918, "postDate": "2025-08-10T00:58:57.233Z", "content": "

Ok then you have single model bordering 8.5 which is a big improvement compared to best published single model- 0.82 iirc. So why you say 'but was not able to improve from the public notebook' šŸ˜… If you want people to share secrets you have to share secrets first! šŸ¤£ BTW I have 1 model 5 fold IMU only 0.829, so just know you can improve! but is quite volatile, the same model with different seed can get as low as 0.814. Maybe top scores have single imu only 0.835-0.84! šŸ˜±

", "rawMarkdown": "Ok then you have single model bordering 8.5 which is a big improvement compared to best published single model- 0.82 iirc. So why you say 'but was not able to improve from the public notebook' šŸ˜… If you want people to share secrets you have to share secrets first! šŸ¤£ BTW I have 1 model 5 fold IMU only 0.829, so just know you can improve! but is quite volatile, the same model with different seed can get as low as 0.814. Maybe top scores have single imu only 0.835-0.84! šŸ˜±", "votes": 1 }, { "id": 3266930, "postDate": "2025-08-10T01:37:06.040Z", "content": "

Haha I more just wanted to share for others that got great value out of the public model and wished to have a start for training their own. No secrets sadly haha. Yes a little too late in the game to be sharing .85 solo model lol. I have seen lots of discussion about numbers much better than ours so I am optimistic for sure. This public model helped our ensemble so I figure it could for others too!

", "rawMarkdown": "Haha I more just wanted to share for others that got great value out of the public model and wished to have a start for training their own. No secrets sadly haha. Yes a little too late in the game to be sharing .85 solo model lol. I have seen lots of discussion about numbers much better than ours so I am optimistic for sure. This public model helped our ensemble so I figure it could for others too!", "votes": 3 }, { "id": 3266970, "postDate": "2025-08-10T04:54:58.370Z", "content": "

Regarding the models with scores of .829 and .814, do their cross-validation results exhibit a similar performance gap?

", "rawMarkdown": "Regarding the models with scores of .829 and .814, do their cross-validation results exhibit a similar performance gap?" }, { "id": 3266989, "postDate": "2025-08-10T06:23:52.453Z", "content": "

Hi ,The best single model in our team is 0.85. Perhaps the way our ensemble strategy was not quite right, which resulted in the score not showing much improvement. By the way, the current best score for our imu only model is LB 0.823.šŸ˜€

", "rawMarkdown": "Hi ,The best single model in our team is 0.85. Perhaps the way our ensemble strategy was not quite right, which resulted in the score not showing much improvement. By the way, the current best score for our imu only model is LB 0.823.šŸ˜€" }, { "id": 3267002, "postDate": "2025-08-10T07:01:20.470Z", "content": "

@wscmisx A bit smaller- something like 0.812-0.82.

", "rawMarkdown": "@wscmisx A bit smaller- something like 0.812-0.82. ", "votes": 2 }, { "id": 3267362, "postDate": "2025-08-11T03:07:18.337Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3268232, "postDate": "2025-08-12T16:28:03Z", "content": "

Thank you. Iā€™m curious, why is the demographic of subjects not being used in this approach\"\"

", "rawMarkdown": "Thank you. Iā€™m curious, why is the demographic of subjects not being used in this approach![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3842406%2F4d815b71167c11c205515205d93ba0a0%2Fdemograph.png?generation=1755016078492607&alt=media)", "replies": [ { "id": 3268291, "postDate": "2025-08-12T17:47:01.810Z", "content": "

I have considered this a lot, I think it could make some difference but I worry about overfitting with these metrics. For example if one of these metrics was massively valuable and then in test it shifted a bit I would worry it would not translate. Not sure if that will be an issue or not. I also personally hadnt gotten a great amount of value from these.

", "rawMarkdown": "I have considered this a lot, I think it could make some difference but I worry about overfitting with these metrics. For example if one of these metrics was massively valuable and then in test it shifted a bit I would worry it would not translate. Not sure if that will be an issue or not. I also personally hadnt gotten a great amount of value from these. ", "votes": 1 }, { "id": 3268882, "postDate": "2025-08-13T14:46:48.023Z", "content": "

I did the performance analysis for model not using demographics features, I saw:

\n\n
\nEXPERIMENT: IMU-THM-TOF\n\nBinary Performance by Adult/Child\n\nChi-square test: chi2=0.534, p-value=0.465\ncorrectcorrect\nadultcorrect\nadult
\n\n
: IMU-THM-TOF\n r (binary) = -., p = .\n r (binary) = -., p = .\n r (macro) = ., p = .\n r (macro) = ., p = .\n
\n

\"height\"

\n
: IMU-THM-TOF\n r (binary) = -., p = .e-\n r (binary) = -., p = .\n r (macro) = -., p = .\n r (macro) = ., p = .\n
\n

\"shoulder-to-wrist\"

\n
: IMU-THM-TOF\n r (binary) = -., p = .e-\n r (binary) = -., p = .e-\n r (macro) = -., p = .\n r (macro) = ., p = .\n
\n

\"elbow-to-wrist\"

", "rawMarkdown": "I did the performance analysis for model not using demographics features, I saw:\n* the model performs slighly better for males, right handedness, and adult than females, left handedness, and children --> this may be because we have more data for males, right handedness, and adults than the counterparts. However, this difference is very small, not worth to integrate those features in training.\nFor example:\n```\n********************************************************************************\nEXPERIMENT: IMU-THM-TOF\n\nBinary Performance by Adult/Child\n\nChi-square test: chi2=0.534, p-value=0.465\ncorrect_bin 0 1 percent_correct\nadult_child \nChild 173 3700 95.533178\nAdult 176 4102 95.885928\n==================================================\n\nMacro Performance by Adult/Child\n\nChi-square test: chi2=9.545, p-value=0.002\ncorrect 0 1 percent_correct\nadult_child \nChild 1067 2806 72.450297\nAdult 1049 3229 75.479196\n```\n* There is not significant difference in height, shoulder to wrist, and elbow to wrist --> I think that these features are not very important because the model performs agnostically to them.\n\n```\nEXPERIMENT: IMU-THM-TOF\nPearson r (binary) = -0.017, p = 0.1234\nSpearman r (binary) = -0.021, p = 0.05342\nPearson r (macro) = 0.020, p = 0.07798\nSpearman r (macro) = 0.023, p = 0.03685\n```\n![height](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F878726%2F6f016495c884c937e571091e0d93b27a%2F__results___34_3.png?generation=1755096179390697&alt=media)\n\n```\nEXPERIMENT: IMU-THM-TOF\nPearson r (binary) = -0.097, p = 1.802e-18\nSpearman r (binary) = -0.017, p = 0.1197\nPearson r (macro) = -0.013, p = 0.2388\nSpearman r (macro) = 0.018, p = 0.1137\n```\n![shoulder-to-wrist](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F878726%2F26d5d9fe31faf897fe815fdf66a7561e%2F__results___37_3.png?generation=1755096107627545&alt=media)\n\n```\nEXPERIMENT: IMU-THM-TOF\nPearson r (binary) = -0.189, p = 2.856e-66\nSpearman r (binary) = -0.049, p = 8.484e-06\nPearson r (macro) = -0.032, p = 0.003555\nSpearman r (macro) = 0.013, p = 0.2525\n```\n![elbow-to-wrist](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F878726%2F650a8eb063e9e85ea637f635c23434ed%2F__results___39_3.png?generation=1755096341962550&alt=media)\n", "votes": 7 } ] }, { "id": 3268230, "postDate": "2025-08-12T16:27:10.803Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3268229, "postDate": "2025-08-12T16:25:54.807Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3266877, "author_name": "wasupandceacar", "author_url": "", "post_date": "2025-08-09T20:58:23.433000", "content": "

In my training pipeline(only saying the version I published):

\n
    \n
  1. More augment, like cutmix, mixup, time_scale, etc. Mention that imu only augment is also important
    2. \n
  2. I find adamw has some unstable converge than adam, maybe the dataset is too small
    3. \n
  3. ReduceLROnPlateau with parameters tuning
    4. \n
  4. More loss combination
    5. \n
", "votes": 14, "replies": [] }, { "id": 3267203, "author_name": "yukiZ", "author_url": "", "post_date": "2025-08-10T17:13:25.077000", "content": "

Thank you for sharing this great code! @cody11null
\nI have a question about the Validation Strategy and would appreciate some feedback.

\n

This train & original use StratifiedKFold

\n
\nStratifiedKFold(=n_folds, =, =random_seed)\n\nself.skf = StratifiedKFold(=n_folds, =, =random_seed)\n
\n

In my opinion, SKF leaks data because data providers are not grouped.
\nTherefore, I believe that SGKF by subject has a better CV/LB correlation.

\n

I look forward to your feedback.

", "votes": 1, "replies": [ { "id": 3267207, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-08-10T17:16:47.803000", "content": "

I think you are right and I totally agree!

", "votes": 1, "replies": [] } ] }, { "id": 3266884, "author_name": "greySnow", "author_url": "", "post_date": "2025-08-09T21:26:17.850000", "content": "

Your team has a score of 0.86, surely you have a better single model than 0.82?

", "votes": 1, "replies": [ { "id": 3266916, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-08-10T00:38:12.457000", "content": "

My teams work is very unique in this competition I think. My teammates strategy gives us great boost with ensemble. Currently our best solo model is below .85 and best imu <82.5. We hope to team with someone with better solo models as well as improve our own. We have had quite some issues with a few parts that have recently been resolved.

", "votes": 0, "replies": [ { "id": 3266918, "author_name": "greySnow", "author_url": "", "post_date": "2025-08-10T00:58:57.233000", "content": "

Ok then you have single model bordering 8.5 which is a big improvement compared to best published single model- 0.82 iirc. So why you say 'but was not able to improve from the public notebook' šŸ˜… If you want people to share secrets you have to share secrets first! šŸ¤£ BTW I have 1 model 5 fold IMU only 0.829, so just know you can improve! but is quite volatile, the same model with different seed can get as low as 0.814. Maybe top scores have single imu only 0.835-0.84! šŸ˜±

", "votes": 1, "replies": [] }, { "id": 3266930, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-08-10T01:37:06.040000", "content": "

Haha I more just wanted to share for others that got great value out of the public model and wished to have a start for training their own. No secrets sadly haha. Yes a little too late in the game to be sharing .85 solo model lol. I have seen lots of discussion about numbers much better than ours so I am optimistic for sure. This public model helped our ensemble so I figure it could for others too!

", "votes": 3, "replies": [] }, { "id": 3266970, "author_name": "wsc-MISX", "author_url": "", "post_date": "2025-08-10T04:54:58.370000", "content": "

Regarding the models with scores of .829 and .814, do their cross-validation results exhibit a similar performance gap?

", "votes": 0, "replies": [] }, { "id": 3266989, "author_name": "XXX", "author_url": "", "post_date": "2025-08-10T06:23:52.453000", "content": "

Hi ,The best single model in our team is 0.85. Perhaps the way our ensemble strategy was not quite right, which resulted in the score not showing much improvement. By the way, the current best score for our imu only model is LB 0.823.šŸ˜€

", "votes": 0, "replies": [] }, { "id": 3267002, "author_name": "greySnow", "author_url": "", "post_date": "2025-08-10T07:01:20.470000", "content": "

@wscmisx A bit smaller- something like 0.812-0.82.

", "votes": 2, "replies": [] }, { "id": 3267362, "author_name": "", "author_url": "", "post_date": "2025-08-11T03:07:18.337000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3268232, "author_name": "Cyrus", "author_url": "", "post_date": "2025-08-12T16:28:03", "content": "

Thank you. Iā€™m curious, why is the demographic of subjects not being used in this approach\"\"

", "votes": 0, "replies": [ { "id": 3268291, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-08-12T17:47:01.810000", "content": "

I have considered this a lot, I think it could make some difference but I worry about overfitting with these metrics. For example if one of these metrics was massively valuable and then in test it shifted a bit I would worry it would not translate. Not sure if that will be an issue or not. I also personally hadnt gotten a great amount of value from these.

", "votes": 1, "replies": [] }, { "id": 3268882, "author_name": "Pi", "author_url": "", "post_date": "2025-08-13T14:46:48.023000", "content": "

I did the performance analysis for model not using demographics features, I saw:

\n\n
\nEXPERIMENT: IMU-THM-TOF\n\nBinary Performance by Adult/Child\n\nChi-square test: chi2=0.534, p-value=0.465\ncorrectcorrect\nadultcorrect\nadult
\n\n
: IMU-THM-TOF\n r (binary) = -., p = .\n r (binary) = -., p = .\n r (macro) = ., p = .\n r (macro) = ., p = .\n
\n

\"height\"

\n
: IMU-THM-TOF\n r (binary) = -., p = .e-\n r (binary) = -., p = .\n r (macro) = -., p = .\n r (macro) = ., p = .\n
\n

\"shoulder-to-wrist\"

\n
: IMU-THM-TOF\n r (binary) = -., p = .e-\n r (binary) = -., p = .e-\n r (macro) = -., p = .\n r (macro) = ., p = .\n
\n

\"elbow-to-wrist\"

", "votes": 7, "replies": [] } ] }, { "id": 3268230, "author_name": "", "author_url": "", "post_date": "2025-08-12T16:27:10.803000", "content": "", "votes": 0, "replies": [] }, { "id": 3268229, "author_name": "", "author_url": "", "post_date": "2025-08-12T16:25:54.807000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3266877": "In my training pipeline(only saying the version I published):\n1. More augment, like cutmix, mixup, time_scale, etc. Mention that imu only augment is also important\n2. I find adamw has some unstable converge than adam, maybe the dataset is too small\n3. ReduceLROnPlateau with parameters tuning\n4. More loss combination", "3266830": "After the release of the 5 fold ensembled bert by @wasupandceacar I tried to replicate the training script. I will share this version of that script: https://www.kaggle.com/code/cody11null/public-bert-training-attempt I did a number of other trials with this but was not able to improve from the public notebook. Only get a near score. I would be curious if anyone else has had any luck and what they have tried?", "3267203": "Thank you for sharing this great code! @cody11null \nI have a question about the Validation Strategy and would appreciate some feedback.\n\nThis train & original use **StratifiedKFold**\n```\n# Origin\nStratifiedKFold(n_splits=n_folds, shuffle=True, random_state=random_seed)\n# This Train\nself.skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=random_seed)\n```\n\nIn my opinion, **SKF** leaks data because data providers are not grouped.\nTherefore, I believe that **SGKF** by subject has a better CV/LB correlation.\n\nI look forward to your feedback.\n", "3266884": "Your team has a score of 0.86, surely you have a better single model than 0.82?", "3268232": "Thank you. Iā€™m curious, why is the demographic of subjects not being used in this approach![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3842406%2F4d815b71167c11c205515205d93ba0a0%2Fdemograph.png?generation=1755016078492607&alt=media)", "3268230": "", "3268229": "" } }