Links to code:
\n\nCongratulations to all the winners and a big respect to the organizers for putting together such a challenging and rewarding competition.
\nSpecial thanks to my teammates yuanzhe zhou and Devin Anzelmo for the great collaboration and team spirit.
Large ensembles for IMU-only and all-features samples, assembled as weighted sum of logits
Post-processing to take into account limited number of samples with each class and with each orientation per subject (already described here)
Additional models for orientation prediction
1. Preprocessing/feature engineering:
\nRemoved subjects who wore device incorrectly from both training and validation
For TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)
Some models trained with temporal difference of above TOF features and thermal
IMU: raw 7 + temporal diff of acc, rotational diff as posted in notebooks, as well as magnitude of acc/rot; 16 total
preprocess left handed to match right handed for IMU and THM/TOF. Flip acc_x, convert rot to xyz euler and flip sign for y and z
robust scaler
2. Augmentation:
\nup scaling/down scaling all features
random noise
3. Models:
\nCNN GRU with different stems for different features. Feature extractor is four branches of CNN with no pooling
Pure CNN. Same feature extractor as above but three blocks of residual at end with expanding kernal (5, 9, 13)
dense GRU, scores lower but helps a little in ensemble
some self excitation models, some not combining helps
3D CNN for TOF data scores about 0.83 but blends well
4. Training:
\nbatch size 64
train for 60-80 (depending on model) iters with AdamW lr = 0.0005
5. Ensemble:
\nblend IMU models with all-features models
blend TOF-only model improves all data score
1. Filtering data:
\n[all models] remove sequence SEQ_011975 without gesture
[all models] remove sequences with gesture/total ratio < 0.2 (this subset includes almost all shortest sequences)
[models with THM and TOF data] remove sequences with TOF completely missing (missing values patterns in TOF - 0%, 20% or 100%; 20% missing is ok)
few models were trained without \"problematic\" subjects SUBJ_045235 and SUBJ_019262
2. Preprocessing:
\nfill quaternion missing values, if possible
mirror data for left-handed subjects
replace -1 TOF values with 500 because -1 encodes large distance
ultimate filling NaNs with seq.ffill().bfill().fillna(0)
pad/crop by 94 percentile in each fold (~120 max seq length)
no scalers - it gave the largest improvement at early stage
3. Feature engineering:
\nnothing for THM and TOF
35 IMU features:
[', ', ', ', ', ', ', ', ', ', ', ', \nrot_w', ', ', ', ', ', ', ', ', ', ', \nacc_x2', ', ', ', ', ', ', ', ', ', ', ']\nacc_x2 and other ***2 features are important part of the solution.
\nThese features are calculated on values transformed by remove_gravity_from_acc().
4. Models:
\nbunch of models with different architectures for training on IMU-only data and similar models for training with all features
main differences in time dimension processing: LSTM-, attention- or CNN-based
key part - independent processing groups of features at first layers
5. Augmentations:
\nmixup with MIXUP_ALPHA = 0.4
augmentation from CMI | Sequence Data Augmentation (thanks moth):
transforms = Compose, \n TimeStretch, \n])\nwith some variations.
\nIt was hard to pick up right augmetations with right probability & strength and apply it in the correct order.
\n6. Training:
\nCV5 with group by subject K-fold, 5 models are used on inference
200 or 220 epochs, patience 30 or 35
torch.optim.Adam with lr = 4e-4 or lr = 5e-4 and weight_decay = 3e-4
CosineAnnealingWarmRestarts; LinearLR + CosineAnnealingWarmRestarts for attention-based models
one model was trained with focal loss and class weights (class weights obviously mean nothing for mixup samples, maybe it can serve as regularization)
one model contains bug in cross-attention (shared weights for 3 different attentions) but works pretty well
7. Some tricks:
\npick best result for each fold among 3 runs
increase sequence length by 5 on inference (it doesn't improve CV score but effectively decorrelates models)
1.1. Using the pipeline structure from CMI3 Pyroch Baseline+model_add, Aug, folds
\n1.2. Fixing many bugs improved my score to 0.85.
\nModels: simple combinations of RNN and CNN1D.
Feature engineering using pytorch, which avoids problem of consistency but makes the training slower.
Basic preprocessing mentioned by most teams: fillnan, handedness, etc.
Post-processing based on data structure.
Augmentation idea from teammates:
6.1. Mask data during training helps.
", "messages": [ { "id": 3280919, "postDate": "2025-09-03T11:14:57.397Z", "content": "Links to code:
\n\nCongratulations to all the winners and a big respect to the organizers for putting together such a challenging and rewarding competition.
\nSpecial thanks to my teammates yuanzhe zhou and Devin Anzelmo for the great collaboration and team spirit.
Large ensembles for IMU-only and all-features samples, assembled as weighted sum of logits
Post-processing to take into account limited number of samples with each class and with each orientation per subject (already described here)
Additional models for orientation prediction
1. Preprocessing/feature engineering:
\nRemoved subjects who wore device incorrectly from both training and validation
For TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)
Some models trained with temporal difference of above TOF features and thermal
IMU: raw 7 + temporal diff of acc, rotational diff as posted in notebooks, as well as magnitude of acc/rot; 16 total
preprocess left handed to match right handed for IMU and THM/TOF. Flip acc_x, convert rot to xyz euler and flip sign for y and z
robust scaler
2. Augmentation:
\nup scaling/down scaling all features
random noise
3. Models:
\nCNN GRU with different stems for different features. Feature extractor is four branches of CNN with no pooling
Pure CNN. Same feature extractor as above but three blocks of residual at end with expanding kernal (5, 9, 13)
dense GRU, scores lower but helps a little in ensemble
some self excitation models, some not combining helps
3D CNN for TOF data scores about 0.83 but blends well
4. Training:
\nbatch size 64
train for 60-80 (depending on model) iters with AdamW lr = 0.0005
5. Ensemble:
\nblend IMU models with all-features models
blend TOF-only model improves all data score
1. Filtering data:
\n[all models] remove sequence SEQ_011975 without gesture
[all models] remove sequences with gesture/total ratio < 0.2 (this subset includes almost all shortest sequences)
[models with THM and TOF data] remove sequences with TOF completely missing (missing values patterns in TOF - 0%, 20% or 100%; 20% missing is ok)
few models were trained without \"problematic\" subjects SUBJ_045235 and SUBJ_019262
2. Preprocessing:
\nfill quaternion missing values, if possible
mirror data for left-handed subjects
replace -1 TOF values with 500 because -1 encodes large distance
ultimate filling NaNs with seq.ffill().bfill().fillna(0)
pad/crop by 94 percentile in each fold (~120 max seq length)
no scalers - it gave the largest improvement at early stage
3. Feature engineering:
\nnothing for THM and TOF
35 IMU features:
[', ', ', ', ', ', ', ', ', ', ', ', \nrot_w', ', ', ', ', ', ', ', ', ', ', \nacc_x2', ', ', ', ', ', ', ', ', ', ', ']\nacc_x2 and other ***2 features are important part of the solution.
\nThese features are calculated on values transformed by remove_gravity_from_acc().
4. Models:
\nbunch of models with different architectures for training on IMU-only data and similar models for training with all features
main differences in time dimension processing: LSTM-, attention- or CNN-based
key part - independent processing groups of features at first layers
5. Augmentations:
\nmixup with MIXUP_ALPHA = 0.4
augmentation from CMI | Sequence Data Augmentation (thanks moth):
transforms = Compose, \n TimeStretch, \n])\nwith some variations.
\nIt was hard to pick up right augmetations with right probability & strength and apply it in the correct order.
\n6. Training:
\nCV5 with group by subject K-fold, 5 models are used on inference
200 or 220 epochs, patience 30 or 35
torch.optim.Adam with lr = 4e-4 or lr = 5e-4 and weight_decay = 3e-4
CosineAnnealingWarmRestarts; LinearLR + CosineAnnealingWarmRestarts for attention-based models
one model was trained with focal loss and class weights (class weights obviously mean nothing for mixup samples, maybe it can serve as regularization)
one model contains bug in cross-attention (shared weights for 3 different attentions) but works pretty well
7. Some tricks:
\npick best result for each fold among 3 runs
increase sequence length by 5 on inference (it doesn't improve CV score but effectively decorrelates models)
1.1. Using the pipeline structure from CMI3 Pyroch Baseline+model_add, Aug, folds
\n1.2. Fixing many bugs improved my score to 0.85.
\nModels: simple combinations of RNN and CNN1D.
Feature engineering using pytorch, which avoids problem of consistency but makes the training slower.
Basic preprocessing mentioned by most teams: fillnan, handedness, etc.
Post-processing based on data structure.
Augmentation idea from teammates:
6.1. Mask data during training helps.
", "rawMarkdown": "**Links to code:**\n- [Training code on github](https://github.com/statist-bhfz/kaggle_cmi_1st_place_solution)\n- [Inference notebook](https://www.kaggle.com/code/yuanzhezhou/final-sub-cmi3)\n\n## Solution Writeup\n\nCongratulations to all the winners and a big respect to the organizers for putting together such a challenging and rewarding competition. \nSpecial thanks to my teammates [yuanzhe zhou](https://www.kaggle.com/yuanzhezhou) and [Devin Anzelmo](https://www.kaggle.com/devinanzelmo) for the great collaboration and team spirit.\n\n## Solution Overview\n\n- Large ensembles for IMU-only and all-features samples, assembled as weighted sum of logits\n\n- Post-processing to take into account limited number of samples with each class and with each orientation per subject (already described [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544))\n\n- Additional models for orientation prediction\n\n------------------------------------------------------------------------\n\n## Devin's part\n\n**1. Preprocessing/feature engineering:**\n\n- Removed subjects who wore device incorrectly from both training and validation\n\n- For TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)\n\n- Some models trained with temporal difference of above TOF features and thermal\n\n- IMU: raw 7 + temporal diff of `acc`, rotational diff as posted in notebooks, as well as magnitude of `acc`/`rot`; 16 total\n\n- preprocess left handed to match right handed for IMU and THM/TOF. Flip `acc_x`, convert `rot` to xyz euler and flip sign for y and z\n\n- robust scaler \n\n**2. Augmentation:**\n\n- up scaling/down scaling all features \n\n- random noise \n\n**3. Models:**\n\n- CNN GRU with different stems for different features. Feature extractor is four branches of CNN with no pooling\n\n- Pure CNN. Same feature extractor as above but three blocks of residual at end with expanding kernal (5, 9, 13)\n\n- dense GRU, scores lower but helps a little in ensemble\n\n- some self excitation models, some not combining helps\n\n- 3D CNN for TOF data scores about 0.83 but blends well\n\n**4. Training:**\n\n- batch size 64\n\n- train for 60-80 (depending on model) iters with AdamW `lr = 0.0005`\n\n**5. Ensemble:**\n\n- blend IMU models with all-features models \n\n- blend TOF-only model improves all data score\n\n------------------------------------------------------------------------\n\n## Ogurtsov's part\n\n**1. Filtering data:**\t\n\t\n- [all models] remove sequence SEQ_011975 without gesture\t\n\t\n- [all models] remove sequences with gesture/total ratio < 0.2 (this subset includes almost all shortest sequences)\t\n\t\n- [models with THM and TOF data] remove sequences with TOF completely missing\t(missing values patterns in TOF - 0%, 20% or 100%; 20% missing is ok)\t\n\n- few models were trained without \"problematic\" subjects SUBJ_045235 and SUBJ_019262\n\t\n**2. Preprocessing:**\n\t\n- fill quaternion missing values, if possible\t\n\t\n- mirror data for left-handed subjects \n\n- replace -1 TOF values with 500 because -1 encodes large distance\t\n\t\n- ultimate filling NaNs with `seq.ffill().bfill().fillna(0)`\n\t\n- pad/crop by 94 percentile in each fold (~120 max seq length)\t\n\t\n- no scalers - it gave the largest improvement at early stage\t\n\t\n**3. Feature engineering:**\t\n\t\n- nothing for THM and TOF\t\n\t\n- 35 IMU features:\t\n```\n['acc_x', 'acc_y', 'acc_z', 'acc_mag', 'acc_mag_jerk', 'jerk_x', 'jerk_y', 'jerk_z', 'jerk_magnitude', 'acc_xy_corr', 'acc_xz_corr', 'acc_yz_corr',\t\nrot_w', 'rot_x', 'rot_y', 'rot_z', 'rot_angle', 'rot_angle_vel', 'angular_vel_x', 'angular_vel_y', 'angular_vel_z', 'angular_vel_magnitude', 'angular_distance',\t\nacc_x2', 'acc_y2', 'acc_z2', 'acc_mag2', 'acc_mag_jerk2', 'jerk_x2', 'jerk_y2', 'jerk_z2', 'jerk_magnitude2', 'acc_xy_corr2', 'acc_xz_corr2', 'acc_yz_corr2']\n```\n\n`acc_x2` and other `***2` features are important part of the solution. \nThese features are calculated on values transformed by `remove_gravity_from_acc()`.\t\n\t\n**4. Models:**\t\n\t\n- bunch of models with different architectures for training on IMU-only data and similar models for training with all features\t\n\t\n- main differences in time dimension processing: LSTM-, attention- or CNN-based\t\n\t\n- key part - independent processing groups of features at first layers\t\n\t\n**5. Augmentations:**\t\n\t\n- mixup with `MIXUP_ALPHA = 0.4`\n\t\n- augmentation from [CMI | Sequence Data Augmentation](https://www.kaggle.com/code/alejopaullier/cmi-sequence-data-augmentation) (thanks [moth](https://www.kaggle.com/alejopaullier)):\t\n```\ntransforms = Compose([\t\n TimeShift(p = 0.35, padding_mode = \"zero\", max_shift_pct = 0.25),\t\n TimeStretch(p = 0.35, max_rate = 1.5, min_rate = 0.5),\t\n])\n```\nwith some variations.\n\nIt was hard to pick up right augmetations with right probability & strength and apply it in the correct order.\t\n\t\n**6. Training:**\t\n\t\n- CV5 with group by subject K-fold, 5 models are used on inference\t\n\t\n- 200 or 220 epochs, patience 30 or 35\t\n\t\n- `torch.optim.Adam` with `lr = 4e-4` or `lr = 5e-4` and `weight_decay = 3e-4`\t\n\t\n- `CosineAnnealingWarmRestarts`; `LinearLR` + `CosineAnnealingWarmRestarts` for attention-based models\t\n\t\n- one model was trained with focal loss and class weights\t(class weights obviously mean nothing for mixup samples, maybe it can serve as regularization)\t\n\n- one model contains bug in cross-attention (shared weights for 3 different attentions) but works pretty well\n\n**7. Some tricks:**\t\n\t\n- pick best result for each fold among 3 runs\n\n- increase sequence length by 5 on inference (it doesn't improve CV score but effectively decorrelates models)\n\n------------------------------------------------------------------------\n\n## zyz's part\n\n1.1. Using the pipeline structure from [CMI3 Pyroch Baseline+model_add, Aug, folds](https://www.kaggle.com/code/myso1987/cmi3-pyroch-baseline-model-add-aug-folds?scriptVersionId=248310337)\n\n1.2. Fixing many bugs improved my score to 0.85.\n\n2. Models: simple combinations of RNN and CNN1D.\n\n3. Feature engineering using pytorch, which avoids problem of consistency but makes the training slower.\n\n4. Basic preprocessing mentioned by most teams: fillnan, handedness, etc.\n\n5. Post-processing based on data structure.\n\n6. Augmentation idea from teammates:\n\n6.1. Mask data during training helps.\n\n", "votes": 81 }, { "id": 3281797, "postDate": "2025-09-05T00:49:45.107Z", "content": "Congratulations! I have some question about Devin's part's Preprocessing.
\n\n\nRemoved subjects who wore device incorrectly from both training and validation
\n
How do you know that subjects wore device incorrectly?
\n\n\nFor TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)
\n
How do you guy do this? I mean, TOF is a matrix 8x8, using 2x2 square so it must need 16 squares. Or if you do with stride=2?
", "rawMarkdown": "Congratulations! I have some question about Devin's part's Preprocessing.\n> Removed subjects who wore device incorrectly from both training and validation\n\nHow do you know that subjects wore device incorrectly?\n> For TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)\n\nHow do you guy do this? I mean, TOF is a matrix 8x8, using 2x2 square so it must need 16 squares. Or if you do with stride=2?", "votes": 3 }, { "id": 3287738, "postDate": "2025-09-12T06:24:00.133Z", "content": "Congrats on the 1st place! Really impressive ensemble design and feature engineering across IMU, TOF, and THM.
\nI had a question regarding CV and data imbalance: since some classes/orientations are underrepresented, did you find that group K-fold by subject was sufficient, or did you try any balancing strategies to make sure rare cases were well-covered during validation?
", "rawMarkdown": "Congrats on the 1st place! Really impressive ensemble design and feature engineering across IMU, TOF, and THM.\n\nI had a question regarding CV and data imbalance: since some classes/orientations are underrepresented, did you find that group K-fold by subject was sufficient, or did you try any balancing strategies to make sure rare cases were well-covered during validation?", "votes": 1, "replies": [ { "id": 3287751, "postDate": "2025-09-12T07:15:39.703Z", "content": "I used K-fold by subject from the very beginning and didn't try anything else.
", "rawMarkdown": "I used K-fold by subject from the very beginning and didn't try anything else." } ] }, { "id": 3280951, "postDate": "2025-09-03T11:33:36.080Z", "content": "Congratz !
\nI have some questions :)
\nThanks! Exact pure score without post-processing for best submit is unknown, we literally had no submits left to check. Our sub-ensembles at the team merging moment scored like 0.873-0.874 (public scores) and were significantly improved further. We used optimization heuristics, targets/positions distribution information was explicitly available only for small fraction of base models (72-classes).
", "rawMarkdown": "Thanks! Exact pure score without post-processing for best submit is unknown, we literally had no submits left to check. Our sub-ensembles at the team merging moment scored like 0.873-0.874 (public scores) and were significantly improved further. We used optimization heuristics, targets/positions distribution information was explicitly available only for small fraction of base models (72-classes).", "votes": 1, "replies": [ { "id": 3280982, "postDate": "2025-09-03T11:45:44.377Z", "content": "Thanks for the quick replies.
\nLooks like we have comparable ensembles without PP but your PP is stronger !
Congratulations! I have a question about your scales. I also train on no scales data which is better for my model but the scaler data is also better for my inference. I don't know why it will work.
", "rawMarkdown": "Congratulations! I have a question about your scales. I also train on no scales data which is better for my model but the scaler data is also better for my inference. I don't know why it will work.", "votes": 2, "replies": [ { "id": 3281315, "postDate": "2025-09-04T03:37:20.263Z", "content": "It is more about the initialization of the neural network, which is more related to lucky seed. The scale of the provided data is already very good, just divide by 10 is enough.
", "rawMarkdown": "It is more about the initialization of the neural network, which is more related to lucky seed. The scale of the provided data is already very good, just divide by 10 is enough.", "votes": 1, "replies": [ { "id": 3281321, "postDate": "2025-09-04T03:47:11.770Z", "content": "which more related to lucky seed.\n@yuanzhezhou may i ask which seed value or values do you use in traning ?
", "rawMarkdown": "```python\n\nwhich is more related to lucky seed.\n\n```\n@yuanzhezhou may i ask which seed value or values do you use in traning ?" }, { "id": 3281330, "postDate": "2025-09-04T04:23:54.630Z", "content": "I did not change the seed use by the public notebook, which is 42. it is important not to tune seed for better score
", "rawMarkdown": "I did not change the seed use by the public notebook, which is 42. it is important not to tune seed for better score", "votes": 2 } ] }, { "id": 3281353, "postDate": "2025-09-04T05:56:27.460Z", "content": "Scaled data on train + not scaled on inference could work only if scaling changes data to a very small extent.
", "rawMarkdown": "Scaled data on train + not scaled on inference could work only if scaling changes data to a very small extent.", "votes": 2 } ] }, { "id": 3280988, "postDate": "2025-09-03T12:01:00.963Z", "content": "Congratulations! I am glad my augmentations worked for you. For me the one that worked the best (besides mixup) was adding random peaks or valleys. This is similar to masking. Another good one is channel dropout :)
", "rawMarkdown": "Congratulations! I am glad my augmentations worked for you. For me the one that worked the best (besides mixup) was adding random peaks or valleys. This is similar to masking. Another good one is channel dropout :)", "votes": 2, "replies": [ { "id": 3280994, "postDate": "2025-09-03T12:10:29.173Z", "content": "Thanks! Your aug notebook helped me make progress when I got stuck at some moment.
", "rawMarkdown": "Thanks! Your aug notebook helped me make progress when I got stuck at some moment.", "votes": 1 } ] }, { "id": 3281833, "postDate": "2025-09-05T03:47:15.280Z", "content": "Congrats! Do you mean bugs in my code? If so, could you share what you fixed?
", "rawMarkdown": "Congrats! Do you mean bugs in my code? If so, could you share what you fixed?", "replies": [ { "id": 3281842, "postDate": "2025-09-05T04:21:00.827Z", "content": "Hello, it is a good notebook. It was the only pytorch notebook that could train&submit to get good score at that time. The problem is that, many details are limitting further development. For example, ema is never used for validation, Scaler should not use global statistics, etc.
", "rawMarkdown": "Hello, it is a good notebook. It was the only pytorch notebook that could train&submit to get good score at that time. The problem is that, many details are limitting further development. For example, ema is never used for validation, Scaler should not use global statistics, etc.", "votes": 1 } ] }, { "id": 3281603, "postDate": "2025-09-04T14:53:55.727Z", "content": "Wow! Congratulations ! that’s great work. It was clear from the beginning that feature engineering would have a significant impact, the dataset has many faces 😅
\nOne question: What was the level of scores for your GRU model ?
Congratulations
", "rawMarkdown": "Congratulations" } ], "comments": [ { "id": 3281797, "author_name": "Sam Bui", "author_url": "", "post_date": "2025-09-05T00:49:45.107000", "content": "Congratulations! I have some question about Devin's part's Preprocessing.
\n\n\nRemoved subjects who wore device incorrectly from both training and validation
\n
How do you know that subjects wore device incorrectly?
\n\n\nFor TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)
\n
How do you guy do this? I mean, TOF is a matrix 8x8, using 2x2 square so it must need 16 squares. Or if you do with stride=2?
", "votes": 3, "replies": [] }, { "id": 3287738, "author_name": "Fae Gaze", "author_url": "", "post_date": "2025-09-12T06:24:00.133000", "content": "Congrats on the 1st place! Really impressive ensemble design and feature engineering across IMU, TOF, and THM.
\nI had a question regarding CV and data imbalance: since some classes/orientations are underrepresented, did you find that group K-fold by subject was sufficient, or did you try any balancing strategies to make sure rare cases were well-covered during validation?
", "votes": 1, "replies": [ { "id": 3287751, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-09-12T07:15:39.703000", "content": "I used K-fold by subject from the very beginning and didn't try anything else.
", "votes": 0, "replies": [] } ] }, { "id": 3280951, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-09-03T11:33:36.080000", "content": "Congratz !
\nI have some questions :)
\nThanks! Exact pure score without post-processing for best submit is unknown, we literally had no submits left to check. Our sub-ensembles at the team merging moment scored like 0.873-0.874 (public scores) and were significantly improved further. We used optimization heuristics, targets/positions distribution information was explicitly available only for small fraction of base models (72-classes).
", "votes": 1, "replies": [ { "id": 3280982, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-09-03T11:45:44.377000", "content": "Thanks for the quick replies.
\nLooks like we have comparable ensembles without PP but your PP is stronger !
Congratulations! I have a question about your scales. I also train on no scales data which is better for my model but the scaler data is also better for my inference. I don't know why it will work.
", "votes": 2, "replies": [ { "id": 3281315, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-09-04T03:37:20.263000", "content": "It is more about the initialization of the neural network, which is more related to lucky seed. The scale of the provided data is already very good, just divide by 10 is enough.
", "votes": 1, "replies": [ { "id": 3281321, "author_name": "Sangram Patil", "author_url": "", "post_date": "2025-09-04T03:47:11.770000", "content": "which more related to lucky seed.\n@yuanzhezhou may i ask which seed value or values do you use in traning ?
", "votes": 0, "replies": [] }, { "id": 3281330, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-09-04T04:23:54.630000", "content": "I did not change the seed use by the public notebook, which is 42. it is important not to tune seed for better score
", "votes": 2, "replies": [] } ] }, { "id": 3281353, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-09-04T05:56:27.460000", "content": "Scaled data on train + not scaled on inference could work only if scaling changes data to a very small extent.
", "votes": 2, "replies": [] } ] }, { "id": 3280988, "author_name": "moth", "author_url": "", "post_date": "2025-09-03T12:01:00.963000", "content": "Congratulations! I am glad my augmentations worked for you. For me the one that worked the best (besides mixup) was adding random peaks or valleys. This is similar to masking. Another good one is channel dropout :)
", "votes": 2, "replies": [ { "id": 3280994, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-09-03T12:10:29.173000", "content": "Thanks! Your aug notebook helped me make progress when I got stuck at some moment.
", "votes": 1, "replies": [] } ] }, { "id": 3281833, "author_name": "MYSO", "author_url": "", "post_date": "2025-09-05T03:47:15.280000", "content": "Congrats! Do you mean bugs in my code? If so, could you share what you fixed?
", "votes": 0, "replies": [ { "id": 3281842, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-09-05T04:21:00.827000", "content": "Hello, it is a good notebook. It was the only pytorch notebook that could train&submit to get good score at that time. The problem is that, many details are limitting further development. For example, ema is never used for validation, Scaler should not use global statistics, etc.
", "votes": 1, "replies": [] } ] }, { "id": 3281603, "author_name": "Ricardo Pirabed A.A.", "author_url": "", "post_date": "2025-09-04T14:53:55.727000", "content": "Wow! Congratulations ! that’s great work. It was clear from the beginning that feature engineering would have a significant impact, the dataset has many faces 😅
\nOne question: What was the level of scores for your GRU model ?
Congratulations
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3280919": "**Links to code:**\n- [Training code on github](https://github.com/statist-bhfz/kaggle_cmi_1st_place_solution)\n- [Inference notebook](https://www.kaggle.com/code/yuanzhezhou/final-sub-cmi3)\n\n## Solution Writeup\n\nCongratulations to all the winners and a big respect to the organizers for putting together such a challenging and rewarding competition. \nSpecial thanks to my teammates [yuanzhe zhou](https://www.kaggle.com/yuanzhezhou) and [Devin Anzelmo](https://www.kaggle.com/devinanzelmo) for the great collaboration and team spirit.\n\n## Solution Overview\n\n- Large ensembles for IMU-only and all-features samples, assembled as weighted sum of logits\n\n- Post-processing to take into account limited number of samples with each class and with each orientation per subject (already described [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544))\n\n- Additional models for orientation prediction\n\n------------------------------------------------------------------------\n\n## Devin's part\n\n**1. Preprocessing/feature engineering:**\n\n- Removed subjects who wore device incorrectly from both training and validation\n\n- For TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)\n\n- Some models trained with temporal difference of above TOF features and thermal\n\n- IMU: raw 7 + temporal diff of `acc`, rotational diff as posted in notebooks, as well as magnitude of `acc`/`rot`; 16 total\n\n- preprocess left handed to match right handed for IMU and THM/TOF. Flip `acc_x`, convert `rot` to xyz euler and flip sign for y and z\n\n- robust scaler \n\n**2. Augmentation:**\n\n- up scaling/down scaling all features \n\n- random noise \n\n**3. Models:**\n\n- CNN GRU with different stems for different features. Feature extractor is four branches of CNN with no pooling\n\n- Pure CNN. Same feature extractor as above but three blocks of residual at end with expanding kernal (5, 9, 13)\n\n- dense GRU, scores lower but helps a little in ensemble\n\n- some self excitation models, some not combining helps\n\n- 3D CNN for TOF data scores about 0.83 but blends well\n\n**4. Training:**\n\n- batch size 64\n\n- train for 60-80 (depending on model) iters with AdamW `lr = 0.0005`\n\n**5. Ensemble:**\n\n- blend IMU models with all-features models \n\n- blend TOF-only model improves all data score\n\n------------------------------------------------------------------------\n\n## Ogurtsov's part\n\n**1. Filtering data:**\t\n\t\n- [all models] remove sequence SEQ_011975 without gesture\t\n\t\n- [all models] remove sequences with gesture/total ratio < 0.2 (this subset includes almost all shortest sequences)\t\n\t\n- [models with THM and TOF data] remove sequences with TOF completely missing\t(missing values patterns in TOF - 0%, 20% or 100%; 20% missing is ok)\t\n\n- few models were trained without \"problematic\" subjects SUBJ_045235 and SUBJ_019262\n\t\n**2. Preprocessing:**\n\t\n- fill quaternion missing values, if possible\t\n\t\n- mirror data for left-handed subjects \n\n- replace -1 TOF values with 500 because -1 encodes large distance\t\n\t\n- ultimate filling NaNs with `seq.ffill().bfill().fillna(0)`\n\t\n- pad/crop by 94 percentile in each fold (~120 max seq length)\t\n\t\n- no scalers - it gave the largest improvement at early stage\t\n\t\n**3. Feature engineering:**\t\n\t\n- nothing for THM and TOF\t\n\t\n- 35 IMU features:\t\n```\n['acc_x', 'acc_y', 'acc_z', 'acc_mag', 'acc_mag_jerk', 'jerk_x', 'jerk_y', 'jerk_z', 'jerk_magnitude', 'acc_xy_corr', 'acc_xz_corr', 'acc_yz_corr',\t\nrot_w', 'rot_x', 'rot_y', 'rot_z', 'rot_angle', 'rot_angle_vel', 'angular_vel_x', 'angular_vel_y', 'angular_vel_z', 'angular_vel_magnitude', 'angular_distance',\t\nacc_x2', 'acc_y2', 'acc_z2', 'acc_mag2', 'acc_mag_jerk2', 'jerk_x2', 'jerk_y2', 'jerk_z2', 'jerk_magnitude2', 'acc_xy_corr2', 'acc_xz_corr2', 'acc_yz_corr2']\n```\n\n`acc_x2` and other `***2` features are important part of the solution. \nThese features are calculated on values transformed by `remove_gravity_from_acc()`.\t\n\t\n**4. Models:**\t\n\t\n- bunch of models with different architectures for training on IMU-only data and similar models for training with all features\t\n\t\n- main differences in time dimension processing: LSTM-, attention- or CNN-based\t\n\t\n- key part - independent processing groups of features at first layers\t\n\t\n**5. Augmentations:**\t\n\t\n- mixup with `MIXUP_ALPHA = 0.4`\n\t\n- augmentation from [CMI | Sequence Data Augmentation](https://www.kaggle.com/code/alejopaullier/cmi-sequence-data-augmentation) (thanks [moth](https://www.kaggle.com/alejopaullier)):\t\n```\ntransforms = Compose([\t\n TimeShift(p = 0.35, padding_mode = \"zero\", max_shift_pct = 0.25),\t\n TimeStretch(p = 0.35, max_rate = 1.5, min_rate = 0.5),\t\n])\n```\nwith some variations.\n\nIt was hard to pick up right augmetations with right probability & strength and apply it in the correct order.\t\n\t\n**6. Training:**\t\n\t\n- CV5 with group by subject K-fold, 5 models are used on inference\t\n\t\n- 200 or 220 epochs, patience 30 or 35\t\n\t\n- `torch.optim.Adam` with `lr = 4e-4` or `lr = 5e-4` and `weight_decay = 3e-4`\t\n\t\n- `CosineAnnealingWarmRestarts`; `LinearLR` + `CosineAnnealingWarmRestarts` for attention-based models\t\n\t\n- one model was trained with focal loss and class weights\t(class weights obviously mean nothing for mixup samples, maybe it can serve as regularization)\t\n\n- one model contains bug in cross-attention (shared weights for 3 different attentions) but works pretty well\n\n**7. Some tricks:**\t\n\t\n- pick best result for each fold among 3 runs\n\n- increase sequence length by 5 on inference (it doesn't improve CV score but effectively decorrelates models)\n\n------------------------------------------------------------------------\n\n## zyz's part\n\n1.1. Using the pipeline structure from [CMI3 Pyroch Baseline+model_add, Aug, folds](https://www.kaggle.com/code/myso1987/cmi3-pyroch-baseline-model-add-aug-folds?scriptVersionId=248310337)\n\n1.2. Fixing many bugs improved my score to 0.85.\n\n2. Models: simple combinations of RNN and CNN1D.\n\n3. Feature engineering using pytorch, which avoids problem of consistency but makes the training slower.\n\n4. Basic preprocessing mentioned by most teams: fillnan, handedness, etc.\n\n5. Post-processing based on data structure.\n\n6. Augmentation idea from teammates:\n\n6.1. Mask data during training helps.\n\n", "3281797": "Congratulations! I have some question about Devin's part's Preprocessing.\n> Removed subjects who wore device incorrectly from both training and validation\n\nHow do you know that subjects wore device incorrectly?\n> For TOF subsample by taking nine 2x2 squares and calculating mean for each (turn -1 to null first)\n\nHow do you guy do this? I mean, TOF is a matrix 8x8, using 2x2 square so it must need 16 squares. Or if you do with stride=2?", "3287738": "Congrats on the 1st place! Really impressive ensemble design and feature engineering across IMU, TOF, and THM.\n\nI had a question regarding CV and data imbalance: since some classes/orientations are underrepresented, did you find that group K-fold by subject was sufficient, or did you try any balancing strategies to make sure rare cases were well-covered during validation?", "3280951": "Congratz !\n\nI have some questions :)\n- How does your ensemble perform without PP ?\n- How did you implement PP ? Using optimization heuristics / using a transformer model ?", "3281291": "Congratulations! I have a question about your scales. I also train on no scales data which is better for my model but the scaler data is also better for my inference. I don't know why it will work.", "3280988": "Congratulations! I am glad my augmentations worked for you. For me the one that worked the best (besides mixup) was adding random peaks or valleys. This is similar to masking. Another good one is channel dropout :)", "3281833": "Congrats! Do you mean bugs in my code? If so, could you share what you fixed?", "3281603": "Wow! Congratulations ! that’s great work. It was clear from the beginning that feature engineering would have a significant impact, the dataset has many faces 😅\n**One question**: What was the level of scores for your GRU model ?", "3280926": "", "3286789": "Congratulations" }, "complete": true, "source": "existing_discussion", "competition_id": 102335, "competition_slug": "cmi-detect-behavior-with-sensor-data", "team_id": 13896582, "solution_url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/cmi-1st-place-solution" }