![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2514988%2Fa46a932746d245849836b4dd656b2dd7%2Fanil_model_arch.png?generation=1756880030185722&alt=media\")
Big congratulations for all winners and thanks to the organizers! We unfortunately couldn't leverage subject-wise boosting tricks that many teams found out. But we are happy seeing that we got a robust ensemble! Our solution is an equal blend of two pipelines:
\n
Accelerometer-Only Features
\nRotation-Only Features
\nACC-ROT Fusion Features
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2514988%2Ffa896c00ddd19f8bc67176ca76ba4da4%2Fanil_model_arch_layers.png?generation=1756880604880838&alt=media\")
NaN Handling Layers
\nIMU Branch
\nTHM Branch
\nToF Branch
\nThree-Modal Attention
\nBiLSTM + Attention
\nMulti-Task Aux Learning
\nI am going to share how my final pipeline looks like and try to explain how I got there. I did a lot of ablation study in the end showing that a quite simple solution could be competitive.
\nThe dataset in this competition is quite small so it seemed important to represent each sequence in a more standardized way.
\nI did a lot of augmentations on the fly:
\nI did not feel very comfortable using demographics data as there are two few subjects to be meaningfull for training. Nevertheless I tried to incorporate as much knowledge as possible to my models by forcing them to predict as many thing as possible (different heads):
\nLocation, action and child were probably useless.
\nI did little feature engineering:
\nI ended up with a relatively simple architecture, I only used one single model that learns to ignore missing information. I did not have one specific model for IMU only and full sensors: I simply randomly mask the thm and tof feats during training.
\nFirst encode information at the time level
\nThen each stream goes through a gating unit which decides the importance of each stream of sensors:
\nThen I combine all feature by summing them which forces the model to share the same representation for each stream of data:
\n = accel_features + orient_features + temp_features + tof_features\n = gate_accel + gate_orient + gate_temp + gate_tof\n\n = numerator / (denominator + e-)\nThen I have an attention pooling mechanism to go from time series to series representation then basic MLP heads.
\nFollowing the above pipeline I could reach avg CV 85.8 (IMU only 82.9 - all 88.7) with upside down subject flipped correctly.
\nEnsembling a few models with different training parameters could reach 86.4 CV.
My LB never went above 86.6 with my own models, I was never able to overfit public LB, not sure why.
\nSimply merging one 86.5 Anil's sub with one 86.5 single model of mine yielded LB 87.6 so I guess diversity in approaches was important here!
\nThe main reason I did not managed to improve my pipeline on the past month was because of stubbornness and CV distillation. If using the OOF of a model to train a model from scratch following same CV scheme I could easily reach CV 0.88 (LB dropped significantly). It took me almost a month to accept that this was just pure leakage. If I decayed overtime the importance of OOF labels the CV went down but LB went up. I then tried 50 shades of leakage to reduce my CV and improve my LB like a crazy person.
\nI should have:
\nAnyway! Thanks for reading this far, congratulations again to the winners and happy kaggling!
", "messages": [ { "id": "3280773", "postDate": "09/03/2025 06:22:10", "content": "Big congratulations for all winners and thanks to the organizers! We unfortunately couldn't leverage subject-wise boosting tricks that many teams found out. But we are happy seeing that we got a robust ensemble! Our solution is an equal blend of two pipelines:
\n
Accelerometer-Only Features
\nRotation-Only Features
\nACC-ROT Fusion Features
\n
NaN Handling Layers
\nIMU Branch
\nTHM Branch
\nToF Branch
\nThree-Modal Attention
\nBiLSTM + Attention
\nMulti-Task Aux Learning
\nI am going to share how my final pipeline looks like and try to explain how I got there. I did a lot of ablation study in the end showing that a quite simple solution could be competitive.
\nThe dataset in this competition is quite small so it seemed important to represent each sequence in a more standardized way.
\nI did a lot of augmentations on the fly:
\nI did not feel very comfortable using demographics data as there are two few subjects to be meaningfull for training. Nevertheless I tried to incorporate as much knowledge as possible to my models by forcing them to predict as many thing as possible (different heads):
\nLocation, action and child were probably useless.
\nI did little feature engineering:
\nI ended up with a relatively simple architecture, I only used one single model that learns to ignore missing information. I did not have one specific model for IMU only and full sensors: I simply randomly mask the thm and tof feats during training.
\nFirst encode information at the time level
\nThen each stream goes through a gating unit which decides the importance of each stream of sensors:
\nThen I combine all feature by summing them which forces the model to share the same representation for each stream of data:
\n = accel_features + orient_features + temp_features + tof_features\n = gate_accel + gate_orient + gate_temp + gate_tof\n\n = numerator / (denominator + e-)\nThen I have an attention pooling mechanism to go from time series to series representation then basic MLP heads.
\nFollowing the above pipeline I could reach avg CV 85.8 (IMU only 82.9 - all 88.7) with upside down subject flipped correctly.
\nEnsembling a few models with different training parameters could reach 86.4 CV.
My LB never went above 86.6 with my own models, I was never able to overfit public LB, not sure why.
\nSimply merging one 86.5 Anil's sub with one 86.5 single model of mine yielded LB 87.6 so I guess diversity in approaches was important here!
\nThe main reason I did not managed to improve my pipeline on the past month was because of stubbornness and CV distillation. If using the OOF of a model to train a model from scratch following same CV scheme I could easily reach CV 0.88 (LB dropped significantly). It took me almost a month to accept that this was just pure leakage. If I decayed overtime the importance of OOF labels the CV went down but LB went up. I then tried 50 shades of leakage to reduce my CV and improve my LB like a crazy person.
\nI should have:
\nAnyway! Thanks for reading this far, congratulations again to the winners and happy kaggling!
", "rawMarkdown": "Big congratulations for all winners and thanks to the organizers! We unfortunately couldn't leverage subject-wise boosting tricks that many teams found out. But we are happy seeing that we got a robust ensemble! Our solution is an equal blend of two pipelines:\n\n---\n\n# 1. Anil's Pipeline\n\n## 1.1. Modelling\n\n- **Mode 1: IMU Only** - Uses accelerometer and rotation features when THM/ToF sensors are unavailable\n- **Mode 2: Full Sensor Suite** - Leverages all three sensor modalities (IMU + THM + ToF)\n\n## 1.2. Features\n**Accelerometer-Only Features**\n- **Raw acceleration:** acc_x, acc_y, acc_z\n- **Jerk (acceleration derivatives):** acc_jerk_x, acc_jerk_y, acc_jerk_z\n\n**Rotation-Only Features**\n- **6D rotation matrix differences:** rot_6d_diff_col1_x/y/z, rot_6d_diff_col2_x/y/z\n- **Absolute 6D rotation differences:** abs_rot_6d_diff_col1_x/y/z, abs_rot_6d_diff_col2_x/y/z\n- **Relative rotation between frames:** rot_rel_6d_col1_x/y/z, rot_rel_6d_col2_x/y/z, abs_rot_rel_6d_col1_x/y/z, abs_rot_rel_6d_col2_x/y/z\n\n**ACC-ROT Fusion Features**\n- **Gravity Estimation:** gravity_est_x/y/z, gravity_est_jerk_x/y/z\n- **Linear Acceleration:** linear_acc_x/y/z, abs_linear_acc_x/y/z\n- **Jerk Features:** jerk_x/y/z\n- **Subject-Aligned Features:** acc_tangential/lateral/vertical, jerk_tangential/lateral/vertical\n- **Tilt Calculation:** tilt, tilt_diff\n\n## 1.3. Architecture\n\n\n**NaN Handling Layers**\n- **IMU/THM**: Learnable embedding vectors for missing values\n- **ToF**: Single tunable parameter broadcasted to all missing pixels\n\n**IMU Branch**\n- Two residual CNN blocks with SE\n- Kernel sizes: 3 and 5\n- Output: 128 features per timestep\n- Mode-specific dropout: 0.3 (Mode 1), 0.1 (Mode 2)\n\n**THM Branch** \n- Two residual CNN blocks with SE\n- Output: 64 features per timestep\n\n**ToF Branch**\n- Spatial attention\n- Depthwise separable convolutions\n- Residual connections\n- Multi-scale feature extraction\n- Input: (5, 8, 8) per timestep → Output: 128 features\n\n**Three-Modal Attention**\n- **Cross-modal attention** between IMU, ToF, and THM\n- Bidirectional\n- Residual connections\n\n**BiLSTM + Attention**\n- 128 hidden units, bidirectional (256 total)\n- Attention pooling for sequence-level tasks\n- Dropout: 0.4\n\n**Multi-Task Aux Learning**\n- **Gesture** (main): 20 classes\n- **Orientation**: 4 classes \n- **Handedness**: 2 classes\n- **Behavior** (per-timestep): 4 classes\n\n---\n\n# 2. Optimo's Pipeline\n\nI am going to share how my final pipeline looks like and try to explain how I got there. I did a lot of ablation study in the end showing that a quite simple solution could be competitive.\n\n## 2.1 Data normalization\n\nThe dataset in this competition is quite small so it seemed important to represent each sequence in a more standardized way.\n\n- **tof alignement**: as I was using 2CNN for tof feature extraction (see architecture later) it felt important to have pixels across sensors to correspond. So following the scheme of the device I rotate 90° tof sensors 3 and 5.\n- **left handed alignement**: I did exactly the same thing as describe by @tatamikenn [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603566) but his drawing abilities are far better than mine so I won't compete here! Transformation consists in change sign of correct acc axis and rot axis, as well as switch sensors 3 and 5 (thm and tof) as well as flipping tof images. The idea is that when a right handed person bring is hand to his head/body, the acceleration is from right to left while it's the opposite for left handed person.\n- **upside down alignements**: it became obvious when looking at my oof scores that something wrong was happening to \"SUBJ_045235\", \"SUBJ_019262\". After a lot of data visualisation I came to the conclusion that they were wearing the device upside down. When applying the corresponding transfos (switching sensors, negating some acc and rot axis) those two subjects' scores went back to normal. To make sure that my transfo was correct I trained an \"upside down detector\": randomly applying the transfo to all sequences and predicting whether it was used or not. The model was very good at it and correctly said that those two subjects were indeed upside down. Problem is you do not have a similar info as 'handedness' to know that someone is wearing the device wrong. So I tried a sub with first using the detector and applying the transfo if needed before making my prediction: got a similar score (it was the two digit LB era at that moment). I thought \"the test set is clean and everybody is wearing their device the correct way\", so I just flipped those two subjects once and for all and continue improving my CV. **BIG MISTAKE**: some private subjects are indeed wearing their device upside down and the detector worked well (LB: 863 - PLB: 857). I should have done it the correct way: do not allow my CV to improve after altering the data myself, my solution should have included the detector in order to improve CV and I should have trusted it!\n- **sequence length**: all sequence are right cropped/padded to 96 (got similar results 64 or 128)\n\n## 2.2 Data augmentation\n\nI did a lot of augmentations on the fly:\n- Alignement aware MIXUP: I randomly average sequences after aligning the start of the gesture so that averaging made more sense\n- random rotaion: when looking at the videos of the lady performing the gestures on Youtube, you can see that the device is not always perfectly worn. So I add random small rotation (+-30°) around Z-axis to quaternions.\n- random stretching\n- randomly ignore some points at the beginning of sequences\n\n## 2.3 Losses\n\nI did not feel very comfortable using demographics data as there are two few subjects to be meaningfull for training. Nevertheless I tried to incorporate as much knowledge as possible to my models by forcing them to predict as many thing as possible (different heads):\n- 18 gestures\n- binary target\n- orientation\n- location: a custom mix of gestures into 4 locations (leg - face - neck - front)\n- action: a custom mix of gestures into 5 actions (pull hair - pinch - scratch - write - other)\n- is child\n\nLocation, action and child were probably useless.\n\n## 2.4 Feature engineering\n\nI did little feature engineering:\n- acceleration: original accelerations, temporal diff 1, acc_mal_jerk, normalized position in sequence (0 to 1)\n- quaternions: orig quaternions, rotation angle and angle velocity (x, y, z) and distances, angular velocity norm and 10 window rolling mean of it\n- thm: original thm\n- rof: original tof\n\n## 2.5 Architecture\n\nI ended up with a relatively simple architecture, I only used one single model that learns to ignore missing information. I did not have one specific model for IMU only and full sensors: I simply randomly mask the thm and tof feats during training.\n\nFirst encode information at the time level\n- independant conv1D for acc, quats and thm for time level encoding (with kernels 3-5-7) with N_OUT features for each\n- 2D CNN with 5 channels transforming 5x8x8 images to N_OUT (no time series info shared)\n\nThen each stream goes through a gating unit which decides the importance of each stream of sensors:\n```\nclass GatingUnit_v2(nn.Module):\n def __init__(self, input_dim, hidden_dim=64):\n super().__init__()\n\n self.pool = nn.AdaptiveAvgPool1d(1)\n self.gater = nn.Sequential(\n nn.Linear(input_dim, hidden_dim),\n nn.ReLU(),\n nn.Linear(hidden_dim, 1),\n nn.Sigmoid()\n )\n \n def forward(self, x):\n # x: (batch, feat_dim, seq_len)\n pooled_x = self.pool(x).squeeze(-1) # (batch, feat_dim)\n gate = self.gater(pooled_x).unsqueeze(-1) # (batch, 1, 1)\n gated_x = gate * x\n return gated_x, gate\n``` \n\nThen I combine all feature by summing them which forces the model to share the same representation for each stream of data:\n```\nnumerator = accel_features + orient_features + temp_features + tof_features\ndenominator = gate_accel + gate_orient + gate_temp + gate_tof\n\n combined_feats = numerator / (denominator + 1e-6)\n```\n\nThen I have an attention pooling mechanism to go from time series to series representation then basic MLP heads.\n\n\n## 2.6 CV scores\n\nFollowing the above pipeline I could reach avg CV 85.8 (IMU only 82.9 - all 88.7) with upside down subject flipped correctly.\nEnsembling a few models with different training parameters could reach 86.4 CV.\n\nMy LB never went above 86.6 with my own models, I was never able to overfit public LB, not sure why.\n\nSimply merging one 86.5 Anil's sub with one 86.5 single model of mine yielded LB 87.6 so I guess diversity in approaches was important here!\n\n## 2.7 Few mistakes\n\nThe main reason I did not managed to improve my pipeline on the past month was because of stubbornness and CV distillation. If using the OOF of a model to train a model from scratch following same CV scheme I could easily reach CV 0.88 (LB dropped significantly). It took me almost a month to accept that this was just pure leakage. If I decayed overtime the importance of OOF labels the CV went down but LB went up. I then tried 50 shades of leakage to reduce my CV and improve my LB like a crazy person.\n\nI should have:\n- accepted that distillation using OOF creates a powefull leakage\n- compute my CV without allowing to arbitrarily put two subject upside down (I may try my upside down detector as late sub but I suspect a huge improvement on PLB)\n- prob LB to exploit all available info as a real Kaggle champion should\n\n\nAnyway! Thanks for reading this far, congratulations again to the winners and happy kaggling!", "votes": null }, { "id": "3280781", "postDate": "09/03/2025 06:29:19", "content": "Curious about the multi-task experiment results
", "rawMarkdown": "Curious about the multi-task experiment results", "votes": null }, { "id": "3280783", "postDate": "09/03/2025 06:30:44", "content": "Congratulations! Was there any intuition behind the features or was it just random testing?
", "rawMarkdown": "Congratulations! Was there any intuition behind the features or was it just random testing?", "votes": null }, { "id": "3280788", "postDate": "09/03/2025 06:35:32", "content": "Great solution and strong models! Did 6d quaternions representations give a boost to the score? I also came across the paper https://arxiv.org/pdf/1812.07035 and tried many different representations for the quaternions, but in the end the original raw values were the best
", "rawMarkdown": "Great solution and strong models! Did 6d quaternions representations give a boost to the score? I also came across the paper https://arxiv.org/pdf/1812.07035 and tried many different representations for the quaternions, but in the end the original raw values were the best", "votes": null }, { "id": "3280789", "postDate": "09/03/2025 06:35:50", "content": "Congrats! Nice to see you managed to get the ensemble working. It didn’t work in my case at all; my final submission was a single model 😅
", "rawMarkdown": "Congrats! Nice to see you managed to get the ensemble working. It didn’t work in my case at all; my final submission was a single model 😅", "votes": null }, { "id": "3280795", "postDate": "09/03/2025 06:50:32", "content": "Orientation/behavior loss boosted your validation? How much?
\nI tried both and they only hurt me.
for me when I sarted to use orientation, my CV and LB improved 0.01
", "rawMarkdown": "for me when I sarted to use orientation, my CV and LB improved 0.01", "votes": null }, { "id": "3280886", "postDate": "09/03/2025 09:52:46", "content": "I think I started the competition by doing that so I can't say :)
", "rawMarkdown": "I think I started the competition by doing that so I can't say :)", "votes": null }, { "id": "3280921", "postDate": "09/03/2025 11:15:57", "content": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds,and seems like I should've done that. 😅
", "rawMarkdown": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds,and seems like I should've done that. 😅", "votes": null }, { "id": "3280922", "postDate": "09/03/2025 11:16:46", "content": "They boosted my CV like 0.002-3 when compared using roll/pitch/yaw and their diffs.
", "rawMarkdown": "They boosted my CV like 0.002-3 when compared using roll/pitch/yaw and their diffs.", "votes": null }, { "id": "3280923", "postDate": "09/03/2025 11:17:35", "content": "We have very different approaches in terms of both features and architectures. I'm also glad that worked out well for us. 😃
", "rawMarkdown": "We have very different approaches in terms of both features and architectures. I'm also glad that worked out well for us. 😃", "votes": null }, { "id": "3280925", "postDate": "09/03/2025 11:17:54", "content": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds.
", "rawMarkdown": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds.", "votes": null }, { "id": "3280927", "postDate": "09/03/2025 11:18:21", "content": "I've converged to this feature subset by iterating on both CV/LB feedback.
", "rawMarkdown": "I've converged to this feature subset by iterating on both CV/LB feedback.", "votes": null }, { "id": "3280985", "postDate": "09/03/2025 11:57:20", "content": "Congrats!!
", "rawMarkdown": "Congrats!!", "votes": null }, { "id": "3280999", "postDate": "09/03/2025 12:25:34", "content": "I think you misunderstood me. My question was how you thought of what features to test. Was it based on some sort of logic or just random guessing?
", "rawMarkdown": "I think you misunderstood me. My question was how you thought of what features to test. Was it based on some sort of logic or just random guessing?", "votes": null }, { "id": "3281002", "postDate": "09/03/2025 12:31:57", "content": "Derivatives and subject-referenced representations performed better than the absolute position values since the subjects’ position and orientation can vary a lot. When you think that way you will be quickly converged to a subset such like this, consists of many first and second derivatives of relative positional features.
", "rawMarkdown": "Derivatives and subject-referenced representations performed better than the absolute position values since the subjects’ position and orientation can vary a lot. When you think that way you will be quickly converged to a subset such like this, consists of many first and second derivatives of relative positional features.", "votes": null }, { "id": "3281007", "postDate": "09/03/2025 12:35:33", "content": "I see, thanks for the quick reply!
", "rawMarkdown": "I see, thanks for the quick reply!", "votes": null }, { "id": "3281670", "postDate": "09/04/2025 16:38:52", "content": "\n\n\nall sequence are right cropped
\n
Are you really cropping right?
", "rawMarkdown": "optimo \n> all sequence are right cropped\n\nAre you really cropping right?", "votes": null }, { "id": "3281697", "postDate": "09/04/2025 17:55:21", "content": "I keep the right part (end of sequence) and crop if necessary, so I should probably say a left crop sorry! :)
", "rawMarkdown": "I keep the right part (end of sequence) and crop if necessary, so I should probably say a left crop sorry! :)", "votes": null } ], "comments": [ { "id": 3280781, "author_name": "tom99763", "author_url": "", "post_date": "09/03/2025 06:29:19", "content": "Curious about the multi-task experiment results
", "votes": null, "replies": [ { "id": 3280925, "author_name": "nlztrk", "author_url": "", "post_date": "09/03/2025 11:17:54", "content": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds.
", "votes": null, "replies": [] } ] }, { "id": 3280783, "author_name": "mahoganybuttstrings", "author_url": "", "post_date": "09/03/2025 06:30:44", "content": "Congratulations! Was there any intuition behind the features or was it just random testing?
", "votes": null, "replies": [ { "id": 3280927, "author_name": "nlztrk", "author_url": "", "post_date": "09/03/2025 11:18:21", "content": "I've converged to this feature subset by iterating on both CV/LB feedback.
", "votes": null, "replies": [ { "id": 3280999, "author_name": "mahoganybuttstrings", "author_url": "", "post_date": "09/03/2025 12:25:34", "content": "I think you misunderstood me. My question was how you thought of what features to test. Was it based on some sort of logic or just random guessing?
", "votes": null, "replies": [ { "id": 3281002, "author_name": "nlztrk", "author_url": "", "post_date": "09/03/2025 12:31:57", "content": "Derivatives and subject-referenced representations performed better than the absolute position values since the subjects’ position and orientation can vary a lot. When you think that way you will be quickly converged to a subset such like this, consists of many first and second derivatives of relative positional features.
", "votes": null, "replies": [ { "id": 3281007, "author_name": "mahoganybuttstrings", "author_url": "", "post_date": "09/03/2025 12:35:33", "content": "I see, thanks for the quick reply!
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3280788, "author_name": "l1ghtsource", "author_url": "", "post_date": "09/03/2025 06:35:32", "content": "Great solution and strong models! Did 6d quaternions representations give a boost to the score? I also came across the paper https://arxiv.org/pdf/1812.07035 and tried many different representations for the quaternions, but in the end the original raw values were the best
", "votes": null, "replies": [ { "id": 3280922, "author_name": "nlztrk", "author_url": "", "post_date": "09/03/2025 11:16:46", "content": "They boosted my CV like 0.002-3 when compared using roll/pitch/yaw and their diffs.
", "votes": null, "replies": [] } ] }, { "id": 3280789, "author_name": "datafan07", "author_url": "", "post_date": "09/03/2025 06:35:50", "content": "Congrats! Nice to see you managed to get the ensemble working. It didn’t work in my case at all; my final submission was a single model 😅
", "votes": null, "replies": [ { "id": 3280923, "author_name": "nlztrk", "author_url": "", "post_date": "09/03/2025 11:17:35", "content": "We have very different approaches in terms of both features and architectures. I'm also glad that worked out well for us. 😃
", "votes": null, "replies": [] } ] }, { "id": 3280795, "author_name": "shlomoron", "author_url": "", "post_date": "09/03/2025 06:50:32", "content": "Orientation/behavior loss boosted your validation? How much?
\nI tried both and they only hurt me.
for me when I sarted to use orientation, my CV and LB improved 0.01
", "votes": null, "replies": [ { "id": 3280886, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 09:52:46", "content": "I think I started the competition by doing that so I can't say :)
", "votes": null, "replies": [] } ] }, { "id": 3280921, "author_name": "nlztrk", "author_url": "", "post_date": "09/03/2025 11:15:57", "content": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds,and seems like I should've done that. 😅
", "votes": null, "replies": [] } ] }, { "id": 3280985, "author_name": "ahmetemirakdogan", "author_url": "", "post_date": "09/03/2025 11:57:20", "content": "Congrats!!
", "votes": null, "replies": [] }, { "id": 3281670, "author_name": "nazarov", "author_url": "", "post_date": "09/04/2025 16:38:52", "content": "\n\n\nall sequence are right cropped
\n
Are you really cropping right?
", "votes": null, "replies": [ { "id": 3281697, "author_name": "optimo", "author_url": "", "post_date": "09/04/2025 17:55:21", "content": "I keep the right part (end of sequence) and crop if necessary, so I should probably say a left crop sorry! :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3280773": "Big congratulations for all winners and thanks to the organizers! We unfortunately couldn't leverage subject-wise boosting tricks that many teams found out. But we are happy seeing that we got a robust ensemble! Our solution is an equal blend of two pipelines:\n\n---\n\n# 1. Anil's Pipeline\n\n## 1.1. Modelling\n\n- **Mode 1: IMU Only** - Uses accelerometer and rotation features when THM/ToF sensors are unavailable\n- **Mode 2: Full Sensor Suite** - Leverages all three sensor modalities (IMU + THM + ToF)\n\n## 1.2. Features\n**Accelerometer-Only Features**\n- **Raw acceleration:** acc_x, acc_y, acc_z\n- **Jerk (acceleration derivatives):** acc_jerk_x, acc_jerk_y, acc_jerk_z\n\n**Rotation-Only Features**\n- **6D rotation matrix differences:** rot_6d_diff_col1_x/y/z, rot_6d_diff_col2_x/y/z\n- **Absolute 6D rotation differences:** abs_rot_6d_diff_col1_x/y/z, abs_rot_6d_diff_col2_x/y/z\n- **Relative rotation between frames:** rot_rel_6d_col1_x/y/z, rot_rel_6d_col2_x/y/z, abs_rot_rel_6d_col1_x/y/z, abs_rot_rel_6d_col2_x/y/z\n\n**ACC-ROT Fusion Features**\n- **Gravity Estimation:** gravity_est_x/y/z, gravity_est_jerk_x/y/z\n- **Linear Acceleration:** linear_acc_x/y/z, abs_linear_acc_x/y/z\n- **Jerk Features:** jerk_x/y/z\n- **Subject-Aligned Features:** acc_tangential/lateral/vertical, jerk_tangential/lateral/vertical\n- **Tilt Calculation:** tilt, tilt_diff\n\n## 1.3. Architecture\n\n\n**NaN Handling Layers**\n- **IMU/THM**: Learnable embedding vectors for missing values\n- **ToF**: Single tunable parameter broadcasted to all missing pixels\n\n**IMU Branch**\n- Two residual CNN blocks with SE\n- Kernel sizes: 3 and 5\n- Output: 128 features per timestep\n- Mode-specific dropout: 0.3 (Mode 1), 0.1 (Mode 2)\n\n**THM Branch** \n- Two residual CNN blocks with SE\n- Output: 64 features per timestep\n\n**ToF Branch**\n- Spatial attention\n- Depthwise separable convolutions\n- Residual connections\n- Multi-scale feature extraction\n- Input: (5, 8, 8) per timestep → Output: 128 features\n\n**Three-Modal Attention**\n- **Cross-modal attention** between IMU, ToF, and THM\n- Bidirectional\n- Residual connections\n\n**BiLSTM + Attention**\n- 128 hidden units, bidirectional (256 total)\n- Attention pooling for sequence-level tasks\n- Dropout: 0.4\n\n**Multi-Task Aux Learning**\n- **Gesture** (main): 20 classes\n- **Orientation**: 4 classes \n- **Handedness**: 2 classes\n- **Behavior** (per-timestep): 4 classes\n\n---\n\n# 2. Optimo's Pipeline\n\nI am going to share how my final pipeline looks like and try to explain how I got there. I did a lot of ablation study in the end showing that a quite simple solution could be competitive.\n\n## 2.1 Data normalization\n\nThe dataset in this competition is quite small so it seemed important to represent each sequence in a more standardized way.\n\n- **tof alignement**: as I was using 2CNN for tof feature extraction (see architecture later) it felt important to have pixels across sensors to correspond. So following the scheme of the device I rotate 90° tof sensors 3 and 5.\n- **left handed alignement**: I did exactly the same thing as describe by @tatamikenn [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603566) but his drawing abilities are far better than mine so I won't compete here! Transformation consists in change sign of correct acc axis and rot axis, as well as switch sensors 3 and 5 (thm and tof) as well as flipping tof images. The idea is that when a right handed person bring is hand to his head/body, the acceleration is from right to left while it's the opposite for left handed person.\n- **upside down alignements**: it became obvious when looking at my oof scores that something wrong was happening to \"SUBJ_045235\", \"SUBJ_019262\". After a lot of data visualisation I came to the conclusion that they were wearing the device upside down. When applying the corresponding transfos (switching sensors, negating some acc and rot axis) those two subjects' scores went back to normal. To make sure that my transfo was correct I trained an \"upside down detector\": randomly applying the transfo to all sequences and predicting whether it was used or not. The model was very good at it and correctly said that those two subjects were indeed upside down. Problem is you do not have a similar info as 'handedness' to know that someone is wearing the device wrong. So I tried a sub with first using the detector and applying the transfo if needed before making my prediction: got a similar score (it was the two digit LB era at that moment). I thought \"the test set is clean and everybody is wearing their device the correct way\", so I just flipped those two subjects once and for all and continue improving my CV. **BIG MISTAKE**: some private subjects are indeed wearing their device upside down and the detector worked well (LB: 863 - PLB: 857). I should have done it the correct way: do not allow my CV to improve after altering the data myself, my solution should have included the detector in order to improve CV and I should have trusted it!\n- **sequence length**: all sequence are right cropped/padded to 96 (got similar results 64 or 128)\n\n## 2.2 Data augmentation\n\nI did a lot of augmentations on the fly:\n- Alignement aware MIXUP: I randomly average sequences after aligning the start of the gesture so that averaging made more sense\n- random rotaion: when looking at the videos of the lady performing the gestures on Youtube, you can see that the device is not always perfectly worn. So I add random small rotation (+-30°) around Z-axis to quaternions.\n- random stretching\n- randomly ignore some points at the beginning of sequences\n\n## 2.3 Losses\n\nI did not feel very comfortable using demographics data as there are two few subjects to be meaningfull for training. Nevertheless I tried to incorporate as much knowledge as possible to my models by forcing them to predict as many thing as possible (different heads):\n- 18 gestures\n- binary target\n- orientation\n- location: a custom mix of gestures into 4 locations (leg - face - neck - front)\n- action: a custom mix of gestures into 5 actions (pull hair - pinch - scratch - write - other)\n- is child\n\nLocation, action and child were probably useless.\n\n## 2.4 Feature engineering\n\nI did little feature engineering:\n- acceleration: original accelerations, temporal diff 1, acc_mal_jerk, normalized position in sequence (0 to 1)\n- quaternions: orig quaternions, rotation angle and angle velocity (x, y, z) and distances, angular velocity norm and 10 window rolling mean of it\n- thm: original thm\n- rof: original tof\n\n## 2.5 Architecture\n\nI ended up with a relatively simple architecture, I only used one single model that learns to ignore missing information. I did not have one specific model for IMU only and full sensors: I simply randomly mask the thm and tof feats during training.\n\nFirst encode information at the time level\n- independant conv1D for acc, quats and thm for time level encoding (with kernels 3-5-7) with N_OUT features for each\n- 2D CNN with 5 channels transforming 5x8x8 images to N_OUT (no time series info shared)\n\nThen each stream goes through a gating unit which decides the importance of each stream of sensors:\n```\nclass GatingUnit_v2(nn.Module):\n def __init__(self, input_dim, hidden_dim=64):\n super().__init__()\n\n self.pool = nn.AdaptiveAvgPool1d(1)\n self.gater = nn.Sequential(\n nn.Linear(input_dim, hidden_dim),\n nn.ReLU(),\n nn.Linear(hidden_dim, 1),\n nn.Sigmoid()\n )\n \n def forward(self, x):\n # x: (batch, feat_dim, seq_len)\n pooled_x = self.pool(x).squeeze(-1) # (batch, feat_dim)\n gate = self.gater(pooled_x).unsqueeze(-1) # (batch, 1, 1)\n gated_x = gate * x\n return gated_x, gate\n``` \n\nThen I combine all feature by summing them which forces the model to share the same representation for each stream of data:\n```\nnumerator = accel_features + orient_features + temp_features + tof_features\ndenominator = gate_accel + gate_orient + gate_temp + gate_tof\n\n combined_feats = numerator / (denominator + 1e-6)\n```\n\nThen I have an attention pooling mechanism to go from time series to series representation then basic MLP heads.\n\n\n## 2.6 CV scores\n\nFollowing the above pipeline I could reach avg CV 85.8 (IMU only 82.9 - all 88.7) with upside down subject flipped correctly.\nEnsembling a few models with different training parameters could reach 86.4 CV.\n\nMy LB never went above 86.6 with my own models, I was never able to overfit public LB, not sure why.\n\nSimply merging one 86.5 Anil's sub with one 86.5 single model of mine yielded LB 87.6 so I guess diversity in approaches was important here!\n\n## 2.7 Few mistakes\n\nThe main reason I did not managed to improve my pipeline on the past month was because of stubbornness and CV distillation. If using the OOF of a model to train a model from scratch following same CV scheme I could easily reach CV 0.88 (LB dropped significantly). It took me almost a month to accept that this was just pure leakage. If I decayed overtime the importance of OOF labels the CV went down but LB went up. I then tried 50 shades of leakage to reduce my CV and improve my LB like a crazy person.\n\nI should have:\n- accepted that distillation using OOF creates a powefull leakage\n- compute my CV without allowing to arbitrarily put two subject upside down (I may try my upside down detector as late sub but I suspect a huge improvement on PLB)\n- prob LB to exploit all available info as a real Kaggle champion should\n\n\nAnyway! Thanks for reading this far, congratulations again to the winners and happy kaggling!", "3280781": "Curious about the multi-task experiment results", "3280783": "Congratulations! Was there any intuition behind the features or was it just random testing?", "3280788": "Great solution and strong models! Did 6d quaternions representations give a boost to the score? I also came across the paper https://arxiv.org/pdf/1812.07035 and tried many different representations for the quaternions, but in the end the original raw values were the best", "3280789": "Congrats! Nice to see you managed to get the ensemble working. It didn’t work in my case at all; my final submission was a single model 😅", "3280795": "Orientation/behavior loss boosted your validation? How much?\nI tried both and they only hurt me.", "3280884": "for me when I sarted to use orientation, my CV and LB improved 0.01", "3280886": "I think I started the competition by doing that so I can't say :)", "3280921": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds,and seems like I should've done that. 😅", "3280922": "They boosted my CV like 0.002-3 when compared using roll/pitch/yaw and their diffs.", "3280923": "We have very different approaches in terms of both features and architectures. I'm also glad that worked out well for us. 😃", "3280925": "It didn't affect my CV in a good or bad direction. I kept them in case of using these preds.", "3280927": "I've converged to this feature subset by iterating on both CV/LB feedback.", "3280985": "Congrats!!", "3280999": "I think you misunderstood me. My question was how you thought of what features to test. Was it based on some sort of logic or just random guessing?", "3281002": "Derivatives and subject-referenced representations performed better than the absolute position values since the subjects’ position and orientation can vary a lot. When you think that way you will be quickly converged to a subset such like this, consists of many first and second derivatives of relative positional features.", "3281007": "I see, thanks for the quick reply!", "3281670": "optimo \n> all sequence are right cropped\n\nAre you really cropping right?", "3281697": "I keep the right part (end of sequence) and crop if necessary, so I should probably say a left crop sorry! :)" }, "source": "meta" }