![\"CMI-1\"](\"https://i.ibb.co/4ZdvWJqH/CMI-1.png\")
Thanks to all the participants, this was a really fun and challenging competition. We are very happy with the 3rd place finish :)
\nOur work is a result of great team-work and synergy by all team members - @minerppdy, @nazarov @ren4yu & @theoviel - and everyone contributed to achieving 3rd place.
\nOur solution is an ensemble of pipelines carefully tweaked by individual members of the team. As-is, the ensemble achieves Public 0.882 - Private 0.864.
\nWe jumped to Public 0.897 - Private 0.878 (3rd) by limiting each subject to 2 predictions per (gesture, orientation) pair. We'll explain this post-processing at the end of this section.
\n
The team pipeline contains one branch for IMU-only sequences, and one branch for full-data sequences (that reuses some IMU-only models), as illustrated below.
\n![\"CMI-2\"](\"https://i.ibb.co/wFf2pWNv/CMI-2.png\")
As noted above, we applied a post-processing step that limits each subject to two predictions per (gesture, orientation) pair. This optimization is done online and uses a Hungarian method.
\nWe keep a cache of model predictions for each (subject, orientation) bucket in the predict function. For each new sample to be classified, we append it to the list for that bucket; let the list length be N. In this case, we have (N, num_classes) prediction matrix, and create (N, num_classes * 2) cost matrix, and run linear_sum_assignment. Here * 2 means we allow each class to be assigned twice. Then we return the class assigned to the last row (the new sample).
\nAn important point is that we optimize from scratch for each sample arrival, ignoring previous results returned to the API. We were also aware that some (gesture, orientation) pairs do not exist but did not include that in the algorithm (we thought of it too late).
Post processing results vary a lot depending on sequence order, resulting in us finishing at worst 6th - although a luckier choice might have gotten us 2nd.
\nHere are some key things shared across our pipelines. Although implementations might differ a bit, they provided key boosts to all models.
\nrot_block(quat) + rot_block(-quat)SUBJ_019262 and SUBJ_045235 did not wear the device correctly, resulting in reversed y_acc and x_acc and a 180-degree rotation of the quat and tof data. They were fixed during training. Although we trained an outlier detection model to probe LB for similar outliers, we did not find anything. Probably an oversight on our end since @rsakata reported +0.01 with a similar strategy. Written by @theoviel
\nI initially worked on a transformer-based model, by starting from this really strong baseline.
\nAfter fixing the left/right handed issue, I reworked augmentations, tweaked the architecture, added auxiliary heads and everything I could find that would help my CV. I tweaked the architecture separately for IMU-only and full data.
Although I could come up with useful features (using spectrograms, highpass filtering for instance), transformer-based models blended quite poorly together, and I think were a bit over-engineered.
\n@nazarov had a lot of success with feeding raw input into 2D CNNs. Although that did not really make sense to me because it meant feature interaction would happen late in the model.
\nIt worked almost out of the box in my pipeline, by doing a simple resizing to the desired image size. The key was to be a bit more careful with feature selection (use fewer features), and use (most) features twice in different orders to alleviate the feature interaction issue.
\nI got rid of the gesture mask head, and used an orientation head instead, which was reused for the post-processing. I ended up having best results with maxvit and convnext-v2 backbones.
The CV of CNNs was similar to transformers, but they blended much better in the team ensemble, and were stronger on LB. In the end, the following models were used:
\nWritten by @ren4yu
\nI'm using two models: one with IMU+THM+ToF, and another IMU-only.
\nBoth models have an efficientnet-like 1D CNN backbone.
\nFor the IMU-only model, I also used a ModernBERT backbone.
\nFeatures are similar to those of the team
Temporal scaling, mixup, 2d blockdrop for tof, randomly drop 3rd or 5th channel of thm and tof, gaussian noise for imu features
\nTrain with auxiliary heads to predict behavior and orientation.
\nAdamW optimizer with cosine annealing, 128 epochs, batch size = 32, learning rate = 2e-3, weight decay = 0.2, and an exponential moving average (EMA) decay of 0.9995.
\nWritten by @nazarov
\nThis is my first experience with image processing, and some aspects of the approach may seem naive, but it worked. The core idea was simple: if I can visually distinguish between gestures on a feature plot, why shouldn't a pre-trained image classification model be able to do the same?
\nI consulted Qwen-coder for recommendations on suitable models and preprocessing techniques. The resulting pipeline is straightforward.
\nThe input data, which has the shape (n_timestamps, n_features), was first resized to (image_size, image_size). To make it compatible with models that expect three-channel images, the single-channel input was repeated three times, resulting in a final shape of (3, image_size, image_size). This processed input was then fed into a model with a modified last layer to suit the classification task.
\nI experimented with all popular models under 200 MB, and the EfficientNet family appeared to be the most efficient.
\nI used EfficientNetB0, B3, B5, V2_S and V2_M in the ensemble. All models were retrained on full data.
EfficientNetB5 with CV 0.8310 on imu-only and 0.8868 on all-sensors data was the strongest. The full ensemble score was CV / Public / Private = 0.8636 / 0.863 / 0.859.
\nA subset of it was included in the team ensemble to maximize diversity and optimize the team CV.
Written by @minerppdy
\nThe architecture is illustrated below, key idea is to use multiple branches to deal with different input channels.
\n![\"model\"](\"https://i.ibb.co/GvTryd3Z/model.png\")
Thanks for reading!
", "messages": [ { "id": "3281703", "postDate": "09/04/2025 18:09:45", "content": "Thanks to all the participants, this was a really fun and challenging competition. We are very happy with the 3rd place finish :)
\nOur work is a result of great team-work and synergy by all team members - @minerppdy, @nazarov @ren4yu & @theoviel - and everyone contributed to achieving 3rd place.
\nOur solution is an ensemble of pipelines carefully tweaked by individual members of the team. As-is, the ensemble achieves Public 0.882 - Private 0.864.
\nWe jumped to Public 0.897 - Private 0.878 (3rd) by limiting each subject to 2 predictions per (gesture, orientation) pair. We'll explain this post-processing at the end of this section.
\n
The team pipeline contains one branch for IMU-only sequences, and one branch for full-data sequences (that reuses some IMU-only models), as illustrated below.
\n
As noted above, we applied a post-processing step that limits each subject to two predictions per (gesture, orientation) pair. This optimization is done online and uses a Hungarian method.
\nWe keep a cache of model predictions for each (subject, orientation) bucket in the predict function. For each new sample to be classified, we append it to the list for that bucket; let the list length be N. In this case, we have (N, num_classes) prediction matrix, and create (N, num_classes * 2) cost matrix, and run linear_sum_assignment. Here * 2 means we allow each class to be assigned twice. Then we return the class assigned to the last row (the new sample).
\nAn important point is that we optimize from scratch for each sample arrival, ignoring previous results returned to the API. We were also aware that some (gesture, orientation) pairs do not exist but did not include that in the algorithm (we thought of it too late).
Post processing results vary a lot depending on sequence order, resulting in us finishing at worst 6th - although a luckier choice might have gotten us 2nd.
\nHere are some key things shared across our pipelines. Although implementations might differ a bit, they provided key boosts to all models.
\nrot_block(quat) + rot_block(-quat)SUBJ_019262 and SUBJ_045235 did not wear the device correctly, resulting in reversed y_acc and x_acc and a 180-degree rotation of the quat and tof data. They were fixed during training. Although we trained an outlier detection model to probe LB for similar outliers, we did not find anything. Probably an oversight on our end since @rsakata reported +0.01 with a similar strategy. Written by @theoviel
\nI initially worked on a transformer-based model, by starting from this really strong baseline.
\nAfter fixing the left/right handed issue, I reworked augmentations, tweaked the architecture, added auxiliary heads and everything I could find that would help my CV. I tweaked the architecture separately for IMU-only and full data.
Although I could come up with useful features (using spectrograms, highpass filtering for instance), transformer-based models blended quite poorly together, and I think were a bit over-engineered.
\n@nazarov had a lot of success with feeding raw input into 2D CNNs. Although that did not really make sense to me because it meant feature interaction would happen late in the model.
\nIt worked almost out of the box in my pipeline, by doing a simple resizing to the desired image size. The key was to be a bit more careful with feature selection (use fewer features), and use (most) features twice in different orders to alleviate the feature interaction issue.
\nI got rid of the gesture mask head, and used an orientation head instead, which was reused for the post-processing. I ended up having best results with maxvit and convnext-v2 backbones.
The CV of CNNs was similar to transformers, but they blended much better in the team ensemble, and were stronger on LB. In the end, the following models were used:
\nWritten by @ren4yu
\nI'm using two models: one with IMU+THM+ToF, and another IMU-only.
\nBoth models have an efficientnet-like 1D CNN backbone.
\nFor the IMU-only model, I also used a ModernBERT backbone.
\nFeatures are similar to those of the team
Temporal scaling, mixup, 2d blockdrop for tof, randomly drop 3rd or 5th channel of thm and tof, gaussian noise for imu features
\nTrain with auxiliary heads to predict behavior and orientation.
\nAdamW optimizer with cosine annealing, 128 epochs, batch size = 32, learning rate = 2e-3, weight decay = 0.2, and an exponential moving average (EMA) decay of 0.9995.
\nWritten by @nazarov
\nThis is my first experience with image processing, and some aspects of the approach may seem naive, but it worked. The core idea was simple: if I can visually distinguish between gestures on a feature plot, why shouldn't a pre-trained image classification model be able to do the same?
\nI consulted Qwen-coder for recommendations on suitable models and preprocessing techniques. The resulting pipeline is straightforward.
\nThe input data, which has the shape (n_timestamps, n_features), was first resized to (image_size, image_size). To make it compatible with models that expect three-channel images, the single-channel input was repeated three times, resulting in a final shape of (3, image_size, image_size). This processed input was then fed into a model with a modified last layer to suit the classification task.
\nI experimented with all popular models under 200 MB, and the EfficientNet family appeared to be the most efficient.
\nI used EfficientNetB0, B3, B5, V2_S and V2_M in the ensemble. All models were retrained on full data.
EfficientNetB5 with CV 0.8310 on imu-only and 0.8868 on all-sensors data was the strongest. The full ensemble score was CV / Public / Private = 0.8636 / 0.863 / 0.859.
\nA subset of it was included in the team ensemble to maximize diversity and optimize the team CV.
Written by @minerppdy
\nThe architecture is illustrated below, key idea is to use multiple branches to deal with different input channels.
\n
Thanks for reading!
", "rawMarkdown": "Thanks to all the participants, this was a really fun and challenging competition. We are very happy with the 3rd place finish :) \n\nOur work is a result of great team-work and synergy by all team members - @minerppdy, @nazarov @ren4yu & @theoviel - and everyone contributed to achieving 3rd place.\n\n## Overview\n\nOur solution is an ensemble of pipelines carefully tweaked by individual members of the team. As-is, the ensemble achieves **Public 0.882 - Private 0.864**.\n\nWe jumped to **Public 0.897 - Private 0.878 (3rd)** by limiting each subject to 2 predictions per (gesture, orientation) pair. We'll explain this post-processing at the end of this section.\n\n\n\nThe team pipeline contains one branch for IMU-only sequences, and one branch for full-data sequences (that reuses some IMU-only models), as illustrated below.\n\n\n\n## Post-processing\n\nAs noted above, we applied a post-processing step that limits each subject to two predictions per (gesture, orientation) pair. This optimization is done online and uses a Hungarian method.\nWe keep a cache of model predictions for each (subject, orientation) bucket in the predict function. For each new sample to be classified, we append it to the list for that bucket; let the list length be N. In this case, we have `(N, num_classes)` prediction matrix, and create `(N, num_classes * 2)` cost matrix, and run `linear_sum_assignment`. Here `* 2` means we allow each class to be assigned twice. Then we return the class assigned to the last row (the new sample).\nAn important point is that we optimize from scratch for each sample arrival, ignoring previous results returned to the API. We were also aware that some (gesture, orientation) pairs do not exist but did not include that in the algorithm (we thought of it too late).\n\nPost processing results vary a lot depending on sequence order, resulting in us finishing at worst 6th - although a luckier choice might have gotten us 2nd.\n\n## Overall ideas\nHere are some key things shared across our pipelines. Although implementations might differ a bit, they provided key boosts to all models.\n\n- Handedness normalization: For samples with handedness = 1, we canonicalize features to a common side.\n -IMU: left-right flip using a quaternion rotation of -110° on the z-axis first, or by simply swapping signs.\n - THM: swap the 3rd and 5th channels\n - ToF: swap the 3rd and 5th channels. We also flipped some of the the 8×8 images although it was less important\n- Padding and truncating on the left makes more sense. \n- Convention for quaternions is to have rot_w positive. This results in sequence discontinuity. Some of us smoothed the data to make training easier. To make this actually improve models, we added sign flip augmentation or symmetric blocks for quaternions : `rot_block(quat) + rot_block(-quat)`\n- The users `SUBJ_019262` and `SUBJ_045235` did not wear the device correctly, resulting in reversed y_acc and x_acc and a 180-degree rotation of the quat and tof data. They were fixed during training. Although we trained an outlier detection model to probe LB for similar outliers, we did not find anything. Probably an oversight on our end since @rsakata reported +0.01 with a similar strategy. \n- Data augmentation was important, among the most useful we found:\n - Mixup\n - Tof dropout\n - Stretch & shift the sequence\n - Rotation using the quaternion representation\n - Cutmix of sequences\n- Features shared publicly were very strong, so we used them as-is or used some variants. It was quite hard to find new things.\n\n## Theo's pipeline\n\n*Written by @theoviel*\n\n### Transformers\nI initially worked on a transformer-based model, by starting from [this really strong baseline](https://www.kaggle.com/code/jiazhuang/cmi-imu-only-lstm). \nAfter fixing the left/right handed issue, I reworked augmentations, tweaked the architecture, added auxiliary heads and everything I could find that would help my CV. I tweaked the architecture separately for IMU-only and full data.\n\nAlthough I could come up with useful features (using spectrograms, highpass filtering for instance), transformer-based models blended quite poorly together, and I think were a bit over-engineered. \n\n### Some ideas\n- Features are computed after applying augmentations, which made more sense from the physical standpoint\nTof features (mostly region aggregation ones) were computed inside the model for GPU acceleration, and I used a 3D-CNN as well\n- I tweaked the SE-1D-CNN block from public kernels a bit but I'm not sure it really helped. Some models used pooling to reduce the stride while others did not. Some also used parallel 1D-CNN blocks with different kernel sizes. \n- I also tweaked different transformers / RNN layers for diversity (GRU, Deberta, squeezeformer, skip-connections)\n- I added a gesture/transition mask head with BCE which is also used for pooling, since most of the signal happens after the transition.\n\n### CNNs\n\n@nazarov had a lot of success with feeding raw input into 2D CNNs. Although that did not really make sense to me because it meant feature interaction would happen late in the model.\nIt worked almost out of the box in my pipeline, by doing a simple resizing to the desired image size. The key was to be a bit more careful with feature selection (use fewer features), and use (most) features twice in different orders to alleviate the feature interaction issue.\nI got rid of the gesture mask head, and used an orientation head instead, which was reused for the post-processing. I ended up having best results with maxvit and convnext-v2 backbones.\n\n### Results\n\nThe CV of CNNs was similar to transformers, but they blended much better in the team ensemble, and were stronger on LB. In the end, the following models were used:\n\n- IMU - CV 0.844\n- IMU + TOF + THM - CV 0.897\n- **Public 0.868 and Private 0.852**\n\n\n## Yu4u's pipeline\n\n*Written by @ren4yu*\n\n### Overview\n\n#### Model\nI'm using two models: one with IMU+THM+ToF, and another IMU-only.\nBoth models have an efficientnet-like 1D CNN backbone.\nFor the IMU-only model, I also used a ModernBERT backbone.\nFeatures are similar to those of the team\n\n#### Augmentations\nTemporal scaling, mixup, 2d blockdrop for tof, randomly drop 3rd or 5th channel of thm and tof, gaussian noise for imu features\n\n#### Auxiliary tasks\nTrain with auxiliary heads to predict behavior and orientation.\n\n#### Training\nAdamW optimizer with cosine annealing, 128 epochs, batch size = 32, learning rate = 2e-3, weight decay = 0.2, and an exponential moving average (EMA) decay of 0.9995.\n\n### Results\n\n- IMU only models - CV: 0.8423\n- IMU+THM+ToF model - CV: 0.8857\n- **Public LB: 0.862, Private LB: 0.847**\n\n## Leonid's pipeline\n\n*Written by @nazarov*\n\n### Summary\n\nThis is my first experience with image processing, and some aspects of the approach may seem naive, but it worked. The core idea was simple: if I can visually distinguish between gestures on a feature plot, why shouldn't a pre-trained image classification model be able to do the same?\n\nI consulted Qwen-coder for recommendations on suitable models and preprocessing techniques. The resulting pipeline is straightforward. \nThe input data, which has the shape `(n_timestamps, n_features)`, was first resized to `(image_size, image_size)`. To make it compatible with models that expect three-channel images, the single-channel input was repeated three times, resulting in a final shape of `(3, image_size, image_size)`. This processed input was then fed into a model with a modified last layer to suit the classification task.\nI experimented with all popular models under 200 MB, and the EfficientNet family appeared to be the most efficient.\nI used EfficientNetB0, B3, B5, V2_S and V2_M in the ensemble. All models were retrained on full data. \n\n### Other things that worked\n\n- Separating groups of features by arrays of zeros.\n- Loss function, that was the weighted sum of 3 losses: CE for 18 classes, CE for 9 classes (non-target gestures were combined into one class) and CE for 2 classes (target/non-target).\n\n### Results\n\nEfficientNetB5 with CV 0.8310 on imu-only and 0.8868 on all-sensors data was the strongest. The full ensemble score was **CV / Public / Private = 0.8636 / 0.863 / 0.859**.\nA subset of it was included in the team ensemble to maximize diversity and optimize the team CV.\n\n## Minerppdy's pipeline\n\n*Written by @minerppdy*\n\n### Key ideas\n- Features\n - Relative quaternion to the next step based on the local coordinate system.\n - Angular jerk/snap\n - Acc jerk/snap\n - Linear acc\n- Augmentation \n - Rotate the quaternion around the z-axis of the world coordinate system by random angles (from -60 to 60 degrees) to increase robustness against the different directions the trial subjects face.\n - Rotate both the accelerometer and quaternion data around the y-axis of the local coordinate system by small random angles (from -7 to 7 degrees) to increase robustness against slight variations in how the device is worn.\n\n### Model\n\nThe architecture is illustrated below, key idea is to use multiple branches to deal with different input channels.\n\n\n\n### Results\n\n- IMU only models - CV: 0.8283\n- IMU+THM+ToF model - CV: 0.8850\n- **Public LB: 0.859, Private LB: 0.848**\n\n## --\n\n*Thanks for reading!*",
"votes": null
},
{
"id": "3281730",
"postDate": "09/04/2025 19:14:12",
"content": "Great work, I think the handedness normalization was an important piece of surviving the private leaderboard that we missed. Didnt measure much performance difference locally or on leaderboard so I dropped it. I started everything from scratch to try to unify my models and which features they used and I dropped that then. Wish I had kept it in.
", "rawMarkdown": "Great work, I think the handedness normalization was an important piece of surviving the private leaderboard that we missed. Didnt measure much performance difference locally or on leaderboard so I dropped it. I started everything from scratch to try to unify my models and which features they used and I dropped that then. Wish I had kept it in.", "votes": null }, { "id": "3281731", "postDate": "09/04/2025 19:18:23", "content": "I'm curious about the impact of the hungarian matching. Our postprocessing logic was similar but didnt use that. We just sorted by highest confidence for a slot and tried alternatives with some threshold so we wouldnt consider options a certain amount less confident than the initial models selection. Like if the first choice was 85% but the next best was 5% then we wouldnt drop that low.
", "rawMarkdown": "I'm curious about the impact of the hungarian matching. Our postprocessing logic was similar but didnt use that. We just sorted by highest confidence for a slot and tried alternatives with some threshold so we wouldnt consider options a certain amount less confident than the initial models selection. Like if the first choice was 85% but the next best was 5% then we wouldnt drop that low.", "votes": null }, { "id": "3281758", "postDate": "09/04/2025 21:11:37", "content": "Our first version of the PP was something similar to that as well. The hungarian matching approach had was about +0.006 CV better.
", "rawMarkdown": "Our first version of the PP was something similar to that as well. The hungarian matching approach had was about +0.006 CV better.", "votes": null }, { "id": "3281759", "postDate": "09/04/2025 21:13:12", "content": "Interesting that it did not help CV for you, maybe your models learnt to be robust to that.
\nFor me it gave +0.01 very early when I started tweaking models.
well explained, thanks!
", "rawMarkdown": "well explained, thanks!", "votes": null } ], "comments": [ { "id": 3281730, "author_name": "ryches", "author_url": "", "post_date": "09/04/2025 19:14:12", "content": "Great work, I think the handedness normalization was an important piece of surviving the private leaderboard that we missed. Didnt measure much performance difference locally or on leaderboard so I dropped it. I started everything from scratch to try to unify my models and which features they used and I dropped that then. Wish I had kept it in.
", "votes": null, "replies": [ { "id": 3281759, "author_name": "theoviel", "author_url": "", "post_date": "09/04/2025 21:13:12", "content": "Interesting that it did not help CV for you, maybe your models learnt to be robust to that.
\nFor me it gave +0.01 very early when I started tweaking models.
I'm curious about the impact of the hungarian matching. Our postprocessing logic was similar but didnt use that. We just sorted by highest confidence for a slot and tried alternatives with some threshold so we wouldnt consider options a certain amount less confident than the initial models selection. Like if the first choice was 85% but the next best was 5% then we wouldnt drop that low.
", "votes": null, "replies": [ { "id": 3281758, "author_name": "theoviel", "author_url": "", "post_date": "09/04/2025 21:11:37", "content": "Our first version of the PP was something similar to that as well. The hungarian matching approach had was about +0.006 CV better.
", "votes": null, "replies": [] } ] }, { "id": 3282779, "author_name": "irakozekelly", "author_url": "", "post_date": "09/06/2025 19:44:34", "content": "well explained, thanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3281703": "Thanks to all the participants, this was a really fun and challenging competition. We are very happy with the 3rd place finish :) \n\nOur work is a result of great team-work and synergy by all team members - @minerppdy, @nazarov @ren4yu & @theoviel - and everyone contributed to achieving 3rd place.\n\n## Overview\n\nOur solution is an ensemble of pipelines carefully tweaked by individual members of the team. As-is, the ensemble achieves **Public 0.882 - Private 0.864**.\n\nWe jumped to **Public 0.897 - Private 0.878 (3rd)** by limiting each subject to 2 predictions per (gesture, orientation) pair. We'll explain this post-processing at the end of this section.\n\n\n\nThe team pipeline contains one branch for IMU-only sequences, and one branch for full-data sequences (that reuses some IMU-only models), as illustrated below.\n\n\n\n## Post-processing\n\nAs noted above, we applied a post-processing step that limits each subject to two predictions per (gesture, orientation) pair. This optimization is done online and uses a Hungarian method.\nWe keep a cache of model predictions for each (subject, orientation) bucket in the predict function. For each new sample to be classified, we append it to the list for that bucket; let the list length be N. In this case, we have `(N, num_classes)` prediction matrix, and create `(N, num_classes * 2)` cost matrix, and run `linear_sum_assignment`. Here `* 2` means we allow each class to be assigned twice. Then we return the class assigned to the last row (the new sample).\nAn important point is that we optimize from scratch for each sample arrival, ignoring previous results returned to the API. We were also aware that some (gesture, orientation) pairs do not exist but did not include that in the algorithm (we thought of it too late).\n\nPost processing results vary a lot depending on sequence order, resulting in us finishing at worst 6th - although a luckier choice might have gotten us 2nd.\n\n## Overall ideas\nHere are some key things shared across our pipelines. Although implementations might differ a bit, they provided key boosts to all models.\n\n- Handedness normalization: For samples with handedness = 1, we canonicalize features to a common side.\n -IMU: left-right flip using a quaternion rotation of -110° on the z-axis first, or by simply swapping signs.\n - THM: swap the 3rd and 5th channels\n - ToF: swap the 3rd and 5th channels. We also flipped some of the the 8×8 images although it was less important\n- Padding and truncating on the left makes more sense. \n- Convention for quaternions is to have rot_w positive. This results in sequence discontinuity. Some of us smoothed the data to make training easier. To make this actually improve models, we added sign flip augmentation or symmetric blocks for quaternions : `rot_block(quat) + rot_block(-quat)`\n- The users `SUBJ_019262` and `SUBJ_045235` did not wear the device correctly, resulting in reversed y_acc and x_acc and a 180-degree rotation of the quat and tof data. They were fixed during training. Although we trained an outlier detection model to probe LB for similar outliers, we did not find anything. Probably an oversight on our end since @rsakata reported +0.01 with a similar strategy. \n- Data augmentation was important, among the most useful we found:\n - Mixup\n - Tof dropout\n - Stretch & shift the sequence\n - Rotation using the quaternion representation\n - Cutmix of sequences\n- Features shared publicly were very strong, so we used them as-is or used some variants. It was quite hard to find new things.\n\n## Theo's pipeline\n\n*Written by @theoviel*\n\n### Transformers\nI initially worked on a transformer-based model, by starting from [this really strong baseline](https://www.kaggle.com/code/jiazhuang/cmi-imu-only-lstm). \nAfter fixing the left/right handed issue, I reworked augmentations, tweaked the architecture, added auxiliary heads and everything I could find that would help my CV. I tweaked the architecture separately for IMU-only and full data.\n\nAlthough I could come up with useful features (using spectrograms, highpass filtering for instance), transformer-based models blended quite poorly together, and I think were a bit over-engineered. \n\n### Some ideas\n- Features are computed after applying augmentations, which made more sense from the physical standpoint\nTof features (mostly region aggregation ones) were computed inside the model for GPU acceleration, and I used a 3D-CNN as well\n- I tweaked the SE-1D-CNN block from public kernels a bit but I'm not sure it really helped. Some models used pooling to reduce the stride while others did not. Some also used parallel 1D-CNN blocks with different kernel sizes. \n- I also tweaked different transformers / RNN layers for diversity (GRU, Deberta, squeezeformer, skip-connections)\n- I added a gesture/transition mask head with BCE which is also used for pooling, since most of the signal happens after the transition.\n\n### CNNs\n\n@nazarov had a lot of success with feeding raw input into 2D CNNs. Although that did not really make sense to me because it meant feature interaction would happen late in the model.\nIt worked almost out of the box in my pipeline, by doing a simple resizing to the desired image size. The key was to be a bit more careful with feature selection (use fewer features), and use (most) features twice in different orders to alleviate the feature interaction issue.\nI got rid of the gesture mask head, and used an orientation head instead, which was reused for the post-processing. I ended up having best results with maxvit and convnext-v2 backbones.\n\n### Results\n\nThe CV of CNNs was similar to transformers, but they blended much better in the team ensemble, and were stronger on LB. In the end, the following models were used:\n\n- IMU - CV 0.844\n- IMU + TOF + THM - CV 0.897\n- **Public 0.868 and Private 0.852**\n\n\n## Yu4u's pipeline\n\n*Written by @ren4yu*\n\n### Overview\n\n#### Model\nI'm using two models: one with IMU+THM+ToF, and another IMU-only.\nBoth models have an efficientnet-like 1D CNN backbone.\nFor the IMU-only model, I also used a ModernBERT backbone.\nFeatures are similar to those of the team\n\n#### Augmentations\nTemporal scaling, mixup, 2d blockdrop for tof, randomly drop 3rd or 5th channel of thm and tof, gaussian noise for imu features\n\n#### Auxiliary tasks\nTrain with auxiliary heads to predict behavior and orientation.\n\n#### Training\nAdamW optimizer with cosine annealing, 128 epochs, batch size = 32, learning rate = 2e-3, weight decay = 0.2, and an exponential moving average (EMA) decay of 0.9995.\n\n### Results\n\n- IMU only models - CV: 0.8423\n- IMU+THM+ToF model - CV: 0.8857\n- **Public LB: 0.862, Private LB: 0.847**\n\n## Leonid's pipeline\n\n*Written by @nazarov*\n\n### Summary\n\nThis is my first experience with image processing, and some aspects of the approach may seem naive, but it worked. The core idea was simple: if I can visually distinguish between gestures on a feature plot, why shouldn't a pre-trained image classification model be able to do the same?\n\nI consulted Qwen-coder for recommendations on suitable models and preprocessing techniques. The resulting pipeline is straightforward. \nThe input data, which has the shape `(n_timestamps, n_features)`, was first resized to `(image_size, image_size)`. To make it compatible with models that expect three-channel images, the single-channel input was repeated three times, resulting in a final shape of `(3, image_size, image_size)`. This processed input was then fed into a model with a modified last layer to suit the classification task.\nI experimented with all popular models under 200 MB, and the EfficientNet family appeared to be the most efficient.\nI used EfficientNetB0, B3, B5, V2_S and V2_M in the ensemble. All models were retrained on full data. \n\n### Other things that worked\n\n- Separating groups of features by arrays of zeros.\n- Loss function, that was the weighted sum of 3 losses: CE for 18 classes, CE for 9 classes (non-target gestures were combined into one class) and CE for 2 classes (target/non-target).\n\n### Results\n\nEfficientNetB5 with CV 0.8310 on imu-only and 0.8868 on all-sensors data was the strongest. The full ensemble score was **CV / Public / Private = 0.8636 / 0.863 / 0.859**.\nA subset of it was included in the team ensemble to maximize diversity and optimize the team CV.\n\n## Minerppdy's pipeline\n\n*Written by @minerppdy*\n\n### Key ideas\n- Features\n - Relative quaternion to the next step based on the local coordinate system.\n - Angular jerk/snap\n - Acc jerk/snap\n - Linear acc\n- Augmentation \n - Rotate the quaternion around the z-axis of the world coordinate system by random angles (from -60 to 60 degrees) to increase robustness against the different directions the trial subjects face.\n - Rotate both the accelerometer and quaternion data around the y-axis of the local coordinate system by small random angles (from -7 to 7 degrees) to increase robustness against slight variations in how the device is worn.\n\n### Model\n\nThe architecture is illustrated below, key idea is to use multiple branches to deal with different input channels.\n\n\n\n### Results\n\n- IMU only models - CV: 0.8283\n- IMU+THM+ToF model - CV: 0.8850\n- **Public LB: 0.859, Private LB: 0.848**\n\n## --\n\n*Thanks for reading!*",
"3281730": "Great work, I think the handedness normalization was an important piece of surviving the private leaderboard that we missed. Didnt measure much performance difference locally or on leaderboard so I dropped it. I started everything from scratch to try to unify my models and which features they used and I dropped that then. Wish I had kept it in.",
"3281731": "I'm curious about the impact of the hungarian matching. Our postprocessing logic was similar but didnt use that. We just sorted by highest confidence for a slot and tried alternatives with some threshold so we wouldnt consider options a certain amount less confident than the initial models selection. Like if the first choice was 85% but the next best was 5% then we wouldnt drop that low.",
"3281758": "Our first version of the PP was something similar to that as well. The hungarian matching approach had was about +0.006 CV better.",
"3281759": "Interesting that it did not help CV for you, maybe your models learnt to be robust to that.\nFor me it gave +0.01 very early when I started tweaking models.",
"3282779": "well explained, thanks!"
},
"source": "existing_discussion",
"complete": true,
"competition_id": 102335,
"competition_slug": "cmi-detect-behavior-with-sensor-data",
"team_id": 14087656,
"solution_url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/3rd-place-solution"
}