{
  "id": 603580,
  "title": "11th Place Solution",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/11th-place-solution",
  "author_name": "",
  "post_date": "2025-09-03T07:16:25.107Z",
  "votes": 42,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Very lucky to have teamed with lightsource. My solution was decent, but my imu modeling was greatly underpowered and his was very strong. We teamed rather late so it was lucky that when we revealed our cards to each other we had good complimenting pieces. The main theme of our solution was optimized modeling and then a little bit of extra postprocessing sprinkled on top to get a decent boost. The main postprocessing insight is <a href=\"https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544\" target=\"_blank\">here</a>. Our ensembling stayed relatively simple, just 50/50 averaging between our approaches. We teamed late enough that we used totally different code bases and just ran them separately and then averaged results. </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2F2e22497f53da286ed120d74ccfd01771%2Fthreshold_sweep_analysis.png?generation=1756882621755379&amp;alt=media\" alt=\"\"></p>\n<p>Plots analyzing how postprocessing works with different relaxations from the 2 gesture limit per subject orientation</p>\n<h1>lightsource part</h1>\n<p>full code: <a href=\"https://github.com/l1ghtsource/cmi-detect-behavior-w-sensor-data\" target=\"_blank\">github link</a></p>\n<h2>Features</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>IMU: base (<code>acc_x/y/z</code>, <code>rot_w/x/y/z</code>)</li>\n<li>From public solutions: <code>acc_mag</code>, <code>rot_angle</code>, <code>acc_mag_jerk</code>, <code>rot_angle_vel</code>, <code>linear_acc_x/y/z</code>, <code>linear_acc_mag</code>, <code>linear_acc_mag_jerk</code>, <code>angular_vel_x/y/z</code>, <code>angular_distance</code></li>\n<li>TOF: raw 320 features</li>\n<li>THM: raw 5 features</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>IMU: 6d quat representations (<a href=\"https://arxiv.org/pdf/1812.07035)\" target=\"_blank\">https://arxiv.org/pdf/1812.07035)</a>, sliding-window statistics, integrals (except linear velocity), euler angles, gravity-based features, positional features</li>\n<li>TOF/THM: inter-sensor differences, statistics, center of mass, gradients</li>\n<li>Demographics: failed due to limited number of subjects</li>\n</ul>\n<h2>Preprocessing</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>No normalization (better than per-sequence scaling and StandardScaler)</li>\n<li>TOF: replaced -1 with 255, then divided by 255</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>Filtering methods (firwin, wavelet, savgol, butterworth, median) degraded performance</li>\n<li>Kalman filtering was neutral</li>\n<li>Left-to-right handedness transformations were not useful (did not increase cv and lb, although for private it would probably be important)</li>\n</ul>\n<h2>Modeling</h2>\n<p><strong>Two models:</strong></p>\n<ul>\n<li>IMU only</li>\n<li>IMU + TOF + THM (model selected at inference based on TOF availability)</li>\n</ul>\n<p><strong>Branches:</strong></p>\n<ul>\n<li>acc: <code>acc_x/y/z</code></li>\n<li>rot: <code>rot_w/x/y/z</code></li>\n<li>fe1: hand-crafted features (jerks, velocities, angles, distance,<br>\netc.)</li>\n<li>fe2: lag/lead diff, cumsum</li>\n<li>full: all IMU features</li>\n<li>thm: all THM features (only in IMU+TOF+THM model)</li>\n<li>tof1--tof5: TOF features from individual sensors (only in IMU+TOF+THM model)</li>\n</ul>\n<p><strong>Extractors:</strong></p>\n<ul>\n<li>Public_SingleSensor - based on the most popular public model</li>\n<li>FilterNetFeatureExtractor (only IMU only) - modified from <a href=\"https://github.com/WhistleLabs/FilterNet\" target=\"_blank\">https://github.com/WhistleLabs/FilterNet</a></li>\n<li>ConvTran_SingleSensor_NoTranLol modified from <a href=\"https://github.com/Navidfoumani/ConvTran\" target=\"_blank\">https://github.com/Navidfoumani/ConvTran</a></li>\n<li>ResNet1D</li>\n<li>Public2_SingleSensor (only IMU only) - based on another public model that uses an extractor for each channel</li>\n<li>MultiResidualBiGRU - modified from <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\" target=\"_blank\">https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410</a></li>\n</ul>\n<p><strong>Aggregation:</strong></p>\n<ul>\n<li>Branch level: concat (better than MHA, comparable to GAT, GCN worse)</li>\n<li>Extractor level: multihead attention</li>\n</ul>\n<p><strong>Heads:</strong></p>\n<ul>\n<li>gesture</li>\n<li>sequence type</li>\n<li>orientation</li>\n<li>additional gesture heads directly over extractors</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>Architectures: inceptiontime, efficientnet, harmamba, transformers (husformer, medformer, squeezeformer, timemil, etc.), moderntcn, wavenet, 2D/3D extractors for TOF</li>\n<li>Using spectrograms, scalograms, line plots</li>\n<li>Heads: gesture start prediction, full behavior mask, demographic features</li>\n</ul>\n<h2>Augmentations</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>mixup and variants (cutmix, channelmix, zebra)</li>\n<li>channel masking</li>\n<li>jitter</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>time shift, stretch, warp</li>\n<li>rotations, left/right reflections</li>\n<li>low pass filter augmentation</li>\n</ul>\n<h2>Training</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>Optimizers: AdamW, MuonWithAuxAdam (slightly better in some cases)</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>TorchJD for multi-task learning</li>\n<li>Metric learning components to improve the separation of similar classes, also did not provide any gain (arcface, tripletloss, …)</li>\n<li>EMA</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fb588d6a0c9329dfa1cef7aafd2aa9206%2Fperformance_by_demographics.png?generation=1756882712543880&amp;alt=media\" alt=\"\"></p>\n<h1>Ryan part</h1>\n<p>full code: <a href=\"https://github.com/ryanchesler/cmi\" target=\"_blank\">github</a></p>\n<h2>Features</h2>\n<p>Virtually the same as lightsource, mostly things from the public kernels. I tried many different processings of the features. Trying to spectrogram them, turn them into random squiggle plots like in <a href=\"https://www.nature.com/articles/s41598-022-25108-2\" target=\"_blank\">this paper</a> but the regular features with gravity removed and derivatives seemed the most useful. I tried to convert things to world frame or a representation where the starting direction was always consistent, but didnt find any success with this. </p>\n<h2>Preprocessing</h2>\n<ul>\n<li>Similar to lightsource, only normalizing with batchnorm as the first layer</li>\n<li>TOF: replaced -1 with 300, then divided by 255. Similar except for I marked -1 as even further away because I didnt want it confused between an object far away and something that was totally out of view. Dont think it really mattered</li>\n</ul>\n<h2>Modeling</h2>\n<p>I had many different variations of my models. Some were imu only and full sensor only, but most of them functioned in both scenarios. Training them with heavy sensor dropout so they were robust to many sensors being nulled out or as imu only. </p>\n<p>My main conclusion with the modeling was that we really did not want depth and high feature interaction and we did not want methods that encouraged global patterns. One early experiment that was really telling to me was training a model that had a linear layer that learned feature interactions before applying any convolutions to the sequences. This fit so quickly and overfit immediately. We really wanted simple methods that would learn the best from every channel and grouping of channels individually. </p>\n<p>This landed me on a solution that was similar to lightsources and the public kernels that had multiple different branches that were being late and simply fused. My best model was a model with lots of branches but very shallow depth. It had a different branch with two conv1d layers of kernel size 3, 5, 9. And also the same for group convs so only groups of features got interactions. Then it also had branches of lstm, gru and a two layer transformer. All of these then fed into a conv1d layer that finally merged and interacted these features. I used a form of branch dropout and also some temporal and channel dropout in between every layer. Really trying to encourage it not to get dominated by any one branch and not to learn from a specific time step or feature. </p>\n<h2>Augmentations</h2>\n<p>I tried a ton of options but didnt really find anything that added beyond mixup. I ended up using very high amounts of this. The difference between alpha of 0.4 and 0.8 was quite large in terms of both loss and validation f1</p>\n<p>I tried various forms of warping, rotations, scaling, noise, sensor drift, hand flipping, but I didnt really seem to find anything that boosted things. </p>\n<h2>Ensembling and Postprocessing</h2>\n<p>I think the main contribution I had to the team was my postprocessing. My modeling was ok, but lightsource's was significantly better on imu and my whole ensemble was approaching similar full performance but it was with brute force and many models being ensembled together. When we merged a simple average between our solutions gave us a huge boost, but when I ported over my postprocessing to our combined solution that was when we were able to take first for a while. The biggest thing was just taking advantage of the relationship between subject gesture orientation constraints. We saw from our training data that only two gestures were done per orientation at most and locking this in significantly helped our score. The second piece on top of this was making sure illegal pairings were never predicted. 21 orientation gesture combos were never done in our data so turning these off didnt have a huge effect but it was still a decent gain in reference to how close our leaderboard was. </p>\n<p>We got another big boost when we converting the probablities that we were averaging before with a conversion to ranking. This was rather simple. Just assigning 18/18 to the highest confidence class 17/18 to the next and 16/18 to the third and so on. It was surprising to me that this was better than giving the models some expressive power by giving probability to each class, but maybe this worked to calibrate predictions per model. I noticed that lightsources predictions were way more confident than mine. Mine were often extremely blurry decision boundaries, probably because of the extremely high mixup. </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fa4fe2da9cee47e87a495f388214eb8da%2FAbove_ear___pull_hair_SEQ_002046_97timesteps.png?generation=1756882731104941&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "3280793",
      "postDate": "09/03/2025 06:48:48",
      "content": "<p>Very lucky to have teamed with lightsource. My solution was decent, but my imu modeling was greatly underpowered and his was very strong. We teamed rather late so it was lucky that when we revealed our cards to each other we had good complimenting pieces. The main theme of our solution was optimized modeling and then a little bit of extra postprocessing sprinkled on top to get a decent boost. The main postprocessing insight is <a href=\"https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544\" target=\"_blank\">here</a>. Our ensembling stayed relatively simple, just 50/50 averaging between our approaches. We teamed late enough that we used totally different code bases and just ran them separately and then averaged results. </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2F2e22497f53da286ed120d74ccfd01771%2Fthreshold_sweep_analysis.png?generation=1756882621755379&amp;alt=media\" alt=\"\"></p>\n<p>Plots analyzing how postprocessing works with different relaxations from the 2 gesture limit per subject orientation</p>\n<h1>lightsource part</h1>\n<p>full code: <a href=\"https://github.com/l1ghtsource/cmi-detect-behavior-w-sensor-data\" target=\"_blank\">github link</a></p>\n<h2>Features</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>IMU: base (<code>acc_x/y/z</code>, <code>rot_w/x/y/z</code>)</li>\n<li>From public solutions: <code>acc_mag</code>, <code>rot_angle</code>, <code>acc_mag_jerk</code>, <code>rot_angle_vel</code>, <code>linear_acc_x/y/z</code>, <code>linear_acc_mag</code>, <code>linear_acc_mag_jerk</code>, <code>angular_vel_x/y/z</code>, <code>angular_distance</code></li>\n<li>TOF: raw 320 features</li>\n<li>THM: raw 5 features</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>IMU: 6d quat representations (<a href=\"https://arxiv.org/pdf/1812.07035)\" target=\"_blank\">https://arxiv.org/pdf/1812.07035)</a>, sliding-window statistics, integrals (except linear velocity), euler angles, gravity-based features, positional features</li>\n<li>TOF/THM: inter-sensor differences, statistics, center of mass, gradients</li>\n<li>Demographics: failed due to limited number of subjects</li>\n</ul>\n<h2>Preprocessing</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>No normalization (better than per-sequence scaling and StandardScaler)</li>\n<li>TOF: replaced -1 with 255, then divided by 255</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>Filtering methods (firwin, wavelet, savgol, butterworth, median) degraded performance</li>\n<li>Kalman filtering was neutral</li>\n<li>Left-to-right handedness transformations were not useful (did not increase cv and lb, although for private it would probably be important)</li>\n</ul>\n<h2>Modeling</h2>\n<p><strong>Two models:</strong></p>\n<ul>\n<li>IMU only</li>\n<li>IMU + TOF + THM (model selected at inference based on TOF availability)</li>\n</ul>\n<p><strong>Branches:</strong></p>\n<ul>\n<li>acc: <code>acc_x/y/z</code></li>\n<li>rot: <code>rot_w/x/y/z</code></li>\n<li>fe1: hand-crafted features (jerks, velocities, angles, distance,<br>\netc.)</li>\n<li>fe2: lag/lead diff, cumsum</li>\n<li>full: all IMU features</li>\n<li>thm: all THM features (only in IMU+TOF+THM model)</li>\n<li>tof1--tof5: TOF features from individual sensors (only in IMU+TOF+THM model)</li>\n</ul>\n<p><strong>Extractors:</strong></p>\n<ul>\n<li>Public_SingleSensor - based on the most popular public model</li>\n<li>FilterNetFeatureExtractor (only IMU only) - modified from <a href=\"https://github.com/WhistleLabs/FilterNet\" target=\"_blank\">https://github.com/WhistleLabs/FilterNet</a></li>\n<li>ConvTran_SingleSensor_NoTranLol modified from <a href=\"https://github.com/Navidfoumani/ConvTran\" target=\"_blank\">https://github.com/Navidfoumani/ConvTran</a></li>\n<li>ResNet1D</li>\n<li>Public2_SingleSensor (only IMU only) - based on another public model that uses an extractor for each channel</li>\n<li>MultiResidualBiGRU - modified from <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\" target=\"_blank\">https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410</a></li>\n</ul>\n<p><strong>Aggregation:</strong></p>\n<ul>\n<li>Branch level: concat (better than MHA, comparable to GAT, GCN worse)</li>\n<li>Extractor level: multihead attention</li>\n</ul>\n<p><strong>Heads:</strong></p>\n<ul>\n<li>gesture</li>\n<li>sequence type</li>\n<li>orientation</li>\n<li>additional gesture heads directly over extractors</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>Architectures: inceptiontime, efficientnet, harmamba, transformers (husformer, medformer, squeezeformer, timemil, etc.), moderntcn, wavenet, 2D/3D extractors for TOF</li>\n<li>Using spectrograms, scalograms, line plots</li>\n<li>Heads: gesture start prediction, full behavior mask, demographic features</li>\n</ul>\n<h2>Augmentations</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>mixup and variants (cutmix, channelmix, zebra)</li>\n<li>channel masking</li>\n<li>jitter</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>time shift, stretch, warp</li>\n<li>rotations, left/right reflections</li>\n<li>low pass filter augmentation</li>\n</ul>\n<h2>Training</h2>\n<p><strong>Used:</strong></p>\n<ul>\n<li>Optimizers: AdamW, MuonWithAuxAdam (slightly better in some cases)</li>\n</ul>\n<p><strong>Unsuccessful experiments:</strong></p>\n<ul>\n<li>TorchJD for multi-task learning</li>\n<li>Metric learning components to improve the separation of similar classes, also did not provide any gain (arcface, tripletloss, …)</li>\n<li>EMA</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fb588d6a0c9329dfa1cef7aafd2aa9206%2Fperformance_by_demographics.png?generation=1756882712543880&amp;alt=media\" alt=\"\"></p>\n<h1>Ryan part</h1>\n<p>full code: <a href=\"https://github.com/ryanchesler/cmi\" target=\"_blank\">github</a></p>\n<h2>Features</h2>\n<p>Virtually the same as lightsource, mostly things from the public kernels. I tried many different processings of the features. Trying to spectrogram them, turn them into random squiggle plots like in <a href=\"https://www.nature.com/articles/s41598-022-25108-2\" target=\"_blank\">this paper</a> but the regular features with gravity removed and derivatives seemed the most useful. I tried to convert things to world frame or a representation where the starting direction was always consistent, but didnt find any success with this. </p>\n<h2>Preprocessing</h2>\n<ul>\n<li>Similar to lightsource, only normalizing with batchnorm as the first layer</li>\n<li>TOF: replaced -1 with 300, then divided by 255. Similar except for I marked -1 as even further away because I didnt want it confused between an object far away and something that was totally out of view. Dont think it really mattered</li>\n</ul>\n<h2>Modeling</h2>\n<p>I had many different variations of my models. Some were imu only and full sensor only, but most of them functioned in both scenarios. Training them with heavy sensor dropout so they were robust to many sensors being nulled out or as imu only. </p>\n<p>My main conclusion with the modeling was that we really did not want depth and high feature interaction and we did not want methods that encouraged global patterns. One early experiment that was really telling to me was training a model that had a linear layer that learned feature interactions before applying any convolutions to the sequences. This fit so quickly and overfit immediately. We really wanted simple methods that would learn the best from every channel and grouping of channels individually. </p>\n<p>This landed me on a solution that was similar to lightsources and the public kernels that had multiple different branches that were being late and simply fused. My best model was a model with lots of branches but very shallow depth. It had a different branch with two conv1d layers of kernel size 3, 5, 9. And also the same for group convs so only groups of features got interactions. Then it also had branches of lstm, gru and a two layer transformer. All of these then fed into a conv1d layer that finally merged and interacted these features. I used a form of branch dropout and also some temporal and channel dropout in between every layer. Really trying to encourage it not to get dominated by any one branch and not to learn from a specific time step or feature. </p>\n<h2>Augmentations</h2>\n<p>I tried a ton of options but didnt really find anything that added beyond mixup. I ended up using very high amounts of this. The difference between alpha of 0.4 and 0.8 was quite large in terms of both loss and validation f1</p>\n<p>I tried various forms of warping, rotations, scaling, noise, sensor drift, hand flipping, but I didnt really seem to find anything that boosted things. </p>\n<h2>Ensembling and Postprocessing</h2>\n<p>I think the main contribution I had to the team was my postprocessing. My modeling was ok, but lightsource's was significantly better on imu and my whole ensemble was approaching similar full performance but it was with brute force and many models being ensembled together. When we merged a simple average between our solutions gave us a huge boost, but when I ported over my postprocessing to our combined solution that was when we were able to take first for a while. The biggest thing was just taking advantage of the relationship between subject gesture orientation constraints. We saw from our training data that only two gestures were done per orientation at most and locking this in significantly helped our score. The second piece on top of this was making sure illegal pairings were never predicted. 21 orientation gesture combos were never done in our data so turning these off didnt have a huge effect but it was still a decent gain in reference to how close our leaderboard was. </p>\n<p>We got another big boost when we converting the probablities that we were averaging before with a conversion to ranking. This was rather simple. Just assigning 18/18 to the highest confidence class 17/18 to the next and 16/18 to the third and so on. It was surprising to me that this was better than giving the models some expressive power by giving probability to each class, but maybe this worked to calibrate predictions per model. I noticed that lightsources predictions were way more confident than mine. Mine were often extremely blurry decision boundaries, probably because of the extremely high mixup. </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fa4fe2da9cee47e87a495f388214eb8da%2FAbove_ear___pull_hair_SEQ_002046_97timesteps.png?generation=1756882731104941&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Very lucky to have teamed with lightsource. My solution was decent, but my imu modeling was greatly underpowered and his was very strong. We teamed rather late so it was lucky that when we revealed our cards to each other we had good complimenting pieces. The main theme of our solution was optimized modeling and then a little bit of extra postprocessing sprinkled on top to get a decent boost. The main postprocessing insight is [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544). Our ensembling stayed relatively simple, just 50/50 averaging between our approaches. We teamed late enough that we used totally different code bases and just ran them separately and then averaged results. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2F2e22497f53da286ed120d74ccfd01771%2Fthreshold_sweep_analysis.png?generation=1756882621755379&alt=media)\n\nPlots analyzing how postprocessing works with different relaxations from the 2 gesture limit per subject orientation\n\n# lightsource part\n\nfull code: [github link](https://github.com/l1ghtsource/cmi-detect-behavior-w-sensor-data)\n\n\n## Features\n\n**Used:**\n- IMU: base (`acc_x/y/z`, `rot_w/x/y/z`)\n- From public solutions: `acc_mag`, `rot_angle`, `acc_mag_jerk`, `rot_angle_vel`, `linear_acc_x/y/z`, `linear_acc_mag`, `linear_acc_mag_jerk`, `angular_vel_x/y/z`, `angular_distance`\n- TOF: raw 320 features\n- THM: raw 5 features\n\n**Unsuccessful experiments:**\n- IMU: 6d quat representations (https://arxiv.org/pdf/1812.07035), sliding-window statistics, integrals (except linear velocity), euler angles, gravity-based features, positional features\n- TOF/THM: inter-sensor differences, statistics, center of mass, gradients\n- Demographics: failed due to limited number of subjects\n\n## Preprocessing\n\n**Used:**\n- No normalization (better than per-sequence scaling and StandardScaler)\n- TOF: replaced -1 with 255, then divided by 255\n\n**Unsuccessful experiments:**\n- Filtering methods (firwin, wavelet, savgol, butterworth, median) degraded performance\n- Kalman filtering was neutral\n- Left-to-right handedness transformations were not useful (did not increase cv and lb, although for private it would probably be important)\n\n## Modeling\n\n**Two models:**\n- IMU only\n- IMU + TOF + THM (model selected at inference based on TOF availability)\n\n**Branches:**\n- acc: `acc_x/y/z`\n- rot: `rot_w/x/y/z`\n- fe1: hand-crafted features (jerks, velocities, angles, distance,\netc.)\n- fe2: lag/lead diff, cumsum\n- full: all IMU features\n- thm: all THM features (only in IMU+TOF+THM model)\n- tof1--tof5: TOF features from individual sensors (only in IMU+TOF+THM model)\n\n**Extractors:**\n- Public_SingleSensor - based on the most popular public model\n- FilterNetFeatureExtractor (only IMU only) - modified from https://github.com/WhistleLabs/FilterNet\n- ConvTran_SingleSensor_NoTranLol modified from https://github.com/Navidfoumani/ConvTran\n- ResNet1D\n- Public2_SingleSensor (only IMU only) - based on another public model that uses an extractor for each channel\n- MultiResidualBiGRU - modified from https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\n\n**Aggregation:**\n- Branch level: concat (better than MHA, comparable to GAT, GCN worse)\n- Extractor level: multihead attention\n\n**Heads:**\n- gesture\n- sequence type\n- orientation\n- additional gesture heads directly over extractors\n\n**Unsuccessful experiments:**\n- Architectures: inceptiontime, efficientnet, harmamba, transformers (husformer, medformer, squeezeformer, timemil, etc.), moderntcn, wavenet, 2D/3D extractors for TOF\n- Using spectrograms, scalograms, line plots\n- Heads: gesture start prediction, full behavior mask, demographic features\n\n## Augmentations\n\n**Used:**\n- mixup and variants (cutmix, channelmix, zebra)\n- channel masking\n- jitter\n\n**Unsuccessful experiments:**\n- time shift, stretch, warp\n- rotations, left/right reflections\n- low pass filter augmentation\n\n## Training\n\n**Used:**\n- Optimizers: AdamW, MuonWithAuxAdam (slightly better in some cases)\n\n**Unsuccessful experiments:**\n- TorchJD for multi-task learning\n- Metric learning components to improve the separation of similar classes, also did not provide any gain (arcface, tripletloss, ...)\n- EMA\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fb588d6a0c9329dfa1cef7aafd2aa9206%2Fperformance_by_demographics.png?generation=1756882712543880&alt=media)\n# Ryan part\nfull code: [github](https://github.com/ryanchesler/cmi)\n\n## Features\n\nVirtually the same as lightsource, mostly things from the public kernels. I tried many different processings of the features. Trying to spectrogram them, turn them into random squiggle plots like in [this paper](https://www.nature.com/articles/s41598-022-25108-2) but the regular features with gravity removed and derivatives seemed the most useful. I tried to convert things to world frame or a representation where the starting direction was always consistent, but didnt find any success with this. \n\n## Preprocessing\n\n- Similar to lightsource, only normalizing with batchnorm as the first layer\n- TOF: replaced -1 with 300, then divided by 255. Similar except for I marked -1 as even further away because I didnt want it confused between an object far away and something that was totally out of view. Dont think it really mattered\n\n## Modeling\nI had many different variations of my models. Some were imu only and full sensor only, but most of them functioned in both scenarios. Training them with heavy sensor dropout so they were robust to many sensors being nulled out or as imu only. \n\nMy main conclusion with the modeling was that we really did not want depth and high feature interaction and we did not want methods that encouraged global patterns. One early experiment that was really telling to me was training a model that had a linear layer that learned feature interactions before applying any convolutions to the sequences. This fit so quickly and overfit immediately. We really wanted simple methods that would learn the best from every channel and grouping of channels individually. \n\nThis landed me on a solution that was similar to lightsources and the public kernels that had multiple different branches that were being late and simply fused. My best model was a model with lots of branches but very shallow depth. It had a different branch with two conv1d layers of kernel size 3, 5, 9. And also the same for group convs so only groups of features got interactions. Then it also had branches of lstm, gru and a two layer transformer. All of these then fed into a conv1d layer that finally merged and interacted these features. I used a form of branch dropout and also some temporal and channel dropout in between every layer. Really trying to encourage it not to get dominated by any one branch and not to learn from a specific time step or feature. \n\n## Augmentations\n\nI tried a ton of options but didnt really find anything that added beyond mixup. I ended up using very high amounts of this. The difference between alpha of 0.4 and 0.8 was quite large in terms of both loss and validation f1\n\nI tried various forms of warping, rotations, scaling, noise, sensor drift, hand flipping, but I didnt really seem to find anything that boosted things. \n\n## Ensembling and Postprocessing\nI think the main contribution I had to the team was my postprocessing. My modeling was ok, but lightsource's was significantly better on imu and my whole ensemble was approaching similar full performance but it was with brute force and many models being ensembled together. When we merged a simple average between our solutions gave us a huge boost, but when I ported over my postprocessing to our combined solution that was when we were able to take first for a while. The biggest thing was just taking advantage of the relationship between subject gesture orientation constraints. We saw from our training data that only two gestures were done per orientation at most and locking this in significantly helped our score. The second piece on top of this was making sure illegal pairings were never predicted. 21 orientation gesture combos were never done in our data so turning these off didnt have a huge effect but it was still a decent gain in reference to how close our leaderboard was. \n\nWe got another big boost when we converting the probablities that we were averaging before with a conversion to ranking. This was rather simple. Just assigning 18/18 to the highest confidence class 17/18 to the next and 16/18 to the third and so on. It was surprising to me that this was better than giving the models some expressive power by giving probability to each class, but maybe this worked to calibrate predictions per model. I noticed that lightsources predictions were way more confident than mine. Mine were often extremely blurry decision boundaries, probably because of the extremely high mixup. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fa4fe2da9cee47e87a495f388214eb8da%2FAbove_ear___pull_hair_SEQ_002046_97timesteps.png?generation=1756882731104941&alt=media)",
      "votes": null
    },
    {
      "id": "3280801",
      "postDate": "09/03/2025 06:58:30",
      "content": "<p>Can you please share how much boost you got from postprocessing on Public LB? And how much from converting probabilities to ranking? If you have the data. And congrats</p>",
      "rawMarkdown": "Can you please share how much boost you got from postprocessing on Public LB? And how much from converting probabilities to ranking? If you have the data. And congrats",
      "votes": null
    },
    {
      "id": "3280808",
      "postDate": "09/03/2025 07:11:44",
      "content": "<p>On public leaderboard we were about 0.87 with no postprocessing and then 0.880 with. Then converting to ranking boosted us to 0.888 and then a couple extra models in the mix and tuning our threshold got us to 0.891. This was shaky in the end though, because we know we had probed a little bit extra out by fiddling with thresholds</p>",
      "rawMarkdown": "On public leaderboard we were about 0.87 with no postprocessing and then 0.880 with. Then converting to ranking boosted us to 0.888 and then a couple extra models in the mix and tuning our threshold got us to 0.891. This was shaky in the end though, because we know we had probed a little bit extra out by fiddling with thresholds",
      "votes": null
    },
    {
      "id": "3280811",
      "postDate": "09/03/2025 07:14:11",
      "content": "<p>Nice. Yeah if you probed for thresholds you deserve to be shaky! 🤣</p>",
      "rawMarkdown": "Nice. Yeah if you probed for thresholds you deserve to be shaky! 🤣",
      "votes": null
    },
    {
      "id": "3280817",
      "postDate": "09/03/2025 07:27:19",
      "content": "<p>Wasn't the intention to inflate our score. Just testing for variation between runs and if the thresholding interacted with lb like we saw on cv or not and ended up with some lucky runs that boosted our public ranking. We roughly knew where we actually lined up. </p>",
      "rawMarkdown": "Wasn't the intention to inflate our score. Just testing for variation between runs and if the thresholding interacted with lb like we saw on cv or not and ended up with some lucky runs that boosted our public ranking. We roughly knew where we actually lined up.",
      "votes": null
    },
    {
      "id": "3280818",
      "postDate": "09/03/2025 07:32:43",
      "content": "<p>Well this competition is also a bit shaky inherently. My best ensemble was 0.865/0.854, second best 0.86/0.857 public/private. And both were chosen only based on local validation.</p>",
      "rawMarkdown": "Well this competition is also a bit shaky inherently. My best ensemble was 0.865/0.854, second best 0.86/0.857 public/private. And both were chosen only based on local validation.",
      "votes": null
    },
    {
      "id": "3280824",
      "postDate": "09/03/2025 07:40:01",
      "content": "<p>We had a couple that scored 0.871 on private. In hindsight we should have picked them, they had newer versions trained on all of the data with my best model. They tanked the public leaderboard enough that I didnt fully trust them, because I had no local validation on them.</p>",
      "rawMarkdown": "We had a couple that scored 0.871 on private. In hindsight we should have picked them, they had newer versions trained on all of the data with my best model. They tanked the public leaderboard enough that I didnt fully trust them, because I had no local validation on them.",
      "votes": null
    },
    {
      "id": "3280828",
      "postDate": "09/03/2025 07:44:34",
      "content": "<p>your github link seems private <a href=\"https://www.kaggle.com/ryches\" target=\"_blank\">@ryches</a> </p>",
      "rawMarkdown": "your github link seems private @ryches",
      "votes": null
    },
    {
      "id": "3280831",
      "postDate": "09/03/2025 07:55:29",
      "content": "<p>Fixed it now. Should be public</p>",
      "rawMarkdown": "Fixed it now. Should be public",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3280801,
      "author_name": "shlomoron",
      "author_url": "",
      "post_date": "09/03/2025 06:58:30",
      "content": "<p>Can you please share how much boost you got from postprocessing on Public LB? And how much from converting probabilities to ranking? If you have the data. And congrats</p>",
      "votes": null,
      "replies": [
        {
          "id": 3280808,
          "author_name": "ryches",
          "author_url": "",
          "post_date": "09/03/2025 07:11:44",
          "content": "<p>On public leaderboard we were about 0.87 with no postprocessing and then 0.880 with. Then converting to ranking boosted us to 0.888 and then a couple extra models in the mix and tuning our threshold got us to 0.891. This was shaky in the end though, because we know we had probed a little bit extra out by fiddling with thresholds</p>",
          "votes": null,
          "replies": [
            {
              "id": 3280811,
              "author_name": "shlomoron",
              "author_url": "",
              "post_date": "09/03/2025 07:14:11",
              "content": "<p>Nice. Yeah if you probed for thresholds you deserve to be shaky! 🤣</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3280817,
                  "author_name": "ryches",
                  "author_url": "",
                  "post_date": "09/03/2025 07:27:19",
                  "content": "<p>Wasn't the intention to inflate our score. Just testing for variation between runs and if the thresholding interacted with lb like we saw on cv or not and ended up with some lucky runs that boosted our public ranking. We roughly knew where we actually lined up. </p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3280818,
                      "author_name": "shlomoron",
                      "author_url": "",
                      "post_date": "09/03/2025 07:32:43",
                      "content": "<p>Well this competition is also a bit shaky inherently. My best ensemble was 0.865/0.854, second best 0.86/0.857 public/private. And both were chosen only based on local validation.</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 3280824,
                          "author_name": "ryches",
                          "author_url": "",
                          "post_date": "09/03/2025 07:40:01",
                          "content": "<p>We had a couple that scored 0.871 on private. In hindsight we should have picked them, they had newer versions trained on all of the data with my best model. They tanked the public leaderboard enough that I didnt fully trust them, because I had no local validation on them.</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 3280828,
      "author_name": "mks2192",
      "author_url": "",
      "post_date": "09/03/2025 07:44:34",
      "content": "<p>your github link seems private <a href=\"https://www.kaggle.com/ryches\" target=\"_blank\">@ryches</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 3280831,
          "author_name": "ryches",
          "author_url": "",
          "post_date": "09/03/2025 07:55:29",
          "content": "<p>Fixed it now. Should be public</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3280793": "Very lucky to have teamed with lightsource. My solution was decent, but my imu modeling was greatly underpowered and his was very strong. We teamed rather late so it was lucky that when we revealed our cards to each other we had good complimenting pieces. The main theme of our solution was optimized modeling and then a little bit of extra postprocessing sprinkled on top to get a decent boost. The main postprocessing insight is [here](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/603544). Our ensembling stayed relatively simple, just 50/50 averaging between our approaches. We teamed late enough that we used totally different code bases and just ran them separately and then averaged results. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2F2e22497f53da286ed120d74ccfd01771%2Fthreshold_sweep_analysis.png?generation=1756882621755379&alt=media)\n\nPlots analyzing how postprocessing works with different relaxations from the 2 gesture limit per subject orientation\n\n# lightsource part\n\nfull code: [github link](https://github.com/l1ghtsource/cmi-detect-behavior-w-sensor-data)\n\n\n## Features\n\n**Used:**\n- IMU: base (`acc_x/y/z`, `rot_w/x/y/z`)\n- From public solutions: `acc_mag`, `rot_angle`, `acc_mag_jerk`, `rot_angle_vel`, `linear_acc_x/y/z`, `linear_acc_mag`, `linear_acc_mag_jerk`, `angular_vel_x/y/z`, `angular_distance`\n- TOF: raw 320 features\n- THM: raw 5 features\n\n**Unsuccessful experiments:**\n- IMU: 6d quat representations (https://arxiv.org/pdf/1812.07035), sliding-window statistics, integrals (except linear velocity), euler angles, gravity-based features, positional features\n- TOF/THM: inter-sensor differences, statistics, center of mass, gradients\n- Demographics: failed due to limited number of subjects\n\n## Preprocessing\n\n**Used:**\n- No normalization (better than per-sequence scaling and StandardScaler)\n- TOF: replaced -1 with 255, then divided by 255\n\n**Unsuccessful experiments:**\n- Filtering methods (firwin, wavelet, savgol, butterworth, median) degraded performance\n- Kalman filtering was neutral\n- Left-to-right handedness transformations were not useful (did not increase cv and lb, although for private it would probably be important)\n\n## Modeling\n\n**Two models:**\n- IMU only\n- IMU + TOF + THM (model selected at inference based on TOF availability)\n\n**Branches:**\n- acc: `acc_x/y/z`\n- rot: `rot_w/x/y/z`\n- fe1: hand-crafted features (jerks, velocities, angles, distance,\netc.)\n- fe2: lag/lead diff, cumsum\n- full: all IMU features\n- thm: all THM features (only in IMU+TOF+THM model)\n- tof1--tof5: TOF features from individual sensors (only in IMU+TOF+THM model)\n\n**Extractors:**\n- Public_SingleSensor - based on the most popular public model\n- FilterNetFeatureExtractor (only IMU only) - modified from https://github.com/WhistleLabs/FilterNet\n- ConvTran_SingleSensor_NoTranLol modified from https://github.com/Navidfoumani/ConvTran\n- ResNet1D\n- Public2_SingleSensor (only IMU only) - based on another public model that uses an extractor for each channel\n- MultiResidualBiGRU - modified from https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/416410\n\n**Aggregation:**\n- Branch level: concat (better than MHA, comparable to GAT, GCN worse)\n- Extractor level: multihead attention\n\n**Heads:**\n- gesture\n- sequence type\n- orientation\n- additional gesture heads directly over extractors\n\n**Unsuccessful experiments:**\n- Architectures: inceptiontime, efficientnet, harmamba, transformers (husformer, medformer, squeezeformer, timemil, etc.), moderntcn, wavenet, 2D/3D extractors for TOF\n- Using spectrograms, scalograms, line plots\n- Heads: gesture start prediction, full behavior mask, demographic features\n\n## Augmentations\n\n**Used:**\n- mixup and variants (cutmix, channelmix, zebra)\n- channel masking\n- jitter\n\n**Unsuccessful experiments:**\n- time shift, stretch, warp\n- rotations, left/right reflections\n- low pass filter augmentation\n\n## Training\n\n**Used:**\n- Optimizers: AdamW, MuonWithAuxAdam (slightly better in some cases)\n\n**Unsuccessful experiments:**\n- TorchJD for multi-task learning\n- Metric learning components to improve the separation of similar classes, also did not provide any gain (arcface, tripletloss, ...)\n- EMA\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fb588d6a0c9329dfa1cef7aafd2aa9206%2Fperformance_by_demographics.png?generation=1756882712543880&alt=media)\n# Ryan part\nfull code: [github](https://github.com/ryanchesler/cmi)\n\n## Features\n\nVirtually the same as lightsource, mostly things from the public kernels. I tried many different processings of the features. Trying to spectrogram them, turn them into random squiggle plots like in [this paper](https://www.nature.com/articles/s41598-022-25108-2) but the regular features with gravity removed and derivatives seemed the most useful. I tried to convert things to world frame or a representation where the starting direction was always consistent, but didnt find any success with this. \n\n## Preprocessing\n\n- Similar to lightsource, only normalizing with batchnorm as the first layer\n- TOF: replaced -1 with 300, then divided by 255. Similar except for I marked -1 as even further away because I didnt want it confused between an object far away and something that was totally out of view. Dont think it really mattered\n\n## Modeling\nI had many different variations of my models. Some were imu only and full sensor only, but most of them functioned in both scenarios. Training them with heavy sensor dropout so they were robust to many sensors being nulled out or as imu only. \n\nMy main conclusion with the modeling was that we really did not want depth and high feature interaction and we did not want methods that encouraged global patterns. One early experiment that was really telling to me was training a model that had a linear layer that learned feature interactions before applying any convolutions to the sequences. This fit so quickly and overfit immediately. We really wanted simple methods that would learn the best from every channel and grouping of channels individually. \n\nThis landed me on a solution that was similar to lightsources and the public kernels that had multiple different branches that were being late and simply fused. My best model was a model with lots of branches but very shallow depth. It had a different branch with two conv1d layers of kernel size 3, 5, 9. And also the same for group convs so only groups of features got interactions. Then it also had branches of lstm, gru and a two layer transformer. All of these then fed into a conv1d layer that finally merged and interacted these features. I used a form of branch dropout and also some temporal and channel dropout in between every layer. Really trying to encourage it not to get dominated by any one branch and not to learn from a specific time step or feature. \n\n## Augmentations\n\nI tried a ton of options but didnt really find anything that added beyond mixup. I ended up using very high amounts of this. The difference between alpha of 0.4 and 0.8 was quite large in terms of both loss and validation f1\n\nI tried various forms of warping, rotations, scaling, noise, sensor drift, hand flipping, but I didnt really seem to find anything that boosted things. \n\n## Ensembling and Postprocessing\nI think the main contribution I had to the team was my postprocessing. My modeling was ok, but lightsource's was significantly better on imu and my whole ensemble was approaching similar full performance but it was with brute force and many models being ensembled together. When we merged a simple average between our solutions gave us a huge boost, but when I ported over my postprocessing to our combined solution that was when we were able to take first for a while. The biggest thing was just taking advantage of the relationship between subject gesture orientation constraints. We saw from our training data that only two gestures were done per orientation at most and locking this in significantly helped our score. The second piece on top of this was making sure illegal pairings were never predicted. 21 orientation gesture combos were never done in our data so turning these off didnt have a huge effect but it was still a decent gain in reference to how close our leaderboard was. \n\nWe got another big boost when we converting the probablities that we were averaging before with a conversion to ranking. This was rather simple. Just assigning 18/18 to the highest confidence class 17/18 to the next and 16/18 to the third and so on. It was surprising to me that this was better than giving the models some expressive power by giving probability to each class, but maybe this worked to calibrate predictions per model. I noticed that lightsources predictions were way more confident than mine. Mine were often extremely blurry decision boundaries, probably because of the extremely high mixup. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fa4fe2da9cee47e87a495f388214eb8da%2FAbove_ear___pull_hair_SEQ_002046_97timesteps.png?generation=1756882731104941&alt=media)",
    "3280801": "Can you please share how much boost you got from postprocessing on Public LB? And how much from converting probabilities to ranking? If you have the data. And congrats",
    "3280808": "On public leaderboard we were about 0.87 with no postprocessing and then 0.880 with. Then converting to ranking boosted us to 0.888 and then a couple extra models in the mix and tuning our threshold got us to 0.891. This was shaky in the end though, because we know we had probed a little bit extra out by fiddling with thresholds",
    "3280811": "Nice. Yeah if you probed for thresholds you deserve to be shaky! 🤣",
    "3280817": "Wasn't the intention to inflate our score. Just testing for variation between runs and if the thresholding interacted with lb like we saw on cv or not and ended up with some lucky runs that boosted our public ranking. We roughly knew where we actually lined up.",
    "3280818": "Well this competition is also a bit shaky inherently. My best ensemble was 0.865/0.854, second best 0.86/0.857 public/private. And both were chosen only based on local validation.",
    "3280824": "We had a couple that scored 0.871 on private. In hindsight we should have picked them, they had newer versions trained on all of the data with my best model. They tanked the public leaderboard enough that I didnt fully trust them, because I had no local validation on them.",
    "3280828": "your github link seems private @ryches",
    "3280831": "Fixed it now. Should be public"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 102335,
  "competition_slug": "cmi-detect-behavior-with-sensor-data",
  "team_id": 13944810,
  "solution_url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/11th-place-solution"
}