![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4671348%2Fa5238a6b31f34a1af26d3491a28aaf48%2FBiMamba.png?generation=1757130865992885&alt=media\")
First and foremost, I would like to extend my gratitude to the organizing team and staff for planning such a wonderful competition.
\nI also want to thank all the participants who shared numerous insightful findings.
Since the evaluation metric was unstable and there was a risk of leaderboard shake, and because single-model inference was fairly lightweight, my main strategy was to develop a variety of models and combine them into a robust ensemble with hill climbing.
\nFrom the beginning, I focused on creating a diverse set of models for ensembling, considering the unstable metric and the short inference time of a single model. I used a Triple Branch Model when ToF/Thermal data was available and a Single Branch Model for IMU-only cases.
\nI developed this model inspired by recent research showcasing the effectiveness of Mamba-based architectures in Human Activity Recognition.
\nTo capture the time-series features of each sensor (IMU, ToF, Thermal), I designed three independent encoder branches. Each encoder consists of Bidirectional Mamba Blocks. While many papers make only the State Space Model (SSM) part bidirectional, my experiments showed that making the entire Mamba block bidirectional resulted in better performance. An ablation study on this Mamba Block showed a +0.005 improvement in CV score.
\nAfter capturing long-term dependencies with Mamba in each branch, a Transformer Encoder aggregates them. This focuses not on temporal dependencies, but on capturing inter-sensor interactions at each timestep.
\nThink of it as experts from IMU, ToF, and Thermal domains discussing their initial findings to form a consensus. An ablation study on this Transformer Encoder also showed a +0.005 improvement in CV score.
For IMU-only data, a model based on an SE-Transformer block outperformed the Mamba-based one. This architecture applies a Squeeze-and-Excitation (SE) block after a Transformer Encoder to learn channel-wise interactions. Ablation studies demonstrated a +0.01 improvement from the Transformer Encoder and +0.005 from the SE block.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4671348%2Fe4be1976681d22670e6f9a988b8a119b%2FSETransformer.png?generation=1757130881445358&alt=media\"){kind=spacer}
Basically, I used the following features:
\nHowever, for sequences where rot data was missing, results using rot-related features (like linear_acc, angular_vel) became deteriorated.
\nFor these cases, I switched to an acc-only feature set. This set included 1st and 2nd derivatives of acceleration and rotation proxy features mimicking cross-products of acceleration vectors.
\nThese additional features significantly improved the accuracy on sequences with missing rot values.
To address the scarcity of data from left-handed subjects, I flipped the relevant axes to treat it as right-handed data.
\nThis allows the model to focus on the gesture itself, regardless of handedness.
As pointed out in the discussions, two subjects (SUBJ_019262, SUBJ_045235) likely wore the device upside-down.
\nI corrected their data for training.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4671348%2F69ae3b3c8652b3a92bc1a7d2c67006e0%2Fconfusion_matrix_with_tof_thm.png?generation=1757130920358613&alt=media\")
I trained a large number of diverse models (using different feature sets, architectures, and random seeds).
\nThen I used Hill Climbing to find the optimal combination of models and their weights that maximized the CV score. The final submission was an ensemble of approximately 20 models.
Final day in the competition, I realized that top competitors might using a trick based on the constraint that each gesture appears exactly limited times within a subject's session…
\nWith limited time for thorough validation, I opted for a safer approach: for each subject, if the predicted count for a gesture exceeded 8, I changed the prediction with the lowest logit to the class with the second-highest logit.
\nThis simple post-processing step alone improved my LB score by +0.003.
Inspired by several researches (e.g., INERTIAL HALLUCINATIONS), I explored a technique to mimic intermediate representations of ToF/Thermal data using only IMU data.
\nIt resulted in a slight CV improvement but was not significant.
This method worked very well for EEG data in the HMS competition, but it didn’t perform as well in this competition.
\nThat might be because of the large differences in movement rhythms between individuals, or because the raw waveform values played a more important role.
I attempted to use a specialized model in a second stage to classify confusable gestures, but this approach was not successful.
", "messages": [ { "id": "3282529", "postDate": "09/06/2025 12:33:48", "content": "First and foremost, I would like to extend my gratitude to the organizing team and staff for planning such a wonderful competition.
\nI also want to thank all the participants who shared numerous insightful findings.
Since the evaluation metric was unstable and there was a risk of leaderboard shake, and because single-model inference was fairly lightweight, my main strategy was to develop a variety of models and combine them into a robust ensemble with hill climbing.
\nFrom the beginning, I focused on creating a diverse set of models for ensembling, considering the unstable metric and the short inference time of a single model. I used a Triple Branch Model when ToF/Thermal data was available and a Single Branch Model for IMU-only cases.
\nI developed this model inspired by recent research showcasing the effectiveness of Mamba-based architectures in Human Activity Recognition.
\nTo capture the time-series features of each sensor (IMU, ToF, Thermal), I designed three independent encoder branches. Each encoder consists of Bidirectional Mamba Blocks. While many papers make only the State Space Model (SSM) part bidirectional, my experiments showed that making the entire Mamba block bidirectional resulted in better performance. An ablation study on this Mamba Block showed a +0.005 improvement in CV score.
\nAfter capturing long-term dependencies with Mamba in each branch, a Transformer Encoder aggregates them. This focuses not on temporal dependencies, but on capturing inter-sensor interactions at each timestep.
\nThink of it as experts from IMU, ToF, and Thermal domains discussing their initial findings to form a consensus. An ablation study on this Transformer Encoder also showed a +0.005 improvement in CV score.
For IMU-only data, a model based on an SE-Transformer block outperformed the Mamba-based one. This architecture applies a Squeeze-and-Excitation (SE) block after a Transformer Encoder to learn channel-wise interactions. Ablation studies demonstrated a +0.01 improvement from the Transformer Encoder and +0.005 from the SE block.
\n
Basically, I used the following features:
\nHowever, for sequences where rot data was missing, results using rot-related features (like linear_acc, angular_vel) became deteriorated.
\nFor these cases, I switched to an acc-only feature set. This set included 1st and 2nd derivatives of acceleration and rotation proxy features mimicking cross-products of acceleration vectors.
\nThese additional features significantly improved the accuracy on sequences with missing rot values.
To address the scarcity of data from left-handed subjects, I flipped the relevant axes to treat it as right-handed data.
\nThis allows the model to focus on the gesture itself, regardless of handedness.
As pointed out in the discussions, two subjects (SUBJ_019262, SUBJ_045235) likely wore the device upside-down.
\nI corrected their data for training.
I trained a large number of diverse models (using different feature sets, architectures, and random seeds).
\nThen I used Hill Climbing to find the optimal combination of models and their weights that maximized the CV score. The final submission was an ensemble of approximately 20 models.
Final day in the competition, I realized that top competitors might using a trick based on the constraint that each gesture appears exactly limited times within a subject's session…
\nWith limited time for thorough validation, I opted for a safer approach: for each subject, if the predicted count for a gesture exceeded 8, I changed the prediction with the lowest logit to the class with the second-highest logit.
\nThis simple post-processing step alone improved my LB score by +0.003.
Inspired by several researches (e.g., INERTIAL HALLUCINATIONS), I explored a technique to mimic intermediate representations of ToF/Thermal data using only IMU data.
\nIt resulted in a slight CV improvement but was not significant.
This method worked very well for EEG data in the HMS competition, but it didn’t perform as well in this competition.
\nThat might be because of the large differences in movement rhythms between individuals, or because the raw waveform values played a more important role.
I attempted to use a specialized model in a second stage to classify confusable gestures, but this approach was not successful.
", "rawMarkdown": "First and foremost, I would like to extend my gratitude to the organizing team and staff for planning such a wonderful competition. \nI also want to thank all the participants who shared numerous insightful findings.\n\n---\n## Overview\nSince the evaluation metric was unstable and there was a risk of leaderboard shake, and because single-model inference was fairly lightweight, my main strategy was to develop a variety of models and combine them into a robust ensemble with hill climbing.\n\n#### Model\n- Triple Branch Model (IMU + ToF + THM)\n - **Learns long-term temporal dependencies** for each sensor modality using a **Mamba-based architecture.**\n - **Captures sensor interactions** at each timestep using a **Transformer Encoder**.\n- Single Branch Model (IMU only)\n - Based on an **SE-Transformer architecture**, which yielded the better accuracy for IMU-only sequences.\n\n#### Feature Engineering & Data Processing\n- Handedness Correction: Flipped data axes for left-handed subjects to standardize the dataset.\n- Device Orientation Correction: Corrected data from subjects who wore the device upside-down.\n- Handling Missing Data: Generated a variety of acceleration-based features for sequences with missing rotation (rot) data.\n\n\n### Final Score\n- CV: **0.9027** (Full Model) / **0.8537** (IMU only Model)\n- LB: **0.873**\n- PB: **0.861**\n\n---\n## 1. Model\nFrom the beginning, I focused on creating a diverse set of models for ensembling, considering the unstable metric and the short inference time of a single model. I used a Triple Branch Model when ToF/Thermal data was available and a Single Branch Model for IMU-only cases.\n\n### 1.1. Triple Branch Model (Mamba + Transformer)\nI developed this model inspired by recent research showcasing the effectiveness of Mamba-based architectures in Human Activity Recognition.\n\n#### Bidirectional Mamba Block:\nTo capture the time-series features of each sensor (IMU, ToF, Thermal), I designed three independent encoder branches. Each encoder consists of Bidirectional Mamba Blocks. While many papers make only the State Space Model (SSM) part bidirectional, my experiments showed that making the entire Mamba block bidirectional resulted in better performance. An ablation study on this Mamba Block showed a +0.005 improvement in CV score.\n\n#### Transformer Encoder for Sensor Fusion:\nAfter capturing long-term dependencies with Mamba in each branch, a Transformer Encoder aggregates them. This focuses not on temporal dependencies, but on capturing inter-sensor interactions at each timestep.\nThink of it as experts from IMU, ToF, and Thermal domains discussing their initial findings to form a consensus. An ablation study on this Transformer Encoder also showed a +0.005 improvement in CV score.\n\n\n\n### 1.2. Single Branch Model (SE-Transformer)\nFor IMU-only data, a model based on an SE-Transformer block outperformed the Mamba-based one. This architecture applies a Squeeze-and-Excitation (SE) block after a Transformer Encoder to learn channel-wise interactions. Ablation studies demonstrated a +0.01 improvement from the Transformer Encoder and +0.005 from the SE block.\n\n\n\n---\n## 2. Feature Engineering / Data Processing\n\n### 2.1. Feature Engineering\nBasically, I used the following features:\n- imu_cols: acc_x/y/z, linear_acc_x/y/z, angular_vel_x/y/z, angular_distance\n- tof_cols: mean, std, max, min, mean_delta, centroid_x/y for each ToF sensor (1-5)\n- thm_cols: raw value and delta for each Thermal sensor (1-5)\n\nHowever, for sequences where rot data was missing, results using rot-related features (like linear_acc, angular_vel) became deteriorated.\nFor these cases, I switched to an acc-only feature set. This set included 1st and 2nd derivatives of acceleration and rotation proxy features mimicking cross-products of acceleration vectors.\nThese additional features significantly improved the accuracy on sequences with missing rot values.\n- d_acc_x/y/z (Jerk)\n- d2_acc_x/y/z\n- rot_proxy_xy/yz/zx (e.g. rot_proxy_xy -> acc_x * d_acc_y - acc_y * d_acc_x)\n\n### 2.2. Data processing\n#### Left-handed Data Conversion:\nTo address the scarcity of data from left-handed subjects, I flipped the relevant axes to treat it as right-handed data.\nThis allows the model to focus on the gesture itself, regardless of handedness.\n- IMU: Flipped the sign of acc_x-related features and angular_vel_y/z.\n- ToF/THM: Swapped sensors 3 and 5.\n- This process was also applied during inference.\n\n#### Upside-down Data Correction:\nAs pointed out in the discussions, two subjects (SUBJ_019262, SUBJ_045235) likely wore the device upside-down.\nI corrected their data for training.\n- IMU: Flipped the sign of features related to the x and y axes.\n- ToF/THM: Swapped sensors 2 and 4, and sensors 3 and 5, respectively.\n\n---\n## 3. Training\n- Fold: StratifiedGroupKFold (n_splits=5)\n- Loss Function: Soft Macro F1 Loss + Hard Margin Loss\n - The Hard Margin Loss was introduced to push the decision boundary for difficult-to-distinguish samples (e.g., \"Eyelash\" vs. \"Eyebrow\"), which was a notable issue in the confusion matrix.\n\n- Learning rate: 7.5e-4\n- Batch size: 64\n- Optimizer: AdamW\n- Scheduler: CosineAnnealingWarmRestarts\n- EMA (Exponential Moving Average): \n - By using EMA(ModelEmaV3), more stable training and improved generalization.\n - decay: 0.9993 \n- Hard Mining: \n - In a 2nd stage of training, I increased the sampling rate of hard-to-classify samples identified by the model from the 1st stage. This led to a slight improvement in performance.\n\n\n---\n## 4. Ensemble\nI trained a large number of diverse models (using different feature sets, architectures, and random seeds). \nThen I used Hill Climbing to find the optimal combination of models and their weights that maximized the CV score. The final submission was an ensemble of approximately 20 models.\n- I thought the hill climbing score had some risk of overfitting, so instead of relying only on the CV score,\n I also considered the variety of models, the balance of weights, and the LB score.\n As a result, this balanced choice turned out to be my best private submission.\n\n\n---\n## 5. Post-processing\nFinal day in the competition, I realized that top competitors might using a trick based on the constraint that each gesture appears exactly limited times within a subject's session...\nWith limited time for thorough validation, I opted for a safer approach: for each subject, if the predicted count for a gesture exceeded 8, I changed the prediction with the lowest logit to the class with the second-highest logit. \nThis simple post-processing step alone improved my LB score by +0.003.\n\n---\n## 6. Other Ideas (Did Not Work)\n### Modality Hallucination\nInspired by several researches (e.g., [INERTIAL HALLUCINATIONS](https://arxiv.org/pdf/2207.06789)), I explored a technique to mimic intermediate representations of ToF/Thermal data using only IMU data. \nIt resulted in a slight CV improvement but was not significant.\n### Spectrogram Features\nThis method worked very well for EEG data in the HMS competition, but it didn’t perform as well in this competition.\nThat might be because of the large differences in movement rhythms between individuals, or because the raw waveform values played a more important role.\n### 2-stage Model for Hard Classes\nI attempted to use a specialized model in a second stage to classify confusable gestures, but this approach was not successful.", "votes": null }, { "id": "3282597", "postDate": "09/06/2025 14:31:33", "content": "@kurisew
\nThank you for your comment!
\nI moved my write-up to this Solution page, so I’m sharing again my replies to your questions that were raised in the earlier Discussion page.
\n\nCongrates to the solo gold medal! About ensemble, may I ask whether you tried stacking by using OOF, or other ensemble strategy you have tried, and why you choose hill climbing ensemble?
\n
I haven't evaluated OOF stacking this time, so I can’t say which one performs better.
\nThat said, I chose hill climbing mainly for the following reasons:
It doesn’t require preparing a separate model for stacking, and the weights are adjusted automatically, which makes it easier to manage.
\nSince the evaluation metric is relatively unstable, I felt that OOF stacking might have a higher risk of overfitting
\n(Hill climbing also has a risk of overfitting, so this is just my view)
@kurisew
\nThank you for your comment!
\nI moved my write-up to this Solution page, so I’m sharing again my replies to your questions that were raised in the earlier Discussion page.
\n\nCongrates to the solo gold medal! About ensemble, may I ask whether you tried stacking by using OOF, or other ensemble strategy you have tried, and why you choose hill climbing ensemble?
\n
I haven't evaluated OOF stacking this time, so I can’t say which one performs better.
\nThat said, I chose hill climbing mainly for the following reasons:
It doesn’t require preparing a separate model for stacking, and the weights are adjusted automatically, which makes it easier to manage.
\nSince the evaluation metric is relatively unstable, I felt that OOF stacking might have a higher risk of overfitting
\n(Hill climbing also has a risk of overfitting, so this is just my view)