# 166th Place Solution

First of all, thanks to the organizers and hosts for an interesting competition. I learned a lot during this competition, and I also appreciate all the participants who openly shared their work. My final solution is largely built upon ideas and code from public notebooks.
Approach
BERT-based model
Started with a BERT base model, and experimented with RoBERTa and DeBERTa configurations.
Added extra feature interactions (addition, subtraction, multiplication,jerk..etc).
Tried different pooling strategies (mean, CLS token, max pooling).
Reference notebook:
[https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model](https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model)
CNN-hybrid based model
created a CNN-based model, adding more features and experimenting with different architecture configurations.
Reference notebook:
[https://www.kaggle.com/code/mahoganybuttstrings/5fold-single-model-with-split-sensors-train](https://www.kaggle.com/code/mahoganybuttstrings/5fold-single-model-with-split-sensors-train)
Model Combination
Combined the BERT-based and CNN-based model architectures.
Auxiliary Loss
Introduced an auxiliary loss for all models, calculated using both the target and non-target outputs, which improved generalization.
Train two models: one using only IMU data and another using the full dataset.
During submission, check the thm and tof values — if they are null, use the IMU-only model; otherwise, use the full-data model.
imu only score:0.8045 - 10 models ensemple
full data score:0.875 - 8 models ensemple
Public LB:0.854
Private LB:0.829
What didn’t work:
Using CatBoost with more than 10k features for full data and 6k features for IMU.
IMU score: 0.7712, Full data score: 0.8211, Public LB: 0.794, Private LB: 0.795
Applying dynamic padding and truncation with 512 tokens — results got worse.
stacking with cat,lgb,xgb.then ensemple.
training notebook:
[https://www.kaggle.com/code/xaviertce/cmi-training-4-models](https://www.kaggle.com/code/xaviertce/cmi-training-4-models)
inference notebook:
[https://www.kaggle.com/code/xaviertce/more-ensemple-0-08028-0-8725-imu-full?scriptVersionId=259610263](https://www.kaggle.com/code/xaviertce/more-ensemple-0-08028-0-8725-imu-full?scriptVersionId=259610263)
inference notebook catboost:
[https://www.kaggle.com/code/xaviertce/imu-features-cat](https://www.kaggle.com/code/xaviertce/imu-features-cat)
other reference:
[https://www.kaggle.com/code/peilwang/train-infer-pytorch](https://www.kaggle.com/code/peilwang/train-infer-pytorch)