{ "id": 603656, "title": "166th Place Solution", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/603656", "author_name": "xavier fernando", "post_date": "2025-09-03T12:45:04.311000", "votes": 4, "comment_count": 0, "views": 0, "content": "

First of all, thanks to the organizers and hosts for an interesting competition. I learned a lot during this competition, and I also appreciate all the participants who openly shared their work. My final solution is largely built upon ideas and code from public notebooks.

\n

Approach

\n
    \n

  1. BERT-based model

    \n

    Started with a BERT base model, and experimented with RoBERTa and DeBERTa configurations.

    \n

    Added extra feature interactions (addition, subtraction, multiplication,jerk..etc).

    \n

    Tried different pooling strategies (mean, CLS token, max pooling).

    \n

    Reference notebook: https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model

    2. \n

  2. CNN-hybrid based model

    \n

    created a CNN-based model, adding more features and experimenting with different architecture configurations.

    \n

    Reference notebook: https://www.kaggle.com/code/mahoganybuttstrings/5fold-single-model-with-split-sensors-train

    3. \n

  3. Model Combination

    \n

    Combined the BERT-based and CNN-based model architectures.

    4. \n

  4. Auxiliary Loss

    \n
    5. \n
\n

Train two models: one using only IMU data and another using the full dataset.
\nDuring submission, check the thm and tof values — if they are null, use the IMU-only model; otherwise, use the full-data model.

\n

imu only score:0.8045 - 10 models ensemple
\nfull data score:0.875 - 8 models ensemple

\n

Public LB:0.854
\nPrivate LB:0.829

\n

What didn’t work:

\n\n

training notebook: https://www.kaggle.com/code/xaviertce/cmi-training-4-models
\ninference notebook: https://www.kaggle.com/code/xaviertce/more-ensemple-0-08028-0-8725-imu-full?scriptVersionId=259610263
\ninference notebook catboost: https://www.kaggle.com/code/xaviertce/imu-features-cat

\n

other reference:
\nhttps://www.kaggle.com/code/peilwang/train-infer-pytorch

", "messages": [ { "id": 3281021, "postDate": "2025-09-03T12:45:04.310Z", "content": "

First of all, thanks to the organizers and hosts for an interesting competition. I learned a lot during this competition, and I also appreciate all the participants who openly shared their work. My final solution is largely built upon ideas and code from public notebooks.

\n

Approach

\n
    \n

  1. BERT-based model

    \n

    Started with a BERT base model, and experimented with RoBERTa and DeBERTa configurations.

    \n

    Added extra feature interactions (addition, subtraction, multiplication,jerk..etc).

    \n

    Tried different pooling strategies (mean, CLS token, max pooling).

    \n

    Reference notebook: https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model

    2. \n

  2. CNN-hybrid based model

    \n

    created a CNN-based model, adding more features and experimenting with different architecture configurations.

    \n

    Reference notebook: https://www.kaggle.com/code/mahoganybuttstrings/5fold-single-model-with-split-sensors-train

    3. \n

  3. Model Combination

    \n

    Combined the BERT-based and CNN-based model architectures.

    4. \n

  4. Auxiliary Loss

    \n
    5. \n
\n

Train two models: one using only IMU data and another using the full dataset.
\nDuring submission, check the thm and tof values — if they are null, use the IMU-only model; otherwise, use the full-data model.

\n

imu only score:0.8045 - 10 models ensemple
\nfull data score:0.875 - 8 models ensemple

\n

Public LB:0.854
\nPrivate LB:0.829

\n

What didn’t work:

\n\n

training notebook: https://www.kaggle.com/code/xaviertce/cmi-training-4-models
\ninference notebook: https://www.kaggle.com/code/xaviertce/more-ensemple-0-08028-0-8725-imu-full?scriptVersionId=259610263
\ninference notebook catboost: https://www.kaggle.com/code/xaviertce/imu-features-cat

\n

other reference:
\nhttps://www.kaggle.com/code/peilwang/train-infer-pytorch

", "rawMarkdown": "First of all, thanks to the organizers and hosts for an interesting competition. I learned a lot during this competition, and I also appreciate all the participants who openly shared their work. My final solution is largely built upon ideas and code from public notebooks.\n\nApproach\n1. BERT-based model\n\n\tStarted with a BERT base model, and experimented with RoBERTa and DeBERTa configurations.\n\n\tAdded extra feature interactions (addition, subtraction, multiplication,jerk..etc).\n\n\tTried different pooling strategies (mean, CLS token, max pooling).\n\n\tReference notebook: https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model\n\n\n2. CNN-hybrid based model\n\n\tcreated a CNN-based model, adding more features and experimenting with different architecture configurations.\n\n\tReference notebook: https://www.kaggle.com/code/mahoganybuttstrings/5fold-single-model-with-split-sensors-train\n\n\n3. Model Combination\n\n\tCombined the BERT-based and CNN-based model architectures.\n\n4. Auxiliary Loss\n\n\t- Introduced an auxiliary loss for all models, calculated using both the target and non-target outputs, which improved generalization.\n\t\nTrain two models: one using only IMU data and another using the full dataset.\nDuring submission, check the thm and tof values — if they are null, use the IMU-only model; otherwise, use the full-data model.\n\nimu only score:0.8045 - 10 models ensemple\nfull data score:0.875 - 8 models ensemple\n\nPublic LB:0.854\nPrivate LB:0.829\n\nWhat didn’t work:\n\n - Using CatBoost with more than 10k features for full data and 6k features for IMU.\n\n - IMU score: 0.7712, Full data score: 0.8211, Public LB: 0.794, Private LB: 0.795\n\n - Applying dynamic padding and truncation with 512 tokens — results got worse.\n - stacking with cat,lgb,xgb.then ensemple.\n\ntraining notebook: https://www.kaggle.com/code/xaviertce/cmi-training-4-models\ninference notebook: https://www.kaggle.com/code/xaviertce/more-ensemple-0-08028-0-8725-imu-full?scriptVersionId=259610263\ninference notebook catboost: https://www.kaggle.com/code/xaviertce/imu-features-cat\n\nother reference:\nhttps://www.kaggle.com/code/peilwang/train-infer-pytorch", "votes": 4 } ], "comments": [], "raw_markdown_by_id": { "3281021": "First of all, thanks to the organizers and hosts for an interesting competition. I learned a lot during this competition, and I also appreciate all the participants who openly shared their work. My final solution is largely built upon ideas and code from public notebooks.\n\nApproach\n1. BERT-based model\n\n\tStarted with a BERT base model, and experimented with RoBERTa and DeBERTa configurations.\n\n\tAdded extra feature interactions (addition, subtraction, multiplication,jerk..etc).\n\n\tTried different pooling strategies (mean, CLS token, max pooling).\n\n\tReference notebook: https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model\n\n\n2. CNN-hybrid based model\n\n\tcreated a CNN-based model, adding more features and experimenting with different architecture configurations.\n\n\tReference notebook: https://www.kaggle.com/code/mahoganybuttstrings/5fold-single-model-with-split-sensors-train\n\n\n3. Model Combination\n\n\tCombined the BERT-based and CNN-based model architectures.\n\n4. Auxiliary Loss\n\n\t- Introduced an auxiliary loss for all models, calculated using both the target and non-target outputs, which improved generalization.\n\t\nTrain two models: one using only IMU data and another using the full dataset.\nDuring submission, check the thm and tof values — if they are null, use the IMU-only model; otherwise, use the full-data model.\n\nimu only score:0.8045 - 10 models ensemple\nfull data score:0.875 - 8 models ensemple\n\nPublic LB:0.854\nPrivate LB:0.829\n\nWhat didn’t work:\n\n - Using CatBoost with more than 10k features for full data and 6k features for IMU.\n\n - IMU score: 0.7712, Full data score: 0.8211, Public LB: 0.794, Private LB: 0.795\n\n - Applying dynamic padding and truncation with 512 tokens — results got worse.\n - stacking with cat,lgb,xgb.then ensemple.\n\ntraining notebook: https://www.kaggle.com/code/xaviertce/cmi-training-4-models\ninference notebook: https://www.kaggle.com/code/xaviertce/more-ensemple-0-08028-0-8725-imu-full?scriptVersionId=259610263\ninference notebook catboost: https://www.kaggle.com/code/xaviertce/imu-features-cat\n\nother reference:\nhttps://www.kaggle.com/code/peilwang/train-infer-pytorch" }, "complete": true, "source": "existing_discussion", "competition_id": 102335, "competition_slug": "cmi-detect-behavior-with-sensor-data", "team_id": 14153047, "solution_url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/166th-place-solution" }