Thanks to everyone and congratulations to the winners! Let me give a quick summary of my solution and share the code.
\nThe target for all my models was a combination of the 3: gesture, orientation and hidden Relax&Move / Move from the initial state of \"behavior\" column. This new target has 18 * 4 * 2 potential values and no subject had more than 1 sequence of each value recorded. That was the basis of the post-processing I did.
\nDuring the inference I accumulate the history of predicted probabilities per subject. When a new subject comes in, I run a maximum llh optimization under the constraint that each target class can occur no more than once. You can use the optimization output of the new record as the prediction, but to get a tiny improvement on early records, when the history is too short yet, I just force set probabilities of the classes from past sequences to be zero for the currently processed sequence. Then I sum probabilities by gesture and take argmax.
\nAs the data came in random order, that created a significant variance in the scores, final choice was kind of a lottery. Resubmits of my final solution scored from 0.868 to 0.880, I ended up selecting 0.876.
I used a blend of 2 model architectures, each heavily bagged. I separately training imu-only and full-data models, using the same architectures with thm and tof branches cut for imu-only variants. Imu-only models gave a very good boost when blended with full-data models.
\n1) cnn - attention - pooling architecture with separate branches for velocity, position, thm, thm diff, tof and tof diff branches. You can check the code if you are curious about details, but I think there was nothing special there.
\n2) Bert-based architecture and derived features borrowed from https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model. Kudos to @wasupandceacar for publishing it! I only dropped some of imu derived variables as they hurt my cv score.
I am not sure how big was the benefit of the augmentations was given the high variance of LB scores, but I used:
\nQuite straightforward here, I didn't find value in complex derived features on cv, so I used a very limited set of features, like scaled thm and tof, acceleration without gravity components and angles derived from rot vectors. There were a couple of records with missing rot vectors, for them I used some rough estimates from acc, and also removed moving average from acc vectors to move the values closer to the desired distribution.
\nAnd, of course, flipping the values of some variables for handedness == 0 subjects.
I am sharing my code in full here https://www.kaggle.com/code/dott1718/cmikn-ensemble3b-re5
\nThis is the inference notebook of my top private score (the resubmit I picked). The attached datasets contain the model weights as well as the code for feature engineering, training and inference. You can also find a lot of repetitive and redundant code of my failed ideas there :)
Thanks to everyone and congratulations to the winners! Let me give a quick summary of my solution and share the code.
\nThe target for all my models was a combination of the 3: gesture, orientation and hidden Relax&Move / Move from the initial state of \"behavior\" column. This new target has 18 * 4 * 2 potential values and no subject had more than 1 sequence of each value recorded. That was the basis of the post-processing I did.
\nDuring the inference I accumulate the history of predicted probabilities per subject. When a new subject comes in, I run a maximum llh optimization under the constraint that each target class can occur no more than once. You can use the optimization output of the new record as the prediction, but to get a tiny improvement on early records, when the history is too short yet, I just force set probabilities of the classes from past sequences to be zero for the currently processed sequence. Then I sum probabilities by gesture and take argmax.
\nAs the data came in random order, that created a significant variance in the scores, final choice was kind of a lottery. Resubmits of my final solution scored from 0.868 to 0.880, I ended up selecting 0.876.
I used a blend of 2 model architectures, each heavily bagged. I separately training imu-only and full-data models, using the same architectures with thm and tof branches cut for imu-only variants. Imu-only models gave a very good boost when blended with full-data models.
\n1) cnn - attention - pooling architecture with separate branches for velocity, position, thm, thm diff, tof and tof diff branches. You can check the code if you are curious about details, but I think there was nothing special there.
\n2) Bert-based architecture and derived features borrowed from https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model. Kudos to @wasupandceacar for publishing it! I only dropped some of imu derived variables as they hurt my cv score.
I am not sure how big was the benefit of the augmentations was given the high variance of LB scores, but I used:
\nQuite straightforward here, I didn't find value in complex derived features on cv, so I used a very limited set of features, like scaled thm and tof, acceleration without gravity components and angles derived from rot vectors. There were a couple of records with missing rot vectors, for them I used some rough estimates from acc, and also removed moving average from acc vectors to move the values closer to the desired distribution.
\nAnd, of course, flipping the values of some variables for handedness == 0 subjects.
I am sharing my code in full here https://www.kaggle.com/code/dott1718/cmikn-ensemble3b-re5
\nThis is the inference notebook of my top private score (the resubmit I picked). The attached datasets contain the model weights as well as the code for feature engineering, training and inference. You can also find a lot of repetitive and redundant code of my failed ideas there :)
First of all, congrats to you, and I appreciate your point of view on codes and about main processing.
", "rawMarkdown": "First of all, congrats to you, and I appreciate your point of view on codes and about main processing.", "votes": null }, { "id": "3281000", "postDate": "09/03/2025 12:29:13", "content": "Congratulations! When you increased the number of classes from ~18 to ~144 (new target) how did you evaluate this new model and what was its performance? For example, which metric and loss did you choose and what was its score
", "rawMarkdown": "Congratulations! When you increased the number of classes from ~18 to ~144 (new target) how did you evaluate this new model and what was its performance? For example, which metric and loss did you choose and what was its score", "votes": null }, { "id": "3281254", "postDate": "09/03/2025 23:28:16", "content": "I don't have the numbers of how much boost exactly it gave me, because it was during a continuous process of improving multiple parts. I switched to a different target at some point when I figured out the post-processing that worked the best for me on cv. Moreover, I discovered the 3rd part of the target (behavior) quite late, so I had to retrain all the models again.
\nI used the standard cross entropy loss, which made the loss values incomparable across models after switching the targets. I relied on the competition metric for my cv. To be more focused on the end score, after switching to the new target I've always measured the metric after post-processing.
Thanks for the reply. My question was more about if adding significantly more classes it decreased the loss or metric performance as there would be less samples of each class. Did you experience any of this?
", "rawMarkdown": "Thanks for the reply. My question was more about if adding significantly more classes it decreased the loss or metric performance as there would be less samples of each class. Did you experience any of this?", "votes": null }, { "id": "3281773", "postDate": "09/04/2025 22:42:13", "content": "Got it. The alternative would be to have 3 outputs in the network with 18, 4 and 2 classes. I tried both approaches at some point and having a single output with large number of classes was at least not worse on cv in terms of the target metric. My intuition here is that with more classes the model is more flexible and does not suffer much from decreased number of observations per class.
\nA few years back there was a competition where in the same situation using a single output with more classes gave a significant boost in model accuracy https://www.kaggle.com/competitions/bengaliai-cv19/overview.
this will help us learn thank you for sharing
", "rawMarkdown": "this will help us learn thank you for sharing", "votes": null } ], "comments": [ { "id": 3280907, "author_name": "muhammaddanyalmalik", "author_url": "", "post_date": "09/03/2025 10:50:02", "content": "First of all, congrats to you, and I appreciate your point of view on codes and about main processing.
", "votes": null, "replies": [] }, { "id": 3281000, "author_name": "alejopaullier", "author_url": "", "post_date": "09/03/2025 12:29:13", "content": "Congratulations! When you increased the number of classes from ~18 to ~144 (new target) how did you evaluate this new model and what was its performance? For example, which metric and loss did you choose and what was its score
", "votes": null, "replies": [ { "id": 3281254, "author_name": "dott1718", "author_url": "", "post_date": "09/03/2025 23:28:16", "content": "I don't have the numbers of how much boost exactly it gave me, because it was during a continuous process of improving multiple parts. I switched to a different target at some point when I figured out the post-processing that worked the best for me on cv. Moreover, I discovered the 3rd part of the target (behavior) quite late, so I had to retrain all the models again.
\nI used the standard cross entropy loss, which made the loss values incomparable across models after switching the targets. I relied on the competition metric for my cv. To be more focused on the end score, after switching to the new target I've always measured the metric after post-processing.
Thanks for the reply. My question was more about if adding significantly more classes it decreased the loss or metric performance as there would be less samples of each class. Did you experience any of this?
", "votes": null, "replies": [ { "id": 3281773, "author_name": "dott1718", "author_url": "", "post_date": "09/04/2025 22:42:13", "content": "Got it. The alternative would be to have 3 outputs in the network with 18, 4 and 2 classes. I tried both approaches at some point and having a single output with large number of classes was at least not worse on cv in terms of the target metric. My intuition here is that with more classes the model is more flexible and does not suffer much from decreased number of observations per class.
\nA few years back there was a competition where in the same situation using a single output with more classes gave a significant boost in model accuracy https://www.kaggle.com/competitions/bengaliai-cv19/overview.
this will help us learn thank you for sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3280888": "Thanks to everyone and congratulations to the winners! Let me give a quick summary of my solution and share the code.\n\n# Target\nThe target for all my models was a combination of the 3: gesture, orientation and hidden Relax&Move / Move from the initial state of \"behavior\" column. This new target has 18 * 4 * 2 potential values and no subject had more than 1 sequence of each value recorded. That was the basis of the post-processing I did.\n\n# Post-processing\nDuring the inference I accumulate the history of predicted probabilities per subject. When a new subject comes in, I run a maximum llh optimization under the constraint that each target class can occur no more than once. You can use the optimization output of the new record as the prediction, but to get a tiny improvement on early records, when the history is too short yet, I just force set probabilities of the classes from past sequences to be zero for the currently processed sequence. Then I sum probabilities by gesture and take argmax.\nAs the data came in random order, that created a significant variance in the scores, final choice was kind of a lottery. Resubmits of my final solution scored from 0.868 to 0.880, I ended up selecting 0.876.\n\n# Models\nI used a blend of 2 model architectures, each heavily bagged. I separately training imu-only and full-data models, using the same architectures with thm and tof branches cut for imu-only variants. Imu-only models gave a very good boost when blended with full-data models.\n1) cnn - attention - pooling architecture with separate branches for velocity, position, thm, thm diff, tof and tof diff branches. You can check the code if you are curious about details, but I think there was nothing special there.\n2) Bert-based architecture and derived features borrowed from https://www.kaggle.com/code/wasupandceacar/lb-0-82-5fold-single-bert-model. Kudos to @wasupandceacar for publishing it! I only dropped some of imu derived variables as they hurt my cv score.\n\n# Augmentations\nI am not sure how big was the benefit of the augmentations was given the high variance of LB scores, but I used:\n- Drop out of first records with phase == \"Transition\"\n- Drop out of the last records with phase == \"Gesture\"\n- Random noise\n- Mixup of \"Transition\" and \"Gesture\" parts from different training records having the same target\n\n# Feature engineering\nQuite straightforward here, I didn't find value in complex derived features on cv, so I used a very limited set of features, like scaled thm and tof, acceleration without gravity components and angles derived from rot vectors. There were a couple of records with missing rot vectors, for them I used some rough estimates from acc, and also removed moving average from acc vectors to move the values closer to the desired distribution.\nAnd, of course, flipping the values of some variables for handedness == 0 subjects.\n\nI am sharing my code in full here https://www.kaggle.com/code/dott1718/cmikn-ensemble3b-re5\nThis is the inference notebook of my top private score (the resubmit I picked). The attached datasets contain the model weights as well as the code for feature engineering, training and inference. You can also find a lot of repetitive and redundant code of my failed ideas there :)", "3280907": "First of all, congrats to you, and I appreciate your point of view on codes and about main processing.", "3281000": "Congratulations! When you increased the number of classes from ~18 to ~144 (new target) how did you evaluate this new model and what was its performance? For example, which metric and loss did you choose and what was its score", "3281254": "I don't have the numbers of how much boost exactly it gave me, because it was during a continuous process of improving multiple parts. I switched to a different target at some point when I figured out the post-processing that worked the best for me on cv. Moreover, I discovered the 3rd part of the target (behavior) quite late, so I had to retrain all the models again.\nI used the standard cross entropy loss, which made the loss values incomparable across models after switching the targets. I relied on the competition metric for my cv. To be more focused on the end score, after switching to the new target I've always measured the metric after post-processing.", "3281264": "Thanks for the reply. My question was more about if adding significantly more classes it decreased the loss or metric performance as there would be less samples of each class. Did you experience any of this?", "3281773": "Got it. The alternative would be to have 3 outputs in the network with 18, 4 and 2 classes. I tried both approaches at some point and having a single output with large number of classes was at least not worse on cv in terms of the target metric. My intuition here is that with more classes the model is more flexible and does not suffer much from decreased number of observations per class.\nA few years back there was a competition where in the same situation using a single output with more classes gave a significant boost in model accuracy https://www.kaggle.com/competitions/bengaliai-cv19/overview.", "3282143": "this will help us learn thank you for sharing" }, "source": "existing_discussion", "complete": true, "competition_id": 102335, "competition_slug": "cmi-detect-behavior-with-sensor-data", "team_id": 13904657, "solution_url": "/competitions/cmi-detect-behavior-with-sensor-data/writeups/4th-place-solution" }