![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F745525%2F29376aa0fc9e5f0e4d918a13d112ada7%2FGemini_Generated_Image_su9owosu9owosu9o.jpg?generation=1765878783281637&alt=media\")
\nThis illustration was created with reference to this article.\nThis illustration was created with reference to this article.
We would like to express our gratitude to the competition host and the Kaggle staff for organizing this outstanding competition.\nBelow, we introduce yu4u’s part of the solution from Team monnu and yu4u. Please refer to monnu’s awesome solution here.\nOur submission is a weighted ensemble of monnu's models, yu4u's models, and a public notebook.
\nWhen we started working on this competition, we first explored how to design an architecture that could handle the diversity of Labs, trackers, annotated actions, and both self-actions and interactions.\nAs a result, we decided to adopt an approach in which a single model is trained jointly on all labs, trackers, and actions.\nIt may be worth noting that even self-actions and interactions are handled together within this one model - although of course many of you might already be doing something similar.
\nMy model makes independent predictions for each mouse pair, even when four mice are present.\nTherefore, for each video, we extracted features for all pairs listed in train.csv.\nWe extracted features from the keypoint coordinates.
\nSince the available keypoint types differ across trackers, we selected the following 11 keypoints that are available as consistently as possible across trackers:\near_left, ear_right, tail_base, nose, hip_left, hip_right, neck, body_center, lateral_left, lateral_right, and tail_tip.\nAs preprocessing, we resampled all frames to 30 fps and normalized the coordinates by video_width_pix.
Then, the following features were extracted:
\n| Feature | \nChannels | \nDescription | \n
|---|---|---|
| Velocity (dx, dy) | \n22 | \nVelocity vectors for each keypoint (11 body parts × 2 coords). | \n
| Velocity (norm) | \n11 | \nSpeed magnitude for each keypoint (11 body parts). | \n
| Acceleration | \n11 | \nAcceleration for each keypoint (11 body parts). | \n
| Relative Position | \n33 | \nTarget position in actor's coordinate system (dx, dy, distance) × 11 parts. | \n
| Metadata Embedding | \n72 | \nTracker ID, Arena ID, Lab ID, Mouse1/2's sex, strain, and color (9 attributes × 8 dims). | \n
All missing values were filled with -1.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F745525%2F22adcdc04740acd6ec13cf4da81a4e3f%2Fmodel.png?generation=1765843833369256&alt=media\")
The model architecture is shown in the figure above.\nThis model takes four types of inputs: mouse1’s features, mouse2’s features, relative features, and meta embeddings.\nThe features for mouse1 and mouse2 each consist of 44 channels: 22 for velocity, 11 for speed, and 11 for acceleration.\nThe relative features have 33 channels, and the metadata embeddings have 72 channels.\nWhen predicting self-actions, the same feature set is fed into both the mouse1 and mouse2 branches.
\nEach of these four feature types is first mapped to the same dimensionality by its own stem convolution, and then passed into the backbone network.\nA key characteristic of this model is that it alternates between Grouped Inverted Residual Blocks, which process the four inputs independently, and standard Inverted Residual Blocks, which mix information across them.\nThe Grouped Inverted Residual Blocks are implemented using grouped convolutions, allowing the model to learn features for each input type independently.
\nWe used three types of backbones: a convolution-based backbone, a GRU-based backbone, and a ModernBERT-based backbone. In all of them, the internal blocks are grouped.
\nSince the task in this competition is essentially a frame-wise classification problem, it is natural to use cross-entropy loss.\nHowever, actions that are not listed in train.csv (unannotated actions) cannot be explicitly treated as negative samples.\nTo address this, we add −∞ to the logits of unannotated actions before computing the cross-entropy loss. This way, the model is not penalized even if it predicts an unannotated action as positive.\nIn addition, we introduced separate background classes for self-actions and interactions.
\nWe pre-trained the backbone using a masked keypoint reconstruction task, where a subset of keypoints is masked and then predicted.\nWe conducted two types of pretraining: one including the MABe22 data and one excluding it.\nAlthough this did not yield a significant improvement in metrics compared with training from scratch, we used both pre-trained models in the ensemble, expecting them to increase model diversity.
\nAdamW optimizer with cosine annealing, 64 epochs, batch size = 256, learning rate = 5e-3, weight decay = 0.2, and an exponential moving average (EMA) decay of 0.997. 5-fold cross-validation stratified by lab_id.\nFor validation, we excluded these labs CalMS21_*, CRIM13, and MABe22, which are not included in the test data, and used the official competition metric.
| Model | \nPretrain | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|
| Conv | \nNone | \n0.5206 | \n\n | \n |
| Conv | \nw/o MABe22 | \n0.5305 | \n\n | \n |
| Conv | \nw/ MABe22 | \n0.5233 | \n\n | \n |
| GRU | \nNone | \n0.5221 | \n\n | \n |
| GRU | \nw/o MABe22 | \n0.5228 | \n\n | \n |
| GRU | \nw/ MABe22 | \n0.5225 | \n\n | \n |
| ModernBERT | \nNone | \n0.5150 | \n\n | \n |
| ModernBERT | \nw/o MABe22 | \n0.5225 | \n\n | \n |
| ModernBERT | \nw/ MABe22 | \n0.5192 | \n\n | \n |
| yu4u Ensemble | \n\n | 0.5430 | \n0.543 | \n0.508 | \n
| Team Ensemble | \n\n | 0.5702 | \n0.556 | \n0.525 | \n
\nThis illustration was created with reference to this article.
We would like to express our gratitude to the competition host and the Kaggle staff for organizing this outstanding competition.\nBelow, we introduce yu4u’s part of the solution from Team monnu and yu4u. Please refer to monnu’s awesome solution here.\nOur submission is a weighted ensemble of monnu's models, yu4u's models, and a public notebook.
\nWhen we started working on this competition, we first explored how to design an architecture that could handle the diversity of Labs, trackers, annotated actions, and both self-actions and interactions.\nAs a result, we decided to adopt an approach in which a single model is trained jointly on all labs, trackers, and actions.\nIt may be worth noting that even self-actions and interactions are handled together within this one model - although of course many of you might already be doing something similar.
\nMy model makes independent predictions for each mouse pair, even when four mice are present.\nTherefore, for each video, we extracted features for all pairs listed in train.csv.\nWe extracted features from the keypoint coordinates.
\nSince the available keypoint types differ across trackers, we selected the following 11 keypoints that are available as consistently as possible across trackers:\near_left, ear_right, tail_base, nose, hip_left, hip_right, neck, body_center, lateral_left, lateral_right, and tail_tip.\nAs preprocessing, we resampled all frames to 30 fps and normalized the coordinates by video_width_pix.
Then, the following features were extracted:
\n| Feature | \nChannels | \nDescription | \n
|---|---|---|
| Velocity (dx, dy) | \n22 | \nVelocity vectors for each keypoint (11 body parts × 2 coords). | \n
| Velocity (norm) | \n11 | \nSpeed magnitude for each keypoint (11 body parts). | \n
| Acceleration | \n11 | \nAcceleration for each keypoint (11 body parts). | \n
| Relative Position | \n33 | \nTarget position in actor's coordinate system (dx, dy, distance) × 11 parts. | \n
| Metadata Embedding | \n72 | \nTracker ID, Arena ID, Lab ID, Mouse1/2's sex, strain, and color (9 attributes × 8 dims). | \n
All missing values were filled with -1.
\n
The model architecture is shown in the figure above.\nThis model takes four types of inputs: mouse1’s features, mouse2’s features, relative features, and meta embeddings.\nThe features for mouse1 and mouse2 each consist of 44 channels: 22 for velocity, 11 for speed, and 11 for acceleration.\nThe relative features have 33 channels, and the metadata embeddings have 72 channels.\nWhen predicting self-actions, the same feature set is fed into both the mouse1 and mouse2 branches.
\nEach of these four feature types is first mapped to the same dimensionality by its own stem convolution, and then passed into the backbone network.\nA key characteristic of this model is that it alternates between Grouped Inverted Residual Blocks, which process the four inputs independently, and standard Inverted Residual Blocks, which mix information across them.\nThe Grouped Inverted Residual Blocks are implemented using grouped convolutions, allowing the model to learn features for each input type independently.
\nWe used three types of backbones: a convolution-based backbone, a GRU-based backbone, and a ModernBERT-based backbone. In all of them, the internal blocks are grouped.
\nSince the task in this competition is essentially a frame-wise classification problem, it is natural to use cross-entropy loss.\nHowever, actions that are not listed in train.csv (unannotated actions) cannot be explicitly treated as negative samples.\nTo address this, we add −∞ to the logits of unannotated actions before computing the cross-entropy loss. This way, the model is not penalized even if it predicts an unannotated action as positive.\nIn addition, we introduced separate background classes for self-actions and interactions.
\nWe pre-trained the backbone using a masked keypoint reconstruction task, where a subset of keypoints is masked and then predicted.\nWe conducted two types of pretraining: one including the MABe22 data and one excluding it.\nAlthough this did not yield a significant improvement in metrics compared with training from scratch, we used both pre-trained models in the ensemble, expecting them to increase model diversity.
\nAdamW optimizer with cosine annealing, 64 epochs, batch size = 256, learning rate = 5e-3, weight decay = 0.2, and an exponential moving average (EMA) decay of 0.997. 5-fold cross-validation stratified by lab_id.\nFor validation, we excluded these labs CalMS21_*, CRIM13, and MABe22, which are not included in the test data, and used the official competition metric.
| Model | \nPretrain | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|
| Conv | \nNone | \n0.5206 | \n\n | \n |
| Conv | \nw/o MABe22 | \n0.5305 | \n\n | \n |
| Conv | \nw/ MABe22 | \n0.5233 | \n\n | \n |
| GRU | \nNone | \n0.5221 | \n\n | \n |
| GRU | \nw/o MABe22 | \n0.5228 | \n\n | \n |
| GRU | \nw/ MABe22 | \n0.5225 | \n\n | \n |
| ModernBERT | \nNone | \n0.5150 | \n\n | \n |
| ModernBERT | \nw/o MABe22 | \n0.5225 | \n\n | \n |
| ModernBERT | \nw/ MABe22 | \n0.5192 | \n\n | \n |
| yu4u Ensemble | \n\n | 0.5430 | \n0.543 | \n0.508 | \n
| Team Ensemble | \n\n | 0.5702 | \n0.556 | \n0.525 | \n
Congratulations winning 6th place gold. I've never used a BERT transformer outside of NLP. How does it work, how do you adapt it to work here? I like how you pretrained the models with MABe unlabeled data.
\nHow did you handle AdaptableSnail fps=25 during test inference?
", "rawMarkdown": "Congratulations winning 6th place gold. I've never used a BERT transformer outside of NLP. How does it work, how do you adapt it to work here? I like how you pretrained the models with MABe unlabeled data.\n\nHow did you handle AdaptableSnail fps=25 during test inference?", "votes": null }, { "id": "3378361", "postDate": "12/18/2025 00:22:25", "content": "Congratulations to you as well on your 7th place finish.
\n\n\nI've never used a BERT transformer outside of NLP. How does it work, how do you adapt it to work here?
\n
I simply replaced the Inverted Residual Block with ModernBertEncoderLayer.\nNotably, I did not need to tune the hyperparameters at all; I could just reuse the same hyperparameters as for the 1D CNN, and I was able to achieve comparable scores while also gaining model diversity. This is based on a successful experience we had in CMI3.\nFor the grouped convolution, I created a wrapper layer that holds the same numer (=4) of ModernBertEncoderLayers as the group internally.
from transformers.models.modernbert.modeling_modernbert import ModernBertEncoderLayer
\n\nI like how you pretrained the models with MABe unlabeled data.
\n
I defined a task where I mask the original keypoints, extract features from those masked keypoints, and then reconstruct the original keypoints from those features. The loss is an MSE applied only on the masked keypoints.
\n\n\nHow did you handle AdaptableSnail fps=25 during test inference?
\n
During training, I excluded the 25 fps data, but at inference time I did nothing.\nIf I remove the 25 fps videos also at inference, the private LB score changes from 0.52524 -> 0.53039.\nIf I instead change the start/end to * 30 / 25, it goes from 0.52524 -> 0.52644.\nIf the original 25 fps data are not properly corrected and used for training, it might simply be better to drop them.
Congratulations winning 6th place gold. I've never used a BERT transformer outside of NLP. How does it work, how do you adapt it to work here? I like how you pretrained the models with MABe unlabeled data.
\nHow did you handle AdaptableSnail fps=25 during test inference?
", "votes": null, "replies": [ { "id": 3378361, "author_name": "ren4yu", "author_url": "", "post_date": "12/18/2025 00:22:25", "content": "Congratulations to you as well on your 7th place finish.
\n\n\nI've never used a BERT transformer outside of NLP. How does it work, how do you adapt it to work here?
\n
I simply replaced the Inverted Residual Block with ModernBertEncoderLayer.\nNotably, I did not need to tune the hyperparameters at all; I could just reuse the same hyperparameters as for the 1D CNN, and I was able to achieve comparable scores while also gaining model diversity. This is based on a successful experience we had in CMI3.\nFor the grouped convolution, I created a wrapper layer that holds the same numer (=4) of ModernBertEncoderLayers as the group internally.
from transformers.models.modernbert.modeling_modernbert import ModernBertEncoderLayer
\n\nI like how you pretrained the models with MABe unlabeled data.
\n
I defined a task where I mask the original keypoints, extract features from those masked keypoints, and then reconstruct the original keypoints from those features. The loss is an MSE applied only on the masked keypoints.
\n\n\nHow did you handle AdaptableSnail fps=25 during test inference?
\n
During training, I excluded the 25 fps data, but at inference time I did nothing.\nIf I remove the 25 fps videos also at inference, the private LB score changes from 0.52524 -> 0.53039.\nIf I instead change the start/end to * 30 / 25, it goes from 0.52524 -> 0.52644.\nIf the original 25 fps data are not properly corrected and used for training, it might simply be better to drop them.