![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1648129%2F7caddca4a01a12565048f45946638afe%2Fmice_results.png?generation=1765914104181718&alt=media\")
First of all thanks the host and Kaggle for hosting this competition! Also, a huge shoutout to @tomirol and @aerdem4 for the partnership and effort working in this challenge!
\nOur solution is based on both NN and XGB trained per lab with different setups and parameters. For most labs we simply pick the model with highest CV for that lab, except for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we perform majority voting frame-wise.
\nOur models' per-lab CV scores are the following:\n
Our code was originally forked from the great public notebook from @ravaghi, which was already a pretty strong baseline.\nBelow we list the main points of our solution and the differences from the original notebook:
\nnose, ear_left, ear_right, body_center and tail_base.body_center is not available we can derive it from hip_left + hip_right or lateral_left + lateral_right. As a last resort (only 2 labs) we interpolate it from tail_base + ears_l+r.nose is not available we simply replace it by headlab_id and action, i.e., binary classifier per actionaction threshold by optimizing the F1 score on that lab's OOF predictionsresidual_action_i = pred_action_i - threshold_action_i, then taking the argmax across all residual_action_i. Finally, for the highest residual we simply check if residual_action_i > 0 to choose between action_i and no action.group[\"x\"] = group[\"x\"].interpolate(method=\"linear\", limit_direction=\"both\")) across the frames in order the reduce the large amount of NaN in the input.For the single mouse models we performed feature engieering focusing on the topics below:
\nFor pair of mice we simply try to create features that would capture the relationship between the mice:
\nis_same_sex, is_same_color, etc..Although part of our feature engineering pipeline was already capturing some features at different windows, we found really beneficial to simply \"borrow\" the features from neighbour frames as well. We did it in two different ways: \"simple copy\" and \"statistics over window\".
\nThe first thing we tried was to simply use a sliding window to copy features from frames around frame_i. For example, we applied window=4 and stride=5 meaning that for frame_i we would concat the features from frames [frame_i-20, frame_i-15, frame_i-10, frame_i-5, frame_i, frame_i+5, frame_i+10, frame_i+15, frame_i+20].
Of course this approach lead to a sharp increase in the number of features but we found XGB really robust to high dimensionality. We denote the model trained with this \"simple copy\" approach as \"Matheus XGB\" in the picture with scores.
\nThe second option for temporal features we tried was to compute statistics over each window instead of copying the frame features as is. We used median, min, max and std as aggregators for each window. This model used window=2 and stride=15 and is denoted as \"Edu XGB\".
Similarly to our XGB pipeline, we standardized the sensor keypoints by:
\nlateral_left -> hip_leftlateral_right -> hip_righthead -> noseheadpiece and spineFor each 24 frame, we calculate the following features and provide it as main input to the model:
\nAfter the initial feature extraction backbone, we late inject extra features into the model:
\nRegarding the our architecutre, we opted for the following design choices:
\nGroupNormsInstanceNorm3d on the inputsELU activationsoftmax over all mice pairs including selfFor our training, the main considerations are the following:
\n(batch_size, features, n_mice, n_mice, seq_len)BCEWithLogitsLoss is used as loos function on masked target (no backprop on NaN targets)sqrt(binary_label_mean) to balance rare classesargmaxFor our final solution we simply selected, for each lab, the model with greatest CV (see picture for reference).\nThe only exceptions are for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we found beneficial to perform major voting across all 3 models.
", "messages": [ { "id": 3377697, "postDate": "2025-12-16T19:56:14.783Z", "content": "First of all thanks the host and Kaggle for hosting this competition! Also, a huge shoutout to @tomirol and @aerdem4 for the partnership and effort working in this challenge!
\nOur solution is based on both NN and XGB trained per lab with different setups and parameters. For most labs we simply pick the model with highest CV for that lab, except for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we perform majority voting frame-wise.
\nOur models' per-lab CV scores are the following:\n
Our code was originally forked from the great public notebook from @ravaghi, which was already a pretty strong baseline.\nBelow we list the main points of our solution and the differences from the original notebook:
\nnose, ear_left, ear_right, body_center and tail_base.body_center is not available we can derive it from hip_left + hip_right or lateral_left + lateral_right. As a last resort (only 2 labs) we interpolate it from tail_base + ears_l+r.nose is not available we simply replace it by headlab_id and action, i.e., binary classifier per actionaction threshold by optimizing the F1 score on that lab's OOF predictionsresidual_action_i = pred_action_i - threshold_action_i, then taking the argmax across all residual_action_i. Finally, for the highest residual we simply check if residual_action_i > 0 to choose between action_i and no action.group[\"x\"] = group[\"x\"].interpolate(method=\"linear\", limit_direction=\"both\")) across the frames in order the reduce the large amount of NaN in the input.For the single mouse models we performed feature engieering focusing on the topics below:
\nFor pair of mice we simply try to create features that would capture the relationship between the mice:
\nis_same_sex, is_same_color, etc..Although part of our feature engineering pipeline was already capturing some features at different windows, we found really beneficial to simply \"borrow\" the features from neighbour frames as well. We did it in two different ways: \"simple copy\" and \"statistics over window\".
\nThe first thing we tried was to simply use a sliding window to copy features from frames around frame_i. For example, we applied window=4 and stride=5 meaning that for frame_i we would concat the features from frames [frame_i-20, frame_i-15, frame_i-10, frame_i-5, frame_i, frame_i+5, frame_i+10, frame_i+15, frame_i+20].
Of course this approach lead to a sharp increase in the number of features but we found XGB really robust to high dimensionality. We denote the model trained with this \"simple copy\" approach as \"Matheus XGB\" in the picture with scores.
\nThe second option for temporal features we tried was to compute statistics over each window instead of copying the frame features as is. We used median, min, max and std as aggregators for each window. This model used window=2 and stride=15 and is denoted as \"Edu XGB\".
Similarly to our XGB pipeline, we standardized the sensor keypoints by:
\nlateral_left -> hip_leftlateral_right -> hip_righthead -> noseheadpiece and spineFor each 24 frame, we calculate the following features and provide it as main input to the model:
\nAfter the initial feature extraction backbone, we late inject extra features into the model:
\nRegarding the our architecutre, we opted for the following design choices:
\nGroupNormsInstanceNorm3d on the inputsELU activationsoftmax over all mice pairs including selfFor our training, the main considerations are the following:
\n(batch_size, features, n_mice, n_mice, seq_len)BCEWithLogitsLoss is used as loos function on masked target (no backprop on NaN targets)sqrt(binary_label_mean) to balance rare classesargmaxFor our final solution we simply selected, for each lab, the model with greatest CV (see picture for reference).\nThe only exceptions are for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we found beneficial to perform major voting across all 3 models.
", "rawMarkdown": "First of all thanks the host and Kaggle for hosting this competition! Also, a huge shoutout to @tomirol and @aerdem4 for the partnership and effort working in this challenge!\n\n# TLDR\nOur solution is based on both NN and XGB trained per lab with different setups and parameters. For most labs we simply pick the model with highest CV for that lab, except for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we perform majority voting frame-wise.\n\nOur models' per-lab CV scores are the following:\n\n\n# XGB Pipeline\nOur code was originally forked from the great [public notebook](https://www.kaggle.com/code/ravaghi/social-action-recognition-in-mice-xgboost) from @ravaghi, which was already a pretty strong baseline.\nBelow we list the main points of our solution and the differences from the original notebook:\n- We normalize the FPS of all videos to 30fps and use the mode to get the label in the window\n- We standardize all sensors to only use `nose`, `ear_left`, `ear_right`, `body_center` and `tail_base`.\n - For labs where `body_center` is not available we can derive it from `hip_left` + `hip_right` or `lateral_left` + `lateral_right`. As a last resort (only 2 labs) we interpolate it from `tail_base` + `ears_l+r`.\n - For labs where `nose` is not available we simply replace it by `head`\n- We train separated models from single mouse and pair setups\n- We train 3 groups of 4 models (4Fold CV groupped per video and stratified by action label) for each `lab_id` and `action`, i.e., binary classifier per action\n - We change the model initialization and KFold seed for each model group and average the probs in the end to get the prediction for the action\n - We compute the individual `action` threshold by optimizing the F1 score on that lab's OOF predictions\n - We combine the individual action predictions into multi-class by first computing the residual between prediction and its threshold, i.e., `residual_action_i = pred_action_i - threshold_action_i`, then taking the `argmax` across all `residual_action_i`. Finally, for the highest residual we simply check if `residual_action_i > 0` to choose between `action_i` and no action.\n - Since we train binary models for each action we simply ignore any frame where the label of the action in question is NaN\n- For \"CautiousGiraffe\", \"DeliriousFly\", \"GroovyShrew\" and \"TranquilPanther\" we preprocess the keypoints by temporally interpolating (`group[\"x\"] = group[\"x\"].interpolate(method=\"linear\", limit_direction=\"both\")`) across the frames in order the reduce the large amount of NaN in the input.\n - This preproc improved significantly the labs aforementioned but not for others, which we suspect could be due to a large amount of id-switch in the tracking system (it makes the interpolation results terrible)\n - When applying interpolation we also add binary features for each keypoint flagging if the value is interpolated or not\n\n## Single mouse features\nFor the single mouse models we performed feature engieering focusing on the topics below:\n- Pair-wise distance between sensor's keypoints\n- Velocity for ear and tail\n- Elongation vs. compactness features\n- Body angle features\n- Body center displacement features\n- Body center acceleration and jerk features across different windows\n- Angular velocity and turn-rate\n- Grooming specific features\n- Mouse metadata, e.g., sex, color, has device, etc..\n- Arena boundary feats, e.g., how close mouse is from center or edges of the arena\n\n## Pair features\nFor pair of mice we simply try to create features that would capture the relationship between the mice:\n- Self-mouse features similarly to the described above\n- Cross-mice pair-wise distance between sensor's keypoints\n- Relative speed between mice\n- Relative acceleration and jerk\n- Relative orientation\n- Allogroom specific features\n- Pair metadata feat, e.g., `is_same_sex`, `is_same_color`, etc..\n\n## Temporal features\nAlthough part of our feature engineering pipeline was already capturing some features at different windows, we found really beneficial to simply \"borrow\" the features from neighbour frames as well. We did it in two different ways: \"simple copy\" and \"statistics over window\".\n\n### Simple copy\nThe first thing we tried was to simply use a sliding window to copy features from frames around `frame_i`. For example, we applied `window=4` and `stride=5` meaning that for `frame_i` we would concat the features from frames `[frame_i-20, frame_i-15, frame_i-10, frame_i-5, frame_i, frame_i+5, frame_i+10, frame_i+15, frame_i+20]`.\n\nOf course this approach lead to a sharp increase in the number of features but we found XGB really robust to high dimensionality. We denote the model trained with this \"simple copy\" approach as \"Matheus XGB\" in the picture with scores.\n\n### Statistics over window\nThe second option for temporal features we tried was to compute statistics over each window instead of copying the frame features as is. We used `median`, `min`, `max` and `std` as aggregators for each window. This model used `window=2` and `stride=15` and is denoted as \"Edu XGB\".\n\n\n# NN Pipeline\nSimilarly to our XGB pipeline, we standardized the sensor keypoints by:\n- `lateral_left` -> `hip_left`\n- `lateral_right` -> `hip_right`\n- `head` -> `nose`\n- Remove `headpiece` and `spine`\n\nFor each 24 frame, we calculate the following features and provide it as main input to the model:\n- Body part availability mean, max\n- X axis mean, std\n- Y axis mean, std\n- Interpolate missing x, y but keep availability (not missing) info\n- Min distance to arena edge as 7th feature\n\nAfter the initial feature extraction backbone, we late inject extra features into the model:\n- Mouse to mouse body parts distance\n- Speed itself and speed and heading vector similarities\n- Behaviors labelled array\n\nRegarding the our architecutre, we opted for the following design choices:\n- Convolutions of different dilations with residual connections after `GroupNorms`\n- `InstanceNorm3d` on the inputs\n- `ELU` activation\n- Large avg and max pool windows are fed in the last layer\n- For some part of layers, take `softmax` over all mice pairs including self\n\nFor our training, the main considerations are the following:\n- Data is represented as tensor of shape `(batch_size, features, n_mice, n_mice, seq_len)`\n- `BCEWithLogitsLoss` is used as loos function on masked target (no backprop on NaN targets)\n- Loss is weighted by `sqrt(binary_label_mean)` to balance rare classes\n- Gated max classification\n - We allow the model to pick between its classification or max classification. Great for the cases where mice are mixed.\n- We start our training on all labs together, then finetuned for each lab\n- As postprocessing, we multiply some rare/harder action's logits by 1.5 to improve F1 before applying `argmax`\n\n# Ensemble\nFor our final solution we simply selected, for each lab, the model with greatest CV (see picture for reference).\nThe only exceptions are for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we found beneficial to perform major voting across all 3 models.\n\n\n", "votes": 26 } ], "comments": [], "raw_markdown_by_id": { "3377697": "First of all thanks the host and Kaggle for hosting this competition! Also, a huge shoutout to @tomirol and @aerdem4 for the partnership and effort working in this challenge!\n\n# TLDR\nOur solution is based on both NN and XGB trained per lab with different setups and parameters. For most labs we simply pick the model with highest CV for that lab, except for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we perform majority voting frame-wise.\n\nOur models' per-lab CV scores are the following:\n\n\n# XGB Pipeline\nOur code was originally forked from the great [public notebook](https://www.kaggle.com/code/ravaghi/social-action-recognition-in-mice-xgboost) from @ravaghi, which was already a pretty strong baseline.\nBelow we list the main points of our solution and the differences from the original notebook:\n- We normalize the FPS of all videos to 30fps and use the mode to get the label in the window\n- We standardize all sensors to only use `nose`, `ear_left`, `ear_right`, `body_center` and `tail_base`.\n - For labs where `body_center` is not available we can derive it from `hip_left` + `hip_right` or `lateral_left` + `lateral_right`. As a last resort (only 2 labs) we interpolate it from `tail_base` + `ears_l+r`.\n - For labs where `nose` is not available we simply replace it by `head`\n- We train separated models from single mouse and pair setups\n- We train 3 groups of 4 models (4Fold CV groupped per video and stratified by action label) for each `lab_id` and `action`, i.e., binary classifier per action\n - We change the model initialization and KFold seed for each model group and average the probs in the end to get the prediction for the action\n - We compute the individual `action` threshold by optimizing the F1 score on that lab's OOF predictions\n - We combine the individual action predictions into multi-class by first computing the residual between prediction and its threshold, i.e., `residual_action_i = pred_action_i - threshold_action_i`, then taking the `argmax` across all `residual_action_i`. Finally, for the highest residual we simply check if `residual_action_i > 0` to choose between `action_i` and no action.\n - Since we train binary models for each action we simply ignore any frame where the label of the action in question is NaN\n- For \"CautiousGiraffe\", \"DeliriousFly\", \"GroovyShrew\" and \"TranquilPanther\" we preprocess the keypoints by temporally interpolating (`group[\"x\"] = group[\"x\"].interpolate(method=\"linear\", limit_direction=\"both\")`) across the frames in order the reduce the large amount of NaN in the input.\n - This preproc improved significantly the labs aforementioned but not for others, which we suspect could be due to a large amount of id-switch in the tracking system (it makes the interpolation results terrible)\n - When applying interpolation we also add binary features for each keypoint flagging if the value is interpolated or not\n\n## Single mouse features\nFor the single mouse models we performed feature engieering focusing on the topics below:\n- Pair-wise distance between sensor's keypoints\n- Velocity for ear and tail\n- Elongation vs. compactness features\n- Body angle features\n- Body center displacement features\n- Body center acceleration and jerk features across different windows\n- Angular velocity and turn-rate\n- Grooming specific features\n- Mouse metadata, e.g., sex, color, has device, etc..\n- Arena boundary feats, e.g., how close mouse is from center or edges of the arena\n\n## Pair features\nFor pair of mice we simply try to create features that would capture the relationship between the mice:\n- Self-mouse features similarly to the described above\n- Cross-mice pair-wise distance between sensor's keypoints\n- Relative speed between mice\n- Relative acceleration and jerk\n- Relative orientation\n- Allogroom specific features\n- Pair metadata feat, e.g., `is_same_sex`, `is_same_color`, etc..\n\n## Temporal features\nAlthough part of our feature engineering pipeline was already capturing some features at different windows, we found really beneficial to simply \"borrow\" the features from neighbour frames as well. We did it in two different ways: \"simple copy\" and \"statistics over window\".\n\n### Simple copy\nThe first thing we tried was to simply use a sliding window to copy features from frames around `frame_i`. For example, we applied `window=4` and `stride=5` meaning that for `frame_i` we would concat the features from frames `[frame_i-20, frame_i-15, frame_i-10, frame_i-5, frame_i, frame_i+5, frame_i+10, frame_i+15, frame_i+20]`.\n\nOf course this approach lead to a sharp increase in the number of features but we found XGB really robust to high dimensionality. We denote the model trained with this \"simple copy\" approach as \"Matheus XGB\" in the picture with scores.\n\n### Statistics over window\nThe second option for temporal features we tried was to compute statistics over each window instead of copying the frame features as is. We used `median`, `min`, `max` and `std` as aggregators for each window. This model used `window=2` and `stride=15` and is denoted as \"Edu XGB\".\n\n\n# NN Pipeline\nSimilarly to our XGB pipeline, we standardized the sensor keypoints by:\n- `lateral_left` -> `hip_left`\n- `lateral_right` -> `hip_right`\n- `head` -> `nose`\n- Remove `headpiece` and `spine`\n\nFor each 24 frame, we calculate the following features and provide it as main input to the model:\n- Body part availability mean, max\n- X axis mean, std\n- Y axis mean, std\n- Interpolate missing x, y but keep availability (not missing) info\n- Min distance to arena edge as 7th feature\n\nAfter the initial feature extraction backbone, we late inject extra features into the model:\n- Mouse to mouse body parts distance\n- Speed itself and speed and heading vector similarities\n- Behaviors labelled array\n\nRegarding the our architecutre, we opted for the following design choices:\n- Convolutions of different dilations with residual connections after `GroupNorms`\n- `InstanceNorm3d` on the inputs\n- `ELU` activation\n- Large avg and max pool windows are fed in the last layer\n- For some part of layers, take `softmax` over all mice pairs including self\n\nFor our training, the main considerations are the following:\n- Data is represented as tensor of shape `(batch_size, features, n_mice, n_mice, seq_len)`\n- `BCEWithLogitsLoss` is used as loos function on masked target (no backprop on NaN targets)\n- Loss is weighted by `sqrt(binary_label_mean)` to balance rare classes\n- Gated max classification\n - We allow the model to pick between its classification or max classification. Great for the cases where mice are mixed.\n- We start our training on all labs together, then finetuned for each lab\n- As postprocessing, we multiply some rare/harder action's logits by 1.5 to improve F1 before applying `argmax`\n\n# Ensemble\nFor our final solution we simply selected, for each lab, the model with greatest CV (see picture for reference).\nThe only exceptions are for \"LyricalHare\", \"InvincibleJellyfish\" and \"ElegantMink\" where we found beneficial to perform major voting across all 3 models.\n\n\n" } }