{ "id": 663063, "title": " 10th Place Solution (ST-GCN + Transformer)", "url": "/competitions/MABe-mouse-behavior-detection/writeups/10th-place-solution-st-gcn-transformer", "author_name": "", "post_date": "2025-12-16T06:59:13.283Z", "votes": 27, "comment_count": 1, "views": 0, "content": "

10th Place Solution

\n

Overview

\n\n

Preprocessing

\n

FPS Normalization

\n\n

Coordinate Normalization (per lab)

\n\n

Keypoint Mapping

\n\n

Feature Engineering (24-dim per keypoint)

\n

From normalized (x, y) coordinates, compute 24 features:

\n
Per-keypoint features (8 dim):\n  - x, y         (2): normalized coordinates\n  - vx, vy       (2): velocity (Δt=2 frames)\n  - ax, ay       (2): acceleration (Δt=2 frames)\n\nInter-mouse features (16 dim, relative to other 3 mice):\n  - rel_x, rel_y (6): relative position to mouse i (i=1,2,3)\n  - rel_vx, rel_vy (6): relative velocity to mouse i\n  - dist         (3): Euclidean distance to mouse i\n  - approach_vel (3): approach velocity (d(dist)/dt)\n\nTotal: 8 + 16 = 24 features\n
\n

Cross Validation

\n\n

Model Architecture

\n

ST-GCN + Transformer

\n
Input:           [B, 4, K, 24, F]   # batch, mice, keypoints, features, frames\n    ↓\nST-GCN (×4) + Temporal Pooling:\n                 [B, 4, K, 128, F']  # spatial graph conv + temporal conv + pool (F'=F/p)\n    ↓\nKeypoint Attention Pooling:\n                 [B, 4, 128, F']    # learnable attention over keypoints\n    ↓\nPairwise Feature Extraction:\n  concat(agent, target, agent-target, target-agent) for 4×4 pairs\n                 [B, 16, 512, F']\n    ↓\nConcat Embeddings:\n  + Lab emb (16) + FPS emb (16) + Action emb (32)\n                 [B, 16, 576, F']\n    ↓\nFeature Compression:\n                 [B, 16, 192, F']\n    ↓\nTransformer with RoPE (per pair):\n                 [B, 16, F', 192]\n    ↓\nTemporal Upsample:\n                 [B, 16, F, 192]    # restore to original frame length\n    ↓\nCross-Pair Attention:\n  16 pairs attend to each other\n                 [B, 4, 4, F, 192]\n    ↓\nClassifier:      [B, 4, 4, 38, F]   # agent, target, classes, frames\n
\n

Note: Non-pairwise model (without Pairwise Feature Extraction / Cross-Pair Attention) also used in ensemble.

\n

Loss Function

\n

logits: [B, 4, 4, 38, F] → flatten → [N, 152] # N = B × 4 agents × F

\n

Triplet Mask (from behaviors_labeled):

\n\n

Loss = α × CE + (1-α) × MacroSoftF1

\n\n

Data Augmentation

\n

Training

\n\n

TTA

\n\n

Ensemble

\n\n

What Didn't Work

\n", "messages": [ { "id": "3377356", "postDate": "12/16/2025 06:57:16", "content": "

10th Place Solution

\n

Overview

\n\n

Preprocessing

\n

FPS Normalization

\n\n

Coordinate Normalization (per lab)

\n\n

Keypoint Mapping

\n\n

Feature Engineering (24-dim per keypoint)

\n

From normalized (x, y) coordinates, compute 24 features:

\n
Per-keypoint features (8 dim):\n  - x, y         (2): normalized coordinates\n  - vx, vy       (2): velocity (Δt=2 frames)\n  - ax, ay       (2): acceleration (Δt=2 frames)\n\nInter-mouse features (16 dim, relative to other 3 mice):\n  - rel_x, rel_y (6): relative position to mouse i (i=1,2,3)\n  - rel_vx, rel_vy (6): relative velocity to mouse i\n  - dist         (3): Euclidean distance to mouse i\n  - approach_vel (3): approach velocity (d(dist)/dt)\n\nTotal: 8 + 16 = 24 features\n
\n

Cross Validation

\n\n

Model Architecture

\n

ST-GCN + Transformer

\n
Input:           [B, 4, K, 24, F]   # batch, mice, keypoints, features, frames\n    ↓\nST-GCN (×4) + Temporal Pooling:\n                 [B, 4, K, 128, F']  # spatial graph conv + temporal conv + pool (F'=F/p)\n    ↓\nKeypoint Attention Pooling:\n                 [B, 4, 128, F']    # learnable attention over keypoints\n    ↓\nPairwise Feature Extraction:\n  concat(agent, target, agent-target, target-agent) for 4×4 pairs\n                 [B, 16, 512, F']\n    ↓\nConcat Embeddings:\n  + Lab emb (16) + FPS emb (16) + Action emb (32)\n                 [B, 16, 576, F']\n    ↓\nFeature Compression:\n                 [B, 16, 192, F']\n    ↓\nTransformer with RoPE (per pair):\n                 [B, 16, F', 192]\n    ↓\nTemporal Upsample:\n                 [B, 16, F, 192]    # restore to original frame length\n    ↓\nCross-Pair Attention:\n  16 pairs attend to each other\n                 [B, 4, 4, F, 192]\n    ↓\nClassifier:      [B, 4, 4, 38, F]   # agent, target, classes, frames\n
\n

Note: Non-pairwise model (without Pairwise Feature Extraction / Cross-Pair Attention) also used in ensemble.

\n

Loss Function

\n

logits: [B, 4, 4, 38, F] → flatten → [N, 152] # N = B × 4 agents × F

\n

Triplet Mask (from behaviors_labeled):

\n\n

Loss = α × CE + (1-α) × MacroSoftF1

\n\n

Data Augmentation

\n

Training

\n\n

TTA

\n\n

Ensemble

\n\n

What Didn't Work

\n", "rawMarkdown": "# 10th Place Solution\n\n## Overview\n\n- ST-GCN + Transformer\n - Model all 16 (agent, target) pairs jointly\n - Directly optimize competition metric (Macro Soft F1 Loss)\n - Ensemble models with different keypoints (4-7) and sequence lengths\n\n## Preprocessing\n\n### FPS Normalization\n- Resample all videos to **30 FPS** (linear interpolation)\n- Original FPS passed to model as additional feature\n\n### Coordinate Normalization (per lab)\n- Normalize (x, y) coordinates per lab: `x_norm = (x - mean) / std`\n\n### Keypoint Mapping\n- Labs have different keypoint names → map to unified names\n - head → nose\n - spine_1 → neck\n - hip_left/right → lateral_left/right\n\n### Feature Engineering (24-dim per keypoint)\n\nFrom normalized (x, y) coordinates, compute 24 features:\n\n```\nPer-keypoint features (8 dim):\n - x, y (2): normalized coordinates\n - vx, vy (2): velocity (Δt=2 frames)\n - ax, ay (2): acceleration (Δt=2 frames)\n\nInter-mouse features (16 dim, relative to other 3 mice):\n - rel_x, rel_y (6): relative position to mouse i (i=1,2,3)\n - rel_vx, rel_vy (6): relative velocity to mouse i\n - dist (3): Euclidean distance to mouse i\n - approach_vel (3): approach velocity (d(dist)/dt)\n\nTotal: 8 + 16 = 24 features\n```\n\n## Cross Validation\n\n- **4-fold StratifiedGroupKFold**\n - Group: video_id\n - Stratify: lab_id\n - CalMS21, CRIM13 labs always in train\n- **OOF evaluation**: Train on 3 folds, validate on 1 fold\n- **Final submission**: Train on all data (no validation split)\n\n## Model Architecture\n\n### ST-GCN + Transformer\n\n```\nInput: [B, 4, K, 24, F] # batch, mice, keypoints, features, frames\n ↓\nST-GCN (×4) + Temporal Pooling:\n [B, 4, K, 128, F'] # spatial graph conv + temporal conv + pool (F'=F/p)\n ↓\nKeypoint Attention Pooling:\n [B, 4, 128, F'] # learnable attention over keypoints\n ↓\nPairwise Feature Extraction:\n concat(agent, target, agent-target, target-agent) for 4×4 pairs\n [B, 16, 512, F']\n ↓\nConcat Embeddings:\n + Lab emb (16) + FPS emb (16) + Action emb (32)\n [B, 16, 576, F']\n ↓\nFeature Compression:\n [B, 16, 192, F']\n ↓\nTransformer with RoPE (per pair):\n [B, 16, F', 192]\n ↓\nTemporal Upsample:\n [B, 16, F, 192] # restore to original frame length\n ↓\nCross-Pair Attention:\n 16 pairs attend to each other\n [B, 4, 4, F, 192]\n ↓\nClassifier: [B, 4, 4, 38, F] # agent, target, classes, frames\n```\n\nNote: Non-pairwise model (without Pairwise Feature Extraction / Cross-Pair Attention) also used in ensemble.\n\n## Loss Function\n\nlogits: [B, 4, 4, 38, F] → flatten → [N, 152] # N = B × 4 agents × F\n\nTriplet Mask (from behaviors_labeled):\n - Set logits[invalid] = -inf before softmax\n - Only (agent, target, action) in behaviors_labeled are valid\n\nLoss = α × CE + (1-α) × MacroSoftF1\n\n - CE: Cross-Entropy over all (target × action: 152) per agent\n - MacroSoftF1: Differentiable approximation of competition metric (soft TP/FP/FN, excludes background)\n - Schedule: α transitions 0.2 → 0 over 32 epochs (pure F1 at end)\n\n## Data Augmentation\n\n**Training**\n- Affine transform: rotation, scale, translation, shear, horizontal/vertical flip\n- Mouse shuffle: permute mouse IDs\n- CutMix: mix temporal segments (same lab & num_mice)\n- Keypoint dropout\n\n**TTA**\n- Horizontal flip averaging\n\n## Ensemble\n\n- Different keypoints and sequence lengths (160-224)\n - 4kp: ear×2, nose, tail_base\n - 5kp: + neck\n - 7kp: + lateral×2\n- Pairwise model (LB: ~0.525, CV: ~0.510) + non-pairwise models (LB: ~0.500, CV: ~0.500)\n- Probability averaging across models\n\n## What Didn't Work\n\n- Pseudo-labeling on test data\n- Semi-supervised learning with MABe22 unlabeled data\n- Merging rare classes per lab in training (e.g., AdaptableSnail: avoid → escape)", "votes": null }, { "id": "3377546", "postDate": "12/16/2025 14:49:32", "content": "

Thanks for sharing. The use of graph to model bodyparts and the soft F1 loss are very innovative.

", "rawMarkdown": "Thanks for sharing. The use of graph to model bodyparts and the soft F1 loss are very innovative.", "votes": null } ], "comments": [ { "id": 3377546, "author_name": "xiaoz259", "author_url": "", "post_date": "12/16/2025 14:49:32", "content": "

Thanks for sharing. The use of graph to model bodyparts and the soft F1 loss are very innovative.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3377356": "# 10th Place Solution\n\n## Overview\n\n- ST-GCN + Transformer\n - Model all 16 (agent, target) pairs jointly\n - Directly optimize competition metric (Macro Soft F1 Loss)\n - Ensemble models with different keypoints (4-7) and sequence lengths\n\n## Preprocessing\n\n### FPS Normalization\n- Resample all videos to **30 FPS** (linear interpolation)\n- Original FPS passed to model as additional feature\n\n### Coordinate Normalization (per lab)\n- Normalize (x, y) coordinates per lab: `x_norm = (x - mean) / std`\n\n### Keypoint Mapping\n- Labs have different keypoint names → map to unified names\n - head → nose\n - spine_1 → neck\n - hip_left/right → lateral_left/right\n\n### Feature Engineering (24-dim per keypoint)\n\nFrom normalized (x, y) coordinates, compute 24 features:\n\n```\nPer-keypoint features (8 dim):\n - x, y (2): normalized coordinates\n - vx, vy (2): velocity (Δt=2 frames)\n - ax, ay (2): acceleration (Δt=2 frames)\n\nInter-mouse features (16 dim, relative to other 3 mice):\n - rel_x, rel_y (6): relative position to mouse i (i=1,2,3)\n - rel_vx, rel_vy (6): relative velocity to mouse i\n - dist (3): Euclidean distance to mouse i\n - approach_vel (3): approach velocity (d(dist)/dt)\n\nTotal: 8 + 16 = 24 features\n```\n\n## Cross Validation\n\n- **4-fold StratifiedGroupKFold**\n - Group: video_id\n - Stratify: lab_id\n - CalMS21, CRIM13 labs always in train\n- **OOF evaluation**: Train on 3 folds, validate on 1 fold\n- **Final submission**: Train on all data (no validation split)\n\n## Model Architecture\n\n### ST-GCN + Transformer\n\n```\nInput: [B, 4, K, 24, F] # batch, mice, keypoints, features, frames\n ↓\nST-GCN (×4) + Temporal Pooling:\n [B, 4, K, 128, F'] # spatial graph conv + temporal conv + pool (F'=F/p)\n ↓\nKeypoint Attention Pooling:\n [B, 4, 128, F'] # learnable attention over keypoints\n ↓\nPairwise Feature Extraction:\n concat(agent, target, agent-target, target-agent) for 4×4 pairs\n [B, 16, 512, F']\n ↓\nConcat Embeddings:\n + Lab emb (16) + FPS emb (16) + Action emb (32)\n [B, 16, 576, F']\n ↓\nFeature Compression:\n [B, 16, 192, F']\n ↓\nTransformer with RoPE (per pair):\n [B, 16, F', 192]\n ↓\nTemporal Upsample:\n [B, 16, F, 192] # restore to original frame length\n ↓\nCross-Pair Attention:\n 16 pairs attend to each other\n [B, 4, 4, F, 192]\n ↓\nClassifier: [B, 4, 4, 38, F] # agent, target, classes, frames\n```\n\nNote: Non-pairwise model (without Pairwise Feature Extraction / Cross-Pair Attention) also used in ensemble.\n\n## Loss Function\n\nlogits: [B, 4, 4, 38, F] → flatten → [N, 152] # N = B × 4 agents × F\n\nTriplet Mask (from behaviors_labeled):\n - Set logits[invalid] = -inf before softmax\n - Only (agent, target, action) in behaviors_labeled are valid\n\nLoss = α × CE + (1-α) × MacroSoftF1\n\n - CE: Cross-Entropy over all (target × action: 152) per agent\n - MacroSoftF1: Differentiable approximation of competition metric (soft TP/FP/FN, excludes background)\n - Schedule: α transitions 0.2 → 0 over 32 epochs (pure F1 at end)\n\n## Data Augmentation\n\n**Training**\n- Affine transform: rotation, scale, translation, shear, horizontal/vertical flip\n- Mouse shuffle: permute mouse IDs\n- CutMix: mix temporal segments (same lab & num_mice)\n- Keypoint dropout\n\n**TTA**\n- Horizontal flip averaging\n\n## Ensemble\n\n- Different keypoints and sequence lengths (160-224)\n - 4kp: ear×2, nose, tail_base\n - 5kp: + neck\n - 7kp: + lateral×2\n- Pairwise model (LB: ~0.525, CV: ~0.510) + non-pairwise models (LB: ~0.500, CV: ~0.500)\n- Probability averaging across models\n\n## What Didn't Work\n\n- Pseudo-labeling on test data\n- Semi-supervised learning with MABe22 unlabeled data\n- Merging rare classes per lab in training (e.g., AdaptableSnail: avoid → escape)", "3377546": "Thanks for sharing. The use of graph to model bodyparts and the soft F1 loss are very innovative." }, "source": "meta" }