![\"Ensembling](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F172360%2Fffc9c533191bcf9cf641544fc24d8e6c%2Fensembling_plot.png?generation=1765899796201861&alt=media\")
Huge thanks to the organisers for this interesting and insightful competition — I really enjoyed it!
\nIt was also nice to see that good results were possible with modest compute.
\nPer-frame classification via a weighted ensemble of GBDT models (XGBoost / LightGBM / CatBoost) and TCN models. Both trained on windowed temporal features.
\nRich feature engineering + augmentations (TCN only) + greedy hill-climbing ensembling.
\nSource code: https://github.com/PavelSavchenkov/kaggle-gold-9th-place-mabe
\nInitial plan was to implement this in order:
\nMy assumption was that given enough quality data each successive model would be better than previous. In practice, GBDT-only and TCN-only pipelines performed similarly and the biggest gain came from ensembling.
\nI didn’t manage to beat my TCN baseline with Transformer (and didn’t end up ensembling Transformer models - now I think it might have helped).
\nI modeled the task as per-frame classification using features from a fixed window centered at the target frame.
\nFor each approach, the training loop was roughly:
\nIt was slightly tricky because:
\nI assigned folds for each video, optimizing for evenly spread positive segment duration across folds for each (action, lab).
\nThe first goal was to build XGBoost baseline beating public notebook scores.
\nA central idea in feature engineering was normalizing coordinates into the agent’s body frame:
\nFeature families (see file with all GBDT configs here):
\nAt the beginning I repeated this a few times: trained a few models, inspected feature importances, adjusted feature set, go back to training.
\nThere was a separate model per each action. Data was downsampled to ~1M frames per training run. Also adjusted weights of specific labs (at most one at a time) and weights of all positive samples.
\nFirst non-zero XGBoost submission: 0.474 Public LB (10th at the time, wow!).
\nThen I've trained LightGBM and CatBoost models in a similar fashion. Best \"globally best per (action, lab)\" GBDT: 0.507 Public LB. Ensembling all GBDTs gave only a modest bump: up to 0.511 Public LB.
\nUnexpectedly, horizontal flip didn't help the score. At this point, only a few weeks were left until the end of the comp, so I switched to TCN.
\nIn my solution it was a few stacked 1D Conv blocks with increasing dilation (e.g. 4 temporal blocks with dilations [1, 2, 4, 8]). One sample is a [T, F] tensor there T is time steps from a fixed window and F is a number of features. One model predictions probability distribution for all actions and all labs. Implementation.
\nFeature set was similar in nature to GBDT, but taking into account that now we explicitly have landmarks from each of T time steps around target frame. I used windows of length 2s-4s and T around 30-45. Every accessible categorical info was mapped into a learnable embedding.
\nCompared to GBDT, it was more convenient to manage feature processing - it could be done entirely on GPU, so data loader basically had to supply only a few integers per sample (video id, agent, target, frame). Implementation.
\nI've trained dozens of models maximising total F1 on one fold, then selected best model for each action, trained them on 5 folds.
\nAugmentations were very important. Most of them are inspired by 1st place solution ASL Fingerspelling [1]
\nThe most impactful were (implementation):
\nDidn't help:
\nBest TCN-only (no ensembling): Public LB 0.509. Ensembling improved TCN-only Public LB to 0.521.
\nI used greedy hill climbing (inspired by this writeup [3] by Chris Deotte). Additional features include: different objectives (F1, nested F1, log-loss, etc), beam search, logit vs probability space. Implementation
\nThe best approach turned out to be just maximising nested F1 and allow as many models as possible. In the best solution I used up to 12 per (action, lab).
\nCombining GBDT and TCN models was the crucial part of the solution and boosted Public LB to 0.540.
\nSmoothing predictions with a small temporal kernel improved the score by ~0.001–0.003.
\nI implemented smoothing as \"strided inference\" (best: stride = 7), which also significantly sped up inference:
base from {0, 1, ..., stride-1}.base, base + stride, base + 2·stride, ...base = (base + 1) % stride.Another thing that boosted score by ~0.002–0.004 was paired action thresholds, designed to improve action selection when >1 actions are bigger than their thresholds:
\nDidn't help:
\n| Pipeline | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| xgboost, per-action | \n- | \n0.474 | \n0.458 | \n
| xgboost, per-action, per-lab | \n0.497 | \n0.501 | \n0.482 | \n
| xgboost, lgbm, catboost, per-action, per-lab | \n0.506 | \n0.507 | \n0.486 | \n
| ensemble (xgboost, lgbm, catboost, up to 3 models) | \n0.524* | \n0.511 | \n0.492 | \n
| TCN, per-action, per-lab | \n0.518 | \n0.509 | \n0.478 | \n
| ensemble (TCN, up to 6 models) | \n0.540* | \n0.515 | \n0.489 | \n
| above, but replace with models on 100% of data | \n- | \n0.521 | \n0.490 | \n
| ensemble (GBDT, TCN, up to 6 models) | \n0.556* | \n0.540 | \n0.514 | \n
| ensemble (GBDT, TCN, up to 12 models) | \n0.558* | \n0.539 | \n0.515 | \n
| above + pair threshold tournament | \n0.560* | \n0.538 | \n0.516 | \n
* ensembling CV scores are highly inflated, because it is a direct optimisation objective of huge ensembling space
\nI checked ensembling parameters in OOF fashion:
\nThis was much more honest assessment. In particular, 6 model ensemble was best on Public LB, but the cross validation showed that more models = better score, which turned out to be true on private LB.
\n
I didn't use any unlabeled data, and also noticed issues with 25 fps Adaptable Snail data only few days till the end, so was not able to incorporate this knowledge meaningfully :(
\nHuge thanks to the organisers for this interesting and insightful competition — I really enjoyed it!
\nIt was also nice to see that good results were possible with modest compute.
\nPer-frame classification via a weighted ensemble of GBDT models (XGBoost / LightGBM / CatBoost) and TCN models. Both trained on windowed temporal features.
\nRich feature engineering + augmentations (TCN only) + greedy hill-climbing ensembling.
\nSource code: https://github.com/PavelSavchenkov/kaggle-gold-9th-place-mabe
\nInitial plan was to implement this in order:
\nMy assumption was that given enough quality data each successive model would be better than previous. In practice, GBDT-only and TCN-only pipelines performed similarly and the biggest gain came from ensembling.
\nI didn’t manage to beat my TCN baseline with Transformer (and didn’t end up ensembling Transformer models - now I think it might have helped).
\nI modeled the task as per-frame classification using features from a fixed window centered at the target frame.
\nFor each approach, the training loop was roughly:
\nIt was slightly tricky because:
\nI assigned folds for each video, optimizing for evenly spread positive segment duration across folds for each (action, lab).
\nThe first goal was to build XGBoost baseline beating public notebook scores.
\nA central idea in feature engineering was normalizing coordinates into the agent’s body frame:
\nFeature families (see file with all GBDT configs here):
\nAt the beginning I repeated this a few times: trained a few models, inspected feature importances, adjusted feature set, go back to training.
\nThere was a separate model per each action. Data was downsampled to ~1M frames per training run. Also adjusted weights of specific labs (at most one at a time) and weights of all positive samples.
\nFirst non-zero XGBoost submission: 0.474 Public LB (10th at the time, wow!).
\nThen I've trained LightGBM and CatBoost models in a similar fashion. Best \"globally best per (action, lab)\" GBDT: 0.507 Public LB. Ensembling all GBDTs gave only a modest bump: up to 0.511 Public LB.
\nUnexpectedly, horizontal flip didn't help the score. At this point, only a few weeks were left until the end of the comp, so I switched to TCN.
\nIn my solution it was a few stacked 1D Conv blocks with increasing dilation (e.g. 4 temporal blocks with dilations [1, 2, 4, 8]). One sample is a [T, F] tensor there T is time steps from a fixed window and F is a number of features. One model predictions probability distribution for all actions and all labs. Implementation.
\nFeature set was similar in nature to GBDT, but taking into account that now we explicitly have landmarks from each of T time steps around target frame. I used windows of length 2s-4s and T around 30-45. Every accessible categorical info was mapped into a learnable embedding.
\nCompared to GBDT, it was more convenient to manage feature processing - it could be done entirely on GPU, so data loader basically had to supply only a few integers per sample (video id, agent, target, frame). Implementation.
\nI've trained dozens of models maximising total F1 on one fold, then selected best model for each action, trained them on 5 folds.
\nAugmentations were very important. Most of them are inspired by 1st place solution ASL Fingerspelling [1]
\nThe most impactful were (implementation):
\nDidn't help:
\nBest TCN-only (no ensembling): Public LB 0.509. Ensembling improved TCN-only Public LB to 0.521.
\nI used greedy hill climbing (inspired by this writeup [3] by Chris Deotte). Additional features include: different objectives (F1, nested F1, log-loss, etc), beam search, logit vs probability space. Implementation
\nThe best approach turned out to be just maximising nested F1 and allow as many models as possible. In the best solution I used up to 12 per (action, lab).
\nCombining GBDT and TCN models was the crucial part of the solution and boosted Public LB to 0.540.
\nSmoothing predictions with a small temporal kernel improved the score by ~0.001–0.003.
\nI implemented smoothing as \"strided inference\" (best: stride = 7), which also significantly sped up inference:
base from {0, 1, ..., stride-1}.base, base + stride, base + 2·stride, ...base = (base + 1) % stride.Another thing that boosted score by ~0.002–0.004 was paired action thresholds, designed to improve action selection when >1 actions are bigger than their thresholds:
\nDidn't help:
\n| Pipeline | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| xgboost, per-action | \n- | \n0.474 | \n0.458 | \n
| xgboost, per-action, per-lab | \n0.497 | \n0.501 | \n0.482 | \n
| xgboost, lgbm, catboost, per-action, per-lab | \n0.506 | \n0.507 | \n0.486 | \n
| ensemble (xgboost, lgbm, catboost, up to 3 models) | \n0.524* | \n0.511 | \n0.492 | \n
| TCN, per-action, per-lab | \n0.518 | \n0.509 | \n0.478 | \n
| ensemble (TCN, up to 6 models) | \n0.540* | \n0.515 | \n0.489 | \n
| above, but replace with models on 100% of data | \n- | \n0.521 | \n0.490 | \n
| ensemble (GBDT, TCN, up to 6 models) | \n0.556* | \n0.540 | \n0.514 | \n
| ensemble (GBDT, TCN, up to 12 models) | \n0.558* | \n0.539 | \n0.515 | \n
| above + pair threshold tournament | \n0.560* | \n0.538 | \n0.516 | \n
* ensembling CV scores are highly inflated, because it is a direct optimisation objective of huge ensembling space
\nI checked ensembling parameters in OOF fashion:
\nThis was much more honest assessment. In particular, 6 model ensemble was best on Public LB, but the cross validation showed that more models = better score, which turned out to be true on private LB.
\n
I didn't use any unlabeled data, and also noticed issues with 25 fps Adaptable Snail data only few days till the end, so was not able to incorporate this knowledge meaningfully :(
\n