I've been exploring the potential of ensemble methods for this competition. I trained both TabM and LightGBM models, each achieving a cross-validation score of 0.77 and a public leaderboard score of 0.77 individually. However, when I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability, the leaderboard score drops significantly to 0.69.
\nHas anyone else experienced similar issues with weighted or simple averaging ensembles involving different model architectures?
", "messages": [ { "id": 3236268, "postDate": "2025-06-30T07:36:12.313Z", "content": "I've been exploring the potential of ensemble methods for this competition. I trained both TabM and LightGBM models, each achieving a cross-validation score of 0.77 and a public leaderboard score of 0.77 individually. However, when I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability, the leaderboard score drops significantly to 0.69.
\nHas anyone else experienced similar issues with weighted or simple averaging ensembles involving different model architectures?
", "rawMarkdown": "I've been exploring the potential of ensemble methods for this competition. I trained both TabM and LightGBM models, each achieving a cross-validation score of 0.77 and a public leaderboard score of 0.77 individually. However, when I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability, the leaderboard score drops significantly to 0.69.\n\nHas anyone else experienced similar issues with weighted or simple averaging ensembles involving different model architectures?", "votes": 8 }, { "id": 3236284, "postDate": "2025-06-30T08:08:13.030Z", "content": "Qwen told me several possible reasons, and I feel like calibration is needed. Looking forward to your progress！
\n\"It is recommended to calibrate the output probabilities of each model, for example using Platt Scaling or Isotonic Regression. You can use sklearn.calibration.CalibratedClassifierCV to calibrate each model individually before ensembling them.\"
Will have a look into that. Thank you :)
", "rawMarkdown": "Will have a look into that. Thank you :)", "votes": 1 } ] }, { "id": 3236482, "postDate": "2025-06-30T10:33:29.490Z", "content": "What kind of result do you get when you do the blending locally with using their OOF predictions?
", "rawMarkdown": "What kind of result do you get when you do the blending locally with using their OOF predictions?", "votes": 4 }, { "id": 3245260, "postDate": "2025-07-09T06:12:39.333Z", "content": "I also encountered the same problem. Two models with LB of 0.79 were integrated, and the highest average probability was chosen. In the end, LB was 0.58。
", "rawMarkdown": "I also encountered the same problem. Two models with LB of 0.79 were integrated, and the highest average probability was chosen. In the end, LB was 0.58。", "votes": 1 }, { "id": 3239928, "postDate": "2025-07-03T09:38:34Z", "content": "\n\nwhen I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability
\n
Their prediction distribution could be different. Try voting on class predictions.
", "rawMarkdown": "> when I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability\n\nTheir prediction distribution could be different. Try voting on class predictions.", "votes": 1, "replies": [ { "id": 3240054, "postDate": "2025-07-03T12:13:48.023Z", "content": "I tried ranking to counter this, but the score was equally bad.
", "rawMarkdown": "I tried ranking to counter this, but the score was equally bad.", "votes": 1, "replies": [ { "id": 3240059, "postDate": "2025-07-03T12:27:45.253Z", "content": "I don't think ranking would work here since each prediction vector is ranked independently from each other. For a given sample, p(class i) > p(class j) no longer guarantees rank(class i) > rank(class j), so the alignment breaks. I would save soft predictions of each class and visualize them on top of each other for those models. If they are different from each other, try voting on class predictions.
", "rawMarkdown": "I don't think ranking would work here since each prediction vector is ranked independently from each other. For a given sample, p(class i) > p(class j) no longer guarantees rank(class i) > rank(class j), so the alignment breaks. I would save soft predictions of each class and visualize them on top of each other for those models. If they are different from each other, try voting on class predictions.", "votes": 2 } ] } ] }, { "id": 3236321, "postDate": "2025-06-30T08:50:31.803Z", "content": "Maybe it's worth to check the order of labels in both models.
", "rawMarkdown": "Maybe it's worth to check the order of labels in both models.", "votes": 2, "replies": [ { "id": 3236329, "postDate": "2025-06-30T08:57:25.013Z", "content": "I checked, one common Gesture mapper. And tried ranking them instead (to eliminate probability distributions) which gave the same results.
", "rawMarkdown": "I checked, one common Gesture mapper. And tried ranking them instead (to eliminate probability distributions) which gave the same results. ", "votes": 2, "replies": [ { "id": 3239496, "postDate": "2025-07-02T21:29:17.413Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3248068, "postDate": "2025-07-14T01:48:37.023Z", "content": "so far we havent done any elaborate ensembles just maybe tensorflow and a pytorch model with simple means and logits -> softmax for torch to get same probs scale .. Just I am bit confused in the comp as seems IMU is so dominating the LB scores how do we really trust one over the other .
", "rawMarkdown": "so far we havent done any elaborate ensembles just maybe tensorflow and a pytorch model with simple means and logits -> softmax for torch to get same probs scale .. Just I am bit confused in the comp as seems IMU is so dominating the LB scores how do we really trust one over the other . " } ], "comments": [ { "id": 3236284, "author_name": "Less", "author_url": "", "post_date": "2025-06-30T08:08:13.030000", "content": "Qwen told me several possible reasons, and I feel like calibration is needed. Looking forward to your progress！
\n\"It is recommended to calibrate the output probabilities of each model, for example using Platt Scaling or Isotonic Regression. You can use sklearn.calibration.CalibratedClassifierCV to calibrate each model individually before ensembling them.\"
Will have a look into that. Thank you :)
", "votes": 1, "replies": [] } ] }, { "id": 3236482, "author_name": "Anil Ozturk", "author_url": "", "post_date": "2025-06-30T10:33:29.490000", "content": "What kind of result do you get when you do the blending locally with using their OOF predictions?
", "votes": 4, "replies": [] }, { "id": 3245260, "author_name": "Less", "author_url": "", "post_date": "2025-07-09T06:12:39.333000", "content": "I also encountered the same problem. Two models with LB of 0.79 were integrated, and the highest average probability was chosen. In the end, LB was 0.58。
", "votes": 1, "replies": [] }, { "id": 3239928, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2025-07-03T09:38:34", "content": "\n\nwhen I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability
\n
Their prediction distribution could be different. Try voting on class predictions.
", "votes": 1, "replies": [ { "id": 3240054, "author_name": "stefanoclss", "author_url": "", "post_date": "2025-07-03T12:13:48.023000", "content": "I tried ranking to counter this, but the score was equally bad.
", "votes": 1, "replies": [ { "id": 3240059, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2025-07-03T12:27:45.253000", "content": "I don't think ranking would work here since each prediction vector is ranked independently from each other. For a given sample, p(class i) > p(class j) no longer guarantees rank(class i) > rank(class j), so the alignment breaks. I would save soft predictions of each class and visualize them on top of each other for those models. If they are different from each other, try voting on class predictions.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3236321, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-06-30T08:50:31.803000", "content": "Maybe it's worth to check the order of labels in both models.
", "votes": 2, "replies": [ { "id": 3236329, "author_name": "stefanoclss", "author_url": "", "post_date": "2025-06-30T08:57:25.013000", "content": "I checked, one common Gesture mapper. And tried ranking them instead (to eliminate probability distributions) which gave the same results.
", "votes": 2, "replies": [ { "id": 3239496, "author_name": "", "author_url": "", "post_date": "2025-07-02T21:29:17.413000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3248068, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2025-07-14T01:48:37.023000", "content": "so far we havent done any elaborate ensembles just maybe tensorflow and a pytorch model with simple means and logits -> softmax for torch to get same probs scale .. Just I am bit confused in the comp as seems IMU is so dominating the LB scores how do we really trust one over the other .
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3236268": "I've been exploring the potential of ensemble methods for this competition. I trained both TabM and LightGBM models, each achieving a cross-validation score of 0.77 and a public leaderboard score of 0.77 individually. However, when I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability, the leaderboard score drops significantly to 0.69.\n\nHas anyone else experienced similar issues with weighted or simple averaging ensembles involving different model architectures?", "3236284": "Qwen told me several possible reasons, and I feel like calibration is needed. Looking forward to your progress！\n\"It is recommended to calibrate the output probabilities of each model, for example using Platt Scaling or Isotonic Regression. You can use sklearn.calibration.CalibratedClassifierCV to calibrate each model individually before ensembling them.\"", "3236482": "What kind of result do you get when you do the blending locally with using their OOF predictions?", "3245260": "I also encountered the same problem. Two models with LB of 0.79 were integrated, and the highest average probability was chosen. In the end, LB was 0.58。", "3239928": "> when I ensemble them by simply averaging their predictions and selecting the gesture with the highest probability\n\nTheir prediction distribution could be different. Try voting on class predictions.", "3236321": "Maybe it's worth to check the order of labels in both models.", "3248068": "so far we havent done any elaborate ensembles just maybe tensorflow and a pytorch model with simple means and logits -> softmax for torch to get same probs scale .. Just I am bit confused in the comp as seems IMU is so dominating the LB scores how do we really trust one over the other . " } }