Hi, I'm new to ML and the competition itself, and I saw that there are more and more people trying to combine prediction on the top-voted models publicly posted, so I try it too.
\nAs I was testing, model 1 (with pred1) had 0.82 LB, model 2 (pred2) had 0.82 LB, and the prediction made by pred10.5+pred20.5 can achieve 0.84 LB. I believe this was caused via bias among models, I understand that there are many more tests needed to prove it. But can someone help me explain this? Am I wrong, or is this a real way to improve real scenario model prediction?
", "messages": [ { "id": "3255101", "postDate": "07/28/2025 00:50:06", "content": "Hi, I'm new to ML and the competition itself, and I saw that there are more and more people trying to combine prediction on the top-voted models publicly posted, so I try it too.
\nAs I was testing, model 1 (with pred1) had 0.82 LB, model 2 (pred2) had 0.82 LB, and the prediction made by pred10.5+pred20.5 can achieve 0.84 LB. I believe this was caused via bias among models, I understand that there are many more tests needed to prove it. But can someone help me explain this? Am I wrong, or is this a real way to improve real scenario model prediction?
", "rawMarkdown": "Hi, I'm new to ML and the competition itself, and I saw that there are more and more people trying to combine prediction on the top-voted models publicly posted, so I try it too.\n \nAs I was testing, model 1 (with pred1) had 0.82 LB, model 2 (pred2) had 0.82 LB, and the prediction made by pred1*0.5+pred2*0.5 can achieve 0.84 LB. I believe this was caused via bias among models, I understand that there are many more tests needed to prove it. But can someone help me explain this? Am I wrong, or is this a real way to improve real scenario model prediction?", "votes": null }, { "id": "3255248", "postDate": "07/28/2025 09:36:22", "content": "Not just about high bias (underfitting), but also high variance (overfitting). Combining model helps reduce both bias and variance. Ensemble models usually leverage the strengths of individual models and tend to perform better overall.
", "rawMarkdown": "Not just about high bias (underfitting), but also high variance (overfitting). Combining model helps reduce both bias and variance. Ensemble models usually leverage the strengths of individual models and tend to perform better overall.", "votes": null }, { "id": "3255311", "postDate": "07/28/2025 12:27:52", "content": "This page summarize well the benefits of ensemble: https://machinelearningmastery.com/why-use-ensemble-learning/
\nFor mathematical reasoning behind ensemble, this paper explained well: https://dl.acm.org/doi/pdf/10.5555/1046920.1194899
\nBased on the above paper, error can be decomposed into:
\n$$
\n\\mathbb{E}[(y-\\hat{f}(x))^2] = \\text{Bias}^2 \\ (\\text{underfitting}) + \\text{Variance} \\ (\\text{overfitting})
\n$$
When we do ensemble, the error becomes:
\n$$
\nE\\left[(\\bar{f} - t)^2\\right] = \\overline{\\text{bias}}^2 + \\frac{1}{M} \\, \\overline{\\text{var}} + \\left(1 - \\frac{1}{M} \\right) \\overline{\\text{covar}}
\n$$
where:
\nBased on the ensemble error, I think that ensembling improve variance more often than bias. Given the best case is that our ensemble models are not correlated, \\(\\overline{\\text{covar}}\\) ~ 0, the ensemble methods can reduce the variance linearly with the number of models. To make these ensemble models not correlated or lowly correlated, we can train with many model architectures or train on different subsets of data. However, in reality these models are not fully uncorrelated so in case we ensemble too many models \\(\\overline{\\text{covar}}\\) will contribute to increasing error.
", "rawMarkdown": "This page summarize well the benefits of ensemble: https://machinelearningmastery.com/why-use-ensemble-learning/\n\nFor mathematical reasoning behind ensemble, this paper explained well: https://dl.acm.org/doi/pdf/10.5555/1046920.1194899\n\nBased on the above paper, error can be decomposed into:\n\n$$\n\\mathbb{E}[(y-\\hat{f}(x))^2] = \\text{Bias}^2 \\ (\\text{underfitting}) + \\text{Variance} \\ (\\text{overfitting})\n$$\n\nWhen we do ensemble, the error becomes:\n\n$$\nE\\left[(\\bar{f} - t)^2\\right] = \\overline{\\text{bias}}^2 + \\frac{1}{M} \\, \\overline{\\text{var}} + \\left(1 - \\frac{1}{M} \\right) \\overline{\\text{covar}}\n$$\n\nwhere:\n* \\\\(\\overline{\\text{bias}}\\\\) is the averaged bias of the ensemble members\n* \\\\(\\overline{\\text{var}}\\\\) is the averaged variance of the ensemble members\n* \\\\(\\overline{\\text{covar}}\\\\) is the averaged covariance of the ensemble members\n\nBased on the ensemble error, I think that ensembling improve variance more often than bias. Given the best case is that our ensemble models are not correlated, \\\\(\\overline{\\text{covar}}\\\\) ~ 0, the ensemble methods can reduce the variance linearly with the number of models. To make these ensemble models not correlated or lowly correlated, we can train with many model architectures or train on different subsets of data. However, in reality these models are not fully uncorrelated so in case we ensemble too many models \\\\(\\overline{\\text{covar}}\\\\) will contribute to increasing error.", "votes": null }, { "id": "3255342", "postDate": "07/28/2025 13:25:39", "content": "thank you Hòa, I was spending my whole afternoon finding related papers, this is the best answer I got
", "rawMarkdown": "thank you Hòa, I was spending my whole afternoon finding related papers, this is the best answer I got", "votes": null }, { "id": "3263348", "postDate": "08/05/2025 04:54:16", "content": "I tried to train a per class weights on the two models for ensemble. However, I found that model2 seems over fitting. My per class weights results ranges from 0.82-0.83
", "rawMarkdown": "I tried to train a per class weights on the two models for ensemble. However, I found that model2 seems over fitting. My per class weights results ranges from 0.82-0.83", "votes": null } ], "comments": [ { "id": 3255248, "author_name": "kerta27", "author_url": "", "post_date": "07/28/2025 09:36:22", "content": "Not just about high bias (underfitting), but also high variance (overfitting). Combining model helps reduce both bias and variance. Ensemble models usually leverage the strengths of individual models and tend to perform better overall.
", "votes": null, "replies": [] }, { "id": 3255311, "author_name": "nguyenhoa", "author_url": "", "post_date": "07/28/2025 12:27:52", "content": "This page summarize well the benefits of ensemble: https://machinelearningmastery.com/why-use-ensemble-learning/
\nFor mathematical reasoning behind ensemble, this paper explained well: https://dl.acm.org/doi/pdf/10.5555/1046920.1194899
\nBased on the above paper, error can be decomposed into:
\n$$
\n\\mathbb{E}[(y-\\hat{f}(x))^2] = \\text{Bias}^2 \\ (\\text{underfitting}) + \\text{Variance} \\ (\\text{overfitting})
\n$$
When we do ensemble, the error becomes:
\n$$
\nE\\left[(\\bar{f} - t)^2\\right] = \\overline{\\text{bias}}^2 + \\frac{1}{M} \\, \\overline{\\text{var}} + \\left(1 - \\frac{1}{M} \\right) \\overline{\\text{covar}}
\n$$
where:
\nBased on the ensemble error, I think that ensembling improve variance more often than bias. Given the best case is that our ensemble models are not correlated, \\(\\overline{\\text{covar}}\\) ~ 0, the ensemble methods can reduce the variance linearly with the number of models. To make these ensemble models not correlated or lowly correlated, we can train with many model architectures or train on different subsets of data. However, in reality these models are not fully uncorrelated so in case we ensemble too many models \\(\\overline{\\text{covar}}\\) will contribute to increasing error.
", "votes": null, "replies": [ { "id": 3255342, "author_name": "hahuyy", "author_url": "", "post_date": "07/28/2025 13:25:39", "content": "thank you Hòa, I was spending my whole afternoon finding related papers, this is the best answer I got
", "votes": null, "replies": [] } ] }, { "id": 3263348, "author_name": "yuemingl", "author_url": "", "post_date": "08/05/2025 04:54:16", "content": "I tried to train a per class weights on the two models for ensemble. However, I found that model2 seems over fitting. My per class weights results ranges from 0.82-0.83
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3255101": "Hi, I'm new to ML and the competition itself, and I saw that there are more and more people trying to combine prediction on the top-voted models publicly posted, so I try it too.\n \nAs I was testing, model 1 (with pred1) had 0.82 LB, model 2 (pred2) had 0.82 LB, and the prediction made by pred1*0.5+pred2*0.5 can achieve 0.84 LB. I believe this was caused via bias among models, I understand that there are many more tests needed to prove it. But can someone help me explain this? Am I wrong, or is this a real way to improve real scenario model prediction?", "3255248": "Not just about high bias (underfitting), but also high variance (overfitting). Combining model helps reduce both bias and variance. Ensemble models usually leverage the strengths of individual models and tend to perform better overall.", "3255311": "This page summarize well the benefits of ensemble: https://machinelearningmastery.com/why-use-ensemble-learning/\n\nFor mathematical reasoning behind ensemble, this paper explained well: https://dl.acm.org/doi/pdf/10.5555/1046920.1194899\n\nBased on the above paper, error can be decomposed into:\n\n$$\n\\mathbb{E}[(y-\\hat{f}(x))^2] = \\text{Bias}^2 \\ (\\text{underfitting}) + \\text{Variance} \\ (\\text{overfitting})\n$$\n\nWhen we do ensemble, the error becomes:\n\n$$\nE\\left[(\\bar{f} - t)^2\\right] = \\overline{\\text{bias}}^2 + \\frac{1}{M} \\, \\overline{\\text{var}} + \\left(1 - \\frac{1}{M} \\right) \\overline{\\text{covar}}\n$$\n\nwhere:\n* \\\\(\\overline{\\text{bias}}\\\\) is the averaged bias of the ensemble members\n* \\\\(\\overline{\\text{var}}\\\\) is the averaged variance of the ensemble members\n* \\\\(\\overline{\\text{covar}}\\\\) is the averaged covariance of the ensemble members\n\nBased on the ensemble error, I think that ensembling improve variance more often than bias. Given the best case is that our ensemble models are not correlated, \\\\(\\overline{\\text{covar}}\\\\) ~ 0, the ensemble methods can reduce the variance linearly with the number of models. To make these ensemble models not correlated or lowly correlated, we can train with many model architectures or train on different subsets of data. However, in reality these models are not fully uncorrelated so in case we ensemble too many models \\\\(\\overline{\\text{covar}}\\\\) will contribute to increasing error.", "3255342": "thank you Hòa, I was spending my whole afternoon finding related papers, this is the best answer I got", "3263348": "I tried to train a per class weights on the two models for ensemble. However, I found that model2 seems over fitting. My per class weights results ranges from 0.82-0.83" }, "source": "meta" }