{
  "id": 586562,
  "title": "Avoid Blindly Ensembling and Merging Public kernels",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/586562",
  "author_name": "",
  "post_date": "2025-06-27T07:09:34.942819600Z",
  "votes": 26,
  "comment_count": 14,
  "views": 0,
  "content": "<p>I have noticed this in almost any Kaggle competition midway or even at the start someone submits a high scoring public kernel which is just simple ensemble of many good scoring public kernels employing techniques like simple averaging or weighted averaging . While the allure of a quick leaderboard jump is understandable, especially in a Human Activity Recognition (HAR) multi-class classification problem, blindly blending models from public kernels, particularly through simple averaging, can be a detrimental strategy in the long run.</p>\n<p><strong>Why Blind Blending (especially simple averaging) is Problematic for HAR (and generally):</strong></p>\n<p>1.<strong>Lack of model diversity and redundancy :</strong>Most of the public kernels are basically for of similar ideas with just some minor tweaks and augmentations because of this the model diversity can be low and the variance in the predictions of model can be low making the entire process of ensembling sub optimal</p>\n<ol>\n<li><p><strong>Overfitting to public leaderboard :</strong> Public kernels are designed or optimized keeping public leaderboard in the mind so if you blend them without understanding their underlying validation strategy (or lack of thereof) you risk creating an ensemble that is heavily overfitted to the public test set, leading to a significant drop in performance on the private leaderboard. This is specially true since different kernels might be using different validation schemes.</p></li>\n<li><p><strong>Ignoring Models weakness and strengths:</strong> Each model specializes in classifying certain activities or handling specific types of noise in the sensor data. Simple averaging treats all models equally, even if one performs poorly on a particular class or a specific segment of the data. For multi-class classification like HAR, some models might have a better recall for rare activities, while others might be more precise for common ones.</p></li>\n</ol>\n<p>4.<strong>No interpretability and learning :</strong> Blind blending of models provides no insight as to which model is better or which contributes most effectively . This hinders your understanding of the problem and prevents you from developing more robust and insightful solutions in the future.</p>\n<ol>\n<li><strong>Data Leakage and Inconsistent Preprocessing :</strong> Public kernels might use different preprocessing steps and folds simply using them without understanding and blending can lead to data leakage and development of overly optimistic models</li>\n</ol>\n<p><strong>Better ways to do Ensembling</strong></p>\n<p><strong>1. Weighted Averaging:</strong></p>\n<p><strong>Concept</strong>: Instead of equal weights assigned different weights to each model based on its performance on your local cross validation set.</p>\n<p><strong>How to do it:</strong> Train models individually and evaluate them on cross validation set and based on its performance , You can even optimize these weights using a simple optimization algorithm (e.g., grid search or Nelder-Mead) on your validation predictions.</p>\n<p><strong>2. Stacking:</strong></p>\n<p><strong>Concept :</strong> One of the most powerful concepts in ensembling where output of your base models become input to the meta model . The meta-model then learns how to optimally combine these predictions.</p>\n<p><strong>How to do it:</strong><br>\n1.Train several diverse base models (e.g., a CNN for feature extraction, an LSTM for sequential patterns, a Gradient Boosting model on hand-crafted features).</p>\n<ol>\n<li>Generate \"out-of-fold\" predictions from these base models on your training data (using cross-validation to avoid data leakage).</li>\n</ol>\n<p>3.Use these out-of-fold predictions as features to train a new meta-model (e.g., Logistic Regression, Ridge Classifier, or even a small neural network).</p>\n<ol>\n<li><p>When predicting on the test set, first get predictions from all base models, then feed these into your trained meta-model.</p></li>\n<li><p><strong>Bagging (Bootstrap Aggregating):</strong></p></li>\n</ol>\n<p><strong>Concept:</strong> Trains multiple instances of the same type of model on different bootstrap samples (random subsets with replacement) of the training data. Predictions are then aggregated (e.g., by majority voting for classification). Random Forest is a prime example of a bagging algorithm.</p>\n<p><strong>Benefit:</strong> Reduces variance and helps prevent overfitting, leading to more stable and robust models. While often applied to single model types, you can also consider bagging heterogeneous models if carefully designed.</p>\n<p><strong>Key considerations while ensembling:</strong></p>\n<ol>\n<li><p>Use diverse model architectures and modelling strategy to capture different aspects of data which improves ensembling</p></li>\n<li><p>Build robust local cross validation scheme which is well aligned with LB . This helps not only overfitting public LB but also building powerful and strong confident solution ensemble.</p></li>\n<li><p>Tune the hyperparameters of your base models and, if applicable, your meta-model.</p></li>\n<li><p>Dont over ensemble as ensembling too many models can lead to diminishing returns and higher inference time which can impact models in production.</p></li>\n</ol>\n<p>Expecting to see much reliant and creative solution by end of this competition.<br>\nHappy Kaggling :)</p>",
  "messages": [
    {
      "id": "3233792",
      "postDate": "06/27/2025 07:09:34",
      "content": "<p>I have noticed this in almost any Kaggle competition midway or even at the start someone submits a high scoring public kernel which is just simple ensemble of many good scoring public kernels employing techniques like simple averaging or weighted averaging . While the allure of a quick leaderboard jump is understandable, especially in a Human Activity Recognition (HAR) multi-class classification problem, blindly blending models from public kernels, particularly through simple averaging, can be a detrimental strategy in the long run.</p>\n<p><strong>Why Blind Blending (especially simple averaging) is Problematic for HAR (and generally):</strong></p>\n<p>1.<strong>Lack of model diversity and redundancy :</strong>Most of the public kernels are basically for of similar ideas with just some minor tweaks and augmentations because of this the model diversity can be low and the variance in the predictions of model can be low making the entire process of ensembling sub optimal</p>\n<ol>\n<li><p><strong>Overfitting to public leaderboard :</strong> Public kernels are designed or optimized keeping public leaderboard in the mind so if you blend them without understanding their underlying validation strategy (or lack of thereof) you risk creating an ensemble that is heavily overfitted to the public test set, leading to a significant drop in performance on the private leaderboard. This is specially true since different kernels might be using different validation schemes.</p></li>\n<li><p><strong>Ignoring Models weakness and strengths:</strong> Each model specializes in classifying certain activities or handling specific types of noise in the sensor data. Simple averaging treats all models equally, even if one performs poorly on a particular class or a specific segment of the data. For multi-class classification like HAR, some models might have a better recall for rare activities, while others might be more precise for common ones.</p></li>\n</ol>\n<p>4.<strong>No interpretability and learning :</strong> Blind blending of models provides no insight as to which model is better or which contributes most effectively . This hinders your understanding of the problem and prevents you from developing more robust and insightful solutions in the future.</p>\n<ol>\n<li><strong>Data Leakage and Inconsistent Preprocessing :</strong> Public kernels might use different preprocessing steps and folds simply using them without understanding and blending can lead to data leakage and development of overly optimistic models</li>\n</ol>\n<p><strong>Better ways to do Ensembling</strong></p>\n<p><strong>1. Weighted Averaging:</strong></p>\n<p><strong>Concept</strong>: Instead of equal weights assigned different weights to each model based on its performance on your local cross validation set.</p>\n<p><strong>How to do it:</strong> Train models individually and evaluate them on cross validation set and based on its performance , You can even optimize these weights using a simple optimization algorithm (e.g., grid search or Nelder-Mead) on your validation predictions.</p>\n<p><strong>2. Stacking:</strong></p>\n<p><strong>Concept :</strong> One of the most powerful concepts in ensembling where output of your base models become input to the meta model . The meta-model then learns how to optimally combine these predictions.</p>\n<p><strong>How to do it:</strong><br>\n1.Train several diverse base models (e.g., a CNN for feature extraction, an LSTM for sequential patterns, a Gradient Boosting model on hand-crafted features).</p>\n<ol>\n<li>Generate \"out-of-fold\" predictions from these base models on your training data (using cross-validation to avoid data leakage).</li>\n</ol>\n<p>3.Use these out-of-fold predictions as features to train a new meta-model (e.g., Logistic Regression, Ridge Classifier, or even a small neural network).</p>\n<ol>\n<li><p>When predicting on the test set, first get predictions from all base models, then feed these into your trained meta-model.</p></li>\n<li><p><strong>Bagging (Bootstrap Aggregating):</strong></p></li>\n</ol>\n<p><strong>Concept:</strong> Trains multiple instances of the same type of model on different bootstrap samples (random subsets with replacement) of the training data. Predictions are then aggregated (e.g., by majority voting for classification). Random Forest is a prime example of a bagging algorithm.</p>\n<p><strong>Benefit:</strong> Reduces variance and helps prevent overfitting, leading to more stable and robust models. While often applied to single model types, you can also consider bagging heterogeneous models if carefully designed.</p>\n<p><strong>Key considerations while ensembling:</strong></p>\n<ol>\n<li><p>Use diverse model architectures and modelling strategy to capture different aspects of data which improves ensembling</p></li>\n<li><p>Build robust local cross validation scheme which is well aligned with LB . This helps not only overfitting public LB but also building powerful and strong confident solution ensemble.</p></li>\n<li><p>Tune the hyperparameters of your base models and, if applicable, your meta-model.</p></li>\n<li><p>Dont over ensemble as ensembling too many models can lead to diminishing returns and higher inference time which can impact models in production.</p></li>\n</ol>\n<p>Expecting to see much reliant and creative solution by end of this competition.<br>\nHappy Kaggling :)</p>",
      "rawMarkdown": "I have noticed this in almost any Kaggle competition midway or even at the start someone submits a high scoring public kernel which is just simple ensemble of many good scoring public kernels employing techniques like simple averaging or weighted averaging . While the allure of a quick leaderboard jump is understandable, especially in a Human Activity Recognition (HAR) multi-class classification problem, blindly blending models from public kernels, particularly through simple averaging, can be a detrimental strategy in the long run.\n\n**Why Blind Blending (especially simple averaging) is Problematic for HAR (and generally):**\n\n1.**Lack of model diversity and redundancy :**Most of the public kernels are basically for of similar ideas with just some minor tweaks and augmentations because of this the model diversity can be low and the variance in the predictions of model can be low making the entire process of ensembling sub optimal\n\n2. **Overfitting to public leaderboard :** Public kernels are designed or optimized keeping public leaderboard in the mind so if you blend them without understanding their underlying validation strategy (or lack of thereof) you risk creating an ensemble that is heavily overfitted to the public test set, leading to a significant drop in performance on the private leaderboard. This is specially true since different kernels might be using different validation schemes.\n\n3. **Ignoring Models weakness and strengths:** Each model specializes in classifying certain activities or handling specific types of noise in the sensor data. Simple averaging treats all models equally, even if one performs poorly on a particular class or a specific segment of the data. For multi-class classification like HAR, some models might have a better recall for rare activities, while others might be more precise for common ones.\n\n4.**No interpretability and learning :** Blind blending of models provides no insight as to which model is better or which contributes most effectively . This hinders your understanding of the problem and prevents you from developing more robust and insightful solutions in the future.\n\n5. **Data Leakage and Inconsistent Preprocessing :** Public kernels might use different preprocessing steps and folds simply using them without understanding and blending can lead to data leakage and development of overly optimistic models\n\n\n**Better ways to do Ensembling**\n\n**1. Weighted Averaging:**\n\n**Concept**: Instead of equal weights assigned different weights to each model based on its performance on your local cross validation set.\n\n**How to do it:** Train models individually and evaluate them on cross validation set and based on its performance , You can even optimize these weights using a simple optimization algorithm (e.g., grid search or Nelder-Mead) on your validation predictions.\n\n**2. Stacking:**\n\n**Concept :** One of the most powerful concepts in ensembling where output of your base models become input to the meta model . The meta-model then learns how to optimally combine these predictions.\n\n\n**How to do it:**\n1.Train several diverse base models (e.g., a CNN for feature extraction, an LSTM for sequential patterns, a Gradient Boosting model on hand-crafted features).\n\n2. Generate \"out-of-fold\" predictions from these base models on your training data (using cross-validation to avoid data leakage).\n\n3.Use these out-of-fold predictions as features to train a new meta-model (e.g., Logistic Regression, Ridge Classifier, or even a small neural network).\n\n4. When predicting on the test set, first get predictions from all base models, then feed these into your trained meta-model.\n\n3. **Bagging (Bootstrap Aggregating):**\n\n**Concept:** Trains multiple instances of the same type of model on different bootstrap samples (random subsets with replacement) of the training data. Predictions are then aggregated (e.g., by majority voting for classification). Random Forest is a prime example of a bagging algorithm.\n\n**Benefit:** Reduces variance and helps prevent overfitting, leading to more stable and robust models. While often applied to single model types, you can also consider bagging heterogeneous models if carefully designed.\n\n**Key considerations while ensembling:**\n\n1. Use diverse model architectures and modelling strategy to capture different aspects of data which improves ensembling\n\n2. Build robust local cross validation scheme which is well aligned with LB . This helps not only overfitting public LB but also building powerful and strong confident solution ensemble.\n\n3. Tune the hyperparameters of your base models and, if applicable, your meta-model.\n\n4. Dont over ensemble as ensembling too many models can lead to diminishing returns and higher inference time which can impact models in production.\n\nExpecting to see much reliant and creative solution by end of this competition.\nHappy Kaggling :)",
      "votes": null
    },
    {
      "id": "3233889",
      "postDate": "06/27/2025 09:29:15",
      "content": "<p>Stacking is also quite risky approach when train data are scarce and train/public test/private test domain shift is expected.</p>",
      "rawMarkdown": "Stacking is also quite risky approach when train data are scarce and train/public test/private test domain shift is expected.",
      "votes": null
    },
    {
      "id": "3233926",
      "postDate": "06/27/2025 10:42:13",
      "content": "<p>This is a big problem, especially in non-code competitions. It makes it impossible to find good quality notebooks when the notebook section is flooded with these kinds of notebooks. It also misleads beginners into thinking they are good approaches since they score high on the public LB, even though most of the time they are just overfitting to public LB. Sadly, there is nothing we can do to fix this issue, and Kaggle isn’t taking any action either. Reporting these notebooks doesn’t help either, I’ve tried a few times before. <em>Heavy sigh…</em></p>",
      "rawMarkdown": "This is a big problem, especially in non-code competitions. It makes it impossible to find good quality notebooks when the notebook section is flooded with these kinds of notebooks. It also misleads beginners into thinking they are good approaches since they score high on the public LB, even though most of the time they are just overfitting to public LB. Sadly, there is nothing we can do to fix this issue, and Kaggle isn’t taking any action either. Reporting these notebooks doesn’t help either, I’ve tried a few times before. *Heavy sigh...*",
      "votes": null
    },
    {
      "id": "3233963",
      "postDate": "06/27/2025 11:22:43",
      "content": "<p>The problematic thing is some of this notebooks are submitted few days before competition is about to end and it hampers other serious competitors rank and their thought process</p>",
      "rawMarkdown": "The problematic thing is some of this notebooks are submitted few days before competition is about to end and it hampers other serious competitors rank and their thought process",
      "votes": null
    },
    {
      "id": "3234029",
      "postDate": "06/27/2025 12:49:33",
      "content": "<p>Been there done that CV without stacking: 0.77 LB: 0.77. Then I started stacking CV: 0.82 and LB: 0.7.</p>",
      "rawMarkdown": "Been there done that CV without stacking: 0.77 LB: 0.77. Then I started stacking CV: 0.82 and LB: 0.7.",
      "votes": null
    },
    {
      "id": "3234031",
      "postDate": "06/27/2025 12:50:29",
      "content": "<p>Even averaging isn't really helping.</p>",
      "rawMarkdown": "Even averaging isn't really helping.",
      "votes": null
    },
    {
      "id": "3234037",
      "postDate": "06/27/2025 12:53:40",
      "content": "<p>Yes but higher scores <em>usually</em> mean more notebook medals</p>",
      "rawMarkdown": "Yes but higher scores *usually* mean more notebook medals",
      "votes": null
    },
    {
      "id": "3235153",
      "postDate": "06/28/2025 19:02:17",
      "content": "<p>is a shake-up expected this time too?</p>",
      "rawMarkdown": "is a shake-up expected this time too?",
      "votes": null
    },
    {
      "id": "3235190",
      "postDate": "06/28/2025 20:38:52",
      "content": "<p>I do think some shake-up can be expected. Especially for ensembles.</p>",
      "rawMarkdown": "I do think some shake-up can be expected. Especially for ensembles.",
      "votes": null
    },
    {
      "id": "3235345",
      "postDate": "06/29/2025 05:13:20",
      "content": "<p>When I try to generalized my model by checking per class f1 score every time, my lb score kinda goes down 👇, is this a normal behaviour?</p>",
      "rawMarkdown": "When I try to generalized my model by checking per class f1 score every time, my lb score kinda goes down 👇, is this a normal behaviour?",
      "votes": null
    },
    {
      "id": "3235378",
      "postDate": "06/29/2025 06:29:45",
      "content": "<p>The cure is to improve public ensemble by your models or to improve ensemble itself.</p>",
      "rawMarkdown": "The cure is to improve public ensemble by your models or to improve ensemble itself.",
      "votes": null
    },
    {
      "id": "3236946",
      "postDate": "06/30/2025 19:07:30",
      "content": "<p>THis is dangerous</p>",
      "rawMarkdown": "THis is dangerous",
      "votes": null
    },
    {
      "id": "3237092",
      "postDate": "06/30/2025 22:57:49",
      "content": "<p>Are you recalled Birdclef hahaha… Now we have reliable CV.</p>",
      "rawMarkdown": "Are you recalled Birdclef hahaha... Now we have reliable CV.",
      "votes": null
    },
    {
      "id": "3241153",
      "postDate": "07/04/2025 15:37:20",
      "content": "<p>Birdclef is an example of Kaggle competition where merging blindly might benefit because the hosts are too lazy to prepare a good reliable validation sets however still in competition where we have reliable Cv Ensembling blindly can led to selection of sub optimal solutions </p>",
      "rawMarkdown": "Birdclef is an example of Kaggle competition where merging blindly might benefit because the hosts are too lazy to prepare a good reliable validation sets however still in competition where we have reliable Cv Ensembling blindly can led to selection of sub optimal solutions",
      "votes": null
    },
    {
      "id": "3241158",
      "postDate": "07/04/2025 15:38:18",
      "content": "<p>That is what I meant when I said such notebooks are optimized for Public Leaderboard so that they can get more upvotes</p>",
      "rawMarkdown": "That is what I meant when I said such notebooks are optimized for Public Leaderboard so that they can get more upvotes",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3233889,
      "author_name": "ogurtsov",
      "author_url": "",
      "post_date": "06/27/2025 09:29:15",
      "content": "<p>Stacking is also quite risky approach when train data are scarce and train/public test/private test domain shift is expected.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3234029,
          "author_name": "stefanoclss",
          "author_url": "",
          "post_date": "06/27/2025 12:49:33",
          "content": "<p>Been there done that CV without stacking: 0.77 LB: 0.77. Then I started stacking CV: 0.82 and LB: 0.7.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3234031,
              "author_name": "stefanoclss",
              "author_url": "",
              "post_date": "06/27/2025 12:50:29",
              "content": "<p>Even averaging isn't really helping.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3233926,
      "author_name": "ravaghi",
      "author_url": "",
      "post_date": "06/27/2025 10:42:13",
      "content": "<p>This is a big problem, especially in non-code competitions. It makes it impossible to find good quality notebooks when the notebook section is flooded with these kinds of notebooks. It also misleads beginners into thinking they are good approaches since they score high on the public LB, even though most of the time they are just overfitting to public LB. Sadly, there is nothing we can do to fix this issue, and Kaggle isn’t taking any action either. Reporting these notebooks doesn’t help either, I’ve tried a few times before. <em>Heavy sigh…</em></p>",
      "votes": null,
      "replies": [
        {
          "id": 3233963,
          "author_name": "sayedathar11",
          "author_url": "",
          "post_date": "06/27/2025 11:22:43",
          "content": "<p>The problematic thing is some of this notebooks are submitted few days before competition is about to end and it hampers other serious competitors rank and their thought process</p>",
          "votes": null,
          "replies": [
            {
              "id": 3235378,
              "author_name": "ogurtsov",
              "author_url": "",
              "post_date": "06/29/2025 06:29:45",
              "content": "<p>The cure is to improve public ensemble by your models or to improve ensemble itself.</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 3235153,
          "author_name": "vaishnavimudaliar",
          "author_url": "",
          "post_date": "06/28/2025 19:02:17",
          "content": "<p>is a shake-up expected this time too?</p>",
          "votes": null,
          "replies": [
            {
              "id": 3235190,
              "author_name": "stefanoclss",
              "author_url": "",
              "post_date": "06/28/2025 20:38:52",
              "content": "<p>I do think some shake-up can be expected. Especially for ensembles.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3235345,
                  "author_name": "vaishnavimudaliar",
                  "author_url": "",
                  "post_date": "06/29/2025 05:13:20",
                  "content": "<p>When I try to generalized my model by checking per class f1 score every time, my lb score kinda goes down 👇, is this a normal behaviour?</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 3234037,
      "author_name": "gerwynng",
      "author_url": "",
      "post_date": "06/27/2025 12:53:40",
      "content": "<p>Yes but higher scores <em>usually</em> mean more notebook medals</p>",
      "votes": null,
      "replies": [
        {
          "id": 3241158,
          "author_name": "sayedathar11",
          "author_url": "",
          "post_date": "07/04/2025 15:38:18",
          "content": "<p>That is what I meant when I said such notebooks are optimized for Public Leaderboard so that they can get more upvotes</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3236946,
      "author_name": "lukesun07",
      "author_url": "",
      "post_date": "06/30/2025 19:07:30",
      "content": "<p>THis is dangerous</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3237092,
      "author_name": "kurisew",
      "author_url": "",
      "post_date": "06/30/2025 22:57:49",
      "content": "<p>Are you recalled Birdclef hahaha… Now we have reliable CV.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3241153,
          "author_name": "sayedathar11",
          "author_url": "",
          "post_date": "07/04/2025 15:37:20",
          "content": "<p>Birdclef is an example of Kaggle competition where merging blindly might benefit because the hosts are too lazy to prepare a good reliable validation sets however still in competition where we have reliable Cv Ensembling blindly can led to selection of sub optimal solutions </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3233792": "I have noticed this in almost any Kaggle competition midway or even at the start someone submits a high scoring public kernel which is just simple ensemble of many good scoring public kernels employing techniques like simple averaging or weighted averaging . While the allure of a quick leaderboard jump is understandable, especially in a Human Activity Recognition (HAR) multi-class classification problem, blindly blending models from public kernels, particularly through simple averaging, can be a detrimental strategy in the long run.\n\n**Why Blind Blending (especially simple averaging) is Problematic for HAR (and generally):**\n\n1.**Lack of model diversity and redundancy :**Most of the public kernels are basically for of similar ideas with just some minor tweaks and augmentations because of this the model diversity can be low and the variance in the predictions of model can be low making the entire process of ensembling sub optimal\n\n2. **Overfitting to public leaderboard :** Public kernels are designed or optimized keeping public leaderboard in the mind so if you blend them without understanding their underlying validation strategy (or lack of thereof) you risk creating an ensemble that is heavily overfitted to the public test set, leading to a significant drop in performance on the private leaderboard. This is specially true since different kernels might be using different validation schemes.\n\n3. **Ignoring Models weakness and strengths:** Each model specializes in classifying certain activities or handling specific types of noise in the sensor data. Simple averaging treats all models equally, even if one performs poorly on a particular class or a specific segment of the data. For multi-class classification like HAR, some models might have a better recall for rare activities, while others might be more precise for common ones.\n\n4.**No interpretability and learning :** Blind blending of models provides no insight as to which model is better or which contributes most effectively . This hinders your understanding of the problem and prevents you from developing more robust and insightful solutions in the future.\n\n5. **Data Leakage and Inconsistent Preprocessing :** Public kernels might use different preprocessing steps and folds simply using them without understanding and blending can lead to data leakage and development of overly optimistic models\n\n\n**Better ways to do Ensembling**\n\n**1. Weighted Averaging:**\n\n**Concept**: Instead of equal weights assigned different weights to each model based on its performance on your local cross validation set.\n\n**How to do it:** Train models individually and evaluate them on cross validation set and based on its performance , You can even optimize these weights using a simple optimization algorithm (e.g., grid search or Nelder-Mead) on your validation predictions.\n\n**2. Stacking:**\n\n**Concept :** One of the most powerful concepts in ensembling where output of your base models become input to the meta model . The meta-model then learns how to optimally combine these predictions.\n\n\n**How to do it:**\n1.Train several diverse base models (e.g., a CNN for feature extraction, an LSTM for sequential patterns, a Gradient Boosting model on hand-crafted features).\n\n2. Generate \"out-of-fold\" predictions from these base models on your training data (using cross-validation to avoid data leakage).\n\n3.Use these out-of-fold predictions as features to train a new meta-model (e.g., Logistic Regression, Ridge Classifier, or even a small neural network).\n\n4. When predicting on the test set, first get predictions from all base models, then feed these into your trained meta-model.\n\n3. **Bagging (Bootstrap Aggregating):**\n\n**Concept:** Trains multiple instances of the same type of model on different bootstrap samples (random subsets with replacement) of the training data. Predictions are then aggregated (e.g., by majority voting for classification). Random Forest is a prime example of a bagging algorithm.\n\n**Benefit:** Reduces variance and helps prevent overfitting, leading to more stable and robust models. While often applied to single model types, you can also consider bagging heterogeneous models if carefully designed.\n\n**Key considerations while ensembling:**\n\n1. Use diverse model architectures and modelling strategy to capture different aspects of data which improves ensembling\n\n2. Build robust local cross validation scheme which is well aligned with LB . This helps not only overfitting public LB but also building powerful and strong confident solution ensemble.\n\n3. Tune the hyperparameters of your base models and, if applicable, your meta-model.\n\n4. Dont over ensemble as ensembling too many models can lead to diminishing returns and higher inference time which can impact models in production.\n\nExpecting to see much reliant and creative solution by end of this competition.\nHappy Kaggling :)",
    "3233889": "Stacking is also quite risky approach when train data are scarce and train/public test/private test domain shift is expected.",
    "3233926": "This is a big problem, especially in non-code competitions. It makes it impossible to find good quality notebooks when the notebook section is flooded with these kinds of notebooks. It also misleads beginners into thinking they are good approaches since they score high on the public LB, even though most of the time they are just overfitting to public LB. Sadly, there is nothing we can do to fix this issue, and Kaggle isn’t taking any action either. Reporting these notebooks doesn’t help either, I’ve tried a few times before. *Heavy sigh...*",
    "3233963": "The problematic thing is some of this notebooks are submitted few days before competition is about to end and it hampers other serious competitors rank and their thought process",
    "3234029": "Been there done that CV without stacking: 0.77 LB: 0.77. Then I started stacking CV: 0.82 and LB: 0.7.",
    "3234031": "Even averaging isn't really helping.",
    "3234037": "Yes but higher scores *usually* mean more notebook medals",
    "3235153": "is a shake-up expected this time too?",
    "3235190": "I do think some shake-up can be expected. Especially for ensembles.",
    "3235345": "When I try to generalized my model by checking per class f1 score every time, my lb score kinda goes down 👇, is this a normal behaviour?",
    "3235378": "The cure is to improve public ensemble by your models or to improve ensemble itself.",
    "3236946": "THis is dangerous",
    "3237092": "Are you recalled Birdclef hahaha... Now we have reliable CV.",
    "3241153": "Birdclef is an example of Kaggle competition where merging blindly might benefit because the hosts are too lazy to prepare a good reliable validation sets however still in competition where we have reliable Cv Ensembling blindly can led to selection of sub optimal solutions",
    "3241158": "That is what I meant when I said such notebooks are optimized for Public Leaderboard so that they can get more upvotes"
  },
  "source": "meta"
}