{
  "id": 457534,
  "title": "What's the best way to ensemble multiple models?",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/457534",
  "author_name": "",
  "post_date": "2023-11-25T10:21:28.875874100Z",
  "votes": 8,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I was wondering what's the best way to ensemble predictions from multiple models for this competition.</p>\n<p>Our team has tried the following naive approaches so far:</p>\n<ol>\n<li>Taking the union of predictions.</li>\n<li>Taking the highest confidence prediction (onset / awake) for each segment of time (e.g., 360 steps - a hyperparameter).</li>\n</ol>\n<p>But the above didn't work out for us.</p>\n<p>Any ideas / tips?</p>",
  "messages": [
    {
      "id": "2537599",
      "postDate": "11/25/2023 10:21:28",
      "content": "<p>I was wondering what's the best way to ensemble predictions from multiple models for this competition.</p>\n<p>Our team has tried the following naive approaches so far:</p>\n<ol>\n<li>Taking the union of predictions.</li>\n<li>Taking the highest confidence prediction (onset / awake) for each segment of time (e.g., 360 steps - a hyperparameter).</li>\n</ol>\n<p>But the above didn't work out for us.</p>\n<p>Any ideas / tips?</p>",
      "rawMarkdown": "I was wondering what's the best way to ensemble predictions from multiple models for this competition.\n\nOur team has tried the following naive approaches so far:\n\n1. Taking the union of predictions.\n2. Taking the highest confidence prediction (onset / awake) for each segment of time (e.g., 360 steps - a hyperparameter).\n\nBut the above didn't work out for us.\n\nAny ideas / tips?",
      "votes": null
    },
    {
      "id": "2537753",
      "postDate": "11/25/2023 13:07:16",
      "content": "<p>If your models predict each step, try to take an average of the models output before passing to post processing.</p>",
      "rawMarkdown": "If your models predict each step, try to take an average of the models output before passing to post processing.",
      "votes": null
    },
    {
      "id": "2537810",
      "postDate": "11/25/2023 13:56:56",
      "content": "<p>I'm not sure about your model training approach. My model predicts the probabilities of two events at each step, so for the ensemble, I took the average probabilities of multiple models before entering the post-processing stage.</p>",
      "rawMarkdown": "I'm not sure about your model training approach. My model predicts the probabilities of two events at each step, so for the ensemble, I took the average probabilities of multiple models before entering the post-processing stage.",
      "votes": null
    },
    {
      "id": "2538576",
      "postDate": "11/26/2023 09:24:10",
      "content": "<p>In your model, try mixing up your models' predictions. Weighted averaging is cool (give more weight to better models), or try stacking where a new model learns from your old ones. Experiment to see what rocks! 🚀🤓</p>",
      "rawMarkdown": "In your model, try mixing up your models' predictions. Weighted averaging is cool (give more weight to better models), or try stacking where a new model learns from your old ones. Experiment to see what rocks! 🚀🤓",
      "votes": null
    },
    {
      "id": "2539685",
      "postDate": "11/27/2023 09:06:31",
      "content": "<p>hey <a href=\"https://www.kaggle.com/sambitmukherjee\" target=\"_blank\">@sambitmukherjee</a> I just observe a bug in your strategy. For multiple models, they just make predictions on every seperate step, suppor model A has a peak of 36, but how would it be when model B has a peak of 26/30, how to ensemble them?</p>",
      "rawMarkdown": "hey @sambitmukherjee I just observe a bug in your strategy. For multiple models, they just make predictions on every seperate step, suppor model A has a peak of 36, but how would it be when model B has a peak of 26/30, how to ensemble them?",
      "votes": null
    },
    {
      "id": "2540925",
      "postDate": "11/28/2023 05:26:56",
      "content": "<p>Yes you are right. Now we have moved to a different approach: taking the weighted average of probabilities that are output by the different models.</p>",
      "rawMarkdown": "Yes you are right. Now we have moved to a different approach: taking the weighted average of probabilities that are output by the different models.",
      "votes": null
    },
    {
      "id": "2541415",
      "postDate": "11/28/2023 13:27:38",
      "content": "<p>Thanks for the ideas, it enlights me</p>",
      "rawMarkdown": "Thanks for the ideas, it enlights me",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2537753,
      "author_name": "akorol",
      "author_url": "",
      "post_date": "11/25/2023 13:07:16",
      "content": "<p>If your models predict each step, try to take an average of the models output before passing to post processing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2537810,
      "author_name": "koalas",
      "author_url": "",
      "post_date": "11/25/2023 13:56:56",
      "content": "<p>I'm not sure about your model training approach. My model predicts the probabilities of two events at each step, so for the ensemble, I took the average probabilities of multiple models before entering the post-processing stage.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2538576,
      "author_name": "ziyuechen3",
      "author_url": "",
      "post_date": "11/26/2023 09:24:10",
      "content": "<p>In your model, try mixing up your models' predictions. Weighted averaging is cool (give more weight to better models), or try stacking where a new model learns from your old ones. Experiment to see what rocks! 🚀🤓</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2539685,
      "author_name": "liushundawang",
      "author_url": "",
      "post_date": "11/27/2023 09:06:31",
      "content": "<p>hey <a href=\"https://www.kaggle.com/sambitmukherjee\" target=\"_blank\">@sambitmukherjee</a> I just observe a bug in your strategy. For multiple models, they just make predictions on every seperate step, suppor model A has a peak of 36, but how would it be when model B has a peak of 26/30, how to ensemble them?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2540925,
          "author_name": "sambitmukherjee",
          "author_url": "",
          "post_date": "11/28/2023 05:26:56",
          "content": "<p>Yes you are right. Now we have moved to a different approach: taking the weighted average of probabilities that are output by the different models.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2541415,
      "author_name": "jackdhr",
      "author_url": "",
      "post_date": "11/28/2023 13:27:38",
      "content": "<p>Thanks for the ideas, it enlights me</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2537599": "I was wondering what's the best way to ensemble predictions from multiple models for this competition.\n\nOur team has tried the following naive approaches so far:\n\n1. Taking the union of predictions.\n2. Taking the highest confidence prediction (onset / awake) for each segment of time (e.g., 360 steps - a hyperparameter).\n\nBut the above didn't work out for us.\n\nAny ideas / tips?",
    "2537753": "If your models predict each step, try to take an average of the models output before passing to post processing.",
    "2537810": "I'm not sure about your model training approach. My model predicts the probabilities of two events at each step, so for the ensemble, I took the average probabilities of multiple models before entering the post-processing stage.",
    "2538576": "In your model, try mixing up your models' predictions. Weighted averaging is cool (give more weight to better models), or try stacking where a new model learns from your old ones. Experiment to see what rocks! 🚀🤓",
    "2539685": "hey @sambitmukherjee I just observe a bug in your strategy. For multiple models, they just make predictions on every seperate step, suppor model A has a peak of 36, but how would it be when model B has a peak of 26/30, how to ensemble them?",
    "2540925": "Yes you are right. Now we have moved to a different approach: taking the weighted average of probabilities that are output by the different models.",
    "2541415": "Thanks for the ideas, it enlights me"
  },
  "source": "meta"
}