{
  "id": 598567,
  "title": "Model performance analysis",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/598567",
  "author_name": "",
  "post_date": "2025-08-11T17:27:02.869832100Z",
  "votes": 4,
  "comment_count": 4,
  "views": 0,
  "content": "<p>How do your <code>IMU only</code> model and <code>IMU-THM-TOF</code> model perform against different features? I made a notebook to analyze:</p>\n<ol>\n<li>Binary vs. multi-class performance — target vs. non-target gestures, and detailed target gesture classification.</li>\n<li>Gesture confusion patterns — which gestures are most often mistaken for each other.</li>\n<li>Impact of missing data — rotation, THM, and TOF sensor gaps.</li>\n<li>Demographic &amp; physical factors — adult/child, age, handedness, height, shoulder-to-wrist distance, elbow-to-wrist distance.</li>\n</ol>\n<p>Through this analysis, I saw that my <code>IMU only</code> model performs better than my <code>IMU-THM-TOF</code> when THM &amp; TOF are missing so I changed my ensemble strategy:</p>\n<p><strong>Before:</strong> <code>0.7*IMU-THM-TOF + 0.3*IMU-only</code> --&gt; LB: 0.834<br>\n<strong>After:</strong> If THM &amp; TOF are not available: <code>0.4*IMU-THM-TOF + 0.6*IMU-only</code> else <code>0.7*IMU-THM-TOF + 0.3*IMU-only</code> --&gt; LB: 0.837</p>\n<p>Link: <a href=\"https://www.kaggle.com/code/nguyenhoa/model-performance-analysis\" target=\"_blank\">Model performance analysis</a></p>",
  "messages": [
    {
      "id": "3267695",
      "postDate": "08/11/2025 17:27:02",
      "content": "<p>How do your <code>IMU only</code> model and <code>IMU-THM-TOF</code> model perform against different features? I made a notebook to analyze:</p>\n<ol>\n<li>Binary vs. multi-class performance — target vs. non-target gestures, and detailed target gesture classification.</li>\n<li>Gesture confusion patterns — which gestures are most often mistaken for each other.</li>\n<li>Impact of missing data — rotation, THM, and TOF sensor gaps.</li>\n<li>Demographic &amp; physical factors — adult/child, age, handedness, height, shoulder-to-wrist distance, elbow-to-wrist distance.</li>\n</ol>\n<p>Through this analysis, I saw that my <code>IMU only</code> model performs better than my <code>IMU-THM-TOF</code> when THM &amp; TOF are missing so I changed my ensemble strategy:</p>\n<p><strong>Before:</strong> <code>0.7*IMU-THM-TOF + 0.3*IMU-only</code> --&gt; LB: 0.834<br>\n<strong>After:</strong> If THM &amp; TOF are not available: <code>0.4*IMU-THM-TOF + 0.6*IMU-only</code> else <code>0.7*IMU-THM-TOF + 0.3*IMU-only</code> --&gt; LB: 0.837</p>\n<p>Link: <a href=\"https://www.kaggle.com/code/nguyenhoa/model-performance-analysis\" target=\"_blank\">Model performance analysis</a></p>",
      "rawMarkdown": "How do your `IMU only` model and `IMU-THM-TOF` model perform against different features? I made a notebook to analyze:\n\n1. Binary vs. multi-class performance — target vs. non-target gestures, and detailed target gesture classification.\n2. Gesture confusion patterns — which gestures are most often mistaken for each other.\n3. Impact of missing data — rotation, THM, and TOF sensor gaps.\n4. Demographic & physical factors — adult/child, age, handedness, height, shoulder-to-wrist distance, elbow-to-wrist distance.\n\nThrough this analysis, I saw that my `IMU only` model performs better than my `IMU-THM-TOF` when THM & TOF are missing so I changed my ensemble strategy:\n\n**Before:** `0.7*IMU-THM-TOF + 0.3*IMU-only` --> LB: 0.834\n**After:** If THM & TOF are not available: `0.4*IMU-THM-TOF + 0.6*IMU-only` else `0.7*IMU-THM-TOF + 0.3*IMU-only` --> LB: 0.837\n\nLink: [Model performance analysis](https://www.kaggle.com/code/nguyenhoa/model-performance-analysis)",
      "votes": null
    },
    {
      "id": "3267708",
      "postDate": "08/11/2025 18:01:54",
      "content": "<p>I thought we should just switch the model completely if THM &amp; TOF not available:</p>\n<p>If THM &amp; TOF are not available: 1 * IMU-only else 1 * IMU-THM-TOF </p>",
      "rawMarkdown": "I thought we should just switch the model completely if THM & TOF not available:\n\nIf THM & TOF are not available: 1 * IMU-only else 1 * IMU-THM-TOF",
      "votes": null
    },
    {
      "id": "3267769",
      "postDate": "08/11/2025 20:30:19",
      "content": "<p>It’s interesting that the LB dropped when I totally switch models. Perhaps that’s thanks to the strength of ensemble. Even though IMU-THM-TOF performs less than IMU only when THM and TOF are missing, ensemble these two types of models with reasonable weights still give us better results.</p>",
      "rawMarkdown": "It’s interesting that the LB dropped when I totally switch models. Perhaps that’s thanks to the strength of ensemble. Even though IMU-THM-TOF performs less than IMU only when THM and TOF are missing, ensemble these two types of models with reasonable weights still give us better results.",
      "votes": null
    },
    {
      "id": "3267833",
      "postDate": "08/12/2025 01:29:29",
      "content": "<p>may i know what cross-validation strategy are you using for IMU only? thanks.</p>",
      "rawMarkdown": "may i know what cross-validation strategy are you using for IMU only? thanks.",
      "votes": null
    },
    {
      "id": "3268173",
      "postDate": "08/12/2025 14:03:31",
      "content": "<p>I used StratifiedGroupKFold (stratified on gestures and group by subject_id) for both IMU only and IMU-THM-TOF models.</p>",
      "rawMarkdown": "I used StratifiedGroupKFold (stratified on gestures and group by subject_id) for both IMU only and IMU-THM-TOF models.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3267708,
      "author_name": "s617603",
      "author_url": "",
      "post_date": "08/11/2025 18:01:54",
      "content": "<p>I thought we should just switch the model completely if THM &amp; TOF not available:</p>\n<p>If THM &amp; TOF are not available: 1 * IMU-only else 1 * IMU-THM-TOF </p>",
      "votes": null,
      "replies": [
        {
          "id": 3267769,
          "author_name": "nguyenhoa",
          "author_url": "",
          "post_date": "08/11/2025 20:30:19",
          "content": "<p>It’s interesting that the LB dropped when I totally switch models. Perhaps that’s thanks to the strength of ensemble. Even though IMU-THM-TOF performs less than IMU only when THM and TOF are missing, ensemble these two types of models with reasonable weights still give us better results.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3267833,
      "author_name": "proj1234",
      "author_url": "",
      "post_date": "08/12/2025 01:29:29",
      "content": "<p>may i know what cross-validation strategy are you using for IMU only? thanks.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3268173,
          "author_name": "nguyenhoa",
          "author_url": "",
          "post_date": "08/12/2025 14:03:31",
          "content": "<p>I used StratifiedGroupKFold (stratified on gestures and group by subject_id) for both IMU only and IMU-THM-TOF models.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3267695": "How do your `IMU only` model and `IMU-THM-TOF` model perform against different features? I made a notebook to analyze:\n\n1. Binary vs. multi-class performance — target vs. non-target gestures, and detailed target gesture classification.\n2. Gesture confusion patterns — which gestures are most often mistaken for each other.\n3. Impact of missing data — rotation, THM, and TOF sensor gaps.\n4. Demographic & physical factors — adult/child, age, handedness, height, shoulder-to-wrist distance, elbow-to-wrist distance.\n\nThrough this analysis, I saw that my `IMU only` model performs better than my `IMU-THM-TOF` when THM & TOF are missing so I changed my ensemble strategy:\n\n**Before:** `0.7*IMU-THM-TOF + 0.3*IMU-only` --> LB: 0.834\n**After:** If THM & TOF are not available: `0.4*IMU-THM-TOF + 0.6*IMU-only` else `0.7*IMU-THM-TOF + 0.3*IMU-only` --> LB: 0.837\n\nLink: [Model performance analysis](https://www.kaggle.com/code/nguyenhoa/model-performance-analysis)",
    "3267708": "I thought we should just switch the model completely if THM & TOF not available:\n\nIf THM & TOF are not available: 1 * IMU-only else 1 * IMU-THM-TOF",
    "3267769": "It’s interesting that the LB dropped when I totally switch models. Perhaps that’s thanks to the strength of ensemble. Even though IMU-THM-TOF performs less than IMU only when THM and TOF are missing, ensemble these two types of models with reasonable weights still give us better results.",
    "3267833": "may i know what cross-validation strategy are you using for IMU only? thanks.",
    "3268173": "I used StratifiedGroupKFold (stratified on gestures and group by subject_id) for both IMU only and IMU-THM-TOF models."
  },
  "source": "meta"
}