{
  "id": 583952,
  "title": "Missing Thermopile Data - Subject SUBJ_011323 (95% missing)",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/583952",
  "author_name": "",
  "post_date": "2025-06-10T14:53:39.236259200Z",
  "votes": 6,
  "comment_count": 2,
  "views": 0,
  "content": "<p><strong>TLDR:</strong>  <code>SUBJ_011323</code> has 95% of <code>thm_</code> data missing - that's 101 sequences.</p>\n<p>We know that there's lot of missing data in ToF sensors in the training and test. </p>\n<p>We also have some missing data in Thm sensors </p>\n<pre><code>train_df = pl.read_csv(DATA_DIR / )\nthm_cols = [col  col  train_df.columns  col.startswith()]\n\n\nthermopile_missing_by_subject = (\n    train_df\n    .with_columns([pl.col(col).(, None)  col  thm_cols])  \n    .group_by([, ])\n    .agg(\n        ( * pl.sum_horizontal([pl.col(col).null_count()  col  thm_cols]) \n         / (pl.() * (thm_cols))).()\n    )\n    .group_by()\n    .agg(pl.col().mean().())\n    .(, =True)\n    .head()\n)\n\nthermopile_missing_by_subject\n</code></pre>\n<p><strong>Output: Top 10 Subjects by avg thm_ Missing %:</strong></p>\n<pre><code>(, )\n┌─────────────┬─────────────────┐\n│     ┆ avg_missing_pct │\n│ ---         ┆ ---             │\n│ str         ┆ f64             │\n╞═════════════╪═════════════════╡\n│ ┆ .       │\n│ ┆ .            │\n│ ┆ .            │\n│ ┆ .       │\n│ ┆ .       │\n│ ┆ .        │\n│ ┆ .        │\n│ ┆ .        │\n│ ┆ .        │\n│ ┆ .             │\n└─────────────┴─────────────────┘\n</code></pre>\n<p>SUBJ_011323 has 95% of Thm data missing - that's 101 sequences.</p>\n<p>From the data page - ~50% data missing with Thm and ToF sensors </p>\n<blockquote>\n  <p>Expect approximately 3,500 sequences in the hidden test set.<br>\n  Half of the hidden-test sequences are recorded with IMU only; the thermopile (thm_) and time-of-flight (tof__v*) columns are still present but contain null values for those sequences.<br>\n  This will allow us to determine whether adding the time-of-flight and thermopile sensors improves our ability to detect BFRBs. Note also that there is known sensor communication failure in this dataset, resulting in missing data from some sensors in some sequences.</p>\n</blockquote>",
  "messages": [
    {
      "id": "3221199",
      "postDate": "06/10/2025 14:53:39",
      "content": "<p><strong>TLDR:</strong>  <code>SUBJ_011323</code> has 95% of <code>thm_</code> data missing - that's 101 sequences.</p>\n<p>We know that there's lot of missing data in ToF sensors in the training and test. </p>\n<p>We also have some missing data in Thm sensors </p>\n<pre><code>train_df = pl.read_csv(DATA_DIR / )\nthm_cols = [col  col  train_df.columns  col.startswith()]\n\n\nthermopile_missing_by_subject = (\n    train_df\n    .with_columns([pl.col(col).(, None)  col  thm_cols])  \n    .group_by([, ])\n    .agg(\n        ( * pl.sum_horizontal([pl.col(col).null_count()  col  thm_cols]) \n         / (pl.() * (thm_cols))).()\n    )\n    .group_by()\n    .agg(pl.col().mean().())\n    .(, =True)\n    .head()\n)\n\nthermopile_missing_by_subject\n</code></pre>\n<p><strong>Output: Top 10 Subjects by avg thm_ Missing %:</strong></p>\n<pre><code>(, )\n┌─────────────┬─────────────────┐\n│     ┆ avg_missing_pct │\n│ ---         ┆ ---             │\n│ str         ┆ f64             │\n╞═════════════╪═════════════════╡\n│ ┆ .       │\n│ ┆ .            │\n│ ┆ .            │\n│ ┆ .       │\n│ ┆ .       │\n│ ┆ .        │\n│ ┆ .        │\n│ ┆ .        │\n│ ┆ .        │\n│ ┆ .             │\n└─────────────┴─────────────────┘\n</code></pre>\n<p>SUBJ_011323 has 95% of Thm data missing - that's 101 sequences.</p>\n<p>From the data page - ~50% data missing with Thm and ToF sensors </p>\n<blockquote>\n  <p>Expect approximately 3,500 sequences in the hidden test set.<br>\n  Half of the hidden-test sequences are recorded with IMU only; the thermopile (thm_) and time-of-flight (tof__v*) columns are still present but contain null values for those sequences.<br>\n  This will allow us to determine whether adding the time-of-flight and thermopile sensors improves our ability to detect BFRBs. Note also that there is known sensor communication failure in this dataset, resulting in missing data from some sensors in some sequences.</p>\n</blockquote>",
      "rawMarkdown": "**TLDR:**  `SUBJ_011323` has 95% of `thm_` data missing - that's 101 sequences.\n\n\nWe know that there's lot of missing data in ToF sensors in the training and test. \n\nWe also have some missing data in Thm sensors \n\n```\ntrain_df = pl.read_csv(DATA_DIR / \"train.csv\")\nthm_cols = [col for col in train_df.columns if col.startswith('thm_')]\n\n# Calculate average thermopile sensor missing data percentage by subject\nthermopile_missing_by_subject = (\n    train_df\n    .with_columns([pl.col(col).replace(-1, None) for col in thm_cols])  # Convert -1 to null\n    .group_by(['subject', 'sequence_id'])\n    .agg(\n        (100 * pl.sum_horizontal([pl.col(col).null_count() for col in thm_cols]) \n         / (pl.len() * len(thm_cols))).alias('missing_pct')\n    )\n    .group_by('subject')\n    .agg(pl.col('missing_pct').mean().alias('avg_missing_pct'))\n    .sort('avg_missing_pct', descending=True)\n    .head(10)\n)\n\nthermopile_missing_by_subject\n```\n**Output: Top 10 Subjects by avg thm_ Missing %:**\n\n```\nshape: (10, 2)\n┌─────────────┬─────────────────┐\n│ subject     ┆ avg_missing_pct │\n│ ---         ┆ ---             │\n│ str         ┆ f64             │\n╞═════════════╪═════════════════╡\n│ SUBJ_011323 ┆ 95.049505       │\n│ SUBJ_016552 ┆ 20.0            │\n│ SUBJ_044680 ┆ 20.0            │\n│ SUBJ_036450 ┆ 18.039216       │\n│ SUBJ_036405 ┆ 15.294118       │\n│ SUBJ_039498 ┆ 9.411765        │\n│ SUBJ_053217 ┆ 6.213152        │\n│ SUBJ_055211 ┆ 4.061625        │\n│ SUBJ_019262 ┆ 0.784314        │\n│ SUBJ_019297 ┆ 0.0             │\n└─────────────┴─────────────────┘\n```\n\nSUBJ_011323 has 95% of Thm data missing - that's 101 sequences.\n\nFrom the data page - ~50% data missing with Thm and ToF sensors \n\n>Expect approximately 3,500 sequences in the hidden test set.\nHalf of the hidden-test sequences are recorded with IMU only; the thermopile (thm_) and time-of-flight (tof__v*) columns are still present but contain null values for those sequences.\nThis will allow us to determine whether adding the time-of-flight and thermopile sensors improves our ability to detect BFRBs. Note also that there is known sensor communication failure in this dataset, resulting in missing data from some sensors in some sequences.",
      "votes": null
    },
    {
      "id": "3255097",
      "postDate": "07/28/2025 00:26:36",
      "content": "<p>There are also 50 sequences that miss quaterion data (rot_*).</p>",
      "rawMarkdown": "There are also 50 sequences that miss quaterion data (rot_*).",
      "votes": null
    },
    {
      "id": "3255100",
      "postDate": "07/28/2025 00:43:52",
      "content": "<p>This is a great discovery. Thank you so much.</p>",
      "rawMarkdown": "This is a great discovery. Thank you so much.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3255097,
      "author_name": "nguyenhoa",
      "author_url": "",
      "post_date": "07/28/2025 00:26:36",
      "content": "<p>There are also 50 sequences that miss quaterion data (rot_*).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3255100,
      "author_name": "kerta27",
      "author_url": "",
      "post_date": "07/28/2025 00:43:52",
      "content": "<p>This is a great discovery. Thank you so much.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3221199": "**TLDR:**  `SUBJ_011323` has 95% of `thm_` data missing - that's 101 sequences.\n\n\nWe know that there's lot of missing data in ToF sensors in the training and test. \n\nWe also have some missing data in Thm sensors \n\n```\ntrain_df = pl.read_csv(DATA_DIR / \"train.csv\")\nthm_cols = [col for col in train_df.columns if col.startswith('thm_')]\n\n# Calculate average thermopile sensor missing data percentage by subject\nthermopile_missing_by_subject = (\n    train_df\n    .with_columns([pl.col(col).replace(-1, None) for col in thm_cols])  # Convert -1 to null\n    .group_by(['subject', 'sequence_id'])\n    .agg(\n        (100 * pl.sum_horizontal([pl.col(col).null_count() for col in thm_cols]) \n         / (pl.len() * len(thm_cols))).alias('missing_pct')\n    )\n    .group_by('subject')\n    .agg(pl.col('missing_pct').mean().alias('avg_missing_pct'))\n    .sort('avg_missing_pct', descending=True)\n    .head(10)\n)\n\nthermopile_missing_by_subject\n```\n**Output: Top 10 Subjects by avg thm_ Missing %:**\n\n```\nshape: (10, 2)\n┌─────────────┬─────────────────┐\n│ subject     ┆ avg_missing_pct │\n│ ---         ┆ ---             │\n│ str         ┆ f64             │\n╞═════════════╪═════════════════╡\n│ SUBJ_011323 ┆ 95.049505       │\n│ SUBJ_016552 ┆ 20.0            │\n│ SUBJ_044680 ┆ 20.0            │\n│ SUBJ_036450 ┆ 18.039216       │\n│ SUBJ_036405 ┆ 15.294118       │\n│ SUBJ_039498 ┆ 9.411765        │\n│ SUBJ_053217 ┆ 6.213152        │\n│ SUBJ_055211 ┆ 4.061625        │\n│ SUBJ_019262 ┆ 0.784314        │\n│ SUBJ_019297 ┆ 0.0             │\n└─────────────┴─────────────────┘\n```\n\nSUBJ_011323 has 95% of Thm data missing - that's 101 sequences.\n\nFrom the data page - ~50% data missing with Thm and ToF sensors \n\n>Expect approximately 3,500 sequences in the hidden test set.\nHalf of the hidden-test sequences are recorded with IMU only; the thermopile (thm_) and time-of-flight (tof__v*) columns are still present but contain null values for those sequences.\nThis will allow us to determine whether adding the time-of-flight and thermopile sensors improves our ability to detect BFRBs. Note also that there is known sensor communication failure in this dataset, resulting in missing data from some sensors in some sequences.",
    "3255097": "There are also 50 sequences that miss quaterion data (rot_*).",
    "3255100": "This is a great discovery. Thank you so much."
  },
  "source": "meta"
}