{
  "id": 593675,
  "title": "Sensor Data Preprocessing",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/593675",
  "author_name": "",
  "post_date": "2025-07-30T01:09:40.505122800Z",
  "votes": 31,
  "comment_count": 6,
  "views": 0,
  "content": "<p>If you want explore EDA as well as data cleaning</p>\n<p>=&gt; <a href=\"https://www.kaggle.com/code/seoyunje/cmi2025-sensor-data-analysis-with-spectrum-eda\" target=\"_blank\">https://www.kaggle.com/code/seoyunje/cmi2025-sensor-data-analysis-with-spectrum-eda</a></p>\n<h2>📌 Behavior Flow</h2>\n<p>Every sequence contains <strong>3 behavior phases</strong>, categorized as follows:</p>\n<h3>🔹 Flow 1</h3>\n<table>\n<thead>\n<tr>\n<th>Step</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Step 1</td>\n<td>Relaxes and moves hand to target location</td>\n</tr>\n<tr>\n<td>Step 2</td>\n<td>Hand at target location</td>\n</tr>\n<tr>\n<td>Step 3</td>\n<td>Performs gesture</td>\n</tr>\n</tbody>\n</table>\n<h3>🔹 Flow 2</h3>\n<table>\n<thead>\n<tr>\n<th>Step</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Step 1</td>\n<td>Moves hand to target location</td>\n</tr>\n<tr>\n<td>Step 2</td>\n<td>Hand at target location</td>\n</tr>\n<tr>\n<td>Step 3</td>\n<td>Performs gesture</td>\n</tr>\n</tbody>\n</table>\n<p>According to the CMI task description, participants performed a series of repeated gestures while wearing the Helios device:</p>\n<blockquote>\n  <p><strong>Transition → Pause → Gesture</strong></p>\n</blockquote>\n<h3>❗ Behavior Anomaly</h3>\n<table>\n<thead>\n<tr>\n<th>Behavior Count</th>\n<th>Sequence Count</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>3</td>\n<td>8150</td>\n</tr>\n<tr>\n<td>2</td>\n<td>1 (<code>SEQ_011975</code>)</td>\n</tr>\n</tbody>\n</table>\n<p>🗑️ Since <code>SEQ_011975</code> contains <strong>only 2 behavior phases</strong> (Relaxes and moves hand to target location -&gt; Hand at target location), it was excluded from the dataset.</p>\n<hr>\n<h2>⚖️ Phase Ratio: Transition vs Gesture</h2>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>Value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>✅ Average Gesture Ratio</td>\n<td><strong>44.5%</strong></td>\n</tr>\n<tr>\n<td>✅ Average Transition Ratio</td>\n<td><strong>55.5%</strong></td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F66b51ce3b6622111fdc16308bb283bfc%2FPHASE%20ratio.JPG?generation=1753837076541573&amp;alt=media\" alt=\"Phase Ratio\"></p>\n<p>🔴 Red shaded areas in the plot indicate potential <strong>outliers</strong>:</p>\n<ul>\n<li><strong>Transition Ratio &gt; 95th percentile</strong></li>\n<li><strong>Gesture Ratio &gt; 85th percentile</strong></li>\n</ul>\n<h3>🚨 Potential Outliers</h3>\n<table>\n<thead>\n<tr>\n<th>Condition</th>\n<th>Sequence IDs</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Gesture Ratio &gt; 85</strong></td>\n<td><code>SEQ_011728</code>, <code>SEQ_038837</code>, <code>SEQ_011271</code>, <code>SEQ_065309</code>, <code>SEQ_032114</code>, <code>SEQ_059162</code></td>\n</tr>\n<tr>\n<td><strong>Transition Ratio &gt; 95</strong></td>\n<td><code>SEQ_014205</code>, <code>SEQ_010353</code>, <code>SEQ_060431</code>, <code>SEQ_015261</code>, <code>SEQ_060593</code></td>\n</tr>\n</tbody>\n</table>\n<p>A heavy imbalance in these ratios for specific gesture categories may suggest flaws in the experimental process and could affect model training.</p>\n<hr>\n<h2>🔧 IMU Device</h2>\n<table>\n<thead>\n<tr>\n<th>Sensor Type</th>\n<th>Features</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Accelerometer</td>\n<td><code>acc_x</code>, <code>acc_y</code>, <code>acc_z</code></td>\n</tr>\n<tr>\n<td>Gyroscope &amp; Magnetometer</td>\n<td><code>rot_w</code>, <code>rot_x</code>, <code>rot_y</code>, <code>rot_z</code></td>\n</tr>\n</tbody>\n</table>\n<h3>📉 Null Summary</h3>\n<table>\n<thead>\n<tr>\n<th>Feature</th>\n<th>0% Null</th>\n<th>100% Null</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>acc_x</code> / <code>acc_y</code> / <code>acc_z</code></td>\n<td>8139</td>\n<td>0</td>\n</tr>\n<tr>\n<td><code>rot_*</code> (x, y, z, w)</td>\n<td>8089</td>\n<td>50</td>\n</tr>\n</tbody>\n</table>\n<p>📌 If a sequence has nulls in one <code>rot_*</code> column, <strong>all 4 rot columns</strong> are null.</p>\n<hr>\n<h2>🌡️ Thermopile Sensor (TMP)</h2>\n<table>\n<thead>\n<tr>\n<th>Column</th>\n<th>0% Null</th>\n<th>71.4% Null</th>\n<th>77.6% Null</th>\n<th>100% Null</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>TMP1</td>\n<td>8034</td>\n<td>0</td>\n<td>0</td>\n<td>104</td>\n</tr>\n<tr>\n<td>TMP2</td>\n<td>8021</td>\n<td>1</td>\n<td>0</td>\n<td>116</td>\n</tr>\n<tr>\n<td>TMP3</td>\n<td>8038</td>\n<td>0</td>\n<td>0</td>\n<td>100</td>\n</tr>\n<tr>\n<td>TMP4</td>\n<td>8042</td>\n<td>0</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TMP5</td>\n<td>7655</td>\n<td>0</td>\n<td>271</td>\n<td>482</td>\n</tr>\n</tbody>\n</table>\n<p>✅ <strong>No overlap</strong> between sequences with nulls in TMP and those in IMU.</p>\n<p>📝 TMP nulls appear mostly as full-column nulls (100%).  <br>\nA total of <strong>508 unique sequence_ids</strong> have TMP nulls.</p>\n<h3>⚠️ Pause Phase Stability Check</h3>\n<p>During <strong>Pause</strong>, hands are stationary at the target, so TMP should be <strong>stable</strong>.</p>\n<p>To measure subtle fluctuations:</p>\n<pre><code>np.mean(np.(np.diff(x)))\n</code></pre>\n<h3>📌 Example – <code>SEQ_012617</code></h3>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F86a6db1c3b32c982907c7910020de6f1%2Foutline_1.JPG?generation=1753837095658701&amp;alt=media\" alt=\"TMP Example\"></p>\n<hr>\n<h2>🛰️ Time-of-Flight Sensor (TOF)</h2>\n<table>\n<thead>\n<tr>\n<th>Column</th>\n<th>0% Null</th>\n<th>77.6% Null</th>\n<th>100% Null</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>TOF1</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF2</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF3</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF4</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF5</td>\n<td>7703</td>\n<td>1</td>\n<td>434</td>\n</tr>\n</tbody>\n</table>\n<p>🧪 If <strong>even one</strong> element in a TOF device contains a null value, <strong>all 64 elements</strong> in that sensor are fully null.  <br>\nA total of <strong>435 unique <code>sequence_id</code>s</strong> contain TOF nulls.</p>\n<h3>⚠️ Pause Phase Stability Check (Same as TMP)</h3>\n<p>To detect subtle movement during the <strong>Pause</strong> phase, the following method is used:</p>\n<pre><code>np.mean(np.(np.diff(x)))\n</code></pre>\n<h3>📌 Example – <code>SEQ_047121</code>, <code>SEQ_030299</code></h3>\n<p>These sequences show <strong>unusual variation</strong> during the Pause phase.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F9662b18ab448ae148103d9b388dcc701%2Foutline_2.JPG?generation=1753837109975778&amp;alt=media\" alt=\"TOF Example\"></p>",
  "messages": [
    {
      "id": "3258040",
      "postDate": "07/30/2025 01:09:40",
      "content": "<p>If you want explore EDA as well as data cleaning</p>\n<p>=&gt; <a href=\"https://www.kaggle.com/code/seoyunje/cmi2025-sensor-data-analysis-with-spectrum-eda\" target=\"_blank\">https://www.kaggle.com/code/seoyunje/cmi2025-sensor-data-analysis-with-spectrum-eda</a></p>\n<h2>📌 Behavior Flow</h2>\n<p>Every sequence contains <strong>3 behavior phases</strong>, categorized as follows:</p>\n<h3>🔹 Flow 1</h3>\n<table>\n<thead>\n<tr>\n<th>Step</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Step 1</td>\n<td>Relaxes and moves hand to target location</td>\n</tr>\n<tr>\n<td>Step 2</td>\n<td>Hand at target location</td>\n</tr>\n<tr>\n<td>Step 3</td>\n<td>Performs gesture</td>\n</tr>\n</tbody>\n</table>\n<h3>🔹 Flow 2</h3>\n<table>\n<thead>\n<tr>\n<th>Step</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Step 1</td>\n<td>Moves hand to target location</td>\n</tr>\n<tr>\n<td>Step 2</td>\n<td>Hand at target location</td>\n</tr>\n<tr>\n<td>Step 3</td>\n<td>Performs gesture</td>\n</tr>\n</tbody>\n</table>\n<p>According to the CMI task description, participants performed a series of repeated gestures while wearing the Helios device:</p>\n<blockquote>\n  <p><strong>Transition → Pause → Gesture</strong></p>\n</blockquote>\n<h3>❗ Behavior Anomaly</h3>\n<table>\n<thead>\n<tr>\n<th>Behavior Count</th>\n<th>Sequence Count</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>3</td>\n<td>8150</td>\n</tr>\n<tr>\n<td>2</td>\n<td>1 (<code>SEQ_011975</code>)</td>\n</tr>\n</tbody>\n</table>\n<p>🗑️ Since <code>SEQ_011975</code> contains <strong>only 2 behavior phases</strong> (Relaxes and moves hand to target location -&gt; Hand at target location), it was excluded from the dataset.</p>\n<hr>\n<h2>⚖️ Phase Ratio: Transition vs Gesture</h2>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>Value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>✅ Average Gesture Ratio</td>\n<td><strong>44.5%</strong></td>\n</tr>\n<tr>\n<td>✅ Average Transition Ratio</td>\n<td><strong>55.5%</strong></td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F66b51ce3b6622111fdc16308bb283bfc%2FPHASE%20ratio.JPG?generation=1753837076541573&amp;alt=media\" alt=\"Phase Ratio\"></p>\n<p>🔴 Red shaded areas in the plot indicate potential <strong>outliers</strong>:</p>\n<ul>\n<li><strong>Transition Ratio &gt; 95th percentile</strong></li>\n<li><strong>Gesture Ratio &gt; 85th percentile</strong></li>\n</ul>\n<h3>🚨 Potential Outliers</h3>\n<table>\n<thead>\n<tr>\n<th>Condition</th>\n<th>Sequence IDs</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Gesture Ratio &gt; 85</strong></td>\n<td><code>SEQ_011728</code>, <code>SEQ_038837</code>, <code>SEQ_011271</code>, <code>SEQ_065309</code>, <code>SEQ_032114</code>, <code>SEQ_059162</code></td>\n</tr>\n<tr>\n<td><strong>Transition Ratio &gt; 95</strong></td>\n<td><code>SEQ_014205</code>, <code>SEQ_010353</code>, <code>SEQ_060431</code>, <code>SEQ_015261</code>, <code>SEQ_060593</code></td>\n</tr>\n</tbody>\n</table>\n<p>A heavy imbalance in these ratios for specific gesture categories may suggest flaws in the experimental process and could affect model training.</p>\n<hr>\n<h2>🔧 IMU Device</h2>\n<table>\n<thead>\n<tr>\n<th>Sensor Type</th>\n<th>Features</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Accelerometer</td>\n<td><code>acc_x</code>, <code>acc_y</code>, <code>acc_z</code></td>\n</tr>\n<tr>\n<td>Gyroscope &amp; Magnetometer</td>\n<td><code>rot_w</code>, <code>rot_x</code>, <code>rot_y</code>, <code>rot_z</code></td>\n</tr>\n</tbody>\n</table>\n<h3>📉 Null Summary</h3>\n<table>\n<thead>\n<tr>\n<th>Feature</th>\n<th>0% Null</th>\n<th>100% Null</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>acc_x</code> / <code>acc_y</code> / <code>acc_z</code></td>\n<td>8139</td>\n<td>0</td>\n</tr>\n<tr>\n<td><code>rot_*</code> (x, y, z, w)</td>\n<td>8089</td>\n<td>50</td>\n</tr>\n</tbody>\n</table>\n<p>📌 If a sequence has nulls in one <code>rot_*</code> column, <strong>all 4 rot columns</strong> are null.</p>\n<hr>\n<h2>🌡️ Thermopile Sensor (TMP)</h2>\n<table>\n<thead>\n<tr>\n<th>Column</th>\n<th>0% Null</th>\n<th>71.4% Null</th>\n<th>77.6% Null</th>\n<th>100% Null</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>TMP1</td>\n<td>8034</td>\n<td>0</td>\n<td>0</td>\n<td>104</td>\n</tr>\n<tr>\n<td>TMP2</td>\n<td>8021</td>\n<td>1</td>\n<td>0</td>\n<td>116</td>\n</tr>\n<tr>\n<td>TMP3</td>\n<td>8038</td>\n<td>0</td>\n<td>0</td>\n<td>100</td>\n</tr>\n<tr>\n<td>TMP4</td>\n<td>8042</td>\n<td>0</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TMP5</td>\n<td>7655</td>\n<td>0</td>\n<td>271</td>\n<td>482</td>\n</tr>\n</tbody>\n</table>\n<p>✅ <strong>No overlap</strong> between sequences with nulls in TMP and those in IMU.</p>\n<p>📝 TMP nulls appear mostly as full-column nulls (100%).  <br>\nA total of <strong>508 unique sequence_ids</strong> have TMP nulls.</p>\n<h3>⚠️ Pause Phase Stability Check</h3>\n<p>During <strong>Pause</strong>, hands are stationary at the target, so TMP should be <strong>stable</strong>.</p>\n<p>To measure subtle fluctuations:</p>\n<pre><code>np.mean(np.(np.diff(x)))\n</code></pre>\n<h3>📌 Example – <code>SEQ_012617</code></h3>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F86a6db1c3b32c982907c7910020de6f1%2Foutline_1.JPG?generation=1753837095658701&amp;alt=media\" alt=\"TMP Example\"></p>\n<hr>\n<h2>🛰️ Time-of-Flight Sensor (TOF)</h2>\n<table>\n<thead>\n<tr>\n<th>Column</th>\n<th>0% Null</th>\n<th>77.6% Null</th>\n<th>100% Null</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>TOF1</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF2</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF3</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF4</td>\n<td>8042</td>\n<td>0</td>\n<td>96</td>\n</tr>\n<tr>\n<td>TOF5</td>\n<td>7703</td>\n<td>1</td>\n<td>434</td>\n</tr>\n</tbody>\n</table>\n<p>🧪 If <strong>even one</strong> element in a TOF device contains a null value, <strong>all 64 elements</strong> in that sensor are fully null.  <br>\nA total of <strong>435 unique <code>sequence_id</code>s</strong> contain TOF nulls.</p>\n<h3>⚠️ Pause Phase Stability Check (Same as TMP)</h3>\n<p>To detect subtle movement during the <strong>Pause</strong> phase, the following method is used:</p>\n<pre><code>np.mean(np.(np.diff(x)))\n</code></pre>\n<h3>📌 Example – <code>SEQ_047121</code>, <code>SEQ_030299</code></h3>\n<p>These sequences show <strong>unusual variation</strong> during the Pause phase.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F9662b18ab448ae148103d9b388dcc701%2Foutline_2.JPG?generation=1753837109975778&amp;alt=media\" alt=\"TOF Example\"></p>",
      "rawMarkdown": "If you want explore EDA as well as data cleaning\n\n=> https://www.kaggle.com/code/seoyunje/cmi2025-sensor-data-analysis-with-spectrum-eda\n\n## 📌 Behavior Flow\n\nEvery sequence contains **3 behavior phases**, categorized as follows:\n\n### 🔹 Flow 1\n| Step | Description |\n|------|-------------|\n| Step 1 | Relaxes and moves hand to target location |\n| Step 2 | Hand at target location |\n| Step 3 | Performs gesture |\n\n### 🔹 Flow 2\n| Step | Description |\n|------|-------------|\n| Step 1 | Moves hand to target location |\n| Step 2 | Hand at target location |\n| Step 3 | Performs gesture |\n\nAccording to the CMI task description, participants performed a series of repeated gestures while wearing the Helios device:\n\n> **Transition → Pause → Gesture**\n\n### ❗ Behavior Anomaly\n\n| Behavior Count | Sequence Count |\n|----------------|----------------|\n| 3              | 8150           |\n| 2              | 1 (`SEQ_011975`) |\n\n🗑️ Since `SEQ_011975` contains **only 2 behavior phases** (Relaxes and moves hand to target location -> Hand at target location), it was excluded from the dataset.\n\n---\n\n## ⚖️ Phase Ratio: Transition vs Gesture\n\n| Metric | Value |\n|--------|-------|\n| ✅ Average Gesture Ratio | **44.5%** |\n| ✅ Average Transition Ratio | **55.5%** |\n\n![Phase Ratio](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F66b51ce3b6622111fdc16308bb283bfc%2FPHASE%20ratio.JPG?generation=1753837076541573&alt=media)\n\n🔴 Red shaded areas in the plot indicate potential **outliers**:\n- **Transition Ratio > 95th percentile**\n- **Gesture Ratio > 85th percentile**\n\n### 🚨 Potential Outliers\n\n| Condition | Sequence IDs |\n|-----------|--------------|\n| **Gesture Ratio > 85** | `SEQ_011728`, `SEQ_038837`, `SEQ_011271`, `SEQ_065309`, `SEQ_032114`, `SEQ_059162` |\n| **Transition Ratio > 95** | `SEQ_014205`, `SEQ_010353`, `SEQ_060431`, `SEQ_015261`, `SEQ_060593` |\n\nA heavy imbalance in these ratios for specific gesture categories may suggest flaws in the experimental process and could affect model training.\n\n---\n\n## 🔧 IMU Device\n\n| Sensor Type             | Features              |\n|-------------------------|------------------------|\n| Accelerometer           | `acc_x`, `acc_y`, `acc_z` |\n| Gyroscope & Magnetometer| `rot_w`, `rot_x`, `rot_y`, `rot_z` |\n\n### 📉 Null Summary\n\n| Feature | 0% Null | 100% Null |\n|---------|--------|------------|\n| `acc_x` / `acc_y` / `acc_z` | 8139 | 0 |\n| `rot_*` (x, y, z, w) | 8089 | 50 |\n\n📌 If a sequence has nulls in one `rot_*` column, **all 4 rot columns** are null.\n\n---\n\n## 🌡️ Thermopile Sensor (TMP)\n\n| Column | 0% Null | 71.4% Null | 77.6% Null | 100% Null |\n|--------|---------|------------|------------|------------|\n| TMP1   | 8034    | 0          | 0          | 104        |\n| TMP2   | 8021    | 1          | 0          | 116        |\n| TMP3   | 8038    | 0          | 0          | 100        |\n| TMP4   | 8042    | 0          | 0          | 96         |\n| TMP5   | 7655    | 0          | 271        | 482        |\n\n✅ **No overlap** between sequences with nulls in TMP and those in IMU.\n\n📝 TMP nulls appear mostly as full-column nulls (100%).  \nA total of **508 unique sequence_ids** have TMP nulls.\n\n### ⚠️ Pause Phase Stability Check\n\nDuring **Pause**, hands are stationary at the target, so TMP should be **stable**.\n\nTo measure subtle fluctuations:\n```python\nnp.mean(np.abs(np.diff(x)))\n```\n\n### 📌 Example – `SEQ_012617`\n![TMP Example](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F86a6db1c3b32c982907c7910020de6f1%2Foutline_1.JPG?generation=1753837095658701&alt=media)\n\n---\n\n## 🛰️ Time-of-Flight Sensor (TOF)\n\n| Column | 0% Null | 77.6% Null | 100% Null |\n|--------|---------|-------------|------------|\n| TOF1   | 8042    | 0           | 96         |\n| TOF2   | 8042    | 0           | 96         |\n| TOF3   | 8042    | 0           | 96         |\n| TOF4   | 8042    | 0           | 96         |\n| TOF5   | 7703    | 1           | 434        |\n\n🧪 If **even one** element in a TOF device contains a null value, **all 64 elements** in that sensor are fully null.  \nA total of **435 unique `sequence_id`s** contain TOF nulls.\n\n### ⚠️ Pause Phase Stability Check (Same as TMP)\n\nTo detect subtle movement during the **Pause** phase, the following method is used:\n\n```python\nnp.mean(np.abs(np.diff(x)))\n```\n\n### 📌 Example – `SEQ_047121`, `SEQ_030299`\nThese sequences show **unusual variation** during the Pause phase.\n\n![TOF Example](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F9662b18ab448ae148103d9b388dcc701%2Foutline_2.JPG?generation=1753837109975778&alt=media)",
      "votes": null
    },
    {
      "id": "3258223",
      "postDate": "07/30/2025 11:12:11",
      "content": "<p>awesome!</p>",
      "rawMarkdown": "awesome!",
      "votes": null
    },
    {
      "id": "3258699",
      "postDate": "07/31/2025 04:31:19",
      "content": "<p>Excellent observation! Just a small typo:<br>\nSEQ_011975 only includes 2 behavior phases:</p>\n<p>Relaxes and moves hand to target location: 79<br>\nHand at target location: 5<br>\nwith the gesture = \"Text on phone\" occurring for all 84 steps.</p>\n<p>I agree that this data can be excluded, but it also suggests that \"text on phone\" is typically a brief action, which might be why it isn’t consistently captured in the behavior phase.</p>",
      "rawMarkdown": "Excellent observation! Just a small typo:\nSEQ_011975 only includes 2 behavior phases:\n\nRelaxes and moves hand to target location: 79\nHand at target location: 5\nwith the gesture = \"Text on phone\" occurring for all 84 steps.\n\nI agree that this data can be excluded, but it also suggests that \"text on phone\" is typically a brief action, which might be why it isn’t consistently captured in the behavior phase.",
      "votes": null
    },
    {
      "id": "3258767",
      "postDate": "07/31/2025 07:46:31",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/khushikyad001\" target=\"_blank\">@khushikyad001</a> </p>",
      "rawMarkdown": "Thank you @khushikyad001",
      "votes": null
    },
    {
      "id": "3258768",
      "postDate": "07/31/2025 07:54:39",
      "content": "<p>Oops, It's my mistake. SEQ_011975 has two behavior[Relaxes and moves hand to target location -&gt; Hand at target location]</p>\n<p><strong>FLOW 1</strong>: <code>Relaxes and moves hand to target location -&gt; Hand at target location -&gt; Performs gesture</code><br>\n<strong>FLOW 2</strong>: <code>Moves hand to target location -&gt; Hand at target location -&gt; Performs gesture</code></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2Fffb1e5c1bcfebd5058617d1022f88fd3%2Ftext%20on%20phone.JPG?generation=1753948476369747&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Oops, It's my mistake. SEQ_011975 has two behavior[Relaxes and moves hand to target location -> Hand at target location]\n\n**FLOW 1**: `Relaxes and moves hand to target location -> Hand at target location -> Performs gesture`\n**FLOW 2**: `Moves hand to target location -> Hand at target location -> Performs gesture`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2Fffb1e5c1bcfebd5058617d1022f88fd3%2Ftext%20on%20phone.JPG?generation=1753948476369747&alt=media)",
      "votes": null
    },
    {
      "id": "3258771",
      "postDate": "07/31/2025 07:58:32",
      "content": "<p>Considering that a large proportion of \"text on phone\" occurs during the perform gesture phase, it is more likely a measurement error than a case of the action being too brief to capture.</p>",
      "rawMarkdown": "Considering that a large proportion of \"text on phone\" occurs during the perform gesture phase, it is more likely a measurement error than a case of the action being too brief to capture.",
      "votes": null
    },
    {
      "id": "3261840",
      "postDate": "08/02/2025 10:26:33",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3258223,
      "author_name": "khushikyad001",
      "author_url": "",
      "post_date": "07/30/2025 11:12:11",
      "content": "<p>awesome!</p>",
      "votes": null,
      "replies": [
        {
          "id": 3258767,
          "author_name": "seoyunje",
          "author_url": "",
          "post_date": "07/31/2025 07:46:31",
          "content": "<p>Thank you <a href=\"https://www.kaggle.com/khushikyad001\" target=\"_blank\">@khushikyad001</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3258699,
      "author_name": "heyyousum",
      "author_url": "",
      "post_date": "07/31/2025 04:31:19",
      "content": "<p>Excellent observation! Just a small typo:<br>\nSEQ_011975 only includes 2 behavior phases:</p>\n<p>Relaxes and moves hand to target location: 79<br>\nHand at target location: 5<br>\nwith the gesture = \"Text on phone\" occurring for all 84 steps.</p>\n<p>I agree that this data can be excluded, but it also suggests that \"text on phone\" is typically a brief action, which might be why it isn’t consistently captured in the behavior phase.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3258768,
          "author_name": "seoyunje",
          "author_url": "",
          "post_date": "07/31/2025 07:54:39",
          "content": "<p>Oops, It's my mistake. SEQ_011975 has two behavior[Relaxes and moves hand to target location -&gt; Hand at target location]</p>\n<p><strong>FLOW 1</strong>: <code>Relaxes and moves hand to target location -&gt; Hand at target location -&gt; Performs gesture</code><br>\n<strong>FLOW 2</strong>: <code>Moves hand to target location -&gt; Hand at target location -&gt; Performs gesture</code></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2Fffb1e5c1bcfebd5058617d1022f88fd3%2Ftext%20on%20phone.JPG?generation=1753948476369747&amp;alt=media\" alt=\"\"></p>",
          "votes": null,
          "replies": [
            {
              "id": 3258771,
              "author_name": "seoyunje",
              "author_url": "",
              "post_date": "07/31/2025 07:58:32",
              "content": "<p>Considering that a large proportion of \"text on phone\" occurs during the perform gesture phase, it is more likely a measurement error than a case of the action being too brief to capture.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3261840,
      "author_name": "volodymyrpivoshenko",
      "author_url": "",
      "post_date": "08/02/2025 10:26:33",
      "content": "<p>Thanks for sharing!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3258040": "If you want explore EDA as well as data cleaning\n\n=> https://www.kaggle.com/code/seoyunje/cmi2025-sensor-data-analysis-with-spectrum-eda\n\n## 📌 Behavior Flow\n\nEvery sequence contains **3 behavior phases**, categorized as follows:\n\n### 🔹 Flow 1\n| Step | Description |\n|------|-------------|\n| Step 1 | Relaxes and moves hand to target location |\n| Step 2 | Hand at target location |\n| Step 3 | Performs gesture |\n\n### 🔹 Flow 2\n| Step | Description |\n|------|-------------|\n| Step 1 | Moves hand to target location |\n| Step 2 | Hand at target location |\n| Step 3 | Performs gesture |\n\nAccording to the CMI task description, participants performed a series of repeated gestures while wearing the Helios device:\n\n> **Transition → Pause → Gesture**\n\n### ❗ Behavior Anomaly\n\n| Behavior Count | Sequence Count |\n|----------------|----------------|\n| 3              | 8150           |\n| 2              | 1 (`SEQ_011975`) |\n\n🗑️ Since `SEQ_011975` contains **only 2 behavior phases** (Relaxes and moves hand to target location -> Hand at target location), it was excluded from the dataset.\n\n---\n\n## ⚖️ Phase Ratio: Transition vs Gesture\n\n| Metric | Value |\n|--------|-------|\n| ✅ Average Gesture Ratio | **44.5%** |\n| ✅ Average Transition Ratio | **55.5%** |\n\n![Phase Ratio](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F66b51ce3b6622111fdc16308bb283bfc%2FPHASE%20ratio.JPG?generation=1753837076541573&alt=media)\n\n🔴 Red shaded areas in the plot indicate potential **outliers**:\n- **Transition Ratio > 95th percentile**\n- **Gesture Ratio > 85th percentile**\n\n### 🚨 Potential Outliers\n\n| Condition | Sequence IDs |\n|-----------|--------------|\n| **Gesture Ratio > 85** | `SEQ_011728`, `SEQ_038837`, `SEQ_011271`, `SEQ_065309`, `SEQ_032114`, `SEQ_059162` |\n| **Transition Ratio > 95** | `SEQ_014205`, `SEQ_010353`, `SEQ_060431`, `SEQ_015261`, `SEQ_060593` |\n\nA heavy imbalance in these ratios for specific gesture categories may suggest flaws in the experimental process and could affect model training.\n\n---\n\n## 🔧 IMU Device\n\n| Sensor Type             | Features              |\n|-------------------------|------------------------|\n| Accelerometer           | `acc_x`, `acc_y`, `acc_z` |\n| Gyroscope & Magnetometer| `rot_w`, `rot_x`, `rot_y`, `rot_z` |\n\n### 📉 Null Summary\n\n| Feature | 0% Null | 100% Null |\n|---------|--------|------------|\n| `acc_x` / `acc_y` / `acc_z` | 8139 | 0 |\n| `rot_*` (x, y, z, w) | 8089 | 50 |\n\n📌 If a sequence has nulls in one `rot_*` column, **all 4 rot columns** are null.\n\n---\n\n## 🌡️ Thermopile Sensor (TMP)\n\n| Column | 0% Null | 71.4% Null | 77.6% Null | 100% Null |\n|--------|---------|------------|------------|------------|\n| TMP1   | 8034    | 0          | 0          | 104        |\n| TMP2   | 8021    | 1          | 0          | 116        |\n| TMP3   | 8038    | 0          | 0          | 100        |\n| TMP4   | 8042    | 0          | 0          | 96         |\n| TMP5   | 7655    | 0          | 271        | 482        |\n\n✅ **No overlap** between sequences with nulls in TMP and those in IMU.\n\n📝 TMP nulls appear mostly as full-column nulls (100%).  \nA total of **508 unique sequence_ids** have TMP nulls.\n\n### ⚠️ Pause Phase Stability Check\n\nDuring **Pause**, hands are stationary at the target, so TMP should be **stable**.\n\nTo measure subtle fluctuations:\n```python\nnp.mean(np.abs(np.diff(x)))\n```\n\n### 📌 Example – `SEQ_012617`\n![TMP Example](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F86a6db1c3b32c982907c7910020de6f1%2Foutline_1.JPG?generation=1753837095658701&alt=media)\n\n---\n\n## 🛰️ Time-of-Flight Sensor (TOF)\n\n| Column | 0% Null | 77.6% Null | 100% Null |\n|--------|---------|-------------|------------|\n| TOF1   | 8042    | 0           | 96         |\n| TOF2   | 8042    | 0           | 96         |\n| TOF3   | 8042    | 0           | 96         |\n| TOF4   | 8042    | 0           | 96         |\n| TOF5   | 7703    | 1           | 434        |\n\n🧪 If **even one** element in a TOF device contains a null value, **all 64 elements** in that sensor are fully null.  \nA total of **435 unique `sequence_id`s** contain TOF nulls.\n\n### ⚠️ Pause Phase Stability Check (Same as TMP)\n\nTo detect subtle movement during the **Pause** phase, the following method is used:\n\n```python\nnp.mean(np.abs(np.diff(x)))\n```\n\n### 📌 Example – `SEQ_047121`, `SEQ_030299`\nThese sequences show **unusual variation** during the Pause phase.\n\n![TOF Example](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2F9662b18ab448ae148103d9b388dcc701%2Foutline_2.JPG?generation=1753837109975778&alt=media)",
    "3258223": "awesome!",
    "3258699": "Excellent observation! Just a small typo:\nSEQ_011975 only includes 2 behavior phases:\n\nRelaxes and moves hand to target location: 79\nHand at target location: 5\nwith the gesture = \"Text on phone\" occurring for all 84 steps.\n\nI agree that this data can be excluded, but it also suggests that \"text on phone\" is typically a brief action, which might be why it isn’t consistently captured in the behavior phase.",
    "3258767": "Thank you @khushikyad001",
    "3258768": "Oops, It's my mistake. SEQ_011975 has two behavior[Relaxes and moves hand to target location -> Hand at target location]\n\n**FLOW 1**: `Relaxes and moves hand to target location -> Hand at target location -> Performs gesture`\n**FLOW 2**: `Moves hand to target location -> Hand at target location -> Performs gesture`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16438831%2Fffb1e5c1bcfebd5058617d1022f88fd3%2Ftext%20on%20phone.JPG?generation=1753948476369747&alt=media)",
    "3258771": "Considering that a large proportion of \"text on phone\" occurs during the perform gesture phase, it is more likely a measurement error than a case of the action being too brief to capture.",
    "3261840": "Thanks for sharing!"
  },
  "source": "meta"
}