{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 📘 Table of Contents\n1. [Introduction](#introduction)\n2. [Dataset Overview](#dataset-overview)\n3. [Missing Values](#missing-values)\n4. [Target Distribution](#target-distribution)\n5. [Sensor Signal Snapshots](#sensor-signal-snapshots)\n6. [Orientation & Behavior](#orientation--behavior)\n7. [Demographic Insights](#demographic-insights)\n8. [Feature Engineering Preview](#feature-engineering-preview)","metadata":{}},{"cell_type":"markdown","source":"## 1. 🧠 Introduction\n\nThis notebook presents an exploratory data analysis (EDA) of the sensor data provided in the [Detect Behavior with Sensor Data](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data) competition by the Child Mind Institute.\n\n### 🌟 Objective: \nBuild a model that classifies **body-focused repetitive behaviors (BFRBs)** (e.g., hair pulling, skin picking) versus **non-BFRB everyday gestures** (e.g., drinking water, waving).\n\n🎯 These insights could inform the development of wearable devices that detect harmful behavior patterns and support real-time interventions for mental health.\n\n### 📎 Why this matters:\n- BFRBs often co-occur with anxiety or OCD.\n- Accurate gesture detection enables smarter, real-time interventions.\n\n### 📂 Dataset Summary:\n\n- **IMU sensors**: acceleration + rotation (motion)\n- **Thermopiles**: temperature\n- **ToF sensors**: distance to objects\n- Temporal gesture sequences (e.g., \"Transition\", \"Pause\", \"Gesture\")\n- Participant metadata (age, height, handedness, etc.)\n\n### EDA Goals:\n\n- Understand gesture types and their frequency\n\n- Explore missing values and signal quality per sensor\n\n- Visualize key sensor patterns across phases and gestures\n\n- Investigate how participant demographics might influence movements\n\n","metadata":{}},{"cell_type":"markdown","source":"## 2. 📦 Dataset Overview \nBelow is a preview of the first few rows of the training data set.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nsns.set(style=\"whitegrid\")\n%matplotlib inline","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:43:54.524311Z","iopub.execute_input":"2025-07-12T10:43:54.524649Z","iopub.status.idle":"2025-07-12T10:43:54.530629Z","shell.execute_reply.started":"2025-07-12T10:43:54.524628Z","shell.execute_reply":"2025-07-12T10:43:54.529787Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n\nWe have the following data files available in this competition:\n\n- **train.csv**: Sensor data with gesture labels for training.\n- **test.csv**: Sensor data without labels for evaluation.\n- **train_demographics.csv**: Participant demographics for training subjects.\n- **test_demographics.csv**: Participant demographics for test subjects.","metadata":{}},{"cell_type":"code","source":"# File paths\ndata_path = \"/kaggle/input/cmi-detect-behavior-with-sensor-data/\"\n\n# Load datasets\ntrain_df      = pd.read_csv(data_path + \"train.csv\")\ntest_df       = pd.read_csv(data_path + \"test.csv\")\ntrain_dem_df  = pd.read_csv(data_path + \"train_demographics.csv\")\ntest_dem_df   = pd.read_csv(data_path + \"test_demographics.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:43:56.943183Z","iopub.execute_input":"2025-07-12T10:43:56.943508Z","iopub.status.idle":"2025-07-12T10:44:09.267852Z","shell.execute_reply.started":"2025-07-12T10:43:56.943488Z","shell.execute_reply":"2025-07-12T10:44:09.266262Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Display dataset shapes in a DataFrame \nshapes_df = pd.DataFrame({\n    \"Dataset\": [\"Train Sensor Data\", \"Test Sensor Data\", \"Train Demographics\", \"Test Demographics\"],\n    \"Rows\": [train_df.shape[0], test_df.shape[0], train_dem_df.shape[0], test_dem_df.shape[0]],\n    \"Columns\": [train_df.shape[1], test_df.shape[1], train_dem_df.shape[1], test_dem_df.shape[1]]\n})\n\nshapes_df.style.set_caption(\"📦 Dataset Shapes Overview\").background_gradient(cmap='Blues', subset=[\"Rows\", \"Columns\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:09.269310Z","iopub.execute_input":"2025-07-12T10:44:09.269686Z","iopub.status.idle":"2025-07-12T10:44:09.283185Z","shell.execute_reply.started":"2025-07-12T10:44:09.269663Z","shell.execute_reply":"2025-07-12T10:44:09.282382Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Preview of `train.csv` and `train_demographics.csv`\n\nBelow is a preview of the first few rows of the training","metadata":{}},{"cell_type":"code","source":"print(\"Sample rows from train.csv:\")\ndisplay(train_df.head(2))\n\nprint(\"\\nSample rows from train_demographics.csv:\")\ndisplay(train_dem_df.head(2))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:09.284129Z","iopub.execute_input":"2025-07-12T10:44:09.284390Z","iopub.status.idle":"2025-07-12T10:44:09.319999Z","shell.execute_reply.started":"2025-07-12T10:44:09.284373Z","shell.execute_reply":"2025-07-12T10:44:09.319274Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- 📌 **81 subjects**, performing **17 unique gestures**\n- 📈 Over **2,400 sequences** in training\n- Demographic data includes: age, sex, handedness, height, forearm length","metadata":{}},{"cell_type":"markdown","source":"### Preview of `test.csv` and `test_demographics.csv`\n\nBelow is a preview of the first few rows of the training","metadata":{}},{"cell_type":"code","source":"print(\"Sample rows from test.csv:\")\ndisplay(test_df.head(2))\n\nprint(\"\\nSample rows from test_demographics.csv:\")\ndisplay(test_dem_df.head(2))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:09.321207Z","iopub.execute_input":"2025-07-12T10:44:09.321393Z","iopub.status.idle":"2025-07-12T10:44:09.349074Z","shell.execute_reply.started":"2025-07-12T10:44:09.321374Z","shell.execute_reply":"2025-07-12T10:44:09.348390Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 Summary Statistics","metadata":{}},{"cell_type":"code","source":"# Select numeric columns only\nnumeric_cols = train_df.select_dtypes(include=[np.number]).columns\n\n# Generate summary statistics\nsummary_stats = train_df[numeric_cols].describe().transpose()\nsummary_stats = summary_stats[['mean', 'std', 'min', '25%', '50%', '75%', 'max']]\nsummary_stats","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Unique Identifiers","metadata":{}},{"cell_type":"code","source":"unique_sequences = train_df['sequence_id'].nunique()\nunique_gestures = train_df['gesture'].nunique()\nunique_subjects = train_df['subject'].nunique()\n\nsummary_df = pd.DataFrame({\n    \"Metric\": [\"Unique sequences\", \"Unique gestures\", \"Unique subjects\"],\n    \"Count\": [unique_sequences, unique_gestures, unique_subjects]\n})\n\nsummary_df.style.set_caption(\"🔢 Unique Values Summary\").background_gradient(cmap='Greens', subset=[\"Count\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:09.371697Z","iopub.execute_input":"2025-07-12T10:44:09.371957Z","iopub.status.idle":"2025-07-12T10:44:09.465355Z","shell.execute_reply.started":"2025-07-12T10:44:09.371935Z","shell.execute_reply":"2025-07-12T10:44:09.464469Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Notes**:\n- The train sensor dataset has over half a million rows with 341 columns capturing time series and sensor details.\n\n- The test sensor dataset is smaller and contains 336 columns (no gesture labels).\n\n- The demographics datasets contain participant info such as age, sex, handedness, height, and arm measurements.\n\n- There are 81 unique subjects in training data performing 17 different gestures across 2,430 unique sequences.\n\n- This variety will help us explore gesture patterns and sensor signals across different people and actions.","metadata":{}},{"cell_type":"markdown","source":"## 3. 📉 Missing Values\n\n### 🎯 Objective:\n\n- Understand the completeness of sensor data across different sensor groups:\n  - **IMU sensors** (acceleration + rotation)\n  - **Thermopile sensors** (`thm_1` to `thm_5`)\n  - **Time-of-Flight (ToF) sensors** (`tof_1_v0` to `tof_5_v63`)\n- Detect missingness patterns and special cases (e.g., `-1` values for ToF)\n","metadata":{}},{"cell_type":"markdown","source":"**Step 1:** Group columns by sensor type","metadata":{}},{"cell_type":"code","source":"imu_cols = [col for col in train_df.columns if col.startswith('acc_') or col.startswith('rot_')]\nthm_cols = [col for col in train_df.columns if col.startswith('thm_')]\ntof_cols = [col for col in train_df.columns if col.startswith('tof_')]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:11.188071Z","iopub.execute_input":"2025-07-12T10:44:11.188315Z","iopub.status.idle":"2025-07-12T10:44:11.193168Z","shell.execute_reply.started":"2025-07-12T10:44:11.188300Z","shell.execute_reply":"2025-07-12T10:44:11.192170Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Step 2:** Calculate missing value fractions for each group","metadata":{}},{"cell_type":"code","source":"# Fraction of missing values per sensor group\nimu_missing_frac = train_df[imu_cols].isnull().mean().mean()\nthm_missing_frac = train_df[thm_cols].isnull().mean().mean()\ntof_missing_frac = train_df[tof_cols].isnull().mean().mean()\n\nmissing_summary = pd.DataFrame({\n    'Sensor Group': ['IMU (acc + rot)', 'Thermopiles (thm_1 to thm_5)', 'ToF sensors (tof_1_v* to tof_5_v*)'],\n    'Fraction Missing': [imu_missing_frac, thm_missing_frac, tof_missing_frac]\n})\n\nmissing_summary.style.format({\"Fraction Missing\": \"{:.4f}\"}).set_caption(\"Missing Values Fraction by Sensor Group\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:13.262631Z","iopub.execute_input":"2025-07-12T10:44:13.262918Z","iopub.status.idle":"2025-07-12T10:44:13.746195Z","shell.execute_reply.started":"2025-07-12T10:44:13.262899Z","shell.execute_reply":"2025-07-12T10:44:13.745265Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Step 3:** Show % of -1 values in ToF sensors (no reflection)","metadata":{}},{"cell_type":"code","source":"# Calculate % of -1 per ToF sensor (grouped by tof_1, tof_2, ..., tof_5)\n\ntof_sensor_ids = [f\"tof_{i}\" for i in range(1,6)]\ntof_neg1_percent = {}\n\nfor sensor_id in tof_sensor_ids:\n    sensor_cols = [col for col in tof_cols if col.startswith(sensor_id)]\n    # Flatten data for all these columns and calculate percentage of -1\n    neg1_count = (train_df[sensor_cols] == -1).sum().sum()\n    total_count = train_df[sensor_cols].size\n    tof_neg1_percent[sensor_id] = neg1_count / total_count\n\ntof_neg1_df = pd.DataFrame.from_dict(tof_neg1_percent, orient='index', columns=['% No Reflection (-1)'])\ntof_neg1_df.style.format({\"% No Reflection (-1)\": \"{:.4%}\"}).set_caption(\"Percentage of -1 (No Reflection) Values in ToF Sensors\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:16.969164Z","iopub.execute_input":"2025-07-12T10:44:16.969872Z","iopub.status.idle":"2025-07-12T10:44:17.627493Z","shell.execute_reply.started":"2025-07-12T10:44:16.969843Z","shell.execute_reply":"2025-07-12T10:44:17.626678Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tof_sensor_ids = [f\"tof_{i}\" for i in range(1,6)]\ntof_neg1_percent = {}\ntof_nan_percent = {}\n\nfor sensor_id in tof_sensor_ids:\n    sensor_cols = [col for col in tof_cols if col.startswith(sensor_id)]\n    \n    total_count = train_df[sensor_cols].size\n    neg1_count = (train_df[sensor_cols] == -1).sum().sum()\n    nan_count = train_df[sensor_cols].isna().sum().sum()\n    \n    tof_neg1_percent[sensor_id] = neg1_count / total_count * 100\n    tof_nan_percent[sensor_id] = nan_count / total_count * 100\n\n# Combine into a single DataFrame\ntof_quality_df = pd.DataFrame({\n    'No Reflection (-1) %': tof_neg1_percent,\n    'Missing (NaN) %': tof_nan_percent\n}).T\n\n# Plot\nx = np.arange(len(tof_sensor_ids))\nwidth = 0.35\n\nfig, ax = plt.subplots(figsize=(12, 5))\nbars1 = ax.bar(x - width/2, list(tof_nan_percent.values()), width, label='% NaN', color='skyblue')\nbars2 = ax.bar(x + width/2, list(tof_neg1_percent.values()), width, label='% -1', color='salmon')\n\n# Labels and formatting\nax.set_ylabel('Percentage')\nax.set_title('ToF Sensor Groups: Missing (NaN) vs. No Reflection (-1)')\nax.set_xticks(x)\nax.set_xticklabels(tof_sensor_ids)\nax.legend()\nax.grid(True, axis='y', linestyle='--', alpha=0.6)\n\n# Annotate bars\nfor bar in bars1 + bars2:\n    height = bar.get_height()\n    ax.annotate(f'{height:.1f}%',\n                xy=(bar.get_x() + bar.get_width() / 2, height),\n                xytext=(0, 3),\n                textcoords=\"offset points\",\n                ha='center', va='bottom', fontsize=9)\n\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:18.890456Z","iopub.execute_input":"2025-07-12T10:44:18.890733Z","iopub.status.idle":"2025-07-12T10:44:20.161332Z","shell.execute_reply.started":"2025-07-12T10:44:18.890713Z","shell.execute_reply":"2025-07-12T10:44:20.160522Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔍 Observations & Insights\n\n- **IMU sensors** (acceleration + rotation) are the most reliable, with only **~0.37% missing values**. This makes them a consistent and trustworthy source of motion information throughout sequences.\n\n- **Thermopile sensors** have **~2.1% missing data**, suggesting occasional gaps. These may stem from environmental conditions or sensor orientation (e.g., not pointed toward the body), and may require interpolation or smoothing.\n\n- **ToF sensors** show **~1.9% missing data**, but the more significant issue is the high proportion of **`-1` values (indicating no IR reflection)**:\n  - Over **50% of readings** across all ToF sensors are `-1`, with `tof_2` being the most affected at ~**62%**.\n  - These `-1` values likely occur when no object (e.g., hands) is within reflective range or when surfaces deflect/absorb IR poorly.\n\n","metadata":{}},{"cell_type":"markdown","source":"### 💡 Modeling Considerations:\n\n- Handle both **`NaN`** and **`-1`** values in ToF sensor data carefully.\n- Use **IMU sensors as primary input** for motion-based features as they're clean and consistent.\n- For ToF:\n  - Investigate if **`-1` values correlate with specific gestures** or behavior phases.\n  - Use derived features (e.g., reflection presence rate) as potential signals.","metadata":{}},{"cell_type":"markdown","source":"### 4. 📊 Target Distribution\n\nWe'll explore the following aspects of the target labels:\n\n- **Gesture distribution:** Understand the overall spread of gesture labels in the dataset, including how frequently each occurs.\n\n- **Proportion of target vs. non-target sequences:** Examine how balanced the dataset is between body-focused repetitive behaviors (BFRBs) and non-BFRBs.\n\n- **Gesture count per sequence type:** Analyze how gesture types are distributed within each sequence class (target vs. non-target), helping identify potential biases or signal strengths.","metadata":{}},{"cell_type":"markdown","source":" **Step 1:** Gesture Distribution (Colored by Target vs. Non-Target)","metadata":{}},{"cell_type":"code","source":"# Count gestures grouped by sequence type (Target or Non-Target)\ngesture_counts = train_df.groupby(['gesture', 'sequence_type'])['sequence_id'].nunique().reset_index()\ngesture_counts.columns = ['Gesture', 'Sequence Type', 'Unique Sequence Count']\n# Define custom palette matching your colors\ncustom_palette = {\n    'Target': '#ff9999',      # soft red\n    'Non-Target': '#66b3ff'   # soft blue\n}\n# Sort by total frequency for consistent ordering\ngesture_order = gesture_counts.groupby('Gesture')['Unique Sequence Count'].sum().sort_values(ascending=False).index\nplt.figure(figsize=(12, 6))\nsns.barplot(data=gesture_counts, x='Gesture', y='Unique Sequence Count', hue='Sequence Type', order=gesture_order, palette=custom_palette)\nplt.xticks(rotation=45, ha='right')\nplt.title('Unique Gesture Sequence Counts by Type')\nplt.xlabel('Gesture')\nplt.ylabel('Number of Unique Sequences')\nplt.legend(title='Sequence Type')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:22.740448Z","iopub.execute_input":"2025-07-12T10:44:22.740719Z","iopub.status.idle":"2025-07-12T10:44:23.274000Z","shell.execute_reply.started":"2025-07-12T10:44:22.740700Z","shell.execute_reply":"2025-07-12T10:44:23.273331Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Step 2:** Overall Target vs. Non-Target Distribution","metadata":{}},{"cell_type":"code","source":"# Count total number of sequences by type\nsequence_type_counts = train_df[['sequence_id', 'sequence_type']].drop_duplicates()['sequence_type'].value_counts()\n\n# Plot\nplt.figure(figsize=(6, 4))\nsequence_type_counts.plot(kind='pie', autopct='%1.1f%%', startangle=140, colors=['#ff9999','#66b3ff'])\nplt.title('Proportion of Target vs. Non-Target Sequences')\nplt.ylabel('')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:25.324102Z","iopub.execute_input":"2025-07-12T10:44:25.324684Z","iopub.status.idle":"2025-07-12T10:44:25.458840Z","shell.execute_reply.started":"2025-07-12T10:44:25.324661Z","shell.execute_reply":"2025-07-12T10:44:25.458083Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Step 3:** Number of Unique Sequences Per Gesture","metadata":{}},{"cell_type":"code","source":"# Count unique sequences per gesture only\ngesture_seq_counts = train_df.groupby('gesture')['sequence_id'].nunique().sort_values(ascending=False)\n\n# Display as styled DataFrame\ngesture_seq_counts_df = gesture_seq_counts.reset_index()\ngesture_seq_counts_df.columns = ['Gesture', 'Unique Sequences']\ngesture_seq_counts_df.style.set_caption(\" Unique Sequences per Gesture\").background_gradient(cmap='Purples', subset=[\"Unique Sequences\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:30.985225Z","iopub.execute_input":"2025-07-12T10:44:30.985570Z","iopub.status.idle":"2025-07-12T10:44:31.057466Z","shell.execute_reply.started":"2025-07-12T10:44:30.985548Z","shell.execute_reply":"2025-07-12T10:44:31.056468Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🎯 Target Distribution Insights\n\n- The dataset includes **18 unique gestures**, split between **Target** (BFRB) and **Non-Target** (everyday gestures).\n- The distribution is imbalanced: some gestures (e.g., *Text on phone*, *Write name in air*) are far more frequent than others.\n- Around **62.7%** of sequences are labeled as **Target**, while **37.3%** are **Non-Target** (based on the pie chart).\n- This imbalance should be considered when modeling — for instance, by applying stratified sampling or using class weights.\n\nWe'll now proceed to visually explore the sensor signals for representative sequences.","metadata":{}},{"cell_type":"markdown","source":"## 5. 📡 Sensor Signal Snapshots \nLet's explore how the sensor signals behave for different gesture sequences.\n\n### Select Example Sequences","metadata":{}},{"cell_type":"code","source":"# Select a few example sequence_ids for visualization (stratified by gesture type)\nnp.random.seed(42)\nexample_sequences = []\nfor gesture_type in ['Target', 'Non-Target']:\n    seqs = train_df[train_df['sequence_type'] == gesture_type]['sequence_id'].unique()\n    example_sequences.extend(np.random.choice(seqs, size=2, replace=False))\nexample_sequences","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:38.854006Z","iopub.execute_input":"2025-07-12T10:44:38.854288Z","iopub.status.idle":"2025-07-12T10:44:39.473960Z","shell.execute_reply.started":"2025-07-12T10:44:38.854267Z","shell.execute_reply":"2025-07-12T10:44:39.472835Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Plot IMU Acceleration Over Time","metadata":{}},{"cell_type":"code","source":"for seq_id in example_sequences:\n    seq_data = train_df[train_df['sequence_id'] == seq_id]\n    plt.figure(figsize=(12, 4))\n    plt.plot(seq_data['sequence_counter'], seq_data['acc_x'], label='acc_x')\n    plt.plot(seq_data['sequence_counter'], seq_data['acc_y'], label='acc_y')\n    plt.plot(seq_data['sequence_counter'], seq_data['acc_z'], label='acc_z')\n    plt.title(f'IMU Acceleration for Sequence {seq_id} ({seq_data[\"gesture\"].iloc[0]})')\n    plt.xlabel('Time Step')\n    plt.ylabel('Acceleration (m/s²)')\n    plt.legend()\n    plt.tight_layout()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:41.593630Z","iopub.execute_input":"2025-07-12T10:44:41.593895Z","iopub.status.idle":"2025-07-12T10:44:42.672123Z","shell.execute_reply.started":"2025-07-12T10:44:41.593875Z","shell.execute_reply":"2025-07-12T10:44:42.671107Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Plot Thermopile Sensor Values Over Time","metadata":{}},{"cell_type":"code","source":"thm_cols = [f'thm_{i}' for i in range(1, 6)]\nfor seq_id in example_sequences:\n    seq_data = train_df[train_df['sequence_id'] == seq_id]\n    plt.figure(figsize=(12, 4))\n    for col in thm_cols:\n        plt.plot(seq_data['sequence_counter'], seq_data[col], label=col)\n    plt.title(f'Thermopile Sensor Readings for Sequence {seq_id}')\n    plt.xlabel('Time Step')\n    plt.ylabel('Temperature (°C)')\n    plt.legend()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:47.042342Z","iopub.execute_input":"2025-07-12T10:44:47.042700Z","iopub.status.idle":"2025-07-12T10:44:48.025819Z","shell.execute_reply.started":"2025-07-12T10:44:47.042679Z","shell.execute_reply":"2025-07-12T10:44:48.024693Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Plot ToF Sensor Averages and Missingness","metadata":{}},{"cell_type":"code","source":"tof_cols = [col for col in train_df.columns if col.startswith('tof_')]\nfor seq_id in example_sequences:\n    seq_data = train_df[train_df['sequence_id'] == seq_id]\n    plt.figure(figsize=(12, 4))\n    for i in range(1, 6):\n        tof_sensor_cols = [f'tof_{i}_v{j}' for j in range(64)]\n        mean_dist = seq_data[tof_sensor_cols].replace(-1, np.nan).mean(axis=1)\n        pct_neg1 = (seq_data[tof_sensor_cols] == -1).mean(axis=1) * 100\n        plt.plot(seq_data['sequence_counter'], mean_dist, label=f'ToF {i} Mean Dist')\n        plt.plot(seq_data['sequence_counter'], pct_neg1, '--', label=f'ToF {i} % -1')\n    plt.title(f'ToF Sensor Mean Distance and Missingness for Sequence {seq_id}')\n    plt.xlabel('Time Step')\n    plt.ylabel('Distance / % -1')\n    plt.legend()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:44:52.381203Z","iopub.execute_input":"2025-07-12T10:44:52.381501Z","iopub.status.idle":"2025-07-12T10:44:53.548600Z","shell.execute_reply.started":"2025-07-12T10:44:52.381481Z","shell.execute_reply":"2025-07-12T10:44:53.547708Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ToF Heatmap Visualization","metadata":{}},{"cell_type":"code","source":"seq_id = example_sequences[0]\nseq_data = train_df[train_df['sequence_id'] == seq_id]\n\ntime_step = 10\nif time_step >= len(seq_data):\n    time_step = len(seq_data) // 2  # fallback if index is out of bounds\n\ntof_sensor = 1\ntof_grid_cols = [f'tof_{tof_sensor}_v{j}' for j in range(64)]\n\n# Extract the row \nrow = seq_data.iloc[time_step][tof_grid_cols]\n\n# Convert to numeric (coerce errors to NaN)\ntof_values = pd.to_numeric(row, errors='coerce').values.astype(float)\n\n#  Reshape into 8x8 grid\ntof_grid = tof_values.reshape(8, 8)\n\n#  - mask -1 values as NaN for better visualization\ntof_grid[tof_grid == -1] = np.nan\n\n# Plot\nplt.figure(figsize=(6, 5))\nsns.heatmap(tof_grid, cmap='viridis', annot=False, cbar=True)\nplt.title(f'ToF Sensor {tof_sensor} Heatmap at Time Step {time_step} (Sequence {seq_id})')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:45:00.385951Z","iopub.execute_input":"2025-07-12T10:45:00.386222Z","iopub.status.idle":"2025-07-12T10:45:00.692647Z","shell.execute_reply.started":"2025-07-12T10:45:00.386204Z","shell.execute_reply":"2025-07-12T10:45:00.691572Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. 🧭 Orientation & Behavior\nLet's analyze the distribution of orientation and behavior phases.\n\n### Orientation Distribution","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8, 4))\nsns.countplot(data=train_df, x='orientation', order=train_df['orientation'].value_counts().index, palette='Set2')\nplt.title('Orientation Distribution')\nplt.xticks(rotation=45, ha='right')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:49:07.484523Z","iopub.execute_input":"2025-07-12T10:49:07.484773Z","iopub.status.idle":"2025-07-12T10:49:07.802573Z","shell.execute_reply.started":"2025-07-12T10:49:07.484758Z","shell.execute_reply":"2025-07-12T10:49:07.801478Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Behavior Phase Distribution","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8, 4))\nsns.countplot(data=train_df, x='behavior', order=train_df['behavior'].value_counts().index, palette='Pastel1')\nplt.title('Behavior Phase Distribution')\nplt.xticks(rotation=45, ha='right')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:49:35.491651Z","iopub.execute_input":"2025-07-12T10:49:35.491938Z","iopub.status.idle":"2025-07-12T10:49:35.804173Z","shell.execute_reply.started":"2025-07-12T10:49:35.491920Z","shell.execute_reply":"2025-07-12T10:49:35.803089Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Rows Per Phase in Each Sequence","metadata":{}},{"cell_type":"code","source":"phase_counts = train_df.groupby(['sequence_id', 'behavior']).size().unstack(fill_value=0)\nphase_counts.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:45:17.934941Z","iopub.execute_input":"2025-07-12T10:45:17.935193Z","iopub.status.idle":"2025-07-12T10:45:18.020002Z","shell.execute_reply.started":"2025-07-12T10:45:17.935177Z","shell.execute_reply":"2025-07-12T10:45:18.019312Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. 👥 Demographic Insights\nMerge demographics and analyze age, sex, and gesture counts by group.\n\n### Merge Demographics","metadata":{}},{"cell_type":"code","source":"train_merged = train_df.merge(train_dem_df, on='subject', how='left')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:45:20.484469Z","iopub.execute_input":"2025-07-12T10:45:20.484736Z","iopub.status.idle":"2025-07-12T10:45:20.827226Z","shell.execute_reply.started":"2025-07-12T10:45:20.484720Z","shell.execute_reply":"2025-07-12T10:45:20.826169Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Age Distribution","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(6, 4))\nsns.histplot(train_merged['age'], bins=15, kde=True)\nplt.title('Age Distribution')\nplt.xlabel('Age')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:45:23.612074Z","iopub.execute_input":"2025-07-12T10:45:23.612312Z","iopub.status.idle":"2025-07-12T10:45:25.628546Z","shell.execute_reply.started":"2025-07-12T10:45:23.612295Z","shell.execute_reply":"2025-07-12T10:45:25.627496Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Sex Distribution\n","metadata":{}},{"cell_type":"code","source":"train_merged['sex_label'] = train_merged['sex'].map({0: 'Female', 1: 'Male'})\n\nplt.figure(figsize=(4, 4))\nsns.countplot(data=train_merged, x='sex_label', palette=['#f4a582', '#92c5de'])  # salmon & sky blue\nplt.title('Sex Distribution')\nplt.xlabel('Sex')\nplt.ylabel('Count')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:45:53.391866Z","iopub.execute_input":"2025-07-12T10:45:53.392125Z","iopub.status.idle":"2025-07-12T10:45:53.720601Z","shell.execute_reply.started":"2025-07-12T10:45:53.392110Z","shell.execute_reply":"2025-07-12T10:45:53.719491Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Gestures Per Demographic Group (e.g., Child vs. Adult)","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nsns.countplot(\n    data=train_merged,\n    x='gesture',\n    hue='adult_child',\n    order=train_merged['gesture'].value_counts().index,\n    palette={0: '#8dd3c7', 1: '#fb8072'}  # teal for children, red-orange for adults\n)\nplt.title('Gesture Counts by Age Group')\nplt.xlabel('Gesture')\nplt.ylabel('Count')\nplt.xticks(rotation=45, ha='right')\nplt.legend(title='Age Group', labels=['Child', 'Adult'])\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:46:20.548003Z","iopub.execute_input":"2025-07-12T10:46:20.548269Z","iopub.status.idle":"2025-07-12T10:46:21.139242Z","shell.execute_reply.started":"2025-07-12T10:46:20.548251Z","shell.execute_reply":"2025-07-12T10:46:21.138388Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 8. 🧪 Feature Engineering Preview\nPreview some engineered features to guide modeling.\n\n### IMU Signal Magnitude","metadata":{}},{"cell_type":"code","source":"train_df['imu_magnitude'] = np.sqrt(train_df['acc_x']**2 + train_df['acc_y']**2 + train_df['acc_z']**2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:46:35.283367Z","iopub.execute_input":"2025-07-12T10:46:35.283685Z","iopub.status.idle":"2025-07-12T10:46:35.292139Z","shell.execute_reply.started":"2025-07-12T10:46:35.283669Z","shell.execute_reply":"2025-07-12T10:46:35.291392Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Quaternion Norm","metadata":{}},{"cell_type":"code","source":"train_df['quat_norm'] = np.sqrt(train_df['rot_w']**2 + train_df['rot_x']**2 + train_df['rot_y']**2 + train_df['rot_z']**2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:46:37.458118Z","iopub.execute_input":"2025-07-12T10:46:37.459068Z","iopub.status.idle":"2025-07-12T10:46:37.470083Z","shell.execute_reply.started":"2025-07-12T10:46:37.459029Z","shell.execute_reply":"2025-07-12T10:46:37.469088Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ToF: % of -1s Per Row","metadata":{}},{"cell_type":"code","source":"tof_neg1_pct_row = (train_df[tof_cols] == -1).mean(axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:46:39.514391Z","iopub.execute_input":"2025-07-12T10:46:39.514773Z","iopub.status.idle":"2025-07-12T10:46:39.946081Z","shell.execute_reply.started":"2025-07-12T10:46:39.514752Z","shell.execute_reply":"2025-07-12T10:46:39.945321Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Thermopile: Temperature Range Across Sensors","metadata":{}},{"cell_type":"code","source":"train_df['thm_range'] = train_df[thm_cols].max(axis=1) - train_df[thm_cols].min(axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:46:45.900159Z","iopub.execute_input":"2025-07-12T10:46:45.900443Z","iopub.status.idle":"2025-07-12T10:46:46.004788Z","shell.execute_reply.started":"2025-07-12T10:46:45.900396Z","shell.execute_reply":"2025-07-12T10:46:46.003918Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visualize Feature Distributions","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 4))\nsns.histplot(train_df['imu_magnitude'], bins=50, kde=True)\nplt.title('IMU Signal Magnitude Distribution')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-12T10:47:16.496254Z","iopub.execute_input":"2025-07-12T10:47:16.496604Z","iopub.status.idle":"2025-07-12T10:47:19.077226Z","shell.execute_reply.started":"2025-07-12T10:47:16.496578Z","shell.execute_reply":"2025-07-12T10:47:19.076192Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔚 Conclusion & Next Steps\n\nThis EDA highlighted key patterns, missingness issues, and demographic variation in the sensor data. Based on these insights, modeling efforts should focus on:\n\n- Using IMU data as a primary signal source\n- Carefully imputing or modeling around -1 values in ToF sensors\n- Addressing class imbalance in gestures\n- Incorporating demographic features when relevant","metadata":{}}]}