{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1 style=\"font-family:Arial; text-align:center; font-weight:bold;\">\n  Understanding the Competition + Deep EDA 🧠📉\n</h1>","metadata":{"_uuid":"0680892b-85a7-4702-a6b3-a245dc6b9524","_cell_guid":"b7384530-9ad2-4446-b4d2-47925cf515e3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"---\n## 🧠 Understanding the Competition — TL;DR (Too Lazy; Didn't Read 👀🤨)\n\n\n### 🧐 What's It About?\nThis competition involves analyzing **time-series sensor data** from wearable devices to classify **specific behaviors** (gestures). The data includes IMU (accelerometer, gyroscope, orientation), thermopiles (temperature), and time-of-flight (distance) readings, with each record representing a time window labeled with a gesture.\n\n\n### 🎯 What's the Goal?\nBuild a machine learning model to **predict gesture classes** accurately. The evaluation metric is likely a classification metric (e.g., macro F1-score), emphasizing balanced performance across imbalanced gesture classes. Feature extraction and pattern recognition are key.\n\n\n### 📊 What Kind of Data Do We Have?\n- **Training Data**: Time-series sensor readings with gesture labels per sequence.\n- **Test Data**: Similar sensor readings without labels.\n- **Demographics**: Participant info (e.g., age, gender) for train and test sets.\n- **Features**: Multivariate time-series (acc_x/y/z, rot_x/y/z, thermopile, time-of-flight).\n- **Note**: Data may have missing values, and gesture classes are imbalanced.","metadata":{"_uuid":"f79871dc-f323-49bb-88a8-d6203155a699","_cell_guid":"7254d4b5-f3f7-4592-9f66-4e7292fcc19e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## ⚠️ What You *Really* Need to Know!\n\nCan you tell when someone’s scratching their head vs. pulling their hair out (literally) using wrist sensor data? Let’s find out! \n\n###  18 gestures in total:\n- **8 BFRB-like** (target behaviors)\n- **10 non-BFRB** (non-targets)\n\n###  Two tasks:\n- **Binary Classification**: BFRB vs. not-BFRB\n- **Multiclass Classification**: Which gesture exactly?\n\n###  Test Set Surprise:\n- Half = **IMU only**\n- Half = **all sensors**\n- Each sequence: `Transition → Pause → Gesture`\n","metadata":{}},{"cell_type":"code","source":"# Import libraries\nfrom tqdm.notebook import tqdm\nimport pandas as pd\nimport numpy as np\nimport plotly.io as pio\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nfrom scipy.fft import fft\nfrom scipy.stats import kruskal, zscore\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Set professional plotting style\nplt.style.use('ggplot')\nplt.rcParams['figure.facecolor'] = 'white'\nplt.rcParams['font.family'] = 'Arial'\ncustom_palette = [\"#3498db\", \"#e74c3c\", \"#2ecc71\", \"#f39c12\", \"#9b59b6\"]\nsns.set_palette(custom_palette)\n\n# Suppress warnings for cleaner output\nimport warnings\nwarnings.filterwarnings('ignore', category=FutureWarning)\n","metadata":{"_uuid":"406b3b3d-a297-43fe-be6d-232a9650dbb7","_cell_guid":"623a6af6-8b06-4e6f-a505-4046d279a01e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:29:21.669428Z","iopub.execute_input":"2025-06-01T18:29:21.669703Z","iopub.status.idle":"2025-06-01T18:29:24.942267Z","shell.execute_reply.started":"2025-06-01T18:29:21.669679Z","shell.execute_reply":"2025-06-01T18:29:24.941179Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load data with progress bar\ndef load_data():\n    data_files = {\n        'train': '/kaggle/input/cmi-detect-behavior-with-sensor-data/train.csv',\n        'test': '/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv',\n        'train_demo': '/kaggle/input/cmi-detect-behavior-with-sensor-data/train_demographics.csv',\n        'test_demo': '/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv'\n    }\n    data = {}\n    with tqdm(total=len(data_files), desc=\"Loading Data\") as pbar:\n        for name, file in data_files.items():\n            try:\n                data[name] = pd.read_csv(file)\n                pbar.update(1)\n            except FileNotFoundError:\n                print(f\"Error: {file} not found. Please check the file path.\")\n                return None\n    return data\n\ndata = load_data()\nif data is None:\n    raise FileNotFoundError(\"Data loading failed.\")\ntrain, test, demo_train, demo_test = data['train'], data['test'], data['train_demo'], data['test_demo']\n\n# Display basic dataset info\nprint(\"Train Shape:\", train.shape)\nprint(\"Test Shape:\", test.shape)\nprint(\"Train Demographics Shape:\", demo_train.shape)\nprint(\"Test Demographics Shape:\", demo_test.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-01T18:29:24.943766Z","iopub.execute_input":"2025-06-01T18:29:24.944526Z","iopub.status.idle":"2025-06-01T18:30:02.558633Z","shell.execute_reply.started":"2025-06-01T18:29:24.944500Z","shell.execute_reply":"2025-06-01T18:30:02.556860Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 1️⃣ Gesture Distribution - The Big Picture","metadata":{"_uuid":"ba73729a-ecd0-4e32-8d00-cd220ba9d821","_cell_guid":"4f48ff51-2752-4016-afc8-11cd35e415db","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_gesture_distribution(df):\n\n    gesture_counts = df['gesture'].value_counts().sort_values(ascending=False)\n    gesture_percentages = gesture_counts / gesture_counts.sum() * 100\n\n    # Bar plot with counts\n    plt.figure(figsize=(14, 6))\n    ax = sns.barplot(x=gesture_counts.index, y=gesture_counts.values, \n                     order=gesture_counts.index, palette='viridis')\n    plt.title('Gesture Distribution in Training Data\\n(Log Scale)', fontsize=16, pad=20)\n    plt.xlabel('Gesture Type', fontsize=12)\n    plt.ylabel('Count (log scale)', fontsize=12)\n    plt.yscale('log')\n    plt.xticks(rotation=45, ha='right')\n    \n    # Add count annotations\n    for p in ax.patches:\n        ax.annotate(f\"{int(p.get_height()):,}\", \n                    (p.get_x() + p.get_width() / 2., p.get_height()),\n                    ha='center', va='center', xytext=(0, 10), \n                    textcoords='offset points', fontsize=9)\n    \n    plt.tight_layout()\n    plt.show()\n\n    # Percentage plot\n    plt.figure(figsize=(14, 6))\n    ax = sns.barplot(x=gesture_percentages.index, y=gesture_percentages.values, \n                     order=gesture_counts.index, palette='viridis')\n    plt.title('Gesture Distribution (% of Total)', fontsize=16)\n    plt.xlabel('Gesture Type', fontsize=12)\n    plt.ylabel('Percentage (%)', fontsize=12)\n    plt.xticks(rotation=45, ha='right')\n    \n    # Add percentage annotations\n    for p in ax.patches:\n        ax.annotate(f\"{p.get_height():.1f}%\", \n                    (p.get_x() + p.get_width() / 2., p.get_height()),\n                    ha='center', va='center', xytext=(0, 10), \n                    textcoords='offset points', fontsize=9)\n    \n    plt.tight_layout()\n    plt.show()\n\nplot_gesture_distribution(train)","metadata":{"_uuid":"db21117a-7d6f-414a-9d64-90693ea284f7","_cell_guid":"93e8f4c4-8e2a-4ee6-a7e1-e981e361c3bd","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:30:02.561848Z","iopub.execute_input":"2025-06-01T18:30:02.562699Z","iopub.status.idle":"2025-06-01T18:30:03.924610Z","shell.execute_reply.started":"2025-06-01T18:30:02.562666Z","shell.execute_reply":"2025-06-01T18:30:03.923410Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: The gesture distribution is imbalanced, with some gestures dominating. This suggests using techniques like class weights or oversampling during modeling to handle minority classes.","metadata":{"_uuid":"15251309-556c-4efd-a143-2bd8c7a35c31","_cell_guid":"14f35441-290e-4d72-b9e6-4ae6c0a9aed0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 2️⃣ Sensor Data Deep Dive","metadata":{"_uuid":"37694e54-edec-422e-b90d-20c02de9f006","_cell_guid":"8b3b7812-e95b-4664-863a-7d9a49e2a9f6","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_sensor_distributions(df, sensor_type, cols_per_row=3):\n    sensor_cols = [col for col in df.columns if sensor_type in col]\n    if not sensor_cols:\n        print(f\"No columns found for {sensor_type}.\")\n        return\n    \n    n_cols = len(sensor_cols)\n    n_rows = (n_cols + cols_per_row - 1) // cols_per_row\n    \n    plt.figure(figsize=(18, 4 * n_rows))\n    for i, col in enumerate(sensor_cols, 1):\n        plt.subplot(n_rows, cols_per_row, i)\n        sns.kdeplot(df[col].dropna(), fill=True, alpha=0.7, color=custom_palette[i % len(custom_palette)])\n        plt.title(f'Distribution of {col}', fontsize=10)\n        plt.xlabel('Value')\n        plt.ylabel('Density')\n    plt.tight_layout()\n    plt.suptitle(f'{sensor_type.upper()} Sensor Distributions', y=1.02, fontsize=16)\n    plt.show()\n\n    # Summary statistics\n    print(f\"\\nSummary Statistics for {sensor_type} Sensors:\")\n    print(df[sensor_cols].describe())\n\n# Plot accelerometer and rotation distributions\nplot_sensor_distributions(train, 'acc_')\nplot_sensor_distributions(train, 'rot_')\n\n# Correlation heatmap\nsensor_cols = [col for col in train.columns if 'acc_' in col or 'rot_' in col]\nif sensor_cols:\n    plt.figure(figsize=(10, 8))\n    sns.heatmap(train[sensor_cols].corr(), annot=True, cmap='coolwarm', fmt='.2f')\n    plt.title('Correlation Matrix of Sensor Features', fontsize=16)\n    plt.tight_layout()\n    plt.show()","metadata":{"_uuid":"0f6ab7ff-920f-4790-8186-ee6234aa4d34","_cell_guid":"19e7b60a-f7e3-4788-9e89-ed93018d1902","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:30:03.926928Z","iopub.execute_input":"2025-06-01T18:30:03.927289Z","iopub.status.idle":"2025-06-01T18:30:23.322141Z","shell.execute_reply.started":"2025-06-01T18:30:03.927253Z","shell.execute_reply":"2025-06-01T18:30:23.321043Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Sensor distributions vary by axis, with potential correlations between features (e.g., acc_x and acc_y). This suggests feature selection or dimensionality reduction (e.g., PCA) may be useful.","metadata":{"_uuid":"a6abbdd3-ef6d-44ff-b73c-6849ed4bbddb","_cell_guid":"725dd037-fafd-453a-bebd-2ce87b733c14","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 3️⃣ Time Series Storytelling","metadata":{"_uuid":"bdc16e1d-bd4b-4e52-a999-30062b47bbf6","_cell_guid":"62dbd86f-8572-4df3-90e8-2950b485ce6d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def create_interactive_sequence(df, sequence_id):\n    pio.renderers.default = 'iframe'\n    seq_data = df[df['sequence_id'] == sequence_id].copy()\n    if seq_data.empty:\n        print(f\"Sequence {sequence_id} not found.\")\n        return\n    \n    gesture = seq_data['gesture'].iloc[0]\n    fig = make_subplots(specs=[[{\"secondary_y\": True}]])\n    \n    # Add acceleration traces\n    for axis, color in zip(['x', 'y', 'z'], ['#e74c3c', '#3498db', '#2ecc71']):\n        fig.add_trace(\n            go.Scatter(\n                x=seq_data['sequence_counter'],\n                y=seq_data[f'acc_{axis}'],\n                name=f'Acc {axis.upper()}',\n                line=dict(color=color),\n                opacity=0.8\n            ),\n            secondary_y=False\n        )\n    \n    # Add behavior phases as background\n    behaviors = seq_data['behavior'].unique()\n    colors = px.colors.qualitative.Plotly\n    for i, behavior in enumerate(behaviors):\n        behavior_data = seq_data[seq_data['behavior'] == behavior]\n        fig.add_trace(\n            go.Scatter(\n                x=behavior_data['sequence_counter'],\n                y=[0] * len(behavior_data),\n                mode='markers',\n                marker=dict(size=15, color=colors[i % len(colors)], opacity=0.2),\n                name=behavior,\n                showlegend=True\n            ),\n            secondary_y=True\n        )\n    \n    # Add rotation traces\n    for axis, color in zip(['x', 'y', 'z'], ['#e67e22', '#16a085', '#c0392b']):\n        fig.add_trace(\n            go.Scatter(\n                x=seq_data['sequence_counter'],\n                y=seq_data[f'rot_{axis}'],\n                name=f'Rot {axis.upper()}',\n                line=dict(color=color, dash='dot'),\n                opacity=0.6\n            ),\n            secondary_y=False\n        )\n    \n    fig.update_layout(\n        title=f'Sequence {sequence_id} - {gesture}<br><sup>Interactive Sensor Timeline (Sequence Counter = Time Steps)</sup>',\n        hovermode='x unified',\n        height=600,\n        template='plotly_white'\n    )\n    fig.update_yaxes(title_text=\"Sensor Values\", secondary_y=False)\n    fig.update_yaxes(title_text=\"Behavior Phases\", secondary_y=True)\n    fig.show()\n\n    # FFT analysis for one sensor\n    def plot_fft(data, sensor='acc_x'):\n        fft_vals = np.abs(fft(data[sensor].dropna()))[:len(data)//2]\n        freqs = np.linspace(0, 0.5, len(fft_vals))\n        plt.figure(figsize=(10, 4))\n        plt.plot(freqs, fft_vals, color=custom_palette[0])\n        plt.title(f'FFT of {sensor} for Sequence {sequence_id}')\n        plt.xlabel('Frequency')\n        plt.ylabel('Amplitude')\n        plt.tight_layout()\n        plt.show()\n    \n    plot_fft(seq_data, 'acc_x')\n\n# Plot for a sample sequence\nsample_seq = train['sequence_id'].unique()[10]\ncreate_interactive_sequence(train, sample_seq)","metadata":{"_uuid":"e91246ce-6e2a-4e48-9771-ddcb7a0dbcfd","_cell_guid":"e6f90e89-6279-4570-bd95-131f4c9df5e8","trusted":true,"execution":{"iopub.status.busy":"2025-06-01T18:36:19.054448Z","iopub.execute_input":"2025-06-01T18:36:19.054791Z","iopub.status.idle":"2025-06-01T18:36:19.435799Z","shell.execute_reply.started":"2025-06-01T18:36:19.054761Z","shell.execute_reply":"2025-06-01T18:36:19.434921Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Time-series patterns vary by gesture, with distinct behavior phases. Frequency-domain analysis (FFT) reveals periodic components, useful for feature engineering.","metadata":{"_uuid":"cd0df481-e524-4b9a-a995-da84c5371fa6","_cell_guid":"37f9e46d-5937-4eb2-98c9-f2d8729ee6ea","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 4️⃣ Missing Data Investigation","metadata":{"_uuid":"502bde06-965b-40d9-8711-57d7201a59a9","_cell_guid":"090211d0-b1f2-4698-aa5f-b0da71ff91cc","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Import required libraries\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport missingno as msno\nimport numpy as np\n\n# Reuse custom palette\ncustom_palette = [\"#3498db\", \"#e74c3c\", \"#2ecc71\", \"#f39c12\", \"#9b59b6\"]\nsns.set_palette(custom_palette)\n\ndef plot_missing_data(df, top_n=10):\n    # Calculate missing data percentages\n    missing_data = df.isna().mean() * 100\n    missing_data = missing_data[missing_data > 0]  # Filter columns with missing values\n    \n    if missing_data.empty:\n        print(\"No missing data found.\")\n        return\n    \n    # --- 1. Grouped Bar Plot by Sensor Type ---\n    # Define sensor types based on column prefixes\n    sensor_types = ['acc_', 'rot_', 'thermopile_', 'tof_']  \n    missing_by_type = {}\n    for stype in sensor_types:\n        cols = [col for col in missing_data.index if stype in col]\n        if cols:\n            missing_by_type[stype.replace('_', '')] = missing_data[cols].mean()\n    \n    if missing_by_type:\n        plt.figure(figsize=(10, 6))\n        ax = sns.barplot(x=list(missing_by_type.keys()), y=list(missing_by_type.values()))\n        plt.title('Average Missing Data by Sensor Type', fontsize=16)\n        plt.xlabel('Sensor Type', fontsize=12)\n        plt.ylabel('Missing (%)', fontsize=12)\n        # Add annotations\n        for p in ax.patches:\n            ax.annotate(f\"{p.get_height():.1f}%\", \n                        (p.get_x() + p.get_width() / 2., p.get_height()),\n                        ha='center', va='center', xytext=(0, 10), \n                        textcoords='offset points', fontsize=9)\n        plt.tight_layout()\n        plt.show()\n\n    # --- 2. Top-N Features Bar Plot ---\n    top_missing = missing_data.sort_values(ascending=False).head(top_n)\n    plt.figure(figsize=(12, 6))\n    ax = sns.barplot(x=top_missing.index, y=top_missing.values, palette='magma')\n    plt.title(f'Top {top_n} Features with Highest Missing Data', fontsize=16)\n    plt.xlabel('Feature', fontsize=12)\n    plt.ylabel('Missing (%)', fontsize=12)\n    plt.xticks(rotation=45, ha='right')\n    # Add annotations\n    for p in ax.patches:\n        ax.annotate(f\"{p.get_height():.1f}%\", \n                    (p.get_x() + p.get_width() / 2., p.get_height()),\n                    ha='center', va='center', xytext=(0, 10), \n                    textcoords='offset points', fontsize=9)\n    plt.tight_layout()\n    plt.show()\n\n    # --- 3. Heatmap of Missing Data by Gesture ---\n    sensor_cols = [col for col in df.columns if any(stype in col for stype in sensor_types)]\n    if sensor_cols:\n        missing_by_gesture = df.groupby('gesture')[sensor_cols].apply(lambda x: x.isna().mean() * 100)\n        plt.figure(figsize=(12, 8))\n        sns.heatmap(missing_by_gesture, cmap='magma', annot=True, fmt='.1f', cbar_kws={'label': 'Missing (%)'})\n        plt.title('Missing Data (%) by Gesture and Feature', fontsize=16)\n        plt.xlabel('Feature')\n        plt.ylabel('Gesture')\n        plt.tight_layout()\n        plt.show()\n\n    # --- 5. Summary Table ---\n    print(\"\\nSummary of Missing Data (%):\")\n    print(missing_data.to_frame(name='Missing (%)').sort_values(by='Missing (%)', ascending=False))\n\n\nplot_missing_data(train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-01T18:30:23.343751Z","iopub.execute_input":"2025-06-01T18:30:23.344043Z","iopub.status.idle":"2025-06-01T18:30:36.915588Z","shell.execute_reply.started":"2025-06-01T18:30:23.344016Z","shell.execute_reply":"2025-06-01T18:30:36.914419Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Missing data varies by sensor and gesture. Imputation strategies (e.g., forward fill for time-series continuity, mean imputation for small gaps) should be tested.","metadata":{"_uuid":"c92619a0-ff43-497d-9b71-ff42be6117a7","_cell_guid":"bfb287f4-831d-4ce9-a6ba-12205afcc06a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 5️⃣ Demographic Insights","metadata":{"_uuid":"d21bc9d7-7405-4173-ae35-0a18bebd5d7a","_cell_guid":"e65b425f-fa63-4891-a983-2ce8fed6400d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_demographic_insights(train_df, demo_df):\n    # Merge train and demographic data\n    merged = train_df.merge(demo_df, on='subject', how='left')\n    \n    # Age distribution by gender\n    plt.figure(figsize=(10, 6))\n    sns.histplot(data=demo_df, x='age', hue='sex', multiple='stack', palette='Set2')\n    plt.title('Age Distribution by Gender', fontsize=16)\n    plt.xlabel('Age')\n    plt.ylabel('Count')\n    plt.tight_layout()\n    plt.show()\n    \n    # Gesture distribution by gender\n    gesture_by_gender = merged.groupby(['sex', 'gesture']).size().unstack().fillna(0)\n    gesture_by_gender.plot(kind='bar', stacked=True, figsize=(10, 6), color=custom_palette)\n    plt.title('Gesture Distribution by Gender', fontsize=16)\n    plt.xlabel('Sex')\n    plt.ylabel('Count')\n    plt.xticks(rotation=0)\n    plt.legend(title='Gesture', bbox_to_anchor=(1.05, 1), loc='upper left')\n    plt.tight_layout()\n    plt.show()\n\nplot_demographic_insights(train, demo_train)","metadata":{"_uuid":"ffba8686-feb4-4b34-92aa-00aeac42da59","_cell_guid":"e5c37338-750a-4c09-8d12-589313012961","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:30:36.916962Z","iopub.execute_input":"2025-06-01T18:30:36.917322Z","iopub.status.idle":"2025-06-01T18:30:38.834818Z","shell.execute_reply.started":"2025-06-01T18:30:36.917290Z","shell.execute_reply":"2025-06-01T18:30:38.833516Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Demographic factors (e.g., age, gender) may influence gesture patterns. Consider including demographic features in the model or stratifying data during training.","metadata":{"_uuid":"92849a0c-fa5f-4e49-8106-e1168ffcf8ad","_cell_guid":"92a9f0c9-5200-496c-831c-3eb8f55bb9ef","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 6️⃣ Gesture Duration Analysis","metadata":{"_uuid":"8e35234f-2ef4-4172-8ad8-cbd1dd694f86","_cell_guid":"14383170-bd85-4fe1-9614-b208c676de79","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from scipy.stats import kruskal\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ndef plot_gesture_duration(df):\n    # Compute gesture counts for ordering\n    gesture_counts = df['gesture'].value_counts().sort_values(ascending=False)\n    gesture_durations = df.groupby(['gesture', 'sequence_id']).size().reset_index(name='duration')\n    \n    plt.figure(figsize=(14, 6))\n    sns.boxplot(data=gesture_durations, x='gesture', y='duration', \n                order=gesture_counts.index, palette='viridis', showfliers=False)\n    plt.title('Gesture Duration Distribution (Without Outliers)', fontsize=16)\n    plt.xlabel('Gesture')\n    plt.ylabel('Duration (frames)')\n    plt.xticks(rotation=45, ha='right')\n    \n    # Add median labels\n    medians = gesture_durations.groupby('gesture')['duration'].median().loc[gesture_counts.index]\n    for i, median in enumerate(medians):\n        plt.text(i, median + 5, f\"Median: {median}\", ha='center', va='bottom', fontsize=9)\n    \n    plt.tight_layout()\n    plt.show()\n    \n    # Statistical test for duration differences\n    groups = [gesture_durations[gesture_durations['gesture'] == g]['duration'] for g in gesture_counts.index]\n    stat, p = kruskal(*groups)\n    print(f\"Kruskal-Wallis Test for Duration Differences: Statistic={stat:.2f}, p-value={p:.4f}\")\n    \nplot_gesture_duration(train)","metadata":{"_uuid":"5d798f0a-4fe9-4fc6-8a17-f77674e6dee8","_cell_guid":"1f2b714a-2301-4be8-9bf1-9042addc9e2c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:30:38.836165Z","iopub.execute_input":"2025-06-01T18:30:38.837356Z","iopub.status.idle":"2025-06-01T18:30:39.460511Z","shell.execute_reply.started":"2025-06-01T18:30:38.837304Z","shell.execute_reply":"2025-06-01T18:30:39.459352Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Gesture durations vary significantly, suggesting time-window normalization or duration-based features for modeling.","metadata":{"_uuid":"2b56901b-4297-482b-b545-aa58cfbcf03c","_cell_guid":"8ccd84a0-2df4-49c0-9431-3a8b01a3a181","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 7️⃣ Train vs. Test Comparison","metadata":{"_uuid":"9d6459bd-ffe5-456c-9a69-5ff05314d6bb","_cell_guid":"636dd452-1b9b-4327-babb-57c395f8e868","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def compare_train_test(train_df, test_df):\n    sensor_cols = [col for col in train_df.columns if 'acc_' in col or 'rot_' in col]\n    for sensor in sensor_cols[:3]:  # Limit to first 3 for brevity\n        plt.figure(figsize=(10, 6))\n        sns.kdeplot(train_df[sensor].dropna(), label='Train', color=custom_palette[0])\n        sns.kdeplot(test_df[sensor].dropna(), label='Test', color=custom_palette[1])\n        plt.title(f'Train vs. Test: {sensor} Distribution', fontsize=16)\n        plt.xlabel('Value')\n        plt.ylabel('Density')\n        plt.legend()\n        plt.tight_layout()\n        plt.show()\n\ncompare_train_test(train, test)","metadata":{"_uuid":"191bafcb-0aac-414f-add5-ddee25a0d148","_cell_guid":"7bdd92ae-fe00-4856-bfe3-8e8563a802cc","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:30:39.461784Z","iopub.execute_input":"2025-06-01T18:30:39.462105Z","iopub.status.idle":"2025-06-01T18:30:48.375578Z","shell.execute_reply.started":"2025-06-01T18:30:39.462078Z","shell.execute_reply":"2025-06-01T18:30:48.374330Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Train and test distributions are similar, but slight differences suggest robust preprocessing (e.g., normalization) to handle potential data drift.","metadata":{"_uuid":"99a215c9-4bc6-4acc-b02e-64f0021f2c02","_cell_guid":"f3e0882c-576c-45e0-b3e8-22b5939932b1","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 8️⃣ Feature Importance Exploration","metadata":{"_uuid":"281613b9-5250-4ae6-be69-5de2499c1087","_cell_guid":"b06843f2-919d-4cd8-bd7e-24ed24190d23","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_feature_importance(df):\n    sensor_cols = [col for col in df.columns if 'acc_' in col or 'rot_' in col]\n    X = df[sensor_cols].fillna(0)  # Simple imputation for demo\n    y = df['gesture']\n    \n    rf = RandomForestClassifier(n_estimators=100, random_state=42)\n    rf.fit(X, y)\n    \n    importances = pd.Series(rf.feature_importances_, index=sensor_cols).sort_values()\n    plt.figure(figsize=(10, 6))\n    importances.plot(kind='barh', color=custom_palette[2])\n    plt.title('Feature Importance (Random Forest)', fontsize=16)\n    plt.xlabel('Importance')\n    plt.tight_layout()\n    plt.show()\n\nplot_feature_importance(train)","metadata":{"_uuid":"52686b63-2e89-4aee-8fac-e3e1444e7d1f","_cell_guid":"e8b6f0c0-6f7a-4173-a62f-e2ac30cfe93e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:30:48.378254Z","iopub.execute_input":"2025-06-01T18:30:48.378519Z","iopub.status.idle":"2025-06-01T18:34:47.062460Z","shell.execute_reply.started":"2025-06-01T18:30:48.378501Z","shell.execute_reply":"2025-06-01T18:34:47.061472Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Some sensors (e.g., rot_x) are more predictive than others. Focus on high-importance features for modeling.","metadata":{"_uuid":"a1db6268-bc78-4e38-a598-c90a089ecd69","_cell_guid":"f40c2eef-af2a-4328-8ac7-796fc2340b87","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 9️⃣ Outlier Detection","metadata":{"_uuid":"f920212e-e37c-4dba-bb34-8d946b7a5b50","_cell_guid":"1cf59f18-45e9-41a2-a09d-174ab553c85c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from scipy.stats import zscore\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport numpy as np\n\ndef detect_outliers(df, z_threshold=3):\n    # Define sensor columns\n    sensor_cols = [col for col in df.columns if 'acc_' in col or 'rot_' in col]\n    if not sensor_cols:\n        print(\"No sensor columns found (acc_ or rot_).\")\n        return\n    \n    # Initialize dictionary to store outlier counts\n    outlier_counts = {}\n    \n    # Clean data and compute Z-scores\n    for col in sensor_cols:\n        # Drop NaN and inf values for this column\n        data = df[col].replace([np.inf, -np.inf], np.nan).dropna()\n        if data.empty:\n            outlier_counts[col] = 0\n            continue\n        \n        # Check for zero variance\n        if data.std() == 0:\n            print(f\"Warning: Column {col} has zero variance. Skipping outlier detection.\")\n            outlier_counts[col] = 0\n            continue\n        \n        # Compute Z-scores\n        z_scores = zscore(data, nan_policy='omit')\n        # Count outliers (|Z| > z_threshold)\n        outlier_counts[col] = np.sum(np.abs(z_scores) > z_threshold)\n    \n    # Print outlier counts\n    print(\"\\nOutliers per Sensor (Z-score > {}):\".format(z_threshold))\n    print(pd.Series(outlier_counts))\n    \n    # Visualize outliers for acc_x (if available)\n    if 'acc_x' in sensor_cols and 'sequence_counter' in df.columns:\n        # Create a mask for outliers\n        data = df['acc_x'].replace([np.inf, -np.inf], np.nan)\n        if data.std() > 0:\n            z_scores = zscore(data.dropna(), nan_policy='omit')\n            # Align Z-scores with original data\n            outlier_mask = pd.Series(False, index=df.index)\n            outlier_mask[data.dropna().index] = np.abs(z_scores) > z_threshold\n            \n            plt.figure(figsize=(10, 6))\n            sns.scatterplot(x=df['sequence_counter'], y=df['acc_x'], \n                            hue=outlier_mask, palette={False: custom_palette[0], True: custom_palette[1]})\n            plt.title('Outliers in acc_x (Z-score > {})'.format(z_threshold), fontsize=16)\n            plt.xlabel('Sequence Counter')\n            plt.ylabel('acc_x')\n            plt.tight_layout()\n            plt.show()\n        else:\n            print(\"Warning: Cannot visualize acc_x due to zero variance or insufficient data.\")\n\ndetect_outliers(train)","metadata":{"_uuid":"4cab61e8-eb23-44ef-8073-a71da1322f18","_cell_guid":"a4910d6f-e74e-466d-8a11-b5a89a24f90a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-06-01T18:34:47.063628Z","iopub.execute_input":"2025-06-01T18:34:47.063931Z","iopub.status.idle":"2025-06-01T18:35:04.441901Z","shell.execute_reply.started":"2025-06-01T18:34:47.063908Z","shell.execute_reply":"2025-06-01T18:35:04.440734Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights**: Outliers are present in some sensors, suggesting robust scaling or clipping during preprocessing.","metadata":{"_uuid":"5d41398e-fde7-471f-a515-bb586a8e17a7","_cell_guid":"21aa1628-e780-438b-af69-146332a02a0c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### Key Takeaways:\n1. **Gesture Imbalance**: Use class weights or oversampling to handle imbalanced classes.\n2. **Sensor Patterns**: Distinct distributions and correlations suggest feature selection or PCA.\n3. **Time-Series Insights**: Behavior phases and frequency-domain features (FFT) are promising for modeling.\n4. **Missing Data**: Test imputation strategies like forward fill or mean imputation.\n5. **Demographics**: Include age and gender as features to capture participant variability.\n6. **Durations**: Normalize sequence lengths or use duration-based features.\n7. **Train-Test Alignment**: Similar distributions, but normalization is needed to handle drift.\n8. **Outliers**: Apply robust scaling or clipping to mitigate outlier effects.\n#\n### Next Steps:\n- **Preprocessing**: Normalize sensor data, impute missing values (forward fill), clip outliers.\n- **Feature Engineering**: Extract statistical features (mean, std), frequency-domain features (FFT), and demographic features.\n- **Modeling**: Use time-series models (LSTM, TCN) with class weights to address imbalance.","metadata":{}},{"cell_type":"markdown","source":"# 🔟  Labels & Evaluation\nIn this competion you're scored on:\n\n**Binary F1: Is it a BFRB or not?**\n\n**Macro F1: What gesture is it?**\n\n💡 All non-target gestures are grouped into one “non_target” class in the final evaluation.\n\nSo… your model should:\n\n* Be very good at binary separation\n\n* And somewhat decent at gesture-specific predictions (esp. for BFRBs)","metadata":{}},{"cell_type":"markdown","source":"# 🔚 Last Words (Don't Leaderboard Too Early!)\n\nSlow down and explore the data thoroughly before building models. Don't rush the leaderboard like it's a haircut on a Monday morning. EDA is your map before the treasure hunt!  \n \nGood luck everyone! Feel free to share your insights and suggestions in the comments!\n\n","metadata":{"_uuid":"9055d8c7-f3cd-4cb5-8a8a-9d6709b0d82f","_cell_guid":"615627e7-b709-4b91-8753-9f0861b5be33","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}