{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"},{"sourceId":12736345,"sourceType":"datasetVersion","datasetId":8049864},{"sourceId":242561727,"sourceType":"kernelVersion"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🧠 Model Performance Analysis\n\n**Introduction**  \nThis notebook dives deep into the results of our models to uncover the patterns behind their successes and mistakes.  \nBy understanding these patterns, we can identify clear opportunities to improve performance.\n\n---\n\n**Objective** \nAnalyze model predictions against different features to discover which factors most influence accuracy and provide guidance for model improvements.\n\n---\n\n**Required input:**\nA results CSV file containing at least the following columns:\n* `sequence_id`\n* `gesture`\n* `fold`\n\n---\n\n**We will explore** \n1. **Binary vs. multi-class performance** — target vs. non-target gestures, and detailed target gesture classification.  \n2. **Gesture confusion patterns** — which gestures are most often mistaken for each other.  \n3. **Impact of missing data** — rotation, THM, and TOF sensor gaps.  \n4. **Demographic & physical factors** — adult/child, age, handedness, height, shoulder-to-wrist distance, elbow-to-wrist distance.\n\n---\n\n**📊 Analysis Workflow** ","metadata":{}},{"cell_type":"code","source":"from IPython.display import SVG\nSVG(\"/kaggle/input/cmi-performance-analysis/chart.svg\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:20:08.132245Z","iopub.execute_input":"2025-09-03T17:20:08.132571Z","iopub.status.idle":"2025-09-03T17:20:08.170345Z","shell.execute_reply.started":"2025-09-03T17:20:08.132536Z","shell.execute_reply":"2025-09-03T17:20:08.168308Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**TL/DR** \n\n1. When THM and TOF data are missing, the IMU only model outperforms the IMU-THM-TOF model --> asign higher weights for IMU_only model when THM & TOF are not available.\n2. Both models achieve higher accuracy on adults, males, and right hands compared to children, females, and left hands. However, this gap may be influenced by the co-occurrence of missing rotation/THM/TOF data with these groups. --> upsampling for children, females, and left hands during training.\n3. No significant performance differences are observed across height, shoulder-to-wrist length, or elbow-to-wrist length. --> no need to use these features during training.","metadata":{}},{"cell_type":"markdown","source":"## Load data","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os, sys\nfrom pathlib import Path\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nsys.path.append(\"/kaggle/usr/lib/cmi-2025\")\nfrom metric import score\n\ndatadir = Path(\"/kaggle/input/cmi-detect-behavior-with-sensor-data\")\noutdir = Path(\"/kaggle/input/cmi-performance-analysis\")\n\nimu_only = pd.read_csv(f\"{outdir}/imu_only_5folds_results.csv\")\nimu_thm_tof = pd.read_csv(f\"{outdir}/imu_thm_tof_10folds_results.csv\")\n\ndf = pd.read_csv(datadir / \"train.csv\")\ndem_df = pd.read_csv(datadir / \"train_demographics.csv\")\n\ntarget_gestures = [\n    'Above ear - pull hair',\n    'Cheek - pinch skin',\n    'Eyebrow - pull hair',\n    'Eyelash - pull hair',\n    'Forehead - pull hairline',\n    'Forehead - scratch',\n    'Neck - pinch skin',\n    'Neck - scratch'\n]\n\nnon_target_gestures = [\n    'Write name on leg',\n    'Wave hello',\n    'Glasses on/off',\n    'Text on phone',\n    'Write name in air',\n    'Feel around in tray and pull out an object',\n    'Scratch knee/leg skin',\n    'Pull air toward your face',\n    'Drink from bottle/cup',\n    'Pinch knee/leg skin'\n]\n\ngt = df.groupby('sequence_id').first().reset_index()[['sequence_id', 'gesture']]\ngt[\"gesture_can\"] = gt[\"gesture\"].apply(lambda x: x if x in target_gestures else \"non_target\")\n\nprint(\"\"\"\n#########\n# LABEL #\n#########\n\"\"\")\nprint(\"\\nTarget gestures:\")\nprint(gt[\"gesture_can\"].value_counts())\nprint(\"\\ntarget vs non-target:\")\nprint(gt[\"gesture_can\"].apply(lambda x: \"target\" if x != \"non_target\" else \"non_target\").value_counts())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:20:08.172411Z","iopub.execute_input":"2025-09-03T17:20:08.173298Z","iopub.status.idle":"2025-09-03T17:21:04.015426Z","shell.execute_reply.started":"2025-09-03T17:20:08.173250Z","shell.execute_reply":"2025-09-03T17:21:04.013935Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Overall scores","metadata":{}},{"cell_type":"code","source":"def print_exp_score(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    res_df[\"label\"] = res_df[\"sequence_id\"].map(gt.set_index(\"sequence_id\")[\"gesture_can\"])\n    res_df[\"pred\"] = res_df[\"gesture\"].apply(lambda x: x if x in target_gestures else \"non_target\")\n    res_df[\"label_bin\"] = res_df[\"label\"].apply(lambda x: \"target\" if x != \"non_target\" else \"non_target\")\n    res_df[\"pred_bin\"] = res_df[\"pred\"].apply(lambda x: \"target\" if x != \"non_target\" else \"non_target\")\n    \n    folds = res_df['fold'].unique()\n    \n    for fold in folds:\n        fold_pred = res_df[res_df['fold'] == fold].sort_values(\"sequence_id\").reset_index(drop=True)\n        fold_gt = gt[gt['sequence_id'].isin(fold_pred['sequence_id'])].sort_values(\"sequence_id\").reset_index(drop=True)\n        \n        score_value = score(fold_gt, fold_pred, row_id_column_name=\"sequence_id\")\n        print(f\"\\tFold {fold} - Score: {score_value:.4f}\")\n        \n    print(\"=\"*50)\n    print(\"Overall Score:\", score(gt.sort_values(\"sequence_id\").reset_index(drop=True), \n                                  res_df.sort_values(\"sequence_id\").reset_index(drop=True), \n                                  row_id_column_name=\"sequence_id\"))\n\nprint_exp_score(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nprint_exp_score(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:04.016508Z","iopub.execute_input":"2025-09-03T17:21:04.016793Z","iopub.status.idle":"2025-09-03T17:21:04.494292Z","shell.execute_reply.started":"2025-09-03T17:21:04.016770Z","shell.execute_reply":"2025-09-03T17:21:04.493254Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Score break-down","metadata":{}},{"cell_type":"code","source":"from sklearn import metrics\nfrom sklearn.metrics import confusion_matrix\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ndef print_score_break_down(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    # Compute binary F1 (Target vs Non-Target)\n    y_true_bin = gt.sort_values(\"sequence_id\").reset_index(drop=True)['gesture'].isin(target_gestures).values\n    y_pred_bin = res_df.sort_values(\"sequence_id\").reset_index(drop=True)['gesture'].isin(target_gestures).values\n    \n    f1_binary = metrics.f1_score(y_true_bin, y_pred_bin, pos_label=True, zero_division=0, average='binary')\n    recall_bin = metrics.recall_score(y_true_bin, y_pred_bin, pos_label=True, zero_division=0, average='binary')\n    precision_bin = metrics.precision_score(y_true_bin, y_pred_bin, pos_label=True, zero_division=0, average='binary')\n    \n    # Compute macro F1 over all gesture classes\n    y_true_mc = gt.sort_values(\"sequence_id\").reset_index(drop=True)['gesture'].apply(\n        lambda x: x if x in target_gestures else 'non_target')\n    y_pred_mc = res_df.sort_values(\"sequence_id\").reset_index(drop=True)['gesture'].apply(\n        lambda x: x if x in target_gestures else 'non_target')\n    f1_macro = metrics.f1_score(y_true_mc, y_pred_mc, average='macro', zero_division=0)\n    recall_macro = metrics.recall_score(y_true_mc, y_pred_mc, average='macro', zero_division=0)\n    precision_macro = metrics.precision_score(y_true_mc, y_pred_mc, average='macro', zero_division=0)\n    \n    print(\"=\"*10 + \" Binary (target vs non-target) \" + \"=\"*10)\n    print(f\"  F1 score: {f1_binary:.4f}\")\n    print(f\"  Recall: {recall_bin:.4f}\")\n    print(f\"  Precision: {precision_bin:.4f}\")\n    \n    print(\"\\n\" + \"=\"*10 + \" Macro F1 over all gesture classes \" + \"=\"*10)\n    print(f\"  F1 score (macro): {f1_macro:.4f}\")\n    print(f\"  Recall (macro): {recall_macro:.4f}\")\n    print(f\"  Precision (macro): {precision_macro:.4f}\")\n    \n    print(\"\\n\" + \"=\"*10 + \" Folds\" + \"=\"*10)\n    folds = res_df['fold'].unique()\n    \n    for fold in folds:\n        fold_pred = res_df[res_df['fold'] == fold].sort_values(\"sequence_id\").reset_index(drop=True)\n        fold_gt = gt[gt['sequence_id'].isin(fold_pred['sequence_id'])].sort_values(\"sequence_id\").reset_index(drop=True)\n        \n        y_true_bin = fold_gt['gesture'].isin(target_gestures).values\n        y_pred_bin = fold_pred['gesture'].isin(target_gestures).values\n        \n        f1_binary_fold = metrics.f1_score(y_true_bin, y_pred_bin, pos_label=True, zero_division=0, average='binary')\n        recall_bin_fold = metrics.recall_score(y_true_bin, y_pred_bin, pos_label=True, zero_division=0, average='binary')\n        precision_bin_fold = metrics.precision_score(y_true_bin, y_pred_bin, pos_label=True, zero_division=0, average='binary')\n        \n        # Compute macro F1 over all gesture classes\n        y_true_mc_fold = fold_gt['gesture'].apply(lambda x: x if x in target_gestures else 'non_target')\n        y_pred_mc_fold = fold_pred['gesture'].apply(lambda x: x if x in target_gestures else 'non_target')\n        f1_macro_fold = metrics.f1_score(y_true_mc_fold, y_pred_mc_fold, average='macro', zero_division=0)\n        recall_macro_fold = metrics.recall_score(y_true_mc_fold, y_pred_mc_fold, average='macro', zero_division=0)\n        precision_macro_fold = metrics.precision_score(y_true_mc_fold, y_pred_mc_fold, average='macro', zero_division=0)\n        \n        print(f\"  Fold {fold} - Binary F1: {f1_binary_fold:.4f}, Bin Recall: {recall_bin_fold:.4f}, Bin Precision: {precision_bin_fold:.4f}\" +\n              f\" | Macro F1: {f1_macro_fold:.4f}, Macro Recall: {recall_macro_fold:.4f}, Macro Precision: {precision_macro_fold:.4f}\")\n\nprint_score_break_down(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nprint_score_break_down(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:04.495450Z","iopub.execute_input":"2025-09-03T17:21:04.495802Z","iopub.status.idle":"2025-09-03T17:21:06.543810Z","shell.execute_reply.started":"2025-09-03T17:21:04.495767Z","shell.execute_reply":"2025-09-03T17:21:06.542305Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_binary_cm(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    # Get labels in a consistent order\n    labels = sorted(list(set(res_df['label_bin']) | set(res_df['pred_bin'])))\n    \n    # Compute confusion matrix (counts)\n    cm = confusion_matrix(res_df['label_bin'], res_df['pred_bin'], labels=labels)\n    \n    # Normalized confusion matrix\n    cm_norm = cm.astype(float) / cm.sum(axis=1)[:, np.newaxis]\n    \n    # Plot side-by-side\n    fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n    \n    # --- Counts ---\n    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',\n                xticklabels=labels, yticklabels=labels, ax=axes[0])\n    axes[0].set_xlabel(\"Predicted\")\n    axes[0].set_ylabel(\"True\")\n    axes[0].set_title(\"target vs. non-target (counts)\")\n    \n    # --- Normalized ---\n    sns.heatmap(cm_norm, annot=True, fmt=\".3f\", cmap='Blues',\n                xticklabels=labels, yticklabels=labels, ax=axes[1])\n    axes[1].set_xlabel(\"Predicted\")\n    axes[1].set_ylabel(\"True\")\n    axes[1].set_title(\"target vs. non-target (normalized)\")\n    \n    plt.tight_layout()\n    plt.show()\n\nprint(\"\"\"\n#########################\n# BINARY CLASSIFICATION #\n#########################\n\"\"\")\nplot_binary_cm(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nplot_binary_cm(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:06.545518Z","iopub.execute_input":"2025-09-03T17:21:06.546087Z","iopub.status.idle":"2025-09-03T17:21:07.932954Z","shell.execute_reply.started":"2025-09-03T17:21:06.546049Z","shell.execute_reply":"2025-09-03T17:21:07.931519Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"`IMU only` seems to perform slightly better than `IMU-THM-TOF` in binary classification task.","metadata":{}},{"cell_type":"code","source":"def plot_gestures_cm(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    # Get labels in a consistent order\n    labels = sorted(list(set(res_df['label']) | set(res_df['pred'])))\n    \n    # Compute confusion matrix (counts)\n    cm = confusion_matrix(res_df['label'], res_df['pred'], labels=labels)\n    \n    # Normalized confusion matrix\n    cm_norm = cm.astype(float) / cm.sum(axis=1)[:, np.newaxis]\n    \n    # Plot side-by-side\n    fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n    \n    # --- Counts ---\n    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',\n                xticklabels=labels, yticklabels=labels, ax=axes[0])\n    axes[0].set_xlabel(\"Predicted\")\n    axes[0].set_ylabel(\"True\")\n    axes[0].set_title(\"gesture classification (counts)\")\n    \n    # --- Normalized ---\n    sns.heatmap(cm_norm, annot=True, fmt=\".2f\", cmap='Blues',\n                xticklabels=labels, yticklabels=labels, ax=axes[1])\n    axes[1].set_xlabel(\"Predicted\")\n    axes[1].set_ylabel(\"True\")\n    axes[1].set_title(\"gesture classification (normalized)\")\n    \n    plt.tight_layout()\n    plt.show()\n\nprint(\"\"\"\n##########################\n# GESTURE CLASSIFICATION #\n##########################\n\"\"\")\nplot_gestures_cm(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nplot_gestures_cm(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:07.935546Z","iopub.execute_input":"2025-09-03T17:21:07.935884Z","iopub.status.idle":"2025-09-03T17:21:10.083823Z","shell.execute_reply.started":"2025-09-03T17:21:07.935855Z","shell.execute_reply":"2025-09-03T17:21:10.082652Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Some confusing pairs:\n* Eyebrow - pull hair & Eyelash - pull  hair\n* Neck - pinch skin\n* Neck - scratch\n* ...\n\n--> Prioritize targeted improvements for these specific pairs, such as collecting more representative training samples, refining feature extraction, or incorporating additional context cues.","metadata":{}},{"cell_type":"markdown","source":"## Missing data impact","metadata":{}},{"cell_type":"markdown","source":"### Performance vs. missing rotation","metadata":{}},{"cell_type":"code","source":"from scipy.stats import pearsonr\n\ndef print_perf_missing_rot(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    # Subset rotation columns\n    rot_cols = ['rot_w', 'rot_x', 'rot_y', 'rot_z']\n\n    # Count missing values per sequence\n    rotation_missing = df.groupby('sequence_id')[rot_cols].apply(\n        lambda x: x.isna().sum().sum() / x.size\n    ).reset_index(name='rotation_missing_fraction')\n    \n    # Add accuracy column (1 = correct, 0 = wrong)\n    res_df['correct_bin'] = (res_df['label_bin'] == res_df['pred_bin']).astype(int)\n    \n    # Add accuracy column (1 = correct, 0 = wrong)\n    res_df['correct'] = (res_df['label'] == res_df['pred']).astype(int)\n    \n    # Merge rotation missing info into res_df\n    res_with_missing = res_df.merge(rotation_missing, on='sequence_id', how='left')\n\n    # Calculate correlation and p-value\n    r_bin, p_bin = pearsonr(res_with_missing['rotation_missing_fraction'], res_with_missing['correct_bin'])\n    r_label, p_label = pearsonr(res_with_missing['rotation_missing_fraction'], res_with_missing['correct'])\n    print(\"  Pearson correlation\")\n    print(f\"\\tBinary accuracy: r = {r_bin:.3f}, p = {p_bin:.3g}\")\n    print(f\"\\tGesture accuracy:  r = {r_label:.3f}, p = {p_label:.3g}\")\n\n    print(\"\"\"\n    ---------------------\n    Binary classification\n    ---------------------\n    \"\"\")\n    ct = pd.crosstab(res_with_missing['rotation_missing_fraction'], res_with_missing['correct_bin'])\n    # Calculate percentage of correct_bin == 1 for each row\n    ct['%_correct'] = ct[1] / ct.sum(axis=1) * 100\n    print(ct.round(2))\n\n    print(\"\"\"\n    ----------------------\n    Gesture classification\n    ----------------------\n    \"\"\")\n    ct = pd.crosstab(res_with_missing['rotation_missing_fraction'], res_with_missing['correct'])\n    # Calculate percentage of correct_bin == 1 for each row\n    ct['%_correct'] = ct[1] / ct.sum(axis=1) * 100\n    print(ct.round(2))\n\nprint_perf_missing_rot(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nprint_perf_missing_rot(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:10.085012Z","iopub.execute_input":"2025-09-03T17:21:10.085427Z","iopub.status.idle":"2025-09-03T17:21:15.273475Z","shell.execute_reply.started":"2025-09-03T17:21:10.085391Z","shell.execute_reply":"2025-09-03T17:21:15.272242Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Both `IMU only` and `IMU-THM-TOF` models experience a performance drop when rotation data is missing. However, `IMU-THM-TOF` still outperforms `IMU only`, likely due to the compensatory effect of the additional THM and TOF features.","metadata":{}},{"cell_type":"markdown","source":"### Performance vs missing THM","metadata":{}},{"cell_type":"code","source":"def plot_perf_missing_thm(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    # Thermopile columns\n    thm_cols = [f'thm_{i}' for i in range(1, 6)]\n    \n    # Per sequence: check if any thermopile value is missing\n    thm_missing_flag = (\n        df.groupby('sequence_id')[thm_cols]\n          .apply(lambda x: x.isna().sum().sum() / x.size)\n          .reset_index(name='has_missing_thm')\n    )\n    # Add accuracy column (1 = correct, 0 = wrong)\n    res_df['correct_bin'] = (res_df['label_bin'] == res_df['pred_bin']).astype(int)\n    # Accuracy column\n    res_df['correct'] = (res_df['label'] == res_df['pred']).astype(int)\n    \n    # Merge\n    res_with_thm_flag = res_df.merge(thm_missing_flag, on='sequence_id', how='left')\n    # Calculate correlation and p-value\n    r_bin, p_bin = pearsonr(res_with_thm_flag['has_missing_thm'], res_with_thm_flag['correct_bin'])\n    r_label, p_label = pearsonr(res_with_thm_flag['has_missing_thm'], res_with_thm_flag['correct'])\n    # Plot\n    plt.figure(figsize=(12, 6))\n    \n    # Plot for correct_bin\n    plt.subplot(1, 2, 1)\n    sns.scatterplot(\n        data=res_with_thm_flag,\n        x='has_missing_thm',\n        y='correct_bin',\n        alpha=0.5\n    )\n    sns.regplot(\n        data=res_with_thm_flag,\n        x='has_missing_thm',\n        y='correct_bin',\n        scatter=False,\n        color='red'\n    )\n    plt.xlabel(\"Fraction of Missing thm Data\")\n    plt.ylabel(\"Correct Binary Prediction (1=True, 0=False)\")\n    plt.title(f\"Accuracy vs Missing thm Data\\nr = {r_bin:.3f}, p = {p_bin:.3g}\")\n    \n    # Plot for correct\n    plt.subplot(1, 2, 2)\n    sns.scatterplot(\n        data=res_with_thm_flag,\n        x='has_missing_thm',\n        y='correct',\n        alpha=0.5\n    )\n    sns.regplot(\n        data=res_with_thm_flag,\n        x='has_missing_thm',\n        y='correct',\n        scatter=False,\n        color='red'\n    )\n    plt.xlabel(\"Fraction of Missing thm Data\")\n    plt.ylabel(\"Correct Gesture Prediction (1=True, 0=False)\")\n    plt.title(f\"Accuracy vs Missing thm Data\\nr = {r_label:.3f}, p = {p_label:.3g}\")\n    \n    plt.tight_layout()\n    plt.show()\n\n    # Crosstab counts\n    print(\"\"\"\n    ---------------------\n    Binary classification\n    ---------------------\n    \"\"\")\n    ct = pd.crosstab(res_with_thm_flag['has_missing_thm'], res_with_thm_flag['correct_bin'])\n    # Calculate percentage of correct_bin == 1 for each row\n    ct['%_correct'] = ct[1] / ct.sum(axis=1) * 100\n    print(ct.round(2))\n\n    print(\"\"\"\n    ----------------------\n    Gesture classification\n    ----------------------\n    \"\"\")\n    ct = pd.crosstab(res_with_thm_flag['has_missing_thm'], res_with_thm_flag['correct'])\n    # Calculate percentage of correct_bin == 1 for each row\n    ct['%_correct'] = ct[1] / ct.sum(axis=1) * 100\n    print(ct.round(2))\n\nplot_perf_missing_thm(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nplot_perf_missing_thm(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:15.275010Z","iopub.execute_input":"2025-09-03T17:21:15.275736Z","iopub.status.idle":"2025-09-03T17:21:23.584442Z","shell.execute_reply.started":"2025-09-03T17:21:15.275694Z","shell.execute_reply":"2025-09-03T17:21:23.582003Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"When THM data is missing, the `IMU only` model outperforms `IMU-THM-TOF`. --> Ensemble strategy: assign higher weight to the `IMU only` model in scenarios where THM is unavailable.","metadata":{}},{"cell_type":"markdown","source":"### Performance vs. missing TOF","metadata":{}},{"cell_type":"code","source":"def plot_perf_missing_tof(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    tof_cols = [x for x in df.columns if x.startswith('tof_')]\n    \n    # Per sequence: check if any thermopile value is missing\n    tof_missing = (\n        df.groupby('sequence_id')[tof_cols]\n          .apply(lambda x: x.isna().sum().sum() / x.size)\n          .reset_index(name='tof_missing')\n    )\n    \n    # Add accuracy column (1 = correct, 0 = wrong)\n    res_df['correct_bin'] = (res_df['label_bin'] == res_df['pred_bin']).astype(int)\n    res_df['correct'] = (res_df['label'] == res_df['pred']).astype(int)\n    \n    # Merge\n    res_with_tof = res_df.merge(tof_missing, on='sequence_id', how='left')\n\n    # Calculate correlation and p-value\n    r_bin, p_bin = pearsonr(res_with_tof['tof_missing'], res_with_tof['correct_bin'])\n    r_label, p_label = pearsonr(res_with_tof['tof_missing'], res_with_tof['correct'])\n    \n    # Plot\n    plt.figure(figsize=(12, 6))\n    \n    # Plot for correct_bin\n    plt.subplot(1, 2, 1)\n    sns.scatterplot(\n        data=res_with_tof,\n        x='tof_missing',\n        y='correct_bin',\n        alpha=0.5\n    )\n    sns.regplot(\n        data=res_with_tof,\n        x='tof_missing',\n        y='correct_bin',\n        scatter=False,\n        color='red'\n    )\n    plt.xlabel(\"Fraction of Missing tof Data\")\n    plt.ylabel(\"Correct Binary Prediction (1=True, 0=False)\")\n    plt.title(f\"Accuracy vs Missing tof Data\\nr = {r_bin:.3f}, p = {p_bin:.3g}\")\n    \n    # Plot for correct\n    plt.subplot(1, 2, 2)\n    sns.scatterplot(\n        data=res_with_tof,\n        x='tof_missing',\n        y='correct',\n        alpha=0.5\n    )\n    sns.regplot(\n        data=res_with_tof,\n        x='tof_missing',\n        y='correct',\n        scatter=False,\n        color='red'\n    )\n    plt.xlabel(\"Fraction of Missing tof Data\")\n    plt.ylabel(\"Correct Gesture Prediction (1=True, 0=False)\")\n    plt.title(f\"Accuracy vs Missing tof Data\\nr = {r_label:.3f}, p = {p_label:.3g}\")\n    \n    plt.tight_layout()\n    plt.show()\n\n    # Crosstab counts\n    print(\"\"\"\n    ---------------------\n    Binary classification\n    ---------------------\n    \"\"\")\n    ct = pd.crosstab(res_with_tof['tof_missing'], res_with_tof['correct_bin'])\n    # Calculate percentage of correct_bin == 1 for each row\n    ct['%_correct'] = ct[1] / ct.sum(axis=1) * 100\n    print(ct.round(2))\n\n    print(\"\"\"\n    ----------------------\n    Gesture classification\n    ----------------------\n    \"\"\")\n    ct = pd.crosstab(res_with_tof['tof_missing'], res_with_tof['correct_bin'])\n    # Calculate percentage of correct_bin == 1 for each row\n    ct['%_correct'] = ct[1] / ct.sum(axis=1) * 100\n    print(ct.round(2))\n\nplot_perf_missing_tof(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nplot_perf_missing_tof(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:23.587574Z","iopub.execute_input":"2025-09-03T17:21:23.588414Z","iopub.status.idle":"2025-09-03T17:21:41.801558Z","shell.execute_reply.started":"2025-09-03T17:21:23.588342Z","shell.execute_reply":"2025-09-03T17:21:41.800198Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Similary, when TOF data is missing, the `IMU only` model outperforms `IMU-THM-TOF`. --> Ensemble strategy: assign higher weight to the `IMU only` model in scenarios where TOF is unavailable.","metadata":{}},{"cell_type":"markdown","source":"## Performance vs. demographics","metadata":{}},{"cell_type":"markdown","source":"### Performance vs. adult/child","metadata":{}},{"cell_type":"code","source":"import scipy.stats as stats\n\ndef print_perf_adult_child(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n\n    merged_df = res_df.copy()\n    merged_df[\"subject\"] = merged_df[\"sequence_id\"].map(df.drop_duplicates([\"subject\", \"sequence_id\"]).set_index(\"sequence_id\").to_dict()[\"subject\"])\n    merged_df = merged_df.merge(dem_df, on='subject', how='left')\n    \n    print(\"\\nBinary Performance by Adult/Child\")\n    # Crosstab\n    ct = pd.crosstab(merged_df['adult_child'], merged_df['correct_bin'])\n    # Chi-square test\n    chi2, p, dof, ex = stats.chi2_contingency(ct)\n    print(f\"\\nChi-square test: chi2={chi2:.3f}, p-value={p:.3g}\")\n    # Add percentage of correct predictions in each group\n    ct['percent_correct'] = ct[1] / ct.sum(axis=1) * 100\n    \n    # Map adult_child to labels for readability\n    ct.index = ct.index.map({0: \"Child\", 1: \"Adult\"})\n    print(ct)\n    \n    print(\"=\"*50)\n    print(\"\\nMacro Performance by Adult/Child\")\n    # Crosstab\n    ct = pd.crosstab(merged_df['adult_child'], merged_df['correct'])\n    # Chi-square test\n    chi2, p, dof, ex = stats.chi2_contingency(ct)\n    print(f\"\\nChi-square test: chi2={chi2:.3f}, p-value={p:.3g}\")\n    # Add percentage of correct predictions in each group\n    ct['percent_correct'] = ct[1] / ct.sum(axis=1) * 100\n    \n    # Map adult_child to labels for readability\n    ct.index = ct.index.map({0: \"Child\", 1: \"Adult\"})\n    \n    print(ct)\n\nprint_perf_adult_child(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nprint_perf_adult_child(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:41.802855Z","iopub.execute_input":"2025-09-03T17:21:41.803198Z","iopub.status.idle":"2025-09-03T17:21:44.322932Z","shell.execute_reply.started":"2025-09-03T17:21:41.803143Z","shell.execute_reply":"2025-09-03T17:21:44.321636Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"`IMU only` model performs better on adults than on children while `IMU-THM-TOF` doesn't show significant differences.","metadata":{}},{"cell_type":"markdown","source":"### Performance vs. age","metadata":{}},{"cell_type":"code","source":"from scipy.stats import pearsonr, spearmanr\n\ndef plot_perf_param(res_df, exp_name, param):\n    print(f\"EXPERIMENT: {exp_name}\")\n    merged_df = res_df.copy()\n    merged_df[\"subject\"] = merged_df[\"sequence_id\"].map(df.drop_duplicates([\"subject\", \"sequence_id\"]).set_index(\"sequence_id\").to_dict()[\"subject\"])\n    merged_df = merged_df.merge(dem_df, on='subject', how='left')\n    # Drop rows with missing height\n    df_param = merged_df.dropna(subset=[param])\n    \n    # Pearson correlation\n    pearson_r, pearson_p = pearsonr(df_param[param], df_param['correct_bin'])\n    print(f\"Pearson r (binary) = {pearson_r:.3f}, p = {pearson_p:.4g}\")\n    \n    # Spearman correlation (rank-based, more robust to non-linearity)\n    spearman_r, spearman_p = spearmanr(df_param[param], df_param['correct_bin'])\n    print(f\"Spearman r (binary) = {spearman_r:.3f}, p = {spearman_p:.4g}\")\n    \n    # Pearson correlation\n    pearson_r, pearson_p = pearsonr(df_param[param], df_param['correct'])\n    print(f\"Pearson r (macro) = {pearson_r:.3f}, p = {pearson_p:.4g}\")\n    \n    # Spearman correlation (rank-based, more robust to non-linearity)\n    spearman_r, spearman_p = spearmanr(df_param[param], df_param['correct'])\n    print(f\"Spearman r (macro) = {spearman_r:.3f}, p = {spearman_p:.4g}\")\n    \n    df_param_perf = merged_df.dropna(subset=[param, 'correct_bin'])\n    \n    plt.figure(figsize=(10, 5))\n    \n    # Left: Boxplot + swarmplot for detail\n    plt.subplot(1, 2, 1)\n    sns.kdeplot(data=df_param_perf[df_param_perf['correct_bin'] == 1], x=param, label='Correct', fill=True, alpha=0.5)\n    sns.kdeplot(data=df_param_perf[df_param_perf['correct_bin'] == 0], x=param, label='Wrong', fill=True, alpha=0.5)\n    plt.xlabel(param)\n    plt.ylabel(\"Density\")\n    plt.title(\"binary (correct vs wrong)\")\n    \n    # Right: KDE (density) plot\n    plt.subplot(1, 2, 2)\n    sns.kdeplot(data=df_param_perf[df_param_perf['correct'] == 1], x=param, label='Correct', fill=True, alpha=0.5)\n    sns.kdeplot(data=df_param_perf[df_param_perf['correct'] == 0], x=param, label='Wrong', fill=True, alpha=0.5)\n    plt.xlabel(param)\n    plt.ylabel(\"Density\")\n    plt.title(\"macro (correct vs wrong)\")\n    plt.legend()\n    \n    plt.tight_layout()\n    plt.show()\n\nplot_perf_param(imu_only, exp_name=\"IMU only\", param=\"age\")\nprint(\"\\n\" + \"*\"*80)\nplot_perf_param(imu_thm_tof, exp_name=\"IMU-THM-TOF\", param=\"age\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:44.324232Z","iopub.execute_input":"2025-09-03T17:21:44.324614Z","iopub.status.idle":"2025-09-03T17:21:47.984572Z","shell.execute_reply.started":"2025-09-03T17:21:44.324581Z","shell.execute_reply":"2025-09-03T17:21:47.981290Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Performance vs. sex","metadata":{}},{"cell_type":"code","source":"def print_perf_sex(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    merged_df = res_df.copy()\n    merged_df[\"subject\"] = merged_df[\"sequence_id\"].map(df.drop_duplicates([\"subject\", \"sequence_id\"]).set_index(\"sequence_id\").to_dict()[\"subject\"])\n    merged_df = merged_df.merge(dem_df, on='subject', how='left')\n    \n    print(\"\\nBinary Performance by female/male\")\n    # Crosstab\n    ct = pd.crosstab(merged_df['sex'], merged_df['correct_bin'])\n    # Chi-square test\n    chi2, p, dof, ex = stats.chi2_contingency(ct)\n    print(f\"\\nChi-square test: chi2={chi2:.3f}, p-value={p:.3g}\")\n    # Add percentage of correct predictions in each group\n    ct['percent_correct'] = ct[1] / ct.sum(axis=1) * 100\n    \n    # Map adult_child to labels for readability\n    ct.index = ct.index.map({0: \"female\", 1: \"male\"})\n    print(ct)\n    \n    print(\"=\"*50)\n    print(\"\\nMacro Performance by female/male\")\n    # Crosstab\n    ct = pd.crosstab(merged_df['sex'], merged_df['correct'])\n    # Chi-square test\n    chi2, p, dof, ex = stats.chi2_contingency(ct)\n    print(f\"\\nChi-square test: chi2={chi2:.3f}, p-value={p:.3g}\")\n    # Add percentage of correct predictions in each group\n    ct['percent_correct'] = ct[1] / ct.sum(axis=1) * 100\n    \n    # Map adult_child to labels for readability\n    ct.index = ct.index.map({0: \"female\", 1: \"male\"})\n    \n    print(ct)\n\nprint_perf_sex(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nprint_perf_sex(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:47.986081Z","iopub.execute_input":"2025-09-03T17:21:47.986575Z","iopub.status.idle":"2025-09-03T17:21:50.492213Z","shell.execute_reply.started":"2025-09-03T17:21:47.986546Z","shell.execute_reply":"2025-09-03T17:21:50.491251Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Performance vs. handedness","metadata":{}},{"cell_type":"code","source":"def print_perf_handedness(res_df, exp_name):\n    print(f\"EXPERIMENT: {exp_name}\")\n    merged_df = res_df.copy()\n    merged_df[\"subject\"] = merged_df[\"sequence_id\"].map(df.drop_duplicates([\"subject\", \"sequence_id\"]).set_index(\"sequence_id\").to_dict()[\"subject\"])\n    merged_df = merged_df.merge(dem_df, on='subject', how='left')\n    \n    print(\"\\nBinary Performance by handedness\")\n    # Crosstab\n    ct = pd.crosstab(merged_df['handedness'], merged_df['correct_bin'])\n    # Chi-square test\n    chi2, p, dof, ex = stats.chi2_contingency(ct)\n    print(f\"\\nChi-square test: chi2={chi2:.3f}, p-value={p:.3g}\")\n    # Add percentage of correct predictions in each group\n    ct['percent_correct'] = ct[1] / ct.sum(axis=1) * 100\n    \n    # Map adult_child to labels for readability\n    ct.index = ct.index.map({0: \"left\", 1: \"right\"})\n    print(ct)\n    \n    print(\"=\"*50)\n    print(\"\\nMacro Performance by handedness\")\n    # Crosstab\n    ct = pd.crosstab(merged_df['handedness'], merged_df['correct'])\n    # Chi-square test\n    chi2, p, dof, ex = stats.chi2_contingency(ct)\n    print(f\"\\nChi-square test: chi2={chi2:.3f}, p-value={p:.3g}\")\n    # Add percentage of correct predictions in each group\n    ct['percent_correct'] = ct[1] / ct.sum(axis=1) * 100\n    \n    # Map adult_child to labels for readability\n    ct.index = ct.index.map({0: \"left\", 1: \"right\"})\n    \n    print(ct)\n\nprint_perf_handedness(imu_only, exp_name=\"IMU only\")\nprint(\"\\n\" + \"*\"*80)\nprint_perf_handedness(imu_thm_tof, exp_name=\"IMU-THM-TOF\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:50.493376Z","iopub.execute_input":"2025-09-03T17:21:50.493721Z","iopub.status.idle":"2025-09-03T17:21:53.069068Z","shell.execute_reply.started":"2025-09-03T17:21:50.493684Z","shell.execute_reply":"2025-09-03T17:21:53.067333Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Both models perform better on adults, males, right hands compared to children, females, and left hands.\nThis disparity may not stem directly from age or gender but could be influenced by the co-occurrence of these groups with missing rotation, THM, or TOF data. Further investigation is needed to disentangle the effect of demographic factors from missing data patterns.","metadata":{}},{"cell_type":"markdown","source":"### Performance with height","metadata":{}},{"cell_type":"code","source":"plot_perf_param(imu_only, exp_name=\"IMU only\", param=\"height_cm\")\nprint(\"\\n\" + \"*\"*80)\nplot_perf_param(imu_thm_tof, exp_name=\"IMU-THM-TOF\", param=\"height_cm\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:53.070908Z","iopub.execute_input":"2025-09-03T17:21:53.071662Z","iopub.status.idle":"2025-09-03T17:21:56.657582Z","shell.execute_reply.started":"2025-09-03T17:21:53.071619Z","shell.execute_reply":"2025-09-03T17:21:56.656277Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Both models have no significant difference for performance vs. height.","metadata":{}},{"cell_type":"markdown","source":"### Performance vs should_to_wrist","metadata":{}},{"cell_type":"code","source":"plot_perf_param(imu_only, exp_name=\"IMU only\", param=\"shoulder_to_wrist_cm\")\nprint(\"\\n\" + \"*\"*80)\nplot_perf_param(imu_thm_tof, exp_name=\"IMU-THM-TOF\", param=\"shoulder_to_wrist_cm\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:21:56.659018Z","iopub.execute_input":"2025-09-03T17:21:56.659354Z","iopub.status.idle":"2025-09-03T17:22:00.332493Z","shell.execute_reply.started":"2025-09-03T17:21:56.659330Z","shell.execute_reply":"2025-09-03T17:22:00.330278Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Performance vs elbow-to-wrist","metadata":{}},{"cell_type":"code","source":"plot_perf_param(imu_only, exp_name=\"IMU only\", param=\"elbow_to_wrist_cm\")\nprint(\"\\n\" + \"*\"*80)\nplot_perf_param(imu_thm_tof, exp_name=\"IMU-THM-TOF\", param=\"elbow_to_wrist_cm\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-03T17:22:00.334100Z","iopub.execute_input":"2025-09-03T17:22:00.334549Z","iopub.status.idle":"2025-09-03T17:22:04.117862Z","shell.execute_reply.started":"2025-09-03T17:22:00.334510Z","shell.execute_reply":"2025-09-03T17:22:04.116792Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Conclusion","metadata":{}},{"cell_type":"markdown","source":"1. Missing rotation values reduce performance for both the `IMU only` and `IMU-THM-TOF` models, with a stronger impact on the IMU-only model.\n2. When THM and TOF data are missing, the `IMU only` model outperforms the `IMU-THM-TOF` model --> asign higher weights for `IMU_only` model when THM & TOF are not available.\n3. Both models achieve higher accuracy on adults, males, and right hands compared to children, females, and left hands. However, this gap may be influenced by the co-occurrence of missing rotation/THM/TOF data with these groups. Further analysis is needed to separate demographic effects from missing-data effects.\n4. No significant performance differences are observed across height, shoulder-to-wrist length, or elbow-to-wrist length.","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true,"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null}]}