{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"},{"sourceId":12245948,"sourceType":"datasetVersion","datasetId":7715985},{"sourceId":242954653,"sourceType":"kernelVersion"}],"dockerImageVersionId":31041,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Thanks https://www.kaggle.com/rktqwe for the previous features. I was able to find **9** more features that improve score.\n\n### New Feature: Angular Velocity from Quaternion Derivatives\n\nI've added a new function, `calculate_angular_velocity_from_quat`, which computes angular velocity (ω) based on quaternion data (rot_x, rot_y, rot_z, rot_w).\n\n**Why is this needed?**\nWhile gyroscopes directly measure angular velocity, deriving it from quaternion data provides a smoother and more generalized estimate, especially when IMU filters are applied in the system. This metric helps to more accurately capture subtle differences in rotational movements, which is crucial for detailed gesture analysis and recognition in human-computer interaction applications.\n\n**How does it work?**\n* **Inputs:** The function takes a sequence of quaternions (rot_x, rot_y, rot_z, rot_w) and a time step (time_delta, defaulting to 1/200 seconds, corresponding to a sampling rate of 200 Hz).\n* **Step-by-step Calculation:** For each pair of successive quaternions ($q(t)$ and $q(t+Δt)$), it:\n    * Converts the quaternions into `scipy.spatial.transform.Rotation` objects.\n    * Calculates the relative rotation between $q(t)$ and $q(t+Δt)$ as $Δrot = q(t)^{(-1)} * q(t+Δt)$.\n    * Converts this relative rotation into a Rotation Vector, which represents the axis of rotation multiplied by the angle of rotation.\n    * Divides the resulting Rotation Vector by `time_delta` to obtain the angular velocity.\n\nThree new features have been included in the `imu_engineered_features` list:\n* `angular_vel_x`\n* `angular_vel_y`\n* `angular_vel_z`\n\n### New Feature: Angular Jerk\n\nAngular Jerk is the time derivative of Angular Velocity. It quantifies the smoothness or abruptness of rotational movements, providing insights beyond just speed.\n\n**Why is this needed?**\nSimilar to linear jerk, angular jerk helps in capturing the quality of rotational movements. For complex gestures involving twists or turns, the smoothness of rotation (e.g., a fluid wrist rotation versus an abrupt flick) can be a key differentiator for gesture classification. It can also indicate fatigue or lack of control in movements.\n\n**How does it work?**\nAngular jerk is calculated by differentiating the angular velocity (derived from quaternions) over time.\n\nThree additional features have been included in the `imu_engineered_features` list:\n* `angular_jerk_x`\n* `angular_jerk_y`\n* `angular_jerk_z`\n\n### New Feature: Angular Snap\n\nAngular Snap (also known as angular crackle or angular jounce) is the time derivative of Angular Jerk, and thus the second derivative of angular velocity. It represents the rate of change of angular jerk.\n\n**Why is this needed?**\nAngular snap provides an even more fine-grained understanding of rapid and sudden changes in rotational motion. While angular jerk captures the \"jerkiness\" of a movement, angular snap highlights the *rate at which that jerkiness changes*. This can be particularly useful for distinguishing very quick, sharp, or impactful rotational gestures, where even subtle variations in the onset or cessation of the jerk motion are critical. It helps to characterize highly dynamic and impulsive rotational components of a gesture, which might be missed by lower-order derivatives.\n\n**How does it work?**\nAngular snap is calculated by differentiating the angular jerk over time.\n\nThree new features have been included in the `imu_engineered_features` list:\n* `angular_snap_x`\n* `angular_snap_y`\n* `angular_snap_z`\n\n### Overall Impact\n\nFeatures derived from quaternion-based angular dynamics, including angular velocity, jerk, and snap, provide superior accuracy and reliability for complex gesture recognition. These capabilities enable the recognition of more intricate, nuanced, and precise gestures. This goes beyond simple actions, allowing for the recognition of the manner of action (e.g., a smooth, controlled movement versus a hesitant, jerky one, or a very sharp, sudden rotational impact). This facilitates the creation of more intuitive and responsive user interfaces, and potentially personalization based on individual movement styles.\n\n","metadata":{}},{"cell_type":"markdown","source":"## I hope this feature engineering will be useful to someone else.\n## Please vote for this notebook!","metadata":{}},{"cell_type":"markdown","source":"### 》》》**Importing the necessary Libraries**","metadata":{}},{"cell_type":"code","source":"import os, json, joblib, numpy as np, pandas as pd\nfrom pathlib import Path\nimport warnings \nwarnings.filterwarnings(\"ignore\")\n\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\n\nfrom tensorflow.keras.utils import Sequence, to_categorical, pad_sequences\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.layers import (\n    Input, Conv1D, BatchNormalization, Activation, add, MaxPooling1D, Dropout,\n    Bidirectional, LSTM, GlobalAveragePooling1D, Dense, Multiply, Reshape,\n    Lambda, Concatenate, GRU, GaussianNoise\n)\nfrom tensorflow.keras.regularizers import l2\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom tensorflow.keras import backend as K\nimport tensorflow as tf\nimport polars as pl\nfrom sklearn.model_selection import StratifiedGroupKFold\nfrom scipy.spatial.transform import Rotation as R\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:05.743313Z","iopub.execute_input":"2025-06-25T16:34:05.743621Z","iopub.status.idle":"2025-06-25T16:34:22.056097Z","shell.execute_reply.started":"2025-06-25T16:34:05.743601Z","shell.execute_reply":"2025-06-25T16:34:22.055533Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Fix Seed**","metadata":{}},{"cell_type":"code","source":"import random\ndef seed_everything(seed):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    tf.experimental.numpy.random.seed(seed)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\nseed_everything(seed=42)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:22.057117Z","iopub.execute_input":"2025-06-25T16:34:22.057642Z","iopub.status.idle":"2025-06-25T16:34:22.062386Z","shell.execute_reply.started":"2025-06-25T16:34:22.057615Z","shell.execute_reply":"2025-06-25T16:34:22.061607Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Configuration**","metadata":{}},{"cell_type":"code","source":"# (Competition metric will only be imported when TRAINing)\nTRAIN = False                     # ← set to True when you want to train\nRAW_DIR = Path(\"/kaggle/input/cmi-detect-behavior-with-sensor-data\")\nPRETRAINED_DIR = Path(\"/kaggle/input/quit-diff2\")  # used when TRAIN=False\nEXPORT_DIR = Path(\"./\")                                    # artefacts will be saved here\nBATCH_SIZE = 64\nPAD_PERCENTILE = 95\nLR_INIT = 5e-4\nWD = 3e-3\nMIXUP_ALPHA = 0.4\nEPOCHS = 160\nPATIENCE = 40\n\n\nprint(\"▶ imports ready · tensorflow\", tf.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:22.063516Z","iopub.execute_input":"2025-06-25T16:34:22.063843Z","iopub.status.idle":"2025-06-25T16:34:22.085773Z","shell.execute_reply.started":"2025-06-25T16:34:22.063825Z","shell.execute_reply":"2025-06-25T16:34:22.085179Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Utility Functions**","metadata":{}},{"cell_type":"code","source":"#Tensor Manipulations\ndef time_sum(x):\n    return K.sum(x, axis=1)\n\ndef squeeze_last_axis(x):\n    return tf.squeeze(x, axis=-1)\n\ndef expand_last_axis(x):\n    return tf.expand_dims(x, axis=-1)\n\ndef se_block(x, reduction=8):\n    ch = x.shape[-1]\n    se = GlobalAveragePooling1D()(x)\n    se = Dense(ch // reduction, activation='relu')(se)\n    se = Dense(ch, activation='sigmoid')(se)\n    se = Reshape((1, ch))(se)\n    return Multiply()([x, se])\n\n# Residual CNN Block with SE\ndef residual_se_cnn_block(x, filters, kernel_size, pool_size=2, drop=0.3, wd=1e-4):\n    shortcut = x\n    for _ in range(2):\n        x = Conv1D(filters, kernel_size, padding='same', use_bias=False,\n                   kernel_regularizer=l2(wd))(x)\n        x = BatchNormalization()(x)\n        x = Activation('relu')(x)\n    x = se_block(x)\n    if shortcut.shape[-1] != filters:\n        shortcut = Conv1D(filters, 1, padding='same', use_bias=False,\n                          kernel_regularizer=l2(wd))(shortcut)\n        shortcut = BatchNormalization()(shortcut)\n    x = add([x, shortcut])\n    x = Activation('relu')(x)\n    x = MaxPooling1D(pool_size)(x)\n    x = Dropout(drop)(x)\n    return x\n\ndef attention_layer(inputs):\n    score = Dense(1, activation='tanh')(inputs)\n    score = Lambda(squeeze_last_axis)(score)\n    weights = Activation('softmax')(score)\n    weights = Lambda(expand_last_axis)(weights)\n    context = Multiply()([inputs, weights])\n    context = Lambda(time_sum)(context)\n    return context","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:22.087613Z","iopub.execute_input":"2025-06-25T16:34:22.087813Z","iopub.status.idle":"2025-06-25T16:34:22.100806Z","shell.execute_reply.started":"2025-06-25T16:34:22.087798Z","shell.execute_reply":"2025-06-25T16:34:22.100115Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Data Helpers**","metadata":{}},{"cell_type":"code","source":"# Normalizes and cleans the time series sequence. \n\ndef preprocess_sequence(df_seq: pd.DataFrame, feature_cols: list[str], scaler: StandardScaler):\n    mat = df_seq[feature_cols].ffill().bfill().fillna(0).values\n    return scaler.transform(mat).astype('float32')\n\n# MixUp the data argumentation in order to regularize the neural network. \n\nclass MixupGenerator(Sequence):\n    def __init__(self, X, y, batch_size, alpha=0.2):\n        self.X, self.y = X, y\n        self.batch = batch_size\n        self.alpha = alpha\n        self.indices = np.arange(len(X))\n    def __len__(self):\n        return int(np.ceil(len(self.X) / self.batch))\n    def __getitem__(self, i):\n        idx = self.indices[i*self.batch:(i+1)*self.batch]\n        Xb, yb = self.X[idx], self.y[idx]\n        lam = np.random.beta(self.alpha, self.alpha)\n        perm = np.random.permutation(len(Xb))\n        X_mix = lam * Xb + (1-lam) * Xb[perm]\n        y_mix = lam * yb + (1-lam) * yb[perm]\n        return X_mix, y_mix\n    def on_epoch_end(self):\n        np.random.shuffle(self.indices)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:22.101386Z","iopub.execute_input":"2025-06-25T16:34:22.101584Z","iopub.status.idle":"2025-06-25T16:34:22.119270Z","shell.execute_reply.started":"2025-06-25T16:34:22.101546Z","shell.execute_reply":"2025-06-25T16:34:22.118735Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def remove_gravity_from_acc(acc_data, rot_data):\n\n    if isinstance(acc_data, pd.DataFrame):\n        acc_values = acc_data[['acc_x', 'acc_y', 'acc_z']].values\n    else:\n        acc_values = acc_data\n\n    if isinstance(rot_data, pd.DataFrame):\n        quat_values = rot_data[['rot_x', 'rot_y', 'rot_z', 'rot_w']].values\n    else:\n        quat_values = rot_data\n\n    num_samples = acc_values.shape[0]\n    linear_accel = np.zeros_like(acc_values)\n    \n    gravity_world = np.array([0, 0, 9.81])\n\n    for i in range(num_samples):\n        if np.all(np.isnan(quat_values[i])) or np.all(np.isclose(quat_values[i], 0)):\n            linear_accel[i, :] = acc_values[i, :] \n            continue\n\n        try:\n            rotation = R.from_quat(quat_values[i])\n            gravity_sensor_frame = rotation.apply(gravity_world, inverse=True)\n            linear_accel[i, :] = acc_values[i, :] - gravity_sensor_frame\n        except ValueError:\n             linear_accel[i, :] = acc_values[i, :]\n             \n    return linear_accel\n\ndef calculate_angular_velocity_from_quat(rot_data, time_delta=1/200): # Assuming 200Hz sampling rate\n    if isinstance(rot_data, pd.DataFrame):\n        quat_values = rot_data[['rot_x', 'rot_y', 'rot_z', 'rot_w']].values\n    else:\n        quat_values = rot_data\n\n    num_samples = quat_values.shape[0]\n    angular_vel = np.zeros((num_samples, 3))\n\n    for i in range(num_samples - 1):\n        q_t = quat_values[i]\n        q_t_plus_dt = quat_values[i+1]\n\n        if np.all(np.isnan(q_t)) or np.all(np.isclose(q_t, 0)) or \\\n           np.all(np.isnan(q_t_plus_dt)) or np.all(np.isclose(q_t_plus_dt, 0)):\n            continue\n\n        try:\n            rot_t = R.from_quat(q_t)\n            rot_t_plus_dt = R.from_quat(q_t_plus_dt)\n\n            # Calculate the relative rotation\n            delta_rot = rot_t.inv() * rot_t_plus_dt\n            \n            # Convert delta rotation to angular velocity vector\n            # The rotation vector (Euler axis * angle) scaled by 1/dt\n            # is a good approximation for small delta_rot\n            angular_vel[i, :] = delta_rot.as_rotvec() / time_delta\n        except ValueError:\n            # If quaternion is invalid, angular velocity remains zero\n            pass\n            \n    return angular_vel","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:22.119982Z","iopub.execute_input":"2025-06-25T16:34:22.120302Z","iopub.status.idle":"2025-06-25T16:34:22.142098Z","shell.execute_reply.started":"2025-06-25T16:34:22.120278Z","shell.execute_reply":"2025-06-25T16:34:22.141511Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Model Definition - Two Branch Architecture**","metadata":{}},{"cell_type":"code","source":"def build_two_branch_model(pad_len, imu_dim, tof_dim, n_classes, wd=1e-4):\n    inp = Input(shape=(pad_len, imu_dim+tof_dim))\n    imu = Lambda(lambda t: t[:, :, :imu_dim])(inp)\n    tof = Lambda(lambda t: t[:, :, imu_dim:])(inp)\n\n    # IMU deep branch\n    x1 = residual_se_cnn_block(imu, 64, 3, drop=0.1, wd=wd)\n    x1 = residual_se_cnn_block(x1, 128, 5, drop=0.1, wd=wd)\n\n    # TOF/Thermal lighter branch\n    x2 = Conv1D(64, 3, padding='same', use_bias=False, kernel_regularizer=l2(wd))(tof)\n    x2 = BatchNormalization()(x2); x2 = Activation('relu')(x2)\n    x2 = MaxPooling1D(2)(x2); x2 = Dropout(0.2)(x2)\n    x2 = Conv1D(128, 3, padding='same', use_bias=False, kernel_regularizer=l2(wd))(x2)\n    x2 = BatchNormalization()(x2); x2 = Activation('relu')(x2)\n    x2 = MaxPooling1D(2)(x2); x2 = Dropout(0.2)(x2)\n\n    merged = Concatenate()([x1, x2])\n\n    xa = Bidirectional(LSTM(128, return_sequences=True, kernel_regularizer=l2(wd)))(merged)\n    xb = Bidirectional(GRU(128, return_sequences=True, kernel_regularizer=l2(wd)))(merged)\n    xc = GaussianNoise(0.09)(merged)\n    xc = Dense(16, activation='elu')(xc)\n    \n    x = Concatenate()([xa, xb, xc])\n    x = Dropout(0.4)(x)\n    x = attention_layer(x)\n\n    for units, drop in [(256, 0.5), (128, 0.3)]:\n        x = Dense(units, use_bias=False, kernel_regularizer=l2(wd))(x)\n        x = BatchNormalization()(x); x = Activation('relu')(x)\n        x = Dropout(drop)(x)\n\n    out = Dense(n_classes, activation='softmax', kernel_regularizer=l2(wd))(x)\n    return Model(inp, out)\n\ntmp_model = build_two_branch_model(127,7,325,18)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:22.142744Z","iopub.execute_input":"2025-06-25T16:34:22.142928Z","iopub.status.idle":"2025-06-25T16:34:25.054968Z","shell.execute_reply.started":"2025-06-25T16:34:22.142914Z","shell.execute_reply":"2025-06-25T16:34:25.054387Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Training / Inference Pipeline**","metadata":{}},{"cell_type":"code","source":"if TRAIN:\n    print(\"▶ TRAIN MODE – loading dataset …\")\n    df = pd.read_csv(RAW_DIR / \"train.csv\")\n\n    train_dem_df = pd.read_csv(RAW_DIR / \"train_demographics.csv\")\n    df_for_groups = pd.merge(df.copy(), train_dem_df, on='subject', how='left')\n\n    le = LabelEncoder()\n    df['gesture_int'] = le.fit_transform(df['gesture'])\n    np.save(EXPORT_DIR / \"gesture_classes.npy\", le.classes_)\n    gesture_classes = le.classes_\n\n    print(\"  Calculating base engineered IMU features (magnitude, angle)...\")\n    df['acc_mag'] = np.sqrt(df['acc_x']**2 + df['acc_y']**2 + df['acc_z']**2)\n    df['rot_angle'] = 2 * np.arccos(df['rot_w'].clip(-1, 1))\n    \n    print(\"  Calculating engineered IMU derivatives (jerk, angular velocity) for original acc_mag...\")\n    df['acc_mag_jerk'] = df.groupby('sequence_id')['acc_mag'].diff().fillna(0)\n    df['rot_angle_vel'] = df.groupby('sequence_id')['rot_angle'].diff().fillna(0)\n\n    print(\"  Removing gravity and calculating linear acceleration features...\")\n    \n    linear_accel_list = []\n    for _, group in df.groupby('sequence_id'):\n        acc_data_group = group[['acc_x', 'acc_y', 'acc_z']]\n        rot_data_group = group[['rot_x', 'rot_y', 'rot_z', 'rot_w']]\n        linear_accel_group = remove_gravity_from_acc(acc_data_group, rot_data_group)\n        linear_accel_list.append(pd.DataFrame(linear_accel_group, columns=['linear_acc_x', 'linear_acc_y', 'linear_acc_z'], index=group.index))\n    \n    df_linear_accel = pd.concat(linear_accel_list)\n    df = pd.concat([df, df_linear_accel], axis=1)\n\n    df['linear_acc_mag'] = np.sqrt(df['linear_acc_x']**2 + df['linear_acc_y']**2 + df['linear_acc_z']**2)\n    df['linear_acc_mag_jerk'] = df.groupby('sequence_id')['linear_acc_mag'].diff().fillna(0)\n\n    print(\"  Calculating angular velocity from quaternion derivatives...\")\n    angular_vel_list = []\n    for _, group in df.groupby('sequence_id'):\n        rot_data_group = group[['rot_x', 'rot_y', 'rot_z', 'rot_w']]\n        angular_vel_group = calculate_angular_velocity_from_quat(rot_data_group)\n        angular_vel_list.append(pd.DataFrame(angular_vel_group, columns=['angular_vel_x', 'angular_vel_y', 'angular_vel_z'], index=group.index))\n    \n    df_angular_vel = pd.concat(angular_vel_list)\n    df = pd.concat([df, df_angular_vel], axis=1)\n\n    print(\"  Calculating angular jerk from angular velocity...\")\n    df['angular_jerk_x'] = df.groupby('sequence_id')['angular_vel_x'].diff().fillna(0)\n    df['angular_jerk_y'] = df.groupby('sequence_id')['angular_vel_y'].diff().fillna(0)\n    df['angular_jerk_z'] = df.groupby('sequence_id')['angular_vel_z'].diff().fillna(0)\n\n    print(\"  Calculating angular snap from angular jerk...\")\n    df['angular_snap_x'] = df.groupby('sequence_id')['angular_jerk_x'].diff().fillna(0)\n    df['angular_snap_y'] = df.groupby('sequence_id')['angular_jerk_y'].diff().fillna(0)\n    df['angular_snap_z'] = df.groupby('sequence_id')['angular_jerk_z'].diff().fillna(0)\n\n    meta_cols = { } # This was an empty dict in your provided code, keeping it as is.\n\n    imu_cols_base = ['linear_acc_x', 'linear_acc_y', 'linear_acc_z']\n    imu_cols_base.extend([c for c in df.columns if c.startswith('rot_') and c not in ['rot_angle', 'rot_angle_vel']])\n    \n    imu_engineered_features = [\n        'acc_mag', 'rot_angle',\n        'acc_mag_jerk', 'rot_angle_vel',\n        'linear_acc_mag', 'linear_acc_mag_jerk',\n        'angular_vel_x', 'angular_vel_y', 'angular_vel_z',\n        'angular_jerk_x', 'angular_jerk_y', 'angular_jerk_z',\n        'angular_snap_x', 'angular_snap_y', 'angular_snap_z' # Added new angular snap features\n    ]\n    imu_cols = imu_cols_base + imu_engineered_features\n    imu_cols = list(dict.fromkeys(imu_cols))\n\n    thm_cols_original = [c for c in df.columns if c.startswith('thm_')]\n    \n    tof_aggregated_cols_template = []\n    for i in range(1, 6):\n        tof_aggregated_cols_template.extend([f'tof_{i}_mean', f'tof_{i}_std', f'tof_{i}_min', f'tof_{i}_max'])\n\n    final_feature_cols = imu_cols + thm_cols_original + tof_aggregated_cols_template\n    imu_dim_final = len(imu_cols)\n    tof_thm_aggregated_dim_final = len(thm_cols_original) + len(tof_aggregated_cols_template)\n    \n    print(f\"  IMU (incl. engineered & derivatives) {imu_dim_final} | THM + Aggregated TOF {tof_thm_aggregated_dim_final} | total {len(final_feature_cols)} features\")\n    np.save(EXPORT_DIR / \"feature_cols.npy\", np.array(final_feature_cols))\n\n    print(\"  Building sequences with aggregated TOF and preparing data for scaler...\")\n    seq_gp = df.groupby('sequence_id') \n    \n    all_steps_for_scaler_list = []\n    X_list_unscaled, y_list_int_for_stratify, lens = [], [], [] \n\n    for seq_id, seq_df_orig in seq_gp:\n        seq_df = seq_df_orig.copy()\n\n        for i in range(1, 6):\n            pixel_cols_tof = [f\"tof_{i}_v{p}\" for p in range(64)]\n            tof_sensor_data = seq_df[pixel_cols_tof].replace(-1, np.nan)\n            seq_df[f'tof_{i}_mean'] = tof_sensor_data.mean(axis=1)\n            seq_df[f'tof_{i}_std']  = tof_sensor_data.std(axis=1)\n            seq_df[f'tof_{i}_min']  = tof_sensor_data.min(axis=1)\n            seq_df[f'tof_{i}_max']  = tof_sensor_data.max(axis=1)\n        \n        mat_unscaled = seq_df[final_feature_cols].ffill().bfill().fillna(0).values.astype('float32')\n        \n        all_steps_for_scaler_list.append(mat_unscaled)\n        X_list_unscaled.append(mat_unscaled)\n        y_list_int_for_stratify.append(seq_df['gesture_int'].iloc[0])\n        lens.append(len(mat_unscaled))\n\n    print(\"  Fitting StandardScaler...\")\n    all_steps_concatenated = np.concatenate(all_steps_for_scaler_list, axis=0)\n    scaler = StandardScaler().fit(all_steps_concatenated)\n    joblib.dump(scaler, EXPORT_DIR / \"scaler.pkl\")\n    del all_steps_for_scaler_list, all_steps_concatenated\n\n    print(\"  Scaling and padding sequences...\")\n    X_scaled_list = [scaler.transform(x_seq) for x_seq in X_list_unscaled]\n    del X_list_unscaled\n\n    pad_len = int(np.percentile(lens, PAD_PERCENTILE))\n    np.save(EXPORT_DIR / \"sequence_maxlen.npy\", pad_len)\n    \n    X = pad_sequences(X_scaled_list, maxlen=pad_len, padding='post', truncating='post', dtype='float32')\n    del X_scaled_list\n    \n    y_int_for_stratify = np.array(y_list_int_for_stratify)\n    y = to_categorical(y_int_for_stratify, num_classes=len(le.classes_))\n\n    print(\"  Splitting data and preparing for training...\")\n    X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=82, stratify=y_int_for_stratify)\n\n    cw_vals = compute_class_weight('balanced', classes=np.arange(len(le.classes_)), y=y_int_for_stratify)\n    class_weight = dict(enumerate(cw_vals))\n\n    model = build_two_branch_model(pad_len, imu_dim_final, tof_thm_aggregated_dim_final, len(le.classes_), wd=WD)\n    \n    steps = len(X_tr) // BATCH_SIZE\n    lr_sched = tf.keras.optimizers.schedules.CosineDecayRestarts(5e-4, first_decay_steps=15 * steps) \n    \n    model.compile(optimizer=Adam(lr_sched),\n                  loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1),\n                  metrics=['accuracy'])\n\n    train_gen = MixupGenerator(X_tr, y_tr, batch_size=BATCH_SIZE, alpha=MIXUP_ALPHA)\n    cb = EarlyStopping(patience=PATIENCE, restore_best_weights=True, verbose=1, monitor='val_accuracy', mode='max')\n    \n    print(\"  Starting model training...\")\n    model.fit(train_gen, epochs=EPOCHS, validation_data=(X_val, y_val),\n              class_weight=class_weight, callbacks=[cb], verbose=1)\n\n    model.save(EXPORT_DIR / \"gesture_two_branch_mixup.h5\")\n    print(\"✔ Training done – artefacts saved in\", EXPORT_DIR)\n\n    from cmi_2025_metric_copy_for_import import CompetitionMetric\n    preds_val = model.predict(X_val).argmax(1)\n    true_val_int  = y_val.argmax(1)\n    \n    h_f1 = CompetitionMetric().calculate_hierarchical_f1(\n        pd.DataFrame({'gesture': le.classes_[true_val_int]}),\n        pd.DataFrame({'gesture': le.classes_[preds_val]}))\n    print(\"Hold‑out H‑F1 =\", round(h_f1, 4))\nelse:\n    print(\"▶ INFERENCE MODE – loading artefacts from\", PRETRAINED_DIR)\n    final_feature_cols = np.load(PRETRAINED_DIR / \"feature_cols.npy\", allow_pickle=True).tolist()\n    pad_len        = int(np.load(PRETRAINED_DIR / \"sequence_maxlen.npy\"))\n    scaler         = joblib.load(PRETRAINED_DIR / \"scaler.pkl\")\n    gesture_classes = np.load(PRETRAINED_DIR / \"gesture_classes.npy\", allow_pickle=True)\n\n    # Re-calculate imu_dim_final based on the actual features that will be used\n    imu_features_in_final_cols = [c for c in final_feature_cols if any(c.startswith(prefix) for prefix in ['linear_acc_', 'acc_', 'rot_', 'angular_vel_', 'angular_jerk_', 'angular_snap_'])]\n    imu_dim_final = len(imu_features_in_final_cols)\n\n    tof_thm_aggregated_dim_final = len(final_feature_cols) - imu_dim_final\n\n    custom_objs = {\n        'time_sum': time_sum,\n        'squeeze_last_axis': squeeze_last_axis,\n        'expand_last_axis': expand_last_axis,\n        'se_block': se_block,\n        'residual_se_cnn_block': residual_se_cnn_block,\n        'attention_layer': attention_layer,\n    }\n    model = load_model(PRETRAINED_DIR / \"gesture_two_branch_mixup.h5\",\n                       compile=False, custom_objects=custom_objs)\n    print(\"  Model, scaler, feature_cols, pad_len loaded – ready for evaluation\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:25.055897Z","iopub.execute_input":"2025-06-25T16:34:25.056568Z","iopub.status.idle":"2025-06-25T16:34:25.632171Z","shell.execute_reply.started":"2025-06-25T16:34:25.056520Z","shell.execute_reply":"2025-06-25T16:34:25.631592Z"},"scrolled":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Predict**","metadata":{}},{"cell_type":"code","source":"def predict(sequence: pl.DataFrame, demographics: pl.DataFrame) -> str:\n    df_seq = sequence.to_pandas()\n\n    df_seq['acc_mag'] = np.sqrt(df_seq['acc_x']**2 + df_seq['acc_y']**2 + df_seq['acc_z']**2)\n    df_seq['rot_angle'] = 2 * np.arccos(df_seq['rot_w'].clip(-1, 1))\n    df_seq['acc_mag_jerk'] = df_seq['acc_mag'].diff().fillna(0)\n    df_seq['rot_angle_vel'] = df_seq['rot_angle'].diff().fillna(0)\n\n    acc_cols_for_gravity_removal = ['acc_x', 'acc_y', 'acc_z']\n    rot_cols_for_gravity_removal = ['rot_x', 'rot_y', 'rot_z', 'rot_w']\n\n    if not all(col in df_seq.columns for col in acc_cols_for_gravity_removal + rot_cols_for_gravity_removal):\n        print(f\"Warning: Missing raw acc/rot columns for gravity removal in predict for sequence. Using raw acc as linear.\")\n        df_seq['linear_acc_x'] = df_seq.get('acc_x', 0)\n        df_seq['linear_acc_y'] = df_seq.get('acc_y', 0)\n        df_seq['linear_acc_z'] = df_seq.get('acc_z', 0)\n    else:\n        acc_data_seq = df_seq[acc_cols_for_gravity_removal]\n        rot_data_seq = df_seq[rot_cols_for_gravity_removal]\n        linear_accel_seq_arr = remove_gravity_from_acc(acc_data_seq, rot_data_seq)\n        \n        df_seq['linear_acc_x'] = linear_accel_seq_arr[:, 0]\n        df_seq['linear_acc_y'] = linear_accel_seq_arr[:, 1]\n        df_seq['linear_acc_z'] = linear_accel_seq_arr[:, 2]\n    \n    df_seq['linear_acc_mag'] = np.sqrt(df_seq['linear_acc_x']**2 + df_seq['linear_acc_y']**2 + df_seq['linear_acc_z']**2)\n    df_seq['linear_acc_mag_jerk'] = df_seq['linear_acc_mag'].diff().fillna(0)\n    \n    # Calculate angular velocity from quaternions in predict function\n    if all(col in df_seq.columns for col in rot_cols_for_gravity_removal):\n        angular_vel_seq_arr = calculate_angular_velocity_from_quat(df_seq[rot_cols_for_gravity_removal])\n        df_seq['angular_vel_x'] = angular_vel_seq_arr[:, 0]\n        df_seq['angular_vel_y'] = angular_vel_seq_arr[:, 1]\n        df_seq['angular_vel_z'] = angular_vel_seq_arr[:, 2]\n\n        # Calculate angular jerk from angular velocity\n        df_seq['angular_jerk_x'] = df_seq['angular_vel_x'].diff().fillna(0)\n        df_seq['angular_jerk_y'] = df_seq['angular_vel_y'].diff().fillna(0)\n        df_seq['angular_jerk_z'] = df_seq['angular_vel_z'].diff().fillna(0)\n\n        # Calculate angular snap from angular jerk\n        df_seq['angular_snap_x'] = df_seq['angular_jerk_x'].diff().fillna(0)\n        df_seq['angular_snap_y'] = df_seq['angular_jerk_y'].diff().fillna(0)\n        df_seq['angular_snap_z'] = df_seq['angular_jerk_z'].diff().fillna(0)\n\n    else:\n        print(f\"Warning: Missing quaternion columns for angular velocity, jerk, and snap calculation in predict. Filling with 0.\")\n        df_seq['angular_vel_x'] = 0\n        df_seq['angular_vel_y'] = 0\n        df_seq['angular_vel_z'] = 0\n        df_seq['angular_jerk_x'] = 0\n        df_seq['angular_jerk_y'] = 0\n        df_seq['angular_jerk_z'] = 0\n        df_seq['angular_snap_x'] = 0\n        df_seq['angular_snap_y'] = 0\n        df_seq['angular_snap_z'] = 0\n\n    for i in range(1, 6): \n        pixel_cols_tof = [f\"tof_{i}_v{p}\" for p in range(64)]\n        if not all(col in df_seq.columns for col in pixel_cols_tof):\n            print(f\"Warning: Missing some TOF pixel columns for tof_{i} in predict. Filling aggregates with 0.\")\n            df_seq[f'tof_{i}_mean'] = 0\n            df_seq[f'tof_{i}_std']  = 0\n            df_seq[f'tof_{i}_min']  = 0\n            df_seq[f'tof_{i}_max']  = 0\n            continue\n\n        tof_sensor_data = df_seq[pixel_cols_tof].replace(-1, np.nan)\n        df_seq[f'tof_{i}_mean'] = tof_sensor_data.mean(axis=1)\n        df_seq[f'tof_{i}_std']  = tof_sensor_data.std(axis=1)\n        df_seq[f'tof_{i}_min']  = tof_sensor_data.min(axis=1)\n        df_seq[f'tof_{i}_max']  = tof_sensor_data.max(axis=1)\n        \n    if 'tof_range_across_sensors' in final_feature_cols:\n        tof_mean_cols_for_contrast = [f'tof_{i}_mean' for i in range(1, 6) if f'tof_{i}_mean' in df_seq.columns]\n        thm_cols_for_contrast = [f'thm_{i}' for i in range(1, 6) if f'thm_{i}' in df_seq.columns]\n\n        if tof_mean_cols_for_contrast:\n            tof_values_for_contrast = df_seq[tof_mean_cols_for_contrast]\n            df_seq['tof_range_across_sensors'] = tof_values_for_contrast.max(axis=1) - tof_values_for_contrast.min(axis=1)\n            df_seq['tof_std_across_sensors'] = tof_values_for_contrast.std(axis=1)\n        else:\n            df_seq['tof_range_across_sensors'] = 0\n            df_seq['tof_std_across_sensors'] = 0\n\n        if thm_cols_for_contrast:\n            thm_values_for_contrast = df_seq[thm_cols_for_contrast]\n            df_seq['thm_range_across_sensors'] = thm_values_for_contrast.max(axis=1) - thm_values_for_contrast.min(axis=1)\n            df_seq['thm_std_across_sensors'] = thm_values_for_contrast.std(axis=1)\n        else:\n            df_seq['thm_range_across_sensors'] = 0\n            df_seq['thm_std_across_sensors'] = 0\n        \n    df_seq_final_features = pd.DataFrame(index=df_seq.index)\n    for col_name in final_feature_cols:\n        if col_name in df_seq.columns:\n            df_seq_final_features[col_name] = df_seq[col_name]\n        else:\n            print(f\"CRITICAL ERROR IN PREDICT: Feature '{col_name}' expected by model (from final_feature_cols) was NOT generated in df_seq. Filling with 0. THIS IS LIKELY A BUG.\")\n            df_seq_final_features[col_name] = 0 \n            \n    mat_unscaled = df_seq_final_features.ffill().bfill().fillna(0).values.astype('float32')\n    \n    mat_scaled = scaler.transform(mat_unscaled)\n    \n    pad_input = pad_sequences([mat_scaled], maxlen=pad_len, padding='post', truncating='post', dtype='float32')\n    \n    idx = int(model.predict(pad_input, verbose=0).argmax(1)[0])\n    return str(gesture_classes[idx])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:25.632883Z","iopub.execute_input":"2025-06-25T16:34:25.633061Z","iopub.status.idle":"2025-06-25T16:34:25.889908Z","shell.execute_reply.started":"2025-06-25T16:34:25.633047Z","shell.execute_reply":"2025-06-25T16:34:25.889145Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 》》》**Submit Inference server**","metadata":{}},{"cell_type":"code","source":"import kaggle_evaluation.cmi_inference_server\ninference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n    inference_server.serve()\nelse:\n    inference_server.run_local_gateway(\n        data_paths=(\n            '/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv',\n            '/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv',\n        )\n    )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T16:34:25.891746Z","iopub.execute_input":"2025-06-25T16:34:25.891937Z","iopub.status.idle":"2025-06-25T16:34:29.560320Z","shell.execute_reply.started":"2025-06-25T16:34:25.891923Z","shell.execute_reply":"2025-06-25T16:34:29.559743Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}