{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"isSourceIdPinned":false,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Gesture Recognition: Training and Inference\n\nThis notebook implements a gesture recognition system using sensor data from the `cmi-detect-behavior-with-sensor-data` dataset. It is organized into three main sections:\n\n1. **`Imports and Setup`**: Imports required libraries and sets up the configuration and device (GPU if available).\n2. **`Training`**: Preprocesses training data, defines the model architecture, and trains the model to classify gestures.\n3. **`Inference`**: Loads the trained model and artifacts to predict gestures on test data, compatible with Kaggle's evaluation server.\n\n**Prerequisites**:\n- **`Dataset`**: Attach `/kaggle/input/cmi-detect-behavior-with-sensor-data` to the notebook.\n- **`Artifacts`**: Ensure `gesture_model.pt`, `global_scaler.pkl`, `gesture_classes.npy`, and `sequence_maxlen.npy`","metadata":{}},{"cell_type":"markdown","source":"## 1. Imports and Setup\n\nThis section imports libraries for data processing, model building, and evaluation. We define hyperparameters in a configuration dictionary and set the device to GPU if available.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport joblib\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.metrics import accuracy_score, f1_score\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences as keras_pad_sequences\n\n# Configuration\nconfig = {\n    \"batch_size\": 128,\n    \"lr\": 1e-3,\n    \"num_epochs\": 100,\n    \"patience\": 10,\n    \"mixup_alpha\": 0.2,\n    \"use_gru\": False,\n    \"augmentations\": True,\n}\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using device: {device}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-17T12:21:46.586039Z","iopub.execute_input":"2025-06-17T12:21:46.586501Z","iopub.status.idle":"2025-06-17T12:22:15.766111Z","shell.execute_reply.started":"2025-06-17T12:21:46.586465Z","shell.execute_reply":"2025-06-17T12:22:15.765234Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Training\nThis section handles data preprocessing, model definition, and training. It includes:\n\nPreprocessing: Encodes gesture labels, scales features, pads sequences, and splits data.\nModel: Defines a hybrid neural network with IMU and ToF branches, LSTM, and attention.\nTraining: Uses augmentation, Adam optimizer, and early stopping to train the model.\n\n### 2.1 Data Preprocessing\nLoad training data, encode labels, scale features, and pad sequences to a fixed length (90th percentile). Split into train/validation sets and create data loaders with augmentation functions.","metadata":{}},{"cell_type":"code","source":"# Load and preprocess training data\ntrain_df = pd.read_csv(\"/kaggle/input/cmi-detect-behavior-with-sensor-data/train.csv\")\nlabel_encoder = LabelEncoder()\ntrain_df['gesture'] = label_encoder.fit_transform(train_df['gesture'].astype(str))\ngesture_classes = label_encoder.classes_\nnp.save(\"gesture_classes.npy\", gesture_classes)\n\n# Define feature columns\nexcluded = {'gesture', 'sequence_type', 'behavior', 'orientation', 'row_id', 'subject', 'phase', 'sequence_id', 'sequence_counter'}\nfeature_cols = [c for c in train_df.columns if c not in excluded]\nimu_cols = [c for c in feature_cols if not (c.startswith(\"thm_\") or c.startswith(\"tof_\"))]\ntof_cols = [c for c in feature_cols if c.startswith(\"thm_\") or c.startswith(\"tof_\")]\nimu_dim, tof_dim = len(imu_cols), len(tof_cols)\n\n# Scale features\nscaler = StandardScaler().fit(train_df[feature_cols].ffill().bfill().fillna(0).values)\njoblib.dump(scaler, \"global_scaler.pkl\")\n\n# Group by sequence and pad\nX_list, y_list, lengths = [], [], []\nfor _, seq in train_df.groupby(\"sequence_id\"):\n    arr = scaler.transform(seq[feature_cols].ffill().bfill().fillna(0).values)\n    X_list.append(arr)\n    y_list.append(seq['gesture'].iloc[0])\n    lengths.append(len(arr))\n\npad_len = int(np.percentile(lengths, 90))\nnp.save(\"sequence_maxlen.npy\", pad_len)\n\nX = keras_pad_sequences(X_list, maxlen=pad_len, dtype='float32', padding='post', truncating='post')\ny = np.eye(len(np.unique(y_list)))[np.array(y_list)]\nX = np.transpose(X, (0, 2, 1))\n\n# Split data\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y_list)\nclass_weights = torch.tensor(compute_class_weight('balanced', classes=np.unique(y_list), y=np.array(y_list)), dtype=torch.float)\n\n# Augmentation functions\ndef jitter(x, sigma=0.01):\n    return x + sigma * torch.randn_like(x)\n\ndef time_mask(x, mask_ratio=0.1):\n    if x.dim() == 2:\n        C, T = x.shape\n        num_mask = int(T * mask_ratio)\n        mask_indices = torch.randint(0, T, (num_mask,))\n        x[:, mask_indices] = 0\n    elif x.dim() == 3:\n        B, C, T = x.shape\n        num_mask = int(T * mask_ratio)\n        mask_indices = torch.randint(0, T, (B, num_mask))\n        for i in range(B):\n            x[i, :, mask_indices[i]] = 0\n    return x\n\ndef mixup_data(x, y, alpha=0.2):\n    lam = np.random.beta(alpha, alpha) if alpha > 0 else 1.0\n    index = torch.randperm(x.size(0)).to(x.device)\n    return lam * x + (1 - lam) * x[index], lam * y + (1 - lam) * y[index]\n\n# Dataset class\nclass SequenceDataset(Dataset):\n    def __init__(self, X, y=None, augment=False):\n        self.X = torch.from_numpy(X).float()\n        self.y = torch.from_numpy(y).float() if y is not None else None\n        self.augment = augment\n\n    def __len__(self):\n        return self.X.size(0)\n\n    def __getitem__(self, idx):\n        x = self.X[idx]\n        if self.augment:\n            x = jitter(x)\n            x = time_mask(x)\n        return (x, self.y[idx]) if self.y is not None else x\n\n# Create data loaders\ntrain_loader = DataLoader(SequenceDataset(X_train, y_train, augment=config[\"augmentations\"]), batch_size=config[\"batch_size\"], shuffle=True, drop_last=True)\nval_loader = DataLoader(SequenceDataset(X_val, y_val), batch_size=config[\"batch_size\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-17T12:24:04.144871Z","iopub.execute_input":"2025-06-17T12:24:04.145192Z","iopub.status.idle":"2025-06-17T12:25:15.950056Z","shell.execute_reply.started":"2025-06-17T12:24:04.14517Z","shell.execute_reply":"2025-06-17T12:25:15.949061Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 2.2 Model Architecture\nDefine the HARModel, a hybrid neural network with:\n\n- `IMU Branch`: Two ResidualSEBlocks for motion sensor data.\n- `ToF Branch`: Convolutional layers for distance sensor data.\n- `LSTM and Attention`: Captures temporal dependencies and focuses on key time steps.\n- `Fully Connected Layers`: Outputs gesture class probabilities.","metadata":{}},{"cell_type":"code","source":"class SEBlock(nn.Module):\n    def __init__(self, channels):\n        super().__init__()\n        self.fc1 = nn.Linear(channels, channels // 8)\n        self.fc2 = nn.Linear(channels // 8, channels)\n\n    def forward(self, x):\n        se = x.mean(dim=2)\n        se = F.relu(self.fc1(se))\n        se = torch.sigmoid(self.fc2(se)).unsqueeze(2)\n        return x * se\n\nclass ResidualSEBlock(nn.Module):\n    def __init__(self, in_ch, out_ch, k=3, p=2):\n        super().__init__()\n        self.conv1 = nn.Conv1d(in_ch, out_ch, k, padding=k//2)\n        self.bn1 = nn.BatchNorm1d(out_ch)\n        self.conv2 = nn.Conv1d(out_ch, out_ch, k, padding=k//2)\n        self.bn2 = nn.BatchNorm1d(out_ch)\n        self.se = SEBlock(out_ch)\n        self.down = nn.Conv1d(in_ch, out_ch, 1) if in_ch != out_ch else nn.Identity()\n        self.pool = nn.MaxPool1d(p)\n        self.drop = nn.Dropout(0.3)\n\n    def forward(self, x):\n        identity = self.down(x)\n        out = F.relu(self.bn1(self.conv1(x)))\n        out = self.bn2(self.conv2(out))\n        out = self.se(out)\n        out = F.relu(out + identity)\n        return self.drop(self.pool(out))\n\nclass Attention(nn.Module):\n    def __init__(self, dim):\n        super().__init__()\n        self.score = nn.Linear(dim, 1)\n\n    def forward(self, x):\n        weights = F.softmax(self.score(x).squeeze(2), dim=1).unsqueeze(2)\n        return (x * weights).sum(dim=1)\n\nclass HARModel(nn.Module):\n    def __init__(self, imu_dim, tof_dim, seq_len, num_classes):\n        super().__init__()\n        self.imu_branch = nn.Sequential(ResidualSEBlock(imu_dim, 64), ResidualSEBlock(64, 128))\n        self.tof_branch = nn.Sequential(\n            nn.Conv1d(tof_dim, 64, 3, padding=1), nn.BatchNorm1d(64), nn.ReLU(), nn.MaxPool1d(2), nn.Dropout(0.3),\n            nn.Conv1d(64, 128, 3, padding=1), nn.BatchNorm1d(128), nn.ReLU(), nn.MaxPool1d(2), nn.Dropout(0.3)\n        )\n        self.rnn = (nn.GRU if config[\"use_gru\"] else nn.LSTM)(256, 128, batch_first=True, bidirectional=True)\n        self.attn = Attention(256)\n        self.fc = nn.Sequential(nn.Linear(256, 128), nn.BatchNorm1d(128), nn.ReLU(), nn.Dropout(0.3), nn.Linear(128, num_classes))\n\n    def forward(self, x):\n        x_imu, x_tof = x[:, :imu_dim], x[:, imu_dim:]\n        b1 = self.imu_branch(x_imu)\n        b2 = self.tof_branch(x_tof)\n        x = torch.cat([b1, b2], dim=1).permute(0, 2, 1)\n        x, _ = self.rnn(x)\n        return self.fc(self.attn(x))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-17T12:28:59.219817Z","iopub.execute_input":"2025-06-17T12:28:59.221131Z","iopub.status.idle":"2025-06-17T12:28:59.254261Z","shell.execute_reply.started":"2025-06-17T12:28:59.221065Z","shell.execute_reply":"2025-06-17T12:28:59.253111Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 2.3 Training Loop\nTrain the model using:\n\n- `Loss`: Soft cross-entropy (supports mixup augmentation).\n- `Optimizer`: Adam with weight decay.\n- `Scheduler`: Cosine annealing with warm restarts.\n- `Early Stopping`: Stops if validation loss doesn’t improve for 10 epochs.\n- `Metrics`: Tracks loss, F1 score, and accuracy.","metadata":{}},{"cell_type":"code","source":"# Initialize model, optimizer, and scheduler\nmodel = HARModel(imu_dim, tof_dim, pad_len, len(gesture_classes)).to(device)\nopt = optim.Adam(model.parameters(), lr=config[\"lr\"], weight_decay=1e-4)\nsched = optim.lr_scheduler.CosineAnnealingWarmRestarts(opt, T_0=5*len(train_loader), T_mult=2)\n\ndef soft_ce(pred, target):\n    return -(F.log_softmax(pred, dim=1) * target).sum(dim=1).mean()\n\n# Training loop\nbest_loss = np.inf\nfor epoch in range(config[\"num_epochs\"]):\n    model.train()\n    train_loss = 0\n    for xb, yb in train_loader:\n        xb, yb = xb.to(device), yb.to(device)\n        xb, yb = mixup_data(xb, yb, config[\"mixup_alpha\"])\n        opt.zero_grad()\n        loss = soft_ce(model(xb), yb)\n        loss.backward()\n        opt.step()\n        sched.step()\n        train_loss += loss.item() * xb.size(0)\n    train_loss /= len(train_loader.dataset)\n\n    model.eval()\n    val_loss, all_preds, all_true = 0, [], []\n    with torch.no_grad():\n        for xb, yb in val_loader:\n            xb, yb = xb.to(device), yb.to(device)\n            preds = model(xb)\n            val_loss += soft_ce(preds, yb).item() * xb.size(0)\n            all_preds.extend(preds.argmax(1).cpu().numpy())\n            all_true.extend(yb.argmax(1).cpu().numpy())\n    val_loss /= len(val_loader.dataset)\n    f1 = f1_score(all_true, all_preds, average=\"macro\")\n    acc = accuracy_score(all_true, all_preds)\n    print(f\"Epoch {epoch+1}: Train={train_loss:.4f}, Val={val_loss:.4f}, F1={f1:.4f}, Acc={acc:.4f}\")\n    if val_loss < best_loss:\n        best_loss = val_loss\n        best_model = model.state_dict()\n    else:\n        if epoch >= config[\"patience\"]:\n            print(\"Early stopping.\")\n            break\n\ntorch.save(best_model, \"gesture_model.pt\")\nprint(\"Training complete. Model saved as gesture_model.pt\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-17T12:37:50.294015Z","iopub.execute_input":"2025-06-17T12:37:50.294348Z","iopub.status.idle":"2025-06-17T12:44:13.722277Z","shell.execute_reply.started":"2025-06-17T12:37:50.294323Z","shell.execute_reply":"2025-06-17T12:44:13.7214Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Inference\nThis section loads the trained model and artifacts to predict gestures on test data. It is designed for Kaggle’s evaluation server or local testing.\n\n### 3.1 Inference Setup\nLoad artifacts, reconstruct the model, and define the prediction function. Check for Kaggle’s evaluation environment to handle server or local modes.","metadata":{}},{"cell_type":"code","source":"# Check for Kaggle environment\ntry:\n    import kaggle_evaluation.cmi_inference_server as cmi_server\n    is_kaggle = True\nexcept ImportError:\n    print(\"[WARNING] kaggle_evaluation module not found. Running outside of Kaggle environment.\")\n    is_kaggle = False\n\n# Load\ngesture_classes = np.load(\"/kaggle/working/gesture_classes.npy\", allow_pickle=True)\npad_len = int(np.load(\"/kaggle/working/sequence_maxlen.npy\"))\nscaler = joblib.load(\"/kaggle/working/global_scaler.pkl\")\n\n# Define feature columns (same as training)\ntemp_df = pd.read_csv(\"/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv\", nrows=1)\nfeature_cols = [c for c in temp_df.columns if c not in excluded]\nimu_cols = [c for c in feature_cols if not (c.startswith(\"thm_\") or c.startswith(\"tof_\"))]\ntof_cols = [c for c in feature_cols if c.startswith(\"thm_\") or c.startswith(\"tof_\")]\nimu_dim, tof_dim = len(imu_cols), len(tof_cols)\n\n# Load model\nmodel = HARModel(imu_dim, tof_dim, pad_len, len(gesture_classes)).to(device)\nmodel.load_state_dict(torch.load(\"/kaggle/working/gesture_model.pt\", map_location=device))\nmodel.eval()\n\n# Inference helper\ndef preprocess_sequence(df):\n    arr = scaler.transform(df[feature_cols].ffill().bfill().fillna(0).values)\n    padded = keras_pad_sequences([arr], maxlen=pad_len, dtype='float32', padding='post', truncating='post')[0]\n    return torch.from_numpy(padded.T).unsqueeze(0).float()\n\ndef predict(sequence: pl.DataFrame, demographics: pl.DataFrame) -> str:\n    x = preprocess_sequence(sequence.to_pandas()).to(device)\n    with torch.no_grad():\n        out = model(x)\n        return str(gesture_classes[out.argmax().item()])\n\n# Launch server (Kaggle only)\nif is_kaggle:\n    inference_server = cmi_server.CMIInferenceServer(predict)\n    if os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n        inference_server.serve()\n    else:\n        inference_server.run_local_gateway(\n            data_paths=(\n                '/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv',\n                '/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv'\n            )\n        )\nelse:\n    print(\"[INFO] Not running in Kaggle evaluation mode. Skipping server startup.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-17T12:49:00.671539Z","iopub.execute_input":"2025-06-17T12:49:00.672672Z","iopub.status.idle":"2025-06-17T12:49:02.156203Z","shell.execute_reply.started":"2025-06-17T12:49:00.672638Z","shell.execute_reply":"2025-06-17T12:49:02.155298Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#  work in progresss","metadata":{}},{"cell_type":"markdown","source":"This solution was developed by building upon the foundational work of key contributor:\n\nSensor Pulse🧠| Viz & EDA for BFRB Detection by Tarun Mishra","metadata":{}}]}