{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"},{"sourceId":242954653,"sourceType":"kernelVersion"}],"dockerImageVersionId":31041,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Changelog:\n\n- **Version 1**: Initial copy and conversion to Pytorch (LB 0.51)\n  - This was a very naive conversion from the original Tensorflow/Keras to Pytorch using Gemini Flash 2.5 LLM for an assist.\n  - Likely not correctly copied as predictions were only ~50%\n- **Version 2**: N/A\n- **Version 3**: Replaced the main model with a different, refactored simplification\n- **Version 4**: Some speed improvements; checkpoint for when no GPU\n- **Version 5**: Clean-up\n- **Version 6 & 7**: Fix typo; allow for clean-up of checkpoints\n- **Version 8**: Split up training and inference logic\n\n## Via original author: \n\n> This solution was developed by building upon the foundational work of two key contributors:\n> - **CMI25 | IMU+THM/TOF |TF BiLSTM+GRU+Attention|LB.75** by [yukiZ](https://www.kaggle.com/code/hideyukizushi/cmi25-imu-thm-tof-tf-bilstm-gru-attention-lb-75)\n> - **imu-tof** by [Erich von Mainstein](https://www.kaggle.com/code/vonmainstein/imu-tof)\n> \n> Special recognition for the exceptional exploratory data analysis:\n> - **Sensor Pulse🧠| Viz & EDA for BFRB Detection** by [Tarun Mishra](https://www.kaggle.com/code/tarundirector/sensor-pulse-viz-eda-for-bfrb-detection)\n> ## Implemented Enhancements\n>\n> ### IMU Signal Processing Optimization (Core Improvement)\n>\n> **Implemented Features:**\n> - Derived kinematic metrics:\n>  - Acceleration vector magnitude (`acc_mag`)\n>   - Scalar rotation angle (`rot_angle`)\n>   - Acceleration derivative (`acc_mag_jerk`)\n>   - Angular velocity (`rot_angle_vel`)\n> - Initial learning rate reduction from `1e-3` to `5e-4`\n> - Implemented cosine decay scheduling:\n>   - Parameters: `first_decay_steps = 15 × steps_per_epoch`\n> \n> **Unsuccessful Experiments (Reverted):**\n> \n> - Further LR Reduction: `LR_INIT = 2-e4` worsened score.\n> - Dropout Rate Adjustments: No improvement over baseline dropout.\n> - TOF/THEM Branch Architecture Change: Replacing simpler CNNs with `residual_se_cnn_blocks` was detrimental.","metadata":{}},{"cell_type":"markdown","source":"## Import Necessary Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport json\nimport joblib\nimport numpy as np\nimport pandas as pd\nimport random\nfrom pathlib import Path\nimport warnings\nimport shutil\nwarnings.filterwarnings(\"ignore\")\n\n# Scikit-learn\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.model_selection import StratifiedGroupKFold # Still from sklearn\n\n# PyTorch\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.nn.utils.rnn import pad_sequence\nfrom torch.optim.lr_scheduler import LambdaLR\n\nimport polars as pl","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:21.377759Z","iopub.execute_input":"2025-06-21T16:23:21.377969Z","iopub.status.idle":"2025-06-21T16:23:36.114644Z","shell.execute_reply.started":"2025-06-21T16:23:21.377951Z","shell.execute_reply":"2025-06-21T16:23:36.113991Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Set Seed","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed):\n    \"\"\"\n    Seeds various random number generators for reproducibility in PyTorch.\n\n    Args:\n        seed (int): The seed value to use.\n    \"\"\"\n    os.environ['PYTHONHASHSEED'] = str(seed) # Set Python hash seed\n    random.seed(seed)                        # Seed Python's random module\n    np.random.seed(seed)                     # Seed NumPy\n    torch.manual_seed(seed)                  # Seed PyTorch for CPU operations\n    torch.cuda.manual_seed(seed)             # Seed PyTorch for all GPU operations (if available)\n    torch.cuda.manual_seed_all(seed)         # Seed all GPUs if multiple are used\n\n    # Ensure deterministic operations for cuDNN (GPU backend)\n    # This can sometimes slightly reduce performance but ensures reproducibility\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False   # Disable cuDNN benchmarking for reproducibility\n\n# Call the function to set the seed\nseed_everything(seed=42)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.116402Z","iopub.execute_input":"2025-06-21T16:23:36.116766Z","iopub.status.idle":"2025-06-21T16:23:36.131213Z","shell.execute_reply.started":"2025-06-21T16:23:36.116745Z","shell.execute_reply":"2025-06-21T16:23:36.130562Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Configuration","metadata":{}},{"cell_type":"code","source":"# (Competition metric will only be imported when TRAINing)\nTRAIN = True\nDELETE_EXISTING_CHECKPOINTS = False\nRAW_DIR = Path(\"/kaggle/input/cmi-detect-behavior-with-sensor-data\")\n\n# used when TRAIN=False\nPRETRAINED_DIR = Path(\"/kaggle/input/pretrained-model\")\n\n# artifacts will be saved here\nEXPORT_DIR = Path(\"./\")                 \n\nBATCH_SIZE = 64\nPAD_PERCENTILE = 95\nLR_INIT = 5e-4\nWD = 3e-3\nMIXUP_ALPHA = 0.4\nEPOCHS = 100\nPATIENCE = 40\n\nprint(\"▶ imports ready · pytorch\", torch.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.132097Z","iopub.execute_input":"2025-06-21T16:23:36.132412Z","iopub.status.idle":"2025-06-21T16:23:36.151624Z","shell.execute_reply.started":"2025-06-21T16:23:36.132393Z","shell.execute_reply":"2025-06-21T16:23:36.150974Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Utility Functions","metadata":{}},{"cell_type":"code","source":"# --- Tensor Manipulations ---\ndef time_sum(x):\n    \"\"\"Sums a tensor along the time dimension (dimension 1).\"\"\"\n    return torch.sum(x, dim=1)\n\ndef squeeze_last_axis(x):\n    \"\"\"Removes the last dimension of a tensor.\"\"\"\n    return x.squeeze(-1)\n\ndef expand_last_axis(x):\n    \"\"\"Adds a new dimension at the end of a tensor.\"\"\"\n    return x.unsqueeze(-1)\n\n# --- SE Block (Squeeze-and-Excitation Block) ---\nclass SEBlock(nn.Module):\n    \"\"\"\n    Squeeze-and-Excitation block that adaptively recalibrates channel-wise feature responses.\n    \"\"\"\n    def __init__(self, channels, reduction=8):\n        super(SEBlock, self).__init__()\n        self.avg_pool = nn.AdaptiveAvgPool1d(1)\n        self.fc1 = nn.Linear(channels, channels // reduction, bias=False)\n        self.relu = nn.ReLU(inplace=True)\n        self.fc2 = nn.Linear(channels // reduction, channels, bias=False)\n        self.sigmoid = nn.Sigmoid()\n\n    def forward(self, x):\n        b, c, _ = x.size()\n        se = self.avg_pool(x).view(b, c)\n        se = self.fc1(se)\n        se = self.relu(se)\n        se = self.fc2(se)\n        se = self.sigmoid(se).view(b, c, 1)\n        return x * se\n\n# --- Residual CNN Block with SE ---\nclass ResidualSECNNBlock(nn.Module):\n    \"\"\"\n    Residual CNN block with a Squeeze-and-Excitation block.\n    Includes an option to freeze layers for fine-tuning.\n    \"\"\"\n    def __init__(self, in_filters, out_filters, kernel_size, pool_size=2, drop_rate=0.3, wd=1e-4, requires_grad=True):\n        super(ResidualSECNNBlock, self).__init__()\n        self.conv_block = nn.Sequential(\n            nn.Conv1d(in_filters, out_filters, kernel_size, padding='same', bias=False),\n            nn.BatchNorm1d(out_filters),\n            nn.ReLU(inplace=True),\n            nn.Conv1d(out_filters, out_filters, kernel_size, padding='same', bias=False),\n            nn.BatchNorm1d(out_filters),\n            nn.ReLU(inplace=True)\n        )\n        self.se_block = SEBlock(out_filters)\n\n        self.shortcut_conv = None\n        if in_filters != out_filters:\n            self.shortcut_conv = nn.Sequential(\n                nn.Conv1d(in_filters, out_filters, 1, padding='same', bias=False),\n                nn.BatchNorm1d(out_filters)\n            )\n\n        self.max_pool = nn.MaxPool1d(pool_size)\n        self.dropout = nn.Dropout(drop_rate)\n        self.wd = wd\n\n        for param in self.parameters():\n            param.requires_grad = requires_grad\n\n    def forward(self, x):\n        shortcut = x\n        x = self.conv_block(x)\n        x = self.se_block(x)\n\n        if self.shortcut_conv:\n            shortcut = self.shortcut_conv(shortcut)\n        x = x + shortcut\n        \n        x = F.relu(x)\n        x = self.max_pool(x)\n        x = self.dropout(x)\n        return x\n\n# --- Attention Layer ---\nclass AttentionLayer(nn.Module):\n    \"\"\"\n    Attention layer to compute context vectors.\n    \"\"\"\n    def __init__(self, input_dim):\n        super(AttentionLayer, self).__init__()\n        self.score_dense = nn.Linear(input_dim, 1)\n        self.tanh = nn.Tanh()\n\n    def forward(self, inputs):\n        score = self.score_dense(inputs)\n        score = self.tanh(score)\n        score = squeeze_last_axis(score)\n        weights = F.softmax(score, dim=1)\n        weights = expand_last_axis(weights)\n        context = inputs * weights\n        context = time_sum(context)\n        return context\n\n# --- A Simple CNN Block for the ToF Branch ---\nclass SimpleCNNBlock(nn.Module):\n    \"\"\"\n    A simple CNN block.\n    Includes an option to freeze layers for fine-tuning.\n    \"\"\"\n    def __init__(self, in_filters, out_filters, kernel_size=3, pool_size=2, drop_rate=0.2, requires_grad=True):\n        super(SimpleCNNBlock, self).__init__()\n        self.block = nn.Sequential(\n            nn.Conv1d(in_filters, out_filters, kernel_size, padding='same', bias=False),\n            nn.BatchNorm1d(out_filters),\n            nn.ReLU(inplace=True),\n            nn.MaxPool1d(pool_size),\n            nn.Dropout(drop_rate)\n        )\n        for param in self.parameters():\n            param.requires_grad = requires_grad\n\n    def forward(self, x):\n        return self.block(x)\n\n# --- Two-Branch Model ---\nclass TwoBranchModel(nn.Module):\n    \"\"\"\n    A two-branch model for IMU and ToF data.\n    Provides functionality to freeze specific branches for fine-tuning.\n    \"\"\"\n    def __init__(self, pad_len, imu_dim, tof_dim, n_classes, requires_grad_imu=True, requires_grad_tof=True, requires_grad_classifier=True):\n        super(TwoBranchModel, self).__init__()\n        \n        self.imu_dim = imu_dim\n        self.tof_dim = tof_dim\n\n        # --- IMU Branch ---\n        self.imu_branch_block1 = ResidualSECNNBlock(imu_dim, 64, 3, drop_rate=0.1, requires_grad=requires_grad_imu)\n        self.imu_branch_block2 = ResidualSECNNBlock(64, 128, 5, drop_rate=0.1, requires_grad=requires_grad_imu)\n\n        # --- TOF Branch ---\n        self.tof_branch_block1 = SimpleCNNBlock(tof_dim, 64, drop_rate=0.2, requires_grad=requires_grad_tof)\n        self.tof_branch_block2 = SimpleCNNBlock(64, 128, drop_rate=0.2, requires_grad=requires_grad_tof)\n        \n        # --- Shared Recurrent Layer ---\n        merged_cnn_features = 128 + 128\n        \n        self.recurrent_layer = nn.LSTM(\n            input_size=merged_cnn_features, \n            hidden_size=128, \n            bidirectional=True, \n            batch_first=True\n        )\n        \n        recurrent_output_features = 128 * 2\n\n        # --- Attention Layer ---\n        self.attention_layer = AttentionLayer(input_dim=recurrent_output_features)\n        \n        # --- Classifier Head ---\n        self.classifier_head = nn.Sequential(\n            nn.Linear(recurrent_output_features, 256, bias=False),\n            nn.BatchNorm1d(256),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.5),\n            nn.Linear(256, 128, bias=False),\n            nn.BatchNorm1d(128),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.3)\n        )\n\n        self.output_layer = nn.Linear(128, n_classes)\n\n        # Freeze the classifier head if specified\n        if not requires_grad_classifier:\n            for param in self.classifier_head.parameters():\n                param.requires_grad = False\n            for param in self.output_layer.parameters():\n                param.requires_grad = False\n\n\n    def forward(self, x):\n        # Split branches\n        imu = x[:, :, :self.imu_dim]\n        tof = x[:, :, self.imu_dim:]\n\n        # Permute for Conv1D\n        imu = imu.permute(0, 2, 1)\n        tof = tof.permute(0, 2, 1)\n\n        # --- Process Branches ---\n        x1 = self.imu_branch_block1(imu)\n        x1 = self.imu_branch_block2(x1)\n\n        x2 = self.tof_branch_block1(tof)\n        x2 = self.tof_branch_block2(x2)\n        \n        # --- Merge and Process Sequentially ---\n        merged = torch.cat([x1, x2], dim=1)\n        \n        # Permute for Recurrent Layer\n        merged = merged.permute(0, 2, 1)\n\n        recurrent_out, _ = self.recurrent_layer(merged)\n        \n        attention_out = self.attention_layer(recurrent_out)\n        \n        # --- Classifier ---\n        classified = self.classifier_head(attention_out)\n        \n        # --- Output ---\n        logits = self.output_layer(classified)\n        \n        return logits","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.152783Z","iopub.execute_input":"2025-06-21T16:23:36.153105Z","iopub.status.idle":"2025-06-21T16:23:36.311966Z","shell.execute_reply.started":"2025-06-21T16:23:36.153085Z","shell.execute_reply":"2025-06-21T16:23:36.311309Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Data Helpers**","metadata":{}},{"cell_type":"code","source":"# --- Preprocessing Function ---\nclass MixupDataset(Dataset):\n    def __init__(self, X: torch.Tensor, y: torch.Tensor, alpha: float = 0.2):\n        self.X = X\n        self.y = y\n\n        self.alpha = alpha\n\n    def __len__(self):\n        return len(self.X)\n\n    def __getitem__(self, idx: int):\n        x1, y1 = self.X[idx], self.y[idx]\n\n        # Select a random sample from the dataset for mixing\n        rand_idx = random.randint(0, len(self.X) - 1)\n        x2, y2 = self.X[rand_idx], self.y[rand_idx]\n\n        # Generate lambda from Beta distribution. Ensure alpha > 0.\n        lam = np.random.beta(self.alpha, self.alpha)\n        lam = max(0, min(1, lam))\n\n        x_mixed = lam * x1 + (1 - lam) * x2\n        y_mixed = lam * y1 + (1 - lam) * y2\n\n        return x_mixed, y_mixed","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.313171Z","iopub.execute_input":"2025-06-21T16:23:36.313808Z","iopub.status.idle":"2025-06-21T16:23:36.339008Z","shell.execute_reply.started":"2025-06-21T16:23:36.313770Z","shell.execute_reply":"2025-06-21T16:23:36.338396Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Instantiate the model\nmodel = TwoBranchModel(pad_len=127, imu_dim=7, tof_dim=325, n_classes=18)\nprint(model) # To see the model's structure","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.339700Z","iopub.execute_input":"2025-06-21T16:23:36.339918Z","iopub.status.idle":"2025-06-21T16:23:36.482183Z","shell.execute_reply.started":"2025-06-21T16:23:36.339900Z","shell.execute_reply":"2025-06-21T16:23:36.481235Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Training / Inference Pipeline","metadata":{}},{"cell_type":"markdown","source":"### Step 1: Checkpoint and Directory Management","metadata":{}},{"cell_type":"code","source":"def setup_environment():\n    \"\"\"Creates directories and handles existing checkpoints.\"\"\"\n    print(\"▶ Setting up environment...\")\n    os.makedirs(EXPORT_DIR, exist_ok=True)\n    \n    if DELETE_EXISTING_CHECKPOINTS:\n        print(f\"--- Deleting existing checkpoints in {EXPORT_DIR} ---\")\n        for filename in [\"latest_checkpoint.pth\", \"best_checkpoint.pth\"]:\n            filepath = os.path.join(EXPORT_DIR, filename)\n            if os.path.isfile(filepath):\n                os.remove(filepath)\n                print(f\"  Removed: {filepath}\")\n            else:\n                print(f\"  No existing checkpoint found: {filepath}\")\n\ndef save_checkpoint(state, is_best, directory=\".\", filename=\"latest_checkpoint.pth\", best_filename=\"best_checkpoint.pth\"):\n    \"\"\"Saves model checkpoint.\"\"\"\n    filepath = os.path.join(directory, filename)\n    torch.save(state, filepath)\n    if is_best:\n        shutil.copyfile(filepath, os.path.join(directory, best_filename))\n        print(f\"    New best model saved to {best_filename}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.484351Z","iopub.execute_input":"2025-06-21T16:23:36.484574Z","iopub.status.idle":"2025-06-21T16:23:36.490529Z","shell.execute_reply.started":"2025-06-21T16:23:36.484557Z","shell.execute_reply":"2025-06-21T16:23:36.489914Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Step 2: Data Loading and Preprocessing","metadata":{}},{"cell_type":"code","source":"def load_and_preprocess_data():\n    \"\"\"Loads, processes, and prepares data for training.\"\"\"\n    print(\"▶ Loading and preprocessing data...\")\n    df = pd.read_csv(os.path.join(RAW_DIR, \"train.csv\"))\n\n    # Label Encoding\n    le = LabelEncoder()\n    df['gesture_int'] = le.fit_transform(df['gesture'])\n    np.save(os.path.join(EXPORT_DIR, \"gesture_classes.npy\"), le.classes_)\n    \n    # Feature Engineering\n    print(\"  - Engineering features...\")\n    df['acc_mag'] = np.sqrt(df['acc_x']**2 + df['acc_y']**2 + df['acc_z']**2)\n    df['rot_angle'] = 2 * np.arccos(df['rot_w'].clip(-1, 1))\n    df['acc_mag_jerk'] = df.groupby('sequence_id')['acc_mag'].diff().fillna(0)\n    df['rot_angle_vel'] = df.groupby('sequence_id')['rot_angle'].diff().fillna(0)\n    \n    tof_pixel_cols = [f\"tof_{i}_v{p}\" for i in range(1, 6) for p in range(64)]\n    reshaped_tof = df[tof_pixel_cols].replace(-1, np.nan).to_numpy().reshape(len(df), 5, 64)\n    with warnings.catch_warnings():\n        warnings.simplefilter(\"ignore\", category=RuntimeWarning)\n        mean_vals, std_vals = np.nanmean(reshaped_tof, axis=2), np.nanstd(reshaped_tof, axis=2)\n        min_vals, max_vals = np.nanmin(reshaped_tof, axis=2), np.nanmax(reshaped_tof, axis=2)\n    \n    tof_agg_cols = []\n    for i in range(1, 6):\n        df[f'tof_{i}_mean'], df[f'tof_{i}_std'] = mean_vals[:, i-1], std_vals[:, i-1]\n        df[f'tof_{i}_min'], df[f'tof_{i}_max'] = min_vals[:, i-1], max_vals[:, i-1]\n        tof_agg_cols.extend([f'tof_{i}_mean', f'tof_{i}_std', f'tof_{i}_min', f'tof_{i}_max'])\n\n    # Final feature selection and dimension calculation\n    imu_cols = [c for c in df.columns if c.startswith(('acc_', 'rot_'))]\n    thm_cols = [c for c in df.columns if c.startswith('thm_')]\n    final_feature_cols = imu_cols + thm_cols + tof_agg_cols\n    imu_dim = len(imu_cols)\n    tof_thm_dim = len(thm_cols) + len(tof_agg_cols)\n    np.save(os.path.join(EXPORT_DIR, \"feature_cols.npy\"), np.array(final_feature_cols))\n    \n    # Sequence creation, scaling, and padding\n    print(\"  - Building, scaling, and padding sequences...\")\n    X_list, y_list, lens = [], [], []\n    for _, seq_df in df.groupby('sequence_id'):\n        X_list.append(seq_df[final_feature_cols].ffill().bfill().fillna(0).values.astype('float32'))\n        y_list.append(seq_df['gesture_int'].iloc[0])\n        lens.append(len(seq_df))\n        \n    feature_scaler = StandardScaler().fit(np.concatenate(X_list, axis=0))\n    joblib.dump(feature_scaler, os.path.join(EXPORT_DIR, \"scaler.pkl\"))\n    X_scaled_list = [feature_scaler.transform(x) for x in X_list]\n    \n    pad_len = int(np.percentile(lens, PAD_PERCENTILE))\n    np.save(os.path.join(EXPORT_DIR, \"sequence_maxlen.npy\"), pad_len)\n    \n    X_padded = np.zeros((len(X_scaled_list), pad_len, len(final_feature_cols)), dtype='float32')\n    for i, seq in enumerate(X_scaled_list):\n        seq_len = min(len(seq), pad_len)\n        X_padded[i, :seq_len] = seq[:seq_len]\n\n    y_np = np.array(y_list)\n    y_one_hot = F.one_hot(torch.from_numpy(y_np).long(), num_classes=len(le.classes_)).float().numpy()\n    \n    # Train-validation split\n    X_tr, X_val, y_tr_oh, y_val_oh, y_tr_int, _ = train_test_split(\n        X_padded, y_one_hot, y_np, test_size=0.1, random_state=82, stratify=y_np\n    )\n    \n    # Create DataLoaders\n    train_dataset = MixupDataset(torch.from_numpy(X_tr).float(), torch.from_numpy(y_tr_oh).float(), alpha=MIXUP_ALPHA)\n    train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=os.cpu_count()//2 or 1, pin_memory=True)\n    \n    val_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_val).float(), torch.from_numpy(y_val_oh).float())\n    val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=os.cpu_count()//2 or 1, pin_memory=True)\n    \n    # Compute class weights\n    class_weights = compute_class_weight('balanced', classes=np.unique(y_tr_int), y=y_tr_int)\n    \n    # Return all necessary objects for training\n    return {\n        \"train_loader\": train_loader,\n        \"val_loader\": val_loader,\n        \"class_weights\": torch.from_numpy(class_weights).float(),\n        \"pad_len\": pad_len,\n        \"imu_dim\": imu_dim,\n        \"tof_thm_dim\": tof_thm_dim,\n        \"n_classes\": len(le.classes_)\n    }\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.491391Z","iopub.execute_input":"2025-06-21T16:23:36.491618Z","iopub.status.idle":"2025-06-21T16:23:36.521720Z","shell.execute_reply.started":"2025-06-21T16:23:36.491600Z","shell.execute_reply":"2025-06-21T16:23:36.521098Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Step 3: Model Training Pipeline","metadata":{}},{"cell_type":"code","source":"# Step 3a: Epoch-Level Training and Validation Functions\ndef train_epoch(model, loader, optimizer, scheduler, grad_scaler, criterion_weights, n_classes, device):\n    \"\"\"Handles the training loop for a single epoch.\"\"\"\n    model.train()\n    running_loss, correct_predictions, total_samples = 0.0, 0, 0\n    \n    for inputs, targets in loader:\n        inputs, targets = inputs.to(device), targets.to(device)\n        targets_idx = torch.argmax(targets, dim=1)\n        \n        with torch.cuda.amp.autocast(enabled=(device.type == 'cuda')):\n            outputs = model(inputs)\n            # Apply label smoothing and class weights\n            smoothed_targets = targets * (1 - 0.1) + (0.1 / n_classes)\n            per_sample_loss = F.cross_entropy(outputs, smoothed_targets, reduction='none')\n            loss = (per_sample_loss * criterion_weights[targets_idx]).mean()\n\n        optimizer.zero_grad(set_to_none=True)\n        grad_scaler.scale(loss).backward()\n        grad_scaler.step(optimizer)\n        grad_scaler.update()\n        scheduler.step()\n        \n        running_loss += loss.item() * inputs.size(0)\n        _, predicted = torch.max(outputs.data, 1)\n        total_samples += targets_idx.size(0)\n        correct_predictions += (predicted == targets_idx).sum().item()\n        \n    epoch_loss = running_loss / total_samples\n    epoch_acc = correct_predictions / total_samples\n    return epoch_loss, epoch_acc\n\ndef validate_epoch(model, loader, criterion_weights, device):\n    \"\"\"Handles the validation loop for a single epoch.\"\"\"\n    model.eval()\n    val_loss, val_correct, val_total = 0.0, 0, 0\n    with torch.no_grad():\n        for inputs, targets in loader:\n            inputs, targets = inputs.to(device), targets.to(device)\n            targets_idx = torch.argmax(targets, dim=1)\n            \n            with torch.cuda.amp.autocast(enabled=(device.type == 'cuda')):\n                outputs = model(inputs)\n                loss = F.cross_entropy(outputs, targets_idx, weight=criterion_weights)\n            \n            val_loss += loss.item() * inputs.size(0)\n            _, predicted = torch.max(outputs.data, 1)\n            val_total += targets_idx.size(0)\n            val_correct += (predicted == targets_idx).sum().item()\n            \n    epoch_acc = val_correct / val_total\n    epoch_loss = val_loss / val_total\n    return epoch_loss, epoch_acc","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:23:36.522539Z","iopub.execute_input":"2025-06-21T16:23:36.522759Z","iopub.status.idle":"2025-06-21T16:23:36.547436Z","shell.execute_reply.started":"2025-06-21T16:23:36.522742Z","shell.execute_reply":"2025-06-21T16:23:36.546588Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Step 3b: Main Training Orchestrator\ndef train_model(data):\n    \"\"\"Handles the entire model training and validation loop.\"\"\"\n    print(\"▶ Starting model training...\")\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    print(f\"  Using device: {device}\")\n    \n    class_weights = data['class_weights'].to(device)\n    \n    # Initialize model\n    model = TwoBranchModel(data['pad_len'], data['imu_dim'], data['tof_thm_dim'], data['n_classes'])\n    \n    # Optimizer and Scheduler\n    optimizer = optim.Adam(model.parameters(), lr=LR_INIT, weight_decay=WD)\n    lr_scheduler = LambdaLR(optimizer, lambda step: 0.5 * (1 + np.cos(np.pi * step / (EPOCHS * len(data['train_loader'])))))\n    grad_scaler = torch.cuda.amp.GradScaler(enabled=(device.type == 'cuda'))\n\n    # Load from checkpoint if available\n    start_epoch, best_val_accuracy = 0, -1.0\n    checkpoint_path = os.path.join(EXPORT_DIR, \"latest_checkpoint.pth\")\n    if os.path.exists(checkpoint_path):\n        print(f\"--- Resuming training from checkpoint: {checkpoint_path} ---\")\n        checkpoint = torch.load(checkpoint_path, map_location=device, weights_only=False)\n        state_dict = checkpoint['model_state_dict']\n        if any(k.startswith('_orig_mod.') for k in state_dict):\n            print(\"  - Cleaning state_dict keys from compiled model.\")\n            state_dict = {k.replace('_orig_mod.', ''): v for k, v in state_dict.items()}\n        model.load_state_dict(state_dict)\n        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n        lr_scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n        grad_scaler.load_state_dict(checkpoint['scaler_state_dict'])\n        start_epoch = checkpoint['epoch'] + 1\n        best_val_accuracy = checkpoint['best_val_accuracy']\n        print(f\"--- Resumed from epoch {start_epoch}. Best accuracy so far: {best_val_accuracy:.4f} ---\")\n    else:\n        print(\"--- No checkpoint found, starting training from scratch. ---\")\n\n    model.to(device)\n    try:\n        model = torch.compile(model)\n        print(\"  - Model compiled successfully.\")\n    except Exception as e:\n        print(f\"  Could not compile model: {e}. Running uncompiled.\")\n\n    # Training Loop\n    epochs_no_improve = 0\n    for epoch in range(start_epoch, EPOCHS):\n        train_loss, train_acc = train_epoch(\n            model, data['train_loader'], optimizer, lr_scheduler, grad_scaler, class_weights, data['n_classes'], device\n        )\n        val_loss, val_acc = validate_epoch(model, data['val_loader'], class_weights, device)\n\n        print(f\"Epoch {epoch+1}/{EPOCHS} | Train Loss: {train_loss:.4f} Acc: {train_acc:.4f} | Val Loss: {val_loss:.4f} Acc: {val_acc:.4f}\")\n        \n        # Save checkpoint and check for early stopping\n        is_best = val_acc > best_val_accuracy\n        if is_best:\n            best_val_accuracy = val_acc\n            epochs_no_improve = 0\n            print(f\"  Validation accuracy improved to {best_val_accuracy:.4f}.\")\n        else:\n            epochs_no_improve += 1\n        \n        model_state_to_save = model._orig_mod.state_dict() if hasattr(model, '_orig_mod') else model.state_dict()\n        checkpoint_state = {\n            'epoch': epoch, 'model_state_dict': model_state_to_save,\n            'optimizer_state_dict': optimizer.state_dict(), 'scheduler_state_dict': lr_scheduler.state_dict(),\n            'scaler_state_dict': grad_scaler.state_dict(), 'best_val_accuracy': best_val_accuracy\n        }\n        save_checkpoint(checkpoint_state, is_best=is_best, directory=EXPORT_DIR)\n        \n        if epochs_no_improve >= PATIENCE:\n            print(f\"  Early stopping after {PATIENCE} epochs without improvement.\")\n            break\n            \n    print(\"✔ Training done.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:45:08.029297Z","iopub.execute_input":"2025-06-21T16:45:08.029571Z","iopub.status.idle":"2025-06-21T16:45:08.042209Z","shell.execute_reply.started":"2025-06-21T16:45:08.029551Z","shell.execute_reply":"2025-06-21T16:45:08.041644Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Step 4: Train & Inference","metadata":{}},{"cell_type":"code","source":"if TRAIN:\n    setup_environment()\n    processed_data = load_and_preprocess_data()\n    train_model(processed_data)\n\n# --- INFERENCE SETUP ---\n# This block runs regardless of whether we just trained or are in a submission-only environment.\n# It ensures the `predict` function has everything it needs.\nprint(\"\\n▶ Setting up environment for inference...\")\n\n# Define global variables for the predict() function\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nmodel = None\nfeature_scaler = None\nfinal_feature_cols = None\npad_len = None\ngesture_classes = None\n\n# Determine which directory to load artifacts from\nartifact_dir = EXPORT_DIR if TRAIN else PRETRAINED_DIR\nprint(f\"  Loading artifacts from: {artifact_dir}\")\n\n# Load all necessary artifacts\nfinal_feature_cols = np.load(os.path.join(artifact_dir, \"feature_cols.npy\"), allow_pickle=True).tolist()\npad_len = int(np.load(os.path.join(artifact_dir, \"sequence_maxlen.npy\")))\ngesture_classes = np.load(os.path.join(artifact_dir, \"gesture_classes.npy\"), allow_pickle=True)\nfeature_scaler = joblib.load(os.path.join(artifact_dir, \"scaler.pkl\"))\n\n# Calculate model dimensions from loaded artifacts\nimu_dim = len([c for c in final_feature_cols if c.startswith(('acc_', 'rot_'))])\ntof_thm_dim = len(final_feature_cols) - imu_dim\nn_classes = len(gesture_classes)\n\n# Load the trained model state\nprint(\"  - Loading trained model weights...\")\nmodel = TwoBranchModel(pad_len, imu_dim, tof_thm_dim, n_classes)\nbest_model_path = os.path.join(artifact_dir, \"best_checkpoint.pth\")\ncheckpoint = torch.load(best_model_path, map_location=device, weights_only=False)\n\n# Handle potential model compilation keys\nstate_dict = checkpoint['model_state_dict']\nif any(k.startswith('_orig_mod.') for k in state_dict):\n    print(\"  - Cleaning state_dict keys from compiled model for inference.\")\n    state_dict = {k.replace('_orig_mod.', ''): v for k, v in state_dict.items()}\n\nmodel.load_state_dict(state_dict)\nmodel.to(device)\nmodel.eval()\nprint(\"✔ Inference environment ready.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:48:58.106050Z","iopub.execute_input":"2025-06-21T16:48:58.106880Z","iopub.status.idle":"2025-06-21T16:49:25.335943Z","shell.execute_reply.started":"2025-06-21T16:48:58.106852Z","shell.execute_reply":"2025-06-21T16:49:25.334713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Prediction","metadata":{}},{"cell_type":"code","source":"def predict(sequence: pl.DataFrame, demographics: pl.DataFrame) -> str:\n    \"\"\"\n    Predicts the gesture for a single sequence using the trained PyTorch model.\n    This is an optimized version with vectorized feature engineering.\n    \"\"\"\n    df_seq = sequence.to_pandas()\n\n    # --- OPTIMIZATION 1: Vectorized Feature Engineering ---\n    # IMU Features (this part was already efficient)\n    df_seq['acc_mag'] = np.sqrt(df_seq['acc_x']**2 + df_seq['acc_y']**2 + df_seq['acc_z']**2)\n    df_seq['rot_angle'] = 2 * np.arccos(df_seq['rot_w'].clip(-1, 1))\n    df_seq['acc_mag_jerk'] = df_seq['acc_mag'].diff().fillna(0)\n    df_seq['rot_angle_vel'] = df_seq['rot_angle'].diff().fillna(0)\n\n    # ToF Features (using the fast NumPy method)\n    tof_pixel_cols = [f\"tof_{i}_v{p}\" for i in range(1, 6) for p in range(64)]\n    tof_data_np = df_seq[tof_pixel_cols].replace(-1, np.nan).to_numpy()\n    reshaped_tof = tof_data_np.reshape(len(df_seq), 5, 64)\n    with warnings.catch_warnings():\n        warnings.filterwarnings('ignore', r'Mean of empty slice'); warnings.filterwarnings('ignore', r'Degrees of freedom <= 0 for slice')\n        mean_vals, std_vals = np.nanmean(reshaped_tof, axis=2), np.nanstd(reshaped_tof, axis=2)\n        min_vals, max_vals = np.nanmin(reshaped_tof, axis=2), np.nanmax(reshaped_tof, axis=2)\n    for i in range(1, 6):\n        df_seq[f'tof_{i}_mean'], df_seq[f'tof_{i}_std'] = mean_vals[:, i-1], std_vals[:, i-1]\n        df_seq[f'tof_{i}_min'], df_seq[f'tof_{i}_max'] = min_vals[:, i-1], max_vals[:, i-1]\n\n    # --- OPTIMIZATION 2: Efficient Column Reordering & Preprocessing ---\n    # Select columns in the correct order and fill any missing values\n    mat_unscaled = df_seq[final_feature_cols].ffill().bfill().fillna(0).values.astype('float32')\n\n    # --- FIX: Use the correct 'feature_scaler' variable ---\n    mat_scaled = feature_scaler.transform(mat_unscaled)\n\n    # --- Padding ---\n    padded_sequence = np.zeros((pad_len, len(final_feature_cols)), dtype='float32')\n    seq_len = min(len(mat_scaled), pad_len)\n    padded_sequence[:seq_len] = mat_scaled[:seq_len]\n\n    # --- Prepare for Model ---\n    model_input = torch.from_numpy(padded_sequence).float().unsqueeze(0).to(device)\n\n    # --- Model Inference ---\n    model.eval()\n    model.to(device)\n    with torch.no_grad():\n        outputs = model(model_input) # Get raw logits\n        # --- OPTIMIZATION 3: No need for extra softmax ---\n        predicted_idx = torch.argmax(outputs, dim=1).item()\n\n    # --- Map to Gesture Class ---\n    return str(gesture_classes[predicted_idx])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:46:25.452986Z","iopub.execute_input":"2025-06-21T16:46:25.453334Z","iopub.status.idle":"2025-06-21T16:46:25.463634Z","shell.execute_reply.started":"2025-06-21T16:46:25.453270Z","shell.execute_reply":"2025-06-21T16:46:25.462973Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Submit Inference server","metadata":{}},{"cell_type":"code","source":"import kaggle_evaluation.cmi_inference_server\n\ninference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n    inference_server.serve()\nelse:\n    inference_server.run_local_gateway(\n        data_paths=(\n            '/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv',\n            '/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv',\n        )\n    )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-21T16:46:30.011829Z","iopub.execute_input":"2025-06-21T16:46:30.012436Z","iopub.status.idle":"2025-06-21T16:46:33.510796Z","shell.execute_reply.started":"2025-06-21T16:46:30.012412Z","shell.execute_reply":"2025-06-21T16:46:33.510160Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}