{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"isSourceIdPinned":false,"sourceType":"competition"},{"sourceId":244801492,"sourceType":"kernelVersion"}],"dockerImageVersionId":31041,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# LSTM Ensemble for Human Activity Recognition (Paper Implementation)\n\nThis notebook implements an advanced ensemble learning framework for Human Activity Recognition (HAR) based on the methods described in the following research paper:\n\n**Paper:** [Ensemble of deep LSTM learners for semi-supervised human activity recognition](https://arxiv.org/pdf/1703.09370)\n\n## Concept\n\nThe core idea, inspired by the paper, is that an **ensemble of diverse LSTM models** is more robust and accurate than any single model, especially when dealing with noisy and imbalanced real-world sensor data like the one in this competition.\n\nThis script trains multiple LSTM models (`CFG.N_MODELS`) that will later be combined for inference.\n\n## Key Implementation Points\n\nThis notebook translates the core ideas of the paper into code:\n\n### 1. Ensemble of LSTM Models\nInstead of training a single model, this script initializes and trains multiple independent `LSTM_Model` instances in a loop. Each model will learn slightly different aspects of the data, and their collective prediction will be more reliable.\n\n### 2. Epoch-wise Bagging\nThis is the paper's most novel technique for ensuring diversity among the ensemble members. Instead of using a fixed data loader, we **re-create the `DataLoader` at the beginning of every single epoch**.\n\n-   **Random Batch Size:** The batch size for each epoch is randomly selected from a range (`CFG.BATCH_SIZE_RANGE`).\n-   **Random Sequence Length:** The length of the sequences fed to the model is also randomized in each epoch (`CFG.SEQ_LEN_RANGE`).\n\nThis process ensures that each model sees slightly different \"views\" of the data in every epoch, which effectively prevents overfitting and encourages the creation of diverse base models for the ensemble.\n\n### 3. Hybrid Loss Function\nTo further enhance model diversity, we use a hybrid loss function. Instead of relying solely on the standard `CrossEntropyLoss`, we combine it with a custom, differentiable `F1Loss`.\n\n-   The final loss is a weighted average: \n    `loss = alpha * CrossEntropyLoss + (1 - alpha) * F1Loss`\n-   This forces some models to focus on optimizing the F1-score directly, making them behave differently from models that only optimize for classification accuracy.\n\n### 4. Score-level Fusion (for Inference)\nThis training notebook saves all the trained LSTM models as separate `.pth` files. In the final inference step (in a different notebook), the prediction probabilities from all these models will be loaded and **averaged** to produce a single, robust final prediction. This final step is known as \"score-level fusion.\"","metadata":{}},{"cell_type":"code","source":"# import polars as pl\n# import pandas as pd\n# import numpy as np\n# import torch\n# import torch.nn as nn\n# import torch.nn.functional as F\n# from torch.utils.data import Dataset, DataLoader\n# from torch.optim.lr_scheduler import CosineAnnealingLR\n# from sklearn.model_selection import StratifiedGroupKFold\n# from sklearn.preprocessing import LabelEncoder\n# from tqdm.auto import tqdm\n# import joblib\n# import os\n# import random\n# import warnings\n\n# warnings.filterwarnings(\"ignore\")\n\n# # --- 1. Configuration Block ---\n# class CFG:\n#     # --- Final Ensemble Configuration ---\n#     N_MODELS = 10 \n#     EPOCHS = 50 # Final training epochs\n    \n#     # --- Winning Architecture: \"2-Layer Wider + Higher Dropout\" ---\n#     LSTM_UNITS = [256, 256]\n#     LSTM_LAYERS = len(LSTM_UNITS) # Automatically determined\n#     LSTM_DROPOUT = 0.6\n    \n#     # --- Training Parameters (Using best known settings) ---\n#     LEARNING_RATE = 1e-3\n#     WEIGHT_DECAY = 1e-5\n#     LOSS_ALPHA = 0.7 \n\n#     # --- Epoch-wise Bagging Parameters ---\n#     BATCH_SIZE_RANGE = (32, 128)\n#     SEQ_LEN_RANGE = (96, 128)\n\n#     # --- Fixed Parameters ---\n#     N_SPLITS = 5\n#     N_CLASSES = 18\n#     IMU_DIM = 7\n#     DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n#     SEED = 42\n\n# def seed_everything(seed):\n#     random.seed(seed); os.environ['PYTHONHASHSEED'] = str(seed); np.random.seed(seed)\n#     torch.manual_seed(seed); torch.cuda.manual_seed(seed)\n#     torch.backends.cudnn.deterministic = True; torch.backends.cudnn.benchmark = True\n\n# seed_everything(CFG.SEED)\n\n# # --- 2. Helper Functions & Classes ---\n# class F1Loss(nn.Module):\n#     def __init__(self, epsilon=1e-7):\n#         super().__init__(); self.epsilon = epsilon\n#     def forward(self, y_pred, y_true):\n#         y_pred_softmax = torch.softmax(y_pred, dim=1)\n#         y_true_one_hot = F.one_hot(y_true, num_classes=CFG.N_CLASSES).float()\n#         tp = (y_true_one_hot * y_pred_softmax).sum(dim=0)\n#         fp = ((1 - y_true_one_hot) * y_pred_softmax).sum(dim=0)\n#         fn = (y_true_one_hot * (1 - y_pred_softmax)).sum(dim=0)\n#         precision = tp / (tp + fp + self.epsilon); recall = tp / (tp + fn + self.epsilon)\n#         f1 = 2 * (precision * recall) / (precision + recall + self.epsilon)\n#         return 1 - f1.clamp(min=self.epsilon, max=1-self.epsilon).mean()\n\n# class FullSequenceDataset(Dataset):\n#     def __init__(self, samples, indices, is_train=False):\n#         self.samples = samples; self.indices = indices; self.is_train = is_train\n#     def __len__(self):\n#         return len(self.indices)\n#     def __getitem__(self, idx):\n#         true_idx = self.indices[idx]\n#         data, label = self.samples[true_idx]['data'], self.samples[true_idx]['label']\n#         if self.is_train:\n#             noise = np.random.normal(0, 0.05, data.shape).astype(np.float32); data = data + noise\n#         return data, label\n\n# def create_collate_fn(seq_len, input_dim):\n#     def collate_fn(batch):\n#         sequences, labels = zip(*batch)\n#         padded_sequences = []\n#         for seq in sequences:\n#             current_len = seq.shape[1]\n#             if current_len < seq_len:\n#                 padding = np.zeros((input_dim, seq_len - current_len), dtype=np.float32)\n#                 padded_seq = np.concatenate([seq, padding], axis=1)\n#             else:\n#                 start_point = np.random.randint(0, current_len - seq_len + 1)\n#                 padded_seq = seq[:, start_point : start_point + seq_len]\n#             padded_sequences.append(padded_seq)\n#         return torch.tensor(np.array(padded_sequences), dtype=torch.float32), torch.tensor(labels, dtype=torch.long)\n#     return collate_fn\n\n# class LSTM_Model_Flexible(nn.Module):\n#     def __init__(self, cfg):\n#         super().__init__()\n#         self.lstm_layers = nn.ModuleList()\n#         input_dim = cfg.IMU_DIM\n#         for i, hidden_dim in enumerate(cfg.LSTM_UNITS):\n#             dropout_val = cfg.LSTM_DROPOUT if i < len(cfg.LSTM_UNITS) - 1 else 0\n#             self.lstm_layers.append(\n#                 nn.LSTM(input_dim, hidden_dim, num_layers=1, batch_first=True, dropout=dropout_val, bidirectional=True)\n#             )\n#             input_dim = hidden_dim * 2\n#         self.classifier = nn.Linear(input_dim, cfg.N_CLASSES)\n\n#     def forward(self, x):\n#         x = x.permute(0, 2, 1)\n#         for lstm_layer in self.lstm_layers:\n#             x, _ = lstm_layer(x)\n#         return self.classifier(x[:, -1, :])\n\n# # --- 3. Main Training Execution ---\n# print(\"Loading and preparing data into a unified list...\")\n# train_df = pl.read_csv('/kaggle/input/cmi-detect-behavior-with-sensor-data/train.csv')\n# imu_cols = [col for col in train_df.columns if 'acc_' in col or 'rot_' in col]\n# all_samples = []\n# le = LabelEncoder().fit(train_df['gesture'].unique().sort().to_numpy())\n# for seq_id, group_df in tqdm(train_df.group_by(\"sequence_id\", maintain_order=True), desc=\"Processing Sequences\"):\n#     imu_data = group_df[imu_cols].fill_null(0.0).to_numpy().T.astype(np.float32)\n#     all_samples.append({\"data\": imu_data, \"label\": le.transform([group_df['gesture'][0]])[0], \"group\": group_df['subject'][0]})\n\n# train_indices = np.arange(len(all_samples))\n\n# print(f\"\\n--- Starting Final Ensemble Training for {CFG.N_MODELS} models ---\")\n# for model_idx in range(CFG.N_MODELS):\n#     print(f\"\\n\" + \"=\"*20 + f\" Training Model {model_idx+1}/{CFG.N_MODELS} \" + \"=\"*20)\n    \n#     model = LSTM_Model_Flexible(CFG).to(CFG.DEVICE)\n#     optimizer = torch.optim.AdamW(model.parameters(), lr=CFG.LEARNING_RATE, weight_decay=CFG.WEIGHT_DECAY)\n#     scheduler = CosineAnnealingLR(optimizer, T_max=CFG.EPOCHS, eta_min=1e-6)\n#     criterion_ce, criterion_f1 = nn.CrossEntropyLoss(), F1Loss()\n\n#     for epoch in range(CFG.EPOCHS):\n#         epoch_batch_size = random.randint(*CFG.BATCH_SIZE_RANGE)\n#         epoch_seq_len = random.randint(*CFG.SEQ_LEN_RANGE)\n        \n#         train_dataset = FullSequenceDataset(all_samples, train_indices, is_train=True)\n#         train_loader = DataLoader(\n#             train_dataset, batch_size=epoch_batch_size, shuffle=True, \n#             num_workers=0, collate_fn=create_collate_fn(epoch_seq_len, CFG.IMU_DIM)\n#         )\n        \n#         model.train()\n#         pbar = tqdm(train_loader, desc=f\"Epoch {epoch+1}/{CFG.EPOCHS} [Model {model_idx+1}]\")\n#         for data, batch_labels in pbar:\n#             data, batch_labels = data.to(CFG.DEVICE), batch_labels.to(CFG.DEVICE)\n#             optimizer.zero_grad()\n#             outputs = model(data)\n#             loss = CFG.LOSS_ALPHA * criterion_ce(outputs, batch_labels) + (1 - CFG.LOSS_ALPHA) * criterion_f1(outputs, batch_labels)\n#             loss.backward()\n#             optimizer.step()\n#             pbar.set_postfix(loss=f\"{loss.item():.4f}\")\n        \n#         scheduler.step() # Step the scheduler at the end of each epoch\n            \n#     model_save_path = f'lstm_ensemble_model_{model_idx}.pth'\n#     print(f\"Finished training model {model_idx+1}. Saving to {model_save_path}\")\n#     torch.save(model.state_dict(), model_save_path)\n\n# joblib.dump(le, 'lstm_ensemble_label_encoder.pkl')\n# print(\"\\n--- All models for the LSTM ensemble have been trained and saved successfully! ---\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-06-11T05:23:22.224818Z","iopub.execute_input":"2025-06-11T05:23:22.225082Z","iopub.status.idle":"2025-06-11T05:23:22.231307Z","shell.execute_reply.started":"2025-06-11T05:23:22.225062Z","shell.execute_reply":"2025-06-11T05:23:22.230557Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport polars as pl\nimport numpy as np\nimport torch\nimport torch.nn as nn\nfrom sklearn.preprocessing import LabelEncoder\nimport joblib\nimport kaggle_evaluation.cmi_inference_server\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")\n\n# --- 1. Configuration & Model Definition ---\nclass CFG:\n    N_MODELS = 10 # The number of models in the ensemble\n    \n    # --- Data Shape (must match training) ---\n    MAX_LEN = 128 \n    N_CLASSES = 18\n    IMU_DIM = 7\n    \n    # --- Architecture (must match training) ---\n    LSTM_UNITS = [256, 256]\n    LSTM_DROPOUT = 0.6\n    \n    # --- Environment ---\n    DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# The model architecture MUST be defined to load the weights.\nclass LSTM_Model_Flexible(nn.Module):\n    def __init__(self, cfg):\n        super().__init__()\n        self.lstm_layers = nn.ModuleList()\n        input_dim = cfg.IMU_DIM\n        \n        for i, hidden_dim in enumerate(cfg.LSTM_UNITS):\n            dropout_val = cfg.LSTM_DROPOUT if i < len(cfg.LSTM_UNITS) - 1 else 0\n            self.lstm_layers.append(\n                nn.LSTM(\n                    input_dim, hidden_dim, num_layers=1, batch_first=True,\n                    dropout=dropout_val, bidirectional=True\n                )\n            )\n            input_dim = hidden_dim * 2\n            \n        self.classifier = nn.Linear(input_dim, cfg.N_CLASSES)\n\n    def forward(self, x):\n        x = x.permute(0, 2, 1)\n        for lstm_layer in self.lstm_layers:\n            x, _ = lstm_layer(x)\n        out = self.classifier(x[:, -1, :])\n        return out\n\n# --- 2. Load All Pre-trained Models and Artifacts ---\nINPUT_DIR = '/kaggle/input/cmi2025-lstm-ensemble/' \n\nprint(f\"Loading {CFG.N_MODELS} LSTM ensemble models and assets...\")\nmodels = []\nfor i in range(CFG.N_MODELS):\n    model = LSTM_Model_Flexible(CFG).to(CFG.DEVICE)\n    model_path = os.path.join(INPUT_DIR, f'lstm_ensemble_model_{i}.pth')\n    model.load_state_dict(torch.load(model_path, map_location=CFG.DEVICE))\n    model.eval()\n    models.append(model)\n\nle = joblib.load(os.path.join(INPUT_DIR, 'lstm_ensemble_label_encoder.pkl'))\n\nprint(f\"All {len(models)} models loaded successfully.\")\n\n\n# --- 3. Define Predict Function ---\ndef prepare_inference_tensor(sequence: pl.DataFrame) -> torch.Tensor:\n    \"\"\"Prepares a single raw sequence into a padded tensor for inference.\"\"\"\n    imu_cols = [col for col in sequence.columns if 'acc_' in col or 'rot_' in col]\n    imu_data_lists = [sequence.get_column(col).fill_null(0.0).to_numpy() for col in imu_cols]\n    imu_data_np = np.array(imu_data_lists, dtype=np.float32)\n    \n    current_len = imu_data_np.shape[1]\n    if current_len < CFG.MAX_LEN:\n        padding = np.zeros((CFG.IMU_DIM, CFG.MAX_LEN - current_len), dtype=np.float32)\n        data_padded = np.concatenate([imu_data_np, padding], axis=1)\n    else:\n        data_padded = imu_data_np[:, -CFG.MAX_LEN:] # Take the last part of the sequence\n        \n    return torch.tensor(data_padded, dtype=torch.float32).unsqueeze(0).to(CFG.DEVICE)\n\n\ndef predict(sequence: pl.DataFrame, demographics: pl.DataFrame) -> str:\n    \"\"\"\n    Predicts a gesture by ensembling the predictions of all loaded LSTM models.\n    \"\"\"\n    # 1. Prepare the input tensor\n    X_test = prepare_inference_tensor(sequence)\n    \n    # 2. Get predictions from all models\n    all_probs = []\n    with torch.no_grad():\n        for model in models:\n            outputs = model(X_test)\n            probs = torch.softmax(outputs, dim=1)\n            all_probs.append(probs)\n            \n    # 3. Score-level Fusion: Average the probabilities across all models\n    avg_probs = torch.mean(torch.stack(all_probs), dim=0).cpu().numpy()\n    \n    # 4. Get the final prediction\n    final_pred_idx = np.argmax(avg_probs, axis=1)\n    \n    # 5. Convert index back to string label\n    pred_label = le.inverse_transform(final_pred_idx)[0]\n    \n    return pred_label\n\n# --- 4. Start the Inference Server ---\nprint(\"Starting inference server...\")\ninference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n    inference_server.serve()\nelse:\n    inference_server.run_local_gateway(\n        data_paths=(\n            '/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv',\n            '/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv',\n        )\n    )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-11T05:23:22.237000Z","iopub.execute_input":"2025-06-11T05:23:22.237230Z","iopub.status.idle":"2025-06-11T05:23:25.881024Z","shell.execute_reply.started":"2025-06-11T05:23:22.237215Z","shell.execute_reply":"2025-06-11T05:23:25.880425Z"}},"outputs":[],"execution_count":null}]}