{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.13"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":102335},{"sourceType":"datasetVersion","sourceId":13108724},{"sourceType":"datasetVersion","sourceId":12917259},{"sourceType":"datasetVersion","sourceId":12909581},{"sourceType":"datasetVersion","sourceId":12938086},{"sourceType":"datasetVersion","sourceId":12917342},{"sourceType":"datasetVersion","sourceId":12917332},{"sourceType":"datasetVersion","sourceId":12920136},{"sourceType":"datasetVersion","sourceId":12929851},{"sourceType":"datasetVersion","sourceId":12898879},{"sourceType":"datasetVersion","sourceId":13078963},{"sourceType":"datasetVersion","sourceId":12932896},{"sourceType":"datasetVersion","sourceId":12888675},{"sourceType":"kernelVersion","sourceId":259543862},{"sourceType":"kernelVersion","sourceId":261256427}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":372.214826,"end_time":"2026-04-15T09:02:06.507119","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2026-04-15T08:55:54.292293","version":"2.6.0"},"vscode":{"interpreter":{"hash":"f7241b2af102f7e024509099765066b36197b195077f7bfac6e5bc041ba17c8c"}}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"84140563-5c90-434b-8f39-ac7542cc4b96","cell_type":"markdown","source":"# CMI Sensor Gesture Classification\n### Metric-aware Theo ensemble with separate full-sensor and IMU-only inference\n\nThis notebook keeps the strong Theo checkpoint pipeline and makes the experimentation easier to understand and validate.\n\n**Main ideas**\n\n- use separate model families for full-sensor and IMU-only sequences\n- tune ensemble weights from out-of-fold predictions\n- optimize the BFRB vs non-BFRB boundary instead of relying only on 18-class argmax\n- keep validation diagnostics close to the model decisions\n- inspect class balance, sequence length, sensor availability, ensemble weights, confusion patterns and prediction confidence\n\n> OOF predictions are used only for ensemble/threshold selection. Final predictions still come from the trained checkpoints.\n","metadata":{}},{"id":"034c5e02-9ee6-43d0-a3c5-d656e2d5c69e","cell_type":"markdown","source":"## 1. Run configuration\n\nI keep the main switches in one place so it is easy to move between local validation, debugging and Kaggle rerun mode.\n","metadata":{}},{"id":"f479d7ed-5828-4a85-a139-713c20594b53","cell_type":"code","source":"EVAL = True\nDEBUG = False\n\nimport os\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n    EVAL = False\n    DEBUG = False\n\nIMU_ONLY = False\nFULLFIT = True\n\n# small OOF tuning for the ensemble\nUSE_OOF_BLEND = True\nN_BLEND_TRIALS = 220\nMAX_IMU_MIX = 0.30\n\nORIENTATION_PP = False\nLIM = 2\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:41.696313Z","iopub.execute_input":"2026-08-17T10:37:41.696575Z","iopub.status.idle":"2026-08-17T10:37:41.701919Z","shell.execute_reply.started":"2026-08-17T10:37:41.696558Z","shell.execute_reply":"2026-08-17T10:37:41.701052Z"}},"outputs":[],"execution_count":null},{"id":"ac4584c1-a425-4355-aa29-f36d27eaa86e","cell_type":"markdown","source":"## 2. Imports and competition utilities\n\nThe notebook uses the original competition code package for preprocessing, model creation and the hierarchical F1 metric.\n","metadata":{}},{"id":"073b964a-4e0b-47ac-82a3-db977fa1902e","cell_type":"code","source":"import os\nimport gc\nimport sys\nimport glob\nimport json\nfrom pathlib import Path\n\nimport torch\nimport shutil\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\nfrom scipy.special import softmax\nfrom collections import defaultdict\n\nwarnings.simplefilter(action=\"ignore\", category=UserWarning)\npd.set_option('display.max_columns', 100)\npd.set_option(\"display.precision\", 5)\nfrom IPython.display import display, Markdown\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:43.816581Z","iopub.execute_input":"2026-08-17T10:37:43.817662Z","iopub.status.idle":"2026-08-17T10:37:43.823914Z","shell.execute_reply.started":"2026-08-17T10:37:43.817595Z","shell.execute_reply":"2026-08-17T10:37:43.823190Z"}},"outputs":[],"execution_count":null},{"id":"b6bace5f-0834-423b-ad69-60b30c642627","cell_type":"code","source":"sys.path.append(\"/kaggle/input/cmi-behavior-code/src/\")\n\nfrom params import *\nfrom util.logger import Config\nfrom util.metrics import hierarchical_f1\nfrom util.torch import load_model_weights\nfrom model_zoo.models import define_model\nfrom inference.predict import predict as predict_\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:44.920886Z","iopub.execute_input":"2026-08-17T10:37:44.921542Z","iopub.status.idle":"2026-08-17T10:37:44.925545Z","shell.execute_reply.started":"2026-08-17T10:37:44.921517Z","shell.execute_reply":"2026-08-17T10:37:44.924827Z"}},"outputs":[],"execution_count":null},{"id":"0a19a883-2b99-4875-a85f-cf0c15267794","cell_type":"markdown","source":"## 3. Theo predictor wrapper\n\n`TheoPredictor` handles the preprocessing expected by each saved experiment, loads its fold checkpoints, and returns probabilities in one consistent format.\n\nKeeping this logic in one wrapper makes the ensemble code much easier to read.\n","metadata":{}},{"id":"c8b4c6f5-9110-4ca4-adaa-59698c102fbd","cell_type":"code","source":"from inference.utils import process, handle_imu_nans, gravity_only_acc\nfrom data.processing import flip_acc_rot\n\n\nclass TheoPredictor:\n    def __init__(self, exp_folder, folds, imu_only=False, device=\"cuda\", use_fp16=True, orientation=False):\n        self.exp_folder = exp_folder\n        self.folds = folds\n        self.device = device\n        self.use_fp16 = use_fp16\n        self.imu_only = imu_only\n        self.orientation = orientation\n\n        config = Config(json.load(open(exp_folder + \"config.json\", \"r\")))\n\n        self.max_len = config.max_len\n        self.resize = config.resize if hasattr(config, \"resize\") else False\n        self.fix_quat = config.fix_quat\n\n        self.models = []\n        for fold in folds:\n            weights = exp_folder + f\"{config.name}_{fold}.pt\"\n    \n            model = define_model(\n                config.name,\n                config.model_args,\n                num_classes=config.num_classes,\n                verbose=True,\n                pretrained=False,\n                config_dir=CFG_FOLDER,\n                cnn_2d=config.cnn_2d if hasattr(config, \"cnn_2d\") else False,\n            )\n            model = model.to(device).eval()\n    \n            model = load_model_weights(model, weights, verbose=True)\n            self.models.append(model)        \n\n    def preprocess(self, sequence, demographics):\n        fts = {\n            \"imu\": sequence[IMU_FEATURES].values.astype(np.float32),\n        }    \n        fts[\"imu\"] = handle_imu_nans(fts[\"imu\"])\n    \n        if not self.imu_only:\n            fts[\"thm\"] = sequence[THM_FEATURES].values.astype(np.float32)\n            fts[\"tof\"] = sequence[TOF_FEATURES].fillna(FILL_VALUES['tof']).values.astype(np.uint8)\n    \n        handedness = demographics.handedness.values[0]\n        if handedness == 0:\n            try:\n                fts[\"imu\"][:, 3:] = flip_acc_rot(fts[\"imu\"][:, 3:].copy())\n                fts[\"imu\"][:, 0] *= -1\n            except ValueError:\n                # print(\"ValueError\")\n                if self.fix_quat:\n                    fts[\"imu\"][:, [0, 4, 5]] *= -1\n                else:\n                    fts[\"imu\"][:, [0, 3, 6]] *= -1\n    \n            if \"thm\" in fts:\n                fts[\"thm\"] = fts[\"thm\"][:, [0, 1, 4, 3, 2]]\n    \n            if \"tof\" in fts:\n                tof = fts[\"tof\"].reshape(-1, 5, 8, 8)\n                tof = tof[:, [0, 1, 4, 3, 2]]\n                tof[:, [0, 1, 3]] = tof[:, [0, 1, 3], :, ::-1]\n                tof[:, [2, 4]] = tof[:, [2, 4], ::-1]\n                tof = tof.reshape(-1, 320)\n                fts[\"tof\"] = tof\n    \n        fts = process(\n            fts,\n            self.max_len,\n            demographics,\n            resize=self.resize,\n            fix_quat=self.fix_quat,\n            device=self.device\n        )\n        return fts\n\n    def infer(self, fts, activation=\"softmax\"):\n        all_preds = []\n        for model in self.models:\n            with torch.inference_mode(), torch.amp.autocast(\"cuda\", enabled=self.use_fp16):\n                if self.orientation:\n                    pred = model(fts)[2]\n                else:\n                    pred = model(fts)[0]\n            pred = pred[0].cpu()\n            if activation == \"softmax\":\n                pred = pred.softmax(0)\n            pred = pred.numpy().astype(np.float64)\n            all_preds.append(pred)\n        return all_preds\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:47.249370Z","iopub.execute_input":"2026-08-17T10:37:47.249932Z","iopub.status.idle":"2026-08-17T10:37:47.261872Z","shell.execute_reply.started":"2026-08-17T10:37:47.249904Z","shell.execute_reply":"2026-08-17T10:37:47.261136Z"}},"outputs":[],"execution_count":null},{"id":"e742407b-a5e5-486b-b449-64b43c506d98","cell_type":"markdown","source":"## 4. Experiment configuration\n\nValidation uses a single fold for fast local checking, while Kaggle reruns can use the full-fit checkpoints configured in each experiment folder.\n","metadata":{}},{"id":"0dada2de-36ce-4e96-96ef-92b94cc54d07","cell_type":"code","source":"FOLDS = [i for i in range(5)]\nif EVAL:\n    FULLFIT = False\n    FOLDS = [0] \nif DEBUG:\n    FULLFIT = False\n    FOLDS = [3]\nFOLDS\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:50.129153Z","iopub.execute_input":"2026-08-17T10:37:50.129893Z","iopub.status.idle":"2026-08-17T10:37:50.135465Z","shell.execute_reply.started":"2026-08-17T10:37:50.129866Z","shell.execute_reply":"2026-08-17T10:37:50.134581Z"}},"outputs":[],"execution_count":null},{"id":"8357fd6f-7b77-4856-993a-378c4d4325f0","cell_type":"code","source":"DATA_PATH = \"/kaggle/input/cmi-detect-behavior-with-sensor-data/\"\nDATASET = \"/kaggle/input/cmi-weights-1/\"\nCFG_FOLDER = \"/kaggle/input/cmi-behavior-code/src/configs/\"\n\nUSE_FP16 = True\nNUM_WORKERS = 8\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:51.501465Z","iopub.execute_input":"2026-08-17T10:37:51.502100Z","iopub.status.idle":"2026-08-17T10:37:51.505905Z","shell.execute_reply.started":"2026-08-17T10:37:51.502073Z","shell.execute_reply":"2026-08-17T10:37:51.505094Z"}},"outputs":[],"execution_count":null},{"id":"9473fc4a-516d-4174-926a-3bf2f2f88f37","cell_type":"markdown","source":"### 4.1 IMU-only model family\n\nThese models are used when thermal / ToF information is unavailable. Their weights are tuned independently because the IMU-only problem behaves differently from the full-sensor problem.\n","metadata":{}},{"id":"c36eb060-b20c-4efb-af82-fedb56dcd71c","cell_type":"code","source":"EXP_FOLDERS_IMU = [\n    # (\"/kaggle/input/cmi-weights-2/2025-08-28_47/\", [f'fullfit_{i}' for i in range(3)] if FULLFIT else FOLDS),  # deberta-rnn      - 0.8321   ->3 FF\n    (\"/kaggle/input/cmi-weights-2/2025-08-28_19/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # maxvit-base 224  - 0.8331\n    (\"/kaggle/input/cmi-weights-2/2025-08-28_38/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # convnext 224     - 0.8299\n    (\"/kaggle/input/cmi-weights-2/2025-08-28_41/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # maxvit-base 384  - 0.8319\n]\n\nIMU_MODEL_NAMES = [\n    \"MaxViT 224\",\n    \"ConvNeXt 224\",\n    \"MaxViT 384\",\n]\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:53.853182Z","iopub.execute_input":"2026-08-17T10:37:53.853427Z","iopub.status.idle":"2026-08-17T10:37:53.858166Z","shell.execute_reply.started":"2026-08-17T10:37:53.853410Z","shell.execute_reply":"2026-08-17T10:37:53.857308Z"}},"outputs":[],"execution_count":null},{"id":"e1079294-91de-4ad0-adc2-5ccb58c8901f","cell_type":"code","source":"models_imu = []\nfor exp_folder, folds in EXP_FOLDERS_IMU:\n    models_imu.append(\n        TheoPredictor(\n            exp_folder,\n            folds,\n            imu_only=True,\n            device=DEVICE,\n            use_fp16=USE_FP16\n        )\n    )\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:37:55.135045Z","iopub.execute_input":"2026-08-17T10:37:55.135813Z","iopub.status.idle":"2026-08-17T10:38:13.688177Z","shell.execute_reply.started":"2026-08-17T10:37:55.135789Z","shell.execute_reply":"2026-08-17T10:38:13.687445Z"}},"outputs":[],"execution_count":null},{"id":"39a5483e-ffd2-4361-af2f-0771b2beab0e","cell_type":"markdown","source":"### 4.2 Full-sensor model family\n\nThe full ensemble mixes sequence and image-style architectures. The goal is not only to average strong models, but also to exploit their different mistakes.\n","metadata":{}},{"id":"29fbd4f2-3b3d-4ab4-b0da-406c05902fc9","cell_type":"code","source":"# EXP_FOLDERS = [\n#     (\"/kaggle/input/cmi-weights-2/2025-08-29_0/\",  [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # deberta          - 0.8845\n#     (\"/kaggle/input/cmi-weights-1/2025-08-29_44/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # deberta-rnn      - 0.8805\n#     # (\"/kaggle/input/cmi-weights-1/2025-08-29_45/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # modernbert       - 0.8806\n#     (\"/kaggle/input/cmi-weights-1/2025-08-29_47/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # Squeezeformer    - 0.8830\n\n#     (\"/kaggle/input/cmi-weights-2/2025-08-29_23/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # maxvit-base 224  - 0.8832\n#     (\"/kaggle/input/cmi-weights-2/2025-08-29_24/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # convnext 224     - 0.8790\n#     (\"/kaggle/input/cmi-weights-2/2025-08-29_25/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # maxvit-base 384  - 0.8827\n# ]\n\nEXP_FOLDERS = [\n    (\"/kaggle/input/cmi-weights-2/2025-08-29_0/\",  [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # deberta             - 0.8845\n    (\"/kaggle/input/cmi-weights-2/2025-08-29_25/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # maxvit-base 384     - 0.8827\n    (\"/kaggle/input/cmi-weights-3/2025-08-30_44/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # maxvit-base 224     - 0.8848\n    (\"/kaggle/input/cmi-weights-3/2025-08-31_7/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),   # convnext224 len120  - 0.8890\n    (\"/kaggle/input/cmi-weights-3/2025-08-31_10/\", [f'fullfit_{i}' for i in range(1)] if FULLFIT else FOLDS),  # convnext224 len110  - 0.8863\n]\n\n\nif IMU_ONLY: \n    EXP_FOLDERS = []\n\nFULL_MODEL_NAMES = [\n    \"DeBERTa\",\n    \"MaxViT 384\",\n    \"MaxViT 224\",\n    \"ConvNeXt L120\",\n    \"ConvNeXt L110\",\n]\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:38:51.137141Z","iopub.execute_input":"2026-08-17T10:38:51.137727Z","iopub.status.idle":"2026-08-17T10:38:51.143099Z","shell.execute_reply.started":"2026-08-17T10:38:51.137695Z","shell.execute_reply":"2026-08-17T10:38:51.142428Z"}},"outputs":[],"execution_count":null},{"id":"77cdb5fd-5507-4216-8afc-61c05d1541eb","cell_type":"markdown","source":"## 5. Metric-aware OOF blending\n\nA plain 18-class argmax does not fully reflect the competition objective.\n\nThe competition metric cares about two things:\n\n1. whether a sequence belongs to one of the BFRB classes\n2. which BFRB gesture is predicted when it is a target behavior\n\nSo the notebook first sums the probability mass of the eight BFRB classes. A threshold decides **BFRB vs non-BFRB**, and the best class inside the selected group is then returned.\n\nThe threshold and model weights are tuned only on OOF predictions.\n","metadata":{}},{"id":"f45a741f-737c-44c2-ae01-e76162a9cde5","cell_type":"code","source":"# target gestures used by the competition metric\nBFRB_CLASSES = [\n    \"Above ear - pull hair\",\n    \"Cheek - pinch skin\",\n    \"Eyebrow - pull hair\",\n    \"Eyelash - pull hair\",\n    \"Forehead - pull hairline\",\n    \"Forehead - scratch\",\n    \"Neck - pinch skin\",\n    \"Neck - scratch\",\n]\n\nCLASS_ARRAY = np.asarray(CLASSES, dtype=object)\nBFRB_IDX = np.asarray([CLASSES.index(x) for x in BFRB_CLASSES], dtype=int)\nNON_BFRB_IDX = np.asarray([i for i in range(len(CLASSES)) if i not in set(BFRB_IDX)], dtype=int)\n\n\ndef as_probs(x):\n    x = np.asarray(x, dtype=np.float64)\n\n    if x.ndim == 1:\n        x = x[None, :]\n\n    row_sum = x.sum(-1, keepdims=True)\n    looks_like_logits = (x.min() < 0) or np.mean(np.abs(row_sum - 1.0)) > 0.05\n\n    if looks_like_logits:\n        x = softmax(x, axis=-1)\n    else:\n        x = np.clip(x, 1e-9, None)\n        x = x / x.sum(-1, keepdims=True)\n\n    return x\n\n\ndef blend_model_probs(preds, weights=None, method=\"mean\"):\n    preds = [as_probs(p) for p in preds]\n    stack = np.stack(preds, axis=0)\n\n    n = len(preds)\n    if weights is None or len(weights) != n:\n        weights = np.ones(n, dtype=np.float64) / n\n    else:\n        weights = np.asarray(weights, dtype=np.float64)\n        weights = np.clip(weights, 1e-8, None)\n        weights = weights / weights.sum()\n\n    wshape = [n] + [1] * (stack.ndim - 1)\n    w = weights.reshape(wshape)\n\n    if method == \"geom\":\n        out = np.exp(np.sum(w * np.log(np.clip(stack, 1e-9, 1.0)), axis=0))\n        out = out / out.sum(-1, keepdims=True)\n    else:\n        out = np.sum(w * stack, axis=0)\n        out = out / out.sum(-1, keepdims=True)\n\n    return out\n\n\ndef labels_from_probs(probs, threshold=0.50):\n    probs = as_probs(probs)\n    target_mass = probs[:, BFRB_IDX].sum(1)\n\n    target_best = BFRB_IDX[np.argmax(probs[:, BFRB_IDX], axis=1)]\n    other_best = NON_BFRB_IDX[np.argmax(probs[:, NON_BFRB_IDX], axis=1)]\n\n    pred_idx = np.where(target_mass >= threshold, target_best, other_best)\n    return CLASS_ARRAY[pred_idx]\n\n\ndef score_probs(gt, probs, threshold=0.50):\n    pred = labels_from_probs(probs, threshold)\n    return hierarchical_f1(np.asarray(gt), pred)\n\n\ndef common_oof_folds(exp_folders, max_folds=5):\n    if not exp_folders:\n        return []\n\n    good = []\n    for fold in range(max_folds):\n        ok = True\n        for folder, _ in exp_folders:\n            if not os.path.exists(folder + f\"pred_val_{fold}.npy\"):\n                ok = False\n                break\n        if ok and os.path.exists(exp_folders[0][0] + f\"df_val_{fold}.csv\"):\n            good.append(fold)\n    return good\n\n\ndef load_oof(exp_folders, folds):\n    all_models = []\n\n    for folder, _ in exp_folders:\n        fold_preds = []\n        for fold in folds:\n            fold_preds.append(as_probs(np.load(folder + f\"pred_val_{fold}.npy\")))\n        all_models.append(np.concatenate(fold_preds, axis=0))\n\n    gt = []\n    seq_ids = []\n    for fold in folds:\n        df = pd.read_csv(exp_folders[0][0] + f\"df_val_{fold}.csv\")\n        gt.extend(df[\"gesture\"].astype(str).tolist())\n        seq_ids.extend(df[\"sequence_id\"].astype(str).tolist())\n\n    return all_models, np.asarray(gt), np.asarray(seq_ids)\n\n\ndef tune_blend(preds, gt, seed=42, trials=220):\n    rng = np.random.default_rng(seed)\n    n = len(preds)\n\n    equal = np.ones(n, dtype=np.float64) / n\n    best = {\n        \"weights\": equal.copy(),\n        \"method\": \"mean\",\n        \"threshold\": 0.50,\n        \"score\": -1.0,\n    }\n\n    candidates = [equal]\n    candidates += list(rng.dirichlet(np.ones(n) * 4.0, size=trials))\n\n    for method in [\"mean\", \"geom\"]:\n        for weights in candidates:\n            probs = blend_model_probs(preds, weights, method)\n            sc = score_probs(gt, probs, 0.50)\n\n            if sc > best[\"score\"]:\n                best.update(\n                    weights=np.asarray(weights).copy(),\n                    method=method,\n                    threshold=0.50,\n                    score=float(sc),\n                )\n\n    # tune the BFRB / non-BFRB boundary\n    probs = blend_model_probs(preds, best[\"weights\"], best[\"method\"])\n    for threshold in np.arange(0.30, 0.701, 0.01):\n        sc = score_probs(gt, probs, float(threshold))\n        if sc > best[\"score\"]:\n            best[\"threshold\"] = float(threshold)\n            best[\"score\"] = float(sc)\n\n    # one small local pass around the best weights\n    for _ in range(80):\n        weights = best[\"weights\"] + rng.normal(0, 0.035, n)\n        weights = np.clip(weights, 0.02, None)\n        weights = weights / weights.sum()\n\n        probs = blend_model_probs(preds, weights, best[\"method\"])\n        sc = score_probs(gt, probs, best[\"threshold\"])\n\n        if sc > best[\"score\"]:\n            best[\"weights\"] = weights.copy()\n            best[\"score\"] = float(sc)\n\n    # threshold again after the local pass\n    probs = blend_model_probs(preds, best[\"weights\"], best[\"method\"])\n    for threshold in np.arange(0.30, 0.701, 0.005):\n        sc = score_probs(gt, probs, float(threshold))\n        if sc > best[\"score\"]:\n            best[\"threshold\"] = float(threshold)\n            best[\"score\"] = float(sc)\n\n    best[\"weights\"] = best[\"weights\"].tolist()\n    return best\n\n\ndef default_blend(n):\n    return {\n        \"weights\": (np.ones(n) / max(n, 1)).tolist() if n else [],\n        \"method\": \"mean\",\n        \"threshold\": 0.50,\n        \"score\": None,\n    }\n\n\nFULL_BLEND = default_blend(len(EXP_FOLDERS))\nIMU_BLEND = default_blend(len(EXP_FOLDERS_IMU))\nFULL_IMU_MIX = 0.0\nFULL_THRESHOLD = 0.50\nIMU_THRESHOLD = 0.50\nOOF_FOLDS = []\n\nif USE_OOF_BLEND:\n    folders_for_search = EXP_FOLDERS_IMU if IMU_ONLY else EXP_FOLDERS + EXP_FOLDERS_IMU\n    OOF_FOLDS = common_oof_folds(folders_for_search)\n\n    if len(OOF_FOLDS):\n        print(\"OOF folds:\", OOF_FOLDS)\n\n        imu_oof, gt_oof, oof_ids = load_oof(EXP_FOLDERS_IMU, OOF_FOLDS)\n        IMU_BLEND = tune_blend(\n            imu_oof,\n            gt_oof,\n            seed=123,\n            trials=N_BLEND_TRIALS,\n        )\n        IMU_THRESHOLD = IMU_BLEND[\"threshold\"]\n\n        print(\n            f\"IMU blend  : {IMU_BLEND['score']:.4f} | \"\n            f\"{IMU_BLEND['method']} | th={IMU_THRESHOLD:.3f}\"\n        )\n        print(\"IMU weights:\", np.round(IMU_BLEND[\"weights\"], 3).tolist())\n\n        if not IMU_ONLY and len(EXP_FOLDERS):\n            full_oof, gt_full, full_ids = load_oof(EXP_FOLDERS, OOF_FOLDS)\n\n            if not np.array_equal(oof_ids, full_ids):\n                raise ValueError(\"OOF rows are not aligned between full and IMU models\")\n\n            FULL_BLEND = tune_blend(\n                full_oof,\n                gt_full,\n                seed=321,\n                trials=N_BLEND_TRIALS,\n            )\n\n            p_full = blend_model_probs(\n                full_oof,\n                FULL_BLEND[\"weights\"],\n                FULL_BLEND[\"method\"],\n            )\n            p_imu = blend_model_probs(\n                imu_oof,\n                IMU_BLEND[\"weights\"],\n                IMU_BLEND[\"method\"],\n            )\n\n            # a little IMU contribution can help some ambiguous full-sensor cases\n            best_mix_score = -1.0\n            best_mix = 0.0\n            best_threshold = FULL_BLEND[\"threshold\"]\n\n            for mix in np.arange(0.0, MAX_IMU_MIX + 1e-9, 0.025):\n                p = (1.0 - mix) * p_full + mix * p_imu\n\n                for threshold in np.arange(0.30, 0.701, 0.01):\n                    sc = score_probs(gt_full, p, float(threshold))\n\n                    if sc > best_mix_score:\n                        best_mix_score = float(sc)\n                        best_mix = float(mix)\n                        best_threshold = float(threshold)\n\n            FULL_IMU_MIX = best_mix\n            FULL_THRESHOLD = best_threshold\n\n            print(\n                f\"Full blend : {best_mix_score:.4f} | \"\n                f\"{FULL_BLEND['method']} | th={FULL_THRESHOLD:.3f} | \"\n                f\"imu_mix={FULL_IMU_MIX:.3f}\"\n            )\n            print(\"Full weights:\", np.round(FULL_BLEND[\"weights\"], 3).tolist())\n\n            FULL_BLEND[\"score_with_imu_mix\"] = best_mix_score\n\n        settings = {\n            \"oof_folds\": OOF_FOLDS,\n            \"full\": FULL_BLEND,\n            \"imu\": IMU_BLEND,\n            \"full_imu_mix\": FULL_IMU_MIX,\n            \"full_threshold\": FULL_THRESHOLD,\n            \"imu_threshold\": IMU_THRESHOLD,\n        }\n\n        with open(\"/kaggle/working/blend_settings.json\", \"w\") as f:\n            json.dump(settings, f, indent=2)\n    else:\n        print(\"OOF predictions were not found. Using equal averaging.\")\nelse:\n    print(\"OOF tuning disabled.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:38:58.593088Z","iopub.execute_input":"2026-08-17T10:38:58.593768Z","iopub.status.idle":"2026-08-17T10:40:49.054011Z","shell.execute_reply.started":"2026-08-17T10:38:58.593742Z","shell.execute_reply":"2026-08-17T10:40:49.053293Z"}},"outputs":[],"execution_count":null},{"id":"d27d5a35-5a73-46e1-9362-3dfabf98cb64","cell_type":"markdown","source":"### 5.1 What did the OOF optimizer choose?\n\nThese plots make the ensemble easier to inspect. A model with a larger bar receives more influence in the final blend. The threshold curve shows whether `0.50` is actually optimal for the hierarchical metric.\n","metadata":{}},{"id":"e05ff9cd-0a01-4b60-aae0-6cadf605ca19","cell_type":"code","source":"if len(OOF_FOLDS):\n    if len(EXP_FOLDERS_IMU):\n        imu_names = IMU_MODEL_NAMES[:len(IMU_BLEND[\"weights\"])]\n        fig, ax = plt.subplots(figsize=(8, 4))\n        ax.barh(imu_names, IMU_BLEND[\"weights\"])\n        ax.set_xlabel(\"Blend weight\")\n        ax.set_title(\"OOF optimized weights — IMU-only ensemble\")\n        ax.set_xlim(0, max(IMU_BLEND[\"weights\"]) * 1.20 if len(IMU_BLEND[\"weights\"]) else 1)\n        for i, w in enumerate(IMU_BLEND[\"weights\"]):\n            ax.text(w + 0.005, i, f\"{w:.3f}\", va=\"center\")\n        plt.tight_layout()\n        plt.show()\n\n    if (not IMU_ONLY) and len(EXP_FOLDERS):\n        full_names = FULL_MODEL_NAMES[:len(FULL_BLEND[\"weights\"])]\n        fig, ax = plt.subplots(figsize=(8, 5))\n        ax.barh(full_names, FULL_BLEND[\"weights\"])\n        ax.set_xlabel(\"Blend weight\")\n        ax.set_title(\"OOF optimized weights — full-sensor ensemble\")\n        ax.set_xlim(0, max(FULL_BLEND[\"weights\"]) * 1.20 if len(FULL_BLEND[\"weights\"]) else 1)\n        for i, w in enumerate(FULL_BLEND[\"weights\"]):\n            ax.text(w + 0.005, i, f\"{w:.3f}\", va=\"center\")\n        plt.tight_layout()\n        plt.show()\n\n        # Rebuild the final full-sensor OOF probability used for threshold selection\n        p_full_plot = blend_model_probs(\n            full_oof,\n            FULL_BLEND[\"weights\"],\n            FULL_BLEND[\"method\"],\n        )\n        p_imu_plot = blend_model_probs(\n            imu_oof,\n            IMU_BLEND[\"weights\"],\n            IMU_BLEND[\"method\"],\n        )\n        p_plot = (1.0 - FULL_IMU_MIX) * p_full_plot + FULL_IMU_MIX * p_imu_plot\n\n        threshold_grid = np.arange(0.30, 0.701, 0.01)\n        threshold_scores = [\n            score_probs(gt_full, p_plot, float(t))\n            for t in threshold_grid\n        ]\n\n        fig, ax = plt.subplots(figsize=(8, 4))\n        ax.plot(threshold_grid, threshold_scores, marker=\"o\", markersize=3)\n        ax.axvline(FULL_THRESHOLD, linestyle=\"--\", label=f\"selected = {FULL_THRESHOLD:.3f}\")\n        ax.set_xlabel(\"BFRB probability threshold\")\n        ax.set_ylabel(\"Hierarchical F1\")\n        ax.set_title(\"OOF threshold search\")\n        ax.legend()\n        plt.tight_layout()\n        plt.show()\n\n        print(\n            f\"Selected full threshold: {FULL_THRESHOLD:.3f} | \"\n            f\"IMU mix: {FULL_IMU_MIX:.3f} | \"\n            f\"blend method: {FULL_BLEND['method']}\"\n        )\nelse:\n    print(\"OOF files were not available, so the notebook is using equal model averaging.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:40:49.055466Z","iopub.execute_input":"2026-08-17T10:40:49.056272Z","iopub.status.idle":"2026-08-17T10:40:51.789827Z","shell.execute_reply.started":"2026-08-17T10:40:49.056210Z","shell.execute_reply":"2026-08-17T10:40:51.788975Z"}},"outputs":[],"execution_count":null},{"id":"7b6f47e8-fb1e-4b0c-a2b6-105b17d6dd07","cell_type":"markdown","source":"## 6. Load full-sensor predictors\n\nModel objects are loaded only after the ensemble configuration is ready.\n","metadata":{}},{"id":"eb79f2a1-3f0b-48df-80fd-15bc0faa227a","cell_type":"code","source":"models = []\nfor exp_folder, folds in EXP_FOLDERS:\n    models.append(\n        TheoPredictor(\n            exp_folder,\n            folds,\n            imu_only=False,\n            device=DEVICE,\n            use_fp16=USE_FP16\n        )\n    )\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:40:51.790664Z","iopub.execute_input":"2026-08-17T10:40:51.790929Z","iopub.status.idle":"2026-08-17T10:41:16.680319Z","shell.execute_reply.started":"2026-08-17T10:40:51.790907Z","shell.execute_reply":"2026-08-17T10:41:16.679683Z"}},"outputs":[],"execution_count":null},{"id":"a06afc4c-275e-49d8-92b7-48a932d0e780","cell_type":"markdown","source":"## 7. Inference logic\n\nAt prediction time the sequence is first checked for thermal / ToF availability.\n\n- **IMU-only sequence:** use the IMU ensemble and its own tuned threshold\n- **full-sensor sequence:** use the full ensemble and optionally mix in a small IMU contribution if OOF validation found that useful\n\nThis keeps the decision rule identical between validation and competition inference.\n","metadata":{}},{"id":"e97e037c-2ca4-42e5-87e7-296045b1d368","cell_type":"code","source":"import polars as pl\nimport pandas as pd\n\n\ndef detect_imu_only(sequence: pd.DataFrame) -> bool:\n    has_thm = all(feature in sequence.columns for feature in THM_FEATURES)\n    has_tof = all(feature in sequence.columns for feature in TOF_FEATURES)\n\n    thm_missing = (not has_thm) or sequence[THM_FEATURES].isna().values.all()\n    tof_missing = (not has_tof) or (sequence[TOF_FEATURES].fillna(-1).to_numpy().max() <= 0)\n\n    return bool(thm_missing or tof_missing)\n\n\ndef run_family(sequence, demographics, predictors):\n    model_probs = []\n\n    for predictor in predictors:\n        fts = predictor.preprocess(sequence.copy(), demographics.copy())\n        fold_preds = predictor.infer(fts, activation=\"softmax\")\n\n        # if a predictor contains more than one fold, average those first\n        model_probs.append(np.mean(np.stack(fold_preds, axis=0), axis=0))\n\n    return model_probs\n\n\ndef predict(sequence: pl.DataFrame, demographics: pl.DataFrame) -> str:\n    if not isinstance(sequence, pd.core.frame.DataFrame):\n        sequence = sequence.to_pandas()\n    if not isinstance(demographics, pd.core.frame.DataFrame):\n        demographics = demographics.to_pandas()\n\n    imu_only = detect_imu_only(sequence)\n    if IMU_ONLY:\n        imu_only = True\n\n    if DEBUG:\n        seq = sequence[\"sequence_id\"].values[0]\n        idx = df_val[\"sequence_id\"].values.tolist().index(seq)\n        gt = df_val.gesture.values[idx]\n        mode = \"imu_only\" if imu_only else \"all\"\n        print()\n        print(f\"-> Sequence {seq} - GT: {gt} ({mode})\")\n\n    if imu_only:\n        imu_preds = run_family(sequence, demographics, models_imu)\n        probs = blend_model_probs(\n            imu_preds,\n            IMU_BLEND[\"weights\"],\n            IMU_BLEND[\"method\"],\n        )\n        y = labels_from_probs(probs, IMU_THRESHOLD)[0]\n\n    else:\n        full_preds = run_family(sequence, demographics, models)\n        p_full = blend_model_probs(\n            full_preds,\n            FULL_BLEND[\"weights\"],\n            FULL_BLEND[\"method\"],\n        )\n\n        if FULL_IMU_MIX > 0 and len(models_imu):\n            imu_preds = run_family(sequence, demographics, models_imu)\n            p_imu = blend_model_probs(\n                imu_preds,\n                IMU_BLEND[\"weights\"],\n                IMU_BLEND[\"method\"],\n            )\n            probs = (1.0 - FULL_IMU_MIX) * p_full + FULL_IMU_MIX * p_imu\n        else:\n            probs = p_full\n\n        y = labels_from_probs(probs, FULL_THRESHOLD)[0]\n\n    if DEBUG:\n        print(\"probs:\", np.round(probs[0], 3).tolist())\n        print(\"->\", y)\n\n    return str(y)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:41:16.681509Z","iopub.execute_input":"2026-08-17T10:41:16.682318Z","iopub.status.idle":"2026-08-17T10:41:17.127759Z","shell.execute_reply.started":"2026-08-17T10:41:16.682296Z","shell.execute_reply":"2026-08-17T10:41:17.127107Z"}},"outputs":[],"execution_count":null},{"id":"2d626b33-f8eb-449d-90b1-00d7b501154b","cell_type":"markdown","source":"## 8. Local validation setup\n\nThe validation path mirrors the Kaggle inference path. This is useful because a good cached OOF score is not enough if the live `predict()` function behaves differently.\n","metadata":{}},{"id":"aa07c833-126f-4c56-9acc-1d95af6bdb2c","cell_type":"code","source":"if EVAL:\n    data_paths = (DATASET + 'val_0.csv', DATA_PATH + \"train_demographics.csv\")\n    # data_paths = (DATA_PATH + 'train.csv', DATA_PATH + \"train_demographics.csv\")  # MORE DATA\nelif DEBUG:\n    data_paths = (DATASET + 'debug_3.csv', DATA_PATH + \"train_demographics.csv\")\nelse:\n    data_paths = (DATA_PATH + 'test.csv', DATA_PATH + \"test_demographics.csv\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:41:22.880463Z","iopub.execute_input":"2026-08-17T10:41:22.881181Z","iopub.status.idle":"2026-08-17T10:41:22.885163Z","shell.execute_reply.started":"2026-08-17T10:41:22.881150Z","shell.execute_reply":"2026-08-17T10:41:22.884478Z"}},"outputs":[],"execution_count":null},{"id":"69c92b1d-799b-422e-abb5-04f57fb75d62","cell_type":"code","source":"if EVAL or DEBUG:\n    df_val = pd.read_csv(EXP_FOLDERS_IMU[0][0] + f\"df_val_{FOLDS[0]}.csv\")\n\n    imu_fold_preds = [\n        np.load(folder + f\"pred_val_{FOLDS[0]}.npy\")\n        for folder, _ in EXP_FOLDERS_IMU\n    ]\n    p_imu_val = blend_model_probs(\n        imu_fold_preds,\n        IMU_BLEND[\"weights\"],\n        IMU_BLEND[\"method\"],\n    )\n    df_val[\"pred_val_imu\"] = labels_from_probs(p_imu_val, IMU_THRESHOLD)\n    df_val[\"imu_confidence\"] = p_imu_val.max(axis=1)\n    df_val[\"imu_bfrb_probability\"] = p_imu_val[:, BFRB_IDX].sum(axis=1)\n\n    if not IMU_ONLY:\n        full_fold_preds = [\n            np.load(folder + f\"pred_val_{FOLDS[0]}.npy\")\n            for folder, _ in EXP_FOLDERS\n        ]\n        p_full_val = blend_model_probs(\n            full_fold_preds,\n            FULL_BLEND[\"weights\"],\n            FULL_BLEND[\"method\"],\n        )\n\n        if FULL_IMU_MIX > 0:\n            p_val = (1.0 - FULL_IMU_MIX) * p_full_val + FULL_IMU_MIX * p_imu_val\n        else:\n            p_val = p_full_val\n\n        df_val[\"pred_val\"] = labels_from_probs(p_val, FULL_THRESHOLD)\n        df_val[\"blend_confidence\"] = p_val.max(axis=1)\n        df_val[\"bfrb_probability\"] = p_val[:, BFRB_IDX].sum(axis=1)\n\n        sorted_p = np.sort(p_val, axis=1)\n        df_val[\"prediction_margin\"] = sorted_p[:, -1] - sorted_p[:, -2]\n    else:\n        df_val[\"pred_val\"] = df_val[\"pred_val_imu\"]\n        df_val[\"blend_confidence\"] = df_val[\"imu_confidence\"]\n        df_val[\"bfrb_probability\"] = df_val[\"imu_bfrb_probability\"]\n        df_val[\"prediction_margin\"] = 0.0\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:41:24.003494Z","iopub.execute_input":"2026-08-17T10:41:24.003984Z","iopub.status.idle":"2026-08-17T10:41:24.037608Z","shell.execute_reply.started":"2026-08-17T10:41:24.003960Z","shell.execute_reply":"2026-08-17T10:41:24.037064Z"}},"outputs":[],"execution_count":null},{"id":"a92b086a-ab8b-4348-aa7b-898d5485e435","cell_type":"markdown","source":"### 8.1 Run the real prediction function\n\nThis section evaluates the exact `predict()` function sequence-by-sequence, so preprocessing, sensor routing and post-processing are all tested together.\n","metadata":{}},{"id":"f727b98e-fa07-4aa1-8735-820af866e47a","cell_type":"code","source":"if EVAL or DEBUG:\n    data = pd.read_csv(data_paths[0])\n    demo = pd.read_csv(data_paths[1])\n\n    preds = []\n    for seq, dfg in tqdm(data.groupby(\"sequence_id\"), disable=DEBUG):\n        demo_seq = demo[demo[\"subject\"] == dfg[\"subject\"].values[0]]\n        y = predict(dfg, demo_seq)\n        preds.append({\"sequence_id\": seq, \"gesture\": y})\n\n    sub_ = pd.DataFrame(preds)\n\n    diag_cols = [\n        \"sequence_id\",\n        \"pred_val\",\n        \"pred_val_imu\",\n        \"gesture\",\n        \"blend_confidence\",\n        \"bfrb_probability\",\n        \"prediction_margin\",\n    ]\n    diag_cols = [c for c in diag_cols if c in df_val.columns]\n\n    sub_ = sub_.merge(\n        df_val[diag_cols].rename(columns={\"gesture\": \"gt\"}),\n        on=\"sequence_id\",\n        how=\"inner\",\n        validate=\"one_to_one\",\n    )\n\n    if len(sub_) != len(preds):\n        raise ValueError(f\"Row count changed after merge: {len(preds)} -> {len(sub_)}\")\n\n    sub_[\"correct\"] = sub_[\"gesture\"] == sub_[\"gt\"]\n\n    if EVAL:\n        val_f1 = hierarchical_f1(sub_[\"gt\"], sub_[\"pred_val\"])\n        val_f1_imu = hierarchical_f1(sub_[\"gt\"], sub_[\"pred_val_imu\"])\n        inf_f1 = hierarchical_f1(sub_[\"gt\"], sub_[\"gesture\"])\n\n        print(f\"Cached blend F1 : {val_f1:.4f}\")\n        print(f\"IMU-only F1     : {val_f1_imu:.4f}\")\n        print(f\"Live predict F1 : {inf_f1:.4f}\")\n        print(\n            f\"Full threshold  : {FULL_THRESHOLD:.3f} | \"\n            f\"IMU threshold: {IMU_THRESHOLD:.3f} | \"\n            f\"IMU mix: {FULL_IMU_MIX:.3f}\"\n        )\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:41:26.158497Z","iopub.execute_input":"2026-08-17T10:41:26.159096Z","iopub.status.idle":"2026-08-17T10:57:40.022302Z","shell.execute_reply.started":"2026-08-17T10:41:26.159070Z","shell.execute_reply":"2026-08-17T10:57:40.021445Z"}},"outputs":[],"execution_count":null},{"id":"a66bc04e-e21f-4ed3-874d-4dc39fd876b4","cell_type":"markdown","source":"## 9. Validation data understanding\n\nThese plots are deliberately lightweight. They answer the questions that matter most for this model:\n\n- Is the validation set class-balanced?\n- Are some gestures represented by much longer sequences?\n- How much sensor information is missing?\n","metadata":{}},{"id":"b4457bee-853b-4034-96fc-1ae20298c8fe","cell_type":"code","source":"if EVAL or DEBUG:\n    # one row per sequence\n    seq_meta = (\n        data.groupby(\"sequence_id\")\n        .agg(\n            sequence_length=(\"sequence_id\", \"size\"),\n            subject=(\"subject\", \"first\"),\n        )\n        .reset_index()\n    )\n\n    label_meta = df_val[[\"sequence_id\", \"gesture\"]].copy()\n    seq_meta = seq_meta.merge(label_meta, on=\"sequence_id\", how=\"left\")\n\n    class_counts = (\n        seq_meta[\"gesture\"]\n        .value_counts()\n        .reindex(CLASSES, fill_value=0)\n        .sort_values()\n    )\n\n    fig, ax = plt.subplots(figsize=(10, 7))\n    ax.barh(class_counts.index, class_counts.values)\n    ax.set_xlabel(\"Number of validation sequences\")\n    ax.set_title(\"Validation class distribution\")\n    for i, value in enumerate(class_counts.values):\n        ax.text(value + 0.2, i, str(int(value)), va=\"center\", fontsize=8)\n    plt.tight_layout()\n    plt.show()\n\n    fig, ax = plt.subplots(figsize=(8, 4))\n    ax.hist(seq_meta[\"sequence_length\"], bins=25)\n    ax.axvline(\n        seq_meta[\"sequence_length\"].median(),\n        linestyle=\"--\",\n        label=f\"median = {seq_meta['sequence_length'].median():.0f}\",\n    )\n    ax.set_xlabel(\"Timesteps per sequence\")\n    ax.set_ylabel(\"Count\")\n    ax.set_title(\"Sequence length distribution\")\n    ax.legend()\n    plt.tight_layout()\n    plt.show()\n\n    sensor_rows = []\n\n    for group_name, columns in [\n        (\"IMU\", IMU_FEATURES),\n        (\"Thermal\", THM_FEATURES),\n        (\"ToF\", TOF_FEATURES),\n    ]:\n        available = [c for c in columns if c in data.columns]\n\n        if available:\n            missing_pct = data[available].isna().mean().mean() * 100\n            sensor_rows.append(\n                {\n                    \"sensor_group\": group_name,\n                    \"columns\": len(available),\n                    \"missing_percent\": missing_pct,\n                }\n            )\n\n    sensor_summary = pd.DataFrame(sensor_rows)\n    display(sensor_summary)\n\n    if len(sensor_summary):\n        fig, ax = plt.subplots(figsize=(7, 4))\n        ax.bar(sensor_summary[\"sensor_group\"], sensor_summary[\"missing_percent\"])\n        ax.set_ylabel(\"Missing values (%)\")\n        ax.set_title(\"Average missingness by sensor group\")\n        ax.set_ylim(0, max(5, sensor_summary[\"missing_percent\"].max() * 1.15))\n        for i, value in enumerate(sensor_summary[\"missing_percent\"]):\n            ax.text(i, value + 0.3, f\"{value:.1f}%\", ha=\"center\")\n        plt.tight_layout()\n        plt.show()\n\n    display(\n        seq_meta.groupby(\"gesture\")[\"sequence_length\"]\n        .agg([\"count\", \"mean\", \"median\", \"min\", \"max\"])\n        .round(1)\n        .sort_values(\"count\")\n    )\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:57:40.023548Z","iopub.execute_input":"2026-08-17T10:57:40.024081Z","iopub.status.idle":"2026-08-17T10:57:40.898226Z","shell.execute_reply.started":"2026-08-17T10:57:40.024061Z","shell.execute_reply":"2026-08-17T10:57:40.897687Z"}},"outputs":[],"execution_count":null},{"id":"08a79814-24f5-49a7-9b50-ef5917537956","cell_type":"markdown","source":"## 10. Validation diagnostics\n\nThe overall score tells only part of the story. The following diagnostics show **which classes are difficult**, **which mistakes are confident**, and **whether the BFRB boundary is the main source of errors**.\n","metadata":{}},{"id":"86ae0060-0ca6-4eb4-81be-3b3632d67645","cell_type":"code","source":"if EVAL:\n    from sklearn.metrics import confusion_matrix, precision_recall_fscore_support\n\n    y_true = sub_[\"gt\"].astype(str).to_numpy()\n    y_pred = sub_[\"gesture\"].astype(str).to_numpy()\n    labels = list(CLASSES)\n\n    precision, recall, per_class_f1, support = precision_recall_fscore_support(\n        y_true,\n        y_pred,\n        labels=labels,\n        zero_division=0,\n    )\n\n    class_metrics = pd.DataFrame(\n        {\n            \"gesture\": labels,\n            \"precision\": precision,\n            \"recall\": recall,\n            \"f1\": per_class_f1,\n            \"support\": support,\n        }\n    )\n\n    class_metrics[\"group\"] = np.where(\n        class_metrics[\"gesture\"].isin(BFRB_CLASSES),\n        \"BFRB\",\n        \"Non-BFRB\",\n    )\n\n    display(\n        class_metrics.sort_values([\"f1\", \"support\"])\n        .reset_index(drop=True)\n        .style.format(\n            {\n                \"precision\": \"{:.3f}\",\n                \"recall\": \"{:.3f}\",\n                \"f1\": \"{:.3f}\",\n            }\n        )\n    )\n\n    sorted_metrics = class_metrics.sort_values(\"f1\")\n\n    fig, ax = plt.subplots(figsize=(10, 7))\n    ax.barh(sorted_metrics[\"gesture\"], sorted_metrics[\"f1\"])\n    ax.set_xlabel(\"F1\")\n    ax.set_xlim(0, 1)\n    ax.set_title(\"Per-class F1 — live validation inference\")\n    for i, value in enumerate(sorted_metrics[\"f1\"]):\n        ax.text(value + 0.01, i, f\"{value:.2f}\", va=\"center\", fontsize=8)\n    plt.tight_layout()\n    plt.show()\n\n    cm = confusion_matrix(\n        y_true,\n        y_pred,\n        labels=labels,\n        normalize=\"true\",\n    )\n\n    fig, ax = plt.subplots(figsize=(13, 11))\n    im = ax.imshow(cm, vmin=0, vmax=1)\n    fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)\n    ax.set_xticks(range(len(labels)))\n    ax.set_yticks(range(len(labels)))\n    ax.set_xticklabels(labels, rotation=90, fontsize=8)\n    ax.set_yticklabels(labels, fontsize=8)\n    ax.set_xlabel(\"Predicted\")\n    ax.set_ylabel(\"True\")\n    ax.set_title(\"Normalized confusion matrix\")\n    plt.tight_layout()\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:57:40.898915Z","iopub.execute_input":"2026-08-17T10:57:40.899173Z","iopub.status.idle":"2026-08-17T10:57:41.672530Z","shell.execute_reply.started":"2026-08-17T10:57:40.899153Z","shell.execute_reply":"2026-08-17T10:57:41.671596Z"}},"outputs":[],"execution_count":null},{"id":"8c069d1c-4cb2-44a8-b89e-f343fc7f4780","cell_type":"markdown","source":"### 10.1 Confidence and high-value error inspection\n\nHigh-confidence mistakes are usually more useful than random errors because they reveal systematic confusion rather than simple uncertainty.\n","metadata":{}},{"id":"a90e3d6b-2190-46c2-8185-f3354954e45f","cell_type":"code","source":"if EVAL and \"blend_confidence\" in sub_.columns:\n    correct_conf = sub_.loc[sub_[\"correct\"], \"blend_confidence\"].dropna()\n    wrong_conf = sub_.loc[~sub_[\"correct\"], \"blend_confidence\"].dropna()\n\n    fig, ax = plt.subplots(figsize=(8, 4))\n    ax.hist(correct_conf, bins=20, alpha=0.65, label=\"correct\")\n    ax.hist(wrong_conf, bins=20, alpha=0.65, label=\"wrong\")\n    ax.set_xlabel(\"Maximum blended probability\")\n    ax.set_ylabel(\"Sequences\")\n    ax.set_title(\"Prediction confidence: correct vs wrong\")\n    ax.legend()\n    plt.tight_layout()\n    plt.show()\n\n    error_table = (\n        sub_.loc[~sub_[\"correct\"]]\n        .sort_values(\"blend_confidence\", ascending=False)\n        [\n            [\n                \"sequence_id\",\n                \"gt\",\n                \"gesture\",\n                \"blend_confidence\",\n                \"bfrb_probability\",\n                \"prediction_margin\",\n            ]\n        ]\n        .head(20)\n        .reset_index(drop=True)\n    )\n\n    print(\"Highest-confidence mistakes\")\n    display(error_table.style.format({\n        \"blend_confidence\": \"{:.3f}\",\n        \"bfrb_probability\": \"{:.3f}\",\n        \"prediction_margin\": \"{:.3f}\",\n    }))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:57:41.673873Z","iopub.execute_input":"2026-08-17T10:57:41.674168Z","iopub.status.idle":"2026-08-17T10:57:41.968406Z","shell.execute_reply.started":"2026-08-17T10:57:41.674144Z","shell.execute_reply":"2026-08-17T10:57:41.967759Z"}},"outputs":[],"execution_count":null},{"id":"77a07cbf-4bd4-4ba1-a01c-a15e29643706","cell_type":"markdown","source":"## 11. Discussion\n\nThis section converts the validation outputs into a short experiment summary instead of leaving the reader to interpret every plot manually.\n","metadata":{}},{"id":"750afd33-69cf-4a69-9902-b27170728189","cell_type":"code","source":"if EVAL:\n    hardest = class_metrics.nsmallest(5, \"f1\")\n    strongest = class_metrics.nlargest(5, \"f1\")\n\n    bfrb_macro = class_metrics.loc[class_metrics[\"group\"] == \"BFRB\", \"f1\"].mean()\n    non_bfrb_macro = class_metrics.loc[class_metrics[\"group\"] == \"Non-BFRB\", \"f1\"].mean()\n\n    high_conf_errors = 0\n    if \"blend_confidence\" in sub_.columns:\n        high_conf_errors = int(\n            ((~sub_[\"correct\"]) & (sub_[\"blend_confidence\"] >= 0.80)).sum()\n        )\n\n    discussion = f\"\"\"\n### Validation summary\n\n- **Live hierarchical F1:** `{inf_f1:.4f}`\n- **Cached blended F1:** `{val_f1:.4f}`\n- **IMU-only hierarchical F1:** `{val_f1_imu:.4f}`\n- **Mean BFRB class F1:** `{bfrb_macro:.4f}`\n- **Mean non-BFRB class F1:** `{non_bfrb_macro:.4f}`\n- **High-confidence errors (p ≥ 0.80):** `{high_conf_errors}`\n\n### What I would improve next\n\n1. Focus new experiments on the lowest-F1 classes rather than adding models blindly.\n2. Check the high-confidence error table for repeated class pairs; these are the best candidates for targeted augmentation or specialist models.\n3. Keep full-sensor and IMU-only validation separate. A change that helps one path can hurt the other.\n4. Re-tune the BFRB threshold whenever the ensemble composition changes.\n5. Prefer models that add diversity to the ensemble, not only models with a slightly higher standalone score.\n\"\"\"\n\n    display(Markdown(discussion))\n\n    print(\"Hardest classes\")\n    display(hardest[[\"gesture\", \"f1\", \"support\", \"group\"]].reset_index(drop=True))\n\n    print(\"Strongest classes\")\n    display(strongest[[\"gesture\", \"f1\", \"support\", \"group\"]].reset_index(drop=True))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:57:41.969340Z","iopub.execute_input":"2026-08-17T10:57:41.969657Z","iopub.status.idle":"2026-08-17T10:57:41.993171Z","shell.execute_reply.started":"2026-08-17T10:57:41.969607Z","shell.execute_reply":"2026-08-17T10:57:41.992388Z"}},"outputs":[],"execution_count":null},{"id":"70dcbfcc-8b77-4a52-a836-429d43e94655","cell_type":"markdown","source":"## 12. Kaggle inference server\n\nThe final competition path uses the same `predict()` function that was tested above.\n","metadata":{}},{"id":"4bee355f-05ef-45e6-8bed-c5c49fade41e","cell_type":"code","source":"import kaggle_evaluation.cmi_inference_server\ninference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n    inference_server.serve()\nelse:\n    if EVAL and not DEBUG:\n        data_paths = (DATASET + 'val_0.csv', DATA_PATH + \"train_demographics.csv\")\n    elif DEBUG:\n        data_paths = (DATASET + 'debug_3.csv', DATA_PATH + \"train_demographics.csv\")\n    else:\n        data_paths = (DATA_PATH + 'test.csv', DATA_PATH + \"test_demographics.csv\")\n\n    print(f\"Infering using data: {data_paths} ...\\n\")\n    inference_server.run_local_gateway(data_paths=data_paths)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T10:57:41.993982Z","iopub.execute_input":"2026-08-17T10:57:41.994609Z","iopub.status.idle":"2026-08-17T11:15:18.207908Z","shell.execute_reply.started":"2026-08-17T10:57:41.994585Z","shell.execute_reply":"2026-08-17T11:15:18.207212Z"}},"outputs":[],"execution_count":null},{"id":"fd123582-c347-48db-a53d-12c75baf877e","cell_type":"markdown","source":"### Submission preview\n","metadata":{}},{"id":"05e29bd0-46ed-4736-8c54-84581f4e8f67","cell_type":"code","source":"sub = pd.read_parquet('submission.parquet')\nif EVAL or DEBUG:\n    sub = sub.merge(df_val[[\"sequence_id\", \"pred_val\", \"gesture\"]].rename(columns={\"gesture\": \"gt\"}), how=\"left\")\nsub.head(10)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T11:15:18.208771Z","iopub.execute_input":"2026-08-17T11:15:18.209151Z","iopub.status.idle":"2026-08-17T11:15:18.354334Z","shell.execute_reply.started":"2026-08-17T11:15:18.209109Z","shell.execute_reply":"2026-08-17T11:15:18.353686Z"}},"outputs":[],"execution_count":null},{"id":"5f1502ea-4bb5-4782-849b-f9704cf968fd","cell_type":"code","source":"# save the actual competition submission\nsave_path = \"/kaggle/working/submission.parquet\"\nsub.to_parquet(save_path, index=False)\n\nprint(\"Saved to:\", save_path)\nprint(\"Shape:\", sub.shape)\ndisplay(sub.head(10))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-17T11:17:57.292921Z","iopub.execute_input":"2026-08-17T11:17:57.293552Z","iopub.status.idle":"2026-08-17T11:17:57.310678Z","shell.execute_reply.started":"2026-08-17T11:17:57.293526Z","shell.execute_reply":"2026-08-17T11:17:57.310048Z"}},"outputs":[],"execution_count":null},{"id":"5f6187f2-ca11-45be-ba6c-0b0955aee114","cell_type":"markdown","source":"## 13. Final notes\n\nThis version is structured so each experiment answers a specific question:\n\n- **Data plots:** what makes the validation set difficult?\n- **OOF plots:** why were these ensemble weights and thresholds selected?\n- **Per-class metrics:** where is the model weak?\n- **Confidence analysis:** which errors are systematic?\n- **Live inference score:** does the real submission path match the cached validation logic?\n\nThat makes future improvements easier to justify and compare instead of relying only on a single leaderboard number.\n","metadata":{}}]}