{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"},{"sourceId":242561727,"sourceType":"kernelVersion"}],"dockerImageVersionId":31041,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CMI - LSTM Classifier Starter\n\nThis is meant as a training notebook to get started. We train a simple LSTM classifier on the IMU signals. A corresponding inference notebook is done afterwards.\n\n**All comments welcome!**\n\n\n\nBefore we start let us discuss and elaborate on the individual parts.\n\n1. Load training data (IMU).\n    - For simplicity we skip demographics data and other, sometime missing features.\n\n2. Split data into training and validation set. This must be done on the ``sequence_id`` level in order to avoid mixing different sequences.\n\n3. Implement a scikit-learn like preprocessing that can also be stored.\n\n4. Create a dataset that prepares the sequences. \n    - Sequences have a maximum length.\n    - Since gesture is at the end, take the maximum lenght from the right side.\n\n5. Define a data collator that pads sequences to equal length, optionally pads target and outputs an additional mask.\n\n6. There are different options to classify:\n    - Output logits are only processed for the last hidden state. In this way one has only 1 output, but I'm not sure how to code this correctly.\n    - Each hidden state passes thru the same classifier to create an logits for each instance. This is more straightforward, however we can have issues in the beginning when gesture did not even start and about the overall weight of the sequence.\n    - Output logits are only considered when gesture has started. This might be doable by using an appropriate ``gesture_mask``, but again might suffer from weighting effects.\n    - There is also the possibility to consider aggregates from all hidden states, something like an average before the classifier.\n    - I think ultimately the code for each one is very similar, when the main feature is using the loss masking feature, i.e., setting ``target=-100`` so that it is not included in the loss computation.\n\n7. Define model using average over all hidden states as a start.\n\n8. Create training loop with validation and checkpointing.\n\n9. Compute competition metric on validation set.\n\n## Change Log\n- V2: Add model components like dropout or bidirectionality. Include scheduler and different checkpointing mechanism.\n- V4: Fix model eval mode, add label smoothing.","metadata":{}},{"cell_type":"markdown","source":"# Import & Settings","metadata":{}},{"cell_type":"code","source":"import os\nimport pathlib\nimport time\nimport collections\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nfrom tqdm.auto import tqdm\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom sklearn.model_selection import StratifiedKFold\nfrom metric import score as cmi_score","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:18.118312Z","iopub.execute_input":"2025-06-25T21:10:18.118659Z","iopub.status.idle":"2025-06-25T21:10:21.901611Z","shell.execute_reply.started":"2025-06-25T21:10:18.118631Z","shell.execute_reply":"2025-06-25T21:10:21.900832Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"SEED = 2025\nBASE_DIR = pathlib.Path(\"/kaggle/input/cmi-detect-behavior-with-sensor-data\")\nOUT_DIR = pathlib.Path(\"/kaggle/working/\")\nprint(\"output dir:\", OUT_DIR)\n\n# data\nID_COL = \"sequence_id\"\nSEQ_COL = \"sequence_counter\"\nFEATURES = [\n    \"acc_x\",\n    \"acc_y\",\n    \"acc_z\",\n    \"rot_w\",\n    \"rot_x\",\n    \"rot_y\",\n    \"rot_z\",\n]\nTARGET = \"gesture\"\nMAX_LENGTH = 40\n\n# model\nNUM_CLASSES = 18\nHIDDEN_SIZE = 256\nN_LAYERS = 3\nN_CLF_LAYERS = 2\nBIDIRECTIONAL = True\nDROP = 0.15\nLABEL_SMOOTHING = 0.1\n\n# trainig\nBATCH_SIZE = 16\nEPOCHS = 70\nLR = 1e-3\nLR_MIN = 1e-5\nDEVICE = torch.device(\"cuda\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:21.902849Z","iopub.execute_input":"2025-06-25T21:10:21.903311Z","iopub.status.idle":"2025-06-25T21:10:21.909175Z","shell.execute_reply.started":"2025-06-25T21:10:21.903287Z","shell.execute_reply":"2025-06-25T21:10:21.908330Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"os.listdir(BASE_DIR)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:21.909978Z","iopub.execute_input":"2025-06-25T21:10:21.910238Z","iopub.status.idle":"2025-06-25T21:10:21.933301Z","shell.execute_reply.started":"2025-06-25T21:10:21.910215Z","shell.execute_reply":"2025-06-25T21:10:21.932590Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Load","metadata":{}},{"cell_type":"code","source":"path_data = BASE_DIR / \"train.csv\"\nheader = pd.read_csv(path_data, nrows=0)\ncolumns = header.columns\ncolumns = [x for x in columns if not x.startswith(\"tof\")]\ncolumns = [x for x in columns if not x.startswith(\"thm\")]\ncolumns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:21.934113Z","iopub.execute_input":"2025-06-25T21:10:21.934366Z","iopub.status.idle":"2025-06-25T21:10:21.991900Z","shell.execute_reply.started":"2025-06-25T21:10:21.934336Z","shell.execute_reply":"2025-06-25T21:10:21.991207Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data = pd.read_csv(path_data, usecols=columns)\ndata.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:21.994001Z","iopub.execute_input":"2025-06-25T21:10:21.994444Z","iopub.status.idle":"2025-06-25T21:10:39.558001Z","shell.execute_reply.started":"2025-06-25T21:10:21.994426Z","shell.execute_reply":"2025-06-25T21:10:39.557324Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.iloc[0].to_dict()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.558752Z","iopub.execute_input":"2025-06-25T21:10:39.558981Z","iopub.status.idle":"2025-06-25T21:10:39.564836Z","shell.execute_reply.started":"2025-06-25T21:10:39.558957Z","shell.execute_reply":"2025-06-25T21:10:39.564356Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cat = pd.Categorical(data[TARGET], ordered=True).dtype\ndata[TARGET] = pd.Categorical(data[TARGET], ordered=True)\ndata[TARGET] = data[TARGET].cat.codes","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.565419Z","iopub.execute_input":"2025-06-25T21:10:39.565578Z","iopub.status.idle":"2025-06-25T21:10:39.685356Z","shell.execute_reply.started":"2025-06-25T21:10:39.565565Z","shell.execute_reply":"2025-06-25T21:10:39.684632Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cat.categories, cat.ordered","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.686168Z","iopub.execute_input":"2025-06-25T21:10:39.686398Z","iopub.status.idle":"2025-06-25T21:10:39.691582Z","shell.execute_reply.started":"2025-06-25T21:10:39.686380Z","shell.execute_reply":"2025-06-25T21:10:39.691040Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"assert NUM_CLASSES == len(cat.categories)\nprint(f\"#{NUM_CLASSES} classes found.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.692218Z","iopub.execute_input":"2025-06-25T21:10:39.692687Z","iopub.status.idle":"2025-06-25T21:10:39.705513Z","shell.execute_reply.started":"2025-06-25T21:10:39.692671Z","shell.execute_reply":"2025-06-25T21:10:39.704982Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Split","metadata":{}},{"cell_type":"code","source":"def add_split(data):\n    data[\"split\"] = -1\n    df = data[[ID_COL, TARGET]].value_counts().reset_index()[[ID_COL, TARGET]].reset_index(drop=True)\n    splitter = StratifiedKFold(shuffle=True, random_state=SEED)\n    for i, (__, i_valid) in enumerate(splitter.split(df, y=df[TARGET])):\n        ind = df.iloc[i_valid].index\n        ids = df.loc[ind][ID_COL].values\n        ind = data[data[ID_COL].isin(ids)].index\n        data.loc[ind, \"split\"] = i","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.706049Z","iopub.execute_input":"2025-06-25T21:10:39.706250Z","iopub.status.idle":"2025-06-25T21:10:39.719705Z","shell.execute_reply.started":"2025-06-25T21:10:39.706236Z","shell.execute_reply":"2025-06-25T21:10:39.719047Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"add_split(data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.720281Z","iopub.execute_input":"2025-06-25T21:10:39.720482Z","iopub.status.idle":"2025-06-25T21:10:39.954582Z","shell.execute_reply.started":"2025-06-25T21:10:39.720469Z","shell.execute_reply":"2025-06-25T21:10:39.953969Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"splits = {}\nfor x in sorted(data[\"split\"].unique()):\n    splits[f\"split_{x}\"] = data[data[\"split\"] == x].groupby(ID_COL)[TARGET].first().value_counts()\npd.DataFrame(splits)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:39.955282Z","iopub.execute_input":"2025-06-25T21:10:39.955490Z","iopub.status.idle":"2025-06-25T21:10:40.083592Z","shell.execute_reply.started":"2025-06-25T21:10:39.955474Z","shell.execute_reply":"2025-06-25T21:10:40.082962Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ind_train = data[data[\"split\"] != 0].index\nind_valid = data[data[\"split\"] == 0].index\ntrain = data.loc[ind_train].copy()\nvalid = data.loc[ind_valid].copy()\nlen(train), len(valid)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.084225Z","iopub.execute_input":"2025-06-25T21:10:40.084471Z","iopub.status.idle":"2025-06-25T21:10:40.312514Z","shell.execute_reply.started":"2025-06-25T21:10:40.084454Z","shell.execute_reply":"2025-06-25T21:10:40.311775Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ids_train = train[ID_COL].unique()\nids_valid = valid[ID_COL].unique()\nlen(ids_train), len(ids_valid), set(ids_valid).isdisjoint(ids_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.315573Z","iopub.execute_input":"2025-06-25T21:10:40.315833Z","iopub.status.idle":"2025-06-25T21:10:40.345736Z","shell.execute_reply.started":"2025-06-25T21:10:40.315816Z","shell.execute_reply":"2025-06-25T21:10:40.345152Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Focus on a single fold","metadata":{}},{"cell_type":"code","source":"import joblib\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import Pipeline","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.346378Z","iopub.execute_input":"2025-06-25T21:10:40.346606Z","iopub.status.idle":"2025-06-25T21:10:40.529401Z","shell.execute_reply.started":"2025-06-25T21:10:40.346584Z","shell.execute_reply":"2025-06-25T21:10:40.528834Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"transformer = ColumnTransformer(\n    transformers=[\n        (\n            \"pipe\",\n            Pipeline([(\"im\", SimpleImputer()), (\"sc\", StandardScaler())]),\n            FEATURES,\n        )\n    ],\n    # remainder=\"passthrough\",\n    verbose_feature_names_out=False,\n)\ntransformer.set_output(transform=\"pandas\")\ntransformer.fit(train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.530048Z","iopub.execute_input":"2025-06-25T21:10:40.530343Z","iopub.status.idle":"2025-06-25T21:10:40.715585Z","shell.execute_reply.started":"2025-06-25T21:10:40.530318Z","shell.execute_reply":"2025-06-25T21:10:40.714845Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.loc[11111].to_dict()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.716481Z","iopub.execute_input":"2025-06-25T21:10:40.717162Z","iopub.status.idle":"2025-06-25T21:10:40.734317Z","shell.execute_reply.started":"2025-06-25T21:10:40.717138Z","shell.execute_reply":"2025-06-25T21:10:40.733699Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train[FEATURES] = transformer.transform(train[FEATURES])\nvalid[FEATURES] = transformer.transform(valid[FEATURES])\ntrain.loc[11111].to_dict()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.734939Z","iopub.execute_input":"2025-06-25T21:10:40.735170Z","iopub.status.idle":"2025-06-25T21:10:40.869259Z","shell.execute_reply.started":"2025-06-25T21:10:40.735154Z","shell.execute_reply":"2025-06-25T21:10:40.868617Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"joblib.dump(transformer, OUT_DIR / \"preprocessing.joblib\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.870060Z","iopub.execute_input":"2025-06-25T21:10:40.870566Z","iopub.status.idle":"2025-06-25T21:10:40.877384Z","shell.execute_reply.started":"2025-06-25T21:10:40.870543Z","shell.execute_reply":"2025-06-25T21:10:40.876857Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Dataset & Dataloader\n- Note that we ultimately want to have the input to be of the form ``[seq, batch, feature]``.\n- Mask is only of size ``[seq, batch]``.","metadata":{}},{"cell_type":"code","source":"from torch.utils.data import Dataset, DataLoader\nfrom torch.nn.utils.rnn import pad_sequence","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.878216Z","iopub.execute_input":"2025-06-25T21:10:40.878428Z","iopub.status.idle":"2025-06-25T21:10:40.887439Z","shell.execute_reply.started":"2025-06-25T21:10:40.878403Z","shell.execute_reply":"2025-06-25T21:10:40.886816Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CMIData(Dataset):\n    def __init__(self, data, max_length=None):\n        super().__init__()\n        self.d_data = dict(list(data.groupby(ID_COL)))\n        self.features = FEATURES\n        self.target = TARGET\n        self.max_length = max_length\n        self.keys = list(self.d_data)\n\n    def __len__(self):\n        return len(self.keys)\n\n    def __getitem__(self, index):\n        df = self.d_data[self.keys[index]]\n        if self.max_length is not None:\n            df = df.iloc[-self.max_length :]\n        return (\n            torch.tensor(df[self.features].values.astype(np.float32)),\n            torch.tensor(df[self.target].values[-1].astype(np.int64)),\n        )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.888167Z","iopub.execute_input":"2025-06-25T21:10:40.888400Z","iopub.status.idle":"2025-06-25T21:10:40.903524Z","shell.execute_reply.started":"2025-06-25T21:10:40.888386Z","shell.execute_reply":"2025-06-25T21:10:40.902926Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ds = CMIData(train, max_length=10)\nlen(ds)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:40.904166Z","iopub.execute_input":"2025-06-25T21:10:40.904369Z","iopub.status.idle":"2025-06-25T21:10:41.624112Z","shell.execute_reply.started":"2025-06-25T21:10:40.904355Z","shell.execute_reply":"2025-06-25T21:10:41.623382Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"x, y = ds[0]\nx, y","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:41.624852Z","iopub.execute_input":"2025-06-25T21:10:41.625134Z","iopub.status.idle":"2025-06-25T21:10:41.713979Z","shell.execute_reply.started":"2025-06-25T21:10:41.625080Z","shell.execute_reply":"2025-06-25T21:10:41.713406Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def collate_fn(batch):\n    x_tensors = [item[0] for item in batch]\n    y_tensors = [item[1] for item in batch]\n\n    batch_x = pad_sequence(x_tensors, batch_first=False, padding_value=0.0)\n    batch_y = torch.tensor(y_tensors)\n    # batch_y = pad_sequence(y_tensors, batch_first=False, padding_value=-100)\n    mask = pad_sequence(\n        [torch.ones_like(x[..., -1]) for x in x_tensors], padding_value=0\n    )\n    return batch_x, batch_y, mask","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:41.714626Z","iopub.execute_input":"2025-06-25T21:10:41.714796Z","iopub.status.idle":"2025-06-25T21:10:41.719370Z","shell.execute_reply.started":"2025-06-25T21:10:41.714783Z","shell.execute_reply":"2025-06-25T21:10:41.718651Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ds = CMIData(train, max_length=100)\ndl = DataLoader(ds, batch_size=2, shuffle=False, collate_fn=collate_fn)\nx, y, mask = next(iter(dl))\nx.shape, y.shape, mask.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:41.719988Z","iopub.execute_input":"2025-06-25T21:10:41.720182Z","iopub.status.idle":"2025-06-25T21:10:42.285253Z","shell.execute_reply.started":"2025-06-25T21:10:41.720167Z","shell.execute_reply":"2025-06-25T21:10:42.284626Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mask[-10:]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.285890Z","iopub.execute_input":"2025-06-25T21:10:42.286072Z","iopub.status.idle":"2025-06-25T21:10:42.291720Z","shell.execute_reply.started":"2025-06-25T21:10:42.286058Z","shell.execute_reply":"2025-06-25T21:10:42.291023Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model\nStart with the following architecture:\n- input passes thru LSTM layers\n- compute mean over all hidden states respecting mask\n- apply dropout\n- use output classifier to get logits","metadata":{}},{"cell_type":"code","source":"class LSTMClassifier(nn.Module):\n\n    def __init__(self):\n        super().__init__()\n        self.drop = nn.Dropout(DROP)\n        self.lstm = nn.LSTM(\n            input_size=len(FEATURES),\n            hidden_size=HIDDEN_SIZE,\n            num_layers=N_LAYERS,\n            bidirectional=BIDIRECTIONAL,\n            dropout=DROP if N_LAYERS > 1 else 0.0,\n            batch_first=False,\n        )\n        output_size = 2 * HIDDEN_SIZE if BIDIRECTIONAL else HIDDEN_SIZE\n        clf_layers = []\n        for i in range(N_CLF_LAYERS - 1):\n            clf_layers.append(nn.Linear(output_size, output_size))\n            clf_layers.append(nn.ReLU())\n        clf_layers.append(nn.Linear(output_size, NUM_CLASSES))\n        self.clf = nn.Sequential(*clf_layers)\n\n    def forward(self, x, mask):\n        x, _ = self.lstm(x)\n        x = torch.mean(x * mask.unsqueeze(-1), dim=0)\n        x = self.drop(x)\n        return self.clf(x)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.292916Z","iopub.execute_input":"2025-06-25T21:10:42.293581Z","iopub.status.idle":"2025-06-25T21:10:42.303851Z","shell.execute_reply.started":"2025-06-25T21:10:42.293564Z","shell.execute_reply":"2025-06-25T21:10:42.303234Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = LSTMClassifier()\nmodel","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.304461Z","iopub.execute_input":"2025-06-25T21:10:42.304669Z","iopub.status.idle":"2025-06-25T21:10:42.346959Z","shell.execute_reply.started":"2025-06-25T21:10:42.304647Z","shell.execute_reply":"2025-06-25T21:10:42.346436Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"with torch.no_grad():\n    logits = model(x, mask)\nlogits","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.347524Z","iopub.execute_input":"2025-06-25T21:10:42.347681Z","iopub.status.idle":"2025-06-25T21:10:42.463783Z","shell.execute_reply.started":"2025-06-25T21:10:42.347669Z","shell.execute_reply":"2025-06-25T21:10:42.463192Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"loss_fn = nn.CrossEntropyLoss(label_smoothing=LABEL_SMOOTHING)\nloss_fn","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.464486Z","iopub.execute_input":"2025-06-25T21:10:42.464823Z","iopub.status.idle":"2025-06-25T21:10:42.469702Z","shell.execute_reply.started":"2025-06-25T21:10:42.464806Z","shell.execute_reply":"2025-06-25T21:10:42.468990Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"loss_fn(logits, y)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.470469Z","iopub.execute_input":"2025-06-25T21:10:42.470650Z","iopub.status.idle":"2025-06-25T21:10:42.492698Z","shell.execute_reply.started":"2025-06-25T21:10:42.470636Z","shell.execute_reply":"2025-06-25T21:10:42.492199Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"ds_train = CMIData(train, max_length=MAX_LENGTH)\ntrain_loader = DataLoader(\n    ds_train, \n    batch_size=BATCH_SIZE, \n    shuffle=True, \n    collate_fn=collate_fn, \n    drop_last=True\n)\nprint(f\"Training dataset contains: {len(ds_train)}\")\nds_valid = CMIData(valid, max_length=MAX_LENGTH)\nvalid_loader = DataLoader(\n    ds_valid,\n    shuffle=False,\n    batch_size=BATCH_SIZE,\n    collate_fn=collate_fn,\n    drop_last=True,\n)\nprint(f\"Valid dataset contains: {len(ds_valid)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:42.493332Z","iopub.execute_input":"2025-06-25T21:10:42.493512Z","iopub.status.idle":"2025-06-25T21:10:43.298037Z","shell.execute_reply.started":"2025-06-25T21:10:42.493497Z","shell.execute_reply":"2025-06-25T21:10:43.297489Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = LSTMClassifier().to(DEVICE)\nloss_fn = nn.CrossEntropyLoss(label_smoothing=LABEL_SMOOTHING)\noptimizer = torch.optim.Adam(model.parameters(), lr=LR)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n    optimizer, factor=0.1, patience=10, min_lr=LR_MIN\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:43.298695Z","iopub.execute_input":"2025-06-25T21:10:43.298871Z","iopub.status.idle":"2025-06-25T21:10:45.982560Z","shell.execute_reply.started":"2025-06-25T21:10:43.298858Z","shell.execute_reply":"2025-06-25T21:10:45.982008Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def evalute(model, data_loader, loss_fn):\n    model.eval()\n    loss = 0.0\n    with torch.no_grad():\n        for x, y, mask in data_loader:\n            x = x.to(DEVICE)\n            y = y.to(DEVICE)\n            mask = mask.to(DEVICE)\n            logits = model(x, mask)\n            loss += loss_fn(logits, y).item()\n    model.train()\n    return loss / len(data_loader)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:45.983317Z","iopub.execute_input":"2025-06-25T21:10:45.983738Z","iopub.status.idle":"2025-06-25T21:10:45.989673Z","shell.execute_reply.started":"2025-06-25T21:10:45.983712Z","shell.execute_reply":"2025-06-25T21:10:45.989013Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def predict(model, data):\n    model.eval()\n    ds = CMIData(data, max_length=MAX_LENGTH)\n    dl = DataLoader(\n        ds, shuffle=False, batch_size=BATCH_SIZE, collate_fn=collate_fn, drop_last=False\n    )\n    y_true = []\n    y_pred = []\n    with torch.no_grad():\n        for x, y, m in dl:\n            x = x.to(DEVICE)\n            y = y.to(DEVICE)\n            m = m.to(DEVICE)\n            logits = model(x, m)\n            y_pred.append(torch.argmax(logits, dim=-1).cpu().numpy())\n            y_true.append(y.cpu().numpy())\n    sol = pd.DataFrame(\n        {\"id\": ds.keys, \"gesture\": cat.categories[np.concatenate(y_true)].values}\n    )\n    sub = pd.DataFrame(\n        {\"id\": ds.keys, \"gesture\": cat.categories[np.concatenate(y_pred)].values}\n    )\n    return sol, sub","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:45.990320Z","iopub.execute_input":"2025-06-25T21:10:45.990599Z","iopub.status.idle":"2025-06-25T21:10:46.016321Z","shell.execute_reply.started":"2025-06-25T21:10:45.990575Z","shell.execute_reply":"2025-06-25T21:10:46.015648Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sol, sub = predict(model, valid)\ncmi_score(sol, sub, \"id\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:46.017157Z","iopub.execute_input":"2025-06-25T21:10:46.017401Z","iopub.status.idle":"2025-06-25T21:10:47.778831Z","shell.execute_reply.started":"2025-06-25T21:10:46.017377Z","shell.execute_reply":"2025-06-25T21:10:47.778120Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class Monitor:\n    def __init__(self):\n        self.records = collections.defaultdict(list)\n\n    def add(self, metric_name, epoch, value):\n        self.records[metric_name].append({\"epoch\": epoch, \"value\": value})\n        print(f\"Epoch {epoch}/{EPOCHS} - {metric_name}: {value:.4g}\")\n\n    @property\n    def dataframe(self):\n        return pd.DataFrame(\n            {\n                k: pd.DataFrame(v).rename(columns={\"value\": k}).set_index(\"epoch\")[k]\n                for k, v in self.records.items()\n            }\n        )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:47.779842Z","iopub.execute_input":"2025-06-25T21:10:47.780394Z","iopub.status.idle":"2025-06-25T21:10:47.785923Z","shell.execute_reply.started":"2025-06-25T21:10:47.780358Z","shell.execute_reply":"2025-06-25T21:10:47.785146Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\nbest_val_loss = float('inf')\nbest_val_score = float('-inf')\n\n# training loop\nmonitor = Monitor()\nfor epoch in tqdm(range(1, EPOCHS + 1)):\n    \n    # training\n    model.train()\n    losses = []\n    for x, y, mask in train_loader:\n        x = x.to(DEVICE)\n        y = y.to(DEVICE)\n        mask = mask.to(DEVICE)\n        logits = model(x, mask)\n        loss = loss_fn(logits, y)\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n        losses.append(loss.item())\n    train_loss = np.mean(losses)\n    monitor.add(\"train_loss\", epoch, train_loss)\n    \n    # validation\n    val_loss = evalute(model, valid_loader, loss_fn)\n    monitor.add(\"val_loss\", epoch, val_loss)\n    sol, sub = predict(model, valid)\n    val_score = cmi_score(sol, sub, \"id\")\n    monitor.add(\"val_cmi\", epoch, val_score)\n    scheduler.step(val_loss)\n    monitor.add(\"lr\", epoch, optimizer.param_groups[0]['lr'])\n\n    # checkpointing\n    if val_loss < best_val_loss:\n        print(f\"Validation loss improved from {best_val_loss:.4f} to {val_loss:.4}\")\n        path_model = OUT_DIR / \"model_best.pt\"\n        torch.save(model.state_dict(), path_model)\n        best_val_loss = val_loss\n\n    # checkpointing\n    if val_score > best_val_score:\n        print(f\"Validation score improved from {best_val_score:.4f} to {val_score:.4}\")\n        path_model_score = OUT_DIR / \"model_best_score.pt\"\n        torch.save(model.state_dict(), path_model_score)\n        best_val_score = val_score","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:10:47.800151Z","iopub.execute_input":"2025-06-25T21:10:47.800360Z","iopub.status.idle":"2025-06-25T21:11:11.495957Z","shell.execute_reply.started":"2025-06-25T21:10:47.800338Z","shell.execute_reply":"2025-06-25T21:11:11.494874Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"hist = monitor.dataframe\nhist.to_csv(OUT_DIR / \"history.csv\")\nhist","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:11.496914Z","iopub.execute_input":"2025-06-25T21:11:11.497211Z","iopub.status.idle":"2025-06-25T21:11:11.515480Z","shell.execute_reply.started":"2025-06-25T21:11:11.497187Z","shell.execute_reply":"2025-06-25T21:11:11.514878Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"hist.loc[hist[\"val_loss\"].idxmin()]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:11.519623Z","iopub.execute_input":"2025-06-25T21:11:11.519811Z","iopub.status.idle":"2025-06-25T21:11:11.525513Z","shell.execute_reply.started":"2025-06-25T21:11:11.519797Z","shell.execute_reply":"2025-06-25T21:11:11.524885Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"hist.loc[hist[\"val_cmi\"].idxmax()]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:11.526074Z","iopub.execute_input":"2025-06-25T21:11:11.526368Z","iopub.status.idle":"2025-06-25T21:11:11.542366Z","shell.execute_reply.started":"2025-06-25T21:11:11.526350Z","shell.execute_reply":"2025-06-25T21:11:11.541761Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axs = plt.subplots(nrows=2, figsize=(12, 8), sharex=\"col\")\nplt_kwargs = dict(grid=True, marker='x')\nhist[[\"train_loss\", \"val_loss\"]].plot(ax=axs[0], **plt_kwargs)\nhist[[\"val_cmi\"]].plot(ax=axs[1], **plt_kwargs)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:11.542980Z","iopub.execute_input":"2025-06-25T21:11:11.543197Z","iopub.status.idle":"2025-06-25T21:11:11.951254Z","shell.execute_reply.started":"2025-06-25T21:11:11.543181Z","shell.execute_reply":"2025-06-25T21:11:11.950566Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Eval","metadata":{}},{"cell_type":"code","source":"model.load_state_dict(torch.load(path_model, weights_only=True))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:11.952032Z","iopub.execute_input":"2025-06-25T21:11:11.952319Z","iopub.status.idle":"2025-06-25T21:11:11.975026Z","shell.execute_reply.started":"2025-06-25T21:11:11.952302Z","shell.execute_reply":"2025-06-25T21:11:11.974493Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sol, sub = predict(model, valid)\ncmi_score(sol, sub, \"id\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:11.975674Z","iopub.execute_input":"2025-06-25T21:11:11.975884Z","iopub.status.idle":"2025-06-25T21:11:13.426405Z","shell.execute_reply.started":"2025-06-25T21:11:11.975861Z","shell.execute_reply":"2025-06-25T21:11:13.425641Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model.load_state_dict(torch.load(path_model_score, weights_only=True))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:13.427164Z","iopub.execute_input":"2025-06-25T21:11:13.427442Z","iopub.status.idle":"2025-06-25T21:11:13.443949Z","shell.execute_reply.started":"2025-06-25T21:11:13.427421Z","shell.execute_reply":"2025-06-25T21:11:13.443148Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sol, sub = predict(model, valid)\ncmi_score(sol, sub, \"id\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T21:11:13.444683Z","iopub.execute_input":"2025-06-25T21:11:13.444933Z","iopub.status.idle":"2025-06-25T21:11:14.897962Z","shell.execute_reply.started":"2025-06-25T21:11:13.444913Z","shell.execute_reply":"2025-06-25T21:11:14.897303Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}