{
  "id": 582464,
  "title": "A metric computation function",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/582464",
  "author_name": "",
  "post_date": "2025-05-31T13:14:54.587797800Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Share a metric evaluation function.  </p>\n<pre><code> ():\n\n    \n    binary_true = [Cfg.label_map.all_collapse(label)  label  labels]\n    binary_pred = [Cfg.label_map.all_collapse(pred)  pred  preds]\n    binary_f1 = f1_score(binary_true, binary_pred)\n\n    \n    collapsed_true = [Cfg.label_map.nobfrb_collapse(label)  label  labels]\n    collapsed_pred = [Cfg.label_map.nobfrb_collapse(pred)  pred  preds]\n\n    unique_labels = ((collapsed_true + collapsed_pred))  \n    macro_f1 = f1_score(collapsed_true, collapsed_pred, average=, labels=unique_labels)\n\n    final_score = (binary_f1 + macro_f1) / \n     final_score, binary_f1, macro_f1\n\n\n :\n     ():\n        .label2idx = {\n            : , : , \n            : , : , \n            : , : , \n            : , : ,\n            : , : , \n            : , : , \n            : , : , \n            : , : , \n            : , : \n        }\n        .idx2label = {\n            : , : , \n            : , : , \n            : , : , \n            : , : ,\n            : , : , \n            : , : , \n            : , : , \n            : , : , \n            : , : \n        }\n        .collapse_nobfrb = \n        (, (.label2idx), , (.idx2label))\n\n     ():\n         label  .label2idx\n         .label2idx[label]\n\n     ():        \n         idx  .idx2label\n         .idx2label[idx]\n\n     ():\n         [k  k, v  .label2idx.items()  v &lt; .collapse_nobfrb]\n\n     ():\n         (.label2idx.keys())\n\n     ():\n          &lt;= idx &lt;= \n            &lt;= idx &lt; .collapse_nobfrb  \n\n     ():\n          &lt;= idx &lt;= \n         idx   &lt;= idx &lt; .collapse_nobfrb  .collapse_nobfrb\n</code></pre>",
  "messages": [
    {
      "id": "3214407",
      "postDate": "05/31/2025 13:14:54",
      "content": "<p>Share a metric evaluation function.  </p>\n<pre><code> ():\n\n    \n    binary_true = [Cfg.label_map.all_collapse(label)  label  labels]\n    binary_pred = [Cfg.label_map.all_collapse(pred)  pred  preds]\n    binary_f1 = f1_score(binary_true, binary_pred)\n\n    \n    collapsed_true = [Cfg.label_map.nobfrb_collapse(label)  label  labels]\n    collapsed_pred = [Cfg.label_map.nobfrb_collapse(pred)  pred  preds]\n\n    unique_labels = ((collapsed_true + collapsed_pred))  \n    macro_f1 = f1_score(collapsed_true, collapsed_pred, average=, labels=unique_labels)\n\n    final_score = (binary_f1 + macro_f1) / \n     final_score, binary_f1, macro_f1\n\n\n :\n     ():\n        .label2idx = {\n            : , : , \n            : , : , \n            : , : , \n            : , : ,\n            : , : , \n            : , : , \n            : , : , \n            : , : , \n            : , : \n        }\n        .idx2label = {\n            : , : , \n            : , : , \n            : , : , \n            : , : ,\n            : , : , \n            : , : , \n            : , : , \n            : , : , \n            : , : \n        }\n        .collapse_nobfrb = \n        (, (.label2idx), , (.idx2label))\n\n     ():\n         label  .label2idx\n         .label2idx[label]\n\n     ():        \n         idx  .idx2label\n         .idx2label[idx]\n\n     ():\n         [k  k, v  .label2idx.items()  v &lt; .collapse_nobfrb]\n\n     ():\n         (.label2idx.keys())\n\n     ():\n          &lt;= idx &lt;= \n            &lt;= idx &lt; .collapse_nobfrb  \n\n     ():\n          &lt;= idx &lt;= \n         idx   &lt;= idx &lt; .collapse_nobfrb  .collapse_nobfrb\n</code></pre>",
      "rawMarkdown": "Share a metric evaluation function.  \n\n```\ndef evaluate_f1_metric(labels, preds):\n    \n    # Binary F1: 1 = target, 0 = non-target\n    binary_true = [Cfg.label_map.all_collapse(label) for label in labels]\n    binary_pred = [Cfg.label_map.all_collapse(pred) for pred in preds]\n    binary_f1 = f1_score(binary_true, binary_pred)\n\n    # Macro F1: collapse all non-target into 'non_target' class\n    collapsed_true = [Cfg.label_map.nobfrb_collapse(label) for label in labels]\n    collapsed_pred = [Cfg.label_map.nobfrb_collapse(pred) for pred in preds]\n    \n    unique_labels = list(set(collapsed_true + collapsed_pred))  # Ensure all present in macro\n    macro_f1 = f1_score(collapsed_true, collapsed_pred, average='macro', labels=unique_labels)\n\n    final_score = (binary_f1 + macro_f1) / 2\n    return final_score, binary_f1, macro_f1\n\n\nclass LabelMap:\n    def __init__(self):\n        self.label2idx = {\n            'Neck - scratch': 0, 'Eyebrow - pull hair': 1, \n            'Forehead - scratch': 2, 'Forehead - pull hairline': 3, \n            'Above ear - pull hair': 4, 'Neck - pinch skin': 5, \n            'Eyelash - pull hair': 6, 'Cheek - pinch skin': 7,\n            'Text on phone': 8, 'Wave hello': 9, \n            'Write name in air': 10, 'Pull air toward your face': 11, \n            'Feel around in tray and pull out an object': 12, 'Glasses on/off': 13, \n            'Drink from bottle/cup': 14, 'Scratch knee/leg skin': 15, \n            'Write name on leg': 16, 'Pinch knee/leg skin': 17\n        }\n        self.idx2label = {\n            0: 'Neck-scratch', 1: 'Eyebrow - pull hair', \n            2: 'Forehead - scratch', 3: 'Forehead - pull hairline', \n            4: 'Above ear - pull hair', 5: 'Neck - pinch skin', \n            6: 'Eyelash - pull hair', 7: 'Cheek - pinch skin',\n            8: 'Text on phone', 9: 'Wave hello', \n            10: 'Write name in air', 11: 'Pull air toward your face', \n            12: 'Feel around in tray and pull out an object', 13: 'Glasses on/off', \n            14: 'Drink from bottle/cup', 15: 'Scratch knee/leg skin', \n            16: 'Write name on leg', 17: 'Pinch knee/leg skin'\n        }\n        self.collapse_nobfrb = 8\n        print('label2idx len', len(self.label2idx), 'idx2label len', len(self.idx2label))\n\n    def get_idx(self, label):\n        assert label in self.label2idx\n        return self.label2idx[label]\n\n    def get_label(self, idx):        \n        assert idx in self.idx2label\n        return self.idx2label[idx]\n\n    def get_bfrb_gestures(self):\n        return [k for k, v in self.label2idx.items() if v < self.collapse_nobfrb]\n\n    def keys(self):\n        return list(self.label2idx.keys())\n\n    def all_collapse(self, idx):\n        assert 0 <= idx <= 17\n        return 1 if 0 <= idx < self.collapse_nobfrb else 0\n\n    def nobfrb_collapse(self, idx):\n        assert 0 <= idx <= 17\n        return idx if 0 <= idx < self.collapse_nobfrb else self.collapse_nobfrb\n\n```",
      "votes": null
    },
    {
      "id": "3214409",
      "postDate": "05/31/2025 13:16:46",
      "content": "<p>We already have this here - <a href=\"https://www.kaggle.com/code/metric/cmi-2025\" target=\"_blank\">https://www.kaggle.com/code/metric/cmi-2025</a> <a href=\"https://www.kaggle.com/andrewguanzc\" target=\"_blank\">@andrewguanzc</a> </p>",
      "rawMarkdown": "We already have this here - https://www.kaggle.com/code/metric/cmi-2025 @andrewguanzc",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3214409,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "05/31/2025 13:16:46",
      "content": "<p>We already have this here - <a href=\"https://www.kaggle.com/code/metric/cmi-2025\" target=\"_blank\">https://www.kaggle.com/code/metric/cmi-2025</a> <a href=\"https://www.kaggle.com/andrewguanzc\" target=\"_blank\">@andrewguanzc</a> </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3214407": "Share a metric evaluation function.  \n\n```\ndef evaluate_f1_metric(labels, preds):\n    \n    # Binary F1: 1 = target, 0 = non-target\n    binary_true = [Cfg.label_map.all_collapse(label) for label in labels]\n    binary_pred = [Cfg.label_map.all_collapse(pred) for pred in preds]\n    binary_f1 = f1_score(binary_true, binary_pred)\n\n    # Macro F1: collapse all non-target into 'non_target' class\n    collapsed_true = [Cfg.label_map.nobfrb_collapse(label) for label in labels]\n    collapsed_pred = [Cfg.label_map.nobfrb_collapse(pred) for pred in preds]\n    \n    unique_labels = list(set(collapsed_true + collapsed_pred))  # Ensure all present in macro\n    macro_f1 = f1_score(collapsed_true, collapsed_pred, average='macro', labels=unique_labels)\n\n    final_score = (binary_f1 + macro_f1) / 2\n    return final_score, binary_f1, macro_f1\n\n\nclass LabelMap:\n    def __init__(self):\n        self.label2idx = {\n            'Neck - scratch': 0, 'Eyebrow - pull hair': 1, \n            'Forehead - scratch': 2, 'Forehead - pull hairline': 3, \n            'Above ear - pull hair': 4, 'Neck - pinch skin': 5, \n            'Eyelash - pull hair': 6, 'Cheek - pinch skin': 7,\n            'Text on phone': 8, 'Wave hello': 9, \n            'Write name in air': 10, 'Pull air toward your face': 11, \n            'Feel around in tray and pull out an object': 12, 'Glasses on/off': 13, \n            'Drink from bottle/cup': 14, 'Scratch knee/leg skin': 15, \n            'Write name on leg': 16, 'Pinch knee/leg skin': 17\n        }\n        self.idx2label = {\n            0: 'Neck-scratch', 1: 'Eyebrow - pull hair', \n            2: 'Forehead - scratch', 3: 'Forehead - pull hairline', \n            4: 'Above ear - pull hair', 5: 'Neck - pinch skin', \n            6: 'Eyelash - pull hair', 7: 'Cheek - pinch skin',\n            8: 'Text on phone', 9: 'Wave hello', \n            10: 'Write name in air', 11: 'Pull air toward your face', \n            12: 'Feel around in tray and pull out an object', 13: 'Glasses on/off', \n            14: 'Drink from bottle/cup', 15: 'Scratch knee/leg skin', \n            16: 'Write name on leg', 17: 'Pinch knee/leg skin'\n        }\n        self.collapse_nobfrb = 8\n        print('label2idx len', len(self.label2idx), 'idx2label len', len(self.idx2label))\n\n    def get_idx(self, label):\n        assert label in self.label2idx\n        return self.label2idx[label]\n\n    def get_label(self, idx):        \n        assert idx in self.idx2label\n        return self.idx2label[idx]\n\n    def get_bfrb_gestures(self):\n        return [k for k, v in self.label2idx.items() if v < self.collapse_nobfrb]\n\n    def keys(self):\n        return list(self.label2idx.keys())\n\n    def all_collapse(self, idx):\n        assert 0 <= idx <= 17\n        return 1 if 0 <= idx < self.collapse_nobfrb else 0\n\n    def nobfrb_collapse(self, idx):\n        assert 0 <= idx <= 17\n        return idx if 0 <= idx < self.collapse_nobfrb else self.collapse_nobfrb\n\n```",
    "3214409": "We already have this here - https://www.kaggle.com/code/metric/cmi-2025 @andrewguanzc"
  },
  "source": "meta"
}