{
  "id": 586595,
  "title": "CV 0.76, LB 0.18",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/586595",
  "author_name": "",
  "post_date": "2025-06-27T11:11:49.564020500Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I thought I'm smart. Trained and validated only on the ~ last 20 TS with RNN (with some random ~ 10 steps back). My grouping and CV is based on 'sequence_id' rather than 'subject', which I understand is an issue.<br>\nMy CV - 0.76, LB 0.18.<br>\nI want to ask whether adding a detection mechanism is needed? Where is actually the gesture?<br>\nFrom your investigation, is the test more or less behaving like the training, is the gesture towards the end, or was it just wishful thinking on my end?<br>\nAny other tips how to see better LB results, closer to what I get with CV?    </p>",
  "messages": [
    {
      "id": "3233948",
      "postDate": "06/27/2025 11:11:49",
      "content": "<p>I thought I'm smart. Trained and validated only on the ~ last 20 TS with RNN (with some random ~ 10 steps back). My grouping and CV is based on 'sequence_id' rather than 'subject', which I understand is an issue.<br>\nMy CV - 0.76, LB 0.18.<br>\nI want to ask whether adding a detection mechanism is needed? Where is actually the gesture?<br>\nFrom your investigation, is the test more or less behaving like the training, is the gesture towards the end, or was it just wishful thinking on my end?<br>\nAny other tips how to see better LB results, closer to what I get with CV?    </p>",
      "rawMarkdown": "I thought I'm smart. Trained and validated only on the ~ last 20 TS with RNN (with some random ~ 10 steps back). My grouping and CV is based on 'sequence_id' rather than 'subject', which I understand is an issue.\nMy CV - 0.76, LB 0.18.\nI want to ask whether adding a detection mechanism is needed? Where is actually the gesture?\nFrom your investigation, is the test more or less behaving like the training, is the gesture towards the end, or was it just wishful thinking on my end?\nAny other tips how to see better LB results, closer to what I get with CV?",
      "votes": null
    },
    {
      "id": "3233966",
      "postDate": "06/27/2025 11:26:59",
      "content": "<p>This is almost certainly a bug in your code somewhere. At lb 0.18 it is predicting too many non gesture also. Just predicting any target gesture should get almost 0.4.  It there is a bug in your label processing or something else. </p>",
      "rawMarkdown": "This is almost certainly a bug in your code somewhere. At lb 0.18 it is predicting too many non gesture also. Just predicting any target gesture should get almost 0.4.  It there is a bug in your label processing or something else.",
      "votes": null
    },
    {
      "id": "3234006",
      "postDate": "06/27/2025 12:13:29",
      "content": "<p>Thank you Devin. Taking your inputs into account. Will look for the bug. Can you please also opinion on \"detecting the gesture on the last ~20 TS (also in the test?)\"?</p>",
      "rawMarkdown": "Thank you Devin. Taking your inputs into account. Will look for the bug. Can you please also opinion on \"detecting the gesture on the last ~20 TS (also in the test?)\"?",
      "votes": null
    },
    {
      "id": "3234178",
      "postDate": "06/27/2025 15:37:39",
      "content": "<p>It might be a little short. Most gestures are about 30 long. And there is signal in the pregesture movements. Experiment and see how your score changes. </p>",
      "rawMarkdown": "It might be a little short. Most gestures are about 30 long. And there is signal in the pregesture movements. Experiment and see how your score changes.",
      "votes": null
    },
    {
      "id": "3234663",
      "postDate": "06/28/2025 07:00:22",
      "content": "<p>I standardized the data during training, but I forgot to standardize it during prediction, and my LB score ended up being 0.21😅</p>",
      "rawMarkdown": "I standardized the data during training, but I forgot to standardize it during prediction, and my LB score ended up being 0.21😅",
      "votes": null
    },
    {
      "id": "3235163",
      "postDate": "06/28/2025 19:11:34",
      "content": "<p>Hello, <a href=\"https://www.kaggle.com/abosol\" target=\"_blank\">@abosol</a>! I am not really sure if stratifying by sequence_id is meaningful, or may be I dont understand the statement. </p>\n<p>Speaking of applying segmentation of Transition/gesture parts if a sequence, it has increased my cv and lb scores, but I am not using time-aware nodels like RNN. <br>\nImpo, segmenting would work well for simple models like RNN, but for more complicated models it would be bad (ive tested it on a transformer)</p>",
      "rawMarkdown": "Hello, @abosol! I am not really sure if stratifying by sequence_id is meaningful, or may be I dont understand the statement. \n\nSpeaking of applying segmentation of Transition/gesture parts if a sequence, it has increased my cv and lb scores, but I am not using time-aware nodels like RNN. \nImpo, segmenting would work well for simple models like RNN, but for more complicated models it would be bad (ive tested it on a transformer)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3233966,
      "author_name": "devinanzelmo",
      "author_url": "",
      "post_date": "06/27/2025 11:26:59",
      "content": "<p>This is almost certainly a bug in your code somewhere. At lb 0.18 it is predicting too many non gesture also. Just predicting any target gesture should get almost 0.4.  It there is a bug in your label processing or something else. </p>",
      "votes": null,
      "replies": [
        {
          "id": 3234006,
          "author_name": "abosol",
          "author_url": "",
          "post_date": "06/27/2025 12:13:29",
          "content": "<p>Thank you Devin. Taking your inputs into account. Will look for the bug. Can you please also opinion on \"detecting the gesture on the last ~20 TS (also in the test?)\"?</p>",
          "votes": null,
          "replies": [
            {
              "id": 3234178,
              "author_name": "devinanzelmo",
              "author_url": "",
              "post_date": "06/27/2025 15:37:39",
              "content": "<p>It might be a little short. Most gestures are about 30 long. And there is signal in the pregesture movements. Experiment and see how your score changes. </p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3234663,
      "author_name": "youheitomio",
      "author_url": "",
      "post_date": "06/28/2025 07:00:22",
      "content": "<p>I standardized the data during training, but I forgot to standardize it during prediction, and my LB score ended up being 0.21😅</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3235163,
      "author_name": "oostg0t",
      "author_url": "",
      "post_date": "06/28/2025 19:11:34",
      "content": "<p>Hello, <a href=\"https://www.kaggle.com/abosol\" target=\"_blank\">@abosol</a>! I am not really sure if stratifying by sequence_id is meaningful, or may be I dont understand the statement. </p>\n<p>Speaking of applying segmentation of Transition/gesture parts if a sequence, it has increased my cv and lb scores, but I am not using time-aware nodels like RNN. <br>\nImpo, segmenting would work well for simple models like RNN, but for more complicated models it would be bad (ive tested it on a transformer)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3233948": "I thought I'm smart. Trained and validated only on the ~ last 20 TS with RNN (with some random ~ 10 steps back). My grouping and CV is based on 'sequence_id' rather than 'subject', which I understand is an issue.\nMy CV - 0.76, LB 0.18.\nI want to ask whether adding a detection mechanism is needed? Where is actually the gesture?\nFrom your investigation, is the test more or less behaving like the training, is the gesture towards the end, or was it just wishful thinking on my end?\nAny other tips how to see better LB results, closer to what I get with CV?",
    "3233966": "This is almost certainly a bug in your code somewhere. At lb 0.18 it is predicting too many non gesture also. Just predicting any target gesture should get almost 0.4.  It there is a bug in your label processing or something else.",
    "3234006": "Thank you Devin. Taking your inputs into account. Will look for the bug. Can you please also opinion on \"detecting the gesture on the last ~20 TS (also in the test?)\"?",
    "3234178": "It might be a little short. Most gestures are about 30 long. And there is signal in the pregesture movements. Experiment and see how your score changes.",
    "3234663": "I standardized the data during training, but I forgot to standardize it during prediction, and my LB score ended up being 0.21😅",
    "3235163": "Hello, @abosol! I am not really sure if stratifying by sequence_id is meaningful, or may be I dont understand the statement. \n\nSpeaking of applying segmentation of Transition/gesture parts if a sequence, it has increased my cv and lb scores, but I am not using time-aware nodels like RNN. \nImpo, segmenting would work well for simple models like RNN, but for more complicated models it would be bad (ive tested it on a transformer)"
  },
  "source": "meta"
}