{
  "id": 596266,
  "title": "Local CV 0.8+, but only got 0.73 lb score ",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/596266",
  "author_name": "",
  "post_date": "2025-08-02T16:04:34.776305600Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I followed the <a href=\"https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251203240\" target=\"_blank\">notebook </a> shared by <a href=\"https://www.kaggle.com/gauravparkhedkar\" target=\"_blank\">@gauravparkhedkar</a>, with training code in <a href=\"https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251713729\" target=\"_blank\">version 27</a>, tried to train my own model. I got incredibly good local cv, but the lb score is only 0.73. The learning curve is as the folllowing. Why does that happen? How do I fix this issue?  </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F534882%2F79ee7d5a26ccb9a58833378e41e2a8c9%2F2025-08-03%20000415.png?generation=1754150663476953&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "3261955",
      "postDate": "08/02/2025 16:04:34",
      "content": "<p>I followed the <a href=\"https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251203240\" target=\"_blank\">notebook </a> shared by <a href=\"https://www.kaggle.com/gauravparkhedkar\" target=\"_blank\">@gauravparkhedkar</a>, with training code in <a href=\"https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251713729\" target=\"_blank\">version 27</a>, tried to train my own model. I got incredibly good local cv, but the lb score is only 0.73. The learning curve is as the folllowing. Why does that happen? How do I fix this issue?  </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F534882%2F79ee7d5a26ccb9a58833378e41e2a8c9%2F2025-08-03%20000415.png?generation=1754150663476953&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "I followed the [notebook ](https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251203240) shared by @gauravparkhedkar, with training code in [version 27](https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251713729), tried to train my own model. I got incredibly good local cv, but the lb score is only 0.73. The learning curve is as the folllowing. Why does that happen? How do I fix this issue?  \n\n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F534882%2F79ee7d5a26ccb9a58833378e41e2a8c9%2F2025-08-03%20000415.png?generation=1754150663476953&alt=media)",
      "votes": null
    },
    {
      "id": "3262118",
      "postDate": "08/03/2025 00:14:27",
      "content": "<p>How are you computing your CV scores?<br>\nI noticed that the notebook you mentioned is using full sensor data (IMU + thermal + ToF) to compute cross-validation scores. If thats the approach you're using, it's likely the main vause of the discrepancy, since the test set only has 50% IMU data your CV should simulate the test conditions to get realistic performance estimates.</p>",
      "rawMarkdown": "How are you computing your CV scores?\nI noticed that the notebook you mentioned is using full sensor data (IMU + thermal + ToF) to compute cross-validation scores. If thats the approach you're using, it's likely the main vause of the discrepancy, since the test set only has 50% IMU data your CV should simulate the test conditions to get realistic performance estimates.",
      "votes": null
    },
    {
      "id": "3262181",
      "postDate": "08/03/2025 06:11:41",
      "content": "<p>Yes, I used IMU+THM+ToF for both train and validation phase. Thank you for pointing that out. Do you suggest simulating the 50% IMU-only data during training phase besides validation?</p>",
      "rawMarkdown": "Yes, I used IMU+THM+ToF for both train and validation phase. Thank you for pointing that out. Do you suggest simulating the 50% IMU-only data during training phase besides validation?",
      "votes": null
    },
    {
      "id": "3262228",
      "postDate": "08/03/2025 07:52:23",
      "content": "<p>You should definitely try that.<br>\nI have experimented with simulating the IMU only scenario using different masking probabilities (15%, 25%, 35%, and 50%). Currently, I'm using gesture specific Masking prob applying a different Masking prob for each gesture which is giving me the best lb so far.<br>\nOf course, results may vary depending on your architecture, so it's worth testing a few options.</p>",
      "rawMarkdown": "You should definitely try that.\nI have experimented with simulating the IMU only scenario using different masking probabilities (15%, 25%, 35%, and 50%). Currently, I'm using gesture specific Masking prob applying a different Masking prob for each gesture which is giving me the best lb so far.\n\nOf course, results may vary depending on your architecture, so it's worth testing a few options.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3262118,
      "author_name": "abdulhanan677",
      "author_url": "",
      "post_date": "08/03/2025 00:14:27",
      "content": "<p>How are you computing your CV scores?<br>\nI noticed that the notebook you mentioned is using full sensor data (IMU + thermal + ToF) to compute cross-validation scores. If thats the approach you're using, it's likely the main vause of the discrepancy, since the test set only has 50% IMU data your CV should simulate the test conditions to get realistic performance estimates.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3262181,
          "author_name": "shanyun",
          "author_url": "",
          "post_date": "08/03/2025 06:11:41",
          "content": "<p>Yes, I used IMU+THM+ToF for both train and validation phase. Thank you for pointing that out. Do you suggest simulating the 50% IMU-only data during training phase besides validation?</p>",
          "votes": null,
          "replies": [
            {
              "id": 3262228,
              "author_name": "abdulhanan677",
              "author_url": "",
              "post_date": "08/03/2025 07:52:23",
              "content": "<p>You should definitely try that.<br>\nI have experimented with simulating the IMU only scenario using different masking probabilities (15%, 25%, 35%, and 50%). Currently, I'm using gesture specific Masking prob applying a different Masking prob for each gesture which is giving me the best lb so far.<br>\nOf course, results may vary depending on your architecture, so it's worth testing a few options.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3261955": "I followed the [notebook ](https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251203240) shared by @gauravparkhedkar, with training code in [version 27](https://www.kaggle.com/code/gauravparkhedkar/sensordata?scriptVersionId=251713729), tried to train my own model. I got incredibly good local cv, but the lb score is only 0.73. The learning curve is as the folllowing. Why does that happen? How do I fix this issue?  \n\n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F534882%2F79ee7d5a26ccb9a58833378e41e2a8c9%2F2025-08-03%20000415.png?generation=1754150663476953&alt=media)",
    "3262118": "How are you computing your CV scores?\nI noticed that the notebook you mentioned is using full sensor data (IMU + thermal + ToF) to compute cross-validation scores. If thats the approach you're using, it's likely the main vause of the discrepancy, since the test set only has 50% IMU data your CV should simulate the test conditions to get realistic performance estimates.",
    "3262181": "Yes, I used IMU+THM+ToF for both train and validation phase. Thank you for pointing that out. Do you suggest simulating the 50% IMU-only data during training phase besides validation?",
    "3262228": "You should definitely try that.\nI have experimented with simulating the IMU only scenario using different masking probabilities (15%, 25%, 35%, and 50%). Currently, I'm using gesture specific Masking prob applying a different Masking prob for each gesture which is giving me the best lb so far.\n\nOf course, results may vary depending on your architecture, so it's worth testing a few options."
  },
  "source": "meta"
}