{
  "id": 586651,
  "title": "Cross Validation Setup",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/586651",
  "author_name": "",
  "post_date": "2025-06-27T16:15:42.793434900Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>My cv with stratification on subject is 0.79, yet my imu only model on the LB is only 0.69, I'm wondering if there is still data leakage opportunities even I am doing the stratifying on subject? Also, do you think there is a large gap between the distribution of the LB data and the train.csv?</p>",
  "messages": [
    {
      "id": "3234218",
      "postDate": "06/27/2025 16:15:42",
      "content": "<p>My cv with stratification on subject is 0.79, yet my imu only model on the LB is only 0.69, I'm wondering if there is still data leakage opportunities even I am doing the stratifying on subject? Also, do you think there is a large gap between the distribution of the LB data and the train.csv?</p>",
      "rawMarkdown": "My cv with stratification on subject is 0.79, yet my imu only model on the LB is only 0.69, I'm wondering if there is still data leakage opportunities even I am doing the stratifying on subject? Also, do you think there is a large gap between the distribution of the LB data and the train.csv?",
      "votes": null
    },
    {
      "id": "3235057",
      "postDate": "06/28/2025 17:00:21",
      "content": "<p>Public LB and CV correlate perfectly, so double-check your code, first of all cv split and metric calculation (asking code assistants also may help). Metric calculation on train set is a very popular issue.</p>",
      "rawMarkdown": "Public LB and CV correlate perfectly, so double-check your code, first of all cv split and metric calculation (asking code assistants also may help). Metric calculation on train set is a very popular issue.",
      "votes": null
    },
    {
      "id": "3235376",
      "postDate": "06/29/2025 06:09:10",
      "content": "<p>Thanks for the help! I have checked the cv pipeline and I think it should work fine. I'm wondering if you used any data imputation technique during the inference? I am currently filling all the nan values with 0, I'm thinking if that might be the reason.</p>",
      "rawMarkdown": "Thanks for the help! I have checked the cv pipeline and I think it should work fine. I'm wondering if you used any data imputation technique during the inference? I am currently filling all the nan values with 0, I'm thinking if that might be the reason.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3235057,
      "author_name": "ogurtsov",
      "author_url": "",
      "post_date": "06/28/2025 17:00:21",
      "content": "<p>Public LB and CV correlate perfectly, so double-check your code, first of all cv split and metric calculation (asking code assistants also may help). Metric calculation on train set is a very popular issue.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3235376,
          "author_name": "georgewang73",
          "author_url": "",
          "post_date": "06/29/2025 06:09:10",
          "content": "<p>Thanks for the help! I have checked the cv pipeline and I think it should work fine. I'm wondering if you used any data imputation technique during the inference? I am currently filling all the nan values with 0, I'm thinking if that might be the reason.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3234218": "My cv with stratification on subject is 0.79, yet my imu only model on the LB is only 0.69, I'm wondering if there is still data leakage opportunities even I am doing the stratifying on subject? Also, do you think there is a large gap between the distribution of the LB data and the train.csv?",
    "3235057": "Public LB and CV correlate perfectly, so double-check your code, first of all cv split and metric calculation (asking code assistants also may help). Metric calculation on train set is a very popular issue.",
    "3235376": "Thanks for the help! I have checked the cv pipeline and I think it should work fine. I'm wondering if you used any data imputation technique during the inference? I am currently filling all the nan values with 0, I'm thinking if that might be the reason."
  },
  "source": "meta"
}