{
  "id": 585472,
  "title": "Good performance during local training and testing and bad performance when evaluating using the server",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/585472",
  "author_name": "menglei",
  "post_date": "2025-06-20T16:10:01.385000",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Has anyone met the situation where you have very good result in local training and testing, but very bad performance as you submit for the server? Mine score dropped more than 20 percent when submitting</p>",
  "messages": [
    {
      "id": 3228883,
      "postDate": "2025-06-20T16:15:01.453Z",
      "content": "<p>I faced the same issue, but my score dropped by 8 percent. </p>",
      "rawMarkdown": "I faced the same issue, but my score dropped by 8 percent. ",
      "votes": 1,
      "replies": [
        {
          "id": 3228896,
          "postDate": "2025-06-20T16:40:00.937Z",
          "content": "<p>did you do any techniques to handle the potential missing values during the test?</p>",
          "rawMarkdown": "did you do any techniques to handle the potential missing values during the test?",
          "replies": [
            {
              "id": 3228901,
              "postDate": "2025-06-20T16:43:28.067Z",
              "rawMarkdown": "",
              "votes": 1,
              "isDeleted": true
            },
            {
              "id": 3228955,
              "postDate": "2025-06-20T18:22:03.180Z",
              "content": "<p>We did not do the cross validation, but we retrained the model several times, and the test result are similar. The samples in our dataset are sequences, which is a dataframe grouped by the sequence id, so I don't think it will lead to data leakage either.</p>",
              "rawMarkdown": "We did not do the cross validation, but we retrained the model several times, and the test result are similar. The samples in our dataset are sequences, which is a dataframe grouped by the sequence id, so I don't think it will lead to data leakage either."
            },
            {
              "id": 3228973,
              "postDate": "2025-06-20T18:59:37.267Z",
              "rawMarkdown": "",
              "isDeleted": true
            },
            {
              "id": 3228976,
              "postDate": "2025-06-20T19:11:03.147Z",
              "content": "<p>Thanks for those suggestions! We also just thought it might due to half of the data are only from imu, we did implement a mask mechanism in the model, but that was random mask, so we might need to change it a bit, also for the grouping by subject we will try that, thanks for those suggestions!</p>",
              "rawMarkdown": "Thanks for those suggestions! We also just thought it might due to half of the data are only from imu, we did implement a mask mechanism in the model, but that was random mask, so we might need to change it a bit, also for the grouping by subject we will try that, thanks for those suggestions!"
            }
          ]
        }
      ]
    },
    {
      "id": 3228881,
      "postDate": "2025-06-20T16:10:01.387Z",
      "content": "<p>Has anyone met the situation where you have very good result in local training and testing, but very bad performance as you submit for the server? Mine score dropped more than 20 percent when submitting</p>",
      "rawMarkdown": "Has anyone met the situation where you have very good result in local training and testing, but very bad performance as you submit for the server? Mine score dropped more than 20 percent when submitting\n",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 3228883,
      "author_name": "Abdul Rehman",
      "author_url": "",
      "post_date": "2025-06-20T16:15:01.453000",
      "content": "<p>I faced the same issue, but my score dropped by 8 percent. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 3228896,
          "author_name": "menglei",
          "author_url": "",
          "post_date": "2025-06-20T16:40:00.937000",
          "content": "<p>did you do any techniques to handle the potential missing values during the test?</p>",
          "votes": 0,
          "replies": [
            {
              "id": 3228901,
              "author_name": "",
              "author_url": "",
              "post_date": "2025-06-20T16:43:28.067000",
              "content": "",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3228955,
              "author_name": "menglei",
              "author_url": "",
              "post_date": "2025-06-20T18:22:03.180000",
              "content": "<p>We did not do the cross validation, but we retrained the model several times, and the test result are similar. The samples in our dataset are sequences, which is a dataframe grouped by the sequence id, so I don't think it will lead to data leakage either.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3228973,
              "author_name": "",
              "author_url": "",
              "post_date": "2025-06-20T18:59:37.267000",
              "content": "",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3228976,
              "author_name": "menglei",
              "author_url": "",
              "post_date": "2025-06-20T19:11:03.147000",
              "content": "<p>Thanks for those suggestions! We also just thought it might due to half of the data are only from imu, we did implement a mask mechanism in the model, but that was random mask, so we might need to change it a bit, also for the grouping by subject we will try that, thanks for those suggestions!</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3228883": "I faced the same issue, but my score dropped by 8 percent. ",
    "3228881": "Has anyone met the situation where you have very good result in local training and testing, but very bad performance as you submit for the server? Mine score dropped more than 20 percent when submitting\n"
  }
}