{ "id": 590391, "title": "Seeking Advice on Performance Discrepancy When Combining IMU and TOF Data", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/590391", "author_name": "", "post_date": "2025-07-20T13:28:56.760414900Z", "votes": 4, "comment_count": 4, "views": 0, "content": "

I’m currently facing an issue where my model achieves around 0.80 using IMU-only features. However, when I incorporate TOF data, the CV score improves slightly to ~0.82, but the LB score drops significantly to 0.67.

\n

For context:

\n

I’ve used data augmentation to simulate cases where the TOF sensor is inactive.

\n

To further debug, I also tried setting all TOF data to zero during training. In this case, the CV score was ~0.79, and the LB score was similarly ~0.79. This makes me think the issue is not with my model’s inference logic, but perhaps with how TOF data interacts with the training/validation distribution.

\n

Has anyone encountered a similar situation or have insights into why this might happen? Any suggestions would be greatly appreciated!

\n

Thank you in advance for your help.🥺

", "messages": [ { "id": "3251327", "postDate": "07/20/2025 13:28:56", "content": "

I’m currently facing an issue where my model achieves around 0.80 using IMU-only features. However, when I incorporate TOF data, the CV score improves slightly to ~0.82, but the LB score drops significantly to 0.67.

\n

For context:

\n

I’ve used data augmentation to simulate cases where the TOF sensor is inactive.

\n

To further debug, I also tried setting all TOF data to zero during training. In this case, the CV score was ~0.79, and the LB score was similarly ~0.79. This makes me think the issue is not with my model’s inference logic, but perhaps with how TOF data interacts with the training/validation distribution.

\n

Has anyone encountered a similar situation or have insights into why this might happen? Any suggestions would be greatly appreciated!

\n

Thank you in advance for your help.🥺

", "rawMarkdown": "I’m currently facing an issue where my model achieves around 0.80 using IMU-only features. However, when I incorporate TOF data, the CV score improves slightly to ~0.82, but the LB score drops significantly to 0.67.\n\nFor context:\n\nI’ve used data augmentation to simulate cases where the TOF sensor is inactive.\n\nTo further debug, I also tried setting all TOF data to zero during training. In this case, the CV score was ~0.79, and the LB score was similarly ~0.79. This makes me think the issue is not with my model’s inference logic, but perhaps with how TOF data interacts with the training/validation distribution.\n\nHas anyone encountered a similar situation or have insights into why this might happen? Any suggestions would be greatly appreciated!\n\nThank you in advance for your help.🥺", "votes": null }, { "id": "3258117", "postDate": "07/30/2025 06:09:58", "content": "

I have the same problem

", "rawMarkdown": "I have the same problem", "votes": null }, { "id": "3261631", "postDate": "08/01/2025 20:43:30", "content": "

Drop from LB 0.8 -> 0.67 most probably shows you have a bug in your inference code. I think TOF FEs in training and inference time are different…you can give your training and inference code to an LLM like Gemini on ai studio and ask it to compare the TOF FE parts.

", "rawMarkdown": "Drop from LB 0.8 -> 0.67 most probably shows you have a bug in your inference code. I think TOF FEs in training and inference time are different...you can give your training and inference code to an LLM like Gemini on ai studio and ask it to compare the TOF FE parts.", "votes": null }, { "id": "3261736", "postDate": "08/02/2025 06:12:53", "content": "

Thank you. I eventually found that the fill value for the missing TOF data during inference conflicted with the one used during training. Thanks again for your help🌹.

", "rawMarkdown": "Thank you. I eventually found that the fill value for the missing TOF data during inference conflicted with the one used during training. Thanks again for your help🌹.", "votes": null }, { "id": "3261739", "postDate": "08/02/2025 06:22:52", "content": "

在数据增强时，我在模拟传感器失活的情况下，将 TOF 数据全部设置为 0。但这样会造成一个问题：在原始数据中本身就存在真实的 0 值，这会让模型无法区分 0 是缺失值还是实际测量值。我怀疑正是因为这种歧义，导致在 IMU-only 正常的情况下，加入 TOF 数据反而使模型性能下降。

\n

为了解决这个问题，我首先将数据中的 -1 转换为 300（这是一个超出实际最大测量范围的值），然后在缺失值和模拟传感器失活的情况下，将 TOF 值统一设为 -1。这样，在训练和推理时，模型就能够正确区分缺失值和真实测量值，从而恢复了正常的分类性能。希望能帮助到你🫡

", "rawMarkdown": "在数据增强时，我在模拟传感器失活的情况下，将 TOF 数据全部设置为 0。但这样会造成一个问题：在原始数据中本身就存在真实的 0 值，这会让模型无法区分 0 是缺失值还是实际测量值。我怀疑正是因为这种歧义，导致在 IMU-only 正常的情况下，加入 TOF 数据反而使模型性能下降。\n\n为了解决这个问题，我首先将数据中的 -1 转换为 300（这是一个超出实际最大测量范围的值），然后在缺失值和模拟传感器失活的情况下，将 TOF 值统一设为 -1。这样，在训练和推理时，模型就能够正确区分缺失值和真实测量值，从而恢复了正常的分类性能。希望能帮助到你🫡", "votes": null } ], "comments": [ { "id": 3258117, "author_name": "xukongji", "author_url": "", "post_date": "07/30/2025 06:09:58", "content": "

I have the same problem

", "votes": null, "replies": [ { "id": 3261739, "author_name": "tudouking", "author_url": "", "post_date": "08/02/2025 06:22:52", "content": "

在数据增强时，我在模拟传感器失活的情况下，将 TOF 数据全部设置为 0。但这样会造成一个问题：在原始数据中本身就存在真实的 0 值，这会让模型无法区分 0 是缺失值还是实际测量值。我怀疑正是因为这种歧义，导致在 IMU-only 正常的情况下，加入 TOF 数据反而使模型性能下降。

\n

为了解决这个问题，我首先将数据中的 -1 转换为 300（这是一个超出实际最大测量范围的值），然后在缺失值和模拟传感器失活的情况下，将 TOF 值统一设为 -1。这样，在训练和推理时，模型就能够正确区分缺失值和真实测量值，从而恢复了正常的分类性能。希望能帮助到你🫡

", "votes": null, "replies": [] } ] }, { "id": 3261631, "author_name": "kainsama", "author_url": "", "post_date": "08/01/2025 20:43:30", "content": "

Drop from LB 0.8 -> 0.67 most probably shows you have a bug in your inference code. I think TOF FEs in training and inference time are different…you can give your training and inference code to an LLM like Gemini on ai studio and ask it to compare the TOF FE parts.

", "votes": null, "replies": [ { "id": 3261736, "author_name": "tudouking", "author_url": "", "post_date": "08/02/2025 06:12:53", "content": "

Thank you. I eventually found that the fill value for the missing TOF data during inference conflicted with the one used during training. Thanks again for your help🌹.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3251327": "I’m currently facing an issue where my model achieves around 0.80 using IMU-only features. However, when I incorporate TOF data, the CV score improves slightly to ~0.82, but the LB score drops significantly to 0.67.\n\nFor context:\n\nI’ve used data augmentation to simulate cases where the TOF sensor is inactive.\n\nTo further debug, I also tried setting all TOF data to zero during training. In this case, the CV score was ~0.79, and the LB score was similarly ~0.79. This makes me think the issue is not with my model’s inference logic, but perhaps with how TOF data interacts with the training/validation distribution.\n\nHas anyone encountered a similar situation or have insights into why this might happen? Any suggestions would be greatly appreciated!\n\nThank you in advance for your help.🥺", "3258117": "I have the same problem", "3261631": "Drop from LB 0.8 -> 0.67 most probably shows you have a bug in your inference code. I think TOF FEs in training and inference time are different...you can give your training and inference code to an LLM like Gemini on ai studio and ask it to compare the TOF FE parts.", "3261736": "Thank you. I eventually found that the fill value for the missing TOF data during inference conflicted with the one used during training. Thanks again for your help🌹.", "3261739": "在数据增强时，我在模拟传感器失活的情况下，将 TOF 数据全部设置为 0。但这样会造成一个问题：在原始数据中本身就存在真实的 0 值，这会让模型无法区分 0 是缺失值还是实际测量值。我怀疑正是因为这种歧义，导致在 IMU-only 正常的情况下，加入 TOF 数据反而使模型性能下降。\n\n为了解决这个问题，我首先将数据中的 -1 转换为 300（这是一个超出实际最大测量范围的值），然后在缺失值和模拟传感器失活的情况下，将 TOF 值统一设为 -1。这样，在训练和推理时，模型就能够正确区分缺失值和真实测量值，从而恢复了正常的分类性能。希望能帮助到你🫡" }, "source": "meta" }