![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F416ee1a6f71b8bde43f255dd40b04a08%2FScreenshot%202025-09-03%20at%2010.05.30.png?generation=1756868762432753&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fe1076e17f55f463e164e67389cd26c7e%2FScreenshot%202025-09-03%20at%2010.08.52.png?generation=1756868952435987&alt=media\")
Is it true that the private dataset contains left subjects, while the public dataset only has right subjects? And one important thing is that in the private dataset, the handedness feature for left subjects is NaN, or it could also be NaN for right subjects.
\nhttps://www.kaggle.com/code/quan0095/cmi-2025-demo-submission/notebook?scriptVersionId=255668869
Is it true that the private dataset contains left subjects, while the public dataset only has right subjects? And one important thing is that in the private dataset, the handedness feature for left subjects is NaN, or it could also be NaN for right subjects.
\nhttps://www.kaggle.com/code/quan0095/cmi-2025-demo-submission/notebook?scriptVersionId=255668869
The results of running this notebook show that there are only right subjects in the dataset. However, I know that there are NaN values in the demographics, though I'm not sure where exactly. So, if the results of this notebook are incorrect, it is highly likely that the handedness feature is NaN in the private dataset. I can confirm that the presence of NaN values does not affect the results of the Public leaderboard, meaning the public dataset only includes right subjects.
", "rawMarkdown": "The results of running this notebook show that there are only right subjects in the dataset. However, I know that there are NaN values in the demographics, though I'm not sure where exactly. So, if the results of this notebook are incorrect, it is highly likely that the handedness feature is NaN in the private dataset. I can confirm that the presence of NaN values does not affect the results of the Public leaderboard, meaning the public dataset only includes right subjects.", "votes": null }, { "id": "3280680", "postDate": "09/03/2025 03:04:04", "content": "I did not notice that demographic column includes NaN.
\nMy approach is highly dependent on handedness. It might explains gap between CV and LB. Thanks for sharing.
Anyway, maybe LB probing was must for this competition 😅
", "rawMarkdown": "I did not notice that demographic column includes NaN.\nMy approach is highly dependent on handedness. It might explains gap between CV and LB. Thanks for sharing.\n\nAnyway, maybe LB probing was must for this competition 😅", "votes": null }, { "id": "3280682", "postDate": "09/03/2025 03:09:48", "content": "
\n
I didn't know that the demographic had NaN values until I encountered a Submission Format Error. At that time, I was using demographic as a feature of the model, which led to the output being NaN. However, np.argmax with an input of a NaN vector will produce index 0, which resulted in the private dataset score being lower than expected. I only discovered this issue when using the code to find the max as shown above.
", "rawMarkdown": "I didn't know that the demographic had NaN values until I encountered a Submission Format Error. At that time, I was using demographic as a feature of the model, which led to the output being NaN. However, np.argmax with an input of a NaN vector will produce index 0, which resulted in the private dataset score being lower than expected. I only discovered this issue when using the code to find the max as shown above.", "votes": null }, { "id": "3280687", "postDate": "09/03/2025 03:15:14", "content": "Another factor contributing to the large gap between CV and the private LB is the difference in distribution between the public LB and the private LB. It appears that there are so few left subjects that the host has shifted all left subjects to the private LB. I trained the model based on the entire dataset. I believe that if I had known this information, training two models—one for right subjects and one for left subjects—and then using demographic data to decide which model to call would have achieved higher accuracy.
", "rawMarkdown": "Another factor contributing to the large gap between CV and the private LB is the difference in distribution between the public LB and the private LB. It appears that there are so few left subjects that the host has shifted all left subjects to the private LB. I trained the model based on the entire dataset. I believe that if I had known this information, training two models—one for right subjects and one for left subjects—and then using demographic data to decide which model to call would have achieved higher accuracy.", "votes": null }, { "id": "3280693", "postDate": "09/03/2025 03:25:14", "content": "This only happens with the private dataset. So if you only monitor the public score, you won't notice it. As you can see in the image above, before fixing the demographic NaN, my private score was quite low at 0.81.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F7c6efa4c5b71ad400f7f5ac052a5b645%2FScreenshot%202025-09-03%20at%2012.00.10.png?generation=1756875687207240&alt=media\")
Some people in private are wearing the device upside down, not in the public.
", "rawMarkdown": "Some people in private are wearing the device upside down, not in the public.", "votes": null }, { "id": "3280742", "postDate": "09/03/2025 05:12:37", "content": "I regret I didn't probed😂 Thanks for sharing.
", "rawMarkdown": "I regret I didn't probed😂 Thanks for sharing.", "votes": null }, { "id": "3280747", "postDate": "09/03/2025 05:20:42", "content": "@quan0095 0.810->0.837 is significant effect. I'll also try late sub. Thanks.
", "rawMarkdown": "quan0095 0.810->0.837 is significant effect. I'll also try late sub. Thanks.", "votes": null }, { "id": "3280748", "postDate": "09/03/2025 05:21:32", "content": "I actually did augment by flipping, but later gave up because it had no effect on the public leaderboard and CV. It's really hard to make decisions without prob information from the private dataset.
", "rawMarkdown": "I actually did augment by flipping, but later gave up because it had no effect on the public leaderboard and CV. It's really hard to make decisions without prob information from the private dataset.", "votes": null }, { "id": "3280751", "postDate": "09/03/2025 05:28:32", "content": "What is the fix? Nan = right or left?
", "rawMarkdown": "What is the fix? Nan = right or left?", "votes": null }, { "id": "3280752", "postDate": "09/03/2025 05:30:28", "content": "I removed demographic feature from input of my models.
", "rawMarkdown": "I removed demographic feature from input of my models.", "votes": null }, { "id": "3280759", "postDate": "09/03/2025 05:38:52", "content": "Do you mean like these 2 ['SUBJ_019262','SUBJ_045235']? Or some other ways of wearing it? @optimo
", "rawMarkdown": "Do you mean like these 2 ['SUBJ_019262','SUBJ_045235']? Or some other ways of wearing it? @optimo", "votes": null }, { "id": "3281074", "postDate": "09/03/2025 15:14:56", "content": "Excuse me, could you please tell me how you remove demographic features? And why did the leaderboard score not change after removing them?thx
", "rawMarkdown": "Excuse me, could you please tell me how you remove demographic features? And why did the leaderboard score not change after removing them?thx", "votes": null }, { "id": "3281302", "postDate": "09/04/2025 03:08:04", "content": "Initially, I added demographic features to the model's input. After discovering a bug, I removed the demographic features from the model's input. The public leaderboard was unaffected, possibly because the demographics of the subjects in the public dataset were normal. Initially, I also hoped that removing the demographic features would increase the leaderboard score, but there was no change except that I no longer encountered Submission Format Errors. However, the private score improved significantly.
", "rawMarkdown": "Initially, I added demographic features to the model's input. After discovering a bug, I removed the demographic features from the model's input. The public leaderboard was unaffected, possibly because the demographics of the subjects in the public dataset were normal. Initially, I also hoped that removing the demographic features would increase the leaderboard score, but there was no change except that I no longer encountered Submission Format Errors. However, the private score improved significantly.", "votes": null }, { "id": "3289492", "postDate": "09/16/2025 06:03:04", "content": "@minerppdy
\nHow did you find out that ['SUBJ_019262','SUBJ_045235'] were flipped?
https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/588051#3240789
", "rawMarkdown": "https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/588051#3240789", "votes": null }, { "id": "3290023", "postDate": "09/17/2025 01:47:44", "content": "1.The F1 scores of these two users are very low.
\n2.By comparing the scatter plots of acceleration for the same orientation and gesture, it can be observed that these two users are clearly outliers. @shanyun
The results of running this notebook show that there are only right subjects in the dataset. However, I know that there are NaN values in the demographics, though I'm not sure where exactly. So, if the results of this notebook are incorrect, it is highly likely that the handedness feature is NaN in the private dataset. I can confirm that the presence of NaN values does not affect the results of the Public leaderboard, meaning the public dataset only includes right subjects.
", "votes": null, "replies": [] }, { "id": 3280680, "author_name": "tatamikenn", "author_url": "", "post_date": "09/03/2025 03:04:04", "content": "I did not notice that demographic column includes NaN.
\nMy approach is highly dependent on handedness. It might explains gap between CV and LB. Thanks for sharing.
Anyway, maybe LB probing was must for this competition 😅
", "votes": null, "replies": [ { "id": 3280682, "author_name": "quan0095", "author_url": "", "post_date": "09/03/2025 03:09:48", "content": "
\n
I didn't know that the demographic had NaN values until I encountered a Submission Format Error. At that time, I was using demographic as a feature of the model, which led to the output being NaN. However, np.argmax with an input of a NaN vector will produce index 0, which resulted in the private dataset score being lower than expected. I only discovered this issue when using the code to find the max as shown above.
", "votes": null, "replies": [] }, { "id": 3280687, "author_name": "quan0095", "author_url": "", "post_date": "09/03/2025 03:15:14", "content": "Another factor contributing to the large gap between CV and the private LB is the difference in distribution between the public LB and the private LB. It appears that there are so few left subjects that the host has shifted all left subjects to the private LB. I trained the model based on the entire dataset. I believe that if I had known this information, training two models—one for right subjects and one for left subjects—and then using demographic data to decide which model to call would have achieved higher accuracy.
", "votes": null, "replies": [] }, { "id": 3280693, "author_name": "quan0095", "author_url": "", "post_date": "09/03/2025 03:25:14", "content": "This only happens with the private dataset. So if you only monitor the public score, you won't notice it. As you can see in the image above, before fixing the demographic NaN, my private score was quite low at 0.81.
@quan0095 0.810->0.837 is significant effect. I'll also try late sub. Thanks.
", "votes": null, "replies": [] }, { "id": 3280751, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 05:28:32", "content": "What is the fix? Nan = right or left?
", "votes": null, "replies": [ { "id": 3280752, "author_name": "quan0095", "author_url": "", "post_date": "09/03/2025 05:30:28", "content": "I removed demographic feature from input of my models.
", "votes": null, "replies": [ { "id": 3281074, "author_name": "yanzhou06", "author_url": "", "post_date": "09/03/2025 15:14:56", "content": "Excuse me, could you please tell me how you remove demographic features? And why did the leaderboard score not change after removing them?thx
", "votes": null, "replies": [ { "id": 3281302, "author_name": "quan0095", "author_url": "", "post_date": "09/04/2025 03:08:04", "content": "Initially, I added demographic features to the model's input. After discovering a bug, I removed the demographic features from the model's input. The public leaderboard was unaffected, possibly because the demographics of the subjects in the public dataset were normal. Initially, I also hoped that removing the demographic features would increase the leaderboard score, but there was no change except that I no longer encountered Submission Format Errors. However, the private score improved significantly.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3280737, "author_name": "optimo", "author_url": "", "post_date": "09/03/2025 05:05:05", "content": "Some people in private are wearing the device upside down, not in the public.
", "votes": null, "replies": [ { "id": 3280742, "author_name": "tatamikenn", "author_url": "", "post_date": "09/03/2025 05:12:37", "content": "I regret I didn't probed😂 Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 3280748, "author_name": "quan0095", "author_url": "", "post_date": "09/03/2025 05:21:32", "content": "I actually did augment by flipping, but later gave up because it had no effect on the public leaderboard and CV. It's really hard to make decisions without prob information from the private dataset.
", "votes": null, "replies": [] }, { "id": 3280759, "author_name": "minerppdy", "author_url": "", "post_date": "09/03/2025 05:38:52", "content": "Do you mean like these 2 ['SUBJ_019262','SUBJ_045235']? Or some other ways of wearing it? @optimo
", "votes": null, "replies": [ { "id": 3289492, "author_name": "shanyun", "author_url": "", "post_date": "09/16/2025 06:03:04", "content": "@minerppdy
\nHow did you find out that ['SUBJ_019262','SUBJ_045235'] were flipped?
https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/588051#3240789
", "votes": null, "replies": [] }, { "id": 3290023, "author_name": "minerppdy", "author_url": "", "post_date": "09/17/2025 01:47:44", "content": "1.The F1 scores of these two users are very low.
\n2.By comparing the scatter plots of acceleration for the same orientation and gesture, it can be observed that these two users are clearly outliers. @shanyun